2023-12-19T07:30:47.873 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T07:30:47.874 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --archive /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886 --owner scheduled_rishabh@teuthology --verbose -- /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/orig.config.yaml 2023-12-19T07:30:47.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T07:30:48.010 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886 branch: rishabh-mdscap-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: null first_in_suite: false job_id: '7495886' kernel: flavor: default kdb: true sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: rishabh-mdscap-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-mdscap-quincy sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4463 sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 sleep_before_teardown: 0 subset: 344/512 suite: fs suite_branch: rishabh-mdscap-quincy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_05:13:44 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-19T07:30:48.010 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa; will attempt to use it 2023-12-19T07:30:48.011 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa/tasks 2023-12-19T07:30:48.011 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T07:30:48.012 INFO:teuthology.task.internal:Checking packages... 2023-12-19T07:30:48.041 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c7312db04d305d59a3441862c84d6d4da3817eb0' 2023-12-19T07:30:48.041 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T07:30:48.041 INFO:teuthology.packaging:ref: None 2023-12-19T07:30:48.041 INFO:teuthology.packaging:tag: None 2023-12-19T07:30:48.041 INFO:teuthology.packaging:branch: rishabh-mdscap-quincy 2023-12-19T07:30:48.041 INFO:teuthology.packaging:sha1: c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T07:30:48.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=rishabh-mdscap-quincy 2023-12-19T07:30:48.187 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-141-gc7312db0-1focal 2023-12-19T07:30:48.189 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T07:30:48.196 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T07:30:48.196 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T07:30:48.217 INFO:teuthology.task.internal:Saving configuration 2023-12-19T07:30:48.238 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T07:30:48.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T07:30:48.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 07:24:25.336278', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-19T07:30:48.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 07:24:25.337776', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-19T07:30:48.293 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-19T07:30:48.300 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-12-19T07:30:48.300 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-12-19T07:30:48.300 INFO:teuthology.run_tasks:Running task console_log... 2023-12-19T07:30:48.374 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff088779790>, signals=[15]) 2023-12-19T07:30:48.375 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-19T07:30:48.380 INFO:teuthology.task.internal:Opening connections... 2023-12-19T07:30:48.381 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:30:48.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:30:48.433 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:30:48.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:30:48.482 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-19T07:30:48.489 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-12-19T07:30:48.509 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-12-19T07:30:48.510 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-12-19T07:30:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Ubuntu" 2023-12-19T07:30:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-19T07:30:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:ID=ubuntu 2023-12-19T07:30:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE=debian 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="20.04" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-19T07:30:48.517 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_CODENAME=focal 2023-12-19T07:30:48.518 INFO:teuthology.orchestra.run.smithi111.stdout:UBUNTU_CODENAME=focal 2023-12-19T07:30:48.518 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-12-19T07:30:48.540 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2023-12-19T07:30:48.566 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2023-12-19T07:30:48.566 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="20.04" 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-19T07:30:48.615 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-19T07:30:48.616 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-19T07:30:48.616 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-19T07:30:48.616 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=focal 2023-12-19T07:30:48.616 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=focal 2023-12-19T07:30:48.616 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2023-12-19T07:30:48.640 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-19T07:30:48.652 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-19T07:30:48.657 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-19T07:30:48.658 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-12-19T07:30:48.660 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2023-12-19T07:30:48.665 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-19T07:30:48.672 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-19T07:30:48.672 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-12-19T07:30:48.674 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2023-12-19T07:30:48.697 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-19T07:30:48.752 INFO:teuthology.run_tasks:Running task kernel... 2023-12-19T07:30:48.766 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'} 2023-12-19T07:30:48.767 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}}, timeout 300 2023-12-19T07:30:48.767 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T07:30:48.767 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T07:30:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:30:48.773 DEBUG:teuthology.task.kernel:sha1 for mon.a is cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.773 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "cd68252d7b4d7802f3cae7f5945b38e9b20dad4c"... 2023-12-19T07:30:48.773 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-12-19T07:30:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:30:48.774 DEBUG:teuthology.task.kernel:sha1 for mon.b is cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.774 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "cd68252d7b4d7802f3cae7f5945b38e9b20dad4c"... 2023-12-19T07:30:48.774 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2023-12-19T07:30:48.780 INFO:teuthology.orchestra.run.smithi111.stdout:5.4.0-124-generic 2023-12-19T07:30:48.781 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-124-generic vs cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.781 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-12-19T07:30:48.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.785 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-124-generic 2023-12-19T07:30:48.785 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-124-generic vs cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.785 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-12-19T07:30:48.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:30:48.921 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2023-12-19T07:30:48.921 DEBUG:teuthology.orchestra.run.smithi111:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-12-19T07:30:48.933 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2023-12-19T07:30:48.933 DEBUG:teuthology.orchestra.run.smithi143:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-12-19T07:30:48.950 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 0: cd: /lib/firmware/updates: No such file or directory 2023-12-19T07:30:48.954 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 0: cd: /lib/firmware/updates: No such file or directory 2023-12-19T07:30:49.063 INFO:teuthology.orchestra.run.smithi111.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-12-19T07:30:49.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-12-19T07:30:49.066 INFO:teuthology.orchestra.run.smithi143.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-12-19T07:30:49.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-12-19T07:30:49.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-12-19T07:30:49.113 DEBUG:teuthology.orchestra.run.smithi143:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-12-19T07:30:49.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-12-19T07:30:49.150 DEBUG:teuthology.orchestra.run.smithi111:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-12-19T07:31:35.952 INFO:teuthology.orchestra.run.smithi111.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-12-19T07:31:35.952 INFO:teuthology.orchestra.run.smithi111.stderr: * [new branch] main -> origin/main 2023-12-19T07:31:36.004 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190312 -> 20190312 2023-12-19T07:31:36.007 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190416 -> 20190416 2023-12-19T07:31:36.007 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190514 -> 20190514 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190618 -> 20190618 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190717 -> 20190717 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190815 -> 20190815 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20190923 -> 20190923 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20191022 -> 20191022 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20191215 -> 20191215 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200122 -> 20200122 2023-12-19T07:31:36.008 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200316 -> 20200316 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200421 -> 20200421 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200519 -> 20200519 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200619 -> 20200619 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200721 -> 20200721 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200817 -> 20200817 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20200918 -> 20200918 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20201022 -> 20201022 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20201118 -> 20201118 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20201218 -> 20201218 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210208 -> 20210208 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210315 -> 20210315 2023-12-19T07:31:36.012 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210511 -> 20210511 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210716 -> 20210716 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210818 -> 20210818 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20210919 -> 20210919 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20211027 -> 20211027 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20211216 -> 20211216 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220209 -> 20220209 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220310 -> 20220310 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220411 -> 20220411 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220509 -> 20220509 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220610 -> 20220610 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220708 -> 20220708 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220815 -> 20220815 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20220913 -> 20220913 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20221012 -> 20221012 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20221109 -> 20221109 2023-12-19T07:31:36.013 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20221214 -> 20221214 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230117 -> 20230117 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230210 -> 20230210 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230310 -> 20230310 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230404 -> 20230404 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230515 -> 20230515 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230625 -> 20230625 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230804 -> 20230804 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20230919 -> 20230919 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20231030 -> 20231030 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20231111 -> 20231111 2023-12-19T07:31:36.014 INFO:teuthology.orchestra.run.smithi111.stderr: * [new tag] 20231211 -> 20231211 2023-12-19T07:31:37.295 INFO:teuthology.orchestra.run.smithi143.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-12-19T07:31:37.296 INFO:teuthology.orchestra.run.smithi143.stderr: * [new branch] main -> origin/main 2023-12-19T07:31:37.355 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190312 -> 20190312 2023-12-19T07:31:37.355 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190416 -> 20190416 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190514 -> 20190514 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190618 -> 20190618 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190717 -> 20190717 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190815 -> 20190815 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20190923 -> 20190923 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20191022 -> 20191022 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20191215 -> 20191215 2023-12-19T07:31:37.356 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200122 -> 20200122 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200316 -> 20200316 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200421 -> 20200421 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200519 -> 20200519 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200619 -> 20200619 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200721 -> 20200721 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200817 -> 20200817 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20200918 -> 20200918 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20201022 -> 20201022 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20201118 -> 20201118 2023-12-19T07:31:37.358 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20201218 -> 20201218 2023-12-19T07:31:37.359 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210208 -> 20210208 2023-12-19T07:31:37.359 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210315 -> 20210315 2023-12-19T07:31:37.359 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210511 -> 20210511 2023-12-19T07:31:37.359 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210716 -> 20210716 2023-12-19T07:31:37.359 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210818 -> 20210818 2023-12-19T07:31:37.361 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20210919 -> 20210919 2023-12-19T07:31:37.361 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20211027 -> 20211027 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20211216 -> 20211216 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220209 -> 20220209 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220310 -> 20220310 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220411 -> 20220411 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220509 -> 20220509 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220610 -> 20220610 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220708 -> 20220708 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220815 -> 20220815 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20220913 -> 20220913 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20221012 -> 20221012 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20221109 -> 20221109 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20221214 -> 20221214 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230117 -> 20230117 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230210 -> 20230210 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230310 -> 20230310 2023-12-19T07:31:37.362 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230404 -> 20230404 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230515 -> 20230515 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230625 -> 20230625 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230804 -> 20230804 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20230919 -> 20230919 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20231030 -> 20231030 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20231111 -> 20231111 2023-12-19T07:31:37.363 INFO:teuthology.orchestra.run.smithi143.stderr: * [new tag] 20231211 -> 20231211 2023-12-19T07:31:39.952 INFO:teuthology.orchestra.run.smithi111.stderr:Updating files: 51% (1688/3256) Updating files: 52% (1694/3256) Updating files: 53% (1726/3256) Updating files: 54% (1759/3256) Updating files: 55% (1791/3256) Updating files: 56% (1824/3256) Updating files: 57% (1856/3256) Updating files: 58% (1889/3256) Updating files: 59% (1922/3256) Updating files: 60% (1954/3256) Updating files: 61% (1987/3256) Updating files: 62% (2019/3256) Updating files: 63% (2052/3256) Updating files: 64% (2084/3256) Updating files: 65% (2117/3256) Updating files: 65% (2130/3256) Updating files: 66% (2149/3256) Updating files: 67% (2182/3256) Updating files: 68% (2215/3256) Updating files: 69% (2247/3256) Updating files: 70% (2280/3256) Updating files: 71% (2312/3256) Updating files: 72% (2345/3256) Updating files: 73% (2377/3256) Updating files: 74% (2410/3256) Updating files: 75% (2442/3256) Updating files: 76% (2475/3256) Updating files: 77% (2508/3256) Updating files: 78% (2540/3256) Updating files: 79% (2573/3256) Updating files: 80% (2605/3256) Updating files: 81% (2638/3256) Updating files: 81% (2659/3256) Updating files: 82% (2670/3256) Updating files: 83% (2703/3256) Updating files: 84% (2736/3256) Updating files: 85% (2768/3256) Updating files: 86% (2801/3256) Updating files: 87% (2833/3256) Updating files: 88% (2866/3256) Updating files: 89% (2898/3256) Updating files: 90% (2931/3256) Updating files: 91% (2963/3256) Updating files: 92% (2996/3256) Updating files: 93% (3029/3256) Updating files: 94% (3061/3256) Updating files: 95% (3094/3256) Updating files: 96% (3126/3256) Updating files: 97% (3159/3256) Updating files: 98% (3191/3256) Updating files: 99% (3224/3256) Updating files: 100% (3256/3256) Updating files: 100% (3256/3256), done. 2023-12-19T07:31:39.957 INFO:teuthology.orchestra.run.smithi111.stdout:HEAD is now at c156e6be Merge branch 'robot/pr-0-1702625169' into 'main' 2023-12-19T07:31:39.974 INFO:teuthology.task.kernel:Downloading kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.a... 2023-12-19T07:31:39.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:31:40.124 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ 2023-12-19T07:31:40.125 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ --input-file=- 2023-12-19T07:31:40.128 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi111.front.sepia.ceph.com to complete... 2023-12-19T07:31:41.321 INFO:teuthology.orchestra.run.smithi143.stderr:Updating files: 51% (1688/3256) Updating files: 52% (1694/3256) Updating files: 53% (1726/3256) Updating files: 54% (1759/3256) Updating files: 55% (1791/3256) Updating files: 56% (1824/3256) Updating files: 57% (1856/3256) Updating files: 58% (1889/3256) Updating files: 59% (1922/3256) Updating files: 60% (1954/3256) Updating files: 61% (1987/3256) Updating files: 62% (2019/3256) Updating files: 63% (2052/3256) Updating files: 64% (2084/3256) Updating files: 65% (2117/3256) Updating files: 65% (2130/3256) Updating files: 66% (2149/3256) Updating files: 67% (2182/3256) Updating files: 68% (2215/3256) Updating files: 69% (2247/3256) Updating files: 70% (2280/3256) Updating files: 71% (2312/3256) Updating files: 72% (2345/3256) Updating files: 73% (2377/3256) Updating files: 74% (2410/3256) Updating files: 75% (2442/3256) Updating files: 76% (2475/3256) Updating files: 77% (2508/3256) Updating files: 78% (2540/3256) Updating files: 79% (2573/3256) Updating files: 80% (2605/3256) Updating files: 81% (2638/3256) Updating files: 81% (2659/3256) Updating files: 82% (2670/3256) Updating files: 83% (2703/3256) Updating files: 84% (2736/3256) Updating files: 85% (2768/3256) Updating files: 86% (2801/3256) Updating files: 87% (2833/3256) Updating files: 88% (2866/3256) Updating files: 89% (2898/3256) Updating files: 90% (2931/3256) Updating files: 91% (2963/3256) Updating files: 92% (2996/3256) Updating files: 93% (3029/3256) Updating files: 94% (3061/3256) Updating files: 95% (3094/3256) Updating files: 96% (3126/3256) Updating files: 97% (3159/3256) Updating files: 98% (3191/3256) Updating files: 99% (3224/3256) Updating files: 100% (3256/3256) Updating files: 100% (3256/3256), done. 2023-12-19T07:31:41.325 INFO:teuthology.orchestra.run.smithi143.stdout:HEAD is now at c156e6be Merge branch 'robot/pr-0-1702625169' into 'main' 2023-12-19T07:31:41.342 INFO:teuthology.task.kernel:Downloading kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.b... 2023-12-19T07:31:41.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cd68252d7b4d7802f3cae7f5945b38e9b20dad4c 2023-12-19T07:31:41.562 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ 2023-12-19T07:31:41.562 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/ --input-file=- 2023-12-19T07:31:41.565 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi143.front.sepia.ceph.com to complete... 2023-12-19T07:31:41.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19 07:31:41 URL:https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb [38928782/38928782] -> "/tmp/linux-image.deb" [1] 2023-12-19T07:31:41.646 INFO:teuthology.orchestra.run.smithi111.stderr:FINISHED --2023-12-19 07:31:41-- 2023-12-19T07:31:41.646 INFO:teuthology.orchestra.run.smithi111.stderr:Total wall clock time: 1.5s 2023-12-19T07:31:41.646 INFO:teuthology.orchestra.run.smithi111.stderr:Downloaded: 1 files, 37M in 1.4s (27.2 MB/s) 2023-12-19T07:31:41.649 INFO:teuthology.task.kernel:Installing kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.a... 2023-12-19T07:31:41.649 DEBUG:teuthology.orchestra.run.smithi111:> sudo dpkg -i /tmp/linux-image.deb 2023-12-19T07:31:41.958 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-image-6.6.0-gcd68252d7b4d. 2023-12-19T07:31:43.285 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19 07:31:43 URL:https://3.chacra.ceph.com/r/kernel/testing/cd68252d7b4d7802f3cae7f5945b38e9b20dad4c/ubuntu/focal/flavors/default/pool/main/l/linux-6.6.0-gcd68252d7b4d/linux-image-6.6.0-gcd68252d7b4d_6.6.0-gcd68252d7b4d-1_amd64.deb [38928782/38928782] -> "/tmp/linux-image.deb" [1] 2023-12-19T07:31:43.285 INFO:teuthology.orchestra.run.smithi143.stderr:FINISHED --2023-12-19 07:31:43-- 2023-12-19T07:31:43.285 INFO:teuthology.orchestra.run.smithi143.stderr:Total wall clock time: 1.7s 2023-12-19T07:31:43.285 INFO:teuthology.orchestra.run.smithi143.stderr:Downloaded: 1 files, 37M in 1.5s (25.1 MB/s) 2023-12-19T07:31:43.289 INFO:teuthology.task.kernel:Installing kernel cd68252d7b4d7802f3cae7f5945b38e9b20dad4c on mon.b... 2023-12-19T07:31:43.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo dpkg -i /tmp/linux-image.deb 2023-12-19T07:31:43.708 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-image-6.6.0-gcd68252d7b4d. 2023-12-19T07:31:43.730 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... 104373 files and directories currently installed.) 2023-12-19T07:31:43.735 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-12-19T07:31:43.927 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T07:31:45.421 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... 104373 files and directories currently installed.) 2023-12-19T07:31:45.426 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-12-19T07:31:45.570 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T07:31:46.960 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T07:31:47.079 INFO:teuthology.orchestra.run.smithi111.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:31:48.556 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-6.6.0-gcd68252d7b4d (6.6.0-gcd68252d7b4d-1) ... 2023-12-19T07:31:48.673 INFO:teuthology.orchestra.run.smithi143.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:32:13.409 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub' 2023-12-19T07:32:13.418 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T07:32:13.439 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2023-12-19T07:32:13.779 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:13.819 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:32:14.217 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T07:32:14.228 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T07:32:14.903 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub' 2023-12-19T07:32:14.911 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T07:32:14.933 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2023-12-19T07:32:15.302 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:15.320 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:32:15.686 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T07:32:15.699 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T07:32:16.896 INFO:teuthology.orchestra.run.smithi111.stderr:done 2023-12-19T07:32:17.103 DEBUG:teuthology.orchestra.run.smithi111:> dpkg-deb -c /tmp/linux-image.deb 2023-12-19T07:32:17.155 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./ 2023-12-19T07:32:17.156 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./boot/ 2023-12-19T07:32:17.156 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 4415617 2023-12-08 17:50 ./boot/System.map-6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.172 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 169214 2023-12-08 17:50 ./boot/config-6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.172 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11352864 2023-12-08 17:50 ./boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postinst.d/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postrm.d/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/preinst.d/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/prerm.d/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/ 2023-12-19T07:32:17.194 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ 2023-12-19T07:32:17.195 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 490233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/aesni-intel.ko 2023-12-19T07:32:17.197 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-12-19T07:32:17.197 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-12-19T07:32:17.197 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 93113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 106729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 152241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 83889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-12-19T07:32:17.198 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/ 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/ 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/ 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpuid.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/msr.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/ 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 354433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-amd.ko 2023-12-19T07:32:17.199 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 881625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-intel.ko 2023-12-19T07:32:17.200 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 2275865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/af_alg.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_hash.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_skcipher.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ansi_cprng.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/anubis.ko 2023-12-19T07:32:17.217 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/arc4.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/ 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/ 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_memcpy.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_pq.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_tx.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_xor.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/raid6test.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authenc.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authencesn.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blake2b_generic.ko 2023-12-19T07:32:17.218 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_common.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_generic.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/camellia_generic.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast5_generic.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast6_generic.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast_common.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ccm.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cmac.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crc32_generic.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cryptd.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_engine.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_simd.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_user.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cts.ko 2023-12-19T07:32:17.219 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/des_generic.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/dh_generic.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/echainiv.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/essiv.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/fcrypt.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/khazad.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lrw.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4hc.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/md4.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/michael_mic.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcbc.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcrypt.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/rmd160.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/seed.ko 2023-12-19T07:32:17.220 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/serpent_generic.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 149193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tcrypt.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tea.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_common.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_generic.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/vmac.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/wp512.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xcbc.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xor.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xts.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xxhash_generic.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_extlog.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_ipmi.ko 2023-12-19T07:32:17.221 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_pad.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/ 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/einj.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ec_sys.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/platform_profile.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbs.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbshc.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 103025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/video.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/acard-ahci.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 88865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci_platform.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 88289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci_platform.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_acpi.ko 2023-12-19T07:32:17.222 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ali.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_amd.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_artop.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atiixp.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atp867x.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd640.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd64x.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cypress.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_efar.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt366.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt37x.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x3.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it8213.ko 2023-12-19T07:32:17.223 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it821x.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_jmicron.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_legacy.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_marvell.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_mpiix.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_netcell.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ninja32.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87410.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87415.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_oldpiix.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_opti.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_optidma.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pcmcia.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc2027x.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_piccolo.ko 2023-12-19T07:32:17.224 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_radisys.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rdc.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rz1000.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sch.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_serverworks.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sil680.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sl82c105.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_triflex.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_via.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pdc_adma.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_inic162x.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 83345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_mv.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_nv.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_promise.ko 2023-12-19T07:32:17.225 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_qstor.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil24.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sis.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_svw.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sx4.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_uli.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_via.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_vsc.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ks0108.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/ 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 124473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/bcma.ko 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/ 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/ 2023-12-19T07:32:17.226 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 113665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/aoe.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/ 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 965585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/drbd.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 173225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/floppy.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/ 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 94297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 115793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/nbd.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/ 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 110385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/null_blk.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 98193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/pktcdvd.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 263769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/rbd.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/ 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 98145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/ 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/zram.ko 2023-12-19T07:32:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/applicom.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hangcheck-timer.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/ 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/amd-rng.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/intel-rng.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/via-rng.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 148601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 139313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-12-19T07:32:17.228 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/lp.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/ 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 84097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/mwave.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/nvram.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ppdev.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tlclk.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/ 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-12-19T07:32:17.229 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/ 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_class.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_core.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/ 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-palmas.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-si5351.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-twl6040.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-wm831x.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/ 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-12-19T07:32:17.230 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 243025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp.ko 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/ 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/ 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/ 2023-12-19T07:32:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 438761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-aes.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-sha.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/ 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/ 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/dca.ko 2023-12-19T07:32:17.277 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/ 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/dw_dmac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 122569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ioatdma.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 79953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/amd64_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/e752x_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 63617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/edac_mce_amd.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3000_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3200_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5100_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5400_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7300_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7core_edac.ko 2023-12-19T07:32:17.278 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i82975x_edac.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ie31200_edac.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 56825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/sb_edac.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/x38_edac.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/ 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-gpio.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max14577.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max77693.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max8997.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-palmas.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-rt8973a.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-sm5502.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/ 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/dmi-sysfs.ko 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/ 2023-12-19T07:32:17.279 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/efi-pstore.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/iscsi_ibft.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/ 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-adp5520.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-amd8111.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-arizona.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9052.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9055.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-dln2.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 77849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-f7188x.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-generic.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ich.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-kempld.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-lp3943.ko 2023-12-19T07:32:17.280 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max7300.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max730x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max732x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pca953x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pcf857x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-rdc321x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch311x.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-tps65912.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl4030.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl6040.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-viperboard.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-vx855.ko 2023-12-19T07:32:17.281 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm831x.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8350.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8994.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/ 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-a4tech.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-apple.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-appleir.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-aureal.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-axff.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-belkin.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cherry.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-chicony.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cp2112.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cypress.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-dr.ko 2023-12-19T07:32:17.282 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elecom.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elo.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-emsff.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ezkey.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gaff.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-generic.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gt683r.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gyration.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtekff.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-hyperv.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-icade.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kensington.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-keytouch.ko 2023-12-19T07:32:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kye.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lcpower.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-led.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lenovo.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lg-g15.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-dj.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 80993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-magicmouse.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-microsoft.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-monterey.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-multitouch.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ntrig.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ortek.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-penmount.ko 2023-12-19T07:32:17.284 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-petalynx.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-picolcd.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-pl.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-plantronics.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-primax.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-rmi.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-common.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-isku.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kone.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-lua.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-12-19T07:32:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-savu.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-saitek.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-samsung.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sensor-hub.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sjoy.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sony.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-speedlink.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-steelseries.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sunplus.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-thrustmaster.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tivo.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tmff.ko 2023-12-19T07:32:17.286 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-topseed.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-twinhan.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 71073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-uclogic.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-waltop.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 143473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-wiimote.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-xinmo.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zpff.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zydacron.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 267825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/uhid.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/ 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbhid.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbkbd.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbmouse.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 217873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/wacom.ko 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/ 2023-12-19T07:32:17.287 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 64201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_balloon.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_utils.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 305465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_vmbus.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru3.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7414.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7418.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adc128d818.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1025.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 97977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1026.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1029.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1031.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm9240.ko 2023-12-19T07:32:17.288 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ads7828.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7410.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7411.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 74433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7462.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7470.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7475.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7x10.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/amc6821.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/applesmc.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asb100.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asc7621.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asus_atk0110.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/atxp1.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/coretemp.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9052-hwmon.ko 2023-12-19T07:32:17.289 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9055-hwmon.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 99601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dme1737.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds1621.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds620.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc1403.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc2103.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc6w201.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71805f.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 94457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71882fg.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f75375s.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fam15h_power.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fschmd.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g760a.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g762.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl518sm.ko 2023-12-19T07:32:17.290 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl520sm.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hih6130.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hwmon-vid.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5500_temp.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5k_amb.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmaem.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmpex.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina209.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina2xx.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 130449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/it87.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/jc42.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k10temp.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k8temp.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lineage-pem.ko 2023-12-19T07:32:17.291 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm63.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm73.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm75.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm77.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm78.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm80.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm83.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm85.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm87.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm90.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm92.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 108841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm93.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95234.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95241.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95245.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc2945.ko 2023-12-19T07:32:17.292 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4151.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4215.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4222.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4245.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4260.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4261.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max16065.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1619.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1668.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max197.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6639.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6650.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6697.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mc13783-adc.ko 2023-12-19T07:32:17.293 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mcp3021.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6683.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 119817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775-core.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct7802.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 82337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87360.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87427.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pcf8591.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-12-19T07:32:17.294 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 82577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/powr1220.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5627.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5636.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch56xx-common.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht15.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht21.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/shtc1.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sis5595.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47b397.ko 2023-12-19T07:32:17.295 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m1.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m192.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/thmc50.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp102.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp103.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp401.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp421.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via-cputemp.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via686a.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt1211.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt8231.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627ehf.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627hf.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 71345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83781d.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 63225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83791d.ko 2023-12-19T07:32:17.296 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 71321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83792d.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83793.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 119505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83795.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l785ts.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l786ng.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/ 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/ 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/ 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-12-19T07:32:17.297 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-12-19T07:32:17.298 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-via.ko 2023-12-19T07:32:17.299 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-mux.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-smbus.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-stub.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/ 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 273609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_cm.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 879969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_core.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 69273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_umad.ko 2023-12-19T07:32:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 324217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 97985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/iw_cm.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 282273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 493073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 419737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 739025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 296625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 153697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-12-19T07:32:17.301 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 168969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 263081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 100241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 111425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 128753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 129529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/evbug.ko 2023-12-19T07:32:17.302 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ff-memless.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/ 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/fm801-gp.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/gameport.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/joydev.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/ 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lkkbd.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8323.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8333.ko 2023-12-19T07:32:17.303 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/newtonkbd.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt1070.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt2160.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/stowaway.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/sunkbd.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/xtkbd.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/matrix-keymap.ko 2023-12-19T07:32:17.304 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/ 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/appletouch.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/bcm5974.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 123681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/cyapatp.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/elan_i2c.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/gpio_mouse.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 335929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/psmouse.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/sermouse.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/ 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 109873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/rmi_core.ko 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ 2023-12-19T07:32:17.305 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/altera_ps2.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/arc_ps2.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ct82c710.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/parkbd.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/pcips2.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ps2mult.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serio_raw.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serport.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/sparse-keymap.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/ 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/ 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/ 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-88pm860x.ko 2023-12-19T07:32:17.306 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-adp5520.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-bd2802.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-blinkm.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da903x.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da9052.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-gpio.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3530.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3533.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm355x.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3642.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp3944.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp8788.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-max8997.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-mc13783.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-menf21bmc.ko 2023-12-19T07:32:17.307 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca9532.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca955x.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca963x.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-ss4200.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-tca6507.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm831x-status.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm8350.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-12-19T07:32:17.308 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/ 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/ 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 654969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/bcache.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bio-prison.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bufio.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache-smq.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 156009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 106897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-crypt.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-delay.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-era.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-flakey.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log-userspace.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-mirror.ko 2023-12-19T07:32:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-multipath.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-queue-length.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-raid.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-region-hash.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-round-robin.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-service-time.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 109529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-snapshot.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-switch.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 181865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-thin-pool.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-verity.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-zero.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/faulty.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/linear.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/multipath.ko 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/ 2023-12-19T07:32:17.310 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 199937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid0.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 113073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid1.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 158681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid10.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 395721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid456.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/ 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/ 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 178137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptbase.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 63385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptctl.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptfc.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptlan.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 123321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptsas.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 79041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptscsih.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptspi.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm800.ko 2023-12-19T07:32:17.311 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm805.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm80x.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona-i2c.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 123721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/bcm590xx.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/cros_ec_dev.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/dln2.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/janz-cmodio.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/kempld-core.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-core.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lp3943.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_ich.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_sch.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/max8907.ko 2023-12-19T07:32:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-core.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/menf21bmc.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-adc.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-gpio.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/retu-mfd.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 80425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/si476x-core.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/sm501.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-12-19T07:32:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps65010.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps6507x.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/viperboard.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/vx855.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wl1273-core.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wm8994.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/ 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds9802als.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds990x.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/bh1770glc.ko 2023-12-19T07:32:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/ 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/core.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/ 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/cb710.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ds1682.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/dummy-irq.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/ 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/echo.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/ 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/at24.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/max6875.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/enclosure.ko 2023-12-19T07:32:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/ 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 152201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hmc6352.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hpilo.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ics932s401.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29003.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29020.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/ 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/ 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-me.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 56521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-txe.ko 2023-12-19T07:32:17.316 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 310113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/phantom.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 74265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/st_drv.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_7xx1.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_core.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tsl2550.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_balloon.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 183609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 490129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/bonding.ko 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/ 2023-12-19T07:32:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 106033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6060.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/ 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 453737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dummy.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/eql.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/ 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 130553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 68849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/8390.ko 2023-12-19T07:32:17.318 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 99049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/ 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/ 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 69137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/ 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/ 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/ 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 87297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-12-19T07:32:17.319 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 97337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 104673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 90257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 81569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 69729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 176761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1421497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-12-19T07:32:17.320 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 128569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 132089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 312081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 323513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 193497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 347417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1044497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-12-19T07:32:17.321 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 166193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/ 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/ 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 204345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/ 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/ 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 69449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-12-19T07:32:17.322 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 141225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-12-19T07:32:17.354 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/ 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dnet.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ec_bhf.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/ 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/ 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 281137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ethoc.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fealnx.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/ 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-12-19T07:32:17.355 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 90465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e100.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 295577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 584497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 246209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 925825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 302937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 514641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/ 2023-12-19T07:32:17.356 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 93593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 734305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 151769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 86921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/jme.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 102289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 122649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 727601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 278937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-12-19T07:32:17.357 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1113681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 95665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/ 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 108649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-12-19T07:32:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 150905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 136721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 227897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 87081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 672049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/ 2023-12-19T07:32:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 83985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 189801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 104241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1052497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/ 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-12-19T07:32:17.360 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 63337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/ 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 68177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/ 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 487025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/ 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 103337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 166097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/niu.ko 2023-12-19T07:32:17.361 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/ 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/ 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 86529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/ 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 80273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 83633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/ 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/ 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-12-19T07:32:17.362 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/defxx.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 216985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/skfp.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 77345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/geneve.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 235521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/fakelb.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ifb.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 83769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvlan.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvtap.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/ 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-12-19T07:32:17.363 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-gpio.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-regmap.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mii.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/netconsole.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/nlmon.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/ 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs-lynx.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/amd.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm7xxx.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm87xx.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/broadcom.ko 2023-12-19T07:32:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/cicada.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/davicom.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/et1011c.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/icplus.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/lxt.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 82049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/marvell.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 76113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/micrel.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/national.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 97857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/phylink.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/qsemi.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/realtek.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/smsc.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ste10Xp.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/vitesse.ko 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/ 2023-12-19T07:32:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/plip.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/bsd_comp.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_async.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppoe.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppox.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pptp.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sb1000.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/ 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/slip.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sungem_phy.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 64449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/tap.ko 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/ 2023-12-19T07:32:17.366 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 106169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_random.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 87017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/veth.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/ 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 144297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/ 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 297329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/vxlan.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/ 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 144585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ntb.ko 2023-12-19T07:32:17.367 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/ 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/ 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 265121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme-core.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 102209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/ 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 133841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_cs.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 84465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_pc.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_serial.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/ 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/ 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/pci-stub.ko 2023-12-19T07:32:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/xen-pcifront.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/i82092.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 145809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_core.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pd6729.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 112889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/yenta_socket.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/ 2023-12-19T07:32:17.369 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/ 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/ 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acer-wmi.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acerhdf.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/apple-gmux.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-laptop.ko 2023-12-19T07:32:17.370 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 127441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-wmi.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/hdaps.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ibm_rtl.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/ 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel_ips.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/msi-wmi.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-12-19T07:32:17.371 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-q10.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/ 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_battery.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_charger.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq2415x_charger.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24190_charger.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24735-charger.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-12-19T07:32:17.372 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9030_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9052-battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2760_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2780_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2781_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2782_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/gpio-charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/isp1704_charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/lp8727_charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max14577_charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17040_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17042_battery.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8903_charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8925_power.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/pcf50633-charger.ko 2023-12-19T07:32:17.373 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/sbs-battery.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/test_power.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/tps65090-charger.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_backup.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_power.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm8350_power.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/ 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_common.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/ 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/ 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-gpio.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps_parport.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/pps_core.ko 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ 2023-12-19T07:32:17.374 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp_kvm.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/ 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm80x.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm860x.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-bq32k.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9052.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9055.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9063.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1286.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 68577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1307.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1374.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1511.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1553.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1672.ko 2023-12-19T07:32:17.375 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1742.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds2404.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds3232.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-em3027.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-fm3130.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl12022.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl1208.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-lp8788.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m41t80.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t35.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t59.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t86.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max6900.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8907.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8925.ko 2023-12-19T07:32:17.376 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8997.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8998.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-msm6242.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-palmas.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf50633.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rc5t583.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-12-19T07:32:17.377 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8025.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8581.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-s35390a.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps6586x.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps65910.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm831x.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm8350.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-x1205.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 77617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-9xxx.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-sas.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-xxxx.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 135185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/BusLogic.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/a100u2w.ko 2023-12-19T07:32:17.378 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 272305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 100617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/advansys.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 356201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 295801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 189185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/am53c974.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 106881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/atp870u.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 246985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/ 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1171945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/bfa.ko 2023-12-19T07:32:17.379 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 237273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 115369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ch.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 308921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 76145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 146689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 75553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dc395x.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dmx3191d.ko 2023-12-19T07:32:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/ 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 212425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esp_scsi.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/ 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 158505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fdomain.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/ 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 263489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/fnic.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 201817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hpsa.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hptiop.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hv_storvsc.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/imm.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/initio.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 266537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ipr.ko 2023-12-19T07:32:17.381 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ips.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/ 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 317153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/isci.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_tcp.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/ 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 314865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/libfc.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 145585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/ 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 216665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/libsas.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/ 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 2197249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/ 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-12-19T07:32:17.382 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 310617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 84721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/ 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 659825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/ 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 119561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvumi.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/ 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 81625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/ 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 364105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-12-19T07:32:17.383 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 123617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pmcraid.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ppa.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla1280.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/ 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1876689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 85289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/ 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 559033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qlogicfas408.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/raid_class.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 238761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_debug.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 173937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 279313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 97729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-12-19T07:32:17.384 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ses.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 115889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/st.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/stex.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/ 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 162121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/virtio_scsi.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/wd719x.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/xen-scsifront.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 129441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ssb.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/ 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/ 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 736601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-12-19T07:32:17.385 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/ 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/tcm_loop.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_file.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_iblock.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 930257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_mod.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_pscsi.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 122521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_user.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/ 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 75593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/ 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/ 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/ 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-12-19T07:32:17.386 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ 2023-12-19T07:32:17.387 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/moxa.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 50953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/mxser.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 108601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_gsm.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_hdlc.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/nozomi.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/ 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/ 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_uart.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/arc_uart.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/ 2023-12-19T07:32:17.388 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 103073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-12-19T07:32:17.389 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/rp2.ko 2023-12-19T07:32:17.389 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/sc16is7xx.ko 2023-12-19T07:32:17.389 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/ 2023-12-19T07:32:17.389 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ 2023-12-19T07:32:17.389 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 531265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-12-19T07:32:17.418 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ 2023-12-19T07:32:17.418 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-12-19T07:32:17.418 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-12-19T07:32:17.418 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/ 2023-12-19T07:32:17.418 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_aec.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_cif.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_dmem_genirq.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_mf624.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_netx.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pci_generic.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_sercos3.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/ 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/ 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/c67x00.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 144377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-12-19T07:32:17.419 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/ 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 87089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-acm.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-wdm.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usblp.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usbtmc.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ulpi.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/ 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/ 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-12-19T07:32:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 448505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/ 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/ 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_ether.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_serial.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 92489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 122081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-12-19T07:32:17.421 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 85609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/ 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-12-19T07:32:17.422 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 62185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 161225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/libcomposite.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/ 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/ 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 157681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 69705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 56033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-12-19T07:32:17.423 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 76057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2272.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 75073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2280.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 73169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 65809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 81505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 156073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/bcma-hcd.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 80113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/isp116x-hcd.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 87353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811-hcd.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811_cs.ko 2023-12-19T07:32:17.424 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ssb-hcd.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/ 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/mdc800.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/microtek.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/adutux.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/appledisplay.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cytherm.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ehset.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi26.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi62.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ezusb.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/idmouse.ko 2023-12-19T07:32:17.425 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/iowarrior.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/isight_firmware.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ldusb.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/legousbtower.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/lvstest.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/ 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 64841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/trancevibrator.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usb3503.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usblcd.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbsevseg.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbtest.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/uss720.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/yurex.ko 2023-12-19T07:32:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/ 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/usbmon.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/ 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 301697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/musb_hdrc.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/ 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-generic.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-isp1301.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-tahvo.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/aircable.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ark3116.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/belkin_sa.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ch341.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cp210x.ko 2023-12-19T07:32:17.427 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cyberjack.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cypress_m8.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/empeg.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/f81232.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 175137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/garmin_gps.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 137081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_edgeport.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 124833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_ti.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipaq.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipw.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ir-usb.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 107521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-12-19T07:32:17.428 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kobil_sct.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mct_u232.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/metro-usb.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7720.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 78217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7840.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 56337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mxuport.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/navman.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/omninet.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/opticon.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 191137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/option.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/oti6858.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/pl2303.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcaux.ko 2023-12-19T07:32:17.429 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcserial.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/quatech2.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/safe_serial.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 51625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/sierra.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/spcp8x5.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ssu100.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/symbolserial.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_debug.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_wwan.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usbserial.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/visor.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/whiteheat.ko 2023-12-19T07:32:17.430 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/wishbone-serial.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/xsens_mt.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 52913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/uas.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-alauda.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-cypress.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-datafab.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-freecom.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-isd200.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-karma.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-realtek.ko 2023-12-19T07:32:17.431 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-usbat.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 179969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/usb-storage.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/ 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-core.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-host.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 107001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/ 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/ 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 154297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 108737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 77609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/ 2023-12-19T07:32:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 113785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_iotlb.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 58089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_net.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_scsi.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/ 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/ 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/aat2870_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp5520_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8860_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8870_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/apple_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/as3711_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/bd6107.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da903x_bl.ko 2023-12-19T07:32:17.433 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da9052_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/gpio_backlight.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/kb3886_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lcd.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3533_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3639_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lv5207lp.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/max8925_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pandora_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/platform_lcd.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/wm831x_bl.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/ 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2482.ko 2023-12-19T07:32:17.434 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2490.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/matrox_w1.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/w1-gpio.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/ 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_smem.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 56425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_therm.ko 2023-12-19T07:32:17.435 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 102673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/wire.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/acquirewdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/advantechwdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/cpu5wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9052_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9055_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9063_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/dw_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/eurotechwdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/hpwdt.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/i6300esb.ko 2023-12-19T07:32:17.436 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ib700wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ibmasr.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it87_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/kempld_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/machzwd.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/mena21_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-12-19T07:32:17.437 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/nv_tco.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pc87413_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_pci.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_usb.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/retu_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sc1200wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/softdog.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sp5100_tco.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/twl4030_wdt.ko 2023-12-19T07:32:17.438 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/via_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wafer5823wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wdt_pci.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm831x_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm8350_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/xen_wdt.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/ 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-evtchn.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntalloc.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntdev.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/ 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 176377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-privcmd.ko 2023-12-19T07:32:17.439 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 71249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-scsiback.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/xenfs.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 166425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/9p.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/adfs.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 163593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/affs.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 932817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/kafs.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/ 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/autofs4.ko 2023-12-19T07:32:17.440 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 64545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/befs.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/bfs.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/binfmt_misc.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 3402729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/btrfs.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 306009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/cachefiles.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1333017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ceph.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 100801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/coda.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/ 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 108393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/configfs.ko 2023-12-19T07:32:17.441 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 29625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/cramfs.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 642513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/dlm.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/efs.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1505025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/f2fs.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/msdos.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/ 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/freevxfs.ko 2023-12-19T07:32:17.442 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 226481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/fscache.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/cuse.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1231529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/gfs2.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 141209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/hfs.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 222817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/hfsplus.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 192329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/hpfs.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/ 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 74065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/isofs.ko 2023-12-19T07:32:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 419497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/jfs.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 280313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/lockd.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/minix.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 124649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/netfs.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 81449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 59449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/ 2023-12-19T07:32:17.444 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 117465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1055721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfs.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv2.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 120177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv3.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 2042137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv4.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/ 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/grace.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/nfs_acl.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/ 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1516873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/nfsd.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/ 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 592833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/nilfs2.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/ 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-celtic.ko 2023-12-19T07:32:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-centeuro.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-croatian.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-cyrillic.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-gaelic.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-greek.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-iceland.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-inuit.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-roman.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-romanian.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-turkish.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ascii.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1250.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1251.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1255.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp737.ko 2023-12-19T07:32:17.446 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp775.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp850.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp852.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp855.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp857.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp860.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp861.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp862.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp863.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp864.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp865.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp866.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp869.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp874.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 94137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp932.ko 2023-12-19T07:32:17.447 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 142673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp936.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 165745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp949.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 118121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp950.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_euc-jp.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-1.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-13.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-14.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-15.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-2.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-3.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-4.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-5.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-6.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-7.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-9.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-r.ko 2023-12-19T07:32:17.448 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-ru.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-u.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ucs2_utils.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_utf8.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 219417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ntfs.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 244409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 440737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 2592433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2.ko 2023-12-19T07:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/omfs.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 352161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/overlay.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ramoops.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/qnx4.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 46689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/qnx6.ko 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/ 2023-12-19T07:32:17.450 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_tree.ko 2023-12-19T07:32:17.451 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v1.ko 2023-12-19T07:32:17.451 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v2.ko 2023-12-19T07:32:17.451 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/ 2023-12-19T07:32:17.451 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 532673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/reiserfs.ko 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/ 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/romfs.ko 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/ 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/ 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 2668401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/cifs.ko 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/ 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_arc4.ko 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_md4.ko 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/ 2023-12-19T07:32:17.481 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 134641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/squashfs.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 90721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/sysv.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 247737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/udf.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 162497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ufs.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 4011017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/xfs.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 75489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/locktorture.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/ 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/test_udelay.ko 2023-12-19T07:32:17.482 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/torture.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc-itu-t.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc7.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc8.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/ 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libarc4.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libdes.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/interval_tree_test.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/libcrc32c.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lru_cache.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/ 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4_compress.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4hc_compress.ko 2023-12-19T07:32:17.483 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/ 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/cordic.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/memory-notifier-error-inject.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/notifier-error-inject.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/percpu_test.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/pm-notifier-error-inject.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/ 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 143433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/raid6_pq.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/rbtree_test.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/ 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/reed_solomon.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-kstrtox.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-string_helpers.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 4760425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_bpf.ko 2023-12-19T07:32:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_firmware.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_module.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_user_copy.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_bm.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_fsm.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_kmp.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/ 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/xz_dec_test.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/ 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/hwpoison-inject.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/ 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/6lowpan.ko 2023-12-19T07:32:17.485 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_dest.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_fragment.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_hop.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 6985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_ipv6.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_mobility.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_routing.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_udp.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/ 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/garp.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/mrp.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/p8022.ko 2023-12-19T07:32:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/psnap.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/stp.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/ 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 81737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/8021q.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/ 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 198337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_fd.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_rdma.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_virtio.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/ 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 649297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/batman-adv.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/ 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 63809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/br_netfilter.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 777689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/bridge.ko 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ 2023-12-19T07:32:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_among.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_log.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-12-19T07:32:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 55929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtables.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/ 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1046681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/libceph.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/ 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/pktgen.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/ 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 194209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp.ko 2023-12-19T07:32:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_diag.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 57881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv4.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv6.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/ 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 390473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/dsa_core.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_brcm.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_dsa.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_none.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_trailer.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/ 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 99729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/hsr.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 45233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 254625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154.ko 2023-12-19T07:32:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154_socket.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ah4.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/esp4.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/fou.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/gre.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/inet_diag.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_gre.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_tunnel.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_vti.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipcomp.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipip.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-12-19T07:32:17.491 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ip_tables.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-12-19T07:32:17.492 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_bic.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_dctcp.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_diag.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_highspeed.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_htcp.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_hybla.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_illinois.ko 2023-12-19T07:32:17.493 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_lp.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_scalable.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_vegas.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_veno.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_westwood.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_yeah.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tunnel4.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_diag.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 41665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_tunnel.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ah6.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/esp6.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/fou6.ko 2023-12-19T07:32:17.494 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 80897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_gre.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 93577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_tunnel.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_vti.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ipcomp6.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/mip6.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-12-19T07:32:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-12-19T07:32:17.496 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 74073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/sit.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/tunnel6.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/ 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 82833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/af_key.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/ 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_core.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_debugfs.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 25953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_eth.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip6.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_netlink.ko 2023-12-19T07:32:17.497 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 53241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ppp.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 47977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/lapb.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 267913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc2.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 290417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/mac802154.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/mpls_gso.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 119585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-12-19T07:32:17.498 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 100297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 101441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 102817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 103417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 112849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 60329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 109593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 113129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 112585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 111585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 114465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ 2023-12-19T07:32:17.499 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 411729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conncount.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 408257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack.ko 2023-12-19T07:32:17.500 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 118145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 107233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_dup_netdev.ko 2023-12-19T07:32:17.501 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 37129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_log_syslog.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 95353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_amanda.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_ftp.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_irc.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_sip.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_tftp.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 44065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_synproxy_core.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 615649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_tables.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 42561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_acct.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 39785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_log.ko 2023-12-19T07:32:17.502 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_osf.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_queue.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_chain_nat.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_compat.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 40721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_ct.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_dup_netdev.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fib.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_hash.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_limit.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_log.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_masq.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_nat.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_numgen.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_osf.ko 2023-12-19T07:32:17.503 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_queue.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_redir.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject_inet.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_socket.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_synproxy.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_tproxy.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 89313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/x_tables.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_AUDIT.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 18169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CT.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_DSCP.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HL.ko 2023-12-19T07:32:17.504 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HMARK.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LED.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LOG.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NETMAP.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFLOG.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_RATEEST.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_REDIRECT.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 14337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_SECMARK.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPMSS.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-12-19T07:32:17.505 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TEE.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TPROXY.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TRACE.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_addrtype.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_bpf.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cgroup.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cluster.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_comment.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connbytes.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlabel.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlimit.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connmark.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_conntrack.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cpu.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dccp.ko 2023-12-19T07:32:17.506 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_devgroup.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dscp.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ecn.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_esp.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hashlimit.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_helper.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hl.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipcomp.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_iprange.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipvs.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_l2tp.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_length.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_limit.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mac.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mark.ko 2023-12-19T07:32:17.507 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_multiport.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nat.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nfacct.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_osf.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_owner.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_physdev.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_pkttype.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_policy.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_quota.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_rateest.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_realm.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_recent.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_sctp.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_set.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 16089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_socket.ko 2023-12-19T07:32:17.508 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_state.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_statistic.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_string.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 8553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpmss.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpudp.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_time.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_u32.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/ 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/netlink_diag.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/ 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/nsh.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/ 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 380033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/openvswitch.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-geneve.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-gre.ko 2023-12-19T07:32:17.509 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 11793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-vxlan.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/ 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/af_packet_diag.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/ 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 110201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/phonet.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/pn_pep.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/ 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 251481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 164937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_rdma.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 67377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_tcp.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/ 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 819657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/rxrpc.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/ 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_csum.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_gact.ko 2023-12-19T07:32:17.510 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_ipt.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_mirred.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_nat.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 43449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_pedit.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_police.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_simple.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_skbedit.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_vlan.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_basic.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_bpf.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_cgroup.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_flow.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_fw.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 38425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_route.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 61409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_u32.ko 2023-12-19T07:32:17.511 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_cmp.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 13409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_ipset.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_meta.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_nbyte.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_text.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_u32.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 21649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_choke.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_codel.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 34633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_drr.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 33609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq_codel.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 35433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_gred.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hfsc.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hhf.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 66393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_htb.ko 2023-12-19T07:32:17.512 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_ingress.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 30273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio_lib.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_multiq.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 36137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_netem.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 27497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_pie.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_plug.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_prio.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 48729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_qfq.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 32905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_red.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 26881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfb.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 31353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfq.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_tbf.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 28105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_teql.ko 2023-12-19T07:32:17.513 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 929745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 19033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp_diag.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 336529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 54737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 1520097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/sunrpc.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 707657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/diag.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 862665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/tipc.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/unix_diag.ko 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/ 2023-12-19T07:32:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 85409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 68049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 96193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 10705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_diag.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 17521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/ 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 125177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/x25.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/ 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 22657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_algo.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 20649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 102257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_user.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/ 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/ 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 15537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/irqbypass.ko 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 640081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 624107 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias.bin 2023-12-19T07:32:17.515 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 7663 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 23428 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.alias.bin 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 9821 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.bin 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 70004 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.modinfo 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 146313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 232548 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep.bin 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 210 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.devname 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 72606 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.order 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 879 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.softdep 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 254438 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 309363 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols.bin 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/ 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/ 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/ 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/ 2023-12-19T07:32:17.516 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 170 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/changelog.Debian.gz 2023-12-19T07:32:17.517 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r-- root/root 691 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/copyright 2023-12-19T07:32:17.517 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.518 INFO:teuthology.task.kernel:searching for kernel 6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.518 DEBUG:teuthology.orchestra.run.smithi111:> egrep '(submenu|menuentry.*6.6.0-gcd68252d7b4d).*{' /boot/grub/grub.cfg 2023-12-19T07:32:17.526 INFO:teuthology.orchestra.run.smithi111.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-recovery-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:17.527 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-12-19T07:32:17.527 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-gcd68252d7b4d 2023-12-19T07:32:17.527 DEBUG:teuthology.orchestra.run.smithi111:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-12-19T07:32:17.568 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi111.front.sepia.ceph.com to complete... 2023-12-19T07:32:18.296 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub' 2023-12-19T07:32:18.303 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T07:32:18.314 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2023-12-19T07:32:18.321 INFO:teuthology.orchestra.run.smithi143.stderr:done 2023-12-19T07:32:18.530 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-deb -c /tmp/linux-image.deb 2023-12-19T07:32:18.539 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./ 2023-12-19T07:32:18.540 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./boot/ 2023-12-19T07:32:18.540 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 4415617 2023-12-08 17:50 ./boot/System.map-6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.556 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 169214 2023-12-08 17:50 ./boot/config-6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.556 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11352864 2023-12-08 17:50 ./boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.576 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.578 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/ 2023-12-19T07:32:18.578 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/ 2023-12-19T07:32:18.578 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postinst.d/ 2023-12-19T07:32:18.578 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/postrm.d/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/preinst.d/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./etc/kernel/prerm.d/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ 2023-12-19T07:32:18.579 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 490233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/aesni-intel.ko 2023-12-19T07:32:18.581 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-12-19T07:32:18.581 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 93113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 106729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 152241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 83889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-avx2.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-12-19T07:32:18.582 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/ 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/ 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/ 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/cpuid.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kernel/msr.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/ 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 354433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-amd.ko 2023-12-19T07:32:18.583 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 881625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm-intel.ko 2023-12-19T07:32:18.584 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 2275865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/arch/x86/kvm/kvm.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/af_alg.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_hash.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/algif_skcipher.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ansi_cprng.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/anubis.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/arc4.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/ 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-12-19T07:32:18.598 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/ 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_memcpy.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_pq.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_raid6_recov.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_tx.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/async_xor.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/async_tx/raid6test.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authenc.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/authencesn.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blake2b_generic.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_common.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/blowfish_generic.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/camellia_generic.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast5_generic.ko 2023-12-19T07:32:18.599 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast6_generic.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cast_common.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/ccm.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cmac.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crc32_generic.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cryptd.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_engine.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_simd.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/crypto_user.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/cts.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/des_generic.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/dh_generic.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/echainiv.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/essiv.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/fcrypt.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/khazad.ko 2023-12-19T07:32:18.600 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lrw.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/lz4hc.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/md4.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/michael_mic.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcbc.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/pcrypt.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/rmd160.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/seed.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/serpent_generic.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 149193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tcrypt.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/tea.ko 2023-12-19T07:32:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_common.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/twofish_generic.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/vmac.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/wp512.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xcbc.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xor.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xts.ko 2023-12-19T07:32:18.604 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/crypto/xxhash_generic.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_extlog.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_ipmi.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/acpi_pad.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/ 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/apei/einj.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/ec_sys.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/platform_profile.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbs.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/sbshc.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 103025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/acpi/video.ko 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ 2023-12-19T07:32:18.605 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/acard-ahci.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 88865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/ahci_platform.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 88289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/libahci_platform.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_acpi.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ali.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_amd.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_artop.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atiixp.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_atp867x.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd640.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cmd64x.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_cypress.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_efar.ko 2023-12-19T07:32:18.606 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt366.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt37x.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x2n.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_hpt3x3.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it8213.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_it821x.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_jmicron.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_legacy.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_marvell.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_mpiix.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_netcell.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ninja32.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87410.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_ns87415.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_oldpiix.ko 2023-12-19T07:32:18.607 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_opti.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_optidma.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pcmcia.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc2027x.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_piccolo.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_radisys.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rdc.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_rz1000.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sch.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_serverworks.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sil680.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_sl82c105.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_triflex.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pata_via.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/pdc_adma.ko 2023-12-19T07:32:18.608 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_inic162x.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 83345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_mv.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_nv.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_promise.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_qstor.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sil24.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sis.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_svw.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_sx4.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_uli.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_via.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ata/sata_vsc.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/auxdisplay/ks0108.ko 2023-12-19T07:32:18.609 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 124473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/bcma/bcma.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 113665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/aoe/aoe.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 965585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/drbd/drbd.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 173225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/floppy.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 94297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 115793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/nbd.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 110385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/null_blk/null_blk.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 98193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/pktcdvd.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 263769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/rbd.ko 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/ 2023-12-19T07:32:18.610 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 98145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/ 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/block/zram/zram.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/applicom.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hangcheck-timer.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/ 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/amd-rng.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/intel-rng.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-12-19T07:32:18.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/via-rng.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/hw_random/virtio-rng.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 148601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 139313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_si.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/lp.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/ 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 84097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/mwave/mwave.ko 2023-12-19T07:32:18.664 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/nvram.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/ppdev.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tlclk.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/ 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_atmel.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_infineon.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/tpm_nsc.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/ 2023-12-19T07:32:18.665 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_class.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_core.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/ 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-palmas.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-si5351.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-twl6040.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/clk/clk-wm831x.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/ 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/p4-clockmod.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/cpufreq/speedstep-lib.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-12-19T07:32:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 243025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/ccp/ccp.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 438761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-aes.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/padlock-sha.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dca/dca.ko 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/ 2023-12-19T07:32:18.667 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/dw/dw_dmac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 122569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/dma/ioat/ioatdma.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 79953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/amd64_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/e752x_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 63617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/edac_mce_amd.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3000_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i3200_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5100_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i5400_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7300_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i7core_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/i82975x_edac.ko 2023-12-19T07:32:18.668 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/ie31200_edac.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 56825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/sb_edac.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/edac/x38_edac.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/ 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-gpio.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max14577.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max77693.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-max8997.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-palmas.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-rt8973a.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/extcon/extcon-sm5502.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/ 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/dmi-sysfs.ko 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/ 2023-12-19T07:32:18.669 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/efi/efi-pstore.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/firmware/iscsi_ibft.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/ 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-adp5520.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-amd8111.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-arizona.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9052.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-da9055.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-dln2.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 77849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-f7188x.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-generic.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ich.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-12-19T07:32:18.670 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-kempld.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-lp3943.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max7300.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max730x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-max732x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pca953x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-pcf857x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-rdc321x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-sch311x.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-tps65912.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl4030.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-twl6040.ko 2023-12-19T07:32:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-viperboard.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-vx855.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm831x.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8350.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/gpio/gpio-wm8994.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/ 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-a4tech.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-apple.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-appleir.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-aureal.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-axff.ko 2023-12-19T07:32:18.672 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-belkin.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cherry.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-chicony.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cp2112.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-cypress.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-dr.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elecom.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-elo.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-emsff.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ezkey.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gaff.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-generic.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gt683r.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-gyration.ko 2023-12-19T07:32:18.673 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-kbd.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtek-mouse.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-holtekff.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-hyperv.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-icade.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kensington.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-keytouch.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-kye.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lcpower.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-led.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lenovo.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-lg-g15.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-dj.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-12-19T07:32:18.674 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 80993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-logitech.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-magicmouse.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-microsoft.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-monterey.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-multitouch.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ntrig.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-ortek.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-penmount.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-petalynx.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-picolcd.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-pl.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-plantronics.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-primax.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-rmi.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-arvo.ko 2023-12-19T07:32:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-common.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-isku.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kone.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-konepure.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-lua.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-pyra.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-ryos.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat-savu.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-roccat.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-saitek.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-samsung.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sensor-hub.ko 2023-12-19T07:32:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sjoy.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sony.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-speedlink.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-steelseries.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-sunplus.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-thrustmaster.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tivo.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-tmff.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-topseed.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-twinhan.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 71073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-uclogic.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-waltop.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 143473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-wiimote.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-xinmo.ko 2023-12-19T07:32:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zpff.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid-zydacron.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 267825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/hid.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/uhid.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/ 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbhid.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbkbd.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/usbhid/usbmouse.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 217873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hid/wacom.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/ 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 64201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_balloon.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_utils.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 305465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hv/hv_vmbus.ko 2023-12-19T07:32:18.678 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/abituguru3.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/acpi_power_meter.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7414.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ad7418.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adc128d818.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1025.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 97977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1026.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1029.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm1031.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adm9240.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ads7828.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7410.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7411.ko 2023-12-19T07:32:18.679 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 74433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7462.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7470.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7475.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/adt7x10.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/amc6821.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/applesmc.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asb100.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asc7621.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/asus_atk0110.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/atxp1.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/coretemp.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9052-hwmon.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/da9055-hwmon.ko 2023-12-19T07:32:18.680 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 99601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/dme1737.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds1621.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ds620.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc1403.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc2103.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/emc6w201.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71805f.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 94457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f71882fg.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/f75375s.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fam15h_power.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/fschmd.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g760a.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/g762.ko 2023-12-19T07:32:18.681 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl518sm.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/gl520sm.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hih6130.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/hwmon-vid.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5500_temp.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/i5k_amb.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmaem.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ibmpex.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina209.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ina2xx.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 130449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/it87.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/jc42.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k10temp.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/k8temp.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lineage-pem.ko 2023-12-19T07:32:18.682 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm63.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm73.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm75.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm77.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm78.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm80.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm83.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm85.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm87.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm90.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm92.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 108841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm93.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95234.ko 2023-12-19T07:32:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95241.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/lm95245.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc2945.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4151.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4215.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4222.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4245.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4260.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/ltc4261.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max16065.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1619.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max1668.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max197.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6639.ko 2023-12-19T07:32:18.684 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6650.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/max6697.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mc13783-adc.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/mcp3021.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6683.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 119817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775-core.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct6775.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/nct7802.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 82337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87360.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pc87427.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pcf8591.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-12-19T07:32:18.685 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max16064.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max34440.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/max8688.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 82577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/powr1220.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5627.ko 2023-12-19T07:32:18.686 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch5636.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sch56xx-common.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht15.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sht21.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/shtc1.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/sis5595.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47b397.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m1.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/smsc47m192.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/thmc50.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp102.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp103.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp401.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/tmp421.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via-cputemp.ko 2023-12-19T07:32:18.687 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/via686a.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt1211.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/vt8231.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627ehf.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83627hf.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 71345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83781d.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 63225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83791d.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 71321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83792d.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83793.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 119505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83795.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l785ts.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/w83l786ng.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-12-19T07:32:18.688 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/ 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/ 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/ 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-12-19T07:32:18.689 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-i801.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-isch.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-12-19T07:32:18.690 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-parport.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-via.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-12-19T07:32:18.691 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-mux.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-smbus.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/i2c-stub.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/ 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 273609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_cm.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 879969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_core.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 69273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_umad.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 324217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 97985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/iw_cm.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 282273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_cm.ko 2023-12-19T07:32:18.692 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 493073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 419737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 739025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 296625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 153697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/ 2023-12-19T07:32:18.693 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 168969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 263081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 100241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 111425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 128753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 129529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/evbug.ko 2023-12-19T07:32:18.694 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/ff-memless.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/ 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/fm801-gp.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/gameport/gameport.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/joydev.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/ 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lkkbd.ko 2023-12-19T07:32:18.695 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8323.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/lm8333.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/newtonkbd.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt1070.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/qt2160.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/stowaway.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/sunkbd.ko 2023-12-19T07:32:18.696 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/keyboard/xtkbd.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/matrix-keymap.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/ 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/appletouch.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/bcm5974.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 123681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/cyapatp.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/elan_i2c.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/gpio_mouse.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 335929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/psmouse.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/sermouse.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-12-19T07:32:18.697 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/synaptics_usb.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/mouse/vsxxxaa.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/ 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 109873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/rmi4/rmi_core.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/altera_ps2.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/arc_ps2.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ct82c710.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/parkbd.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/pcips2.ko 2023-12-19T07:32:18.698 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/ps2mult.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serio_raw.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/serio/serport.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/input/sparse-keymap.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/ 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/ 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/iommu/amd/iommu_v2.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/ 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-88pm860x.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-adp5520.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-bd2802.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-blinkm.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da903x.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-da9052.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-gpio.ko 2023-12-19T07:32:18.699 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3530.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3533.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm355x.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lm3642.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp3944.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-lp8788.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-max8997.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-mc13783.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-menf21bmc.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca9532.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca955x.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-pca963x.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-ss4200.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-tca6507.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm831x-status.ko 2023-12-19T07:32:18.700 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/leds-wm8350.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/ 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/ 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 654969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/bcache/bcache.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bio-prison.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-bufio.ko 2023-12-19T07:32:18.701 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache-smq.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 156009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-cache.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 106897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-crypt.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-delay.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-era.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-flakey.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log-userspace.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-log.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-mirror.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-multipath.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-queue-length.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-raid.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-region-hash.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-round-robin.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-service-time.ko 2023-12-19T07:32:18.702 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 109529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-snapshot.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-switch.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 181865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-thin-pool.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-verity.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/dm-zero.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/faulty.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/linear.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/multipath.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/ 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 199937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid0.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 113073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid1.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 158681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid10.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 395721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/md/raid456.ko 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/ 2023-12-19T07:32:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/ 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 178137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptbase.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 63385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptctl.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptfc.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptlan.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 123321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptsas.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 79041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptscsih.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/message/fusion/mptspi.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm800.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm805.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/88pm80x.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona-i2c.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 123721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/arizona.ko 2023-12-19T07:32:18.704 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/bcm590xx.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/cros_ec_dev.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/dln2.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/janz-cmodio.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/kempld-core.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-core.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lp3943.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_ich.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/lpc_sch.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/max8907.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-core.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/menf21bmc.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-adc.ko 2023-12-19T07:32:18.705 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633-gpio.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/pcf50633.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/retu-mfd.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 80425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/si476x-core.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/sm501.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps65010.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/tps6507x.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/viperboard.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/vx855.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wl1273-core.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/mfd/wm8994.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ad525x_dpot.ko 2023-12-19T07:32:18.706 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/ 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds9802als.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/apds990x.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/bh1770glc.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/ 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/c2port/core.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/ 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/cb710/cb710.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ds1682.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/dummy-irq.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/ 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/echo/echo.ko 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/ 2023-12-19T07:32:18.707 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/at24.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/eeprom/max6875.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/enclosure.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/ 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 152201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hmc6352.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/hpilo.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ics932s401.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29003.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/isl29020.ko 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/ 2023-12-19T07:32:18.708 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/ 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-me.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 56521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei-txe.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 310113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/mei/mei.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/phantom.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/ 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 74265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/ti-st/st_drv.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_7xx1.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tifm_core.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/tsl2550.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_balloon.ko 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/ 2023-12-19T07:32:18.709 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 183609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 490129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/bonding/bonding.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 106033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/b53/b53_common.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/bcm-sf2.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6060.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 453737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/dummy.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/eql.ko 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ 2023-12-19T07:32:18.710 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/ 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 130553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 68849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/8390.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 99049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/ 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/ 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 69137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/agere/et131x.ko 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/ 2023-12-19T07:32:18.711 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 87297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 97337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 104673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 90257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/ 2023-12-19T07:32:18.712 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 81569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 69729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/ 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 176761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-12-19T07:32:18.713 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1421497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 128569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/ 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 132089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 312081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/ 2023-12-19T07:32:18.745 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 323513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 193497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 347417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1044497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-12-19T07:32:18.746 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 166193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/ 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/ 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 204345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/ 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/ 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 69449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 141225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-12-19T07:32:18.747 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/ 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/dnet.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ec_bhf.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/ 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/ 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 281137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ethoc.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fealnx.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/ 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ 2023-12-19T07:32:18.748 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 90465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e100.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 295577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 584497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 246209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 925825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 302937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/ 2023-12-19T07:32:18.749 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 514641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/ 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 93593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 734305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 151769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 86921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/jme.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/ 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 102289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/skge.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 122649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/ 2023-12-19T07:32:18.750 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 727601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 278937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1113681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 95665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-12-19T07:32:18.751 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 108649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/ 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 150905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/ 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 136721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/ 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/ 2023-12-19T07:32:18.752 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 227897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 87081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 672049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 83985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/atp.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 189801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 104241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-12-19T07:32:18.753 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1052497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis190.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 63337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sis/sis900.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 68177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-12-19T07:32:18.754 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 487025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/ 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 103337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/cassini.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 166097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/niu.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sungem.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/ 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/ 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 86529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/ti/tlan.ko 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/ 2023-12-19T07:32:18.755 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 80273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 83633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/ 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/ 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/ 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/defxx.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/ 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 216985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/fddi/skfp/skfp.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 77345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/geneve.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/ 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 235521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-12-19T07:32:18.756 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/ 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ieee802154/fakelb.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ifb.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 83769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ipvlan/ipvlan.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvlan.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/macvtap.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/ 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-gpio.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio/mdio-regmap.ko 2023-12-19T07:32:18.757 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mdio.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/mii.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/netconsole.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/nlmon.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/ 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs-lynx.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/amd.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm7xxx.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/bcm87xx.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/broadcom.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/cicada.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/davicom.ko 2023-12-19T07:32:18.758 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/et1011c.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/icplus.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/lxt.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 82049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/marvell.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 76113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/micrel.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/national.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 97857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/phylink.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/qsemi.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/realtek.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/smsc.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/ste10Xp.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/phy/vitesse.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/ 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/plip/plip.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/bsd_comp.ko 2023-12-19T07:32:18.759 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_async.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_deflate.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_mppe.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/ppp_synctty.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppoe.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pppox.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/ppp/pptp.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sb1000.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/ 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/slip/slip.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/sungem_phy.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 64449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/tap.ko 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/ 2023-12-19T07:32:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 106169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_activebackup.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_broadcast.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_random.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 87017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/veth.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/ 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 144297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/ 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 297329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/vxlan/vxlan.ko 2023-12-19T07:32:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 144585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/net/xen-netback/xen-netback.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ntb/ntb.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 265121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme-core.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 102209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/nvme/host/nvme.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 133841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_cs.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 84465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_pc.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/parport/parport_serial.ko 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/ 2023-12-19T07:32:18.762 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/pci-stub.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pci/xen-pcifront.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/ 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/i82092.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 145809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_core.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/pd6729.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 112889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pcmcia/yenta_socket.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/ 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/ 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-12-19T07:32:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ 2023-12-19T07:32:18.764 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acer-wmi.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/acerhdf.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/apple-gmux.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-laptop.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 127441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/asus-wmi.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/classmate-laptop.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/hdaps.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/ibm_rtl.ko 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/ 2023-12-19T07:32:18.765 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/intel_ips.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/msi-wmi.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/mxm-wmi.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-laptop.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/samsung-q10.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/topstar-laptop.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/toshiba_haps.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi-bmof.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/platform/x86/wmi.ko 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/ 2023-12-19T07:32:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/88pm860x_charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq2415x_charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24190_charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/bq24735-charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9030_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/da9052-battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2760_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2780_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2781_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/ds2782_battery.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/gpio-charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/isp1704_charger.ko 2023-12-19T07:32:18.767 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/lp8727_charger.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max14577_charger.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17040_battery.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max17042_battery.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8903_charger.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/max8925_power.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/pcf50633-charger.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/sbs-battery.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/test_power.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/tps65090-charger.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_backup.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm831x_power.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/power/supply/wm8350_power.ko 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/ 2023-12-19T07:32:18.768 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_common.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/powercap/intel_rapl_msr.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/ 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/ 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-gpio.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps-ldisc.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/clients/pps_parport.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/pps/pps_core.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ptp/ptp_kvm.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/ 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm80x.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-88pm860x.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-bq32k.ko 2023-12-19T07:32:18.769 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9052.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9055.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-da9063.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1286.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 68577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1307.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1374.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1511.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1553.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1672.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds1742.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds2404.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-ds3232.ko 2023-12-19T07:32:18.770 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-em3027.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-fm3130.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl12022.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-isl1208.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-lp8788.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m41t80.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t35.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t59.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-m48t86.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max6900.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8907.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8925.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8997.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-max8998.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-12-19T07:32:18.771 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-msm6242.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-palmas.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf2127.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf50633.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf85063.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8523.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8563.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-pcf8583.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rc5t583.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rp5c01.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rs5c372.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8025.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-rx8581.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-s35390a.ko 2023-12-19T07:32:18.772 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps6586x.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-tps65910.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm831x.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-wm8350.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/rtc/rtc-x1205.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 77617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-9xxx.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-sas.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/3w-xxxx.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 135185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/BusLogic.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/a100u2w.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/ 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 272305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aacraid/aacraid.ko 2023-12-19T07:32:18.773 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 100617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/advansys.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 356201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 295801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 189185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/am53c974.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 106881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/atp870u.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 246985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1171945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bfa/bfa.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/ 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 237273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-12-19T07:32:18.774 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 115369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ch.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 308921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/csiostor/csiostor.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 76145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 146689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 75553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dc395x.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/dmx3191d.ko 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/ 2023-12-19T07:32:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 212425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esas2r/esas2r.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/esp_scsi.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/ 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/fcoe.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 158505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fdomain.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/ 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 263489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/fnic/fnic.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 201817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hpsa.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hptiop.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/hv_storvsc.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/imm.ko 2023-12-19T07:32:18.776 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/initio.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 266537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ipr.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ips.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/ 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 317153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/isci/isci.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/iscsi_tcp.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/ 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 314865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libfc/libfc.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 145585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libiscsi_tcp.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/ 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 216665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/libsas/libsas.ko 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/ 2023-12-19T07:32:18.777 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 2197249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/lpfc/lpfc.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/ 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 310617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 84721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/megaraid.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/ 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 659825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/ 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 119561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvsas/mvsas.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/mvumi.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/ 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 81625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-12-19T07:32:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/ 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 364105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 123617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/pmcraid.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ppa.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla1280.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/ 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1876689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 85289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/ 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 559033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/qlogicfas408.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/raid_class.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 238761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_debug.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 173937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_fc.ko 2023-12-19T07:32:18.779 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 279313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 97729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_sas.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_spi.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/scsi_transport_srp.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/ses.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 115889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/st.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/stex.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/ 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 162121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/virtio_scsi.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/vmw_pvscsi.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/wd719x.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/scsi/xen-scsifront.ko 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ 2023-12-19T07:32:18.780 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 129441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ssb/ssb.ko 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/ 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/ 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 736601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/ 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/loopback/tcm_loop.ko 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_file.ko 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_iblock.ko 2023-12-19T07:32:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 930257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_mod.ko 2023-12-19T07:32:18.815 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_pscsi.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 122521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/target_core_user.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/ 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 75593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/ 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/ 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/ 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-12-19T07:32:18.816 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/moxa.ko 2023-12-19T07:32:18.817 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 50953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/mxser.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 108601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_gsm.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/n_hdlc.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/nozomi.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/ 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/ 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/altera_uart.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/arc_uart.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/ 2023-12-19T07:32:18.818 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 103073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/jsm/jsm.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/rp2.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/tty/serial/sc16is7xx.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/ 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 531265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/core/ufshcd-core.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/ 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_aec.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_cif.ko 2023-12-19T07:32:18.819 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_dmem_genirq.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_mf624.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_netx.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pci_generic.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/uio/uio_sercos3.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/ 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/ 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/c67x00/c67x00.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 144377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-12-19T07:32:18.820 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/ 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 87089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-acm.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/cdc-wdm.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usblp.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/class/usbtmc.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/common/ulpi.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/ 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/ 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 448505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/dwc3/dwc3.ko 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/ 2023-12-19T07:32:18.821 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/ 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_ether.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/u_serial.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 92489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 122081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 85609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-12-19T07:32:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/ 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-12-19T07:32:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 62185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 161225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/libcomposite.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/ 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/ 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 157681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 69705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 56033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 76057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2272.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 75073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/net2280.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 73169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 65809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-12-19T07:32:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 81505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 156073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/bcma-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 80113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/isp116x-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 87353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/sl811_cs.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/ssb-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/ 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/mdc800.ko 2023-12-19T07:32:18.825 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/image/microtek.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/adutux.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/appledisplay.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/cytherm.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ehset.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi26.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/emi62.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ezusb.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/idmouse.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/iowarrior.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/isight_firmware.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/ldusb.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/legousbtower.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/lvstest.ko 2023-12-19T07:32:18.826 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/ 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 64841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/trancevibrator.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usb3503.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usblcd.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbsevseg.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/usbtest.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/uss720.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/misc/yurex.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/ 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/mon/usbmon.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/ 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 301697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/musb/musb_hdrc.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/ 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-generic.ko 2023-12-19T07:32:18.827 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-isp1301.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/phy/phy-tahvo.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/aircable.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ark3116.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/belkin_sa.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ch341.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cp210x.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cyberjack.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/cypress_m8.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/digi_acceleport.ko 2023-12-19T07:32:18.828 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/empeg.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/f81232.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 175137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ftdi_sio.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/garmin_gps.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 137081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_edgeport.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 124833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/io_ti.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipaq.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ipw.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ir-usb.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 107521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/keyspan_pda.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kl5kusb105.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/kobil_sct.ko 2023-12-19T07:32:18.829 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mct_u232.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/metro-usb.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7720.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 78217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mos7840.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 56337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/mxuport.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/navman.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/omninet.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/opticon.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 191137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/option.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/oti6858.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/pl2303.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcaux.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/qcserial.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/quatech2.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/safe_serial.ko 2023-12-19T07:32:18.830 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 51625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/sierra.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/spcp8x5.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ssu100.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/symbolserial.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_debug.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usb_wwan.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/usbserial.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/visor.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/whiteheat.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/wishbone-serial.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/serial/xsens_mt.ko 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ 2023-12-19T07:32:18.831 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 52913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/uas.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-alauda.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-cypress.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-datafab.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-freecom.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-isd200.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-karma.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-onetouch.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-realtek.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr09.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-sddr55.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/ums-usbat.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 179969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/storage/usb-storage.ko 2023-12-19T07:32:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/ 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-core.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/usbip-host.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 107001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/ 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/ 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 154297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/pci/vfio-pci.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 108737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 77609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/ 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 113785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost.ko 2023-12-19T07:32:18.833 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_iotlb.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 58089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_net.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/vhost/vhost_scsi.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/ 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/ 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/aat2870_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp5520_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8860_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/adp8870_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/apple_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/as3711_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/bd6107.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da903x_bl.ko 2023-12-19T07:32:18.834 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/da9052_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/gpio_backlight.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/kb3886_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lcd.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3533_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lm3639_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/lv5207lp.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/max8925_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pandora_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/platform_lcd.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/video/backlight/wm831x_bl.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/ 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2482.ko 2023-12-19T07:32:18.835 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/ds2490.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/matrox_w1.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/masters/w1-gpio.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/ 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14153 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_smem.ko 2023-12-19T07:32:18.836 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 56425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/slaves/w1_therm.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 102673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/w1/wire.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/acquirewdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/advantechwdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim1535_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/alim7101_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/cpu5wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9052_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9055_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/da9063_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/dw_wdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/eurotechwdt.ko 2023-12-19T07:32:18.837 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/f71808e_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/hpwdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/i6300esb.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/iTCO_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ib700wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ibmasr.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it8712f_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/it87_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/kempld_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/machzwd.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/mena21_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-12-19T07:32:18.838 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/nv_tco.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22265 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pc87413_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_pci.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/pcwd_usb.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/retu_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sc1200wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sch311x_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/softdog.ko 2023-12-19T07:32:18.839 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/sp5100_tco.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/twl4030_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/via_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83877f_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/w83977f_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wafer5823wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wdt_pci.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm831x_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/wm8350_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/watchdog/xen_wdt.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/ 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-evtchn.ko 2023-12-19T07:32:18.840 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntalloc.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-gntdev.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 176377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-privcmd.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 71249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xen-scsiback.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/drivers/xen/xenfs/xenfs.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 166425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/9p/9p.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/adfs/adfs.ko 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/ 2023-12-19T07:32:18.841 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 163593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/affs/affs.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 932817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/afs/kafs.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/autofs/autofs4.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 64545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/befs/befs.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/bfs/bfs.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/binfmt_misc.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 3402729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/btrfs/btrfs.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 306009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cachefiles/cachefiles.ko 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ 2023-12-19T07:32:18.842 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1333017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ceph/ceph.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/ 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 100801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/coda/coda.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/ 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 108393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/configfs/configfs.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/ 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 29625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/cramfs/cramfs.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/ 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 642513 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/dlm/dlm.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/ 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/efs/efs.ko 2023-12-19T07:32:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1505025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/f2fs/f2fs.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fat/msdos.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/freevxfs/freevxfs.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 226481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fscache/fscache.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/fuse/cuse.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1231529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/gfs2/gfs2.ko 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/ 2023-12-19T07:32:18.844 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 141209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfs/hfs.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 222817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hfsplus/hfsplus.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 192329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/hpfs/hpfs.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 74065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/isofs/isofs.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 419497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/jfs/jfs.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 280313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/lockd/lockd.ko 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/ 2023-12-19T07:32:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/minix/minix.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/ 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 124649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/netfs/netfs.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/ 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/ 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 81449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/ 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 59449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/ 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 117465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1055721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfs.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv2.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 120177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv3.ko 2023-12-19T07:32:18.846 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 2042137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs/nfsv4.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/ 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/grace.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfs_common/nfs_acl.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/ 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1516873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nfsd/nfsd.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/ 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 592833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nilfs2/nilfs2.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/ 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-celtic.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-centeuro.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-croatian.ko 2023-12-19T07:32:18.847 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-cyrillic.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-gaelic.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-greek.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-iceland.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-inuit.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-roman.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-romanian.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/mac-turkish.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ascii.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1250.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1251.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp1255.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp737.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp775.ko 2023-12-19T07:32:18.848 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp850.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp852.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp855.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp857.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp860.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp861.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp862.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp863.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp864.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp865.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp866.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp869.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp874.ko 2023-12-19T07:32:18.849 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 94137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp932.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 142673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp936.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 165745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp949.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 118121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_cp950.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_euc-jp.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-1.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-13.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-14.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-15.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-2.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-3.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-4.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-5.ko 2023-12-19T07:32:18.850 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-6.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-7.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_iso8859-9.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-r.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-ru.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_koi8-u.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_ucs2_utils.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/nls/nls_utf8.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 219417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ntfs/ntfs.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 244409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-12-19T07:32:18.851 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 440737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 2592433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/omfs/omfs.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 352161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/overlayfs/overlay.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/pstore/ramoops.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/ 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx4/qnx4.ko 2023-12-19T07:32:18.852 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 46689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/qnx6/qnx6.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_tree.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v1.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/quota/quota_v2.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 532673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/reiserfs/reiserfs.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/romfs/romfs.ko 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/ 2023-12-19T07:32:18.853 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 2668401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/client/cifs.ko 2023-12-19T07:32:18.883 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/ 2023-12-19T07:32:18.883 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_arc4.ko 2023-12-19T07:32:18.883 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/smb/common/cifs_md4.ko 2023-12-19T07:32:18.883 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 134641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/squashfs/squashfs.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 90721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/sysv/sysv.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 247737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/udf/udf.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 162497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/ufs/ufs.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 4011017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/fs/xfs/xfs.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 75489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/locking/locktorture.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/ 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/time/test_udelay.ko 2023-12-19T07:32:18.884 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/kernel/torture.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc-itu-t.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc7.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crc8.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/ 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libarc4.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/crypto/libdes.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/interval_tree_test.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/libcrc32c.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lru_cache.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/ 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4_compress.ko 2023-12-19T07:32:18.885 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/lz4/lz4hc_compress.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/ 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/math/cordic.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/memory-notifier-error-inject.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/notifier-error-inject.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/percpu_test.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/pm-notifier-error-inject.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/ 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 143433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/raid6/raid6_pq.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/rbtree_test.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/ 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/reed_solomon/reed_solomon.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-kstrtox.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test-string_helpers.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 4760425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_bpf.ko 2023-12-19T07:32:18.886 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_firmware.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_module.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/test_user_copy.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_bm.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_fsm.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/ts_kmp.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/ 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/lib/xz/xz_dec_test.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/ 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/mm/hwpoison-inject.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/ 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/6lowpan.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_dest.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_fragment.ko 2023-12-19T07:32:18.887 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_hop.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 6985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_ipv6.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_mobility.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_routing.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/6lowpan/nhc_udp.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/ 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/garp.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/mrp.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/p8022.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/psnap.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/802/stp.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/ 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 81737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/8021q/8021q.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/ 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 198337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet.ko 2023-12-19T07:32:18.888 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_fd.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26345 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_rdma.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/9p/9pnet_virtio.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/ 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 649297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/batman-adv/batman-adv.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/ 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 63809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/br_netfilter.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 777689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/bridge.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_among.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arp.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-12-19T07:32:18.889 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_limit.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13217 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_log.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8385 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_snat.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_stp.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-12-19T07:32:18.890 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 55929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/ebtables.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/ 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1046681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ceph/libceph.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/ 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/core/pktgen.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/ 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 194209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_diag.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 57881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv4.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dccp/dccp_ipv6.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/ 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 390473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/dsa_core.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_brcm.ko 2023-12-19T07:32:18.891 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_dsa.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_none.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/dsa/tag_trailer.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/ 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 99729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/hsr/hsr.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 45233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 254625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ieee802154/ieee802154_socket.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ah4.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/esp4.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/fou.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/gre.ko 2023-12-19T07:32:18.892 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/inet_diag.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_gre.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61225 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_tunnel.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ip_vti.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipcomp.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/ipip.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arp_tables.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ip_tables.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-12-19T07:32:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11305 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/iptable_security.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18161 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-12-19T07:32:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_bic.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_dctcp.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_diag.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_highspeed.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_htcp.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_hybla.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_illinois.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_lp.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_scalable.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_vegas.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_veno.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_westwood.ko 2023-12-19T07:32:18.895 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tcp_yeah.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/tunnel4.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_diag.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 41665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/udp_tunnel.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv4/xfrm4_tunnel.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ah6.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/esp6.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/fou6.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 80897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_gre.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 93577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_tunnel.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ip6_vti.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/ipcomp6.ko 2023-12-19T07:32:18.896 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19017 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/mip6.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-12-19T07:32:18.897 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11121 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25945 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9009 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 74073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/sit.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/tunnel6.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/ipv6/xfrm6_tunnel.ko 2023-12-19T07:32:18.898 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/ 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 82833 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/key/af_key.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/ 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125281 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_core.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_debugfs.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 25953 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_eth.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36713 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ip6.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_netlink.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 53241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/l2tp/l2tp_ppp.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/ 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 47977 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/lapb/lapb.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/ 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 267913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/llc/llc2.ko 2023-12-19T07:32:18.899 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/ 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 290417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mac802154/mac802154.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/ 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/mpls/mpls_gso.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 119585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26913 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 100297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 101441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 102817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 103417 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 112849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-12-19T07:32:18.900 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 60329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 109593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 113129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 112585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 111585 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 114465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 411729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-12-19T07:32:18.901 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7689 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10681 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7697 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10993 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conncount.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 408257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33761 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 118145 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_h323.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_irc.ko 2023-12-19T07:32:18.902 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 107233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sane.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66641 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_sip.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10113 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_dup_netdev.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9593 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_flow_table_inet.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 37129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_log_syslog.ko 2023-12-19T07:32:18.903 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 95353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_amanda.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_ftp.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11857 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_irc.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28929 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_sip.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_nat_tftp.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 44065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_synproxy_core.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 615649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nf_tables.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 42561 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30233 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_acct.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30569 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 39785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_log.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_osf.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nfnetlink_queue.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10457 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_chain_nat.ko 2023-12-19T07:32:18.904 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_compat.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 40721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_ct.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9601 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_dup_netdev.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fib.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_fwd_netdev.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14185 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_hash.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_limit.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_log.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_masq.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_nat.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13673 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_numgen.ko 2023-12-19T07:32:18.905 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11841 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_osf.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_queue.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_redir.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_reject_inet.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_socket.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_synproxy.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/nft_tproxy.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 89313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/x_tables.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_AUDIT.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CHECKSUM.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CLASSIFY.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 18169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_CT.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_DSCP.ko 2023-12-19T07:32:18.906 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HL.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13369 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_HMARK.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_IDLETIMER.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13889 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LED.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11489 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_LOG.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11753 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_MASQUERADE.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NETMAP.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFLOG.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_NFQUEUE.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_RATEEST.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_REDIRECT.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 14337 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_SECMARK.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPMSS.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-12-19T07:32:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17361 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TEE.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TPROXY.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8321 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_TRACE.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_addrtype.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_bpf.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cgroup.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cluster.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_comment.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10465 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connbytes.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlabel.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connlimit.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13129 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_connmark.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_conntrack.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_cpu.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dccp.ko 2023-12-19T07:32:18.908 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_devgroup.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8801 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_dscp.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ecn.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_esp.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hashlimit.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_helper.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7865 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_hl.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11921 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipcomp.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_iprange.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11785 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_ipvs.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16041 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_l2tp.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8001 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_length.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10209 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_limit.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7545 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mac.ko 2023-12-19T07:32:18.909 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_mark.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11825 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_multiport.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nat.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_nfacct.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_osf.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_owner.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_physdev.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_pkttype.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11985 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_policy.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_quota.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_rateest.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_realm.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_recent.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13329 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_sctp.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_set.ko 2023-12-19T07:32:18.910 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 16089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_socket.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9505 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_state.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9473 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_statistic.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8809 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_string.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 8553 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpmss.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15777 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_tcpudp.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_time.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netfilter/xt_u32.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/ 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13969 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/netlink/netlink_diag.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/ 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 12425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/nsh/nsh.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/ 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 380033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/openvswitch.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-geneve.ko 2023-12-19T07:32:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-gre.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 11793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/openvswitch/vport-vxlan.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/ 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13057 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/packet/af_packet_diag.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/ 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 110201 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/phonet.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54721 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/phonet/pn_pep.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/ 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 251481 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 164937 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_rdma.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 67377 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rds/rds_tcp.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/ 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 819657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/rxrpc/rxrpc.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/ 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32241 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_csum.ko 2023-12-19T07:32:18.912 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23577 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_gact.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26873 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_ipt.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_mirred.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_nat.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 43449 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_pedit.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_police.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_simple.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35297 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_skbedit.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/act_vlan.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_basic.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33961 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_bpf.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19169 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_cgroup.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31401 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_flow.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31769 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_fw.ko 2023-12-19T07:32:18.913 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 38425 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_route.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 61409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/cls_u32.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7441 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_cmp.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 13409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_ipset.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33289 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_meta.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_nbyte.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_text.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7249 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/em_u32.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 21649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_choke.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20849 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_codel.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 34633 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_drr.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 33609 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31617 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_fq_codel.ko 2023-12-19T07:32:18.914 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 35433 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_gred.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 49817 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hfsc.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22089 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_hhf.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 66393 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_htb.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26073 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_ingress.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 30273 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10793 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_mqprio_lib.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26065 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_multiq.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 36137 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_netem.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 27497 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_pie.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10625 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_plug.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 24729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_prio.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 48729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_qfq.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 32905 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_red.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 26881 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfb.ko 2023-12-19T07:32:18.915 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 31353 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_sfq.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28897 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_tbf.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 28105 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sched/sch_teql.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/ 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 929745 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 19033 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sctp/sctp_diag.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/ 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/ 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 336529 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 54737 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 1520097 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/sunrpc.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/ 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 707657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/ 2023-12-19T07:32:18.916 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10025 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/diag.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 862665 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/tipc/tipc.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/ 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15729 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/unix/unix_diag.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/ 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 85409 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 68049 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 96193 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 10705 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_diag.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 17521 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/vmw_vsock/vsock_loopback.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/ 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 125177 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/x25/x25.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/ 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 22657 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_algo.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 20649 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_ipcomp.ko 2023-12-19T07:32:18.917 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 102257 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/net/xfrm/xfrm_user.ko 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/ 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/ 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 15537 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/kernel/virt/lib/irqbypass.ko 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 640081 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 624107 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.alias.bin 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 7663 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 23428 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.alias.bin 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 9821 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.bin 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 70004 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.builtin.modinfo 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 146313 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 232548 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.dep.bin 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 210 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.devname 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 72606 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.order 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 879 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.softdep 2023-12-19T07:32:18.918 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 254438 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 309363 2023-12-08 17:50 ./lib/modules/6.6.0-gcd68252d7b4d/modules.symbols.bin 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/ 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/ 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/ 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:drwxr-xr-x root/root 0 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/ 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 170 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/changelog.Debian.gz 2023-12-19T07:32:18.919 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r-- root/root 691 2023-12-08 17:50 ./usr/share/doc/linux-image-6.6.0-gcd68252d7b4d/copyright 2023-12-19T07:32:18.920 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.920 INFO:teuthology.task.kernel:searching for kernel 6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.920 DEBUG:teuthology.orchestra.run.smithi143:> egrep '(submenu|menuentry.*6.6.0-gcd68252d7b4d).*{' /boot/grub/grub.cfg 2023-12-19T07:32:18.928 INFO:teuthology.orchestra.run.smithi143.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:18.929 INFO:teuthology.orchestra.run.smithi143.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-advanced-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:18.929 INFO:teuthology.orchestra.run.smithi143.stdout: menuentry 'Ubuntu, with Linux 6.6.0-gcd68252d7b4d (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-gcd68252d7b4d-recovery-1168b92c-82ac-47e1-87d9-5d77414c7738' { 2023-12-19T07:32:18.929 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-12-19T07:32:18.930 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-gcd68252d7b4d 2023-12-19T07:32:18.930 DEBUG:teuthology.orchestra.run.smithi143:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-12-19T07:32:18.932 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi143.front.sepia.ceph.com to complete... 2023-12-19T07:32:18.991 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T07:32:19.004 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T07:32:19.585 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub' 2023-12-19T07:32:19.590 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-19T07:32:19.599 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2023-12-19T07:32:19.842 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-6.6.0-gcd68252d7b4d 2023-12-19T07:32:19.853 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-6.6.0-gcd68252d7b4d 2023-12-19T07:32:20.211 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-19T07:32:20.223 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-19T07:32:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:done 2023-12-19T07:32:21.851 INFO:teuthology.orchestra.run.smithi143.stderr:done 2023-12-19T07:32:21.867 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-12-19T07:32:51.897 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-19T07:32:51.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:32:51.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:32:52.966 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-19T07:32:52.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-12-19T07:32:52.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:33:26.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-12-19T07:33:35.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:33:35.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:33:38.481 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-12-19T07:33:50.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:33:50.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:33:52.969 DEBUG:teuthology.orchestra.remote:timed out 2023-12-19T07:34:01.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-12-19T07:34:01.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:34:02.374 DEBUG:teuthology.orchestra.run.smithi143:> true 2023-12-19T07:34:03.479 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-12-19T07:34:03.480 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-gcd68252d7b4d"... 2023-12-19T07:34:03.480 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2023-12-19T07:34:03.539 INFO:teuthology.orchestra.run.smithi143.stdout:6.6.0-gcd68252d7b4d 2023-12-19T07:34:03.540 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gcd68252d7b4d vs 6.6.0-gcd68252d7b4d 2023-12-19T07:34:03.540 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T07:34:03.540 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-19T07:34:04.541 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-19T07:34:04.541 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T07:34:05.085 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2023-12-19T07:34:05.148 DEBUG:teuthology.parallel:result is None 2023-12-19T07:34:08.913 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-12-19T07:34:23.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:34:23.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:34:24.304 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-12-19T07:34:25.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:34:25.051 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-gcd68252d7b4d"... 2023-12-19T07:34:25.051 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-12-19T07:34:25.114 INFO:teuthology.orchestra.run.smithi111.stdout:6.6.0-gcd68252d7b4d 2023-12-19T07:34:25.114 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-gcd68252d7b4d vs 6.6.0-gcd68252d7b4d 2023-12-19T07:34:25.114 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T07:34:25.114 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-19T07:34:26.115 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-19T07:34:26.115 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T07:34:26.237 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2023-12-19T07:34:26.285 DEBUG:teuthology.parallel:result is None 2023-12-19T07:34:26.285 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-19T07:34:26.293 INFO:teuthology.task.internal:Creating test directory... 2023-12-19T07:34:26.294 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T07:34:26.300 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T07:34:26.320 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-19T07:34:26.326 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-19T07:34:26.332 INFO:teuthology.task.internal:Creating archive directory... 2023-12-19T07:34:26.333 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T07:34:26.366 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T07:34:26.423 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-19T07:34:26.429 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-19T07:34:26.429 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T07:34:26.442 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T07:34:26.477 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T07:34:26.497 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T07:34:26.508 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T07:34:26.532 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T07:34:26.534 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-19T07:34:26.541 INFO:teuthology.task.internal:Configuring sudo... 2023-12-19T07:34:26.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T07:34:26.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T07:34:26.614 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-19T07:34:26.624 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-19T07:34:26.624 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T07:34:26.634 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T07:34:26.677 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T07:34:26.701 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T07:34:26.766 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:34:26.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T07:34:26.846 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T07:34:26.869 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T07:34:26.933 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:34:26.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T07:34:27.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-12-19T07:34:27.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2023-12-19T07:34:27.180 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-19T07:34:27.187 INFO:teuthology.task.internal:Starting timer... 2023-12-19T07:34:27.187 INFO:teuthology.run_tasks:Running task pcp... 2023-12-19T07:34:27.214 INFO:teuthology.run_tasks:Running task selinux... 2023-12-19T07:34:27.223 DEBUG:teuthology.task.selinux:Excluding smithi111: OS 'ubuntu' does not support SELinux 2023-12-19T07:34:27.223 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2023-12-19T07:34:27.223 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-19T07:34:27.223 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-19T07:34:27.223 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-19T07:34:27.223 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-19T07:34:27.232 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-19T07:34:27.232 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-19T07:34:27.255 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-19T07:34:27.256 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-19T07:42:42.350 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2023-12-19T07:42:42.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:42:42.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:42:42.457 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-12-19T07:42:42.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-19T07:42:42.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-12-19T07:42:42.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T07:42:42.811 DEBUG:teuthology.orchestra.run.smithi143:> true 2023-12-19T07:42:43.026 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-12-19T07:42:43.026 INFO:teuthology.run_tasks:Running task clock... 2023-12-19T07:42:43.048 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-19T07:42:43.048 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T07:42:43.049 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T07:42:43.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T07:42:43.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T07:42:43.130 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-19T07:42:43.130 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Command line: ntpd -gq 2023-12-19T07:42:43.130 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: proto: precision = 0.041 usec (-24) 2023-12-19T07:42:43.131 INFO:teuthology.orchestra.run.smithi111.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T07:42:43.132 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T07:42:43.132 INFO:teuthology.orchestra.run.smithi111.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: restrict ::: KOD does nothing without LIMITED. 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen and drop on 0 v6wildcard [::]:123 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen normally on 2 lo 127.0.0.1:123 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen normally on 3 enp3s0f1 172.21.15.111:123 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen normally on 4 lo [::1]:123 2023-12-19T07:42:43.133 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc2f%5]:123 2023-12-19T07:42:43.134 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:43 ntpd[23454]: Listening on routing socket on fd #22 for interface updates 2023-12-19T07:42:43.150 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-19T07:42:43.150 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Command line: ntpd -gq 2023-12-19T07:42:43.151 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: proto: precision = 0.055 usec (-24) 2023-12-19T07:42:43.151 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T07:42:43.152 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-19T07:42:43.152 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: restrict ::: KOD does nothing without LIMITED. 2023-12-19T07:42:43.152 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen and drop on 0 v6wildcard [::]:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen normally on 2 lo 127.0.0.1:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen normally on 4 lo [::1]:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2023-12-19T07:42:43.153 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:43 ntpd[23439]: Listening on routing socket on fd #22 for interface updates 2023-12-19T07:42:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:44 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:44.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:44.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:44.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:44.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:44.153 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:44.153 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:44.153 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:44.153 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:44 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:45.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:45 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:46.134 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:46 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:46.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:46.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:46 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:47.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:47.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:47 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.134 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:48.134 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:48 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:48.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:48 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:49.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:49 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:50.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.131 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stdout:19 Dec 07:42:50 ntpd[23454]: ntpd: time slew +0.001744 s 2023-12-19T07:42:50.132 INFO:teuthology.orchestra.run.smithi111.stdout:ntpd: time slew +0.001744s 2023-12-19T07:42:50.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-19T07:42:50.133 INFO:teuthology.orchestra.run.smithi111.stderr:19 Dec 07:42:50 ntpd[23454]: can't open /var/log/ntpstats/loopstats.20231219: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: can't open /var/log/ntpstats/rawstats.20231219: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: can't open /var/log/ntpstats/peerstats.20231219: Permission denied 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stdout:19 Dec 07:42:50 ntpd[23439]: ntpd: time slew -0.001122 s 2023-12-19T07:42:50.151 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew -0.001122s 2023-12-19T07:42:50.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-19T07:42:50.152 INFO:teuthology.orchestra.run.smithi143.stderr:19 Dec 07:42:50 ntpd[23439]: can't open /var/log/ntpstats/loopstats.20231219: Permission denied 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.289 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.304 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T07:42:50.305 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2023-12-19T07:42:50.305 INFO:teuthology.orchestra.run.smithi111.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.305 INFO:teuthology.orchestra.run.smithi111.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.305 INFO:teuthology.orchestra.run.smithi111.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.305 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-19T07:42:50.306 INFO:teuthology.run_tasks:Running task install... 2023-12-19T07:42:50.321 DEBUG:teuthology.task.install:project ceph 2023-12-19T07:42:50.321 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}} 2023-12-19T07:42:50.321 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'} 2023-12-19T07:42:50.321 INFO:teuthology.task.install:Using flavor: default 2023-12-19T07:42:50.327 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-19T07:42:50.327 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-12-19T07:42:50.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-key list | grep Ceph 2023-12-19T07:42:50.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2023-12-19T07:42:50.549 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-19T07:42:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-19T07:42:50.696 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-19T07:42:50.696 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2023-12-19T07:42:50.698 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-19T07:42:50.698 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-12-19T07:42:50.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T07:42:50.712 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-19T07:42:50.712 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph.com (release key) 2023-12-19T07:42:50.714 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-19T07:42:50.714 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-12-19T07:42:50.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T07:42:50.840 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default/ 2023-12-19T07:42:50.840 INFO:teuthology.task.install.deb:Package version is 17.2.7-141-gc7312db0-1focal 2023-12-19T07:42:50.861 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default/ 2023-12-19T07:42:50.861 INFO:teuthology.task.install.deb:Package version is 17.2.7-141-gc7312db0-1focal 2023-12-19T07:42:50.981 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:42:50.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-19T07:42:51.008 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:42:51.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-19T07:42:51.023 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2023-12-19T07:42:51.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2023-12-19T07:42:51.301 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-19T07:42:51.332 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-19T07:42:51.399 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-19T07:42:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal InRelease 2023-12-19T07:42:51.425 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-19T07:42:51.433 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal InRelease 2023-12-19T07:42:51.445 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-12-19T07:42:51.470 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-12-19T07:42:51.480 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-19T07:42:51.483 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release.gpg 2023-12-19T07:42:51.509 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal Release.gpg 2023-12-19T07:42:51.509 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-19T07:42:51.520 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main i386 Packages [3,368 B] 2023-12-19T07:42:51.546 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main i386 Packages [3,368 B] 2023-12-19T07:42:51.563 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-19T07:42:51.581 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-12-19T07:42:51.590 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-19T07:42:51.605 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-12-19T07:42:52.735 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 28.4 kB in 1s (21.0 kB/s) 2023-12-19T07:42:52.754 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 28.4 kB in 1s (20.5 kB/s) 2023-12-19T07:42:53.958 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2023-12-19T07:42:53.994 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-12-19T07:42:54.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-141-gc7312db0-1focal cephadm=17.2.7-141-gc7312db0-1focal ceph-mds=17.2.7-141-gc7312db0-1focal ceph-mgr=17.2.7-141-gc7312db0-1focal ceph-common=17.2.7-141-gc7312db0-1focal ceph-fuse=17.2.7-141-gc7312db0-1focal ceph-test=17.2.7-141-gc7312db0-1focal ceph-volume=17.2.7-141-gc7312db0-1focal radosgw=17.2.7-141-gc7312db0-1focal python3-rados=17.2.7-141-gc7312db0-1focal python3-rgw=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal python3-rbd=17.2.7-141-gc7312db0-1focal libcephfs2=17.2.7-141-gc7312db0-1focal libcephfs-dev=17.2.7-141-gc7312db0-1focal librados2=17.2.7-141-gc7312db0-1focal librbd1=17.2.7-141-gc7312db0-1focal rbd-fuse=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal cephfs-shell=17.2.7-141-gc7312db0-1focal cephfs-top=17.2.7-141-gc7312db0-1focal cephfs-mirror=17.2.7-141-gc7312db0-1focal 2023-12-19T07:42:54.038 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-141-gc7312db0-1focal cephadm=17.2.7-141-gc7312db0-1focal ceph-mds=17.2.7-141-gc7312db0-1focal ceph-mgr=17.2.7-141-gc7312db0-1focal ceph-common=17.2.7-141-gc7312db0-1focal ceph-fuse=17.2.7-141-gc7312db0-1focal ceph-test=17.2.7-141-gc7312db0-1focal ceph-volume=17.2.7-141-gc7312db0-1focal radosgw=17.2.7-141-gc7312db0-1focal python3-rados=17.2.7-141-gc7312db0-1focal python3-rgw=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal python3-rbd=17.2.7-141-gc7312db0-1focal libcephfs2=17.2.7-141-gc7312db0-1focal libcephfs-dev=17.2.7-141-gc7312db0-1focal librados2=17.2.7-141-gc7312db0-1focal librbd1=17.2.7-141-gc7312db0-1focal rbd-fuse=17.2.7-141-gc7312db0-1focal python3-cephfs=17.2.7-141-gc7312db0-1focal cephfs-shell=17.2.7-141-gc7312db0-1focal cephfs-top=17.2.7-141-gc7312db0-1focal cephfs-mirror=17.2.7-141-gc7312db0-1focal 2023-12-19T07:42:54.131 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2023-12-19T07:42:54.159 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-12-19T07:42:54.333 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2023-12-19T07:42:54.333 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2023-12-19T07:42:54.372 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-12-19T07:42:54.372 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-12-19T07:42:54.511 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T07:42:54.512 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T07:42:54.512 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T07:42:54.512 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2023-12-19T07:42:54.512 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-19T07:42:54.513 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-12-19T07:42:54.513 INFO:teuthology.orchestra.run.smithi143.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-12-19T07:42:54.513 INFO:teuthology.orchestra.run.smithi143.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-12-19T07:42:54.513 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T07:42:54.514 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: qttranslations5-l10n socat xmlstarlet 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-19T07:42:54.515 INFO:teuthology.orchestra.run.smithi143.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2023-12-19T07:42:54.516 INFO:teuthology.orchestra.run.smithi143.stdout: btrfs-tools 2023-12-19T07:42:54.576 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T07:42:54.577 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T07:42:54.577 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T07:42:54.577 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2023-12-19T07:42:54.577 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-12-19T07:42:54.578 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T07:42:54.579 INFO:teuthology.orchestra.run.smithi111.stdout: qttranslations5-l10n socat xmlstarlet 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout:Recommended packages: 2023-12-19T07:42:54.580 INFO:teuthology.orchestra.run.smithi111.stdout: btrfs-tools 2023-12-19T07:42:54.719 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2023-12-19T07:42:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-19T07:42:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-19T07:42:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-19T07:42:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-12-19T07:42:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-12-19T07:42:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-12-19T07:42:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-12-19T07:42:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T07:42:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T07:42:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T07:42:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T07:42:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-12-19T07:42:54.722 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2023-12-19T07:42:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 librbd1 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-12-19T07:42:54.779 INFO:teuthology.orchestra.run.smithi111.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-19T07:42:54.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-12-19T07:42:54.781 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2023-12-19T07:42:54.782 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 librbd1 2023-12-19T07:42:54.882 INFO:teuthology.orchestra.run.smithi111.stdout:2 upgraded, 114 newly installed, 0 to remove and 262 not upgraded. 2023-12-19T07:42:54.882 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 210 MB of archives. 2023-12-19T07:42:54.882 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 982 MB of additional disk space will be used. 2023-12-19T07:42:54.883 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-12-19T07:42:54.905 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-19T07:42:54.913 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-12-19T07:42:54.947 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-19T07:42:54.978 INFO:teuthology.orchestra.run.smithi143.stdout:2 upgraded, 114 newly installed, 0 to remove and 262 not upgraded. 2023-12-19T07:42:54.978 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 210 MB of archives. 2023-12-19T07:42:54.978 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 982 MB of additional disk space will be used. 2023-12-19T07:42:54.978 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-141-gc7312db0-1focal [3,501 kB] 2023-12-19T07:42:54.979 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-19T07:42:54.981 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-19T07:42:54.993 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-19T07:42:54.995 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-19T07:42:55.017 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-12-19T07:42:55.019 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-12-19T07:42:55.023 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-12-19T07:42:55.042 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-141-gc7312db0-1focal [3,501 kB] 2023-12-19T07:42:55.055 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-12-19T07:42:55.058 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-12-19T07:42:55.065 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-12-19T07:42:55.067 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-19T07:42:55.068 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-19T07:42:55.071 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-19T07:42:55.071 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-19T07:42:55.072 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-19T07:42:55.073 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-12-19T07:42:55.082 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-19T07:42:55.083 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-19T07:42:55.102 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-19T07:42:55.103 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-19T07:42:55.105 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-19T07:42:55.106 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-19T07:42:55.107 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-19T07:42:55.108 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-19T07:42:55.109 INFO:teuthology.orchestra.run.smithi111.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-19T07:42:55.111 INFO:teuthology.orchestra.run.smithi111.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-19T07:42:55.122 INFO:teuthology.orchestra.run.smithi111.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-19T07:42:55.122 INFO:teuthology.orchestra.run.smithi111.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-19T07:42:55.142 INFO:teuthology.orchestra.run.smithi111.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-19T07:42:55.142 INFO:teuthology.orchestra.run.smithi111.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-19T07:42:55.143 INFO:teuthology.orchestra.run.smithi111.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-19T07:42:55.145 INFO:teuthology.orchestra.run.smithi111.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-19T07:42:55.148 INFO:teuthology.orchestra.run.smithi111.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-19T07:42:55.152 INFO:teuthology.orchestra.run.smithi111.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-19T07:42:55.153 INFO:teuthology.orchestra.run.smithi111.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-19T07:42:55.153 INFO:teuthology.orchestra.run.smithi111.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-19T07:42:55.161 INFO:teuthology.orchestra.run.smithi111.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-19T07:42:55.162 INFO:teuthology.orchestra.run.smithi111.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-19T07:42:55.175 INFO:teuthology.orchestra.run.smithi111.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-19T07:42:55.176 INFO:teuthology.orchestra.run.smithi111.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-19T07:42:55.177 INFO:teuthology.orchestra.run.smithi111.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-19T07:42:55.300 INFO:teuthology.orchestra.run.smithi111.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-19T07:42:55.324 INFO:teuthology.orchestra.run.smithi111.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-19T07:42:55.326 INFO:teuthology.orchestra.run.smithi111.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-19T07:42:55.341 INFO:teuthology.orchestra.run.smithi111.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-19T07:42:55.342 INFO:teuthology.orchestra.run.smithi111.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-19T07:42:55.344 INFO:teuthology.orchestra.run.smithi111.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-19T07:42:55.344 INFO:teuthology.orchestra.run.smithi111.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-19T07:42:55.345 INFO:teuthology.orchestra.run.smithi111.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-19T07:42:55.345 INFO:teuthology.orchestra.run.smithi111.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-19T07:42:55.346 INFO:teuthology.orchestra.run.smithi111.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-19T07:42:55.352 INFO:teuthology.orchestra.run.smithi111.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-19T07:42:55.353 INFO:teuthology.orchestra.run.smithi111.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-19T07:42:55.355 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-141-gc7312db0-1focal [3,553 kB] 2023-12-19T07:42:55.356 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-12-19T07:42:55.373 INFO:teuthology.orchestra.run.smithi111.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-19T07:42:55.376 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-19T07:42:55.379 INFO:teuthology.orchestra.run.smithi111.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-19T07:42:55.384 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-12-19T07:42:55.385 INFO:teuthology.orchestra.run.smithi111.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-19T07:42:55.387 INFO:teuthology.orchestra.run.smithi111.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-19T07:42:55.388 INFO:teuthology.orchestra.run.smithi111.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-19T07:42:55.390 INFO:teuthology.orchestra.run.smithi111.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-19T07:42:55.393 INFO:teuthology.orchestra.run.smithi111.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-12-19T07:42:55.393 INFO:teuthology.orchestra.run.smithi111.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-12-19T07:42:55.411 INFO:teuthology.orchestra.run.smithi111.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-19T07:42:55.412 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-19T07:42:55.414 INFO:teuthology.orchestra.run.smithi111.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-12-19T07:42:55.440 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-19T07:42:55.442 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-19T07:42:55.451 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-19T07:42:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-19T07:42:55.472 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-12-19T07:42:55.472 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-12-19T07:42:55.477 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-12-19T07:42:55.479 INFO:teuthology.orchestra.run.smithi111.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-12-19T07:42:55.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-19T07:42:55.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-19T07:42:55.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-19T07:42:55.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-19T07:42:55.484 INFO:teuthology.orchestra.run.smithi111.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-19T07:42:55.496 INFO:teuthology.orchestra.run.smithi111.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-19T07:42:55.496 INFO:teuthology.orchestra.run.smithi111.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-19T07:42:55.497 INFO:teuthology.orchestra.run.smithi111.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-19T07:42:55.505 INFO:teuthology.orchestra.run.smithi111.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-19T07:42:55.506 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-12-19T07:42:55.507 INFO:teuthology.orchestra.run.smithi111.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-19T07:42:55.507 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-141-gc7312db0-1focal [759 kB] 2023-12-19T07:42:55.507 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-12-19T07:42:55.508 INFO:teuthology.orchestra.run.smithi111.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-19T07:42:55.508 INFO:teuthology.orchestra.run.smithi111.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-19T07:42:55.509 INFO:teuthology.orchestra.run.smithi111.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-19T07:42:55.510 INFO:teuthology.orchestra.run.smithi111.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-12-19T07:42:55.515 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-12-19T07:42:55.517 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-19T07:42:55.518 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-19T07:42:55.520 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-19T07:42:55.520 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-19T07:42:55.521 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-19T07:42:55.523 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-12-19T07:42:55.527 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-19T07:42:55.527 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-19T07:42:55.533 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-141-gc7312db0-1focal [414 kB] 2023-12-19T07:42:55.544 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-19T07:42:55.545 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-19T07:42:55.547 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-19T07:42:55.548 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-19T07:42:55.549 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-19T07:42:55.552 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-19T07:42:55.553 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-19T07:42:55.554 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-19T07:42:55.557 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-141-gc7312db0-1focal [22.6 kB] 2023-12-19T07:42:55.558 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-141-gc7312db0-1focal [200 kB] 2023-12-19T07:42:55.561 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-19T07:42:55.561 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-19T07:42:55.565 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-141-gc7312db0-1focal [47.2 kB] 2023-12-19T07:42:55.568 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-141-gc7312db0-1focal [386 kB] 2023-12-19T07:42:55.578 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-19T07:42:55.578 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-19T07:42:55.581 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-19T07:42:55.582 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-141-gc7312db0-1focal [5,578 kB] 2023-12-19T07:42:55.583 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-19T07:42:55.586 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-19T07:42:55.590 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-19T07:42:55.591 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-19T07:42:55.592 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-19T07:42:55.594 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-19T07:42:55.596 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-19T07:42:55.611 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-19T07:42:55.612 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-19T07:42:55.613 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-19T07:42:55.654 INFO:teuthology.orchestra.run.smithi111.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-141-gc7312db0-1focal [3,553 kB] 2023-12-19T07:42:55.733 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-19T07:42:55.758 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-19T07:42:55.761 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-19T07:42:55.780 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-19T07:42:55.780 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-19T07:42:55.781 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-19T07:42:55.782 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-19T07:42:55.782 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-19T07:42:55.784 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-19T07:42:55.784 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-19T07:42:55.785 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-19T07:42:55.793 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-19T07:42:55.803 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-19T07:42:55.808 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-19T07:42:55.812 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-19T07:42:55.814 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-19T07:42:55.815 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-19T07:42:55.821 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-19T07:42:55.821 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-141-gc7312db0-1focal [116 kB] 2023-12-19T07:42:55.823 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-141-gc7312db0-1focal [479 kB] 2023-12-19T07:42:55.825 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-12-19T07:42:55.825 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-12-19T07:42:55.835 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-141-gc7312db0-1focal [27.8 MB] 2023-12-19T07:42:55.855 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-19T07:42:55.858 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-12-19T07:42:55.913 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-12-19T07:42:55.914 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-19T07:42:55.914 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-19T07:42:55.915 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-19T07:42:55.915 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-19T07:42:55.917 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-19T07:42:55.935 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-19T07:42:55.935 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-19T07:42:55.935 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-19T07:42:55.936 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-19T07:42:55.947 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-19T07:42:55.948 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-19T07:42:55.949 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-19T07:42:55.951 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-19T07:42:55.952 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-12-19T07:42:56.104 INFO:teuthology.orchestra.run.smithi111.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-141-gc7312db0-1focal [759 kB] 2023-12-19T07:42:56.190 INFO:teuthology.orchestra.run.smithi111.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-141-gc7312db0-1focal [414 kB] 2023-12-19T07:42:56.272 INFO:teuthology.orchestra.run.smithi111.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-141-gc7312db0-1focal [22.6 kB] 2023-12-19T07:42:56.274 INFO:teuthology.orchestra.run.smithi111.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-141-gc7312db0-1focal [200 kB] 2023-12-19T07:42:56.279 INFO:teuthology.orchestra.run.smithi111.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-141-gc7312db0-1focal [47.2 kB] 2023-12-19T07:42:56.282 INFO:teuthology.orchestra.run.smithi111.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-141-gc7312db0-1focal [386 kB] 2023-12-19T07:42:56.293 INFO:teuthology.orchestra.run.smithi111.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-141-gc7312db0-1focal [5,578 kB] 2023-12-19T07:42:57.022 INFO:teuthology.orchestra.run.smithi111.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-141-gc7312db0-1focal [116 kB] 2023-12-19T07:42:57.023 INFO:teuthology.orchestra.run.smithi111.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-141-gc7312db0-1focal [479 kB] 2023-12-19T07:42:57.075 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-141-gc7312db0-1focal [6,455 kB] 2023-12-19T07:42:57.094 INFO:teuthology.orchestra.run.smithi111.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-141-gc7312db0-1focal [27.8 MB] 2023-12-19T07:42:57.356 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-141-gc7312db0-1focal [221 kB] 2023-12-19T07:42:57.359 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-141-gc7312db0-1focal [152 kB] 2023-12-19T07:42:57.363 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-141-gc7312db0-1focal [1,590 kB] 2023-12-19T07:42:57.436 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-141-gc7312db0-1focal [7,850 kB] 2023-12-19T07:42:57.779 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-141-gc7312db0-1focal [28.7 MB] 2023-12-19T07:42:59.079 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-141-gc7312db0-1focal [3,940 B] 2023-12-19T07:42:59.080 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-141-gc7312db0-1focal [868 kB] 2023-12-19T07:42:59.094 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-141-gc7312db0-1focal [2,408 kB] 2023-12-19T07:42:59.198 INFO:teuthology.orchestra.run.smithi143.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-141-gc7312db0-1focal [79.2 kB] 2023-12-19T07:42:59.199 INFO:teuthology.orchestra.run.smithi143.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-141-gc7312db0-1focal [117 kB] 2023-12-19T07:42:59.203 INFO:teuthology.orchestra.run.smithi143.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-141-gc7312db0-1focal [1,760 kB] 2023-12-19T07:42:59.286 INFO:teuthology.orchestra.run.smithi143.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-141-gc7312db0-1focal [7,876 kB] 2023-12-19T07:42:59.641 INFO:teuthology.orchestra.run.smithi143.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-141-gc7312db0-1focal [4,056 B] 2023-12-19T07:42:59.641 INFO:teuthology.orchestra.run.smithi143.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-141-gc7312db0-1focal [56.3 MB] 2023-12-19T07:43:00.624 INFO:teuthology.orchestra.run.smithi111.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-141-gc7312db0-1focal [6,455 kB] 2023-12-19T07:43:01.317 INFO:teuthology.orchestra.run.smithi111.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-141-gc7312db0-1focal [221 kB] 2023-12-19T07:43:01.320 INFO:teuthology.orchestra.run.smithi111.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-141-gc7312db0-1focal [152 kB] 2023-12-19T07:43:01.323 INFO:teuthology.orchestra.run.smithi111.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-141-gc7312db0-1focal [1,590 kB] 2023-12-19T07:43:01.478 INFO:teuthology.orchestra.run.smithi111.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-141-gc7312db0-1focal [7,850 kB] 2023-12-19T07:43:02.293 INFO:teuthology.orchestra.run.smithi143.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-141-gc7312db0-1focal [229 kB] 2023-12-19T07:43:02.295 INFO:teuthology.orchestra.run.smithi143.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.7-141-gc7312db0-1focal [240 kB] 2023-12-19T07:43:02.297 INFO:teuthology.orchestra.run.smithi143.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.7-141-gc7312db0-1focal [22.0 kB] 2023-12-19T07:43:02.297 INFO:teuthology.orchestra.run.smithi143.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.7-141-gc7312db0-1focal [13.1 kB] 2023-12-19T07:43:02.298 INFO:teuthology.orchestra.run.smithi143.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-141-gc7312db0-1focal [21.9 kB] 2023-12-19T07:43:02.298 INFO:teuthology.orchestra.run.smithi143.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-141-gc7312db0-1focal [17.1 MB] 2023-12-19T07:43:02.321 INFO:teuthology.orchestra.run.smithi111.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-141-gc7312db0-1focal [28.7 MB] 2023-12-19T07:43:03.107 INFO:teuthology.orchestra.run.smithi143.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-141-gc7312db0-1focal [79.9 kB] 2023-12-19T07:43:04.031 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 210 MB in 8s (25.1 MB/s) 2023-12-19T07:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-141-gc7312db0-1focal [3,940 B] 2023-12-19T07:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-141-gc7312db0-1focal [868 kB] 2023-12-19T07:43:04.144 INFO:teuthology.orchestra.run.smithi111.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-141-gc7312db0-1focal [2,408 kB] 2023-12-19T07:43:04.235 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-19T07:43:04.255 INFO:teuthology.orchestra.run.smithi111.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-141-gc7312db0-1focal [79.2 kB] 2023-12-19T07:43:04.257 INFO:teuthology.orchestra.run.smithi111.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-141-gc7312db0-1focal [117 kB] 2023-12-19T07:43:04.259 INFO:teuthology.orchestra.run.smithi111.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-141-gc7312db0-1focal [1,760 kB] 2023-12-19T07:43:04.307 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 107756 files and directories currently installed.) 2023-12-19T07:43:04.310 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-12-19T07:43:04.336 INFO:teuthology.orchestra.run.smithi111.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-141-gc7312db0-1focal [7,876 kB] 2023-12-19T07:43:04.355 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-12-19T07:43:04.683 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-19T07:43:04.701 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-19T07:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-141-gc7312db0-1focal [4,056 B] 2023-12-19T07:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-141-gc7312db0-1focal [56.3 MB] 2023-12-19T07:43:04.741 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T07:43:05.103 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cmd2. 2023-12-19T07:43:05.123 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-12-19T07:43:05.161 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-12-19T07:43:05.539 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-19T07:43:05.557 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-19T07:43:05.639 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T07:43:06.058 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-librbd1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:06.184 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T07:43:06.947 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-librados2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:07.105 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T07:43:07.825 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2023-12-19T07:43:07.845 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-libcephfs2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:07.904 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:08.196 INFO:teuthology.orchestra.run.smithi111.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-141-gc7312db0-1focal [229 kB] 2023-12-19T07:43:08.197 INFO:teuthology.orchestra.run.smithi111.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.7-141-gc7312db0-1focal [240 kB] 2023-12-19T07:43:08.200 INFO:teuthology.orchestra.run.smithi111.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.7-141-gc7312db0-1focal [22.0 kB] 2023-12-19T07:43:08.243 INFO:teuthology.orchestra.run.smithi111.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.7-141-gc7312db0-1focal [13.1 kB] 2023-12-19T07:43:08.243 INFO:teuthology.orchestra.run.smithi111.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-141-gc7312db0-1focal [21.9 kB] 2023-12-19T07:43:08.243 INFO:teuthology.orchestra.run.smithi111.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-141-gc7312db0-1focal [17.1 MB] 2023-12-19T07:43:08.269 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2023-12-19T07:43:08.290 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-python3-rados_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:08.362 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:08.773 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-19T07:43:08.789 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:08.830 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:09.086 INFO:teuthology.orchestra.run.smithi111.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/rishabh-mdscap-quincy/c7312db04d305d59a3441862c84d6d4da3817eb0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-141-gc7312db0-1focal [79.9 kB] 2023-12-19T07:43:09.176 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2023-12-19T07:43:09.198 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-python3-cephfs_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:09.248 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:09.643 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-19T07:43:09.663 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:09.701 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:10.030 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 210 MB in 14s (14.7 MB/s) 2023-12-19T07:43:10.242 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-19T07:43:10.313 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 107756 files and directories currently installed.) 2023-12-19T07:43:10.316 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-12-19T07:43:10.340 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2023-12-19T07:43:10.355 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-12-19T07:43:10.361 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-12-19T07:43:10.398 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-19T07:43:10.701 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2023-12-19T07:43:10.716 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-19T07:43:10.722 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-python3-rbd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:10.737 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-19T07:43:10.759 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:10.783 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T07:43:11.152 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cmd2. 2023-12-19T07:43:11.153 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-19T07:43:11.169 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-12-19T07:43:11.173 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-19T07:43:11.211 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T07:43:11.237 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-12-19T07:43:11.632 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-19T07:43:11.648 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-19T07:43:11.653 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-19T07:43:11.668 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-19T07:43:11.723 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T07:43:11.748 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T07:43:12.127 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-19T07:43:12.135 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../004-librbd1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:12.147 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-19T07:43:12.185 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T07:43:12.285 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librbd1 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T07:43:12.613 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-19T07:43:12.633 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-12-19T07:43:12.679 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T07:43:13.066 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-19T07:43:13.085 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../005-librados2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:13.087 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-12-19T07:43:13.124 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T07:43:13.215 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librados2 (17.2.7-141-gc7312db0-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-19T07:43:13.644 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-19T07:43:13.666 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:13.736 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:13.960 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs2. 2023-12-19T07:43:13.983 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../006-libcephfs2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:14.043 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:14.309 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-19T07:43:14.327 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:14.364 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:14.412 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rados. 2023-12-19T07:43:14.431 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../007-python3-rados_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:14.470 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:14.784 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-19T07:43:14.797 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:14.842 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:14.874 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-19T07:43:14.894 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:14.932 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:15.243 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cephfs. 2023-12-19T07:43:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../009-python3-cephfs_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:15.279 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-12-19T07:43:15.300 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-12-19T07:43:15.309 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:15.339 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T07:43:15.712 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-19T07:43:15.730 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:15.732 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2023-12-19T07:43:15.752 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-librgw2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:15.770 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:15.790 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:16.174 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-prettytable. 2023-12-19T07:43:16.192 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-12-19T07:43:16.232 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-19T07:43:16.551 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rbd. 2023-12-19T07:43:16.573 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../012-python3-rbd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:16.609 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:16.635 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2023-12-19T07:43:16.657 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-python3-rgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:16.701 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:16.954 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-19T07:43:16.975 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-19T07:43:17.012 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T07:43:17.104 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0. 2023-12-19T07:43:17.122 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-19T07:43:17.201 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-19T07:43:17.432 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-19T07:43:17.453 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-19T07:43:17.490 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T07:43:17.566 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2023-12-19T07:43:17.584 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-libradosstriper1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:17.649 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:17.944 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-19T07:43:17.962 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-19T07:43:18.002 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T07:43:18.153 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2023-12-19T07:43:18.172 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-ceph-common_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:18.210 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:18.413 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-19T07:43:18.433 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-12-19T07:43:18.471 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T07:43:18.875 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-19T07:43:18.898 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-12-19T07:43:18.941 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T07:43:19.487 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-19T07:43:19.506 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:19.560 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:20.390 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-19T07:43:20.407 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:20.470 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:20.893 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-19T07:43:20.913 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-12-19T07:43:20.951 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:43:21.225 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2023-12-19T07:43:21.246 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-ceph-base_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:21.314 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:21.396 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-12-19T07:43:21.416 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-12-19T07:43:21.454 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T07:43:21.874 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librgw2. 2023-12-19T07:43:21.894 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../022-librgw2_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:21.949 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:22.313 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cryptography. 2023-12-19T07:43:22.332 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-19T07:43:22.370 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T07:43:22.785 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rgw. 2023-12-19T07:43:22.802 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../023-python3-rgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:22.843 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:22.879 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2023-12-19T07:43:22.898 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-19T07:43:22.941 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-19T07:43:23.213 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liboath0. 2023-12-19T07:43:23.233 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-19T07:43:23.270 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-19T07:43:23.275 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-19T07:43:23.297 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:23.344 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:23.632 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libradosstriper1. 2023-12-19T07:43:23.656 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../025-libradosstriper1_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:23.715 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:23.800 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-19T07:43:23.825 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-19T07:43:23.866 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T07:43:24.219 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-19T07:43:24.237 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-19T07:43:24.260 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-common. 2023-12-19T07:43:24.277 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-19T07:43:24.280 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../026-ceph-common_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:24.318 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:24.689 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-natsort. 2023-12-19T07:43:24.709 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-12-19T07:43:24.747 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-12-19T07:43:25.125 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jwt. 2023-12-19T07:43:25.146 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-19T07:43:25.183 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T07:43:25.571 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2023-12-19T07:43:25.591 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-12-19T07:43:25.628 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-19T07:43:26.715 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-19T07:43:26.733 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-19T07:43:26.773 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-19T07:43:27.132 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-base. 2023-12-19T07:43:27.135 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2023-12-19T07:43:27.154 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../027-ceph-base_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:27.156 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-19T07:43:27.201 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T07:43:27.224 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:27.589 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-19T07:43:27.609 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-19T07:43:27.646 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-19T07:43:27.991 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-19T07:43:28.011 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-19T07:43:28.049 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T07:43:28.304 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cryptography. 2023-12-19T07:43:28.325 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-19T07:43:28.361 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T07:43:28.427 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2023-12-19T07:43:28.449 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-19T07:43:28.539 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-19T07:43:28.816 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-openssl. 2023-12-19T07:43:28.836 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-19T07:43:28.899 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-19T07:43:28.905 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-19T07:43:28.925 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-19T07:43:28.980 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T07:43:29.252 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-19T07:43:29.272 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:29.318 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:29.365 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bs4. 2023-12-19T07:43:29.382 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-12-19T07:43:29.433 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-19T07:43:29.806 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-19T07:43:29.823 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-19T07:43:29.845 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2023-12-19T07:43:29.864 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T07:43:29.865 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-19T07:43:29.909 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T07:43:30.226 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-19T07:43:30.246 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-19T07:43:30.281 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2023-12-19T07:43:30.283 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-19T07:43:30.298 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-12-19T07:43:30.339 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-19T07:43:30.621 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-natsort. 2023-12-19T07:43:30.641 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-12-19T07:43:30.704 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-12-19T07:43:30.709 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2023-12-19T07:43:30.730 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-19T07:43:30.767 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T07:43:31.066 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jwt. 2023-12-19T07:43:31.083 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-19T07:43:31.123 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T07:43:31.154 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-19T07:43:31.174 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-19T07:43:31.212 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T07:43:31.502 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-logutils. 2023-12-19T07:43:31.522 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-12-19T07:43:31.560 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-19T07:43:31.566 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-19T07:43:31.586 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-19T07:43:31.623 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-19T07:43:31.905 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-19T07:43:31.926 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-19T07:43:31.963 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-19T07:43:31.994 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2023-12-19T07:43:32.011 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-12-19T07:43:32.051 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-19T07:43:32.309 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-mako. 2023-12-19T07:43:32.326 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-19T07:43:32.366 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T07:43:32.421 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2023-12-19T07:43:32.441 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-19T07:43:32.479 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T07:43:32.744 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-19T07:43:32.765 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-19T07:43:32.827 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-19T07:43:32.926 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-19T07:43:32.949 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-19T07:43:32.992 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T07:43:33.180 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-19T07:43:33.201 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-19T07:43:33.247 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T07:43:33.396 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-19T07:43:33.416 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:33.454 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:33.583 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webob. 2023-12-19T07:43:33.603 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-19T07:43:33.641 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-19T07:43:33.858 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2023-12-19T07:43:33.878 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-ceph-mgr_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:33.932 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:34.036 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-19T07:43:34.057 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-19T07:43:34.094 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T07:43:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-bs4. 2023-12-19T07:43:34.461 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2023-12-19T07:43:34.478 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-12-19T07:43:34.478 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-ceph-mon_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:34.516 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-19T07:43:34.518 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:34.919 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-waitress. 2023-12-19T07:43:34.939 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-19T07:43:35.017 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T07:43:35.430 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-tempita. 2023-12-19T07:43:35.451 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-12-19T07:43:35.488 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-19T07:43:35.706 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2023-12-19T07:43:35.731 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-ceph-osd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:35.772 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:35.825 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-paste. 2023-12-19T07:43:35.846 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-19T07:43:35.882 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T07:43:36.253 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-19T07:43:36.274 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-19T07:43:36.318 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T07:43:36.664 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-19T07:43:36.683 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-19T07:43:36.721 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-19T07:43:37.125 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webtest. 2023-12-19T07:43:37.144 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-12-19T07:43:37.182 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-19T07:43:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pecan. 2023-12-19T07:43:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-19T07:43:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T07:43:38.048 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-19T07:43:38.069 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-19T07:43:38.106 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T07:43:38.292 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2023-12-19T07:43:38.309 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:38.375 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:38.552 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-19T07:43:38.572 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:38.617 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:38.720 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2023-12-19T07:43:38.740 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-ceph-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:38.794 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:39.013 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr. 2023-12-19T07:43:39.033 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../052-ceph-mgr_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:39.078 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:39.281 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2023-12-19T07:43:39.301 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-ceph-mds_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:39.339 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:39.648 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mon. 2023-12-19T07:43:39.670 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../053-ceph-mon_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:39.715 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:39.951 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2023-12-19T07:43:39.972 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-cephadm_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:40.042 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:40.437 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2023-12-19T07:43:40.450 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-12-19T07:43:40.495 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-19T07:43:40.890 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-19T07:43:40.908 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:40.910 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-osd. 2023-12-19T07:43:40.928 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../054-ceph-osd_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:40.948 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:40.968 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:41.376 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-19T07:43:41.393 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-12-19T07:43:41.433 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-19T07:43:41.771 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-underscore. 2023-12-19T07:43:41.791 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-19T07:43:41.828 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T07:43:42.132 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-19T07:43:42.152 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-19T07:43:42.181 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T07:43:42.534 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2023-12-19T07:43:42.553 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-12-19T07:43:42.592 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-19T07:43:42.996 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-19T07:43:43.018 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:43.062 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:43.538 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph. 2023-12-19T07:43:43.558 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../055-ceph_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:43.596 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:43.649 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-decorator. 2023-12-19T07:43:43.669 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-19T07:43:43.707 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T07:43:43.982 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-fuse. 2023-12-19T07:43:44.002 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../056-ceph-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:44.069 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-19T07:43:44.074 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:44.089 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-19T07:43:44.126 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T07:43:44.497 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-scipy. 2023-12-19T07:43:44.517 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-19T07:43:44.552 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mds. 2023-12-19T07:43:44.572 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../057-ceph-mds_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:44.574 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-19T07:43:44.610 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:45.196 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephadm. 2023-12-19T07:43:45.210 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../058-cephadm_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:45.254 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:45.582 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jinja2. 2023-12-19T07:43:45.605 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-12-19T07:43:45.648 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-19T07:43:46.035 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-19T07:43:46.043 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-19T07:43:46.056 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:46.057 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-19T07:43:46.101 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:46.101 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T07:43:46.496 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-19T07:43:46.521 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-12-19T07:43:46.587 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-19T07:43:46.738 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2023-12-19T07:43:46.758 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-12-19T07:43:46.796 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-19T07:43:46.906 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjs-underscore. 2023-12-19T07:43:46.923 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-19T07:43:46.973 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T07:43:47.217 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2023-12-19T07:43:47.237 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-19T07:43:47.283 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T07:43:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-19T07:43:47.316 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-19T07:43:47.356 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T07:43:47.717 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-routes. 2023-12-19T07:43:47.735 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-12-19T07:43:47.775 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-19T07:43:48.109 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-19T07:43:48.127 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:48.162 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-19T07:43:48.166 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:48.179 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:48.219 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:48.790 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-decorator. 2023-12-19T07:43:48.811 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-19T07:43:48.856 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T07:43:49.234 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-19T07:43:49.247 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-19T07:43:49.292 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T07:43:49.420 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cachetools. 2023-12-19T07:43:49.442 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-12-19T07:43:49.486 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-19T07:43:49.670 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-scipy. 2023-12-19T07:43:49.690 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-19T07:43:49.750 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-19T07:43:49.865 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-19T07:43:49.882 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-19T07:43:49.924 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T07:43:50.320 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-19T07:43:50.340 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-19T07:43:50.386 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T07:43:50.772 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2023-12-19T07:43:50.794 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-12-19T07:43:50.872 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-19T07:43:51.275 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-google-auth. 2023-12-19T07:43:51.290 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-12-19T07:43:51.333 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-19T07:43:51.466 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-19T07:43:51.486 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-19T07:43:51.523 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T07:43:51.720 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-blinker. 2023-12-19T07:43:51.741 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-19T07:43:52.053 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T07:43:52.128 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-joblib. 2023-12-19T07:43:52.146 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-12-19T07:43:52.185 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-19T07:43:52.415 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-19T07:43:52.436 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-19T07:43:52.481 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T07:43:52.630 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn. 2023-12-19T07:43:52.650 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-19T07:43:52.712 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T07:43:52.868 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-19T07:43:52.886 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-19T07:43:52.951 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T07:43:53.521 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-19T07:43:53.532 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2023-12-19T07:43:53.542 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:53.550 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-19T07:43:53.587 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:53.589 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T07:43:53.978 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-19T07:43:53.997 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-19T07:43:54.035 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T07:43:54.693 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-19T07:43:54.714 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:54.751 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:54.824 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cachetools. 2023-12-19T07:43:54.844 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-12-19T07:43:54.915 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-19T07:43:55.121 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-19T07:43:55.143 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-12-19T07:43:55.187 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-19T07:43:55.293 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-19T07:43:55.315 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-19T07:43:55.359 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T07:43:55.599 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-19T07:43:55.621 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:43:55.666 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:43:55.771 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-19T07:43:55.792 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-19T07:43:55.829 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T07:43:56.002 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2023-12-19T07:43:56.022 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:43:56.060 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:43:56.199 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rsa. 2023-12-19T07:43:56.216 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-12-19T07:43:56.277 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-19T07:43:56.355 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2023-12-19T07:43:56.375 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-12-19T07:43:56.420 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-19T07:43:56.677 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-google-auth. 2023-12-19T07:43:56.697 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-12-19T07:43:56.734 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-19T07:43:56.791 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2023-12-19T07:43:56.812 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-19T07:43:56.857 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-19T07:43:57.104 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-blinker. 2023-12-19T07:43:57.126 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-19T07:43:57.170 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T07:43:57.271 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2023-12-19T07:43:57.293 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-ceph-test_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:43:57.336 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:43:57.515 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-19T07:43:57.529 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-19T07:43:57.573 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T07:43:57.951 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-19T07:43:57.965 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-19T07:43:58.010 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T07:43:58.473 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-websocket. 2023-12-19T07:43:58.492 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-19T07:43:58.531 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T07:43:58.902 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-19T07:43:58.924 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-19T07:43:58.968 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T07:43:59.641 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-19T07:43:59.655 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:43:59.716 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:00.103 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-19T07:44:00.119 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-12-19T07:44:00.160 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-19T07:44:00.606 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-19T07:44:00.619 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:44:00.663 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:00.991 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package jq. 2023-12-19T07:44:01.004 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:44:01.049 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:01.377 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package socat. 2023-12-19T07:44:01.390 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-12-19T07:44:01.435 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-19T07:44:01.797 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package xmlstarlet. 2023-12-19T07:44:01.819 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-19T07:44:01.863 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-19T07:44:02.033 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-volume. 2023-12-19T07:44:02.053 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-ceph-volume_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:02.090 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:02.259 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-test. 2023-12-19T07:44:02.280 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../089-ceph-test_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:02.342 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:02.615 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-19T07:44:02.635 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:02.679 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:03.057 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-colorama. 2023-12-19T07:44:03.071 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-12-19T07:44:03.115 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-12-19T07:44:03.452 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephfs-shell. 2023-12-19T07:44:03.472 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-cephfs-shell_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:03.518 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:03.787 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephfs-top. 2023-12-19T07:44:03.801 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-cephfs-top_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:03.879 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:04.148 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package fonts-mathjax. 2023-12-19T07:44:04.162 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T07:44:04.206 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:04.786 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-19T07:44:04.807 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:04.844 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:05.264 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2023-12-19T07:44:05.285 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-19T07:44:05.355 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T07:44:05.770 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-mathjax. 2023-12-19T07:44:05.793 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T07:44:05.836 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:07.121 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-volume. 2023-12-19T07:44:07.141 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../090-ceph-volume_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:07.204 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:07.634 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-19T07:44:07.652 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:07.692 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:07.885 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-natsort-doc. 2023-12-19T07:44:07.909 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-12-19T07:44:07.951 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-12-19T07:44:08.104 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-colorama. 2023-12-19T07:44:08.125 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-12-19T07:44:08.170 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-12-19T07:44:08.322 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-19T07:44:08.343 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-19T07:44:08.387 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T07:44:08.498 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephfs-shell. 2023-12-19T07:44:08.516 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../093-cephfs-shell_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:08.555 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:08.741 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-attr. 2023-12-19T07:44:08.763 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-12-19T07:44:08.807 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-19T07:44:08.850 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephfs-top. 2023-12-19T07:44:08.873 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../094-cephfs-top_17.2.7-141-gc7312db0-1focal_all.deb ... 2023-12-19T07:44:08.907 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:09.197 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webencodings. 2023-12-19T07:44:09.218 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-19T07:44:09.236 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package fonts-mathjax. 2023-12-19T07:44:09.256 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T07:44:09.279 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T07:44:09.327 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:09.600 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-html5lib. 2023-12-19T07:44:09.618 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-12-19T07:44:09.690 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-19T07:44:09.923 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-19T07:44:09.946 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:10.011 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:10.088 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-19T07:44:10.108 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-19T07:44:10.152 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T07:44:10.460 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package nvme-cli. 2023-12-19T07:44:10.478 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-19T07:44:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T07:44:10.683 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-packaging. 2023-12-19T07:44:10.706 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-12-19T07:44:10.749 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-19T07:44:10.914 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjs-mathjax. 2023-12-19T07:44:10.932 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-12-19T07:44:10.963 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:11.113 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2023-12-19T07:44:11.138 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-19T07:44:11.177 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-19T07:44:11.548 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2023-12-19T07:44:11.567 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-12-19T07:44:11.605 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-19T07:44:11.984 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-psutil. 2023-12-19T07:44:12.005 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-19T07:44:12.041 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T07:44:12.487 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2023-12-19T07:44:12.506 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-19T07:44:12.544 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T07:44:12.957 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-19T07:44:12.981 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-19T07:44:13.031 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T07:44:13.228 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-natsort-doc. 2023-12-19T07:44:13.251 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-12-19T07:44:13.294 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-12-19T07:44:13.376 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2023-12-19T07:44:13.398 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-12-19T07:44:13.450 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-19T07:44:13.684 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-19T07:44:13.703 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-19T07:44:13.766 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T07:44:13.863 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2023-12-19T07:44:13.887 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-19T07:44:13.929 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T07:44:14.119 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-attr. 2023-12-19T07:44:14.138 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-12-19T07:44:14.177 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-19T07:44:14.274 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-19T07:44:14.294 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-12-19T07:44:14.332 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T07:44:14.539 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webencodings. 2023-12-19T07:44:14.560 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-19T07:44:14.605 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T07:44:14.961 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-html5lib. 2023-12-19T07:44:14.982 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-12-19T07:44:15.027 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-19T07:44:15.177 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2023-12-19T07:44:15.192 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../114-radosgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:15.235 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:15.414 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-19T07:44:15.435 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-19T07:44:15.480 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T07:44:15.967 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-packaging. 2023-12-19T07:44:15.985 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-12-19T07:44:16.024 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-19T07:44:16.372 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastescript. 2023-12-19T07:44:16.390 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-19T07:44:16.429 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-19T07:44:16.824 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pluggy. 2023-12-19T07:44:16.847 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-12-19T07:44:16.890 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-19T07:44:16.923 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2023-12-19T07:44:16.942 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../115-rbd-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:16.980 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:17.268 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-psutil. 2023-12-19T07:44:17.283 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-19T07:44:17.355 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-19T07:44:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T07:44:17.727 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T07:44:17.763 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-py. 2023-12-19T07:44:17.778 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-19T07:44:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T07:44:17.854 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T07:44:17.980 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T07:44:18.216 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-19T07:44:18.223 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:18.238 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-19T07:44:18.282 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T07:44:18.627 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pytest. 2023-12-19T07:44:18.646 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-12-19T07:44:18.684 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-19T07:44:18.948 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2023-12-19T07:44:19.088 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplejson. 2023-12-19T07:44:19.108 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-19T07:44:19.140 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:19.170 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T07:44:19.299 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T07:44:19.516 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-19T07:44:19.537 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-12-19T07:44:19.573 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T07:44:19.734 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-19T07:44:20.126 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:20.252 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-19T07:44:20.344 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package radosgw. 2023-12-19T07:44:20.366 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../114-radosgw_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:20.378 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T07:44:20.410 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-19T07:44:21.274 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T07:44:21.650 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-19T07:44:22.001 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-12-19T07:44:22.360 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T07:44:22.422 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package rbd-fuse. 2023-12-19T07:44:22.447 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../115-rbd-fuse_17.2.7-141-gc7312db0-1focal_amd64.deb ... 2023-12-19T07:44:22.488 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:22.852 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T07:44:22.856 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-19T07:44:23.220 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:23.260 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-19T07:44:23.386 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-12-19T07:44:23.511 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-19T07:44:23.596 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T07:44:23.722 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-12-19T07:44:23.771 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:24.064 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-19T07:44:24.415 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T07:44:24.463 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user cephadm....done 2023-12-19T07:44:24.644 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:24.783 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T07:44:24.811 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-19T07:44:24.909 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T07:44:25.238 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-19T07:44:25.385 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-12-19T07:44:25.589 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-12-19T07:44:25.715 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-19T07:44:25.744 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T07:44:25.874 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-19T07:44:26.137 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T07:44:26.276 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-19T07:44:26.538 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T07:44:26.762 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-19T07:44:26.890 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-19T07:44:27.146 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-19T07:44:27.301 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-19T07:44:27.531 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-12-19T07:44:27.661 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-19T07:44:27.932 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-19T07:44:28.053 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T07:44:28.188 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T07:44:28.308 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-19T07:44:28.313 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T07:44:28.717 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-argparse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:28.723 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T07:44:29.093 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-19T07:44:29.216 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-12-19T07:44:29.218 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-12-19T07:44:29.570 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-19T07:44:29.617 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T07:44:29.929 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-19T07:44:30.118 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T07:44:30.280 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-12-19T07:44:30.406 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-19T07:44:30.561 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-19T07:44:30.696 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:30.924 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-12-19T07:44:31.089 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:31.241 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-19T07:44:31.284 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-19T07:44:31.635 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-19T07:44:31.710 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T07:44:31.869 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-19T07:44:31.995 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-19T07:44:32.044 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-19T07:44:32.363 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T07:44:32.387 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-19T07:44:32.747 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:32.821 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-19T07:44:32.873 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T07:44:33.024 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T07:44:33.206 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-19T07:44:33.450 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T07:44:33.665 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-19T07:44:33.785 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-12-19T07:44:33.792 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-19T07:44:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-19T07:44:34.178 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T07:44:34.313 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-19T07:44:34.338 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-19T07:44:34.698 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-19T07:44:34.831 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-12-19T07:44:35.048 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T07:44:35.199 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-19T07:44:35.457 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T07:44:35.651 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-19T07:44:35.875 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-19T07:44:36.093 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-19T07:44:36.226 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-19T07:44:36.253 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:36.377 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T07:44:36.646 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:36.770 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-19T07:44:36.797 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-19T07:44:37.221 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T07:44:37.281 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-19T07:44:37.347 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T07:44:37.416 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-19T07:44:37.473 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-19T07:44:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-19T07:44:37.907 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T07:44:37.962 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-19T07:44:38.300 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:38.364 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:38.451 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-19T07:44:38.489 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-19T07:44:38.615 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-19T07:44:39.050 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-19T07:44:39.401 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-19T07:44:39.760 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-19T07:44:39.886 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-19T07:44:40.229 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-19T07:44:40.273 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:40.399 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:40.558 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-19T07:44:40.597 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-19T07:44:40.974 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-19T07:44:41.026 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-19T07:44:41.380 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-19T07:44:41.402 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-19T07:44:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-19T07:44:41.770 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:41.894 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-19T07:44:41.895 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-19T07:44:42.295 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-19T07:44:42.313 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:42.439 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T07:44:42.730 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-19T07:44:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:42.856 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-19T07:44:43.007 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-19T07:44:43.025 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T07:44:43.400 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-19T07:44:43.453 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T07:44:43.751 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librados2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:43.877 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-19T07:44:44.709 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:44.832 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T07:44:44.958 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-12-19T07:44:45.109 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:45.235 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-19T07:44:45.603 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T07:44:45.665 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libsqlite3-mod-ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:45.737 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-19T07:44:45.825 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:45.951 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-19T07:44:46.240 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:46.403 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-19T07:44:46.408 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:46.672 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T07:44:46.672 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T07:44:46.842 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-19T07:44:47.230 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-12-19T07:44:47.355 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-19T07:44:47.500 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:47.660 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T07:44:47.765 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:47.891 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-19T07:44:48.011 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-19T07:44:48.351 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libradosstriper1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:48.429 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T07:44:48.477 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-19T07:44:48.873 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:48.886 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-19T07:44:48.999 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:49.125 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:49.293 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:49.418 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:49.553 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-19T07:44:49.963 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-19T07:44:50.117 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librbd1 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-19T07:44:50.368 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-12-19T07:44:50.372 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:50.528 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-19T07:44:50.531 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:50.657 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:50.662 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-19T07:44:50.783 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T07:44:51.022 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-12-19T07:44:51.189 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-19T07:44:51.651 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-modules-core (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:51.777 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:51.961 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:52.075 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T07:44:52.075 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-19T07:44:52.612 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2023-12-19T07:44:52.978 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs-dev (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:53.091 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2023-12-19T07:44:53.105 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-19T07:44:53.466 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-19T07:44:53.730 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2023-12-19T07:44:53.760 INFO:teuthology.orchestra.run.smithi143.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-19T07:44:53.840 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-19T07:44:53.962 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-19T07:44:54.326 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rados (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:54.469 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librgw2 (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:54.503 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-19T07:44:54.597 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephfs-top (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:54.750 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rbd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:54.874 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up rbd-fuse (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:55.042 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-19T07:44:55.410 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-19T07:44:55.480 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:55.738 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T07:44:55.738 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T07:44:55.769 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:55.895 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cephfs (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:56.021 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephfs-shell (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:56.181 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-19T07:44:56.598 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:56.765 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:57.076 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T07:44:57.076 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T07:44:57.326 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-common (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:58.034 INFO:teuthology.orchestra.run.smithi111.stdout:Adding group ceph....done 2023-12-19T07:44:58.301 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:58.516 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user ceph....done 2023-12-19T07:44:58.753 INFO:teuthology.orchestra.run.smithi111.stdout:Setting system user ceph properties....done 2023-12-19T07:44:58.795 INFO:teuthology.orchestra.run.smithi111.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-19T07:44:58.822 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-19T07:44:58.989 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-19T07:44:59.522 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-19T07:44:59.711 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:44:59.990 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T07:44:59.990 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T07:45:00.470 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephfs-mirror (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:00.729 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T07:45:00.729 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-19T07:45:00.836 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:01.148 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T07:45:01.148 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T07:45:01.546 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-test (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:01.672 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up radosgw (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:02.004 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:02.024 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T07:45:02.024 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-19T07:45:02.797 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T07:45:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T07:45:03.275 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-base (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:03.647 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:03.759 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-19T07:45:03.874 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:04.025 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:04.281 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T07:45:04.281 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T07:45:04.635 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mds (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:04.916 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T07:45:04.916 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-19T07:45:05.085 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:05.236 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:05.361 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:05.546 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:05.673 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T07:45:05.794 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:05.861 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T07:45:06.075 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T07:45:06.075 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-19T07:45:06.359 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T07:45:06.912 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-osd (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:07.717 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T07:45:07.717 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-19T07:45:08.573 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-k8sevents (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:08.699 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:08.883 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mon (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:09.147 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T07:45:09.147 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-19T07:45:10.051 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-cephadm (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:10.194 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:10.353 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-dashboard (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:10.513 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-volume (17.2.7-141-gc7312db0-1focal) ... 2023-12-19T07:45:10.673 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T07:45:10.833 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T07:45:11.334 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T07:45:11.519 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-12-19T07:45:12.312 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T07:45:12.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-12-19T07:45:12.442 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2023-12-19T07:45:12.648 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2023-12-19T07:45:12.649 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2023-12-19T07:45:12.854 INFO:teuthology.orchestra.run.smithi143.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-12-19T07:45:12.854 INFO:teuthology.orchestra.run.smithi143.stdout:flex is already the newest version (2.6.4-6.2). 2023-12-19T07:45:12.854 INFO:teuthology.orchestra.run.smithi143.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-12-19T07:45:12.854 INFO:teuthology.orchestra.run.smithi143.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-12-19T07:45:12.854 INFO:teuthology.orchestra.run.smithi143.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-12-19T07:45:12.855 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T07:45:12.855 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T07:45:12.855 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T07:45:12.856 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2023-12-19T07:45:12.856 INFO:teuthology.orchestra.run.smithi143.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-12-19T07:45:12.856 INFO:teuthology.orchestra.run.smithi143.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-12-19T07:45:12.856 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-12-19T07:45:12.856 INFO:teuthology.orchestra.run.smithi143.stdout: usb-modeswitch-data wpasupplicant 2023-12-19T07:45:12.858 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2023-12-19T07:45:12.858 INFO:teuthology.orchestra.run.smithi143.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-12-19T07:45:12.858 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-12-19T07:45:12.948 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2023-12-19T07:45:12.949 INFO:teuthology.orchestra.run.smithi143.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-12-19T07:45:12.949 INFO:teuthology.orchestra.run.smithi143.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-12-19T07:45:12.949 INFO:teuthology.orchestra.run.smithi143.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-12-19T07:45:12.949 INFO:teuthology.orchestra.run.smithi143.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-19T07:45:12.950 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2023-12-19T07:45:12.951 INFO:teuthology.orchestra.run.smithi143.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-12-19T07:45:13.020 INFO:teuthology.orchestra.run.smithi143.stdout:6 upgraded, 23 newly installed, 0 to remove and 256 not upgraded. 2023-12-19T07:45:13.020 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 10.4 MB of archives. 2023-12-19T07:45:13.021 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-12-19T07:45:13.021 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-12-19T07:45:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-12-19T07:45:13.181 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-12-19T07:45:13.182 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-12-19T07:45:13.190 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-12-19T07:45:13.191 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.20 [1,321 kB] 2023-12-19T07:45:13.208 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-12-19T07:45:13.209 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-12-19T07:45:13.210 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-12-19T07:45:13.212 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-12-19T07:45:13.212 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.7 [60.7 kB] 2023-12-19T07:45:13.218 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-12-19T07:45:13.219 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-12-19T07:45:13.221 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-12-19T07:45:13.221 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-12-19T07:45:13.238 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-12-19T07:45:13.239 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-12-19T07:45:13.249 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-12-19T07:45:13.256 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-12-19T07:45:13.256 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.20 [1,584 kB] 2023-12-19T07:45:13.274 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-12-19T07:45:13.275 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-12-19T07:45:13.284 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-12-19T07:45:13.296 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-12-19T07:45:13.315 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-12-19T07:45:13.318 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-12-19T07:45:13.318 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-12-19T07:45:13.319 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-12-19T07:45:13.319 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-12-19T07:45:13.744 INFO:teuthology.orchestra.run.smithi143.stdout:Preconfiguring packages ... 2023-12-19T07:45:13.918 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 10.4 MB in 0s (29.0 MB/s) 2023-12-19T07:45:14.358 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:14.362 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:14.553 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:16.202 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:16.578 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-12-19T07:45:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T07:45:17.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-12-19T07:45:17.521 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2023-12-19T07:45:17.736 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2023-12-19T07:45:17.737 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2023-12-19T07:45:17.936 INFO:teuthology.orchestra.run.smithi111.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-12-19T07:45:17.937 INFO:teuthology.orchestra.run.smithi111.stdout:flex is already the newest version (2.6.4-6.2). 2023-12-19T07:45:17.937 INFO:teuthology.orchestra.run.smithi111.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-12-19T07:45:17.937 INFO:teuthology.orchestra.run.smithi111.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-12-19T07:45:17.937 INFO:teuthology.orchestra.run.smithi111.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-12-19T07:45:17.937 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2023-12-19T07:45:17.938 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-19T07:45:17.938 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-19T07:45:17.938 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2023-12-19T07:45:17.938 INFO:teuthology.orchestra.run.smithi111.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-12-19T07:45:17.938 INFO:teuthology.orchestra.run.smithi111.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-12-19T07:45:17.939 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-12-19T07:45:17.939 INFO:teuthology.orchestra.run.smithi111.stdout: usb-modeswitch-data wpasupplicant 2023-12-19T07:45:17.940 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2023-12-19T07:45:17.940 INFO:teuthology.orchestra.run.smithi111.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-12-19T07:45:17.940 INFO:teuthology.orchestra.run.smithi111.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-12-19T07:45:18.027 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2023-12-19T07:45:18.027 INFO:teuthology.orchestra.run.smithi111.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-12-19T07:45:18.027 INFO:teuthology.orchestra.run.smithi111.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-12-19T07:45:18.028 INFO:teuthology.orchestra.run.smithi111.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-12-19T07:45:18.028 INFO:teuthology.orchestra.run.smithi111.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-19T07:45:18.029 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2023-12-19T07:45:18.030 INFO:teuthology.orchestra.run.smithi111.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-12-19T07:45:18.099 INFO:teuthology.orchestra.run.smithi111.stdout:6 upgraded, 23 newly installed, 0 to remove and 256 not upgraded. 2023-12-19T07:45:18.099 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 10.4 MB of archives. 2023-12-19T07:45:18.099 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-12-19T07:45:18.099 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-12-19T07:45:18.233 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-12-19T07:45:18.235 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-12-19T07:45:18.236 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-12-19T07:45:18.241 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-12-19T07:45:18.242 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.20 [1,321 kB] 2023-12-19T07:45:18.245 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:18.248 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:18.261 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-12-19T07:45:18.261 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-12-19T07:45:18.263 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-12-19T07:45:18.264 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-12-19T07:45:18.264 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.7 [60.7 kB] 2023-12-19T07:45:18.266 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-12-19T07:45:18.267 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-12-19T07:45:18.269 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-12-19T07:45:18.269 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-12-19T07:45:18.283 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-12-19T07:45:18.284 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-12-19T07:45:18.292 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-12-19T07:45:18.301 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-12-19T07:45:18.302 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.20 [1,584 kB] 2023-12-19T07:45:18.319 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-12-19T07:45:18.319 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-12-19T07:45:18.329 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-12-19T07:45:18.341 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-12-19T07:45:18.359 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-12-19T07:45:18.362 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-12-19T07:45:18.363 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-12-19T07:45:18.363 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-12-19T07:45:18.364 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-12-19T07:45:18.417 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:18.786 INFO:teuthology.orchestra.run.smithi111.stdout:Preconfiguring packages ... 2023-12-19T07:45:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:18.969 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 10.4 MB in 0s (32.0 MB/s) 2023-12-19T07:45:19.178 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:19.317 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:19.320 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:19.529 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:19.545 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:20.097 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:20.100 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:20.232 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T07:45:20.733 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:20.912 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T07:45:21.173 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:21.463 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T07:45:21.600 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.20) over (1.1.1f-1ubuntu2.16) ... 2023-12-19T07:45:22.154 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-19T07:45:22.176 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-12-19T07:45:22.220 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-12-19T07:45:22.581 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package dump. 2023-12-19T07:45:22.605 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-12-19T07:45:22.648 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking dump (0.4b46-6) ... 2023-12-19T07:45:22.985 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package indent. 2023-12-19T07:45:23.005 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:45:23.043 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T07:45:23.162 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:23.166 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:23.288 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:23.395 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-19T07:45:23.418 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-12-19T07:45:23.453 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T07:45:23.807 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-19T07:45:23.828 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.7_amd64.deb ... 2023-12-19T07:45:23.873 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T07:45:24.233 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-19T07:45:24.249 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:24.305 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-12-19T07:45:24.479 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-12-19T07:45:24.855 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:24.876 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:25.280 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-19T07:45:25.302 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:25.346 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:25.369 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 118685 files and directories currently installed.) 2023-12-19T07:45:25.372 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:25.512 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T07:45:25.691 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-19T07:45:25.710 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:25.749 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:26.078 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:26.127 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-19T07:45:26.148 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:26.193 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:26.208 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-12-19T07:45:26.604 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-19T07:45:26.625 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-12-19T07:45:26.671 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T07:45:26.760 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T07:45:26.905 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.20) over (1.1.1f-1ubuntu2.16) ... 2023-12-19T07:45:27.032 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-19T07:45:27.053 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T07:45:27.090 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-19T07:45:27.513 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-12-19T07:45:27.526 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-19T07:45:27.545 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:27.551 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-12-19T07:45:27.584 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:27.938 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package dump. 2023-12-19T07:45:27.959 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-12-19T07:45:27.962 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-19T07:45:27.983 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:28.004 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking dump (0.4b46-6) ... 2023-12-19T07:45:28.028 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:28.331 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-19T07:45:28.333 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package indent. 2023-12-19T07:45:28.353 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T07:45:28.354 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-12-19T07:45:28.389 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:28.399 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T07:45:28.753 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-19T07:45:28.775 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-12-19T07:45:28.818 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T07:45:28.959 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-19T07:45:28.980 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-12-19T07:45:29.025 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-12-19T07:45:29.406 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package modemmanager. 2023-12-19T07:45:29.429 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:29.484 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:29.499 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-19T07:45:29.520 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.7_amd64.deb ... 2023-12-19T07:45:29.557 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T07:45:29.869 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-19T07:45:29.890 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-12-19T07:45:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:30.262 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package wpasupplicant. 2023-12-19T07:45:30.284 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-12-19T07:45:30.328 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T07:45:30.372 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-19T07:45:30.394 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:30.438 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:30.758 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-19T07:45:30.776 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:30.816 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:30.926 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package network-manager. 2023-12-19T07:45:30.948 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T07:45:31.064 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:31.194 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-19T07:45:31.216 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:31.260 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:31.680 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-19T07:45:31.702 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-12-19T07:45:31.780 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T07:45:31.900 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ppp. 2023-12-19T07:45:31.922 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-12-19T07:45:32.158 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-19T07:45:32.180 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T07:45:32.224 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:32.396 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T07:45:32.661 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-19T07:45:32.675 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:32.719 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:32.823 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package pptp-linux. 2023-12-19T07:45:32.842 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-12-19T07:45:32.880 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-12-19T07:45:33.105 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-19T07:45:33.131 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:33.205 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:33.301 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-19T07:45:33.323 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-12-19T07:45:33.403 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-12-19T07:45:33.491 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-19T07:45:33.513 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-19T07:45:33.574 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:33.729 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-19T07:45:33.752 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-12-19T07:45:33.795 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-12-19T07:45:34.119 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-19T07:45:34.138 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-12-19T07:45:34.177 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-12-19T07:45:34.249 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-19T07:45:34.272 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-12-19T07:45:34.364 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T07:45:34.648 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package modemmanager. 2023-12-19T07:45:34.667 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-12-19T07:45:34.699 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:34.731 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:35.080 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-12-19T07:45:35.240 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T07:45:35.366 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T07:45:35.437 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package wpasupplicant. 2023-12-19T07:45:35.457 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-12-19T07:45:35.495 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T07:45:35.846 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T07:45:35.846 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T07:45:36.142 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package network-manager. 2023-12-19T07:45:36.163 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-12-19T07:45:36.256 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:36.744 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:36.904 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-12-19T07:45:37.030 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:37.091 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ppp. 2023-12-19T07:45:37.113 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-12-19T07:45:37.157 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-12-19T07:45:37.332 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:37.450 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T07:45:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T07:45:38.005 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package pptp-linux. 2023-12-19T07:45:38.027 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-12-19T07:45:38.071 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-12-19T07:45:38.542 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-19T07:45:38.561 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-12-19T07:45:38.631 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-12-19T07:45:38.725 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-12-19T07:45:38.928 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-19T07:45:38.947 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-12-19T07:45:38.986 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-12-19T07:45:39.123 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up dump (0.4b46-6) ... 2023-12-19T07:45:39.230 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-19T07:45:39.333 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:39.418 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-19T07:45:39.435 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-12-19T07:45:39.459 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T07:45:39.492 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T07:45:39.585 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T07:45:39.754 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T07:45:39.861 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:39.947 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T07:45:40.081 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:40.241 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:40.262 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-12-19T07:45:40.367 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-12-19T07:45:40.421 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-12-19T07:45:40.547 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-12-19T07:45:40.560 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:40.686 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:41.031 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T07:45:41.031 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout:which means they are currently configured by ifupdown: 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout:- enp3s0f1 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout:remove their configuration from /etc/network/interfaces. 2023-12-19T07:45:41.813 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:45:41.941 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-12-19T07:45:42.034 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-19T07:45:42.092 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-12-19T07:45:42.218 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-19T07:45:42.344 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-12-19T07:45:42.512 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:42.599 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-19T07:45:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-12-19T07:45:43.562 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-19T07:45:43.891 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-12-19T07:45:44.327 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up dump (0.4b46-6) ... 2023-12-19T07:45:44.436 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-19T07:45:44.448 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:44.504 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:44.583 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-12-19T07:45:44.663 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.7) ... 2023-12-19T07:45:44.751 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:44.789 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-12-19T07:45:44.885 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:44.916 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-12-19T07:45:45.011 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:45.125 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-12-19T07:45:45.163 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:45.251 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:45.411 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:45.545 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-12-19T07:45:45.738 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-12-19T07:45:45.877 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T07:45:45.877 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T07:45:46.857 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-12-19T07:45:46.983 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout:which means they are currently configured by ifupdown: 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout:- enp3s0f1 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout:remove their configuration from /etc/network/interfaces. 2023-12-19T07:45:47.074 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:45:47.117 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T07:45:47.311 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T07:45:47.318 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-19T07:45:47.821 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T07:45:47.876 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-19T07:45:48.848 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-19T07:45:50.151 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-12-19T07:45:50.286 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-12-19T07:45:50.478 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-12-19T07:45:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:50.730 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-12-19T07:45:50.890 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-12-19T07:45:51.216 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T07:45:51.216 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-19T07:45:52.201 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-12-19T07:45:52.318 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-19T07:45:52.453 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-19T07:45:52.655 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-19T07:45:53.122 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-19T07:45:54.815 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-12-19T07:45:54.923 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-19T07:45:56.463 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T07:45:56.472 DEBUG:teuthology.parallel:result is None 2023-12-19T07:45:59.999 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-12-19T07:46:00.108 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-19T07:46:01.354 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-19T07:46:01.357 DEBUG:teuthology.parallel:result is None 2023-12-19T07:46:01.358 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T07:46:01.577 DEBUG:teuthology.orchestra.run.smithi111:> dpkg-query -W -f '${Version}' ceph 2023-12-19T07:46:01.614 INFO:teuthology.orchestra.run.smithi111.stdout:17.2.7-141-gc7312db0-1focal 2023-12-19T07:46:01.614 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-141-gc7312db0-1focal 2023-12-19T07:46:01.615 INFO:teuthology.task.install:The correct ceph version 17.2.7-141-gc7312db0-1focal is installed. 2023-12-19T07:46:01.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=c7312db04d305d59a3441862c84d6d4da3817eb0 2023-12-19T07:46:01.756 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2023-12-19T07:46:01.793 INFO:teuthology.orchestra.run.smithi143.stdout:17.2.7-141-gc7312db0-1focal 2023-12-19T07:46:01.793 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-141-gc7312db0-1focal 2023-12-19T07:46:01.793 INFO:teuthology.task.install:The correct ceph version 17.2.7-141-gc7312db0-1focal is installed. 2023-12-19T07:46:01.795 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-19T07:46:01.795 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:01.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T07:46:01.830 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:01.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T07:46:01.872 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-19T07:46:01.872 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:01.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T07:46:01.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T07:46:01.995 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:01.996 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T07:46:02.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T07:46:02.108 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-19T07:46:02.108 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:02.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T07:46:02.146 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T07:46:02.222 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:02.223 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T07:46:02.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T07:46:02.332 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-19T07:46:02.332 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:02.332 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T07:46:02.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T07:46:02.450 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:02.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T07:46:02.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T07:46:02.570 INFO:teuthology.run_tasks:Running task ceph... 2023-12-19T07:46:02.651 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-19T07:46:02.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2023-12-19T07:46:02.658 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2023-12-19T07:46:02.690 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-19T07:46:02.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-19T07:46:02.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-19T07:46:02.771 INFO:tasks.ceph:Creating extra log directories... 2023-12-19T07:46:02.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-19T07:46:02.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-19T07:46:02.865 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-19T07:46:02.865 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-19T07:46:02.865 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-19T07:46:02.865 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:02.865 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-19T07:46:02.898 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-19T07:46:03.030 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:03.030 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-19T07:46:03.051 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-19T07:46:03.190 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-19T07:46:03.190 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-19T07:46:03.190 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886', 'branch': 'rishabh-mdscap-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': None, 'first_in_suite': False, 'job_id': '7495886', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'rishabh-mdscap-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-mdscap-quincy', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4463, 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'sleep_before_teardown': 0, 'subset': '344/512', 'suite': 'fs', 'suite_branch': 'rishabh-mdscap-quincy', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c7312db04d305d59a3441862c84d6d4da3817eb0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'targets': {'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'cd68252d7b4d7802f3cae7f5945b38e9b20dad4c'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7312db04d305d59a3441862c84d6d4da3817eb0', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-19_05:13:44', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-19T07:46:03.191 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-19T07:46:03.198 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-19T07:46:03.220 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2023-12-19T07:46:03.260 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2023-12-19T07:46:03.299 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:03.299 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T07:46:03.350 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T07:46:03.350 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 497 Links: 1 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-12-19 07:40:46.699661935 +0000 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-12-19 07:40:46.695662028 +0000 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-12-19 07:40:46.695662028 +0000 2023-12-19T07:46:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-12-19T07:46:03.413 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T07:46:03.489 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-12-19T07:46:03.489 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-12-19T07:46:03.489 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000700688 s, 731 kB/s 2023-12-19T07:46:03.491 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T07:46:03.559 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 503 Links: 1 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-12-19 07:40:47.907634077 +0000 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-12-19 07:40:47.431645054 +0000 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-12-19 07:40:47.431645054 +0000 2023-12-19T07:46:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-12-19T07:46:03.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T07:46:03.695 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-12-19T07:46:03.695 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-12-19T07:46:03.695 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000704552 s, 727 kB/s 2023-12-19T07:46:03.697 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T07:46:03.762 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 509 Links: 1 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-12-19 07:40:48.679616274 +0000 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-12-19 07:40:48.211627067 +0000 2023-12-19T07:46:03.827 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-12-19 07:40:48.211627067 +0000 2023-12-19T07:46:03.828 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-12-19T07:46:03.828 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T07:46:03.902 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-12-19T07:46:03.902 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-12-19T07:46:03.902 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000656529 s, 780 kB/s 2023-12-19T07:46:03.905 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T07:46:03.976 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 516 Links: 1 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-12-19 07:40:49.463598194 +0000 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-12-19 07:40:48.963609725 +0000 2023-12-19T07:46:04.035 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-12-19 07:40:48.963609725 +0000 2023-12-19T07:46:04.036 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-12-19T07:46:04.036 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T07:46:04.114 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-12-19T07:46:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-12-19T07:46:04.115 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000671625 s, 762 kB/s 2023-12-19T07:46:04.117 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T07:46:04.180 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:04.180 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:04.180 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T07:46:04.198 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T07:46:04.198 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2023-12-19T07:46:04.263 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T07:46:04.263 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:04.263 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 498 Links: 1 2023-12-19T07:46:04.263 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:04.264 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-12-19 07:40:45.147749350 +0000 2023-12-19T07:46:04.264 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-12-19 07:40:44.667760536 +0000 2023-12-19T07:46:04.264 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-12-19 07:40:44.667760536 +0000 2023-12-19T07:46:04.264 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-12-19T07:46:04.264 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T07:46:04.338 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-12-19T07:46:04.339 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-12-19T07:46:04.339 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000699198 s, 732 kB/s 2023-12-19T07:46:04.341 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T07:46:04.408 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 505 Links: 1 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-12-19 07:40:45.447742360 +0000 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-12-19 07:40:45.443742452 +0000 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-12-19 07:40:45.443742452 +0000 2023-12-19T07:46:04.470 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-12-19T07:46:04.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T07:46:04.548 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-12-19T07:46:04.548 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-12-19T07:46:04.548 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000657858 s, 778 kB/s 2023-12-19T07:46:04.550 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T07:46:04.616 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 510 Links: 1 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-12-19 07:40:46.687713464 +0000 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-12-19 07:40:46.239723903 +0000 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-12-19 07:40:46.239723903 +0000 2023-12-19T07:46:04.681 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-12-19T07:46:04.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T07:46:04.755 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-12-19T07:46:04.755 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-12-19T07:46:04.755 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000482348 s, 1.1 MB/s 2023-12-19T07:46:04.757 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T07:46:04.827 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 516 Links: 1 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-12-19 07:40:47.011705914 +0000 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-12-19 07:40:47.007706007 +0000 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-12-19 07:40:47.007706007 +0000 2023-12-19T07:46:04.890 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-12-19T07:46:04.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T07:46:04.962 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-12-19T07:46:04.962 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-12-19T07:46:04.962 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000707872 s, 723 kB/s 2023-12-19T07:46:04.964 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T07:46:05.030 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:05.030 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-19T07:46:05.030 INFO:tasks.ceph:Generating config... 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] debug client = 20 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] debug ms = 1 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] pid file = 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-19T07:46:05.031 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-19T07:46:05.031 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mgr] debug client = 20 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-19T07:46:05.032 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-19T07:46:05.033 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-19T07:46:05.033 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-19T07:46:05.034 INFO:tasks.ceph:Setting up mon.a... 2023-12-19T07:46:05.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-19T07:46:05.093 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.keyring 2023-12-19T07:46:05.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-19T07:46:05.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T07:46:05.191 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.111'), ('mon.b', '172.21.15.143'), ('mon.c', '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]')] 2023-12-19T07:46:05.191 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.111,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-19T07:46:05.192 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:05.192 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T07:46:05.254 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.111 --add b 172.21.15.143 --addv c '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: generated fsid 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = octopus 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 0 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:last_changed 2023-12-19T07:46:05.385692+0000 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-12-19T07:46:05.385692+0000 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:min_mon_release 15 (octopus) 2023-12-19T07:46:05.365 INFO:teuthology.orchestra.run.smithi111.stdout:election_strategy: 1 2023-12-19T07:46:05.366 INFO:teuthology.orchestra.run.smithi111.stdout:0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2023-12-19T07:46:05.366 INFO:teuthology.orchestra.run.smithi111.stdout:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2023-12-19T07:46:05.366 INFO:teuthology.orchestra.run.smithi111.stdout:2: [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] mon.c 2023-12-19T07:46:05.366 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-19T07:46:05.368 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T07:46:05.381 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4ae666be-184d-45ea-a875-cac9a231c35b... 2023-12-19T07:46:05.382 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T07:46:05.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.512 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-12-19T07:46:05.513 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-12-19T07:46:05.514 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.111,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.515 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-12-19T07:46:05.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 20 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 20 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 10 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-12-19T07:46:05.517 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug finisher = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-12-19T07:46:05.518 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T07:46:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T07:46:05.520 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = "" 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2023-12-19T07:46:05.521 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2023-12-19T07:46:05.522 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T07:46:05.523 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.524 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.111,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.525 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.526 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2023-12-19T07:46:05.527 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2023-12-19T07:46:05.528 INFO:teuthology.orchestra.run.smithi143.stdout: debug finisher = 20 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T07:46:05.529 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2023-12-19T07:46:05.530 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = "" 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.d] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2023-12-19T07:46:05.531 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-12-19T07:46:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-12-19T07:46:05.542 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-19T07:46:05.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-19T07:46:05.653 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-19T07:46:05.653 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:05.653 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-19T07:46:05.670 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:05.670 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-19T07:46:05.730 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:46:05.730 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:05.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-19T07:46:05.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T07:46:05.828 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:05.829 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:05.893 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:46:05.893 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:05.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-19T07:46:05.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T07:46:05.958 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:05.958 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:06.015 INFO:tasks.ceph:Setting up mon nodes... 2023-12-19T07:46:06.016 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-19T07:46:06.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-19T07:46:06.105 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-19T07:46:06.112 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T07:46:06.200 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T07:46:06.208 INFO:tasks.ceph:Setting up mds nodes... 2023-12-19T07:46:06.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-19T07:46:06.295 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-19T07:46:06.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-19T07:46:06.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-19T07:46:06.468 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-19T07:46:06.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-19T07:46:06.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-19T07:46:06.600 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-19T07:46:06.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-19T07:46:06.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-19T07:46:06.774 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-19T07:46:06.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-19T07:46:06.809 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-19T07:46:06.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-19T07:46:06.874 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-19T07:46:06.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-12-19T07:46:06.966 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-12-19T07:46:06.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-19T07:46:07.062 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-19T07:46:07.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-12-19T07:46:07.151 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-12-19T07:46:07.181 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-19T07:46:07.181 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-19T07:46:07.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-19T07:46:07.218 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:07.218 INFO:tasks.ceph:role: osd.0 2023-12-19T07:46:07.218 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:46:07.218 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-19T07:46:07.809 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:07.809 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:07.810 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:07.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-12-19T07:46:07.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-19T07:46:07.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-19T07:46:07.966 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:07.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-19T07:46:08.051 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:08.052 INFO:tasks.ceph:role: osd.1 2023-12-19T07:46:08.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:46:08.052 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-19T07:46:08.569 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:08.569 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:08.569 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:08.570 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:08.574 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-12-19T07:46:08.574 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-19T07:46:08.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-19T07:46:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:08.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-19T07:46:08.757 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:08.757 INFO:tasks.ceph:role: osd.2 2023-12-19T07:46:08.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:46:08.758 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:09.328 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:09.331 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-12-19T07:46:09.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-19T07:46:09.458 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-19T07:46:09.481 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:09.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-19T07:46:09.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:09.557 INFO:tasks.ceph:role: osd.3 2023-12-19T07:46:09.558 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:46:09.558 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-19T07:46:10.139 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:10.140 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:10.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-12-19T07:46:10.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-19T07:46:10.268 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-19T07:46:10.291 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:10.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:10.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.465+0000 7fd1006da3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-19T07:46:10.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.469+0000 7fd1006da3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-19T07:46:10.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.469+0000 7fd1006da3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T07:46:10.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.469+0000 7fd1006da3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T07:46:10.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.469+0000 7fd1006da3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-19T07:46:10.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:10.473+0000 7fd1006da3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-19T07:46:11.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:11.045+0000 7fd1006da3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:11.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:11.841+0000 7fd1006da3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:12.629+0000 7fd1006da3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:13.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-19T07:46:13.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:13.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.693+0000 7fec7a1c03c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-19T07:46:13.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.693+0000 7fec7a1c03c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-19T07:46:13.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.693+0000 7fec7a1c03c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T07:46:13.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.693+0000 7fec7a1c03c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T07:46:13.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.693+0000 7fec7a1c03c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-19T07:46:13.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:13.697+0000 7fec7a1c03c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-19T07:46:14.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:14.265+0000 7fec7a1c03c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:15.061+0000 7fec7a1c03c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:15.857+0000 7fec7a1c03c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:16.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-19T07:46:16.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:16.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.917+0000 7fa1765a13c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-19T07:46:16.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.917+0000 7fa1765a13c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-19T07:46:16.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.921+0000 7fa1765a13c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T07:46:16.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.921+0000 7fa1765a13c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T07:46:16.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.921+0000 7fa1765a13c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-19T07:46:16.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:16.921+0000 7fa1765a13c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-19T07:46:17.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:17.505+0000 7fa1765a13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:18.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:18.293+0000 7fa1765a13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:19.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:19.097+0000 7fa1765a13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:19.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-19T07:46:20.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:20.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.165+0000 7f1d077143c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-19T07:46:20.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.169+0000 7f1d077143c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-19T07:46:20.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.169+0000 7f1d077143c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T07:46:20.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.169+0000 7f1d077143c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T07:46:20.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.169+0000 7f1d077143c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-19T07:46:20.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.169+0000 7f1d077143c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-19T07:46:20.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:20.741+0000 7f1d077143c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:21.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:21.525+0000 7f1d077143c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:22.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:22.321+0000 7f1d077143c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:23.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-19T07:46:23.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-19T07:46:23.247 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:23.247 INFO:tasks.ceph:role: osd.4 2023-12-19T07:46:23.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:46:23.248 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:23.854 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:23.860 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-12-19T07:46:23.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-19T07:46:23.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-19T07:46:24.015 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:24.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-19T07:46:24.098 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:24.098 INFO:tasks.ceph:role: osd.5 2023-12-19T07:46:24.098 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:46:24.099 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:24.625 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:24.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-12-19T07:46:24.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-19T07:46:24.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-19T07:46:24.744 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:24.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-19T07:46:24.826 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:24.826 INFO:tasks.ceph:role: osd.6 2023-12-19T07:46:24.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:46:24.826 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:25.406 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:25.407 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:25.407 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:25.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-12-19T07:46:25.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-19T07:46:25.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-19T07:46:25.585 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:25.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-19T07:46:25.665 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-19T07:46:25.665 INFO:tasks.ceph:role: osd.7 2023-12-19T07:46:25.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:46:25.665 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T07:46:26.248 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T07:46:26.252 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-12-19T07:46:26.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-19T07:46:26.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-19T07:46:26.447 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2023-12-19T07:46:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:46:26.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:26.602 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.618+0000 7f359a6e63c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-19T07:46:26.603 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.618+0000 7f359a6e63c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-19T07:46:26.603 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.622+0000 7f359a6e63c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T07:46:26.603 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.622+0000 7f359a6e63c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T07:46:26.603 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.622+0000 7f359a6e63c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-19T07:46:26.606 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:26.622+0000 7f359a6e63c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-19T07:46:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:27.198+0000 7f359a6e63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:27.982 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:27.998+0000 7f359a6e63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:28.766 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:28.782+0000 7f359a6e63c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:29.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-19T07:46:29.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:29.819 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.834+0000 7f951eef43c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-19T07:46:29.820 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.838+0000 7f951eef43c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-19T07:46:29.820 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.838+0000 7f951eef43c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T07:46:29.820 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.838+0000 7f951eef43c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T07:46:29.820 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.838+0000 7f951eef43c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-19T07:46:29.822 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:29.838+0000 7f951eef43c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-19T07:46:30.397 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:30.414+0000 7f951eef43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:31.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:31.198+0000 7f951eef43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:31.974 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:31.990+0000 7f951eef43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:32.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-19T07:46:32.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:33.019 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.034+0000 7f7ae84dc3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-19T07:46:33.020 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.038+0000 7f7ae84dc3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-19T07:46:33.020 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.038+0000 7f7ae84dc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T07:46:33.020 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.038+0000 7f7ae84dc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T07:46:33.021 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.038+0000 7f7ae84dc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-19T07:46:33.023 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.038+0000 7f7ae84dc3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-19T07:46:33.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:46:33.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:46:33.594 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:33.610+0000 7f7ae84dc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:34.396 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:34.414+0000 7f7ae84dc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:35.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:35.202+0000 7f7ae84dc3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:36.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-19T07:46:36.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:36.235 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.254+0000 7f1f892de3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-19T07:46:36.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.254+0000 7f1f892de3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-19T07:46:36.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.254+0000 7f1f892de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T07:46:36.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.254+0000 7f1f892de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T07:46:36.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.254+0000 7f1f892de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-19T07:46:36.239 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.258+0000 7f1f892de3c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-19T07:46:36.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:36.830+0000 7f1f892de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:37.602 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:37.618+0000 7f1f892de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:38.386 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:38.401+0000 7f1f892de3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-19T07:46:39.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-19T07:46:39.316 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-19T07:46:39.317 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-19T07:46:39.351 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-19T07:46:39.429 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-19T07:46:39.511 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-19T07:46:39.593 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-19T07:46:39.670 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.670 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-19T07:46:39.746 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:39.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-19T07:46:39.826 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:39.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-19T07:46:39.859 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:39.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-19T07:46:39.939 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:39.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-19T07:46:40.019 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-19T07:46:40.095 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-19T07:46:40.173 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.173 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-19T07:46:40.252 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-19T07:46:40.330 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:40.330 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-19T07:46:40.349 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:40.350 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-12-19T07:46:40.416 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.416 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-19T07:46:40.435 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:40.435 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-12-19T07:46:40.497 INFO:tasks.ceph:Adding keys to all mons... 2023-12-19T07:46:40.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-19T07:46:40.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-19T07:46:40.547 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw== 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ== 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c] 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw== 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2023-12-19T07:46:40.584 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q== 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ== 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ== 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw== 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw== 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2023-12-19T07:46:40.585 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[client.2] 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ== 2023-12-19T07:46:40.586 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2023-12-19T07:46:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg== 2023-12-19T07:46:40.587 INFO:teuthology.orchestra.run.smithi111.stdout:[client.3] 2023-12-19T07:46:40.587 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A== 2023-12-19T07:46:40.588 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.y] 2023-12-19T07:46:40.588 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw== 2023-12-19T07:46:40.588 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2023-12-19T07:46:40.588 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw== 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2023-12-19T07:46:40.589 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.d] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.6] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.7] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q== 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2023-12-19T07:46:40.590 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA== 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout:[client.2] 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ== 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout:[client.1] 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg== 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout:[client.3] 2023-12-19T07:46:40.591 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A== 2023-12-19T07:46:40.592 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T07:46:40.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T07:46:40.708 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:40.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:40.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:40.801 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:40.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:40.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:40.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:40.964 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T07:46:41.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T07:46:41.274 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:41.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:41.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:41.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-19T07:46:41.439 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.446 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.516 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T07:46:41.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:41.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:41.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:41.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:41.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:41.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:42.006 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:42.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T07:46:42.094 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-19T07:46:42.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-19T07:46:42.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T07:46:42.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-19T07:46:42.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-19T07:46:42.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T07:46:42.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-19T07:46:42.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-19T07:46:42.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T07:46:42.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-19T07:46:42.751 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:42.757 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-19T07:46:42.817 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-19T07:46:42.818 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-19T07:46:42.818 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-19T07:46:42.825 INFO:tasks.ceph.mon.a:Started 2023-12-19T07:46:42.825 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-19T07:46:42.825 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-19T07:46:42.866 INFO:tasks.ceph.mon.b:Started 2023-12-19T07:46:42.866 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-19T07:46:42.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-19T07:46:42.870 INFO:tasks.ceph.mon.c:Started 2023-12-19T07:46:42.870 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-19T07:46:42.871 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-19T07:46:42.871 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-19T07:46:42.878 INFO:tasks.ceph.mgr.y:Started 2023-12-19T07:46:42.878 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-19T07:46:42.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-19T07:46:42.883 INFO:tasks.ceph.mgr.x:Started 2023-12-19T07:46:42.883 INFO:tasks.ceph:Setting crush tunables to default 2023-12-19T07:46:42.884 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd crush tunables default 2023-12-19T07:46:43.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.049+0000 7faa5e3c5700 1 Processor -- start 2023-12-19T07:46:43.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.053+0000 7faa5e3c5700 1 -- start start 2023-12-19T07:46:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.053+0000 7faa5e3c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.053+0000 7faa5e3c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5806d5e0 con 0x7faa580ca2c0 2023-12-19T07:46:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.053+0000 7faa5e3c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa580cbdf0 con 0x7faa580cb970 2023-12-19T07:46:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.053+0000 7faa5e3c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa580cc0d0 con 0x7faa58158e20 2023-12-19T07:46:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa5c962700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7faa580ca2c0 0x7faa5806ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36104/0 (socket says 172.21.15.111:36104) 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 learned_addr learned my addr 172.21.15.111:0/409249760 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa577fe700 1 --1- 172.21.15.111:0/409249760 >> v1:172.21.15.143:6789/0 conn(0x7faa580cb970 0x7faa5806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:52872/0 (socket says 172.21.15.111:52872) 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:43.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:43.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:43.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:43.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:43.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:43.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:43.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:44.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:44.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:44.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:44.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:44.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:44.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:44.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:44.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:44.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:44.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:46:46.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 msgr2=0x7faa580ce5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:46.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa58158e20 0x7faa580ce5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:46.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v1:172.21.15.143:6789/0 conn(0x7faa580cb970 legacy=0x7faa5806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:46.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v1:172.21.15.111:6789/0 conn(0x7faa580ca2c0 legacy=0x7faa5806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:46.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:46.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:46.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa580cc0d0 con 0x7faa44006200 2023-12-19T07:46:46.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa44008d50 con 0x7faa44003760 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:46.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:46.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:47.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:47.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:47.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:47.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:47.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:46:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.053+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 msgr2=0x7faa44005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.053+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003760 0x7faa44005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 msgr2=0x7faa440086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44006200 0x7faa440086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 reap_dead start 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa580cc3b0 con 0x7faa44006be0 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa580cc0d0 con 0x7faa44009100 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa44008d50 con 0x7faa44003ba0 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:49.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa57fff700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:50.469+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:46:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 msgr2=0x7faa440066a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.143:3300/0 conn(0x7faa44003ba0 0x7faa440066a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 msgr2=0x7faa4400b4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> v2:172.21.15.111:3300/0 conn(0x7faa44009100 0x7faa4400b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 >> v1:172.21.15.111:6789/0 conn(0x7faa44006be0 legacy=0x7faa44007080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa440089e0 con 0x7faa4400dc50 2023-12-19T07:46:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa55ffb700 1 -- 172.21.15.111:0/409249760 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa580cc0d0 con 0x7faa44011b40 2023-12-19T07:46:52.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:52.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:52.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:52.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:52.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.057+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:46:52.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:52.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:52.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.261+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:52.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:52.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:46:52.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:52.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-19T07:46:52.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.665+0000 7faa577fe700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-19T07:46:52.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T07:46:52.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:52.665+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:46:53.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.257+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960092053 0 0) 0x7faa440089e0 con 0x7faa4400dc50 2023-12-19T07:46:53.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.257+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40003570 con 0x7faa4400dc50 2023-12-19T07:46:53.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756526621 0 0) 0x7faa40003570 con 0x7faa4400dc50 2023-12-19T07:46:53.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 msgr2=0x7faa44013f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa44011b40 0x7faa44013f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44015470 con 0x7faa4400dc50 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7faa480176f0 con 0x7faa4400dc50 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 >> v1:172.21.15.143:6789/0 conn(0x7faa4400dc50 legacy=0x7faa4400e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 reap_dead start 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40003a60 0x7faa40003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 0x7faa40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 0x7faa4000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa580cc0d0 con 0x7faa40005230 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4000c200 con 0x7faa40004810 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4000c3c0 con 0x7faa40003a60 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa57fff700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40003a60 0x7faa40003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 0x7faa40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 0x7faa4000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40003a60 msgr2=0x7faa40003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40003a60 0x7faa40003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 msgr2=0x7faa40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 0x7faa40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.261+0000 7faa5c962700 1 -- 172.21.15.111:0/409249760 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40003570 con 0x7faa40005230 2023-12-19T07:46:53.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa577fe700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 0x7faa40004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:53.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5c962700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 0x7faa4000ba60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa580cdc10 tx=0x7faa50001850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacc8c9a8b17773 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:53.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000c800 con 0x7faa40005230 2023-12-19T07:46:53.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5000c980 con 0x7faa40005230 2023-12-19T07:46:53.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa56ffd700 1 -- 172.21.15.111:0/409249760 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000cd40 con 0x7faa40005230 2023-12-19T07:46:53.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 -- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 msgr2=0x7faa4000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 0x7faa4000ba60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa580cdc10 tx=0x7faa50001850 comp rx=0 tx=0).stop 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 -- 172.21.15.111:0/409249760 shutdown_connections 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40004810 0x7faa40004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40005230 0x7faa4000ba60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/409249760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40003a60 0x7faa40003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 -- 172.21.15.111:0/409249760 >> 172.21.15.111:0/409249760 conn(0x7faa58067bb0 msgr2=0x7faa580d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 -- 172.21.15.111:0/409249760 shutdown_connections 2023-12-19T07:46:53.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.265+0000 7faa5e3c5700 1 -- 172.21.15.111:0/409249760 wait complete. 2023-12-19T07:46:53.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 Processor -- start 2023-12-19T07:46:53.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 -- start start 2023-12-19T07:46:53.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 0x7faa580b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 0x7faa580c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:53.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa580b56a0 con 0x7faa40003a60 2023-12-19T07:46:53.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5815c750 con 0x7faa40005230 2023-12-19T07:46:53.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa5e3c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5815ca30 con 0x7faa40004810 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa57fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.269+0000 7faa57fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59778/0 (socket says 172.21.15.111:59778) 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa577fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 0x7faa580b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa5c962700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 0x7faa580c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 -- 172.21.15.111:0/2640133771 learned_addr learned my addr 172.21.15.111:0/2640133771 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 -- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 msgr2=0x7faa580b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 0x7faa580b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 -- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 msgr2=0x7faa580c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 0x7faa580c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 -- 172.21.15.111:0/2640133771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5815cd10 con 0x7faa40003a60 2023-12-19T07:46:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa577fe700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 0x7faa580b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa5c962700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 0x7faa580c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa57fff700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa48006b00 tx=0x7faa48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48016d20 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4801e040 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa480256b0 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5815cff0 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa58155150 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7faa48025ae0 con 0x7faa40003a60 2023-12-19T07:46:53.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7faa48005c30 con 0x7faa40003a60 2023-12-19T07:46:53.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.273+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa58071120 con 0x7faa40003a60 2023-12-19T07:46:53.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.277+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7faa4804c6d0 con 0x7faa40003a60 2023-12-19T07:46:53.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.357+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7faa580cc690 con 0x7faa40003a60 2023-12-19T07:46:53.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.361+0000 7faa3ffff700 1 -- 172.21.15.111:0/2640133771 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7faa580cc690 con 0x7faa40003a60 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 msgr2=0x7faa58154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faa48006b00 tx=0x7faa48008040 comp rx=0 tx=0).stop 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 shutdown_connections 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa40005230 0x7faa580c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa40004810 0x7faa580b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 --2- 172.21.15.111:0/2640133771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40003a60 0x7faa58154c10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 >> 172.21.15.111:0/2640133771 conn(0x7faa58067bb0 msgr2=0x7faa5806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:53.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 shutdown_connections 2023-12-19T07:46:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:53.365+0000 7faa5e3c5700 1 -- 172.21.15.111:0/2640133771 wait complete. 2023-12-19T07:46:53.348 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2023-12-19T07:46:53.370 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-19T07:46:53.370 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:53.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-19T07:46:53.402 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:53.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-19T07:46:53.470 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:53.485+0000 7fde3d5fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:46:53.471 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:53.471 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-19T07:46:53.547 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-19T07:46:53.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-19T07:46:53.624 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:53.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-19T07:46:53.648 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:53.665+0000 7faf0dbc7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:46:53.657 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:53.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-19T07:46:53.731 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:53.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-19T07:46:53.807 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-12-19T07:46:53.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-19T07:46:53.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 176b55d9-24fe-41e0-8a08-156ec028a881 0 2023-12-19T07:46:54.071 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 Processor -- start 2023-12-19T07:46:54.072 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 -- start start 2023-12-19T07:46:54.073 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c900cab10 0x7f9c900ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.073 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 0x7f9c9006b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.073 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c900c7450 con 0x7f9c9015d410 2023-12-19T07:46:54.073 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9006bea0 con 0x7f9c900cab10 2023-12-19T07:46:54.074 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c980f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900c6e00 con 0x7f9c9006b1b0 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c900cab10 0x7f9c900ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c900cab10 0x7f9c900ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:38630/0 (socket says 172.21.15.143:38630) 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 -- 172.21.15.143:0/3586442307 learned_addr learned my addr 172.21.15.143:0/3586442307 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9668d700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 0x7f9c9006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 -- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 msgr2=0x7f9c9006b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 0x7f9c9006b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 -- 172.21.15.143:0/3586442307 >> v1:172.21.15.111:6789/0 conn(0x7f9c9015d410 legacy=0x7f9c900ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 -- 172.21.15.143:0/3586442307 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900c70e0 con 0x7f9c900cab10 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9668d700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 0x7f9c9006b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:46:54.075 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.089+0000 7f9c9568b700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c900cab10 0x7f9c900ab6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c800099b0 tx=0x7f9c80009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6886341d6377dc7d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c94e8a700 1 -- 172.21.15.143:0/3586442307 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000b860 con 0x7f9c900cab10 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c94e8a700 1 -- 172.21.15.143:0/3586442307 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000b9c0 con 0x7f9c900cab10 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c94e8a700 1 -- 172.21.15.143:0/3586442307 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8000bd60 con 0x7f9c900cab10 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c900cab10 msgr2=0x7f9c900ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c900cab10 0x7f9c900ab6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9c800099b0 tx=0x7f9c80009250 comp rx=0 tx=0).stop 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- 172.21.15.143:0/3586442307 shutdown_connections 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9006b1b0 0x7f9c9006b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.077 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3586442307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c900cab10 0x7f9c900ab6c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.078 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- 172.21.15.143:0/3586442307 >> 172.21.15.143:0/3586442307 conn(0x7f9c90067bb0 msgr2=0x7f9c900cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.078 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- 172.21.15.143:0/3586442307 shutdown_connections 2023-12-19T07:46:54.078 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- 172.21.15.143:0/3586442307 wait complete. 2023-12-19T07:46:54.079 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 Processor -- start 2023-12-19T07:46:54.079 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.093+0000 7f9c980f0700 1 -- start start 2023-12-19T07:46:54.082 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 0x7f9c901709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.082 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 0x7f9c9015e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90160230 con 0x7f9c9006b1b0 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900afa90 con 0x7f9c9015d410 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c901715f0 con 0x7f9c900cab10 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9668d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 0x7f9c9015e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:53838/0 (socket says 172.21.15.143:53838) 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9668d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 0x7f9c9015e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:38640/0 (socket says 172.21.15.143:38640) 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 -- 172.21.15.143:0/3394212677 learned_addr learned my addr 172.21.15.143:0/3394212677 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c95e8c700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 0x7f9c901709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 -- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 msgr2=0x7f9c9015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 0x7f9c9015e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 -- 172.21.15.143:0/3394212677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 msgr2=0x7f9c901709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.083 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 0x7f9c901709d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 -- 172.21.15.143:0/3394212677 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c800094f0 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c95e8c700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 0x7f9c901709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c9568b700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009290 tx=0x7f9c80017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8001a450 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8001a5b0 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c900aeba0 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8001a970 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c900aef30 con 0x7f9c900cab10 2023-12-19T07:46:54.084 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9c8001ace0 con 0x7f9c900cab10 2023-12-19T07:46:54.085 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.097+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9c80025bb0 con 0x7f9c900cab10 2023-12-19T07:46:54.085 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.101+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c90071120 con 0x7f9c900cab10 2023-12-19T07:46:54.089 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.105+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9c80020070 con 0x7f9c900cab10 2023-12-19T07:46:54.169 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.185+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "176b55d9-24fe-41e0-8a08-156ec028a881", "id": 0} v 0) v1 -- 0x7f9c900c9510 con 0x7f9c900cab10 2023-12-19T07:46:54.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c85ffb700 1 -- 172.21.15.143:0/3394212677 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "176b55d9-24fe-41e0-8a08-156ec028a881", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9c80015020 con 0x7f9c900cab10 2023-12-19T07:46:54.181 INFO:teuthology.orchestra.run.smithi143.stdout:0 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 msgr2=0x7f9c900b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009290 tx=0x7f9c80017000 comp rx=0 tx=0).stop 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 shutdown_connections 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9015d410 0x7f9c9015e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c900cab10 0x7f9c900b1330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 --2- 172.21.15.143:0/3394212677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9006b1b0 0x7f9c901709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.197+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 >> 172.21.15.143:0/3394212677 conn(0x7f9c90067bb0 msgr2=0x7f9c900c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.201+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 shutdown_connections 2023-12-19T07:46:54.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.201+0000 7f9c980f0700 1 -- 172.21.15.143:0/3394212677 wait complete. 2023-12-19T07:46:54.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 178e68a7-dd99-4b0c-8e58-5d49ee2ff22e 1 2023-12-19T07:46:54.216 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:54.233+0000 7fde3d5fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:46:54.295 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:54.309+0000 7fde3d5fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:46:54.340 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:54.357+0000 7faf0dbc7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:46:54.350 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.365+0000 7f76d90e3700 1 Processor -- start 2023-12-19T07:46:54.351 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.365+0000 7f76d90e3700 1 -- start start 2023-12-19T07:46:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 0x7f76d40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40cbbb0 con 0x7f76d40caa10 2023-12-19T07:46:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d406dc90 con 0x7f76d40ce2d0 2023-12-19T07:46:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40cafa0 con 0x7f76d40cc040 2023-12-19T07:46:54.353 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.353 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48246/0 (socket says 172.21.15.143:48246) 2023-12-19T07:46:54.353 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 -- 172.21.15.143:0/2179741242 learned_addr learned my addr 172.21.15.143:0/2179741242 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.354 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d3fff700 1 --2- 172.21.15.143:0/2179741242 >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 0x7f76d40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 -- 172.21.15.143:0/2179741242 >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 msgr2=0x7f76d40ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 --2- 172.21.15.143:0/2179741242 >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 0x7f76d40ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 -- 172.21.15.143:0/2179741242 >> v1:172.21.15.111:6789/0 conn(0x7f76d40caa10 legacy=0x7f76d4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 -- 172.21.15.143:0/2179741242 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40cb280 con 0x7f76d40cc040 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d3fff700 1 --2- 172.21.15.143:0/2179741242 >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 0x7f76d40ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d88e2700 1 --2- 172.21.15.143:0/2179741242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009f20 tx=0x7f76cc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3b47c27cdf0d40 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d37fe700 1 -- 172.21.15.143:0/2179741242 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc00e070 con 0x7f76d40cc040 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d37fe700 1 -- 172.21.15.143:0/2179741242 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc00ba40 con 0x7f76d40cc040 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d37fe700 1 -- 172.21.15.143:0/2179741242 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0092e0 con 0x7f76d40cc040 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- 172.21.15.143:0/2179741242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 msgr2=0x7f76d4159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 --2- 172.21.15.143:0/2179741242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009f20 tx=0x7f76cc009840 comp rx=0 tx=0).stop 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- 172.21.15.143:0/2179741242 shutdown_connections 2023-12-19T07:46:54.355 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 --2- 172.21.15.143:0/2179741242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40cc040 0x7f76d4159010 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 --2- 172.21.15.143:0/2179741242 >> v2:172.21.15.111:3300/0 conn(0x7f76d40ce2d0 0x7f76d40ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- 172.21.15.143:0/2179741242 >> 172.21.15.143:0/2179741242 conn(0x7f76d4067bb0 msgr2=0x7f76d40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- 172.21.15.143:0/2179741242 shutdown_connections 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.369+0000 7f76d90e3700 1 -- 172.21.15.143:0/2179741242 wait complete. 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 Processor -- start 2023-12-19T07:46:54.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 -- start start 2023-12-19T07:46:54.358 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76d40caa10 0x7f76d415bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.358 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 0x7f76d40b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.358 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 0x7f76d40b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.358 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40b3bc0 con 0x7f76d40caa10 2023-12-19T07:46:54.358 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40b3fe0 con 0x7f76d40cc040 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40b2cf0 con 0x7f76d40ce2d0 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d88e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 0x7f76d40b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 0x7f76d40b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d88e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 0x7f76d40b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48258/0 (socket says 172.21.15.143:48258) 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d88e2700 1 -- 172.21.15.143:0/1239328238 learned_addr learned my addr 172.21.15.143:0/1239328238 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d3fff700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76d40caa10 0x7f76d415bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 -- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 msgr2=0x7f76d40b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 0x7f76d40b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 -- 172.21.15.143:0/1239328238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76d40caa10 msgr2=0x7f76d415bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76d40caa10 0x7f76d415bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 -- 172.21.15.143:0/1239328238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc009ae0 con 0x7f76d40cc040 2023-12-19T07:46:54.359 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76cbfff700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 0x7f76d40b1060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c510 tx=0x7f76bc007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.360 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc011590 con 0x7f76d40cc040 2023-12-19T07:46:54.360 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc011710 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d40b2f40 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc011ad0 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.373+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f76bc01c6e0 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.377+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d40b3500 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.377+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f76bc012420 con 0x7f76d40cc040 2023-12-19T07:46:54.361 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.377+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d40b3d00 con 0x7f76d40cc040 2023-12-19T07:46:54.365 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.381+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f76d40b3d00 con 0x7f76d40cc040 2023-12-19T07:46:54.417 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:54.433+0000 7faf0dbc7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:46:54.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:54.453+0000 7fde3d5fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:46:54.443 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.457+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "178e68a7-dd99-4b0c-8e58-5d49ee2ff22e", "id": 1} v 0) v1 -- 0x7f76d404be40 con 0x7f76d40cc040 2023-12-19T07:46:54.454 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.469+0000 7f76d0ff9700 1 -- 172.21.15.143:0/1239328238 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "178e68a7-dd99-4b0c-8e58-5d49ee2ff22e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f76bc02b0b0 con 0x7f76d40cc040 2023-12-19T07:46:54.454 INFO:teuthology.orchestra.run.smithi143.stdout:1 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 msgr2=0x7f76d40b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 0x7f76d40b1060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c510 tx=0x7f76bc007ea0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 shutdown_connections 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40cc040 0x7f76d40b1060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40ce2d0 0x7f76d40b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 --2- 172.21.15.143:0/1239328238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76d40caa10 0x7f76d415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 >> 172.21.15.143:0/1239328238 conn(0x7f76d4067bb0 msgr2=0x7f76d40c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 shutdown_connections 2023-12-19T07:46:54.456 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.473+0000 7f76d90e3700 1 -- 172.21.15.143:0/1239328238 wait complete. 2023-12-19T07:46:54.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new fb2e19b3-ea94-4568-ba79-b7a7eb32cf64 2 2023-12-19T07:46:54.546 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:54.565+0000 7faf0dbc7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:46:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.681+0000 7ff27dee3700 1 Processor -- start 2023-12-19T07:46:54.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.681+0000 7ff27dee3700 1 -- start start 2023-12-19T07:46:54.669 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff27dee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff278158b00 0x7ff278158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.669 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff27dee3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27806b6c0 con 0x7ff2780cb290 2023-12-19T07:46:54.670 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff27dee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27806aab0 con 0x7ff27806bb50 2023-12-19T07:46:54.670 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff27dee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806ad90 con 0x7ff278158b00 2023-12-19T07:46:54.670 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff277fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff27806bb50 0x7ff2780cab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51980/0 (socket says 172.21.15.143:51980) 2023-12-19T07:46:54.670 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff277fff700 1 -- 172.21.15.143:0/3673278099 learned_addr learned my addr 172.21.15.143:0/3673278099 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff27cee1700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff278158b00 0x7ff278158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823232075 0 0) 0x7ff27806aab0 con 0x7ff27806bb50 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff260003570 con 0x7ff27806bb50 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512241271 0 0) 0x7ff27806b6c0 con 0x7ff2780cb290 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27806aab0 con 0x7ff2780cb290 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200011029 0 0) 0x7ff260003570 con 0x7ff27806bb50 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff278158b00 msgr2=0x7ff278158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff278158b00 0x7ff278158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 >> v1:172.21.15.111:6789/0 conn(0x7ff2780cb290 legacy=0x7ff2780afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27806b070 con 0x7ff27806bb50 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7ff268001470 con 0x7ff27806bb50 2023-12-19T07:46:54.672 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.685+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 >> v1:172.21.15.143:6789/0 conn(0x7ff27806bb50 legacy=0x7ff2780cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff260003bf0 0x7ff260003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff260004810 0x7ff260004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff26000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27806ad90 con 0x7ff2600051c0 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26000c000 con 0x7ff260003bf0 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26000c1c0 con 0x7ff260004810 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27cee1700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff278158b00 0x7ff278158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff277fff700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff260004810 0x7ff260004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff26000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.673 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27cee1700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff260003bf0 0x7ff260003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 -- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff260004810 msgr2=0x7ff260004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff260004810 0x7ff260004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 -- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff260003bf0 msgr2=0x7ff260003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff260003bf0 0x7ff260003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 -- 172.21.15.143:0/3673278099 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff260003570 con 0x7ff2600051c0 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff277fff700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff260004810 0x7ff260004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff26000ba60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff2700020f0 tx=0x7ff2700017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fffe3a0717e367 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000c8b0 con 0x7ff2600051c0 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000ca10 con 0x7ff2600051c0 2023-12-19T07:46:54.675 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff2777fe700 1 -- 172.21.15.143:0/3673278099 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000cdb0 con 0x7ff2600051c0 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 -- 172.21.15.143:0/3673278099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 msgr2=0x7ff26000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 --2- 172.21.15.143:0/3673278099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff26000ba60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff2700020f0 tx=0x7ff2700017a0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27d6e2700 1 -- 172.21.15.143:0/3673278099 reap_dead start 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 -- 172.21.15.143:0/3673278099 shutdown_connections 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 -- 172.21.15.143:0/3673278099 >> 172.21.15.143:0/3673278099 conn(0x7ff278067bb0 msgr2=0x7ff2780cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 -- 172.21.15.143:0/3673278099 shutdown_connections 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.689+0000 7ff27dee3700 1 -- 172.21.15.143:0/3673278099 wait complete. 2023-12-19T07:46:54.676 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 Processor -- start 2023-12-19T07:46:54.677 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 -- start start 2023-12-19T07:46:54.679 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff27814d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.679 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780b3040 con 0x7ff2600051c0 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780b2d60 con 0x7ff2780b34d0 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.693+0000 7ff27dee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780b3e10 con 0x7ff2780b3940 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27d6e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56440/0 (socket says 172.21.15.143:56440) 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27d6e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48292/0 (socket says 172.21.15.143:48292) 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27cee1700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff27814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.680 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 -- 172.21.15.143:0/3139645976 learned_addr learned my addr 172.21.15.143:0/3139645976 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.681 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 -- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 msgr2=0x7ff2780cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 -- 172.21.15.143:0/3139645976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 msgr2=0x7ff27814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff27814d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 -- 172.21.15.143:0/3139645976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27814c860 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27d6e2700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27cee1700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff27814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff277fff700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff268007710 tx=0x7ff27806b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268007e40 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26800f9a0 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27814cb40 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800fd40 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2780ce5a0 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff268013a50 con 0x7ff2780b34d0 2023-12-19T07:46:54.682 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ff268013d00 con 0x7ff2780b34d0 2023-12-19T07:46:54.683 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.697+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780b31f0 con 0x7ff2780b34d0 2023-12-19T07:46:54.685 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.701+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff2780b31f0 con 0x7ff2780b34d0 2023-12-19T07:46:54.726 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:54.741+0000 7fde3d5fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:46:54.764 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.781+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fb2e19b3-ea94-4568-ba79-b7a7eb32cf64", "id": 2} v 0) v1 -- 0x7ff27804be40 con 0x7ff2780b34d0 2023-12-19T07:46:54.774 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.789+0000 7ff274ff9700 1 -- 172.21.15.143:0/3139645976 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fb2e19b3-ea94-4568-ba79-b7a7eb32cf64", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff268024060 con 0x7ff2780b34d0 2023-12-19T07:46:54.774 INFO:teuthology.orchestra.run.smithi143.stdout:2 2023-12-19T07:46:54.775 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.789+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 msgr2=0x7ff2780cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.775 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.789+0000 7ff27dee3700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff268007710 tx=0x7ff27806b630 comp rx=0 tx=0).stop 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 shutdown_connections 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2780b34d0 0x7ff2780cd900 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2780b3940 0x7ff2780cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 --2- 172.21.15.143:0/3139645976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2600051c0 0x7ff27814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 >> 172.21.15.143:0/3139645976 conn(0x7ff278067bb0 msgr2=0x7ff278157390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 shutdown_connections 2023-12-19T07:46:54.776 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.793+0000 7ff27dee3700 1 -- 172.21.15.143:0/3139645976 wait complete. 2023-12-19T07:46:54.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86 3 2023-12-19T07:46:54.801 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:54.813+0000 7fde3d5fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:46:54.834 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:54.853+0000 7faf0dbc7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:46:54.903 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:54.921+0000 7faf0dbc7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:46:54.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.961+0000 7f14292f0700 1 Processor -- start 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.961+0000 7f14292f0700 1 -- start start 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14292f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1424158e20 0x7f14240ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14292f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142406d5e0 con 0x7f14240ca2c0 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14292f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14240cbdf0 con 0x7f14240cb970 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14292f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14240cc0d0 con 0x7f1424158e20 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1422ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1424158e20 0x7f14240ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14227fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f14240cb970 0x7f142406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51984/0 (socket says 172.21.15.143:51984) 2023-12-19T07:46:54.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f14227fc700 1 -- 172.21.15.143:0/3960153293 learned_addr learned my addr 172.21.15.143:0/3960153293 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034615722 0 0) 0x7f14240cbdf0 con 0x7f14240cb970 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f140c003570 con 0x7f14240cb970 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118202822 0 0) 0x7f142406d5e0 con 0x7f14240ca2c0 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1422ffd700 1 -- 172.21.15.143:0/3960153293 >> v1:172.21.15.143:6789/0 conn(0x7f14240cb970 legacy=0x7f142406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1422ffd700 1 -- 172.21.15.143:0/3960153293 >> v1:172.21.15.111:6789/0 conn(0x7f14240ca2c0 legacy=0x7f142406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.965+0000 7f1422ffd700 1 -- 172.21.15.143:0/3960153293 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240cc3b0 con 0x7f1424158e20 2023-12-19T07:46:54.952 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f1422ffd700 1 --2- 172.21.15.143:0/3960153293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1424158e20 0x7f14240ce5c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1414009910 tx=0x7f14140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97a0a7bb0a757c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.952 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400b640 con 0x7f1424158e20 2023-12-19T07:46:54.952 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400b7a0 con 0x7f1424158e20 2023-12-19T07:46:54.952 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f1421ffb700 1 -- 172.21.15.143:0/3960153293 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400bb60 con 0x7f1424158e20 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 -- 172.21.15.143:0/3960153293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1424158e20 msgr2=0x7f14240ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 --2- 172.21.15.143:0/3960153293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1424158e20 0x7f14240ce5c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1414009910 tx=0x7f14140091a0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 -- 172.21.15.143:0/3960153293 shutdown_connections 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 --2- 172.21.15.143:0/3960153293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1424158e20 0x7f14240ce5c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 -- 172.21.15.143:0/3960153293 >> 172.21.15.143:0/3960153293 conn(0x7f1424067bb0 msgr2=0x7f14240d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 -- 172.21.15.143:0/3960153293 shutdown_connections 2023-12-19T07:46:54.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.969+0000 7f14292f0700 1 -- 172.21.15.143:0/3960153293 wait complete. 2023-12-19T07:46:54.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 Processor -- start 2023-12-19T07:46:54.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- start start 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1424158e20 0x7f14240c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14240c5a50 0x7f14241447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240c9d80 con 0x7f1424158e20 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240c8d30 con 0x7f14240ca2c0 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14240c9170 con 0x7f14240c5a50 2023-12-19T07:46:54.958 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56450/0 (socket says 172.21.15.143:56450) 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 -- 172.21.15.143:0/2959612226 learned_addr learned my addr 172.21.15.143:0/2959612226 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14237fe700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14240c5a50 0x7f14241447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14227fc700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1424158e20 0x7f14240c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 -- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14240c5a50 msgr2=0x7f14241447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14240c5a50 0x7f14241447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 -- 172.21.15.143:0/2959612226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1424158e20 msgr2=0x7f14240c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1424158e20 0x7f14240c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:54.959 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 -- 172.21.15.143:0/2959612226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009420 con 0x7f14240ca2c0 2023-12-19T07:46:54.960 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f1422ffd700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14240cf4e0 tx=0x7f1414017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:54.960 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141401f070 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414000bc0 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14240c9450 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400bb60 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.973+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14240c9a10 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.977+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1414023070 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.977+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f1414011420 con 0x7f14240ca2c0 2023-12-19T07:46:54.961 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.977+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14240b60d0 con 0x7f14240ca2c0 2023-12-19T07:46:54.964 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:54.981+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f14240b60d0 con 0x7f14240ca2c0 2023-12-19T07:46:55.027 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:55.041+0000 7fde3d5fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:46:55.047 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.065+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86", "id": 3} v 0) v1 -- 0x7f1424071120 con 0x7f14240ca2c0 2023-12-19T07:46:55.058 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.073+0000 7f140affd700 1 -- 172.21.15.143:0/2959612226 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f141402a070 con 0x7f14240ca2c0 2023-12-19T07:46:55.058 INFO:teuthology.orchestra.run.smithi143.stdout:3 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 msgr2=0x7f14240c87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f14240cf4e0 tx=0x7f1414017000 comp rx=0 tx=0).stop 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 shutdown_connections 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14240c5a50 0x7f14241447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1424158e20 0x7f14240c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.060 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 --2- 172.21.15.143:0/2959612226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14240ca2c0 0x7f14240c87f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.061 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 >> 172.21.15.143:0/2959612226 conn(0x7f1424067bb0 msgr2=0x7f142414fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.061 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 shutdown_connections 2023-12-19T07:46:55.061 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.077+0000 7f14292f0700 1 -- 172.21.15.143:0/2959612226 wait complete. 2023-12-19T07:46:55.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 33ab5082-d40a-45c3-a9e5-770ca2530812 4 2023-12-19T07:46:55.124 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:55.141+0000 7fde3d5fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:46:55.142 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:55.161+0000 7faf0dbc7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:46:55.214 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:55.229+0000 7fde3d5fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:46:55.220 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:55.237+0000 7faf0dbc7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:46:55.234 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.249+0000 7f43680cd700 1 Processor -- start 2023-12-19T07:46:55.235 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.249+0000 7f43680cd700 1 -- start start 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f436006b8c0 0x7f43600ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43600c7450 con 0x7f436015d410 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436006bea0 con 0x7f43600cab10 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43600c6e00 con 0x7f436006b8c0 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f436006b8c0 0x7f43600ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f436006b8c0 0x7f43600ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56456/0 (socket says 172.21.15.143:56456) 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 -- 172.21.15.143:0/122962572 learned_addr learned my addr 172.21.15.143:0/122962572 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:55.237 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 -- 172.21.15.143:0/122962572 >> v1:172.21.15.143:6789/0 conn(0x7f43600cab10 legacy=0x7f436006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.238 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 -- 172.21.15.143:0/122962572 >> v1:172.21.15.111:6789/0 conn(0x7f436015d410 legacy=0x7f43600ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.238 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 -- 172.21.15.143:0/122962572 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600c70e0 con 0x7f436006b8c0 2023-12-19T07:46:55.239 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f436666a700 1 --2- 172.21.15.143:0/122962572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436006b8c0 0x7f43600ab6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4358009f20 tx=0x7f4358009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=608255f497aaa16a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.239 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f4364e67700 1 -- 172.21.15.143:0/122962572 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800e070 con 0x7f436006b8c0 2023-12-19T07:46:55.239 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f4364e67700 1 -- 172.21.15.143:0/122962572 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435800ba80 con 0x7f436006b8c0 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f4364e67700 1 -- 172.21.15.143:0/122962572 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43580092e0 con 0x7f436006b8c0 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- 172.21.15.143:0/122962572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436006b8c0 msgr2=0x7f43600ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 --2- 172.21.15.143:0/122962572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436006b8c0 0x7f43600ab6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4358009f20 tx=0x7f4358009840 comp rx=0 tx=0).stop 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- 172.21.15.143:0/122962572 shutdown_connections 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 --2- 172.21.15.143:0/122962572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436006b8c0 0x7f43600ab6c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- 172.21.15.143:0/122962572 >> 172.21.15.143:0/122962572 conn(0x7f4360067bb0 msgr2=0x7f43600cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- 172.21.15.143:0/122962572 shutdown_connections 2023-12-19T07:46:55.240 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.253+0000 7f43680cd700 1 -- 172.21.15.143:0/122962572 wait complete. 2023-12-19T07:46:55.241 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 Processor -- start 2023-12-19T07:46:55.241 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 -- start start 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43600cab10 0x7f4360170950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 0x7f43600b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4360159990 con 0x7f43600cab10 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4360170e90 con 0x7f436015d410 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f43680cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43601710c0 con 0x7f436006b8c0 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f436666a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 0x7f43600b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f4365e69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.243 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f4365e69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48320/0 (socket says 172.21.15.143:48320) 2023-12-19T07:46:55.244 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f4365e69700 1 -- 172.21.15.143:0/2219514712 learned_addr learned my addr 172.21.15.143:0/2219514712 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:55.244 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.257+0000 7f4365668700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43600cab10 0x7f4360170950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 -- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 msgr2=0x7f43600b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 0x7f43600b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 -- 172.21.15.143:0/2219514712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43600cab10 msgr2=0x7f4360170950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43600cab10 0x7f4360170950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 -- 172.21.15.143:0/2219514712 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358009ae0 con 0x7f436006b8c0 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f436666a700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 0x7f43600b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4365e69700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f435c00c4e0 tx=0x7f435c00c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c011590 con 0x7f436006b8c0 2023-12-19T07:46:55.245 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c011710 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c011ad0 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43601713a0 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f435c0126a0 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436016d610 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f435c012b80 con 0x7f436006b8c0 2023-12-19T07:46:55.246 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.261+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436016d830 con 0x7f436006b8c0 2023-12-19T07:46:55.250 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.265+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f435c017070 con 0x7f436006b8c0 2023-12-19T07:46:55.308 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:55.325+0000 7faf0dbc7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:46:55.329 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.345+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "33ab5082-d40a-45c3-a9e5-770ca2530812", "id": 4} v 0) v1 -- 0x7f4360071120 con 0x7f436006b8c0 2023-12-19T07:46:55.341 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f4355ffb700 1 -- 172.21.15.143:0/2219514712 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "33ab5082-d40a-45c3-a9e5-770ca2530812", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f435c012d40 con 0x7f436006b8c0 2023-12-19T07:46:55.342 INFO:teuthology.orchestra.run.smithi143.stdout:4 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 msgr2=0x7f4360160200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f435c00c4e0 tx=0x7f435c00c7f0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 shutdown_connections 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f436015d410 0x7f43600b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43600cab10 0x7f4360170950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.357+0000 7f43680cd700 1 --2- 172.21.15.143:0/2219514712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f436006b8c0 0x7f4360160200 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.361+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 >> 172.21.15.143:0/2219514712 conn(0x7f4360067bb0 msgr2=0x7f436015aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.361+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 shutdown_connections 2023-12-19T07:46:55.345 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.361+0000 7f43680cd700 1 -- 172.21.15.143:0/2219514712 wait complete. 2023-12-19T07:46:55.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 303e543d-2f45-4bb2-9397-6ded49636bfc 5 2023-12-19T07:46:55.560 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 Processor -- start 2023-12-19T07:46:55.560 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 -- start start 2023-12-19T07:46:55.562 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9270154780 0x7f927015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927015ae30 con 0x7f927006baa0 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92700abf70 con 0x7f92700adab0 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f9274e4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9270154b50 con 0x7f9270154780 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.577+0000 7f926ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f92700adab0 0x7f92700adeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51994/0 (socket says 172.21.15.143:51994) 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926ffff700 1 -- 172.21.15.143:0/1431399134 learned_addr learned my addr 172.21.15.143:0/1431399134 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-12-19T07:46:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926f7fe700 1 --2- 172.21.15.143:0/1431399134 >> v2:172.21.15.111:3300/0 conn(0x7f9270154780 0x7f927015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109289231 0 0) 0x7f92700abf70 con 0x7f92700adab0 2023-12-19T07:46:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f925c003570 con 0x7f92700adab0 2023-12-19T07:46:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554862342 0 0) 0x7f927015ae30 con 0x7f927006baa0 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92700abf70 con 0x7f927006baa0 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926f7fe700 1 -- 172.21.15.143:0/1431399134 >> v1:172.21.15.143:6789/0 conn(0x7f92700adab0 legacy=0x7f92700adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926f7fe700 1 -- 172.21.15.143:0/1431399134 >> v1:172.21.15.111:6789/0 conn(0x7f927006baa0 legacy=0x7f9270157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926f7fe700 1 -- 172.21.15.143:0/1431399134 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92700ac210 con 0x7f9270154780 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926f7fe700 1 --2- 172.21.15.143:0/1431399134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9270154780 0x7f927015b2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f92600098e0 tx=0x7f92600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a57ec39db87b11 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.565 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260011630 con 0x7f9270154780 2023-12-19T07:46:55.566 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9260011790 con 0x7f9270154780 2023-12-19T07:46:55.566 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f926e7fc700 1 -- 172.21.15.143:0/1431399134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260011b30 con 0x7f9270154780 2023-12-19T07:46:55.566 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 -- 172.21.15.143:0/1431399134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9270154780 msgr2=0x7f927015b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.566 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 --2- 172.21.15.143:0/1431399134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9270154780 0x7f927015b2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f92600098e0 tx=0x7f92600091a0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.567 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 -- 172.21.15.143:0/1431399134 shutdown_connections 2023-12-19T07:46:55.567 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 --2- 172.21.15.143:0/1431399134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9270154780 0x7f927015b2c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.567 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 -- 172.21.15.143:0/1431399134 >> 172.21.15.143:0/1431399134 conn(0x7f9270067b50 msgr2=0x7f92700cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.567 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 -- 172.21.15.143:0/1431399134 shutdown_connections 2023-12-19T07:46:55.567 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.581+0000 7f9274e4c700 1 -- 172.21.15.143:0/1431399134 wait complete. 2023-12-19T07:46:55.569 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 Processor -- start 2023-12-19T07:46:55.570 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 -- start start 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927006baa0 0x7f9270150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 0x7f92701561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92700c1920 con 0x7f927006baa0 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92701557a0 con 0x7f92700adab0 2023-12-19T07:46:55.571 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.585+0000 7f9274e4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9270155d10 con 0x7f9270154780 2023-12-19T07:46:55.573 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 0x7f92701561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.573 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.573 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48332/0 (socket says 172.21.15.143:48332) 2023-12-19T07:46:55.573 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 0x7f92701561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56474/0 (socket says 172.21.15.143:56474) 2023-12-19T07:46:55.573 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 -- 172.21.15.143:0/2223319119 learned_addr learned my addr 172.21.15.143:0/2223319119 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926f7fe700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927006baa0 0x7f9270150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 -- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 msgr2=0x7f92701561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 0x7f92701561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 -- 172.21.15.143:0/2223319119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927006baa0 msgr2=0x7f9270150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927006baa0 0x7f9270150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 -- 172.21.15.143:0/2223319119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9260009420 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f926ffff700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f926800cd30 tx=0x7f9268001ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800ce30 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800ab30 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92700c1360 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680070f0 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92701559a0 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f92680074a0 con 0x7f9270154780 2023-12-19T07:46:55.574 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f9268007750 con 0x7f9270154780 2023-12-19T07:46:55.575 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.589+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927004be40 con 0x7f9270154780 2023-12-19T07:46:55.577 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.593+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f92680079a0 con 0x7f9270154780 2023-12-19T07:46:55.657 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.673+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "303e543d-2f45-4bb2-9397-6ded49636bfc", "id": 5} v 0) v1 -- 0x7f9270155e50 con 0x7f9270154780 2023-12-19T07:46:55.664 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f92677fe700 1 -- 172.21.15.143:0/2223319119 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "303e543d-2f45-4bb2-9397-6ded49636bfc", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9268025480 con 0x7f9270154780 2023-12-19T07:46:55.664 INFO:teuthology.orchestra.run.smithi143.stdout:5 2023-12-19T07:46:55.666 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 msgr2=0x7f92700c2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.666 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f926800cd30 tx=0x7f9268001ac0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.666 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 shutdown_connections 2023-12-19T07:46:55.666 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92700adab0 0x7f92701561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.666 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9270154780 0x7f92700c2dd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 --2- 172.21.15.143:0/2223319119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927006baa0 0x7f9270150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 >> 172.21.15.143:0/2223319119 conn(0x7f9270067b50 msgr2=0x7f9270152010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 shutdown_connections 2023-12-19T07:46:55.667 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.681+0000 7f9274e4c700 1 -- 172.21.15.143:0/2223319119 wait complete. 2023-12-19T07:46:55.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 8d84908d-e5c3-455b-b350-f80681f1cf24 6 2023-12-19T07:46:55.710 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:55.725+0000 7fde3d5fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:46:55.781 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:55.797+0000 7faf0dbc7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:46:55.838 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.853+0000 7f256e2e3700 1 Processor -- start 2023-12-19T07:46:55.838 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.853+0000 7f256e2e3700 1 -- start start 2023-12-19T07:46:55.841 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256e2e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f256813f2d0 0x7f25680ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256e2e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25680c7780 0x7f25680c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256e2e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25680adc60 con 0x7f256813f6a0 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256e2e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256806d7d0 con 0x7f25680c7780 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256e2e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25680c8140 con 0x7f256813f2d0 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256cae0700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f256813f6a0 0x7f256815a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:52010/0 (socket says 172.21.15.143:52010) 2023-12-19T07:46:55.842 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256cae0700 1 -- 172.21.15.143:0/4011308205 learned_addr learned my addr 172.21.15.143:0/4011308205 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256d2e1700 1 --2- 172.21.15.143:0/4011308205 >> v2:172.21.15.143:3300/0 conn(0x7f256813f2d0 0x7f25680ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256dae2700 1 --2- 172.21.15.143:0/4011308205 >> v2:172.21.15.111:3300/0 conn(0x7f25680c7780 0x7f25680c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f255ffff700 1 -- 172.21.15.143:0/4011308205 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775149857 0 0) 0x7f25680adc60 con 0x7f256813f6a0 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f255ffff700 1 -- 172.21.15.143:0/4011308205 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2554003570 con 0x7f256813f6a0 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256d2e1700 1 -- 172.21.15.143:0/4011308205 >> v2:172.21.15.111:3300/0 conn(0x7f25680c7780 msgr2=0x7f25680c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256d2e1700 1 --2- 172.21.15.143:0/4011308205 >> v2:172.21.15.111:3300/0 conn(0x7f25680c7780 0x7f25680c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256d2e1700 1 -- 172.21.15.143:0/4011308205 >> v1:172.21.15.143:6789/0 conn(0x7f256813f6a0 legacy=0x7f256815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.843 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.857+0000 7f256d2e1700 1 -- 172.21.15.143:0/4011308205 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25680ad050 con 0x7f256813f2d0 2023-12-19T07:46:55.844 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256d2e1700 1 --2- 172.21.15.143:0/4011308205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f2d0 0x7f25680ac9b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2568069bf0 tx=0x7f25640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffecb0bf18e0ee82 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.844 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f255ffff700 1 -- 172.21.15.143:0/4011308205 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564011630 con 0x7f256813f2d0 2023-12-19T07:46:55.844 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f255ffff700 1 -- 172.21.15.143:0/4011308205 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25640117b0 con 0x7f256813f2d0 2023-12-19T07:46:55.844 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f255ffff700 1 -- 172.21.15.143:0/4011308205 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564011b70 con 0x7f256813f2d0 2023-12-19T07:46:55.845 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 -- 172.21.15.143:0/4011308205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f2d0 msgr2=0x7f25680ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.845 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 --2- 172.21.15.143:0/4011308205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f2d0 0x7f25680ac9b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2568069bf0 tx=0x7f25640091a0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.845 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 -- 172.21.15.143:0/4011308205 shutdown_connections 2023-12-19T07:46:55.846 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 --2- 172.21.15.143:0/4011308205 >> v2:172.21.15.111:3300/0 conn(0x7f25680c7780 0x7f25680c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.846 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 --2- 172.21.15.143:0/4011308205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f2d0 0x7f25680ac9b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.846 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 -- 172.21.15.143:0/4011308205 >> 172.21.15.143:0/4011308205 conn(0x7f2568067b50 msgr2=0x7f25680cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.846 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 -- 172.21.15.143:0/4011308205 shutdown_connections 2023-12-19T07:46:55.846 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.861+0000 7f256e2e3700 1 -- 172.21.15.143:0/4011308205 wait complete. 2023-12-19T07:46:55.848 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 Processor -- start 2023-12-19T07:46:55.848 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 -- start start 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256813f2d0 0x7f2568156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 0x7f2568157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568168f30 con 0x7f256813f2d0 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568157700 con 0x7f256813f6a0 2023-12-19T07:46:55.850 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256e2e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2568157420 con 0x7f25680c7780 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256dae2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 0x7f2568157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256dae2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 0x7f2568157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56484/0 (socket says 172.21.15.143:56484) 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256dae2700 1 -- 172.21.15.143:0/325117763 learned_addr learned my addr 172.21.15.143:0/325117763 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48348/0 (socket says 172.21.15.143:48348) 2023-12-19T07:46:55.851 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256cae0700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256813f2d0 0x7f2568156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:55.852 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 -- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 msgr2=0x7f2568157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.852 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 0x7f2568157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.852 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 -- 172.21.15.143:0/325117763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256813f2d0 msgr2=0x7f2568156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.852 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256813f2d0 0x7f2568156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.865+0000 7f256d2e1700 1 -- 172.21.15.143:0/325117763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564009420 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f256d2e1700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2564009fd0 tx=0x7f2564017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256401f070 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564000bc0 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2568169110 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2564011b50 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25681696d0 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2564023070 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f25640258b0 con 0x7f25680c7780 2023-12-19T07:46:55.853 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.869+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256804be40 con 0x7f25680c7780 2023-12-19T07:46:55.856 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.873+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f256402a070 con 0x7f25680c7780 2023-12-19T07:46:55.876 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:55.893+0000 7fde3d5fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:46:55.934 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.949+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8d84908d-e5c3-455b-b350-f80681f1cf24", "id": 6} v 0) v1 -- 0x7f256813e130 con 0x7f25680c7780 2023-12-19T07:46:55.943 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.957+0000 7f255d7fa700 1 -- 172.21.15.143:0/325117763 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8d84908d-e5c3-455b-b350-f80681f1cf24", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2564015020 con 0x7f25680c7780 2023-12-19T07:46:55.944 INFO:teuthology.orchestra.run.smithi143.stdout:6 2023-12-19T07:46:55.944 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 msgr2=0x7f2568155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:55.944 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2564009fd0 tx=0x7f2564017000 comp rx=0 tx=0).stop 2023-12-19T07:46:55.945 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:55.961+0000 7faf0dbc7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:46:55.946 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 shutdown_connections 2023-12-19T07:46:55.946 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256813f6a0 0x7f2568157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.946 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256813f2d0 0x7f2568156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.947 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 --2- 172.21.15.143:0/325117763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25680c7780 0x7f2568155800 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:55.947 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 >> 172.21.15.143:0/325117763 conn(0x7f2568067b50 msgr2=0x7f25680c6600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:55.947 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 shutdown_connections 2023-12-19T07:46:55.947 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:55.961+0000 7f256e2e3700 1 -- 172.21.15.143:0/325117763 wait complete. 2023-12-19T07:46:55.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 0f7bf890-21b8-40bd-94c4-73967d7a4f46 7 2023-12-19T07:46:56.170 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.185+0000 7f41cd86e700 1 Processor -- start 2023-12-19T07:46:56.170 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.185+0000 7f41cd86e700 1 -- start start 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 0x7f41c806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c806cca0 con 0x7f41c80d02a0 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c806cf80 con 0x7f41c80cfed0 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c806d260 con 0x7f41c806c7c0 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.172 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48358/0 (socket says 172.21.15.143:48358) 2023-12-19T07:46:56.173 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 -- 172.21.15.143:0/3062953168 learned_addr learned my addr 172.21.15.143:0/3062953168 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c6ffd700 1 --2- 172.21.15.143:0/3062953168 >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 0x7f41c806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 -- 172.21.15.143:0/3062953168 >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 msgr2=0x7f41c806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 --2- 172.21.15.143:0/3062953168 >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 0x7f41c806dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 -- 172.21.15.143:0/3062953168 >> v1:172.21.15.111:6789/0 conn(0x7f41c80d02a0 legacy=0x7f41c8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 -- 172.21.15.143:0/3062953168 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c806d540 con 0x7f41c806c7c0 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c6ffd700 1 --2- 172.21.15.143:0/3062953168 >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 0x7f41c806dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c67fc700 1 --2- 172.21.15.143:0/3062953168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0099b0 tx=0x7f41bc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2b65478f5f3ab0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c5ffb700 1 -- 172.21.15.143:0/3062953168 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc00b820 con 0x7f41c806c7c0 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c5ffb700 1 -- 172.21.15.143:0/3062953168 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc00b980 con 0x7f41c806c7c0 2023-12-19T07:46:56.174 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41c5ffb700 1 -- 172.21.15.143:0/3062953168 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc00bd20 con 0x7f41c806c7c0 2023-12-19T07:46:56.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- 172.21.15.143:0/3062953168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 msgr2=0x7f41c80b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 --2- 172.21.15.143:0/3062953168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0099b0 tx=0x7f41bc009250 comp rx=0 tx=0).stop 2023-12-19T07:46:56.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- 172.21.15.143:0/3062953168 shutdown_connections 2023-12-19T07:46:56.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 --2- 172.21.15.143:0/3062953168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80b2e00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 --2- 172.21.15.143:0/3062953168 >> v2:172.21.15.111:3300/0 conn(0x7f41c80cfed0 0x7f41c806dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- 172.21.15.143:0/3062953168 >> 172.21.15.143:0/3062953168 conn(0x7f41c80670e0 msgr2=0x7f41c80674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:56.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.189+0000 7f41cd86e700 1 -- 172.21.15.143:0/3062953168 shutdown_connections 2023-12-19T07:46:56.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.193+0000 7f41cd86e700 1 -- 172.21.15.143:0/3062953168 wait complete. 2023-12-19T07:46:56.179 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.193+0000 7f41cd86e700 1 Processor -- start 2023-12-19T07:46:56.179 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.193+0000 7f41cd86e700 1 -- start start 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c80cfed0 0x7f41c81591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c80d02a0 0x7f41c815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80cec40 con 0x7f41c80d02a0 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80ce270 con 0x7f41c80cfed0 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80cd660 con 0x7f41c806c7c0 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:48360/0 (socket says 172.21.15.143:48360) 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 -- 172.21.15.143:0/2379673384 learned_addr learned my addr 172.21.15.143:0/2379673384 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-12-19T07:46:56.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c67fc700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c80cfed0 0x7f41c81591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c77fe700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c80d02a0 0x7f41c815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 -- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c80cfed0 msgr2=0x7f41c81591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c80cfed0 0x7f41c81591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 -- 172.21.15.143:0/2379673384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c80d02a0 msgr2=0x7f41c815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c80d02a0 0x7f41c815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 -- 172.21.15.143:0/2379673384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0094f0 con 0x7f41c806c7c0 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41c6ffd700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c830 tx=0x7f41b8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011600 con 0x7f41c806c7c0 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8011760 con 0x7f41c806c7c0 2023-12-19T07:46:56.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c80cd940 con 0x7f41c806c7c0 2023-12-19T07:46:56.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011b20 con 0x7f41c806c7c0 2023-12-19T07:46:56.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c80cdf80 con 0x7f41c806c7c0 2023-12-19T07:46:56.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f41b801a6d0 con 0x7f41c806c7c0 2023-12-19T07:46:56.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.197+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f41b801a980 con 0x7f41c806c7c0 2023-12-19T07:46:56.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.201+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804be40 con 0x7f41c806c7c0 2023-12-19T07:46:56.187 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.205+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f41b80396d0 con 0x7f41c806c7c0 2023-12-19T07:46:56.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.285+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0f7bf890-21b8-40bd-94c4-73967d7a4f46", "id": 7} v 0) v1 -- 0x7f41c806d820 con 0x7f41c806c7c0 2023-12-19T07:46:56.282 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.297+0000 7f41aeffd700 1 -- 172.21.15.143:0/2379673384 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f7bf890-21b8-40bd-94c4-73967d7a4f46", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f41b8028020 con 0x7f41c806c7c0 2023-12-19T07:46:56.283 INFO:teuthology.orchestra.run.smithi143.stdout:7 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 msgr2=0x7f41c80ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c830 tx=0x7f41b8008040 comp rx=0 tx=0).stop 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 shutdown_connections 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c80cfed0 0x7f41c81591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41c80d02a0 0x7f41c815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.284 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 --2- 172.21.15.143:0/2379673384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41c806c7c0 0x7f41c80ce700 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.285 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:56.301+0000 7fde3d5fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:46:56.285 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 >> 172.21.15.143:0/2379673384 conn(0x7f41c80670e0 msgr2=0x7f41c814f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:56.286 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 shutdown_connections 2023-12-19T07:46:56.286 INFO:teuthology.orchestra.run.smithi143.stderr:2023-12-19T07:46:56.301+0000 7f41cd86e700 1 -- 172.21.15.143:0/2379673384 wait complete. 2023-12-19T07:46:56.301 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-19T07:46:56.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-19T07:46:56.308 INFO:tasks.ceph.osd.0:Started 2023-12-19T07:46:56.308 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-19T07:46:56.309 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-19T07:46:56.310 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:56.325+0000 7faf0dbc7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:46:56.314 INFO:tasks.ceph.osd.1:Started 2023-12-19T07:46:56.314 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-19T07:46:56.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-19T07:46:56.320 INFO:tasks.ceph.osd.2:Started 2023-12-19T07:46:56.320 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-19T07:46:56.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-19T07:46:56.327 INFO:tasks.ceph.osd.3:Started 2023-12-19T07:46:56.327 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-19T07:46:56.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-19T07:46:56.333 INFO:tasks.ceph.osd.4:Started 2023-12-19T07:46:56.333 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-19T07:46:56.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-19T07:46:56.340 INFO:tasks.ceph.osd.5:Started 2023-12-19T07:46:56.341 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-19T07:46:56.341 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-19T07:46:56.345 INFO:tasks.ceph.osd.6:Started 2023-12-19T07:46:56.345 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-19T07:46:56.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-19T07:46:56.350 INFO:tasks.ceph.osd.7:Started 2023-12-19T07:46:56.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:46:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.537+0000 7fcbd9bc3700 1 Processor -- start 2023-12-19T07:46:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.537+0000 7fcbd9bc3700 1 -- start start 2023-12-19T07:46:56.522 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:56.537+0000 7fde3d5fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:46:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd9bc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcbd41591f0 0x7fcbd406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd9bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd9bc3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd406b730 con 0x7fcbd4158e20 2023-12-19T07:46:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd9bc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd406ab20 con 0x7fcbd41591f0 2023-12-19T07:46:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd9bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd406ae00 con 0x7fcbd40ce810 2023-12-19T07:46:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:44908/0 (socket says 172.21.15.111:44908) 2023-12-19T07:46:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 -- 172.21.15.111:0/139328050 learned_addr learned my addr 172.21.15.111:0/139328050 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:46:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd3fff700 1 --2- 172.21.15.111:0/139328050 >> v2:172.21.15.143:3300/0 conn(0x7fcbd41591f0 0x7fcbd406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 -- 172.21.15.111:0/139328050 >> v2:172.21.15.143:3300/0 conn(0x7fcbd41591f0 msgr2=0x7fcbd406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 --2- 172.21.15.111:0/139328050 >> v2:172.21.15.143:3300/0 conn(0x7fcbd41591f0 0x7fcbd406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 -- 172.21.15.111:0/139328050 >> v1:172.21.15.143:6789/0 conn(0x7fcbd4158e20 legacy=0x7fcbd40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.541+0000 7fcbd93c2700 1 -- 172.21.15.111:0/139328050 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd406b0e0 con 0x7fcbd40ce810 2023-12-19T07:46:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd93c2700 1 --2- 172.21.15.111:0/139328050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009f20 tx=0x7fcbcc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe7f04a29056dab server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd37fe700 1 -- 172.21.15.111:0/139328050 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc00e070 con 0x7fcbd40ce810 2023-12-19T07:46:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd37fe700 1 -- 172.21.15.111:0/139328050 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc00b9d0 con 0x7fcbd40ce810 2023-12-19T07:46:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd37fe700 1 -- 172.21.15.111:0/139328050 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0092e0 con 0x7fcbd40ce810 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/139328050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 msgr2=0x7fcbd40ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/139328050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009f20 tx=0x7fcbcc009840 comp rx=0 tx=0).stop 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/139328050 shutdown_connections 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/139328050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd40ce810 0x7fcbd40ca950 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/139328050 >> v2:172.21.15.143:3300/0 conn(0x7fcbd41591f0 0x7fcbd406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/139328050 >> 172.21.15.111:0/139328050 conn(0x7fcbd4067bb0 msgr2=0x7fcbd40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/139328050 shutdown_connections 2023-12-19T07:46:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/139328050 wait complete. 2023-12-19T07:46:56.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 Processor -- start 2023-12-19T07:46:56.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.545+0000 7fcbd9bc3700 1 -- start start 2023-12-19T07:46:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbd40ce810 0x7fcbd4155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 0x7fcbd40c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd40c71a0 con 0x7fcbd41591f0 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd40c6e70 con 0x7fcbd40ce810 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd415a9f0 con 0x7fcbd4158e20 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59810/0 (socket says 172.21.15.111:59810) 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 -- 172.21.15.111:0/715174534 learned_addr learned my addr 172.21.15.111:0/715174534 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd3fff700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 0x7fcbd40c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd8bc1700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbd40ce810 0x7fcbd4155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 -- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 msgr2=0x7fcbd40c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 0x7fcbd40c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 -- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbd40ce810 msgr2=0x7fcbd4155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbd40ce810 0x7fcbd4155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 -- 172.21.15.111:0/715174534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc009ae0 con 0x7fcbd41591f0 2023-12-19T07:46:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd3fff700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 0x7fcbd40c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:46:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd93c2700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00b610 tx=0x7fcbcc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc008c30 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc008d90 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc026690 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd40c9670 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd40c9c30 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcbcc003040 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.549+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fcbcc026ac0 con 0x7fcbd41591f0 2023-12-19T07:46:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.553+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd40c9df0 con 0x7fcbd41591f0 2023-12-19T07:46:56.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.557+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcbcc021070 con 0x7fcbd41591f0 2023-12-19T07:46:56.558 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:56.577+0000 7faf0dbc7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:46:56.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbd40c7310 con 0x7fcbd41591f0 2023-12-19T07:46:56.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd0ff9700 1 -- 172.21.15.111:0/715174534 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fcbcc032430 con 0x7fcbd41591f0 2023-12-19T07:46:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:46:56.611 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":10,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:46:56.290869+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:46:56.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 msgr2=0x7fcbd40c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00b610 tx=0x7fcbcc018000 comp rx=0 tx=0).stop 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 shutdown_connections 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbd41591f0 0x7fcbd40c8b00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbd4158e20 0x7fcbd40c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 --2- 172.21.15.111:0/715174534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbd40ce810 0x7fcbd4155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:56.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 >> 172.21.15.111:0/715174534 conn(0x7fcbd4067bb0 msgr2=0x7fcbd4152400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:56.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 shutdown_connections 2023-12-19T07:46:56.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:46:56.629+0000 7fcbd9bc3700 1 -- 172.21.15.111:0/715174534 wait complete. 2023-12-19T07:46:56.619 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:56.637+0000 7fde3d5fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:46:56.630 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-19T07:46:56.630 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-19T07:46:56.658 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:56.677+0000 7faf0dbc7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:46:56.700 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:56.717+0000 7fde3d5fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:46:56.746 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:56.765+0000 7faf0dbc7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:46:56.790 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:56.805+0000 7fde3d5fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:46:56.832 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:56.849+0000 7faf0dbc7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:46:56.969 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-12-19T07:46:56.985+0000 7f3bf7ce83c0 -1 Falling back to public interface 2023-12-19T07:46:56.983 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-12-19T07:46:57.001+0000 7f06bbd493c0 -1 Falling back to public interface 2023-12-19T07:46:56.990 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-12-19T07:46:57.005+0000 7fafc20ea3c0 -1 Falling back to public interface 2023-12-19T07:46:57.004 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-12-19T07:46:57.021+0000 7f2eda0dc3c0 -1 Falling back to public interface 2023-12-19T07:46:57.014 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-12-19T07:46:57.029+0000 7f72c92f53c0 -1 Falling back to public interface 2023-12-19T07:46:57.014 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-12-19T07:46:57.029+0000 7f9445eee3c0 -1 Falling back to public interface 2023-12-19T07:46:57.020 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-12-19T07:46:57.037+0000 7f5b6cb9b3c0 -1 Falling back to public interface 2023-12-19T07:46:57.032 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-12-19T07:46:57.049+0000 7f4d62c8d3c0 -1 Falling back to public interface 2023-12-19T07:46:57.140 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:57.157+0000 7fde3d5fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:46:57.178 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:57.197+0000 7faf0dbc7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:46:57.225 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:57.241+0000 7fde3d5fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:46:57.261 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:57.281+0000 7faf0dbc7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:46:57.318 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:57.333+0000 7fde3d5fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:46:57.360 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:57.377+0000 7faf0dbc7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:46:57.436 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T07:46:57.539 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:57.553+0000 7fde3d5fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:46:57.574 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 Processor -- start 2023-12-19T07:46:57.574 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 -- start start 2023-12-19T07:46:57.575 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efbf8156bc0 0x7efbf80b1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:57.575 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 0x7efbf806c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:57.575 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf806efa0 con 0x7efbf806ae80 2023-12-19T07:46:57.575 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf80afc60 con 0x7efbf8156bc0 2023-12-19T07:46:57.576 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd923700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf80af450 con 0x7efbf8069da0 2023-12-19T07:46:57.576 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efbf8156bc0 0x7efbf80b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:57.576 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbf7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efbf806ae80 0x7efbf8069690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:51316/0 (socket says 172.21.15.111:51316) 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbf7fff700 1 -- 172.21.15.111:0/4252603527 learned_addr learned my addr 172.21.15.111:0/4252603527 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 --2- 172.21.15.111:0/4252603527 >> v2:172.21.15.111:3300/0 conn(0x7efbf8156bc0 0x7efbf80b1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59818/0 (socket says 172.21.15.111:59818) 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd122700 1 --2- 172.21.15.111:0/4252603527 >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 0x7efbf806c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 -- 172.21.15.111:0/4252603527 >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 msgr2=0x7efbf806c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 --2- 172.21.15.111:0/4252603527 >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 0x7efbf806c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 -- 172.21.15.111:0/4252603527 >> v1:172.21.15.111:6789/0 conn(0x7efbf806ae80 legacy=0x7efbf8069690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 -- 172.21.15.111:0/4252603527 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf806bc00 con 0x7efbf8156bc0 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbf77fe700 1 -- 172.21.15.111:0/4252603527 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289531208 0 0) 0x7efbf806efa0 con 0x7efbf806ae80 2023-12-19T07:46:57.577 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfd122700 1 --2- 172.21.15.111:0/4252603527 >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 0x7efbf806c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:46:57.578 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.593+0000 7efbfc921700 1 --2- 172.21.15.111:0/4252603527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf80b1110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efbf8158450 tx=0x7efbec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa06902fa1db59e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:57.578 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbf77fe700 1 -- 172.21.15.111:0/4252603527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec00b6b0 con 0x7efbf8156bc0 2023-12-19T07:46:57.578 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbf77fe700 1 -- 172.21.15.111:0/4252603527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec00b810 con 0x7efbf8156bc0 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbf77fe700 1 -- 172.21.15.111:0/4252603527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec00bbb0 con 0x7efbf8156bc0 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- 172.21.15.111:0/4252603527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 msgr2=0x7efbf80b1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 --2- 172.21.15.111:0/4252603527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf80b1110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efbf8158450 tx=0x7efbec0091a0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- 172.21.15.111:0/4252603527 shutdown_connections 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 --2- 172.21.15.111:0/4252603527 >> v2:172.21.15.143:3300/0 conn(0x7efbf8069da0 0x7efbf806c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 --2- 172.21.15.111:0/4252603527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf80b1110 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- 172.21.15.111:0/4252603527 >> 172.21.15.111:0/4252603527 conn(0x7efbf80c8420 msgr2=0x7efbf80c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- 172.21.15.111:0/4252603527 shutdown_connections 2023-12-19T07:46:57.579 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- 172.21.15.111:0/4252603527 wait complete. 2023-12-19T07:46:57.580 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 Processor -- start 2023-12-19T07:46:57.580 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:57.597+0000 7faf0dbc7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:46:57.581 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.597+0000 7efbfd923700 1 -- start start 2023-12-19T07:46:57.582 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8069da0 0x7efbf8154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:57.582 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf806ae80 0x7efbf80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:57.582 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:46:57.582 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8154da0 con 0x7efbf8156bc0 2023-12-19T07:46:57.583 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8155bd0 con 0x7efbf8069da0 2023-12-19T07:46:57.583 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf814d5b0 con 0x7efbf806ae80 2023-12-19T07:46:57.583 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:57.583 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59834/0 (socket says 172.21.15.111:59834) 2023-12-19T07:46:57.583 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 -- 172.21.15.111:0/1830708016 learned_addr learned my addr 172.21.15.111:0/1830708016 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfc921700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8069da0 0x7efbf8154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf7fff700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf806ae80 0x7efbf80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 -- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf806ae80 msgr2=0x7efbf80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf806ae80 0x7efbf80c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 -- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8069da0 msgr2=0x7efbf8154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:57.584 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8069da0 0x7efbf8154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 -- 172.21.15.111:0/1830708016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec009420 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd122700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009f90 tx=0x7efbf00099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000dd40 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000f040 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf814db40 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0015690 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf814e100 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7efbf0015a40 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.601+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7efbf001ea90 con 0x7efbf8156bc0 2023-12-19T07:46:57.585 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.605+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf814e350 con 0x7efbf8156bc0 2023-12-19T07:46:57.589 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.609+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7efbf001a070 con 0x7efbf8156bc0 2023-12-19T07:46:57.636 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:57.653+0000 7fde3d5fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:46:57.659 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.677+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efbf804be40 con 0x7efbf8156bc0 2023-12-19T07:46:57.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.677+0000 7efbf4ff9700 1 -- 172.21.15.111:0/1830708016 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7efbf001ecc0 con 0x7efbf8156bc0 2023-12-19T07:46:57.660 INFO:teuthology.misc.health.smithi111.stdout: 2023-12-19T07:46:57.660 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":10,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:46:56.290869+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:46:57.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 msgr2=0x7efbf814b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:46:57.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009f90 tx=0x7efbf00099b0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 shutdown_connections 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf8156bc0 0x7efbf814b150 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbf806ae80 0x7efbf80c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 --2- 172.21.15.111:0/1830708016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8069da0 0x7efbf8154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 >> 172.21.15.111:0/1830708016 conn(0x7efbf80c8420 msgr2=0x7efbf8157a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 shutdown_connections 2023-12-19T07:46:57.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:46:57.681+0000 7efbfd923700 1 -- 172.21.15.111:0/1830708016 wait complete. 2023-12-19T07:46:57.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-19T07:46:57.683 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:57.701+0000 7faf0dbc7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:46:58.559 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:58.573+0000 7fde3d5fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:46:58.609 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:46:58.625+0000 7faf0dbc7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:46:59.598 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:59.613+0000 7fde2672e700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-19T07:46:59.599 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:46:59.617+0000 7fde2672e700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-19T07:47:00.867 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-12-19T07:47:00.881+0000 7fafc20ea3c0 -1 osd.6 0 log_to_monitors true 2023-12-19T07:47:00.962 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-12-19T07:47:00.977+0000 7f72c92f53c0 -1 osd.5 0 log_to_monitors true 2023-12-19T07:47:00.965 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-12-19T07:47:00.981+0000 7f9445eee3c0 -1 osd.4 0 log_to_monitors true 2023-12-19T07:47:00.990 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-12-19T07:47:01.009+0000 7f06bbd493c0 -1 osd.1 0 log_to_monitors true 2023-12-19T07:47:01.028 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-12-19T07:47:01.045+0000 7f4d62c8d3c0 -1 osd.7 0 log_to_monitors true 2023-12-19T07:47:01.184 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-12-19T07:47:01.201+0000 7f3bf7ce83c0 -1 osd.0 0 log_to_monitors true 2023-12-19T07:47:01.210 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-12-19T07:47:01.229+0000 7f5b6cb9b3c0 -1 osd.2 0 log_to_monitors true 2023-12-19T07:47:01.255 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-12-19T07:47:01.273+0000 7f2eda0dc3c0 -1 osd.3 0 log_to_monitors true 2023-12-19T07:47:03.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:47:03.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:47:04.481 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:04.657 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.673+0000 7f756900c700 1 Processor -- start 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.673+0000 7f756900c700 1 -- start start 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756900c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f755c0e24e0 0x7f755c0071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756900c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 0x7f755c03b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756900c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c006620 con 0x7f755c0e2110 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756900c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c007bc0 con 0x7f755c0e24e0 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756900c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c053ec0 con 0x7f755c007720 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f7563fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f755c0e2110 0x7f755c03bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:60632/0 (socket says 172.21.15.111:60632) 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f7563fff700 1 -- 172.21.15.111:0/2235657419 learned_addr learned my addr 172.21.15.111:0/2235657419 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:04.660 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f75637fe700 1 --2- 172.21.15.111:0/2235657419 >> v2:172.21.15.111:3300/0 conn(0x7f755c0e24e0 0x7f755c0071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756880b700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 0x7f755c03b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f7562ffd700 1 -- 172.21.15.111:0/2235657419 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738398455 0 0) 0x7f755c006620 con 0x7f755c0e2110 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f7562ffd700 1 -- 172.21.15.111:0/2235657419 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554003570 con 0x7f755c0e2110 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f75637fe700 1 -- 172.21.15.111:0/2235657419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 msgr2=0x7f755c03b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f75637fe700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 0x7f755c03b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f75637fe700 1 -- 172.21.15.111:0/2235657419 >> v1:172.21.15.143:6789/0 conn(0x7f755c0e2110 legacy=0x7f755c03bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f75637fe700 1 -- 172.21.15.111:0/2235657419 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c054a40 con 0x7f755c0e24e0 2023-12-19T07:47:04.661 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.677+0000 7f756880b700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 0x7f755c03b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:04.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f75637fe700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e24e0 0x7f755c0071e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75580098e0 tx=0x7f75580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb520a4aa0fe2804 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:04.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f7562ffd700 1 -- 172.21.15.111:0/2235657419 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558011630 con 0x7f755c0e24e0 2023-12-19T07:47:04.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f7562ffd700 1 -- 172.21.15.111:0/2235657419 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558011790 con 0x7f755c0e24e0 2023-12-19T07:47:04.662 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f7562ffd700 1 -- 172.21.15.111:0/2235657419 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558011b30 con 0x7f755c0e24e0 2023-12-19T07:47:04.663 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 -- 172.21.15.111:0/2235657419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e24e0 msgr2=0x7f755c0071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.663 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e24e0 0x7f755c0071e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75580098e0 tx=0x7f75580091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.663 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 -- 172.21.15.111:0/2235657419 shutdown_connections 2023-12-19T07:47:04.664 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c007720 0x7f755c03b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.664 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 --2- 172.21.15.111:0/2235657419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e24e0 0x7f755c0071e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.664 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 -- 172.21.15.111:0/2235657419 >> 172.21.15.111:0/2235657419 conn(0x7f755c00f280 msgr2=0x7f755c0591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:04.664 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 -- 172.21.15.111:0/2235657419 shutdown_connections 2023-12-19T07:47:04.665 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.681+0000 7f756900c700 1 -- 172.21.15.111:0/2235657419 wait complete. 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 Processor -- start 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 -- start start 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c007720 0x7f755c04eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c0e24e0 0x7f755c0382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0dcda0 con 0x7f755c0e2110 2023-12-19T07:47:04.668 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0dc190 con 0x7f755c007720 2023-12-19T07:47:04.669 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.685+0000 7f756900c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0dc470 con 0x7f755c0e24e0 2023-12-19T07:47:04.670 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:04.670 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60328/0 (socket says 172.21.15.111:60328) 2023-12-19T07:47:04.670 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 -- 172.21.15.111:0/2387946297 learned_addr learned my addr 172.21.15.111:0/2387946297 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:04.671 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 -- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c0e24e0 msgr2=0x7f755c0382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.671 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c0e24e0 0x7f755c0382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.671 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 -- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c007720 msgr2=0x7f755c04eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.671 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c007720 0x7f755c04eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.671 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 -- 172.21.15.111:0/2387946297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009420 con 0x7f755c0e2110 2023-12-19T07:47:04.672 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f75637fe700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75580093f0 tx=0x7f7558016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:04.672 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755801bea0 con 0x7f755c0e2110 2023-12-19T07:47:04.674 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558020070 con 0x7f755c0e2110 2023-12-19T07:47:04.674 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.689+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75580276d0 con 0x7f755c0e2110 2023-12-19T07:47:04.674 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.693+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c0dc750 con 0x7f755c0e2110 2023-12-19T07:47:04.675 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.693+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c04f090 con 0x7f755c0e2110 2023-12-19T07:47:04.677 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.693+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c0dcbd0 con 0x7f755c0e2110 2023-12-19T07:47:04.678 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.697+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 79905+0+0 (secure 0 0 0) 0x7f7558027ac0 con 0x7f755c0e2110 2023-12-19T07:47:04.680 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.697+0000 7f754ffff700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 0x7f7548062a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:04.680 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.697+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f7558022070 con 0x7f755c0e2110 2023-12-19T07:47:04.680 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.697+0000 7f7563fff700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 0x7f7548062a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:04.682 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.701+0000 7f7563fff700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 0x7f7548062a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75500011e0 tx=0x7f75500079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:04.690 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.709+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f755804ed30 con 0x7f755c0e2110 2023-12-19T07:47:04.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.993+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f755c005950 con 0x7f755c0e2110 2023-12-19T07:47:04.977 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:04.997+0000 7f754ffff700 1 -- 172.21.15.111:0/2387946297 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f7558052350 con 0x7f755c0e2110 2023-12-19T07:47:04.978 INFO:teuthology.misc.health.smithi111.stdout: 2023-12-19T07:47:04.978 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":12,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:02.630340+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:04.986 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 msgr2=0x7f7548062a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.986 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 0x7f7548062a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75500011e0 tx=0x7f75500079d0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.986 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 msgr2=0x7f755c0dd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:04.986 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75580093f0 tx=0x7f7558016000 comp rx=0 tx=0).stop 2023-12-19T07:47:04.986 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 shutdown_connections 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7548060590 0x7f7548062a40 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75500011e0 tx=0x7f75500079d0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f755c0e24e0 0x7f755c0382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f755c0e2110 0x7f755c0dd400 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75580093f0 tx=0x7f7558016000 comp rx=0 tx=0).stop 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 --2- 172.21.15.111:0/2387946297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c007720 0x7f755c04eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.001+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 >> 172.21.15.111:0/2387946297 conn(0x7f755c00f280 msgr2=0x7f755c03d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:04.987 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.005+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 shutdown_connections 2023-12-19T07:47:04.988 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:05.005+0000 7f756900c700 1 -- 172.21.15.111:0/2387946297 wait complete. 2023-12-19T07:47:05.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-19T07:47:05.031 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-12-19T07:47:05.045+0000 7f4d5e42d700 -1 osd.7 0 waiting for initial osdmap 2023-12-19T07:47:05.038 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-12-19T07:47:05.053+0000 7fafbd039700 -1 osd.6 0 waiting for initial osdmap 2023-12-19T07:47:05.047 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-12-19T07:47:05.061+0000 7fafb5ddd700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.049 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-12-19T07:47:05.065+0000 7f4d571d1700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.081 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-12-19T07:47:05.097+0000 7f72c4203700 -1 osd.5 0 waiting for initial osdmap 2023-12-19T07:47:05.094 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-12-19T07:47:05.109+0000 7f72bcfa7700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.114 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-12-19T07:47:05.133+0000 7f06b74e9700 -1 osd.1 0 waiting for initial osdmap 2023-12-19T07:47:05.135 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-12-19T07:47:05.153+0000 7f06b028d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.141 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-12-19T07:47:05.157+0000 7f9440e1b700 -1 osd.4 0 waiting for initial osdmap 2023-12-19T07:47:05.156 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-12-19T07:47:05.173+0000 7f9439bbf700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.163 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-12-19T07:47:05.181+0000 7f3bf2c0d700 -1 osd.0 0 waiting for initial osdmap 2023-12-19T07:47:05.171 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-12-19T07:47:05.189+0000 7f3beb9b1700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.205 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-12-19T07:47:05.225+0000 7f2ed4ef8700 -1 osd.3 0 waiting for initial osdmap 2023-12-19T07:47:05.215 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-12-19T07:47:05.233+0000 7f2ecdc9c700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:05.259 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-12-19T07:47:05.277+0000 7f5b6833b700 -1 osd.2 0 waiting for initial osdmap 2023-12-19T07:47:05.268 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-12-19T07:47:05.285+0000 7f5b610df700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T07:47:11.814 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:11.965 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.980+0000 7f9135e8e700 1 Processor -- start 2023-12-19T07:47:11.965 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.980+0000 7f9135e8e700 1 -- start start 2023-12-19T07:47:11.969 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9135e8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9130068230 0x7f9130069c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:11.969 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9135e8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913006e350 con 0x7f91301592a0 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9135e8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91300b35c0 con 0x7f91300b28c0 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9135e8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913006d740 con 0x7f9130068230 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f912ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f91301592a0 0x7f91300b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37116/0 (socket says 172.21.15.111:37116) 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f912ffff700 1 -- 172.21.15.111:0/1694158952 learned_addr learned my addr 172.21.15.111:0/1694158952 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9134e8c700 1 --2- 172.21.15.111:0/1694158952 >> v2:172.21.15.111:3300/0 conn(0x7f9130068230 0x7f9130069c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9134e8c700 1 -- 172.21.15.111:0/1694158952 >> v1:172.21.15.143:6789/0 conn(0x7f91300b28c0 legacy=0x7f913006e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9134e8c700 1 -- 172.21.15.111:0/1694158952 >> v1:172.21.15.111:6789/0 conn(0x7f91301592a0 legacy=0x7f91300b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f912f7fe700 1 -- 172.21.15.111:0/1694158952 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177781946 0 0) 0x7f913006e350 con 0x7f91301592a0 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.984+0000 7f9134e8c700 1 -- 172.21.15.111:0/1694158952 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913006da20 con 0x7f9130068230 2023-12-19T07:47:11.970 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9134e8c700 1 --2- 172.21.15.111:0/1694158952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9130068230 0x7f9130069c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91240098e0 tx=0x7f91240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a69b22fb8be7bcf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:11.971 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f912f7fe700 1 -- 172.21.15.111:0/1694158952 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124009e80 con 0x7f9130068230 2023-12-19T07:47:11.971 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f912f7fe700 1 -- 172.21.15.111:0/1694158952 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912400b860 con 0x7f9130068230 2023-12-19T07:47:11.971 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f912f7fe700 1 -- 172.21.15.111:0/1694158952 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912400bbe0 con 0x7f9130068230 2023-12-19T07:47:11.971 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 -- 172.21.15.111:0/1694158952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9130068230 msgr2=0x7f9130069c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 --2- 172.21.15.111:0/1694158952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9130068230 0x7f9130069c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91240098e0 tx=0x7f91240091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 -- 172.21.15.111:0/1694158952 shutdown_connections 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 --2- 172.21.15.111:0/1694158952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9130068230 0x7f9130069c30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 -- 172.21.15.111:0/1694158952 >> 172.21.15.111:0/1694158952 conn(0x7f91300ae060 msgr2=0x7f91300b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 -- 172.21.15.111:0/1694158952 shutdown_connections 2023-12-19T07:47:11.972 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 -- 172.21.15.111:0/1694158952 wait complete. 2023-12-19T07:47:11.973 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.988+0000 7f9135e8e700 1 Processor -- start 2023-12-19T07:47:11.974 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 -- start start 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9130068230 0x7f9130155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91300b28c0 0x7f91300c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c9040 con 0x7f91301592a0 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c8d10 con 0x7f9130068230 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9135e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91300c76b0 con 0x7f91300b28c0 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50102/0 (socket says 172.21.15.111:50102) 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 -- 172.21.15.111:0/948071083 learned_addr learned my addr 172.21.15.111:0/948071083 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f9134e8c700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9130068230 0x7f9130155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:11.976 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f912ffff700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91300b28c0 0x7f91300c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 -- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91300b28c0 msgr2=0x7f91300c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91300b28c0 0x7f91300c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 -- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9130068230 msgr2=0x7f9130155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9130068230 0x7f9130155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 -- 172.21.15.111:0/948071083 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124009420 con 0x7f91301592a0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.992+0000 7f913568d700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f912800c910 tx=0x7f91280016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128012630 con 0x7f91301592a0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91280127b0 con 0x7f91301592a0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91300c7990 con 0x7f91301592a0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128012b70 con 0x7f91301592a0 2023-12-19T07:47:11.978 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91301558d0 con 0x7f91301592a0 2023-12-19T07:47:11.979 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91300c91f0 con 0x7f91301592a0 2023-12-19T07:47:11.981 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:11.996+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f912800a3b0 con 0x7f91301592a0 2023-12-19T07:47:11.982 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.000+0000 7f912cff9700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 0x7f9114062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:11.982 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.000+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f912807c770 con 0x7f91301592a0 2023-12-19T07:47:11.982 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.000+0000 7f9134e8c700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 0x7f9114062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:11.984 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.000+0000 7f9134e8c700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 0x7f9114062a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91240093f0 tx=0x7f9124017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:11.989 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.008+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f91300c91f0 con 0x7f91301592a0 2023-12-19T07:47:12.162 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.180+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f913004be40 con 0x7f91301592a0 2023-12-19T07:47:12.163 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.180+0000 7f912cff9700 1 -- 172.21.15.111:0/948071083 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12175 (secure 0 0 0) 0x7f9128050aa0 con 0x7f91301592a0 2023-12-19T07:47:12.164 INFO:teuthology.misc.health.smithi111.stdout: 2023-12-19T07:47:12.164 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":16,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:09.095827+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:12.169 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 msgr2=0x7f9114062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 0x7f9114062a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91240093f0 tx=0x7f9124017000 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 msgr2=0x7f9130169170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f912800c910 tx=0x7f91280016f0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 shutdown_connections 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9114060550 0x7f9114062a00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91301592a0 0x7f9130169170 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91300b28c0 0x7f91300c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 --2- 172.21.15.111:0/948071083 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9130068230 0x7f9130155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 >> 172.21.15.111:0/948071083 conn(0x7f91300ae060 msgr2=0x7f91301561e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 shutdown_connections 2023-12-19T07:47:12.170 INFO:teuthology.misc.health.smithi111.stderr:2023-12-19T07:47:12.184+0000 7f9135e8e700 1 -- 172.21.15.111:0/948071083 wait complete. 2023-12-19T07:47:12.182 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-19T07:47:12.182 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-19T07:47:12.182 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T07:47:12.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T07:47:12.189 INFO:tasks.ceph.mds.a:Started 2023-12-19T07:47:12.189 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T07:47:12.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T07:47:12.194 INFO:tasks.ceph.mds.c:Started 2023-12-19T07:47:12.194 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T07:47:12.194 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T07:47:12.201 INFO:tasks.ceph.mds.b:Started 2023-12-19T07:47:12.201 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T07:47:12.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T07:47:12.207 INFO:tasks.ceph.mds.d:Started 2023-12-19T07:47:12.207 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-19T07:47:12.207 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:12.295 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T07:47:12.298 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T07:47:12.314 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T07:47:12.318 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T07:47:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.364+0000 7ff1766e3700 1 Processor -- start 2023-12-19T07:47:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 -- start start 2023-12-19T07:47:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff170157320 0x7ff1701576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1700aca70 0x7ff1700acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1700c8620 con 0x7ff17006de90 2023-12-19T07:47:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff17006d1d0 con 0x7ff1700aca70 2023-12-19T07:47:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1766e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff17006d3b0 con 0x7ff170157320 2023-12-19T07:47:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1700aca70 0x7ff1700acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1700aca70 0x7ff1700acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50130/0 (socket says 172.21.15.111:50130) 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 -- 172.21.15.111:0/3014237879 learned_addr learned my addr 172.21.15.111:0/3014237879 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff1756e1700 1 --2- 172.21.15.111:0/3014237879 >> v2:172.21.15.143:3300/0 conn(0x7ff170157320 0x7ff1701576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 -- 172.21.15.111:0/3014237879 >> v2:172.21.15.143:3300/0 conn(0x7ff170157320 msgr2=0x7ff1701576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 --2- 172.21.15.111:0/3014237879 >> v2:172.21.15.143:3300/0 conn(0x7ff170157320 0x7ff1701576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 -- 172.21.15.111:0/3014237879 >> v1:172.21.15.143:6789/0 conn(0x7ff17006de90 legacy=0x7ff1700c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 -- 172.21.15.111:0/3014237879 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17006d690 con 0x7ff1700aca70 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.368+0000 7ff175ee2700 1 --2- 172.21.15.111:0/3014237879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1700aca70 0x7ff1700acf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff170064370 tx=0x7ff168009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab00cd70d77813d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff167fff700 1 -- 172.21.15.111:0/3014237879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16800e070 con 0x7ff1700aca70 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff167fff700 1 -- 172.21.15.111:0/3014237879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16800ba50 con 0x7ff1700aca70 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff167fff700 1 -- 172.21.15.111:0/3014237879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1680092e0 con 0x7ff1700aca70 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 -- 172.21.15.111:0/3014237879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1700aca70 msgr2=0x7ff1700acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 --2- 172.21.15.111:0/3014237879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1700aca70 0x7ff1700acf10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff170064370 tx=0x7ff168009840 comp rx=0 tx=0).stop 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 -- 172.21.15.111:0/3014237879 shutdown_connections 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 --2- 172.21.15.111:0/3014237879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1700aca70 0x7ff1700acf10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 --2- 172.21.15.111:0/3014237879 >> v2:172.21.15.143:3300/0 conn(0x7ff170157320 0x7ff1701576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 -- 172.21.15.111:0/3014237879 >> 172.21.15.111:0/3014237879 conn(0x7ff1700cdc20 msgr2=0x7ff1700ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 -- 172.21.15.111:0/3014237879 shutdown_connections 2023-12-19T07:47:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.372+0000 7ff1766e3700 1 -- 172.21.15.111:0/3014237879 wait complete. 2023-12-19T07:47:12.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 Processor -- start 2023-12-19T07:47:12.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 -- start start 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff17006de90 0x7ff1700b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1700aca70 0x7ff1700b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700b22e0 con 0x7ff170157320 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700b25c0 con 0x7ff17006de90 2023-12-19T07:47:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1766e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700b28a0 con 0x7ff1700aca70 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50136/0 (socket says 172.21.15.111:50136) 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 -- 172.21.15.111:0/859398798 learned_addr learned my addr 172.21.15.111:0/859398798 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff1756e1700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff17006de90 0x7ff1700b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff174ee0700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1700aca70 0x7ff1700b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 -- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1700aca70 msgr2=0x7ff1700b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1700aca70 0x7ff1700b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 -- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff17006de90 msgr2=0x7ff1700b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff17006de90 0x7ff1700b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 -- 172.21.15.111:0/859398798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff168009ae0 con 0x7ff170157320 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.376+0000 7ff175ee2700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff17006e6d0 tx=0x7ff168018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168008e80 con 0x7ff170157320 2023-12-19T07:47:12.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16800be60 con 0x7ff170157320 2023-12-19T07:47:12.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168026690 con 0x7ff170157320 2023-12-19T07:47:12.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1700c4d70 con 0x7ff170157320 2023-12-19T07:47:12.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1700b1f00 con 0x7ff170157320 2023-12-19T07:47:12.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17004be40 con 0x7ff170157320 2023-12-19T07:47:12.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.380+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff168003040 con 0x7ff170157320 2023-12-19T07:47:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.384+0000 7ff1657fa700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 0x7ff154062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.384+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff16807e4e0 con 0x7ff170157320 2023-12-19T07:47:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.384+0000 7ff1756e1700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 0x7ff154062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.384+0000 7ff1756e1700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 0x7ff154062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1700c4740 tx=0x7ff16c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.388+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff16804ee20 con 0x7ff170157320 2023-12-19T07:47:12.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.556+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff17006d970 con 0x7ff170157320 2023-12-19T07:47:12.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.556+0000 7ff1657fa700 1 -- 172.21.15.111:0/859398798 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12175 (secure 0 0 0) 0x7ff16807e030 con 0x7ff170157320 2023-12-19T07:47:12.541 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:12.541 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:09.095827+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:12.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 msgr2=0x7ff154062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 0x7ff154062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1700c4740 tx=0x7ff16c008040 comp rx=0 tx=0).stop 2023-12-19T07:47:12.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 msgr2=0x7ff170153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff17006e6d0 tx=0x7ff168018000 comp rx=0 tx=0).stop 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 shutdown_connections 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff154060550 0x7ff154062a00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff170157320 0x7ff170153d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1700aca70 0x7ff1700b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 --2- 172.21.15.111:0/859398798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff17006de90 0x7ff1700b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 >> 172.21.15.111:0/859398798 conn(0x7ff1700cdc20 msgr2=0x7ff170156500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 shutdown_connections 2023-12-19T07:47:12.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.560+0000 7ff1766e3700 1 -- 172.21.15.111:0/859398798 wait complete. 2023-12-19T07:47:12.564 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T07:47:12.564 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:12.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.744+0000 7f3ac7016700 1 Processor -- start 2023-12-19T07:47:12.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.744+0000 7f3ac7016700 1 -- start start 2023-12-19T07:47:12.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac7016700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3ac0158c90 0x7f3ac00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac7016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3ac006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac7016700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac006c850 con 0x7f3ac006aeb0 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac7016700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac006b470 con 0x7f3ac0158c90 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac7016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac006b750 con 0x7f3ac0159770 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5813700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3ac006aeb0 0x7f3ac0159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37130/0 (socket says 172.21.15.111:37130) 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5813700 1 -- 172.21.15.111:0/503530641 learned_addr learned my addr 172.21.15.111:0/503530641 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac6014700 1 --2- 172.21.15.111:0/503530641 >> v2:172.21.15.143:3300/0 conn(0x7f3ac0158c90 0x7f3ac00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac6815700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3ac006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894210025 0 0) 0x7f3ac006c850 con 0x7f3ac006aeb0 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac003570 con 0x7f3ac006aeb0 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3359713777 0 0) 0x7f3aac003570 con 0x7f3ac006aeb0 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 msgr2=0x7f3ac006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3ac006cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 >> v2:172.21.15.143:3300/0 conn(0x7f3ac0158c90 msgr2=0x7f3ac00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 --2- 172.21.15.111:0/503530641 >> v2:172.21.15.143:3300/0 conn(0x7f3ac0158c90 0x7f3ac00caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac006ba30 con 0x7f3ac006aeb0 2023-12-19T07:47:12.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3ab0001470 con 0x7f3ac006aeb0 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 >> v1:172.21.15.111:6789/0 conn(0x7f3ac006aeb0 legacy=0x7f3ac0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3aac003a60 0x7f3aac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac004810 0x7f3aac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac006b470 con 0x7f3aac004810 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac006b750 con 0x7f3aac003a60 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00ba60 con 0x7f3ac0159770 2023-12-19T07:47:12.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.748+0000 7f3ac5813700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac004810 0x7f3aac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac6014700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3aac003a60 0x7f3aac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac6815700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 -- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 msgr2=0x7f3aac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac0159770 0x7f3aac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 -- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3aac003a60 msgr2=0x7f3aac003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3aac003a60 0x7f3aac003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 -- 172.21.15.111:0/503530641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac003570 con 0x7f3aac004810 2023-12-19T07:47:12.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5813700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac004810 0x7f3aac004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c3f0 tx=0x7f3ab0001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977258644a4b6171 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000cc80 con 0x7f3aac004810 2023-12-19T07:47:12.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab000ce20 con 0x7f3aac004810 2023-12-19T07:47:12.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac5012700 1 -- 172.21.15.111:0/503530641 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0005b70 con 0x7f3aac004810 2023-12-19T07:47:12.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 -- 172.21.15.111:0/503530641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac004810 msgr2=0x7f3aac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 --2- 172.21.15.111:0/503530641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac004810 0x7f3aac004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c3f0 tx=0x7f3ab0001e90 comp rx=0 tx=0).stop 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac6815700 1 -- 172.21.15.111:0/503530641 reap_dead start 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 -- 172.21.15.111:0/503530641 shutdown_connections 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 -- 172.21.15.111:0/503530641 >> 172.21.15.111:0/503530641 conn(0x7f3ac0067bb0 msgr2=0x7f3ac00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 -- 172.21.15.111:0/503530641 shutdown_connections 2023-12-19T07:47:12.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.752+0000 7f3ac7016700 1 -- 172.21.15.111:0/503530641 wait complete. 2023-12-19T07:47:12.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.756+0000 7f3ac7016700 1 Processor -- start 2023-12-19T07:47:12.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.756+0000 7f3ac7016700 1 -- start start 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3aac004810 0x7f3ac0148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac0154b20 0x7f3ac0154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00c85d0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00c8e10 con 0x7f3ac0154b20 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00c5e60 con 0x7f3aac004810 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50150/0 (socket says 172.21.15.111:50150) 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 -- 172.21.15.111:0/1539028443 learned_addr learned my addr 172.21.15.111:0/1539028443 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:12.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac6815700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac0154b20 0x7f3ac0154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac6014700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3aac004810 0x7f3ac0148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 -- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3aac004810 msgr2=0x7f3ac0148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3aac004810 0x7f3ac0148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 -- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac0154b20 msgr2=0x7f3ac0154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac0154b20 0x7f3ac0154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 -- 172.21.15.111:0/1539028443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00c8800 con 0x7f3ac00c92a0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac5813700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0011000 tx=0x7f3ab0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000cc80 con 0x7f3ac00c92a0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab00072e0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac0148da0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab00076a0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.760+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac00c8aa0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.764+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac00c8f80 con 0x7f3ac00c92a0 2023-12-19T07:47:12.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.764+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3ab0019070 con 0x7f3ac00c92a0 2023-12-19T07:47:12.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.764+0000 7f3ab67fc700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 0x7f3aa4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:12.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.764+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3ab007d3e0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.764+0000 7f3ac6014700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 0x7f3aa4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:12.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.768+0000 7f3ac6014700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 0x7f3aa4062a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006f00 tx=0x7f3abc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:12.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.772+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3ab004dc80 con 0x7f3ac00c92a0 2023-12-19T07:47:12.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.940+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ac004be40 con 0x7f3ac00c92a0 2023-12-19T07:47:12.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ab67fc700 1 -- 172.21.15.111:0/1539028443 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ab00512a0 con 0x7f3ac00c92a0 2023-12-19T07:47:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 msgr2=0x7f3aa4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 0x7f3aa4062a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3abc006f00 tx=0x7f3abc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 msgr2=0x7f3ac00c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0011000 tx=0x7f3ab0015000 comp rx=0 tx=0).stop 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 shutdown_connections 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3aa4060550 0x7f3aa4062a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac0154b20 0x7f3ac0154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac00c92a0 0x7f3ac00c9710 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 --2- 172.21.15.111:0/1539028443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3aac004810 0x7f3ac0148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 >> 172.21.15.111:0/1539028443 conn(0x7f3ac0067bb0 msgr2=0x7f3ac00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 shutdown_connections 2023-12-19T07:47:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:12.944+0000 7f3ac7016700 1 -- 172.21.15.111:0/1539028443 wait complete. 2023-12-19T07:47:12.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:13.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.140+0000 7effe5c4c700 1 Processor -- start 2023-12-19T07:47:13.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.140+0000 7effe5c4c700 1 -- start start 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe5c4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7effe00cad10 0x7effe00cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe5c4c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effe006b750 0x7effe006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe5c4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe006b2c0 con 0x7effe006df20 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe5c4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe006a6b0 con 0x7effe00cad10 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe5c4c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe006a990 con 0x7effe006b750 2023-12-19T07:47:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7effe00cad10 0x7effe00cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe544b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7effe006df20 0x7effe015bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37138/0 (socket says 172.21.15.111:37138) 2023-12-19T07:47:13.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe544b700 1 -- 172.21.15.111:0/720364118 learned_addr learned my addr 172.21.15.111:0/720364118 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:13.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effdffff700 1 --2- 172.21.15.111:0/720364118 >> v2:172.21.15.143:3300/0 conn(0x7effe006b750 0x7effe006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 --2- 172.21.15.111:0/720364118 >> v2:172.21.15.111:3300/0 conn(0x7effe00cad10 0x7effe00cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50162/0 (socket says 172.21.15.111:50162) 2023-12-19T07:47:13.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 -- 172.21.15.111:0/720364118 >> v2:172.21.15.143:3300/0 conn(0x7effe006b750 msgr2=0x7effe006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:13.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 --2- 172.21.15.111:0/720364118 >> v2:172.21.15.143:3300/0 conn(0x7effe006b750 0x7effe006d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 -- 172.21.15.111:0/720364118 >> v1:172.21.15.111:6789/0 conn(0x7effe006df20 legacy=0x7effe015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 -- 172.21.15.111:0/720364118 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe006ac70 con 0x7effe00cad10 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effdf7fe700 1 -- 172.21.15.111:0/720364118 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171707989 0 0) 0x7effe006b2c0 con 0x7effe006df20 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effe4c4a700 1 --2- 172.21.15.111:0/720364118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe00cad10 0x7effe00cb0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7effe00696f0 tx=0x7effd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8651dfdd393b2234 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.144+0000 7effdf7fe700 1 -- 172.21.15.111:0/720364118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4011630 con 0x7effe00cad10 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effdf7fe700 1 -- 172.21.15.111:0/720364118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4011790 con 0x7effe00cad10 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effdf7fe700 1 -- 172.21.15.111:0/720364118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4011b30 con 0x7effe00cad10 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- 172.21.15.111:0/720364118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe00cad10 msgr2=0x7effe00cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 --2- 172.21.15.111:0/720364118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe00cad10 0x7effe00cb0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7effe00696f0 tx=0x7effd40091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- 172.21.15.111:0/720364118 shutdown_connections 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 --2- 172.21.15.111:0/720364118 >> v2:172.21.15.143:3300/0 conn(0x7effe006b750 0x7effe006d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 --2- 172.21.15.111:0/720364118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe00cad10 0x7effe00cb0e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- 172.21.15.111:0/720364118 >> 172.21.15.111:0/720364118 conn(0x7effe0067bb0 msgr2=0x7effe00cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- 172.21.15.111:0/720364118 shutdown_connections 2023-12-19T07:47:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- 172.21.15.111:0/720364118 wait complete. 2023-12-19T07:47:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 Processor -- start 2023-12-19T07:47:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.148+0000 7effe5c4c700 1 -- start start 2023-12-19T07:47:13.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe006df20 0x7effe00b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 0x7effe00b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01543d0 con 0x7effe006b750 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0154860 con 0x7effe00cad10 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe0153830 con 0x7effe006df20 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50178/0 (socket says 172.21.15.111:50178) 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 -- 172.21.15.111:0/3758185538 learned_addr learned my addr 172.21.15.111:0/3758185538 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe544b700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 0x7effe00b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effdffff700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe006df20 0x7effe00b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 -- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe006df20 msgr2=0x7effe00b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe006df20 0x7effe00b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 -- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 msgr2=0x7effe00b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 0x7effe00b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 -- 172.21.15.111:0/3758185538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4009420 con 0x7effe006b750 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe544b700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 0x7effe00b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe4c4a700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7effd4009f50 tx=0x7effd4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4011b10 con 0x7effe006b750 2023-12-19T07:47:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd40253f0 con 0x7effe006b750 2023-12-19T07:47:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4025790 con 0x7effe006b750 2023-12-19T07:47:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0153aa0 con 0x7effe006b750 2023-12-19T07:47:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.152+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe0154060 con 0x7effe006b750 2023-12-19T07:47:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.156+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe004be40 con 0x7effe006b750 2023-12-19T07:47:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.156+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7effd4025b80 con 0x7effe006b750 2023-12-19T07:47:13.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.156+0000 7effdcff9700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 0x7effc4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.156+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7effd407dcb0 con 0x7effe006b750 2023-12-19T07:47:13.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.156+0000 7effdffff700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 0x7effc4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.160+0000 7effdffff700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 0x7effc4062a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7effe016c400 tx=0x7effcc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.164+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7effd404e5f0 con 0x7effe006b750 2023-12-19T07:47:13.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.376+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effe006af50 con 0x7effe006b750 2023-12-19T07:47:13.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.376+0000 7effdcff9700 1 -- 172.21.15.111:0/3758185538 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7effd4015020 con 0x7effe006b750 2023-12-19T07:47:13.361 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:13.361 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-12-19T07:47:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 msgr2=0x7effc4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 0x7effc4062a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7effe016c400 tx=0x7effcc008040 comp rx=0 tx=0).stop 2023-12-19T07:47:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 msgr2=0x7effe0151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7effd4009f50 tx=0x7effd4017000 comp rx=0 tx=0).stop 2023-12-19T07:47:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 shutdown_connections 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7effc4060550 0x7effc4062a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe00cad10 0x7effe00b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe006df20 0x7effe00b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 --2- 172.21.15.111:0/3758185538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effe006b750 0x7effe0151710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 >> 172.21.15.111:0/3758185538 conn(0x7effe0067bb0 msgr2=0x7effe00b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.380+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 shutdown_connections 2023-12-19T07:47:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.384+0000 7effe5c4c700 1 -- 172.21.15.111:0/3758185538 wait complete. 2023-12-19T07:47:13.368 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 3 2023-12-19T07:47:13.388 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:13.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 Processor -- start 2023-12-19T07:47:13.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 -- start start 2023-12-19T07:47:13.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064140ae0 0x7f50640680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5064068630 0x7f50640ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506406e150 con 0x7f5064140710 2023-12-19T07:47:13.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5064068ab0 con 0x7f5064068630 2023-12-19T07:47:13.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f506a6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506406d540 con 0x7f5064140ae0 2023-12-19T07:47:13.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f5063fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5064140710 0x7f50640c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37144/0 (socket says 172.21.15.111:37144) 2023-12-19T07:47:13.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.568+0000 7f5063fff700 1 -- 172.21.15.111:0/2995586050 learned_addr learned my addr 172.21.15.111:0/2995586050 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:13.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 --2- 172.21.15.111:0/2995586050 >> v2:172.21.15.111:3300/0 conn(0x7f5064068630 0x7f50640ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f505bfff700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064140ae0 0x7f50640680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f50637fe700 1 -- 172.21.15.111:0/2995586050 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88323678 0 0) 0x7f506406e150 con 0x7f5064140710 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f50637fe700 1 -- 172.21.15.111:0/2995586050 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5050003570 con 0x7f5064140710 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 -- 172.21.15.111:0/2995586050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064140ae0 msgr2=0x7f50640680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064140ae0 0x7f50640680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 -- 172.21.15.111:0/2995586050 >> v1:172.21.15.111:6789/0 conn(0x7f5064140710 legacy=0x7f50640c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 -- 172.21.15.111:0/2995586050 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506406d820 con 0x7f5064068630 2023-12-19T07:47:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f5068c8d700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064068630 0x7f50640ac250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f505c009e60 tx=0x7f505c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9442c399059940f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f50637fe700 1 -- 172.21.15.111:0/2995586050 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00e070 con 0x7f5064068630 2023-12-19T07:47:13.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f50637fe700 1 -- 172.21.15.111:0/2995586050 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c00bab0 con 0x7f5064068630 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f50637fe700 1 -- 172.21.15.111:0/2995586050 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00be70 con 0x7f5064068630 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 -- 172.21.15.111:0/2995586050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064068630 msgr2=0x7f50640ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064068630 0x7f50640ac250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f505c009e60 tx=0x7f505c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 -- 172.21.15.111:0/2995586050 shutdown_connections 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064068630 0x7f50640ac250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2995586050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064140ae0 0x7f50640680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 -- 172.21.15.111:0/2995586050 >> 172.21.15.111:0/2995586050 conn(0x7f50640c8850 msgr2=0x7f50640c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 -- 172.21.15.111:0/2995586050 shutdown_connections 2023-12-19T07:47:13.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.572+0000 7f506a6f0700 1 -- 172.21.15.111:0/2995586050 wait complete. 2023-12-19T07:47:13.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 Processor -- start 2023-12-19T07:47:13.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 -- start start 2023-12-19T07:47:13.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064068630 0x7f5064156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064140ae0 0x7f50640a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50640aaa10 con 0x7f5064140710 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5064168a20 con 0x7f5064140ae0 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f506a6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5064158370 con 0x7f5064068630 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f505bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f505bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50200/0 (socket says 172.21.15.111:50200) 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.576+0000 7f505bfff700 1 -- 172.21.15.111:0/2205024017 learned_addr learned my addr 172.21.15.111:0/2205024017 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f5063fff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064068630 0x7f5064156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f5068c8d700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064140ae0 0x7f50640a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 -- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064068630 msgr2=0x7f5064156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064068630 0x7f5064156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 -- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064140ae0 msgr2=0x7f50640a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064140ae0 0x7f50640a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 -- 172.21.15.111:0/2205024017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c009a20 con 0x7f5064140710 2023-12-19T07:47:13.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f505bfff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f504c00ec80 tx=0x7f504c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c00c9d0 con 0x7f5064140710 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504c00cb30 con 0x7f5064140710 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5064168ce0 con 0x7f5064140710 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c013470 con 0x7f5064140710 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50641692a0 con 0x7f5064140710 2023-12-19T07:47:13.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.580+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064071120 con 0x7f5064140710 2023-12-19T07:47:13.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.584+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f504c0137e0 con 0x7f5064140710 2023-12-19T07:47:13.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.584+0000 7f5060ff9700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 0x7f5044062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.584+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f504c07c880 con 0x7f5064140710 2023-12-19T07:47:13.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.584+0000 7f5063fff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 0x7f5044062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.588+0000 7f5063fff700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 0x7f5044062a80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50540058d0 tx=0x7f5054009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.592+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f504c04d120 con 0x7f5064140710 2023-12-19T07:47:13.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.764+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50640642a0 con 0x7f5064140710 2023-12-19T07:47:13.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f5060ff9700 1 -- 172.21.15.111:0/2205024017 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12175 (secure 0 0 0) 0x7f504c050740 con 0x7f5064140710 2023-12-19T07:47:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:13.751 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:09.095827+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:13.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 msgr2=0x7f5044062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 0x7f5044062a80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50540058d0 tx=0x7f5054009040 comp rx=0 tx=0).stop 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 msgr2=0x7f50640a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f504c00ec80 tx=0x7f504c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 shutdown_connections 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50440605d0 0x7f5044062a80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064140ae0 0x7f50640a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5064140710 0x7f50640a9820 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 --2- 172.21.15.111:0/2205024017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5064068630 0x7f5064156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 >> 172.21.15.111:0/2205024017 conn(0x7f50640c8850 msgr2=0x7f50640ac250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:13.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 shutdown_connections 2023-12-19T07:47:13.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.768+0000 7f506a6f0700 1 -- 172.21.15.111:0/2205024017 wait complete. 2023-12-19T07:47:13.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T07:47:13.773 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.948+0000 7f8d12278700 1 Processor -- start 2023-12-19T07:47:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.948+0000 7f8d12278700 1 -- start start 2023-12-19T07:47:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d0c0c6d20 0x7f8d0c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d0c06c7c0 0x7f8d0c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c06a7a0 con 0x7f8d0c15aa30 2023-12-19T07:47:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c7630 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c06d4b0 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d0c0c6d20 0x7f8d0c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d10815700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8d0c15aa30 0x7f8d0c15ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38022/0 (socket says 172.21.15.111:38022) 2023-12-19T07:47:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d10815700 1 -- 172.21.15.111:0/2050479440 learned_addr learned my addr 172.21.15.111:0/2050479440 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0b7fe700 1 --2- 172.21.15.111:0/2050479440 >> v2:172.21.15.143:3300/0 conn(0x7f8d0c06c7c0 0x7f8d0c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0affd700 1 -- 172.21.15.111:0/2050479440 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147500827 0 0) 0x7f8d0c06a7a0 con 0x7f8d0c15aa30 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2050479440 >> v2:172.21.15.143:3300/0 conn(0x7f8d0c06c7c0 msgr2=0x7f8d0c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2050479440 >> v2:172.21.15.143:3300/0 conn(0x7f8d0c06c7c0 0x7f8d0c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2050479440 >> v1:172.21.15.143:6789/0 conn(0x7f8d0c15aa30 legacy=0x7f8d0c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2050479440 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c06d790 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2050479440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c0c70f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0c86f0 tx=0x7f8cfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aad79407b3608e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0affd700 1 -- 172.21.15.111:0/2050479440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc011630 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0affd700 1 -- 172.21.15.111:0/2050479440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc011790 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d0affd700 1 -- 172.21.15.111:0/2050479440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc011b30 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 -- 172.21.15.111:0/2050479440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 msgr2=0x7f8d0c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 --2- 172.21.15.111:0/2050479440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c0c70f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0c86f0 tx=0x7f8cfc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.952+0000 7f8d12278700 1 -- 172.21.15.111:0/2050479440 shutdown_connections 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 --2- 172.21.15.111:0/2050479440 >> v2:172.21.15.143:3300/0 conn(0x7f8d0c06c7c0 0x7f8d0c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 --2- 172.21.15.111:0/2050479440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c0c70f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 -- 172.21.15.111:0/2050479440 >> 172.21.15.111:0/2050479440 conn(0x7f8d0c067bb0 msgr2=0x7f8d0c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 -- 172.21.15.111:0/2050479440 shutdown_connections 2023-12-19T07:47:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 -- 172.21.15.111:0/2050479440 wait complete. 2023-12-19T07:47:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 Processor -- start 2023-12-19T07:47:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.956+0000 7f8d12278700 1 -- start start 2023-12-19T07:47:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 0x7f8d0c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d0c15aa30 0x7f8d0c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c155840 con 0x7f8d0c0c6d20 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0c9a60 con 0x7f8d0c15aa30 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c154cc0 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50220/0 (socket says 172.21.15.111:50220) 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0b7fe700 1 -- 172.21.15.111:0/2492177264 learned_addr learned my addr 172.21.15.111:0/2492177264 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 0x7f8d0c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d10815700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d0c15aa30 0x7f8d0c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d0c15aa30 msgr2=0x7f8d0c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d0c15aa30 0x7f8d0c0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2492177264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 msgr2=0x7f8d0c156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 -- 172.21.15.111:0/2492177264 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc009420 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0b7fe700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d0bfff700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 0x7f8d0c1529e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc016000 tx=0x7f8cfc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc000d10 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc011b10 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc025640 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c154fa0 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.960+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c155560 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.964+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c071120 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.964+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8cfc0193b0 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.968+0000 7f8cf3fff700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 0x7f8cf8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.968+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8cfc07dd50 con 0x7f8d0c06c7c0 2023-12-19T07:47:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.968+0000 7f8d0b7fe700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 0x7f8cf8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:13.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.968+0000 7f8d0b7fe700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 0x7f8cf8062a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c154520 tx=0x7f8d000091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:13.980+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8cfc04e5f0 con 0x7f8d0c06c7c0 2023-12-19T07:47:14.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.148+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d0c06da70 con 0x7f8d0c06c7c0 2023-12-19T07:47:14.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.148+0000 7f8cf3fff700 1 -- 172.21.15.111:0/2492177264 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8cfc051c10 con 0x7f8d0c06c7c0 2023-12-19T07:47:14.131 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:14.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 msgr2=0x7f8cf8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 0x7f8cf8062a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c154520 tx=0x7f8d000091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 msgr2=0x7f8d0c1529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 0x7f8d0c1529e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc016000 tx=0x7f8cfc017000 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 shutdown_connections 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8cf8060580 0x7f8cf8062a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d0c15aa30 0x7f8d0c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d0c0c6d20 0x7f8d0c156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 --2- 172.21.15.111:0/2492177264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d0c06c7c0 0x7f8d0c1529e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 >> 172.21.15.111:0/2492177264 conn(0x7f8d0c067bb0 msgr2=0x7f8d0c0ad2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 shutdown_connections 2023-12-19T07:47:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.152+0000 7f8d12278700 1 -- 172.21.15.111:0/2492177264 wait complete. 2023-12-19T07:47:14.157 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T07:47:14.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 Processor -- start 2023-12-19T07:47:14.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 -- start start 2023-12-19T07:47:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fce740c5fd0 0x7fce740cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce740c63a0 con 0x7fce740ae210 2023-12-19T07:47:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7406a7b0 con 0x7fce740c5fd0 2023-12-19T07:47:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.328+0000 7fce79e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce740ad550 con 0x7fce74159160 2023-12-19T07:47:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:37334/0 (socket says 172.21.15.111:37334) 2023-12-19T07:47:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 -- 172.21.15.111:0/1192929067 learned_addr learned my addr 172.21.15.111:0/1192929067 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce7960b700 1 --1- 172.21.15.111:0/1192929067 >> v1:172.21.15.143:6789/0 conn(0x7fce740ae210 0x7fce74158200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38024/0 (socket says 172.21.15.111:38024) 2023-12-19T07:47:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce78e0a700 1 --2- 172.21.15.111:0/1192929067 >> v2:172.21.15.143:3300/0 conn(0x7fce740c5fd0 0x7fce740cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 -- 172.21.15.111:0/1192929067 >> v2:172.21.15.143:3300/0 conn(0x7fce740c5fd0 msgr2=0x7fce740cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 --2- 172.21.15.111:0/1192929067 >> v2:172.21.15.143:3300/0 conn(0x7fce740c5fd0 0x7fce740cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 -- 172.21.15.111:0/1192929067 >> v1:172.21.15.143:6789/0 conn(0x7fce740ae210 legacy=0x7fce74158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 -- 172.21.15.111:0/1192929067 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce740accb0 con 0x7fce74159160 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce737fe700 1 -- 172.21.15.111:0/1192929067 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052848269 0 0) 0x7fce740c63a0 con 0x7fce740ae210 2023-12-19T07:47:14.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce73fff700 1 --2- 172.21.15.111:0/1192929067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce600099b0 tx=0x7fce60009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7921f74367f273 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce737fe700 1 -- 172.21.15.111:0/1192929067 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60009ea0 con 0x7fce74159160 2023-12-19T07:47:14.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce737fe700 1 -- 172.21.15.111:0/1192929067 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6000b860 con 0x7fce74159160 2023-12-19T07:47:14.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce737fe700 1 -- 172.21.15.111:0/1192929067 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6000bc00 con 0x7fce74159160 2023-12-19T07:47:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce79e0c700 1 -- 172.21.15.111:0/1192929067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 msgr2=0x7fce740adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.332+0000 7fce79e0c700 1 --2- 172.21.15.111:0/1192929067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce600099b0 tx=0x7fce60009250 comp rx=0 tx=0).stop 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 -- 172.21.15.111:0/1192929067 shutdown_connections 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 --2- 172.21.15.111:0/1192929067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce74159160 0x7fce740adcd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 --2- 172.21.15.111:0/1192929067 >> v2:172.21.15.143:3300/0 conn(0x7fce740c5fd0 0x7fce740cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 -- 172.21.15.111:0/1192929067 >> 172.21.15.111:0/1192929067 conn(0x7fce74067bb0 msgr2=0x7fce740cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 -- 172.21.15.111:0/1192929067 shutdown_connections 2023-12-19T07:47:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 -- 172.21.15.111:0/1192929067 wait complete. 2023-12-19T07:47:14.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.336+0000 7fce79e0c700 1 Processor -- start 2023-12-19T07:47:14.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 -- start start 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce740ae210 0x7fce74149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74159160 0x7fce7414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74149890 con 0x7fce740c5fd0 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce740b19e0 con 0x7fce74159160 2023-12-19T07:47:14.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce79e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7414a6b0 con 0x7fce740ae210 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50236/0 (socket says 172.21.15.111:50236) 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 -- 172.21.15.111:0/3730432776 learned_addr learned my addr 172.21.15.111:0/3730432776 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce78e0a700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce740ae210 0x7fce74149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce7960b700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74159160 0x7fce7414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 -- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce740ae210 msgr2=0x7fce74149350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce740ae210 0x7fce74149350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 -- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74159160 msgr2=0x7fce7414ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74159160 0x7fce7414ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 -- 172.21.15.111:0/3730432776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600094f0 con 0x7fce740c5fd0 2023-12-19T07:47:14.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.340+0000 7fce73fff700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce7406a8f0 tx=0x7fce60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60000d10 con 0x7fce740c5fd0 2023-12-19T07:47:14.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6000bbe0 con 0x7fce740c5fd0 2023-12-19T07:47:14.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce74149aa0 con 0x7fce740c5fd0 2023-12-19T07:47:14.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60011690 con 0x7fce740c5fd0 2023-12-19T07:47:14.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7414a0e0 con 0x7fce740c5fd0 2023-12-19T07:47:14.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7414a860 con 0x7fce740c5fd0 2023-12-19T07:47:14.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.344+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fce60011ac0 con 0x7fce740c5fd0 2023-12-19T07:47:14.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.348+0000 7fce70ff9700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 0x7fce58062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.348+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fce6007dc00 con 0x7fce740c5fd0 2023-12-19T07:47:14.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.348+0000 7fce78e0a700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 0x7fce58062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.348+0000 7fce78e0a700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 0x7fce58062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce68009db0 tx=0x7fce680091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.352+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce7414a860 con 0x7fce740c5fd0 2023-12-19T07:47:14.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.528+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fce7414a460 con 0x7fce740c5fd0 2023-12-19T07:47:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.528+0000 7fce70ff9700 1 -- 172.21.15.111:0/3730432776 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fce60015090 con 0x7fce740c5fd0 2023-12-19T07:47:14.510 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:47:14.511 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 msgr2=0x7fce58062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 0x7fce58062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce68009db0 tx=0x7fce680091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 msgr2=0x7fce740b14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce7406a8f0 tx=0x7fce60017000 comp rx=0 tx=0).stop 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 shutdown_connections 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce58060580 0x7fce58062a30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce74159160 0x7fce7414ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce740c5fd0 0x7fce740b14a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 --2- 172.21.15.111:0/3730432776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce740ae210 0x7fce74149350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 >> 172.21.15.111:0/3730432776 conn(0x7fce74067bb0 msgr2=0x7fce740c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:14.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 shutdown_connections 2023-12-19T07:47:14.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.532+0000 7fce79e0c700 1 -- 172.21.15.111:0/3730432776 wait complete. 2023-12-19T07:47:14.539 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T07:47:14.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T07:47:14.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 Processor -- start 2023-12-19T07:47:14.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 -- start start 2023-12-19T07:47:14.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 0x7fd76c0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0cb3f0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c0cc140 con 0x7fd76c06af50 2023-12-19T07:47:14.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77180c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c06d220 con 0x7fd76c0aedc0 2023-12-19T07:47:14.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd76bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 0x7fd76c0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77100b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd76c06d6b0 0x7fd76c0af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38032/0 (socket says 172.21.15.111:38032) 2023-12-19T07:47:14.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:37356/0 (socket says 172.21.15.111:37356) 2023-12-19T07:47:14.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77100b700 1 -- 172.21.15.111:0/3058705990 learned_addr learned my addr 172.21.15.111:0/3058705990 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd76b7fe700 1 -- 172.21.15.111:0/3058705990 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146220053 0 0) 0x7fd76c0cb3f0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd76b7fe700 1 -- 172.21.15.111:0/3058705990 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75c003570 con 0x7fd76c06d6b0 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 -- 172.21.15.111:0/3058705990 >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 msgr2=0x7fd76c0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 --2- 172.21.15.111:0/3058705990 >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 0x7fd76c0cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 -- 172.21.15.111:0/3058705990 >> v1:172.21.15.143:6789/0 conn(0x7fd76c06d6b0 legacy=0x7fd76c0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd77080a700 1 -- 172.21.15.111:0/3058705990 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0cb190 con 0x7fd76c0aedc0 2023-12-19T07:47:14.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.724+0000 7fd76bfff700 1 --2- 172.21.15.111:0/3058705990 >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 0x7fd76c0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:14.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77080a700 1 --2- 172.21.15.111:0/3058705990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7600098e0 tx=0x7fd7600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc0744ad124170b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd76b7fe700 1 -- 172.21.15.111:0/3058705990 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760011630 con 0x7fd76c0aedc0 2023-12-19T07:47:14.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd76b7fe700 1 -- 172.21.15.111:0/3058705990 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760011790 con 0x7fd76c0aedc0 2023-12-19T07:47:14.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd76b7fe700 1 -- 172.21.15.111:0/3058705990 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760011b30 con 0x7fd76c0aedc0 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 -- 172.21.15.111:0/3058705990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 msgr2=0x7fd76c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 --2- 172.21.15.111:0/3058705990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7600098e0 tx=0x7fd7600091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 -- 172.21.15.111:0/3058705990 shutdown_connections 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 --2- 172.21.15.111:0/3058705990 >> v2:172.21.15.143:3300/0 conn(0x7fd76c06af50 0x7fd76c0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 --2- 172.21.15.111:0/3058705990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0af190 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 -- 172.21.15.111:0/3058705990 >> 172.21.15.111:0/3058705990 conn(0x7fd76c0cc760 msgr2=0x7fd76c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 -- 172.21.15.111:0/3058705990 shutdown_connections 2023-12-19T07:47:14.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.728+0000 7fd77180c700 1 -- 172.21.15.111:0/3058705990 wait complete. 2023-12-19T07:47:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 Processor -- start 2023-12-19T07:47:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 -- start start 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 0x7fd76c0cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15cab0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15bea0 con 0x7fd76c06af50 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.732+0000 7fd77180c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c15c180 con 0x7fd76c0aedc0 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 0x7fd76c0cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77080a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77100b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77080a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:40230/0 (socket says 172.21.15.111:40230) 2023-12-19T07:47:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77080a700 1 -- 172.21.15.111:0/62245906 learned_addr learned my addr 172.21.15.111:0/62245906 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:14.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 -- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 msgr2=0x7fd76c0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 -- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 msgr2=0x7fd76c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 -- 172.21.15.111:0/62245906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758009420 con 0x7fd76c06d6b0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77100b700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77080a700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd76bfff700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 0x7fd76c0cee10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ea00 tx=0x7fd7580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75800c6a0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd760009420 con 0x7fd76c06d6b0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75800c800 con 0x7fd76c06d6b0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75800cba0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.736+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c15c740 con 0x7fd76c06d6b0 2023-12-19T07:47:14.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.740+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c15cc60 con 0x7fd76c06d6b0 2023-12-19T07:47:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.740+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd758013470 con 0x7fd76c06d6b0 2023-12-19T07:47:14.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.744+0000 7fd768ff9700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.744+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd75807c6b0 con 0x7fd76c06d6b0 2023-12-19T07:47:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.744+0000 7fd77080a700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.744+0000 7fd77080a700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7600091e0 tx=0x7fd760017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:14.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.748+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd76c15cc60 con 0x7fd76c06d6b0 2023-12-19T07:47:14.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:14.924+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd76c04be40 con 0x7fd76c06d6b0 2023-12-19T07:47:15.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.148+0000 7fd768ff9700 1 -- 172.21.15.111:0/62245906 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd75804cff0 con 0x7fd76c06d6b0 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 msgr2=0x7fd750062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7600091e0 tx=0x7fd760017000 comp rx=0 tx=0).stop 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 msgr2=0x7fd76c0cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 0x7fd76c0cee10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ea00 tx=0x7fd7580091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 shutdown_connections 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c0aedc0 0x7fd76c0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd76c06d6b0 0x7fd76c0cee10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 --2- 172.21.15.111:0/62245906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06af50 0x7fd76c0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 >> 172.21.15.111:0/62245906 conn(0x7fd76c0cc760 msgr2=0x7fd76c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 shutdown_connections 2023-12-19T07:47:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.152+0000 7fd77180c700 1 -- 172.21.15.111:0/62245906 wait complete. 2023-12-19T07:47:15.139 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T07:47:15.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-12-19T07:47:15.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.344+0000 7f4f40945700 1 Processor -- start 2023-12-19T07:47:15.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.344+0000 7f4f40945700 1 -- start start 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f40945700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f40945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0ca280 con 0x7f4f3c06d050 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f40945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c15d7c0 con 0x7f4f3c158ed0 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f40945700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cca20 con 0x7f4f3c1592a0 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50242/0 (socket says 172.21.15.111:50242) 2023-12-19T07:47:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 -- 172.21.15.111:0/13707396 learned_addr learned my addr 172.21.15.111:0/13707396 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 -- 172.21.15.111:0/13707396 >> v1:172.21.15.143:6789/0 conn(0x7f4f3c158ed0 legacy=0x7f4f3c15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 -- 172.21.15.111:0/13707396 >> v1:172.21.15.111:6789/0 conn(0x7f4f3c06d050 legacy=0x7f4f3c0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 -- 172.21.15.111:0/13707396 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c15daa0 con 0x7f4f3c1592a0 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f33fff700 1 --2- 172.21.15.111:0/13707396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4f240099b0 tx=0x7f4f24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b91073e8ea36e45 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f39d9b700 1 -- 172.21.15.111:0/13707396 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2400b820 con 0x7f4f3c1592a0 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f39d9b700 1 -- 172.21.15.111:0/13707396 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2400b980 con 0x7f4f3c1592a0 2023-12-19T07:47:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.348+0000 7f4f39d9b700 1 -- 172.21.15.111:0/13707396 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2400bd20 con 0x7f4f3c1592a0 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 -- 172.21.15.111:0/13707396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 msgr2=0x7f4f3c0cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 --2- 172.21.15.111:0/13707396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4f240099b0 tx=0x7f4f24009250 comp rx=0 tx=0).stop 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 -- 172.21.15.111:0/13707396 shutdown_connections 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 --2- 172.21.15.111:0/13707396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c0cc4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 -- 172.21.15.111:0/13707396 >> 172.21.15.111:0/13707396 conn(0x7f4f3c0cd8f0 msgr2=0x7f4f3c0c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 -- 172.21.15.111:0/13707396 shutdown_connections 2023-12-19T07:47:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 -- 172.21.15.111:0/13707396 wait complete. 2023-12-19T07:47:15.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.352+0000 7f4f40945700 1 Processor -- start 2023-12-19T07:47:15.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- start start 2023-12-19T07:47:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f3c06d050 0x7f4f3c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 0x7f4f3c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b07d0 con 0x7f4f3c1592a0 2023-12-19T07:47:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0afbc0 con 0x7f4f3c06d050 2023-12-19T07:47:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0afea0 con 0x7f4f3c158ed0 2023-12-19T07:47:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50244/0 (socket says 172.21.15.111:50244) 2023-12-19T07:47:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 -- 172.21.15.111:0/3469108638 learned_addr learned my addr 172.21.15.111:0/3469108638 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:15.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3a59c700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f3c06d050 0x7f4f3c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:15.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f33fff700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 0x7f4f3c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:15.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 -- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 msgr2=0x7f4f3c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 0x7f4f3c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 -- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f3c06d050 msgr2=0x7f4f3c156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f3c06d050 0x7f4f3c156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 -- 172.21.15.111:0/3469108638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f240094f0 con 0x7f4f3c1592a0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f33fff700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 0x7f4f3c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f3ad9d700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400c8e0 tx=0x7f4f34001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400ce80 con 0x7f4f3c1592a0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400a7e0 con 0x7f4f3c1592a0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c0b0180 con 0x7f4f3c1592a0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400ab80 con 0x7f4f3c1592a0 2023-12-19T07:47:15.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.356+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c157420 con 0x7f4f3c1592a0 2023-12-19T07:47:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.360+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c0b0940 con 0x7f4f3c1592a0 2023-12-19T07:47:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.360+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4f34024420 con 0x7f4f3c1592a0 2023-12-19T07:47:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.364+0000 7f4f327fc700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 0x7f4f1c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.364+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5223+0+0 (secure 0 0 0) 0x7f4f3407d110 con 0x7f4f3c1592a0 2023-12-19T07:47:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.364+0000 7f4f3a59c700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 0x7f4f1c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.364+0000 7f4f3a59c700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 0x7f4f1c062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c003870 tx=0x7f4f2c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:15.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.368+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4f3404d8e0 con 0x7f4f3c1592a0 2023-12-19T07:47:15.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:15.544+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f4f3c071120 con 0x7f4f3c1592a0 2023-12-19T07:47:16.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.164+0000 7f4f327fc700 1 -- 172.21.15.111:0/3469108638 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f4f34050f00 con 0x7f4f3c1592a0 2023-12-19T07:47:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.176+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 msgr2=0x7f4f1c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.176+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 0x7f4f1c062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c003870 tx=0x7f4f2c009040 comp rx=0 tx=0).stop 2023-12-19T07:47:16.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.176+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 msgr2=0x7f4f3c1539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:16.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.176+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400c8e0 tx=0x7f4f34001710 comp rx=0 tx=0).stop 2023-12-19T07:47:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 shutdown_connections 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f1c060580 0x7f4f1c062a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c1592a0 0x7f4f3c1539f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f3c158ed0 0x7f4f3c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 --2- 172.21.15.111:0/3469108638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f3c06d050 0x7f4f3c156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.180+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 >> 172.21.15.111:0/3469108638 conn(0x7f4f3c0cd8f0 msgr2=0x7f4f3c155700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.184+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 shutdown_connections 2023-12-19T07:47:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.184+0000 7f4f40945700 1 -- 172.21.15.111:0/3469108638 wait complete. 2023-12-19T07:47:16.177 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T07:47:16.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T07:47:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.376+0000 7fb6f546e700 1 Processor -- start 2023-12-19T07:47:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.376+0000 7fb6f546e700 1 -- start start 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb6f00c6d20 0x7fb6f00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6f015aa30 0x7fb6f015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f006a7a0 con 0x7fb6f00c6d20 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f00c7630 con 0x7fb6f015aa30 2023-12-19T07:47:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f006d4b0 con 0x7fb6f006c7c0 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb6f00c6d20 0x7fb6f00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb6f00c6d20 0x7fb6f00c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50248/0 (socket says 172.21.15.111:50248) 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 -- 172.21.15.111:0/3892486166 learned_addr learned my addr 172.21.15.111:0/3892486166 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6ef7fe700 1 --2- 172.21.15.111:0/3892486166 >> v2:172.21.15.143:3300/0 conn(0x7fb6f015aa30 0x7fb6f015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6ee7fc700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 -- 172.21.15.111:0/3892486166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 msgr2=0x7fb6f006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 -- 172.21.15.111:0/3892486166 >> v2:172.21.15.143:3300/0 conn(0x7fb6f015aa30 msgr2=0x7fb6f015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/3892486166 >> v2:172.21.15.143:3300/0 conn(0x7fb6f015aa30 0x7fb6f015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 -- 172.21.15.111:0/3892486166 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f006d790 con 0x7fb6f00c6d20 2023-12-19T07:47:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f00c6d20 0x7fb6f00c70f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00cc7b0 tx=0x7fb6e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9726e03f15ff75cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6edffb700 1 -- 172.21.15.111:0/3892486166 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0009e70 con 0x7fb6f00c6d20 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6edffb700 1 -- 172.21.15.111:0/3892486166 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e000b860 con 0x7fb6f00c6d20 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6edffb700 1 -- 172.21.15.111:0/3892486166 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e000bc00 con 0x7fb6f00c6d20 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- 172.21.15.111:0/3892486166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f00c6d20 msgr2=0x7fb6f00c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f00c6d20 0x7fb6f00c70f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00cc7b0 tx=0x7fb6e0009250 comp rx=0 tx=0).stop 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- 172.21.15.111:0/3892486166 shutdown_connections 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- 172.21.15.111:0/3892486166 >> v2:172.21.15.143:3300/0 conn(0x7fb6f015aa30 0x7fb6f015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 --2- 172.21.15.111:0/3892486166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f00c6d20 0x7fb6f00c70f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- 172.21.15.111:0/3892486166 >> 172.21.15.111:0/3892486166 conn(0x7fb6f0067bb0 msgr2=0x7fb6f00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- 172.21.15.111:0/3892486166 shutdown_connections 2023-12-19T07:47:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.380+0000 7fb6f546e700 1 -- 172.21.15.111:0/3892486166 wait complete. 2023-12-19T07:47:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 Processor -- start 2023-12-19T07:47:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 -- start start 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f00a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6f00c6d20 0x7fb6f014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c40a0 con 0x7fb6f015aa30 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00aab80 con 0x7fb6f00c6d20 2023-12-19T07:47:16.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6f546e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c4c50 con 0x7fb6f006c7c0 2023-12-19T07:47:16.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50252/0 (socket says 172.21.15.111:50252) 2023-12-19T07:47:16.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.384+0000 7fb6ef7fe700 1 -- 172.21.15.111:0/1560371546 learned_addr learned my addr 172.21.15.111:0/1560371546 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:16.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ee7fc700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6f00c6d20 0x7fb6f014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f00a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 -- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 msgr2=0x7fb6f00a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f00a8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 -- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6f00c6d20 msgr2=0x7fb6f014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6f00c6d20 0x7fb6f014b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 -- 172.21.15.111:0/1560371546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e00094f0 con 0x7fb6f015aa30 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6ef7fe700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009df0 tx=0x7fb6e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800caa0 con 0x7fb6f015aa30 2023-12-19T07:47:16.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e800f040 con 0x7fb6f015aa30 2023-12-19T07:47:16.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f00a9230 con 0x7fb6f015aa30 2023-12-19T07:47:16.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800ce70 con 0x7fb6f015aa30 2023-12-19T07:47:16.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00ab0b0 con 0x7fb6f015aa30 2023-12-19T07:47:16.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.388+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f00c4dc0 con 0x7fb6f015aa30 2023-12-19T07:47:16.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.392+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb6e8014690 con 0x7fb6f015aa30 2023-12-19T07:47:16.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.392+0000 7fb6d6ffd700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 0x7fb6dc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:16.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.392+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5605+0+0 (secure 0 0 0) 0x7fb6e807cf40 con 0x7fb6f015aa30 2023-12-19T07:47:16.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.392+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 0x7fb6dc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:16.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.396+0000 7fb6eeffd700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 0x7fb6dc062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00cc7b0 tx=0x7fb6e0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:16.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.400+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb6e8051020 con 0x7fb6f015aa30 2023-12-19T07:47:16.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:16.596+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb6f006da70 con 0x7fb6f015aa30 2023-12-19T07:47:17.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.200+0000 7fb6d6ffd700 1 -- 172.21.15.111:0/1560371546 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb6e80180c0 con 0x7fb6f015aa30 2023-12-19T07:47:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.204+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 msgr2=0x7fb6dc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.204+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 0x7fb6dc062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00cc7b0 tx=0x7fb6e0017000 comp rx=0 tx=0).stop 2023-12-19T07:47:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 msgr2=0x7fb6f00ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009df0 tx=0x7fb6e800b040 comp rx=0 tx=0).stop 2023-12-19T07:47:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 shutdown_connections 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb6dc060550 0x7fb6dc062a00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6f00c6d20 0x7fb6f014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6f015aa30 0x7fb6f00ca250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 --2- 172.21.15.111:0/1560371546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6f006c7c0 0x7fb6f00a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 >> 172.21.15.111:0/1560371546 conn(0x7fb6f0067bb0 msgr2=0x7fb6f00ad6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 shutdown_connections 2023-12-19T07:47:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.208+0000 7fb6f546e700 1 -- 172.21.15.111:0/1560371546 wait complete. 2023-12-19T07:47:17.193 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 2 and data pool 3 2023-12-19T07:47:17.214 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-12-19T07:47:17.214 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-12-19T07:47:17.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 Processor -- start 2023-12-19T07:47:17.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 -- start start 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e901570a0 0x7f2e90068440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e900c87a0 0x7f2e90068980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9013a670 con 0x7f2e900c87a0 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900cb4a0 con 0x7f2e901570a0 2023-12-19T07:47:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.396+0000 7f2e95785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900cb710 con 0x7f2e90156cd0 2023-12-19T07:47:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e900c87a0 0x7f2e90068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e901570a0 0x7f2e90068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e900c87a0 0x7f2e90068980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50258/0 (socket says 172.21.15.111:50258) 2023-12-19T07:47:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/521069693 learned_addr learned my addr 172.21.15.111:0/521069693 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/521069693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 msgr2=0x7f2e900c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/521069693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/521069693 >> v2:172.21.15.143:3300/0 conn(0x7f2e901570a0 msgr2=0x7f2e90068440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/521069693 >> v2:172.21.15.143:3300/0 conn(0x7f2e901570a0 0x7f2e90068440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/521069693 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900cb9f0 con 0x7f2e900c87a0 2023-12-19T07:47:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/521069693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e900c87a0 0x7f2e90068980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e9006ef40 tx=0x7f2e88009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8f16b483fd3a1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8dffb700 1 -- 172.21.15.111:0/521069693 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800e070 con 0x7f2e900c87a0 2023-12-19T07:47:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8dffb700 1 -- 172.21.15.111:0/521069693 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8800b970 con 0x7f2e900c87a0 2023-12-19T07:47:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e8dffb700 1 -- 172.21.15.111:0/521069693 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e880092e0 con 0x7f2e900c87a0 2023-12-19T07:47:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 -- 172.21.15.111:0/521069693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e900c87a0 msgr2=0x7f2e90068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 --2- 172.21.15.111:0/521069693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e900c87a0 0x7f2e90068980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e9006ef40 tx=0x7f2e88009840 comp rx=0 tx=0).stop 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 -- 172.21.15.111:0/521069693 shutdown_connections 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 --2- 172.21.15.111:0/521069693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e900c87a0 0x7f2e90068980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 --2- 172.21.15.111:0/521069693 >> v2:172.21.15.143:3300/0 conn(0x7f2e901570a0 0x7f2e90068440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 --2- 172.21.15.111:0/521069693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.400+0000 7f2e95785700 1 -- 172.21.15.111:0/521069693 >> 172.21.15.111:0/521069693 conn(0x7f2e9006adc0 msgr2=0x7f2e9006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.404+0000 7f2e95785700 1 -- 172.21.15.111:0/521069693 shutdown_connections 2023-12-19T07:47:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.404+0000 7f2e95785700 1 -- 172.21.15.111:0/521069693 wait complete. 2023-12-19T07:47:17.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.404+0000 7f2e95785700 1 Processor -- start 2023-12-19T07:47:17.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.404+0000 7f2e95785700 1 -- start start 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 0x7f2e90136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900c6d00 con 0x7f2e901570a0 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90139360 con 0x7f2e900c87a0 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900c66b0 con 0x7f2e90156cd0 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50272/0 (socket says 172.21.15.111:50272) 2023-12-19T07:47:17.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/4107799172 learned_addr learned my addr 172.21.15.111:0/4107799172 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8effd700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 0x7f2e90136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8e7fc700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 msgr2=0x7f2e900c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 msgr2=0x7f2e90136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 0x7f2e90136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 -- 172.21.15.111:0/4107799172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88009ae0 con 0x7f2e901570a0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8effd700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 0x7f2e90136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e8f7fe700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e88012000 tx=0x7f2e88018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88008b80 con 0x7f2e901570a0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e88008ce0 con 0x7f2e901570a0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e900c6990 con 0x7f2e901570a0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e880266b0 con 0x7f2e901570a0 2023-12-19T07:47:17.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.408+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e901371a0 con 0x7f2e901570a0 2023-12-19T07:47:17.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.412+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e900c6eb0 con 0x7f2e901570a0 2023-12-19T07:47:17.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.412+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2e88003040 con 0x7f2e901570a0 2023-12-19T07:47:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.412+0000 7f2e76ffd700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 0x7f2e7c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.412+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5705+0+0 (secure 0 0 0) 0x7f2e8807e070 con 0x7f2e901570a0 2023-12-19T07:47:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.416+0000 7f2e8effd700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 0x7f2e7c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:17.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.416+0000 7f2e8effd700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 0x7f2e7c062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e80006f00 tx=0x7f2e80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:17.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.420+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e900c6eb0 con 0x7f2e901570a0 2023-12-19T07:47:17.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:17.600+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f2e9004be40 con 0x7f2e901570a0 2023-12-19T07:47:18.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.208+0000 7f2e76ffd700 1 -- 172.21.15.111:0/4107799172 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f2e8804e660 con 0x7f2e901570a0 2023-12-19T07:47:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 msgr2=0x7f2e7c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 0x7f2e7c062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e80006f00 tx=0x7f2e80008040 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 msgr2=0x7f2e900c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e88012000 tx=0x7f2e88018000 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 shutdown_connections 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e7c060550 0x7f2e7c062a00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e901570a0 0x7f2e900c5920 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e90156cd0 0x7f2e900c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 --2- 172.21.15.111:0/4107799172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e900c87a0 0x7f2e90136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 >> 172.21.15.111:0/4107799172 conn(0x7f2e9006adc0 msgr2=0x7f2e9006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 shutdown_connections 2023-12-19T07:47:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.212+0000 7f2e95785700 1 -- 172.21.15.111:0/4107799172 wait complete. 2023-12-19T07:47:18.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T07:47:18.340 INFO:tasks.ceph.mds.b.smithi143.stderr:2023-12-19T07:47:18.356+0000 7f4639d2a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T07:47:18.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.412+0000 7fcb824cd700 1 Processor -- start 2023-12-19T07:47:18.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.416+0000 7fcb824cd700 1 -- start start 2023-12-19T07:47:18.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.416+0000 7fcb824cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb7c06b690 0x7fcb7c0afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:18.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.416+0000 7fcb824cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c06c7c0 con 0x7fcb7c0b0110 2023-12-19T07:47:18.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.416+0000 7fcb824cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c06cd80 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.416+0000 7fcb824cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0b0940 con 0x7fcb7c06b690 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb7c06b690 0x7fcb7c0afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb7c06b690 0x7fcb7c0afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50288/0 (socket says 172.21.15.111:50288) 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 -- 172.21.15.111:0/3180529873 learned_addr learned my addr 172.21.15.111:0/3180529873 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 -- 172.21.15.111:0/3180529873 >> v1:172.21.15.143:6789/0 conn(0x7fcb7c06b2c0 legacy=0x7fcb7c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 -- 172.21.15.111:0/3180529873 >> v1:172.21.15.111:6789/0 conn(0x7fcb7c0b0110 legacy=0x7fcb7c158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 -- 172.21.15.111:0/3180529873 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0b0c20 con 0x7fcb7c06b690 2023-12-19T07:47:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb737fe700 1 --2- 172.21.15.111:0/3180529873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b690 0x7fcb7c0afbd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb640099b0 tx=0x7fcb64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559eb55676ad0ced server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:18.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb7b7fe700 1 -- 172.21.15.111:0/3180529873 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400b860 con 0x7fcb7c06b690 2023-12-19T07:47:18.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb7b7fe700 1 -- 172.21.15.111:0/3180529873 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6400b9c0 con 0x7fcb7c06b690 2023-12-19T07:47:18.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb7b7fe700 1 -- 172.21.15.111:0/3180529873 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400bd60 con 0x7fcb7c06b690 2023-12-19T07:47:18.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 -- 172.21.15.111:0/3180529873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b690 msgr2=0x7fcb7c0afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 --2- 172.21.15.111:0/3180529873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b690 0x7fcb7c0afbd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb640099b0 tx=0x7fcb64009250 comp rx=0 tx=0).stop 2023-12-19T07:47:18.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 -- 172.21.15.111:0/3180529873 shutdown_connections 2023-12-19T07:47:18.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 --2- 172.21.15.111:0/3180529873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b690 0x7fcb7c0afbd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 -- 172.21.15.111:0/3180529873 >> 172.21.15.111:0/3180529873 conn(0x7fcb7c0cc580 msgr2=0x7fcb7c0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 -- 172.21.15.111:0/3180529873 shutdown_connections 2023-12-19T07:47:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.420+0000 7fcb824cd700 1 -- 172.21.15.111:0/3180529873 wait complete. 2023-12-19T07:47:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.424+0000 7fcb824cd700 1 Processor -- start 2023-12-19T07:47:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.424+0000 7fcb824cd700 1 -- start start 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 0x7fcb7c15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 0x7fcb7c0c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c1fd0 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c13c0 con 0x7fcb7c0b0110 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c16a0 con 0x7fcb7c06b690 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50302/0 (socket says 172.21.15.111:50302) 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 -- 172.21.15.111:0/750347886 learned_addr learned my addr 172.21.15.111:0/750347886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb737fe700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 0x7fcb7c15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb80a6a700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 0x7fcb7c0c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 -- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 msgr2=0x7fcb7c15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 0x7fcb7c15c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 -- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 msgr2=0x7fcb7c0c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 0x7fcb7c0c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 -- 172.21.15.111:0/750347886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb640094f0 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb737fe700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 0x7fcb7c15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb80a6a700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 0x7fcb7c0c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb7bfff700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c590 tx=0x7fcb6c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c011590 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c0116f0 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c0c1980 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c011ab0 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c15b760 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.428+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c071120 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.432+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fcb6c01b3b0 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.432+0000 7fcb78ff9700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 0x7fcb5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.432+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcb6c07be60 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.432+0000 7fcb737fe700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 0x7fcb5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:18.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.432+0000 7fcb737fe700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 0x7fcb5c062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcb640099b0 tx=0x7fcb64017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.436+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcb6c04c370 con 0x7fcb7c06b2c0 2023-12-19T07:47:18.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:18.616+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcb7c06ca10 con 0x7fcb7c06b2c0 2023-12-19T07:47:20.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.224+0000 7fcb78ff9700 1 -- 172.21.15.111:0/750347886 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7fcb6c017020 con 0x7fcb7c06b2c0 2023-12-19T07:47:20.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 msgr2=0x7fcb5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 0x7fcb5c062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcb640099b0 tx=0x7fcb64017000 comp rx=0 tx=0).stop 2023-12-19T07:47:20.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 msgr2=0x7fcb7c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c590 tx=0x7fcb6c00c950 comp rx=0 tx=0).stop 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 shutdown_connections 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb5c060550 0x7fcb5c062a00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb7c0b0110 0x7fcb7c0c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb7c06b690 0x7fcb7c15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 --2- 172.21.15.111:0/750347886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb7c06b2c0 0x7fcb7c15b220 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00c590 tx=0x7fcb6c00c950 comp rx=0 tx=0).stop 2023-12-19T07:47:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.232+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 >> 172.21.15.111:0/750347886 conn(0x7fcb7c0cc580 msgr2=0x7fcb7c155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.236+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 shutdown_connections 2023-12-19T07:47:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.236+0000 7fcb824cd700 1 -- 172.21.15.111:0/750347886 wait complete. 2023-12-19T07:47:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' created 2023-12-19T07:47:20.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-12-19T07:47:20.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.420+0000 7f49ad2f0700 1 Processor -- start 2023-12-19T07:47:20.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.420+0000 7f49ad2f0700 1 -- start start 2023-12-19T07:47:20.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49ad2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 0x7f49a80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49ad2f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 0x7f49a806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49ad2f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a80b0ac0 con 0x7f49a806a6a0 2023-12-19T07:47:20.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49ad2f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a806b030 con 0x7f49a815c930 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49ad2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a806b210 con 0x7f49a80cb130 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 0x7f49a806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 0x7f49a80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 0x7f49a806aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50312/0 (socket says 172.21.15.111:50312) 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a77fe700 1 -- 172.21.15.111:0/729766903 learned_addr learned my addr 172.21.15.111:0/729766903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a67fc700 1 --1- 172.21.15.111:0/729766903 >> v1:172.21.15.111:6789/0 conn(0x7f49a806a6a0 0x7f49a815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37178/0 (socket says 172.21.15.111:37178) 2023-12-19T07:47:20.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a6ffd700 1 -- 172.21.15.111:0/729766903 >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 msgr2=0x7f49a806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:20.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a6ffd700 1 --2- 172.21.15.111:0/729766903 >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 0x7f49a806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a6ffd700 1 -- 172.21.15.111:0/729766903 >> v1:172.21.15.111:6789/0 conn(0x7f49a806a6a0 legacy=0x7f49a815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.424+0000 7f49a6ffd700 1 -- 172.21.15.111:0/729766903 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a80ca810 con 0x7f49a80cb130 2023-12-19T07:47:20.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49a6ffd700 1 --2- 172.21.15.111:0/729766903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 0x7f49a80cb500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49980098e0 tx=0x7f49980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a268a91680e37b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:20.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49a5ffb700 1 -- 172.21.15.111:0/729766903 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499800b640 con 0x7f49a80cb130 2023-12-19T07:47:20.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49a5ffb700 1 -- 172.21.15.111:0/729766903 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499800b7a0 con 0x7f49a80cb130 2023-12-19T07:47:20.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49a5ffb700 1 -- 172.21.15.111:0/729766903 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499800bb40 con 0x7f49a80cb130 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 -- 172.21.15.111:0/729766903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 msgr2=0x7f49a80cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/729766903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 0x7f49a80cb500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49980098e0 tx=0x7f49980091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 -- 172.21.15.111:0/729766903 shutdown_connections 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/729766903 >> v2:172.21.15.111:3300/0 conn(0x7f49a815c930 0x7f49a806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/729766903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a80cb130 0x7f49a80cb500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 -- 172.21.15.111:0/729766903 >> 172.21.15.111:0/729766903 conn(0x7f49a8067bb0 msgr2=0x7f49a80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:20.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 -- 172.21.15.111:0/729766903 shutdown_connections 2023-12-19T07:47:20.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.428+0000 7f49ad2f0700 1 -- 172.21.15.111:0/729766903 wait complete. 2023-12-19T07:47:20.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 Processor -- start 2023-12-19T07:47:20.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 -- start start 2023-12-19T07:47:20.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a806a6a0 0x7f49a815a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a815c930 0x7f49a8156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a815b540 con 0x7f49a80cb130 2023-12-19T07:47:20.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a80b5a30 con 0x7f49a815c930 2023-12-19T07:47:20.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49ad2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8156980 con 0x7f49a806a6a0 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50314/0 (socket says 172.21.15.111:50314) 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 -- 172.21.15.111:0/2818643882 learned_addr learned my addr 172.21.15.111:0/2818643882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a6ffd700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a806a6a0 0x7f49a815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a77fe700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a815c930 0x7f49a8156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 -- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a806a6a0 msgr2=0x7f49a815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a806a6a0 0x7f49a815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 -- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a815c930 msgr2=0x7f49a8156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a815c930 0x7f49a8156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:20.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.432+0000 7f49a67fc700 1 -- 172.21.15.111:0/2818643882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998009420 con 0x7f49a80cb130 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f49a67fc700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c7c0 tx=0x7f499c007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c005a70 con 0x7f49a80cb130 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c005bd0 con 0x7f49a80cb130 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c0070d0 con 0x7f49a80cb130 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a815b260 con 0x7f49a80cb130 2023-12-19T07:47:20.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a8156610 con 0x7f49a80cb130 2023-12-19T07:47:20.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.436+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a8156190 con 0x7f49a80cb130 2023-12-19T07:47:20.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.440+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f499c007440 con 0x7f49a80cb130 2023-12-19T07:47:20.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.440+0000 7f498effd700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 0x7f4994062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:20.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.440+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f499c07be90 con 0x7f49a80cb130 2023-12-19T07:47:20.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.440+0000 7f49a6ffd700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 0x7f4994062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:20.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.440+0000 7f49a6ffd700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 0x7f4994062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49980093f0 tx=0x7f4998017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:20.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.448+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f49a8156190 con 0x7f49a80cb130 2023-12-19T07:47:20.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:20.624+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f49a80b07c0 con 0x7f49a80cb130 2023-12-19T07:47:21.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.228+0000 7f498effd700 1 -- 172.21.15.111:0/2818643882 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f49a80b07c0 con 0x7f49a80cb130 2023-12-19T07:47:21.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.236+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 msgr2=0x7f4994062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:21.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.236+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 0x7f4994062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49980093f0 tx=0x7f4998017000 comp rx=0 tx=0).stop 2023-12-19T07:47:21.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.236+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 msgr2=0x7f49a815abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:21.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.236+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c7c0 tx=0x7f499c007f30 comp rx=0 tx=0).stop 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 shutdown_connections 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4994060550 0x7f4994062a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a815c930 0x7f49a8156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49a80cb130 0x7f49a815abb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 --2- 172.21.15.111:0/2818643882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a806a6a0 0x7f49a815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 >> 172.21.15.111:0/2818643882 conn(0x7f49a8067bb0 msgr2=0x7f49a814ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:21.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 shutdown_connections 2023-12-19T07:47:21.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.244+0000 7f49ad2f0700 1 -- 172.21.15.111:0/2818643882 wait complete. 2023-12-19T07:47:21.241 INFO:teuthology.orchestra.run.smithi111.stderr:set pool 4 allow_ec_overwrites to true 2023-12-19T07:47:21.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-12-19T07:47:21.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.444+0000 7f87383c5700 1 Processor -- start 2023-12-19T07:47:21.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.444+0000 7f87383c5700 1 -- start start 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f8730154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8730158b60 0x7f8730158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300c7ea0 con 0x7f8730158b60 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8730155430 con 0x7f873006d980 2023-12-19T07:47:21.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f87383c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87300c7290 con 0x7f8730070250 2023-12-19T07:47:21.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8730158b60 0x7f8730158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8735960700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736161700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f8730154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8730158b60 0x7f8730158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50316/0 (socket says 172.21.15.111:50316) 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8735960700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:40310/0 (socket says 172.21.15.111:40310) 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 -- 172.21.15.111:0/640570056 learned_addr learned my addr 172.21.15.111:0/640570056 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 -- 172.21.15.111:0/640570056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 msgr2=0x7f8730154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 --2- 172.21.15.111:0/640570056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f8730154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 -- 172.21.15.111:0/640570056 >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 msgr2=0x7f873006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 --2- 172.21.15.111:0/640570056 >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 -- 172.21.15.111:0/640570056 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87300c7570 con 0x7f8730158b60 2023-12-19T07:47:21.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8735960700 1 --2- 172.21.15.111:0/640570056 >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f8736962700 1 --2- 172.21.15.111:0/640570056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f8730158fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f873006a4f0 tx=0x7f8728009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf183fe0a5d7c83 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f873515f700 1 -- 172.21.15.111:0/640570056 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800e070 con 0x7f8730158b60 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f873515f700 1 -- 172.21.15.111:0/640570056 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872800b970 con 0x7f8730158b60 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.448+0000 7f873515f700 1 -- 172.21.15.111:0/640570056 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87280092e0 con 0x7f8730158b60 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- 172.21.15.111:0/640570056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 msgr2=0x7f8730158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:21.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 --2- 172.21.15.111:0/640570056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f8730158fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f873006a4f0 tx=0x7f8728009840 comp rx=0 tx=0).stop 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- 172.21.15.111:0/640570056 shutdown_connections 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 --2- 172.21.15.111:0/640570056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f8730158fe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 --2- 172.21.15.111:0/640570056 >> v2:172.21.15.143:3300/0 conn(0x7f873006d980 0x7f873006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 --2- 172.21.15.111:0/640570056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f8730154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- 172.21.15.111:0/640570056 >> 172.21.15.111:0/640570056 conn(0x7f87300c8aa0 msgr2=0x7f87300c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- 172.21.15.111:0/640570056 shutdown_connections 2023-12-19T07:47:21.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- 172.21.15.111:0/640570056 wait complete. 2023-12-19T07:47:21.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 Processor -- start 2023-12-19T07:47:21.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.452+0000 7f87383c5700 1 -- start start 2023-12-19T07:47:21.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f873006d980 0x7f87300c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f87300aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87300af430 con 0x7f8730158b60 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87300b0270 con 0x7f873006d980 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87300b0c40 con 0x7f8730070250 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50328/0 (socket says 172.21.15.111:50328) 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 -- 172.21.15.111:0/3762225094 learned_addr learned my addr 172.21.15.111:0/3762225094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736161700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f873006d980 0x7f87300c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8735960700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f87300aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 -- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 msgr2=0x7f87300aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f87300aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 -- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f873006d980 msgr2=0x7f87300c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f873006d980 0x7f87300c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 -- 172.21.15.111:0/3762225094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720009420 con 0x7f8730158b60 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8735960700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f87300aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:21.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f8736962700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8730063910 tx=0x7f8728018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:21.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728008ad0 con 0x7f8730158b60 2023-12-19T07:47:21.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8728008c30 con 0x7f8730158b60 2023-12-19T07:47:21.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8728009ae0 con 0x7f8730158b60 2023-12-19T07:47:21.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872801c410 con 0x7f8730158b60 2023-12-19T07:47:21.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.456+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87300afc20 con 0x7f8730158b60 2023-12-19T07:47:21.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.460+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87300b03e0 con 0x7f8730158b60 2023-12-19T07:47:21.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.460+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8728003040 con 0x7f8730158b60 2023-12-19T07:47:21.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.460+0000 7f87267fc700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 0x7f8714062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:21.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.460+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f872807e220 con 0x7f8730158b60 2023-12-19T07:47:21.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.464+0000 7f8736161700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 0x7f8714062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:21.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.464+0000 7f8736161700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 0x7f8714062a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f872c009d10 tx=0x7f872c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:21.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.468+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f872804e4a0 con 0x7f8730158b60 2023-12-19T07:47:21.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:21.672+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f8730071120 con 0x7f8730158b60 2023-12-19T07:47:22.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.308+0000 7f87267fc700 1 -- 172.21.15.111:0/3762225094 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8728051ac0 con 0x7f8730158b60 2023-12-19T07:47:22.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.308+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 msgr2=0x7f8714062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.308+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 0x7f8714062a30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f872c009d10 tx=0x7f872c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.308+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 msgr2=0x7f87300b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.308+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8730063910 tx=0x7f8728018000 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 shutdown_connections 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8714060580 0x7f8714062a30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8730158b60 0x7f87300b0700 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8730070250 0x7f87300aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 --2- 172.21.15.111:0/3762225094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f873006d980 0x7f87300c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 >> 172.21.15.111:0/3762225094 conn(0x7f87300c8aa0 msgr2=0x7f873006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 shutdown_connections 2023-12-19T07:47:22.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.312+0000 7f87383c5700 1 -- 172.21.15.111:0/3762225094 wait complete. 2023-12-19T07:47:22.296 INFO:teuthology.orchestra.run.smithi111.stderr:added data pool 4 to fsmap 2023-12-19T07:47:22.316 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:22.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.496+0000 7f1701acd700 1 Processor -- start 2023-12-19T07:47:22.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.496+0000 7f1701acd700 1 -- start start 2023-12-19T07:47:22.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f1701acd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f1701acd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16fc068630 0x7f16fc0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f1701acd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc06e150 con 0x7f16fc140710 2023-12-19T07:47:22.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f1701acd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16fc068ab0 con 0x7f16fc068630 2023-12-19T07:47:22.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f1701acd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc06d540 con 0x7f16fc140ae0 2023-12-19T07:47:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fb7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f16fc140710 0x7f16fc0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36154/0 (socket says 172.21.15.111:36154) 2023-12-19T07:47:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fb7fe700 1 -- 172.21.15.111:0/1171355875 learned_addr learned my addr 172.21.15.111:0/1171355875 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fbfff700 1 --2- 172.21.15.111:0/1171355875 >> v2:172.21.15.143:3300/0 conn(0x7f16fc068630 0x7f16fc0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16faffd700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fa7fc700 1 -- 172.21.15.111:0/1171355875 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447697898 0 0) 0x7f16fc06e150 con 0x7f16fc140710 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fa7fc700 1 -- 172.21.15.111:0/1171355875 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4003570 con 0x7f16fc140710 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fbfff700 1 -- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 msgr2=0x7f16fc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fbfff700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.500+0000 7f16fbfff700 1 -- 172.21.15.111:0/1171355875 >> v1:172.21.15.143:6789/0 conn(0x7f16fc140710 legacy=0x7f16fc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f16fbfff700 1 -- 172.21.15.111:0/1171355875 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc06d820 con 0x7f16fc068630 2023-12-19T07:47:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f16fbfff700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc0ac250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009e60 tx=0x7f16f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=322f09d913a39a80 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f16fa7fc700 1 -- 172.21.15.111:0/1171355875 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400e070 con 0x7f16fc068630 2023-12-19T07:47:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f16fa7fc700 1 -- 172.21.15.111:0/1171355875 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f400bab0 con 0x7f16fc068630 2023-12-19T07:47:22.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f16fa7fc700 1 -- 172.21.15.111:0/1171355875 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400be70 con 0x7f16fc068630 2023-12-19T07:47:22.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 -- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 msgr2=0x7f16fc0ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc0ac250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009e60 tx=0x7f16f4009780 comp rx=0 tx=0).stop 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 -- 172.21.15.111:0/1171355875 shutdown_connections 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc0ac250 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 --2- 172.21.15.111:0/1171355875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 -- 172.21.15.111:0/1171355875 >> 172.21.15.111:0/1171355875 conn(0x7f16fc0c8850 msgr2=0x7f16fc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 -- 172.21.15.111:0/1171355875 shutdown_connections 2023-12-19T07:47:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.504+0000 7f1701acd700 1 -- 172.21.15.111:0/1171355875 wait complete. 2023-12-19T07:47:22.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 Processor -- start 2023-12-19T07:47:22.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 -- start start 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc169e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc138090 con 0x7f16fc140710 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc0c0b80 con 0x7f16fc068630 2023-12-19T07:47:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f1701acd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc0c1700 con 0x7f16fc140ae0 2023-12-19T07:47:22.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33364/0 (socket says 172.21.15.111:33364) 2023-12-19T07:47:22.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 -- 172.21.15.111:0/2185058118 learned_addr learned my addr 172.21.15.111:0/2185058118 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16fbfff700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16fb7fe700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc169e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 -- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 msgr2=0x7f16fc0b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 -- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 msgr2=0x7f16fc169e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc169e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16faffd700 1 -- 172.21.15.111:0/2185058118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f4009a20 con 0x7f16fc140710 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.508+0000 7f16fbfff700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f16faffd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16f000ec80 tx=0x7f16f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000c850 con 0x7f16fc140710 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f000c9b0 con 0x7f16fc140710 2023-12-19T07:47:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc0aee50 con 0x7f16fc140710 2023-12-19T07:47:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000cd50 con 0x7f16fc140710 2023-12-19T07:47:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc0c0dd0 con 0x7f16fc140710 2023-12-19T07:47:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.512+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc0c1870 con 0x7f16fc140710 2023-12-19T07:47:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.516+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f16f0013680 con 0x7f16fc140710 2023-12-19T07:47:22.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.516+0000 7f16e37fe700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 0x7f16e8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.516+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f16f007cdc0 con 0x7f16fc140710 2023-12-19T07:47:22.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.516+0000 7f16fb7fe700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 0x7f16e8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.520+0000 7f16fb7fe700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 0x7f16e8062b50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16ec007fd0 tx=0x7f16ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.524+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f16f004d0a0 con 0x7f16fc140710 2023-12-19T07:47:22.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.732+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16fc071120 con 0x7f16fc140710 2023-12-19T07:47:22.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f16e37fe700 1 -- 172.21.15.111:0/2185058118 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f16f00506c0 con 0x7f16fc140710 2023-12-19T07:47:22.718 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:22.718 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:22.294030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T07:47:22.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 msgr2=0x7f16e8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 0x7f16e8062b50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16ec007fd0 tx=0x7f16ec009040 comp rx=0 tx=0).stop 2023-12-19T07:47:22.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 msgr2=0x7f16fc0aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f16f000ec80 tx=0x7f16f00091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 shutdown_connections 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16e80606a0 0x7f16e8062b50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc140ae0 0x7f16fc0b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16fc140710 0x7f16fc0aadd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 --2- 172.21.15.111:0/2185058118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16fc068630 0x7f16fc169e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 >> 172.21.15.111:0/2185058118 conn(0x7f16fc0c8850 msgr2=0x7f16fc0ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:22.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 shutdown_connections 2023-12-19T07:47:22.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.736+0000 7f1701acd700 1 -- 172.21.15.111:0/2185058118 wait complete. 2023-12-19T07:47:22.724 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 6 2023-12-19T07:47:22.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:22.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.908+0000 7f05896ae700 1 Processor -- start 2023-12-19T07:47:22.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.908+0000 7f05896ae700 1 -- start start 2023-12-19T07:47:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05896ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05840c9410 0x7f05840c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05896ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0584155d60 0x7f05840cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05896ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0584159780 con 0x7f05840c9e70 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05896ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840ca850 con 0x7f0584155d60 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05896ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0584159a20 con 0x7f05840c9410 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0584155d60 0x7f05840cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f057bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f05840c9e70 0x7f0584155650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57146/0 (socket says 172.21.15.111:57146) 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f057bfff700 1 -- 172.21.15.111:0/2781624455 learned_addr learned my addr 172.21.15.111:0/2781624455 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 --2- 172.21.15.111:0/2781624455 >> v2:172.21.15.111:3300/0 conn(0x7f0584155d60 0x7f05840cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33366/0 (socket says 172.21.15.111:33366) 2023-12-19T07:47:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0582ffd700 1 --2- 172.21.15.111:0/2781624455 >> v2:172.21.15.143:3300/0 conn(0x7f05840c9410 0x7f05840c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 -- 172.21.15.111:0/2781624455 >> v2:172.21.15.143:3300/0 conn(0x7f05840c9410 msgr2=0x7f05840c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 --2- 172.21.15.111:0/2781624455 >> v2:172.21.15.143:3300/0 conn(0x7f05840c9410 0x7f05840c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 -- 172.21.15.111:0/2781624455 >> v1:172.21.15.111:6789/0 conn(0x7f05840c9e70 legacy=0x7f0584155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 -- 172.21.15.111:0/2781624455 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0584159d00 con 0x7f0584155d60 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05827fc700 1 -- 172.21.15.111:0/2781624455 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411347822 0 0) 0x7f0584159780 con 0x7f05840c9e70 2023-12-19T07:47:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f0583fff700 1 --2- 172.21.15.111:0/2781624455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0584155d60 0x7f05840cc260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05840c9e40 tx=0x7f057c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526e2c38b47b0cfb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05827fc700 1 -- 172.21.15.111:0/2781624455 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c00e070 con 0x7f0584155d60 2023-12-19T07:47:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05827fc700 1 -- 172.21.15.111:0/2781624455 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c00bb60 con 0x7f0584155d60 2023-12-19T07:47:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.912+0000 7f05827fc700 1 -- 172.21.15.111:0/2781624455 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c012430 con 0x7f0584155d60 2023-12-19T07:47:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- 172.21.15.111:0/2781624455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0584155d60 msgr2=0x7f05840cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 --2- 172.21.15.111:0/2781624455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0584155d60 0x7f05840cc260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05840c9e40 tx=0x7f057c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- 172.21.15.111:0/2781624455 shutdown_connections 2023-12-19T07:47:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 --2- 172.21.15.111:0/2781624455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0584155d60 0x7f05840cc260 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 --2- 172.21.15.111:0/2781624455 >> v2:172.21.15.143:3300/0 conn(0x7f05840c9410 0x7f05840c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- 172.21.15.111:0/2781624455 >> 172.21.15.111:0/2781624455 conn(0x7f0584068500 msgr2=0x7f05840688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:22.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- 172.21.15.111:0/2781624455 shutdown_connections 2023-12-19T07:47:22.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- 172.21.15.111:0/2781624455 wait complete. 2023-12-19T07:47:22.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 Processor -- start 2023-12-19T07:47:22.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.916+0000 7f05896ae700 1 -- start start 2023-12-19T07:47:22.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 0x7f058414eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 0x7f05840cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840cb930 con 0x7f05840c9410 2023-12-19T07:47:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840aa090 con 0x7f05840c9e70 2023-12-19T07:47:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840aa350 con 0x7f0584155d60 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33370/0 (socket says 172.21.15.111:33370) 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 -- 172.21.15.111:0/1075923948 learned_addr learned my addr 172.21.15.111:0/1075923948 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0583fff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 0x7f05840cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f057bfff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 0x7f058414eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 -- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 msgr2=0x7f05840cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 0x7f05840cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 -- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 msgr2=0x7f058414eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 0x7f058414eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 -- 172.21.15.111:0/1075923948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c009a20 con 0x7f05840c9410 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0583fff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 0x7f05840cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f057bfff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 0x7f058414eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f0582ffd700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f057400ec80 tx=0x7f05740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400c9d0 con 0x7f05840c9410 2023-12-19T07:47:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400cb30 con 0x7f05840c9410 2023-12-19T07:47:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574013470 con 0x7f05840c9410 2023-12-19T07:47:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05840aa630 con 0x7f05840c9410 2023-12-19T07:47:22.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.920+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05840aabf0 con 0x7f05840c9410 2023-12-19T07:47:22.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.924+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0584071120 con 0x7f05840c9410 2023-12-19T07:47:22.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.924+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0574013820 con 0x7f05840c9410 2023-12-19T07:47:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.928+0000 7f057affd700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 0x7f0564062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.928+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f057407cda0 con 0x7f05840c9410 2023-12-19T07:47:22.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.928+0000 7f057bfff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 0x7f0564062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.928+0000 7f057bfff700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 0x7f0564062a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f05840a89b0 tx=0x7f056c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:22.932+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f057404d0a0 con 0x7f05840c9410 2023-12-19T07:47:23.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.108+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0584159fe0 con 0x7f05840c9410 2023-12-19T07:47:23.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.112+0000 7f057affd700 1 -- 172.21.15.111:0/1075923948 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17252 (secure 0 0 0) 0x7f05740506c0 con 0x7f05840c9410 2023-12-19T07:47:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:23.096 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:21.676752+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:23.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 msgr2=0x7f0564062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 0x7f0564062a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f05840a89b0 tx=0x7f056c009040 comp rx=0 tx=0).stop 2023-12-19T07:47:23.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 msgr2=0x7f05840a9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f057400ec80 tx=0x7f05740091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 shutdown_connections 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0564060550 0x7f0564062a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05840c9e70 0x7f058414eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0584155d60 0x7f05840cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 --2- 172.21.15.111:0/1075923948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05840c9410 0x7f05840a9a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 >> 172.21.15.111:0/1075923948 conn(0x7f0584068500 msgr2=0x7f0584156210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 shutdown_connections 2023-12-19T07:47:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.116+0000 7f05896ae700 1 -- 172.21.15.111:0/1075923948 wait complete. 2023-12-19T07:47:23.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:23.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.288+0000 7fe48324c700 1 Processor -- start 2023-12-19T07:47:23.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.288+0000 7fe48324c700 1 -- start start 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe47c159530 0x7fe47c0c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 0x7fe47c158660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47c0accb0 con 0x7fe47c159160 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47c06a7b0 con 0x7fe47c159530 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0ad550 con 0x7fe47c158200 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe47c159530 0x7fe47c0c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe47c159530 0x7fe47c0c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33386/0 (socket says 172.21.15.111:33386) 2023-12-19T07:47:23.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 -- 172.21.15.111:0/2903941392 learned_addr learned my addr 172.21.15.111:0/2903941392 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:23.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe482a4b700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 0x7fe47c158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 -- 172.21.15.111:0/2903941392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 msgr2=0x7fe47c158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:23.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 0x7fe47c158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 -- 172.21.15.111:0/2903941392 >> v1:172.21.15.143:6789/0 conn(0x7fe47c159160 legacy=0x7fe47c0adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 -- 172.21.15.111:0/2903941392 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0acf90 con 0x7fe47c159530 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe482a4b700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 0x7fe47c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481a49700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c159530 0x7fe47c0c5fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ab50 tx=0x7fe46c00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b6f246e4ddbc0b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481248700 1 -- 172.21.15.111:0/2903941392 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c00ca20 con 0x7fe47c159530 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481248700 1 -- 172.21.15.111:0/2903941392 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c00cb80 con 0x7fe47c159530 2023-12-19T07:47:23.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe481248700 1 -- 172.21.15.111:0/2903941392 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c0183f0 con 0x7fe47c159530 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- 172.21.15.111:0/2903941392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c159530 msgr2=0x7fe47c0c5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c159530 0x7fe47c0c5fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ab50 tx=0x7fe46c00a3f0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- 172.21.15.111:0/2903941392 shutdown_connections 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c158200 0x7fe47c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 --2- 172.21.15.111:0/2903941392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c159530 0x7fe47c0c5fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- 172.21.15.111:0/2903941392 >> 172.21.15.111:0/2903941392 conn(0x7fe47c067bb0 msgr2=0x7fe47c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- 172.21.15.111:0/2903941392 shutdown_connections 2023-12-19T07:47:23.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.292+0000 7fe48324c700 1 -- 172.21.15.111:0/2903941392 wait complete. 2023-12-19T07:47:23.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.296+0000 7fe48324c700 1 Processor -- start 2023-12-19T07:47:23.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.296+0000 7fe48324c700 1 -- start start 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe47c159160 0x7fe47c153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c159530 0x7fe47c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0af3b0 con 0x7fe47c158200 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0c5230 con 0x7fe47c159160 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47c0c5510 con 0x7fe47c159530 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33398/0 (socket says 172.21.15.111:33398) 2023-12-19T07:47:23.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 -- 172.21.15.111:0/3969443209 learned_addr learned my addr 172.21.15.111:0/3969443209 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe482a4b700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c159530 0x7fe47c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe481a49700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe47c159160 0x7fe47c153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 -- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c159530 msgr2=0x7fe47c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c159530 0x7fe47c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 -- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe47c159160 msgr2=0x7fe47c153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe47c159160 0x7fe47c153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 -- 172.21.15.111:0/3969443209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c00a690 con 0x7fe47c158200 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48224a700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe47800c5d0 tx=0x7fe478007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011590 con 0x7fe47c158200 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe478011710 con 0x7fe47c158200 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47c0c57f0 con 0x7fe47c158200 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478011ad0 con 0x7fe47c158200 2023-12-19T07:47:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.300+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47c153340 con 0x7fe47c158200 2023-12-19T07:47:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.304+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47c04be40 con 0x7fe47c158200 2023-12-19T07:47:23.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.304+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe47801c460 con 0x7fe47c158200 2023-12-19T07:47:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.304+0000 7fe4727fc700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 0x7fe460062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.304+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe47807c6a0 con 0x7fe47c158200 2023-12-19T07:47:23.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.304+0000 7fe481a49700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 0x7fe460062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.308+0000 7fe481a49700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 0x7fe460062a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ab50 tx=0x7fe46c019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.312+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe47804c9a0 con 0x7fe47c158200 2023-12-19T07:47:23.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.484+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe47c0aaed0 con 0x7fe47c158200 2023-12-19T07:47:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe4727fc700 1 -- 172.21.15.111:0/3969443209 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7fe47804ffc0 con 0x7fe47c158200 2023-12-19T07:47:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:23.470 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 msgr2=0x7fe460062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 0x7fe460062a30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00ab50 tx=0x7fe46c019000 comp rx=0 tx=0).stop 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 msgr2=0x7fe47c152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe47800c5d0 tx=0x7fe478007f60 comp rx=0 tx=0).stop 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 shutdown_connections 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe460060580 0x7fe460062a30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe47c159160 0x7fe47c153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe47c159530 0x7fe47c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 --2- 172.21.15.111:0/3969443209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe47c158200 0x7fe47c152e00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 >> 172.21.15.111:0/3969443209 conn(0x7fe47c067bb0 msgr2=0x7fe47c0cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 shutdown_connections 2023-12-19T07:47:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.488+0000 7fe48324c700 1 -- 172.21.15.111:0/3969443209 wait complete. 2023-12-19T07:47:23.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:23.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.696+0000 7ff621185700 1 Processor -- start 2023-12-19T07:47:23.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.696+0000 7ff621185700 1 -- start start 2023-12-19T07:47:23.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff621185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff61c15cc80 0x7ff61c15d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff621185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0ad810 con 0x7ff61c158fa0 2023-12-19T07:47:23.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff621185700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0686f0 con 0x7ff61c159370 2023-12-19T07:47:23.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff621185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61c15d600 con 0x7ff61c15cc80 2023-12-19T07:47:23.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff61c158fa0 0x7ff61c068b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57160/0 (socket says 172.21.15.111:57160) 2023-12-19T07:47:23.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61ad9d700 1 -- 172.21.15.111:0/2305515866 learned_addr learned my addr 172.21.15.111:0/2305515866 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:23.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61b59e700 1 --2- 172.21.15.111:0/2305515866 >> v2:172.21.15.143:3300/0 conn(0x7ff61c15cc80 0x7ff61c15d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff619d9b700 1 -- 172.21.15.111:0/2305515866 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807181567 0 0) 0x7ff61c0ad810 con 0x7ff61c158fa0 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff619d9b700 1 -- 172.21.15.111:0/2305515866 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604003570 con 0x7ff61c158fa0 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61b59e700 1 -- 172.21.15.111:0/2305515866 >> v1:172.21.15.143:6789/0 conn(0x7ff61c159370 legacy=0x7ff61c06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61b59e700 1 -- 172.21.15.111:0/2305515866 >> v1:172.21.15.111:6789/0 conn(0x7ff61c158fa0 legacy=0x7ff61c068b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61b59e700 1 -- 172.21.15.111:0/2305515866 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c067ae0 con 0x7ff61c15cc80 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff61b59e700 1 --2- 172.21.15.111:0/2305515866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c15cc80 0x7ff61c15d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1501f88814f5e26 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff619d9b700 1 -- 172.21.15.111:0/2305515866 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400e070 con 0x7ff61c15cc80 2023-12-19T07:47:23.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff619d9b700 1 -- 172.21.15.111:0/2305515866 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61400bbf0 con 0x7ff61c15cc80 2023-12-19T07:47:23.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.700+0000 7ff619d9b700 1 -- 172.21.15.111:0/2305515866 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614012430 con 0x7ff61c15cc80 2023-12-19T07:47:23.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- 172.21.15.111:0/2305515866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c15cc80 msgr2=0x7ff61c15d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 --2- 172.21.15.111:0/2305515866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c15cc80 0x7ff61c15d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e60 tx=0x7ff614009780 comp rx=0 tx=0).stop 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- 172.21.15.111:0/2305515866 shutdown_connections 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 --2- 172.21.15.111:0/2305515866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c15cc80 0x7ff61c15d0c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- 172.21.15.111:0/2305515866 >> 172.21.15.111:0/2305515866 conn(0x7ff61c0ca9d0 msgr2=0x7ff61c0cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- 172.21.15.111:0/2305515866 shutdown_connections 2023-12-19T07:47:23.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- 172.21.15.111:0/2305515866 wait complete. 2023-12-19T07:47:23.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 Processor -- start 2023-12-19T07:47:23.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- start start 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c159370 0x7ff61c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff61c15cc80 0x7ff61c0c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c157ab0 con 0x7ff61c158fa0 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0c77e0 con 0x7ff61c159370 2023-12-19T07:47:23.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.704+0000 7ff621185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0ada20 con 0x7ff61c15cc80 2023-12-19T07:47:23.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33412/0 (socket says 172.21.15.111:33412) 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 -- 172.21.15.111:0/2670353850 learned_addr learned my addr 172.21.15.111:0/2670353850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61a59c700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c159370 0x7ff61c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61b59e700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff61c15cc80 0x7ff61c0c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 -- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff61c15cc80 msgr2=0x7ff61c0c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff61c15cc80 0x7ff61c0c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 -- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c159370 msgr2=0x7ff61c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c159370 0x7ff61c0b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 -- 172.21.15.111:0/2670353850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614009a20 con 0x7ff61c158fa0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff61ad9d700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c7c0 tx=0x7ff60c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c005b30 con 0x7ff61c158fa0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c005c90 con 0x7ff61c158fa0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c007320 con 0x7ff61c158fa0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c0c6f40 con 0x7ff61c158fa0 2023-12-19T07:47:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.708+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c0c7500 con 0x7ff61c158fa0 2023-12-19T07:47:23.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.712+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c071120 con 0x7ff61c158fa0 2023-12-19T07:47:23.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.712+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff60c007690 con 0x7ff61c158fa0 2023-12-19T07:47:23.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.712+0000 7ff602ffd700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:23.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.712+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff60c07bf70 con 0x7ff61c158fa0 2023-12-19T07:47:23.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.712+0000 7ff61a59c700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.716+0000 7ff61a59c700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 0x7ff608062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0ad280 tx=0x7ff610007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:23.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.720+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff60c04c1d0 con 0x7ff61c158fa0 2023-12-19T07:47:23.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.932+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61c067dc0 con 0x7ff61c158fa0 2023-12-19T07:47:23.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.932+0000 7ff602ffd700 1 -- 172.21.15.111:0/2670353850 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4605 (secure 0 0 0) 0x7ff60c053e60 con 0x7ff61c158fa0 2023-12-19T07:47:23.916 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:23.916 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:22.294030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T07:47:23.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 msgr2=0x7ff608062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 0x7ff608062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0ad280 tx=0x7ff610007ec0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 msgr2=0x7ff61c157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:23.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c7c0 tx=0x7ff60c007e00 comp rx=0 tx=0).stop 2023-12-19T07:47:23.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 shutdown_connections 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff61c159370 0x7ff61c0b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff61c15cc80 0x7ff61c0c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 --2- 172.21.15.111:0/2670353850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff61c158fa0 0x7ff61c157570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 >> 172.21.15.111:0/2670353850 conn(0x7ff61c0ca9d0 msgr2=0x7ff61c0cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 shutdown_connections 2023-12-19T07:47:23.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:23.936+0000 7ff621185700 1 -- 172.21.15.111:0/2670353850 wait complete. 2023-12-19T07:47:23.921 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 6 2023-12-19T07:47:23.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:24.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.124+0000 7f4e7d416700 1 Processor -- start 2023-12-19T07:47:24.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.124+0000 7f4e7d416700 1 -- start start 2023-12-19T07:47:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7d416700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e7806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7d416700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7d416700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7806b730 con 0x7f4e7815ca80 2023-12-19T07:47:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7d416700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7806ab20 con 0x7f4e781591f0 2023-12-19T07:47:24.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7d416700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7806ae00 con 0x7f4e78158e20 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e777fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e7806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e777fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49632/0 (socket says 172.21.15.111:49632) 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e777fe700 1 -- 172.21.15.111:0/2263079090 learned_addr learned my addr 172.21.15.111:0/2263079090 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e7cc15700 1 --1- 172.21.15.111:0/2263079090 >> v1:172.21.15.111:6789/0 conn(0x7f4e7815ca80 0x7f4e7806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57176/0 (socket says 172.21.15.111:57176) 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e77fff700 1 -- 172.21.15.111:0/2263079090 >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 msgr2=0x7f4e780ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e77fff700 1 --2- 172.21.15.111:0/2263079090 >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e77fff700 1 -- 172.21.15.111:0/2263079090 >> v1:172.21.15.111:6789/0 conn(0x7f4e7815ca80 legacy=0x7f4e7806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e77fff700 1 -- 172.21.15.111:0/2263079090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7806b0e0 con 0x7f4e78158e20 2023-12-19T07:47:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.128+0000 7f4e777fe700 1 --2- 172.21.15.111:0/2263079090 >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e77fff700 1 --2- 172.21.15.111:0/2263079090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e7806a540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098e0 tx=0x7f4e680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6724c57221ac1aff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e76ffd700 1 -- 172.21.15.111:0/2263079090 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800b640 con 0x7f4e78158e20 2023-12-19T07:47:24.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e76ffd700 1 -- 172.21.15.111:0/2263079090 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6800b7a0 con 0x7f4e78158e20 2023-12-19T07:47:24.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e76ffd700 1 -- 172.21.15.111:0/2263079090 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800bb40 con 0x7f4e78158e20 2023-12-19T07:47:24.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 -- 172.21.15.111:0/2263079090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 msgr2=0x7f4e7806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 --2- 172.21.15.111:0/2263079090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e7806a540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098e0 tx=0x7f4e680091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 -- 172.21.15.111:0/2263079090 shutdown_connections 2023-12-19T07:47:24.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 --2- 172.21.15.111:0/2263079090 >> v2:172.21.15.143:3300/0 conn(0x7f4e781591f0 0x7f4e780ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 --2- 172.21.15.111:0/2263079090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e7806a540 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 -- 172.21.15.111:0/2263079090 >> 172.21.15.111:0/2263079090 conn(0x7f4e78067bb0 msgr2=0x7f4e780cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 -- 172.21.15.111:0/2263079090 shutdown_connections 2023-12-19T07:47:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.132+0000 7f4e7d416700 1 -- 172.21.15.111:0/2263079090 wait complete. 2023-12-19T07:47:24.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 Processor -- start 2023-12-19T07:47:24.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 -- start start 2023-12-19T07:47:24.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e78154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e781591f0 0x7f4e7815bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7815aed0 con 0x7f4e7815ca80 2023-12-19T07:47:24.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7815a5d0 con 0x7f4e781591f0 2023-12-19T07:47:24.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7d416700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78155090 con 0x7f4e78158e20 2023-12-19T07:47:24.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7cc15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e78154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e7cc15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33422/0 (socket says 172.21.15.111:33422) 2023-12-19T07:47:24.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e7cc15700 1 -- 172.21.15.111:0/802768232 learned_addr learned my addr 172.21.15.111:0/802768232 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.136+0000 7f4e777fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e781591f0 0x7f4e7815bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 -- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e781591f0 msgr2=0x7f4e7815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e781591f0 0x7f4e7815bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 -- 172.21.15.111:0/802768232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 msgr2=0x7f4e7816d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 -- 172.21.15.111:0/802768232 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0094f0 con 0x7f4e78158e20 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e7cc15700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:24.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e77fff700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e78154ab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e68016000 tx=0x7f4e68017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6801bc40 con 0x7f4e78158e20 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6801bdc0 con 0x7f4e78158e20 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e68009420 con 0x7f4e78158e20 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68011670 con 0x7f4e78158e20 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7815ab60 con 0x7f4e78158e20 2023-12-19T07:47:24.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.140+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7804be40 con 0x7f4e78158e20 2023-12-19T07:47:24.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.144+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4e68011a20 con 0x7f4e78158e20 2023-12-19T07:47:24.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.144+0000 7f4e5ffff700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 0x7f4e64062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.144+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4e6807dfb0 con 0x7f4e78158e20 2023-12-19T07:47:24.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.144+0000 7f4e777fe700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 0x7f4e64062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.148+0000 7f4e777fe700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 0x7f4e64062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4e780ca6a0 tx=0x7f4e6c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.156+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e6804e210 con 0x7f4e78158e20 2023-12-19T07:47:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.332+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e7806b3c0 con 0x7f4e78158e20 2023-12-19T07:47:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.336+0000 7f4e5ffff700 1 -- 172.21.15.111:0/802768232 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f4e68051830 con 0x7f4e78158e20 2023-12-19T07:47:24.320 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:24.320 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 msgr2=0x7f4e64062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 0x7f4e64062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4e780ca6a0 tx=0x7f4e6c009250 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 msgr2=0x7f4e78154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e78154ab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e68016000 tx=0x7f4e68017000 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 shutdown_connections 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e64060550 0x7f4e64062a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e781591f0 0x7f4e7815bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7815ca80 0x7f4e7816d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 --2- 172.21.15.111:0/802768232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e78158e20 0x7f4e78154ab0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 >> 172.21.15.111:0/802768232 conn(0x7f4e78067bb0 msgr2=0x7f4e7815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 shutdown_connections 2023-12-19T07:47:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.340+0000 7f4e7d416700 1 -- 172.21.15.111:0/802768232 wait complete. 2023-12-19T07:47:24.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T07:47:16.604129+0000', 'modified': '2023-12-19T07:47:22.294030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6835/1214316551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1214316551}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1214316551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-12-19T07:47:24.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T07:47:24.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T07:47:24.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.936+0000 7f80fd84c700 1 Processor -- start 2023-12-19T07:47:24.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.936+0000 7f80fd84c700 1 -- start start 2023-12-19T07:47:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fd84c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80f806a540 0x7f80f806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fd84c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f80f80cc470 0x7f80f80cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fd84c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f80ca5a0 con 0x7f80f8158e70 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fd84c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f806dac0 con 0x7f80f806a540 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fd84c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f806d220 con 0x7f80f80cc470 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80effff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80f806a540 0x7f80f806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80effff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80f806a540 0x7f80f806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33436/0 (socket says 172.21.15.111:33436) 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80effff700 1 -- 172.21.15.111:0/3062731033 learned_addr learned my addr 172.21.15.111:0/3062731033 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.940+0000 7f80fc84a700 1 --1- 172.21.15.111:0/3062731033 >> v1:172.21.15.111:6789/0 conn(0x7f80f8158e70 0x7f80f80afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57190/0 (socket says 172.21.15.111:57190) 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd04b700 1 --2- 172.21.15.111:0/3062731033 >> v2:172.21.15.143:3300/0 conn(0x7f80f80cc470 0x7f80f80cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80effff700 1 -- 172.21.15.111:0/3062731033 >> v2:172.21.15.143:3300/0 conn(0x7f80f80cc470 msgr2=0x7f80f80cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80effff700 1 --2- 172.21.15.111:0/3062731033 >> v2:172.21.15.143:3300/0 conn(0x7f80f80cc470 0x7f80f80cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80effff700 1 -- 172.21.15.111:0/3062731033 >> v1:172.21.15.111:6789/0 conn(0x7f80f8158e70 legacy=0x7f80f80afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80effff700 1 -- 172.21.15.111:0/3062731033 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f806d500 con 0x7f80f806a540 2023-12-19T07:47:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80effff700 1 --2- 172.21.15.111:0/3062731033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f806c9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f80f806df30 tx=0x7f80e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b788b0187fa23a08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80ef7fe700 1 -- 172.21.15.111:0/3062731033 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e800b860 con 0x7f80f806a540 2023-12-19T07:47:24.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80ef7fe700 1 -- 172.21.15.111:0/3062731033 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e800b9c0 con 0x7f80f806a540 2023-12-19T07:47:24.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80ef7fe700 1 -- 172.21.15.111:0/3062731033 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e800bd80 con 0x7f80f806a540 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 -- 172.21.15.111:0/3062731033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 msgr2=0x7f80f806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 --2- 172.21.15.111:0/3062731033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f806c9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f80f806df30 tx=0x7f80e8009250 comp rx=0 tx=0).stop 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 -- 172.21.15.111:0/3062731033 shutdown_connections 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 --2- 172.21.15.111:0/3062731033 >> v2:172.21.15.143:3300/0 conn(0x7f80f80cc470 0x7f80f80cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 --2- 172.21.15.111:0/3062731033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f806c9d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 -- 172.21.15.111:0/3062731033 >> 172.21.15.111:0/3062731033 conn(0x7f80f80670e0 msgr2=0x7f80f80674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 -- 172.21.15.111:0/3062731033 shutdown_connections 2023-12-19T07:47:24.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.944+0000 7f80fd84c700 1 -- 172.21.15.111:0/3062731033 wait complete. 2023-12-19T07:47:24.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.948+0000 7f80fd84c700 1 Processor -- start 2023-12-19T07:47:24.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.948+0000 7f80fd84c700 1 -- start start 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.948+0000 7f80fd84c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f80cc470 0x7f80f8159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f8158e70 0x7f80f8156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80c4ce0 con 0x7f80f806a540 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f816ce60 con 0x7f80f8158e70 2023-12-19T07:47:24.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f816d140 con 0x7f80f80cc470 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33448/0 (socket says 172.21.15.111:33448) 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 -- 172.21.15.111:0/1497772548 learned_addr learned my addr 172.21.15.111:0/1497772548 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd04b700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f8158e70 0x7f80f8156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80effff700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f80cc470 0x7f80f8159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 -- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f80cc470 msgr2=0x7f80f8159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f80cc470 0x7f80f8159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 -- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f8158e70 msgr2=0x7f80f8156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f8158e70 0x7f80f8156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 -- 172.21.15.111:0/1497772548 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e80094f0 con 0x7f80f806a540 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80f400cd80 tx=0x7f80f4001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80ecff9700 1 -- 172.21.15.111:0/1497772548 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4005df0 con 0x7f80f806a540 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80ecff9700 1 -- 172.21.15.111:0/1497772548 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f400a8b0 con 0x7f80f806a540 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80ecff9700 1 -- 172.21.15.111:0/1497772548 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400ac70 con 0x7f80f806a540 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f816d420 con 0x7f80f806a540 2023-12-19T07:47:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.952+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f80b2dc0 con 0x7f80f806a540 2023-12-19T07:47:24.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f80ecff9700 1 -- 172.21.15.111:0/1497772548 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f80f40070f0 con 0x7f80f806a540 2023-12-19T07:47:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f80ecff9700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 0x7f80e4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f80ecff9700 1 -- 172.21.15.111:0/1497772548 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f80f407cc30 con 0x7f80f806a540 2023-12-19T07:47:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f80effff700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 0x7f80e4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f8100d40740 1 Processor -- start 2023-12-19T07:47:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.960+0000 7f8100d40740 1 -- start start 2023-12-19T07:47:24.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80effff700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 0x7f80e4062a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80f806df30 tx=0x7f80e8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1df4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1bc1790 0x1df4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d21f30 con 0x1bc1790 2023-12-19T07:47:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d1ded0 con 0x1b76ae0 2023-12-19T07:47:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ce40b0 con 0x1b76710 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49654/0 (socket says 172.21.15.111:49654) 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 -- 172.21.15.111:0/4164036430 learned_addr learned my addr 172.21.15.111:0/4164036430 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80effff700 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1df4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fd04b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1bc1790 0x1df4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 -- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 msgr2=0x1df4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1df4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 -- 172.21.15.111:0/4164036430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1bc1790 msgr2=0x1df4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1bc1790 0x1df4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 -- 172.21.15.111:0/4164036430 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1bc29e0 con 0x1b76ae0 2023-12-19T07:47:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80fc84a700 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80f400a620 tx=0x7f80f407ad20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a31051ba7dbe30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80ddffb700 1 -- 172.21.15.111:0/4164036430 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4053e60 con 0x1b76ae0 2023-12-19T07:47:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80ddffb700 1 -- 172.21.15.111:0/4164036430 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4057bb0 con 0x1b76ae0 2023-12-19T07:47:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.964+0000 7f80ddffb700 1 -- 172.21.15.111:0/4164036430 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f404ed10 con 0x1b76ae0 2023-12-19T07:47:24.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 msgr2=0x1df3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80f400a620 tx=0x7f80f407ad20 comp rx=0 tx=0).stop 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- 172.21.15.111:0/4164036430 shutdown_connections 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1bc1790 0x1df4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1df4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- 172.21.15.111:0/4164036430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df3f00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- 172.21.15.111:0/4164036430 >> 172.21.15.111:0/4164036430 conn(0x1b76eb0 msgr2=0x1db0930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- 172.21.15.111:0/4164036430 shutdown_connections 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- 172.21.15.111:0/4164036430 wait complete. 2023-12-19T07:47:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- start start 2023-12-19T07:47:24.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 Processor -- start 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1dd9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ce3d50 con 0x1b76710 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cbc970 con 0x1b76ae0 2023-12-19T07:47:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.968+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ce6920 con 0x1bc1790 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 10 client.0 _ll_register_callbacks cb 0x1d676a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33464/0 (socket says 172.21.15.111:33464) 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 -- 172.21.15.111:0/2217853831 learned_addr learned my addr 172.21.15.111:0/2217853831 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fd04b700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1dd9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80effff700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 -- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 msgr2=0x1dd9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1dd9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 -- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 msgr2=0x1df8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 -- 172.21.15.111:0/2217853831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f80b2dc0 con 0x1b76710 2023-12-19T07:47:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80effff700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80fc84a700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f80f4050b60 tx=0x7f80f4078000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76079485ff6f857 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:47:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4053e60 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f80f404ed10 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1cbccc0 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80f40502d0 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 10 client.4410 fetch_fsmap learned FSMap version 6 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x1cec370 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1cbccc0 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f80f405f030 con 0x1b76710 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 10 client.4410 fetch_fsmap finished waiting for FSMap version 6 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 10 client.4410 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 20 client.4410 populate_metadata read hostname 'smithi111' 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1ddb1b0 0x1e036b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 4 client.4410 mds_command: new command op to 4275 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T07:47:24.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 --> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1cc03e0 con 0x1ddb1b0 2023-12-19T07:47:24.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.972+0000 7f80effff700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1ddb1b0 0x1e036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f80effff700 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1ddb1b0 0x1e036b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd55a8b3901ed07 server_cookie=55de96a4b96f5960 in_seq=0 out_seq=0 2023-12-19T07:47:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f80d6ffd700 10 client.4410 ms_handle_connect on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f80d6ffd700 1 -- 172.21.15.111:0/2217853831 <== mds.0 v2:172.21.15.143:6834/1214316551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x1cc03e0 con 0x1ddb1b0 2023-12-19T07:47:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f80d6ffd700 10 client.4410 handle_command_reply: tid=0 2023-12-19T07:47:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 client.4410 shutdown 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 msgr2=0x1df8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f80f4050b60 tx=0x7f80f4078000 comp rx=0 tx=0).stop 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 shutdown_connections 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1ddb1b0 0x1e036b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76ae0 0x1df8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1dd9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f80d6ffd700 0 client.4410 ms_handle_reset on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 --2- 172.21.15.111:0/2217853831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76710 0x1df8510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 >> 172.21.15.111:0/2217853831 conn(0x1b76eb0 msgr2=0x1da8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 shutdown_connections 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 1 -- 172.21.15.111:0/2217853831 wait complete. 2023-12-19T07:47:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.976+0000 7f8100d40740 20 client.4410 trim_cache size 0 max 16384 2023-12-19T07:47:24.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.996+0000 7f8100d40740 1 Processor -- start 2023-12-19T07:47:24.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:24.996+0000 7f8100d40740 1 -- start start 2023-12-19T07:47:24.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76710 0x1dd8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1df3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d1cd60 con 0x1b76ae0 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d22ef0 con 0x1b76710 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1cccc90 con 0x1bc1790 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33472/0 (socket says 172.21.15.111:33472) 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 -- 172.21.15.111:0/44263597 learned_addr learned my addr 172.21.15.111:0/44263597 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80fd04b700 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1df3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80fc84a700 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76710 0x1dd8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 -- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 msgr2=0x1df3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1df3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 -- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76710 msgr2=0x1dd8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76710 0x1dd8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 -- 172.21.15.111:0/44263597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1cd2d80 con 0x1b76ae0 2023-12-19T07:47:24.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80effff700 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f80e80096f0 tx=0x7f80e8029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ffe998a4859b28 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80de7fc700 1 -- 172.21.15.111:0/44263597 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e802c040 con 0x1b76ae0 2023-12-19T07:47:24.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80de7fc700 1 -- 172.21.15.111:0/44263597 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e8020040 con 0x1b76ae0 2023-12-19T07:47:24.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.000+0000 7f80de7fc700 1 -- 172.21.15.111:0/44263597 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8034030 con 0x1b76ae0 2023-12-19T07:47:24.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- 172.21.15.111:0/44263597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 msgr2=0x1dd9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f80e80096f0 tx=0x7f80e8029040 comp rx=0 tx=0).stop 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- 172.21.15.111:0/44263597 shutdown_connections 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1bc1790 0x1df3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1b76ae0 0x1dd9220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- 172.21.15.111:0/44263597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1b76710 0x1dd8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- 172.21.15.111:0/44263597 >> 172.21.15.111:0/44263597 conn(0x1b76eb0 msgr2=0x1df6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- 172.21.15.111:0/44263597 shutdown_connections 2023-12-19T07:47:24.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- 172.21.15.111:0/44263597 wait complete. 2023-12-19T07:47:24.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- start start 2023-12-19T07:47:24.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 Processor -- start 2023-12-19T07:47:24.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1da9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1bc1790 0x1dad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d20590 con 0x1dada00 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d1b890 con 0x1bc1790 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1cf3550 con 0x1b76710 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f8100d40740 10 client.0 _ll_register_callbacks cb 0x1b429d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.004+0000 7f80fd04b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33482/0 (socket says 172.21.15.111:33482) 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 -- 172.21.15.111:0/1968475614 learned_addr learned my addr 172.21.15.111:0/1968475614 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80effff700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1bc1790 0x1dad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1da9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 -- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 msgr2=0x1da9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1da9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 -- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1bc1790 msgr2=0x1dad4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1bc1790 0x1dad4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 -- 172.21.15.111:0/1968475614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f806d500 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fd04b700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f80f0016b50 tx=0x7f80f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6079def540537d06 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f001b070 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f80f0014ab0 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80f4057bb0 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80f0007b10 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 10 client.4416 fetch_fsmap learned FSMap version 6 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x1cc7130 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4057bb0 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f80f001f410 con 0x1dada00 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 10 client.4416 fetch_fsmap finished waiting for FSMap version 6 2023-12-19T07:47:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 10 client.4416 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:47:24.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 20 client.4416 populate_metadata read hostname 'smithi111' 2023-12-19T07:47:24.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1e4d6d0 0x1e4daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:24.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 4 client.4416 mds_command: new command op to 4275 tid=0[{"prefix": "status"}] 2023-12-19T07:47:24.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 --> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1cf2410 con 0x1e4d6d0 2023-12-19T07:47:24.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.008+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1e4d6d0 0x1e4daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:24.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80d6ffd700 10 client.4416 ms_handle_connect on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fc84a700 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1e4d6d0 0x1e4daa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22e0f03caff50e10 server_cookie=e92910b5696f7644 in_seq=0 out_seq=0 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80d6ffd700 1 -- 172.21.15.111:0/1968475614 <== mds.0 v2:172.21.15.143:6834/1214316551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x1cf2410 con 0x1e4d6d0 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80d6ffd700 10 client.4416 handle_command_reply: tid=0 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 client.4416 shutdown 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 msgr2=0x1daddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f80f0016b50 tx=0x7f80f0008040 comp rx=0 tx=0).stop 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 shutdown_connections 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x1e4d6d0 0x1e4daa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1bc1790 0x1dad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1dada00 0x1daddd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80d6ffd700 0 client.4416 ms_handle_reset on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 --2- 172.21.15.111:0/1968475614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1b76710 0x1da9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 >> 172.21.15.111:0/1968475614 conn(0x1b76eb0 msgr2=0x1dd7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 shutdown_connections 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 1 -- 172.21.15.111:0/1968475614 wait complete. 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f8100d40740 20 client.4416 trim_cache size 0 max 16384 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4275, 2023-12-19T07:47:24.997 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 7.8153404960000001, 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 6, 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 25, 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 24, 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.675000606999999 2023-12-19T07:47:24.998 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 msgr2=0x7f80e4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 0x7f80e4062a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f80f806df30 tx=0x7f80e8017000 comp rx=0 tx=0).stop 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 msgr2=0x7f80f816c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80f400cd80 tx=0x7f80f4001b90 comp rx=0 tx=0).stop 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 shutdown_connections 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e4060580 0x7f80e4062a30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:24.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f8158e70 0x7f80f8156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f80cc470 0x7f80f8159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 --2- 172.21.15.111:0/1497772548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f806a540 0x7f80f816c880 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 >> 172.21.15.111:0/1497772548 conn(0x7f80f80670e0 msgr2=0x7f80f80b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:25.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 shutdown_connections 2023-12-19T07:47:25.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.012+0000 7f80fd84c700 1 -- 172.21.15.111:0/1497772548 wait complete. 2023-12-19T07:47:25.022 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:47:25.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.208+0000 7f8570e8e700 1 Processor -- start 2023-12-19T07:47:25.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.208+0000 7f8570e8e700 1 -- start start 2023-12-19T07:47:25.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:25.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c06d470 con 0x7f856c06b4f0 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c06c860 con 0x7f856c0a8850 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c06cb40 con 0x7f856c158cb0 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f856c06b4f0 0x7f856c06d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57204/0 (socket says 172.21.15.111:57204) 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856bfff700 1 -- 172.21.15.111:0/1134175857 learned_addr learned my addr 172.21.15.111:0/1134175857 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:25.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856affd700 1 --1- 172.21.15.111:0/1134175857 >> v1:172.21.15.143:6789/0 conn(0x7f856c0a8850 0x7f856c06ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36168/0 (socket says 172.21.15.111:36168) 2023-12-19T07:47:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542166362 0 0) 0x7f856c06c860 con 0x7f856c0a8850 2023-12-19T07:47:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554003570 con 0x7f856c0a8850 2023-12-19T07:47:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107874435 0 0) 0x7f856c06d470 con 0x7f856c06b4f0 2023-12-19T07:47:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c06c860 con 0x7f856c06b4f0 2023-12-19T07:47:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856b7fe700 1 -- 172.21.15.111:0/1134175857 >> v1:172.21.15.143:6789/0 conn(0x7f856c0a8850 legacy=0x7f856c06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856b7fe700 1 -- 172.21.15.111:0/1134175857 >> v1:172.21.15.111:6789/0 conn(0x7f856c06b4f0 legacy=0x7f856c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856b7fe700 1 -- 172.21.15.111:0/1134175857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c06ce20 con 0x7f856c158cb0 2023-12-19T07:47:25.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856b7fe700 1 --2- 172.21.15.111:0/1134175857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c0b1d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f855c009940 tx=0x7f855c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850679a3441fe61c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c011630 con 0x7f856c158cb0 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c011790 con 0x7f856c158cb0 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f856a7fc700 1 -- 172.21.15.111:0/1134175857 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c011b30 con 0x7f856c158cb0 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- 172.21.15.111:0/1134175857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 msgr2=0x7f856c0b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1134175857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c0b1d40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f855c009940 tx=0x7f855c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- 172.21.15.111:0/1134175857 shutdown_connections 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1134175857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c0b1d40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- 172.21.15.111:0/1134175857 >> 172.21.15.111:0/1134175857 conn(0x7f856c0684d0 msgr2=0x7f856c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- 172.21.15.111:0/1134175857 shutdown_connections 2023-12-19T07:47:25.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.212+0000 7f8570e8e700 1 -- 172.21.15.111:0/1134175857 wait complete. 2023-12-19T07:47:25.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 Processor -- start 2023-12-19T07:47:25.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 -- start start 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f856c0a8850 0x7f856c148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c9960 con 0x7f856c06b4f0 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c8950 con 0x7f856c0a8850 2023-12-19T07:47:25.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.216+0000 7f8570e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c94d0 con 0x7f856c158cb0 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33494/0 (socket says 172.21.15.111:33494) 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 -- 172.21.15.111:0/1165341429 learned_addr learned my addr 172.21.15.111:0/1165341429 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856affd700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f856c0a8850 0x7f856c148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856bfff700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 -- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 msgr2=0x7f856c148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 -- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f856c0a8850 msgr2=0x7f856c148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f856c0a8850 0x7f856c148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 -- 172.21.15.111:0/1165341429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c009420 con 0x7f856c06b4f0 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856bfff700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f856b7fe700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f856c069e80 tx=0x7f855c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c01f070 con 0x7f856c06b4f0 2023-12-19T07:47:25.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c000bc0 con 0x7f856c06b4f0 2023-12-19T07:47:25.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c0c8ba0 con 0x7f856c06b4f0 2023-12-19T07:47:25.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c025420 con 0x7f856c06b4f0 2023-12-19T07:47:25.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c0c9160 con 0x7f856c06b4f0 2023-12-19T07:47:25.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.220+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c04be40 con 0x7f856c06b4f0 2023-12-19T07:47:25.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.224+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f855c023070 con 0x7f856c06b4f0 2023-12-19T07:47:25.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.224+0000 7f85537fe700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 0x7f8558062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:25.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.224+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f855c07dd00 con 0x7f856c06b4f0 2023-12-19T07:47:25.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.224+0000 7f856affd700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 0x7f8558062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:25.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.228+0000 7f856affd700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 0x7f8558062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f856c0cbb60 tx=0x7f8560007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:25.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.228+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f855c051aa0 con 0x7f856c06b4f0 2023-12-19T07:47:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.436+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f856c06ee70 con 0x7f856c06b4f0 2023-12-19T07:47:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.436+0000 7f85537fe700 1 -- 172.21.15.111:0/1165341429 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f855c02a070 con 0x7f856c06b4f0 2023-12-19T07:47:25.420 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 msgr2=0x7f8558062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 0x7f8558062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f856c0cbb60 tx=0x7f8560007df0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 msgr2=0x7f856c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f856c069e80 tx=0x7f855c017000 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 shutdown_connections 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8558060550 0x7f8558062a00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f856c0a8850 0x7f856c148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f856c158cb0 0x7f856c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 --2- 172.21.15.111:0/1165341429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f856c06b4f0 0x7f856c15bee0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 >> 172.21.15.111:0/1165341429 conn(0x7f856c0684d0 msgr2=0x7f856c0ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 shutdown_connections 2023-12-19T07:47:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:25.440+0000 7f8570e8e700 1 -- 172.21.15.111:0/1165341429 wait complete. 2023-12-19T07:47:25.442 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:47:25.443 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-12-19T07:47:25.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.443 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:25.463 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42412sec preferred_lft 42412sec 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:25.464 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:47:25.465 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:25.465 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:47:25.465 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:25.465 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T07:47:25.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.465 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T07:47:25.466 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:25.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.615 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:47:25.649 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T07:47:25.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.652 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T07:47:25.730 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T07:47:25.730 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:47:25.730 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:47:25.730 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:47:25.731 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:47:25.731 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:47:25.731 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:47:25.731 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:47:25.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.731 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:25.731 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:25.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:47:25.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:47:25.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:47:25.732 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:25.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.868 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T07:47:25.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.886 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-12-19T07:47:25.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:25.950 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:25.950 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:25.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-12-19T07:47:25.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-12-19T07:47:25.950 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:26.064 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-12-19T07:47:26.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:26.065 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:26.065 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:26.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-12-19T07:47:26.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T07:47:26.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-12-19T07:47:26.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-12-19T07:47:26.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-12-19T07:47:26.066 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:26.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:26.243 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:26.243 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:26.243 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T07:47:26.243 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T07:47:26.243 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:26.317 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-12-19T07:47:26.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:26.317 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:26.335 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-12-19T07:47:26.335 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:26.335 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:26.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:26.476 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:26.476 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:26.555 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:26.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:26.558 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:26.618 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T07:47:26.618 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-12-19T07:47:26.668 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:26.705 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:26.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:26.780 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:26.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:26.784 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:26.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:2023-12-19T07:47:26.820+0000 7fe4d3dbe380 -1 init, newargv = 0x56544be94510 newargc=15 2023-12-19T07:47:26.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[39945]: starting ceph client 2023-12-19T07:47:26.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[39945]: starting fuse 2023-12-19T07:47:27.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:27.957 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:27.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:28.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:28.033 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:28.098 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T07:47:28.098 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T07:47:28.099 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.admin")) 2023-12-19T07:47:28.100 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T07:47:28.261 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.admin.39945.asok 2023-12-19T07:47:28.437 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.39945.asok 2023-12-19T07:47:28.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:28.437 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.39945.asok status 2023-12-19T07:47:28.615 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "admin", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "39945", 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4428, 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T07:47:28.616 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 4428 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 544825067 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 544825067 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.4428 192.168.0.1:0/544825067", 2023-12-19T07:47:28.617 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/544825067", 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 6, 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 25, 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T07:47:28.618 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:47:28.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:28.630 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T07:47:28.630 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:28.643 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T07:47:28.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:28.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:28.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:28.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:28.713 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-12-19T07:47:29.273 INFO:teuthology.orchestra.run.smithi111.stdout:# file: . 2023-12-19T07:47:29.274 INFO:teuthology.orchestra.run.smithi111.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-12-19T07:47:29.274 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:29.275 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T07:47:29.275 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:47:29.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-12-19T07:47:29.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[39945]: fuse finished with error 0 and tester_r 0 2023-12-19T07:47:29.342 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T07:47:29.389 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:47:29.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:29.389 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-12-19T07:47:29.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:29.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 Processor -- start 2023-12-19T07:47:29.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 -- start start 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44581502b0 0x7f4458068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44580c5900 0x7f44580c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580c6630 con 0x7f4458067c90 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44580c6910 con 0x7f44580c5900 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.632+0000 7f445c8ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580c6bf0 con 0x7f44581502b0 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44580c5900 0x7f44580c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44580c5900 0x7f44580c5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33506/0 (socket says 172.21.15.111:33506) 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 -- 172.21.15.111:0/1023105934 learned_addr learned my addr 172.21.15.111:0/1023105934 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f444dd9b700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44581502b0 0x7f4458068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 -- 172.21.15.111:0/1023105934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44581502b0 msgr2=0x7f4458068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44581502b0 0x7f4458068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 -- 172.21.15.111:0/1023105934 >> v1:172.21.15.143:6789/0 conn(0x7f4458067c90 legacy=0x7f445814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 -- 172.21.15.111:0/1023105934 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580c6ed0 con 0x7f44580c5900 2023-12-19T07:47:29.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4456d9d700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44580c5900 0x7f44580c5d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4450009f20 tx=0x7f4450009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9b5df391ff1c9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:29.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4455d9b700 1 -- 172.21.15.111:0/1023105934 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000e070 con 0x7f44580c5900 2023-12-19T07:47:29.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4455d9b700 1 -- 172.21.15.111:0/1023105934 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445000ba80 con 0x7f44580c5900 2023-12-19T07:47:29.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f4455d9b700 1 -- 172.21.15.111:0/1023105934 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44500092e0 con 0x7f44580c5900 2023-12-19T07:47:29.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 -- 172.21.15.111:0/1023105934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44580c5900 msgr2=0x7f44580c5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44580c5900 0x7f44580c5d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4450009f20 tx=0x7f4450009840 comp rx=0 tx=0).stop 2023-12-19T07:47:29.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 -- 172.21.15.111:0/1023105934 shutdown_connections 2023-12-19T07:47:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44580c5900 0x7f44580c5d30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 --2- 172.21.15.111:0/1023105934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44581502b0 0x7f4458068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 -- 172.21.15.111:0/1023105934 >> 172.21.15.111:0/1023105934 conn(0x7f44580cd6d0 msgr2=0x7f445806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 -- 172.21.15.111:0/1023105934 shutdown_connections 2023-12-19T07:47:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.636+0000 7f445c8ae700 1 -- 172.21.15.111:0/1023105934 wait complete. 2023-12-19T07:47:29.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 Processor -- start 2023-12-19T07:47:29.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 -- start start 2023-12-19T07:47:29.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4458067c90 0x7f445814edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 0x7f44580b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580b3680 con 0x7f44581502b0 2023-12-19T07:47:29.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44580b3050 con 0x7f44580c5900 2023-12-19T07:47:29.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445c8ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580c9d90 con 0x7f4458067c90 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f4456d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f4456d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33512/0 (socket says 172.21.15.111:33512) 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f444dd9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 0x7f44580b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f4456d9d700 1 -- 172.21.15.111:0/3410336873 learned_addr learned my addr 172.21.15.111:0/3410336873 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.640+0000 7f445659c700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4458067c90 0x7f445814edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 -- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4458067c90 msgr2=0x7f445814edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4458067c90 0x7f445814edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 -- 172.21.15.111:0/3410336873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 msgr2=0x7f44580b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 -- 172.21.15.111:0/3410336873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009ae0 con 0x7f44580c5900 2023-12-19T07:47:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f4456d9d700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444dd9b700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 0x7f44580b0d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f444000d860 tx=0x7f4440008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44400125f0 con 0x7f44580c5900 2023-12-19T07:47:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440012750 con 0x7f44580c5900 2023-12-19T07:47:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440012b10 con 0x7f44580c5900 2023-12-19T07:47:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44580b3310 con 0x7f44580c5900 2023-12-19T07:47:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445814f330 con 0x7f44580c5900 2023-12-19T07:47:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.644+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44580b3830 con 0x7f44580c5900 2023-12-19T07:47:29.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.648+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f444001e460 con 0x7f44580c5900 2023-12-19T07:47:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.648+0000 7f444effd700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 0x7f4438062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.648+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f444007cf90 con 0x7f44580c5900 2023-12-19T07:47:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.648+0000 7f445659c700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 0x7f4438062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:29.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.652+0000 7f445659c700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 0x7f4438062a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4448003a80 tx=0x7f4448007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:29.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.656+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44580b3830 con 0x7f44580c5900 2023-12-19T07:47:29.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.828+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44580b37f0 con 0x7f44580c5900 2023-12-19T07:47:29.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.832+0000 7f444effd700 1 -- 172.21.15.111:0/3410336873 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f4440050c80 con 0x7f44580c5900 2023-12-19T07:47:29.816 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:29.816 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 msgr2=0x7f4438062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 0x7f4438062a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4448003a80 tx=0x7f4448007ec0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 msgr2=0x7f44580b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 0x7f44580b0d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f444000d860 tx=0x7f4440008040 comp rx=0 tx=0).stop 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 shutdown_connections 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4438060550 0x7f4438062a00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44580c5900 0x7f44580b0d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44581502b0 0x7f44580b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 --2- 172.21.15.111:0/3410336873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4458067c90 0x7f445814edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:29.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 >> 172.21.15.111:0/3410336873 conn(0x7f44580cd6d0 msgr2=0x7f44580b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:29.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 shutdown_connections 2023-12-19T07:47:29.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:29.836+0000 7f445c8ae700 1 -- 172.21.15.111:0/3410336873 wait complete. 2023-12-19T07:47:29.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.024+0000 7fad24c8e700 1 Processor -- start 2023-12-19T07:47:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.028+0000 7fad24c8e700 1 -- start start 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.028+0000 7fad24c8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 0x7fad200cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.028+0000 7fad24c8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad2015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.028+0000 7fad24c8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad2015cee0 0x7fad20159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.028+0000 7fad24c8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad200cc910 con 0x7fad2015cee0 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad200682a0 con 0x7fad20158d40 2023-12-19T07:47:30.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2015d360 con 0x7fad200cb920 2023-12-19T07:47:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad2015cee0 0x7fad20159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad2015cee0 0x7fad20159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33526/0 (socket says 172.21.15.111:33526) 2023-12-19T07:47:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 -- 172.21.15.111:0/1721085199 learned_addr learned my addr 172.21.15.111:0/1721085199 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:30.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1f7fe700 1 --2- 172.21.15.111:0/1721085199 >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 0x7fad200cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1effd700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad2015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 -- 172.21.15.111:0/1721085199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 msgr2=0x7fad2015c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad2015c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 -- 172.21.15.111:0/1721085199 >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 msgr2=0x7fad200cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 --2- 172.21.15.111:0/1721085199 >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 0x7fad200cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 -- 172.21.15.111:0/1721085199 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad20068dc0 con 0x7fad2015cee0 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1f7fe700 1 --2- 172.21.15.111:0/1721085199 >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 0x7fad200cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1ffff700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad2015cee0 0x7fad20159110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad2006c590 tx=0x7fad18009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cc99c60bd77a71 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1e7fc700 1 -- 172.21.15.111:0/1721085199 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1800e070 con 0x7fad2015cee0 2023-12-19T07:47:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1e7fc700 1 -- 172.21.15.111:0/1721085199 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1800ba40 con 0x7fad2015cee0 2023-12-19T07:47:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad1e7fc700 1 -- 172.21.15.111:0/1721085199 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad180092e0 con 0x7fad2015cee0 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- 172.21.15.111:0/1721085199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad2015cee0 msgr2=0x7fad20159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad2015cee0 0x7fad20159110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad2006c590 tx=0x7fad18009840 comp rx=0 tx=0).stop 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- 172.21.15.111:0/1721085199 shutdown_connections 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad2015cee0 0x7fad20159110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 --2- 172.21.15.111:0/1721085199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad2015c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 --2- 172.21.15.111:0/1721085199 >> v2:172.21.15.143:3300/0 conn(0x7fad20158d40 0x7fad200cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- 172.21.15.111:0/1721085199 >> 172.21.15.111:0/1721085199 conn(0x7fad200cdbc0 msgr2=0x7fad200cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- 172.21.15.111:0/1721085199 shutdown_connections 2023-12-19T07:47:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.032+0000 7fad24c8e700 1 -- 172.21.15.111:0/1721085199 wait complete. 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 Processor -- start 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 -- start start 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad200afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 0x7fad2016cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2016c4e0 con 0x7fad20158d40 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2016c7c0 con 0x7fad2015cee0 2023-12-19T07:47:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad200c9da0 con 0x7fad200cb920 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33534/0 (socket says 172.21.15.111:33534) 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 -- 172.21.15.111:0/2628309094 learned_addr learned my addr 172.21.15.111:0/2628309094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1ffff700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 0x7fad2016cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1f7fe700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad200afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 -- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 msgr2=0x7fad200afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad200afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 -- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 msgr2=0x7fad2016cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 0x7fad2016cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 -- 172.21.15.111:0/2628309094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18009ae0 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1ffff700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 0x7fad2016cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad1effd700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad1000d890 tx=0x7fad10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad100125f0 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10012750 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10012ad0 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.036+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad200ca080 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad200c9a50 con 0x7fad20158d40 2023-12-19T07:47:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2004be40 con 0x7fad20158d40 2023-12-19T07:47:30.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fad1001e460 con 0x7fad20158d40 2023-12-19T07:47:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad177fe700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 0x7fad00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fad1007d110 con 0x7fad20158d40 2023-12-19T07:47:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.040+0000 7fad1f7fe700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 0x7fad00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.044+0000 7fad1f7fe700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 0x7fad00062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad08006f00 tx=0x7fad08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.048+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad1004d410 con 0x7fad20158d40 2023-12-19T07:47:30.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.220+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad200c4860 con 0x7fad20158d40 2023-12-19T07:47:30.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad177fe700 1 -- 172.21.15.111:0/2628309094 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7fad10050a30 con 0x7fad20158d40 2023-12-19T07:47:30.207 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:30.207 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:30.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 msgr2=0x7fad00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 0x7fad00062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad08006f00 tx=0x7fad08008040 comp rx=0 tx=0).stop 2023-12-19T07:47:30.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 msgr2=0x7fad2014bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad1000d890 tx=0x7fad10008040 comp rx=0 tx=0).stop 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 shutdown_connections 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad00060550 0x7fad00062a00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad2015cee0 0x7fad2016cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad20158d40 0x7fad2014bdf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 --2- 172.21.15.111:0/2628309094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad200cb920 0x7fad200afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 >> 172.21.15.111:0/2628309094 conn(0x7fad200cdbc0 msgr2=0x7fad200678e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:30.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.224+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 shutdown_connections 2023-12-19T07:47:30.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.228+0000 7fad24c8e700 1 -- 172.21.15.111:0/2628309094 wait complete. 2023-12-19T07:47:30.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T07:47:30.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 Processor -- start 2023-12-19T07:47:30.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 -- start start 2023-12-19T07:47:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8134158c90 0x7f81340caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f813406aeb0 0x7f8134159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813406c850 con 0x7f81341595a0 2023-12-19T07:47:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813406b470 con 0x7f813406aeb0 2023-12-19T07:47:30.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.416+0000 7f8138923700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813406b750 con 0x7f8134158c90 2023-12-19T07:47:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f813406aeb0 0x7f8134159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8133fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f81341595a0 0x7f813415c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57208/0 (socket says 172.21.15.111:57208) 2023-12-19T07:47:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f813406aeb0 0x7f8134159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33544/0 (socket says 172.21.15.111:33544) 2023-12-19T07:47:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8133fff700 1 -- 172.21.15.111:0/3644530191 learned_addr learned my addr 172.21.15.111:0/3644530191 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:30.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f81337fe700 1 --2- 172.21.15.111:0/3644530191 >> v2:172.21.15.143:3300/0 conn(0x7f8134158c90 0x7f81340caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 -- 172.21.15.111:0/3644530191 >> v2:172.21.15.143:3300/0 conn(0x7f8134158c90 msgr2=0x7f81340caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 --2- 172.21.15.111:0/3644530191 >> v2:172.21.15.143:3300/0 conn(0x7f8134158c90 0x7f81340caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 -- 172.21.15.111:0/3644530191 >> v1:172.21.15.111:6789/0 conn(0x7f81341595a0 legacy=0x7f813415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f81327fc700 1 -- 172.21.15.111:0/3644530191 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093238801 0 0) 0x7f813406c850 con 0x7f81341595a0 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 -- 172.21.15.111:0/3644530191 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813406ba30 con 0x7f813406aeb0 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8132ffd700 1 --2- 172.21.15.111:0/3644530191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813406aeb0 0x7f8134159060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81340698f0 tx=0x7f81280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6affa093bd4a05d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f81327fc700 1 -- 172.21.15.111:0/3644530191 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128011630 con 0x7f813406aeb0 2023-12-19T07:47:30.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f81327fc700 1 -- 172.21.15.111:0/3644530191 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128011790 con 0x7f813406aeb0 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f81327fc700 1 -- 172.21.15.111:0/3644530191 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128011b30 con 0x7f813406aeb0 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 -- 172.21.15.111:0/3644530191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813406aeb0 msgr2=0x7f8134159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 --2- 172.21.15.111:0/3644530191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813406aeb0 0x7f8134159060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81340698f0 tx=0x7f81280091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 -- 172.21.15.111:0/3644530191 shutdown_connections 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 --2- 172.21.15.111:0/3644530191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f813406aeb0 0x7f8134159060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 --2- 172.21.15.111:0/3644530191 >> v2:172.21.15.143:3300/0 conn(0x7f8134158c90 0x7f81340caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 -- 172.21.15.111:0/3644530191 >> 172.21.15.111:0/3644530191 conn(0x7f8134067bb0 msgr2=0x7f81340cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:30.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 -- 172.21.15.111:0/3644530191 shutdown_connections 2023-12-19T07:47:30.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.420+0000 7f8138923700 1 -- 172.21.15.111:0/3644530191 wait complete. 2023-12-19T07:47:30.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.424+0000 7f8138923700 1 Processor -- start 2023-12-19T07:47:30.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.424+0000 7f8138923700 1 -- start start 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f813406aeb0 0x7f8134157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 0x7f813416caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340c9920 con 0x7f8134158c90 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134155710 con 0x7f813406aeb0 2023-12-19T07:47:30.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81341559f0 con 0x7f81341595a0 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f81337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f813406aeb0 0x7f8134157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33554/0 (socket says 172.21.15.111:33554) 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 -- 172.21.15.111:0/85129 learned_addr learned my addr 172.21.15.111:0/85129 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8133fff700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 0x7f813416caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 -- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 msgr2=0x7f813416caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 0x7f813416caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 -- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f813406aeb0 msgr2=0x7f8134157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f813406aeb0 0x7f8134157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 -- 172.21.15.111:0/85129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128009420 con 0x7f8134158c90 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8133fff700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 0x7f813416caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8132ffd700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8134069b70 tx=0x7f8128017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128011b10 con 0x7f8134158c90 2023-12-19T07:47:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81280253f0 con 0x7f8134158c90 2023-12-19T07:47:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 -- 172.21.15.111:0/85129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8134155cd0 con 0x7f8134158c90 2023-12-19T07:47:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128025790 con 0x7f8134158c90 2023-12-19T07:47:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.428+0000 7f8138923700 1 -- 172.21.15.111:0/85129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81340c6f40 con 0x7f8134158c90 2023-12-19T07:47:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.432+0000 7f8138923700 1 -- 172.21.15.111:0/85129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813406c7c0 con 0x7f8134158c90 2023-12-19T07:47:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.432+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8128025b80 con 0x7f8134158c90 2023-12-19T07:47:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.432+0000 7f811b7fe700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 0x7f8120062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.436+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f812807e400 con 0x7f8134158c90 2023-12-19T07:47:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.436+0000 7f81337fe700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 0x7f8120062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:30.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.436+0000 7f81337fe700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 0x7f8120062a30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8124009d20 tx=0x7f81240091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:30.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.440+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f813406c7c0 con 0x7f8134158c90 2023-12-19T07:47:30.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:30.648+0000 7f8138923700 1 -- 172.21.15.111:0/85129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f813406e140 con 0x7f8134158c90 2023-12-19T07:47:31.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.040+0000 7f811b7fe700 1 -- 172.21.15.111:0/85129 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f813406e140 con 0x7f8134158c90 2023-12-19T07:47:31.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 msgr2=0x7f8120062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 0x7f8120062a30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8124009d20 tx=0x7f81240091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 msgr2=0x7f813416c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8134069b70 tx=0x7f8128017000 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 shutdown_connections 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8120060580 0x7f8120062a30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81341595a0 0x7f813416caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134158c90 0x7f813416c5b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 --2- 172.21.15.111:0/85129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f813406aeb0 0x7f8134157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 >> 172.21.15.111:0/85129 conn(0x7f8134067bb0 msgr2=0x7f81340afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 shutdown_connections 2023-12-19T07:47:31.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.044+0000 7f8138923700 1 -- 172.21.15.111:0/85129 wait complete. 2023-12-19T07:47:31.048 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T07:47:31.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.232+0000 7f5b9f59e700 1 Processor -- start 2023-12-19T07:47:31.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.232+0000 7f5b9f59e700 1 -- start start 2023-12-19T07:47:31.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ba013f6a0 0x7f5ba00ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00adc60 con 0x7f5ba013f2d0 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00ad050 con 0x7f5ba00c7780 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba00ad330 con 0x7f5ba013f6a0 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5ba013f2d0 0x7f5ba015a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:57212/0 (socket says 172.21.15.111:57212) 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9ed9d700 1 --1- 172.21.15.111:0/2136911771 >> v1:172.21.15.143:6789/0 conn(0x7f5ba00c7780 0x7f5ba006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36180/0 (socket says 172.21.15.111:36180) 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9e59c700 1 -- 172.21.15.111:0/2136911771 learned_addr learned my addr 172.21.15.111:0/2136911771 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:31.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9dd9b700 1 --2- 172.21.15.111:0/2136911771 >> v2:172.21.15.111:3300/0 conn(0x7f5ba013f6a0 0x7f5ba00ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:31.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271513681 0 0) 0x7f5ba00ad050 con 0x7f5ba00c7780 2023-12-19T07:47:31.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88003570 con 0x7f5ba00c7780 2023-12-19T07:47:31.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397153621 0 0) 0x7f5ba00adc60 con 0x7f5ba013f2d0 2023-12-19T07:47:31.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba00ad050 con 0x7f5ba013f2d0 2023-12-19T07:47:31.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9dd9b700 1 -- 172.21.15.111:0/2136911771 >> v1:172.21.15.143:6789/0 conn(0x7f5ba00c7780 legacy=0x7f5ba006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9dd9b700 1 -- 172.21.15.111:0/2136911771 >> v1:172.21.15.111:6789/0 conn(0x7f5ba013f2d0 legacy=0x7f5ba015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9dd9b700 1 -- 172.21.15.111:0/2136911771 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00ad610 con 0x7f5ba013f6a0 2023-12-19T07:47:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.236+0000 7f5b9dd9b700 1 --2- 172.21.15.111:0/2136911771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba00ac9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098e0 tx=0x7f5b940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36298916e95a5d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94011630 con 0x7f5ba013f6a0 2023-12-19T07:47:31.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b94011790 con 0x7f5ba013f6a0 2023-12-19T07:47:31.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9d59a700 1 -- 172.21.15.111:0/2136911771 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94011b30 con 0x7f5ba013f6a0 2023-12-19T07:47:31.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- 172.21.15.111:0/2136911771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 msgr2=0x7f5ba00ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/2136911771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba00ac9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098e0 tx=0x7f5b940091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- 172.21.15.111:0/2136911771 shutdown_connections 2023-12-19T07:47:31.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/2136911771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba00ac9b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- 172.21.15.111:0/2136911771 >> 172.21.15.111:0/2136911771 conn(0x7f5ba0067b50 msgr2=0x7f5ba00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:31.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- 172.21.15.111:0/2136911771 shutdown_connections 2023-12-19T07:47:31.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- 172.21.15.111:0/2136911771 wait complete. 2023-12-19T07:47:31.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 Processor -- start 2023-12-19T07:47:31.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.240+0000 7f5b9f59e700 1 -- start start 2023-12-19T07:47:31.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ba00c7780 0x7f5ba00c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ba013f2d0 0x7f5ba0155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00c1910 con 0x7f5ba013f6a0 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba013dbe0 con 0x7f5ba00c7780 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00c5980 con 0x7f5ba013f2d0 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:31.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ba00c7780 0x7f5ba00c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:31.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33566/0 (socket says 172.21.15.111:33566) 2023-12-19T07:47:31.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 -- 172.21.15.111:0/1921427129 learned_addr learned my addr 172.21.15.111:0/1921427129 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:31.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9dd9b700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ba013f2d0 0x7f5ba0155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:31.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 -- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ba013f2d0 msgr2=0x7f5ba0155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ba013f2d0 0x7f5ba0155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 -- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ba00c7780 msgr2=0x7f5ba00c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:31.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ba00c7780 0x7f5ba00c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:31.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 -- 172.21.15.111:0/1921427129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94009420 con 0x7f5ba013f6a0 2023-12-19T07:47:31.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9ed9d700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800ccc0 tx=0x7f5b98001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:31.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98005df0 con 0x7f5ba013f6a0 2023-12-19T07:47:31.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9800a690 con 0x7f5ba013f6a0 2023-12-19T07:47:31.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba00c56a0 con 0x7f5ba013f6a0 2023-12-19T07:47:31.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800aa50 con 0x7f5ba013f6a0 2023-12-19T07:47:31.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.244+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba00c0e10 con 0x7f5ba013f6a0 2023-12-19T07:47:31.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.248+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba00c1a80 con 0x7f5ba013f6a0 2023-12-19T07:47:31.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.248+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5b980070f0 con 0x7f5ba013f6a0 2023-12-19T07:47:31.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.248+0000 7f5b8e7fc700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 0x7f5b80062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:31.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.248+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f5b9807cbd0 con 0x7f5ba013f6a0 2023-12-19T07:47:31.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.248+0000 7f5b9e59c700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 0x7f5b80062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:31.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.252+0000 7f5b9e59c700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 0x7f5b80062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5b90007fd0 tx=0x7f5b90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:31.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.256+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5b9804ceb0 con 0x7f5ba013f6a0 2023-12-19T07:47:31.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:31.464+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5ba004be40 con 0x7f5ba013f6a0 2023-12-19T07:47:32.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.048+0000 7f5b8e7fc700 1 -- 172.21.15.111:0/1921427129 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5b980504d0 con 0x7f5ba013f6a0 2023-12-19T07:47:32.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 msgr2=0x7f5b80062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 0x7f5b80062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5b90007fd0 tx=0x7f5b90009040 comp rx=0 tx=0).stop 2023-12-19T07:47:32.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 msgr2=0x7f5ba0168cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800ccc0 tx=0x7f5b98001b40 comp rx=0 tx=0).stop 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 shutdown_connections 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b80060550 0x7f5b80062a00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ba013f6a0 0x7f5ba0168cd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ba013f2d0 0x7f5ba0155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 --2- 172.21.15.111:0/1921427129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ba00c7780 0x7f5ba00c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 >> 172.21.15.111:0/1921427129 conn(0x7f5ba0067b50 msgr2=0x7f5ba00c7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 shutdown_connections 2023-12-19T07:47:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.052+0000 7f5b9f59e700 1 -- 172.21.15.111:0/1921427129 wait complete. 2023-12-19T07:47:32.060 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:32.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.244+0000 7fb869b45700 1 Processor -- start 2023-12-19T07:47:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.244+0000 7fb869b45700 1 -- start start 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.244+0000 7fb869b45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb864159030 0x7fb8640c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8640ca500 0x7fb86406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 0x7fb8640c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86415dbc0 con 0x7fb864159030 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86406d9d0 con 0x7fb8640ca500 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86415cd10 con 0x7fb86406d590 2023-12-19T07:47:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb864159030 0x7fb8640c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb864159030 0x7fb8640c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34438/0 (socket says 172.21.15.111:34438) 2023-12-19T07:47:32.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 -- 172.21.15.111:0/1911675908 learned_addr learned my addr 172.21.15.111:0/1911675908 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:32.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb862ffd700 1 --2- 172.21.15.111:0/1911675908 >> v2:172.21.15.143:3300/0 conn(0x7fb8640ca500 0x7fb86406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb863fff700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 0x7fb8640c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 -- 172.21.15.111:0/1911675908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 msgr2=0x7fb8640c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 0x7fb8640c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 -- 172.21.15.111:0/1911675908 >> v2:172.21.15.143:3300/0 conn(0x7fb8640ca500 msgr2=0x7fb86406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 --2- 172.21.15.111:0/1911675908 >> v2:172.21.15.143:3300/0 conn(0x7fb8640ca500 0x7fb86406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 -- 172.21.15.111:0/1911675908 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86415cfb0 con 0x7fb864159030 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb863fff700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 0x7fb8640c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8637fe700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640c9fc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb8640c3de0 tx=0x7fb854009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df9477219a05a20 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8627fc700 1 -- 172.21.15.111:0/1911675908 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85400b860 con 0x7fb864159030 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8627fc700 1 -- 172.21.15.111:0/1911675908 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85400b9c0 con 0x7fb864159030 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb8627fc700 1 -- 172.21.15.111:0/1911675908 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85400bd60 con 0x7fb864159030 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- 172.21.15.111:0/1911675908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 msgr2=0x7fb8640c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640c9fc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb8640c3de0 tx=0x7fb854009250 comp rx=0 tx=0).stop 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- 172.21.15.111:0/1911675908 shutdown_connections 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86406d590 0x7fb8640c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- 172.21.15.111:0/1911675908 >> v2:172.21.15.143:3300/0 conn(0x7fb8640ca500 0x7fb86406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 --2- 172.21.15.111:0/1911675908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640c9fc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- 172.21.15.111:0/1911675908 >> 172.21.15.111:0/1911675908 conn(0x7fb8640cd8e0 msgr2=0x7fb8640c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- 172.21.15.111:0/1911675908 shutdown_connections 2023-12-19T07:47:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.248+0000 7fb869b45700 1 -- 172.21.15.111:0/1911675908 wait complete. 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 Processor -- start 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 -- start start 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86406d590 0x7fb864068220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8640ca500 0x7fb8641581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864068b70 con 0x7fb864159030 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864068e50 con 0x7fb86406d590 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb869b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb864069130 con 0x7fb8640ca500 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb863fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb863fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34446/0 (socket says 172.21.15.111:34446) 2023-12-19T07:47:32.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb863fff700 1 -- 172.21.15.111:0/2378521639 learned_addr learned my addr 172.21.15.111:0/2378521639 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:32.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb862ffd700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8640ca500 0x7fb8641581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.252+0000 7fb8637fe700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86406d590 0x7fb864068220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 -- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8640ca500 msgr2=0x7fb8641581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8640ca500 0x7fb8641581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 -- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86406d590 msgr2=0x7fb864068220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86406d590 0x7fb864068220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 -- 172.21.15.111:0/2378521639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8540094f0 con 0x7fb864159030 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb863fff700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00a380 tx=0x7fb85c007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c015440 con 0x7fb864159030 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c0155a0 con 0x7fb864159030 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb864069410 con 0x7fb864159030 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c015960 con 0x7fb864159030 2023-12-19T07:47:32.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864068900 con 0x7fb864159030 2023-12-19T07:47:32.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.256+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb864071120 con 0x7fb864159030 2023-12-19T07:47:32.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.260+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb85c00d440 con 0x7fb864159030 2023-12-19T07:47:32.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.260+0000 7fb84b7fe700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 0x7fb850062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.260+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb85c07d550 con 0x7fb864159030 2023-12-19T07:47:32.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.260+0000 7fb8637fe700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 0x7fb850062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.260+0000 7fb8637fe700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 0x7fb850062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8640c3de0 tx=0x7fb854017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.268+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb85c04d850 con 0x7fb864159030 2023-12-19T07:47:32.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.464+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb86415d290 con 0x7fb864159030 2023-12-19T07:47:32.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb84b7fe700 1 -- 172.21.15.111:0/2378521639 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7fb85c050e70 con 0x7fb864159030 2023-12-19T07:47:32.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:32.451 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 msgr2=0x7fb850062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 0x7fb850062a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8640c3de0 tx=0x7fb854017000 comp rx=0 tx=0).stop 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 msgr2=0x7fb8640b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00a380 tx=0x7fb85c007900 comp rx=0 tx=0).stop 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 shutdown_connections 2023-12-19T07:47:32.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb850060550 0x7fb850062a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb864159030 0x7fb8640b19c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8640ca500 0x7fb8641581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 --2- 172.21.15.111:0/2378521639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86406d590 0x7fb864068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 >> 172.21.15.111:0/2378521639 conn(0x7fb8640cd8e0 msgr2=0x7fb8640b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 shutdown_connections 2023-12-19T07:47:32.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.468+0000 7fb869b45700 1 -- 172.21.15.111:0/2378521639 wait complete. 2023-12-19T07:47:32.456 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:47:32.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:32.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.656+0000 7fb7394f0700 1 Processor -- start 2023-12-19T07:47:32.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.656+0000 7fb7394f0700 1 -- start start 2023-12-19T07:47:32.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7394f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 0x7fb734067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7394f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb734159340 0x7fb7340cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7394f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb734068c70 con 0x7fb734069100 2023-12-19T07:47:32.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7394f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb734068060 con 0x7fb734159340 2023-12-19T07:47:32.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7394f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb734068340 con 0x7fb734158f70 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7327fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb734069100 0x7fb7340cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56036/0 (socket says 172.21.15.111:56036) 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7327fc700 1 -- 172.21.15.111:0/3811126553 learned_addr learned my addr 172.21.15.111:0/3811126553 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb732ffd700 1 --2- 172.21.15.111:0/3811126553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 0x7fb734067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb7337fe700 1 --2- 172.21.15.111:0/3811126553 >> v2:172.21.15.143:3300/0 conn(0x7fb734159340 0x7fb7340cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb731ffb700 1 -- 172.21.15.111:0/3811126553 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762809695 0 0) 0x7fb734068c70 con 0x7fb734069100 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.660+0000 7fb731ffb700 1 -- 172.21.15.111:0/3811126553 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb71c003570 con 0x7fb734069100 2023-12-19T07:47:32.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb732ffd700 1 -- 172.21.15.111:0/3811126553 >> v2:172.21.15.143:3300/0 conn(0x7fb734159340 msgr2=0x7fb7340cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:32.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb732ffd700 1 --2- 172.21.15.111:0/3811126553 >> v2:172.21.15.143:3300/0 conn(0x7fb734159340 0x7fb7340cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb732ffd700 1 -- 172.21.15.111:0/3811126553 >> v1:172.21.15.111:6789/0 conn(0x7fb734069100 legacy=0x7fb7340cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb732ffd700 1 -- 172.21.15.111:0/3811126553 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734068620 con 0x7fb734158f70 2023-12-19T07:47:32.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb732ffd700 1 --2- 172.21.15.111:0/3811126553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 0x7fb734067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb7240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=412034864e310a22 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb731ffb700 1 -- 172.21.15.111:0/3811126553 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011630 con 0x7fb734158f70 2023-12-19T07:47:32.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb731ffb700 1 -- 172.21.15.111:0/3811126553 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724011790 con 0x7fb734158f70 2023-12-19T07:47:32.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb731ffb700 1 -- 172.21.15.111:0/3811126553 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011b30 con 0x7fb734158f70 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 -- 172.21.15.111:0/3811126553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 msgr2=0x7fb734067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 --2- 172.21.15.111:0/3811126553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 0x7fb734067980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb7240098e0 tx=0x7fb7240091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 -- 172.21.15.111:0/3811126553 shutdown_connections 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 --2- 172.21.15.111:0/3811126553 >> v2:172.21.15.143:3300/0 conn(0x7fb734159340 0x7fb7340cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 --2- 172.21.15.111:0/3811126553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734158f70 0x7fb734067980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 -- 172.21.15.111:0/3811126553 >> 172.21.15.111:0/3811126553 conn(0x7fb7340ccc40 msgr2=0x7fb7340cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 -- 172.21.15.111:0/3811126553 shutdown_connections 2023-12-19T07:47:32.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.664+0000 7fb7394f0700 1 -- 172.21.15.111:0/3811126553 wait complete. 2023-12-19T07:47:32.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 Processor -- start 2023-12-19T07:47:32.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 -- start start 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb734158f70 0x7fb734150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734159340 0x7fb73406cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73416d930 con 0x7fb734069100 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7341540f0 con 0x7fb734158f70 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7394f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73416d4a0 con 0x7fb734159340 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34462/0 (socket says 172.21.15.111:34462) 2023-12-19T07:47:32.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 -- 172.21.15.111:0/2063235296 learned_addr learned my addr 172.21.15.111:0/2063235296 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:32.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7327fc700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb734158f70 0x7fb734150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb7337fe700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734159340 0x7fb73406cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 -- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734159340 msgr2=0x7fb73406cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734159340 0x7fb73406cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 -- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb734158f70 msgr2=0x7fb734150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb734158f70 0x7fb734150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.668+0000 7fb732ffd700 1 -- 172.21.15.111:0/2063235296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724009420 con 0x7fb734069100 2023-12-19T07:47:32.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb732ffd700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb724009ed0 tx=0x7fb724017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011b10 con 0x7fb734069100 2023-12-19T07:47:32.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7240253f0 con 0x7fb734069100 2023-12-19T07:47:32.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73416cbb0 con 0x7fb734069100 2023-12-19T07:47:32.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7240257b0 con 0x7fb734069100 2023-12-19T07:47:32.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73416d130 con 0x7fb734069100 2023-12-19T07:47:32.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73415cf00 con 0x7fb734069100 2023-12-19T07:47:32.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.672+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb724025b60 con 0x7fb734069100 2023-12-19T07:47:32.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.676+0000 7fb71affd700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 0x7fb720062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:32.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.676+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb72407e460 con 0x7fb734069100 2023-12-19T07:47:32.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.676+0000 7fb7327fc700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 0x7fb720062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:32.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.676+0000 7fb7327fc700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 0x7fb720062a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb734154420 tx=0x7fb728009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:32.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.680+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb73415cf00 con 0x7fb734069100 2023-12-19T07:47:32.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.848+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb73415cf00 con 0x7fb734069100 2023-12-19T07:47:32.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.848+0000 7fb71affd700 1 -- 172.21.15.111:0/2063235296 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7fb73415cf00 con 0x7fb734069100 2023-12-19T07:47:32.834 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:32.834 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:32.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 msgr2=0x7fb720062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 0x7fb720062a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb734154420 tx=0x7fb728009040 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 msgr2=0x7fb7341538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb724009ed0 tx=0x7fb724017000 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 shutdown_connections 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb720060550 0x7fb720062a00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb734158f70 0x7fb734150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb734159340 0x7fb73406cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 --2- 172.21.15.111:0/2063235296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb734069100 0x7fb7341538a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 >> 172.21.15.111:0/2063235296 conn(0x7fb7340ccc40 msgr2=0x7fb73415c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 shutdown_connections 2023-12-19T07:47:32.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:32.856+0000 7fb7394f0700 1 -- 172.21.15.111:0/2063235296 wait complete. 2023-12-19T07:47:32.861 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T07:47:32.861 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:47:33.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.032+0000 7f3445443700 1 Processor -- start 2023-12-19T07:47:33.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- start start 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3440158e20 0x7f344006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f34400ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344006b730 con 0x7f34401591f0 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344006ab20 con 0x7f3440158e20 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344006ae00 con 0x7f34400ce810 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3440158e20 0x7f344006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3440158e20 0x7f344006a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34466/0 (socket says 172.21.15.111:34466) 2023-12-19T07:47:33.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 -- 172.21.15.111:0/1469221215 learned_addr learned my addr 172.21.15.111:0/1469221215 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:33.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3444c42700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f34400ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 -- 172.21.15.111:0/1469221215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 msgr2=0x7f34400ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f34400ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 -- 172.21.15.111:0/1469221215 >> v1:172.21.15.143:6789/0 conn(0x7f34401591f0 legacy=0x7f34400ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 -- 172.21.15.111:0/1469221215 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344006b0e0 con 0x7f3440158e20 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3444c42700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f34400ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343ffff700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f344006a540 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f34300099b0 tx=0x7f3430009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853bc1761747a251 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343effd700 1 -- 172.21.15.111:0/1469221215 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000b860 con 0x7f3440158e20 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343effd700 1 -- 172.21.15.111:0/1469221215 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000b9c0 con 0x7f3440158e20 2023-12-19T07:47:33.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f343effd700 1 -- 172.21.15.111:0/1469221215 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000bd60 con 0x7f3440158e20 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- 172.21.15.111:0/1469221215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 msgr2=0x7f344006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f344006a540 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f34300099b0 tx=0x7f3430009250 comp rx=0 tx=0).stop 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- 172.21.15.111:0/1469221215 shutdown_connections 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f34400ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 --2- 172.21.15.111:0/1469221215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f344006a540 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.036+0000 7f3445443700 1 -- 172.21.15.111:0/1469221215 >> 172.21.15.111:0/1469221215 conn(0x7f3440067bb0 msgr2=0x7f34400cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.040+0000 7f3445443700 1 -- 172.21.15.111:0/1469221215 shutdown_connections 2023-12-19T07:47:33.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.040+0000 7f3445443700 1 -- 172.21.15.111:0/1469221215 wait complete. 2023-12-19T07:47:33.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.040+0000 7f3445443700 1 Processor -- start 2023-12-19T07:47:33.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.040+0000 7f3445443700 1 -- start start 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f3440152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34401591f0 0x7f344015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344016a780 con 0x7f3440158e20 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440153300 con 0x7f34401591f0 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440158210 con 0x7f34400ce810 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34480/0 (socket says 172.21.15.111:34480) 2023-12-19T07:47:33.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 -- 172.21.15.111:0/4251588052 learned_addr learned my addr 172.21.15.111:0/4251588052 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:33.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343ffff700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f3440152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3444c42700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34401591f0 0x7f344015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 -- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 msgr2=0x7f3440152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f3440152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 -- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34401591f0 msgr2=0x7f344015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34401591f0 0x7f344015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 -- 172.21.15.111:0/4251588052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300094f0 con 0x7f3440158e20 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f343f7fe700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f343400c4e0 tx=0x7f343400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434011590 con 0x7f3440158e20 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434011710 con 0x7f3440158e20 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440169b70 con 0x7f3440158e20 2023-12-19T07:47:33.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434011ad0 con 0x7f3440158e20 2023-12-19T07:47:33.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.044+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344016a130 con 0x7f3440158e20 2023-12-19T07:47:33.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.048+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344004be40 con 0x7f3440158e20 2023-12-19T07:47:33.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.048+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f343401b3b0 con 0x7f3440158e20 2023-12-19T07:47:33.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.048+0000 7f3427fff700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 0x7f342c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.052+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f343407be20 con 0x7f3440158e20 2023-12-19T07:47:33.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.052+0000 7f343ffff700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 0x7f342c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.052+0000 7f343ffff700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 0x7f342c062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34300099b0 tx=0x7f3430017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.056+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f343404c120 con 0x7f3440158e20 2023-12-19T07:47:33.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.236+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f344006b3c0 con 0x7f342c060550 2023-12-19T07:47:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.260+0000 7f3427fff700 1 -- 172.21.15.111:0/4251588052 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345063 (secure 0 0 0) 0x7f344006b3c0 con 0x7f342c060550 2023-12-19T07:47:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 msgr2=0x7f342c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 0x7f342c062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f34300099b0 tx=0x7f3430017000 comp rx=0 tx=0).stop 2023-12-19T07:47:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 msgr2=0x7f3440157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f343400c4e0 tx=0x7f343400c7f0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 shutdown_connections 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f342c060550 0x7f342c062a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34401591f0 0x7f344015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3440158e20 0x7f3440157cd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 --2- 172.21.15.111:0/4251588052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34400ce810 0x7f3440152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 >> 172.21.15.111:0/4251588052 conn(0x7f3440067bb0 msgr2=0x7f34400c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 shutdown_connections 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.272+0000 7f3445443700 1 -- 172.21.15.111:0/4251588052 wait complete. 2023-12-19T07:47:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:47:33.275 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-12-19T07:47:32.614682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381604,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593116,"statfs":{"total":773094113280,"available":770655350784,"internally_reserved":0,"allocated":3379200,"data_stored":2041019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003315"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904960+0000","last_change":"2023-12-19T07:47:16.213519+0000","last_active":"2023-12-19T07:47:22.904960+0000","last_peered":"2023-12-19T07:47:22.904960+0000","last_clean":"2023-12-19T07:47:22.904960+0000","last_became_active":"2023-12-19T07:47:16.213172+0000","last_became_peered":"2023-12-19T07:47:16.213172+0000","last_unstale":"2023-12-19T07:47:22.904960+0000","last_undegraded":"2023-12-19T07:47:22.904960+0000","last_fullsized":"2023-12-19T07:47:22.904960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599466+0000","last_change":"2023-12-19T07:47:17.206618+0000","last_active":"2023-12-19T07:47:22.599466+0000","last_peered":"2023-12-19T07:47:22.599466+0000","last_clean":"2023-12-19T07:47:22.599466+0000","last_became_active":"2023-12-19T07:47:17.206300+0000","last_became_peered":"2023-12-19T07:47:17.206300+0000","last_unstale":"2023-12-19T07:47:22.599466+0000","last_undegraded":"2023-12-19T07:47:22.599466+0000","last_fullsized":"2023-12-19T07:47:22.599466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597951+0000","last_change":"2023-12-19T07:47:21.847101+0000","last_active":"2023-12-19T07:47:22.597951+0000","last_peered":"2023-12-19T07:47:22.597951+0000","last_clean":"2023-12-19T07:47:22.597951+0000","last_became_active":"2023-12-19T07:47:21.845076+0000","last_became_peered":"2023-12-19T07:47:21.845076+0000","last_unstale":"2023-12-19T07:47:22.597951+0000","last_undegraded":"2023-12-19T07:47:22.597951+0000","last_fullsized":"2023-12-19T07:47:22.597951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599595+0000","last_change":"2023-12-19T07:47:16.198270+0000","last_active":"2023-12-19T07:47:22.599595+0000","last_peered":"2023-12-19T07:47:22.599595+0000","last_clean":"2023-12-19T07:47:22.599595+0000","last_became_active":"2023-12-19T07:47:16.197562+0000","last_became_peered":"2023-12-19T07:47:16.197562+0000","last_unstale":"2023-12-19T07:47:22.599595+0000","last_undegraded":"2023-12-19T07:47:22.599595+0000","last_fullsized":"2023-12-19T07:47:22.599595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904301+0000","last_change":"2023-12-19T07:47:17.198174+0000","last_active":"2023-12-19T07:47:22.904301+0000","last_peered":"2023-12-19T07:47:22.904301+0000","last_clean":"2023-12-19T07:47:22.904301+0000","last_became_active":"2023-12-19T07:47:17.197876+0000","last_became_peered":"2023-12-19T07:47:17.197876+0000","last_unstale":"2023-12-19T07:47:22.904301+0000","last_undegraded":"2023-12-19T07:47:22.904301+0000","last_fullsized":"2023-12-19T07:47:22.904301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306106+0000","last_change":"2023-12-19T07:47:21.260994+0000","last_active":"2023-12-19T07:47:22.306106+0000","last_peered":"2023-12-19T07:47:22.306106+0000","last_clean":"2023-12-19T07:47:22.306106+0000","last_became_active":"2023-12-19T07:47:21.260122+0000","last_became_peered":"2023-12-19T07:47:21.260122+0000","last_unstale":"2023-12-19T07:47:22.306106+0000","last_undegraded":"2023-12-19T07:47:22.306106+0000","last_fullsized":"2023-12-19T07:47:22.306106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868136+0000","last_change":"2023-12-19T07:47:16.215728+0000","last_active":"2023-12-19T07:47:22.868136+0000","last_peered":"2023-12-19T07:47:22.868136+0000","last_clean":"2023-12-19T07:47:22.868136+0000","last_became_active":"2023-12-19T07:47:16.215434+0000","last_became_peered":"2023-12-19T07:47:16.215434+0000","last_unstale":"2023-12-19T07:47:22.868136+0000","last_undegraded":"2023-12-19T07:47:22.868136+0000","last_fullsized":"2023-12-19T07:47:22.868136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806795+0000","last_change":"2023-12-19T07:47:17.204556+0000","last_active":"2023-12-19T07:47:22.806795+0000","last_peered":"2023-12-19T07:47:22.806795+0000","last_clean":"2023-12-19T07:47:22.806795+0000","last_became_active":"2023-12-19T07:47:17.204225+0000","last_became_peered":"2023-12-19T07:47:17.204225+0000","last_unstale":"2023-12-19T07:47:22.806795+0000","last_undegraded":"2023-12-19T07:47:22.806795+0000","last_fullsized":"2023-12-19T07:47:22.806795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306583+0000","last_change":"2023-12-19T07:47:21.266228+0000","last_active":"2023-12-19T07:47:22.306583+0000","last_peered":"2023-12-19T07:47:22.306583+0000","last_clean":"2023-12-19T07:47:22.306583+0000","last_became_active":"2023-12-19T07:47:21.265305+0000","last_became_peered":"2023-12-19T07:47:21.265305+0000","last_unstale":"2023-12-19T07:47:22.306583+0000","last_undegraded":"2023-12-19T07:47:22.306583+0000","last_fullsized":"2023-12-19T07:47:22.306583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867767+0000","last_change":"2023-12-19T07:47:16.214913+0000","last_active":"2023-12-19T07:47:22.867767+0000","last_peered":"2023-12-19T07:47:22.867767+0000","last_clean":"2023-12-19T07:47:22.867767+0000","last_became_active":"2023-12-19T07:47:16.214514+0000","last_became_peered":"2023-12-19T07:47:16.214514+0000","last_unstale":"2023-12-19T07:47:22.867767+0000","last_undegraded":"2023-12-19T07:47:22.867767+0000","last_fullsized":"2023-12-19T07:47:22.867767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:24.444351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807216+0000","last_change":"2023-12-19T07:47:17.208692+0000","last_active":"2023-12-19T07:47:22.807216+0000","last_peered":"2023-12-19T07:47:22.807216+0000","last_clean":"2023-12-19T07:47:22.807216+0000","last_became_active":"2023-12-19T07:47:17.208320+0000","last_became_peered":"2023-12-19T07:47:17.208320+0000","last_unstale":"2023-12-19T07:47:22.807216+0000","last_undegraded":"2023-12-19T07:47:22.807216+0000","last_fullsized":"2023-12-19T07:47:22.807216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306291+0000","last_change":"2023-12-19T07:47:21.249790+0000","last_active":"2023-12-19T07:47:22.306291+0000","last_peered":"2023-12-19T07:47:22.306291+0000","last_clean":"2023-12-19T07:47:22.306291+0000","last_became_active":"2023-12-19T07:47:21.249266+0000","last_became_peered":"2023-12-19T07:47:21.249266+0000","last_unstale":"2023-12-19T07:47:22.306291+0000","last_undegraded":"2023-12-19T07:47:22.306291+0000","last_fullsized":"2023-12-19T07:47:22.306291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358655+0000","last_change":"2023-12-19T07:47:17.206257+0000","last_active":"2023-12-19T07:47:22.358655+0000","last_peered":"2023-12-19T07:47:22.358655+0000","last_clean":"2023-12-19T07:47:22.358655+0000","last_became_active":"2023-12-19T07:47:17.205737+0000","last_became_peered":"2023-12-19T07:47:17.205737+0000","last_unstale":"2023-12-19T07:47:22.358655+0000","last_undegraded":"2023-12-19T07:47:22.358655+0000","last_fullsized":"2023-12-19T07:47:22.358655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867658+0000","last_change":"2023-12-19T07:47:16.214499+0000","last_active":"2023-12-19T07:47:22.867658+0000","last_peered":"2023-12-19T07:47:22.867658+0000","last_clean":"2023-12-19T07:47:22.867658+0000","last_became_active":"2023-12-19T07:47:16.213693+0000","last_became_peered":"2023-12-19T07:47:16.213693+0000","last_unstale":"2023-12-19T07:47:22.867658+0000","last_undegraded":"2023-12-19T07:47:22.867658+0000","last_fullsized":"2023-12-19T07:47:22.867658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307061+0000","last_change":"2023-12-19T07:47:21.254224+0000","last_active":"2023-12-19T07:47:22.307061+0000","last_peered":"2023-12-19T07:47:22.307061+0000","last_clean":"2023-12-19T07:47:22.307061+0000","last_became_active":"2023-12-19T07:47:21.253786+0000","last_became_peered":"2023-12-19T07:47:21.253786+0000","last_unstale":"2023-12-19T07:47:22.307061+0000","last_undegraded":"2023-12-19T07:47:22.307061+0000","last_fullsized":"2023-12-19T07:47:22.307061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:18:10.044237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598837+0000","last_change":"2023-12-19T07:47:17.205670+0000","last_active":"2023-12-19T07:47:22.598837+0000","last_peered":"2023-12-19T07:47:22.598837+0000","last_clean":"2023-12-19T07:47:22.598837+0000","last_became_active":"2023-12-19T07:47:17.201174+0000","last_became_peered":"2023-12-19T07:47:17.201174+0000","last_unstale":"2023-12-19T07:47:22.598837+0000","last_undegraded":"2023-12-19T07:47:22.598837+0000","last_fullsized":"2023-12-19T07:47:22.598837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352125+0000","last_change":"2023-12-19T07:47:16.196003+0000","last_active":"2023-12-19T07:47:22.352125+0000","last_peered":"2023-12-19T07:47:22.352125+0000","last_clean":"2023-12-19T07:47:22.352125+0000","last_became_active":"2023-12-19T07:47:16.195717+0000","last_became_peered":"2023-12-19T07:47:16.195717+0000","last_unstale":"2023-12-19T07:47:22.352125+0000","last_undegraded":"2023-12-19T07:47:22.352125+0000","last_fullsized":"2023-12-19T07:47:22.352125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306644+0000","last_change":"2023-12-19T07:47:21.259995+0000","last_active":"2023-12-19T07:47:22.306644+0000","last_peered":"2023-12-19T07:47:22.306644+0000","last_clean":"2023-12-19T07:47:22.306644+0000","last_became_active":"2023-12-19T07:47:21.259318+0000","last_became_peered":"2023-12-19T07:47:21.259318+0000","last_unstale":"2023-12-19T07:47:22.306644+0000","last_undegraded":"2023-12-19T07:47:22.306644+0000","last_fullsized":"2023-12-19T07:47:22.306644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598330+0000","last_change":"2023-12-19T07:47:17.205504+0000","last_active":"2023-12-19T07:47:22.598330+0000","last_peered":"2023-12-19T07:47:22.598330+0000","last_clean":"2023-12-19T07:47:22.598330+0000","last_became_active":"2023-12-19T07:47:17.200074+0000","last_became_peered":"2023-12-19T07:47:17.200074+0000","last_unstale":"2023-12-19T07:47:22.598330+0000","last_undegraded":"2023-12-19T07:47:22.598330+0000","last_fullsized":"2023-12-19T07:47:22.598330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351472+0000","last_change":"2023-12-19T07:47:16.206620+0000","last_active":"2023-12-19T07:47:22.351472+0000","last_peered":"2023-12-19T07:47:22.351472+0000","last_clean":"2023-12-19T07:47:22.351472+0000","last_became_active":"2023-12-19T07:47:16.206288+0000","last_became_peered":"2023-12-19T07:47:16.206288+0000","last_unstale":"2023-12-19T07:47:22.351472+0000","last_undegraded":"2023-12-19T07:47:22.351472+0000","last_fullsized":"2023-12-19T07:47:22.351472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351144+0000","last_change":"2023-12-19T07:47:21.841279+0000","last_active":"2023-12-19T07:47:22.351144+0000","last_peered":"2023-12-19T07:47:22.351144+0000","last_clean":"2023-12-19T07:47:22.351144+0000","last_became_active":"2023-12-19T07:47:21.840184+0000","last_became_peered":"2023-12-19T07:47:21.840184+0000","last_unstale":"2023-12-19T07:47:22.351144+0000","last_undegraded":"2023-12-19T07:47:22.351144+0000","last_fullsized":"2023-12-19T07:47:22.351144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:22:46.085295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306581+0000","last_change":"2023-12-19T07:47:17.199136+0000","last_active":"2023-12-19T07:47:22.306581+0000","last_peered":"2023-12-19T07:47:22.306581+0000","last_clean":"2023-12-19T07:47:22.306581+0000","last_became_active":"2023-12-19T07:47:17.198802+0000","last_became_peered":"2023-12-19T07:47:17.198802+0000","last_unstale":"2023-12-19T07:47:22.306581+0000","last_undegraded":"2023-12-19T07:47:22.306581+0000","last_fullsized":"2023-12-19T07:47:22.306581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:35.655311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866892+0000","last_change":"2023-12-19T07:47:16.213635+0000","last_active":"2023-12-19T07:47:22.866892+0000","last_peered":"2023-12-19T07:47:22.866892+0000","last_clean":"2023-12-19T07:47:22.866892+0000","last_became_active":"2023-12-19T07:47:16.213201+0000","last_became_peered":"2023-12-19T07:47:16.213201+0000","last_unstale":"2023-12-19T07:47:22.866892+0000","last_undegraded":"2023-12-19T07:47:22.866892+0000","last_fullsized":"2023-12-19T07:47:22.866892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306787+0000","last_change":"2023-12-19T07:47:21.265928+0000","last_active":"2023-12-19T07:47:22.306787+0000","last_peered":"2023-12-19T07:47:22.306787+0000","last_clean":"2023-12-19T07:47:22.306787+0000","last_became_active":"2023-12-19T07:47:21.265360+0000","last_became_peered":"2023-12-19T07:47:21.265360+0000","last_unstale":"2023-12-19T07:47:22.306787+0000","last_undegraded":"2023-12-19T07:47:22.306787+0000","last_fullsized":"2023-12-19T07:47:22.306787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904688+0000","last_change":"2023-12-19T07:47:17.195176+0000","last_active":"2023-12-19T07:47:22.904688+0000","last_peered":"2023-12-19T07:47:22.904688+0000","last_clean":"2023-12-19T07:47:22.904688+0000","last_became_active":"2023-12-19T07:47:17.194607+0000","last_became_peered":"2023-12-19T07:47:17.194607+0000","last_unstale":"2023-12-19T07:47:22.904688+0000","last_undegraded":"2023-12-19T07:47:22.904688+0000","last_fullsized":"2023-12-19T07:47:22.904688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359926+0000","last_change":"2023-12-19T07:47:16.206513+0000","last_active":"2023-12-19T07:47:22.359926+0000","last_peered":"2023-12-19T07:47:22.359926+0000","last_clean":"2023-12-19T07:47:22.359926+0000","last_became_active":"2023-12-19T07:47:16.206148+0000","last_became_peered":"2023-12-19T07:47:16.206148+0000","last_unstale":"2023-12-19T07:47:22.359926+0000","last_undegraded":"2023-12-19T07:47:22.359926+0000","last_fullsized":"2023-12-19T07:47:22.359926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.305902+0000","last_change":"2023-12-19T07:47:21.841183+0000","last_active":"2023-12-19T07:47:22.305902+0000","last_peered":"2023-12-19T07:47:22.305902+0000","last_clean":"2023-12-19T07:47:22.305902+0000","last_became_active":"2023-12-19T07:47:21.839559+0000","last_became_peered":"2023-12-19T07:47:21.839559+0000","last_unstale":"2023-12-19T07:47:22.305902+0000","last_undegraded":"2023-12-19T07:47:22.305902+0000","last_fullsized":"2023-12-19T07:47:22.305902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:27.954678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599577+0000","last_change":"2023-12-19T07:47:17.208713+0000","last_active":"2023-12-19T07:47:22.599577+0000","last_peered":"2023-12-19T07:47:22.599577+0000","last_clean":"2023-12-19T07:47:22.599577+0000","last_became_active":"2023-12-19T07:47:17.202555+0000","last_became_peered":"2023-12-19T07:47:17.202555+0000","last_unstale":"2023-12-19T07:47:22.599577+0000","last_undegraded":"2023-12-19T07:47:22.599577+0000","last_fullsized":"2023-12-19T07:47:22.599577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.356659+0000","last_change":"2023-12-19T07:47:16.198423+0000","last_active":"2023-12-19T07:47:22.356659+0000","last_peered":"2023-12-19T07:47:22.356659+0000","last_clean":"2023-12-19T07:47:22.356659+0000","last_became_active":"2023-12-19T07:47:16.198088+0000","last_became_peered":"2023-12-19T07:47:16.198088+0000","last_unstale":"2023-12-19T07:47:22.356659+0000","last_undegraded":"2023-12-19T07:47:22.356659+0000","last_fullsized":"2023-12-19T07:47:22.356659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597749+0000","last_change":"2023-12-19T07:47:21.845056+0000","last_active":"2023-12-19T07:47:22.597749+0000","last_peered":"2023-12-19T07:47:22.597749+0000","last_clean":"2023-12-19T07:47:22.597749+0000","last_became_active":"2023-12-19T07:47:21.844217+0000","last_became_peered":"2023-12-19T07:47:21.844217+0000","last_unstale":"2023-12-19T07:47:22.597749+0000","last_undegraded":"2023-12-19T07:47:22.597749+0000","last_fullsized":"2023-12-19T07:47:22.597749+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:53:07.229783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352565+0000","last_change":"2023-12-19T07:47:17.200525+0000","last_active":"2023-12-19T07:47:22.352565+0000","last_peered":"2023-12-19T07:47:22.352565+0000","last_clean":"2023-12-19T07:47:22.352565+0000","last_became_active":"2023-12-19T07:47:17.199921+0000","last_became_peered":"2023-12-19T07:47:17.199921+0000","last_unstale":"2023-12-19T07:47:22.352565+0000","last_undegraded":"2023-12-19T07:47:22.352565+0000","last_fullsized":"2023-12-19T07:47:22.352565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868254+0000","last_change":"2023-12-19T07:47:16.213616+0000","last_active":"2023-12-19T07:47:22.868254+0000","last_peered":"2023-12-19T07:47:22.868254+0000","last_clean":"2023-12-19T07:47:22.868254+0000","last_became_active":"2023-12-19T07:47:16.213264+0000","last_became_peered":"2023-12-19T07:47:16.213264+0000","last_unstale":"2023-12-19T07:47:22.868254+0000","last_undegraded":"2023-12-19T07:47:22.868254+0000","last_fullsized":"2023-12-19T07:47:22.868254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902861+0000","last_change":"2023-12-19T07:47:21.841016+0000","last_active":"2023-12-19T07:47:22.902861+0000","last_peered":"2023-12-19T07:47:22.902861+0000","last_clean":"2023-12-19T07:47:22.902861+0000","last_became_active":"2023-12-19T07:47:21.840031+0000","last_became_peered":"2023-12-19T07:47:21.840031+0000","last_unstale":"2023-12-19T07:47:22.902861+0000","last_undegraded":"2023-12-19T07:47:22.902861+0000","last_fullsized":"2023-12-19T07:47:22.902861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807240+0000","last_change":"2023-12-19T07:47:17.205604+0000","last_active":"2023-12-19T07:47:22.807240+0000","last_peered":"2023-12-19T07:47:22.807240+0000","last_clean":"2023-12-19T07:47:22.807240+0000","last_became_active":"2023-12-19T07:47:17.205354+0000","last_became_peered":"2023-12-19T07:47:17.205354+0000","last_unstale":"2023-12-19T07:47:22.807240+0000","last_undegraded":"2023-12-19T07:47:22.807240+0000","last_fullsized":"2023-12-19T07:47:22.807240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352620+0000","last_change":"2023-12-19T07:47:16.201653+0000","last_active":"2023-12-19T07:47:22.352620+0000","last_peered":"2023-12-19T07:47:22.352620+0000","last_clean":"2023-12-19T07:47:22.352620+0000","last_became_active":"2023-12-19T07:47:16.201071+0000","last_became_peered":"2023-12-19T07:47:16.201071+0000","last_unstale":"2023-12-19T07:47:22.352620+0000","last_undegraded":"2023-12-19T07:47:22.352620+0000","last_fullsized":"2023-12-19T07:47:22.352620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351616+0000","last_change":"2023-12-19T07:47:21.262464+0000","last_active":"2023-12-19T07:47:22.351616+0000","last_peered":"2023-12-19T07:47:22.351616+0000","last_clean":"2023-12-19T07:47:22.351616+0000","last_became_active":"2023-12-19T07:47:21.259084+0000","last_became_peered":"2023-12-19T07:47:21.259084+0000","last_unstale":"2023-12-19T07:47:22.351616+0000","last_undegraded":"2023-12-19T07:47:22.351616+0000","last_fullsized":"2023-12-19T07:47:22.351616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307178+0000","last_change":"2023-12-19T07:47:17.212640+0000","last_active":"2023-12-19T07:47:22.307178+0000","last_peered":"2023-12-19T07:47:22.307178+0000","last_clean":"2023-12-19T07:47:22.307178+0000","last_became_active":"2023-12-19T07:47:17.212189+0000","last_became_peered":"2023-12-19T07:47:17.212189+0000","last_unstale":"2023-12-19T07:47:22.307178+0000","last_undegraded":"2023-12-19T07:47:22.307178+0000","last_fullsized":"2023-12-19T07:47:22.307178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806950+0000","last_change":"2023-12-19T07:47:16.201470+0000","last_active":"2023-12-19T07:47:22.806950+0000","last_peered":"2023-12-19T07:47:22.806950+0000","last_clean":"2023-12-19T07:47:22.806950+0000","last_became_active":"2023-12-19T07:47:16.200990+0000","last_became_peered":"2023-12-19T07:47:16.200990+0000","last_unstale":"2023-12-19T07:47:22.806950+0000","last_undegraded":"2023-12-19T07:47:22.806950+0000","last_fullsized":"2023-12-19T07:47:22.806950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352494+0000","last_change":"2023-12-19T07:47:21.261403+0000","last_active":"2023-12-19T07:47:22.352494+0000","last_peered":"2023-12-19T07:47:22.352494+0000","last_clean":"2023-12-19T07:47:22.352494+0000","last_became_active":"2023-12-19T07:47:21.260934+0000","last_became_peered":"2023-12-19T07:47:21.260934+0000","last_unstale":"2023-12-19T07:47:22.352494+0000","last_undegraded":"2023-12-19T07:47:22.352494+0000","last_fullsized":"2023-12-19T07:47:22.352494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351362+0000","last_change":"2023-12-19T07:47:17.196604+0000","last_active":"2023-12-19T07:47:22.351362+0000","last_peered":"2023-12-19T07:47:22.351362+0000","last_clean":"2023-12-19T07:47:22.351362+0000","last_became_active":"2023-12-19T07:47:17.195716+0000","last_became_peered":"2023-12-19T07:47:17.195716+0000","last_unstale":"2023-12-19T07:47:22.351362+0000","last_undegraded":"2023-12-19T07:47:22.351362+0000","last_fullsized":"2023-12-19T07:47:22.351362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302998+0000","last_change":"2023-12-19T07:47:16.188258+0000","last_active":"2023-12-19T07:47:22.302998+0000","last_peered":"2023-12-19T07:47:22.302998+0000","last_clean":"2023-12-19T07:47:22.302998+0000","last_became_active":"2023-12-19T07:47:16.187018+0000","last_became_peered":"2023-12-19T07:47:16.187018+0000","last_unstale":"2023-12-19T07:47:22.302998+0000","last_undegraded":"2023-12-19T07:47:22.302998+0000","last_fullsized":"2023-12-19T07:47:22.302998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353576+0000","last_change":"2023-12-19T07:47:21.269038+0000","last_active":"2023-12-19T07:47:22.353576+0000","last_peered":"2023-12-19T07:47:22.353576+0000","last_clean":"2023-12-19T07:47:22.353576+0000","last_became_active":"2023-12-19T07:47:21.268433+0000","last_became_peered":"2023-12-19T07:47:21.268433+0000","last_unstale":"2023-12-19T07:47:22.353576+0000","last_undegraded":"2023-12-19T07:47:22.353576+0000","last_fullsized":"2023-12-19T07:47:22.353576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903117+0000","last_change":"2023-12-19T07:47:17.201741+0000","last_active":"2023-12-19T07:47:22.903117+0000","last_peered":"2023-12-19T07:47:22.903117+0000","last_clean":"2023-12-19T07:47:22.903117+0000","last_became_active":"2023-12-19T07:47:17.201369+0000","last_became_peered":"2023-12-19T07:47:17.201369+0000","last_unstale":"2023-12-19T07:47:22.903117+0000","last_undegraded":"2023-12-19T07:47:22.903117+0000","last_fullsized":"2023-12-19T07:47:22.903117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866966+0000","last_change":"2023-12-19T07:47:16.213829+0000","last_active":"2023-12-19T07:47:22.866966+0000","last_peered":"2023-12-19T07:47:22.866966+0000","last_clean":"2023-12-19T07:47:22.866966+0000","last_became_active":"2023-12-19T07:47:16.213228+0000","last_became_peered":"2023-12-19T07:47:16.213228+0000","last_unstale":"2023-12-19T07:47:22.866966+0000","last_undegraded":"2023-12-19T07:47:22.866966+0000","last_fullsized":"2023-12-19T07:47:22.866966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902852+0000","last_change":"2023-12-19T07:47:21.841516+0000","last_active":"2023-12-19T07:47:22.902852+0000","last_peered":"2023-12-19T07:47:22.902852+0000","last_clean":"2023-12-19T07:47:22.902852+0000","last_became_active":"2023-12-19T07:47:21.840477+0000","last_became_peered":"2023-12-19T07:47:21.840477+0000","last_unstale":"2023-12-19T07:47:22.902852+0000","last_undegraded":"2023-12-19T07:47:22.902852+0000","last_fullsized":"2023-12-19T07:47:22.902852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598774+0000","last_change":"2023-12-19T07:47:17.200992+0000","last_active":"2023-12-19T07:47:22.598774+0000","last_peered":"2023-12-19T07:47:22.598774+0000","last_clean":"2023-12-19T07:47:22.598774+0000","last_became_active":"2023-12-19T07:47:17.199427+0000","last_became_peered":"2023-12-19T07:47:17.199427+0000","last_unstale":"2023-12-19T07:47:22.598774+0000","last_undegraded":"2023-12-19T07:47:22.598774+0000","last_fullsized":"2023-12-19T07:47:22.598774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806565+0000","last_change":"2023-12-19T07:47:16.201062+0000","last_active":"2023-12-19T07:47:22.806565+0000","last_peered":"2023-12-19T07:47:22.806565+0000","last_clean":"2023-12-19T07:47:22.806565+0000","last_became_active":"2023-12-19T07:47:16.199674+0000","last_became_peered":"2023-12-19T07:47:16.199674+0000","last_unstale":"2023-12-19T07:47:22.806565+0000","last_undegraded":"2023-12-19T07:47:22.806565+0000","last_fullsized":"2023-12-19T07:47:22.806565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:33.372367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867080+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:47:22.867080+0000","last_peered":"2023-12-19T07:47:22.867080+0000","last_clean":"2023-12-19T07:47:22.867080+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:47:22.867080+0000","last_undegraded":"2023-12-19T07:47:22.867080+0000","last_fullsized":"2023-12-19T07:47:22.867080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359462+0000","last_change":"2023-12-19T07:47:21.844050+0000","last_active":"2023-12-19T07:47:22.359462+0000","last_peered":"2023-12-19T07:47:22.359462+0000","last_clean":"2023-12-19T07:47:22.359462+0000","last_became_active":"2023-12-19T07:47:21.843480+0000","last_became_peered":"2023-12-19T07:47:21.843480+0000","last_unstale":"2023-12-19T07:47:22.359462+0000","last_undegraded":"2023-12-19T07:47:22.359462+0000","last_fullsized":"2023-12-19T07:47:22.359462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:29.237260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867932+0000","last_change":"2023-12-19T07:47:17.201683+0000","last_active":"2023-12-19T07:47:22.867932+0000","last_peered":"2023-12-19T07:47:22.867932+0000","last_clean":"2023-12-19T07:47:22.867932+0000","last_became_active":"2023-12-19T07:47:17.201138+0000","last_became_peered":"2023-12-19T07:47:17.201138+0000","last_unstale":"2023-12-19T07:47:22.867932+0000","last_undegraded":"2023-12-19T07:47:22.867932+0000","last_fullsized":"2023-12-19T07:47:22.867932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:25.176702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303850+0000","last_change":"2023-12-19T07:47:16.199670+0000","last_active":"2023-12-19T07:47:22.303850+0000","last_peered":"2023-12-19T07:47:22.303850+0000","last_clean":"2023-12-19T07:47:22.303850+0000","last_became_active":"2023-12-19T07:47:16.199236+0000","last_became_peered":"2023-12-19T07:47:16.199236+0000","last_unstale":"2023-12-19T07:47:22.303850+0000","last_undegraded":"2023-12-19T07:47:22.303850+0000","last_fullsized":"2023-12-19T07:47:22.303850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358653+0000","last_change":"2023-12-19T07:47:21.844306+0000","last_active":"2023-12-19T07:47:22.358653+0000","last_peered":"2023-12-19T07:47:22.358653+0000","last_clean":"2023-12-19T07:47:22.358653+0000","last_became_active":"2023-12-19T07:47:21.843694+0000","last_became_peered":"2023-12-19T07:47:21.843694+0000","last_unstale":"2023-12-19T07:47:22.358653+0000","last_undegraded":"2023-12-19T07:47:22.358653+0000","last_fullsized":"2023-12-19T07:47:22.358653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:25.534670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806965+0000","last_change":"2023-12-19T07:47:17.210076+0000","last_active":"2023-12-19T07:47:22.806965+0000","last_peered":"2023-12-19T07:47:22.806965+0000","last_clean":"2023-12-19T07:47:22.806965+0000","last_became_active":"2023-12-19T07:47:17.209871+0000","last_became_peered":"2023-12-19T07:47:17.209871+0000","last_unstale":"2023-12-19T07:47:22.806965+0000","last_undegraded":"2023-12-19T07:47:22.806965+0000","last_fullsized":"2023-12-19T07:47:22.806965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353351+0000","last_change":"2023-12-19T07:47:16.187382+0000","last_active":"2023-12-19T07:47:22.353351+0000","last_peered":"2023-12-19T07:47:22.353351+0000","last_clean":"2023-12-19T07:47:22.353351+0000","last_became_active":"2023-12-19T07:47:16.187021+0000","last_became_peered":"2023-12-19T07:47:16.187021+0000","last_unstale":"2023-12-19T07:47:22.353351+0000","last_undegraded":"2023-12-19T07:47:22.353351+0000","last_fullsized":"2023-12-19T07:47:22.353351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:03.039957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903413+0000","last_change":"2023-12-19T07:47:21.265329+0000","last_active":"2023-12-19T07:47:22.903413+0000","last_peered":"2023-12-19T07:47:22.903413+0000","last_clean":"2023-12-19T07:47:22.903413+0000","last_became_active":"2023-12-19T07:47:21.264912+0000","last_became_peered":"2023-12-19T07:47:21.264912+0000","last_unstale":"2023-12-19T07:47:22.903413+0000","last_undegraded":"2023-12-19T07:47:22.903413+0000","last_fullsized":"2023-12-19T07:47:22.903413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868427+0000","last_change":"2023-12-19T07:47:17.215646+0000","last_active":"2023-12-19T07:47:22.868427+0000","last_peered":"2023-12-19T07:47:22.868427+0000","last_clean":"2023-12-19T07:47:22.868427+0000","last_became_active":"2023-12-19T07:47:17.215345+0000","last_became_peered":"2023-12-19T07:47:17.215345+0000","last_unstale":"2023-12-19T07:47:22.868427+0000","last_undegraded":"2023-12-19T07:47:22.868427+0000","last_fullsized":"2023-12-19T07:47:22.868427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:05.660865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304403+0000","last_change":"2023-12-19T07:47:16.195752+0000","last_active":"2023-12-19T07:47:22.304403+0000","last_peered":"2023-12-19T07:47:22.304403+0000","last_clean":"2023-12-19T07:47:22.304403+0000","last_became_active":"2023-12-19T07:47:16.195415+0000","last_became_peered":"2023-12-19T07:47:16.195415+0000","last_unstale":"2023-12-19T07:47:22.304403+0000","last_undegraded":"2023-12-19T07:47:22.304403+0000","last_fullsized":"2023-12-19T07:47:22.304403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805409+0000","last_change":"2023-12-19T07:47:21.840195+0000","last_active":"2023-12-19T07:47:22.805409+0000","last_peered":"2023-12-19T07:47:22.805409+0000","last_clean":"2023-12-19T07:47:22.805409+0000","last_became_active":"2023-12-19T07:47:21.839557+0000","last_became_peered":"2023-12-19T07:47:21.839557+0000","last_unstale":"2023-12-19T07:47:22.805409+0000","last_undegraded":"2023-12-19T07:47:22.805409+0000","last_fullsized":"2023-12-19T07:47:22.805409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867970+0000","last_change":"2023-12-19T07:47:17.211241+0000","last_active":"2023-12-19T07:47:22.867970+0000","last_peered":"2023-12-19T07:47:22.867970+0000","last_clean":"2023-12-19T07:47:22.867970+0000","last_became_active":"2023-12-19T07:47:17.210508+0000","last_became_peered":"2023-12-19T07:47:17.210508+0000","last_unstale":"2023-12-19T07:47:22.867970+0000","last_undegraded":"2023-12-19T07:47:22.867970+0000","last_fullsized":"2023-12-19T07:47:22.867970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:48.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307381+0000","last_change":"2023-12-19T07:47:16.198483+0000","last_active":"2023-12-19T07:47:22.307381+0000","last_peered":"2023-12-19T07:47:22.307381+0000","last_clean":"2023-12-19T07:47:22.307381+0000","last_became_active":"2023-12-19T07:47:16.198150+0000","last_became_peered":"2023-12-19T07:47:16.198150+0000","last_unstale":"2023-12-19T07:47:22.307381+0000","last_undegraded":"2023-12-19T07:47:22.307381+0000","last_fullsized":"2023-12-19T07:47:22.307381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306109+0000","last_change":"2023-12-19T07:47:21.259856+0000","last_active":"2023-12-19T07:47:22.306109+0000","last_peered":"2023-12-19T07:47:22.306109+0000","last_clean":"2023-12-19T07:47:22.306109+0000","last_became_active":"2023-12-19T07:47:21.259147+0000","last_became_peered":"2023-12-19T07:47:21.259147+0000","last_unstale":"2023-12-19T07:47:22.306109+0000","last_undegraded":"2023-12-19T07:47:22.306109+0000","last_fullsized":"2023-12-19T07:47:22.306109+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867790+0000","last_change":"2023-12-19T07:47:17.209489+0000","last_active":"2023-12-19T07:47:22.867790+0000","last_peered":"2023-12-19T07:47:22.867790+0000","last_clean":"2023-12-19T07:47:22.867790+0000","last_became_active":"2023-12-19T07:47:17.209131+0000","last_became_peered":"2023-12-19T07:47:17.209131+0000","last_unstale":"2023-12-19T07:47:22.867790+0000","last_undegraded":"2023-12-19T07:47:22.867790+0000","last_fullsized":"2023-12-19T07:47:22.867790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303258+0000","last_change":"2023-12-19T07:47:16.192274+0000","last_active":"2023-12-19T07:47:22.303258+0000","last_peered":"2023-12-19T07:47:22.303258+0000","last_clean":"2023-12-19T07:47:22.303258+0000","last_became_active":"2023-12-19T07:47:16.190777+0000","last_became_peered":"2023-12-19T07:47:16.190777+0000","last_unstale":"2023-12-19T07:47:22.303258+0000","last_undegraded":"2023-12-19T07:47:22.303258+0000","last_fullsized":"2023-12-19T07:47:22.303258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904035+0000","last_change":"2023-12-19T07:47:21.260731+0000","last_active":"2023-12-19T07:47:22.904035+0000","last_peered":"2023-12-19T07:47:22.904035+0000","last_clean":"2023-12-19T07:47:22.904035+0000","last_became_active":"2023-12-19T07:47:21.258011+0000","last_became_peered":"2023-12-19T07:47:21.258011+0000","last_unstale":"2023-12-19T07:47:22.904035+0000","last_undegraded":"2023-12-19T07:47:22.904035+0000","last_fullsized":"2023-12-19T07:47:22.904035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359125+0000","last_change":"2023-12-19T07:47:17.201434+0000","last_active":"2023-12-19T07:47:22.359125+0000","last_peered":"2023-12-19T07:47:22.359125+0000","last_clean":"2023-12-19T07:47:22.359125+0000","last_became_active":"2023-12-19T07:47:17.200473+0000","last_became_peered":"2023-12-19T07:47:17.200473+0000","last_unstale":"2023-12-19T07:47:22.359125+0000","last_undegraded":"2023-12-19T07:47:22.359125+0000","last_fullsized":"2023-12-19T07:47:22.359125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359065+0000","last_change":"2023-12-19T07:47:16.193445+0000","last_active":"2023-12-19T07:47:22.359065+0000","last_peered":"2023-12-19T07:47:22.359065+0000","last_clean":"2023-12-19T07:47:22.359065+0000","last_became_active":"2023-12-19T07:47:16.192600+0000","last_became_peered":"2023-12-19T07:47:16.192600+0000","last_unstale":"2023-12-19T07:47:22.359065+0000","last_undegraded":"2023-12-19T07:47:22.359065+0000","last_fullsized":"2023-12-19T07:47:22.359065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597903+0000","last_change":"2023-12-19T07:47:21.847662+0000","last_active":"2023-12-19T07:47:22.597903+0000","last_peered":"2023-12-19T07:47:22.597903+0000","last_clean":"2023-12-19T07:47:22.597903+0000","last_became_active":"2023-12-19T07:47:21.847110+0000","last_became_peered":"2023-12-19T07:47:21.847110+0000","last_unstale":"2023-12-19T07:47:22.597903+0000","last_undegraded":"2023-12-19T07:47:22.597903+0000","last_fullsized":"2023-12-19T07:47:22.597903+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:07:23.838326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867579+0000","last_change":"2023-12-19T07:47:17.219151+0000","last_active":"2023-12-19T07:47:22.867579+0000","last_peered":"2023-12-19T07:47:22.867579+0000","last_clean":"2023-12-19T07:47:22.867579+0000","last_became_active":"2023-12-19T07:47:17.218940+0000","last_became_peered":"2023-12-19T07:47:17.218940+0000","last_unstale":"2023-12-19T07:47:22.867579+0000","last_undegraded":"2023-12-19T07:47:22.867579+0000","last_fullsized":"2023-12-19T07:47:22.867579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806721+0000","last_change":"2023-12-19T07:47:16.201106+0000","last_active":"2023-12-19T07:47:22.806721+0000","last_peered":"2023-12-19T07:47:22.806721+0000","last_clean":"2023-12-19T07:47:22.806721+0000","last_became_active":"2023-12-19T07:47:16.200628+0000","last_became_peered":"2023-12-19T07:47:16.200628+0000","last_unstale":"2023-12-19T07:47:22.806721+0000","last_undegraded":"2023-12-19T07:47:22.806721+0000","last_fullsized":"2023-12-19T07:47:22.806721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597986+0000","last_change":"2023-12-19T07:47:21.847256+0000","last_active":"2023-12-19T07:47:22.597986+0000","last_peered":"2023-12-19T07:47:22.597986+0000","last_clean":"2023-12-19T07:47:22.597986+0000","last_became_active":"2023-12-19T07:47:21.843240+0000","last_became_peered":"2023-12-19T07:47:21.843240+0000","last_unstale":"2023-12-19T07:47:22.597986+0000","last_undegraded":"2023-12-19T07:47:22.597986+0000","last_fullsized":"2023-12-19T07:47:22.597986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:45.450328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903865+0000","last_change":"2023-12-19T07:47:17.200704+0000","last_active":"2023-12-19T07:47:22.903865+0000","last_peered":"2023-12-19T07:47:22.903865+0000","last_clean":"2023-12-19T07:47:22.903865+0000","last_became_active":"2023-12-19T07:47:17.200382+0000","last_became_peered":"2023-12-19T07:47:17.200382+0000","last_unstale":"2023-12-19T07:47:22.903865+0000","last_undegraded":"2023-12-19T07:47:22.903865+0000","last_fullsized":"2023-12-19T07:47:22.903865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903656+0000","last_change":"2023-12-19T07:47:16.189068+0000","last_active":"2023-12-19T07:47:22.903656+0000","last_peered":"2023-12-19T07:47:22.903656+0000","last_clean":"2023-12-19T07:47:22.903656+0000","last_became_active":"2023-12-19T07:47:16.188798+0000","last_became_peered":"2023-12-19T07:47:16.188798+0000","last_unstale":"2023-12-19T07:47:22.903656+0000","last_undegraded":"2023-12-19T07:47:22.903656+0000","last_fullsized":"2023-12-19T07:47:22.903656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805530+0000","last_change":"2023-12-19T07:47:21.847444+0000","last_active":"2023-12-19T07:47:22.805530+0000","last_peered":"2023-12-19T07:47:22.805530+0000","last_clean":"2023-12-19T07:47:22.805530+0000","last_became_active":"2023-12-19T07:47:21.847123+0000","last_became_peered":"2023-12-19T07:47:21.847123+0000","last_unstale":"2023-12-19T07:47:22.805530+0000","last_undegraded":"2023-12-19T07:47:22.805530+0000","last_fullsized":"2023-12-19T07:47:22.805530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867681+0000","last_change":"2023-12-19T07:47:17.212222+0000","last_active":"2023-12-19T07:47:22.867681+0000","last_peered":"2023-12-19T07:47:22.867681+0000","last_clean":"2023-12-19T07:47:22.867681+0000","last_became_active":"2023-12-19T07:47:17.211536+0000","last_became_peered":"2023-12-19T07:47:17.211536+0000","last_unstale":"2023-12-19T07:47:22.867681+0000","last_undegraded":"2023-12-19T07:47:22.867681+0000","last_fullsized":"2023-12-19T07:47:22.867681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:40.976961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-19T07:47:29.088364+0000","last_change":"2023-12-19T07:47:16.200498+0000","last_active":"2023-12-19T07:47:29.088364+0000","last_peered":"2023-12-19T07:47:29.088364+0000","last_clean":"2023-12-19T07:47:29.088364+0000","last_became_active":"2023-12-19T07:47:16.200159+0000","last_became_peered":"2023-12-19T07:47:16.200159+0000","last_unstale":"2023-12-19T07:47:29.088364+0000","last_undegraded":"2023-12-19T07:47:29.088364+0000","last_fullsized":"2023-12-19T07:47:29.088364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351798+0000","last_change":"2023-12-19T07:47:21.269837+0000","last_active":"2023-12-19T07:47:22.351798+0000","last_peered":"2023-12-19T07:47:22.351798+0000","last_clean":"2023-12-19T07:47:22.351798+0000","last_became_active":"2023-12-19T07:47:21.269371+0000","last_became_peered":"2023-12-19T07:47:21.269371+0000","last_unstale":"2023-12-19T07:47:22.351798+0000","last_undegraded":"2023-12-19T07:47:22.351798+0000","last_fullsized":"2023-12-19T07:47:22.351798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307009+0000","last_change":"2023-12-19T07:47:17.206123+0000","last_active":"2023-12-19T07:47:22.307009+0000","last_peered":"2023-12-19T07:47:22.307009+0000","last_clean":"2023-12-19T07:47:22.307009+0000","last_became_active":"2023-12-19T07:47:17.202604+0000","last_became_peered":"2023-12-19T07:47:17.202604+0000","last_unstale":"2023-12-19T07:47:22.307009+0000","last_undegraded":"2023-12-19T07:47:22.307009+0000","last_fullsized":"2023-12-19T07:47:22.307009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353492+0000","last_change":"2023-12-19T07:47:16.207087+0000","last_active":"2023-12-19T07:47:22.353492+0000","last_peered":"2023-12-19T07:47:22.353492+0000","last_clean":"2023-12-19T07:47:22.353492+0000","last_became_active":"2023-12-19T07:47:16.205832+0000","last_became_peered":"2023-12-19T07:47:16.205832+0000","last_unstale":"2023-12-19T07:47:22.353492+0000","last_undegraded":"2023-12-19T07:47:22.353492+0000","last_fullsized":"2023-12-19T07:47:22.353492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307124+0000","last_change":"2023-12-19T07:47:21.259755+0000","last_active":"2023-12-19T07:47:22.307124+0000","last_peered":"2023-12-19T07:47:22.307124+0000","last_clean":"2023-12-19T07:47:22.307124+0000","last_became_active":"2023-12-19T07:47:21.259163+0000","last_became_peered":"2023-12-19T07:47:21.259163+0000","last_unstale":"2023-12-19T07:47:22.307124+0000","last_undegraded":"2023-12-19T07:47:22.307124+0000","last_fullsized":"2023-12-19T07:47:22.307124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904807+0000","last_change":"2023-12-19T07:47:17.202501+0000","last_active":"2023-12-19T07:47:22.904807+0000","last_peered":"2023-12-19T07:47:22.904807+0000","last_clean":"2023-12-19T07:47:22.904807+0000","last_became_active":"2023-12-19T07:47:17.199344+0000","last_became_peered":"2023-12-19T07:47:17.199344+0000","last_unstale":"2023-12-19T07:47:22.904807+0000","last_undegraded":"2023-12-19T07:47:22.904807+0000","last_fullsized":"2023-12-19T07:47:22.904807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807353+0000","last_change":"2023-12-19T07:47:16.200080+0000","last_active":"2023-12-19T07:47:22.807353+0000","last_peered":"2023-12-19T07:47:22.807353+0000","last_clean":"2023-12-19T07:47:22.807353+0000","last_became_active":"2023-12-19T07:47:16.199182+0000","last_became_peered":"2023-12-19T07:47:16.199182+0000","last_unstale":"2023-12-19T07:47:22.807353+0000","last_undegraded":"2023-12-19T07:47:22.807353+0000","last_fullsized":"2023-12-19T07:47:22.807353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598231+0000","last_change":"2023-12-19T07:47:21.264957+0000","last_active":"2023-12-19T07:47:22.598231+0000","last_peered":"2023-12-19T07:47:22.598231+0000","last_clean":"2023-12-19T07:47:22.598231+0000","last_became_active":"2023-12-19T07:47:21.264514+0000","last_became_peered":"2023-12-19T07:47:21.264514+0000","last_unstale":"2023-12-19T07:47:22.598231+0000","last_undegraded":"2023-12-19T07:47:22.598231+0000","last_fullsized":"2023-12-19T07:47:22.598231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:20.377759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807125+0000","last_change":"2023-12-19T07:47:17.199047+0000","last_active":"2023-12-19T07:47:22.807125+0000","last_peered":"2023-12-19T07:47:22.807125+0000","last_clean":"2023-12-19T07:47:22.807125+0000","last_became_active":"2023-12-19T07:47:17.198760+0000","last_became_peered":"2023-12-19T07:47:17.198760+0000","last_unstale":"2023-12-19T07:47:22.807125+0000","last_undegraded":"2023-12-19T07:47:22.807125+0000","last_fullsized":"2023-12-19T07:47:22.807125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359707+0000","last_change":"2023-12-19T07:47:16.206922+0000","last_active":"2023-12-19T07:47:22.359707+0000","last_peered":"2023-12-19T07:47:22.359707+0000","last_clean":"2023-12-19T07:47:22.359707+0000","last_became_active":"2023-12-19T07:47:16.206605+0000","last_became_peered":"2023-12-19T07:47:16.206605+0000","last_unstale":"2023-12-19T07:47:22.359707+0000","last_undegraded":"2023-12-19T07:47:22.359707+0000","last_fullsized":"2023-12-19T07:47:22.359707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597807+0000","last_change":"2023-12-19T07:47:21.845593+0000","last_active":"2023-12-19T07:47:22.597807+0000","last_peered":"2023-12-19T07:47:22.597807+0000","last_clean":"2023-12-19T07:47:22.597807+0000","last_became_active":"2023-12-19T07:47:21.843663+0000","last_became_peered":"2023-12-19T07:47:21.843663+0000","last_unstale":"2023-12-19T07:47:22.597807+0000","last_undegraded":"2023-12-19T07:47:22.597807+0000","last_fullsized":"2023-12-19T07:47:22.597807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902979+0000","last_change":"2023-12-19T07:47:21.846558+0000","last_active":"2023-12-19T07:47:22.902979+0000","last_peered":"2023-12-19T07:47:22.902979+0000","last_clean":"2023-12-19T07:47:22.902979+0000","last_became_active":"2023-12-19T07:47:21.845861+0000","last_became_peered":"2023-12-19T07:47:21.845861+0000","last_unstale":"2023-12-19T07:47:22.902979+0000","last_undegraded":"2023-12-19T07:47:22.902979+0000","last_fullsized":"2023-12-19T07:47:22.902979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:28.389344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351325+0000","last_change":"2023-12-19T07:47:17.210958+0000","last_active":"2023-12-19T07:47:22.351325+0000","last_peered":"2023-12-19T07:47:22.351325+0000","last_clean":"2023-12-19T07:47:22.351325+0000","last_became_active":"2023-12-19T07:47:17.210537+0000","last_became_peered":"2023-12-19T07:47:17.210537+0000","last_unstale":"2023-12-19T07:47:22.351325+0000","last_undegraded":"2023-12-19T07:47:22.351325+0000","last_fullsized":"2023-12-19T07:47:22.351325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805998+0000","last_change":"2023-12-19T07:47:16.198817+0000","last_active":"2023-12-19T07:47:22.805998+0000","last_peered":"2023-12-19T07:47:22.805998+0000","last_clean":"2023-12-19T07:47:22.805998+0000","last_became_active":"2023-12-19T07:47:16.198491+0000","last_became_peered":"2023-12-19T07:47:16.198491+0000","last_unstale":"2023-12-19T07:47:22.805998+0000","last_undegraded":"2023-12-19T07:47:22.805998+0000","last_fullsized":"2023-12-19T07:47:22.805998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359608+0000","last_change":"2023-12-19T07:47:21.263980+0000","last_active":"2023-12-19T07:47:22.359608+0000","last_peered":"2023-12-19T07:47:22.359608+0000","last_clean":"2023-12-19T07:47:22.359608+0000","last_became_active":"2023-12-19T07:47:21.263310+0000","last_became_peered":"2023-12-19T07:47:21.263310+0000","last_unstale":"2023-12-19T07:47:22.359608+0000","last_undegraded":"2023-12-19T07:47:22.359608+0000","last_fullsized":"2023-12-19T07:47:22.359608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.138125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598272+0000","last_change":"2023-12-19T07:47:17.196870+0000","last_active":"2023-12-19T07:47:22.598272+0000","last_peered":"2023-12-19T07:47:22.598272+0000","last_clean":"2023-12-19T07:47:22.598272+0000","last_became_active":"2023-12-19T07:47:17.196588+0000","last_became_peered":"2023-12-19T07:47:17.196588+0000","last_unstale":"2023-12-19T07:47:22.598272+0000","last_undegraded":"2023-12-19T07:47:22.598272+0000","last_fullsized":"2023-12-19T07:47:22.598272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598626+0000","last_change":"2023-12-19T07:47:16.198918+0000","last_active":"2023-12-19T07:47:22.598626+0000","last_peered":"2023-12-19T07:47:22.598626+0000","last_clean":"2023-12-19T07:47:22.598626+0000","last_became_active":"2023-12-19T07:47:16.198496+0000","last_became_peered":"2023-12-19T07:47:16.198496+0000","last_unstale":"2023-12-19T07:47:22.598626+0000","last_undegraded":"2023-12-19T07:47:22.598626+0000","last_fullsized":"2023-12-19T07:47:22.598626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358925+0000","last_change":"2023-12-19T07:47:16.192976+0000","last_active":"2023-12-19T07:47:22.358925+0000","last_peered":"2023-12-19T07:47:22.358925+0000","last_clean":"2023-12-19T07:47:22.358925+0000","last_became_active":"2023-12-19T07:47:16.192588+0000","last_became_peered":"2023-12-19T07:47:16.192588+0000","last_unstale":"2023-12-19T07:47:22.358925+0000","last_undegraded":"2023-12-19T07:47:22.358925+0000","last_fullsized":"2023-12-19T07:47:22.358925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903549+0000","last_change":"2023-12-19T07:47:17.205110+0000","last_active":"2023-12-19T07:47:22.903549+0000","last_peered":"2023-12-19T07:47:22.903549+0000","last_clean":"2023-12-19T07:47:22.903549+0000","last_became_active":"2023-12-19T07:47:17.204811+0000","last_became_peered":"2023-12-19T07:47:17.204811+0000","last_unstale":"2023-12-19T07:47:22.903549+0000","last_undegraded":"2023-12-19T07:47:22.903549+0000","last_fullsized":"2023-12-19T07:47:22.903549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:07.078408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303570+0000","last_change":"2023-12-19T07:47:21.259361+0000","last_active":"2023-12-19T07:47:22.303570+0000","last_peered":"2023-12-19T07:47:22.303570+0000","last_clean":"2023-12-19T07:47:22.303570+0000","last_became_active":"2023-12-19T07:47:21.258883+0000","last_became_peered":"2023-12-19T07:47:21.258883+0000","last_unstale":"2023-12-19T07:47:22.303570+0000","last_undegraded":"2023-12-19T07:47:22.303570+0000","last_fullsized":"2023-12-19T07:47:22.303570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:28.728929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598981+0000","last_change":"2023-12-19T07:47:16.206769+0000","last_active":"2023-12-19T07:47:22.598981+0000","last_peered":"2023-12-19T07:47:22.598981+0000","last_clean":"2023-12-19T07:47:22.598981+0000","last_became_active":"2023-12-19T07:47:16.206392+0000","last_became_peered":"2023-12-19T07:47:16.206392+0000","last_unstale":"2023-12-19T07:47:22.598981+0000","last_undegraded":"2023-12-19T07:47:22.598981+0000","last_fullsized":"2023-12-19T07:47:22.598981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306430+0000","last_change":"2023-12-19T07:47:17.204694+0000","last_active":"2023-12-19T07:47:22.306430+0000","last_peered":"2023-12-19T07:47:22.306430+0000","last_clean":"2023-12-19T07:47:22.306430+0000","last_became_active":"2023-12-19T07:47:17.204276+0000","last_became_peered":"2023-12-19T07:47:17.204276+0000","last_unstale":"2023-12-19T07:47:22.306430+0000","last_undegraded":"2023-12-19T07:47:22.306430+0000","last_fullsized":"2023-12-19T07:47:22.306430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599471+0000","last_change":"2023-12-19T07:47:21.258311+0000","last_active":"2023-12-19T07:47:22.599471+0000","last_peered":"2023-12-19T07:47:22.599471+0000","last_clean":"2023-12-19T07:47:22.599471+0000","last_became_active":"2023-12-19T07:47:21.257510+0000","last_became_peered":"2023-12-19T07:47:21.257510+0000","last_unstale":"2023-12-19T07:47:22.599471+0000","last_undegraded":"2023-12-19T07:47:22.599471+0000","last_fullsized":"2023-12-19T07:47:22.599471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:15:44.535034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358817+0000","last_change":"2023-12-19T07:47:17.204892+0000","last_active":"2023-12-19T07:47:22.358817+0000","last_peered":"2023-12-19T07:47:22.358817+0000","last_clean":"2023-12-19T07:47:22.358817+0000","last_became_active":"2023-12-19T07:47:17.201155+0000","last_became_peered":"2023-12-19T07:47:17.201155+0000","last_unstale":"2023-12-19T07:47:22.358817+0000","last_undegraded":"2023-12-19T07:47:22.358817+0000","last_fullsized":"2023-12-19T07:47:22.358817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302909+0000","last_change":"2023-12-19T07:47:17.204307+0000","last_active":"2023-12-19T07:47:22.302909+0000","last_peered":"2023-12-19T07:47:22.302909+0000","last_clean":"2023-12-19T07:47:22.302909+0000","last_became_active":"2023-12-19T07:47:17.203914+0000","last_became_peered":"2023-12-19T07:47:17.203914+0000","last_unstale":"2023-12-19T07:47:22.302909+0000","last_undegraded":"2023-12-19T07:47:22.302909+0000","last_fullsized":"2023-12-19T07:47:22.302909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806686+0000","last_change":"2023-12-19T07:47:17.196779+0000","last_active":"2023-12-19T07:47:22.806686+0000","last_peered":"2023-12-19T07:47:22.806686+0000","last_clean":"2023-12-19T07:47:22.806686+0000","last_became_active":"2023-12-19T07:47:17.196494+0000","last_became_peered":"2023-12-19T07:47:17.196494+0000","last_unstale":"2023-12-19T07:47:22.806686+0000","last_undegraded":"2023-12-19T07:47:22.806686+0000","last_fullsized":"2023-12-19T07:47:22.806686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903605+0000","last_change":"2023-12-19T07:47:17.205970+0000","last_active":"2023-12-19T07:47:22.903605+0000","last_peered":"2023-12-19T07:47:22.903605+0000","last_clean":"2023-12-19T07:47:22.903605+0000","last_became_active":"2023-12-19T07:47:17.205581+0000","last_became_peered":"2023-12-19T07:47:17.205581+0000","last_unstale":"2023-12-19T07:47:22.903605+0000","last_undegraded":"2023-12-19T07:47:22.903605+0000","last_fullsized":"2023-12-19T07:47:22.903605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599435+0000","last_change":"2023-12-19T07:47:17.208151+0000","last_active":"2023-12-19T07:47:22.599435+0000","last_peered":"2023-12-19T07:47:22.599435+0000","last_clean":"2023-12-19T07:47:22.599435+0000","last_became_active":"2023-12-19T07:47:17.207880+0000","last_became_peered":"2023-12-19T07:47:17.207880+0000","last_unstale":"2023-12-19T07:47:22.599435+0000","last_undegraded":"2023-12-19T07:47:22.599435+0000","last_fullsized":"2023-12-19T07:47:22.599435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352831+0000","last_change":"2023-12-19T07:47:17.209159+0000","last_active":"2023-12-19T07:47:22.352831+0000","last_peered":"2023-12-19T07:47:22.352831+0000","last_clean":"2023-12-19T07:47:22.352831+0000","last_became_active":"2023-12-19T07:47:17.208870+0000","last_became_peered":"2023-12-19T07:47:17.208870+0000","last_unstale":"2023-12-19T07:47:22.352831+0000","last_undegraded":"2023-12-19T07:47:22.352831+0000","last_fullsized":"2023-12-19T07:47:22.352831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.360063+0000","last_change":"2023-12-19T07:47:17.209320+0000","last_active":"2023-12-19T07:47:22.360063+0000","last_peered":"2023-12-19T07:47:22.360063+0000","last_clean":"2023-12-19T07:47:22.360063+0000","last_became_active":"2023-12-19T07:47:17.208263+0000","last_became_peered":"2023-12-19T07:47:17.208263+0000","last_unstale":"2023-12-19T07:47:22.360063+0000","last_undegraded":"2023-12-19T07:47:22.360063+0000","last_fullsized":"2023-12-19T07:47:22.360063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599483+0000","last_change":"2023-12-19T07:47:17.211203+0000","last_active":"2023-12-19T07:47:22.599483+0000","last_peered":"2023-12-19T07:47:22.599483+0000","last_clean":"2023-12-19T07:47:22.599483+0000","last_became_active":"2023-12-19T07:47:17.210860+0000","last_became_peered":"2023-12-19T07:47:17.210860+0000","last_unstale":"2023-12-19T07:47:22.599483+0000","last_undegraded":"2023-12-19T07:47:22.599483+0000","last_fullsized":"2023-12-19T07:47:22.599483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805820+0000","last_change":"2023-12-19T07:47:17.210725+0000","last_active":"2023-12-19T07:47:22.805820+0000","last_peered":"2023-12-19T07:47:22.805820+0000","last_clean":"2023-12-19T07:47:22.805820+0000","last_became_active":"2023-12-19T07:47:17.210432+0000","last_became_peered":"2023-12-19T07:47:17.210432+0000","last_unstale":"2023-12-19T07:47:22.805820+0000","last_undegraded":"2023-12-19T07:47:22.805820+0000","last_fullsized":"2023-12-19T07:47:22.805820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599063+0000","last_change":"2023-12-19T07:47:17.200440+0000","last_active":"2023-12-19T07:47:22.599063+0000","last_peered":"2023-12-19T07:47:22.599063+0000","last_clean":"2023-12-19T07:47:22.599063+0000","last_became_active":"2023-12-19T07:47:17.200137+0000","last_became_peered":"2023-12-19T07:47:17.200137+0000","last_unstale":"2023-12-19T07:47:22.599063+0000","last_undegraded":"2023-12-19T07:47:22.599063+0000","last_fullsized":"2023-12-19T07:47:22.599063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806350+0000","last_change":"2023-12-19T07:47:17.200217+0000","last_active":"2023-12-19T07:47:22.806350+0000","last_peered":"2023-12-19T07:47:22.806350+0000","last_clean":"2023-12-19T07:47:22.806350+0000","last_became_active":"2023-12-19T07:47:17.199920+0000","last_became_peered":"2023-12-19T07:47:17.199920+0000","last_unstale":"2023-12-19T07:47:22.806350+0000","last_undegraded":"2023-12-19T07:47:22.806350+0000","last_fullsized":"2023-12-19T07:47:22.806350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903403+0000","last_change":"2023-12-19T07:47:17.200705+0000","last_active":"2023-12-19T07:47:22.903403+0000","last_peered":"2023-12-19T07:47:22.903403+0000","last_clean":"2023-12-19T07:47:22.903403+0000","last_became_active":"2023-12-19T07:47:17.200462+0000","last_became_peered":"2023-12-19T07:47:17.200462+0000","last_unstale":"2023-12-19T07:47:22.903403+0000","last_undegraded":"2023-12-19T07:47:22.903403+0000","last_fullsized":"2023-12-19T07:47:22.903403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359431+0000","last_change":"2023-12-19T07:47:17.212064+0000","last_active":"2023-12-19T07:47:22.359431+0000","last_peered":"2023-12-19T07:47:22.359431+0000","last_clean":"2023-12-19T07:47:22.359431+0000","last_became_active":"2023-12-19T07:47:17.211684+0000","last_became_peered":"2023-12-19T07:47:17.211684+0000","last_unstale":"2023-12-19T07:47:22.359431+0000","last_undegraded":"2023-12-19T07:47:22.359431+0000","last_fullsized":"2023-12-19T07:47:22.359431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:21.908290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868184+0000","last_change":"2023-12-19T07:47:17.212402+0000","last_active":"2023-12-19T07:47:22.868184+0000","last_peered":"2023-12-19T07:47:22.868184+0000","last_clean":"2023-12-19T07:47:22.868184+0000","last_became_active":"2023-12-19T07:47:17.211887+0000","last_became_peered":"2023-12-19T07:47:17.211887+0000","last_unstale":"2023-12-19T07:47:22.868184+0000","last_undegraded":"2023-12-19T07:47:22.868184+0000","last_fullsized":"2023-12-19T07:47:22.868184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304186+0000","last_change":"2023-12-19T07:47:17.207392+0000","last_active":"2023-12-19T07:47:22.304186+0000","last_peered":"2023-12-19T07:47:22.304186+0000","last_clean":"2023-12-19T07:47:22.304186+0000","last_became_active":"2023-12-19T07:47:17.206962+0000","last_became_peered":"2023-12-19T07:47:17.206962+0000","last_unstale":"2023-12-19T07:47:22.304186+0000","last_undegraded":"2023-12-19T07:47:22.304186+0000","last_fullsized":"2023-12-19T07:47:22.304186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807623+0000","last_change":"2023-12-19T07:47:17.217408+0000","last_active":"2023-12-19T07:47:22.807623+0000","last_peered":"2023-12-19T07:47:22.807623+0000","last_clean":"2023-12-19T07:47:22.807623+0000","last_became_active":"2023-12-19T07:47:17.217038+0000","last_became_peered":"2023-12-19T07:47:17.217038+0000","last_unstale":"2023-12-19T07:47:22.807623+0000","last_undegraded":"2023-12-19T07:47:22.807623+0000","last_fullsized":"2023-12-19T07:47:22.807623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:12.732155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805776+0000","last_change":"2023-12-19T07:47:17.194768+0000","last_active":"2023-12-19T07:47:22.805776+0000","last_peered":"2023-12-19T07:47:22.805776+0000","last_clean":"2023-12-19T07:47:22.805776+0000","last_became_active":"2023-12-19T07:47:17.194457+0000","last_became_peered":"2023-12-19T07:47:17.194457+0000","last_unstale":"2023-12-19T07:47:22.805776+0000","last_undegraded":"2023-12-19T07:47:22.805776+0000","last_fullsized":"2023-12-19T07:47:22.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351363+0000","last_change":"2023-12-19T07:47:17.209052+0000","last_active":"2023-12-19T07:47:22.351363+0000","last_peered":"2023-12-19T07:47:22.351363+0000","last_clean":"2023-12-19T07:47:22.351363+0000","last_became_active":"2023-12-19T07:47:17.208671+0000","last_became_peered":"2023-12-19T07:47:17.208671+0000","last_unstale":"2023-12-19T07:47:22.351363+0000","last_undegraded":"2023-12-19T07:47:22.351363+0000","last_fullsized":"2023-12-19T07:47:22.351363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903861+0000","last_change":"2023-12-19T07:47:17.199159+0000","last_active":"2023-12-19T07:47:22.903861+0000","last_peered":"2023-12-19T07:47:22.903861+0000","last_clean":"2023-12-19T07:47:22.903861+0000","last_became_active":"2023-12-19T07:47:17.198548+0000","last_became_peered":"2023-12-19T07:47:17.198548+0000","last_unstale":"2023-12-19T07:47:22.903861+0000","last_undegraded":"2023-12-19T07:47:22.903861+0000","last_fullsized":"2023-12-19T07:47:22.903861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598739+0000","last_change":"2023-12-19T07:47:17.202636+0000","last_active":"2023-12-19T07:47:22.598739+0000","last_peered":"2023-12-19T07:47:22.598739+0000","last_clean":"2023-12-19T07:47:22.598739+0000","last_became_active":"2023-12-19T07:47:17.202331+0000","last_became_peered":"2023-12-19T07:47:17.202331+0000","last_unstale":"2023-12-19T07:47:22.598739+0000","last_undegraded":"2023-12-19T07:47:22.598739+0000","last_fullsized":"2023-12-19T07:47:22.598739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306911+0000","last_change":"2023-12-19T07:47:17.202590+0000","last_active":"2023-12-19T07:47:22.306911+0000","last_peered":"2023-12-19T07:47:22.306911+0000","last_clean":"2023-12-19T07:47:22.306911+0000","last_became_active":"2023-12-19T07:47:17.202206+0000","last_became_peered":"2023-12-19T07:47:17.202206+0000","last_unstale":"2023-12-19T07:47:22.306911+0000","last_undegraded":"2023-12-19T07:47:22.306911+0000","last_fullsized":"2023-12-19T07:47:22.306911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:04.053349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359760+0000","last_change":"2023-12-19T07:47:17.209806+0000","last_active":"2023-12-19T07:47:22.359760+0000","last_peered":"2023-12-19T07:47:22.359760+0000","last_clean":"2023-12-19T07:47:22.359760+0000","last_became_active":"2023-12-19T07:47:17.209424+0000","last_became_peered":"2023-12-19T07:47:17.209424+0000","last_unstale":"2023-12-19T07:47:22.359760+0000","last_undegraded":"2023-12-19T07:47:22.359760+0000","last_fullsized":"2023-12-19T07:47:22.359760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359954+0000","last_change":"2023-12-19T07:47:17.209509+0000","last_active":"2023-12-19T07:47:22.359954+0000","last_peered":"2023-12-19T07:47:22.359954+0000","last_clean":"2023-12-19T07:47:22.359954+0000","last_became_active":"2023-12-19T07:47:17.209026+0000","last_became_peered":"2023-12-19T07:47:17.209026+0000","last_unstale":"2023-12-19T07:47:22.359954+0000","last_undegraded":"2023-12-19T07:47:22.359954+0000","last_fullsized":"2023-12-19T07:47:22.359954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868374+0000","last_change":"2023-12-19T07:47:17.225611+0000","last_active":"2023-12-19T07:47:22.868374+0000","last_peered":"2023-12-19T07:47:22.868374+0000","last_clean":"2023-12-19T07:47:22.868374+0000","last_became_active":"2023-12-19T07:47:17.225305+0000","last_became_peered":"2023-12-19T07:47:17.225305+0000","last_unstale":"2023-12-19T07:47:22.868374+0000","last_undegraded":"2023-12-19T07:47:22.868374+0000","last_fullsized":"2023-12-19T07:47:22.868374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.474113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598130+0000","last_change":"2023-12-19T07:47:17.209463+0000","last_active":"2023-12-19T07:47:22.598130+0000","last_peered":"2023-12-19T07:47:22.598130+0000","last_clean":"2023-12-19T07:47:22.598130+0000","last_became_active":"2023-12-19T07:47:17.201276+0000","last_became_peered":"2023-12-19T07:47:17.201276+0000","last_unstale":"2023-12-19T07:47:22.598130+0000","last_undegraded":"2023-12-19T07:47:22.598130+0000","last_fullsized":"2023-12-19T07:47:22.598130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867603+0000","last_change":"2023-12-19T07:47:17.200808+0000","last_active":"2023-12-19T07:47:22.867603+0000","last_peered":"2023-12-19T07:47:22.867603+0000","last_clean":"2023-12-19T07:47:22.867603+0000","last_became_active":"2023-12-19T07:47:17.200445+0000","last_became_peered":"2023-12-19T07:47:17.200445+0000","last_unstale":"2023-12-19T07:47:22.867603+0000","last_undegraded":"2023-12-19T07:47:22.867603+0000","last_fullsized":"2023-12-19T07:47:22.867603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806351+0000","last_change":"2023-12-19T07:47:17.194801+0000","last_active":"2023-12-19T07:47:22.806351+0000","last_peered":"2023-12-19T07:47:22.806351+0000","last_clean":"2023-12-19T07:47:22.806351+0000","last_became_active":"2023-12-19T07:47:17.194502+0000","last_became_peered":"2023-12-19T07:47:17.194502+0000","last_unstale":"2023-12-19T07:47:22.806351+0000","last_undegraded":"2023-12-19T07:47:22.806351+0000","last_fullsized":"2023-12-19T07:47:22.806351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867343+0000","last_change":"2023-12-19T07:47:17.214803+0000","last_active":"2023-12-19T07:47:22.867343+0000","last_peered":"2023-12-19T07:47:22.867343+0000","last_clean":"2023-12-19T07:47:22.867343+0000","last_became_active":"2023-12-19T07:47:17.214536+0000","last_became_peered":"2023-12-19T07:47:17.214536+0000","last_unstale":"2023-12-19T07:47:22.867343+0000","last_undegraded":"2023-12-19T07:47:22.867343+0000","last_fullsized":"2023-12-19T07:47:22.867343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806760+0000","last_change":"2023-12-19T07:47:17.206649+0000","last_active":"2023-12-19T07:47:22.806760+0000","last_peered":"2023-12-19T07:47:22.806760+0000","last_clean":"2023-12-19T07:47:22.806760+0000","last_became_active":"2023-12-19T07:47:17.206360+0000","last_became_peered":"2023-12-19T07:47:17.206360+0000","last_unstale":"2023-12-19T07:47:22.806760+0000","last_undegraded":"2023-12-19T07:47:22.806760+0000","last_fullsized":"2023-12-19T07:47:22.806760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904329+0000","last_change":"2023-12-19T07:47:17.195685+0000","last_active":"2023-12-19T07:47:22.904329+0000","last_peered":"2023-12-19T07:47:22.904329+0000","last_clean":"2023-12-19T07:47:22.904329+0000","last_became_active":"2023-12-19T07:47:17.195380+0000","last_became_peered":"2023-12-19T07:47:17.195380+0000","last_unstale":"2023-12-19T07:47:22.904329+0000","last_undegraded":"2023-12-19T07:47:22.904329+0000","last_fullsized":"2023-12-19T07:47:22.904329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:47.378624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353591+0000","last_change":"2023-12-19T07:47:17.204379+0000","last_active":"2023-12-19T07:47:22.353591+0000","last_peered":"2023-12-19T07:47:22.353591+0000","last_clean":"2023-12-19T07:47:22.353591+0000","last_became_active":"2023-12-19T07:47:17.204111+0000","last_became_peered":"2023-12-19T07:47:17.204111+0000","last_unstale":"2023-12-19T07:47:22.353591+0000","last_undegraded":"2023-12-19T07:47:22.353591+0000","last_fullsized":"2023-12-19T07:47:22.353591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868221+0000","last_change":"2023-12-19T07:47:17.209821+0000","last_active":"2023-12-19T07:47:22.868221+0000","last_peered":"2023-12-19T07:47:22.868221+0000","last_clean":"2023-12-19T07:47:22.868221+0000","last_became_active":"2023-12-19T07:47:17.209235+0000","last_became_peered":"2023-12-19T07:47:17.209235+0000","last_unstale":"2023-12-19T07:47:22.868221+0000","last_undegraded":"2023-12-19T07:47:22.868221+0000","last_fullsized":"2023-12-19T07:47:22.868221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:23.894121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":864256,"data_stored":697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.544}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":270336,"data_stored":106631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":106667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":106541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2909999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297556,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074284,"statfs":{"total":96636764160,"available":96332066816,"internally_reserved":0,"allocated":274432,"data_stored":106701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":868352,"data_stored":696508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:47:33.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:47:33.401 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:47:33.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:47:33.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.508+0000 7f8a577c5700 1 Processor -- start 2023-12-19T07:47:33.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.508+0000 7f8a577c5700 1 -- start start 2023-12-19T07:47:33.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a577c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a50140710 0x7f8a500c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a577c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a50068800 0x7f8a500ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a577c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5006e150 con 0x7f8a50140ae0 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a577c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5006d540 con 0x7f8a50140710 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a577c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5006d820 con 0x7f8a50068800 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55561700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a50140710 0x7f8a500c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a54d60700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8a50140ae0 0x7f8a500680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46268/0 (socket says 172.21.15.111:46268) 2023-12-19T07:47:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a54d60700 1 -- 172.21.15.111:0/620576402 learned_addr learned my addr 172.21.15.111:0/620576402 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55d62700 1 --2- 172.21.15.111:0/620576402 >> v2:172.21.15.143:3300/0 conn(0x7f8a50068800 0x7f8a500ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a47fff700 1 -- 172.21.15.111:0/620576402 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081158714 0 0) 0x7f8a5006e150 con 0x7f8a50140ae0 2023-12-19T07:47:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a47fff700 1 -- 172.21.15.111:0/620576402 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c003570 con 0x7f8a50140ae0 2023-12-19T07:47:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55561700 1 -- 172.21.15.111:0/620576402 >> v2:172.21.15.143:3300/0 conn(0x7f8a50068800 msgr2=0x7f8a500ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55561700 1 --2- 172.21.15.111:0/620576402 >> v2:172.21.15.143:3300/0 conn(0x7f8a50068800 0x7f8a500ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55561700 1 -- 172.21.15.111:0/620576402 >> v1:172.21.15.143:6789/0 conn(0x7f8a50140ae0 legacy=0x7f8a500680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.512+0000 7f8a55561700 1 -- 172.21.15.111:0/620576402 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5006db00 con 0x7f8a50140710 2023-12-19T07:47:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a55561700 1 --2- 172.21.15.111:0/620576402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50140710 0x7f8a500c70a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a5006bfc0 tx=0x7f8a4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb704ddca667742 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a47fff700 1 -- 172.21.15.111:0/620576402 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00e070 con 0x7f8a50140710 2023-12-19T07:47:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a47fff700 1 -- 172.21.15.111:0/620576402 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c00bab0 con 0x7f8a50140710 2023-12-19T07:47:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a47fff700 1 -- 172.21.15.111:0/620576402 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00be70 con 0x7f8a50140710 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 -- 172.21.15.111:0/620576402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50140710 msgr2=0x7f8a500c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 --2- 172.21.15.111:0/620576402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50140710 0x7f8a500c70a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a5006bfc0 tx=0x7f8a4c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 -- 172.21.15.111:0/620576402 shutdown_connections 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 --2- 172.21.15.111:0/620576402 >> v2:172.21.15.143:3300/0 conn(0x7f8a50068800 0x7f8a500ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 --2- 172.21.15.111:0/620576402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50140710 0x7f8a500c70a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 -- 172.21.15.111:0/620576402 >> 172.21.15.111:0/620576402 conn(0x7f8a500c8850 msgr2=0x7f8a500c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 -- 172.21.15.111:0/620576402 shutdown_connections 2023-12-19T07:47:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 -- 172.21.15.111:0/620576402 wait complete. 2023-12-19T07:47:33.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.516+0000 7f8a577c5700 1 Processor -- start 2023-12-19T07:47:33.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 -- start start 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a50140710 0x7f8a500c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a50140ae0 0x7f8a50159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500c44f0 con 0x7f8a50068800 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500c38e0 con 0x7f8a50140710 2023-12-19T07:47:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500c3bc0 con 0x7f8a50140ae0 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34508/0 (socket says 172.21.15.111:34508) 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 -- 172.21.15.111:0/1339581490 learned_addr learned my addr 172.21.15.111:0/1339581490 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a54d60700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a50140710 0x7f8a500c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55d62700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a50140ae0 0x7f8a50159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 -- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a50140ae0 msgr2=0x7f8a50159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a50140ae0 0x7f8a50159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 -- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a50140710 msgr2=0x7f8a500c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a50140710 0x7f8a500c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 -- 172.21.15.111:0/1339581490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c009a20 con 0x7f8a50068800 2023-12-19T07:47:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a55561700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8a5006a280 tx=0x7f8a4c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c012b60 con 0x7f8a50068800 2023-12-19T07:47:33.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c012cc0 con 0x7f8a50068800 2023-12-19T07:47:33.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a500c3ea0 con 0x7f8a50068800 2023-12-19T07:47:33.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.520+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0276c0 con 0x7f8a50068800 2023-12-19T07:47:33.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.524+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a500c4980 con 0x7f8a50068800 2023-12-19T07:47:33.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.524+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a50071120 con 0x7f8a50068800 2023-12-19T07:47:33.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.524+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8a4c024070 con 0x7f8a50068800 2023-12-19T07:47:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.528+0000 7f8a457fa700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 0x7f8a34062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.528+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8a4c07ef50 con 0x7f8a50068800 2023-12-19T07:47:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.528+0000 7f8a54d60700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 0x7f8a34062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.528+0000 7f8a54d60700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 0x7f8a34062a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a501593d0 tx=0x7f8a40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.532+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a4c04f1b0 con 0x7f8a50068800 2023-12-19T07:47:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.716+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a500c4320 con 0x7f8a34060550 2023-12-19T07:47:33.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.736+0000 7f8a457fa700 1 -- 172.21.15.111:0/1339581490 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345063 (secure 0 0 0) 0x7f8a4000f680 con 0x7f8a34060550 2023-12-19T07:47:33.725 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 msgr2=0x7f8a34062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 0x7f8a34062a00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a501593d0 tx=0x7f8a40009040 comp rx=0 tx=0).stop 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 msgr2=0x7f8a50133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8a5006a280 tx=0x7f8a4c019000 comp rx=0 tx=0).stop 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 shutdown_connections 2023-12-19T07:47:33.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a34060550 0x7f8a34062a00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a50140710 0x7f8a500c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a50140ae0 0x7f8a50159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 --2- 172.21.15.111:0/1339581490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a50068800 0x7f8a50133a60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 >> 172.21.15.111:0/1339581490 conn(0x7f8a500c8850 msgr2=0x7f8a50067d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 shutdown_connections 2023-12-19T07:47:33.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.744+0000 7f8a577c5700 1 -- 172.21.15.111:0/1339581490 wait complete. 2023-12-19T07:47:33.730 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:47:33.749 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-12-19T07:47:32.614682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381604,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593116,"statfs":{"total":773094113280,"available":770655350784,"internally_reserved":0,"allocated":3379200,"data_stored":2041019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003315"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904960+0000","last_change":"2023-12-19T07:47:16.213519+0000","last_active":"2023-12-19T07:47:22.904960+0000","last_peered":"2023-12-19T07:47:22.904960+0000","last_clean":"2023-12-19T07:47:22.904960+0000","last_became_active":"2023-12-19T07:47:16.213172+0000","last_became_peered":"2023-12-19T07:47:16.213172+0000","last_unstale":"2023-12-19T07:47:22.904960+0000","last_undegraded":"2023-12-19T07:47:22.904960+0000","last_fullsized":"2023-12-19T07:47:22.904960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599466+0000","last_change":"2023-12-19T07:47:17.206618+0000","last_active":"2023-12-19T07:47:22.599466+0000","last_peered":"2023-12-19T07:47:22.599466+0000","last_clean":"2023-12-19T07:47:22.599466+0000","last_became_active":"2023-12-19T07:47:17.206300+0000","last_became_peered":"2023-12-19T07:47:17.206300+0000","last_unstale":"2023-12-19T07:47:22.599466+0000","last_undegraded":"2023-12-19T07:47:22.599466+0000","last_fullsized":"2023-12-19T07:47:22.599466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597951+0000","last_change":"2023-12-19T07:47:21.847101+0000","last_active":"2023-12-19T07:47:22.597951+0000","last_peered":"2023-12-19T07:47:22.597951+0000","last_clean":"2023-12-19T07:47:22.597951+0000","last_became_active":"2023-12-19T07:47:21.845076+0000","last_became_peered":"2023-12-19T07:47:21.845076+0000","last_unstale":"2023-12-19T07:47:22.597951+0000","last_undegraded":"2023-12-19T07:47:22.597951+0000","last_fullsized":"2023-12-19T07:47:22.597951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599595+0000","last_change":"2023-12-19T07:47:16.198270+0000","last_active":"2023-12-19T07:47:22.599595+0000","last_peered":"2023-12-19T07:47:22.599595+0000","last_clean":"2023-12-19T07:47:22.599595+0000","last_became_active":"2023-12-19T07:47:16.197562+0000","last_became_peered":"2023-12-19T07:47:16.197562+0000","last_unstale":"2023-12-19T07:47:22.599595+0000","last_undegraded":"2023-12-19T07:47:22.599595+0000","last_fullsized":"2023-12-19T07:47:22.599595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904301+0000","last_change":"2023-12-19T07:47:17.198174+0000","last_active":"2023-12-19T07:47:22.904301+0000","last_peered":"2023-12-19T07:47:22.904301+0000","last_clean":"2023-12-19T07:47:22.904301+0000","last_became_active":"2023-12-19T07:47:17.197876+0000","last_became_peered":"2023-12-19T07:47:17.197876+0000","last_unstale":"2023-12-19T07:47:22.904301+0000","last_undegraded":"2023-12-19T07:47:22.904301+0000","last_fullsized":"2023-12-19T07:47:22.904301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306106+0000","last_change":"2023-12-19T07:47:21.260994+0000","last_active":"2023-12-19T07:47:22.306106+0000","last_peered":"2023-12-19T07:47:22.306106+0000","last_clean":"2023-12-19T07:47:22.306106+0000","last_became_active":"2023-12-19T07:47:21.260122+0000","last_became_peered":"2023-12-19T07:47:21.260122+0000","last_unstale":"2023-12-19T07:47:22.306106+0000","last_undegraded":"2023-12-19T07:47:22.306106+0000","last_fullsized":"2023-12-19T07:47:22.306106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868136+0000","last_change":"2023-12-19T07:47:16.215728+0000","last_active":"2023-12-19T07:47:22.868136+0000","last_peered":"2023-12-19T07:47:22.868136+0000","last_clean":"2023-12-19T07:47:22.868136+0000","last_became_active":"2023-12-19T07:47:16.215434+0000","last_became_peered":"2023-12-19T07:47:16.215434+0000","last_unstale":"2023-12-19T07:47:22.868136+0000","last_undegraded":"2023-12-19T07:47:22.868136+0000","last_fullsized":"2023-12-19T07:47:22.868136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806795+0000","last_change":"2023-12-19T07:47:17.204556+0000","last_active":"2023-12-19T07:47:22.806795+0000","last_peered":"2023-12-19T07:47:22.806795+0000","last_clean":"2023-12-19T07:47:22.806795+0000","last_became_active":"2023-12-19T07:47:17.204225+0000","last_became_peered":"2023-12-19T07:47:17.204225+0000","last_unstale":"2023-12-19T07:47:22.806795+0000","last_undegraded":"2023-12-19T07:47:22.806795+0000","last_fullsized":"2023-12-19T07:47:22.806795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306583+0000","last_change":"2023-12-19T07:47:21.266228+0000","last_active":"2023-12-19T07:47:22.306583+0000","last_peered":"2023-12-19T07:47:22.306583+0000","last_clean":"2023-12-19T07:47:22.306583+0000","last_became_active":"2023-12-19T07:47:21.265305+0000","last_became_peered":"2023-12-19T07:47:21.265305+0000","last_unstale":"2023-12-19T07:47:22.306583+0000","last_undegraded":"2023-12-19T07:47:22.306583+0000","last_fullsized":"2023-12-19T07:47:22.306583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867767+0000","last_change":"2023-12-19T07:47:16.214913+0000","last_active":"2023-12-19T07:47:22.867767+0000","last_peered":"2023-12-19T07:47:22.867767+0000","last_clean":"2023-12-19T07:47:22.867767+0000","last_became_active":"2023-12-19T07:47:16.214514+0000","last_became_peered":"2023-12-19T07:47:16.214514+0000","last_unstale":"2023-12-19T07:47:22.867767+0000","last_undegraded":"2023-12-19T07:47:22.867767+0000","last_fullsized":"2023-12-19T07:47:22.867767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:24.444351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807216+0000","last_change":"2023-12-19T07:47:17.208692+0000","last_active":"2023-12-19T07:47:22.807216+0000","last_peered":"2023-12-19T07:47:22.807216+0000","last_clean":"2023-12-19T07:47:22.807216+0000","last_became_active":"2023-12-19T07:47:17.208320+0000","last_became_peered":"2023-12-19T07:47:17.208320+0000","last_unstale":"2023-12-19T07:47:22.807216+0000","last_undegraded":"2023-12-19T07:47:22.807216+0000","last_fullsized":"2023-12-19T07:47:22.807216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306291+0000","last_change":"2023-12-19T07:47:21.249790+0000","last_active":"2023-12-19T07:47:22.306291+0000","last_peered":"2023-12-19T07:47:22.306291+0000","last_clean":"2023-12-19T07:47:22.306291+0000","last_became_active":"2023-12-19T07:47:21.249266+0000","last_became_peered":"2023-12-19T07:47:21.249266+0000","last_unstale":"2023-12-19T07:47:22.306291+0000","last_undegraded":"2023-12-19T07:47:22.306291+0000","last_fullsized":"2023-12-19T07:47:22.306291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358655+0000","last_change":"2023-12-19T07:47:17.206257+0000","last_active":"2023-12-19T07:47:22.358655+0000","last_peered":"2023-12-19T07:47:22.358655+0000","last_clean":"2023-12-19T07:47:22.358655+0000","last_became_active":"2023-12-19T07:47:17.205737+0000","last_became_peered":"2023-12-19T07:47:17.205737+0000","last_unstale":"2023-12-19T07:47:22.358655+0000","last_undegraded":"2023-12-19T07:47:22.358655+0000","last_fullsized":"2023-12-19T07:47:22.358655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867658+0000","last_change":"2023-12-19T07:47:16.214499+0000","last_active":"2023-12-19T07:47:22.867658+0000","last_peered":"2023-12-19T07:47:22.867658+0000","last_clean":"2023-12-19T07:47:22.867658+0000","last_became_active":"2023-12-19T07:47:16.213693+0000","last_became_peered":"2023-12-19T07:47:16.213693+0000","last_unstale":"2023-12-19T07:47:22.867658+0000","last_undegraded":"2023-12-19T07:47:22.867658+0000","last_fullsized":"2023-12-19T07:47:22.867658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307061+0000","last_change":"2023-12-19T07:47:21.254224+0000","last_active":"2023-12-19T07:47:22.307061+0000","last_peered":"2023-12-19T07:47:22.307061+0000","last_clean":"2023-12-19T07:47:22.307061+0000","last_became_active":"2023-12-19T07:47:21.253786+0000","last_became_peered":"2023-12-19T07:47:21.253786+0000","last_unstale":"2023-12-19T07:47:22.307061+0000","last_undegraded":"2023-12-19T07:47:22.307061+0000","last_fullsized":"2023-12-19T07:47:22.307061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:18:10.044237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598837+0000","last_change":"2023-12-19T07:47:17.205670+0000","last_active":"2023-12-19T07:47:22.598837+0000","last_peered":"2023-12-19T07:47:22.598837+0000","last_clean":"2023-12-19T07:47:22.598837+0000","last_became_active":"2023-12-19T07:47:17.201174+0000","last_became_peered":"2023-12-19T07:47:17.201174+0000","last_unstale":"2023-12-19T07:47:22.598837+0000","last_undegraded":"2023-12-19T07:47:22.598837+0000","last_fullsized":"2023-12-19T07:47:22.598837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352125+0000","last_change":"2023-12-19T07:47:16.196003+0000","last_active":"2023-12-19T07:47:22.352125+0000","last_peered":"2023-12-19T07:47:22.352125+0000","last_clean":"2023-12-19T07:47:22.352125+0000","last_became_active":"2023-12-19T07:47:16.195717+0000","last_became_peered":"2023-12-19T07:47:16.195717+0000","last_unstale":"2023-12-19T07:47:22.352125+0000","last_undegraded":"2023-12-19T07:47:22.352125+0000","last_fullsized":"2023-12-19T07:47:22.352125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306644+0000","last_change":"2023-12-19T07:47:21.259995+0000","last_active":"2023-12-19T07:47:22.306644+0000","last_peered":"2023-12-19T07:47:22.306644+0000","last_clean":"2023-12-19T07:47:22.306644+0000","last_became_active":"2023-12-19T07:47:21.259318+0000","last_became_peered":"2023-12-19T07:47:21.259318+0000","last_unstale":"2023-12-19T07:47:22.306644+0000","last_undegraded":"2023-12-19T07:47:22.306644+0000","last_fullsized":"2023-12-19T07:47:22.306644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598330+0000","last_change":"2023-12-19T07:47:17.205504+0000","last_active":"2023-12-19T07:47:22.598330+0000","last_peered":"2023-12-19T07:47:22.598330+0000","last_clean":"2023-12-19T07:47:22.598330+0000","last_became_active":"2023-12-19T07:47:17.200074+0000","last_became_peered":"2023-12-19T07:47:17.200074+0000","last_unstale":"2023-12-19T07:47:22.598330+0000","last_undegraded":"2023-12-19T07:47:22.598330+0000","last_fullsized":"2023-12-19T07:47:22.598330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351472+0000","last_change":"2023-12-19T07:47:16.206620+0000","last_active":"2023-12-19T07:47:22.351472+0000","last_peered":"2023-12-19T07:47:22.351472+0000","last_clean":"2023-12-19T07:47:22.351472+0000","last_became_active":"2023-12-19T07:47:16.206288+0000","last_became_peered":"2023-12-19T07:47:16.206288+0000","last_unstale":"2023-12-19T07:47:22.351472+0000","last_undegraded":"2023-12-19T07:47:22.351472+0000","last_fullsized":"2023-12-19T07:47:22.351472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351144+0000","last_change":"2023-12-19T07:47:21.841279+0000","last_active":"2023-12-19T07:47:22.351144+0000","last_peered":"2023-12-19T07:47:22.351144+0000","last_clean":"2023-12-19T07:47:22.351144+0000","last_became_active":"2023-12-19T07:47:21.840184+0000","last_became_peered":"2023-12-19T07:47:21.840184+0000","last_unstale":"2023-12-19T07:47:22.351144+0000","last_undegraded":"2023-12-19T07:47:22.351144+0000","last_fullsized":"2023-12-19T07:47:22.351144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:22:46.085295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306581+0000","last_change":"2023-12-19T07:47:17.199136+0000","last_active":"2023-12-19T07:47:22.306581+0000","last_peered":"2023-12-19T07:47:22.306581+0000","last_clean":"2023-12-19T07:47:22.306581+0000","last_became_active":"2023-12-19T07:47:17.198802+0000","last_became_peered":"2023-12-19T07:47:17.198802+0000","last_unstale":"2023-12-19T07:47:22.306581+0000","last_undegraded":"2023-12-19T07:47:22.306581+0000","last_fullsized":"2023-12-19T07:47:22.306581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:35.655311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866892+0000","last_change":"2023-12-19T07:47:16.213635+0000","last_active":"2023-12-19T07:47:22.866892+0000","last_peered":"2023-12-19T07:47:22.866892+0000","last_clean":"2023-12-19T07:47:22.866892+0000","last_became_active":"2023-12-19T07:47:16.213201+0000","last_became_peered":"2023-12-19T07:47:16.213201+0000","last_unstale":"2023-12-19T07:47:22.866892+0000","last_undegraded":"2023-12-19T07:47:22.866892+0000","last_fullsized":"2023-12-19T07:47:22.866892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306787+0000","last_change":"2023-12-19T07:47:21.265928+0000","last_active":"2023-12-19T07:47:22.306787+0000","last_peered":"2023-12-19T07:47:22.306787+0000","last_clean":"2023-12-19T07:47:22.306787+0000","last_became_active":"2023-12-19T07:47:21.265360+0000","last_became_peered":"2023-12-19T07:47:21.265360+0000","last_unstale":"2023-12-19T07:47:22.306787+0000","last_undegraded":"2023-12-19T07:47:22.306787+0000","last_fullsized":"2023-12-19T07:47:22.306787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904688+0000","last_change":"2023-12-19T07:47:17.195176+0000","last_active":"2023-12-19T07:47:22.904688+0000","last_peered":"2023-12-19T07:47:22.904688+0000","last_clean":"2023-12-19T07:47:22.904688+0000","last_became_active":"2023-12-19T07:47:17.194607+0000","last_became_peered":"2023-12-19T07:47:17.194607+0000","last_unstale":"2023-12-19T07:47:22.904688+0000","last_undegraded":"2023-12-19T07:47:22.904688+0000","last_fullsized":"2023-12-19T07:47:22.904688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359926+0000","last_change":"2023-12-19T07:47:16.206513+0000","last_active":"2023-12-19T07:47:22.359926+0000","last_peered":"2023-12-19T07:47:22.359926+0000","last_clean":"2023-12-19T07:47:22.359926+0000","last_became_active":"2023-12-19T07:47:16.206148+0000","last_became_peered":"2023-12-19T07:47:16.206148+0000","last_unstale":"2023-12-19T07:47:22.359926+0000","last_undegraded":"2023-12-19T07:47:22.359926+0000","last_fullsized":"2023-12-19T07:47:22.359926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.305902+0000","last_change":"2023-12-19T07:47:21.841183+0000","last_active":"2023-12-19T07:47:22.305902+0000","last_peered":"2023-12-19T07:47:22.305902+0000","last_clean":"2023-12-19T07:47:22.305902+0000","last_became_active":"2023-12-19T07:47:21.839559+0000","last_became_peered":"2023-12-19T07:47:21.839559+0000","last_unstale":"2023-12-19T07:47:22.305902+0000","last_undegraded":"2023-12-19T07:47:22.305902+0000","last_fullsized":"2023-12-19T07:47:22.305902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:27.954678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599577+0000","last_change":"2023-12-19T07:47:17.208713+0000","last_active":"2023-12-19T07:47:22.599577+0000","last_peered":"2023-12-19T07:47:22.599577+0000","last_clean":"2023-12-19T07:47:22.599577+0000","last_became_active":"2023-12-19T07:47:17.202555+0000","last_became_peered":"2023-12-19T07:47:17.202555+0000","last_unstale":"2023-12-19T07:47:22.599577+0000","last_undegraded":"2023-12-19T07:47:22.599577+0000","last_fullsized":"2023-12-19T07:47:22.599577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.356659+0000","last_change":"2023-12-19T07:47:16.198423+0000","last_active":"2023-12-19T07:47:22.356659+0000","last_peered":"2023-12-19T07:47:22.356659+0000","last_clean":"2023-12-19T07:47:22.356659+0000","last_became_active":"2023-12-19T07:47:16.198088+0000","last_became_peered":"2023-12-19T07:47:16.198088+0000","last_unstale":"2023-12-19T07:47:22.356659+0000","last_undegraded":"2023-12-19T07:47:22.356659+0000","last_fullsized":"2023-12-19T07:47:22.356659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597749+0000","last_change":"2023-12-19T07:47:21.845056+0000","last_active":"2023-12-19T07:47:22.597749+0000","last_peered":"2023-12-19T07:47:22.597749+0000","last_clean":"2023-12-19T07:47:22.597749+0000","last_became_active":"2023-12-19T07:47:21.844217+0000","last_became_peered":"2023-12-19T07:47:21.844217+0000","last_unstale":"2023-12-19T07:47:22.597749+0000","last_undegraded":"2023-12-19T07:47:22.597749+0000","last_fullsized":"2023-12-19T07:47:22.597749+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:53:07.229783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352565+0000","last_change":"2023-12-19T07:47:17.200525+0000","last_active":"2023-12-19T07:47:22.352565+0000","last_peered":"2023-12-19T07:47:22.352565+0000","last_clean":"2023-12-19T07:47:22.352565+0000","last_became_active":"2023-12-19T07:47:17.199921+0000","last_became_peered":"2023-12-19T07:47:17.199921+0000","last_unstale":"2023-12-19T07:47:22.352565+0000","last_undegraded":"2023-12-19T07:47:22.352565+0000","last_fullsized":"2023-12-19T07:47:22.352565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868254+0000","last_change":"2023-12-19T07:47:16.213616+0000","last_active":"2023-12-19T07:47:22.868254+0000","last_peered":"2023-12-19T07:47:22.868254+0000","last_clean":"2023-12-19T07:47:22.868254+0000","last_became_active":"2023-12-19T07:47:16.213264+0000","last_became_peered":"2023-12-19T07:47:16.213264+0000","last_unstale":"2023-12-19T07:47:22.868254+0000","last_undegraded":"2023-12-19T07:47:22.868254+0000","last_fullsized":"2023-12-19T07:47:22.868254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902861+0000","last_change":"2023-12-19T07:47:21.841016+0000","last_active":"2023-12-19T07:47:22.902861+0000","last_peered":"2023-12-19T07:47:22.902861+0000","last_clean":"2023-12-19T07:47:22.902861+0000","last_became_active":"2023-12-19T07:47:21.840031+0000","last_became_peered":"2023-12-19T07:47:21.840031+0000","last_unstale":"2023-12-19T07:47:22.902861+0000","last_undegraded":"2023-12-19T07:47:22.902861+0000","last_fullsized":"2023-12-19T07:47:22.902861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807240+0000","last_change":"2023-12-19T07:47:17.205604+0000","last_active":"2023-12-19T07:47:22.807240+0000","last_peered":"2023-12-19T07:47:22.807240+0000","last_clean":"2023-12-19T07:47:22.807240+0000","last_became_active":"2023-12-19T07:47:17.205354+0000","last_became_peered":"2023-12-19T07:47:17.205354+0000","last_unstale":"2023-12-19T07:47:22.807240+0000","last_undegraded":"2023-12-19T07:47:22.807240+0000","last_fullsized":"2023-12-19T07:47:22.807240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352620+0000","last_change":"2023-12-19T07:47:16.201653+0000","last_active":"2023-12-19T07:47:22.352620+0000","last_peered":"2023-12-19T07:47:22.352620+0000","last_clean":"2023-12-19T07:47:22.352620+0000","last_became_active":"2023-12-19T07:47:16.201071+0000","last_became_peered":"2023-12-19T07:47:16.201071+0000","last_unstale":"2023-12-19T07:47:22.352620+0000","last_undegraded":"2023-12-19T07:47:22.352620+0000","last_fullsized":"2023-12-19T07:47:22.352620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351616+0000","last_change":"2023-12-19T07:47:21.262464+0000","last_active":"2023-12-19T07:47:22.351616+0000","last_peered":"2023-12-19T07:47:22.351616+0000","last_clean":"2023-12-19T07:47:22.351616+0000","last_became_active":"2023-12-19T07:47:21.259084+0000","last_became_peered":"2023-12-19T07:47:21.259084+0000","last_unstale":"2023-12-19T07:47:22.351616+0000","last_undegraded":"2023-12-19T07:47:22.351616+0000","last_fullsized":"2023-12-19T07:47:22.351616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307178+0000","last_change":"2023-12-19T07:47:17.212640+0000","last_active":"2023-12-19T07:47:22.307178+0000","last_peered":"2023-12-19T07:47:22.307178+0000","last_clean":"2023-12-19T07:47:22.307178+0000","last_became_active":"2023-12-19T07:47:17.212189+0000","last_became_peered":"2023-12-19T07:47:17.212189+0000","last_unstale":"2023-12-19T07:47:22.307178+0000","last_undegraded":"2023-12-19T07:47:22.307178+0000","last_fullsized":"2023-12-19T07:47:22.307178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806950+0000","last_change":"2023-12-19T07:47:16.201470+0000","last_active":"2023-12-19T07:47:22.806950+0000","last_peered":"2023-12-19T07:47:22.806950+0000","last_clean":"2023-12-19T07:47:22.806950+0000","last_became_active":"2023-12-19T07:47:16.200990+0000","last_became_peered":"2023-12-19T07:47:16.200990+0000","last_unstale":"2023-12-19T07:47:22.806950+0000","last_undegraded":"2023-12-19T07:47:22.806950+0000","last_fullsized":"2023-12-19T07:47:22.806950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352494+0000","last_change":"2023-12-19T07:47:21.261403+0000","last_active":"2023-12-19T07:47:22.352494+0000","last_peered":"2023-12-19T07:47:22.352494+0000","last_clean":"2023-12-19T07:47:22.352494+0000","last_became_active":"2023-12-19T07:47:21.260934+0000","last_became_peered":"2023-12-19T07:47:21.260934+0000","last_unstale":"2023-12-19T07:47:22.352494+0000","last_undegraded":"2023-12-19T07:47:22.352494+0000","last_fullsized":"2023-12-19T07:47:22.352494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351362+0000","last_change":"2023-12-19T07:47:17.196604+0000","last_active":"2023-12-19T07:47:22.351362+0000","last_peered":"2023-12-19T07:47:22.351362+0000","last_clean":"2023-12-19T07:47:22.351362+0000","last_became_active":"2023-12-19T07:47:17.195716+0000","last_became_peered":"2023-12-19T07:47:17.195716+0000","last_unstale":"2023-12-19T07:47:22.351362+0000","last_undegraded":"2023-12-19T07:47:22.351362+0000","last_fullsized":"2023-12-19T07:47:22.351362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302998+0000","last_change":"2023-12-19T07:47:16.188258+0000","last_active":"2023-12-19T07:47:22.302998+0000","last_peered":"2023-12-19T07:47:22.302998+0000","last_clean":"2023-12-19T07:47:22.302998+0000","last_became_active":"2023-12-19T07:47:16.187018+0000","last_became_peered":"2023-12-19T07:47:16.187018+0000","last_unstale":"2023-12-19T07:47:22.302998+0000","last_undegraded":"2023-12-19T07:47:22.302998+0000","last_fullsized":"2023-12-19T07:47:22.302998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353576+0000","last_change":"2023-12-19T07:47:21.269038+0000","last_active":"2023-12-19T07:47:22.353576+0000","last_peered":"2023-12-19T07:47:22.353576+0000","last_clean":"2023-12-19T07:47:22.353576+0000","last_became_active":"2023-12-19T07:47:21.268433+0000","last_became_peered":"2023-12-19T07:47:21.268433+0000","last_unstale":"2023-12-19T07:47:22.353576+0000","last_undegraded":"2023-12-19T07:47:22.353576+0000","last_fullsized":"2023-12-19T07:47:22.353576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903117+0000","last_change":"2023-12-19T07:47:17.201741+0000","last_active":"2023-12-19T07:47:22.903117+0000","last_peered":"2023-12-19T07:47:22.903117+0000","last_clean":"2023-12-19T07:47:22.903117+0000","last_became_active":"2023-12-19T07:47:17.201369+0000","last_became_peered":"2023-12-19T07:47:17.201369+0000","last_unstale":"2023-12-19T07:47:22.903117+0000","last_undegraded":"2023-12-19T07:47:22.903117+0000","last_fullsized":"2023-12-19T07:47:22.903117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866966+0000","last_change":"2023-12-19T07:47:16.213829+0000","last_active":"2023-12-19T07:47:22.866966+0000","last_peered":"2023-12-19T07:47:22.866966+0000","last_clean":"2023-12-19T07:47:22.866966+0000","last_became_active":"2023-12-19T07:47:16.213228+0000","last_became_peered":"2023-12-19T07:47:16.213228+0000","last_unstale":"2023-12-19T07:47:22.866966+0000","last_undegraded":"2023-12-19T07:47:22.866966+0000","last_fullsized":"2023-12-19T07:47:22.866966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902852+0000","last_change":"2023-12-19T07:47:21.841516+0000","last_active":"2023-12-19T07:47:22.902852+0000","last_peered":"2023-12-19T07:47:22.902852+0000","last_clean":"2023-12-19T07:47:22.902852+0000","last_became_active":"2023-12-19T07:47:21.840477+0000","last_became_peered":"2023-12-19T07:47:21.840477+0000","last_unstale":"2023-12-19T07:47:22.902852+0000","last_undegraded":"2023-12-19T07:47:22.902852+0000","last_fullsized":"2023-12-19T07:47:22.902852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598774+0000","last_change":"2023-12-19T07:47:17.200992+0000","last_active":"2023-12-19T07:47:22.598774+0000","last_peered":"2023-12-19T07:47:22.598774+0000","last_clean":"2023-12-19T07:47:22.598774+0000","last_became_active":"2023-12-19T07:47:17.199427+0000","last_became_peered":"2023-12-19T07:47:17.199427+0000","last_unstale":"2023-12-19T07:47:22.598774+0000","last_undegraded":"2023-12-19T07:47:22.598774+0000","last_fullsized":"2023-12-19T07:47:22.598774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806565+0000","last_change":"2023-12-19T07:47:16.201062+0000","last_active":"2023-12-19T07:47:22.806565+0000","last_peered":"2023-12-19T07:47:22.806565+0000","last_clean":"2023-12-19T07:47:22.806565+0000","last_became_active":"2023-12-19T07:47:16.199674+0000","last_became_peered":"2023-12-19T07:47:16.199674+0000","last_unstale":"2023-12-19T07:47:22.806565+0000","last_undegraded":"2023-12-19T07:47:22.806565+0000","last_fullsized":"2023-12-19T07:47:22.806565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:33.372367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867080+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:47:22.867080+0000","last_peered":"2023-12-19T07:47:22.867080+0000","last_clean":"2023-12-19T07:47:22.867080+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:47:22.867080+0000","last_undegraded":"2023-12-19T07:47:22.867080+0000","last_fullsized":"2023-12-19T07:47:22.867080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359462+0000","last_change":"2023-12-19T07:47:21.844050+0000","last_active":"2023-12-19T07:47:22.359462+0000","last_peered":"2023-12-19T07:47:22.359462+0000","last_clean":"2023-12-19T07:47:22.359462+0000","last_became_active":"2023-12-19T07:47:21.843480+0000","last_became_peered":"2023-12-19T07:47:21.843480+0000","last_unstale":"2023-12-19T07:47:22.359462+0000","last_undegraded":"2023-12-19T07:47:22.359462+0000","last_fullsized":"2023-12-19T07:47:22.359462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:29.237260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867932+0000","last_change":"2023-12-19T07:47:17.201683+0000","last_active":"2023-12-19T07:47:22.867932+0000","last_peered":"2023-12-19T07:47:22.867932+0000","last_clean":"2023-12-19T07:47:22.867932+0000","last_became_active":"2023-12-19T07:47:17.201138+0000","last_became_peered":"2023-12-19T07:47:17.201138+0000","last_unstale":"2023-12-19T07:47:22.867932+0000","last_undegraded":"2023-12-19T07:47:22.867932+0000","last_fullsized":"2023-12-19T07:47:22.867932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:25.176702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303850+0000","last_change":"2023-12-19T07:47:16.199670+0000","last_active":"2023-12-19T07:47:22.303850+0000","last_peered":"2023-12-19T07:47:22.303850+0000","last_clean":"2023-12-19T07:47:22.303850+0000","last_became_active":"2023-12-19T07:47:16.199236+0000","last_became_peered":"2023-12-19T07:47:16.199236+0000","last_unstale":"2023-12-19T07:47:22.303850+0000","last_undegraded":"2023-12-19T07:47:22.303850+0000","last_fullsized":"2023-12-19T07:47:22.303850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358653+0000","last_change":"2023-12-19T07:47:21.844306+0000","last_active":"2023-12-19T07:47:22.358653+0000","last_peered":"2023-12-19T07:47:22.358653+0000","last_clean":"2023-12-19T07:47:22.358653+0000","last_became_active":"2023-12-19T07:47:21.843694+0000","last_became_peered":"2023-12-19T07:47:21.843694+0000","last_unstale":"2023-12-19T07:47:22.358653+0000","last_undegraded":"2023-12-19T07:47:22.358653+0000","last_fullsized":"2023-12-19T07:47:22.358653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:25.534670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806965+0000","last_change":"2023-12-19T07:47:17.210076+0000","last_active":"2023-12-19T07:47:22.806965+0000","last_peered":"2023-12-19T07:47:22.806965+0000","last_clean":"2023-12-19T07:47:22.806965+0000","last_became_active":"2023-12-19T07:47:17.209871+0000","last_became_peered":"2023-12-19T07:47:17.209871+0000","last_unstale":"2023-12-19T07:47:22.806965+0000","last_undegraded":"2023-12-19T07:47:22.806965+0000","last_fullsized":"2023-12-19T07:47:22.806965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353351+0000","last_change":"2023-12-19T07:47:16.187382+0000","last_active":"2023-12-19T07:47:22.353351+0000","last_peered":"2023-12-19T07:47:22.353351+0000","last_clean":"2023-12-19T07:47:22.353351+0000","last_became_active":"2023-12-19T07:47:16.187021+0000","last_became_peered":"2023-12-19T07:47:16.187021+0000","last_unstale":"2023-12-19T07:47:22.353351+0000","last_undegraded":"2023-12-19T07:47:22.353351+0000","last_fullsized":"2023-12-19T07:47:22.353351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:03.039957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903413+0000","last_change":"2023-12-19T07:47:21.265329+0000","last_active":"2023-12-19T07:47:22.903413+0000","last_peered":"2023-12-19T07:47:22.903413+0000","last_clean":"2023-12-19T07:47:22.903413+0000","last_became_active":"2023-12-19T07:47:21.264912+0000","last_became_peered":"2023-12-19T07:47:21.264912+0000","last_unstale":"2023-12-19T07:47:22.903413+0000","last_undegraded":"2023-12-19T07:47:22.903413+0000","last_fullsized":"2023-12-19T07:47:22.903413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868427+0000","last_change":"2023-12-19T07:47:17.215646+0000","last_active":"2023-12-19T07:47:22.868427+0000","last_peered":"2023-12-19T07:47:22.868427+0000","last_clean":"2023-12-19T07:47:22.868427+0000","last_became_active":"2023-12-19T07:47:17.215345+0000","last_became_peered":"2023-12-19T07:47:17.215345+0000","last_unstale":"2023-12-19T07:47:22.868427+0000","last_undegraded":"2023-12-19T07:47:22.868427+0000","last_fullsized":"2023-12-19T07:47:22.868427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:05.660865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304403+0000","last_change":"2023-12-19T07:47:16.195752+0000","last_active":"2023-12-19T07:47:22.304403+0000","last_peered":"2023-12-19T07:47:22.304403+0000","last_clean":"2023-12-19T07:47:22.304403+0000","last_became_active":"2023-12-19T07:47:16.195415+0000","last_became_peered":"2023-12-19T07:47:16.195415+0000","last_unstale":"2023-12-19T07:47:22.304403+0000","last_undegraded":"2023-12-19T07:47:22.304403+0000","last_fullsized":"2023-12-19T07:47:22.304403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805409+0000","last_change":"2023-12-19T07:47:21.840195+0000","last_active":"2023-12-19T07:47:22.805409+0000","last_peered":"2023-12-19T07:47:22.805409+0000","last_clean":"2023-12-19T07:47:22.805409+0000","last_became_active":"2023-12-19T07:47:21.839557+0000","last_became_peered":"2023-12-19T07:47:21.839557+0000","last_unstale":"2023-12-19T07:47:22.805409+0000","last_undegraded":"2023-12-19T07:47:22.805409+0000","last_fullsized":"2023-12-19T07:47:22.805409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867970+0000","last_change":"2023-12-19T07:47:17.211241+0000","last_active":"2023-12-19T07:47:22.867970+0000","last_peered":"2023-12-19T07:47:22.867970+0000","last_clean":"2023-12-19T07:47:22.867970+0000","last_became_active":"2023-12-19T07:47:17.210508+0000","last_became_peered":"2023-12-19T07:47:17.210508+0000","last_unstale":"2023-12-19T07:47:22.867970+0000","last_undegraded":"2023-12-19T07:47:22.867970+0000","last_fullsized":"2023-12-19T07:47:22.867970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:48.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307381+0000","last_change":"2023-12-19T07:47:16.198483+0000","last_active":"2023-12-19T07:47:22.307381+0000","last_peered":"2023-12-19T07:47:22.307381+0000","last_clean":"2023-12-19T07:47:22.307381+0000","last_became_active":"2023-12-19T07:47:16.198150+0000","last_became_peered":"2023-12-19T07:47:16.198150+0000","last_unstale":"2023-12-19T07:47:22.307381+0000","last_undegraded":"2023-12-19T07:47:22.307381+0000","last_fullsized":"2023-12-19T07:47:22.307381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306109+0000","last_change":"2023-12-19T07:47:21.259856+0000","last_active":"2023-12-19T07:47:22.306109+0000","last_peered":"2023-12-19T07:47:22.306109+0000","last_clean":"2023-12-19T07:47:22.306109+0000","last_became_active":"2023-12-19T07:47:21.259147+0000","last_became_peered":"2023-12-19T07:47:21.259147+0000","last_unstale":"2023-12-19T07:47:22.306109+0000","last_undegraded":"2023-12-19T07:47:22.306109+0000","last_fullsized":"2023-12-19T07:47:22.306109+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867790+0000","last_change":"2023-12-19T07:47:17.209489+0000","last_active":"2023-12-19T07:47:22.867790+0000","last_peered":"2023-12-19T07:47:22.867790+0000","last_clean":"2023-12-19T07:47:22.867790+0000","last_became_active":"2023-12-19T07:47:17.209131+0000","last_became_peered":"2023-12-19T07:47:17.209131+0000","last_unstale":"2023-12-19T07:47:22.867790+0000","last_undegraded":"2023-12-19T07:47:22.867790+0000","last_fullsized":"2023-12-19T07:47:22.867790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303258+0000","last_change":"2023-12-19T07:47:16.192274+0000","last_active":"2023-12-19T07:47:22.303258+0000","last_peered":"2023-12-19T07:47:22.303258+0000","last_clean":"2023-12-19T07:47:22.303258+0000","last_became_active":"2023-12-19T07:47:16.190777+0000","last_became_peered":"2023-12-19T07:47:16.190777+0000","last_unstale":"2023-12-19T07:47:22.303258+0000","last_undegraded":"2023-12-19T07:47:22.303258+0000","last_fullsized":"2023-12-19T07:47:22.303258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904035+0000","last_change":"2023-12-19T07:47:21.260731+0000","last_active":"2023-12-19T07:47:22.904035+0000","last_peered":"2023-12-19T07:47:22.904035+0000","last_clean":"2023-12-19T07:47:22.904035+0000","last_became_active":"2023-12-19T07:47:21.258011+0000","last_became_peered":"2023-12-19T07:47:21.258011+0000","last_unstale":"2023-12-19T07:47:22.904035+0000","last_undegraded":"2023-12-19T07:47:22.904035+0000","last_fullsized":"2023-12-19T07:47:22.904035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359125+0000","last_change":"2023-12-19T07:47:17.201434+0000","last_active":"2023-12-19T07:47:22.359125+0000","last_peered":"2023-12-19T07:47:22.359125+0000","last_clean":"2023-12-19T07:47:22.359125+0000","last_became_active":"2023-12-19T07:47:17.200473+0000","last_became_peered":"2023-12-19T07:47:17.200473+0000","last_unstale":"2023-12-19T07:47:22.359125+0000","last_undegraded":"2023-12-19T07:47:22.359125+0000","last_fullsized":"2023-12-19T07:47:22.359125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359065+0000","last_change":"2023-12-19T07:47:16.193445+0000","last_active":"2023-12-19T07:47:22.359065+0000","last_peered":"2023-12-19T07:47:22.359065+0000","last_clean":"2023-12-19T07:47:22.359065+0000","last_became_active":"2023-12-19T07:47:16.192600+0000","last_became_peered":"2023-12-19T07:47:16.192600+0000","last_unstale":"2023-12-19T07:47:22.359065+0000","last_undegraded":"2023-12-19T07:47:22.359065+0000","last_fullsized":"2023-12-19T07:47:22.359065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597903+0000","last_change":"2023-12-19T07:47:21.847662+0000","last_active":"2023-12-19T07:47:22.597903+0000","last_peered":"2023-12-19T07:47:22.597903+0000","last_clean":"2023-12-19T07:47:22.597903+0000","last_became_active":"2023-12-19T07:47:21.847110+0000","last_became_peered":"2023-12-19T07:47:21.847110+0000","last_unstale":"2023-12-19T07:47:22.597903+0000","last_undegraded":"2023-12-19T07:47:22.597903+0000","last_fullsized":"2023-12-19T07:47:22.597903+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:07:23.838326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867579+0000","last_change":"2023-12-19T07:47:17.219151+0000","last_active":"2023-12-19T07:47:22.867579+0000","last_peered":"2023-12-19T07:47:22.867579+0000","last_clean":"2023-12-19T07:47:22.867579+0000","last_became_active":"2023-12-19T07:47:17.218940+0000","last_became_peered":"2023-12-19T07:47:17.218940+0000","last_unstale":"2023-12-19T07:47:22.867579+0000","last_undegraded":"2023-12-19T07:47:22.867579+0000","last_fullsized":"2023-12-19T07:47:22.867579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806721+0000","last_change":"2023-12-19T07:47:16.201106+0000","last_active":"2023-12-19T07:47:22.806721+0000","last_peered":"2023-12-19T07:47:22.806721+0000","last_clean":"2023-12-19T07:47:22.806721+0000","last_became_active":"2023-12-19T07:47:16.200628+0000","last_became_peered":"2023-12-19T07:47:16.200628+0000","last_unstale":"2023-12-19T07:47:22.806721+0000","last_undegraded":"2023-12-19T07:47:22.806721+0000","last_fullsized":"2023-12-19T07:47:22.806721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597986+0000","last_change":"2023-12-19T07:47:21.847256+0000","last_active":"2023-12-19T07:47:22.597986+0000","last_peered":"2023-12-19T07:47:22.597986+0000","last_clean":"2023-12-19T07:47:22.597986+0000","last_became_active":"2023-12-19T07:47:21.843240+0000","last_became_peered":"2023-12-19T07:47:21.843240+0000","last_unstale":"2023-12-19T07:47:22.597986+0000","last_undegraded":"2023-12-19T07:47:22.597986+0000","last_fullsized":"2023-12-19T07:47:22.597986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:45.450328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903865+0000","last_change":"2023-12-19T07:47:17.200704+0000","last_active":"2023-12-19T07:47:22.903865+0000","last_peered":"2023-12-19T07:47:22.903865+0000","last_clean":"2023-12-19T07:47:22.903865+0000","last_became_active":"2023-12-19T07:47:17.200382+0000","last_became_peered":"2023-12-19T07:47:17.200382+0000","last_unstale":"2023-12-19T07:47:22.903865+0000","last_undegraded":"2023-12-19T07:47:22.903865+0000","last_fullsized":"2023-12-19T07:47:22.903865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903656+0000","last_change":"2023-12-19T07:47:16.189068+0000","last_active":"2023-12-19T07:47:22.903656+0000","last_peered":"2023-12-19T07:47:22.903656+0000","last_clean":"2023-12-19T07:47:22.903656+0000","last_became_active":"2023-12-19T07:47:16.188798+0000","last_became_peered":"2023-12-19T07:47:16.188798+0000","last_unstale":"2023-12-19T07:47:22.903656+0000","last_undegraded":"2023-12-19T07:47:22.903656+0000","last_fullsized":"2023-12-19T07:47:22.903656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805530+0000","last_change":"2023-12-19T07:47:21.847444+0000","last_active":"2023-12-19T07:47:22.805530+0000","last_peered":"2023-12-19T07:47:22.805530+0000","last_clean":"2023-12-19T07:47:22.805530+0000","last_became_active":"2023-12-19T07:47:21.847123+0000","last_became_peered":"2023-12-19T07:47:21.847123+0000","last_unstale":"2023-12-19T07:47:22.805530+0000","last_undegraded":"2023-12-19T07:47:22.805530+0000","last_fullsized":"2023-12-19T07:47:22.805530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867681+0000","last_change":"2023-12-19T07:47:17.212222+0000","last_active":"2023-12-19T07:47:22.867681+0000","last_peered":"2023-12-19T07:47:22.867681+0000","last_clean":"2023-12-19T07:47:22.867681+0000","last_became_active":"2023-12-19T07:47:17.211536+0000","last_became_peered":"2023-12-19T07:47:17.211536+0000","last_unstale":"2023-12-19T07:47:22.867681+0000","last_undegraded":"2023-12-19T07:47:22.867681+0000","last_fullsized":"2023-12-19T07:47:22.867681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:40.976961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-19T07:47:29.088364+0000","last_change":"2023-12-19T07:47:16.200498+0000","last_active":"2023-12-19T07:47:29.088364+0000","last_peered":"2023-12-19T07:47:29.088364+0000","last_clean":"2023-12-19T07:47:29.088364+0000","last_became_active":"2023-12-19T07:47:16.200159+0000","last_became_peered":"2023-12-19T07:47:16.200159+0000","last_unstale":"2023-12-19T07:47:29.088364+0000","last_undegraded":"2023-12-19T07:47:29.088364+0000","last_fullsized":"2023-12-19T07:47:29.088364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351798+0000","last_change":"2023-12-19T07:47:21.269837+0000","last_active":"2023-12-19T07:47:22.351798+0000","last_peered":"2023-12-19T07:47:22.351798+0000","last_clean":"2023-12-19T07:47:22.351798+0000","last_became_active":"2023-12-19T07:47:21.269371+0000","last_became_peered":"2023-12-19T07:47:21.269371+0000","last_unstale":"2023-12-19T07:47:22.351798+0000","last_undegraded":"2023-12-19T07:47:22.351798+0000","last_fullsized":"2023-12-19T07:47:22.351798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307009+0000","last_change":"2023-12-19T07:47:17.206123+0000","last_active":"2023-12-19T07:47:22.307009+0000","last_peered":"2023-12-19T07:47:22.307009+0000","last_clean":"2023-12-19T07:47:22.307009+0000","last_became_active":"2023-12-19T07:47:17.202604+0000","last_became_peered":"2023-12-19T07:47:17.202604+0000","last_unstale":"2023-12-19T07:47:22.307009+0000","last_undegraded":"2023-12-19T07:47:22.307009+0000","last_fullsized":"2023-12-19T07:47:22.307009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353492+0000","last_change":"2023-12-19T07:47:16.207087+0000","last_active":"2023-12-19T07:47:22.353492+0000","last_peered":"2023-12-19T07:47:22.353492+0000","last_clean":"2023-12-19T07:47:22.353492+0000","last_became_active":"2023-12-19T07:47:16.205832+0000","last_became_peered":"2023-12-19T07:47:16.205832+0000","last_unstale":"2023-12-19T07:47:22.353492+0000","last_undegraded":"2023-12-19T07:47:22.353492+0000","last_fullsized":"2023-12-19T07:47:22.353492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307124+0000","last_change":"2023-12-19T07:47:21.259755+0000","last_active":"2023-12-19T07:47:22.307124+0000","last_peered":"2023-12-19T07:47:22.307124+0000","last_clean":"2023-12-19T07:47:22.307124+0000","last_became_active":"2023-12-19T07:47:21.259163+0000","last_became_peered":"2023-12-19T07:47:21.259163+0000","last_unstale":"2023-12-19T07:47:22.307124+0000","last_undegraded":"2023-12-19T07:47:22.307124+0000","last_fullsized":"2023-12-19T07:47:22.307124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904807+0000","last_change":"2023-12-19T07:47:17.202501+0000","last_active":"2023-12-19T07:47:22.904807+0000","last_peered":"2023-12-19T07:47:22.904807+0000","last_clean":"2023-12-19T07:47:22.904807+0000","last_became_active":"2023-12-19T07:47:17.199344+0000","last_became_peered":"2023-12-19T07:47:17.199344+0000","last_unstale":"2023-12-19T07:47:22.904807+0000","last_undegraded":"2023-12-19T07:47:22.904807+0000","last_fullsized":"2023-12-19T07:47:22.904807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807353+0000","last_change":"2023-12-19T07:47:16.200080+0000","last_active":"2023-12-19T07:47:22.807353+0000","last_peered":"2023-12-19T07:47:22.807353+0000","last_clean":"2023-12-19T07:47:22.807353+0000","last_became_active":"2023-12-19T07:47:16.199182+0000","last_became_peered":"2023-12-19T07:47:16.199182+0000","last_unstale":"2023-12-19T07:47:22.807353+0000","last_undegraded":"2023-12-19T07:47:22.807353+0000","last_fullsized":"2023-12-19T07:47:22.807353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598231+0000","last_change":"2023-12-19T07:47:21.264957+0000","last_active":"2023-12-19T07:47:22.598231+0000","last_peered":"2023-12-19T07:47:22.598231+0000","last_clean":"2023-12-19T07:47:22.598231+0000","last_became_active":"2023-12-19T07:47:21.264514+0000","last_became_peered":"2023-12-19T07:47:21.264514+0000","last_unstale":"2023-12-19T07:47:22.598231+0000","last_undegraded":"2023-12-19T07:47:22.598231+0000","last_fullsized":"2023-12-19T07:47:22.598231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:20.377759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807125+0000","last_change":"2023-12-19T07:47:17.199047+0000","last_active":"2023-12-19T07:47:22.807125+0000","last_peered":"2023-12-19T07:47:22.807125+0000","last_clean":"2023-12-19T07:47:22.807125+0000","last_became_active":"2023-12-19T07:47:17.198760+0000","last_became_peered":"2023-12-19T07:47:17.198760+0000","last_unstale":"2023-12-19T07:47:22.807125+0000","last_undegraded":"2023-12-19T07:47:22.807125+0000","last_fullsized":"2023-12-19T07:47:22.807125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359707+0000","last_change":"2023-12-19T07:47:16.206922+0000","last_active":"2023-12-19T07:47:22.359707+0000","last_peered":"2023-12-19T07:47:22.359707+0000","last_clean":"2023-12-19T07:47:22.359707+0000","last_became_active":"2023-12-19T07:47:16.206605+0000","last_became_peered":"2023-12-19T07:47:16.206605+0000","last_unstale":"2023-12-19T07:47:22.359707+0000","last_undegraded":"2023-12-19T07:47:22.359707+0000","last_fullsized":"2023-12-19T07:47:22.359707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597807+0000","last_change":"2023-12-19T07:47:21.845593+0000","last_active":"2023-12-19T07:47:22.597807+0000","last_peered":"2023-12-19T07:47:22.597807+0000","last_clean":"2023-12-19T07:47:22.597807+0000","last_became_active":"2023-12-19T07:47:21.843663+0000","last_became_peered":"2023-12-19T07:47:21.843663+0000","last_unstale":"2023-12-19T07:47:22.597807+0000","last_undegraded":"2023-12-19T07:47:22.597807+0000","last_fullsized":"2023-12-19T07:47:22.597807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902979+0000","last_change":"2023-12-19T07:47:21.846558+0000","last_active":"2023-12-19T07:47:22.902979+0000","last_peered":"2023-12-19T07:47:22.902979+0000","last_clean":"2023-12-19T07:47:22.902979+0000","last_became_active":"2023-12-19T07:47:21.845861+0000","last_became_peered":"2023-12-19T07:47:21.845861+0000","last_unstale":"2023-12-19T07:47:22.902979+0000","last_undegraded":"2023-12-19T07:47:22.902979+0000","last_fullsized":"2023-12-19T07:47:22.902979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:28.389344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351325+0000","last_change":"2023-12-19T07:47:17.210958+0000","last_active":"2023-12-19T07:47:22.351325+0000","last_peered":"2023-12-19T07:47:22.351325+0000","last_clean":"2023-12-19T07:47:22.351325+0000","last_became_active":"2023-12-19T07:47:17.210537+0000","last_became_peered":"2023-12-19T07:47:17.210537+0000","last_unstale":"2023-12-19T07:47:22.351325+0000","last_undegraded":"2023-12-19T07:47:22.351325+0000","last_fullsized":"2023-12-19T07:47:22.351325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805998+0000","last_change":"2023-12-19T07:47:16.198817+0000","last_active":"2023-12-19T07:47:22.805998+0000","last_peered":"2023-12-19T07:47:22.805998+0000","last_clean":"2023-12-19T07:47:22.805998+0000","last_became_active":"2023-12-19T07:47:16.198491+0000","last_became_peered":"2023-12-19T07:47:16.198491+0000","last_unstale":"2023-12-19T07:47:22.805998+0000","last_undegraded":"2023-12-19T07:47:22.805998+0000","last_fullsized":"2023-12-19T07:47:22.805998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359608+0000","last_change":"2023-12-19T07:47:21.263980+0000","last_active":"2023-12-19T07:47:22.359608+0000","last_peered":"2023-12-19T07:47:22.359608+0000","last_clean":"2023-12-19T07:47:22.359608+0000","last_became_active":"2023-12-19T07:47:21.263310+0000","last_became_peered":"2023-12-19T07:47:21.263310+0000","last_unstale":"2023-12-19T07:47:22.359608+0000","last_undegraded":"2023-12-19T07:47:22.359608+0000","last_fullsized":"2023-12-19T07:47:22.359608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.138125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598272+0000","last_change":"2023-12-19T07:47:17.196870+0000","last_active":"2023-12-19T07:47:22.598272+0000","last_peered":"2023-12-19T07:47:22.598272+0000","last_clean":"2023-12-19T07:47:22.598272+0000","last_became_active":"2023-12-19T07:47:17.196588+0000","last_became_peered":"2023-12-19T07:47:17.196588+0000","last_unstale":"2023-12-19T07:47:22.598272+0000","last_undegraded":"2023-12-19T07:47:22.598272+0000","last_fullsized":"2023-12-19T07:47:22.598272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598626+0000","last_change":"2023-12-19T07:47:16.198918+0000","last_active":"2023-12-19T07:47:22.598626+0000","last_peered":"2023-12-19T07:47:22.598626+0000","last_clean":"2023-12-19T07:47:22.598626+0000","last_became_active":"2023-12-19T07:47:16.198496+0000","last_became_peered":"2023-12-19T07:47:16.198496+0000","last_unstale":"2023-12-19T07:47:22.598626+0000","last_undegraded":"2023-12-19T07:47:22.598626+0000","last_fullsized":"2023-12-19T07:47:22.598626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358925+0000","last_change":"2023-12-19T07:47:16.192976+0000","last_active":"2023-12-19T07:47:22.358925+0000","last_peered":"2023-12-19T07:47:22.358925+0000","last_clean":"2023-12-19T07:47:22.358925+0000","last_became_active":"2023-12-19T07:47:16.192588+0000","last_became_peered":"2023-12-19T07:47:16.192588+0000","last_unstale":"2023-12-19T07:47:22.358925+0000","last_undegraded":"2023-12-19T07:47:22.358925+0000","last_fullsized":"2023-12-19T07:47:22.358925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903549+0000","last_change":"2023-12-19T07:47:17.205110+0000","last_active":"2023-12-19T07:47:22.903549+0000","last_peered":"2023-12-19T07:47:22.903549+0000","last_clean":"2023-12-19T07:47:22.903549+0000","last_became_active":"2023-12-19T07:47:17.204811+0000","last_became_peered":"2023-12-19T07:47:17.204811+0000","last_unstale":"2023-12-19T07:47:22.903549+0000","last_undegraded":"2023-12-19T07:47:22.903549+0000","last_fullsized":"2023-12-19T07:47:22.903549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:07.078408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303570+0000","last_change":"2023-12-19T07:47:21.259361+0000","last_active":"2023-12-19T07:47:22.303570+0000","last_peered":"2023-12-19T07:47:22.303570+0000","last_clean":"2023-12-19T07:47:22.303570+0000","last_became_active":"2023-12-19T07:47:21.258883+0000","last_became_peered":"2023-12-19T07:47:21.258883+0000","last_unstale":"2023-12-19T07:47:22.303570+0000","last_undegraded":"2023-12-19T07:47:22.303570+0000","last_fullsized":"2023-12-19T07:47:22.303570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:28.728929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598981+0000","last_change":"2023-12-19T07:47:16.206769+0000","last_active":"2023-12-19T07:47:22.598981+0000","last_peered":"2023-12-19T07:47:22.598981+0000","last_clean":"2023-12-19T07:47:22.598981+0000","last_became_active":"2023-12-19T07:47:16.206392+0000","last_became_peered":"2023-12-19T07:47:16.206392+0000","last_unstale":"2023-12-19T07:47:22.598981+0000","last_undegraded":"2023-12-19T07:47:22.598981+0000","last_fullsized":"2023-12-19T07:47:22.598981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306430+0000","last_change":"2023-12-19T07:47:17.204694+0000","last_active":"2023-12-19T07:47:22.306430+0000","last_peered":"2023-12-19T07:47:22.306430+0000","last_clean":"2023-12-19T07:47:22.306430+0000","last_became_active":"2023-12-19T07:47:17.204276+0000","last_became_peered":"2023-12-19T07:47:17.204276+0000","last_unstale":"2023-12-19T07:47:22.306430+0000","last_undegraded":"2023-12-19T07:47:22.306430+0000","last_fullsized":"2023-12-19T07:47:22.306430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599471+0000","last_change":"2023-12-19T07:47:21.258311+0000","last_active":"2023-12-19T07:47:22.599471+0000","last_peered":"2023-12-19T07:47:22.599471+0000","last_clean":"2023-12-19T07:47:22.599471+0000","last_became_active":"2023-12-19T07:47:21.257510+0000","last_became_peered":"2023-12-19T07:47:21.257510+0000","last_unstale":"2023-12-19T07:47:22.599471+0000","last_undegraded":"2023-12-19T07:47:22.599471+0000","last_fullsized":"2023-12-19T07:47:22.599471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:15:44.535034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358817+0000","last_change":"2023-12-19T07:47:17.204892+0000","last_active":"2023-12-19T07:47:22.358817+0000","last_peered":"2023-12-19T07:47:22.358817+0000","last_clean":"2023-12-19T07:47:22.358817+0000","last_became_active":"2023-12-19T07:47:17.201155+0000","last_became_peered":"2023-12-19T07:47:17.201155+0000","last_unstale":"2023-12-19T07:47:22.358817+0000","last_undegraded":"2023-12-19T07:47:22.358817+0000","last_fullsized":"2023-12-19T07:47:22.358817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302909+0000","last_change":"2023-12-19T07:47:17.204307+0000","last_active":"2023-12-19T07:47:22.302909+0000","last_peered":"2023-12-19T07:47:22.302909+0000","last_clean":"2023-12-19T07:47:22.302909+0000","last_became_active":"2023-12-19T07:47:17.203914+0000","last_became_peered":"2023-12-19T07:47:17.203914+0000","last_unstale":"2023-12-19T07:47:22.302909+0000","last_undegraded":"2023-12-19T07:47:22.302909+0000","last_fullsized":"2023-12-19T07:47:22.302909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806686+0000","last_change":"2023-12-19T07:47:17.196779+0000","last_active":"2023-12-19T07:47:22.806686+0000","last_peered":"2023-12-19T07:47:22.806686+0000","last_clean":"2023-12-19T07:47:22.806686+0000","last_became_active":"2023-12-19T07:47:17.196494+0000","last_became_peered":"2023-12-19T07:47:17.196494+0000","last_unstale":"2023-12-19T07:47:22.806686+0000","last_undegraded":"2023-12-19T07:47:22.806686+0000","last_fullsized":"2023-12-19T07:47:22.806686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903605+0000","last_change":"2023-12-19T07:47:17.205970+0000","last_active":"2023-12-19T07:47:22.903605+0000","last_peered":"2023-12-19T07:47:22.903605+0000","last_clean":"2023-12-19T07:47:22.903605+0000","last_became_active":"2023-12-19T07:47:17.205581+0000","last_became_peered":"2023-12-19T07:47:17.205581+0000","last_unstale":"2023-12-19T07:47:22.903605+0000","last_undegraded":"2023-12-19T07:47:22.903605+0000","last_fullsized":"2023-12-19T07:47:22.903605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599435+0000","last_change":"2023-12-19T07:47:17.208151+0000","last_active":"2023-12-19T07:47:22.599435+0000","last_peered":"2023-12-19T07:47:22.599435+0000","last_clean":"2023-12-19T07:47:22.599435+0000","last_became_active":"2023-12-19T07:47:17.207880+0000","last_became_peered":"2023-12-19T07:47:17.207880+0000","last_unstale":"2023-12-19T07:47:22.599435+0000","last_undegraded":"2023-12-19T07:47:22.599435+0000","last_fullsized":"2023-12-19T07:47:22.599435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352831+0000","last_change":"2023-12-19T07:47:17.209159+0000","last_active":"2023-12-19T07:47:22.352831+0000","last_peered":"2023-12-19T07:47:22.352831+0000","last_clean":"2023-12-19T07:47:22.352831+0000","last_became_active":"2023-12-19T07:47:17.208870+0000","last_became_peered":"2023-12-19T07:47:17.208870+0000","last_unstale":"2023-12-19T07:47:22.352831+0000","last_undegraded":"2023-12-19T07:47:22.352831+0000","last_fullsized":"2023-12-19T07:47:22.352831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.360063+0000","last_change":"2023-12-19T07:47:17.209320+0000","last_active":"2023-12-19T07:47:22.360063+0000","last_peered":"2023-12-19T07:47:22.360063+0000","last_clean":"2023-12-19T07:47:22.360063+0000","last_became_active":"2023-12-19T07:47:17.208263+0000","last_became_peered":"2023-12-19T07:47:17.208263+0000","last_unstale":"2023-12-19T07:47:22.360063+0000","last_undegraded":"2023-12-19T07:47:22.360063+0000","last_fullsized":"2023-12-19T07:47:22.360063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599483+0000","last_change":"2023-12-19T07:47:17.211203+0000","last_active":"2023-12-19T07:47:22.599483+0000","last_peered":"2023-12-19T07:47:22.599483+0000","last_clean":"2023-12-19T07:47:22.599483+0000","last_became_active":"2023-12-19T07:47:17.210860+0000","last_became_peered":"2023-12-19T07:47:17.210860+0000","last_unstale":"2023-12-19T07:47:22.599483+0000","last_undegraded":"2023-12-19T07:47:22.599483+0000","last_fullsized":"2023-12-19T07:47:22.599483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805820+0000","last_change":"2023-12-19T07:47:17.210725+0000","last_active":"2023-12-19T07:47:22.805820+0000","last_peered":"2023-12-19T07:47:22.805820+0000","last_clean":"2023-12-19T07:47:22.805820+0000","last_became_active":"2023-12-19T07:47:17.210432+0000","last_became_peered":"2023-12-19T07:47:17.210432+0000","last_unstale":"2023-12-19T07:47:22.805820+0000","last_undegraded":"2023-12-19T07:47:22.805820+0000","last_fullsized":"2023-12-19T07:47:22.805820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599063+0000","last_change":"2023-12-19T07:47:17.200440+0000","last_active":"2023-12-19T07:47:22.599063+0000","last_peered":"2023-12-19T07:47:22.599063+0000","last_clean":"2023-12-19T07:47:22.599063+0000","last_became_active":"2023-12-19T07:47:17.200137+0000","last_became_peered":"2023-12-19T07:47:17.200137+0000","last_unstale":"2023-12-19T07:47:22.599063+0000","last_undegraded":"2023-12-19T07:47:22.599063+0000","last_fullsized":"2023-12-19T07:47:22.599063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806350+0000","last_change":"2023-12-19T07:47:17.200217+0000","last_active":"2023-12-19T07:47:22.806350+0000","last_peered":"2023-12-19T07:47:22.806350+0000","last_clean":"2023-12-19T07:47:22.806350+0000","last_became_active":"2023-12-19T07:47:17.199920+0000","last_became_peered":"2023-12-19T07:47:17.199920+0000","last_unstale":"2023-12-19T07:47:22.806350+0000","last_undegraded":"2023-12-19T07:47:22.806350+0000","last_fullsized":"2023-12-19T07:47:22.806350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903403+0000","last_change":"2023-12-19T07:47:17.200705+0000","last_active":"2023-12-19T07:47:22.903403+0000","last_peered":"2023-12-19T07:47:22.903403+0000","last_clean":"2023-12-19T07:47:22.903403+0000","last_became_active":"2023-12-19T07:47:17.200462+0000","last_became_peered":"2023-12-19T07:47:17.200462+0000","last_unstale":"2023-12-19T07:47:22.903403+0000","last_undegraded":"2023-12-19T07:47:22.903403+0000","last_fullsized":"2023-12-19T07:47:22.903403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359431+0000","last_change":"2023-12-19T07:47:17.212064+0000","last_active":"2023-12-19T07:47:22.359431+0000","last_peered":"2023-12-19T07:47:22.359431+0000","last_clean":"2023-12-19T07:47:22.359431+0000","last_became_active":"2023-12-19T07:47:17.211684+0000","last_became_peered":"2023-12-19T07:47:17.211684+0000","last_unstale":"2023-12-19T07:47:22.359431+0000","last_undegraded":"2023-12-19T07:47:22.359431+0000","last_fullsized":"2023-12-19T07:47:22.359431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:21.908290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868184+0000","last_change":"2023-12-19T07:47:17.212402+0000","last_active":"2023-12-19T07:47:22.868184+0000","last_peered":"2023-12-19T07:47:22.868184+0000","last_clean":"2023-12-19T07:47:22.868184+0000","last_became_active":"2023-12-19T07:47:17.211887+0000","last_became_peered":"2023-12-19T07:47:17.211887+0000","last_unstale":"2023-12-19T07:47:22.868184+0000","last_undegraded":"2023-12-19T07:47:22.868184+0000","last_fullsized":"2023-12-19T07:47:22.868184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304186+0000","last_change":"2023-12-19T07:47:17.207392+0000","last_active":"2023-12-19T07:47:22.304186+0000","last_peered":"2023-12-19T07:47:22.304186+0000","last_clean":"2023-12-19T07:47:22.304186+0000","last_became_active":"2023-12-19T07:47:17.206962+0000","last_became_peered":"2023-12-19T07:47:17.206962+0000","last_unstale":"2023-12-19T07:47:22.304186+0000","last_undegraded":"2023-12-19T07:47:22.304186+0000","last_fullsized":"2023-12-19T07:47:22.304186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807623+0000","last_change":"2023-12-19T07:47:17.217408+0000","last_active":"2023-12-19T07:47:22.807623+0000","last_peered":"2023-12-19T07:47:22.807623+0000","last_clean":"2023-12-19T07:47:22.807623+0000","last_became_active":"2023-12-19T07:47:17.217038+0000","last_became_peered":"2023-12-19T07:47:17.217038+0000","last_unstale":"2023-12-19T07:47:22.807623+0000","last_undegraded":"2023-12-19T07:47:22.807623+0000","last_fullsized":"2023-12-19T07:47:22.807623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:12.732155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805776+0000","last_change":"2023-12-19T07:47:17.194768+0000","last_active":"2023-12-19T07:47:22.805776+0000","last_peered":"2023-12-19T07:47:22.805776+0000","last_clean":"2023-12-19T07:47:22.805776+0000","last_became_active":"2023-12-19T07:47:17.194457+0000","last_became_peered":"2023-12-19T07:47:17.194457+0000","last_unstale":"2023-12-19T07:47:22.805776+0000","last_undegraded":"2023-12-19T07:47:22.805776+0000","last_fullsized":"2023-12-19T07:47:22.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351363+0000","last_change":"2023-12-19T07:47:17.209052+0000","last_active":"2023-12-19T07:47:22.351363+0000","last_peered":"2023-12-19T07:47:22.351363+0000","last_clean":"2023-12-19T07:47:22.351363+0000","last_became_active":"2023-12-19T07:47:17.208671+0000","last_became_peered":"2023-12-19T07:47:17.208671+0000","last_unstale":"2023-12-19T07:47:22.351363+0000","last_undegraded":"2023-12-19T07:47:22.351363+0000","last_fullsized":"2023-12-19T07:47:22.351363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903861+0000","last_change":"2023-12-19T07:47:17.199159+0000","last_active":"2023-12-19T07:47:22.903861+0000","last_peered":"2023-12-19T07:47:22.903861+0000","last_clean":"2023-12-19T07:47:22.903861+0000","last_became_active":"2023-12-19T07:47:17.198548+0000","last_became_peered":"2023-12-19T07:47:17.198548+0000","last_unstale":"2023-12-19T07:47:22.903861+0000","last_undegraded":"2023-12-19T07:47:22.903861+0000","last_fullsized":"2023-12-19T07:47:22.903861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598739+0000","last_change":"2023-12-19T07:47:17.202636+0000","last_active":"2023-12-19T07:47:22.598739+0000","last_peered":"2023-12-19T07:47:22.598739+0000","last_clean":"2023-12-19T07:47:22.598739+0000","last_became_active":"2023-12-19T07:47:17.202331+0000","last_became_peered":"2023-12-19T07:47:17.202331+0000","last_unstale":"2023-12-19T07:47:22.598739+0000","last_undegraded":"2023-12-19T07:47:22.598739+0000","last_fullsized":"2023-12-19T07:47:22.598739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306911+0000","last_change":"2023-12-19T07:47:17.202590+0000","last_active":"2023-12-19T07:47:22.306911+0000","last_peered":"2023-12-19T07:47:22.306911+0000","last_clean":"2023-12-19T07:47:22.306911+0000","last_became_active":"2023-12-19T07:47:17.202206+0000","last_became_peered":"2023-12-19T07:47:17.202206+0000","last_unstale":"2023-12-19T07:47:22.306911+0000","last_undegraded":"2023-12-19T07:47:22.306911+0000","last_fullsized":"2023-12-19T07:47:22.306911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:04.053349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359760+0000","last_change":"2023-12-19T07:47:17.209806+0000","last_active":"2023-12-19T07:47:22.359760+0000","last_peered":"2023-12-19T07:47:22.359760+0000","last_clean":"2023-12-19T07:47:22.359760+0000","last_became_active":"2023-12-19T07:47:17.209424+0000","last_became_peered":"2023-12-19T07:47:17.209424+0000","last_unstale":"2023-12-19T07:47:22.359760+0000","last_undegraded":"2023-12-19T07:47:22.359760+0000","last_fullsized":"2023-12-19T07:47:22.359760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359954+0000","last_change":"2023-12-19T07:47:17.209509+0000","last_active":"2023-12-19T07:47:22.359954+0000","last_peered":"2023-12-19T07:47:22.359954+0000","last_clean":"2023-12-19T07:47:22.359954+0000","last_became_active":"2023-12-19T07:47:17.209026+0000","last_became_peered":"2023-12-19T07:47:17.209026+0000","last_unstale":"2023-12-19T07:47:22.359954+0000","last_undegraded":"2023-12-19T07:47:22.359954+0000","last_fullsized":"2023-12-19T07:47:22.359954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868374+0000","last_change":"2023-12-19T07:47:17.225611+0000","last_active":"2023-12-19T07:47:22.868374+0000","last_peered":"2023-12-19T07:47:22.868374+0000","last_clean":"2023-12-19T07:47:22.868374+0000","last_became_active":"2023-12-19T07:47:17.225305+0000","last_became_peered":"2023-12-19T07:47:17.225305+0000","last_unstale":"2023-12-19T07:47:22.868374+0000","last_undegraded":"2023-12-19T07:47:22.868374+0000","last_fullsized":"2023-12-19T07:47:22.868374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.474113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598130+0000","last_change":"2023-12-19T07:47:17.209463+0000","last_active":"2023-12-19T07:47:22.598130+0000","last_peered":"2023-12-19T07:47:22.598130+0000","last_clean":"2023-12-19T07:47:22.598130+0000","last_became_active":"2023-12-19T07:47:17.201276+0000","last_became_peered":"2023-12-19T07:47:17.201276+0000","last_unstale":"2023-12-19T07:47:22.598130+0000","last_undegraded":"2023-12-19T07:47:22.598130+0000","last_fullsized":"2023-12-19T07:47:22.598130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867603+0000","last_change":"2023-12-19T07:47:17.200808+0000","last_active":"2023-12-19T07:47:22.867603+0000","last_peered":"2023-12-19T07:47:22.867603+0000","last_clean":"2023-12-19T07:47:22.867603+0000","last_became_active":"2023-12-19T07:47:17.200445+0000","last_became_peered":"2023-12-19T07:47:17.200445+0000","last_unstale":"2023-12-19T07:47:22.867603+0000","last_undegraded":"2023-12-19T07:47:22.867603+0000","last_fullsized":"2023-12-19T07:47:22.867603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806351+0000","last_change":"2023-12-19T07:47:17.194801+0000","last_active":"2023-12-19T07:47:22.806351+0000","last_peered":"2023-12-19T07:47:22.806351+0000","last_clean":"2023-12-19T07:47:22.806351+0000","last_became_active":"2023-12-19T07:47:17.194502+0000","last_became_peered":"2023-12-19T07:47:17.194502+0000","last_unstale":"2023-12-19T07:47:22.806351+0000","last_undegraded":"2023-12-19T07:47:22.806351+0000","last_fullsized":"2023-12-19T07:47:22.806351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867343+0000","last_change":"2023-12-19T07:47:17.214803+0000","last_active":"2023-12-19T07:47:22.867343+0000","last_peered":"2023-12-19T07:47:22.867343+0000","last_clean":"2023-12-19T07:47:22.867343+0000","last_became_active":"2023-12-19T07:47:17.214536+0000","last_became_peered":"2023-12-19T07:47:17.214536+0000","last_unstale":"2023-12-19T07:47:22.867343+0000","last_undegraded":"2023-12-19T07:47:22.867343+0000","last_fullsized":"2023-12-19T07:47:22.867343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806760+0000","last_change":"2023-12-19T07:47:17.206649+0000","last_active":"2023-12-19T07:47:22.806760+0000","last_peered":"2023-12-19T07:47:22.806760+0000","last_clean":"2023-12-19T07:47:22.806760+0000","last_became_active":"2023-12-19T07:47:17.206360+0000","last_became_peered":"2023-12-19T07:47:17.206360+0000","last_unstale":"2023-12-19T07:47:22.806760+0000","last_undegraded":"2023-12-19T07:47:22.806760+0000","last_fullsized":"2023-12-19T07:47:22.806760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904329+0000","last_change":"2023-12-19T07:47:17.195685+0000","last_active":"2023-12-19T07:47:22.904329+0000","last_peered":"2023-12-19T07:47:22.904329+0000","last_clean":"2023-12-19T07:47:22.904329+0000","last_became_active":"2023-12-19T07:47:17.195380+0000","last_became_peered":"2023-12-19T07:47:17.195380+0000","last_unstale":"2023-12-19T07:47:22.904329+0000","last_undegraded":"2023-12-19T07:47:22.904329+0000","last_fullsized":"2023-12-19T07:47:22.904329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:47.378624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353591+0000","last_change":"2023-12-19T07:47:17.204379+0000","last_active":"2023-12-19T07:47:22.353591+0000","last_peered":"2023-12-19T07:47:22.353591+0000","last_clean":"2023-12-19T07:47:22.353591+0000","last_became_active":"2023-12-19T07:47:17.204111+0000","last_became_peered":"2023-12-19T07:47:17.204111+0000","last_unstale":"2023-12-19T07:47:22.353591+0000","last_undegraded":"2023-12-19T07:47:22.353591+0000","last_fullsized":"2023-12-19T07:47:22.353591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868221+0000","last_change":"2023-12-19T07:47:17.209821+0000","last_active":"2023-12-19T07:47:22.868221+0000","last_peered":"2023-12-19T07:47:22.868221+0000","last_clean":"2023-12-19T07:47:22.868221+0000","last_became_active":"2023-12-19T07:47:17.209235+0000","last_became_peered":"2023-12-19T07:47:17.209235+0000","last_unstale":"2023-12-19T07:47:22.868221+0000","last_undegraded":"2023-12-19T07:47:22.868221+0000","last_fullsized":"2023-12-19T07:47:22.868221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:23.894121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":864256,"data_stored":697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.544}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":270336,"data_stored":106631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":106667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":106541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2909999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4710000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297556,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074284,"statfs":{"total":96636764160,"available":96332066816,"internally_reserved":0,"allocated":274432,"data_stored":106701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":868352,"data_stored":696508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:47:33.754 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T07:47:33.755 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-19T07:47:33.755 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-19T07:47:33.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-19T07:47:33.755 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-19T07:47:33.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.976+0000 7f08716f0700 1 Processor -- start 2023-12-19T07:47:33.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.976+0000 7f08716f0700 1 -- start start 2023-12-19T07:47:33.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f08716f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f086c0cc3b0 0x7f086c0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f08716f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 0x7f086c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f08716f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c0cd5a0 con 0x7f086c156c90 2023-12-19T07:47:33.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f08716f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c157470 con 0x7f086c0cc3b0 2023-12-19T07:47:33.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f08716f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0cc990 con 0x7f086c157060 2023-12-19T07:47:33.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f086c0cc3b0 0x7f086c0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f086c0cc3b0 0x7f086c0cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34518/0 (socket says 172.21.15.111:34518) 2023-12-19T07:47:33.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 -- 172.21.15.111:0/4029699774 learned_addr learned my addr 172.21.15.111:0/4029699774 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:33.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086b7fe700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 0x7f086c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 -- 172.21.15.111:0/4029699774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 msgr2=0x7f086c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:33.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 0x7f086c15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 -- 172.21.15.111:0/4029699774 >> v1:172.21.15.143:6789/0 conn(0x7f086c156c90 legacy=0x7f086c0c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 -- 172.21.15.111:0/4029699774 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0ccc70 con 0x7f086c0cc3b0 2023-12-19T07:47:33.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086b7fe700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 0x7f086c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:33.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f086a7fc700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0cc3b0 0x7f086c0cda30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08600099b0 tx=0x7f0860009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b7368791dfe247 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f0869ffb700 1 -- 172.21.15.111:0/4029699774 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000b860 con 0x7f086c0cc3b0 2023-12-19T07:47:33.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f0869ffb700 1 -- 172.21.15.111:0/4029699774 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086000b9c0 con 0x7f086c0cc3b0 2023-12-19T07:47:33.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.980+0000 7f0869ffb700 1 -- 172.21.15.111:0/4029699774 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000bd60 con 0x7f086c0cc3b0 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 -- 172.21.15.111:0/4029699774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0cc3b0 msgr2=0x7f086c0cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0cc3b0 0x7f086c0cda30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08600099b0 tx=0x7f0860009250 comp rx=0 tx=0).stop 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 -- 172.21.15.111:0/4029699774 shutdown_connections 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c157060 0x7f086c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 --2- 172.21.15.111:0/4029699774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0cc3b0 0x7f086c0cda30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 -- 172.21.15.111:0/4029699774 >> 172.21.15.111:0/4029699774 conn(0x7f086c06c7c0 msgr2=0x7f086c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 -- 172.21.15.111:0/4029699774 shutdown_connections 2023-12-19T07:47:33.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.984+0000 7f08716f0700 1 -- 172.21.15.111:0/4029699774 wait complete. 2023-12-19T07:47:33.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 Processor -- start 2023-12-19T07:47:33.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 -- start start 2023-12-19T07:47:33.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c0cc3b0 0x7f086c154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f086c157060 0x7f086c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c157ef0 con 0x7f086c156c90 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0b3aa0 con 0x7f086c157060 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f08716f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c155240 con 0x7f086c0cc3b0 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34520/0 (socket says 172.21.15.111:34520) 2023-12-19T07:47:33.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 -- 172.21.15.111:0/998691559 learned_addr learned my addr 172.21.15.111:0/998691559 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086affd700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c0cc3b0 0x7f086c154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086b7fe700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f086c157060 0x7f086c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 -- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c0cc3b0 msgr2=0x7f086c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c0cc3b0 0x7f086c154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 -- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f086c157060 msgr2=0x7f086c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f086c157060 0x7f086c0b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 -- 172.21.15.111:0/998691559 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600094f0 con 0x7f086c156c90 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.988+0000 7f086a7fc700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0860009290 tx=0x7f0860017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860011e40 con 0x7f086c156c90 2023-12-19T07:47:33.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086c067fd0 con 0x7f086c156c90 2023-12-19T07:47:33.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0860024440 con 0x7f086c156c90 2023-12-19T07:47:33.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600247c0 con 0x7f086c156c90 2023-12-19T07:47:33.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c068590 con 0x7f086c156c90 2023-12-19T07:47:33.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c071120 con 0x7f086c156c90 2023-12-19T07:47:33.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0860024bf0 con 0x7f086c156c90 2023-12-19T07:47:33.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.992+0000 7f0852ffd700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 0x7f0858062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:33.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.996+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f086007e770 con 0x7f086c156c90 2023-12-19T07:47:33.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.996+0000 7f086affd700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 0x7f0858062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:33.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:33.996+0000 7f086affd700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 0x7f0858062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f085c003a80 tx=0x7f085c007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:33.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.000+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f086004e9d0 con 0x7f086c156c90 2023-12-19T07:47:34.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.216+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f086c0ccf50 con 0x7f086c156c90 2023-12-19T07:47:34.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.224+0000 7f0852ffd700 1 -- 172.21.15.111:0/998691559 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+154140 (secure 0 0 0) 0x7f0860051ff0 con 0x7f086c156c90 2023-12-19T07:47:34.210 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:34.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 msgr2=0x7f0858062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 0x7f0858062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f085c003a80 tx=0x7f085c007ec0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 msgr2=0x7f086c0b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0860009290 tx=0x7f0860017000 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 shutdown_connections 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0858060550 0x7f0858062a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f086c157060 0x7f086c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c156c90 0x7f086c0b3020 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 --2- 172.21.15.111:0/998691559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f086c0cc3b0 0x7f086c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 >> 172.21.15.111:0/998691559 conn(0x7f086c06c7c0 msgr2=0x7f086c06cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 shutdown_connections 2023-12-19T07:47:34.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.232+0000 7f08716f0700 1 -- 172.21.15.111:0/998691559 wait complete. 2023-12-19T07:47:34.236 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":33796},{"type":"v1","addr":"172.21.15.143:6833","nonce":33796}]},"active_addr":"172.21.15.143:6833/33796","active_change":"2023-12-19T07:46:58.585134+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4109,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1940071424}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":4055025414}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1676361505}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3058403037}]}]} 2023-12-19T07:47:34.239 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-19T07:47:34.239 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-19T07:47:34.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:34.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.408+0000 7f3063465700 1 Processor -- start 2023-12-19T07:47:34.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 -- start start 2023-12-19T07:47:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f305c067c90 0x7f305c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f305c14fba0 0x7f305c150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0c6630 con 0x7f305c0c5900 2023-12-19T07:47:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c0c6910 con 0x7f305c14fba0 2023-12-19T07:47:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.412+0000 7f3063465700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c0c6bf0 con 0x7f305c067c90 2023-12-19T07:47:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f305c14fba0 0x7f305c150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f305c14fba0 0x7f305c150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34528/0 (socket says 172.21.15.111:34528) 2023-12-19T07:47:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 -- 172.21.15.111:0/3198423266 learned_addr learned my addr 172.21.15.111:0/3198423266 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3061201700 1 --2- 172.21.15.111:0/3198423266 >> v2:172.21.15.143:3300/0 conn(0x7f305c067c90 0x7f305c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 -- 172.21.15.111:0/3198423266 >> v2:172.21.15.143:3300/0 conn(0x7f305c067c90 msgr2=0x7f305c068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 --2- 172.21.15.111:0/3198423266 >> v2:172.21.15.143:3300/0 conn(0x7f305c067c90 0x7f305c068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 -- 172.21.15.111:0/3198423266 >> v1:172.21.15.143:6789/0 conn(0x7f305c0c5900 legacy=0x7f305c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 -- 172.21.15.111:0/3198423266 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0c6ed0 con 0x7f305c14fba0 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3053fff700 1 -- 172.21.15.111:0/3198423266 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880931567 0 0) 0x7f305c0c6630 con 0x7f305c0c5900 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3060a00700 1 --2- 172.21.15.111:0/3198423266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c14fba0 0x7f305c150000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f305c06a0c0 tx=0x7f304c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf48925b6e08280 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3053fff700 1 -- 172.21.15.111:0/3198423266 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c00b860 con 0x7f305c14fba0 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3053fff700 1 -- 172.21.15.111:0/3198423266 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c00b9c0 con 0x7f305c14fba0 2023-12-19T07:47:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3053fff700 1 -- 172.21.15.111:0/3198423266 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c00bd60 con 0x7f305c14fba0 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 -- 172.21.15.111:0/3198423266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c14fba0 msgr2=0x7f305c150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 --2- 172.21.15.111:0/3198423266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c14fba0 0x7f305c150000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f305c06a0c0 tx=0x7f304c009250 comp rx=0 tx=0).stop 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 -- 172.21.15.111:0/3198423266 shutdown_connections 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 --2- 172.21.15.111:0/3198423266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c14fba0 0x7f305c150000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 --2- 172.21.15.111:0/3198423266 >> v2:172.21.15.143:3300/0 conn(0x7f305c067c90 0x7f305c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 -- 172.21.15.111:0/3198423266 >> 172.21.15.111:0/3198423266 conn(0x7f305c0cd6d0 msgr2=0x7f305c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 -- 172.21.15.111:0/3198423266 shutdown_connections 2023-12-19T07:47:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.416+0000 7f3063465700 1 -- 172.21.15.111:0/3198423266 wait complete. 2023-12-19T07:47:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 Processor -- start 2023-12-19T07:47:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 -- start start 2023-12-19T07:47:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f305c067c90 0x7f305c141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f305c14fba0 0x7f305c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c14ee30 con 0x7f305c0c5900 2023-12-19T07:47:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0afc70 con 0x7f305c14fba0 2023-12-19T07:47:34.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3063465700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0aff50 con 0x7f305c067c90 2023-12-19T07:47:34.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34532/0 (socket says 172.21.15.111:34532) 2023-12-19T07:47:34.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 -- 172.21.15.111:0/3005616356 learned_addr learned my addr 172.21.15.111:0/3005616356 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:34.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3061a02700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f305c14fba0 0x7f305c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3061201700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f305c067c90 0x7f305c141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 -- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f305c067c90 msgr2=0x7f305c141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f305c067c90 0x7f305c141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 -- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f305c14fba0 msgr2=0x7f305c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f305c14fba0 0x7f305c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.420+0000 7f3060a00700 1 -- 172.21.15.111:0/3005616356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0094f0 con 0x7f305c0c5900 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f3060a00700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f304c006f50 tx=0x7f304c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c024440 con 0x7f305c0c5900 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c0245c0 con 0x7f305c0c5900 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c0b0230 con 0x7f305c0c5900 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c024980 con 0x7f305c0c5900 2023-12-19T07:47:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c14ea90 con 0x7f305c0c5900 2023-12-19T07:47:34.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c071120 con 0x7f305c0c5900 2023-12-19T07:47:34.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.424+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f304c01a420 con 0x7f305c0c5900 2023-12-19T07:47:34.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.428+0000 7f30517fa700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 0x7f3040062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.428+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f304c07e970 con 0x7f305c0c5900 2023-12-19T07:47:34.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.428+0000 7f3061201700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 0x7f3040062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.428+0000 7f3061201700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 0x7f3040062a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.432+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f304c04ecf0 con 0x7f305c0c5900 2023-12-19T07:47:34.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.608+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f305c0642a0 con 0x7f305c0c5900 2023-12-19T07:47:34.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.612+0000 7f30517fa700 1 -- 172.21.15.111:0/3005616356 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f304c052310 con 0x7f305c0c5900 2023-12-19T07:47:34.594 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:34.594 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:34.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.612+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 msgr2=0x7f3040062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.612+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 0x7f3040062a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3058006fd0 tx=0x7f3058008040 comp rx=0 tx=0).stop 2023-12-19T07:47:34.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.612+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 msgr2=0x7f305c15b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.612+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f304c006f50 tx=0x7f304c017000 comp rx=0 tx=0).stop 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 shutdown_connections 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3040060550 0x7f3040062a00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f305c14fba0 0x7f305c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f305c0c5900 0x7f305c15b860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 --2- 172.21.15.111:0/3005616356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f305c067c90 0x7f305c141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 >> 172.21.15.111:0/3005616356 conn(0x7f305c0cd6d0 msgr2=0x7f305c0cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 shutdown_connections 2023-12-19T07:47:34.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.616+0000 7f3063465700 1 -- 172.21.15.111:0/3005616356 wait complete. 2023-12-19T07:47:34.618 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-19T07:47:34.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:34.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.792+0000 7f8bf6763700 1 Processor -- start 2023-12-19T07:47:34.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.792+0000 7f8bf6763700 1 -- start start 2023-12-19T07:47:34.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf6763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf00c6900 0x7f8bf00c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf6763700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8bf006b900 0x7f8bf006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf6763700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf006b470 con 0x7f8bf015cc20 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf6763700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf015d7b0 con 0x7f8bf006b900 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf6763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf0161d10 con 0x7f8bf00c6900 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8bf006b900 0x7f8bf006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf5f62700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8bf015cc20 0x7f8bf015d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46298/0 (socket says 172.21.15.111:46298) 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf5f62700 1 -- 172.21.15.111:0/2383172914 learned_addr learned my addr 172.21.15.111:0/2383172914 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:34.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf5761700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf00c6900 0x7f8bf00c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8be7fff700 1 -- 172.21.15.111:0/2383172914 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921405453 0 0) 0x7f8bf006b470 con 0x7f8bf015cc20 2023-12-19T07:47:34.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8be7fff700 1 -- 172.21.15.111:0/2383172914 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bdc003570 con 0x7f8bf015cc20 2023-12-19T07:47:34.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 -- 172.21.15.111:0/2383172914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf00c6900 msgr2=0x7f8bf00c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:34.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf00c6900 0x7f8bf00c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 -- 172.21.15.111:0/2383172914 >> v1:172.21.15.143:6789/0 conn(0x7f8bf015cc20 legacy=0x7f8bf015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 -- 172.21.15.111:0/2383172914 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf006bd70 con 0x7f8bf006b900 2023-12-19T07:47:34.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8bf4f60700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf006b900 0x7f8bf006da00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009910 tx=0x7f8be00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130107e8855897da server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8be7fff700 1 -- 172.21.15.111:0/2383172914 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0011630 con 0x7f8bf006b900 2023-12-19T07:47:34.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8be7fff700 1 -- 172.21.15.111:0/2383172914 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be0011790 con 0x7f8bf006b900 2023-12-19T07:47:34.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.796+0000 7f8be7fff700 1 -- 172.21.15.111:0/2383172914 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0011b30 con 0x7f8bf006b900 2023-12-19T07:47:34.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- 172.21.15.111:0/2383172914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf006b900 msgr2=0x7f8bf006da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf006b900 0x7f8bf006da00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009910 tx=0x7f8be00091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- 172.21.15.111:0/2383172914 shutdown_connections 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf006b900 0x7f8bf006da00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 --2- 172.21.15.111:0/2383172914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf00c6900 0x7f8bf00c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- 172.21.15.111:0/2383172914 >> 172.21.15.111:0/2383172914 conn(0x7f8bf0067bb0 msgr2=0x7f8bf00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- 172.21.15.111:0/2383172914 shutdown_connections 2023-12-19T07:47:34.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- 172.21.15.111:0/2383172914 wait complete. 2023-12-19T07:47:34.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 Processor -- start 2023-12-19T07:47:34.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.800+0000 7f8bf6763700 1 -- start start 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf006b900 0x7f8bf0160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf015cc20 0x7f8bf0158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00adea0 con 0x7f8bf00c6900 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf015fdc0 con 0x7f8bf006b900 2023-12-19T07:47:34.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf00ad8e0 con 0x7f8bf015cc20 2023-12-19T07:47:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34550/0 (socket says 172.21.15.111:34550) 2023-12-19T07:47:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 -- 172.21.15.111:0/1732696184 learned_addr learned my addr 172.21.15.111:0/1732696184 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf5761700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf006b900 0x7f8bf0160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf5f62700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf015cc20 0x7f8bf0158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 -- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf015cc20 msgr2=0x7f8bf0158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf015cc20 0x7f8bf0158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 -- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf006b900 msgr2=0x7f8bf0160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf006b900 0x7f8bf0160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 -- 172.21.15.111:0/1732696184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be0009420 con 0x7f8bf00c6900 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf4f60700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0064040 tx=0x7f8be0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0011b10 con 0x7f8bf00c6900 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be0025400 con 0x7f8bf00c6900 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf00adbc0 con 0x7f8bf00c6900 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0025780 con 0x7f8bf00c6900 2023-12-19T07:47:34.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.804+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf00af8d0 con 0x7f8bf00c6900 2023-12-19T07:47:34.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.808+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf00afac0 con 0x7f8bf00c6900 2023-12-19T07:47:34.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.808+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8be002e430 con 0x7f8bf00c6900 2023-12-19T07:47:34.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.808+0000 7f8be57fa700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 0x7f8bd4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:34.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.808+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f8be007e2e0 con 0x7f8bf00c6900 2023-12-19T07:47:34.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.808+0000 7f8bf5761700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 0x7f8bd4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:34.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.812+0000 7f8bf5761700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 0x7f8bd4062a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009db0 tx=0x7f8bec0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:34.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.816+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8be004e5e0 con 0x7f8bf00c6900 2023-12-19T07:47:34.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.980+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bf004be40 con 0x7f8bf00c6900 2023-12-19T07:47:34.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8be57fa700 1 -- 172.21.15.111:0/1732696184 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f8be0051c00 con 0x7f8bf00c6900 2023-12-19T07:47:34.966 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:34.967 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 msgr2=0x7f8bd4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 0x7f8bd4062a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009db0 tx=0x7f8bec0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 msgr2=0x7f8bf00af390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0064040 tx=0x7f8be0017000 comp rx=0 tx=0).stop 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 shutdown_connections 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bd4060580 0x7f8bd4062a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf015cc20 0x7f8bf0158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf00c6900 0x7f8bf00af390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 --2- 172.21.15.111:0/1732696184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf006b900 0x7f8bf0160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 >> 172.21.15.111:0/1732696184 conn(0x7f8bf0067bb0 msgr2=0x7f8bf00cace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:34.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 shutdown_connections 2023-12-19T07:47:34.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:34.984+0000 7f8bf6763700 1 -- 172.21.15.111:0/1732696184 wait complete. 2023-12-19T07:47:34.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-19T07:47:34.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-19T07:47:34.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-19T07:47:34.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-19T07:47:34.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-19T07:47:34.991 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-19T07:47:34.991 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-19T07:47:34.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-19T07:47:35.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 Processor -- start 2023-12-19T07:47:35.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 -- start start 2023-12-19T07:47:35.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f465804fea0 0x7f4658052f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4658006370 con 0x7f46580e3bb0 2023-12-19T07:47:35.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4658050310 con 0x7f465804fea0 2023-12-19T07:47:35.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f4665016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4658053490 con 0x7f465804fad0 2023-12-19T07:47:35.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f465ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f465ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46706/0 (socket says 172.21.15.111:46706) 2023-12-19T07:47:35.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7f465ffff700 1 -- 172.21.15.111:0/458198292 learned_addr learned my addr 172.21.15.111:0/458198292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.224+0000 7fa996963700 1 Processor -- start 2023-12-19T07:47:35.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465f7fe700 1 --2- 172.21.15.111:0/458198292 >> v2:172.21.15.143:3300/0 conn(0x7f465804fea0 0x7f4658052f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465ffff700 1 -- 172.21.15.111:0/458198292 >> v2:172.21.15.143:3300/0 conn(0x7f465804fea0 msgr2=0x7f4658052f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465ffff700 1 --2- 172.21.15.111:0/458198292 >> v2:172.21.15.143:3300/0 conn(0x7f465804fea0 0x7f4658052f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465ffff700 1 -- 172.21.15.111:0/458198292 >> v1:172.21.15.143:6789/0 conn(0x7f46580e3bb0 legacy=0x7f46580e4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465ffff700 1 -- 172.21.15.111:0/458198292 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658053710 con 0x7f465804fad0 2023-12-19T07:47:35.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 -- start start 2023-12-19T07:47:35.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465ffff700 1 --2- 172.21.15.111:0/458198292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f464c009990 tx=0x7f464c009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e3b44acd1482abc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990154b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9900abca0 0x7fa99006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 0x7fa99006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465effd700 1 -- 172.21.15.111:0/458198292 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c005c10 con 0x7f465804fad0 2023-12-19T07:47:35.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465effd700 1 -- 172.21.15.111:0/458198292 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c005d70 con 0x7f465804fad0 2023-12-19T07:47:35.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7f465effd700 1 -- 172.21.15.111:0/458198292 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c00b880 con 0x7f465804fad0 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9900c5c70 con 0x7fa9900abca0 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9900c5fd0 con 0x7fa9900ac0f0 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996963700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99006d3a0 con 0x7fa9900cc0a0 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa995160700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9900abca0 0x7fa99006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa995160700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9900abca0 0x7fa99006c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34552/0 (socket says 172.21.15.111:34552) 2023-12-19T07:47:35.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa995160700 1 -- 172.21.15.111:0/2435682498 learned_addr learned my addr 172.21.15.111:0/2435682498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.228+0000 7fa996162700 1 --2- 172.21.15.111:0/2435682498 >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 0x7fa99006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995961700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 -- 172.21.15.111:0/2435682498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 msgr2=0x7fa990154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 Processor -- start 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 -- 172.21.15.111:0/2435682498 >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 msgr2=0x7fa99006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 --2- 172.21.15.111:0/2435682498 >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 0x7fa99006cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 -- 172.21.15.111:0/2435682498 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99006d680 con 0x7fa9900abca0 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 -- 172.21.15.111:0/458198292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 msgr2=0x7f46580e7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 --2- 172.21.15.111:0/458198292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f464c009990 tx=0x7f464c009220 comp rx=0 tx=0).stop 2023-12-19T07:47:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995961700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa996162700 1 --2- 172.21.15.111:0/2435682498 >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 0x7fa99006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa995160700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900abca0 0x7fa99006c7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa9800099b0 tx=0x7fa980009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a71b98396ceeb68 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 -- 172.21.15.111:0/458198292 shutdown_connections 2023-12-19T07:47:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 --2- 172.21.15.111:0/458198292 >> v2:172.21.15.143:3300/0 conn(0x7f465804fea0 0x7f4658052f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 --2- 172.21.15.111:0/458198292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465804fad0 0x7f46580e7700 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 -- 172.21.15.111:0/458198292 >> 172.21.15.111:0/458198292 conn(0x7f4658051b50 msgr2=0x7f4658051f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 -- start start 2023-12-19T07:47:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b406a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b406ba50 con 0x7fc5b4160e60 2023-12-19T07:47:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b40af600 con 0x7fc5b415d050 2023-12-19T07:47:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b986e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b406bee0 con 0x7fc5b40afef0 2023-12-19T07:47:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 -- 172.21.15.111:0/458198292 shutdown_connections 2023-12-19T07:47:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa99495f700 1 -- 172.21.15.111:0/2435682498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980009ea0 con 0x7fa9900abca0 2023-12-19T07:47:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc5b4160e60 0x7fc5b415d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56052/0 (socket says 172.21.15.111:56052) 2023-12-19T07:47:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fa99495f700 1 -- 172.21.15.111:0/2435682498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98000b8b0 con 0x7fa9900abca0 2023-12-19T07:47:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7fc5b37fe700 1 -- 172.21.15.111:0/1205390559 learned_addr learned my addr 172.21.15.111:0/1205390559 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa99495f700 1 -- 172.21.15.111:0/2435682498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98000bc50 con 0x7fa9900abca0 2023-12-19T07:47:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.232+0000 7f4665016700 1 -- 172.21.15.111:0/458198292 wait complete. 2023-12-19T07:47:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b27fc700 1 --2- 172.21.15.111:0/1205390559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f2e8d585700 1 Processor -- start 2023-12-19T07:47:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 -- 172.21.15.111:0/2435682498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900abca0 msgr2=0x7fa99006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900abca0 0x7fa99006c7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa9800099b0 tx=0x7fa980009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410931362 0 0) 0x7fc5b40af600 con 0x7fc5b415d050 2023-12-19T07:47:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a0003570 con 0x7fc5b415d050 2023-12-19T07:47:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031926829 0 0) 0x7fc5b406ba50 con 0x7fc5b4160e60 2023-12-19T07:47:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 -- 172.21.15.111:0/2435682498 shutdown_connections 2023-12-19T07:47:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 --2- 172.21.15.111:0/2435682498 >> v2:172.21.15.143:3300/0 conn(0x7fa9900ac0f0 0x7fa99006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900abca0 0x7fa99006c7c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f4665016700 1 Processor -- start 2023-12-19T07:47:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 --2- 172.21.15.111:0/2435682498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 -- 172.21.15.111:0/2435682498 >> 172.21.15.111:0/2435682498 conn(0x7fa990067bb0 msgr2=0x7fa9900cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b40af600 con 0x7fc5b4160e60 2023-12-19T07:47:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b27fc700 1 -- 172.21.15.111:0/1205390559 >> v1:172.21.15.143:6789/0 conn(0x7fc5b415d050 legacy=0x7fc5b40af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f2e8d585700 1 -- start start 2023-12-19T07:47:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b27fc700 1 -- 172.21.15.111:0/1205390559 >> v1:172.21.15.111:6789/0 conn(0x7fc5b4160e60 legacy=0x7fc5b415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f4665016700 1 -- start start 2023-12-19T07:47:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b27fc700 1 -- 172.21.15.111:0/1205390559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b406ae40 con 0x7fc5b40afef0 2023-12-19T07:47:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b27fc700 1 --2- 172.21.15.111:0/1205390559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b406a740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009e10 tx=0x7fc5a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=896f23c6197101c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 -- 172.21.15.111:0/2435682498 shutdown_connections 2023-12-19T07:47:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f9d2962e700 1 Processor -- start 2023-12-19T07:47:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400e070 con 0x7fc5b40afef0 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a400b820 con 0x7fc5b40afef0 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fc5b1ffb700 1 -- 172.21.15.111:0/1205390559 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac001470 con 0x7fc5b40afef0 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7f9d2962e700 1 -- start start 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.236+0000 7fa996963700 1 -- 172.21.15.111:0/2435682498 wait complete. 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465804fad0 0x7f465804a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465804fea0 0x7f465803a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 -- 172.21.15.111:0/1205390559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 msgr2=0x7fc5b406a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 --2- 172.21.15.111:0/1205390559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b406a740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009e10 tx=0x7fc5a4009780 comp rx=0 tx=0).stop 2023-12-19T07:47:35.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 0x7f465803a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e8d585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e880b3870 0x7f2e880b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e8d585700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8806b0d0 con 0x7f2e8806dc00 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e8d585700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880b4790 con 0x7f2e88158cc0 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e8d585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880b41d0 con 0x7f2e880b3870 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4658039150 con 0x7f465804fad0 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46580380b0 con 0x7f465804fea0 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4665016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4658038390 con 0x7f46580e3bb0 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 -- 172.21.15.111:0/1205390559 shutdown_connections 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 --2- 172.21.15.111:0/1205390559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b406a740 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 -- 172.21.15.111:0/1205390559 >> 172.21.15.111:0/1205390559 conn(0x7fc5b4067b60 msgr2=0x7fc5b40ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f120ccae700 1 Processor -- start 2023-12-19T07:47:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f120ccae700 1 -- start start 2023-12-19T07:47:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e86ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2e88158cc0 0x7f2e8806d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46320/0 (socket says 172.21.15.111:46320) 2023-12-19T07:47:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e86ffd700 1 -- 172.21.15.111:0/1556595339 learned_addr learned my addr 172.21.15.111:0/1556595339 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 0x7f465803a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 0x7f465803a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46740/0 (socket says 172.21.15.111:46740) 2023-12-19T07:47:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 -- 172.21.15.111:0/2744156205 learned_addr learned my addr 172.21.15.111:0/2744156205 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f465f7fe700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465804fea0 0x7f465803a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 -- 172.21.15.111:0/1205390559 shutdown_connections 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d2962e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 0x7f9d24161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d2962e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d2415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d2962e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2406d790 con 0x7f9d2406ca60 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d2962e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d2406d5b0 con 0x7f9d24160f30 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d2962e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2406da70 con 0x7f9d2415a730 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f465ffff700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465804fad0 0x7f465804a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e877fe700 1 --2- 172.21.15.111:0/1556595339 >> v2:172.21.15.143:3300/0 conn(0x7f2e880b3870 0x7f2e880b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 0x7f9d24161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 0x7f9d24161300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34572/0 (socket says 172.21.15.111:34572) 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d22ffd700 1 -- 172.21.15.111:0/1495691809 learned_addr learned my addr 172.21.15.111:0/1495691809 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d227fc700 1 --2- 172.21.15.111:0/1495691809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d2415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 -- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465804fea0 msgr2=0x7f465803a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465804fea0 0x7f465803a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 -- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465804fad0 msgr2=0x7f465804a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465804fad0 0x7f465804a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f4664815700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c009480 con 0x7f46580e3bb0 2023-12-19T07:47:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fa996963700 1 Processor -- start 2023-12-19T07:47:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fc5b986e700 1 -- 172.21.15.111:0/1205390559 wait complete. 2023-12-19T07:47:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e877fe700 1 -- 172.21.15.111:0/1556595339 >> v1:172.21.15.143:6789/0 conn(0x7f2e88158cc0 legacy=0x7f2e8806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e877fe700 1 -- 172.21.15.111:0/1556595339 >> v1:172.21.15.111:6789/0 conn(0x7f2e8806dc00 legacy=0x7f2e8806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e877fe700 1 -- 172.21.15.111:0/1556595339 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8806a7e0 con 0x7f2e880b3870 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7fa996963700 1 -- start start 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f4664815700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 0x7f465803a600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4660075ed0 tx=0x7f4660070c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f120ccae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1208087bd0 0x7f1208161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f120ccae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208087740 con 0x7f12080889d0 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f120ccae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080d0bd0 con 0x7f12080d3c80 2023-12-19T07:47:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f120ccae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1208162010 con 0x7f1208087bd0 2023-12-19T07:47:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553821010 0 0) 0x7f9d2406d790 con 0x7f9d2406ca60 2023-12-19T07:47:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d10003570 con 0x7f9d2406ca60 2023-12-19T07:47:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fc5b986e700 1 Processor -- start 2023-12-19T07:47:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374147327 0 0) 0x7f9d10003570 con 0x7f9d2406ca60 2023-12-19T07:47:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d227fc700 1 -- 172.21.15.111:0/1495691809 >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 msgr2=0x7f9d24161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d227fc700 1 --2- 172.21.15.111:0/1495691809 >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 0x7f9d24161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d227fc700 1 -- 172.21.15.111:0/1495691809 >> v1:172.21.15.143:6789/0 conn(0x7f9d2406ca60 legacy=0x7f9d2406cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d227fc700 1 -- 172.21.15.111:0/1495691809 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2406dd50 con 0x7f9d2415a730 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.240+0000 7f2e877fe700 1 --2- 172.21.15.111:0/1556595339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e880b3c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009860 tx=0x7f2e7c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9f11e258223a971 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1208087bd0 0x7f1208161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fc5b986e700 1 -- start start 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1208087bd0 0x7f1208161300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47760/0 (socket says 172.21.15.111:47760) 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 -- 172.21.15.111:0/103723583 learned_addr learned my addr 172.21.15.111:0/103723583 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660063f50 con 0x7f46580e3bb0 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46600640d0 con 0x7f46580e3bb0 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466006d940 con 0x7f46580e3bb0 2023-12-19T07:47:35.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 0x7fa9900ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900ac0f0 0x7fa990146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa990147bf0 con 0x7fa9900ac0f0 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa990150bf0 con 0x7fa9900abca0 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996963700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa990152bc0 con 0x7fa9900cc0a0 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e85ffb700 1 -- 172.21.15.111:0/1556595339 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c011630 con 0x7f2e880b3870 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e85ffb700 1 -- 172.21.15.111:0/1556595339 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c011790 con 0x7f2e880b3870 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e85ffb700 1 -- 172.21.15.111:0/1556595339 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c011b30 con 0x7f2e880b3870 2023-12-19T07:47:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995160700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900ac0f0 0x7fa990146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f120559a700 1 -- 172.21.15.111:0/103723583 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363569092 0 0) 0x7f1208087740 con 0x7f12080889d0 2023-12-19T07:47:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4658038670 con 0x7f46580e3bb0 2023-12-19T07:47:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 -- 172.21.15.111:0/103723583 >> v1:172.21.15.143:6789/0 conn(0x7f12080d3c80 legacy=0x7f1208161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 -- 172.21.15.111:0/103723583 >> v1:172.21.15.111:6789/0 conn(0x7f12080889d0 legacy=0x7f12080d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 -- 172.21.15.111:0/103723583 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080870f0 con 0x7f1208087bd0 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 -- 172.21.15.111:0/1556595339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 msgr2=0x7f2e880b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 --2- 172.21.15.111:0/1556595339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e880b3c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009860 tx=0x7f2e7c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4658038c30 con 0x7f46580e3bb0 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995961700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 0x7fa9900ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa996162700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995961700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 0x7fa9900ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47770/0 (socket says 172.21.15.111:47770) 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995961700 1 -- 172.21.15.111:0/3385310882 learned_addr learned my addr 172.21.15.111:0/3385310882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 -- 172.21.15.111:0/1556595339 shutdown_connections 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 --2- 172.21.15.111:0/1556595339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e880b3c90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 -- 172.21.15.111:0/1556595339 >> 172.21.15.111:0/1556595339 conn(0x7f2e88067bb0 msgr2=0x7f2e880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f9d227fc700 1 --2- 172.21.15.111:0/1495691809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d2415ab80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0098e0 tx=0x7f9d0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82cba0c38222d4b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f1205d9b700 1 --2- 172.21.15.111:0/103723583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1208087bd0 0x7f1208161300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0099b0 tx=0x7f11fc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e19359b79b96828 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 -- 172.21.15.111:0/1556595339 shutdown_connections 2023-12-19T07:47:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f2e8d585700 1 -- 172.21.15.111:0/1556595339 wait complete. 2023-12-19T07:47:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa062e2e700 1 Processor -- start 2023-12-19T07:47:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fc5b986e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b4153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fc5b986e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b415d050 0x7fc5b4155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fc5b986e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b4160e60 0x7fc5b415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa062e2e700 1 -- start start 2023-12-19T07:47:35.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46580057e0 con 0x7f46580e3bb0 2023-12-19T07:47:35.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995961700 1 -- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 msgr2=0x7fa990146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.244+0000 7fa995961700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9900cc0a0 0x7fa990146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120559a700 1 -- 172.21.15.111:0/103723583 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00b860 con 0x7f1208087bd0 2023-12-19T07:47:35.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120559a700 1 -- 172.21.15.111:0/103723583 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc00b9c0 con 0x7f1208087bd0 2023-12-19T07:47:35.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120559a700 1 -- 172.21.15.111:0/103723583 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00bd60 con 0x7f1208087bd0 2023-12-19T07:47:35.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa995961700 1 -- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900ac0f0 msgr2=0x7fa990146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 -- 172.21.15.111:0/103723583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1208087bd0 msgr2=0x7f1208161300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 --2- 172.21.15.111:0/103723583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1208087bd0 0x7f1208161300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0099b0 tx=0x7f11fc009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 -- 172.21.15.111:0/103723583 shutdown_connections 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 --2- 172.21.15.111:0/103723583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1208087bd0 0x7f1208161300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 -- 172.21.15.111:0/103723583 >> 172.21.15.111:0/103723583 conn(0x7f120806d050 msgr2=0x7f120806d450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b2ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b4153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b2ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b4153e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46772/0 (socket says 172.21.15.111:46772) 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b415d050 0x7fc5b4155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b2ffd700 1 -- 172.21.15.111:0/842646108 learned_addr learned my addr 172.21.15.111:0/842646108 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b4160e60 0x7fc5b415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b4156bd0 con 0x7fc5b415d050 2023-12-19T07:47:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b416d730 con 0x7fc5b4160e60 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b4156330 con 0x7fc5b40afef0 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d140017f0 con 0x7f9d2415a730 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0c011420 con 0x7f9d2415a730 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f9d2962e700 1 -- 172.21.15.111:0/1495691809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 msgr2=0x7f9d2415ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f9d21ffb700 1 -- 172.21.15.111:0/1495691809 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d140017f0 con 0x7f9d2415a730 2023-12-19T07:47:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f2e8d585700 1 Processor -- start 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa062e2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa05c06dc00 0x7fa05c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa062e2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa05c06baa0 0x7fa05c0b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa062e2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c06b0d0 con 0x7fa05c158cc0 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa062e2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c06bec0 con 0x7fa05c06baa0 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa062e2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0b4790 con 0x7fa05c06dc00 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f466006dcf0 con 0x7f46580e3bb0 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4644060620 0x7f4644062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f2e8d585700 1 -- start start 2023-12-19T07:47:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f46600c7770 con 0x7f46580e3bb0 2023-12-19T07:47:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa995961700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9900ac0f0 0x7fa990146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa995961700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c009420 con 0x7fa9900abca0 2023-12-19T07:47:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1495691809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d2415ab80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0098e0 tx=0x7f9d0c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa05c06baa0 0x7fa05c0b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f464406a970 0x7f464406ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa05c06baa0 0x7fa05c0b3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34610/0 (socket says 172.21.15.111:34610) 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 -- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 msgr2=0x7fc5b4153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40afef0 0x7fc5b4153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 -- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b415d050 msgr2=0x7fc5b4155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b415d050 0x7fc5b4155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4009a20 con 0x7fc5b4160e60 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f464406d4d0 con 0x7f464406a970 2023-12-19T07:47:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46600c6e00 con 0x7f46580e3bb0 2023-12-19T07:47:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f465ffff700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4644060620 0x7f4644062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 -- 172.21.15.111:0/3049960645 learned_addr learned my addr 172.21.15.111:0/3049960645 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 -- 172.21.15.111:0/103723583 shutdown_connections 2023-12-19T07:47:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f120ccae700 1 -- 172.21.15.111:0/103723583 wait complete. 2023-12-19T07:47:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7f465f7fe700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f464406a970 0x7f464406ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa995961700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 0x7fa9900ac970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa98800cfb0 tx=0x7fa988007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fc5b37fe700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b4160e60 0x7fc5b415bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c560 tx=0x7fc59c007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 -- 172.21.15.111:0/3049960645 >> v2:172.21.15.143:3300/0 conn(0x7fa05c06dc00 msgr2=0x7fa05c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 --2- 172.21.15.111:0/3049960645 >> v2:172.21.15.143:3300/0 conn(0x7fa05c06dc00 0x7fa05c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 -- 172.21.15.111:0/3049960645 >> v1:172.21.15.111:6789/0 conn(0x7fa05c158cc0 legacy=0x7fa05c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.248+0000 7fa0613cb700 1 -- 172.21.15.111:0/3049960645 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c06a7e0 con 0x7fa05c06baa0 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98800ae30 con 0x7fa9900abca0 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988016420 con 0x7fa9900abca0 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9880167e0 con 0x7fa9900abca0 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9800094f0 con 0x7fa9900abca0 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 -- 172.21.15.111:0/1495691809 shutdown_connections 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1495691809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d2415ab80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1495691809 >> v2:172.21.15.111:3300/0 conn(0x7f9d24160f30 0x7f9d24161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 0x7f2e8815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e881541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f465f7fe700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f464406a970 0x7f464406ce00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e88158cc0 0x7f2e880ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8816ff80 con 0x7f2e88158cc0 2023-12-19T07:47:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88176be0 con 0x7f2e880b3870 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e8d585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8815d880 con 0x7f2e8806dc00 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 Processor -- start 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 0x7f2e8815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 0x7f2e8815d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46778/0 (socket says 172.21.15.111:46778) 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 -- 172.21.15.111:0/2354348062 learned_addr learned my addr 172.21.15.111:0/2354348062 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa990147630 con 0x7fa9900abca0 2023-12-19T07:47:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 -- start start 2023-12-19T07:47:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c005b30 con 0x7fc5b4160e60 2023-12-19T07:47:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 -- 172.21.15.111:0/1495691809 >> 172.21.15.111:0/1495691809 conn(0x7f9d24067bb0 msgr2=0x7f9d240cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa0613cb700 1 --2- 172.21.15.111:0/3049960645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06baa0 0x7fa05c0b3870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa05c068090 tx=0x7fa054009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86d4e943ed17ecb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b4156610 con 0x7fc5b4160e60 2023-12-19T07:47:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b4154340 con 0x7fc5b4160e60 2023-12-19T07:47:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59c005c90 con 0x7fc5b4160e60 2023-12-19T07:47:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c005b30 con 0x7fc5b4160e60 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== osd.0 v2:172.21.15.111:6800/33928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f464406d4d0 con 0x7f464406a970 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e867fc700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e881541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e877fe700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e88158cc0 0x7f2e880ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 -- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 msgr2=0x7f2e881541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e880b3870 0x7f2e881541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 -- 172.21.15.111:0/2354348062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e88158cc0 msgr2=0x7f2e880ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e88158cc0 0x7f2e880ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c009420 con 0x7f2e8806dc00 2023-12-19T07:47:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa05b7fe700 1 -- 172.21.15.111:0/3049960645 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400e070 con 0x7fa05c06baa0 2023-12-19T07:47:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 0x7f120816e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa05b7fe700 1 -- 172.21.15.111:0/3049960645 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05400ba50 con 0x7fa05c06baa0 2023-12-19T07:47:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa05b7fe700 1 -- 172.21.15.111:0/3049960645 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540092e0 con 0x7fa05c06baa0 2023-12-19T07:47:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f465ffff700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4644060620 0x7f4644062ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f464c007fd0 tx=0x7f464c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12080889d0 0x7f12080ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12080d3c80 0x7f120815b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080cdf20 con 0x7f1208087bd0 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080cd490 con 0x7f12080889d0 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120ccae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1208068fb0 con 0x7f12080d3c80 2023-12-19T07:47:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa99004be40 con 0x7fa9900abca0 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f2e86ffd700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 0x7f2e8815d340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c7e0 tx=0x7f2e78007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 0x7f120816e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 0x7f120816e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34638/0 (socket says 172.21.15.111:34638) 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f120659c700 1 -- 172.21.15.111:0/3428098907 learned_addr learned my addr 172.21.15.111:0/3428098907 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12080889d0 0x7f12080ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1206d9d700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12080d3c80 0x7f120815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa062e2e700 1 -- 172.21.15.111:0/3049960645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06baa0 msgr2=0x7fa05c0b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7fa062e2e700 1 --2- 172.21.15.111:0/3049960645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06baa0 0x7fa05c0b3870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa05c068090 tx=0x7fa054009840 comp rx=0 tx=0).stop 2023-12-19T07:47:35.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 -- 172.21.15.111:0/3049960645 shutdown_connections 2023-12-19T07:47:35.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 -- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12080d3c80 msgr2=0x7f120815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 --2- 172.21.15.111:0/3049960645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06baa0 0x7fa05c0b3870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 --2- 172.21.15.111:0/3049960645 >> v2:172.21.15.143:3300/0 conn(0x7fa05c06dc00 0x7fa05c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 -- 172.21.15.111:0/3049960645 >> 172.21.15.111:0/3049960645 conn(0x7fa05c067bb0 msgr2=0x7fa05c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12080d3c80 0x7f120815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 -- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 msgr2=0x7f120816e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 0x7f120816e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f1205d9b700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0094f0 con 0x7f12080889d0 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 -- 172.21.15.111:0/1495691809 shutdown_connections 2023-12-19T07:47:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.252+0000 7f9d2962e700 1 -- 172.21.15.111:0/1495691809 wait complete. 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc598000fd0 con 0x7fc5b4160e60 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f120659c700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208087bd0 0x7f120816e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f1205d9b700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12080889d0 0x7f12080ccf50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12080ceab0 tx=0x7f11fc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78005b30 con 0x7f2e8806dc00 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78005c90 con 0x7f2e8806dc00 2023-12-19T07:47:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78007320 con 0x7f2e8806dc00 2023-12-19T07:47:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc024440 con 0x7f12080889d0 2023-12-19T07:47:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc0245e0 con 0x7f12080889d0 2023-12-19T07:47:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0249a0 con 0x7f12080889d0 2023-12-19T07:47:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f120ccae700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12080cd680 con 0x7f12080889d0 2023-12-19T07:47:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f120ccae700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12080cdc40 con 0x7f12080889d0 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 -- 172.21.15.111:0/3049960645 shutdown_connections 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa062e2e700 1 -- 172.21.15.111:0/3049960645 wait complete. 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1208071120 con 0x7f12080889d0 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f11fc01a420 con 0x7f12080889d0 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc59c0072e0 con 0x7fc5b4160e60 2023-12-19T07:47:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc594060630 0x7fc594062ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc59c07b930 con 0x7fc5b4160e60 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7fc594069c60 0x7fc59406c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc59406c760 con 0x7fc594069c60 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc59c080220 con 0x7fc5b4160e60 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f9d2962e700 1 Processor -- start 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f9d2962e700 1 -- start start 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e88176e30 con 0x7f2e8806dc00 2023-12-19T07:47:35.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e881773f0 con 0x7f2e8806dc00 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.256+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa988016b90 con 0x7fa9900abca0 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa985ffb700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa9740606a0 0x7fa974062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11e8060550 0x7f11e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f11fc07e8d0 con 0x7f12080889d0 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7f11e8069b80 0x7f11e806bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11e806c680 con 0x7f11e8069b80 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11fc080dd0 con 0x7f12080889d0 2023-12-19T07:47:35.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f120659c700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11e8060550 0x7f11e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fc5b2ffd700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc594060630 0x7fc594062ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fc5b27fc700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7fc594069c60 0x7fc59406c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fc5b27fc700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7fc594069c60 0x7fc59406c090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 0x7f9d24152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d24151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 0x7f9d2416d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d22ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 0x7f9d24152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d22ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 0x7f9d24152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47796/0 (socket says 172.21.15.111:47796) 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d227fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d24151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d227fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d24151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46802/0 (socket says 172.21.15.111:46802) 2023-12-19T07:47:35.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e88071120 con 0x7f2e8806dc00 2023-12-19T07:47:35.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa062e2e700 1 Processor -- start 2023-12-19T07:47:35.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa062e2e700 1 -- start start 2023-12-19T07:47:35.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa988012070 con 0x7fa9900abca0 2023-12-19T07:47:35.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== osd.3 v2:172.21.15.111:6816/33942 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc59406c760 con 0x7fc594069c60 2023-12-19T07:47:35.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa985ffb700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fa974069cd0 0x7fa97406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa97406c7d0 con 0x7fa974069cd0 2023-12-19T07:47:35.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa98807f730 con 0x7fa9900abca0 2023-12-19T07:47:35.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2e78007690 con 0x7f2e8806dc00 2023-12-19T07:47:35.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f2e6effd700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e74060620 0x7f2e74062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f1206d9d700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7f11e8069b80 0x7f11e806bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f1206d9d700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7f11e8069b80 0x7f11e806bfb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa995160700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa9740606a0 0x7fa974062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa05c06baa0 0x7fa05c16f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa996162700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fa974069cd0 0x7fa97406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fc5b2ffd700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc594060630 0x7fc594062ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0011e0 tx=0x7fc5ac00a610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 0x7f9d2416d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241522f0 con 0x7f9d24160f30 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d24154ab0 con 0x7f9d2406ca60 2023-12-19T07:47:35.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.260+0000 7f9d2962e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d24154780 con 0x7f9d2415a730 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06dc00 0x7fa05c147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 0x7fa05c088460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c16ec00 con 0x7fa05c06dc00 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c16f5d0 con 0x7fa05c06baa0 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa062e2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c16dff0 con 0x7fa05c158cc0 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 0x7f9d2416d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34648/0 (socket says 172.21.15.111:34648) 2023-12-19T07:47:35.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 -- 172.21.15.111:0/1429027085 learned_addr learned my addr 172.21.15.111:0/1429027085 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 0x7fa05c088460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f120659c700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11e8060550 0x7f11e8062a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11f8007fd0 tx=0x7f11f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 0x7fa05c088460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46814/0 (socket says 172.21.15.111:46814) 2023-12-19T07:47:35.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa060bca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa05c06baa0 0x7fa05c16f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa995160700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa9740606a0 0x7fa974062b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9800099b0 tx=0x7fa980017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2e7807c5e0 con 0x7f2e8806dc00 2023-12-19T07:47:35.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 -- 172.21.15.111:0/2559903923 learned_addr learned my addr 172.21.15.111:0/2559903923 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 -- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 msgr2=0x7f9d24151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2415a730 0x7f9d24151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 -- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 msgr2=0x7f9d24152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa996162700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fa974069cd0 0x7fa97406c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e6effd700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f2e74069c50 0x7f2e7406c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e7406c750 con 0x7f2e74069c50 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e7807eae0 con 0x7f2e8806dc00 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e867fc700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e74060620 0x7f2e74062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== osd.2 v2:172.21.15.111:6824/33933 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f11e806c680 con 0x7f11e8069b80 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 0x7f9d24152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c009420 con 0x7f9d24160f30 2023-12-19T07:47:35.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d22ffd700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d2406ca60 0x7f9d24152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d237fe700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 0x7f9d2416d180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400cc10 tx=0x7f9d14007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e877fe700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f2e74069c50 0x7f2e7406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa05bfff700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06dc00 0x7fa05c147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f2e867fc700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e74060620 0x7f2e74062ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2e88151550 tx=0x7f2e80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400a850 con 0x7f9d24160f30 2023-12-19T07:47:35.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1400a9b0 con 0x7f9d24160f30 2023-12-19T07:47:35.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400ad70 con 0x7f9d24160f30 2023-12-19T07:47:35.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== osd.1 v2:172.21.15.111:6808/33932 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa97406c7d0 con 0x7fa974069cd0 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 -- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa05c06baa0 msgr2=0x7fa05c16f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa05c06baa0 0x7fa05c16f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 -- 172.21.15.111:0/2559903923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06dc00 msgr2=0x7fa05c147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa05c06dc00 0x7fa05c147cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa0613cb700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054009ae0 con 0x7fa05c158cc0 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7fa060bca700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa05c06baa0 0x7fa05c16f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa0613cb700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 0x7fa05c088460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa05c06eb60 tx=0x7fa054018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054008dd0 con 0x7fa05c158cc0 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054025460 con 0x7fa05c158cc0 2023-12-19T07:47:35.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054025820 con 0x7fa05c158cc0 2023-12-19T07:47:35.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2416d6c0 con 0x7f9d24160f30 2023-12-19T07:47:35.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.264+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2416dc40 con 0x7f9d24160f30 2023-12-19T07:47:35.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c16e2d0 con 0x7fa05c158cc0 2023-12-19T07:47:35.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d2404be40 con 0x7f9d24160f30 2023-12-19T07:47:35.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c16e890 con 0x7fa05c158cc0 2023-12-19T07:47:35.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7f6d02545700 1 Processor -- start 2023-12-19T07:47:35.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7f6d02545700 1 -- start start 2023-12-19T07:47:35.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.268+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa05c071120 con 0x7fa05c158cc0 2023-12-19T07:47:35.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f2e877fe700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f2e74069c50 0x7f2e7406c080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f6d02545700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6cfc158cc0 0x7f6cfc06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9d14012610 con 0x7f9d24160f30 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d040606a0 0x7f9d04062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9d1407d720 con 0x7f9d24160f30 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f9d04069cd0 0x7f9d0406c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d0406c7d0 con 0x7f9d04069cd0 2023-12-19T07:47:35.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d1407da50 con 0x7f9d24160f30 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f9d22ffd700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d040606a0 0x7f9d04062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== osd.5 v2:172.21.15.143:6809/34137 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2e7406c750 con 0x7f2e74069c50 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa054003040 con 0x7fa05c158cc0 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f6d02545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f6d02545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cfc06b0d0 con 0x7f6cfc06dc00 2023-12-19T07:47:35.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f6d02545700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cfc0b4790 con 0x7f6cfc158cc0 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.272+0000 7f6d02545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc06d710 con 0x7f6cfc0b3870 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f9d227fc700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f9d04069cd0 0x7f9d0406c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f9d227fc700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f9d04069cd0 0x7f9d0406c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa058ff9700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa0400606a0 0x7fa040062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa05407e7e0 con 0x7fa05c158cc0 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa058ff9700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7fa040069cd0 0x7fa04006c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa04006c7d0 con 0x7fa040069cd0 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa054080d20 con 0x7fa05c158cc0 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46822/0 (socket says 172.21.15.111:46822) 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 -- 172.21.15.111:0/2035241168 learned_addr learned my addr 172.21.15.111:0/2035241168 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6cfbfff700 1 --2- 172.21.15.111:0/2035241168 >> v2:172.21.15.111:3300/0 conn(0x7f6cfc158cc0 0x7f6cfc06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f9d22ffd700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d040606a0 0x7f9d04062b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c006fd0 tx=0x7f9d1c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa060bca700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa0400606a0 0x7fa040062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 -- 172.21.15.111:0/2035241168 >> v2:172.21.15.111:3300/0 conn(0x7f6cfc158cc0 msgr2=0x7f6cfc06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 --2- 172.21.15.111:0/2035241168 >> v2:172.21.15.111:3300/0 conn(0x7f6cfc158cc0 0x7f6cfc06d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 -- 172.21.15.111:0/2035241168 >> v1:172.21.15.111:6789/0 conn(0x7f6cfc06dc00 legacy=0x7f6cfc06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7fa05bfff700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7fa040069cd0 0x7fa04006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== osd.4 v2:172.21.15.143:6808/34138 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9d0406c7d0 con 0x7f9d04069cd0 2023-12-19T07:47:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 -- 172.21.15.111:0/2035241168 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc06a7e0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.276+0000 7f6d00ae2700 1 --2- 172.21.15.111:0/2035241168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0099b0 tx=0x7f6cec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74677e191b72fddf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6cfaffd700 1 -- 172.21.15.111:0/2035241168 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00b860 con 0x7f6cfc0b3870 2023-12-19T07:47:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7fa05bfff700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7fa040069cd0 0x7fa04006c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 --> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46580384d0 con 0x7f464406a970 2023-12-19T07:47:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6cfaffd700 1 -- 172.21.15.111:0/2035241168 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec00b9c0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6cfaffd700 1 -- 172.21.15.111:0/2035241168 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00bd60 con 0x7f6cfc0b3870 2023-12-19T07:47:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7fa060bca700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa0400606a0 0x7fa040062b50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa050003a60 tx=0x7fa050009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f464bfff700 1 -- 172.21.15.111:0/2744156205 <== osd.0 v2:172.21.15.111:6800/33928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46580384d0 con 0x7f464406a970 2023-12-19T07:47:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== osd.6 v2:172.21.15.143:6800/34136 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa04006c7d0 con 0x7fa040069cd0 2023-12-19T07:47:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f464406a970 msgr2=0x7f464406ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f464406a970 0x7f464406ce00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 -- 172.21.15.111:0/2035241168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 msgr2=0x7f6cfc0b3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 --2- 172.21.15.111:0/2035241168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0099b0 tx=0x7f6cec009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4644060620 msgr2=0x7f4644062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4644060620 0x7f4644062ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f464c007fd0 tx=0x7f464c016000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 msgr2=0x7f465803a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 --2- 172.21.15.111:0/2744156205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46580e3bb0 0x7f465803a600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4660075ed0 tx=0x7f4660070c50 comp rx=0 tx=0).stop 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 -- 172.21.15.111:0/2035241168 shutdown_connections 2023-12-19T07:47:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 --2- 172.21.15.111:0/2035241168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc0b3870 0x7f6cfc0b3c60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 --2- 172.21.15.111:0/2035241168 >> v2:172.21.15.111:3300/0 conn(0x7f6cfc158cc0 0x7f6cfc06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 -- 172.21.15.111:0/2035241168 >> 172.21.15.111:0/2035241168 conn(0x7f6cfc067bb0 msgr2=0x7f6cfc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4664815700 1 -- 172.21.15.111:0/2744156205 reap_dead start 2023-12-19T07:47:35.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 shutdown_connections 2023-12-19T07:47:35.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 >> 172.21.15.111:0/2744156205 conn(0x7f4658051b50 msgr2=0x7f4658006700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 -- 172.21.15.111:0/2035241168 shutdown_connections 2023-12-19T07:47:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f6d02545700 1 -- 172.21.15.111:0/2035241168 wait complete. 2023-12-19T07:47:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 shutdown_connections 2023-12-19T07:47:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.280+0000 7f4665016700 1 -- 172.21.15.111:0/2744156205 wait complete. 2023-12-19T07:47:35.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.284+0000 7f6d02545700 1 Processor -- start 2023-12-19T07:47:35.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.284+0000 7f6d02545700 1 -- start start 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc06dc00 0x7f6cfc0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 0x7f6cfc157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cfc158cc0 0x7f6cfc087e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc089de0 con 0x7f6cfc158cc0 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc087240 con 0x7f6cfc0b3870 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d02545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc0894b0 con 0x7f6cfc06dc00 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 0x7f6cfc157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 0x7f6cfc157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47806/0 (socket says 172.21.15.111:47806) 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 -- 172.21.15.111:0/337489765 learned_addr learned my addr 172.21.15.111:0/337489765 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6d00ae2700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cfc158cc0 0x7f6cfc087e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 -- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc06dc00 msgr2=0x7f6cfc0afc50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-19T07:47:35.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 --> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa990147070 con 0x7fa974069cd0 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cfc06dc00 0x7f6cfc0afc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 -- 172.21.15.111:0/337489765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cfc158cc0 msgr2=0x7f6cfc087e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cfc158cc0 0x7f6cfc087e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.288+0000 7f6cfb7fe700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0094f0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6cfb7fe700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 0x7f6cfc157bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000c4e0 tx=0x7f6cf000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011590 con 0x7f6cfc0b3870 2023-12-19T07:47:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7fa985ffb700 1 -- 172.21.15.111:0/3385310882 <== osd.1 v2:172.21.15.111:6808/33932 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa97406c7d0 con 0x7fa974069cd0 2023-12-19T07:47:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc089790 con 0x7f6cfc0b3870 2023-12-19T07:47:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc0b0190 con 0x7f6cfc0b3870 2023-12-19T07:47:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0011710 con 0x7f6cfc0b3870 2023-12-19T07:47:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011ad0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 --> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc598002d30 con 0x7fc594069c60 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.292+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 --> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1208069310 con 0x7f11e8069b80 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fa974069cd0 msgr2=0x7fa97406c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fa974069cd0 0x7fa97406c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa9740606a0 msgr2=0x7fa974062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa9740606a0 0x7fa974062b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9800099b0 tx=0x7fa980017000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 msgr2=0x7fa9900ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 --2- 172.21.15.111:0/3385310882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9900abca0 0x7fa9900ac970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa98800cfb0 tx=0x7fa988007870 comp rx=0 tx=0).stop 2023-12-19T07:47:35.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11f67fc700 1 -- 172.21.15.111:0/3428098907 <== osd.2 v2:172.21.15.111:6824/33933 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1208069310 con 0x7f11e8069b80 2023-12-19T07:47:35.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5aaffd700 1 -- 172.21.15.111:0/842646108 <== osd.3 v2:172.21.15.111:6816/33942 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc598002d30 con 0x7fc594069c60 2023-12-19T07:47:35.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996162700 1 -- 172.21.15.111:0/3385310882 reap_dead start 2023-12-19T07:47:35.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6cdc000fd0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 shutdown_connections 2023-12-19T07:47:35.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 >> 172.21.15.111:0/3385310882 conn(0x7fa990067bb0 msgr2=0x7fa9900ca480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7f11e8069b80 msgr2=0x7f11e806bfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7f11e8069b80 0x7f11e806bfb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 shutdown_connections 2023-12-19T07:47:35.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fa996963700 1 -- 172.21.15.111:0/3385310882 wait complete. 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11e8060550 msgr2=0x7f11e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11e8060550 0x7f11e8062a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11f8007fd0 tx=0x7f11f8009040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12080889d0 msgr2=0x7f12080ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 --2- 172.21.15.111:0/3428098907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12080889d0 0x7f12080ccf50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12080ceab0 tx=0x7f11fc017000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f1206d9d700 1 -- 172.21.15.111:0/3428098907 reap_dead start 2023-12-19T07:47:35.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7fc594069c60 msgr2=0x7fc59406c090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7fc594069c60 0x7fc59406c090 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 shutdown_connections 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc594060630 msgr2=0x7fc594062ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc594060630 0x7fc594062ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0011e0 tx=0x7fc5ac00a610 comp rx=0 tx=0).stop 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b4160e60 msgr2=0x7fc5b415bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 --2- 172.21.15.111:0/842646108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b4160e60 0x7fc5b415bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c560 tx=0x7fc59c007d50 comp rx=0 tx=0).stop 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b37fe700 1 -- 172.21.15.111:0/842646108 reap_dead start 2023-12-19T07:47:35.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 >> 172.21.15.111:0/3428098907 conn(0x7f120806d050 msgr2=0x7f12080b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 shutdown_connections 2023-12-19T07:47:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 >> 172.21.15.111:0/842646108 conn(0x7fc5b4067b60 msgr2=0x7fc5b415d420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 shutdown_connections 2023-12-19T07:47:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 shutdown_connections 2023-12-19T07:47:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7fc5b986e700 1 -- 172.21.15.111:0/842646108 wait complete. 2023-12-19T07:47:35.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6cf001b3b0 con 0x7f6cfc0b3870 2023-12-19T07:47:35.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.296+0000 7f11e7fff700 1 -- 172.21.15.111:0/3428098907 wait complete. 2023-12-19T07:47:35.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6ce3fff700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ce8060550 0x7f6ce8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 --> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e8806aa80 con 0x7f2e74069c50 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6cf007c010 con 0x7f6cfc0b3870 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6ce3fff700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f6ce8069a10 0x7f6ce806be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6cfbfff700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ce8060550 0x7f6ce8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ce806c510 con 0x7f6ce8069a10 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cf007e550 con 0x7f6cfc0b3870 2023-12-19T07:47:35.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e6effd700 1 -- 172.21.15.111:0/2354348062 <== osd.5 v2:172.21.15.143:6809/34137 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e8806aa80 con 0x7f2e74069c50 2023-12-19T07:47:35.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f2e74069c50 msgr2=0x7f2e7406c080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f2e74069c50 0x7f2e7406c080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6d00ae2700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f6ce8069a10 0x7f6ce806be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e74060620 msgr2=0x7f2e74062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e74060620 0x7f2e74062ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2e88151550 tx=0x7f2e80009040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 msgr2=0x7f2e8815d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e8d585700 1 --2- 172.21.15.111:0/2354348062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e8806dc00 0x7f2e8815d340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c7e0 tx=0x7f2e78007f20 comp rx=0 tx=0).stop 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f2e877fe700 1 -- 172.21.15.111:0/2354348062 reap_dead start 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6cfbfff700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ce8060550 0x7f6ce8062a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007460 tx=0x7f6cf4009a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f6d00ae2700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f6ce8069a10 0x7f6ce806be40 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.300+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 --> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9d2406e030 con 0x7f9d04069cd0 2023-12-19T07:47:35.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 shutdown_connections 2023-12-19T07:47:35.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 >> 172.21.15.111:0/2354348062 conn(0x7f2e88067bb0 msgr2=0x7f2e880cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 shutdown_connections 2023-12-19T07:47:35.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d1affd700 1 -- 172.21.15.111:0/1429027085 <== osd.4 v2:172.21.15.143:6808/34138 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9d2406e030 con 0x7f9d04069cd0 2023-12-19T07:47:35.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f2e8d585700 1 -- 172.21.15.111:0/2354348062 wait complete. 2023-12-19T07:47:35.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f9d04069cd0 msgr2=0x7f9d0406c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f9d04069cd0 0x7f9d0406c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d040606a0 msgr2=0x7f9d04062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d040606a0 0x7f9d04062b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c006fd0 tx=0x7f9d1c009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== osd.7 v2:172.21.15.143:6824/34139 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6ce806c510 con 0x7f6ce8069a10 2023-12-19T07:47:35.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 msgr2=0x7f9d2416d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 --2- 172.21.15.111:0/1429027085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d24160f30 0x7f9d2416d180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400cc10 tx=0x7f9d14007730 comp rx=0 tx=0).stop 2023-12-19T07:47:35.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d237fe700 1 -- 172.21.15.111:0/1429027085 reap_dead start 2023-12-19T07:47:35.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 shutdown_connections 2023-12-19T07:47:35.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.304+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 >> 172.21.15.111:0/1429027085 conn(0x7f9d24067bb0 msgr2=0x7f9d240c81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.308+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 shutdown_connections 2023-12-19T07:47:35.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.308+0000 7f9d2962e700 1 -- 172.21.15.111:0/1429027085 wait complete. 2023-12-19T07:47:35.319 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-19T07:47:35.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.312+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 --> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa05c06aa80 con 0x7fa040069cd0 2023-12-19T07:47:35.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa058ff9700 1 -- 172.21.15.111:0/2559903923 <== osd.6 v2:172.21.15.143:6800/34136 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa05c06aa80 con 0x7fa040069cd0 2023-12-19T07:47:35.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7fa040069cd0 msgr2=0x7fa04006c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7fa040069cd0 0x7fa04006c100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa0400606a0 msgr2=0x7fa040062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa0400606a0 0x7fa040062b50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa050003a60 tx=0x7fa050009040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 msgr2=0x7fa05c088460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 --2- 172.21.15.111:0/2559903923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa05c158cc0 0x7fa05c088460 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa05c06eb60 tx=0x7fa054018000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa0613cb700 1 -- 172.21.15.111:0/2559903923 reap_dead start 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 shutdown_connections 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.316+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 >> 172.21.15.111:0/2559903923 conn(0x7fa05c067bb0 msgr2=0x7fa05c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.321 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-12-19T07:47:35.322 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T07:47:35.323 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-19T07:47:35.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.320+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 shutdown_connections 2023-12-19T07:47:35.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.320+0000 7fa062e2e700 1 -- 172.21.15.111:0/2559903923 wait complete. 2023-12-19T07:47:35.324 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.324 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-19T07:47:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 --> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6cdc002d30 con 0x7f6ce8069a10 2023-12-19T07:47:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6ce3fff700 1 -- 172.21.15.111:0/337489765 <== osd.7 v2:172.21.15.143:6824/34139 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6cdc002d30 con 0x7f6ce8069a10 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f6ce8069a10 msgr2=0x7f6ce806be40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f6ce8069a10 0x7f6ce806be40 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ce8060550 msgr2=0x7f6ce8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ce8060550 0x7f6ce8062a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007460 tx=0x7f6cf4009a10 comp rx=0 tx=0).stop 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 msgr2=0x7f6cfc157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 --2- 172.21.15.111:0/337489765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc0b3870 0x7f6cfc157bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000c4e0 tx=0x7f6cf000c8a0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d00ae2700 1 -- 172.21.15.111:0/337489765 reap_dead start 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 shutdown_connections 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 >> 172.21.15.111:0/337489765 conn(0x7f6cfc067bb0 msgr2=0x7f6cfc0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 shutdown_connections 2023-12-19T07:47:35.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.328+0000 7f6d02545700 1 -- 172.21.15.111:0/337489765 wait complete. 2023-12-19T07:47:35.327 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-19T07:47:35.328 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-19T07:47:35.329 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-19T07:47:35.330 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-12-19T07:47:35.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-19T07:47:35.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.592+0000 7f3f4a723700 1 Processor -- start 2023-12-19T07:47:35.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.592+0000 7f3f4a723700 1 -- start start 2023-12-19T07:47:35.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f4a723700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f3c053d10 0x7f3f3c0ddf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f4a723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 0x7f3f3c04fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f4a723700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0348a0 con 0x7f3f3c00c250 2023-12-19T07:47:35.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f4a723700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c054b50 con 0x7f3f3c053d10 2023-12-19T07:47:35.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f4a723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0542b0 con 0x7f3f3c0de440 2023-12-19T07:47:35.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 0x7f3f3c04fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49721700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3f3c00c250 0x7f3f3c034190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56078/0 (socket says 172.21.15.111:56078) 2023-12-19T07:47:35.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49721700 1 -- 172.21.15.111:0/1459750122 learned_addr learned my addr 172.21.15.111:0/1459750122 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 Processor -- start 2023-12-19T07:47:35.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 -- start start 2023-12-19T07:47:35.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd29c036a40 0x7fd29c0e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c0e44a0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c0e3ab0 con 0x7fd29c038ff0 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2aa20c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c0e4780 con 0x7fd29c036a40 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2a920a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2a920a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34670/0 (socket says 172.21.15.111:34670) 2023-12-19T07:47:35.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2a920a700 1 -- 172.21.15.111:0/4177336450 learned_addr learned my addr 172.21.15.111:0/4177336450 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4177336450 >> v2:172.21.15.143:3300/0 conn(0x7fd29c036a40 0x7fd29c0e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4177336450 >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 msgr2=0x7fd29c0393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f3bfff700 1 -- 172.21.15.111:0/1459750122 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744396775 0 0) 0x7f3f34001110 con 0x7f3f3c00c250 2023-12-19T07:47:35.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f3bfff700 1 -- 172.21.15.111:0/1459750122 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f30003570 con 0x7f3f3c00c250 2023-12-19T07:47:35.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49f22700 1 -- 172.21.15.111:0/1459750122 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c053d10 msgr2=0x7f3f3c0ddf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49f22700 1 --2- 172.21.15.111:0/1459750122 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c053d10 0x7f3f3c0ddf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.596+0000 7f3f49f22700 1 -- 172.21.15.111:0/1459750122 >> v1:172.21.15.111:6789/0 conn(0x7f3f3c00c250 legacy=0x7f3f3c034190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f49f22700 1 -- 172.21.15.111:0/1459750122 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c054590 con 0x7f3f3c0de440 2023-12-19T07:47:35.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd29bfff700 1 -- 172.21.15.111:0/4177336450 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735164492 0 0) 0x7fd29c0e44a0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f49f22700 1 --2- 172.21.15.111:0/1459750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 0x7f3f3c04fe30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3f44071030 tx=0x7f3f44066cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddc10aaa6245c8bd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4177336450 >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4177336450 >> v1:172.21.15.143:6789/0 conn(0x7fd29c0e2fd0 legacy=0x7fd29c0e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4177336450 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0e4a60 con 0x7fd29c036a40 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a920a700 1 --2- 172.21.15.111:0/4177336450 >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 Processor -- start 2023-12-19T07:47:35.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 -- start start 2023-12-19T07:47:35.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f3bfff700 1 -- 172.21.15.111:0/1459750122 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4404be40 con 0x7f3f3c0de440 2023-12-19T07:47:35.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f3bfff700 1 -- 172.21.15.111:0/1459750122 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f440008d0 con 0x7f3f3c0de440 2023-12-19T07:47:35.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f3bfff700 1 -- 172.21.15.111:0/1459750122 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44070460 con 0x7f3f3c0de440 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4177336450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c036a40 0x7fd29c0e3c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2a404c2e0 tx=0x7fd2a4070c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69fc5374c1dd52f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd29bfff700 1 -- 172.21.15.111:0/4177336450 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4071120 con 0x7fd29c036a40 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd29bfff700 1 -- 172.21.15.111:0/4177336450 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4070250 con 0x7fd29c036a40 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd29bfff700 1 -- 172.21.15.111:0/4177336450 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a406fbc0 con 0x7fd29c036a40 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f4a723700 1 -- 172.21.15.111:0/1459750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 msgr2=0x7f3f3c04fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f3f4a723700 1 --2- 172.21.15.111:0/1459750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 0x7f3f3c04fe30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3f44071030 tx=0x7f3f44066cd0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc06d7d0 con 0x7f16bc140420 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc06cc50 con 0x7f16bc1407f0 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16c1145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc0ae280 con 0x7f16bc06c7c0 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fe379185700 1 Processor -- start 2023-12-19T07:47:35.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4177336450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c036a40 msgr2=0x7fd29c0e3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4177336450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c036a40 0x7fd29c0e3c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2a404c2e0 tx=0x7fd2a4070c50 comp rx=0 tx=0).stop 2023-12-19T07:47:35.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 -- 172.21.15.111:0/1459750122 shutdown_connections 2023-12-19T07:47:35.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 --2- 172.21.15.111:0/1459750122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c0de440 0x7f3f3c04fe30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 --2- 172.21.15.111:0/1459750122 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c053d10 0x7f3f3c0ddf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 -- 172.21.15.111:0/1459750122 >> 172.21.15.111:0/1459750122 conn(0x7f3f3c054da0 msgr2=0x7f3f3c0551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.600+0000 7f16ba59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34684/0 (socket says 172.21.15.111:34684) 2023-12-19T07:47:35.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 -- 172.21.15.111:0/934370265 learned_addr learned my addr 172.21.15.111:0/934370265 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4177336450 shutdown_connections 2023-12-19T07:47:35.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 -- start start 2023-12-19T07:47:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4177336450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c036a40 0x7fd29c0e3c70 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2a404c2e0 tx=0x7fd2a4070c50 comp rx=0 tx=0).stop 2023-12-19T07:47:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4177336450 >> v2:172.21.15.111:3300/0 conn(0x7fd29c038ff0 0x7fd29c0393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4177336450 >> 172.21.15.111:0/4177336450 conn(0x7fd29c009e80 msgr2=0x7fd29c00a250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 -- 172.21.15.111:0/1459750122 shutdown_connections 2023-12-19T07:47:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 -- 172.21.15.111:0/1459750122 wait complete. 2023-12-19T07:47:35.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 -- 172.21.15.111:0/934370265 >> v1:172.21.15.143:6789/0 conn(0x7f16bc1407f0 legacy=0x7f16bc0ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 -- 172.21.15.111:0/934370265 >> v1:172.21.15.111:6789/0 conn(0x7f16bc140420 legacy=0x7f16bc15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 -- 172.21.15.111:0/934370265 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc06cf30 con 0x7f16bc06c7c0 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe37415d490 0x7fe3740af640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe37415d860 0x7fe3740afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3740b0860 con 0x7fe37406da80 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3740c8a10 con 0x7fe37415d490 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe379185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe374067e60 con 0x7fe37415d860 2023-12-19T07:47:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 Processor -- start 2023-12-19T07:47:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16ba59c700 1 --2- 172.21.15.111:0/934370265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16b00099b0 tx=0x7f16b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb959e97a973da2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe37415d490 0x7fe3740af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f3f4a723700 1 -- start start 2023-12-19T07:47:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4177336450 shutdown_connections 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe37415d490 0x7fe3740af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34696/0 (socket says 172.21.15.111:34696) 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 -- 172.21.15.111:0/709497374 learned_addr learned my addr 172.21.15.111:0/709497374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe37259c700 1 --2- 172.21.15.111:0/709497374 >> v2:172.21.15.143:3300/0 conn(0x7fe37415d860 0x7fe3740afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 -- 172.21.15.111:0/709497374 >> v2:172.21.15.143:3300/0 conn(0x7fe37415d860 msgr2=0x7fe3740afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 --2- 172.21.15.111:0/709497374 >> v2:172.21.15.143:3300/0 conn(0x7fe37415d860 0x7fe3740afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 -- 172.21.15.111:0/709497374 >> v1:172.21.15.143:6789/0 conn(0x7fe37406da80 legacy=0x7fe37406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fe372d9d700 1 -- 172.21.15.111:0/709497374 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe374068140 con 0x7fe37415d490 2023-12-19T07:47:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7f16b9d9b700 1 -- 172.21.15.111:0/934370265 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b000b860 con 0x7f16bc06c7c0 2023-12-19T07:47:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f16b9d9b700 1 -- 172.21.15.111:0/934370265 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b000b9c0 con 0x7f16bc06c7c0 2023-12-19T07:47:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f16b9d9b700 1 -- 172.21.15.111:0/934370265 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b000bd60 con 0x7f16bc06c7c0 2023-12-19T07:47:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.604+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4177336450 wait complete. 2023-12-19T07:47:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe372d9d700 1 --2- 172.21.15.111:0/709497374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37415d490 0x7fe3740af640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe37406a130 tx=0x7fe368009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267a58abbabfda7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c00c250 0x7f3f3c039e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c053d10 0x7f3f3c03a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c039560 con 0x7f3f3c0de440 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0389e0 con 0x7f3f3c00c250 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f4a723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c038cc0 con 0x7f3f3c053d10 2023-12-19T07:47:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f16c1145700 1 -- 172.21.15.111:0/934370265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 msgr2=0x7f16bc0b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f16c1145700 1 --2- 172.21.15.111:0/934370265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16b00099b0 tx=0x7f16b0009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe371d9b700 1 -- 172.21.15.111:0/709497374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe368009e70 con 0x7fe37415d490 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34710/0 (socket says 172.21.15.111:34710) 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 -- 172.21.15.111:0/1555673943 learned_addr learned my addr 172.21.15.111:0/1555673943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe371d9b700 1 -- 172.21.15.111:0/709497374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36800b820 con 0x7fe37415d490 2023-12-19T07:47:35.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe371d9b700 1 -- 172.21.15.111:0/709497374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36800bbc0 con 0x7fe37415d490 2023-12-19T07:47:35.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fd2aa20c700 1 Processor -- start 2023-12-19T07:47:35.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fd2aa20c700 1 -- start start 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 -- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c053d10 msgr2=0x7f3f3c03a3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c053d10 0x7f3f3c03a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 -- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c00c250 msgr2=0x7f3f3c039e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c00c250 0x7f3f3c039e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 -- 172.21.15.111:0/1555673943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c038fa0 con 0x7f3f3c0de440 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7f3f49f22700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f4404cd50 tx=0x7f3f4408a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 -- 172.21.15.111:0/709497374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37415d490 msgr2=0x7fe3740af640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 --2- 172.21.15.111:0/709497374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37415d490 0x7fe3740af640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe37406a130 tx=0x7fe368009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 -- 172.21.15.111:0/709497374 shutdown_connections 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 --2- 172.21.15.111:0/709497374 >> v2:172.21.15.143:3300/0 conn(0x7fe37415d860 0x7fe3740afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 --2- 172.21.15.111:0/709497374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37415d490 0x7fe3740af640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.608+0000 7fe379185700 1 -- 172.21.15.111:0/709497374 >> 172.21.15.111:0/709497374 conn(0x7fe37406ab50 msgr2=0x7fe3740ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 0x7fd29c04a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 0x7fd29c0ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd29c0f5a00 0x7fd29c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fe379185700 1 -- 172.21.15.111:0/709497374 shutdown_connections 2023-12-19T07:47:35.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fe379185700 1 -- 172.21.15.111:0/709497374 wait complete. 2023-12-19T07:47:35.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 -- 172.21.15.111:0/934370265 shutdown_connections 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 --2- 172.21.15.111:0/934370265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc0b0cd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 -- 172.21.15.111:0/934370265 >> 172.21.15.111:0/934370265 conn(0x7f16bc067b60 msgr2=0x7f16bc0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a920a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 0x7fd29c04a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a920a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 0x7fd29c04a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46842/0 (socket says 172.21.15.111:46842) 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a9a0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd29c0f5a00 0x7fd29c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a920a700 1 -- 172.21.15.111:0/4144791733 learned_addr learned my addr 172.21.15.111:0/4144791733 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c04ae60 con 0x7fd29c0f5a00 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c03a5d0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c0ff7f0 con 0x7fd29c038ff0 2023-12-19T07:47:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 0x7fd29c0ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 -- 172.21.15.111:0/934370265 shutdown_connections 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 -- 172.21.15.111:0/934370265 wait complete. 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44094070 con 0x7f3f3c0de440 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44064160 con 0x7f3f3c0de440 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4406d940 con 0x7f3f3c0de440 2023-12-19T07:47:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f4a723700 1 -- 172.21.15.111:0/1555673943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c039280 con 0x7f3f3c0de440 2023-12-19T07:47:35.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f4a723700 1 -- 172.21.15.111:0/1555673943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c0d27f0 con 0x7f3f3c0de440 2023-12-19T07:47:35.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f1abc678700 1 Processor -- start 2023-12-19T07:47:35.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f1abc678700 1 -- start start 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 msgr2=0x7fd29c04a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 0x7fd29c04a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4144791733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd29c0f5a00 msgr2=0x7fd29c03aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd29c0f5a00 0x7fd29c03aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940094f0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2a8a09700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 0x7fd29c0ff2b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4065590 tx=0x7fd2a408c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f3f4a723700 1 -- 172.21.15.111:0/1555673943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c0de8d0 con 0x7f3f3c0de440 2023-12-19T07:47:35.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a409e070 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a406f9a0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4094040 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4093050 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fe379185700 1 Processor -- start 2023-12-19T07:47:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c04f9c0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f16c1145700 1 Processor -- start 2023-12-19T07:47:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.612+0000 7f1abc678700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1ab415a9f0 0x7f1ab40c8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c03a7d0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 -- start start 2023-12-19T07:47:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1abc678700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 0x7f1ab40c8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1abc678700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab415e3a0 con 0x7f1ab406b200 2023-12-19T07:47:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1abc678700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab415e680 con 0x7f1ab415a9f0 2023-12-19T07:47:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 -- start start 2023-12-19T07:47:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1abc678700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab406a720 con 0x7f1ab415adc0 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1ab9c13700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 0x7f1ab40c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1ab9c13700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 0x7f1ab40c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47848/0 (socket says 172.21.15.111:47848) 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1ab9c13700 1 -- 172.21.15.111:0/2234890710 learned_addr learned my addr 172.21.15.111:0/2234890710 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1abac15700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1ab406b200 0x7f1ab415d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46374/0 (socket says 172.21.15.111:46374) 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 --2- 172.21.15.111:0/2234890710 >> v2:172.21.15.111:3300/0 conn(0x7f1ab415a9f0 0x7f1ab40c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 0x7f16bc0c2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 0x7f16bc13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0c13f0 con 0x7f16bc06c7c0 2023-12-19T07:47:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0c1690 con 0x7f16bc140420 2023-12-19T07:47:35.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1ab9412700 1 -- 172.21.15.111:0/2234890710 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966854549 0 0) 0x7f1ab415e3a0 con 0x7f1ab406b200 2023-12-19T07:47:35.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1ab9412700 1 -- 172.21.15.111:0/2234890710 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa0003570 con 0x7f1ab406b200 2023-12-19T07:47:35.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16c1145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0c1970 con 0x7f16bc1407f0 2023-12-19T07:47:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 -- 172.21.15.111:0/2234890710 >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 msgr2=0x7f1ab40c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 --2- 172.21.15.111:0/2234890710 >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 0x7f1ab40c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 -- 172.21.15.111:0/2234890710 >> v1:172.21.15.143:6789/0 conn(0x7f1ab406b200 legacy=0x7f1ab415d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 -- 172.21.15.111:0/2234890710 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab406aa00 con 0x7f1ab415a9f0 2023-12-19T07:47:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16bb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 0x7f16bc13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16bb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 0x7f16bc13bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46850/0 (socket says 172.21.15.111:46850) 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16bb59e700 1 -- 172.21.15.111:0/3342148275 learned_addr learned my addr 172.21.15.111:0/3342148275 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16ba59c700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 0x7f16bc0c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f16bad9d700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f1aba414700 1 --2- 172.21.15.111:0/2234890710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415a9f0 0x7f1ab40c8150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40c5f70 tx=0x7f1aac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa5b28a7043e43e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37406da80 0x7fe3740c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3741517c0 0x7fe3740c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740c7c40 con 0x7fe37406da80 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740c5660 con 0x7fe3741517c0 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fe379185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3740c73a0 con 0x7fe37415d490 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd2a406f070 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.616+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3f4406dd30 con 0x7f3f3c0de440 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 -- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 msgr2=0x7f16bc13bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 0x7f16bc13bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3741517c0 0x7fe3740c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46854/0 (socket says 172.21.15.111:46854) 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 -- 172.21.15.111:0/3363263149 learned_addr learned my addr 172.21.15.111:0/3363263149 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 Processor -- start 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 -- 172.21.15.111:0/3342148275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 msgr2=0x7f16bc138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 -- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3741517c0 msgr2=0x7fe3740c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3741517c0 0x7fe3740c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 -- 172.21.15.111:0/3363263149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37406da80 msgr2=0x7fe3740c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37406da80 0x7fe3740c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 -- 172.21.15.111:0/3363263149 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3680094f0 con 0x7fe37415d490 2023-12-19T07:47:35.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 -- start start 2023-12-19T07:47:35.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 -- 172.21.15.111:0/3342148275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00094f0 con 0x7f16bc140420 2023-12-19T07:47:35.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f323e2ae700 1 Processor -- start 2023-12-19T07:47:35.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe37259c700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe36400ea50 tx=0x7fe3640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f3f397fa700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 0x7f3f28062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fd2997fa700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 0x7fd288062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f16ba59c700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 0x7f16bc0c2450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f16bc06a340 tx=0x7f16b0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364009630 con 0x7fe37415d490 2023-12-19T07:47:35.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f323e2ae700 1 -- start start 2023-12-19T07:47:35.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3f440f3e70 con 0x7f3f3c0de440 2023-12-19T07:47:35.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f3f49721700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 0x7f3f28062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd2a40f4c20 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36400c850 con 0x7fe37415d490 2023-12-19T07:47:35.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400cc10 con 0x7fe37415d490 2023-12-19T07:47:35.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe379185700 1 -- 172.21.15.111:0/3363263149 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3740c7680 con 0x7fe37415d490 2023-12-19T07:47:35.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fe379185700 1 -- 172.21.15.111:0/3363263149 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3740c3e00 con 0x7fe37415d490 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7f3f49721700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 0x7f3f28062ad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f340058d0 tx=0x7f3f34001130 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb806dc00 0x7fcdb806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcdb806baa0 0x7fcdb80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb806b0d0 con 0x7fcdb8158cc0 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb806beb0 con 0x7fcdb806baa0 2023-12-19T07:47:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbf478700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80b4790 con 0x7fcdb806dc00 2023-12-19T07:47:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fd2a920a700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 0x7fd288062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.620+0000 7fcdbd214700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcdb8158cc0 0x7fcdb806d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56102/0 (socket says 172.21.15.111:56102) 2023-12-19T07:47:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 -- 172.21.15.111:0/1208126956 learned_addr learned my addr 172.21.15.111:0/1208126956 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323e2ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32380afef0 0x7f323806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c0052b0 con 0x7fe37415d490 2023-12-19T07:47:35.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3f440c42d0 con 0x7f3f3c0de440 2023-12-19T07:47:35.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323e2ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3238160e60 0x7f323815d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f323815d050 0x7f32380af7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56104/0 (socket says 172.21.15.111:56104) 2023-12-19T07:47:35.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 -- 172.21.15.111:0/3568260308 learned_addr learned my addr 172.21.15.111:0/3568260308 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323806ba50 con 0x7f323815d050 2023-12-19T07:47:35.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbda15700 1 --2- 172.21.15.111:0/1208126956 >> v2:172.21.15.111:3300/0 conn(0x7fcdb806baa0 0x7fcdb80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fd2a920a700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 0x7fd288062a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd294006d40 tx=0x7fd294008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915840260 0 0) 0x7fcdb806b0d0 con 0x7fcdb8158cc0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1ab9412700 1 -- 172.21.15.111:0/2234890710 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00e070 con 0x7f1ab415a9f0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1ab9412700 1 -- 172.21.15.111:0/2234890710 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00bab0 con 0x7f1ab415a9f0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1ab9412700 1 -- 172.21.15.111:0/2234890710 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00be30 con 0x7f1ab415a9f0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323806bee0 con 0x7f32380afef0 2023-12-19T07:47:35.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32380af600 con 0x7f3238160e60 2023-12-19T07:47:35.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbca13700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb806dc00 0x7fcdb806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda4003570 con 0x7fcdb8158cc0 2023-12-19T07:47:35.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f32377fe700 1 --2- 172.21.15.111:0/3568260308 >> v2:172.21.15.111:3300/0 conn(0x7f32380afef0 0x7f323806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700430905 0 0) 0x7f323806ba50 con 0x7f323815d050 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872736219 0 0) 0x7fcda4003570 con 0x7fcdb8158cc0 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb806dc00 msgr2=0x7fcdb806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb806dc00 0x7fcdb806b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 >> v2:172.21.15.111:3300/0 conn(0x7fcdb806baa0 msgr2=0x7fcdb80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 --2- 172.21.15.111:0/1208126956 >> v2:172.21.15.111:3300/0 conn(0x7fcdb806baa0 0x7fcdb80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb806a7e0 con 0x7fcdb8158cc0 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3220003570 con 0x7f323815d050 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323c84b700 1 --2- 172.21.15.111:0/3568260308 >> v2:172.21.15.143:3300/0 conn(0x7f3238160e60 0x7f323815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155819158 0 0) 0x7f3220003570 con 0x7f323815d050 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 >> v2:172.21.15.143:3300/0 conn(0x7f3238160e60 msgr2=0x7f323815d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 --2- 172.21.15.111:0/3568260308 >> v2:172.21.15.143:3300/0 conn(0x7f3238160e60 0x7f323815d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 >> v2:172.21.15.111:3300/0 conn(0x7f32380afef0 msgr2=0x7f323806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 --2- 172.21.15.111:0/3568260308 >> v2:172.21.15.111:3300/0 conn(0x7f32380afef0 0x7f323806a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323806ae40 con 0x7f323815d050 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3228001470 con 0x7f323815d050 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 >> v1:172.21.15.111:6789/0 conn(0x7f323815d050 legacy=0x7f32380af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 0x7f3220003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 -- 172.21.15.111:0/2234890710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415a9f0 msgr2=0x7f1ab40c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 --2- 172.21.15.111:0/2234890710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415a9f0 0x7f1ab40c8150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40c5f70 tx=0x7f1aac009780 comp rx=0 tx=0).stop 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fcdb4001470 con 0x7fcdb8158cc0 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 >> v1:172.21.15.111:6789/0 conn(0x7fcdb8158cc0 legacy=0x7fcdb806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4003a60 0x7fcda4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220004810 0x7f3220004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32200051c0 0x7f322000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323806bee0 con 0x7f32200051c0 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32380af600 con 0x7f3220003a60 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322000c130 con 0x7f3220004810 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 0x7f3220003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f6dce1c5700 1 Processor -- start 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 -- 172.21.15.111:0/2234890710 shutdown_connections 2023-12-19T07:47:35.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 --2- 172.21.15.111:0/2234890710 >> v2:172.21.15.143:3300/0 conn(0x7f1ab415adc0 0x7f1ab40c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4004810 0x7fcda4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda40051c0 0x7fcda400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb806beb0 con 0x7fcda4004810 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80b4790 con 0x7fcda4003a60 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda400c100 con 0x7fcda40051c0 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbca13700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb806dc00 0x7fcdb806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4003a60 0x7fcda4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 --2- 172.21.15.111:0/2234890710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415a9f0 0x7f1ab40c8150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 -- 172.21.15.111:0/2234890710 >> 172.21.15.111:0/2234890710 conn(0x7f1ab40670e0 msgr2=0x7f1ab40674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 -- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220004810 msgr2=0x7f3220004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220004810 0x7f3220004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 -- 172.21.15.111:0/3568260308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32200051c0 msgr2=0x7f322000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32200051c0 0x7f322000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 -- 172.21.15.111:0/3568260308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220003570 con 0x7f3220003a60 2023-12-19T07:47:35.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f323c84b700 1 -- 172.21.15.111:0/3568260308 reap_dead start 2023-12-19T07:47:35.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbda15700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda40051c0 0x7fcda400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 -- 172.21.15.111:0/2234890710 shutdown_connections 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 -- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda40051c0 msgr2=0x7fcda400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda40051c0 0x7fcda400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 -- 172.21.15.111:0/1208126956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4004810 msgr2=0x7fcda4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4004810 0x7fcda4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 -- 172.21.15.111:0/1208126956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4003570 con 0x7fcda4003a60 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f1abc678700 1 -- 172.21.15.111:0/2234890710 wait complete. 2023-12-19T07:47:35.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbda15700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda40051c0 0x7fcda400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:35.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f6dce1c5700 1 -- start start 2023-12-19T07:47:35.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7f3237fff700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 0x7f3220003e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f322800c3c0 tx=0x7f323806b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d123a4f3b36d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe364013470 con 0x7fe37415d490 2023-12-19T07:47:35.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.624+0000 7fcdbd214700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4003a60 0x7fcda4003e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c3c0 tx=0x7fcdb4001e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bfc1181d3d35668 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7fe35affd700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 0x7fe360062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe36407d490 con 0x7fe37415d490 2023-12-19T07:47:35.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7fe372d9d700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 0x7fe360062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dce1c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc80cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dce1c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc806b4e0 con 0x7f6dc80cade0 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dce1c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc8156220 con 0x7f6dc80add40 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dce1c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80cb770 con 0x7f6dc80c8250 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc77fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6dc80add40 0x7f6dc8152900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46388/0 (socket says 172.21.15.111:46388) 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc77fe700 1 -- 172.21.15.111:0/2890324159 learned_addr learned my addr 172.21.15.111:0/2890324159 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/2890324159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc80cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1abc678700 1 Processor -- start 2023-12-19T07:47:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7fe372d9d700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 0x7fe360062a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe37406a130 tx=0x7fe368016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc67fc700 1 -- 172.21.15.111:0/2890324159 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557468714 0 0) 0x7f6dc806b4e0 con 0x7f6dc80cade0 2023-12-19T07:47:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc67fc700 1 -- 172.21.15.111:0/2890324159 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0003570 con 0x7f6dc80cade0 2023-12-19T07:47:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1abc678700 1 -- start start 2023-12-19T07:47:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc7fff700 1 -- 172.21.15.111:0/2890324159 >> v1:172.21.15.143:6789/0 conn(0x7f6dc80add40 legacy=0x7f6dc8152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc7fff700 1 -- 172.21.15.111:0/2890324159 >> v1:172.21.15.111:6789/0 conn(0x7f6dc80cade0 legacy=0x7f6dc80c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc7fff700 1 -- 172.21.15.111:0/2890324159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8155610 con 0x7f6dc80c8250 2023-12-19T07:47:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/2890324159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc80cb230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e90 tx=0x7f6dc0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ebf6af49739505 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1abc678700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1abc678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 0x7f1ab4156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1abc678700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415adc0 0x7f1ab415e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.628+0000 7f1aba414700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abc678700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40b02f0 con 0x7f1ab415adc0 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abc678700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40af6e0 con 0x7f1ab406b200 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1ab9c13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 0x7f1ab4156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abc678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40af9c0 con 0x7f1ab415a9f0 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47888/0 (socket says 172.21.15.111:47888) 2023-12-19T07:47:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 -- 172.21.15.111:0/3722492410 learned_addr learned my addr 172.21.15.111:0/3722492410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abac15700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415adc0 0x7f1ab415e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800cc90 con 0x7f3220003a60 2023-12-19T07:47:35.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb400cc90 con 0x7fcda4003a60 2023-12-19T07:47:35.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322800cdf0 con 0x7f3220003a60 2023-12-19T07:47:35.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f3236ffd700 1 -- 172.21.15.111:0/3568260308 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228005b70 con 0x7f3220003a60 2023-12-19T07:47:35.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 -- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 msgr2=0x7f1ab4156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0024440 con 0x7f16bc140420 2023-12-19T07:47:35.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 0x7f1ab4156610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 -- 172.21.15.111:0/3722492410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415adc0 msgr2=0x7f1ab415e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415adc0 0x7f1ab415e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00245a0 con 0x7f16bc140420 2023-12-19T07:47:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0024440 con 0x7f16bc140420 2023-12-19T07:47:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc0c1c50 con 0x7f16bc140420 2023-12-19T07:47:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdaffff700 1 -- 172.21.15.111:0/1208126956 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb400ce30 con 0x7fcda4003a60 2023-12-19T07:47:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 -- 172.21.15.111:0/3722492410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac009a20 con 0x7f1ab406b200 2023-12-19T07:47:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd29c03a7d0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc139200 con 0x7f16bc140420 2023-12-19T07:47:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aba414700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1aac013000 tx=0x7f1aac019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 msgr2=0x7f3220003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 0x7f3220003e70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f322800c3c0 tx=0x7f323806b9c0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1ab9c13700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 0x7f1ab4156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 -- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4003a60 msgr2=0x7fcda4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 --2- 172.21.15.111:0/1208126956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4003a60 0x7fcda4003e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c3c0 tx=0x7fcdb4001e90 comp rx=0 tx=0).stop 2023-12-19T07:47:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbda15700 1 -- 172.21.15.111:0/1208126956 reap_dead start 2023-12-19T07:47:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac012a40 con 0x7f1ab406b200 2023-12-19T07:47:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac021070 con 0x7f1ab406b200 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac012dd0 con 0x7f1ab406b200 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abc678700 1 -- 172.21.15.111:0/3722492410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab40afca0 con 0x7f1ab406b200 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 shutdown_connections 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 --2- 172.21.15.111:0/3568260308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220003a60 0x7f3220003e70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 >> 172.21.15.111:0/3568260308 conn(0x7f3238067b60 msgr2=0x7f32380ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 -- 172.21.15.111:0/1208126956 shutdown_connections 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 -- 172.21.15.111:0/1208126956 >> 172.21.15.111:0/1208126956 conn(0x7fcdb8067bb0 msgr2=0x7fcdb80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 shutdown_connections 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f323e2ae700 1 -- 172.21.15.111:0/3568260308 wait complete. 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f6dc67fc700 1 -- 172.21.15.111:0/2890324159 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000e070 con 0x7f6dc80c8250 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f6dc67fc700 1 -- 172.21.15.111:0/2890324159 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc000bc30 con 0x7f6dc80c8250 2023-12-19T07:47:35.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f6dc67fc700 1 -- 172.21.15.111:0/2890324159 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0012430 con 0x7f6dc80c8250 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 -- 172.21.15.111:0/1208126956 shutdown_connections 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fcdbf478700 1 -- 172.21.15.111:0/1208126956 wait complete. 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc071120 con 0x7f16bc140420 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f16b002d440 con 0x7f16bc140420 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f16a2ffd700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 0x7f16a8062ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f16b000af20 con 0x7f16bc140420 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f16bad9d700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 0x7f16a8062ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7f1abc678700 1 -- 172.21.15.111:0/3722492410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab4156b50 con 0x7f1ab406b200 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f6dce1c5700 1 -- 172.21.15.111:0/2890324159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 msgr2=0x7f6dc80cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/2890324159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc80cb230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e90 tx=0x7f6dc0009780 comp rx=0 tx=0).stop 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 Processor -- start 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 -- start start 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 0x7f32380ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 0x7f32380acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323816e590 con 0x7f3220003a60 2023-12-19T07:47:35.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32380aa8f0 con 0x7f32380cb150 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f3237fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323e2ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32380aabd0 con 0x7f32380cace0 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f3237fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34794/0 (socket says 172.21.15.111:34794) 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f3237fff700 1 -- 172.21.15.111:0/1459574229 learned_addr learned my addr 172.21.15.111:0/1459574229 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.632+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe3640512a0 con 0x7fe37415d490 2023-12-19T07:47:35.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7fcdbf478700 1 Processor -- start 2023-12-19T07:47:35.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f32377fe700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 0x7f32380ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7f323c84b700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 0x7f32380acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.636+0000 7fcdbf478700 1 -- start start 2023-12-19T07:47:35.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 -- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 msgr2=0x7f32380acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab404be40 con 0x7f1ab406b200 2023-12-19T07:47:35.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f6dce1c5700 1 -- 172.21.15.111:0/2890324159 shutdown_connections 2023-12-19T07:47:35.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/2890324159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc80cb230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f6dce1c5700 1 -- 172.21.15.111:0/2890324159 >> 172.21.15.111:0/2890324159 conn(0x7f6dc8067bb0 msgr2=0x7f6dc80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 0x7f32380acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 -- 172.21.15.111:0/1459574229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 msgr2=0x7f323816e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 -- 172.21.15.111:0/1459574229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32380aaeb0 con 0x7f32380cace0 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f323c84b700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 0x7f32380acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f16bad9d700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 0x7f16a8062ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16ac003860 tx=0x7f16ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f3237fff700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:35.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f32377fe700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 0x7f32380ac620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f322c00d860 tx=0x7f322c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4003a60 0x7fcdb80a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb80aa9d0 0x7fcdb80aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f6dce1c5700 1 -- 172.21.15.111:0/2890324159 shutdown_connections 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c0125f0 con 0x7f32380cace0 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1aac023030 con 0x7f1ab406b200 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f1aaa7fc700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 0x7f1a98062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c012770 con 0x7f32380cace0 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c012b30 con 0x7f32380cace0 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32380ab190 con 0x7f32380cace0 2023-12-19T07:47:35.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32380ab750 con 0x7f32380cace0 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80ab380 con 0x7fcda4003a60 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8176e90 con 0x7fcdb80aa9d0 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbf478700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8087bd0 con 0x7fcdb8087210 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46888/0 (socket says 172.21.15.111:46888) 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 -- 172.21.15.111:0/2186171741 learned_addr learned my addr 172.21.15.111:0/2186171741 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbd214700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4003a60 0x7fcdb80a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 -- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb80aa9d0 msgr2=0x7fcdb80aae40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb80aa9d0 0x7fcdb80aae40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 -- 172.21.15.111:0/2186171741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4003a60 msgr2=0x7fcdb80a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4003a60 0x7fcdb80a9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 -- 172.21.15.111:0/2186171741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8176bb0 con 0x7fcdb8087210 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7fcdbda15700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8067fd0 tx=0x7fcdb0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f6dce1c5700 1 -- 172.21.15.111:0/2890324159 wait complete. 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1aac07e9f0 con 0x7f1ab406b200 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7f1ab9c13700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 0x7f1a98062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.640+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32380ab970 con 0x7f32380cace0 2023-12-19T07:47:35.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0002c80 con 0x7fcdb8087210 2023-12-19T07:47:35.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0002de0 con 0x7fcdb8087210 2023-12-19T07:47:35.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0014650 con 0x7fcdb8087210 2023-12-19T07:47:35.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f16b004eab0 con 0x7f16bc140420 2023-12-19T07:47:35.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7f1ab9c13700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 0x7f1a98062a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4070a00 tx=0x7f1ab0009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb80a9500 con 0x7fcdb8087210 2023-12-19T07:47:35.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb80a9ac0 con 0x7fcdb8087210 2023-12-19T07:47:35.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7f6dce1c5700 1 Processor -- start 2023-12-19T07:47:35.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.644+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80a9ce0 con 0x7fcdb8087210 2023-12-19T07:47:35.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.648+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f322c013480 con 0x7f32380cace0 2023-12-19T07:47:35.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.648+0000 7f321ffff700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 0x7f3224062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.648+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f322c07d7b0 con 0x7f32380cace0 2023-12-19T07:47:35.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.648+0000 7f6dce1c5700 1 -- start start 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.648+0000 7f3237fff700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 0x7f3224062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f3237fff700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 0x7f3224062b00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3228007c20 tx=0x7f3228015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc80add40 0x7f6dc814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc814a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 0x7f6dc814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80ad250 con 0x7f6dc80cade0 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80ac1b0 con 0x7f6dc80add40 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc80ac490 con 0x7f6dc80c8250 2023-12-19T07:47:35.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc814a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc814a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46896/0 (socket says 172.21.15.111:46896) 2023-12-19T07:47:35.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc6ffd700 1 -- 172.21.15.111:0/3931632258 learned_addr learned my addr 172.21.15.111:0/3931632258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:35.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 0x7f6dc814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1aac04ec50 con 0x7f1ab406b200 2023-12-19T07:47:35.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc77fe700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc80add40 0x7f6dc814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 -- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 msgr2=0x7f6dc814a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc814a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 -- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc80add40 msgr2=0x7f6dc814f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc80add40 0x7f6dc814f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 -- 172.21.15.111:0/3931632258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009a20 con 0x7f6dc80cade0 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dc7fff700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 0x7f6dc814bc00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0016000 tx=0x7f6dc0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f322c04db10 con 0x7f32380cace0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0012e70 con 0x7f6dc80cade0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc80ac770 con 0x7f6dc80cade0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc80acd30 con 0x7f6dc80cade0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc001e630 con 0x7f6dc80cade0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.652+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc001ea30 con 0x7f6dc80cade0 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.656+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fcdb0003040 con 0x7fcdb8087210 2023-12-19T07:47:35.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.656+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da80052b0 con 0x7f6dc80cade0 2023-12-19T07:47:35.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.656+0000 7fcdad7fa700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 0x7fcd9c062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.660+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcdb007d5d0 con 0x7fcdb8087210 2023-12-19T07:47:35.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.660+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcdb007da20 con 0x7fcdb8087210 2023-12-19T07:47:35.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.660+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6dc0024070 con 0x7f6dc80cade0 2023-12-19T07:47:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.660+0000 7fcdbd214700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 0x7fcd9c062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.664+0000 7fcdbd214700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 0x7fcd9c062ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c3c0 tx=0x7fcdb4015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.664+0000 7f6daf7fe700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 0x7f6db4062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.664+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f6dc007f920 con 0x7f6dc80cade0 2023-12-19T07:47:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.664+0000 7f6dc77fe700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 0x7f6db4062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:35.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.668+0000 7f6dc77fe700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 0x7f6db4062a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80b0160 tx=0x7f6db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:35.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.672+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6dc004fc00 con 0x7f6dc80cade0 2023-12-19T07:47:35.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.948+0000 7f3f4a723700 1 -- 172.21.15.111:0/1555673943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3f3c0de8d0 con 0x7f3f3c0de440 2023-12-19T07:47:35.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.948+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd29c03a7d0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.933 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.948+0000 7f3f397fa700 1 -- 172.21.15.111:0/1555673943 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f3c0de8d0 con 0x7f3f3c0de440 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 msgr2=0x7f3f28062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 0x7f3f28062ad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f340058d0 tx=0x7f3f34001130 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 msgr2=0x7f3f3c0fa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f4404cd50 tx=0x7f3f4408a000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 shutdown_connections 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3f28060620 0x7f3f28062ad0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f340058d0 tx=0x7f3f34001130 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f3c0de440 0x7f3f3c0fa090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c053d10 0x7f3f3c03a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 --2- 172.21.15.111:0/1555673943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c00c250 0x7f3f3c039e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 >> 172.21.15.111:0/1555673943 conn(0x7f3f3c054da0 msgr2=0x7f3f3c006c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7fd2997fa700 1 -- 172.21.15.111:0/4144791733 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd29c03a7d0 con 0x7fd29c0e2fd0 2023-12-19T07:47:35.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 shutdown_connections 2023-12-19T07:47:35.936 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.952+0000 7f3f2effd700 1 -- 172.21.15.111:0/1555673943 wait complete. 2023-12-19T07:47:35.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.956+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 msgr2=0x7fd288062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.956+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 0x7fd288062a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd294006d40 tx=0x7fd294008040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.956+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 msgr2=0x7fd29c0ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.956+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 0x7fd29c0ff2b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4065590 tx=0x7fd2a408c000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 shutdown_connections 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd288060580 0x7fd288062a30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd29c0e2fd0 0x7fd29c0ff2b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd29c0f5a00 0x7fd29c03aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 --2- 172.21.15.111:0/4144791733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd29c038ff0 0x7fd29c04a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 >> 172.21.15.111:0/4144791733 conn(0x7fd29c009e80 msgr2=0x7fd29c0e10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 shutdown_connections 2023-12-19T07:47:35.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fd2aa20c700 1 -- 172.21.15.111:0/4144791733 wait complete. 2023-12-19T07:47:35.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe34c005830 con 0x7fe37415d490 2023-12-19T07:47:35.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.960+0000 7fe35affd700 1 -- 172.21.15.111:0/3363263149 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe364050e30 con 0x7fe37415d490 2023-12-19T07:47:35.945 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 msgr2=0x7fe360062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 0x7fe360062a00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe37406a130 tx=0x7fe368016000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 msgr2=0x7fe3740c8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe36400ea50 tx=0x7fe3640091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 shutdown_connections 2023-12-19T07:47:35.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f16bc0c17d0 con 0x7f16bc140420 2023-12-19T07:47:35.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe360060550 0x7fe360062a00 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe37406a130 tx=0x7fe368016000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3741517c0 0x7fe3740c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe37415d490 0x7fe3740c8ea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 --2- 172.21.15.111:0/3363263149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37406da80 0x7fe3740c38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.964+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 >> 172.21.15.111:0/3363263149 conn(0x7fe37406ab50 msgr2=0x7fe3740c8030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 shutdown_connections 2023-12-19T07:47:35.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7fe358ff9700 1 -- 172.21.15.111:0/3363263149 wait complete. 2023-12-19T07:47:35.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16a2ffd700 1 -- 172.21.15.111:0/3342148275 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16b0015030 con 0x7f16bc140420 2023-12-19T07:47:35.950 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1ab406ace0 con 0x7f1ab406b200 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 msgr2=0x7f16a8062ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 0x7f16a8062ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f16ac003860 tx=0x7f16ac009040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 msgr2=0x7f16bc0c2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 0x7f16bc0c2450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f16bc06a340 tx=0x7f16b0017000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 shutdown_connections 2023-12-19T07:47:35.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.968+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16a8060610 0x7f16a8062ac0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16bc140420 0x7f16bc0c2450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16bc1407f0 0x7f16bc13bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 --2- 172.21.15.111:0/3342148275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc06c7c0 0x7f16bc138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 >> 172.21.15.111:0/3342148275 conn(0x7f16bc067b60 msgr2=0x7f16bc13dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f1aaa7fc700 1 -- 172.21.15.111:0/3722492410 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1aac052270 con 0x7f1ab406b200 2023-12-19T07:47:35.954 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 shutdown_connections 2023-12-19T07:47:35.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7f16c1145700 1 -- 172.21.15.111:0/3342148275 wait complete. 2023-12-19T07:47:35.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcdb8071120 con 0x7fcdb8087210 2023-12-19T07:47:35.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.972+0000 7fcdad7fa700 1 -- 172.21.15.111:0/2186171741 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcdb004d930 con 0x7fcdb8087210 2023-12-19T07:47:35.956 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:35.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 msgr2=0x7f1a98062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 0x7f1a98062a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4070a00 tx=0x7f1ab0009250 comp rx=0 tx=0).stop 2023-12-19T07:47:35.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 msgr2=0x7f1ab4150f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1aac013000 tx=0x7f1aac019000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 shutdown_connections 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a980605d0 0x7f1a98062a80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ab415adc0 0x7f1ab415e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ab415a9f0 0x7f1ab4156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 --2- 172.21.15.111:0/3722492410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ab406b200 0x7f1ab4150f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 >> 172.21.15.111:0/3722492410 conn(0x7f1ab40670e0 msgr2=0x7f1ab40c1510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 shutdown_connections 2023-12-19T07:47:35.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f1a97fff700 1 -- 172.21.15.111:0/3722492410 wait complete. 2023-12-19T07:47:35.961 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2023-12-19T07:47:35.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 msgr2=0x7fcd9c062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 0x7fcd9c062ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c3c0 tx=0x7fcdb4015000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 msgr2=0x7fcdb8087690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8067fd0 tx=0x7fcdb0009840 comp rx=0 tx=0).stop 2023-12-19T07:47:35.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3238071120 con 0x7f32380cace0 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 shutdown_connections 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcd9c060620 0x7fcd9c062ad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb80aa9d0 0x7fcdb80aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdb8087210 0x7fcdb8087690 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 --2- 172.21.15.111:0/2186171741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcda4003a60 0x7fcdb80a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 >> 172.21.15.111:0/2186171741 conn(0x7fcdb8067bb0 msgr2=0x7fcdb81595a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 shutdown_connections 2023-12-19T07:47:35.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.976+0000 7fcdbf478700 1 -- 172.21.15.111:0/2186171741 wait complete. 2023-12-19T07:47:35.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.980+0000 7f321ffff700 1 -- 172.21.15.111:0/1459574229 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f322c051130 con 0x7f32380cace0 2023-12-19T07:47:35.963 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.980+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 msgr2=0x7f3224062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.980+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 0x7f3224062b00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3228007c20 tx=0x7f3228015000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.980+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 msgr2=0x7f32380ac620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.980+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 0x7f32380ac620 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f322c00d860 tx=0x7f322c008040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 shutdown_connections 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3224060650 0x7f3224062b00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32380cb150 0x7f32380acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32380cace0 0x7f32380ac620 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 --2- 172.21.15.111:0/1459574229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3220003a60 0x7f323816e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 >> 172.21.15.111:0/1459574229 conn(0x7f3238067b60 msgr2=0x7f3238159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 shutdown_connections 2023-12-19T07:47:35.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.984+0000 7f323e2ae700 1 -- 172.21.15.111:0/1459574229 wait complete. 2023-12-19T07:47:35.968 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2023-12-19T07:47:35.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.988+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6da8005830 con 0x7f6dc80cade0 2023-12-19T07:47:35.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.988+0000 7f6daf7fe700 1 -- 172.21.15.111:0/3931632258 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6dc001ee60 con 0x7f6dc80cade0 2023-12-19T07:47:35.971 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 msgr2=0x7f6db4062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 0x7f6db4062a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80b0160 tx=0x7f6db8009040 comp rx=0 tx=0).stop 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 msgr2=0x7f6dc814bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 0x7f6dc814bc00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0016000 tx=0x7f6dc0019000 comp rx=0 tx=0).stop 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 shutdown_connections 2023-12-19T07:47:35.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6db40605d0 0x7f6db4062a80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dc80cade0 0x7f6dc814bc00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dc80c8250 0x7f6dc814a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 --2- 172.21.15.111:0/3931632258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc80add40 0x7f6dc814f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 >> 172.21.15.111:0/3931632258 conn(0x7f6dc8067bb0 msgr2=0x7f6dc806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 shutdown_connections 2023-12-19T07:47:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:35.992+0000 7f6dce1c5700 1 -- 172.21.15.111:0/3931632258 wait complete. 2023-12-19T07:47:35.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2023-12-19T07:47:35.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-12-19T07:47:35.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2023-12-19T07:47:35.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2023-12-19T07:47:35.987 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2023-12-19T07:47:35.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-12-19T07:47:36.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-19T07:47:36.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-19T07:47:36.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-19T07:47:36.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T07:47:36.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-19T07:47:36.985 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-19T07:47:36.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-19T07:47:36.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-19T07:47:37.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.192+0000 7fc23beae700 1 Processor -- start 2023-12-19T07:47:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.192+0000 7fc23beae700 1 -- start start 2023-12-19T07:47:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.192+0000 7fc23beae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23beae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2340c6880 0x7fc2340ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc239c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc239c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46898/0 (socket says 172.21.15.111:46898) 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc239c4a700 1 -- 172.21.15.111:0/2826705961 learned_addr learned my addr 172.21.15.111:0/2826705961 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 --2- 172.21.15.111:0/2826705961 >> v2:172.21.15.111:3300/0 conn(0x7fc2340c6880 0x7fc2340ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23406ceb0 con 0x7fc23406b300 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2340addf0 con 0x7fc2340c6880 2023-12-19T07:47:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340ae330 con 0x7fc234154a10 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc238c48700 1 -- 172.21.15.111:0/2826705961 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96428216 0 0) 0x7fc23406ceb0 con 0x7fc23406b300 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 -- 172.21.15.111:0/2826705961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 msgr2=0x7fc234154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 -- 172.21.15.111:0/2826705961 >> v1:172.21.15.143:6789/0 conn(0x7fc23406b300 legacy=0x7fc2340ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 -- 172.21.15.111:0/2826705961 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340c6ce0 con 0x7fc2340c6880 2023-12-19T07:47:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc239c4a700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.196+0000 7fc23a44b700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340c6880 0x7fc2340ca140 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc23000d9d0 tx=0x7fc2300080e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9d228098146cbb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc238c48700 1 -- 172.21.15.111:0/2826705961 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23000ab30 con 0x7fc2340c6880 2023-12-19T07:47:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc238c48700 1 -- 172.21.15.111:0/2826705961 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23000ac90 con 0x7fc2340c6880 2023-12-19T07:47:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc238c48700 1 -- 172.21.15.111:0/2826705961 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc230015640 con 0x7fc2340c6880 2023-12-19T07:47:37.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340c6880 msgr2=0x7fc2340ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340c6880 0x7fc2340ca140 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc23000d9d0 tx=0x7fc2300080e0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 shutdown_connections 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2340c6880 0x7fc2340ca140 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 --2- 172.21.15.111:0/2826705961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc234154a10 0x7fc234154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.200+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 >> 172.21.15.111:0/2826705961 conn(0x7fc2340670e0 msgr2=0x7fc2340674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.204+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 shutdown_connections 2023-12-19T07:47:37.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.204+0000 7fc23beae700 1 -- 172.21.15.111:0/2826705961 wait complete. 2023-12-19T07:47:37.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.204+0000 7fc23beae700 1 Processor -- start 2023-12-19T07:47:37.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.204+0000 7fc23beae700 1 -- start start 2023-12-19T07:47:37.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340c6880 0x7fc234153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 0x7fc2340af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46914/0 (socket says 172.21.15.111:46914) 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 -- 172.21.15.111:0/3342221261 learned_addr learned my addr 172.21.15.111:0/3342221261 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341691d0 con 0x7fc234154a10 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239449700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340c6880 0x7fc234153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234153fc0 con 0x7fc2340c6880 2023-12-19T07:47:37.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc234168930 con 0x7fc23406b300 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23a44b700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 0x7fc2340af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 -- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340c6880 msgr2=0x7fc234153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340c6880 0x7fc234153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 -- 172.21.15.111:0/3342221261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 msgr2=0x7fc2340af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 0x7fc2340af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc239c4a700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0077d0 con 0x7fc23406b300 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.208+0000 7fc23a44b700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 0x7fc2340af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:37.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc239c4a700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b4e0 tx=0x7fc22c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c00c040 con 0x7fc23406b300 2023-12-19T07:47:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0092e0 con 0x7fc23406b300 2023-12-19T07:47:37.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c009450 con 0x7fc23406b300 2023-12-19T07:47:37.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23000d510 con 0x7fc23406b300 2023-12-19T07:47:37.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.212+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234168ef0 con 0x7fc23406b300 2023-12-19T07:47:37.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.216+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23404be40 con 0x7fc23406b300 2023-12-19T07:47:37.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.216+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc22c016020 con 0x7fc23406b300 2023-12-19T07:47:37.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.220+0000 7fc229ffb700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 0x7fc218062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.220+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc22c07cea0 con 0x7fc23406b300 2023-12-19T07:47:37.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.224+0000 7fc239449700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 0x7fc218062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.224+0000 7fc239449700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 0x7fc218062b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc234162890 tx=0x7fc224009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.232+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc22c04d180 con 0x7fc23406b300 2023-12-19T07:47:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090f216700 1 Processor -- start 2023-12-19T07:47:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090f216700 1 -- start start 2023-12-19T07:47:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090f216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f09080cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090f216700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f090806a540 0x7f090806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090da13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f09080cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090da13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f09080cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46926/0 (socket says 172.21.15.111:46926) 2023-12-19T07:47:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.244+0000 7f090e214700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f0908158c40 0x7f090806b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46396/0 (socket says 172.21.15.111:46396) 2023-12-19T07:47:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090da13700 1 -- 172.21.15.111:0/1026751187 learned_addr learned my addr 172.21.15.111:0/1026751187 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080cb6c0 con 0x7f0908158c40 2023-12-19T07:47:37.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09080cc090 con 0x7f090806a540 2023-12-19T07:47:37.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090ea15700 1 --2- 172.21.15.111:0/1026751187 >> v2:172.21.15.143:3300/0 conn(0x7f090806a540 0x7f090806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090815d9e0 con 0x7f0908159010 2023-12-19T07:47:37.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129527803 0 0) 0x7f09080cb6c0 con 0x7f0908158c40 2023-12-19T07:47:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f4003570 con 0x7f0908158c40 2023-12-19T07:47:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7fd180ef0700 1 Processor -- start 2023-12-19T07:47:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500998831 0 0) 0x7f08f4003570 con 0x7f0908158c40 2023-12-19T07:47:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 msgr2=0x7f09080cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7fd180ef0700 1 -- start start 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f09080cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 >> v2:172.21.15.143:3300/0 conn(0x7f090806a540 msgr2=0x7f090806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 --2- 172.21.15.111:0/1026751187 >> v2:172.21.15.143:3300/0 conn(0x7f090806a540 0x7f090806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080ca810 con 0x7f0908158c40 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0904001470 con 0x7f0908158c40 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 >> v1:172.21.15.143:6789/0 conn(0x7f0908158c40 legacy=0x7f090806b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f08f4003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4004020 0x7f08f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090e214700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f08f4003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090da13700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4004020 0x7f08f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f4004d50 0x7f08f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080cc090 con 0x7f08f4004020 2023-12-19T07:47:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090815d9e0 con 0x7f08f4004d50 2023-12-19T07:47:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f400bae0 con 0x7f0908159010 2023-12-19T07:47:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7fd180ef0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd17c158e50 0x7fd17c06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090ea15700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f4004d50 0x7f08f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.248+0000 7fd180ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06d370 0x7fd17c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd17c06cd00 0x7fd17c15ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56128/0 (socket says 172.21.15.111:56128) 2023-12-19T07:47:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17ad9d700 1 -- 172.21.15.111:0/268529111 learned_addr learned my addr 172.21.15.111:0/268529111 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c06bc30 con 0x7fd17c06cd00 2023-12-19T07:47:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c06d1a0 con 0x7fd17c158e50 2023-12-19T07:47:37.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c06b020 con 0x7fd17c06d370 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 -- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f4004d50 msgr2=0x7f08f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f4004d50 0x7f08f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 -- 172.21.15.111:0/1026751187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4004020 msgr2=0x7f08f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4004020 0x7f08f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 -- 172.21.15.111:0/1026751187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4003570 con 0x7f0908159010 2023-12-19T07:47:37.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090ea15700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f4004d50 0x7f08f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:37.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd179d9b700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06d370 0x7fd17c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 Processor -- start 2023-12-19T07:47:37.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090da13700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4004020 0x7f08f4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:37.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090e214700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f08f4003ae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f090400d750 tx=0x7f09080cb630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db286710866ead8f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 -- start start 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 --2- 172.21.15.111:0/268529111 >> v2:172.21.15.143:3300/0 conn(0x7fd17c158e50 0x7fd17c06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17959a700 1 -- 172.21.15.111:0/268529111 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318900011 0 0) 0x7fd17c06bc30 con 0x7fd17c06cd00 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17959a700 1 -- 172.21.15.111:0/268529111 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd168003570 con 0x7fd17c06cd00 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 -- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06d370 msgr2=0x7fd17c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06d370 0x7fd17c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 -- 172.21.15.111:0/268529111 >> v1:172.21.15.111:6789/0 conn(0x7fd17c06cd00 legacy=0x7fd17c15ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 -- 172.21.15.111:0/268529111 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c06b300 con 0x7fd17c158e50 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904005860 con 0x7f0908159010 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09040059c0 con 0x7f0908159010 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090d212700 1 -- 172.21.15.111:0/1026751187 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904005da0 con 0x7f0908159010 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 msgr2=0x7f08f4003ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090f216700 1 --2- 172.21.15.111:0/1026751187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f08f4003ae0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f090400d750 tx=0x7f09080cb630 comp rx=0 tx=0).stop 2023-12-19T07:47:37.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f090ea15700 1 -- 172.21.15.111:0/1026751187 reap_dead start 2023-12-19T07:47:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7fd17a59c700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c158e50 0x7fd17c06a740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098e0 tx=0x7fd1640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94bd51705f688022 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7fd17959a700 1 -- 172.21.15.111:0/268529111 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164011630 con 0x7fd17c158e50 2023-12-19T07:47:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 0x7f72d80cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d815e310 con 0x7f72d8067520 2023-12-19T07:47:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 shutdown_connections 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 >> 172.21.15.111:0/1026751187 conn(0x7f09080d0810 msgr2=0x7f09080cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d80cdad0 con 0x7f72d815ac30 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.252+0000 7f72df46e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d815e7a0 con 0x7f72d80cd140 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dd20a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f72d815ac30 0x7f72d80b0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46400/0 (socket says 172.21.15.111:46400) 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dd20a700 1 -- 172.21.15.111:0/525560967 learned_addr learned my addr 172.21.15.111:0/525560967 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dca09700 1 --2- 172.21.15.111:0/525560967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 0x7f72d80cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72cffff700 1 -- 172.21.15.111:0/525560967 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871699940 0 0) 0x7f72d80cdad0 con 0x7f72d815ac30 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dca09700 1 -- 172.21.15.111:0/525560967 >> v1:172.21.15.143:6789/0 conn(0x7f72d815ac30 legacy=0x7f72d80b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dca09700 1 -- 172.21.15.111:0/525560967 >> v1:172.21.15.111:6789/0 conn(0x7f72d8067520 legacy=0x7f72d80679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dca09700 1 -- 172.21.15.111:0/525560967 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d815d9e0 con 0x7f72d80cd140 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 shutdown_connections 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f090f216700 1 -- 172.21.15.111:0/1026751187 wait complete. 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7fd17959a700 1 -- 172.21.15.111:0/268529111 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164011790 con 0x7fd17c158e50 2023-12-19T07:47:37.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7fd17959a700 1 -- 172.21.15.111:0/268529111 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164011b10 con 0x7fd17c158e50 2023-12-19T07:47:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72dca09700 1 --2- 172.21.15.111:0/525560967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 0x7f72d80cd590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009910 tx=0x7f72c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd9fe661e3203226 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7ff901585700 1 Processor -- start 2023-12-19T07:47:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c158e50 msgr2=0x7fd17c06a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7fd180ef0700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c158e50 0x7fd17c06a740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098e0 tx=0x7fd1640091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7f72cffff700 1 -- 172.21.15.111:0/525560967 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c800b860 con 0x7f72d80cd140 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 shutdown_connections 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06d370 0x7fd17c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 --2- 172.21.15.111:0/268529111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c158e50 0x7fd17c06a740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 >> 172.21.15.111:0/268529111 conn(0x7fd17c067bb0 msgr2=0x7fd17c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72cffff700 1 -- 172.21.15.111:0/525560967 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c800b9c0 con 0x7f72d80cd140 2023-12-19T07:47:37.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72cffff700 1 -- 172.21.15.111:0/525560967 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c800bd60 con 0x7f72d80cd140 2023-12-19T07:47:37.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.256+0000 7ff901585700 1 -- start start 2023-12-19T07:47:37.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090f216700 1 Processor -- start 2023-12-19T07:47:37.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090f216700 1 -- start start 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72df46e700 1 -- 172.21.15.111:0/525560967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 msgr2=0x7f72d80cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72df46e700 1 --2- 172.21.15.111:0/525560967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 0x7f72d80cd590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009910 tx=0x7f72c80091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72df46e700 1 -- 172.21.15.111:0/525560967 shutdown_connections 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72df46e700 1 --2- 172.21.15.111:0/525560967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80cd140 0x7f72d80cd590 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009910 tx=0x7f72c80091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f72df46e700 1 -- 172.21.15.111:0/525560967 >> 172.21.15.111:0/525560967 conn(0x7f72d80cdda0 msgr2=0x7f72d80ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 shutdown_connections 2023-12-19T07:47:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090f216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f0908150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090f216700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09080b2b20 0x7f09080b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7ff901585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8fc15d050 0x7ff8fc06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7ff901585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 0x7ff8fc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090e214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f0908150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7ff8faffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8fc15d050 0x7ff8fc06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7ff8faffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8fc15d050 0x7ff8fc06a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34858/0 (socket says 172.21.15.111:34858) 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7f090da13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09080b2b20 0x7f09080b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090e214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f0908150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46952/0 (socket says 172.21.15.111:46952) 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 0x7f0908150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080b3e10 con 0x7f09080b2b20 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09080c76e0 con 0x7f0908150a80 2023-12-19T07:47:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090e214700 1 -- 172.21.15.111:0/2053117146 learned_addr learned my addr 172.21.15.111:0/2053117146 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0908151410 con 0x7f0908159010 2023-12-19T07:47:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8fa7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 0x7ff8fc0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f72df46e700 1 -- 172.21.15.111:0/525560967 shutdown_connections 2023-12-19T07:47:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f72df46e700 1 -- 172.21.15.111:0/525560967 wait complete. 2023-12-19T07:47:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 0x7f0908150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 -- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 msgr2=0x7f0908150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f0908150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 -- 172.21.15.111:0/2053117146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09080b2b20 msgr2=0x7f09080b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09080b2b20 0x7f09080b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09080c73b0 con 0x7f0908150a80 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8fa7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 0x7ff8fc0af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47946/0 (socket says 172.21.15.111:47946) 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8fb7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8fc0afd20 0x7ff8fc06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56146/0 (socket says 172.21.15.111:56146) 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8faffd700 1 -- 172.21.15.111:0/1066312464 learned_addr learned my addr 172.21.15.111:0/1066312464 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc06ba50 con 0x7ff8fc0afd20 2023-12-19T07:47:37.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc06bee0 con 0x7ff8fc15d050 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc0af600 con 0x7ff8fc160e60 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8faffd700 1 -- 172.21.15.111:0/1066312464 >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 msgr2=0x7ff8fc0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8faffd700 1 --2- 172.21.15.111:0/1066312464 >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 0x7ff8fc0af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8f9ffb700 1 -- 172.21.15.111:0/1066312464 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846675738 0 0) 0x7ff8fc06ba50 con 0x7ff8fc0afd20 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8faffd700 1 -- 172.21.15.111:0/1066312464 >> v1:172.21.15.111:6789/0 conn(0x7ff8fc0afd20 legacy=0x7ff8fc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7ff8faffd700 1 -- 172.21.15.111:0/1066312464 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0b0140 con 0x7ff8fc15d050 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.260+0000 7fd180ef0700 1 -- 172.21.15.111:0/268529111 wait complete. 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fd180ef0700 1 Processor -- start 2023-12-19T07:47:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fa14d48e700 1 Processor -- start 2023-12-19T07:47:37.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fa14d48e700 1 -- start start 2023-12-19T07:47:37.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fd180ef0700 1 -- start start 2023-12-19T07:47:37.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090ea15700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 0x7f0908150ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0908069990 tx=0x7f090000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f72df46e700 1 Processor -- start 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f72df46e700 1 -- start start 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fa14d48e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 0x7fa14000c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090000fc20 con 0x7f0908150a80 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090000fd80 con 0x7f0908150a80 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09080b3570 con 0x7f0908150a80 2023-12-19T07:47:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09080b3b30 con 0x7f0908150a80 2023-12-19T07:47:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fa14d48e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1400090d0 0x7fa1400e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.264+0000 7fa14d48e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa1400e6780 0x7fa1400ea650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f2a6359e700 1 Processor -- start 2023-12-19T07:47:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14d48e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa140008c40 con 0x7fa1400090d0 2023-12-19T07:47:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa147fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 0x7fa14000c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14d48e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa140008070 con 0x7fa1400e6780 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14d48e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400e6c00 con 0x7fa14000c5e0 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa147fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 0x7fa14000c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46960/0 (socket says 172.21.15.111:46960) 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa147fff700 1 -- 172.21.15.111:0/1228016356 learned_addr learned my addr 172.21.15.111:0/1228016356 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1228016356 >> v2:172.21.15.143:3300/0 conn(0x7fa1400e6780 0x7fa1400ea650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa1477fe700 1 --2- 172.21.15.111:0/1228016356 >> v2:172.21.15.111:3300/0 conn(0x7fa1400090d0 0x7fa1400e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7ff8faffd700 1 --2- 172.21.15.111:0/1066312464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06a740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc06a150 tx=0x7ff8f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2287a44be448c70a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 -- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 msgr2=0x7fa14000c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 0x7fa14000c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 -- 172.21.15.111:0/1228016356 >> v2:172.21.15.111:3300/0 conn(0x7fa1400090d0 msgr2=0x7fa1400e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1228016356 >> v2:172.21.15.111:3300/0 conn(0x7fa1400090d0 0x7fa1400e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 -- 172.21.15.111:0/1228016356 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140008310 con 0x7fa1400e6780 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900016630 con 0x7f0908150a80 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8067520 0x7f72d80af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 0x7f72d80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 0x7f72d816eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d816de10 con 0x7f72d8067520 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dca09700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 0x7f72d80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dd20a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8067520 0x7f72d80af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dca09700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 0x7f72d80cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47980/0 (socket says 172.21.15.111:47980) 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dca09700 1 -- 172.21.15.111:0/890612502 learned_addr learned my addr 172.21.15.111:0/890612502 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80c30e0 con 0x7f72d815ac30 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80c33c0 con 0x7f72d80c2b40 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 0x7f72d816eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f2a6359e700 1 -- start start 2023-12-19T07:47:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400e6780 0x7fa1400ea650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa148071030 tx=0x7fa148066cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52fc17aefcfec14e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa146ffd700 1 -- 172.21.15.111:0/1228016356 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14804bf10 con 0x7fa1400e6780 2023-12-19T07:47:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa146ffd700 1 -- 172.21.15.111:0/1228016356 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1480008d0 con 0x7fa1400e6780 2023-12-19T07:47:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fa146ffd700 1 -- 172.21.15.111:0/1228016356 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148070460 con 0x7fa1400e6780 2023-12-19T07:47:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f00052b0 con 0x7f0908150a80 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 -- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 msgr2=0x7f72d80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 0x7f72d80cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 -- 172.21.15.111:0/890612502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8067520 msgr2=0x7f72d80af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8067520 0x7f72d80af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7f72dda0b700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8009420 con 0x7f72d80c2b40 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 0x7fd17c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 0x7fd17c14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fe2d30ae700 1 Processor -- start 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7ff8f9ffb700 1 -- 172.21.15.111:0/1066312464 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f400e070 con 0x7ff8fc15d050 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7ff8f9ffb700 1 -- 172.21.15.111:0/1066312464 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f400ba50 con 0x7ff8fc15d050 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7ff8f9ffb700 1 -- 172.21.15.111:0/1066312464 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f40092e0 con 0x7ff8fc15d050 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd17c158e50 0x7fd17c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0cac50 con 0x7fd17c158e50 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0ca040 con 0x7fd17c06d370 2023-12-19T07:47:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fd180ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0ca320 con 0x7fd17c06cd00 2023-12-19T07:47:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.268+0000 7fe2d30ae700 1 -- start start 2023-12-19T07:47:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 -- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400e6780 msgr2=0x7fa1400ea650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400e6780 0x7fa1400ea650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa148071030 tx=0x7fa148066cd0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f09000169e0 con 0x7f0908150a80 2023-12-19T07:47:37.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 -- 172.21.15.111:0/1228016356 shutdown_connections 2023-12-19T07:47:37.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400e6780 0x7fa1400ea650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1228016356 >> v2:172.21.15.111:3300/0 conn(0x7fa1400090d0 0x7fa1400e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1228016356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa14000c5e0 0x7fa14000c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 -- 172.21.15.111:0/1228016356 >> 172.21.15.111:0/1228016356 conn(0x7fa140056f90 msgr2=0x7fa140057390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd179d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 0x7fd17c14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a640c6cd0 0x7f2a6406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2a6415cc20 0x7f2a6415d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 msgr2=0x7ff8fc06a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 --2- 172.21.15.111:0/1066312464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06a740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc06a150 tx=0x7ff8f4009840 comp rx=0 tx=0).stop 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd179d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 0x7fd17c14c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47984/0 (socket says 172.21.15.111:47984) 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd179d9b700 1 -- 172.21.15.111:0/2462441314 learned_addr learned my addr 172.21.15.111:0/2462441314 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 -- 172.21.15.111:0/1228016356 shutdown_connections 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 shutdown_connections 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 --2- 172.21.15.111:0/1066312464 >> v2:172.21.15.143:3300/0 conn(0x7ff8fc160e60 0x7ff8fc0af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 --2- 172.21.15.111:0/1066312464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06a740 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72dda0b700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 0x7f72d816eeb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72d000c840 tx=0x7f72d0001620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0012630 con 0x7f72d80c2b40 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d00127b0 con 0x7f72d80c2b40 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0012b70 con 0x7f72d80c2b40 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d80c36a0 con 0x7f72d80c2b40 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d80c3980 con 0x7f72d80c2b40 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 >> 172.21.15.111:0/1066312464 conn(0x7ff8fc067b60 msgr2=0x7ff8fc0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 shutdown_connections 2023-12-19T07:47:37.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 -- 172.21.15.111:0/1066312464 wait complete. 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d30ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2cc0cc9b0 0x7fe2cc0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d30ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2cc15c5b0 0x7fe2cc0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d30ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc06cfb0 con 0x7fe2cc0afc60 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d30ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0cb660 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d30ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0cb380 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f08fe7fc700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 0x7f08ec062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0900083df0 con 0x7f0908150a80 2023-12-19T07:47:37.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f090e214700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 0x7f08ec062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2cbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe2cc0afc60 0x7fe2cc15bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56170/0 (socket says 172.21.15.111:56170) 2023-12-19T07:47:37.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2cbfff700 1 -- 172.21.15.111:0/1380887553 learned_addr learned my addr 172.21.15.111:0/1380887553 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:37.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d164b700 1 --2- 172.21.15.111:0/1380887553 >> v2:172.21.15.143:3300/0 conn(0x7fe2cc15c5b0 0x7fe2cc0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/1380887553 >> v2:172.21.15.111:3300/0 conn(0x7fe2cc0cc9b0 0x7fe2cc0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7ff901585700 1 Processor -- start 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2a640c6900 0x7f2a6406da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56154/0 (socket says 172.21.15.111:56154) 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a61d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a640c6cd0 0x7f2a6406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a61d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a640c6cd0 0x7f2a6406b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:47996/0 (socket says 172.21.15.111:47996) 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6259c700 1 -- 172.21.15.111:0/2691831947 learned_addr learned my addr 172.21.15.111:0/2691831947 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6406b470 con 0x7f2a640c6900 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6406be40 con 0x7f2a6415cc20 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6406e110 con 0x7f2a640c6cd0 2023-12-19T07:47:37.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a62d9d700 1 --2- 172.21.15.111:0/2691831947 >> v2:172.21.15.111:3300/0 conn(0x7f2a6415cc20 0x7f2a6415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7f2a61d9b700 1 -- 172.21.15.111:0/2691831947 >> v2:172.21.15.111:3300/0 conn(0x7f2a6415cc20 msgr2=0x7f2a6415d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f2a61d9b700 1 --2- 172.21.15.111:0/2691831947 >> v2:172.21.15.111:3300/0 conn(0x7f2a6415cc20 0x7f2a6415d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f2a61d9b700 1 -- 172.21.15.111:0/2691831947 >> v1:172.21.15.111:6789/0 conn(0x7f2a640c6900 legacy=0x7f2a6406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f2a61d9b700 1 -- 172.21.15.111:0/2691831947 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64161d10 con 0x7f2a640c6cd0 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 0x7fd17c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17ad9d700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd17c158e50 0x7fd17c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 -- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 msgr2=0x7fd17c14c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 0x7fd17c14c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 -- 172.21.15.111:0/2462441314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd17c158e50 msgr2=0x7fd17c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd17c158e50 0x7fd17c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 -- 172.21.15.111:0/2462441314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164009420 con 0x7fd17c06cd00 2023-12-19T07:47:37.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fd17a59c700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 0x7fd17c0cb0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd164016000 tx=0x7fd164017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2cb7fe700 1 -- 172.21.15.111:0/1380887553 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958008603 0 0) 0x7fe2cc06cfb0 con 0x7fe2cc0afc60 2023-12-19T07:47:37.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2cb7fe700 1 -- 172.21.15.111:0/1380887553 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc003570 con 0x7fe2cc0afc60 2023-12-19T07:47:37.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 -- 172.21.15.111:0/1228016356 wait complete. 2023-12-19T07:47:37.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 -- start start 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d0e4a700 1 -- 172.21.15.111:0/1380887553 >> v2:172.21.15.143:3300/0 conn(0x7fe2cc15c5b0 msgr2=0x7fe2cc0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/1380887553 >> v2:172.21.15.143:3300/0 conn(0x7fe2cc15c5b0 0x7fe2cc0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d0e4a700 1 -- 172.21.15.111:0/1380887553 >> v1:172.21.15.111:6789/0 conn(0x7fe2cc0afc60 legacy=0x7fe2cc15bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fe2d0e4a700 1 -- 172.21.15.111:0/1380887553 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc06c960 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.272+0000 7fa14d48e700 1 Processor -- start 2023-12-19T07:47:37.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d816df80 con 0x7f72d80c2b40 2023-12-19T07:47:37.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/1380887553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc0ccd80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0b5dc0 tx=0x7fe2c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee781535619448b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f090e214700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 0x7f08ec062b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f090400d510 tx=0x7f0904016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f2a61d9b700 1 --2- 172.21.15.111:0/2691831947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6cd0 0x7f2a6406b900 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a6406a030 tx=0x7f2a5c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded9b5e9d3094c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2cb7fe700 1 -- 172.21.15.111:0/1380887553 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0011630 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16401bd80 con 0x7fd17c06cd00 2023-12-19T07:47:37.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164028070 con 0x7fd17c06cd00 2023-12-19T07:47:37.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164024aa0 con 0x7fd17c06cd00 2023-12-19T07:47:37.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2cb7fe700 1 -- 172.21.15.111:0/1380887553 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0011790 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2cb7fe700 1 -- 172.21.15.111:0/1380887553 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0011b10 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f72d0003030 con 0x7f72d80c2b40 2023-12-19T07:47:37.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c0ca600 con 0x7fd17c06cd00 2023-12-19T07:47:37.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c14cb50 con 0x7fd17c06cd00 2023-12-19T07:47:37.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 -- start start 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 -- 172.21.15.111:0/1380887553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 msgr2=0x7fe2cc0ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/1380887553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc0ccd80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0b5dc0 tx=0x7fe2c00091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc0afd20 0x7ff8fc0c3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc160e60 0x7ff8fc0c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc06e160 con 0x7ff8fc15d050 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc06dcd0 con 0x7ff8fc0afd20 2023-12-19T07:47:37.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff901585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0c3680 con 0x7ff8fc160e60 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 -- 172.21.15.111:0/1380887553 shutdown_connections 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/1380887553 >> v2:172.21.15.143:3300/0 conn(0x7fe2cc15c5b0 0x7fe2cc0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/1380887553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc0ccd80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 -- 172.21.15.111:0/1380887553 >> 172.21.15.111:0/1380887553 conn(0x7fe2cc0670e0 msgr2=0x7fe2cc0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c071120 con 0x7fd17c06cd00 2023-12-19T07:47:37.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34898/0 (socket says 172.21.15.111:34898) 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/2503309768 learned_addr learned my addr 172.21.15.111:0/2503309768 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1400e6780 0x7fa1400fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 -- 172.21.15.111:0/1380887553 shutdown_connections 2023-12-19T07:47:37.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fe2d30ae700 1 -- 172.21.15.111:0/1380887553 wait complete. 2023-12-19T07:47:37.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fb7fe700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc160e60 0x7ff8fc0c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 0x7fa140102820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa147fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400e8f30 con 0x7fa1400e7e80 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400e8320 con 0x7fa1400090d0 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14d48e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400e8600 con 0x7fa1400e6780 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa147fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:48014/0 (socket says 172.21.15.111:48014) 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa147fff700 1 -- 172.21.15.111:0/1246711553 learned_addr learned my addr 172.21.15.111:0/1246711553 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa1477fe700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1400e6780 0x7fa1400fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f72cd7fa700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 0x7f72bc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f72d007c9a0 con 0x7f72d80c2b40 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8faffd700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc0afd20 0x7ff8fc0c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc160e60 msgr2=0x7ff8fc0c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc160e60 0x7ff8fc0c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc0afd20 msgr2=0x7ff8fc0c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc0afd20 0x7ff8fc0c3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/2503309768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4009ae0 con 0x7ff8fc15d050 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.276+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 0x7fa140102820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f72dd20a700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 0x7f72bc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff8fa7fc700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000d8c0 tx=0x7ff8f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 -- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1400e6780 msgr2=0x7fa1400fefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1400e6780 0x7fa1400fefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 -- 172.21.15.111:0/1246711553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 msgr2=0x7fa140102820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 0x7fa140102820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 -- 172.21.15.111:0/1246711553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1400e88e0 con 0x7fa1400090d0 2023-12-19T07:47:37.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa14cc8d700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 0x7fa140102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:37.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f090004ce70 con 0x7f0908150a80 2023-12-19T07:47:37.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6159a700 1 -- 172.21.15.111:0/2691831947 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c00e070 con 0x7f2a640c6cd0 2023-12-19T07:47:37.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6159a700 1 -- 172.21.15.111:0/2691831947 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c00bc30 con 0x7f2a640c6cd0 2023-12-19T07:47:37.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6159a700 1 -- 172.21.15.111:0/2691831947 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c012430 con 0x7f2a640c6cd0 2023-12-19T07:47:37.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000de60 con 0x7ff8fc15d050 2023-12-19T07:47:37.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f000b780 con 0x7ff8fc15d050 2023-12-19T07:47:37.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000bb20 con 0x7ff8fc15d050 2023-12-19T07:47:37.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fe2d30ae700 1 Processor -- start 2023-12-19T07:47:37.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa147fff700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e820 tx=0x7fa13c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc06d0c0 con 0x7ff8fc15d050 2023-12-19T07:47:37.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc06d680 con 0x7ff8fc15d050 2023-12-19T07:47:37.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6cd0 msgr2=0x7f2a6406b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2691831947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6cd0 0x7f2a6406b900 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a6406a030 tx=0x7f2a5c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:37.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd16404c8d0 con 0x7fd17c06cd00 2023-12-19T07:47:37.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00c070 con 0x7fa1400090d0 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c013410 con 0x7fa1400090d0 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0137b0 con 0x7fa1400090d0 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1400e8bc0 con 0x7fa1400090d0 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14003cbf0 con 0x7fa1400090d0 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 shutdown_connections 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2691831947 >> v2:172.21.15.111:3300/0 conn(0x7f2a6415cc20 0x7f2a6415d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fd1727fc700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 0x7fd15c062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd16402a070 con 0x7fd17c06cd00 2023-12-19T07:47:37.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2691831947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6cd0 0x7f2a6406b900 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 >> 172.21.15.111:0/2691831947 conn(0x7f2a64067bb0 msgr2=0x7f2a640cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fe2d30ae700 1 -- start start 2023-12-19T07:47:37.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 shutdown_connections 2023-12-19T07:47:37.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc06de40 con 0x7ff8fc15d050 2023-12-19T07:47:37.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7fd179d9b700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 0x7fd15c062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa140005930 con 0x7fa1400090d0 2023-12-19T07:47:37.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fd179d9b700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 0x7fd15c062ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd17c16c1c0 tx=0x7fd1740091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.280+0000 7f72dd20a700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 0x7f72bc062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c8f0 tx=0x7f72d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 0x7fe2cc16db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc16e080 con 0x7fe2cc0cc9b0 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c3480 con 0x7fe2cc0afc60 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d30ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c4230 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:47008/0 (socket says 172.21.15.111:47008) 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 -- 172.21.15.111:0/2384649102 learned_addr learned my addr 172.21.15.111:0/2384649102 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff8f001e460 con 0x7ff8fc15d050 2023-12-19T07:47:37.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa13c013b20 con 0x7fa1400090d0 2023-12-19T07:47:37.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 0x7fe2cc16db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 -- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 msgr2=0x7fe2cc16db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7f2a6359e700 1 -- 172.21.15.111:0/2691831947 wait complete. 2023-12-19T07:47:37.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 0x7fe2cc16db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72d0050730 con 0x7f72d80c2b40 2023-12-19T07:47:37.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7f2a6359e700 1 Processor -- start 2023-12-19T07:47:37.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 -- 172.21.15.111:0/2384649102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 msgr2=0x7fe2cc157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 -- 172.21.15.111:0/2384649102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0009420 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 0x7fe2cc16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 0x7ff8e8062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff8f007d120 con 0x7ff8fc15d050 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7ff8faffd700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 0x7ff8e8062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7f2a6359e700 1 -- start start 2023-12-19T07:47:37.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fa12ffff700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 0x7fa134062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa13c07c6c0 con 0x7fa1400090d0 2023-12-19T07:47:37.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.284+0000 7fe2d164b700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009df0 tx=0x7fe2c400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7ff8faffd700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 0x7ff8e8062ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009880 tx=0x7ff8f4018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fa1477fe700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 0x7fa134062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6900 0x7f2a641515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a640c6cd0 0x7f2a641554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640b11c0 con 0x7f2a640c6cd0 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64156e30 con 0x7f2a640c6900 2023-12-19T07:47:37.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64155f50 con 0x7f2a6415cc20 2023-12-19T07:47:37.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fa1477fe700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 0x7fa134062b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1400f3170 tx=0x7fa138008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:47012/0 (socket says 172.21.15.111:47012) 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 -- 172.21.15.111:0/2879781250 learned_addr learned my addr 172.21.15.111:0/2879781250 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6259c700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6900 0x7f2a641515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a61d9b700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a640c6cd0 0x7f2a641554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 -- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6900 msgr2=0x7f2a641515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6900 0x7f2a641515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 -- 172.21.15.111:0/2879781250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a640c6cd0 msgr2=0x7f2a641554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a640c6cd0 0x7f2a641554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 -- 172.21.15.111:0/2879781250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c009a20 con 0x7f2a6415cc20 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a62d9d700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800d860 tx=0x7f2a58008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a580125f0 con 0x7f2a6415cc20 2023-12-19T07:47:37.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400cb50 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a64156220 con 0x7f2a6415cc20 2023-12-19T07:47:37.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a58012790 con 0x7f2a6415cc20 2023-12-19T07:47:37.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a641567e0 con 0x7f2a6415cc20 2023-12-19T07:47:37.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58012b50 con 0x7f2a6415cc20 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd164057080 con 0x7fd17c06cd00 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c400f040 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc0c4510 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.288+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc0c3a20 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4014480 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a64156fc0 con 0x7f2a6415cc20 2023-12-19T07:47:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2a58013480 con 0x7f2a6415cc20 2023-12-19T07:47:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ac0052b0 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff8f004d4a0 con 0x7ff8fc15d050 2023-12-19T07:47:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa13c04c9a0 con 0x7fa1400090d0 2023-12-19T07:47:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7f2a527fc700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 0x7f2a44062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2a5807ce50 con 0x7f2a6415cc20 2023-12-19T07:47:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.292+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe2c4014830 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7f2a6259c700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 0x7f2a44062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7fe2c8ff9700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 0x7fe2b0062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe2c407d460 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7f2a6259c700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 0x7f2a44062a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a54007130 tx=0x7f2a54007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 0x7fe2b0062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.296+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2a5804d0b0 con 0x7f2a6415cc20 2023-12-19T07:47:37.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.300+0000 7fe2d0e4a700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 0x7fe2b0062a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009fd0 tx=0x7fe2c0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.308+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2c404d6c0 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.540+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc2341541f0 con 0x7fc23406b300 2023-12-19T07:47:37.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.544+0000 7fc229ffb700 1 -- 172.21.15.111:0/3342221261 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc22c0507a0 con 0x7fc23406b300 2023-12-19T07:47:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 msgr2=0x7fc218062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 0x7fc218062b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc234162890 tx=0x7fc224009040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 msgr2=0x7fc2341682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b4e0 tx=0x7fc22c00b040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 shutdown_connections 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2180606a0 0x7fc218062b50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2340c6880 0x7fc234153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc234154a10 0x7fc2340af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 --2- 172.21.15.111:0/3342221261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc23406b300 0x7fc2341682a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 >> 172.21.15.111:0/3342221261 conn(0x7fc2340670e0 msgr2=0x7fc23406a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 shutdown_connections 2023-12-19T07:47:37.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.548+0000 7fc23beae700 1 -- 172.21.15.111:0/3342221261 wait complete. 2023-12-19T07:47:37.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-12-19T07:47:37.557 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.592+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe2ac005560 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.592+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f08f0005830 con 0x7f0908150a80 2023-12-19T07:47:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2c8ff9700 1 -- 172.21.15.111:0/2384649102 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2c4050ce0 con 0x7fe2cc15c5b0 2023-12-19T07:47:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7f08fe7fc700 1 -- 172.21.15.111:0/2053117146 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0900050470 con 0x7f0908150a80 2023-12-19T07:47:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 msgr2=0x7fe2b0062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 0x7fe2b0062a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009fd0 tx=0x7fe2c0017000 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 msgr2=0x7fe2cc158590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009df0 tx=0x7fe2c400b040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 shutdown_connections 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2b00605d0 0x7fe2b0062a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc15c5b0 0x7fe2cc158590 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2cc0cc9b0 0x7fe2cc157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 --2- 172.21.15.111:0/2384649102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc0afc60 0x7fe2cc16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.596+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 >> 172.21.15.111:0/2384649102 conn(0x7fe2cc0670e0 msgr2=0x7fe2cc0b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 shutdown_connections 2023-12-19T07:47:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7fe2d30ae700 1 -- 172.21.15.111:0/2384649102 wait complete. 2023-12-19T07:47:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 msgr2=0x7f08ec062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 0x7f08ec062b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f090400d510 tx=0x7f0904016000 comp rx=0 tx=0).stop 2023-12-19T07:47:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 msgr2=0x7f0908150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 0x7f0908150ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0908069990 tx=0x7f090000b040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 shutdown_connections 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08ec0606a0 0x7f08ec062b50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0908150a80 0x7f0908150ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09080b2b20 0x7f09080b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 --2- 172.21.15.111:0/2053117146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0908159010 0x7f0908150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 >> 172.21.15.111:0/2053117146 conn(0x7f09080d0810 msgr2=0x7f09080c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 shutdown_connections 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f090f216700 1 -- 172.21.15.111:0/2053117146 wait complete. 2023-12-19T07:47:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f72d8071120 con 0x7f72d80c2b40 2023-12-19T07:47:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.600+0000 7f72cd7fa700 1 -- 172.21.15.111:0/890612502 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f72d0050730 con 0x7f72d80c2b40 2023-12-19T07:47:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 msgr2=0x7f72bc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 0x7f72bc062a00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c8f0 tx=0x7f72d4009040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 msgr2=0x7f72d816eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 0x7f72d816eeb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72d000c840 tx=0x7f72d0001620 comp rx=0 tx=0).stop 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 shutdown_connections 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72bc060550 0x7f72bc062a00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72d815ac30 0x7f72d80cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80c2b40 0x7f72d816eeb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 --2- 172.21.15.111:0/890612502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d8067520 0x7f72d80af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 >> 172.21.15.111:0/890612502 conn(0x7f72d80cdda0 msgr2=0x7f72d806cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 shutdown_connections 2023-12-19T07:47:37.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.604+0000 7f72df46e700 1 -- 172.21.15.111:0/890612502 wait complete. 2023-12-19T07:47:37.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.608+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2a6404be40 con 0x7f2a6415cc20 2023-12-19T07:47:37.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.608+0000 7f2a527fc700 1 -- 172.21.15.111:0/2879781250 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2a5807e090 con 0x7f2a6415cc20 2023-12-19T07:47:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-12-19T07:47:37.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd17c0ca460 con 0x7fd17c06cd00 2023-12-19T07:47:37.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa14003cbf0 con 0x7fa1400090d0 2023-12-19T07:47:37.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7fa12ffff700 1 -- 172.21.15.111:0/1246711553 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa14003cbf0 con 0x7fa1400090d0 2023-12-19T07:47:37.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff8fc071120 con 0x7ff8fc15d050 2023-12-19T07:47:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7fd1727fc700 1 -- 172.21.15.111:0/2462441314 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd17c0ca460 con 0x7fd17c06cd00 2023-12-19T07:47:37.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/2503309768 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8f0050ac0 con 0x7ff8fc15d050 2023-12-19T07:47:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-12-19T07:47:37.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 msgr2=0x7f2a44062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 0x7f2a44062a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a54007130 tx=0x7f2a54007eb0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 msgr2=0x7f2a64155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800d860 tx=0x7f2a58008040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 shutdown_connections 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a440605d0 0x7f2a44062a80 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a54007130 tx=0x7f2a54007eb0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6415cc20 0x7f2a64155a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a640c6cd0 0x7f2a641554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 --2- 172.21.15.111:0/2879781250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a640c6900 0x7f2a641515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 >> 172.21.15.111:0/2879781250 conn(0x7f2a64067bb0 msgr2=0x7f2a640abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 shutdown_connections 2023-12-19T07:47:37.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.612+0000 7f2a6359e700 1 -- 172.21.15.111:0/2879781250 wait complete. 2023-12-19T07:47:37.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 msgr2=0x7fa134062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 0x7fa134062b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1400f3170 tx=0x7fa138008040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 msgr2=0x7fa14003c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e820 tx=0x7fa13c008040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 shutdown_connections 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa1340606a0 0x7fa134062b50 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1400f3170 tx=0x7fa138008040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1400e7e80 0x7fa140102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1400e6780 0x7fa1400fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 --2- 172.21.15.111:0/1246711553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1400090d0 0x7fa14003c490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 >> 172.21.15.111:0/1246711553 conn(0x7fa140056f90 msgr2=0x7fa140054ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 shutdown_connections 2023-12-19T07:47:37.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fa14d48e700 1 -- 172.21.15.111:0/1246711553 wait complete. 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 msgr2=0x7ff8e8062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 0x7ff8e8062ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009880 tx=0x7ff8f4018000 comp rx=0 tx=0).stop 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 msgr2=0x7ff8fc06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000d8c0 tx=0x7ff8f0008040 comp rx=0 tx=0).stop 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 shutdown_connections 2023-12-19T07:47:37.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8e8060620 0x7ff8e8062ad0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc160e60 0x7ff8fc0c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc15d050 0x7ff8fc06c7c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 --2- 172.21.15.111:0/2503309768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc0afd20 0x7ff8fc0c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 >> 172.21.15.111:0/2503309768 conn(0x7ff8fc067b60 msgr2=0x7ff8fc15bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 msgr2=0x7fd15c062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 0x7fd15c062ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd17c16c1c0 tx=0x7fd1740091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 msgr2=0x7fd17c0cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 0x7fd17c0cb0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd164016000 tx=0x7fd164017000 comp rx=0 tx=0).stop 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 shutdown_connections 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060620 0x7fd15c062ad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd17c06d370 0x7fd17c14c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd17c158e50 0x7fd17c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 --2- 172.21.15.111:0/2462441314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd17c06cd00 0x7fd17c0cb0e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 >> 172.21.15.111:0/2462441314 conn(0x7fd17c067bb0 msgr2=0x7fd17c0b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-12-19T07:47:37.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 shutdown_connections 2023-12-19T07:47:37.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7fd180ef0700 1 -- 172.21.15.111:0/2462441314 wait complete. 2023-12-19T07:47:37.607 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 shutdown_connections 2023-12-19T07:47:37.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.616+0000 7ff901585700 1 -- 172.21.15.111:0/2503309768 wait complete. 2023-12-19T07:47:37.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2023-12-19T07:47:37.608 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2023-12-19T07:47:37.611 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-12-19T07:47:37.617 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-12-19T07:47:37.619 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2023-12-19T07:47:37.625 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-12-19T07:47:37.627 DEBUG:teuthology.parallel:result is None 2023-12-19T07:47:37.628 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-19T07:47:37.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:47:37.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 Processor -- start 2023-12-19T07:47:37.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 -- start start 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 0x7f1134159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113415a260 con 0x7f113406ad70 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340afad0 con 0x7f113406b1e0 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f113960c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11340afdb0 con 0x7f113406e1d0 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f11337fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f113406ad70 0x7f113414f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56176/0 (socket says 172.21.15.111:56176) 2023-12-19T07:47:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f11337fe700 1 -- 172.21.15.111:0/1751091689 learned_addr learned my addr 172.21.15.111:0/1751091689 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1133fff700 1 --2- 172.21.15.111:0/1751091689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 0x7f1134159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431174431 0 0) 0x7f113415a260 con 0x7f113406ad70 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111c003570 con 0x7f113406ad70 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689233583 0 0) 0x7f11340afad0 con 0x7f113406b1e0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113415a260 con 0x7f113406b1e0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1133fff700 1 -- 172.21.15.111:0/1751091689 >> v1:172.21.15.143:6789/0 conn(0x7f113406b1e0 legacy=0x7f113414fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1133fff700 1 -- 172.21.15.111:0/1751091689 >> v1:172.21.15.111:6789/0 conn(0x7f113406ad70 legacy=0x7f113414f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.848+0000 7f1133fff700 1 -- 172.21.15.111:0/1751091689 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11340b0090 con 0x7f113406e1d0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f1133fff700 1 --2- 172.21.15.111:0/1751091689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 0x7f1134159d20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1124009910 tx=0x7f11240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62807d373939042e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124011630 con 0x7f113406e1d0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1124011790 con 0x7f113406e1d0 2023-12-19T07:47:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f1132ffd700 1 -- 172.21.15.111:0/1751091689 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124011b30 con 0x7f113406e1d0 2023-12-19T07:47:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- 172.21.15.111:0/1751091689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 msgr2=0x7f1134159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 --2- 172.21.15.111:0/1751091689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 0x7f1134159d20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1124009910 tx=0x7f11240091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- 172.21.15.111:0/1751091689 shutdown_connections 2023-12-19T07:47:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 --2- 172.21.15.111:0/1751091689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406e1d0 0x7f1134159d20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- 172.21.15.111:0/1751091689 >> 172.21.15.111:0/1751091689 conn(0x7f1134067bb0 msgr2=0x7f11340add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- 172.21.15.111:0/1751091689 shutdown_connections 2023-12-19T07:47:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- 172.21.15.111:0/1751091689 wait complete. 2023-12-19T07:47:37.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 Processor -- start 2023-12-19T07:47:37.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.852+0000 7f113960c700 1 -- start start 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406ad70 0x7f113414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f113406b1e0 0x7f11340abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340aaf20 con 0x7f113406e1d0 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113414ef00 con 0x7f113406b1e0 2023-12-19T07:47:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11340ad720 con 0x7f113406ad70 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34922/0 (socket says 172.21.15.111:34922) 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 -- 172.21.15.111:0/1177317923 learned_addr learned my addr 172.21.15.111:0/1177317923 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f11337fe700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f113406b1e0 0x7f11340abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1133fff700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406ad70 0x7f113414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 -- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406ad70 msgr2=0x7f113414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406ad70 0x7f113414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 -- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f113406b1e0 msgr2=0x7f11340abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f113406b1e0 0x7f11340abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 -- 172.21.15.111:0/1177317923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1124009420 con 0x7f113406e1d0 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f1138e0b700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f112c00c940 tx=0x7f112c0027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c012630 con 0x7f113406e1d0 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c0127b0 con 0x7f113406e1d0 2023-12-19T07:47:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11340aa8d0 con 0x7f113406e1d0 2023-12-19T07:47:37.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c012b70 con 0x7f113406e1d0 2023-12-19T07:47:37.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.856+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11340ada00 con 0x7f113406e1d0 2023-12-19T07:47:37.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.860+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11340ab090 con 0x7f113406e1d0 2023-12-19T07:47:37.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.860+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f112c01b400 con 0x7f113406e1d0 2023-12-19T07:47:37.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.860+0000 7f111bfff700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 0x7f1120062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:37.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.860+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f112c07cd90 con 0x7f113406e1d0 2023-12-19T07:47:37.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.864+0000 7f1133fff700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 0x7f1120062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:37.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.864+0000 7f1133fff700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 0x7f1120062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f113415bef0 tx=0x7f1124017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:37.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:37.868+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f112c04d090 con 0x7f113406e1d0 2023-12-19T07:47:38.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.020+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f113404be40 con 0x7f1120060580 2023-12-19T07:47:38.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.040+0000 7f111bfff700 1 -- 172.21.15.111:0/1177317923 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345003 (secure 0 0 0) 0x7f113404be40 con 0x7f1120060580 2023-12-19T07:47:38.029 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 msgr2=0x7f1120062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 0x7f1120062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f113415bef0 tx=0x7f1124017000 comp rx=0 tx=0).stop 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 msgr2=0x7f11340ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f112c00c940 tx=0x7f112c0027a0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 shutdown_connections 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1120060580 0x7f1120062a30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f113406b1e0 0x7f11340abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f113406e1d0 0x7f11340ac690 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 --2- 172.21.15.111:0/1177317923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f113406ad70 0x7f113414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 >> 172.21.15.111:0/1177317923 conn(0x7f1134067bb0 msgr2=0x7f1134157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:38.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 shutdown_connections 2023-12-19T07:47:38.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.048+0000 7f113960c700 1 -- 172.21.15.111:0/1177317923 wait complete. 2023-12-19T07:47:38.034 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:47:38.053 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-19T07:47:36.616305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381604,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593116,"statfs":{"total":773094113280,"available":770655350784,"internally_reserved":0,"allocated":3379200,"data_stored":2041019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004304"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904960+0000","last_change":"2023-12-19T07:47:16.213519+0000","last_active":"2023-12-19T07:47:22.904960+0000","last_peered":"2023-12-19T07:47:22.904960+0000","last_clean":"2023-12-19T07:47:22.904960+0000","last_became_active":"2023-12-19T07:47:16.213172+0000","last_became_peered":"2023-12-19T07:47:16.213172+0000","last_unstale":"2023-12-19T07:47:22.904960+0000","last_undegraded":"2023-12-19T07:47:22.904960+0000","last_fullsized":"2023-12-19T07:47:22.904960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599466+0000","last_change":"2023-12-19T07:47:17.206618+0000","last_active":"2023-12-19T07:47:22.599466+0000","last_peered":"2023-12-19T07:47:22.599466+0000","last_clean":"2023-12-19T07:47:22.599466+0000","last_became_active":"2023-12-19T07:47:17.206300+0000","last_became_peered":"2023-12-19T07:47:17.206300+0000","last_unstale":"2023-12-19T07:47:22.599466+0000","last_undegraded":"2023-12-19T07:47:22.599466+0000","last_fullsized":"2023-12-19T07:47:22.599466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597951+0000","last_change":"2023-12-19T07:47:21.847101+0000","last_active":"2023-12-19T07:47:22.597951+0000","last_peered":"2023-12-19T07:47:22.597951+0000","last_clean":"2023-12-19T07:47:22.597951+0000","last_became_active":"2023-12-19T07:47:21.845076+0000","last_became_peered":"2023-12-19T07:47:21.845076+0000","last_unstale":"2023-12-19T07:47:22.597951+0000","last_undegraded":"2023-12-19T07:47:22.597951+0000","last_fullsized":"2023-12-19T07:47:22.597951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599595+0000","last_change":"2023-12-19T07:47:16.198270+0000","last_active":"2023-12-19T07:47:22.599595+0000","last_peered":"2023-12-19T07:47:22.599595+0000","last_clean":"2023-12-19T07:47:22.599595+0000","last_became_active":"2023-12-19T07:47:16.197562+0000","last_became_peered":"2023-12-19T07:47:16.197562+0000","last_unstale":"2023-12-19T07:47:22.599595+0000","last_undegraded":"2023-12-19T07:47:22.599595+0000","last_fullsized":"2023-12-19T07:47:22.599595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904301+0000","last_change":"2023-12-19T07:47:17.198174+0000","last_active":"2023-12-19T07:47:22.904301+0000","last_peered":"2023-12-19T07:47:22.904301+0000","last_clean":"2023-12-19T07:47:22.904301+0000","last_became_active":"2023-12-19T07:47:17.197876+0000","last_became_peered":"2023-12-19T07:47:17.197876+0000","last_unstale":"2023-12-19T07:47:22.904301+0000","last_undegraded":"2023-12-19T07:47:22.904301+0000","last_fullsized":"2023-12-19T07:47:22.904301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306106+0000","last_change":"2023-12-19T07:47:21.260994+0000","last_active":"2023-12-19T07:47:22.306106+0000","last_peered":"2023-12-19T07:47:22.306106+0000","last_clean":"2023-12-19T07:47:22.306106+0000","last_became_active":"2023-12-19T07:47:21.260122+0000","last_became_peered":"2023-12-19T07:47:21.260122+0000","last_unstale":"2023-12-19T07:47:22.306106+0000","last_undegraded":"2023-12-19T07:47:22.306106+0000","last_fullsized":"2023-12-19T07:47:22.306106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868136+0000","last_change":"2023-12-19T07:47:16.215728+0000","last_active":"2023-12-19T07:47:22.868136+0000","last_peered":"2023-12-19T07:47:22.868136+0000","last_clean":"2023-12-19T07:47:22.868136+0000","last_became_active":"2023-12-19T07:47:16.215434+0000","last_became_peered":"2023-12-19T07:47:16.215434+0000","last_unstale":"2023-12-19T07:47:22.868136+0000","last_undegraded":"2023-12-19T07:47:22.868136+0000","last_fullsized":"2023-12-19T07:47:22.868136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806795+0000","last_change":"2023-12-19T07:47:17.204556+0000","last_active":"2023-12-19T07:47:22.806795+0000","last_peered":"2023-12-19T07:47:22.806795+0000","last_clean":"2023-12-19T07:47:22.806795+0000","last_became_active":"2023-12-19T07:47:17.204225+0000","last_became_peered":"2023-12-19T07:47:17.204225+0000","last_unstale":"2023-12-19T07:47:22.806795+0000","last_undegraded":"2023-12-19T07:47:22.806795+0000","last_fullsized":"2023-12-19T07:47:22.806795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306583+0000","last_change":"2023-12-19T07:47:21.266228+0000","last_active":"2023-12-19T07:47:22.306583+0000","last_peered":"2023-12-19T07:47:22.306583+0000","last_clean":"2023-12-19T07:47:22.306583+0000","last_became_active":"2023-12-19T07:47:21.265305+0000","last_became_peered":"2023-12-19T07:47:21.265305+0000","last_unstale":"2023-12-19T07:47:22.306583+0000","last_undegraded":"2023-12-19T07:47:22.306583+0000","last_fullsized":"2023-12-19T07:47:22.306583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867767+0000","last_change":"2023-12-19T07:47:16.214913+0000","last_active":"2023-12-19T07:47:22.867767+0000","last_peered":"2023-12-19T07:47:22.867767+0000","last_clean":"2023-12-19T07:47:22.867767+0000","last_became_active":"2023-12-19T07:47:16.214514+0000","last_became_peered":"2023-12-19T07:47:16.214514+0000","last_unstale":"2023-12-19T07:47:22.867767+0000","last_undegraded":"2023-12-19T07:47:22.867767+0000","last_fullsized":"2023-12-19T07:47:22.867767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:24.444351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807216+0000","last_change":"2023-12-19T07:47:17.208692+0000","last_active":"2023-12-19T07:47:22.807216+0000","last_peered":"2023-12-19T07:47:22.807216+0000","last_clean":"2023-12-19T07:47:22.807216+0000","last_became_active":"2023-12-19T07:47:17.208320+0000","last_became_peered":"2023-12-19T07:47:17.208320+0000","last_unstale":"2023-12-19T07:47:22.807216+0000","last_undegraded":"2023-12-19T07:47:22.807216+0000","last_fullsized":"2023-12-19T07:47:22.807216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306291+0000","last_change":"2023-12-19T07:47:21.249790+0000","last_active":"2023-12-19T07:47:22.306291+0000","last_peered":"2023-12-19T07:47:22.306291+0000","last_clean":"2023-12-19T07:47:22.306291+0000","last_became_active":"2023-12-19T07:47:21.249266+0000","last_became_peered":"2023-12-19T07:47:21.249266+0000","last_unstale":"2023-12-19T07:47:22.306291+0000","last_undegraded":"2023-12-19T07:47:22.306291+0000","last_fullsized":"2023-12-19T07:47:22.306291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358655+0000","last_change":"2023-12-19T07:47:17.206257+0000","last_active":"2023-12-19T07:47:22.358655+0000","last_peered":"2023-12-19T07:47:22.358655+0000","last_clean":"2023-12-19T07:47:22.358655+0000","last_became_active":"2023-12-19T07:47:17.205737+0000","last_became_peered":"2023-12-19T07:47:17.205737+0000","last_unstale":"2023-12-19T07:47:22.358655+0000","last_undegraded":"2023-12-19T07:47:22.358655+0000","last_fullsized":"2023-12-19T07:47:22.358655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867658+0000","last_change":"2023-12-19T07:47:16.214499+0000","last_active":"2023-12-19T07:47:22.867658+0000","last_peered":"2023-12-19T07:47:22.867658+0000","last_clean":"2023-12-19T07:47:22.867658+0000","last_became_active":"2023-12-19T07:47:16.213693+0000","last_became_peered":"2023-12-19T07:47:16.213693+0000","last_unstale":"2023-12-19T07:47:22.867658+0000","last_undegraded":"2023-12-19T07:47:22.867658+0000","last_fullsized":"2023-12-19T07:47:22.867658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307061+0000","last_change":"2023-12-19T07:47:21.254224+0000","last_active":"2023-12-19T07:47:22.307061+0000","last_peered":"2023-12-19T07:47:22.307061+0000","last_clean":"2023-12-19T07:47:22.307061+0000","last_became_active":"2023-12-19T07:47:21.253786+0000","last_became_peered":"2023-12-19T07:47:21.253786+0000","last_unstale":"2023-12-19T07:47:22.307061+0000","last_undegraded":"2023-12-19T07:47:22.307061+0000","last_fullsized":"2023-12-19T07:47:22.307061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:18:10.044237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598837+0000","last_change":"2023-12-19T07:47:17.205670+0000","last_active":"2023-12-19T07:47:22.598837+0000","last_peered":"2023-12-19T07:47:22.598837+0000","last_clean":"2023-12-19T07:47:22.598837+0000","last_became_active":"2023-12-19T07:47:17.201174+0000","last_became_peered":"2023-12-19T07:47:17.201174+0000","last_unstale":"2023-12-19T07:47:22.598837+0000","last_undegraded":"2023-12-19T07:47:22.598837+0000","last_fullsized":"2023-12-19T07:47:22.598837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352125+0000","last_change":"2023-12-19T07:47:16.196003+0000","last_active":"2023-12-19T07:47:22.352125+0000","last_peered":"2023-12-19T07:47:22.352125+0000","last_clean":"2023-12-19T07:47:22.352125+0000","last_became_active":"2023-12-19T07:47:16.195717+0000","last_became_peered":"2023-12-19T07:47:16.195717+0000","last_unstale":"2023-12-19T07:47:22.352125+0000","last_undegraded":"2023-12-19T07:47:22.352125+0000","last_fullsized":"2023-12-19T07:47:22.352125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306644+0000","last_change":"2023-12-19T07:47:21.259995+0000","last_active":"2023-12-19T07:47:22.306644+0000","last_peered":"2023-12-19T07:47:22.306644+0000","last_clean":"2023-12-19T07:47:22.306644+0000","last_became_active":"2023-12-19T07:47:21.259318+0000","last_became_peered":"2023-12-19T07:47:21.259318+0000","last_unstale":"2023-12-19T07:47:22.306644+0000","last_undegraded":"2023-12-19T07:47:22.306644+0000","last_fullsized":"2023-12-19T07:47:22.306644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598330+0000","last_change":"2023-12-19T07:47:17.205504+0000","last_active":"2023-12-19T07:47:22.598330+0000","last_peered":"2023-12-19T07:47:22.598330+0000","last_clean":"2023-12-19T07:47:22.598330+0000","last_became_active":"2023-12-19T07:47:17.200074+0000","last_became_peered":"2023-12-19T07:47:17.200074+0000","last_unstale":"2023-12-19T07:47:22.598330+0000","last_undegraded":"2023-12-19T07:47:22.598330+0000","last_fullsized":"2023-12-19T07:47:22.598330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351472+0000","last_change":"2023-12-19T07:47:16.206620+0000","last_active":"2023-12-19T07:47:22.351472+0000","last_peered":"2023-12-19T07:47:22.351472+0000","last_clean":"2023-12-19T07:47:22.351472+0000","last_became_active":"2023-12-19T07:47:16.206288+0000","last_became_peered":"2023-12-19T07:47:16.206288+0000","last_unstale":"2023-12-19T07:47:22.351472+0000","last_undegraded":"2023-12-19T07:47:22.351472+0000","last_fullsized":"2023-12-19T07:47:22.351472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351144+0000","last_change":"2023-12-19T07:47:21.841279+0000","last_active":"2023-12-19T07:47:22.351144+0000","last_peered":"2023-12-19T07:47:22.351144+0000","last_clean":"2023-12-19T07:47:22.351144+0000","last_became_active":"2023-12-19T07:47:21.840184+0000","last_became_peered":"2023-12-19T07:47:21.840184+0000","last_unstale":"2023-12-19T07:47:22.351144+0000","last_undegraded":"2023-12-19T07:47:22.351144+0000","last_fullsized":"2023-12-19T07:47:22.351144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:22:46.085295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306581+0000","last_change":"2023-12-19T07:47:17.199136+0000","last_active":"2023-12-19T07:47:22.306581+0000","last_peered":"2023-12-19T07:47:22.306581+0000","last_clean":"2023-12-19T07:47:22.306581+0000","last_became_active":"2023-12-19T07:47:17.198802+0000","last_became_peered":"2023-12-19T07:47:17.198802+0000","last_unstale":"2023-12-19T07:47:22.306581+0000","last_undegraded":"2023-12-19T07:47:22.306581+0000","last_fullsized":"2023-12-19T07:47:22.306581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:35.655311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866892+0000","last_change":"2023-12-19T07:47:16.213635+0000","last_active":"2023-12-19T07:47:22.866892+0000","last_peered":"2023-12-19T07:47:22.866892+0000","last_clean":"2023-12-19T07:47:22.866892+0000","last_became_active":"2023-12-19T07:47:16.213201+0000","last_became_peered":"2023-12-19T07:47:16.213201+0000","last_unstale":"2023-12-19T07:47:22.866892+0000","last_undegraded":"2023-12-19T07:47:22.866892+0000","last_fullsized":"2023-12-19T07:47:22.866892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306787+0000","last_change":"2023-12-19T07:47:21.265928+0000","last_active":"2023-12-19T07:47:22.306787+0000","last_peered":"2023-12-19T07:47:22.306787+0000","last_clean":"2023-12-19T07:47:22.306787+0000","last_became_active":"2023-12-19T07:47:21.265360+0000","last_became_peered":"2023-12-19T07:47:21.265360+0000","last_unstale":"2023-12-19T07:47:22.306787+0000","last_undegraded":"2023-12-19T07:47:22.306787+0000","last_fullsized":"2023-12-19T07:47:22.306787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904688+0000","last_change":"2023-12-19T07:47:17.195176+0000","last_active":"2023-12-19T07:47:22.904688+0000","last_peered":"2023-12-19T07:47:22.904688+0000","last_clean":"2023-12-19T07:47:22.904688+0000","last_became_active":"2023-12-19T07:47:17.194607+0000","last_became_peered":"2023-12-19T07:47:17.194607+0000","last_unstale":"2023-12-19T07:47:22.904688+0000","last_undegraded":"2023-12-19T07:47:22.904688+0000","last_fullsized":"2023-12-19T07:47:22.904688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359926+0000","last_change":"2023-12-19T07:47:16.206513+0000","last_active":"2023-12-19T07:47:22.359926+0000","last_peered":"2023-12-19T07:47:22.359926+0000","last_clean":"2023-12-19T07:47:22.359926+0000","last_became_active":"2023-12-19T07:47:16.206148+0000","last_became_peered":"2023-12-19T07:47:16.206148+0000","last_unstale":"2023-12-19T07:47:22.359926+0000","last_undegraded":"2023-12-19T07:47:22.359926+0000","last_fullsized":"2023-12-19T07:47:22.359926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.305902+0000","last_change":"2023-12-19T07:47:21.841183+0000","last_active":"2023-12-19T07:47:22.305902+0000","last_peered":"2023-12-19T07:47:22.305902+0000","last_clean":"2023-12-19T07:47:22.305902+0000","last_became_active":"2023-12-19T07:47:21.839559+0000","last_became_peered":"2023-12-19T07:47:21.839559+0000","last_unstale":"2023-12-19T07:47:22.305902+0000","last_undegraded":"2023-12-19T07:47:22.305902+0000","last_fullsized":"2023-12-19T07:47:22.305902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:27.954678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599577+0000","last_change":"2023-12-19T07:47:17.208713+0000","last_active":"2023-12-19T07:47:22.599577+0000","last_peered":"2023-12-19T07:47:22.599577+0000","last_clean":"2023-12-19T07:47:22.599577+0000","last_became_active":"2023-12-19T07:47:17.202555+0000","last_became_peered":"2023-12-19T07:47:17.202555+0000","last_unstale":"2023-12-19T07:47:22.599577+0000","last_undegraded":"2023-12-19T07:47:22.599577+0000","last_fullsized":"2023-12-19T07:47:22.599577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.356659+0000","last_change":"2023-12-19T07:47:16.198423+0000","last_active":"2023-12-19T07:47:22.356659+0000","last_peered":"2023-12-19T07:47:22.356659+0000","last_clean":"2023-12-19T07:47:22.356659+0000","last_became_active":"2023-12-19T07:47:16.198088+0000","last_became_peered":"2023-12-19T07:47:16.198088+0000","last_unstale":"2023-12-19T07:47:22.356659+0000","last_undegraded":"2023-12-19T07:47:22.356659+0000","last_fullsized":"2023-12-19T07:47:22.356659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597749+0000","last_change":"2023-12-19T07:47:21.845056+0000","last_active":"2023-12-19T07:47:22.597749+0000","last_peered":"2023-12-19T07:47:22.597749+0000","last_clean":"2023-12-19T07:47:22.597749+0000","last_became_active":"2023-12-19T07:47:21.844217+0000","last_became_peered":"2023-12-19T07:47:21.844217+0000","last_unstale":"2023-12-19T07:47:22.597749+0000","last_undegraded":"2023-12-19T07:47:22.597749+0000","last_fullsized":"2023-12-19T07:47:22.597749+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:53:07.229783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352565+0000","last_change":"2023-12-19T07:47:17.200525+0000","last_active":"2023-12-19T07:47:22.352565+0000","last_peered":"2023-12-19T07:47:22.352565+0000","last_clean":"2023-12-19T07:47:22.352565+0000","last_became_active":"2023-12-19T07:47:17.199921+0000","last_became_peered":"2023-12-19T07:47:17.199921+0000","last_unstale":"2023-12-19T07:47:22.352565+0000","last_undegraded":"2023-12-19T07:47:22.352565+0000","last_fullsized":"2023-12-19T07:47:22.352565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868254+0000","last_change":"2023-12-19T07:47:16.213616+0000","last_active":"2023-12-19T07:47:22.868254+0000","last_peered":"2023-12-19T07:47:22.868254+0000","last_clean":"2023-12-19T07:47:22.868254+0000","last_became_active":"2023-12-19T07:47:16.213264+0000","last_became_peered":"2023-12-19T07:47:16.213264+0000","last_unstale":"2023-12-19T07:47:22.868254+0000","last_undegraded":"2023-12-19T07:47:22.868254+0000","last_fullsized":"2023-12-19T07:47:22.868254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902861+0000","last_change":"2023-12-19T07:47:21.841016+0000","last_active":"2023-12-19T07:47:22.902861+0000","last_peered":"2023-12-19T07:47:22.902861+0000","last_clean":"2023-12-19T07:47:22.902861+0000","last_became_active":"2023-12-19T07:47:21.840031+0000","last_became_peered":"2023-12-19T07:47:21.840031+0000","last_unstale":"2023-12-19T07:47:22.902861+0000","last_undegraded":"2023-12-19T07:47:22.902861+0000","last_fullsized":"2023-12-19T07:47:22.902861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807240+0000","last_change":"2023-12-19T07:47:17.205604+0000","last_active":"2023-12-19T07:47:22.807240+0000","last_peered":"2023-12-19T07:47:22.807240+0000","last_clean":"2023-12-19T07:47:22.807240+0000","last_became_active":"2023-12-19T07:47:17.205354+0000","last_became_peered":"2023-12-19T07:47:17.205354+0000","last_unstale":"2023-12-19T07:47:22.807240+0000","last_undegraded":"2023-12-19T07:47:22.807240+0000","last_fullsized":"2023-12-19T07:47:22.807240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352620+0000","last_change":"2023-12-19T07:47:16.201653+0000","last_active":"2023-12-19T07:47:22.352620+0000","last_peered":"2023-12-19T07:47:22.352620+0000","last_clean":"2023-12-19T07:47:22.352620+0000","last_became_active":"2023-12-19T07:47:16.201071+0000","last_became_peered":"2023-12-19T07:47:16.201071+0000","last_unstale":"2023-12-19T07:47:22.352620+0000","last_undegraded":"2023-12-19T07:47:22.352620+0000","last_fullsized":"2023-12-19T07:47:22.352620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351616+0000","last_change":"2023-12-19T07:47:21.262464+0000","last_active":"2023-12-19T07:47:22.351616+0000","last_peered":"2023-12-19T07:47:22.351616+0000","last_clean":"2023-12-19T07:47:22.351616+0000","last_became_active":"2023-12-19T07:47:21.259084+0000","last_became_peered":"2023-12-19T07:47:21.259084+0000","last_unstale":"2023-12-19T07:47:22.351616+0000","last_undegraded":"2023-12-19T07:47:22.351616+0000","last_fullsized":"2023-12-19T07:47:22.351616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307178+0000","last_change":"2023-12-19T07:47:17.212640+0000","last_active":"2023-12-19T07:47:22.307178+0000","last_peered":"2023-12-19T07:47:22.307178+0000","last_clean":"2023-12-19T07:47:22.307178+0000","last_became_active":"2023-12-19T07:47:17.212189+0000","last_became_peered":"2023-12-19T07:47:17.212189+0000","last_unstale":"2023-12-19T07:47:22.307178+0000","last_undegraded":"2023-12-19T07:47:22.307178+0000","last_fullsized":"2023-12-19T07:47:22.307178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806950+0000","last_change":"2023-12-19T07:47:16.201470+0000","last_active":"2023-12-19T07:47:22.806950+0000","last_peered":"2023-12-19T07:47:22.806950+0000","last_clean":"2023-12-19T07:47:22.806950+0000","last_became_active":"2023-12-19T07:47:16.200990+0000","last_became_peered":"2023-12-19T07:47:16.200990+0000","last_unstale":"2023-12-19T07:47:22.806950+0000","last_undegraded":"2023-12-19T07:47:22.806950+0000","last_fullsized":"2023-12-19T07:47:22.806950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352494+0000","last_change":"2023-12-19T07:47:21.261403+0000","last_active":"2023-12-19T07:47:22.352494+0000","last_peered":"2023-12-19T07:47:22.352494+0000","last_clean":"2023-12-19T07:47:22.352494+0000","last_became_active":"2023-12-19T07:47:21.260934+0000","last_became_peered":"2023-12-19T07:47:21.260934+0000","last_unstale":"2023-12-19T07:47:22.352494+0000","last_undegraded":"2023-12-19T07:47:22.352494+0000","last_fullsized":"2023-12-19T07:47:22.352494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351362+0000","last_change":"2023-12-19T07:47:17.196604+0000","last_active":"2023-12-19T07:47:22.351362+0000","last_peered":"2023-12-19T07:47:22.351362+0000","last_clean":"2023-12-19T07:47:22.351362+0000","last_became_active":"2023-12-19T07:47:17.195716+0000","last_became_peered":"2023-12-19T07:47:17.195716+0000","last_unstale":"2023-12-19T07:47:22.351362+0000","last_undegraded":"2023-12-19T07:47:22.351362+0000","last_fullsized":"2023-12-19T07:47:22.351362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302998+0000","last_change":"2023-12-19T07:47:16.188258+0000","last_active":"2023-12-19T07:47:22.302998+0000","last_peered":"2023-12-19T07:47:22.302998+0000","last_clean":"2023-12-19T07:47:22.302998+0000","last_became_active":"2023-12-19T07:47:16.187018+0000","last_became_peered":"2023-12-19T07:47:16.187018+0000","last_unstale":"2023-12-19T07:47:22.302998+0000","last_undegraded":"2023-12-19T07:47:22.302998+0000","last_fullsized":"2023-12-19T07:47:22.302998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353576+0000","last_change":"2023-12-19T07:47:21.269038+0000","last_active":"2023-12-19T07:47:22.353576+0000","last_peered":"2023-12-19T07:47:22.353576+0000","last_clean":"2023-12-19T07:47:22.353576+0000","last_became_active":"2023-12-19T07:47:21.268433+0000","last_became_peered":"2023-12-19T07:47:21.268433+0000","last_unstale":"2023-12-19T07:47:22.353576+0000","last_undegraded":"2023-12-19T07:47:22.353576+0000","last_fullsized":"2023-12-19T07:47:22.353576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903117+0000","last_change":"2023-12-19T07:47:17.201741+0000","last_active":"2023-12-19T07:47:22.903117+0000","last_peered":"2023-12-19T07:47:22.903117+0000","last_clean":"2023-12-19T07:47:22.903117+0000","last_became_active":"2023-12-19T07:47:17.201369+0000","last_became_peered":"2023-12-19T07:47:17.201369+0000","last_unstale":"2023-12-19T07:47:22.903117+0000","last_undegraded":"2023-12-19T07:47:22.903117+0000","last_fullsized":"2023-12-19T07:47:22.903117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866966+0000","last_change":"2023-12-19T07:47:16.213829+0000","last_active":"2023-12-19T07:47:22.866966+0000","last_peered":"2023-12-19T07:47:22.866966+0000","last_clean":"2023-12-19T07:47:22.866966+0000","last_became_active":"2023-12-19T07:47:16.213228+0000","last_became_peered":"2023-12-19T07:47:16.213228+0000","last_unstale":"2023-12-19T07:47:22.866966+0000","last_undegraded":"2023-12-19T07:47:22.866966+0000","last_fullsized":"2023-12-19T07:47:22.866966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902852+0000","last_change":"2023-12-19T07:47:21.841516+0000","last_active":"2023-12-19T07:47:22.902852+0000","last_peered":"2023-12-19T07:47:22.902852+0000","last_clean":"2023-12-19T07:47:22.902852+0000","last_became_active":"2023-12-19T07:47:21.840477+0000","last_became_peered":"2023-12-19T07:47:21.840477+0000","last_unstale":"2023-12-19T07:47:22.902852+0000","last_undegraded":"2023-12-19T07:47:22.902852+0000","last_fullsized":"2023-12-19T07:47:22.902852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598774+0000","last_change":"2023-12-19T07:47:17.200992+0000","last_active":"2023-12-19T07:47:22.598774+0000","last_peered":"2023-12-19T07:47:22.598774+0000","last_clean":"2023-12-19T07:47:22.598774+0000","last_became_active":"2023-12-19T07:47:17.199427+0000","last_became_peered":"2023-12-19T07:47:17.199427+0000","last_unstale":"2023-12-19T07:47:22.598774+0000","last_undegraded":"2023-12-19T07:47:22.598774+0000","last_fullsized":"2023-12-19T07:47:22.598774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806565+0000","last_change":"2023-12-19T07:47:16.201062+0000","last_active":"2023-12-19T07:47:22.806565+0000","last_peered":"2023-12-19T07:47:22.806565+0000","last_clean":"2023-12-19T07:47:22.806565+0000","last_became_active":"2023-12-19T07:47:16.199674+0000","last_became_peered":"2023-12-19T07:47:16.199674+0000","last_unstale":"2023-12-19T07:47:22.806565+0000","last_undegraded":"2023-12-19T07:47:22.806565+0000","last_fullsized":"2023-12-19T07:47:22.806565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:33.372367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867080+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:47:22.867080+0000","last_peered":"2023-12-19T07:47:22.867080+0000","last_clean":"2023-12-19T07:47:22.867080+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:47:22.867080+0000","last_undegraded":"2023-12-19T07:47:22.867080+0000","last_fullsized":"2023-12-19T07:47:22.867080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359462+0000","last_change":"2023-12-19T07:47:21.844050+0000","last_active":"2023-12-19T07:47:22.359462+0000","last_peered":"2023-12-19T07:47:22.359462+0000","last_clean":"2023-12-19T07:47:22.359462+0000","last_became_active":"2023-12-19T07:47:21.843480+0000","last_became_peered":"2023-12-19T07:47:21.843480+0000","last_unstale":"2023-12-19T07:47:22.359462+0000","last_undegraded":"2023-12-19T07:47:22.359462+0000","last_fullsized":"2023-12-19T07:47:22.359462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:29.237260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867932+0000","last_change":"2023-12-19T07:47:17.201683+0000","last_active":"2023-12-19T07:47:22.867932+0000","last_peered":"2023-12-19T07:47:22.867932+0000","last_clean":"2023-12-19T07:47:22.867932+0000","last_became_active":"2023-12-19T07:47:17.201138+0000","last_became_peered":"2023-12-19T07:47:17.201138+0000","last_unstale":"2023-12-19T07:47:22.867932+0000","last_undegraded":"2023-12-19T07:47:22.867932+0000","last_fullsized":"2023-12-19T07:47:22.867932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:25.176702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303850+0000","last_change":"2023-12-19T07:47:16.199670+0000","last_active":"2023-12-19T07:47:22.303850+0000","last_peered":"2023-12-19T07:47:22.303850+0000","last_clean":"2023-12-19T07:47:22.303850+0000","last_became_active":"2023-12-19T07:47:16.199236+0000","last_became_peered":"2023-12-19T07:47:16.199236+0000","last_unstale":"2023-12-19T07:47:22.303850+0000","last_undegraded":"2023-12-19T07:47:22.303850+0000","last_fullsized":"2023-12-19T07:47:22.303850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358653+0000","last_change":"2023-12-19T07:47:21.844306+0000","last_active":"2023-12-19T07:47:22.358653+0000","last_peered":"2023-12-19T07:47:22.358653+0000","last_clean":"2023-12-19T07:47:22.358653+0000","last_became_active":"2023-12-19T07:47:21.843694+0000","last_became_peered":"2023-12-19T07:47:21.843694+0000","last_unstale":"2023-12-19T07:47:22.358653+0000","last_undegraded":"2023-12-19T07:47:22.358653+0000","last_fullsized":"2023-12-19T07:47:22.358653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:25.534670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806965+0000","last_change":"2023-12-19T07:47:17.210076+0000","last_active":"2023-12-19T07:47:22.806965+0000","last_peered":"2023-12-19T07:47:22.806965+0000","last_clean":"2023-12-19T07:47:22.806965+0000","last_became_active":"2023-12-19T07:47:17.209871+0000","last_became_peered":"2023-12-19T07:47:17.209871+0000","last_unstale":"2023-12-19T07:47:22.806965+0000","last_undegraded":"2023-12-19T07:47:22.806965+0000","last_fullsized":"2023-12-19T07:47:22.806965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353351+0000","last_change":"2023-12-19T07:47:16.187382+0000","last_active":"2023-12-19T07:47:22.353351+0000","last_peered":"2023-12-19T07:47:22.353351+0000","last_clean":"2023-12-19T07:47:22.353351+0000","last_became_active":"2023-12-19T07:47:16.187021+0000","last_became_peered":"2023-12-19T07:47:16.187021+0000","last_unstale":"2023-12-19T07:47:22.353351+0000","last_undegraded":"2023-12-19T07:47:22.353351+0000","last_fullsized":"2023-12-19T07:47:22.353351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:03.039957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903413+0000","last_change":"2023-12-19T07:47:21.265329+0000","last_active":"2023-12-19T07:47:22.903413+0000","last_peered":"2023-12-19T07:47:22.903413+0000","last_clean":"2023-12-19T07:47:22.903413+0000","last_became_active":"2023-12-19T07:47:21.264912+0000","last_became_peered":"2023-12-19T07:47:21.264912+0000","last_unstale":"2023-12-19T07:47:22.903413+0000","last_undegraded":"2023-12-19T07:47:22.903413+0000","last_fullsized":"2023-12-19T07:47:22.903413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868427+0000","last_change":"2023-12-19T07:47:17.215646+0000","last_active":"2023-12-19T07:47:22.868427+0000","last_peered":"2023-12-19T07:47:22.868427+0000","last_clean":"2023-12-19T07:47:22.868427+0000","last_became_active":"2023-12-19T07:47:17.215345+0000","last_became_peered":"2023-12-19T07:47:17.215345+0000","last_unstale":"2023-12-19T07:47:22.868427+0000","last_undegraded":"2023-12-19T07:47:22.868427+0000","last_fullsized":"2023-12-19T07:47:22.868427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:05.660865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304403+0000","last_change":"2023-12-19T07:47:16.195752+0000","last_active":"2023-12-19T07:47:22.304403+0000","last_peered":"2023-12-19T07:47:22.304403+0000","last_clean":"2023-12-19T07:47:22.304403+0000","last_became_active":"2023-12-19T07:47:16.195415+0000","last_became_peered":"2023-12-19T07:47:16.195415+0000","last_unstale":"2023-12-19T07:47:22.304403+0000","last_undegraded":"2023-12-19T07:47:22.304403+0000","last_fullsized":"2023-12-19T07:47:22.304403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805409+0000","last_change":"2023-12-19T07:47:21.840195+0000","last_active":"2023-12-19T07:47:22.805409+0000","last_peered":"2023-12-19T07:47:22.805409+0000","last_clean":"2023-12-19T07:47:22.805409+0000","last_became_active":"2023-12-19T07:47:21.839557+0000","last_became_peered":"2023-12-19T07:47:21.839557+0000","last_unstale":"2023-12-19T07:47:22.805409+0000","last_undegraded":"2023-12-19T07:47:22.805409+0000","last_fullsized":"2023-12-19T07:47:22.805409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867970+0000","last_change":"2023-12-19T07:47:17.211241+0000","last_active":"2023-12-19T07:47:22.867970+0000","last_peered":"2023-12-19T07:47:22.867970+0000","last_clean":"2023-12-19T07:47:22.867970+0000","last_became_active":"2023-12-19T07:47:17.210508+0000","last_became_peered":"2023-12-19T07:47:17.210508+0000","last_unstale":"2023-12-19T07:47:22.867970+0000","last_undegraded":"2023-12-19T07:47:22.867970+0000","last_fullsized":"2023-12-19T07:47:22.867970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:48.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307381+0000","last_change":"2023-12-19T07:47:16.198483+0000","last_active":"2023-12-19T07:47:22.307381+0000","last_peered":"2023-12-19T07:47:22.307381+0000","last_clean":"2023-12-19T07:47:22.307381+0000","last_became_active":"2023-12-19T07:47:16.198150+0000","last_became_peered":"2023-12-19T07:47:16.198150+0000","last_unstale":"2023-12-19T07:47:22.307381+0000","last_undegraded":"2023-12-19T07:47:22.307381+0000","last_fullsized":"2023-12-19T07:47:22.307381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306109+0000","last_change":"2023-12-19T07:47:21.259856+0000","last_active":"2023-12-19T07:47:22.306109+0000","last_peered":"2023-12-19T07:47:22.306109+0000","last_clean":"2023-12-19T07:47:22.306109+0000","last_became_active":"2023-12-19T07:47:21.259147+0000","last_became_peered":"2023-12-19T07:47:21.259147+0000","last_unstale":"2023-12-19T07:47:22.306109+0000","last_undegraded":"2023-12-19T07:47:22.306109+0000","last_fullsized":"2023-12-19T07:47:22.306109+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867790+0000","last_change":"2023-12-19T07:47:17.209489+0000","last_active":"2023-12-19T07:47:22.867790+0000","last_peered":"2023-12-19T07:47:22.867790+0000","last_clean":"2023-12-19T07:47:22.867790+0000","last_became_active":"2023-12-19T07:47:17.209131+0000","last_became_peered":"2023-12-19T07:47:17.209131+0000","last_unstale":"2023-12-19T07:47:22.867790+0000","last_undegraded":"2023-12-19T07:47:22.867790+0000","last_fullsized":"2023-12-19T07:47:22.867790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303258+0000","last_change":"2023-12-19T07:47:16.192274+0000","last_active":"2023-12-19T07:47:22.303258+0000","last_peered":"2023-12-19T07:47:22.303258+0000","last_clean":"2023-12-19T07:47:22.303258+0000","last_became_active":"2023-12-19T07:47:16.190777+0000","last_became_peered":"2023-12-19T07:47:16.190777+0000","last_unstale":"2023-12-19T07:47:22.303258+0000","last_undegraded":"2023-12-19T07:47:22.303258+0000","last_fullsized":"2023-12-19T07:47:22.303258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904035+0000","last_change":"2023-12-19T07:47:21.260731+0000","last_active":"2023-12-19T07:47:22.904035+0000","last_peered":"2023-12-19T07:47:22.904035+0000","last_clean":"2023-12-19T07:47:22.904035+0000","last_became_active":"2023-12-19T07:47:21.258011+0000","last_became_peered":"2023-12-19T07:47:21.258011+0000","last_unstale":"2023-12-19T07:47:22.904035+0000","last_undegraded":"2023-12-19T07:47:22.904035+0000","last_fullsized":"2023-12-19T07:47:22.904035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359125+0000","last_change":"2023-12-19T07:47:17.201434+0000","last_active":"2023-12-19T07:47:22.359125+0000","last_peered":"2023-12-19T07:47:22.359125+0000","last_clean":"2023-12-19T07:47:22.359125+0000","last_became_active":"2023-12-19T07:47:17.200473+0000","last_became_peered":"2023-12-19T07:47:17.200473+0000","last_unstale":"2023-12-19T07:47:22.359125+0000","last_undegraded":"2023-12-19T07:47:22.359125+0000","last_fullsized":"2023-12-19T07:47:22.359125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359065+0000","last_change":"2023-12-19T07:47:16.193445+0000","last_active":"2023-12-19T07:47:22.359065+0000","last_peered":"2023-12-19T07:47:22.359065+0000","last_clean":"2023-12-19T07:47:22.359065+0000","last_became_active":"2023-12-19T07:47:16.192600+0000","last_became_peered":"2023-12-19T07:47:16.192600+0000","last_unstale":"2023-12-19T07:47:22.359065+0000","last_undegraded":"2023-12-19T07:47:22.359065+0000","last_fullsized":"2023-12-19T07:47:22.359065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597903+0000","last_change":"2023-12-19T07:47:21.847662+0000","last_active":"2023-12-19T07:47:22.597903+0000","last_peered":"2023-12-19T07:47:22.597903+0000","last_clean":"2023-12-19T07:47:22.597903+0000","last_became_active":"2023-12-19T07:47:21.847110+0000","last_became_peered":"2023-12-19T07:47:21.847110+0000","last_unstale":"2023-12-19T07:47:22.597903+0000","last_undegraded":"2023-12-19T07:47:22.597903+0000","last_fullsized":"2023-12-19T07:47:22.597903+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:07:23.838326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867579+0000","last_change":"2023-12-19T07:47:17.219151+0000","last_active":"2023-12-19T07:47:22.867579+0000","last_peered":"2023-12-19T07:47:22.867579+0000","last_clean":"2023-12-19T07:47:22.867579+0000","last_became_active":"2023-12-19T07:47:17.218940+0000","last_became_peered":"2023-12-19T07:47:17.218940+0000","last_unstale":"2023-12-19T07:47:22.867579+0000","last_undegraded":"2023-12-19T07:47:22.867579+0000","last_fullsized":"2023-12-19T07:47:22.867579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806721+0000","last_change":"2023-12-19T07:47:16.201106+0000","last_active":"2023-12-19T07:47:22.806721+0000","last_peered":"2023-12-19T07:47:22.806721+0000","last_clean":"2023-12-19T07:47:22.806721+0000","last_became_active":"2023-12-19T07:47:16.200628+0000","last_became_peered":"2023-12-19T07:47:16.200628+0000","last_unstale":"2023-12-19T07:47:22.806721+0000","last_undegraded":"2023-12-19T07:47:22.806721+0000","last_fullsized":"2023-12-19T07:47:22.806721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597986+0000","last_change":"2023-12-19T07:47:21.847256+0000","last_active":"2023-12-19T07:47:22.597986+0000","last_peered":"2023-12-19T07:47:22.597986+0000","last_clean":"2023-12-19T07:47:22.597986+0000","last_became_active":"2023-12-19T07:47:21.843240+0000","last_became_peered":"2023-12-19T07:47:21.843240+0000","last_unstale":"2023-12-19T07:47:22.597986+0000","last_undegraded":"2023-12-19T07:47:22.597986+0000","last_fullsized":"2023-12-19T07:47:22.597986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:45.450328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903865+0000","last_change":"2023-12-19T07:47:17.200704+0000","last_active":"2023-12-19T07:47:22.903865+0000","last_peered":"2023-12-19T07:47:22.903865+0000","last_clean":"2023-12-19T07:47:22.903865+0000","last_became_active":"2023-12-19T07:47:17.200382+0000","last_became_peered":"2023-12-19T07:47:17.200382+0000","last_unstale":"2023-12-19T07:47:22.903865+0000","last_undegraded":"2023-12-19T07:47:22.903865+0000","last_fullsized":"2023-12-19T07:47:22.903865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903656+0000","last_change":"2023-12-19T07:47:16.189068+0000","last_active":"2023-12-19T07:47:22.903656+0000","last_peered":"2023-12-19T07:47:22.903656+0000","last_clean":"2023-12-19T07:47:22.903656+0000","last_became_active":"2023-12-19T07:47:16.188798+0000","last_became_peered":"2023-12-19T07:47:16.188798+0000","last_unstale":"2023-12-19T07:47:22.903656+0000","last_undegraded":"2023-12-19T07:47:22.903656+0000","last_fullsized":"2023-12-19T07:47:22.903656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805530+0000","last_change":"2023-12-19T07:47:21.847444+0000","last_active":"2023-12-19T07:47:22.805530+0000","last_peered":"2023-12-19T07:47:22.805530+0000","last_clean":"2023-12-19T07:47:22.805530+0000","last_became_active":"2023-12-19T07:47:21.847123+0000","last_became_peered":"2023-12-19T07:47:21.847123+0000","last_unstale":"2023-12-19T07:47:22.805530+0000","last_undegraded":"2023-12-19T07:47:22.805530+0000","last_fullsized":"2023-12-19T07:47:22.805530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867681+0000","last_change":"2023-12-19T07:47:17.212222+0000","last_active":"2023-12-19T07:47:22.867681+0000","last_peered":"2023-12-19T07:47:22.867681+0000","last_clean":"2023-12-19T07:47:22.867681+0000","last_became_active":"2023-12-19T07:47:17.211536+0000","last_became_peered":"2023-12-19T07:47:17.211536+0000","last_unstale":"2023-12-19T07:47:22.867681+0000","last_undegraded":"2023-12-19T07:47:22.867681+0000","last_fullsized":"2023-12-19T07:47:22.867681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:40.976961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-19T07:47:29.088364+0000","last_change":"2023-12-19T07:47:16.200498+0000","last_active":"2023-12-19T07:47:29.088364+0000","last_peered":"2023-12-19T07:47:29.088364+0000","last_clean":"2023-12-19T07:47:29.088364+0000","last_became_active":"2023-12-19T07:47:16.200159+0000","last_became_peered":"2023-12-19T07:47:16.200159+0000","last_unstale":"2023-12-19T07:47:29.088364+0000","last_undegraded":"2023-12-19T07:47:29.088364+0000","last_fullsized":"2023-12-19T07:47:29.088364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351798+0000","last_change":"2023-12-19T07:47:21.269837+0000","last_active":"2023-12-19T07:47:22.351798+0000","last_peered":"2023-12-19T07:47:22.351798+0000","last_clean":"2023-12-19T07:47:22.351798+0000","last_became_active":"2023-12-19T07:47:21.269371+0000","last_became_peered":"2023-12-19T07:47:21.269371+0000","last_unstale":"2023-12-19T07:47:22.351798+0000","last_undegraded":"2023-12-19T07:47:22.351798+0000","last_fullsized":"2023-12-19T07:47:22.351798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307009+0000","last_change":"2023-12-19T07:47:17.206123+0000","last_active":"2023-12-19T07:47:22.307009+0000","last_peered":"2023-12-19T07:47:22.307009+0000","last_clean":"2023-12-19T07:47:22.307009+0000","last_became_active":"2023-12-19T07:47:17.202604+0000","last_became_peered":"2023-12-19T07:47:17.202604+0000","last_unstale":"2023-12-19T07:47:22.307009+0000","last_undegraded":"2023-12-19T07:47:22.307009+0000","last_fullsized":"2023-12-19T07:47:22.307009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353492+0000","last_change":"2023-12-19T07:47:16.207087+0000","last_active":"2023-12-19T07:47:22.353492+0000","last_peered":"2023-12-19T07:47:22.353492+0000","last_clean":"2023-12-19T07:47:22.353492+0000","last_became_active":"2023-12-19T07:47:16.205832+0000","last_became_peered":"2023-12-19T07:47:16.205832+0000","last_unstale":"2023-12-19T07:47:22.353492+0000","last_undegraded":"2023-12-19T07:47:22.353492+0000","last_fullsized":"2023-12-19T07:47:22.353492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307124+0000","last_change":"2023-12-19T07:47:21.259755+0000","last_active":"2023-12-19T07:47:22.307124+0000","last_peered":"2023-12-19T07:47:22.307124+0000","last_clean":"2023-12-19T07:47:22.307124+0000","last_became_active":"2023-12-19T07:47:21.259163+0000","last_became_peered":"2023-12-19T07:47:21.259163+0000","last_unstale":"2023-12-19T07:47:22.307124+0000","last_undegraded":"2023-12-19T07:47:22.307124+0000","last_fullsized":"2023-12-19T07:47:22.307124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904807+0000","last_change":"2023-12-19T07:47:17.202501+0000","last_active":"2023-12-19T07:47:22.904807+0000","last_peered":"2023-12-19T07:47:22.904807+0000","last_clean":"2023-12-19T07:47:22.904807+0000","last_became_active":"2023-12-19T07:47:17.199344+0000","last_became_peered":"2023-12-19T07:47:17.199344+0000","last_unstale":"2023-12-19T07:47:22.904807+0000","last_undegraded":"2023-12-19T07:47:22.904807+0000","last_fullsized":"2023-12-19T07:47:22.904807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807353+0000","last_change":"2023-12-19T07:47:16.200080+0000","last_active":"2023-12-19T07:47:22.807353+0000","last_peered":"2023-12-19T07:47:22.807353+0000","last_clean":"2023-12-19T07:47:22.807353+0000","last_became_active":"2023-12-19T07:47:16.199182+0000","last_became_peered":"2023-12-19T07:47:16.199182+0000","last_unstale":"2023-12-19T07:47:22.807353+0000","last_undegraded":"2023-12-19T07:47:22.807353+0000","last_fullsized":"2023-12-19T07:47:22.807353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598231+0000","last_change":"2023-12-19T07:47:21.264957+0000","last_active":"2023-12-19T07:47:22.598231+0000","last_peered":"2023-12-19T07:47:22.598231+0000","last_clean":"2023-12-19T07:47:22.598231+0000","last_became_active":"2023-12-19T07:47:21.264514+0000","last_became_peered":"2023-12-19T07:47:21.264514+0000","last_unstale":"2023-12-19T07:47:22.598231+0000","last_undegraded":"2023-12-19T07:47:22.598231+0000","last_fullsized":"2023-12-19T07:47:22.598231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:20.377759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807125+0000","last_change":"2023-12-19T07:47:17.199047+0000","last_active":"2023-12-19T07:47:22.807125+0000","last_peered":"2023-12-19T07:47:22.807125+0000","last_clean":"2023-12-19T07:47:22.807125+0000","last_became_active":"2023-12-19T07:47:17.198760+0000","last_became_peered":"2023-12-19T07:47:17.198760+0000","last_unstale":"2023-12-19T07:47:22.807125+0000","last_undegraded":"2023-12-19T07:47:22.807125+0000","last_fullsized":"2023-12-19T07:47:22.807125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359707+0000","last_change":"2023-12-19T07:47:16.206922+0000","last_active":"2023-12-19T07:47:22.359707+0000","last_peered":"2023-12-19T07:47:22.359707+0000","last_clean":"2023-12-19T07:47:22.359707+0000","last_became_active":"2023-12-19T07:47:16.206605+0000","last_became_peered":"2023-12-19T07:47:16.206605+0000","last_unstale":"2023-12-19T07:47:22.359707+0000","last_undegraded":"2023-12-19T07:47:22.359707+0000","last_fullsized":"2023-12-19T07:47:22.359707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597807+0000","last_change":"2023-12-19T07:47:21.845593+0000","last_active":"2023-12-19T07:47:22.597807+0000","last_peered":"2023-12-19T07:47:22.597807+0000","last_clean":"2023-12-19T07:47:22.597807+0000","last_became_active":"2023-12-19T07:47:21.843663+0000","last_became_peered":"2023-12-19T07:47:21.843663+0000","last_unstale":"2023-12-19T07:47:22.597807+0000","last_undegraded":"2023-12-19T07:47:22.597807+0000","last_fullsized":"2023-12-19T07:47:22.597807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902979+0000","last_change":"2023-12-19T07:47:21.846558+0000","last_active":"2023-12-19T07:47:22.902979+0000","last_peered":"2023-12-19T07:47:22.902979+0000","last_clean":"2023-12-19T07:47:22.902979+0000","last_became_active":"2023-12-19T07:47:21.845861+0000","last_became_peered":"2023-12-19T07:47:21.845861+0000","last_unstale":"2023-12-19T07:47:22.902979+0000","last_undegraded":"2023-12-19T07:47:22.902979+0000","last_fullsized":"2023-12-19T07:47:22.902979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:28.389344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351325+0000","last_change":"2023-12-19T07:47:17.210958+0000","last_active":"2023-12-19T07:47:22.351325+0000","last_peered":"2023-12-19T07:47:22.351325+0000","last_clean":"2023-12-19T07:47:22.351325+0000","last_became_active":"2023-12-19T07:47:17.210537+0000","last_became_peered":"2023-12-19T07:47:17.210537+0000","last_unstale":"2023-12-19T07:47:22.351325+0000","last_undegraded":"2023-12-19T07:47:22.351325+0000","last_fullsized":"2023-12-19T07:47:22.351325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805998+0000","last_change":"2023-12-19T07:47:16.198817+0000","last_active":"2023-12-19T07:47:22.805998+0000","last_peered":"2023-12-19T07:47:22.805998+0000","last_clean":"2023-12-19T07:47:22.805998+0000","last_became_active":"2023-12-19T07:47:16.198491+0000","last_became_peered":"2023-12-19T07:47:16.198491+0000","last_unstale":"2023-12-19T07:47:22.805998+0000","last_undegraded":"2023-12-19T07:47:22.805998+0000","last_fullsized":"2023-12-19T07:47:22.805998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359608+0000","last_change":"2023-12-19T07:47:21.263980+0000","last_active":"2023-12-19T07:47:22.359608+0000","last_peered":"2023-12-19T07:47:22.359608+0000","last_clean":"2023-12-19T07:47:22.359608+0000","last_became_active":"2023-12-19T07:47:21.263310+0000","last_became_peered":"2023-12-19T07:47:21.263310+0000","last_unstale":"2023-12-19T07:47:22.359608+0000","last_undegraded":"2023-12-19T07:47:22.359608+0000","last_fullsized":"2023-12-19T07:47:22.359608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.138125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598272+0000","last_change":"2023-12-19T07:47:17.196870+0000","last_active":"2023-12-19T07:47:22.598272+0000","last_peered":"2023-12-19T07:47:22.598272+0000","last_clean":"2023-12-19T07:47:22.598272+0000","last_became_active":"2023-12-19T07:47:17.196588+0000","last_became_peered":"2023-12-19T07:47:17.196588+0000","last_unstale":"2023-12-19T07:47:22.598272+0000","last_undegraded":"2023-12-19T07:47:22.598272+0000","last_fullsized":"2023-12-19T07:47:22.598272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598626+0000","last_change":"2023-12-19T07:47:16.198918+0000","last_active":"2023-12-19T07:47:22.598626+0000","last_peered":"2023-12-19T07:47:22.598626+0000","last_clean":"2023-12-19T07:47:22.598626+0000","last_became_active":"2023-12-19T07:47:16.198496+0000","last_became_peered":"2023-12-19T07:47:16.198496+0000","last_unstale":"2023-12-19T07:47:22.598626+0000","last_undegraded":"2023-12-19T07:47:22.598626+0000","last_fullsized":"2023-12-19T07:47:22.598626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358925+0000","last_change":"2023-12-19T07:47:16.192976+0000","last_active":"2023-12-19T07:47:22.358925+0000","last_peered":"2023-12-19T07:47:22.358925+0000","last_clean":"2023-12-19T07:47:22.358925+0000","last_became_active":"2023-12-19T07:47:16.192588+0000","last_became_peered":"2023-12-19T07:47:16.192588+0000","last_unstale":"2023-12-19T07:47:22.358925+0000","last_undegraded":"2023-12-19T07:47:22.358925+0000","last_fullsized":"2023-12-19T07:47:22.358925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903549+0000","last_change":"2023-12-19T07:47:17.205110+0000","last_active":"2023-12-19T07:47:22.903549+0000","last_peered":"2023-12-19T07:47:22.903549+0000","last_clean":"2023-12-19T07:47:22.903549+0000","last_became_active":"2023-12-19T07:47:17.204811+0000","last_became_peered":"2023-12-19T07:47:17.204811+0000","last_unstale":"2023-12-19T07:47:22.903549+0000","last_undegraded":"2023-12-19T07:47:22.903549+0000","last_fullsized":"2023-12-19T07:47:22.903549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:07.078408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303570+0000","last_change":"2023-12-19T07:47:21.259361+0000","last_active":"2023-12-19T07:47:22.303570+0000","last_peered":"2023-12-19T07:47:22.303570+0000","last_clean":"2023-12-19T07:47:22.303570+0000","last_became_active":"2023-12-19T07:47:21.258883+0000","last_became_peered":"2023-12-19T07:47:21.258883+0000","last_unstale":"2023-12-19T07:47:22.303570+0000","last_undegraded":"2023-12-19T07:47:22.303570+0000","last_fullsized":"2023-12-19T07:47:22.303570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:28.728929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598981+0000","last_change":"2023-12-19T07:47:16.206769+0000","last_active":"2023-12-19T07:47:22.598981+0000","last_peered":"2023-12-19T07:47:22.598981+0000","last_clean":"2023-12-19T07:47:22.598981+0000","last_became_active":"2023-12-19T07:47:16.206392+0000","last_became_peered":"2023-12-19T07:47:16.206392+0000","last_unstale":"2023-12-19T07:47:22.598981+0000","last_undegraded":"2023-12-19T07:47:22.598981+0000","last_fullsized":"2023-12-19T07:47:22.598981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306430+0000","last_change":"2023-12-19T07:47:17.204694+0000","last_active":"2023-12-19T07:47:22.306430+0000","last_peered":"2023-12-19T07:47:22.306430+0000","last_clean":"2023-12-19T07:47:22.306430+0000","last_became_active":"2023-12-19T07:47:17.204276+0000","last_became_peered":"2023-12-19T07:47:17.204276+0000","last_unstale":"2023-12-19T07:47:22.306430+0000","last_undegraded":"2023-12-19T07:47:22.306430+0000","last_fullsized":"2023-12-19T07:47:22.306430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599471+0000","last_change":"2023-12-19T07:47:21.258311+0000","last_active":"2023-12-19T07:47:22.599471+0000","last_peered":"2023-12-19T07:47:22.599471+0000","last_clean":"2023-12-19T07:47:22.599471+0000","last_became_active":"2023-12-19T07:47:21.257510+0000","last_became_peered":"2023-12-19T07:47:21.257510+0000","last_unstale":"2023-12-19T07:47:22.599471+0000","last_undegraded":"2023-12-19T07:47:22.599471+0000","last_fullsized":"2023-12-19T07:47:22.599471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:15:44.535034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358817+0000","last_change":"2023-12-19T07:47:17.204892+0000","last_active":"2023-12-19T07:47:22.358817+0000","last_peered":"2023-12-19T07:47:22.358817+0000","last_clean":"2023-12-19T07:47:22.358817+0000","last_became_active":"2023-12-19T07:47:17.201155+0000","last_became_peered":"2023-12-19T07:47:17.201155+0000","last_unstale":"2023-12-19T07:47:22.358817+0000","last_undegraded":"2023-12-19T07:47:22.358817+0000","last_fullsized":"2023-12-19T07:47:22.358817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302909+0000","last_change":"2023-12-19T07:47:17.204307+0000","last_active":"2023-12-19T07:47:22.302909+0000","last_peered":"2023-12-19T07:47:22.302909+0000","last_clean":"2023-12-19T07:47:22.302909+0000","last_became_active":"2023-12-19T07:47:17.203914+0000","last_became_peered":"2023-12-19T07:47:17.203914+0000","last_unstale":"2023-12-19T07:47:22.302909+0000","last_undegraded":"2023-12-19T07:47:22.302909+0000","last_fullsized":"2023-12-19T07:47:22.302909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806686+0000","last_change":"2023-12-19T07:47:17.196779+0000","last_active":"2023-12-19T07:47:22.806686+0000","last_peered":"2023-12-19T07:47:22.806686+0000","last_clean":"2023-12-19T07:47:22.806686+0000","last_became_active":"2023-12-19T07:47:17.196494+0000","last_became_peered":"2023-12-19T07:47:17.196494+0000","last_unstale":"2023-12-19T07:47:22.806686+0000","last_undegraded":"2023-12-19T07:47:22.806686+0000","last_fullsized":"2023-12-19T07:47:22.806686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903605+0000","last_change":"2023-12-19T07:47:17.205970+0000","last_active":"2023-12-19T07:47:22.903605+0000","last_peered":"2023-12-19T07:47:22.903605+0000","last_clean":"2023-12-19T07:47:22.903605+0000","last_became_active":"2023-12-19T07:47:17.205581+0000","last_became_peered":"2023-12-19T07:47:17.205581+0000","last_unstale":"2023-12-19T07:47:22.903605+0000","last_undegraded":"2023-12-19T07:47:22.903605+0000","last_fullsized":"2023-12-19T07:47:22.903605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599435+0000","last_change":"2023-12-19T07:47:17.208151+0000","last_active":"2023-12-19T07:47:22.599435+0000","last_peered":"2023-12-19T07:47:22.599435+0000","last_clean":"2023-12-19T07:47:22.599435+0000","last_became_active":"2023-12-19T07:47:17.207880+0000","last_became_peered":"2023-12-19T07:47:17.207880+0000","last_unstale":"2023-12-19T07:47:22.599435+0000","last_undegraded":"2023-12-19T07:47:22.599435+0000","last_fullsized":"2023-12-19T07:47:22.599435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352831+0000","last_change":"2023-12-19T07:47:17.209159+0000","last_active":"2023-12-19T07:47:22.352831+0000","last_peered":"2023-12-19T07:47:22.352831+0000","last_clean":"2023-12-19T07:47:22.352831+0000","last_became_active":"2023-12-19T07:47:17.208870+0000","last_became_peered":"2023-12-19T07:47:17.208870+0000","last_unstale":"2023-12-19T07:47:22.352831+0000","last_undegraded":"2023-12-19T07:47:22.352831+0000","last_fullsized":"2023-12-19T07:47:22.352831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.360063+0000","last_change":"2023-12-19T07:47:17.209320+0000","last_active":"2023-12-19T07:47:22.360063+0000","last_peered":"2023-12-19T07:47:22.360063+0000","last_clean":"2023-12-19T07:47:22.360063+0000","last_became_active":"2023-12-19T07:47:17.208263+0000","last_became_peered":"2023-12-19T07:47:17.208263+0000","last_unstale":"2023-12-19T07:47:22.360063+0000","last_undegraded":"2023-12-19T07:47:22.360063+0000","last_fullsized":"2023-12-19T07:47:22.360063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599483+0000","last_change":"2023-12-19T07:47:17.211203+0000","last_active":"2023-12-19T07:47:22.599483+0000","last_peered":"2023-12-19T07:47:22.599483+0000","last_clean":"2023-12-19T07:47:22.599483+0000","last_became_active":"2023-12-19T07:47:17.210860+0000","last_became_peered":"2023-12-19T07:47:17.210860+0000","last_unstale":"2023-12-19T07:47:22.599483+0000","last_undegraded":"2023-12-19T07:47:22.599483+0000","last_fullsized":"2023-12-19T07:47:22.599483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805820+0000","last_change":"2023-12-19T07:47:17.210725+0000","last_active":"2023-12-19T07:47:22.805820+0000","last_peered":"2023-12-19T07:47:22.805820+0000","last_clean":"2023-12-19T07:47:22.805820+0000","last_became_active":"2023-12-19T07:47:17.210432+0000","last_became_peered":"2023-12-19T07:47:17.210432+0000","last_unstale":"2023-12-19T07:47:22.805820+0000","last_undegraded":"2023-12-19T07:47:22.805820+0000","last_fullsized":"2023-12-19T07:47:22.805820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599063+0000","last_change":"2023-12-19T07:47:17.200440+0000","last_active":"2023-12-19T07:47:22.599063+0000","last_peered":"2023-12-19T07:47:22.599063+0000","last_clean":"2023-12-19T07:47:22.599063+0000","last_became_active":"2023-12-19T07:47:17.200137+0000","last_became_peered":"2023-12-19T07:47:17.200137+0000","last_unstale":"2023-12-19T07:47:22.599063+0000","last_undegraded":"2023-12-19T07:47:22.599063+0000","last_fullsized":"2023-12-19T07:47:22.599063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806350+0000","last_change":"2023-12-19T07:47:17.200217+0000","last_active":"2023-12-19T07:47:22.806350+0000","last_peered":"2023-12-19T07:47:22.806350+0000","last_clean":"2023-12-19T07:47:22.806350+0000","last_became_active":"2023-12-19T07:47:17.199920+0000","last_became_peered":"2023-12-19T07:47:17.199920+0000","last_unstale":"2023-12-19T07:47:22.806350+0000","last_undegraded":"2023-12-19T07:47:22.806350+0000","last_fullsized":"2023-12-19T07:47:22.806350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903403+0000","last_change":"2023-12-19T07:47:17.200705+0000","last_active":"2023-12-19T07:47:22.903403+0000","last_peered":"2023-12-19T07:47:22.903403+0000","last_clean":"2023-12-19T07:47:22.903403+0000","last_became_active":"2023-12-19T07:47:17.200462+0000","last_became_peered":"2023-12-19T07:47:17.200462+0000","last_unstale":"2023-12-19T07:47:22.903403+0000","last_undegraded":"2023-12-19T07:47:22.903403+0000","last_fullsized":"2023-12-19T07:47:22.903403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359431+0000","last_change":"2023-12-19T07:47:17.212064+0000","last_active":"2023-12-19T07:47:22.359431+0000","last_peered":"2023-12-19T07:47:22.359431+0000","last_clean":"2023-12-19T07:47:22.359431+0000","last_became_active":"2023-12-19T07:47:17.211684+0000","last_became_peered":"2023-12-19T07:47:17.211684+0000","last_unstale":"2023-12-19T07:47:22.359431+0000","last_undegraded":"2023-12-19T07:47:22.359431+0000","last_fullsized":"2023-12-19T07:47:22.359431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:21.908290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868184+0000","last_change":"2023-12-19T07:47:17.212402+0000","last_active":"2023-12-19T07:47:22.868184+0000","last_peered":"2023-12-19T07:47:22.868184+0000","last_clean":"2023-12-19T07:47:22.868184+0000","last_became_active":"2023-12-19T07:47:17.211887+0000","last_became_peered":"2023-12-19T07:47:17.211887+0000","last_unstale":"2023-12-19T07:47:22.868184+0000","last_undegraded":"2023-12-19T07:47:22.868184+0000","last_fullsized":"2023-12-19T07:47:22.868184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304186+0000","last_change":"2023-12-19T07:47:17.207392+0000","last_active":"2023-12-19T07:47:22.304186+0000","last_peered":"2023-12-19T07:47:22.304186+0000","last_clean":"2023-12-19T07:47:22.304186+0000","last_became_active":"2023-12-19T07:47:17.206962+0000","last_became_peered":"2023-12-19T07:47:17.206962+0000","last_unstale":"2023-12-19T07:47:22.304186+0000","last_undegraded":"2023-12-19T07:47:22.304186+0000","last_fullsized":"2023-12-19T07:47:22.304186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807623+0000","last_change":"2023-12-19T07:47:17.217408+0000","last_active":"2023-12-19T07:47:22.807623+0000","last_peered":"2023-12-19T07:47:22.807623+0000","last_clean":"2023-12-19T07:47:22.807623+0000","last_became_active":"2023-12-19T07:47:17.217038+0000","last_became_peered":"2023-12-19T07:47:17.217038+0000","last_unstale":"2023-12-19T07:47:22.807623+0000","last_undegraded":"2023-12-19T07:47:22.807623+0000","last_fullsized":"2023-12-19T07:47:22.807623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:12.732155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805776+0000","last_change":"2023-12-19T07:47:17.194768+0000","last_active":"2023-12-19T07:47:22.805776+0000","last_peered":"2023-12-19T07:47:22.805776+0000","last_clean":"2023-12-19T07:47:22.805776+0000","last_became_active":"2023-12-19T07:47:17.194457+0000","last_became_peered":"2023-12-19T07:47:17.194457+0000","last_unstale":"2023-12-19T07:47:22.805776+0000","last_undegraded":"2023-12-19T07:47:22.805776+0000","last_fullsized":"2023-12-19T07:47:22.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351363+0000","last_change":"2023-12-19T07:47:17.209052+0000","last_active":"2023-12-19T07:47:22.351363+0000","last_peered":"2023-12-19T07:47:22.351363+0000","last_clean":"2023-12-19T07:47:22.351363+0000","last_became_active":"2023-12-19T07:47:17.208671+0000","last_became_peered":"2023-12-19T07:47:17.208671+0000","last_unstale":"2023-12-19T07:47:22.351363+0000","last_undegraded":"2023-12-19T07:47:22.351363+0000","last_fullsized":"2023-12-19T07:47:22.351363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903861+0000","last_change":"2023-12-19T07:47:17.199159+0000","last_active":"2023-12-19T07:47:22.903861+0000","last_peered":"2023-12-19T07:47:22.903861+0000","last_clean":"2023-12-19T07:47:22.903861+0000","last_became_active":"2023-12-19T07:47:17.198548+0000","last_became_peered":"2023-12-19T07:47:17.198548+0000","last_unstale":"2023-12-19T07:47:22.903861+0000","last_undegraded":"2023-12-19T07:47:22.903861+0000","last_fullsized":"2023-12-19T07:47:22.903861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598739+0000","last_change":"2023-12-19T07:47:17.202636+0000","last_active":"2023-12-19T07:47:22.598739+0000","last_peered":"2023-12-19T07:47:22.598739+0000","last_clean":"2023-12-19T07:47:22.598739+0000","last_became_active":"2023-12-19T07:47:17.202331+0000","last_became_peered":"2023-12-19T07:47:17.202331+0000","last_unstale":"2023-12-19T07:47:22.598739+0000","last_undegraded":"2023-12-19T07:47:22.598739+0000","last_fullsized":"2023-12-19T07:47:22.598739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306911+0000","last_change":"2023-12-19T07:47:17.202590+0000","last_active":"2023-12-19T07:47:22.306911+0000","last_peered":"2023-12-19T07:47:22.306911+0000","last_clean":"2023-12-19T07:47:22.306911+0000","last_became_active":"2023-12-19T07:47:17.202206+0000","last_became_peered":"2023-12-19T07:47:17.202206+0000","last_unstale":"2023-12-19T07:47:22.306911+0000","last_undegraded":"2023-12-19T07:47:22.306911+0000","last_fullsized":"2023-12-19T07:47:22.306911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:04.053349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359760+0000","last_change":"2023-12-19T07:47:17.209806+0000","last_active":"2023-12-19T07:47:22.359760+0000","last_peered":"2023-12-19T07:47:22.359760+0000","last_clean":"2023-12-19T07:47:22.359760+0000","last_became_active":"2023-12-19T07:47:17.209424+0000","last_became_peered":"2023-12-19T07:47:17.209424+0000","last_unstale":"2023-12-19T07:47:22.359760+0000","last_undegraded":"2023-12-19T07:47:22.359760+0000","last_fullsized":"2023-12-19T07:47:22.359760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359954+0000","last_change":"2023-12-19T07:47:17.209509+0000","last_active":"2023-12-19T07:47:22.359954+0000","last_peered":"2023-12-19T07:47:22.359954+0000","last_clean":"2023-12-19T07:47:22.359954+0000","last_became_active":"2023-12-19T07:47:17.209026+0000","last_became_peered":"2023-12-19T07:47:17.209026+0000","last_unstale":"2023-12-19T07:47:22.359954+0000","last_undegraded":"2023-12-19T07:47:22.359954+0000","last_fullsized":"2023-12-19T07:47:22.359954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868374+0000","last_change":"2023-12-19T07:47:17.225611+0000","last_active":"2023-12-19T07:47:22.868374+0000","last_peered":"2023-12-19T07:47:22.868374+0000","last_clean":"2023-12-19T07:47:22.868374+0000","last_became_active":"2023-12-19T07:47:17.225305+0000","last_became_peered":"2023-12-19T07:47:17.225305+0000","last_unstale":"2023-12-19T07:47:22.868374+0000","last_undegraded":"2023-12-19T07:47:22.868374+0000","last_fullsized":"2023-12-19T07:47:22.868374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.474113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598130+0000","last_change":"2023-12-19T07:47:17.209463+0000","last_active":"2023-12-19T07:47:22.598130+0000","last_peered":"2023-12-19T07:47:22.598130+0000","last_clean":"2023-12-19T07:47:22.598130+0000","last_became_active":"2023-12-19T07:47:17.201276+0000","last_became_peered":"2023-12-19T07:47:17.201276+0000","last_unstale":"2023-12-19T07:47:22.598130+0000","last_undegraded":"2023-12-19T07:47:22.598130+0000","last_fullsized":"2023-12-19T07:47:22.598130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867603+0000","last_change":"2023-12-19T07:47:17.200808+0000","last_active":"2023-12-19T07:47:22.867603+0000","last_peered":"2023-12-19T07:47:22.867603+0000","last_clean":"2023-12-19T07:47:22.867603+0000","last_became_active":"2023-12-19T07:47:17.200445+0000","last_became_peered":"2023-12-19T07:47:17.200445+0000","last_unstale":"2023-12-19T07:47:22.867603+0000","last_undegraded":"2023-12-19T07:47:22.867603+0000","last_fullsized":"2023-12-19T07:47:22.867603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806351+0000","last_change":"2023-12-19T07:47:17.194801+0000","last_active":"2023-12-19T07:47:22.806351+0000","last_peered":"2023-12-19T07:47:22.806351+0000","last_clean":"2023-12-19T07:47:22.806351+0000","last_became_active":"2023-12-19T07:47:17.194502+0000","last_became_peered":"2023-12-19T07:47:17.194502+0000","last_unstale":"2023-12-19T07:47:22.806351+0000","last_undegraded":"2023-12-19T07:47:22.806351+0000","last_fullsized":"2023-12-19T07:47:22.806351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867343+0000","last_change":"2023-12-19T07:47:17.214803+0000","last_active":"2023-12-19T07:47:22.867343+0000","last_peered":"2023-12-19T07:47:22.867343+0000","last_clean":"2023-12-19T07:47:22.867343+0000","last_became_active":"2023-12-19T07:47:17.214536+0000","last_became_peered":"2023-12-19T07:47:17.214536+0000","last_unstale":"2023-12-19T07:47:22.867343+0000","last_undegraded":"2023-12-19T07:47:22.867343+0000","last_fullsized":"2023-12-19T07:47:22.867343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806760+0000","last_change":"2023-12-19T07:47:17.206649+0000","last_active":"2023-12-19T07:47:22.806760+0000","last_peered":"2023-12-19T07:47:22.806760+0000","last_clean":"2023-12-19T07:47:22.806760+0000","last_became_active":"2023-12-19T07:47:17.206360+0000","last_became_peered":"2023-12-19T07:47:17.206360+0000","last_unstale":"2023-12-19T07:47:22.806760+0000","last_undegraded":"2023-12-19T07:47:22.806760+0000","last_fullsized":"2023-12-19T07:47:22.806760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904329+0000","last_change":"2023-12-19T07:47:17.195685+0000","last_active":"2023-12-19T07:47:22.904329+0000","last_peered":"2023-12-19T07:47:22.904329+0000","last_clean":"2023-12-19T07:47:22.904329+0000","last_became_active":"2023-12-19T07:47:17.195380+0000","last_became_peered":"2023-12-19T07:47:17.195380+0000","last_unstale":"2023-12-19T07:47:22.904329+0000","last_undegraded":"2023-12-19T07:47:22.904329+0000","last_fullsized":"2023-12-19T07:47:22.904329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:47.378624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353591+0000","last_change":"2023-12-19T07:47:17.204379+0000","last_active":"2023-12-19T07:47:22.353591+0000","last_peered":"2023-12-19T07:47:22.353591+0000","last_clean":"2023-12-19T07:47:22.353591+0000","last_became_active":"2023-12-19T07:47:17.204111+0000","last_became_peered":"2023-12-19T07:47:17.204111+0000","last_unstale":"2023-12-19T07:47:22.353591+0000","last_undegraded":"2023-12-19T07:47:22.353591+0000","last_fullsized":"2023-12-19T07:47:22.353591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868221+0000","last_change":"2023-12-19T07:47:17.209821+0000","last_active":"2023-12-19T07:47:22.868221+0000","last_peered":"2023-12-19T07:47:22.868221+0000","last_clean":"2023-12-19T07:47:22.868221+0000","last_became_active":"2023-12-19T07:47:17.209235+0000","last_became_peered":"2023-12-19T07:47:17.209235+0000","last_unstale":"2023-12-19T07:47:22.868221+0000","last_undegraded":"2023-12-19T07:47:22.868221+0000","last_fullsized":"2023-12-19T07:47:22.868221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:23.894121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":864256,"data_stored":697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":270336,"data_stored":106631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":106667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":106541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297556,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074284,"statfs":{"total":96636764160,"available":96332066816,"internally_reserved":0,"allocated":274432,"data_stored":106701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":868352,"data_stored":696508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:47:38.059 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:47:38.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.284+0000 7f466346e700 1 Processor -- start 2023-12-19T07:47:38.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.284+0000 7f466346e700 1 -- start start 2023-12-19T07:47:38.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f466346e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f465c0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f466346e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0c6630 con 0x7f465c1502b0 2023-12-19T07:47:38.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f466346e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0c6910 con 0x7f465c067c90 2023-12-19T07:47:38.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f466346e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c6bf0 con 0x7f465c0cc610 2023-12-19T07:47:38.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4660a09700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f465c1502b0 0x7f465c0c5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56190/0 (socket says 172.21.15.111:56190) 2023-12-19T07:47:38.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4660a09700 1 -- 172.21.15.111:0/2644000358 learned_addr learned my addr 172.21.15.111:0/2644000358 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:38.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4661a0b700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f465c0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036236714 0 0) 0x7f465c0c6630 con 0x7f465c1502b0 2023-12-19T07:47:38.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4648003570 con 0x7f465c1502b0 2023-12-19T07:47:38.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621425351 0 0) 0x7f465c0c6910 con 0x7f465c067c90 2023-12-19T07:47:38.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c0c6630 con 0x7f465c067c90 2023-12-19T07:47:38.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2998791221 0 0) 0x7f4648003570 con 0x7f465c1502b0 2023-12-19T07:47:38.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 msgr2=0x7f465c0c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f465c0c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 >> v1:172.21.15.143:6789/0 conn(0x7f465c067c90 legacy=0x7f465c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0c6ed0 con 0x7f465c1502b0 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f464c001470 con 0x7f465c1502b0 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 >> v1:172.21.15.111:6789/0 conn(0x7f465c1502b0 legacy=0x7f465c0c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4648003c20 0x7f4648004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4648004810 0x7f4648004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f46480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c6bf0 con 0x7f4648004810 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464800bb40 con 0x7f4648003c20 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464800bd00 con 0x7f465c0cc610 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4660a09700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4648004810 0x7f4648004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f466120a700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4648003c20 0x7f4648004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.288+0000 7f4661a0b700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f46480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 -- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 msgr2=0x7f46480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f465c0cc610 0x7f46480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 -- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4648003c20 msgr2=0x7f4648004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4648003c20 0x7f4648004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 -- 172.21.15.111:0/2644000358 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648003570 con 0x7f4648004810 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466120a700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4648003c20 0x7f4648004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4660a09700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4648004810 0x7f4648004c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f464c00c2d0 tx=0x7f464c001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7e22875222140b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c00cbc0 con 0x7f4648004810 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c00cd20 con 0x7f4648004810 2023-12-19T07:47:38.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4653fff700 1 -- 172.21.15.111:0/2644000358 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c005b70 con 0x7f4648004810 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 -- 172.21.15.111:0/2644000358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4648004810 msgr2=0x7f4648004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 --2- 172.21.15.111:0/2644000358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4648004810 0x7f4648004c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f464c00c2d0 tx=0x7f464c001e10 comp rx=0 tx=0).stop 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f4661a0b700 1 -- 172.21.15.111:0/2644000358 reap_dead start 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 -- 172.21.15.111:0/2644000358 shutdown_connections 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 -- 172.21.15.111:0/2644000358 >> 172.21.15.111:0/2644000358 conn(0x7f465c0cd6d0 msgr2=0x7f465c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 -- 172.21.15.111:0/2644000358 shutdown_connections 2023-12-19T07:47:38.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.292+0000 7f466346e700 1 -- 172.21.15.111:0/2644000358 wait complete. 2023-12-19T07:47:38.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 Processor -- start 2023-12-19T07:47:38.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 -- start start 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4648004810 0x7f465c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c0ca1f0 0x7f465c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c83a0 con 0x7f465c169090 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c16a0a0 con 0x7f465c0ca1f0 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f466346e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c169520 con 0x7f4648004810 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.296+0000 7f4660a09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34942/0 (socket says 172.21.15.111:34942) 2023-12-19T07:47:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 -- 172.21.15.111:0/3987321519 learned_addr learned my addr 172.21.15.111:0/3987321519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4661a0b700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c0ca1f0 0x7f465c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f466120a700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4648004810 0x7f465c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 -- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4648004810 msgr2=0x7f465c14b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4648004810 0x7f465c14b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 -- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c0ca1f0 msgr2=0x7f465c0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c0ca1f0 0x7f465c0ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 -- 172.21.15.111:0/3987321519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c169800 con 0x7f465c169090 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f4660a09700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f464800bb10 tx=0x7f464c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c0072e0 con 0x7f465c169090 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c007440 con 0x7f465c169090 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c007800 con 0x7f465c169090 2023-12-19T07:47:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c0c83a0 con 0x7f465c169090 2023-12-19T07:47:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c169dc0 con 0x7f465c169090 2023-12-19T07:47:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.300+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c071120 con 0x7f465c169090 2023-12-19T07:47:38.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.304+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f464c019030 con 0x7f465c169090 2023-12-19T07:47:38.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.304+0000 7f46517fa700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 0x7f4640062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.304+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f464c07e3d0 con 0x7f465c169090 2023-12-19T07:47:38.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.304+0000 7f466120a700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 0x7f4640062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.308+0000 7f466120a700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 0x7f4640062ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.312+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f464c04e760 con 0x7f465c169090 2023-12-19T07:47:38.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.464+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f465c0642a0 con 0x7f4640060620 2023-12-19T07:47:38.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.480+0000 7f46517fa700 1 -- 172.21.15.111:0/3987321519 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345003 (secure 0 0 0) 0x7f465c0642a0 con 0x7f4640060620 2023-12-19T07:47:38.471 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 msgr2=0x7f4640062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 0x7f4640062ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658009040 comp rx=0 tx=0).stop 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 msgr2=0x7f465c0c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f464800bb10 tx=0x7f464c015000 comp rx=0 tx=0).stop 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 shutdown_connections 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4640060620 0x7f4640062ad0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c0ca1f0 0x7f465c0ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f465c169090 0x7f465c0c9cb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 --2- 172.21.15.111:0/3987321519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4648004810 0x7f465c14b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 >> 172.21.15.111:0/3987321519 conn(0x7f465c0cd6d0 msgr2=0x7f465c0cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 shutdown_connections 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.488+0000 7f466346e700 1 -- 172.21.15.111:0/3987321519 wait complete. 2023-12-19T07:47:38.475 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:47:38.491 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-12-19T07:47:36.616305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381604,"kb_used_data":3300,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593116,"statfs":{"total":773094113280,"available":770655350784,"internally_reserved":0,"allocated":3379200,"data_stored":2041019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004304"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904960+0000","last_change":"2023-12-19T07:47:16.213519+0000","last_active":"2023-12-19T07:47:22.904960+0000","last_peered":"2023-12-19T07:47:22.904960+0000","last_clean":"2023-12-19T07:47:22.904960+0000","last_became_active":"2023-12-19T07:47:16.213172+0000","last_became_peered":"2023-12-19T07:47:16.213172+0000","last_unstale":"2023-12-19T07:47:22.904960+0000","last_undegraded":"2023-12-19T07:47:22.904960+0000","last_fullsized":"2023-12-19T07:47:22.904960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599466+0000","last_change":"2023-12-19T07:47:17.206618+0000","last_active":"2023-12-19T07:47:22.599466+0000","last_peered":"2023-12-19T07:47:22.599466+0000","last_clean":"2023-12-19T07:47:22.599466+0000","last_became_active":"2023-12-19T07:47:17.206300+0000","last_became_peered":"2023-12-19T07:47:17.206300+0000","last_unstale":"2023-12-19T07:47:22.599466+0000","last_undegraded":"2023-12-19T07:47:22.599466+0000","last_fullsized":"2023-12-19T07:47:22.599466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597951+0000","last_change":"2023-12-19T07:47:21.847101+0000","last_active":"2023-12-19T07:47:22.597951+0000","last_peered":"2023-12-19T07:47:22.597951+0000","last_clean":"2023-12-19T07:47:22.597951+0000","last_became_active":"2023-12-19T07:47:21.845076+0000","last_became_peered":"2023-12-19T07:47:21.845076+0000","last_unstale":"2023-12-19T07:47:22.597951+0000","last_undegraded":"2023-12-19T07:47:22.597951+0000","last_fullsized":"2023-12-19T07:47:22.597951+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599595+0000","last_change":"2023-12-19T07:47:16.198270+0000","last_active":"2023-12-19T07:47:22.599595+0000","last_peered":"2023-12-19T07:47:22.599595+0000","last_clean":"2023-12-19T07:47:22.599595+0000","last_became_active":"2023-12-19T07:47:16.197562+0000","last_became_peered":"2023-12-19T07:47:16.197562+0000","last_unstale":"2023-12-19T07:47:22.599595+0000","last_undegraded":"2023-12-19T07:47:22.599595+0000","last_fullsized":"2023-12-19T07:47:22.599595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904301+0000","last_change":"2023-12-19T07:47:17.198174+0000","last_active":"2023-12-19T07:47:22.904301+0000","last_peered":"2023-12-19T07:47:22.904301+0000","last_clean":"2023-12-19T07:47:22.904301+0000","last_became_active":"2023-12-19T07:47:17.197876+0000","last_became_peered":"2023-12-19T07:47:17.197876+0000","last_unstale":"2023-12-19T07:47:22.904301+0000","last_undegraded":"2023-12-19T07:47:22.904301+0000","last_fullsized":"2023-12-19T07:47:22.904301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306106+0000","last_change":"2023-12-19T07:47:21.260994+0000","last_active":"2023-12-19T07:47:22.306106+0000","last_peered":"2023-12-19T07:47:22.306106+0000","last_clean":"2023-12-19T07:47:22.306106+0000","last_became_active":"2023-12-19T07:47:21.260122+0000","last_became_peered":"2023-12-19T07:47:21.260122+0000","last_unstale":"2023-12-19T07:47:22.306106+0000","last_undegraded":"2023-12-19T07:47:22.306106+0000","last_fullsized":"2023-12-19T07:47:22.306106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868136+0000","last_change":"2023-12-19T07:47:16.215728+0000","last_active":"2023-12-19T07:47:22.868136+0000","last_peered":"2023-12-19T07:47:22.868136+0000","last_clean":"2023-12-19T07:47:22.868136+0000","last_became_active":"2023-12-19T07:47:16.215434+0000","last_became_peered":"2023-12-19T07:47:16.215434+0000","last_unstale":"2023-12-19T07:47:22.868136+0000","last_undegraded":"2023-12-19T07:47:22.868136+0000","last_fullsized":"2023-12-19T07:47:22.868136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806795+0000","last_change":"2023-12-19T07:47:17.204556+0000","last_active":"2023-12-19T07:47:22.806795+0000","last_peered":"2023-12-19T07:47:22.806795+0000","last_clean":"2023-12-19T07:47:22.806795+0000","last_became_active":"2023-12-19T07:47:17.204225+0000","last_became_peered":"2023-12-19T07:47:17.204225+0000","last_unstale":"2023-12-19T07:47:22.806795+0000","last_undegraded":"2023-12-19T07:47:22.806795+0000","last_fullsized":"2023-12-19T07:47:22.806795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306583+0000","last_change":"2023-12-19T07:47:21.266228+0000","last_active":"2023-12-19T07:47:22.306583+0000","last_peered":"2023-12-19T07:47:22.306583+0000","last_clean":"2023-12-19T07:47:22.306583+0000","last_became_active":"2023-12-19T07:47:21.265305+0000","last_became_peered":"2023-12-19T07:47:21.265305+0000","last_unstale":"2023-12-19T07:47:22.306583+0000","last_undegraded":"2023-12-19T07:47:22.306583+0000","last_fullsized":"2023-12-19T07:47:22.306583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867767+0000","last_change":"2023-12-19T07:47:16.214913+0000","last_active":"2023-12-19T07:47:22.867767+0000","last_peered":"2023-12-19T07:47:22.867767+0000","last_clean":"2023-12-19T07:47:22.867767+0000","last_became_active":"2023-12-19T07:47:16.214514+0000","last_became_peered":"2023-12-19T07:47:16.214514+0000","last_unstale":"2023-12-19T07:47:22.867767+0000","last_undegraded":"2023-12-19T07:47:22.867767+0000","last_fullsized":"2023-12-19T07:47:22.867767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:24.444351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807216+0000","last_change":"2023-12-19T07:47:17.208692+0000","last_active":"2023-12-19T07:47:22.807216+0000","last_peered":"2023-12-19T07:47:22.807216+0000","last_clean":"2023-12-19T07:47:22.807216+0000","last_became_active":"2023-12-19T07:47:17.208320+0000","last_became_peered":"2023-12-19T07:47:17.208320+0000","last_unstale":"2023-12-19T07:47:22.807216+0000","last_undegraded":"2023-12-19T07:47:22.807216+0000","last_fullsized":"2023-12-19T07:47:22.807216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306291+0000","last_change":"2023-12-19T07:47:21.249790+0000","last_active":"2023-12-19T07:47:22.306291+0000","last_peered":"2023-12-19T07:47:22.306291+0000","last_clean":"2023-12-19T07:47:22.306291+0000","last_became_active":"2023-12-19T07:47:21.249266+0000","last_became_peered":"2023-12-19T07:47:21.249266+0000","last_unstale":"2023-12-19T07:47:22.306291+0000","last_undegraded":"2023-12-19T07:47:22.306291+0000","last_fullsized":"2023-12-19T07:47:22.306291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358655+0000","last_change":"2023-12-19T07:47:17.206257+0000","last_active":"2023-12-19T07:47:22.358655+0000","last_peered":"2023-12-19T07:47:22.358655+0000","last_clean":"2023-12-19T07:47:22.358655+0000","last_became_active":"2023-12-19T07:47:17.205737+0000","last_became_peered":"2023-12-19T07:47:17.205737+0000","last_unstale":"2023-12-19T07:47:22.358655+0000","last_undegraded":"2023-12-19T07:47:22.358655+0000","last_fullsized":"2023-12-19T07:47:22.358655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867658+0000","last_change":"2023-12-19T07:47:16.214499+0000","last_active":"2023-12-19T07:47:22.867658+0000","last_peered":"2023-12-19T07:47:22.867658+0000","last_clean":"2023-12-19T07:47:22.867658+0000","last_became_active":"2023-12-19T07:47:16.213693+0000","last_became_peered":"2023-12-19T07:47:16.213693+0000","last_unstale":"2023-12-19T07:47:22.867658+0000","last_undegraded":"2023-12-19T07:47:22.867658+0000","last_fullsized":"2023-12-19T07:47:22.867658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307061+0000","last_change":"2023-12-19T07:47:21.254224+0000","last_active":"2023-12-19T07:47:22.307061+0000","last_peered":"2023-12-19T07:47:22.307061+0000","last_clean":"2023-12-19T07:47:22.307061+0000","last_became_active":"2023-12-19T07:47:21.253786+0000","last_became_peered":"2023-12-19T07:47:21.253786+0000","last_unstale":"2023-12-19T07:47:22.307061+0000","last_undegraded":"2023-12-19T07:47:22.307061+0000","last_fullsized":"2023-12-19T07:47:22.307061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:18:10.044237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598837+0000","last_change":"2023-12-19T07:47:17.205670+0000","last_active":"2023-12-19T07:47:22.598837+0000","last_peered":"2023-12-19T07:47:22.598837+0000","last_clean":"2023-12-19T07:47:22.598837+0000","last_became_active":"2023-12-19T07:47:17.201174+0000","last_became_peered":"2023-12-19T07:47:17.201174+0000","last_unstale":"2023-12-19T07:47:22.598837+0000","last_undegraded":"2023-12-19T07:47:22.598837+0000","last_fullsized":"2023-12-19T07:47:22.598837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352125+0000","last_change":"2023-12-19T07:47:16.196003+0000","last_active":"2023-12-19T07:47:22.352125+0000","last_peered":"2023-12-19T07:47:22.352125+0000","last_clean":"2023-12-19T07:47:22.352125+0000","last_became_active":"2023-12-19T07:47:16.195717+0000","last_became_peered":"2023-12-19T07:47:16.195717+0000","last_unstale":"2023-12-19T07:47:22.352125+0000","last_undegraded":"2023-12-19T07:47:22.352125+0000","last_fullsized":"2023-12-19T07:47:22.352125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306644+0000","last_change":"2023-12-19T07:47:21.259995+0000","last_active":"2023-12-19T07:47:22.306644+0000","last_peered":"2023-12-19T07:47:22.306644+0000","last_clean":"2023-12-19T07:47:22.306644+0000","last_became_active":"2023-12-19T07:47:21.259318+0000","last_became_peered":"2023-12-19T07:47:21.259318+0000","last_unstale":"2023-12-19T07:47:22.306644+0000","last_undegraded":"2023-12-19T07:47:22.306644+0000","last_fullsized":"2023-12-19T07:47:22.306644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598330+0000","last_change":"2023-12-19T07:47:17.205504+0000","last_active":"2023-12-19T07:47:22.598330+0000","last_peered":"2023-12-19T07:47:22.598330+0000","last_clean":"2023-12-19T07:47:22.598330+0000","last_became_active":"2023-12-19T07:47:17.200074+0000","last_became_peered":"2023-12-19T07:47:17.200074+0000","last_unstale":"2023-12-19T07:47:22.598330+0000","last_undegraded":"2023-12-19T07:47:22.598330+0000","last_fullsized":"2023-12-19T07:47:22.598330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351472+0000","last_change":"2023-12-19T07:47:16.206620+0000","last_active":"2023-12-19T07:47:22.351472+0000","last_peered":"2023-12-19T07:47:22.351472+0000","last_clean":"2023-12-19T07:47:22.351472+0000","last_became_active":"2023-12-19T07:47:16.206288+0000","last_became_peered":"2023-12-19T07:47:16.206288+0000","last_unstale":"2023-12-19T07:47:22.351472+0000","last_undegraded":"2023-12-19T07:47:22.351472+0000","last_fullsized":"2023-12-19T07:47:22.351472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351144+0000","last_change":"2023-12-19T07:47:21.841279+0000","last_active":"2023-12-19T07:47:22.351144+0000","last_peered":"2023-12-19T07:47:22.351144+0000","last_clean":"2023-12-19T07:47:22.351144+0000","last_became_active":"2023-12-19T07:47:21.840184+0000","last_became_peered":"2023-12-19T07:47:21.840184+0000","last_unstale":"2023-12-19T07:47:22.351144+0000","last_undegraded":"2023-12-19T07:47:22.351144+0000","last_fullsized":"2023-12-19T07:47:22.351144+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:22:46.085295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306581+0000","last_change":"2023-12-19T07:47:17.199136+0000","last_active":"2023-12-19T07:47:22.306581+0000","last_peered":"2023-12-19T07:47:22.306581+0000","last_clean":"2023-12-19T07:47:22.306581+0000","last_became_active":"2023-12-19T07:47:17.198802+0000","last_became_peered":"2023-12-19T07:47:17.198802+0000","last_unstale":"2023-12-19T07:47:22.306581+0000","last_undegraded":"2023-12-19T07:47:22.306581+0000","last_fullsized":"2023-12-19T07:47:22.306581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:35.655311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866892+0000","last_change":"2023-12-19T07:47:16.213635+0000","last_active":"2023-12-19T07:47:22.866892+0000","last_peered":"2023-12-19T07:47:22.866892+0000","last_clean":"2023-12-19T07:47:22.866892+0000","last_became_active":"2023-12-19T07:47:16.213201+0000","last_became_peered":"2023-12-19T07:47:16.213201+0000","last_unstale":"2023-12-19T07:47:22.866892+0000","last_undegraded":"2023-12-19T07:47:22.866892+0000","last_fullsized":"2023-12-19T07:47:22.866892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:06.365757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306787+0000","last_change":"2023-12-19T07:47:21.265928+0000","last_active":"2023-12-19T07:47:22.306787+0000","last_peered":"2023-12-19T07:47:22.306787+0000","last_clean":"2023-12-19T07:47:22.306787+0000","last_became_active":"2023-12-19T07:47:21.265360+0000","last_became_peered":"2023-12-19T07:47:21.265360+0000","last_unstale":"2023-12-19T07:47:22.306787+0000","last_undegraded":"2023-12-19T07:47:22.306787+0000","last_fullsized":"2023-12-19T07:47:22.306787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904688+0000","last_change":"2023-12-19T07:47:17.195176+0000","last_active":"2023-12-19T07:47:22.904688+0000","last_peered":"2023-12-19T07:47:22.904688+0000","last_clean":"2023-12-19T07:47:22.904688+0000","last_became_active":"2023-12-19T07:47:17.194607+0000","last_became_peered":"2023-12-19T07:47:17.194607+0000","last_unstale":"2023-12-19T07:47:22.904688+0000","last_undegraded":"2023-12-19T07:47:22.904688+0000","last_fullsized":"2023-12-19T07:47:22.904688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359926+0000","last_change":"2023-12-19T07:47:16.206513+0000","last_active":"2023-12-19T07:47:22.359926+0000","last_peered":"2023-12-19T07:47:22.359926+0000","last_clean":"2023-12-19T07:47:22.359926+0000","last_became_active":"2023-12-19T07:47:16.206148+0000","last_became_peered":"2023-12-19T07:47:16.206148+0000","last_unstale":"2023-12-19T07:47:22.359926+0000","last_undegraded":"2023-12-19T07:47:22.359926+0000","last_fullsized":"2023-12-19T07:47:22.359926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.305902+0000","last_change":"2023-12-19T07:47:21.841183+0000","last_active":"2023-12-19T07:47:22.305902+0000","last_peered":"2023-12-19T07:47:22.305902+0000","last_clean":"2023-12-19T07:47:22.305902+0000","last_became_active":"2023-12-19T07:47:21.839559+0000","last_became_peered":"2023-12-19T07:47:21.839559+0000","last_unstale":"2023-12-19T07:47:22.305902+0000","last_undegraded":"2023-12-19T07:47:22.305902+0000","last_fullsized":"2023-12-19T07:47:22.305902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:27.954678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599577+0000","last_change":"2023-12-19T07:47:17.208713+0000","last_active":"2023-12-19T07:47:22.599577+0000","last_peered":"2023-12-19T07:47:22.599577+0000","last_clean":"2023-12-19T07:47:22.599577+0000","last_became_active":"2023-12-19T07:47:17.202555+0000","last_became_peered":"2023-12-19T07:47:17.202555+0000","last_unstale":"2023-12-19T07:47:22.599577+0000","last_undegraded":"2023-12-19T07:47:22.599577+0000","last_fullsized":"2023-12-19T07:47:22.599577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.356659+0000","last_change":"2023-12-19T07:47:16.198423+0000","last_active":"2023-12-19T07:47:22.356659+0000","last_peered":"2023-12-19T07:47:22.356659+0000","last_clean":"2023-12-19T07:47:22.356659+0000","last_became_active":"2023-12-19T07:47:16.198088+0000","last_became_peered":"2023-12-19T07:47:16.198088+0000","last_unstale":"2023-12-19T07:47:22.356659+0000","last_undegraded":"2023-12-19T07:47:22.356659+0000","last_fullsized":"2023-12-19T07:47:22.356659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:46.365232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597749+0000","last_change":"2023-12-19T07:47:21.845056+0000","last_active":"2023-12-19T07:47:22.597749+0000","last_peered":"2023-12-19T07:47:22.597749+0000","last_clean":"2023-12-19T07:47:22.597749+0000","last_became_active":"2023-12-19T07:47:21.844217+0000","last_became_peered":"2023-12-19T07:47:21.844217+0000","last_unstale":"2023-12-19T07:47:22.597749+0000","last_undegraded":"2023-12-19T07:47:22.597749+0000","last_fullsized":"2023-12-19T07:47:22.597749+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:53:07.229783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352565+0000","last_change":"2023-12-19T07:47:17.200525+0000","last_active":"2023-12-19T07:47:22.352565+0000","last_peered":"2023-12-19T07:47:22.352565+0000","last_clean":"2023-12-19T07:47:22.352565+0000","last_became_active":"2023-12-19T07:47:17.199921+0000","last_became_peered":"2023-12-19T07:47:17.199921+0000","last_unstale":"2023-12-19T07:47:22.352565+0000","last_undegraded":"2023-12-19T07:47:22.352565+0000","last_fullsized":"2023-12-19T07:47:22.352565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868254+0000","last_change":"2023-12-19T07:47:16.213616+0000","last_active":"2023-12-19T07:47:22.868254+0000","last_peered":"2023-12-19T07:47:22.868254+0000","last_clean":"2023-12-19T07:47:22.868254+0000","last_became_active":"2023-12-19T07:47:16.213264+0000","last_became_peered":"2023-12-19T07:47:16.213264+0000","last_unstale":"2023-12-19T07:47:22.868254+0000","last_undegraded":"2023-12-19T07:47:22.868254+0000","last_fullsized":"2023-12-19T07:47:22.868254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902861+0000","last_change":"2023-12-19T07:47:21.841016+0000","last_active":"2023-12-19T07:47:22.902861+0000","last_peered":"2023-12-19T07:47:22.902861+0000","last_clean":"2023-12-19T07:47:22.902861+0000","last_became_active":"2023-12-19T07:47:21.840031+0000","last_became_peered":"2023-12-19T07:47:21.840031+0000","last_unstale":"2023-12-19T07:47:22.902861+0000","last_undegraded":"2023-12-19T07:47:22.902861+0000","last_fullsized":"2023-12-19T07:47:22.902861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807240+0000","last_change":"2023-12-19T07:47:17.205604+0000","last_active":"2023-12-19T07:47:22.807240+0000","last_peered":"2023-12-19T07:47:22.807240+0000","last_clean":"2023-12-19T07:47:22.807240+0000","last_became_active":"2023-12-19T07:47:17.205354+0000","last_became_peered":"2023-12-19T07:47:17.205354+0000","last_unstale":"2023-12-19T07:47:22.807240+0000","last_undegraded":"2023-12-19T07:47:22.807240+0000","last_fullsized":"2023-12-19T07:47:22.807240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:08.287410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352620+0000","last_change":"2023-12-19T07:47:16.201653+0000","last_active":"2023-12-19T07:47:22.352620+0000","last_peered":"2023-12-19T07:47:22.352620+0000","last_clean":"2023-12-19T07:47:22.352620+0000","last_became_active":"2023-12-19T07:47:16.201071+0000","last_became_peered":"2023-12-19T07:47:16.201071+0000","last_unstale":"2023-12-19T07:47:22.352620+0000","last_undegraded":"2023-12-19T07:47:22.352620+0000","last_fullsized":"2023-12-19T07:47:22.352620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.226098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351616+0000","last_change":"2023-12-19T07:47:21.262464+0000","last_active":"2023-12-19T07:47:22.351616+0000","last_peered":"2023-12-19T07:47:22.351616+0000","last_clean":"2023-12-19T07:47:22.351616+0000","last_became_active":"2023-12-19T07:47:21.259084+0000","last_became_peered":"2023-12-19T07:47:21.259084+0000","last_unstale":"2023-12-19T07:47:22.351616+0000","last_undegraded":"2023-12-19T07:47:22.351616+0000","last_fullsized":"2023-12-19T07:47:22.351616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307178+0000","last_change":"2023-12-19T07:47:17.212640+0000","last_active":"2023-12-19T07:47:22.307178+0000","last_peered":"2023-12-19T07:47:22.307178+0000","last_clean":"2023-12-19T07:47:22.307178+0000","last_became_active":"2023-12-19T07:47:17.212189+0000","last_became_peered":"2023-12-19T07:47:17.212189+0000","last_unstale":"2023-12-19T07:47:22.307178+0000","last_undegraded":"2023-12-19T07:47:22.307178+0000","last_fullsized":"2023-12-19T07:47:22.307178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806950+0000","last_change":"2023-12-19T07:47:16.201470+0000","last_active":"2023-12-19T07:47:22.806950+0000","last_peered":"2023-12-19T07:47:22.806950+0000","last_clean":"2023-12-19T07:47:22.806950+0000","last_became_active":"2023-12-19T07:47:16.200990+0000","last_became_peered":"2023-12-19T07:47:16.200990+0000","last_unstale":"2023-12-19T07:47:22.806950+0000","last_undegraded":"2023-12-19T07:47:22.806950+0000","last_fullsized":"2023-12-19T07:47:22.806950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352494+0000","last_change":"2023-12-19T07:47:21.261403+0000","last_active":"2023-12-19T07:47:22.352494+0000","last_peered":"2023-12-19T07:47:22.352494+0000","last_clean":"2023-12-19T07:47:22.352494+0000","last_became_active":"2023-12-19T07:47:21.260934+0000","last_became_peered":"2023-12-19T07:47:21.260934+0000","last_unstale":"2023-12-19T07:47:22.352494+0000","last_undegraded":"2023-12-19T07:47:22.352494+0000","last_fullsized":"2023-12-19T07:47:22.352494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:24:35.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351362+0000","last_change":"2023-12-19T07:47:17.196604+0000","last_active":"2023-12-19T07:47:22.351362+0000","last_peered":"2023-12-19T07:47:22.351362+0000","last_clean":"2023-12-19T07:47:22.351362+0000","last_became_active":"2023-12-19T07:47:17.195716+0000","last_became_peered":"2023-12-19T07:47:17.195716+0000","last_unstale":"2023-12-19T07:47:22.351362+0000","last_undegraded":"2023-12-19T07:47:22.351362+0000","last_fullsized":"2023-12-19T07:47:22.351362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302998+0000","last_change":"2023-12-19T07:47:16.188258+0000","last_active":"2023-12-19T07:47:22.302998+0000","last_peered":"2023-12-19T07:47:22.302998+0000","last_clean":"2023-12-19T07:47:22.302998+0000","last_became_active":"2023-12-19T07:47:16.187018+0000","last_became_peered":"2023-12-19T07:47:16.187018+0000","last_unstale":"2023-12-19T07:47:22.302998+0000","last_undegraded":"2023-12-19T07:47:22.302998+0000","last_fullsized":"2023-12-19T07:47:22.302998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353576+0000","last_change":"2023-12-19T07:47:21.269038+0000","last_active":"2023-12-19T07:47:22.353576+0000","last_peered":"2023-12-19T07:47:22.353576+0000","last_clean":"2023-12-19T07:47:22.353576+0000","last_became_active":"2023-12-19T07:47:21.268433+0000","last_became_peered":"2023-12-19T07:47:21.268433+0000","last_unstale":"2023-12-19T07:47:22.353576+0000","last_undegraded":"2023-12-19T07:47:22.353576+0000","last_fullsized":"2023-12-19T07:47:22.353576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:30:43.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903117+0000","last_change":"2023-12-19T07:47:17.201741+0000","last_active":"2023-12-19T07:47:22.903117+0000","last_peered":"2023-12-19T07:47:22.903117+0000","last_clean":"2023-12-19T07:47:22.903117+0000","last_became_active":"2023-12-19T07:47:17.201369+0000","last_became_peered":"2023-12-19T07:47:17.201369+0000","last_unstale":"2023-12-19T07:47:22.903117+0000","last_undegraded":"2023-12-19T07:47:22.903117+0000","last_fullsized":"2023-12-19T07:47:22.903117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.866966+0000","last_change":"2023-12-19T07:47:16.213829+0000","last_active":"2023-12-19T07:47:22.866966+0000","last_peered":"2023-12-19T07:47:22.866966+0000","last_clean":"2023-12-19T07:47:22.866966+0000","last_became_active":"2023-12-19T07:47:16.213228+0000","last_became_peered":"2023-12-19T07:47:16.213228+0000","last_unstale":"2023-12-19T07:47:22.866966+0000","last_undegraded":"2023-12-19T07:47:22.866966+0000","last_fullsized":"2023-12-19T07:47:22.866966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:31.758147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902852+0000","last_change":"2023-12-19T07:47:21.841516+0000","last_active":"2023-12-19T07:47:22.902852+0000","last_peered":"2023-12-19T07:47:22.902852+0000","last_clean":"2023-12-19T07:47:22.902852+0000","last_became_active":"2023-12-19T07:47:21.840477+0000","last_became_peered":"2023-12-19T07:47:21.840477+0000","last_unstale":"2023-12-19T07:47:22.902852+0000","last_undegraded":"2023-12-19T07:47:22.902852+0000","last_fullsized":"2023-12-19T07:47:22.902852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:56:24.723395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598774+0000","last_change":"2023-12-19T07:47:17.200992+0000","last_active":"2023-12-19T07:47:22.598774+0000","last_peered":"2023-12-19T07:47:22.598774+0000","last_clean":"2023-12-19T07:47:22.598774+0000","last_became_active":"2023-12-19T07:47:17.199427+0000","last_became_peered":"2023-12-19T07:47:17.199427+0000","last_unstale":"2023-12-19T07:47:22.598774+0000","last_undegraded":"2023-12-19T07:47:22.598774+0000","last_fullsized":"2023-12-19T07:47:22.598774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806565+0000","last_change":"2023-12-19T07:47:16.201062+0000","last_active":"2023-12-19T07:47:22.806565+0000","last_peered":"2023-12-19T07:47:22.806565+0000","last_clean":"2023-12-19T07:47:22.806565+0000","last_became_active":"2023-12-19T07:47:16.199674+0000","last_became_peered":"2023-12-19T07:47:16.199674+0000","last_unstale":"2023-12-19T07:47:22.806565+0000","last_undegraded":"2023-12-19T07:47:22.806565+0000","last_fullsized":"2023-12-19T07:47:22.806565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:33.372367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867080+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:47:22.867080+0000","last_peered":"2023-12-19T07:47:22.867080+0000","last_clean":"2023-12-19T07:47:22.867080+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:47:22.867080+0000","last_undegraded":"2023-12-19T07:47:22.867080+0000","last_fullsized":"2023-12-19T07:47:22.867080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359462+0000","last_change":"2023-12-19T07:47:21.844050+0000","last_active":"2023-12-19T07:47:22.359462+0000","last_peered":"2023-12-19T07:47:22.359462+0000","last_clean":"2023-12-19T07:47:22.359462+0000","last_became_active":"2023-12-19T07:47:21.843480+0000","last_became_peered":"2023-12-19T07:47:21.843480+0000","last_unstale":"2023-12-19T07:47:22.359462+0000","last_undegraded":"2023-12-19T07:47:22.359462+0000","last_fullsized":"2023-12-19T07:47:22.359462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:29.237260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867932+0000","last_change":"2023-12-19T07:47:17.201683+0000","last_active":"2023-12-19T07:47:22.867932+0000","last_peered":"2023-12-19T07:47:22.867932+0000","last_clean":"2023-12-19T07:47:22.867932+0000","last_became_active":"2023-12-19T07:47:17.201138+0000","last_became_peered":"2023-12-19T07:47:17.201138+0000","last_unstale":"2023-12-19T07:47:22.867932+0000","last_undegraded":"2023-12-19T07:47:22.867932+0000","last_fullsized":"2023-12-19T07:47:22.867932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:19:25.176702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303850+0000","last_change":"2023-12-19T07:47:16.199670+0000","last_active":"2023-12-19T07:47:22.303850+0000","last_peered":"2023-12-19T07:47:22.303850+0000","last_clean":"2023-12-19T07:47:22.303850+0000","last_became_active":"2023-12-19T07:47:16.199236+0000","last_became_peered":"2023-12-19T07:47:16.199236+0000","last_unstale":"2023-12-19T07:47:22.303850+0000","last_undegraded":"2023-12-19T07:47:22.303850+0000","last_fullsized":"2023-12-19T07:47:22.303850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358653+0000","last_change":"2023-12-19T07:47:21.844306+0000","last_active":"2023-12-19T07:47:22.358653+0000","last_peered":"2023-12-19T07:47:22.358653+0000","last_clean":"2023-12-19T07:47:22.358653+0000","last_became_active":"2023-12-19T07:47:21.843694+0000","last_became_peered":"2023-12-19T07:47:21.843694+0000","last_unstale":"2023-12-19T07:47:22.358653+0000","last_undegraded":"2023-12-19T07:47:22.358653+0000","last_fullsized":"2023-12-19T07:47:22.358653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:25.534670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806965+0000","last_change":"2023-12-19T07:47:17.210076+0000","last_active":"2023-12-19T07:47:22.806965+0000","last_peered":"2023-12-19T07:47:22.806965+0000","last_clean":"2023-12-19T07:47:22.806965+0000","last_became_active":"2023-12-19T07:47:17.209871+0000","last_became_peered":"2023-12-19T07:47:17.209871+0000","last_unstale":"2023-12-19T07:47:22.806965+0000","last_undegraded":"2023-12-19T07:47:22.806965+0000","last_fullsized":"2023-12-19T07:47:22.806965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353351+0000","last_change":"2023-12-19T07:47:16.187382+0000","last_active":"2023-12-19T07:47:22.353351+0000","last_peered":"2023-12-19T07:47:22.353351+0000","last_clean":"2023-12-19T07:47:22.353351+0000","last_became_active":"2023-12-19T07:47:16.187021+0000","last_became_peered":"2023-12-19T07:47:16.187021+0000","last_unstale":"2023-12-19T07:47:22.353351+0000","last_undegraded":"2023-12-19T07:47:22.353351+0000","last_fullsized":"2023-12-19T07:47:22.353351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:03.039957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903413+0000","last_change":"2023-12-19T07:47:21.265329+0000","last_active":"2023-12-19T07:47:22.903413+0000","last_peered":"2023-12-19T07:47:22.903413+0000","last_clean":"2023-12-19T07:47:22.903413+0000","last_became_active":"2023-12-19T07:47:21.264912+0000","last_became_peered":"2023-12-19T07:47:21.264912+0000","last_unstale":"2023-12-19T07:47:22.903413+0000","last_undegraded":"2023-12-19T07:47:22.903413+0000","last_fullsized":"2023-12-19T07:47:22.903413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868427+0000","last_change":"2023-12-19T07:47:17.215646+0000","last_active":"2023-12-19T07:47:22.868427+0000","last_peered":"2023-12-19T07:47:22.868427+0000","last_clean":"2023-12-19T07:47:22.868427+0000","last_became_active":"2023-12-19T07:47:17.215345+0000","last_became_peered":"2023-12-19T07:47:17.215345+0000","last_unstale":"2023-12-19T07:47:22.868427+0000","last_undegraded":"2023-12-19T07:47:22.868427+0000","last_fullsized":"2023-12-19T07:47:22.868427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:05.660865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304403+0000","last_change":"2023-12-19T07:47:16.195752+0000","last_active":"2023-12-19T07:47:22.304403+0000","last_peered":"2023-12-19T07:47:22.304403+0000","last_clean":"2023-12-19T07:47:22.304403+0000","last_became_active":"2023-12-19T07:47:16.195415+0000","last_became_peered":"2023-12-19T07:47:16.195415+0000","last_unstale":"2023-12-19T07:47:22.304403+0000","last_undegraded":"2023-12-19T07:47:22.304403+0000","last_fullsized":"2023-12-19T07:47:22.304403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805409+0000","last_change":"2023-12-19T07:47:21.840195+0000","last_active":"2023-12-19T07:47:22.805409+0000","last_peered":"2023-12-19T07:47:22.805409+0000","last_clean":"2023-12-19T07:47:22.805409+0000","last_became_active":"2023-12-19T07:47:21.839557+0000","last_became_peered":"2023-12-19T07:47:21.839557+0000","last_unstale":"2023-12-19T07:47:22.805409+0000","last_undegraded":"2023-12-19T07:47:22.805409+0000","last_fullsized":"2023-12-19T07:47:22.805409+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867970+0000","last_change":"2023-12-19T07:47:17.211241+0000","last_active":"2023-12-19T07:47:22.867970+0000","last_peered":"2023-12-19T07:47:22.867970+0000","last_clean":"2023-12-19T07:47:22.867970+0000","last_became_active":"2023-12-19T07:47:17.210508+0000","last_became_peered":"2023-12-19T07:47:17.210508+0000","last_unstale":"2023-12-19T07:47:22.867970+0000","last_undegraded":"2023-12-19T07:47:22.867970+0000","last_fullsized":"2023-12-19T07:47:22.867970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:48.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307381+0000","last_change":"2023-12-19T07:47:16.198483+0000","last_active":"2023-12-19T07:47:22.307381+0000","last_peered":"2023-12-19T07:47:22.307381+0000","last_clean":"2023-12-19T07:47:22.307381+0000","last_became_active":"2023-12-19T07:47:16.198150+0000","last_became_peered":"2023-12-19T07:47:16.198150+0000","last_unstale":"2023-12-19T07:47:22.307381+0000","last_undegraded":"2023-12-19T07:47:22.307381+0000","last_fullsized":"2023-12-19T07:47:22.307381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306109+0000","last_change":"2023-12-19T07:47:21.259856+0000","last_active":"2023-12-19T07:47:22.306109+0000","last_peered":"2023-12-19T07:47:22.306109+0000","last_clean":"2023-12-19T07:47:22.306109+0000","last_became_active":"2023-12-19T07:47:21.259147+0000","last_became_peered":"2023-12-19T07:47:21.259147+0000","last_unstale":"2023-12-19T07:47:22.306109+0000","last_undegraded":"2023-12-19T07:47:22.306109+0000","last_fullsized":"2023-12-19T07:47:22.306109+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:10:18.560125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867790+0000","last_change":"2023-12-19T07:47:17.209489+0000","last_active":"2023-12-19T07:47:22.867790+0000","last_peered":"2023-12-19T07:47:22.867790+0000","last_clean":"2023-12-19T07:47:22.867790+0000","last_became_active":"2023-12-19T07:47:17.209131+0000","last_became_peered":"2023-12-19T07:47:17.209131+0000","last_unstale":"2023-12-19T07:47:22.867790+0000","last_undegraded":"2023-12-19T07:47:22.867790+0000","last_fullsized":"2023-12-19T07:47:22.867790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303258+0000","last_change":"2023-12-19T07:47:16.192274+0000","last_active":"2023-12-19T07:47:22.303258+0000","last_peered":"2023-12-19T07:47:22.303258+0000","last_clean":"2023-12-19T07:47:22.303258+0000","last_became_active":"2023-12-19T07:47:16.190777+0000","last_became_peered":"2023-12-19T07:47:16.190777+0000","last_unstale":"2023-12-19T07:47:22.303258+0000","last_undegraded":"2023-12-19T07:47:22.303258+0000","last_fullsized":"2023-12-19T07:47:22.303258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904035+0000","last_change":"2023-12-19T07:47:21.260731+0000","last_active":"2023-12-19T07:47:22.904035+0000","last_peered":"2023-12-19T07:47:22.904035+0000","last_clean":"2023-12-19T07:47:22.904035+0000","last_became_active":"2023-12-19T07:47:21.258011+0000","last_became_peered":"2023-12-19T07:47:21.258011+0000","last_unstale":"2023-12-19T07:47:22.904035+0000","last_undegraded":"2023-12-19T07:47:22.904035+0000","last_fullsized":"2023-12-19T07:47:22.904035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359125+0000","last_change":"2023-12-19T07:47:17.201434+0000","last_active":"2023-12-19T07:47:22.359125+0000","last_peered":"2023-12-19T07:47:22.359125+0000","last_clean":"2023-12-19T07:47:22.359125+0000","last_became_active":"2023-12-19T07:47:17.200473+0000","last_became_peered":"2023-12-19T07:47:17.200473+0000","last_unstale":"2023-12-19T07:47:22.359125+0000","last_undegraded":"2023-12-19T07:47:22.359125+0000","last_fullsized":"2023-12-19T07:47:22.359125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359065+0000","last_change":"2023-12-19T07:47:16.193445+0000","last_active":"2023-12-19T07:47:22.359065+0000","last_peered":"2023-12-19T07:47:22.359065+0000","last_clean":"2023-12-19T07:47:22.359065+0000","last_became_active":"2023-12-19T07:47:16.192600+0000","last_became_peered":"2023-12-19T07:47:16.192600+0000","last_unstale":"2023-12-19T07:47:22.359065+0000","last_undegraded":"2023-12-19T07:47:22.359065+0000","last_fullsized":"2023-12-19T07:47:22.359065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597903+0000","last_change":"2023-12-19T07:47:21.847662+0000","last_active":"2023-12-19T07:47:22.597903+0000","last_peered":"2023-12-19T07:47:22.597903+0000","last_clean":"2023-12-19T07:47:22.597903+0000","last_became_active":"2023-12-19T07:47:21.847110+0000","last_became_peered":"2023-12-19T07:47:21.847110+0000","last_unstale":"2023-12-19T07:47:22.597903+0000","last_undegraded":"2023-12-19T07:47:22.597903+0000","last_fullsized":"2023-12-19T07:47:22.597903+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:07:23.838326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867579+0000","last_change":"2023-12-19T07:47:17.219151+0000","last_active":"2023-12-19T07:47:22.867579+0000","last_peered":"2023-12-19T07:47:22.867579+0000","last_clean":"2023-12-19T07:47:22.867579+0000","last_became_active":"2023-12-19T07:47:17.218940+0000","last_became_peered":"2023-12-19T07:47:17.218940+0000","last_unstale":"2023-12-19T07:47:22.867579+0000","last_undegraded":"2023-12-19T07:47:22.867579+0000","last_fullsized":"2023-12-19T07:47:22.867579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:03.004488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806721+0000","last_change":"2023-12-19T07:47:16.201106+0000","last_active":"2023-12-19T07:47:22.806721+0000","last_peered":"2023-12-19T07:47:22.806721+0000","last_clean":"2023-12-19T07:47:22.806721+0000","last_became_active":"2023-12-19T07:47:16.200628+0000","last_became_peered":"2023-12-19T07:47:16.200628+0000","last_unstale":"2023-12-19T07:47:22.806721+0000","last_undegraded":"2023-12-19T07:47:22.806721+0000","last_fullsized":"2023-12-19T07:47:22.806721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597986+0000","last_change":"2023-12-19T07:47:21.847256+0000","last_active":"2023-12-19T07:47:22.597986+0000","last_peered":"2023-12-19T07:47:22.597986+0000","last_clean":"2023-12-19T07:47:22.597986+0000","last_became_active":"2023-12-19T07:47:21.843240+0000","last_became_peered":"2023-12-19T07:47:21.843240+0000","last_unstale":"2023-12-19T07:47:22.597986+0000","last_undegraded":"2023-12-19T07:47:22.597986+0000","last_fullsized":"2023-12-19T07:47:22.597986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:45.450328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903865+0000","last_change":"2023-12-19T07:47:17.200704+0000","last_active":"2023-12-19T07:47:22.903865+0000","last_peered":"2023-12-19T07:47:22.903865+0000","last_clean":"2023-12-19T07:47:22.903865+0000","last_became_active":"2023-12-19T07:47:17.200382+0000","last_became_peered":"2023-12-19T07:47:17.200382+0000","last_unstale":"2023-12-19T07:47:22.903865+0000","last_undegraded":"2023-12-19T07:47:22.903865+0000","last_fullsized":"2023-12-19T07:47:22.903865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903656+0000","last_change":"2023-12-19T07:47:16.189068+0000","last_active":"2023-12-19T07:47:22.903656+0000","last_peered":"2023-12-19T07:47:22.903656+0000","last_clean":"2023-12-19T07:47:22.903656+0000","last_became_active":"2023-12-19T07:47:16.188798+0000","last_became_peered":"2023-12-19T07:47:16.188798+0000","last_unstale":"2023-12-19T07:47:22.903656+0000","last_undegraded":"2023-12-19T07:47:22.903656+0000","last_fullsized":"2023-12-19T07:47:22.903656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805530+0000","last_change":"2023-12-19T07:47:21.847444+0000","last_active":"2023-12-19T07:47:22.805530+0000","last_peered":"2023-12-19T07:47:22.805530+0000","last_clean":"2023-12-19T07:47:22.805530+0000","last_became_active":"2023-12-19T07:47:21.847123+0000","last_became_peered":"2023-12-19T07:47:21.847123+0000","last_unstale":"2023-12-19T07:47:22.805530+0000","last_undegraded":"2023-12-19T07:47:22.805530+0000","last_fullsized":"2023-12-19T07:47:22.805530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:43.062440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867681+0000","last_change":"2023-12-19T07:47:17.212222+0000","last_active":"2023-12-19T07:47:22.867681+0000","last_peered":"2023-12-19T07:47:22.867681+0000","last_clean":"2023-12-19T07:47:22.867681+0000","last_became_active":"2023-12-19T07:47:17.211536+0000","last_became_peered":"2023-12-19T07:47:17.211536+0000","last_unstale":"2023-12-19T07:47:22.867681+0000","last_undegraded":"2023-12-19T07:47:22.867681+0000","last_fullsized":"2023-12-19T07:47:22.867681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:40.976961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-12-19T07:47:29.088364+0000","last_change":"2023-12-19T07:47:16.200498+0000","last_active":"2023-12-19T07:47:29.088364+0000","last_peered":"2023-12-19T07:47:29.088364+0000","last_clean":"2023-12-19T07:47:29.088364+0000","last_became_active":"2023-12-19T07:47:16.200159+0000","last_became_peered":"2023-12-19T07:47:16.200159+0000","last_unstale":"2023-12-19T07:47:29.088364+0000","last_undegraded":"2023-12-19T07:47:29.088364+0000","last_fullsized":"2023-12-19T07:47:29.088364+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:11:00.738860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351798+0000","last_change":"2023-12-19T07:47:21.269837+0000","last_active":"2023-12-19T07:47:22.351798+0000","last_peered":"2023-12-19T07:47:22.351798+0000","last_clean":"2023-12-19T07:47:22.351798+0000","last_became_active":"2023-12-19T07:47:21.269371+0000","last_became_peered":"2023-12-19T07:47:21.269371+0000","last_unstale":"2023-12-19T07:47:22.351798+0000","last_undegraded":"2023-12-19T07:47:22.351798+0000","last_fullsized":"2023-12-19T07:47:22.351798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:50.421948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307009+0000","last_change":"2023-12-19T07:47:17.206123+0000","last_active":"2023-12-19T07:47:22.307009+0000","last_peered":"2023-12-19T07:47:22.307009+0000","last_clean":"2023-12-19T07:47:22.307009+0000","last_became_active":"2023-12-19T07:47:17.202604+0000","last_became_peered":"2023-12-19T07:47:17.202604+0000","last_unstale":"2023-12-19T07:47:22.307009+0000","last_undegraded":"2023-12-19T07:47:22.307009+0000","last_fullsized":"2023-12-19T07:47:22.307009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353492+0000","last_change":"2023-12-19T07:47:16.207087+0000","last_active":"2023-12-19T07:47:22.353492+0000","last_peered":"2023-12-19T07:47:22.353492+0000","last_clean":"2023-12-19T07:47:22.353492+0000","last_became_active":"2023-12-19T07:47:16.205832+0000","last_became_peered":"2023-12-19T07:47:16.205832+0000","last_unstale":"2023-12-19T07:47:22.353492+0000","last_undegraded":"2023-12-19T07:47:22.353492+0000","last_fullsized":"2023-12-19T07:47:22.353492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:20.894898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.307124+0000","last_change":"2023-12-19T07:47:21.259755+0000","last_active":"2023-12-19T07:47:22.307124+0000","last_peered":"2023-12-19T07:47:22.307124+0000","last_clean":"2023-12-19T07:47:22.307124+0000","last_became_active":"2023-12-19T07:47:21.259163+0000","last_became_peered":"2023-12-19T07:47:21.259163+0000","last_unstale":"2023-12-19T07:47:22.307124+0000","last_undegraded":"2023-12-19T07:47:22.307124+0000","last_fullsized":"2023-12-19T07:47:22.307124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:33.797470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904807+0000","last_change":"2023-12-19T07:47:17.202501+0000","last_active":"2023-12-19T07:47:22.904807+0000","last_peered":"2023-12-19T07:47:22.904807+0000","last_clean":"2023-12-19T07:47:22.904807+0000","last_became_active":"2023-12-19T07:47:17.199344+0000","last_became_peered":"2023-12-19T07:47:17.199344+0000","last_unstale":"2023-12-19T07:47:22.904807+0000","last_undegraded":"2023-12-19T07:47:22.904807+0000","last_fullsized":"2023-12-19T07:47:22.904807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807353+0000","last_change":"2023-12-19T07:47:16.200080+0000","last_active":"2023-12-19T07:47:22.807353+0000","last_peered":"2023-12-19T07:47:22.807353+0000","last_clean":"2023-12-19T07:47:22.807353+0000","last_became_active":"2023-12-19T07:47:16.199182+0000","last_became_peered":"2023-12-19T07:47:16.199182+0000","last_unstale":"2023-12-19T07:47:22.807353+0000","last_undegraded":"2023-12-19T07:47:22.807353+0000","last_fullsized":"2023-12-19T07:47:22.807353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:35:51.022055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598231+0000","last_change":"2023-12-19T07:47:21.264957+0000","last_active":"2023-12-19T07:47:22.598231+0000","last_peered":"2023-12-19T07:47:22.598231+0000","last_clean":"2023-12-19T07:47:22.598231+0000","last_became_active":"2023-12-19T07:47:21.264514+0000","last_became_peered":"2023-12-19T07:47:21.264514+0000","last_unstale":"2023-12-19T07:47:22.598231+0000","last_undegraded":"2023-12-19T07:47:22.598231+0000","last_fullsized":"2023-12-19T07:47:22.598231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:20.377759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807125+0000","last_change":"2023-12-19T07:47:17.199047+0000","last_active":"2023-12-19T07:47:22.807125+0000","last_peered":"2023-12-19T07:47:22.807125+0000","last_clean":"2023-12-19T07:47:22.807125+0000","last_became_active":"2023-12-19T07:47:17.198760+0000","last_became_peered":"2023-12-19T07:47:17.198760+0000","last_unstale":"2023-12-19T07:47:22.807125+0000","last_undegraded":"2023-12-19T07:47:22.807125+0000","last_fullsized":"2023-12-19T07:47:22.807125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359707+0000","last_change":"2023-12-19T07:47:16.206922+0000","last_active":"2023-12-19T07:47:22.359707+0000","last_peered":"2023-12-19T07:47:22.359707+0000","last_clean":"2023-12-19T07:47:22.359707+0000","last_became_active":"2023-12-19T07:47:16.206605+0000","last_became_peered":"2023-12-19T07:47:16.206605+0000","last_unstale":"2023-12-19T07:47:22.359707+0000","last_undegraded":"2023-12-19T07:47:22.359707+0000","last_fullsized":"2023-12-19T07:47:22.359707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:51.711538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.597807+0000","last_change":"2023-12-19T07:47:21.845593+0000","last_active":"2023-12-19T07:47:22.597807+0000","last_peered":"2023-12-19T07:47:22.597807+0000","last_clean":"2023-12-19T07:47:22.597807+0000","last_became_active":"2023-12-19T07:47:21.843663+0000","last_became_peered":"2023-12-19T07:47:21.843663+0000","last_unstale":"2023-12-19T07:47:22.597807+0000","last_undegraded":"2023-12-19T07:47:22.597807+0000","last_fullsized":"2023-12-19T07:47:22.597807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:07.490325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.902979+0000","last_change":"2023-12-19T07:47:21.846558+0000","last_active":"2023-12-19T07:47:22.902979+0000","last_peered":"2023-12-19T07:47:22.902979+0000","last_clean":"2023-12-19T07:47:22.902979+0000","last_became_active":"2023-12-19T07:47:21.845861+0000","last_became_peered":"2023-12-19T07:47:21.845861+0000","last_unstale":"2023-12-19T07:47:22.902979+0000","last_undegraded":"2023-12-19T07:47:22.902979+0000","last_fullsized":"2023-12-19T07:47:22.902979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:28.389344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351325+0000","last_change":"2023-12-19T07:47:17.210958+0000","last_active":"2023-12-19T07:47:22.351325+0000","last_peered":"2023-12-19T07:47:22.351325+0000","last_clean":"2023-12-19T07:47:22.351325+0000","last_became_active":"2023-12-19T07:47:17.210537+0000","last_became_peered":"2023-12-19T07:47:17.210537+0000","last_unstale":"2023-12-19T07:47:22.351325+0000","last_undegraded":"2023-12-19T07:47:22.351325+0000","last_fullsized":"2023-12-19T07:47:22.351325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805998+0000","last_change":"2023-12-19T07:47:16.198817+0000","last_active":"2023-12-19T07:47:22.805998+0000","last_peered":"2023-12-19T07:47:22.805998+0000","last_clean":"2023-12-19T07:47:22.805998+0000","last_became_active":"2023-12-19T07:47:16.198491+0000","last_became_peered":"2023-12-19T07:47:16.198491+0000","last_unstale":"2023-12-19T07:47:22.805998+0000","last_undegraded":"2023-12-19T07:47:22.805998+0000","last_fullsized":"2023-12-19T07:47:22.805998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359608+0000","last_change":"2023-12-19T07:47:21.263980+0000","last_active":"2023-12-19T07:47:22.359608+0000","last_peered":"2023-12-19T07:47:22.359608+0000","last_clean":"2023-12-19T07:47:22.359608+0000","last_became_active":"2023-12-19T07:47:21.263310+0000","last_became_peered":"2023-12-19T07:47:21.263310+0000","last_unstale":"2023-12-19T07:47:22.359608+0000","last_undegraded":"2023-12-19T07:47:22.359608+0000","last_fullsized":"2023-12-19T07:47:22.359608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.138125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598272+0000","last_change":"2023-12-19T07:47:17.196870+0000","last_active":"2023-12-19T07:47:22.598272+0000","last_peered":"2023-12-19T07:47:22.598272+0000","last_clean":"2023-12-19T07:47:22.598272+0000","last_became_active":"2023-12-19T07:47:17.196588+0000","last_became_peered":"2023-12-19T07:47:17.196588+0000","last_unstale":"2023-12-19T07:47:22.598272+0000","last_undegraded":"2023-12-19T07:47:22.598272+0000","last_fullsized":"2023-12-19T07:47:22.598272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598626+0000","last_change":"2023-12-19T07:47:16.198918+0000","last_active":"2023-12-19T07:47:22.598626+0000","last_peered":"2023-12-19T07:47:22.598626+0000","last_clean":"2023-12-19T07:47:22.598626+0000","last_became_active":"2023-12-19T07:47:16.198496+0000","last_became_peered":"2023-12-19T07:47:16.198496+0000","last_unstale":"2023-12-19T07:47:22.598626+0000","last_undegraded":"2023-12-19T07:47:22.598626+0000","last_fullsized":"2023-12-19T07:47:22.598626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:30.613221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358925+0000","last_change":"2023-12-19T07:47:16.192976+0000","last_active":"2023-12-19T07:47:22.358925+0000","last_peered":"2023-12-19T07:47:22.358925+0000","last_clean":"2023-12-19T07:47:22.358925+0000","last_became_active":"2023-12-19T07:47:16.192588+0000","last_became_peered":"2023-12-19T07:47:16.192588+0000","last_unstale":"2023-12-19T07:47:22.358925+0000","last_undegraded":"2023-12-19T07:47:22.358925+0000","last_fullsized":"2023-12-19T07:47:22.358925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:21.683592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903549+0000","last_change":"2023-12-19T07:47:17.205110+0000","last_active":"2023-12-19T07:47:22.903549+0000","last_peered":"2023-12-19T07:47:22.903549+0000","last_clean":"2023-12-19T07:47:22.903549+0000","last_became_active":"2023-12-19T07:47:17.204811+0000","last_became_peered":"2023-12-19T07:47:17.204811+0000","last_unstale":"2023-12-19T07:47:22.903549+0000","last_undegraded":"2023-12-19T07:47:22.903549+0000","last_fullsized":"2023-12-19T07:47:22.903549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:07.078408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.303570+0000","last_change":"2023-12-19T07:47:21.259361+0000","last_active":"2023-12-19T07:47:22.303570+0000","last_peered":"2023-12-19T07:47:22.303570+0000","last_clean":"2023-12-19T07:47:22.303570+0000","last_became_active":"2023-12-19T07:47:21.258883+0000","last_became_peered":"2023-12-19T07:47:21.258883+0000","last_unstale":"2023-12-19T07:47:22.303570+0000","last_undegraded":"2023-12-19T07:47:22.303570+0000","last_fullsized":"2023-12-19T07:47:22.303570+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:28.728929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598981+0000","last_change":"2023-12-19T07:47:16.206769+0000","last_active":"2023-12-19T07:47:22.598981+0000","last_peered":"2023-12-19T07:47:22.598981+0000","last_clean":"2023-12-19T07:47:22.598981+0000","last_became_active":"2023-12-19T07:47:16.206392+0000","last_became_peered":"2023-12-19T07:47:16.206392+0000","last_unstale":"2023-12-19T07:47:22.598981+0000","last_undegraded":"2023-12-19T07:47:22.598981+0000","last_fullsized":"2023-12-19T07:47:22.598981+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:15.139092+0000","last_clean_scrub_stamp":"2023-12-19T07:47:15.139092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:19:23.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306430+0000","last_change":"2023-12-19T07:47:17.204694+0000","last_active":"2023-12-19T07:47:22.306430+0000","last_peered":"2023-12-19T07:47:22.306430+0000","last_clean":"2023-12-19T07:47:22.306430+0000","last_became_active":"2023-12-19T07:47:17.204276+0000","last_became_peered":"2023-12-19T07:47:17.204276+0000","last_unstale":"2023-12-19T07:47:22.306430+0000","last_undegraded":"2023-12-19T07:47:22.306430+0000","last_fullsized":"2023-12-19T07:47:22.306430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599471+0000","last_change":"2023-12-19T07:47:21.258311+0000","last_active":"2023-12-19T07:47:22.599471+0000","last_peered":"2023-12-19T07:47:22.599471+0000","last_clean":"2023-12-19T07:47:22.599471+0000","last_became_active":"2023-12-19T07:47:21.257510+0000","last_became_peered":"2023-12-19T07:47:21.257510+0000","last_unstale":"2023-12-19T07:47:22.599471+0000","last_undegraded":"2023-12-19T07:47:22.599471+0000","last_fullsized":"2023-12-19T07:47:22.599471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:20.213041+0000","last_clean_scrub_stamp":"2023-12-19T07:47:20.213041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:15:44.535034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.358817+0000","last_change":"2023-12-19T07:47:17.204892+0000","last_active":"2023-12-19T07:47:22.358817+0000","last_peered":"2023-12-19T07:47:22.358817+0000","last_clean":"2023-12-19T07:47:22.358817+0000","last_became_active":"2023-12-19T07:47:17.201155+0000","last_became_peered":"2023-12-19T07:47:17.201155+0000","last_unstale":"2023-12-19T07:47:22.358817+0000","last_undegraded":"2023-12-19T07:47:22.358817+0000","last_fullsized":"2023-12-19T07:47:22.358817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.302909+0000","last_change":"2023-12-19T07:47:17.204307+0000","last_active":"2023-12-19T07:47:22.302909+0000","last_peered":"2023-12-19T07:47:22.302909+0000","last_clean":"2023-12-19T07:47:22.302909+0000","last_became_active":"2023-12-19T07:47:17.203914+0000","last_became_peered":"2023-12-19T07:47:17.203914+0000","last_unstale":"2023-12-19T07:47:22.302909+0000","last_undegraded":"2023-12-19T07:47:22.302909+0000","last_fullsized":"2023-12-19T07:47:22.302909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:45:07.379149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806686+0000","last_change":"2023-12-19T07:47:17.196779+0000","last_active":"2023-12-19T07:47:22.806686+0000","last_peered":"2023-12-19T07:47:22.806686+0000","last_clean":"2023-12-19T07:47:22.806686+0000","last_became_active":"2023-12-19T07:47:17.196494+0000","last_became_peered":"2023-12-19T07:47:17.196494+0000","last_unstale":"2023-12-19T07:47:22.806686+0000","last_undegraded":"2023-12-19T07:47:22.806686+0000","last_fullsized":"2023-12-19T07:47:22.806686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903605+0000","last_change":"2023-12-19T07:47:17.205970+0000","last_active":"2023-12-19T07:47:22.903605+0000","last_peered":"2023-12-19T07:47:22.903605+0000","last_clean":"2023-12-19T07:47:22.903605+0000","last_became_active":"2023-12-19T07:47:17.205581+0000","last_became_peered":"2023-12-19T07:47:17.205581+0000","last_unstale":"2023-12-19T07:47:22.903605+0000","last_undegraded":"2023-12-19T07:47:22.903605+0000","last_fullsized":"2023-12-19T07:47:22.903605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599435+0000","last_change":"2023-12-19T07:47:17.208151+0000","last_active":"2023-12-19T07:47:22.599435+0000","last_peered":"2023-12-19T07:47:22.599435+0000","last_clean":"2023-12-19T07:47:22.599435+0000","last_became_active":"2023-12-19T07:47:17.207880+0000","last_became_peered":"2023-12-19T07:47:17.207880+0000","last_unstale":"2023-12-19T07:47:22.599435+0000","last_undegraded":"2023-12-19T07:47:22.599435+0000","last_fullsized":"2023-12-19T07:47:22.599435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:08.230851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.352831+0000","last_change":"2023-12-19T07:47:17.209159+0000","last_active":"2023-12-19T07:47:22.352831+0000","last_peered":"2023-12-19T07:47:22.352831+0000","last_clean":"2023-12-19T07:47:22.352831+0000","last_became_active":"2023-12-19T07:47:17.208870+0000","last_became_peered":"2023-12-19T07:47:17.208870+0000","last_unstale":"2023-12-19T07:47:22.352831+0000","last_undegraded":"2023-12-19T07:47:22.352831+0000","last_fullsized":"2023-12-19T07:47:22.352831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:57.433975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.360063+0000","last_change":"2023-12-19T07:47:17.209320+0000","last_active":"2023-12-19T07:47:22.360063+0000","last_peered":"2023-12-19T07:47:22.360063+0000","last_clean":"2023-12-19T07:47:22.360063+0000","last_became_active":"2023-12-19T07:47:17.208263+0000","last_became_peered":"2023-12-19T07:47:17.208263+0000","last_unstale":"2023-12-19T07:47:22.360063+0000","last_undegraded":"2023-12-19T07:47:22.360063+0000","last_fullsized":"2023-12-19T07:47:22.360063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599483+0000","last_change":"2023-12-19T07:47:17.211203+0000","last_active":"2023-12-19T07:47:22.599483+0000","last_peered":"2023-12-19T07:47:22.599483+0000","last_clean":"2023-12-19T07:47:22.599483+0000","last_became_active":"2023-12-19T07:47:17.210860+0000","last_became_peered":"2023-12-19T07:47:17.210860+0000","last_unstale":"2023-12-19T07:47:22.599483+0000","last_undegraded":"2023-12-19T07:47:22.599483+0000","last_fullsized":"2023-12-19T07:47:22.599483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:01.473306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805820+0000","last_change":"2023-12-19T07:47:17.210725+0000","last_active":"2023-12-19T07:47:22.805820+0000","last_peered":"2023-12-19T07:47:22.805820+0000","last_clean":"2023-12-19T07:47:22.805820+0000","last_became_active":"2023-12-19T07:47:17.210432+0000","last_became_peered":"2023-12-19T07:47:17.210432+0000","last_unstale":"2023-12-19T07:47:22.805820+0000","last_undegraded":"2023-12-19T07:47:22.805820+0000","last_fullsized":"2023-12-19T07:47:22.805820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.599063+0000","last_change":"2023-12-19T07:47:17.200440+0000","last_active":"2023-12-19T07:47:22.599063+0000","last_peered":"2023-12-19T07:47:22.599063+0000","last_clean":"2023-12-19T07:47:22.599063+0000","last_became_active":"2023-12-19T07:47:17.200137+0000","last_became_peered":"2023-12-19T07:47:17.200137+0000","last_unstale":"2023-12-19T07:47:22.599063+0000","last_undegraded":"2023-12-19T07:47:22.599063+0000","last_fullsized":"2023-12-19T07:47:22.599063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:58:37.051503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806350+0000","last_change":"2023-12-19T07:47:17.200217+0000","last_active":"2023-12-19T07:47:22.806350+0000","last_peered":"2023-12-19T07:47:22.806350+0000","last_clean":"2023-12-19T07:47:22.806350+0000","last_became_active":"2023-12-19T07:47:17.199920+0000","last_became_peered":"2023-12-19T07:47:17.199920+0000","last_unstale":"2023-12-19T07:47:22.806350+0000","last_undegraded":"2023-12-19T07:47:22.806350+0000","last_fullsized":"2023-12-19T07:47:22.806350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903403+0000","last_change":"2023-12-19T07:47:17.200705+0000","last_active":"2023-12-19T07:47:22.903403+0000","last_peered":"2023-12-19T07:47:22.903403+0000","last_clean":"2023-12-19T07:47:22.903403+0000","last_became_active":"2023-12-19T07:47:17.200462+0000","last_became_peered":"2023-12-19T07:47:17.200462+0000","last_unstale":"2023-12-19T07:47:22.903403+0000","last_undegraded":"2023-12-19T07:47:22.903403+0000","last_fullsized":"2023-12-19T07:47:22.903403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359431+0000","last_change":"2023-12-19T07:47:17.212064+0000","last_active":"2023-12-19T07:47:22.359431+0000","last_peered":"2023-12-19T07:47:22.359431+0000","last_clean":"2023-12-19T07:47:22.359431+0000","last_became_active":"2023-12-19T07:47:17.211684+0000","last_became_peered":"2023-12-19T07:47:17.211684+0000","last_unstale":"2023-12-19T07:47:22.359431+0000","last_undegraded":"2023-12-19T07:47:22.359431+0000","last_fullsized":"2023-12-19T07:47:22.359431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:12:21.908290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868184+0000","last_change":"2023-12-19T07:47:17.212402+0000","last_active":"2023-12-19T07:47:22.868184+0000","last_peered":"2023-12-19T07:47:22.868184+0000","last_clean":"2023-12-19T07:47:22.868184+0000","last_became_active":"2023-12-19T07:47:17.211887+0000","last_became_peered":"2023-12-19T07:47:17.211887+0000","last_unstale":"2023-12-19T07:47:22.868184+0000","last_undegraded":"2023-12-19T07:47:22.868184+0000","last_fullsized":"2023-12-19T07:47:22.868184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:43.642215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.304186+0000","last_change":"2023-12-19T07:47:17.207392+0000","last_active":"2023-12-19T07:47:22.304186+0000","last_peered":"2023-12-19T07:47:22.304186+0000","last_clean":"2023-12-19T07:47:22.304186+0000","last_became_active":"2023-12-19T07:47:17.206962+0000","last_became_peered":"2023-12-19T07:47:17.206962+0000","last_unstale":"2023-12-19T07:47:22.304186+0000","last_undegraded":"2023-12-19T07:47:22.304186+0000","last_fullsized":"2023-12-19T07:47:22.304186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:51:32.771539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.807623+0000","last_change":"2023-12-19T07:47:17.217408+0000","last_active":"2023-12-19T07:47:22.807623+0000","last_peered":"2023-12-19T07:47:22.807623+0000","last_clean":"2023-12-19T07:47:22.807623+0000","last_became_active":"2023-12-19T07:47:17.217038+0000","last_became_peered":"2023-12-19T07:47:17.217038+0000","last_unstale":"2023-12-19T07:47:22.807623+0000","last_undegraded":"2023-12-19T07:47:22.807623+0000","last_fullsized":"2023-12-19T07:47:22.807623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:12.732155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.805776+0000","last_change":"2023-12-19T07:47:17.194768+0000","last_active":"2023-12-19T07:47:22.805776+0000","last_peered":"2023-12-19T07:47:22.805776+0000","last_clean":"2023-12-19T07:47:22.805776+0000","last_became_active":"2023-12-19T07:47:17.194457+0000","last_became_peered":"2023-12-19T07:47:17.194457+0000","last_unstale":"2023-12-19T07:47:22.805776+0000","last_undegraded":"2023-12-19T07:47:22.805776+0000","last_fullsized":"2023-12-19T07:47:22.805776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.351363+0000","last_change":"2023-12-19T07:47:17.209052+0000","last_active":"2023-12-19T07:47:22.351363+0000","last_peered":"2023-12-19T07:47:22.351363+0000","last_clean":"2023-12-19T07:47:22.351363+0000","last_became_active":"2023-12-19T07:47:17.208671+0000","last_became_peered":"2023-12-19T07:47:17.208671+0000","last_unstale":"2023-12-19T07:47:22.351363+0000","last_undegraded":"2023-12-19T07:47:22.351363+0000","last_fullsized":"2023-12-19T07:47:22.351363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:24.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.903861+0000","last_change":"2023-12-19T07:47:17.199159+0000","last_active":"2023-12-19T07:47:22.903861+0000","last_peered":"2023-12-19T07:47:22.903861+0000","last_clean":"2023-12-19T07:47:22.903861+0000","last_became_active":"2023-12-19T07:47:17.198548+0000","last_became_peered":"2023-12-19T07:47:17.198548+0000","last_unstale":"2023-12-19T07:47:22.903861+0000","last_undegraded":"2023-12-19T07:47:22.903861+0000","last_fullsized":"2023-12-19T07:47:22.903861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:25.457743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598739+0000","last_change":"2023-12-19T07:47:17.202636+0000","last_active":"2023-12-19T07:47:22.598739+0000","last_peered":"2023-12-19T07:47:22.598739+0000","last_clean":"2023-12-19T07:47:22.598739+0000","last_became_active":"2023-12-19T07:47:17.202331+0000","last_became_peered":"2023-12-19T07:47:17.202331+0000","last_unstale":"2023-12-19T07:47:22.598739+0000","last_undegraded":"2023-12-19T07:47:22.598739+0000","last_fullsized":"2023-12-19T07:47:22.598739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:13.159718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.306911+0000","last_change":"2023-12-19T07:47:17.202590+0000","last_active":"2023-12-19T07:47:22.306911+0000","last_peered":"2023-12-19T07:47:22.306911+0000","last_clean":"2023-12-19T07:47:22.306911+0000","last_became_active":"2023-12-19T07:47:17.202206+0000","last_became_peered":"2023-12-19T07:47:17.202206+0000","last_unstale":"2023-12-19T07:47:22.306911+0000","last_undegraded":"2023-12-19T07:47:22.306911+0000","last_fullsized":"2023-12-19T07:47:22.306911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:04.053349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359760+0000","last_change":"2023-12-19T07:47:17.209806+0000","last_active":"2023-12-19T07:47:22.359760+0000","last_peered":"2023-12-19T07:47:22.359760+0000","last_clean":"2023-12-19T07:47:22.359760+0000","last_became_active":"2023-12-19T07:47:17.209424+0000","last_became_peered":"2023-12-19T07:47:17.209424+0000","last_unstale":"2023-12-19T07:47:22.359760+0000","last_undegraded":"2023-12-19T07:47:22.359760+0000","last_fullsized":"2023-12-19T07:47:22.359760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:33.848435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.359954+0000","last_change":"2023-12-19T07:47:17.209509+0000","last_active":"2023-12-19T07:47:22.359954+0000","last_peered":"2023-12-19T07:47:22.359954+0000","last_clean":"2023-12-19T07:47:22.359954+0000","last_became_active":"2023-12-19T07:47:17.209026+0000","last_became_peered":"2023-12-19T07:47:17.209026+0000","last_unstale":"2023-12-19T07:47:22.359954+0000","last_undegraded":"2023-12-19T07:47:22.359954+0000","last_fullsized":"2023-12-19T07:47:22.359954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:34:31.433644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868374+0000","last_change":"2023-12-19T07:47:17.225611+0000","last_active":"2023-12-19T07:47:22.868374+0000","last_peered":"2023-12-19T07:47:22.868374+0000","last_clean":"2023-12-19T07:47:22.868374+0000","last_became_active":"2023-12-19T07:47:17.225305+0000","last_became_peered":"2023-12-19T07:47:17.225305+0000","last_unstale":"2023-12-19T07:47:22.868374+0000","last_undegraded":"2023-12-19T07:47:22.868374+0000","last_fullsized":"2023-12-19T07:47:22.868374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.474113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.598130+0000","last_change":"2023-12-19T07:47:17.209463+0000","last_active":"2023-12-19T07:47:22.598130+0000","last_peered":"2023-12-19T07:47:22.598130+0000","last_clean":"2023-12-19T07:47:22.598130+0000","last_became_active":"2023-12-19T07:47:17.201276+0000","last_became_peered":"2023-12-19T07:47:17.201276+0000","last_unstale":"2023-12-19T07:47:22.598130+0000","last_undegraded":"2023-12-19T07:47:22.598130+0000","last_fullsized":"2023-12-19T07:47:22.598130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:22.053149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867603+0000","last_change":"2023-12-19T07:47:17.200808+0000","last_active":"2023-12-19T07:47:22.867603+0000","last_peered":"2023-12-19T07:47:22.867603+0000","last_clean":"2023-12-19T07:47:22.867603+0000","last_became_active":"2023-12-19T07:47:17.200445+0000","last_became_peered":"2023-12-19T07:47:17.200445+0000","last_unstale":"2023-12-19T07:47:22.867603+0000","last_undegraded":"2023-12-19T07:47:22.867603+0000","last_fullsized":"2023-12-19T07:47:22.867603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:50.186735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806351+0000","last_change":"2023-12-19T07:47:17.194801+0000","last_active":"2023-12-19T07:47:22.806351+0000","last_peered":"2023-12-19T07:47:22.806351+0000","last_clean":"2023-12-19T07:47:22.806351+0000","last_became_active":"2023-12-19T07:47:17.194502+0000","last_became_peered":"2023-12-19T07:47:17.194502+0000","last_unstale":"2023-12-19T07:47:22.806351+0000","last_undegraded":"2023-12-19T07:47:22.806351+0000","last_fullsized":"2023-12-19T07:47:22.806351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.867343+0000","last_change":"2023-12-19T07:47:17.214803+0000","last_active":"2023-12-19T07:47:22.867343+0000","last_peered":"2023-12-19T07:47:22.867343+0000","last_clean":"2023-12-19T07:47:22.867343+0000","last_became_active":"2023-12-19T07:47:17.214536+0000","last_became_peered":"2023-12-19T07:47:17.214536+0000","last_unstale":"2023-12-19T07:47:22.867343+0000","last_undegraded":"2023-12-19T07:47:22.867343+0000","last_fullsized":"2023-12-19T07:47:22.867343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:36.849171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.806760+0000","last_change":"2023-12-19T07:47:17.206649+0000","last_active":"2023-12-19T07:47:22.806760+0000","last_peered":"2023-12-19T07:47:22.806760+0000","last_clean":"2023-12-19T07:47:22.806760+0000","last_became_active":"2023-12-19T07:47:17.206360+0000","last_became_peered":"2023-12-19T07:47:17.206360+0000","last_unstale":"2023-12-19T07:47:22.806760+0000","last_undegraded":"2023-12-19T07:47:22.806760+0000","last_fullsized":"2023-12-19T07:47:22.806760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:04:51.020530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.904329+0000","last_change":"2023-12-19T07:47:17.195685+0000","last_active":"2023-12-19T07:47:22.904329+0000","last_peered":"2023-12-19T07:47:22.904329+0000","last_clean":"2023-12-19T07:47:22.904329+0000","last_became_active":"2023-12-19T07:47:17.195380+0000","last_became_peered":"2023-12-19T07:47:17.195380+0000","last_unstale":"2023-12-19T07:47:22.904329+0000","last_undegraded":"2023-12-19T07:47:22.904329+0000","last_fullsized":"2023-12-19T07:47:22.904329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:47.378624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.353591+0000","last_change":"2023-12-19T07:47:17.204379+0000","last_active":"2023-12-19T07:47:22.353591+0000","last_peered":"2023-12-19T07:47:22.353591+0000","last_clean":"2023-12-19T07:47:22.353591+0000","last_became_active":"2023-12-19T07:47:17.204111+0000","last_became_peered":"2023-12-19T07:47:17.204111+0000","last_unstale":"2023-12-19T07:47:22.353591+0000","last_undegraded":"2023-12-19T07:47:22.353591+0000","last_fullsized":"2023-12-19T07:47:22.353591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:17:12.069492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-12-19T07:47:22.868221+0000","last_change":"2023-12-19T07:47:17.209821+0000","last_active":"2023-12-19T07:47:22.868221+0000","last_peered":"2023-12-19T07:47:22.868221+0000","last_clean":"2023-12-19T07:47:22.868221+0000","last_became_active":"2023-12-19T07:47:17.209235+0000","last_became_peered":"2023-12-19T07:47:17.209235+0000","last_unstale":"2023-12-19T07:47:22.868221+0000","last_undegraded":"2023-12-19T07:47:22.868221+0000","last_fullsized":"2023-12-19T07:47:22.868221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:16.152484+0000","last_clean_scrub_stamp":"2023-12-19T07:47:16.152484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:17:23.894121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":864256,"data_stored":697463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0659999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":270336,"data_stored":106631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":106667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":278528,"data_stored":110254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3979999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":266240,"data_stored":106541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297556,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074284,"statfs":{"total":96636764160,"available":96332066816,"internally_reserved":0,"allocated":274432,"data_stored":106701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":868352,"data_stored":696508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:47:38.496 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-19T07:47:38.496 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-19T07:47:38.496 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-19T07:47:38.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-19T07:47:38.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.712+0000 7f78d4d23700 1 Processor -- start 2023-12-19T07:47:38.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.712+0000 7f78d4d23700 1 -- start start 2023-12-19T07:47:38.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78d4d23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78d006de20 0x7f78d00c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78d4d23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d006d990 con 0x7f78d006c7c0 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78d4d23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d006cd80 con 0x7f78d0155960 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78d4d23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d006d060 con 0x7f78d006de20 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ceffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78d006de20 0x7f78d00c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78cf7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f78d0155960 0x7f78d015a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46426/0 (socket says 172.21.15.111:46426) 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78cffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f78d006c7c0 0x7f78d00aca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56200/0 (socket says 172.21.15.111:56200) 2023-12-19T07:47:38.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78cf7fe700 1 -- 172.21.15.111:0/1668813831 learned_addr learned my addr 172.21.15.111:0/1668813831 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592928824 0 0) 0x7f78d006cd80 con 0x7f78d0155960 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b8003570 con 0x7f78d0155960 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044107398 0 0) 0x7f78d006d990 con 0x7f78d006c7c0 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ceffd700 1 -- 172.21.15.111:0/1668813831 >> v1:172.21.15.143:6789/0 conn(0x7f78d0155960 legacy=0x7f78d015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ceffd700 1 -- 172.21.15.111:0/1668813831 >> v1:172.21.15.111:6789/0 conn(0x7f78d006c7c0 legacy=0x7f78d00aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ceffd700 1 -- 172.21.15.111:0/1668813831 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d006d340 con 0x7f78d006de20 2023-12-19T07:47:38.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ceffd700 1 --2- 172.21.15.111:0/1668813831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d006de20 0x7f78d00c77f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f78c40098e0 tx=0x7f78c40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af512a0a35fc05e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400f6b0 con 0x7f78d006de20 2023-12-19T07:47:38.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c400f810 con 0x7f78d006de20 2023-12-19T07:47:38.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.716+0000 7f78ce7fc700 1 -- 172.21.15.111:0/1668813831 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400fbb0 con 0x7f78d006de20 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- 172.21.15.111:0/1668813831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d006de20 msgr2=0x7f78d00c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 --2- 172.21.15.111:0/1668813831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d006de20 0x7f78d00c77f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f78c40098e0 tx=0x7f78c40091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- 172.21.15.111:0/1668813831 shutdown_connections 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 --2- 172.21.15.111:0/1668813831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d006de20 0x7f78d00c77f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- 172.21.15.111:0/1668813831 >> 172.21.15.111:0/1668813831 conn(0x7f78d0067bb0 msgr2=0x7f78d00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- 172.21.15.111:0/1668813831 shutdown_connections 2023-12-19T07:47:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- 172.21.15.111:0/1668813831 wait complete. 2023-12-19T07:47:38.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 Processor -- start 2023-12-19T07:47:38.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- start start 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78d006c7c0 0x7f78d0148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d0155960 0x7f78d00afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.720+0000 7f78d4d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d01551a0 con 0x7f78d006de20 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78d4d23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0148aa0 con 0x7f78d0155960 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78d4d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0154990 con 0x7f78d006c7c0 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34952/0 (socket says 172.21.15.111:34952) 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 -- 172.21.15.111:0/310578077 learned_addr learned my addr 172.21.15.111:0/310578077 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78cf7fe700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78d006c7c0 0x7f78d0148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78cffff700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d0155960 0x7f78d00afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 -- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78d006c7c0 msgr2=0x7f78d0148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78d006c7c0 0x7f78d0148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 -- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d0155960 msgr2=0x7f78d00afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d0155960 0x7f78d00afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 -- 172.21.15.111:0/310578077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c4009420 con 0x7f78d006de20 2023-12-19T07:47:38.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78ceffd700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f78c40093f0 tx=0x7f78c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400f980 con 0x7f78d006de20 2023-12-19T07:47:38.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c400fae0 con 0x7f78d006de20 2023-12-19T07:47:38.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d00c99a0 con 0x7f78d006de20 2023-12-19T07:47:38.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4010680 con 0x7f78d006de20 2023-12-19T07:47:38.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.724+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d0154be0 con 0x7f78d006de20 2023-12-19T07:47:38.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.728+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d004be40 con 0x7f78d006de20 2023-12-19T07:47:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.728+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f78c40243f0 con 0x7f78d006de20 2023-12-19T07:47:38.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.728+0000 7f78b77fe700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 0x7f78bc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:38.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.728+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f78c4014070 con 0x7f78d006de20 2023-12-19T07:47:38.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.732+0000 7f78cf7fe700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 0x7f78bc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.732+0000 7f78cf7fe700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 0x7f78bc062a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78c00058d0 tx=0x7f78c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.736+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f78c404e840 con 0x7f78d006de20 2023-12-19T07:47:38.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.936+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f78d006d620 con 0x7f78d006de20 2023-12-19T07:47:38.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.936+0000 7f78b77fe700 1 -- 172.21.15.111:0/310578077 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f78c4051e60 con 0x7f78d006de20 2023-12-19T07:47:38.920 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:38.920 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 msgr2=0x7f78bc062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 0x7f78bc062a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78c00058d0 tx=0x7f78c0009040 comp rx=0 tx=0).stop 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 msgr2=0x7f78d00c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f78c40093f0 tx=0x7f78c4019000 comp rx=0 tx=0).stop 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 shutdown_connections 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78bc060580 0x7f78bc062a30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78d0155960 0x7f78d00afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78d006de20 0x7f78d00c9460 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 --2- 172.21.15.111:0/310578077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78d006c7c0 0x7f78d0148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:38.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 >> 172.21.15.111:0/310578077 conn(0x7f78d0067bb0 msgr2=0x7f78d00af460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:38.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 shutdown_connections 2023-12-19T07:47:38.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:38.940+0000 7f78d4d23700 1 -- 172.21.15.111:0/310578077 wait complete. 2023-12-19T07:47:38.943 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-19T07:47:38.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:39.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.112+0000 7fa492fc3700 1 Processor -- start 2023-12-19T07:47:39.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.112+0000 7fa492fc3700 1 -- start start 2023-12-19T07:47:39.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa492fc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa48c0c6cd0 0x7fa48c06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa492fc3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa48c15cc20 0x7fa48c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa492fc3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c06b470 con 0x7fa48c0c6900 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa492fc3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c06be40 con 0x7fa48c15cc20 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa492fc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c06e110 con 0x7fa48c0c6cd0 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa491fc1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa48c0c6900 0x7fa48c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56214/0 (socket says 172.21.15.111:56214) 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa491fc1700 1 -- 172.21.15.111:0/1359967672 learned_addr learned my addr 172.21.15.111:0/1359967672 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 --2- 172.21.15.111:0/1359967672 >> v2:172.21.15.111:3300/0 conn(0x7fa48c15cc20 0x7fa48c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4917c0700 1 --2- 172.21.15.111:0/1359967672 >> v2:172.21.15.143:3300/0 conn(0x7fa48c0c6cd0 0x7fa48c06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490059414 0 0) 0x7fa48c06b470 con 0x7fa48c0c6900 2023-12-19T07:47:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478003570 con 0x7fa48c0c6900 2023-12-19T07:47:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 -- 172.21.15.111:0/1359967672 >> v2:172.21.15.143:3300/0 conn(0x7fa48c0c6cd0 msgr2=0x7fa48c06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 --2- 172.21.15.111:0/1359967672 >> v2:172.21.15.143:3300/0 conn(0x7fa48c0c6cd0 0x7fa48c06b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 796545340 0 0) 0x7fa478003570 con 0x7fa48c0c6900 2023-12-19T07:47:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 -- 172.21.15.111:0/1359967672 >> v1:172.21.15.111:6789/0 conn(0x7fa48c0c6900 legacy=0x7fa48c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 -- 172.21.15.111:0/1359967672 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c161d10 con 0x7fa48c15cc20 2023-12-19T07:47:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa4927c2700 1 --2- 172.21.15.111:0/1359967672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c15cc20 0x7fa48c15d080 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa48c06a030 tx=0x7fa484009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2177a6e967001ffa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48400e070 con 0x7fa48c15cc20 2023-12-19T07:47:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48400b7a0 con 0x7fa48c15cc20 2023-12-19T07:47:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.116+0000 7fa490fbf700 1 -- 172.21.15.111:0/1359967672 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488001470 con 0x7fa48c15cc20 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- 172.21.15.111:0/1359967672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c15cc20 msgr2=0x7fa48c15d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 --2- 172.21.15.111:0/1359967672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c15cc20 0x7fa48c15d080 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa48c06a030 tx=0x7fa484009780 comp rx=0 tx=0).stop 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- 172.21.15.111:0/1359967672 shutdown_connections 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 --2- 172.21.15.111:0/1359967672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c15cc20 0x7fa48c15d080 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 --2- 172.21.15.111:0/1359967672 >> v2:172.21.15.143:3300/0 conn(0x7fa48c0c6cd0 0x7fa48c06b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- 172.21.15.111:0/1359967672 >> 172.21.15.111:0/1359967672 conn(0x7fa48c067bb0 msgr2=0x7fa48c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:39.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- 172.21.15.111:0/1359967672 shutdown_connections 2023-12-19T07:47:39.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- 172.21.15.111:0/1359967672 wait complete. 2023-12-19T07:47:39.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 Processor -- start 2023-12-19T07:47:39.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.120+0000 7fa492fc3700 1 -- start start 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c0c6cd0 0x7fa48c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c15cc20 0x7fa48c0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c0b1220 con 0x7fa48c0c6900 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c156800 con 0x7fa48c0c6cd0 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1491d0 con 0x7fa48c15cc20 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34968/0 (socket says 172.21.15.111:34968) 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 -- 172.21.15.111:0/3671280716 learned_addr learned my addr 172.21.15.111:0/3671280716 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:39.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa4917c0700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c0c6cd0 0x7fa48c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa4927c2700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c15cc20 0x7fa48c0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 -- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c15cc20 msgr2=0x7fa48c0c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c15cc20 0x7fa48c0c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 -- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c0c6cd0 msgr2=0x7fa48c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c0c6cd0 0x7fa48c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 -- 172.21.15.111:0/3671280716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa484009a20 con 0x7fa48c0c6900 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa491fc1700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa48800cb90 tx=0x7fa488001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488007360 con 0x7fa48c0c6900 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4880074e0 con 0x7fa48c0c6900 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c155910 con 0x7fa48c0c6900 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488004830 con 0x7fa48c0c6900 2023-12-19T07:47:39.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.124+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c155ed0 con 0x7fa48c0c6900 2023-12-19T07:47:39.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.128+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c04be40 con 0x7fa48c0c6900 2023-12-19T07:47:39.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.128+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa488004bc0 con 0x7fa48c0c6900 2023-12-19T07:47:39.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.128+0000 7fa481ffb700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 0x7fa470062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.128+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa48807c3a0 con 0x7fa48c0c6900 2023-12-19T07:47:39.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.128+0000 7fa4917c0700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 0x7fa470062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.132+0000 7fa4917c0700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 0x7fa470062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0b0540 tx=0x7fa47c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.132+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa48804c680 con 0x7fa48c0c6900 2023-12-19T07:47:39.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.308+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa48c06a580 con 0x7fa48c0c6900 2023-12-19T07:47:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.308+0000 7fa481ffb700 1 -- 172.21.15.111:0/3671280716 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7fa48804fca0 con 0x7fa48c0c6900 2023-12-19T07:47:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:39.293 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 msgr2=0x7fa470062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 0x7fa470062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0b0540 tx=0x7fa47c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 msgr2=0x7fa48c0b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa48800cb90 tx=0x7fa488001b50 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 shutdown_connections 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa470060550 0x7fa470062a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c0c6cd0 0x7fa48c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c15cc20 0x7fa48c0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 --2- 172.21.15.111:0/3671280716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa48c0c6900 0x7fa48c0b0ce0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 >> 172.21.15.111:0/3671280716 conn(0x7fa48c067bb0 msgr2=0x7fa48c15a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 shutdown_connections 2023-12-19T07:47:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.312+0000 7fa492fc3700 1 -- 172.21.15.111:0/3671280716 wait complete. 2023-12-19T07:47:39.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:47:39.316 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:39.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 Processor -- start 2023-12-19T07:47:39.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 -- start start 2023-12-19T07:47:39.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc77c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77c06cec0 con 0x7fc77c0c8b60 2023-12-19T07:47:39.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77c06d1a0 con 0x7fc77c0acea0 2023-12-19T07:47:39.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc781923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c0ccd20 con 0x7fc77c0cbbe0 2023-12-19T07:47:39.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc77bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc77c0c8b60 0x7fc77c15e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56216/0 (socket says 172.21.15.111:56216) 2023-12-19T07:47:39.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.496+0000 7fc77bfff700 1 -- 172.21.15.111:0/1942748120 learned_addr learned my addr 172.21.15.111:0/1942748120 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:39.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc781122700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc77c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102337562 0 0) 0x7fc77c06cec0 con 0x7fc77c0c8b60 2023-12-19T07:47:39.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76c003570 con 0x7fc77c0c8b60 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260609088 0 0) 0x7fc77c06d1a0 con 0x7fc77c0acea0 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77c06cec0 con 0x7fc77c0acea0 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779168616 0 0) 0x7fc76c003570 con 0x7fc77c0c8b60 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 msgr2=0x7fc77c0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc77c0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 >> v1:172.21.15.143:6789/0 conn(0x7fc77c0acea0 legacy=0x7fc77c0c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c06d480 con 0x7fc77c0c8b60 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fc768001470 con 0x7fc77c0c8b60 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 >> v1:172.21.15.111:6789/0 conn(0x7fc77c0c8b60 legacy=0x7fc77c15e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c003c20 0x7fc76c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc76c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc76c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c0ccd20 con 0x7fc76c004810 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c00bb70 con 0x7fc76c003c20 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c00bd30 con 0x7fc77c0cbbe0 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc76c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc781122700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc76c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc780921700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c003c20 0x7fc76c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 -- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 msgr2=0x7fc76c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c0cbbe0 0x7fc76c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 -- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c003c20 msgr2=0x7fc76c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c003c20 0x7fc76c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 -- 172.21.15.111:0/1942748120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c003570 con 0x7fc76c004810 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77bfff700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc76c004c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c2a0 tx=0x7fc768001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107ace399b075bad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76800cb90 con 0x7fc76c004810 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76800ccf0 con 0x7fc76c004810 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.500+0000 7fc77b7fe700 1 -- 172.21.15.111:0/1942748120 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768005b70 con 0x7fc76c004810 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- 172.21.15.111:0/1942748120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 msgr2=0x7fc76c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 --2- 172.21.15.111:0/1942748120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc76c004c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c2a0 tx=0x7fc768001e10 comp rx=0 tx=0).stop 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781122700 1 -- 172.21.15.111:0/1942748120 reap_dead start 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- 172.21.15.111:0/1942748120 shutdown_connections 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- 172.21.15.111:0/1942748120 >> 172.21.15.111:0/1942748120 conn(0x7fc77c0cdc00 msgr2=0x7fc77c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- 172.21.15.111:0/1942748120 shutdown_connections 2023-12-19T07:47:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- 172.21.15.111:0/1942748120 wait complete. 2023-12-19T07:47:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 Processor -- start 2023-12-19T07:47:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.504+0000 7fc781923700 1 -- start start 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c16daf0 0x7fc77c0b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc77c1592a0 0x7fc77c159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c154db0 con 0x7fc76c004810 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c159c50 con 0x7fc77c1592a0 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c0b09e0 con 0x7fc77c16daf0 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34986/0 (socket says 172.21.15.111:34986) 2023-12-19T07:47:39.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 -- 172.21.15.111:0/3388558616 learned_addr learned my addr 172.21.15.111:0/3388558616 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:39.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781122700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc77c1592a0 0x7fc77c159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc77bfff700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c16daf0 0x7fc77c0b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 -- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c16daf0 msgr2=0x7fc77c0b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c16daf0 0x7fc77c0b2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 -- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc77c1592a0 msgr2=0x7fc77c159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc77c1592a0 0x7fc77c159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 -- 172.21.15.111:0/3388558616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c16e060 con 0x7fc76c004810 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc780921700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc770002c00 tx=0x7fc770007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77000f920 con 0x7fc76c004810 2023-12-19T07:47:39.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77000faa0 con 0x7fc76c004810 2023-12-19T07:47:39.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77c16e340 con 0x7fc76c004810 2023-12-19T07:47:39.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77000fe60 con 0x7fc76c004810 2023-12-19T07:47:39.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.508+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77c16e900 con 0x7fc76c004810 2023-12-19T07:47:39.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.512+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77c04be40 con 0x7fc76c004810 2023-12-19T07:47:39.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.512+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc770013600 con 0x7fc76c004810 2023-12-19T07:47:39.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.516+0000 7fc778ff9700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 0x7fc760062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.516+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc77007cff0 con 0x7fc76c004810 2023-12-19T07:47:39.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.516+0000 7fc77bfff700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 0x7fc760062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.516+0000 7fc77bfff700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 0x7fc760062b00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc768008fd0 tx=0x7fc768015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.520+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc77004d370 con 0x7fc76c004810 2023-12-19T07:47:39.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.692+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc77c06d760 con 0x7fc76c004810 2023-12-19T07:47:39.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc778ff9700 1 -- 172.21.15.111:0/3388558616 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc770050990 con 0x7fc76c004810 2023-12-19T07:47:39.677 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:39.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 msgr2=0x7fc760062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 0x7fc760062b00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc768008fd0 tx=0x7fc768015000 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 msgr2=0x7fc77c154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc770002c00 tx=0x7fc770007d30 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 shutdown_connections 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc760060650 0x7fc760062b00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc77c1592a0 0x7fc77c159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc77c16daf0 0x7fc77c0b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 --2- 172.21.15.111:0/3388558616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc76c004810 0x7fc77c154870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 >> 172.21.15.111:0/3388558616 conn(0x7fc77c0cdc00 msgr2=0x7fc77c0c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 shutdown_connections 2023-12-19T07:47:39.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.696+0000 7fc781923700 1 -- 172.21.15.111:0/3388558616 wait complete. 2023-12-19T07:47:39.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:47:39.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.880+0000 7f605b278700 1 Processor -- start 2023-12-19T07:47:39.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 -- start start 2023-12-19T07:47:39.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60540cab10 0x7f60540ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f605406b1b0 0x7f605406b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540c7450 con 0x7f605415d410 2023-12-19T07:47:39.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605406bea0 con 0x7f60540cab10 2023-12-19T07:47:39.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f605b278700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540c6e00 con 0x7f605406b1b0 2023-12-19T07:47:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f6058813700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60540cab10 0x7f60540ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f6059014700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f605415d410 0x7f60540ca400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56218/0 (socket says 172.21.15.111:56218) 2023-12-19T07:47:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f6059014700 1 -- 172.21.15.111:0/193991188 learned_addr learned my addr 172.21.15.111:0/193991188 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.884+0000 7f6059815700 1 --2- 172.21.15.111:0/193991188 >> v2:172.21.15.143:3300/0 conn(0x7f605406b1b0 0x7f605406b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f6058813700 1 -- 172.21.15.111:0/193991188 >> v2:172.21.15.143:3300/0 conn(0x7f605406b1b0 msgr2=0x7f605406b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f6058813700 1 --2- 172.21.15.111:0/193991188 >> v2:172.21.15.143:3300/0 conn(0x7f605406b1b0 0x7f605406b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f6058813700 1 -- 172.21.15.111:0/193991188 >> v1:172.21.15.111:6789/0 conn(0x7f605415d410 legacy=0x7f60540ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f6058813700 1 -- 172.21.15.111:0/193991188 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540c70e0 con 0x7f60540cab10 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f604bfff700 1 -- 172.21.15.111:0/193991188 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351317703 0 0) 0x7f60540c7450 con 0x7f605415d410 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f6058813700 1 --2- 172.21.15.111:0/193991188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60540cab10 0x7f60540ab6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f605406a5a0 tx=0x7f60440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ac3ed8f5f48f13 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f604bfff700 1 -- 172.21.15.111:0/193991188 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400b6b0 con 0x7f60540cab10 2023-12-19T07:47:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f604bfff700 1 -- 172.21.15.111:0/193991188 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604400b810 con 0x7f60540cab10 2023-12-19T07:47:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f604bfff700 1 -- 172.21.15.111:0/193991188 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400bbb0 con 0x7f60540cab10 2023-12-19T07:47:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 -- 172.21.15.111:0/193991188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60540cab10 msgr2=0x7f60540ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 --2- 172.21.15.111:0/193991188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60540cab10 0x7f60540ab6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f605406a5a0 tx=0x7f60440091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 -- 172.21.15.111:0/193991188 shutdown_connections 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 --2- 172.21.15.111:0/193991188 >> v2:172.21.15.143:3300/0 conn(0x7f605406b1b0 0x7f605406b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 --2- 172.21.15.111:0/193991188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60540cab10 0x7f60540ab6c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 -- 172.21.15.111:0/193991188 >> 172.21.15.111:0/193991188 conn(0x7f6054067bb0 msgr2=0x7f60540cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 -- 172.21.15.111:0/193991188 shutdown_connections 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.888+0000 7f605b278700 1 -- 172.21.15.111:0/193991188 wait complete. 2023-12-19T07:47:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 Processor -- start 2023-12-19T07:47:39.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 -- start start 2023-12-19T07:47:39.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f605406b1b0 0x7f6054162110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60540cab10 0x7f6054162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054159a70 con 0x7f605415d410 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054163ae0 con 0x7f605406b1b0 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f605b278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6054162ed0 con 0x7f60540cab10 2023-12-19T07:47:39.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35000/0 (socket says 172.21.15.111:35000) 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059014700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f605406b1b0 0x7f6054162110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 -- 172.21.15.111:0/2767823124 learned_addr learned my addr 172.21.15.111:0/2767823124 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6058813700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60540cab10 0x7f6054162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 -- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60540cab10 msgr2=0x7f6054162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60540cab10 0x7f6054162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 -- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f605406b1b0 msgr2=0x7f6054162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:39.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f605406b1b0 0x7f6054162110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:39.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.892+0000 7f6059815700 1 -- 172.21.15.111:0/2767823124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044009420 con 0x7f605415d410 2023-12-19T07:47:39.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f6059815700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f604c007f20 tx=0x7f604c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c00eb90 con 0x7f605415d410 2023-12-19T07:47:39.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c00ed10 con 0x7f605415d410 2023-12-19T07:47:39.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60541631b0 con 0x7f605415d410 2023-12-19T07:47:39.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c014670 con 0x7f605415d410 2023-12-19T07:47:39.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6054163770 con 0x7f605415d410 2023-12-19T07:47:39.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6054163c50 con 0x7f605415d410 2023-12-19T07:47:39.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.896+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f604c0149e0 con 0x7f605415d410 2023-12-19T07:47:39.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.900+0000 7f60497fa700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 0x7f6038062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:39.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.900+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f604c07d130 con 0x7f605415d410 2023-12-19T07:47:39.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.900+0000 7f6059014700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 0x7f6038062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:39.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.900+0000 7f6059014700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 0x7f6038062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6050007fd0 tx=0x7f6050007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:39.908+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f604c04d390 con 0x7f605415d410 2023-12-19T07:47:40.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.076+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6054071120 con 0x7f605415d410 2023-12-19T07:47:40.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.076+0000 7f60497fa700 1 -- 172.21.15.111:0/2767823124 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f604c0509b0 con 0x7f605415d410 2023-12-19T07:47:40.060 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 msgr2=0x7f6038062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 0x7f6038062a00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6050007fd0 tx=0x7f6050007d30 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 msgr2=0x7f605415e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f604c007f20 tx=0x7f604c009860 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 shutdown_connections 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6038060550 0x7f6038062a00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605415d410 0x7f605415e7b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60540cab10 0x7f6054162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 --2- 172.21.15.111:0/2767823124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f605406b1b0 0x7f6054162110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 >> 172.21.15.111:0/2767823124 conn(0x7f6054067bb0 msgr2=0x7f605415bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 shutdown_connections 2023-12-19T07:47:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.080+0000 7f605b278700 1 -- 172.21.15.111:0/2767823124 wait complete. 2023-12-19T07:47:40.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:47:40.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.280+0000 7f54f2278700 1 Processor -- start 2023-12-19T07:47:40.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.280+0000 7f54f2278700 1 -- start start 2023-12-19T07:47:40.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f2278700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f54ec158e20 0x7f54ec0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f2278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 0x7f54ec0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f2278700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ec0cbdf0 con 0x7f54ec0ca2c0 2023-12-19T07:47:40.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f2278700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ec0cc0d0 con 0x7f54ec158e20 2023-12-19T07:47:40.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f2278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec0cc3b0 con 0x7f54ec0cb970 2023-12-19T07:47:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f0815700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f54ec0ca2c0 0x7f54ec06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56224/0 (socket says 172.21.15.111:56224) 2023-12-19T07:47:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54f0815700 1 -- 172.21.15.111:0/761609056 learned_addr learned my addr 172.21.15.111:0/761609056 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54ebfff700 1 --2- 172.21.15.111:0/761609056 >> v2:172.21.15.143:3300/0 conn(0x7f54ec158e20 0x7f54ec0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 --2- 172.21.15.111:0/761609056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 0x7f54ec0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54eb7fe700 1 -- 172.21.15.111:0/761609056 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618796505 0 0) 0x7f54ec0cbdf0 con 0x7f54ec0ca2c0 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54eb7fe700 1 -- 172.21.15.111:0/761609056 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8003570 con 0x7f54ec0ca2c0 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 -- 172.21.15.111:0/761609056 >> v2:172.21.15.143:3300/0 conn(0x7f54ec158e20 msgr2=0x7f54ec0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 --2- 172.21.15.111:0/761609056 >> v2:172.21.15.143:3300/0 conn(0x7f54ec158e20 0x7f54ec0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 -- 172.21.15.111:0/761609056 >> v1:172.21.15.111:6789/0 conn(0x7f54ec0ca2c0 legacy=0x7f54ec06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 -- 172.21.15.111:0/761609056 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec0cc690 con 0x7f54ec0cb970 2023-12-19T07:47:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54e37fe700 1 --2- 172.21.15.111:0/761609056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 0x7f54ec0ceb00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098e0 tx=0x7f54d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16416d69e9ea095 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54eb7fe700 1 -- 172.21.15.111:0/761609056 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4011630 con 0x7f54ec0cb970 2023-12-19T07:47:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54eb7fe700 1 -- 172.21.15.111:0/761609056 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4011790 con 0x7f54ec0cb970 2023-12-19T07:47:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.284+0000 7f54eb7fe700 1 -- 172.21.15.111:0/761609056 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4011b30 con 0x7f54ec0cb970 2023-12-19T07:47:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 -- 172.21.15.111:0/761609056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 msgr2=0x7f54ec0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 --2- 172.21.15.111:0/761609056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 0x7f54ec0ceb00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098e0 tx=0x7f54d40091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 -- 172.21.15.111:0/761609056 shutdown_connections 2023-12-19T07:47:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 --2- 172.21.15.111:0/761609056 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec0cb970 0x7f54ec0ceb00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 --2- 172.21.15.111:0/761609056 >> v2:172.21.15.143:3300/0 conn(0x7f54ec158e20 0x7f54ec0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 -- 172.21.15.111:0/761609056 >> 172.21.15.111:0/761609056 conn(0x7f54ec067bb0 msgr2=0x7f54ec0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 -- 172.21.15.111:0/761609056 shutdown_connections 2023-12-19T07:47:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 -- 172.21.15.111:0/761609056 wait complete. 2023-12-19T07:47:40.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.288+0000 7f54f2278700 1 Processor -- start 2023-12-19T07:47:40.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 -- start start 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 0x7f54ec16d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec158e20 0x7f54ec145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 0x7f54ec0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec169cd0 con 0x7f54ec16a160 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec0b5d80 con 0x7f54ec0ca2c0 2023-12-19T07:47:40.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f2278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec0b5aa0 con 0x7f54ec158e20 2023-12-19T07:47:40.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 0x7f54ec0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54ebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 0x7f54ec16d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54ebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 0x7f54ec16d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:48146/0 (socket says 172.21.15.111:48146) 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54ebfff700 1 -- 172.21.15.111:0/837367840 learned_addr learned my addr 172.21.15.111:0/837367840 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54e37fe700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec158e20 0x7f54ec145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 -- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec158e20 msgr2=0x7f54ec145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec158e20 0x7f54ec145aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 -- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 msgr2=0x7f54ec16d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 0x7f54ec16d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 -- 172.21.15.111:0/837367840 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4009420 con 0x7f54ec16a160 2023-12-19T07:47:40.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.292+0000 7f54f0815700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 0x7f54ec0cce80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0cf0d0 tx=0x7f54e4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e400a830 con 0x7f54ec16a160 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e400a990 con 0x7f54ec16a160 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e400ad10 con 0x7f54ec16a160 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ec169680 con 0x7f54ec16a160 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ec16d9e0 con 0x7f54ec16a160 2023-12-19T07:47:40.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.296+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ec1694e0 con 0x7f54ec16a160 2023-12-19T07:47:40.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.300+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f54e4007300 con 0x7f54ec16a160 2023-12-19T07:47:40.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.300+0000 7f54e8ff9700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 0x7f54cc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.300+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f54e407cdf0 con 0x7f54ec16a160 2023-12-19T07:47:40.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.300+0000 7f54ebfff700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 0x7f54cc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.300+0000 7f54ebfff700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 0x7f54cc062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009dd0 tx=0x7f54dc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.304+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54ec1694e0 con 0x7f54ec16a160 2023-12-19T07:47:40.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.476+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54ec1694e0 con 0x7f54ec16a160 2023-12-19T07:47:40.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.476+0000 7f54e8ff9700 1 -- 172.21.15.111:0/837367840 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f54ec1694e0 con 0x7f54ec16a160 2023-12-19T07:47:40.461 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:47:40.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 msgr2=0x7f54cc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 0x7f54cc062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009dd0 tx=0x7f54dc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 msgr2=0x7f54ec0cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 0x7f54ec0cce80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0cf0d0 tx=0x7f54e4001a40 comp rx=0 tx=0).stop 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 shutdown_connections 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54cc060550 0x7f54cc062a00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54ec16a160 0x7f54ec0cce80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f54ec158e20 0x7f54ec145aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 --2- 172.21.15.111:0/837367840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54ec0ca2c0 0x7f54ec16d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 >> 172.21.15.111:0/837367840 conn(0x7f54ec067bb0 msgr2=0x7f54ec0c64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:40.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 shutdown_connections 2023-12-19T07:47:40.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.484+0000 7f54f2278700 1 -- 172.21.15.111:0/837367840 wait complete. 2023-12-19T07:47:40.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:47:40.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.684+0000 7f7c8da2e700 1 Processor -- start 2023-12-19T07:47:40.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.684+0000 7f7c8da2e700 1 -- start start 2023-12-19T07:47:40.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c8da2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c88158e90 0x7f7c88159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c8da2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c881598a0 con 0x7f7c880afab0 2023-12-19T07:47:40.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c8da2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88159b80 con 0x7f7c8806b770 2023-12-19T07:47:40.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c8da2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c88159e60 con 0x7f7c88158e90 2023-12-19T07:47:40.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c88158e90 0x7f7c88159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c88158e90 0x7f7c88159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35014/0 (socket says 172.21.15.111:35014) 2023-12-19T07:47:40.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c867fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7c880afab0 0x7f7c8806b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56232/0 (socket says 172.21.15.111:56232) 2023-12-19T07:47:40.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 -- 172.21.15.111:0/686810243 learned_addr learned my addr 172.21.15.111:0/686810243 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:40.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 -- 172.21.15.111:0/686810243 >> v1:172.21.15.143:6789/0 conn(0x7f7c8806b770 legacy=0x7f7c8806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 -- 172.21.15.111:0/686810243 >> v1:172.21.15.111:6789/0 conn(0x7f7c880afab0 legacy=0x7f7c8806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 -- 172.21.15.111:0/686810243 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8815a140 con 0x7f7c88158e90 2023-12-19T07:47:40.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c86ffd700 1 --2- 172.21.15.111:0/686810243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c88158e90 0x7f7c88159260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7c780099b0 tx=0x7f7c78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7853d6be2f21523d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c85ffb700 1 -- 172.21.15.111:0/686810243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800b860 con 0x7f7c88158e90 2023-12-19T07:47:40.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c85ffb700 1 -- 172.21.15.111:0/686810243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7800b9c0 con 0x7f7c88158e90 2023-12-19T07:47:40.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.688+0000 7f7c85ffb700 1 -- 172.21.15.111:0/686810243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800bd60 con 0x7f7c88158e90 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- 172.21.15.111:0/686810243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c88158e90 msgr2=0x7f7c88159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/686810243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c88158e90 0x7f7c88159260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7c780099b0 tx=0x7f7c78009250 comp rx=0 tx=0).stop 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- 172.21.15.111:0/686810243 shutdown_connections 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/686810243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c88158e90 0x7f7c88159260 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- 172.21.15.111:0/686810243 >> 172.21.15.111:0/686810243 conn(0x7f7c880cc580 msgr2=0x7f7c880cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- 172.21.15.111:0/686810243 shutdown_connections 2023-12-19T07:47:40.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- 172.21.15.111:0/686810243 wait complete. 2023-12-19T07:47:40.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 Processor -- start 2023-12-19T07:47:40.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- start start 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 0x7f7c8814eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c8814f080 0x7f7c881513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8814f500 con 0x7f7c8806b770 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880ca0c0 con 0x7f7c880afab0 2023-12-19T07:47:40.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.692+0000 7f7c8da2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880ca360 con 0x7f7c8814f080 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35024/0 (socket says 172.21.15.111:35024) 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 -- 172.21.15.111:0/270571937 learned_addr learned my addr 172.21.15.111:0/270571937 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c867fc700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 0x7f7c8814eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c877fe700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c8814f080 0x7f7c881513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 -- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c8814f080 msgr2=0x7f7c881513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c8814f080 0x7f7c881513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 -- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 msgr2=0x7f7c8814eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 0x7f7c8814eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 -- 172.21.15.111:0/270571937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780094f0 con 0x7f7c8806b770 2023-12-19T07:47:40.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c867fc700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 0x7f7c8814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:40.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c86ffd700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7c881597d0 tx=0x7f7c78017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78024440 con 0x7f7c8806b770 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c780245a0 con 0x7f7c8806b770 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78024960 con 0x7f7c8806b770 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c880ca640 con 0x7f7c8806b770 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.696+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c880cac00 con 0x7f7c8806b770 2023-12-19T07:47:40.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.700+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88071120 con 0x7f7c8806b770 2023-12-19T07:47:40.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.700+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7c7801a420 con 0x7f7c8806b770 2023-12-19T07:47:40.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.700+0000 7f7c6effd700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 0x7f7c74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:40.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.700+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7c7807e7a0 con 0x7f7c8806b770 2023-12-19T07:47:40.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.704+0000 7f7c867fc700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 0x7f7c74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:40.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.704+0000 7f7c867fc700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 0x7f7c74062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c007fd0 tx=0x7f7c7c007e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:40.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.708+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c7804ea00 con 0x7f7c8806b770 2023-12-19T07:47:40.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.880+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7c88155420 con 0x7f7c8806b770 2023-12-19T07:47:40.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.880+0000 7f7c6effd700 1 -- 172.21.15.111:0/270571937 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7c78052020 con 0x7f7c8806b770 2023-12-19T07:47:40.864 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:40.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 msgr2=0x7f7c74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 0x7f7c74062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c007fd0 tx=0x7f7c7c007e10 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 msgr2=0x7f7c880c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7c881597d0 tx=0x7f7c78017000 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 shutdown_connections 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c74060550 0x7f7c74062a00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c880afab0 0x7f7c8814eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c8814f080 0x7f7c881513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 --2- 172.21.15.111:0/270571937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8806b770 0x7f7c880c9b80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 >> 172.21.15.111:0/270571937 conn(0x7f7c880cc580 msgr2=0x7f7c8806b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 shutdown_connections 2023-12-19T07:47:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:40.884+0000 7f7c8da2e700 1 -- 172.21.15.111:0/270571937 wait complete. 2023-12-19T07:47:40.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.056+0000 7fcbbc86e700 1 Processor -- start 2023-12-19T07:47:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 -- start start 2023-12-19T07:47:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcbb813f890 0x7fcbb80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb8068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb806c9f0 con 0x7fcbb80c2980 2023-12-19T07:47:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb80c7f20 con 0x7fcbb813f890 2023-12-19T07:47:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbbc86e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb8068a50 con 0x7fcbb80680f0 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbadd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcbb80c2980 0x7fcbb80c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56242/0 (socket says 172.21.15.111:56242) 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.060+0000 7fcbadd9b700 1 -- 172.21.15.111:0/1421446834 learned_addr learned my addr 172.21.15.111:0/1421446834 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 --2- 172.21.15.111:0/1421446834 >> v2:172.21.15.143:3300/0 conn(0x7fcbb813f890 0x7fcbb80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb6d9d700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb8068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb5d9b700 1 -- 172.21.15.111:0/1421446834 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690011759 0 0) 0x7fcbb806c9f0 con 0x7fcbb80c2980 2023-12-19T07:47:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb5d9b700 1 -- 172.21.15.111:0/1421446834 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba4003570 con 0x7fcbb80c2980 2023-12-19T07:47:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 -- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 msgr2=0x7fcbb8068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb8068510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 -- 172.21.15.111:0/1421446834 >> v1:172.21.15.111:6789/0 conn(0x7fcbb80c2980 legacy=0x7fcbb80c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 -- 172.21.15.111:0/1421446834 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb80c1c50 con 0x7fcbb813f890 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb659c700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb80aeba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009e60 tx=0x7fcba8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7fd6d5fbe3d3c34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb5d9b700 1 -- 172.21.15.111:0/1421446834 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba800e070 con 0x7fcbb813f890 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb5d9b700 1 -- 172.21.15.111:0/1421446834 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba800bab0 con 0x7fcbb813f890 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbb5d9b700 1 -- 172.21.15.111:0/1421446834 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba800be70 con 0x7fcbb813f890 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 -- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 msgr2=0x7fcbb80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb80aeba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009e60 tx=0x7fcba8009780 comp rx=0 tx=0).stop 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 -- 172.21.15.111:0/1421446834 shutdown_connections 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb8068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/1421446834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb80aeba0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 -- 172.21.15.111:0/1421446834 >> 172.21.15.111:0/1421446834 conn(0x7fcbb80c8850 msgr2=0x7fcbb80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:41.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 -- 172.21.15.111:0/1421446834 shutdown_connections 2023-12-19T07:47:41.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.064+0000 7fcbbc86e700 1 -- 172.21.15.111:0/1421446834 wait complete. 2023-12-19T07:47:41.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 Processor -- start 2023-12-19T07:47:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 -- start start 2023-12-19T07:47:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb813e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb8169180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb813a2d0 con 0x7fcbb80c2980 2023-12-19T07:47:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb80c5dc0 con 0x7fcbb813f890 2023-12-19T07:47:41.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbbc86e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb813a4e0 con 0x7fcbb80680f0 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbadd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbadd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35038/0 (socket says 172.21.15.111:35038) 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbadd9b700 1 -- 172.21.15.111:0/274560589 learned_addr learned my addr 172.21.15.111:0/274560589 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbb6d9d700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb8169180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.068+0000 7fcbb659c700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb813e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 -- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 msgr2=0x7fcbb813e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb813e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 -- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 msgr2=0x7fcbb8169180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb8169180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 -- 172.21.15.111:0/274560589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba8009a20 con 0x7fcbb80c2980 2023-12-19T07:47:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbb6d9d700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb8169180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbadd9b700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcba000c7c0 tx=0x7fcba0007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0005c00 con 0x7fcbb80c2980 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0005d60 con 0x7fcbb80c2980 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb813a7c0 con 0x7fcbb80c2980 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0007320 con 0x7fcbb80c2980 2023-12-19T07:47:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb80c62b0 con 0x7fcbb80c2980 2023-12-19T07:47:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.072+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb80c64d0 con 0x7fcbb80c2980 2023-12-19T07:47:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.076+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fcba001b3b0 con 0x7fcbb80c2980 2023-12-19T07:47:41.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.076+0000 7fcbaeffd700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 0x7fcb98062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.076+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fcba007bfe0 con 0x7fcbb80c2980 2023-12-19T07:47:41.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.076+0000 7fcbb659c700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 0x7fcb98062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.076+0000 7fcbb659c700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 0x7fcb98062a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097c0 tx=0x7fcba8019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.080+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcba004c280 con 0x7fcbb80c2980 2023-12-19T07:47:41.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.288+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbb8071120 con 0x7fcbb80c2980 2023-12-19T07:47:41.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbaeffd700 1 -- 172.21.15.111:0/274560589 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7fcba0017090 con 0x7fcbb80c2980 2023-12-19T07:47:41.273 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:41.274 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 msgr2=0x7fcb98062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 0x7fcb98062a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097c0 tx=0x7fcba8019000 comp rx=0 tx=0).stop 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 msgr2=0x7fcbb80c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcba000c7c0 tx=0x7fcba0007dd0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 shutdown_connections 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcb98060550 0x7fcb98062a00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbb813f890 0x7fcbb8169180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbb80c2980 0x7fcbb80c3850 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 --2- 172.21.15.111:0/274560589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbb80680f0 0x7fcbb813e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 >> 172.21.15.111:0/274560589 conn(0x7fcbb80c8850 msgr2=0x7fcbb80ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:41.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 shutdown_connections 2023-12-19T07:47:41.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.292+0000 7fcbbc86e700 1 -- 172.21.15.111:0/274560589 wait complete. 2023-12-19T07:47:41.278 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:47:41.300 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:41.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.468+0000 7f17bc2cd700 1 Processor -- start 2023-12-19T07:47:41.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 -- start start 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 0x7f17b40c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17b4140b80 0x7f17b40cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40b0fe0 0x7f17b40b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b40b1980 con 0x7f17b4140b80 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b406cb90 con 0x7f17b41407b0 2023-12-19T07:47:41.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17bc2cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b406c860 con 0x7f17b40b0fe0 2023-12-19T07:47:41.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17b9868700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17b4140b80 0x7f17b40cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17b9868700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17b4140b80 0x7f17b40cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35048/0 (socket says 172.21.15.111:35048) 2023-12-19T07:47:41.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17b9868700 1 -- 172.21.15.111:0/771483067 learned_addr learned my addr 172.21.15.111:0/771483067 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:41.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17ba069700 1 --2- 172.21.15.111:0/771483067 >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 0x7f17b40c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.472+0000 7f17ba86a700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40b0fe0 0x7f17b40b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 -- 172.21.15.111:0/771483067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40b0fe0 msgr2=0x7f17b40b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:41.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40b0fe0 0x7f17b40b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 -- 172.21.15.111:0/771483067 >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 msgr2=0x7f17b40c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 --2- 172.21.15.111:0/771483067 >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 0x7f17b40c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 -- 172.21.15.111:0/771483067 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b40c8ff0 con 0x7f17b4140b80 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17ba069700 1 --2- 172.21.15.111:0/771483067 >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 0x7f17b40c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9868700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b4140b80 0x7f17b40cb790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17b40b1ec0 tx=0x7f17a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6198728538de540 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9067700 1 -- 172.21.15.111:0/771483067 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a400b640 con 0x7f17b4140b80 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9067700 1 -- 172.21.15.111:0/771483067 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a400b7a0 con 0x7f17b4140b80 2023-12-19T07:47:41.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17b9067700 1 -- 172.21.15.111:0/771483067 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a400bb60 con 0x7f17b4140b80 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 -- 172.21.15.111:0/771483067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b4140b80 msgr2=0x7f17b40cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b4140b80 0x7f17b40cb790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17b40b1ec0 tx=0x7f17a4009250 comp rx=0 tx=0).stop 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 -- 172.21.15.111:0/771483067 shutdown_connections 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b40b0fe0 0x7f17b40b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/771483067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b4140b80 0x7f17b40cb790 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/771483067 >> v2:172.21.15.143:3300/0 conn(0x7f17b41407b0 0x7f17b40c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 -- 172.21.15.111:0/771483067 >> 172.21.15.111:0/771483067 conn(0x7f17b40cc9e0 msgr2=0x7f17b40ad920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 -- 172.21.15.111:0/771483067 shutdown_connections 2023-12-19T07:47:41.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.476+0000 7f17bc2cd700 1 -- 172.21.15.111:0/771483067 wait complete. 2023-12-19T07:47:41.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 Processor -- start 2023-12-19T07:47:41.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 -- start start 2023-12-19T07:47:41.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b41407b0 0x7f17b415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b4140b80 0x7f17b40a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b413d210 con 0x7f17b40b0fe0 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b4169e90 con 0x7f17b4140b80 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17bc2cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40aa810 con 0x7f17b41407b0 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35050/0 (socket says 172.21.15.111:35050) 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 -- 172.21.15.111:0/2307401545 learned_addr learned my addr 172.21.15.111:0/2307401545 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17b9868700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b41407b0 0x7f17b415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba86a700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b4140b80 0x7f17b40a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 -- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b41407b0 msgr2=0x7f17b415e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b41407b0 0x7f17b415e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 -- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b4140b80 msgr2=0x7f17b40a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b4140b80 0x7f17b40a9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 -- 172.21.15.111:0/2307401545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0009420 con 0x7f17b40b0fe0 2023-12-19T07:47:41.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17ba069700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17b000ec80 tx=0x7f17b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.480+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b000ca60 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b000cbc0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a40094f0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0013470 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b40aadd0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b4071120 con 0x7f17b40b0fe0 2023-12-19T07:47:41.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.484+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f17b0013820 con 0x7f17b40b0fe0 2023-12-19T07:47:41.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.488+0000 7f17aa7fc700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 0x7f1798062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.488+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f17b007cfe0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.488+0000 7f17b9868700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 0x7f1798062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.488+0000 7f17b9868700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 0x7f1798062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f17b40b1ec0 tx=0x7f17a4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.492+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f17b004d280 con 0x7f17b40b0fe0 2023-12-19T07:47:41.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.664+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17b40c92d0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17aa7fc700 1 -- 172.21.15.111:0/2307401545 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f17b00508a0 con 0x7f17b40b0fe0 2023-12-19T07:47:41.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:41.651 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:41.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 msgr2=0x7f1798062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 0x7f1798062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f17b40b1ec0 tx=0x7f17a4017000 comp rx=0 tx=0).stop 2023-12-19T07:47:41.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 msgr2=0x7f17b415e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17b000ec80 tx=0x7f17b00091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 shutdown_connections 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1798060550 0x7f1798062a00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17b4140b80 0x7f17b40a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17b41407b0 0x7f17b415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 --2- 172.21.15.111:0/2307401545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17b40b0fe0 0x7f17b415e0b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.668+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 >> 172.21.15.111:0/2307401545 conn(0x7f17b40cc9e0 msgr2=0x7f17b413e780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:41.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.672+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 shutdown_connections 2023-12-19T07:47:41.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.672+0000 7f17bc2cd700 1 -- 172.21.15.111:0/2307401545 wait complete. 2023-12-19T07:47:41.676 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:47:41.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.864+0000 7f0072478700 1 Processor -- start 2023-12-19T07:47:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.864+0000 7f0072478700 1 -- start start 2023-12-19T07:47:41.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0072478700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f006c06c7c0 0x7f006c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0072478700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f006c06ac30 0x7f006c06b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0072478700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c06a7a0 con 0x7f006c0c6d20 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0072478700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c06d4b0 con 0x7f006c06ac30 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0072478700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c06d790 con 0x7f006c06c7c0 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f006c06ac30 0x7f006c06b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f006c0c6d20 0x7f006c0c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43674/0 (socket says 172.21.15.111:43674) 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f006c06ac30 0x7f006c06b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38060/0 (socket says 172.21.15.111:38060) 2023-12-19T07:47:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006bfff700 1 -- 172.21.15.111:0/1025214787 learned_addr learned my addr 172.21.15.111:0/1025214787 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:41.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006b7fe700 1 --2- 172.21.15.111:0/1025214787 >> v2:172.21.15.143:3300/0 conn(0x7f006c06c7c0 0x7f006c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006affd700 1 -- 172.21.15.111:0/1025214787 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319837255 0 0) 0x7f006c06a7a0 con 0x7f006c0c6d20 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006affd700 1 -- 172.21.15.111:0/1025214787 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0054003570 con 0x7f006c0c6d20 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 -- 172.21.15.111:0/1025214787 >> v2:172.21.15.143:3300/0 conn(0x7f006c06c7c0 msgr2=0x7f006c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 --2- 172.21.15.111:0/1025214787 >> v2:172.21.15.143:3300/0 conn(0x7f006c06c7c0 0x7f006c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 -- 172.21.15.111:0/1025214787 >> v1:172.21.15.111:6789/0 conn(0x7f006c0c6d20 legacy=0x7f006c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 -- 172.21.15.111:0/1025214787 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c06da70 con 0x7f006c06ac30 2023-12-19T07:47:41.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f0070a15700 1 --2- 172.21.15.111:0/1025214787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c06ac30 0x7f006c06b000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f006c0c86f0 tx=0x7f0064009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa64bc71cd8f4947 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006affd700 1 -- 172.21.15.111:0/1025214787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006400e070 con 0x7f006c06ac30 2023-12-19T07:47:41.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006affd700 1 -- 172.21.15.111:0/1025214787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006400bab0 con 0x7f006c06ac30 2023-12-19T07:47:41.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.868+0000 7f006affd700 1 -- 172.21.15.111:0/1025214787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006400be70 con 0x7f006c06ac30 2023-12-19T07:47:41.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 -- 172.21.15.111:0/1025214787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c06ac30 msgr2=0x7f006c06b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 --2- 172.21.15.111:0/1025214787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c06ac30 0x7f006c06b000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f006c0c86f0 tx=0x7f0064009780 comp rx=0 tx=0).stop 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 -- 172.21.15.111:0/1025214787 shutdown_connections 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 --2- 172.21.15.111:0/1025214787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c06ac30 0x7f006c06b000 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 --2- 172.21.15.111:0/1025214787 >> v2:172.21.15.143:3300/0 conn(0x7f006c06c7c0 0x7f006c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 -- 172.21.15.111:0/1025214787 >> 172.21.15.111:0/1025214787 conn(0x7f006c067bb0 msgr2=0x7f006c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 -- 172.21.15.111:0/1025214787 shutdown_connections 2023-12-19T07:47:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 -- 172.21.15.111:0/1025214787 wait complete. 2023-12-19T07:47:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.872+0000 7f0072478700 1 Processor -- start 2023-12-19T07:47:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- start start 2023-12-19T07:47:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f006c06ac30 0x7f006c14f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 0x7f006c15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0c9bb0 con 0x7f006c0c6d20 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c14fe50 con 0x7f006c06ac30 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0ad590 con 0x7f006c06c7c0 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38072/0 (socket says 172.21.15.111:38072) 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 -- 172.21.15.111:0/4019601601 learned_addr learned my addr 172.21.15.111:0/4019601601 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f006bfff700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f006c06ac30 0x7f006c14f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f006b7fe700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 0x7f006c15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 -- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 msgr2=0x7f006c15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 0x7f006c15ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 -- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f006c06ac30 msgr2=0x7f006c14f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f006c06ac30 0x7f006c14f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 -- 172.21.15.111:0/4019601601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0064009a20 con 0x7f006c0c6d20 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f006b7fe700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 0x7f006c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0070a15700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f006c069e00 tx=0x7f0064019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064012e80 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064027470 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c0acc60 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064027830 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.876+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c0ad220 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.880+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c0ad720 con 0x7f006c0c6d20 2023-12-19T07:47:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.880+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0064024070 con 0x7f006c0c6d20 2023-12-19T07:47:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.884+0000 7f0053fff700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 0x7f0058062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.884+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f006407f0f0 con 0x7f006c0c6d20 2023-12-19T07:47:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.884+0000 7f006bfff700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 0x7f0058062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:41.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.884+0000 7f006bfff700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 0x7f0058062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f005c007fd0 tx=0x7f005c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:41.888+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f006404f3f0 con 0x7f006c0c6d20 2023-12-19T07:47:42.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.048+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f006c06dd50 con 0x7f006c0c6d20 2023-12-19T07:47:42.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.048+0000 7f0053fff700 1 -- 172.21.15.111:0/4019601601 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0064052a10 con 0x7f006c0c6d20 2023-12-19T07:47:42.032 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:42.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 msgr2=0x7f0058062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 0x7f0058062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f005c007fd0 tx=0x7f005c009040 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 msgr2=0x7f006c0ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f006c069e00 tx=0x7f0064019000 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 shutdown_connections 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0058060550 0x7f0058062a00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006c0c6d20 0x7f006c0ada20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f006c06c7c0 0x7f006c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 --2- 172.21.15.111:0/4019601601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f006c06ac30 0x7f006c14f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 >> 172.21.15.111:0/4019601601 conn(0x7f006c067bb0 msgr2=0x7f006c157990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:42.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 shutdown_connections 2023-12-19T07:47:42.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.052+0000 7f0072478700 1 -- 172.21.15.111:0/4019601601 wait complete. 2023-12-19T07:47:42.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:47:42.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.236+0000 7fd63f9c5700 1 Processor -- start 2023-12-19T07:47:42.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.236+0000 7fd63f9c5700 1 -- start start 2023-12-19T07:47:42.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63f9c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd638069a90 0x7fd6380b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63f9c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 0x7fd6380ab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63f9c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380ae2b0 con 0x7fd6380b1a50 2023-12-19T07:47:42.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63f9c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6381562b0 con 0x7fd638069a90 2023-12-19T07:47:42.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63f9c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b0ec0 con 0x7fd6380ca370 2023-12-19T07:47:42.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63df62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 0x7fd6380ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63cf60700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd638069a90 0x7fd6380b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63d761700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd6380b1a50 0x7fd6380b0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42500/0 (socket says 172.21.15.111:42500) 2023-12-19T07:47:42.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63d761700 1 -- 172.21.15.111:0/850814870 learned_addr learned my addr 172.21.15.111:0/850814870 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd62ffff700 1 -- 172.21.15.111:0/850814870 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759204345 0 0) 0x7fd6380ae2b0 con 0x7fd6380b1a50 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd62ffff700 1 -- 172.21.15.111:0/850814870 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd624003570 con 0x7fd6380b1a50 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63cf60700 1 -- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 msgr2=0x7fd6380ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63cf60700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 0x7fd6380ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63cf60700 1 -- 172.21.15.111:0/850814870 >> v1:172.21.15.143:6789/0 conn(0x7fd6380b1a50 legacy=0x7fd6380b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63cf60700 1 -- 172.21.15.111:0/850814870 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380ad980 con 0x7fd638069a90 2023-12-19T07:47:42.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.240+0000 7fd63df62700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 0x7fd6380ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:42.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63cf60700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380b1e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd628009910 tx=0x7fd6280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=805fd662a66aaf32 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd62ffff700 1 -- 172.21.15.111:0/850814870 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628011630 con 0x7fd638069a90 2023-12-19T07:47:42.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd62ffff700 1 -- 172.21.15.111:0/850814870 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628011790 con 0x7fd638069a90 2023-12-19T07:47:42.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd62ffff700 1 -- 172.21.15.111:0/850814870 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628011b30 con 0x7fd638069a90 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 -- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 msgr2=0x7fd6380b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380b1e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd628009910 tx=0x7fd6280091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 -- 172.21.15.111:0/850814870 shutdown_connections 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380ca370 0x7fd6380ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/850814870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380b1e20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 -- 172.21.15.111:0/850814870 >> 172.21.15.111:0/850814870 conn(0x7fd6380cd870 msgr2=0x7fd638069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:42.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 -- 172.21.15.111:0/850814870 shutdown_connections 2023-12-19T07:47:42.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.244+0000 7fd63f9c5700 1 -- 172.21.15.111:0/850814870 wait complete. 2023-12-19T07:47:42.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 Processor -- start 2023-12-19T07:47:42.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 -- start start 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380b1a50 0x7fd6380ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63816a8f0 con 0x7fd6380ca370 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63816ab40 con 0x7fd638069a90 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63f9c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63816ae20 con 0x7fd6380b1a50 2023-12-19T07:47:42.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38076/0 (socket says 172.21.15.111:38076) 2023-12-19T07:47:42.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 -- 172.21.15.111:0/2617932870 learned_addr learned my addr 172.21.15.111:0/2617932870 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63d761700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63cf60700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380b1a50 0x7fd6380ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 -- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380b1a50 msgr2=0x7fd6380ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380b1a50 0x7fd6380ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 -- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 msgr2=0x7fd6380c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.248+0000 7fd63df62700 1 -- 172.21.15.111:0/2617932870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630009a20 con 0x7fd6380ca370 2023-12-19T07:47:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd63df62700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd630009f20 tx=0x7fd63000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000cc80 con 0x7fd6380ca370 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63000f040 con 0x7fd6380ca370 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd628009420 con 0x7fd6380ca370 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630014480 con 0x7fd6380ca370 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63816b3e0 con 0x7fd6380ca370 2023-12-19T07:47:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63816b600 con 0x7fd6380ca370 2023-12-19T07:47:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.252+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd630003020 con 0x7fd6380ca370 2023-12-19T07:47:42.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.256+0000 7fd62d7fa700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 0x7fd61c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.256+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd63007d4b0 con 0x7fd6380ca370 2023-12-19T07:47:42.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.256+0000 7fd63d761700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 0x7fd61c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.256+0000 7fd63d761700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 0x7fd61c062a30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6340058d0 tx=0x7fd634009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.260+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd63004d710 con 0x7fd6380ca370 2023-12-19T07:47:42.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.428+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd63806f6d0 con 0x7fd6380ca370 2023-12-19T07:47:42.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd62d7fa700 1 -- 172.21.15.111:0/2617932870 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd630050d30 con 0x7fd6380ca370 2023-12-19T07:47:42.414 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 msgr2=0x7fd61c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 0x7fd61c062a30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6340058d0 tx=0x7fd634009040 comp rx=0 tx=0).stop 2023-12-19T07:47:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 msgr2=0x7fd6380aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd630009f20 tx=0x7fd63000b040 comp rx=0 tx=0).stop 2023-12-19T07:47:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 shutdown_connections 2023-12-19T07:47:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd61c060580 0x7fd61c062a30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380ca370 0x7fd6380aeb80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6380b1a50 0x7fd6380ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 --2- 172.21.15.111:0/2617932870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd638069a90 0x7fd6380c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 >> 172.21.15.111:0/2617932870 conn(0x7fd6380cd870 msgr2=0x7fd638154960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:42.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 shutdown_connections 2023-12-19T07:47:42.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.432+0000 7fd63f9c5700 1 -- 172.21.15.111:0/2617932870 wait complete. 2023-12-19T07:47:42.435 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:47:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.632+0000 7f7bf642e700 1 Processor -- start 2023-12-19T07:47:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.632+0000 7f7bf642e700 1 -- start start 2023-12-19T07:47:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bf642e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bf015d410 0x7f7bf00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bf642e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 0x7f7bf00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bf642e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf00c7450 con 0x7f7bf006b1b0 2023-12-19T07:47:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bf642e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf00cad50 con 0x7f7bf015d410 2023-12-19T07:47:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bf642e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf006bea0 con 0x7f7bf00ca400 2023-12-19T07:47:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7beffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bf015d410 0x7f7bf00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7beffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bf015d410 0x7f7bf00ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38086/0 (socket says 172.21.15.111:38086) 2023-12-19T07:47:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7beffff700 1 -- 172.21.15.111:0/685034318 learned_addr learned my addr 172.21.15.111:0/685034318 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.636+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 0x7f7bf00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beffff700 1 -- 172.21.15.111:0/685034318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 msgr2=0x7f7bf00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beffff700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 0x7f7bf00ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beffff700 1 -- 172.21.15.111:0/685034318 >> v1:172.21.15.143:6789/0 conn(0x7f7bf006b1b0 legacy=0x7f7bf006b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beffff700 1 -- 172.21.15.111:0/685034318 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00c6e00 con 0x7f7bf015d410 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 0x7f7bf00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beffff700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf015d410 0x7f7bf00ab6c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7be00099b0 tx=0x7f7be0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5d6f4b22f63d7c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beeffd700 1 -- 172.21.15.111:0/685034318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0009ea0 con 0x7f7bf015d410 2023-12-19T07:47:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beeffd700 1 -- 172.21.15.111:0/685034318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be000b860 con 0x7f7bf015d410 2023-12-19T07:47:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7beeffd700 1 -- 172.21.15.111:0/685034318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be000bc00 con 0x7f7bf015d410 2023-12-19T07:47:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 -- 172.21.15.111:0/685034318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf015d410 msgr2=0x7f7bf00ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf015d410 0x7f7bf00ab6c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7be00099b0 tx=0x7f7be0009250 comp rx=0 tx=0).stop 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 -- 172.21.15.111:0/685034318 shutdown_connections 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf00ca400 0x7f7bf00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 --2- 172.21.15.111:0/685034318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf015d410 0x7f7bf00ab6c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 -- 172.21.15.111:0/685034318 >> 172.21.15.111:0/685034318 conn(0x7f7bf0067bb0 msgr2=0x7f7bf00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 -- 172.21.15.111:0/685034318 shutdown_connections 2023-12-19T07:47:42.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.640+0000 7f7bf642e700 1 -- 172.21.15.111:0/685034318 wait complete. 2023-12-19T07:47:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.644+0000 7f7bf642e700 1 Processor -- start 2023-12-19T07:47:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.644+0000 7f7bf642e700 1 -- start start 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 0x7f7bf00b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 0x7f7bf00c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf014a200 con 0x7f7bf006b1b0 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00b1040 con 0x7f7bf00ca400 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00b1320 con 0x7f7bf015d410 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38102/0 (socket says 172.21.15.111:38102) 2023-12-19T07:47:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 -- 172.21.15.111:0/3596279418 learned_addr learned my addr 172.21.15.111:0/3596279418 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 0x7f7bf00b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf49cb700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 0x7f7bf00c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 -- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 msgr2=0x7f7bf00c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 0x7f7bf00c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 -- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 msgr2=0x7f7bf00b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 0x7f7bf00b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 -- 172.21.15.111:0/3596279418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be4009420 con 0x7f7bf006b1b0 2023-12-19T07:47:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf49cb700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 0x7f7bf00c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 0x7f7bf00b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7beffff700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7bf006a040 tx=0x7f7be0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be000bbe0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be0011420 con 0x7f7bf006b1b0 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be00094f0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00117c0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.648+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf015a0b0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.652+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0071120 con 0x7f7bf006b1b0 2023-12-19T07:47:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.652+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7be0011bf0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.656+0000 7f7bd7fff700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 0x7f7bdc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.656+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f7be007e470 con 0x7f7bf006b1b0 2023-12-19T07:47:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.656+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 0x7f7bdc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:42.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.656+0000 7f7bef7fe700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 0x7f7bdc062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7be40093f0 tx=0x7f7be4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:42.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.660+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7be004e6d0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7bf00c9000 con 0x7f7bf006b1b0 2023-12-19T07:47:42.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bd7fff700 1 -- 172.21.15.111:0/3596279418 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7be0051cf0 con 0x7f7bf006b1b0 2023-12-19T07:47:42.814 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:47:42.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 msgr2=0x7f7bdc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 0x7f7bdc062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7be40093f0 tx=0x7f7be4008040 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 msgr2=0x7f7bf0159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7bf006a040 tx=0x7f7be0017000 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 shutdown_connections 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bdc060550 0x7f7bdc062a00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf00ca400 0x7f7bf00b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf015d410 0x7f7bf00c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 --2- 172.21.15.111:0/3596279418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf006b1b0 0x7f7bf0159610 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.832+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 >> 172.21.15.111:0/3596279418 conn(0x7f7bf0067bb0 msgr2=0x7f7bf00c9780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.836+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 shutdown_connections 2023-12-19T07:47:42.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:42.836+0000 7f7bf642e700 1 -- 172.21.15.111:0/3596279418 wait complete. 2023-12-19T07:47:42.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:47:43.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.020+0000 7f68d9345700 1 Processor -- start 2023-12-19T07:47:43.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.020+0000 7f68d9345700 1 -- start start 2023-12-19T07:47:43.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d9345700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68d40cecd0 0x7f68d406b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d9345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40cbdf0 con 0x7f68d40cb970 2023-12-19T07:47:43.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d9345700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40cc0d0 con 0x7f68d4158e20 2023-12-19T07:47:43.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d9345700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d40cc3b0 con 0x7f68d40cecd0 2023-12-19T07:47:43.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d27fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f68d40cb970 0x7f68d406acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43690/0 (socket says 172.21.15.111:43690) 2023-12-19T07:47:43.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d27fc700 1 -- 172.21.15.111:0/130268856 learned_addr learned my addr 172.21.15.111:0/130268856 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:43.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d37fe700 1 --2- 172.21.15.111:0/130268856 >> v2:172.21.15.143:3300/0 conn(0x7f68d40cecd0 0x7f68d406b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490217849 0 0) 0x7f68d40cbdf0 con 0x7f68d40cb970 2023-12-19T07:47:43.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc003570 con 0x7f68d40cb970 2023-12-19T07:47:43.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578688868 0 0) 0x7f68d40cc0d0 con 0x7f68d4158e20 2023-12-19T07:47:43.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d40cbdf0 con 0x7f68d4158e20 2023-12-19T07:47:43.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d37fe700 1 -- 172.21.15.111:0/130268856 >> v1:172.21.15.143:6789/0 conn(0x7f68d4158e20 legacy=0x7f68d40ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d37fe700 1 -- 172.21.15.111:0/130268856 >> v1:172.21.15.111:6789/0 conn(0x7f68d40cb970 legacy=0x7f68d406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.024+0000 7f68d37fe700 1 -- 172.21.15.111:0/130268856 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40cc690 con 0x7f68d40cecd0 2023-12-19T07:47:43.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d37fe700 1 --2- 172.21.15.111:0/130268856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d406b3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e90 tx=0x7f68cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dfa0ef89ff21f61 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00e070 con 0x7f68d40cecd0 2023-12-19T07:47:43.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc00bb90 con 0x7f68d40cecd0 2023-12-19T07:47:43.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d1ffb700 1 -- 172.21.15.111:0/130268856 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc012430 con 0x7f68d40cecd0 2023-12-19T07:47:43.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 -- 172.21.15.111:0/130268856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 msgr2=0x7f68d406b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 --2- 172.21.15.111:0/130268856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d406b3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e90 tx=0x7f68cc009780 comp rx=0 tx=0).stop 2023-12-19T07:47:43.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 -- 172.21.15.111:0/130268856 shutdown_connections 2023-12-19T07:47:43.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 --2- 172.21.15.111:0/130268856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d406b3d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 -- 172.21.15.111:0/130268856 >> 172.21.15.111:0/130268856 conn(0x7f68d4067bb0 msgr2=0x7f68d40d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 -- 172.21.15.111:0/130268856 shutdown_connections 2023-12-19T07:47:43.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.028+0000 7f68d9345700 1 -- 172.21.15.111:0/130268856 wait complete. 2023-12-19T07:47:43.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 Processor -- start 2023-12-19T07:47:43.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 -- start start 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d414bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d4156410 0x7f68d40c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4156890 con 0x7f68d4158e20 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40c9d30 con 0x7f68d40cecd0 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d9345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d414fff0 con 0x7f68d4156410 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38108/0 (socket says 172.21.15.111:38108) 2023-12-19T07:47:43.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 -- 172.21.15.111:0/186807884 learned_addr learned my addr 172.21.15.111:0/186807884 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:43.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d2ffd700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d414bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d37fe700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d4156410 0x7f68d40c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 -- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d4156410 msgr2=0x7f68d40c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d4156410 0x7f68d40c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 -- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 msgr2=0x7f68d414bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d414bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 -- 172.21.15.111:0/186807884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc009a20 con 0x7f68d4158e20 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.032+0000 7f68d27fc700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f68c800c580 tx=0x7f68c8007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8005b30 con 0x7f68d4158e20 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8005cb0 con 0x7f68d4158e20 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8007320 con 0x7f68d4158e20 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d40c9120 con 0x7f68d4158e20 2023-12-19T07:47:43.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d40c96e0 con 0x7f68d4158e20 2023-12-19T07:47:43.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d40c9ee0 con 0x7f68d4158e20 2023-12-19T07:47:43.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.036+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f68c801c440 con 0x7f68d4158e20 2023-12-19T07:47:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.040+0000 7f68baffd700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 0x7f68c0062b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.040+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f68c807be70 con 0x7f68d4158e20 2023-12-19T07:47:43.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.040+0000 7f68d2ffd700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 0x7f68c0062b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.040+0000 7f68d2ffd700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 0x7f68c0062b00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f68c4007fd0 tx=0x7f68c4007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.044+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f68d40c9ee0 con 0x7f68d4158e20 2023-12-19T07:47:43.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.220+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f68d40c9ea0 con 0x7f68d4158e20 2023-12-19T07:47:43.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.220+0000 7f68baffd700 1 -- 172.21.15.111:0/186807884 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f68c804c1f0 con 0x7f68d4158e20 2023-12-19T07:47:43.204 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 msgr2=0x7f68c0062b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 0x7f68c0062b00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f68c4007fd0 tx=0x7f68c4007d50 comp rx=0 tx=0).stop 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 msgr2=0x7f68d4155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f68c800c580 tx=0x7f68c8007e20 comp rx=0 tx=0).stop 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 shutdown_connections 2023-12-19T07:47:43.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68c0060650 0x7f68c0062b00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d4156410 0x7f68d40c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68d4158e20 0x7f68d4155ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 --2- 172.21.15.111:0/186807884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d40cecd0 0x7f68d414bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 >> 172.21.15.111:0/186807884 conn(0x7f68d4067bb0 msgr2=0x7f68d414caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 shutdown_connections 2023-12-19T07:47:43.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.224+0000 7f68d9345700 1 -- 172.21.15.111:0/186807884 wait complete. 2023-12-19T07:47:43.228 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 Processor -- start 2023-12-19T07:47:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 -- start start 2023-12-19T07:47:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb064158e20 0x7fb06406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0641591f0 0x7fb0640ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb06415ca80 0x7fb0640ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06406b730 con 0x7fb06415ca80 2023-12-19T07:47:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06406ab20 con 0x7fb064158e20 2023-12-19T07:47:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06406ae00 con 0x7fb0641591f0 2023-12-19T07:47:43.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb06415ca80 0x7fb0640ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb06415ca80 0x7fb0640ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38114/0 (socket says 172.21.15.111:38114) 2023-12-19T07:47:43.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb06259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb064158e20 0x7fb06406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 -- 172.21.15.111:0/2400589854 learned_addr learned my addr 172.21.15.111:0/2400589854 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb061d9b700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0641591f0 0x7fb0640ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 -- 172.21.15.111:0/2400589854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0641591f0 msgr2=0x7fb0640ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0641591f0 0x7fb0640ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 -- 172.21.15.111:0/2400589854 >> v2:172.21.15.143:3300/0 conn(0x7fb064158e20 msgr2=0x7fb06406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 --2- 172.21.15.111:0/2400589854 >> v2:172.21.15.143:3300/0 conn(0x7fb064158e20 0x7fb06406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.400+0000 7fb062d9d700 1 -- 172.21.15.111:0/2400589854 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06406b0e0 con 0x7fb06415ca80 2023-12-19T07:47:43.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb062d9d700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0640ce640 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb06406e510 tx=0x7fb05c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ced17fc2fd776f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06159a700 1 -- 172.21.15.111:0/2400589854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00e070 con 0x7fb06415ca80 2023-12-19T07:47:43.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06159a700 1 -- 172.21.15.111:0/2400589854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c00b970 con 0x7fb06415ca80 2023-12-19T07:47:43.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06159a700 1 -- 172.21.15.111:0/2400589854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0092e0 con 0x7fb06415ca80 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 -- 172.21.15.111:0/2400589854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 msgr2=0x7fb0640ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0640ce640 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb06406e510 tx=0x7fb05c009840 comp rx=0 tx=0).stop 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 -- 172.21.15.111:0/2400589854 shutdown_connections 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0640ce640 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 --2- 172.21.15.111:0/2400589854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0641591f0 0x7fb0640ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 --2- 172.21.15.111:0/2400589854 >> v2:172.21.15.143:3300/0 conn(0x7fb064158e20 0x7fb06406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 -- 172.21.15.111:0/2400589854 >> 172.21.15.111:0/2400589854 conn(0x7fb064067bb0 msgr2=0x7fb0640cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 -- 172.21.15.111:0/2400589854 shutdown_connections 2023-12-19T07:47:43.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.404+0000 7fb06359e700 1 -- 172.21.15.111:0/2400589854 wait complete. 2023-12-19T07:47:43.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 Processor -- start 2023-12-19T07:47:43.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 -- start start 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb064158e20 0x7fb0640b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0641591f0 0x7fb064153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb064172090 con 0x7fb06415ca80 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640b2660 con 0x7fb0641591f0 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb06359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06416db30 con 0x7fb064158e20 2023-12-19T07:47:43.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb062d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb062d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38124/0 (socket says 172.21.15.111:38124) 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.408+0000 7fb062d9d700 1 -- 172.21.15.111:0/1622891932 learned_addr learned my addr 172.21.15.111:0/1622891932 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb061d9b700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0641591f0 0x7fb064153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb06259c700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb064158e20 0x7fb0640b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 -- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb064158e20 msgr2=0x7fb0640b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb064158e20 0x7fb0640b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 -- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0641591f0 msgr2=0x7fb064153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0641591f0 0x7fb064153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 -- 172.21.15.111:0/1622891932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c009ae0 con 0x7fb06415ca80 2023-12-19T07:47:43.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb062d9d700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb06406e810 tx=0x7fb05c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0032f0 con 0x7fb06415ca80 2023-12-19T07:47:43.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c008b10 con 0x7fb06415ca80 2023-12-19T07:47:43.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c01c410 con 0x7fb06415ca80 2023-12-19T07:47:43.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06416e390 con 0x7fb06415ca80 2023-12-19T07:47:43.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb064171e30 con 0x7fb06415ca80 2023-12-19T07:47:43.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.412+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06416e520 con 0x7fb06415ca80 2023-12-19T07:47:43.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.416+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb05c003040 con 0x7fb06415ca80 2023-12-19T07:47:43.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.416+0000 7fb0527fc700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 0x7fb044062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.416+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb05c07e0c0 con 0x7fb06415ca80 2023-12-19T07:47:43.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.416+0000 7fb06259c700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 0x7fb044062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.420+0000 7fb06259c700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 0x7fb044062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb054009db0 tx=0x7fb0540091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.424+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb05c04e3c0 con 0x7fb06415ca80 2023-12-19T07:47:43.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.624+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06404be40 con 0x7fb06415ca80 2023-12-19T07:47:43.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.624+0000 7fb0527fc700 1 -- 172.21.15.111:0/1622891932 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7fb05c009940 con 0x7fb06415ca80 2023-12-19T07:47:43.609 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:43.609 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 msgr2=0x7fb044062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 0x7fb044062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb054009db0 tx=0x7fb0540091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 msgr2=0x7fb0641542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb06406e810 tx=0x7fb05c018000 comp rx=0 tx=0).stop 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 shutdown_connections 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb044060550 0x7fb044062a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0641591f0 0x7fb064153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb06415ca80 0x7fb0641542d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 --2- 172.21.15.111:0/1622891932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb064158e20 0x7fb0640b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 >> 172.21.15.111:0/1622891932 conn(0x7fb064067bb0 msgr2=0x7fb064152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 shutdown_connections 2023-12-19T07:47:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.628+0000 7fb06359e700 1 -- 172.21.15.111:0/1622891932 wait complete. 2023-12-19T07:47:43.614 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:47:43.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:43.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.808+0000 7fc301dcc700 1 Processor -- start 2023-12-19T07:47:43.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.808+0000 7fc301dcc700 1 -- start start 2023-12-19T07:47:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc301dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 0x7fc2fc0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc301dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 0x7fc2fc15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc301dcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc06b470 con 0x7fc2fc06b900 2023-12-19T07:47:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc301dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2fc06e110 con 0x7fc2fc0c6900 2023-12-19T07:47:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc301dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc161d10 con 0x7fc2fc15cc20 2023-12-19T07:47:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc300dca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 0x7fc2fc0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 0x7fc2fc15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc2fbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc2fc06b900 0x7fc2fc06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42522/0 (socket says 172.21.15.111:42522) 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc2fbfff700 1 -- 172.21.15.111:0/862498444 learned_addr learned my addr 172.21.15.111:0/862498444 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 -- 172.21.15.111:0/862498444 >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 msgr2=0x7fc2fc0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 --2- 172.21.15.111:0/862498444 >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 0x7fc2fc0c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 -- 172.21.15.111:0/862498444 >> v1:172.21.15.143:6789/0 conn(0x7fc2fc06b900 legacy=0x7fc2fc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 -- 172.21.15.111:0/862498444 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc06bd70 con 0x7fc2fc15cc20 2023-12-19T07:47:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc300dca700 1 --2- 172.21.15.111:0/862498444 >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 0x7fc2fc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:43.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.812+0000 7fc3015cb700 1 --2- 172.21.15.111:0/862498444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 0x7fc2fc15d0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4009e60 tx=0x7fc2f4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f77fef2c061478 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc2fb7fe700 1 -- 172.21.15.111:0/862498444 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400e070 con 0x7fc2fc15cc20 2023-12-19T07:47:43.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc2fb7fe700 1 -- 172.21.15.111:0/862498444 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f400bb60 con 0x7fc2fc15cc20 2023-12-19T07:47:43.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc2fb7fe700 1 -- 172.21.15.111:0/862498444 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4012430 con 0x7fc2fc15cc20 2023-12-19T07:47:43.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 -- 172.21.15.111:0/862498444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 msgr2=0x7fc2fc15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 --2- 172.21.15.111:0/862498444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 0x7fc2fc15d0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4009e60 tx=0x7fc2f4009780 comp rx=0 tx=0).stop 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 -- 172.21.15.111:0/862498444 shutdown_connections 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 --2- 172.21.15.111:0/862498444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc15cc20 0x7fc2fc15d0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 --2- 172.21.15.111:0/862498444 >> v2:172.21.15.143:3300/0 conn(0x7fc2fc0c6900 0x7fc2fc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 -- 172.21.15.111:0/862498444 >> 172.21.15.111:0/862498444 conn(0x7fc2fc067bb0 msgr2=0x7fc2fc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 -- 172.21.15.111:0/862498444 shutdown_connections 2023-12-19T07:47:43.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 -- 172.21.15.111:0/862498444 wait complete. 2023-12-19T07:47:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.816+0000 7fc301dcc700 1 Processor -- start 2023-12-19T07:47:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 -- start start 2023-12-19T07:47:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 0x7fc2fc0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 0x7fc2fc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0b1830 con 0x7fc2fc06b900 2023-12-19T07:47:43.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0b1010 con 0x7fc2fc15cc20 2023-12-19T07:47:43.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc301dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc16d730 con 0x7fc2fc0c6900 2023-12-19T07:47:43.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38126/0 (socket says 172.21.15.111:38126) 2023-12-19T07:47:43.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 -- 172.21.15.111:0/2395049002 learned_addr learned my addr 172.21.15.111:0/2395049002 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc3015cb700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 0x7fc2fc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 0x7fc2fc0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 -- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 msgr2=0x7fc2fc0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 0x7fc2fc0cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 -- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 msgr2=0x7fc2fc1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 0x7fc2fc1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 -- 172.21.15.111:0/2395049002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0009420 con 0x7fc2fc06b900 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc3015cb700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 0x7fc2fc1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 0x7fc2fc0cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.820+0000 7fc300dca700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ec80 tx=0x7fc2f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000ca60 con 0x7fc2fc06b900 2023-12-19T07:47:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f000cbc0 con 0x7fc2fc06b900 2023-12-19T07:47:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0013470 con 0x7fc2fc06b900 2023-12-19T07:47:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4009a20 con 0x7fc2fc06b900 2023-12-19T07:47:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc0aee30 con 0x7fc2fc06b900 2023-12-19T07:47:43.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc04be40 con 0x7fc2fc06b900 2023-12-19T07:47:43.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.824+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc2f0013820 con 0x7fc2fc06b900 2023-12-19T07:47:43.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.828+0000 7fc2f8ff9700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 0x7fc2e0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:43.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.828+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc2f007cfa0 con 0x7fc2fc06b900 2023-12-19T07:47:43.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.828+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 0x7fc2e0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:43.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.828+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 0x7fc2e0062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc06ed20 tx=0x7fc2e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:43.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:43.832+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2f00acbe0 con 0x7fc2fc06b900 2023-12-19T07:47:43.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.004+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2fc06a580 con 0x7fc2fc06b900 2023-12-19T07:47:43.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.008+0000 7fc2f8ff9700 1 -- 172.21.15.111:0/2395049002 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7fc2f004d2a0 con 0x7fc2fc06b900 2023-12-19T07:47:43.991 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:43.991 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:43.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.008+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 msgr2=0x7fc2e0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.008+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 0x7fc2e0062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc06ed20 tx=0x7fc2e8009040 comp rx=0 tx=0).stop 2023-12-19T07:47:43.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.008+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 msgr2=0x7fc2fc0b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:43.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.008+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ec80 tx=0x7fc2f00091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 shutdown_connections 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e0060550 0x7fc2e0062a00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc15cc20 0x7fc2fc1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c6900 0x7fc2fc0cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 --2- 172.21.15.111:0/2395049002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc06b900 0x7fc2fc0b12f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 >> 172.21.15.111:0/2395049002 conn(0x7fc2fc067bb0 msgr2=0x7fc2fc15bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 shutdown_connections 2023-12-19T07:47:43.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.012+0000 7fc301dcc700 1 -- 172.21.15.111:0/2395049002 wait complete. 2023-12-19T07:47:44.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:47:44.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.188+0000 7f3dca88e700 1 Processor -- start 2023-12-19T07:47:44.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 -- start start 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 0x7f3dc406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3dc415cc20 0x7f3dc415d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc406b470 con 0x7f3dc40c6900 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc406be40 con 0x7f3dc415cc20 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca88e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc406e110 con 0x7f3dc40c6cd0 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3dc415cc20 0x7f3dc415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3dc415cc20 0x7f3dc415d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38132/0 (socket says 172.21.15.111:38132) 2023-12-19T07:47:44.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 -- 172.21.15.111:0/1497861994 learned_addr learned my addr 172.21.15.111:0/1497861994 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 -- 172.21.15.111:0/1497861994 >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 msgr2=0x7f3dc406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dc908b700 1 --2- 172.21.15.111:0/1497861994 >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 0x7f3dc406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 --2- 172.21.15.111:0/1497861994 >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 0x7f3dc406b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 -- 172.21.15.111:0/1497861994 >> v1:172.21.15.143:6789/0 conn(0x7f3dc40c6900 legacy=0x7f3dc406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 -- 172.21.15.111:0/1497861994 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc4161d10 con 0x7f3dc415cc20 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dc908b700 1 --2- 172.21.15.111:0/1497861994 >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 0x7f3dc406b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.192+0000 7f3dca08d700 1 --2- 172.21.15.111:0/1497861994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc415d080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3dc406a030 tx=0x7f3dbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c33ffc1e3231af server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dc888a700 1 -- 172.21.15.111:0/1497861994 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc00b860 con 0x7f3dc415cc20 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dc888a700 1 -- 172.21.15.111:0/1497861994 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc00b9c0 con 0x7f3dc415cc20 2023-12-19T07:47:44.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dc888a700 1 -- 172.21.15.111:0/1497861994 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc00bd60 con 0x7f3dc415cc20 2023-12-19T07:47:44.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 -- 172.21.15.111:0/1497861994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 msgr2=0x7f3dc415d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 --2- 172.21.15.111:0/1497861994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc415d080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3dc406a030 tx=0x7f3dbc009250 comp rx=0 tx=0).stop 2023-12-19T07:47:44.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 -- 172.21.15.111:0/1497861994 shutdown_connections 2023-12-19T07:47:44.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 --2- 172.21.15.111:0/1497861994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc415d080 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 --2- 172.21.15.111:0/1497861994 >> v2:172.21.15.143:3300/0 conn(0x7f3dc40c6cd0 0x7f3dc406b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 -- 172.21.15.111:0/1497861994 >> 172.21.15.111:0/1497861994 conn(0x7f3dc4067bb0 msgr2=0x7f3dc40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:44.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 -- 172.21.15.111:0/1497861994 shutdown_connections 2023-12-19T07:47:44.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 -- 172.21.15.111:0/1497861994 wait complete. 2023-12-19T07:47:44.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.196+0000 7f3dca88e700 1 Processor -- start 2023-12-19T07:47:44.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 -- start start 2023-12-19T07:47:44.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc40c6900 0x7f3dc40af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc40c6cd0 0x7f3dc415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b02c0 con 0x7f3dc415cc20 2023-12-19T07:47:44.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4160010 con 0x7f3dc40c6900 2023-12-19T07:47:44.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40af6b0 con 0x7f3dc40c6cd0 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38146/0 (socket says 172.21.15.111:38146) 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 -- 172.21.15.111:0/2199048392 learned_addr learned my addr 172.21.15.111:0/2199048392 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dc988c700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc40c6900 0x7f3dc40af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dc908b700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc40c6cd0 0x7f3dc415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 -- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc40c6cd0 msgr2=0x7f3dc415fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc40c6cd0 0x7f3dc415fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 -- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc40c6900 msgr2=0x7f3dc40af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc40c6900 0x7f3dc40af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 -- 172.21.15.111:0/2199048392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0094f0 con 0x7f3dc415cc20 2023-12-19T07:47:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca08d700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4064040 tx=0x7f3dbc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc011e80 con 0x7f3dc415cc20 2023-12-19T07:47:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc01a630 con 0x7f3dc415cc20 2023-12-19T07:47:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc01a9d0 con 0x7f3dc415cc20 2023-12-19T07:47:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.200+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc40af990 con 0x7f3dc415cc20 2023-12-19T07:47:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.204+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc40aff50 con 0x7f3dc415cc20 2023-12-19T07:47:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.204+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc40b0470 con 0x7f3dc415cc20 2023-12-19T07:47:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.204+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3dbc02d440 con 0x7f3dc415cc20 2023-12-19T07:47:44.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.208+0000 7f3db9ffb700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 0x7f3da8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:44.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.208+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3dbc07e630 con 0x7f3dc415cc20 2023-12-19T07:47:44.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.208+0000 7f3dc988c700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 0x7f3da8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:44.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.208+0000 7f3dc988c700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 0x7f3da8062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0007fd0 tx=0x7f3dc00016f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:44.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.212+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3dc40b0470 con 0x7f3dc415cc20 2023-12-19T07:47:44.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.416+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dc40b0470 con 0x7f3dc415cc20 2023-12-19T07:47:44.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.420+0000 7f3db9ffb700 1 -- 172.21.15.111:0/2199048392 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f3dc40b0470 con 0x7f3dc415cc20 2023-12-19T07:47:44.403 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:44.403 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.420+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 msgr2=0x7f3da8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.420+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 0x7f3da8062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0007fd0 tx=0x7f3dc00016f0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 msgr2=0x7f3dc4141270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4064040 tx=0x7f3dbc017000 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 shutdown_connections 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3da8060550 0x7f3da8062a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dc415cc20 0x7f3dc4141270 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc40c6cd0 0x7f3dc415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 --2- 172.21.15.111:0/2199048392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc40c6900 0x7f3dc40af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 >> 172.21.15.111:0/2199048392 conn(0x7f3dc4067bb0 msgr2=0x7f3dc4162560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 shutdown_connections 2023-12-19T07:47:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:44.424+0000 7f3dca88e700 1 -- 172.21.15.111:0/2199048392 wait complete. 2023-12-19T07:47:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:47:44.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T07:47:16.604129+0000', 'modified': '2023-12-19T07:47:32.042674+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6835/1214316551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1214316551}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1214316551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T07:47:44.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T07:47:44.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T07:47:45.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.016+0000 7f9621aae700 1 Processor -- start 2023-12-19T07:47:45.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.016+0000 7f9621aae700 1 -- start start 2023-12-19T07:47:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9621aae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f961c156ec0 0x7f961c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9621aae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9621aae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961c06d500 con 0x7f961c157290 2023-12-19T07:47:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9621aae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c06c8f0 con 0x7f961c156ec0 2023-12-19T07:47:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9621aae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c06cbd0 con 0x7f961c06ded0 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f961c156ec0 0x7f961c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f961c156ec0 0x7f961c06d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38156/0 (socket says 172.21.15.111:38156) 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 -- 172.21.15.111:0/307111972 learned_addr learned my addr 172.21.15.111:0/307111972 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f9613fff700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 -- 172.21.15.111:0/307111972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 msgr2=0x7f961c0c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 -- 172.21.15.111:0/307111972 >> v1:172.21.15.143:6789/0 conn(0x7f961c157290 legacy=0x7f961c0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 -- 172.21.15.111:0/307111972 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c06ceb0 con 0x7f961c156ec0 2023-12-19T07:47:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.020+0000 7f961b7fe700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c06d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f960c0099b0 tx=0x7f960c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599ddc82af7a2964 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f961affd700 1 -- 172.21.15.111:0/307111972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c009ea0 con 0x7f961c156ec0 2023-12-19T07:47:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f961affd700 1 -- 172.21.15.111:0/307111972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c00b860 con 0x7f961c156ec0 2023-12-19T07:47:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f961affd700 1 -- 172.21.15.111:0/307111972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c00bc00 con 0x7f961c156ec0 2023-12-19T07:47:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 -- 172.21.15.111:0/307111972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 msgr2=0x7f961c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c06d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f960c0099b0 tx=0x7f960c009250 comp rx=0 tx=0).stop 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 -- 172.21.15.111:0/307111972 shutdown_connections 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 --2- 172.21.15.111:0/307111972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c06d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 -- 172.21.15.111:0/307111972 >> 172.21.15.111:0/307111972 conn(0x7f961c069d80 msgr2=0x7f961c0ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 -- 172.21.15.111:0/307111972 shutdown_connections 2023-12-19T07:47:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.024+0000 7f9621aae700 1 -- 172.21.15.111:0/307111972 wait complete. 2023-12-19T07:47:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 Processor -- start 2023-12-19T07:47:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 -- start start 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 0x7f961c0c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0689e0 con 0x7f961c156ec0 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c067b60 con 0x7f961c157290 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9621aae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c067dd0 con 0x7f961c06ded0 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38158/0 (socket says 172.21.15.111:38158) 2023-12-19T07:47:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 -- 172.21.15.111:0/1507619515 learned_addr learned my addr 172.21.15.111:0/1507619515 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f961bfff700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 0x7f961c0c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 -- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 msgr2=0x7f961c0670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 -- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 msgr2=0x7f961c0c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 0x7f961c0c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f9613fff700 1 -- 172.21.15.111:0/1507619515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c0094f0 con 0x7f961c156ec0 2023-12-19T07:47:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.028+0000 7f961bfff700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 0x7f961c0c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f9613fff700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f960400ec80 tx=0x7f96040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f96137fe700 1 -- 172.21.15.111:0/1507619515 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960400c9d0 con 0x7f961c156ec0 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f96137fe700 1 -- 172.21.15.111:0/1507619515 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960400cb30 con 0x7f961c156ec0 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c0680b0 con 0x7f961c156ec0 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f96137fe700 1 -- 172.21.15.111:0/1507619515 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604013470 con 0x7f961c156ec0 2023-12-19T07:47:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c068670 con 0x7f961c156ec0 2023-12-19T07:47:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.032+0000 7f96137fe700 1 -- 172.21.15.111:0/1507619515 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9604013820 con 0x7f961c156ec0 2023-12-19T07:47:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.036+0000 7f96137fe700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 0x7f95fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.036+0000 7f96137fe700 1 -- 172.21.15.111:0/1507619515 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f960407ce40 con 0x7f961c156ec0 2023-12-19T07:47:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.036+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 0x7f95fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.036+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 0x7f95fc062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f960c0099b0 tx=0x7f960c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.036+0000 7f9623d5c740 1 Processor -- start 2023-12-19T07:47:45.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 -- start start 2023-12-19T07:47:45.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219d710 0x2423c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e8790 0x24241d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2348f30 con 0x219dae0 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2344ed0 con 0x219d710 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x230b0b0 con 0x21e8790 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38172/0 (socket says 172.21.15.111:38172) 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 -- 172.21.15.111:0/1038020610 learned_addr learned my addr 172.21.15.111:0/1038020610 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9613fff700 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219d710 0x2423c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961bfff700 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e8790 0x24241d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 -- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e8790 msgr2=0x24241d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e8790 0x24241d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 -- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219d710 msgr2=0x2423c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219d710 0x2423c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 -- 172.21.15.111:0/1038020610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x21e99e0 con 0x219dae0 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f961b7fe700 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f960c006e80 tx=0x7f960c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdadeb4cd3cc69c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9610ff9700 1 -- 172.21.15.111:0/1038020610 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c020070 con 0x219dae0 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.040+0000 7f9610ff9700 1 -- 172.21.15.111:0/1038020610 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c00baa0 con 0x219dae0 2023-12-19T07:47:45.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9610ff9700 1 -- 172.21.15.111:0/1038020610 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c029b70 con 0x219dae0 2023-12-19T07:47:45.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- 172.21.15.111:0/1038020610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 msgr2=0x2423750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f960c006e80 tx=0x7f960c020000 comp rx=0 tx=0).stop 2023-12-19T07:47:45.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- 172.21.15.111:0/1038020610 shutdown_connections 2023-12-19T07:47:45.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219d710 0x2423c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x21e8790 0x24241d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- 172.21.15.111:0/1038020610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2423750 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- 172.21.15.111:0/1038020610 >> 172.21.15.111:0/1038020610 conn(0x219deb0 msgr2=0x23d7930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- 172.21.15.111:0/1038020610 shutdown_connections 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- 172.21.15.111:0/1038020610 wait complete. 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- start start 2023-12-19T07:47:45.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 Processor -- start 2023-12-19T07:47:45.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x23ff140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x23e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x230ad50 con 0x21e8790 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x22e3970 con 0x219dae0 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x22f3e00 con 0x219d710 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.044+0000 7f9623d5c740 10 client.0 _ll_register_callbacks cb 0x238e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38180/0 (socket says 172.21.15.111:38180) 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 -- 172.21.15.111:0/279291195 learned_addr learned my addr 172.21.15.111:0/279291195 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961b7fe700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x23ff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9613fff700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x23e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 -- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 msgr2=0x23ff140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x23ff140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 -- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 msgr2=0x23e0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x23e0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 -- 172.21.15.111:0/279291195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c06ceb0 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961b7fe700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x23ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961bfff700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9614011900 tx=0x7f9614009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d268f30e06b9d45d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614001950 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x22e3cc0 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9614001ab0 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96140030d0 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 10 client.4764 fetch_fsmap learned FSMap version 8 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x2313370 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x22e3cc0 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f9614003230 con 0x21e8790 2023-12-19T07:47:45.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 10 client.4764 fetch_fsmap finished waiting for FSMap version 8 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 10 client.4764 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 20 client.4764 populate_metadata read hostname 'smithi111' 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242ffb0 0x2432390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 4 client.4764 mds_command: new command op to 4275 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 --> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x22e73e0 con 0x242ffb0 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.048+0000 7f961b7fe700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242ffb0 0x2432390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f95f9ffb700 10 client.4764 ms_handle_connect on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:45.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f961b7fe700 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242ffb0 0x2432390 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acec74bafc182d9f server_cookie=abc3c82ed60aa7f7 in_seq=0 out_seq=0 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f95f9ffb700 1 -- 172.21.15.111:0/279291195 <== mds.0 v2:172.21.15.143:6834/1214316551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x22e73e0 con 0x242ffb0 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f95f9ffb700 10 client.4764 handle_command_reply: tid=0 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 client.4764 shutdown 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 msgr2=0x23e13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9614011900 tx=0x7f9614009040 comp rx=0 tx=0).stop 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 shutdown_connections 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242ffb0 0x2432390 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x23e0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x23e13a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f95f9ffb700 0 client.4764 ms_handle_reset on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 --2- 172.21.15.111:0/279291195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x23ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 >> 172.21.15.111:0/279291195 conn(0x219deb0 msgr2=0x23cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 shutdown_connections 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 1 -- 172.21.15.111:0/279291195 wait complete. 2023-12-19T07:47:45.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.052+0000 7f9623d5c740 20 client.4764 trim_cache size 0 max 16384 2023-12-19T07:47:45.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.072+0000 7f9623d5c740 1 Processor -- start 2023-12-19T07:47:45.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.072+0000 7f9623d5c740 1 -- start start 2023-12-19T07:47:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x24783e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e8790 0x2478e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2349ef0 con 0x219dae0 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2318040 con 0x21e8790 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x23489d0 con 0x219d710 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38196/0 (socket says 172.21.15.111:38196) 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 -- 172.21.15.111:0/704716033 learned_addr learned my addr 172.21.15.111:0/704716033 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f961b7fe700 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x24783e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f961bfff700 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e8790 0x2478e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 -- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 msgr2=0x24783e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x24783e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 -- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e8790 msgr2=0x2478e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e8790 0x2478e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 -- 172.21.15.111:0/704716033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x22f9d80 con 0x219dae0 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f9613fff700 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f960407b060 tx=0x7f9604056040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c9f19e39070c04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f96117fa700 1 -- 172.21.15.111:0/704716033 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604053ff0 con 0x219dae0 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f96117fa700 1 -- 172.21.15.111:0/704716033 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604057d40 con 0x219dae0 2023-12-19T07:47:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.076+0000 7f96117fa700 1 -- 172.21.15.111:0/704716033 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96040560b0 con 0x219dae0 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- 172.21.15.111:0/704716033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 msgr2=0x2478920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f960407b060 tx=0x7f9604056040 comp rx=0 tx=0).stop 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- 172.21.15.111:0/704716033 shutdown_connections 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x21e8790 0x2478e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x219dae0 0x2478920 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 --2- 172.21.15.111:0/704716033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x24783e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- 172.21.15.111:0/704716033 >> 172.21.15.111:0/704716033 conn(0x219deb0 msgr2=0x23cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- 172.21.15.111:0/704716033 shutdown_connections 2023-12-19T07:47:45.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- 172.21.15.111:0/704716033 wait complete. 2023-12-19T07:47:45.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 -- start start 2023-12-19T07:47:45.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.080+0000 7f9623d5c740 1 Processor -- start 2023-12-19T07:47:45.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x2429250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x2429790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2342890 con 0x21e8790 2023-12-19T07:47:45.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x22e5b80 con 0x219dae0 2023-12-19T07:47:45.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x219b430 con 0x219d710 2023-12-19T07:47:45.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 10 client.0 _ll_register_callbacks cb 0x21699d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:47:45.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38206/0 (socket says 172.21.15.111:38206) 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 -- 172.21.15.111:0/3564307512 learned_addr learned my addr 172.21.15.111:0/3564307512 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961b7fe700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x2429250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9613fff700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x2429790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 -- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 msgr2=0x2429250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x2429250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 -- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 msgr2=0x2429790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x2429790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 -- 172.21.15.111:0/3564307512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c068670 con 0x21e8790 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f961bfff700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9614003860 tx=0x7f9614020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420f5f36f62bb39c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961402c070 con 0x21e8790 2023-12-19T07:47:45.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9614001ae0 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f960c00baa0 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96140032e0 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 10 client.4770 fetch_fsmap learned FSMap version 8 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x22ee130 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c00baa0 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f961401d350 con 0x21e8790 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 10 client.4770 fetch_fsmap finished waiting for FSMap version 8 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 10 client.4770 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 20 client.4770 populate_metadata read hostname 'smithi111' 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242a210 0x242a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 4 client.4770 mds_command: new command op to 4275 tid=0[{"prefix": "status"}] 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.084+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 --> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] -- command(tid 0: {"prefix": "status"}) v1 -- 0x2319410 con 0x242a210 2023-12-19T07:47:45.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f961b7fe700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242a210 0x242a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f961b7fe700 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242a210 0x242a5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb71c4dcc6a380e1 server_cookie=e1b5473581daaf9f in_seq=0 out_seq=0 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f95f9ffb700 10 client.4770 ms_handle_connect on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f95f9ffb700 1 -- 172.21.15.111:0/3564307512 <== mds.0 v2:172.21.15.143:6834/1214316551 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x2319410 con 0x242a210 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f95f9ffb700 10 client.4770 handle_command_reply: tid=0 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 client.4770 shutdown 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 msgr2=0x2429cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9614003860 tx=0x7f9614020000 comp rx=0 tx=0).stop 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 shutdown_connections 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:6834/1214316551,v1:172.21.15.143:6835/1214316551] conn(0x242a210 0x242a5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x219dae0 0x2429790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x21e8790 0x2429cd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f95f9ffb700 0 client.4770 ms_handle_reset on v2:172.21.15.143:6834/1214316551 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 --2- 172.21.15.111:0/3564307512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x219d710 0x2429250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 >> 172.21.15.111:0/3564307512 conn(0x219deb0 msgr2=0x23d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 shutdown_connections 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 1 -- 172.21.15.111:0/3564307512 wait complete. 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9623d5c740 20 client.4770 trim_cache size 0 max 16384 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T07:47:45.074 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4275, 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 27.892335631000002, 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 8, 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 25, 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 25, 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 32.751998749000002 2023-12-19T07:47:45.075 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 msgr2=0x7f95fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 0x7f95fc062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f960c0099b0 tx=0x7f960c017000 comp rx=0 tx=0).stop 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 msgr2=0x7f961c067620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f960400ec80 tx=0x7f96040091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 shutdown_connections 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f95fc060550 0x7f95fc062a00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f961c157290 0x7f961c0c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f961c156ec0 0x7f961c067620 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 --2- 172.21.15.111:0/1507619515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f961c06ded0 0x7f961c0670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.088+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 >> 172.21.15.111:0/1507619515 conn(0x7f961c069d80 msgr2=0x7f961c0cc120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.092+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 shutdown_connections 2023-12-19T07:47:45.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.092+0000 7f9621aae700 1 -- 172.21.15.111:0/1507619515 wait complete. 2023-12-19T07:47:45.095 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-19T07:47:45.107 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-19T07:47:45.107 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-12-19T07:47:45.108 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-19T07:47:45.108 INFO:tasks.ceph_fuse:client.2 config is {} 2023-12-19T07:47:45.108 INFO:tasks.ceph_fuse:client.1 config is {} 2023-12-19T07:47:45.108 INFO:tasks.ceph_fuse:client.3 config is {} 2023-12-19T07:47:45.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.108 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T07:47:45.128 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-12-19T07:47:45.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-12-19T07:47:45.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.212 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-12-19T07:47:45.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.442 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-12-19T07:47:45.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link delete ceph-brx 2023-12-19T07:47:45.554 INFO:teuthology.orchestra.run.smithi143.stderr:Cannot find device "ceph-brx" 2023-12-19T07:47:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:47:45.559 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-19T07:47:45.559 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:47:45.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 Processor -- start 2023-12-19T07:47:45.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 -- start start 2023-12-19T07:47:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc60c0c6cd0 0x7fc60c06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc60c15cc20 0x7fc60c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60c06b470 con 0x7fc60c0c6900 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c06be40 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.740+0000 7fc610d23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c06e110 con 0x7fc60c15cc20 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc60c0c6cd0 0x7fc60c06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc60c0c6cd0 0x7fc60c06b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38208/0 (socket says 172.21.15.111:38208) 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 -- 172.21.15.111:0/1785292016 learned_addr learned my addr 172.21.15.111:0/1785292016 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc60bfff700 1 --2- 172.21.15.111:0/1785292016 >> v2:172.21.15.143:3300/0 conn(0x7fc60c15cc20 0x7fc60c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 -- 172.21.15.111:0/1785292016 >> v2:172.21.15.143:3300/0 conn(0x7fc60c15cc20 msgr2=0x7fc60c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 --2- 172.21.15.111:0/1785292016 >> v2:172.21.15.143:3300/0 conn(0x7fc60c15cc20 0x7fc60c15d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 -- 172.21.15.111:0/1785292016 >> v1:172.21.15.143:6789/0 conn(0x7fc60c0c6900 legacy=0x7fc60c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 -- 172.21.15.111:0/1785292016 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c161d10 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc603fff700 1 --2- 172.21.15.111:0/1785292016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c06b900 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0c5c50 tx=0x7fc5f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8850f2a7d3943249 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc60affd700 1 -- 172.21.15.111:0/1785292016 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400b860 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc60affd700 1 -- 172.21.15.111:0/1785292016 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f400b9c0 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc60affd700 1 -- 172.21.15.111:0/1785292016 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400bd60 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 -- 172.21.15.111:0/1785292016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 msgr2=0x7fc60c06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 --2- 172.21.15.111:0/1785292016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c06b900 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0c5c50 tx=0x7fc5f4009250 comp rx=0 tx=0).stop 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 -- 172.21.15.111:0/1785292016 shutdown_connections 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 --2- 172.21.15.111:0/1785292016 >> v2:172.21.15.143:3300/0 conn(0x7fc60c15cc20 0x7fc60c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 --2- 172.21.15.111:0/1785292016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c06b900 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 -- 172.21.15.111:0/1785292016 >> 172.21.15.111:0/1785292016 conn(0x7fc60c067bb0 msgr2=0x7fc60c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 -- 172.21.15.111:0/1785292016 shutdown_connections 2023-12-19T07:47:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.744+0000 7fc610d23700 1 -- 172.21.15.111:0/1785292016 wait complete. 2023-12-19T07:47:45.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.748+0000 7fc610d23700 1 Processor -- start 2023-12-19T07:47:45.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.748+0000 7fc610d23700 1 -- start start 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc60c0c6900 0x7fc60c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc60c15cc20 0x7fc60c1508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c151e90 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c155140 con 0x7fc60c15cc20 2023-12-19T07:47:45.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c155420 con 0x7fc60c0c6900 2023-12-19T07:47:45.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38214/0 (socket says 172.21.15.111:38214) 2023-12-19T07:47:45.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 -- 172.21.15.111:0/298567292 learned_addr learned my addr 172.21.15.111:0/298567292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:45.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc60b7fe700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc60c0c6900 0x7fc60c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc60bfff700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc60c15cc20 0x7fc60c1508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 -- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc60c0c6900 msgr2=0x7fc60c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc60c0c6900 0x7fc60c0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 -- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc60c15cc20 msgr2=0x7fc60c1508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc60c15cc20 0x7fc60c1508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 -- 172.21.15.111:0/298567292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40094f0 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc603fff700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc60c064040 tx=0x7fc5f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4024440 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f40245a0 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c155700 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4024960 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.752+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c155d40 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.756+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c155f00 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.756+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc5f401a420 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.760+0000 7fc6037fe700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 0x7fc5ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:45.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.760+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc5f407e870 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.760+0000 7fc60b7fe700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 0x7fc5ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:45.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.760+0000 7fc60b7fe700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 0x7fc5ec062a00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc003a80 tx=0x7fc5fc007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:45.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.764+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc60c155f00 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc60c155f00 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc6037fe700 1 -- 172.21.15.111:0/298567292 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fc60c155f00 con 0x7fc60c0c6cd0 2023-12-19T07:47:45.959 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T07:47:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 msgr2=0x7fc5ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 0x7fc5ec062a00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc003a80 tx=0x7fc5fc007ec0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 msgr2=0x7fc60c0b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.976+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc60c064040 tx=0x7fc5f4017000 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 shutdown_connections 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5ec060550 0x7fc5ec062a00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc60c15cc20 0x7fc60c1508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc60c0c6cd0 0x7fc60c0b0fe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 --2- 172.21.15.111:0/298567292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc60c0c6900 0x7fc60c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 >> 172.21.15.111:0/298567292 conn(0x7fc60c067bb0 msgr2=0x7fc60c0c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 shutdown_connections 2023-12-19T07:47:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:45.980+0000 7fc610d23700 1 -- 172.21.15.111:0/298567292 wait complete. 2023-12-19T07:47:45.981 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T07:47:45.981 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:47:45.981 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:47:45.982 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T07:47:45.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.982 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:45.997 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42392sec preferred_lft 42392sec 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:47:45.998 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:45.999 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T07:47:45.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T07:47:45.999 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:46.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.157 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:47:46.191 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T07:47:46.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.194 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T07:47:46.256 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T07:47:46.256 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:47:46.257 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:47:46.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.258 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:46.258 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:46.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:47:46.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:47:46.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:47:46.259 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:46.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.382 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T07:47:46.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.402 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-12-19T07:47:46.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.470 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:46.471 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:46.471 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T07:47:46.471 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-19T07:47:46.471 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:46.588 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-19T07:47:46.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-19T07:47:46.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-19T07:47:46.589 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-19T07:47:46.589 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:46.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:46.742 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:46.742 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:46.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T07:47:46.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T07:47:46.742 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:46.817 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T07:47:46.817 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:46.817 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:46.834 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T07:47:46.834 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:46.834 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:46.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:46.975 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:46.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:47.055 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:47.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:47.058 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:47.118 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T07:47:47.118 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T07:47:47.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:47.200 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:47.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:47.272 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:47.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:47.277 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:47.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T07:47:47.312+0000 7fdb74381380 -1 init, newargv = 0x5586f4de3d90 newargc=15 2023-12-19T07:47:47.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[41594]: starting ceph client 2023-12-19T07:47:47.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[41594]: starting fuse 2023-12-19T07:47:47.309 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T07:47:47.309 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:47.310 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T07:47:47.311 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T07:47:47.471 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.41594.asok 2023-12-19T07:47:47.658 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41594.asok 2023-12-19T07:47:47.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:47.658 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41594.asok status 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T07:47:47.841 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "41594", 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4782, 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 4782 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3334003585 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:47:47.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3334003585 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.4782 192.168.0.1:0/3334003585", 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/3334003585", 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 8, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 25, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T07:47:47.843 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:47:47.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:47:48.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 Processor -- start 2023-12-19T07:47:48.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 -- start start 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 0x7f3e540cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3e540cc780 0x7f3e54142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540cbea0 con 0x7f3e540cc330 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540cb030 con 0x7f3e540cc780 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e5ae2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540cb2d0 con 0x7f3e540cea90 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e53fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3e540cc330 0x7f3e54068a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43696/0 (socket says 172.21.15.111:43696) 2023-12-19T07:47:48.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.032+0000 7f3e53fff700 1 -- 172.21.15.111:0/3180690974 learned_addr learned my addr 172.21.15.111:0/3180690974 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 --2- 172.21.15.111:0/3180690974 >> v2:172.21.15.111:3300/0 conn(0x7f3e540cc780 0x7f3e54142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e58bca700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 0x7f3e540cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 -- 172.21.15.111:0/3180690974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 msgr2=0x7f3e540cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 0x7f3e540cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 -- 172.21.15.111:0/3180690974 >> v1:172.21.15.111:6789/0 conn(0x7f3e540cc330 legacy=0x7f3e54068a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 -- 172.21.15.111:0/3180690974 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54067e00 con 0x7f3e540cc780 2023-12-19T07:47:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e58bca700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 0x7f3e540cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:48.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e593cb700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cc780 0x7f3e54142720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009e60 tx=0x7f3e4c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a72a2a2ee5d1af5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:48.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e537fe700 1 -- 172.21.15.111:0/3180690974 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00e070 con 0x7f3e540cc780 2023-12-19T07:47:48.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e537fe700 1 -- 172.21.15.111:0/3180690974 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c00bb60 con 0x7f3e540cc780 2023-12-19T07:47:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.036+0000 7f3e537fe700 1 -- 172.21.15.111:0/3180690974 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c012430 con 0x7f3e540cc780 2023-12-19T07:47:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/3180690974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cc780 msgr2=0x7f3e54142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cc780 0x7f3e54142720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009e60 tx=0x7f3e4c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/3180690974 shutdown_connections 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cc780 0x7f3e54142720 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/3180690974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cea90 0x7f3e540cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/3180690974 >> 172.21.15.111:0/3180690974 conn(0x7f3e540cd650 msgr2=0x7f3e5406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/3180690974 shutdown_connections 2023-12-19T07:47:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/3180690974 wait complete. 2023-12-19T07:47:48.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.040+0000 7f3e5ae2e700 1 Processor -- start 2023-12-19T07:47:48.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- start start 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 0x7f3e54141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e540cc780 0x7f3e540b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b0600 con 0x7f3e540cea90 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b0b80 con 0x7f3e540cc780 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54142040 con 0x7f3e540cc330 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38234/0 (socket says 172.21.15.111:38234) 2023-12-19T07:47:48.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 -- 172.21.15.111:0/4192165932 learned_addr learned my addr 172.21.15.111:0/4192165932 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:48.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e58bca700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 0x7f3e54141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e53fff700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e540cc780 0x7f3e540b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 -- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 msgr2=0x7f3e54141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 0x7f3e54141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 -- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e540cc780 msgr2=0x7f3e540b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e540cc780 0x7f3e540b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 -- 172.21.15.111:0/4192165932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48009420 con 0x7f3e540cea90 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e58bca700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 0x7f3e54141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e593cb700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3e5406a0f0 tx=0x7f3e4c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c012e40 con 0x7f3e540cea90 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c01d630 con 0x7f3e540cea90 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c009a20 con 0x7f3e540cea90 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c01d9f0 con 0x7f3e540cea90 2023-12-19T07:47:48.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.044+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5416d260 con 0x7f3e540cea90 2023-12-19T07:47:48.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.048+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e54071120 con 0x7f3e540cea90 2023-12-19T07:47:48.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.048+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3e4c024070 con 0x7f3e540cea90 2023-12-19T07:47:48.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.048+0000 7f3e50ff9700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 0x7f3e38062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:48.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.048+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f3e4c07f320 con 0x7f3e540cea90 2023-12-19T07:47:48.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.048+0000 7f3e58bca700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 0x7f3e38062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:48.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.052+0000 7f3e58bca700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 0x7f3e38062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3e480093f0 tx=0x7f3e48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:48.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.056+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3e4c04f580 con 0x7f3e540cea90 2023-12-19T07:47:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.264+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3e540cb570 con 0x7f3e540cea90 2023-12-19T07:47:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.264+0000 7f3e50ff9700 1 -- 172.21.15.111:0/4192165932 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3e4c052ba0 con 0x7f3e540cea90 2023-12-19T07:47:48.249 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T07:47:48.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 msgr2=0x7f3e38062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 0x7f3e38062a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3e480093f0 tx=0x7f3e48008040 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 msgr2=0x7f3e5416c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3e5406a0f0 tx=0x7f3e4c019000 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 shutdown_connections 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3e38060550 0x7f3e38062a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e540cc780 0x7f3e540b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e540cea90 0x7f3e5416c550 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 --2- 172.21.15.111:0/4192165932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e540cc330 0x7f3e54141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:48.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.268+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 >> 172.21.15.111:0/4192165932 conn(0x7f3e540cd650 msgr2=0x7f3e540c51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:48.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.272+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 shutdown_connections 2023-12-19T07:47:48.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:48.272+0000 7f3e5ae2e700 1 -- 172.21.15.111:0/4192165932 wait complete. 2023-12-19T07:47:48.274 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T07:47:48.274 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:47:48.274 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T07:47:48.274 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-12-19T07:47:48.274 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:48.275 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T07:47:48.275 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:47:48.275 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:47:48.275 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-12-19T07:47:48.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.275 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T07:47:48.294 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.295 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42389sec preferred_lft 42389sec 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T07:47:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.297 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T07:47:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T07:47:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T07:47:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.298 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:47:48.376 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T07:47:48.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.380 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T07:47:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:47:48.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.449 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:48.450 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:48.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:47:48.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:47:48.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:47:48.450 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:48.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.584 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T07:47:48.603 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T07:47:48.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.604 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-12-19T07:47:48.662 INFO:teuthology.orchestra.run.smithi111.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-12-19T07:47:48.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.664 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:48.664 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:48.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T07:47:48.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-12-19T07:47:48.664 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:48.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-19T07:47:48.817 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:48.817 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:48.817 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:48.817 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 8a:1e:30:52:e4:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::881e:30ff:fe52:e4f3/64 scope link 2023-12-19T07:47:48.818 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-19T07:47:48.903 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:48.903 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:48.903 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:48.903 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 8a:1e:30:52:e4:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::881e:30ff:fe52:e4f3/64 scope link 2023-12-19T07:47:48.904 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:48.906 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-12-19T07:47:48.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-12-19T07:47:48.907 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:49.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:49.111 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:47:49.111 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:47:49.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.1 up 2023-12-19T07:47:49.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.1 master ceph-brx 2023-12-19T07:47:49.111 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:47:49.185 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-12-19T07:47:49.186 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:49.186 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:49.199 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-12-19T07:47:49.200 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:49.200 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:49.260 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:49.343 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:49.343 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:49.423 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:49.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:49.430 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:49.491 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T07:47:49.491 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [49] 2023-12-19T07:47:49.492 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-12-19T07:47:49.544 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:47:49.576 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:49.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:49.647 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:49.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:49.650 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:47:49.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi111.stderr:2023-12-19T07:47:49.692+0000 7fcb11bbf380 -1 init, newargv = 0x5606d7814f10 newargc=15 2023-12-19T07:47:49.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi111.stderr:ceph-fuse[41730]: starting ceph client 2023-12-19T07:47:49.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi111.stderr:ceph-fuse[41730]: starting fuse 2023-12-19T07:47:49.689 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T07:47:49.689 INFO:teuthology.orchestra.run.smithi111.stdout:50 2023-12-19T07:47:49.690 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49, 50] 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T07:47:49.690 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T07:47:49.691 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:47:49.692 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:47:49.692 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.2")) 2023-12-19T07:47:49.692 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T07:47:49.851 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.2.41730.asok 2023-12-19T07:47:50.033 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.41730.asok 2023-12-19T07:47:50.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:50.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.41730.asok status 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "2", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "41730", 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:50.225 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4794, 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 4794 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.2:0", 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2205087995 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:50.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.2:0", 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2205087995 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.4794 192.168.0.2:0/2205087995", 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.2:0/2205087995", 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T07:47:50.227 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 8, 2023-12-19T07:47:50.228 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 25, 2023-12-19T07:47:50.228 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:47:50.228 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T07:47:50.228 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T07:47:50.228 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:47:50.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:47:50.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 Processor -- start 2023-12-19T07:47:50.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 -- start start 2023-12-19T07:47:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb14158e20 0x7ffb1406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1406b730 con 0x7ffb141591f0 2023-12-19T07:47:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1406ab20 con 0x7ffb140ce810 2023-12-19T07:47:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.424+0000 7ffb1b06b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1406ae00 con 0x7ffb14158e20 2023-12-19T07:47:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19868700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffb141591f0 0x7ffb140ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43704/0 (socket says 172.21.15.111:43704) 2023-12-19T07:47:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19868700 1 -- 172.21.15.111:0/1316855159 learned_addr learned my addr 172.21.15.111:0/1316855159 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb1a069700 1 --2- 172.21.15.111:0/1316855159 >> v2:172.21.15.143:3300/0 conn(0x7ffb14158e20 0x7ffb1406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511713345 0 0) 0x7ffb1406b730 con 0x7ffb141591f0 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00003570 con 0x7ffb141591f0 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812197866 0 0) 0x7ffb1406ab20 con 0x7ffb140ce810 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1406b730 con 0x7ffb140ce810 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb1a069700 1 -- 172.21.15.111:0/1316855159 >> v1:172.21.15.143:6789/0 conn(0x7ffb140ce810 legacy=0x7ffb140ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb1a069700 1 -- 172.21.15.111:0/1316855159 >> v1:172.21.15.111:6789/0 conn(0x7ffb141591f0 legacy=0x7ffb140ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb1a069700 1 -- 172.21.15.111:0/1316855159 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1406b0e0 con 0x7ffb14158e20 2023-12-19T07:47:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb1a069700 1 --2- 172.21.15.111:0/1316855159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb14158e20 0x7ffb1406a540 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb100098e0 tx=0x7ffb100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe3a329fc4be5929 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10011630 con 0x7ffb14158e20 2023-12-19T07:47:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10011790 con 0x7ffb14158e20 2023-12-19T07:47:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.428+0000 7ffb19067700 1 -- 172.21.15.111:0/1316855159 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10011b30 con 0x7ffb14158e20 2023-12-19T07:47:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 -- 172.21.15.111:0/1316855159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb14158e20 msgr2=0x7ffb1406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/1316855159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb14158e20 0x7ffb1406a540 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb100098e0 tx=0x7ffb100091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 -- 172.21.15.111:0/1316855159 shutdown_connections 2023-12-19T07:47:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/1316855159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb14158e20 0x7ffb1406a540 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 -- 172.21.15.111:0/1316855159 >> 172.21.15.111:0/1316855159 conn(0x7ffb14067bb0 msgr2=0x7ffb140cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 -- 172.21.15.111:0/1316855159 shutdown_connections 2023-12-19T07:47:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.432+0000 7ffb1b06b700 1 -- 172.21.15.111:0/1316855159 wait complete. 2023-12-19T07:47:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 Processor -- start 2023-12-19T07:47:50.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 -- start start 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb140ce810 0x7ffb14141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb141591f0 0x7ffb140c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14159d40 con 0x7ffb14158e20 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140ca190 con 0x7ffb140ce810 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140c9bd0 con 0x7ffb141591f0 2023-12-19T07:47:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38244/0 (socket says 172.21.15.111:38244) 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 -- 172.21.15.111:0/309953809 learned_addr learned my addr 172.21.15.111:0/309953809 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1a069700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb140ce810 0x7ffb14141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1a86a700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb141591f0 0x7ffb140c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 -- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb141591f0 msgr2=0x7ffb140c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb141591f0 0x7ffb140c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 -- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb140ce810 msgr2=0x7ffb14141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb140ce810 0x7ffb14141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 -- 172.21.15.111:0/309953809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009420 con 0x7ffb14158e20 2023-12-19T07:47:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb19868700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c7c0 tx=0x7ffb04007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04005b30 con 0x7ffb14158e20 2023-12-19T07:47:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04005c90 con 0x7ffb14158e20 2023-12-19T07:47:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb140c9eb0 con 0x7ffb14158e20 2023-12-19T07:47:50.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.436+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04007300 con 0x7ffb14158e20 2023-12-19T07:47:50.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.440+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1415c360 con 0x7ffb14158e20 2023-12-19T07:47:50.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.440+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14159ef0 con 0x7ffb14158e20 2023-12-19T07:47:50.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.440+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ffb0401b3b0 con 0x7ffb14158e20 2023-12-19T07:47:50.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.444+0000 7ffb0a7fc700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 0x7ffaf8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:50.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.444+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ffb0407bf10 con 0x7ffb14158e20 2023-12-19T07:47:50.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.444+0000 7ffb1a069700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 0x7ffaf8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:50.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.444+0000 7ffb1a069700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 0x7ffaf8062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffb100093f0 tx=0x7ffb10017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:50.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.452+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb14159ef0 con 0x7ffb14158e20 2023-12-19T07:47:50.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.656+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb14159ef0 con 0x7ffb14158e20 2023-12-19T07:47:50.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.656+0000 7ffb0a7fc700 1 -- 172.21.15.111:0/309953809 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7ffb14159ef0 con 0x7ffb14158e20 2023-12-19T07:47:50.641 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T07:47:50.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 msgr2=0x7ffaf8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 0x7ffaf8062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffb100093f0 tx=0x7ffb10017000 comp rx=0 tx=0).stop 2023-12-19T07:47:50.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 msgr2=0x7ffb140c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c7c0 tx=0x7ffb04007ea0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 shutdown_connections 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffaf8060550 0x7ffaf8062a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb141591f0 0x7ffb140c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb14158e20 0x7ffb140c90a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 --2- 172.21.15.111:0/309953809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb140ce810 0x7ffb14141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 >> 172.21.15.111:0/309953809 conn(0x7ffb14067bb0 msgr2=0x7ffb14153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 shutdown_connections 2023-12-19T07:47:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:50.660+0000 7ffb1b06b700 1 -- 172.21.15.111:0/309953809 wait complete. 2023-12-19T07:47:50.666 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T07:47:50.666 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi143.front.sepia.ceph.com 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:47:50.667 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-19T07:47:50.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:50.667 DEBUG:teuthology.orchestra.run.smithi143:> ip addr 2023-12-19T07:47:50.686 INFO:teuthology.orchestra.run.smithi143.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:94 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: altname enp7s0f0 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:95 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:50.687 INFO:teuthology.orchestra.run.smithi143.stdout: altname enp7s0f1 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:82 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:83 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: inet 172.21.15.143/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft 42370sec preferred_lft 42370sec 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::ec4:7aff:fe88:7783/64 scope link 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 02:42:0e:0d:e8:a5 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:47:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:50.689 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T07:47:50.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link add name ceph-brx type bridge 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip addr flush dev ceph-brx 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set ceph-brx up 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T07:47:50.690 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:50.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:50.846 DEBUG:teuthology.orchestra.run.smithi143:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:47:50.881 INFO:teuthology.orchestra.run.smithi143.stdout:1 2023-12-19T07:47:50.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:50.884 DEBUG:teuthology.orchestra.run.smithi143:> route 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:Kernel IP routing table 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:47:50.962 INFO:teuthology.orchestra.run.smithi143.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:47:50.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:47:50.963 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:51.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:51.099 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-12-19T07:47:51.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:51.121 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list-id 2023-12-19T07:47:51.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:51.181 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:51.181 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:51.181 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T07:47:51.181 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-19T07:47:51.181 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:51.300 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-19T07:47:51.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-19T07:47:51.300 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:51.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:51.479 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:51.480 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:51.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set brx.0 up 2023-12-19T07:47:51.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T07:47:51.480 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:51.552 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-12-19T07:47:51.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:51.553 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:51.571 INFO:teuthology.orchestra.run.smithi143.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-19T07:47:51.571 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:51.571 DEBUG:teuthology.orchestra.run.smithi143:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:51.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo modprobe fuse 2023-12-19T07:47:51.712 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:51.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:51.791 INFO:teuthology.orchestra.run.smithi143.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:51.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:51.795 DEBUG:teuthology.orchestra.run.smithi143:> ls /sys/fs/fuse/connections 2023-12-19T07:47:51.862 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T07:47:51.862 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-12-19T07:47:51.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo modprobe fuse 2023-12-19T07:47:51.950 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:51.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:52.025 INFO:teuthology.orchestra.run.smithi143.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:52.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:52.030 DEBUG:teuthology.orchestra.run.smithi143:> ls /sys/fs/fuse/connections 2023-12-19T07:47:52.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi143.stderr:2023-12-19T07:47:52.060+0000 7f9cfd7c7380 -1 init, newargv = 0x564ab52a4000 newargc=15 2023-12-19T07:47:52.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi143.stderr:ceph-fuse[39323]: starting ceph client 2023-12-19T07:47:52.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi143.stderr:ceph-fuse[39323]: starting fuse 2023-12-19T07:47:52.057 INFO:teuthology.orchestra.run.smithi143.stdout:49 2023-12-19T07:47:52.058 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> import glob 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> import re 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> import os 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> import subprocess 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> def _find_admin_socket(client_name): 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> files = glob.glob(asok_path) 2023-12-19T07:47:52.058 DEBUG:teuthology.orchestra.run.smithi143:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> # Given a non-glob path, it better be there 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> if "*" not in asok_path: 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> assert(len(files) == 1) 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> return files[0] 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> for f in files: 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> contents = proc_f.read() 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> if mountpoint in contents: 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> return f 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> print(_find_admin_socket("client.1")) 2023-12-19T07:47:52.059 DEBUG:teuthology.orchestra.run.smithi143:> ' 2023-12-19T07:47:52.219 INFO:teuthology.orchestra.run.smithi143.stdout:/var/run/ceph/ceph-client.1.39323.asok 2023-12-19T07:47:52.402 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.39323.asok 2023-12-19T07:47:52.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:52.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.39323.asok status 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "metadata": { 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "entity_id": "1", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "hostname": "smithi143", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "pid": "39323", 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "root": "/" 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:52.600 INFO:teuthology.orchestra.run.smithi143.stdout: "dentry_count": 0, 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "dentry_pinned_count": 0, 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "id": 4613, 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "inst": { 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "name": { 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "client", 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "num": 4613 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": { 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1734916444 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: } 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": { 2023-12-19T07:47:52.601 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1734916444 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "inst_str": "client.4613 192.168.0.1:0/1734916444", 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "addr_str": "192.168.0.1:0/1734916444", 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "inode_count": 1, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_epoch": 8, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_epoch": 25, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "blocklisted": false, 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout: "fs_name": "cephfs" 2023-12-19T07:47:52.602 INFO:teuthology.orchestra.run.smithi143.stdout:} 2023-12-19T07:47:52.614 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:47:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 Processor -- start 2023-12-19T07:47:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 -- start start 2023-12-19T07:47:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f160406a580 0x7f160406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 0x7f160406d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160406c7c0 con 0x7f16040b1410 2023-12-19T07:47:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160406af40 con 0x7f160406a580 2023-12-19T07:47:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160af63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160406cd80 con 0x7f160406d210 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f1609760700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f160406a580 0x7f160406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160a762700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 0x7f160406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f1609760700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f160406a580 0x7f160406aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42816/0 (socket says 172.21.15.111:42816) 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f1609f61700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f16040b1410 0x7f16040ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49258/0 (socket says 172.21.15.111:49258) 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f160a762700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 0x7f160406d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:37042/0 (socket says 172.21.15.111:37042) 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.792+0000 7f1609760700 1 -- 172.21.15.111:0/2973318015 learned_addr learned my addr 172.21.15.111:0/2973318015 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1609760700 1 -- 172.21.15.111:0/2973318015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 msgr2=0x7f160406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1609760700 1 --2- 172.21.15.111:0/2973318015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 0x7f160406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1609760700 1 -- 172.21.15.111:0/2973318015 >> v1:172.21.15.143:6789/0 conn(0x7f16040b1410 legacy=0x7f16040ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1609760700 1 -- 172.21.15.111:0/2973318015 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160406b8f0 con 0x7f160406a580 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1609760700 1 --2- 172.21.15.111:0/2973318015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f160406aa00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f15f40099b0 tx=0x7f15f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3038b31cf443b64 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1608f5f700 1 -- 172.21.15.111:0/2973318015 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4009ea0 con 0x7f160406a580 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1608f5f700 1 -- 172.21.15.111:0/2973318015 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f400b860 con 0x7f160406a580 2023-12-19T07:47:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f1608f5f700 1 -- 172.21.15.111:0/2973318015 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f400bc00 con 0x7f160406a580 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 -- 172.21.15.111:0/2973318015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 msgr2=0x7f160406aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 --2- 172.21.15.111:0/2973318015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f160406aa00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f15f40099b0 tx=0x7f15f4009250 comp rx=0 tx=0).stop 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 -- 172.21.15.111:0/2973318015 shutdown_connections 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 --2- 172.21.15.111:0/2973318015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f160406d210 0x7f160406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 --2- 172.21.15.111:0/2973318015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f160406aa00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 -- 172.21.15.111:0/2973318015 >> 172.21.15.111:0/2973318015 conn(0x7f1604067bb0 msgr2=0x7f16040cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 -- 172.21.15.111:0/2973318015 shutdown_connections 2023-12-19T07:47:52.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.796+0000 7f160af63700 1 -- 172.21.15.111:0/2973318015 wait complete. 2023-12-19T07:47:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 Processor -- start 2023-12-19T07:47:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 -- start start 2023-12-19T07:47:52.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 0x7f1604148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 0x7f160414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16040c0ed0 con 0x7f160406a580 2023-12-19T07:47:52.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604149140 con 0x7f160406d210 2023-12-19T07:47:52.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160af63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1604158b30 con 0x7f16040b1410 2023-12-19T07:47:52.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42824/0 (socket says 172.21.15.111:42824) 2023-12-19T07:47:52.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 -- 172.21.15.111:0/383840600 learned_addr learned my addr 172.21.15.111:0/383840600 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:52.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f160a762700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 0x7f160414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 -- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 msgr2=0x7f160414b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:52.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609760700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 0x7f1604148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 0x7f160414b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 -- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 msgr2=0x7f1604148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 0x7f1604148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.800+0000 7f1609f61700 1 -- 172.21.15.111:0/383840600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f40094f0 con 0x7f160406a580 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f160a762700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 0x7f160414b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f1609760700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 0x7f1604148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f1609f61700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f160000c560 tx=0x7f1600007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600005a70 con 0x7f160406a580 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1600005bd0 con 0x7f160406a580 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1604157ca0 con 0x7f160406a580 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16000070d0 con 0x7f160406a580 2023-12-19T07:47:52.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1604158200 con 0x7f160406a580 2023-12-19T07:47:52.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160404be40 con 0x7f160406a580 2023-12-19T07:47:52.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.804+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1600007480 con 0x7f160406a580 2023-12-19T07:47:52.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.808+0000 7f15f9ffb700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 0x7f15e8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:52.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.808+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f160007c000 con 0x7f160406a580 2023-12-19T07:47:52.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.808+0000 7f1609760700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 0x7f15e8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:52.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.808+0000 7f1609760700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 0x7f15e8062a30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f15f40099b0 tx=0x7f15f4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:52.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:52.812+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f160004c300 con 0x7f160406a580 2023-12-19T07:47:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f16040cc340 con 0x7f160406a580 2023-12-19T07:47:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f15f9ffb700 1 -- 172.21.15.111:0/383840600 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1600017070 con 0x7f160406a580 2023-12-19T07:47:53.018 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 msgr2=0x7f15e8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 0x7f15e8062a30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f15f40099b0 tx=0x7f15f4017000 comp rx=0 tx=0).stop 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 msgr2=0x7f1604157760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f160000c560 tx=0x7f1600007f70 comp rx=0 tx=0).stop 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 shutdown_connections 2023-12-19T07:47:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f15e8060580 0x7f15e8062a30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f160406d210 0x7f1604148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16040b1410 0x7f160414b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 --2- 172.21.15.111:0/383840600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f160406a580 0x7f1604157760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 >> 172.21.15.111:0/383840600 conn(0x7f1604067bb0 msgr2=0x7f16040b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 shutdown_connections 2023-12-19T07:47:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:53.036+0000 7f160af63700 1 -- 172.21.15.111:0/383840600 wait complete. 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi143.front.sepia.ceph.com 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:47:53.041 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-12-19T07:47:53.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.042 DEBUG:teuthology.orchestra.run.smithi143:> ip addr 2023-12-19T07:47:53.063 INFO:teuthology.orchestra.run.smithi143.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:53.063 INFO:teuthology.orchestra.run.smithi143.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:53.063 INFO:teuthology.orchestra.run.smithi143.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:53.063 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:94 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: altname enp7s0f0 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:95 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: altname enp7s0f1 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:82 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:83 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: inet 172.21.15.143/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft 42368sec preferred_lft 42368sec 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::ec4:7aff:fe88:7783/64 scope link 2023-12-19T07:47:53.064 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 02:42:0e:0d:e8:a5 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 92:cd:3c:37:60:9a brd ff:ff:ff:ff:ff:ff 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::38ed:79ff:fe2b:ec81/64 scope link 2023-12-19T07:47:53.065 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.066 INFO:teuthology.orchestra.run.smithi143.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T07:47:53.066 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 92:cd:3c:37:60:9a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T07:47:53.066 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::90cd:3cff:fe37:609a/64 scope link tentative 2023-12-19T07:47:53.066 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.067 DEBUG:teuthology.orchestra.run.smithi143:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:47:53.149 INFO:teuthology.orchestra.run.smithi143.stdout:1 2023-12-19T07:47:53.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.154 DEBUG:teuthology.orchestra.run.smithi143:> route 2023-12-19T07:47:53.212 INFO:teuthology.orchestra.run.smithi143.stdout:Kernel IP routing table 2023-12-19T07:47:53.212 INFO:teuthology.orchestra.run.smithi143.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:47:53.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.213 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:53.213 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:53.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:47:53.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:47:53.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:47:53.214 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:53.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.353 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-12-19T07:47:53.368 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-19T07:47:53.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.369 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list-id 2023-12-19T07:47:53.429 INFO:teuthology.orchestra.run.smithi143.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-12-19T07:47:53.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.430 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:53.430 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:53.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-19T07:47:53.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-12-19T07:47:53.430 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:53.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-19T07:47:53.572 INFO:teuthology.orchestra.run.smithi143.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:53.573 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 9a:74:e4:55:e5:92 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T07:47:53.574 INFO:teuthology.orchestra.run.smithi143.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T07:47:53.574 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.574 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::9874:e4ff:fe55:e592/64 scope link 2023-12-19T07:47:53.574 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.577 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-19T07:47:53.655 INFO:teuthology.orchestra.run.smithi143.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:47:53.655 INFO:teuthology.orchestra.run.smithi143.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 ::1/128 scope host 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 9a:74:e4:55:e5:92 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::9874:e4ff:fe55:e592/64 scope link 2023-12-19T07:47:53.656 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:47:53.660 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-12-19T07:47:53.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-12-19T07:47:53.660 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:53.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:47:53.853 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-12-19T07:47:53.853 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-12-19T07:47:53.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set brx.1 up 2023-12-19T07:47:53.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set dev brx.1 master ceph-brx 2023-12-19T07:47:53.853 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-12-19T07:47:53.926 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-12-19T07:47:53.926 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:53.927 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:53.945 INFO:teuthology.orchestra.run.smithi143.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-12-19T07:47:53.945 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:47:53.946 DEBUG:teuthology.orchestra.run.smithi143:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:54.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo modprobe fuse 2023-12-19T07:47:54.086 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:54.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:54.164 INFO:teuthology.orchestra.run.smithi143.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:54.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:54.166 DEBUG:teuthology.orchestra.run.smithi143:> ls /sys/fs/fuse/connections 2023-12-19T07:47:54.223 INFO:teuthology.orchestra.run.smithi143.stdout:49 2023-12-19T07:47:54.224 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [49] 2023-12-19T07:47:54.224 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-12-19T07:47:54.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo modprobe fuse 2023-12-19T07:47:54.313 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:54.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:54.389 INFO:teuthology.orchestra.run.smithi143.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:54.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:54.392 DEBUG:teuthology.orchestra.run.smithi143:> ls /sys/fs/fuse/connections 2023-12-19T07:47:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi143.stderr:2023-12-19T07:47:54.432+0000 7f0737bc2380 -1 init, newargv = 0x555fa8b46b90 newargc=15 2023-12-19T07:47:54.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi143.stderr:ceph-fuse[39432]: starting ceph client 2023-12-19T07:47:54.425 INFO:teuthology.orchestra.run.smithi143.stdout:49 2023-12-19T07:47:54.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi143.stderr:ceph-fuse[39432]: starting fuse 2023-12-19T07:47:55.527 DEBUG:teuthology.orchestra.run.smithi143:> sudo modprobe fuse 2023-12-19T07:47:55.566 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:47:55.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:47:55.646 INFO:teuthology.orchestra.run.smithi143.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:47:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:47:55.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:55.649 DEBUG:teuthology.orchestra.run.smithi143:> ls /sys/fs/fuse/connections 2023-12-19T07:47:55.708 INFO:teuthology.orchestra.run.smithi143.stdout:49 2023-12-19T07:47:55.708 INFO:teuthology.orchestra.run.smithi143.stdout:50 2023-12-19T07:47:55.708 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49, 50] 2023-12-19T07:47:55.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:47:55.708 DEBUG:teuthology.orchestra.run.smithi143:> import glob 2023-12-19T07:47:55.708 DEBUG:teuthology.orchestra.run.smithi143:> import re 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> import os 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> import subprocess 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> def _find_admin_socket(client_name): 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> files = glob.glob(asok_path) 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> # Given a non-glob path, it better be there 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> if "*" not in asok_path: 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> assert(len(files) == 1) 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> return files[0] 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> for f in files: 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:47:55.709 DEBUG:teuthology.orchestra.run.smithi143:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> contents = proc_f.read() 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> if mountpoint in contents: 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> return f 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> print(_find_admin_socket("client.3")) 2023-12-19T07:47:55.710 DEBUG:teuthology.orchestra.run.smithi143:> ' 2023-12-19T07:47:55.872 INFO:teuthology.orchestra.run.smithi143.stdout:/var/run/ceph/ceph-client.3.39432.asok 2023-12-19T07:47:56.059 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.39432.asok 2023-12-19T07:47:56.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.39432.asok status 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "metadata": { 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "entity_id": "3", 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "hostname": "smithi143", 2023-12-19T07:47:56.250 INFO:teuthology.orchestra.run.smithi143.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "pid": "39432", 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "root": "/" 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "dentry_count": 0, 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "dentry_pinned_count": 0, 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "id": 4619, 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "inst": { 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "name": { 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "client", 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "num": 4619 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": { 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "192.168.0.2:0", 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1351550919 2023-12-19T07:47:56.251 INFO:teuthology.orchestra.run.smithi143.stdout: } 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": { 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "type": "v1", 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "addr": "192.168.0.2:0", 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "nonce": 1351550919 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inst_str": "client.4619 192.168.0.2:0/1351550919", 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "addr_str": "192.168.0.2:0/1351550919", 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inode_count": 1, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_epoch": 8, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_epoch": 25, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "blocklisted": false, 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout: "fs_name": "cephfs" 2023-12-19T07:47:56.252 INFO:teuthology.orchestra.run.smithi143.stdout:} 2023-12-19T07:47:56.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.263 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T07:47:56.264 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:56.284 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T07:47:56.284 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:56.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.284 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T07:47:56.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.359 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T07:47:56.359 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:56.423 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T07:47:56.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:56.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-12-19T07:47:56.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.502 DEBUG:teuthology.orchestra.run.smithi143:> stat --file-system '--printf=%T 2023-12-19T07:47:56.502 DEBUG:teuthology.orchestra.run.smithi143:> ' -- /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:56.524 INFO:teuthology.orchestra.run.smithi143.stdout:fuseblk 2023-12-19T07:47:56.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:56.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-19T07:47:56.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.604 DEBUG:teuthology.orchestra.run.smithi143:> stat --file-system '--printf=%T 2023-12-19T07:47:56.604 DEBUG:teuthology.orchestra.run.smithi143:> ' -- /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:56.669 INFO:teuthology.orchestra.run.smithi143.stdout:fuseblk 2023-12-19T07:47:56.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:56.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:47:56.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-12-19T07:47:56.743 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-12-19T07:47:56.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:56.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.951+0000 7f3398f85700 1 Processor -- start 2023-12-19T07:47:56.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.951+0000 7f3398f85700 1 -- start start 2023-12-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3398f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f33940aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3398f85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 0x7f33940c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3398f85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339406c9f0 con 0x7f33940c1a80 2023-12-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3398f85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33940c7f20 con 0x7f33940c2980 2023-12-19T07:47:56.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3398f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33940c8190 con 0x7f339413f890 2023-12-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f33940aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3391d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 0x7f33940c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3392d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f33940c1a80 0x7f33940680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49272/0 (socket says 172.21.15.111:49272) 2023-12-19T07:47:56.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3392d9d700 1 -- 172.21.15.111:0/4276215887 learned_addr learned my addr 172.21.15.111:0/4276215887 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339259c700 1 -- 172.21.15.111:0/4276215887 >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 msgr2=0x7f33940c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339259c700 1 --2- 172.21.15.111:0/4276215887 >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 0x7f33940c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339259c700 1 -- 172.21.15.111:0/4276215887 >> v1:172.21.15.143:6789/0 conn(0x7f33940c1a80 legacy=0x7f33940680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339259c700 1 -- 172.21.15.111:0/4276215887 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33940c8470 con 0x7f339413f890 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f339159a700 1 -- 172.21.15.111:0/4276215887 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427729157 0 0) 0x7f339406c9f0 con 0x7f33940c1a80 2023-12-19T07:47:56.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.955+0000 7f3391d9b700 1 --2- 172.21.15.111:0/4276215887 >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 0x7f33940c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f339259c700 1 --2- 172.21.15.111:0/4276215887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f33940aeba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33840098e0 tx=0x7f33840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d0e644ca39f114 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f339159a700 1 -- 172.21.15.111:0/4276215887 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338400b640 con 0x7f339413f890 2023-12-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f339159a700 1 -- 172.21.15.111:0/4276215887 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338400b7a0 con 0x7f339413f890 2023-12-19T07:47:56.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f339159a700 1 -- 172.21.15.111:0/4276215887 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338400bb40 con 0x7f339413f890 2023-12-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 -- 172.21.15.111:0/4276215887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 msgr2=0x7f33940aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:56.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 --2- 172.21.15.111:0/4276215887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f33940aeba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33840098e0 tx=0x7f33840091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 -- 172.21.15.111:0/4276215887 shutdown_connections 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 --2- 172.21.15.111:0/4276215887 >> v2:172.21.15.143:3300/0 conn(0x7f33940c2980 0x7f33940c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 --2- 172.21.15.111:0/4276215887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f33940aeba0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 -- 172.21.15.111:0/4276215887 >> 172.21.15.111:0/4276215887 conn(0x7f33940c8850 msgr2=0x7f33940c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 -- 172.21.15.111:0/4276215887 shutdown_connections 2023-12-19T07:47:56.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.959+0000 7f3398f85700 1 -- 172.21.15.111:0/4276215887 wait complete. 2023-12-19T07:47:56.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 Processor -- start 2023-12-19T07:47:56.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 -- start start 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f339406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 0x7f33940a91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339406d860 con 0x7f33940c2980 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394168a40 con 0x7f339406d3c0 2023-12-19T07:47:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.963+0000 7f3398f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339413cfe0 con 0x7f339413f890 2023-12-19T07:47:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f339259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f339259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42834/0 (socket says 172.21.15.111:42834) 2023-12-19T07:47:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f339259c700 1 -- 172.21.15.111:0/4257321148 learned_addr learned my addr 172.21.15.111:0/4257321148 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 0x7f33940a91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3391d9b700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f339406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 -- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 msgr2=0x7f339406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f339406ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 -- 172.21.15.111:0/4257321148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 msgr2=0x7f339413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 -- 172.21.15.111:0/4257321148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3388009420 con 0x7f339406d3c0 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f339259c700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3392d9d700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 0x7f33940a91d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33940b1dc0 tx=0x7f338c0019a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:56.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c00a880 con 0x7f339406d3c0 2023-12-19T07:47:56.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c00aa20 con 0x7f339406d3c0 2023-12-19T07:47:56.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c00ade0 con 0x7f339406d3c0 2023-12-19T07:47:56.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3384009420 con 0x7f339406d3c0 2023-12-19T07:47:56.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.967+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394168110 con 0x7f339406d3c0 2023-12-19T07:47:56.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.971+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394071120 con 0x7f339406d3c0 2023-12-19T07:47:56.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.971+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f338c007300 con 0x7f339406d3c0 2023-12-19T07:47:56.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.971+0000 7f33827fc700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 0x7f3374062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:56.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.971+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f338c07cef0 con 0x7f339406d3c0 2023-12-19T07:47:56.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.971+0000 7f339259c700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 0x7f3374062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:56.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.975+0000 7f339259c700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 0x7f3374062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f338400b770 tx=0x7f3384017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:56.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:56.979+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f338c04d150 con 0x7f339406d3c0 2023-12-19T07:47:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.159+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3394168b80 con 0x7f339406d3c0 2023-12-19T07:47:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.159+0000 7f33827fc700 1 -- 172.21.15.111:0/4257321148 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17239 (secure 0 0 0) 0x7f338c050770 con 0x7f339406d3c0 2023-12-19T07:47:57.145 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:57.145 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:23.298000+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 msgr2=0x7f3374062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 0x7f3374062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f338400b770 tx=0x7f3384017000 comp rx=0 tx=0).stop 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 msgr2=0x7f33940a91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 0x7f33940a91d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33940b1dc0 tx=0x7f338c0019a0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 shutdown_connections 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3374060550 0x7f3374062a00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f339406d3c0 0x7f33940a91d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f339413f890 0x7f339406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 --2- 172.21.15.111:0/4257321148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940c2980 0x7f339413caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 >> 172.21.15.111:0/4257321148 conn(0x7f33940c8850 msgr2=0x7f33940c1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:57.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 shutdown_connections 2023-12-19T07:47:57.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.163+0000 7f3398f85700 1 -- 172.21.15.111:0/4257321148 wait complete. 2023-12-19T07:47:57.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:47:57.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:57.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.351+0000 7fdd8bf45700 1 Processor -- start 2023-12-19T07:47:57.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.351+0000 7fdd8bf45700 1 -- start start 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8bf45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd840cecd0 0x7fdd8406b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8bf45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840cbdf0 con 0x7fdd840cb970 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8bf45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840cc0d0 con 0x7fdd84158e20 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8bf45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd840cc3b0 con 0x7fdd840cecd0 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd894e0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdd840cb970 0x7fdd8406acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47276/0 (socket says 172.21.15.111:47276) 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd894e0700 1 -- 172.21.15.111:0/1531445080 learned_addr learned my addr 172.21.15.111:0/1531445080 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8a4e2700 1 --2- 172.21.15.111:0/1531445080 >> v2:172.21.15.143:3300/0 conn(0x7fdd840cecd0 0x7fdd8406b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd88cdf700 1 -- 172.21.15.111:0/1531445080 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892073369 0 0) 0x7fdd840cbdf0 con 0x7fdd840cb970 2023-12-19T07:47:57.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd88cdf700 1 -- 172.21.15.111:0/1531445080 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd70003570 con 0x7fdd840cb970 2023-12-19T07:47:57.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/1531445080 >> v1:172.21.15.143:6789/0 conn(0x7fdd84158e20 legacy=0x7fdd840ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/1531445080 >> v1:172.21.15.111:6789/0 conn(0x7fdd840cb970 legacy=0x7fdd8406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.355+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/1531445080 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840cc690 con 0x7fdd840cecd0 2023-12-19T07:47:57.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8a4e2700 1 --2- 172.21.15.111:0/1531445080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd8406b3d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009e90 tx=0x7fdd7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c93c69bc187b46 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd88cdf700 1 -- 172.21.15.111:0/1531445080 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c00e070 con 0x7fdd840cecd0 2023-12-19T07:47:57.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd88cdf700 1 -- 172.21.15.111:0/1531445080 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c00bc40 con 0x7fdd840cecd0 2023-12-19T07:47:57.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd88cdf700 1 -- 172.21.15.111:0/1531445080 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c012430 con 0x7fdd840cecd0 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- 172.21.15.111:0/1531445080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 msgr2=0x7fdd8406b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/1531445080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd8406b3d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009e90 tx=0x7fdd7c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- 172.21.15.111:0/1531445080 shutdown_connections 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/1531445080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd8406b3d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- 172.21.15.111:0/1531445080 >> 172.21.15.111:0/1531445080 conn(0x7fdd84067bb0 msgr2=0x7fdd840d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- 172.21.15.111:0/1531445080 shutdown_connections 2023-12-19T07:47:57.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- 172.21.15.111:0/1531445080 wait complete. 2023-12-19T07:47:57.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 Processor -- start 2023-12-19T07:47:57.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.359+0000 7fdd8bf45700 1 -- start start 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd840b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 0x7fdd8414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840c7230 con 0x7fdd840c4fa0 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840c74d0 con 0x7fdd840cecd0 2023-12-19T07:47:57.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd840c77b0 con 0x7fdd84158e20 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42850/0 (socket says 172.21.15.111:42850) 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/2279161251 learned_addr learned my addr 172.21.15.111:0/2279161251 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd89ce1700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd840b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd894e0700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 0x7fdd8414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 msgr2=0x7fdd8414f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 0x7fdd8414f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 msgr2=0x7fdd840b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd840b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 -- 172.21.15.111:0/2279161251 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c009a20 con 0x7fdd840c4fa0 2023-12-19T07:47:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd894e0700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 0x7fdd8414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8a4e2700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c013000 tx=0x7fdd7c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c012ea0 con 0x7fdd840c4fa0 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c01b630 con 0x7fdd840c4fa0 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c01b9f0 con 0x7fdd840c4fa0 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd840c7a90 con 0x7fdd840c4fa0 2023-12-19T07:47:57.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.363+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd840b5c30 con 0x7fdd840c4fa0 2023-12-19T07:47:57.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.367+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840b5e20 con 0x7fdd840c4fa0 2023-12-19T07:47:57.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.367+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdd7c01c400 con 0x7fdd840c4fa0 2023-12-19T07:47:57.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.371+0000 7fdd79ffb700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 0x7fdd68062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.371+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fdd7c07e5a0 con 0x7fdd840c4fa0 2023-12-19T07:47:57.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.371+0000 7fdd89ce1700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 0x7fdd68062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.371+0000 7fdd89ce1700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 0x7fdd68062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd80003a80 tx=0x7fdd80007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.375+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd7c052340 con 0x7fdd840c4fa0 2023-12-19T07:47:57.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.543+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd84071120 con 0x7fdd840c4fa0 2023-12-19T07:47:57.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.543+0000 7fdd79ffb700 1 -- 172.21.15.111:0/2279161251 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd7c051ed0 con 0x7fdd840c4fa0 2023-12-19T07:47:57.527 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 msgr2=0x7fdd68062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 0x7fdd68062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd80003a80 tx=0x7fdd80007ec0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 msgr2=0x7fdd840c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c013000 tx=0x7fdd7c019000 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 shutdown_connections 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd68060550 0x7fdd68062a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd840c4fa0 0x7fdd840c9710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd84158e20 0x7fdd8414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 --2- 172.21.15.111:0/2279161251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd840cecd0 0x7fdd840b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 >> 172.21.15.111:0/2279161251 conn(0x7fdd84067bb0 msgr2=0x7fdd840c5910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 shutdown_connections 2023-12-19T07:47:57.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.547+0000 7fdd8bf45700 1 -- 172.21.15.111:0/2279161251 wait complete. 2023-12-19T07:47:57.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:47:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.727+0000 7f0636643700 1 Processor -- start 2023-12-19T07:47:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.727+0000 7f0636643700 1 -- start start 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0636643700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f063006bb20 0x7f06300ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0636643700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063006c9d0 con 0x7f063015d000 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0636643700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063006ccb0 con 0x7f063015d3d0 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0636643700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f063006e180 con 0x7f063006bb20 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635e42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f063006bb20 0x7f06300ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635641700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f063015d000 0x7f063006da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47288/0 (socket says 172.21.15.111:47288) 2023-12-19T07:47:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635641700 1 -- 172.21.15.111:0/2413090399 learned_addr learned my addr 172.21.15.111:0/2413090399 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635e42700 1 -- 172.21.15.111:0/2413090399 >> v1:172.21.15.143:6789/0 conn(0x7f063015d3d0 legacy=0x7f063006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635e42700 1 -- 172.21.15.111:0/2413090399 >> v1:172.21.15.111:6789/0 conn(0x7f063015d000 legacy=0x7f063006da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635e42700 1 -- 172.21.15.111:0/2413090399 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063006cf90 con 0x7f063006bb20 2023-12-19T07:47:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0635e42700 1 --2- 172.21.15.111:0/2413090399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300ca7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e60 tx=0x7f0628009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f459af1f068d48bd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0627fff700 1 -- 172.21.15.111:0/2413090399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062800e070 con 0x7f063006bb20 2023-12-19T07:47:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0627fff700 1 -- 172.21.15.111:0/2413090399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062800bc30 con 0x7f063006bb20 2023-12-19T07:47:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.731+0000 7f0627fff700 1 -- 172.21.15.111:0/2413090399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628012430 con 0x7f063006bb20 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- 172.21.15.111:0/2413090399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 msgr2=0x7f06300ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 --2- 172.21.15.111:0/2413090399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300ca7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e60 tx=0x7f0628009780 comp rx=0 tx=0).stop 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- 172.21.15.111:0/2413090399 shutdown_connections 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 --2- 172.21.15.111:0/2413090399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300ca7b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- 172.21.15.111:0/2413090399 >> 172.21.15.111:0/2413090399 conn(0x7f0630067bb0 msgr2=0x7f06300cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- 172.21.15.111:0/2413090399 shutdown_connections 2023-12-19T07:47:57.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- 172.21.15.111:0/2413090399 wait complete. 2023-12-19T07:47:57.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 Processor -- start 2023-12-19T07:47:57.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.735+0000 7f0636643700 1 -- start start 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f063015d000 0x7f06301550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063015d3d0 0x7f06301500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630160370 con 0x7f063006bb20 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630160c10 con 0x7f063015d000 2023-12-19T07:47:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06301558f0 con 0x7f063015d3d0 2023-12-19T07:47:57.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42870/0 (socket says 172.21.15.111:42870) 2023-12-19T07:47:57.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 -- 172.21.15.111:0/4055532223 learned_addr learned my addr 172.21.15.111:0/4055532223 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:57.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0634e40700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f063015d000 0x7f06301550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635e42700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063015d3d0 0x7f06301500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 -- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063015d3d0 msgr2=0x7f06301500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063015d3d0 0x7f06301500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 -- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f063015d000 msgr2=0x7f06301550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f063015d000 0x7f06301550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 -- 172.21.15.111:0/4055532223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0628009a20 con 0x7f063006bb20 2023-12-19T07:47:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0635641700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c7f0 tx=0x7f062c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c005b30 con 0x7f063006bb20 2023-12-19T07:47:57.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c005cb0 con 0x7f063006bb20 2023-12-19T07:47:57.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c004a80 con 0x7f063006bb20 2023-12-19T07:47:57.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06301605c0 con 0x7f063006bb20 2023-12-19T07:47:57.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.739+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06300b3c50 con 0x7f063006bb20 2023-12-19T07:47:57.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.743+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063004be40 con 0x7f063006bb20 2023-12-19T07:47:57.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.743+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f062c0070f0 con 0x7f063006bb20 2023-12-19T07:47:57.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.743+0000 7f06257fa700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 0x7f0614062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:57.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.743+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f062c07bfa0 con 0x7f063006bb20 2023-12-19T07:47:57.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.743+0000 7f0634e40700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 0x7f0614062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:57.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.747+0000 7f0634e40700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 0x7f0614062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06300b5320 tx=0x7f0620009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:57.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.751+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f062c04c280 con 0x7f063006bb20 2023-12-19T07:47:57.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06300683a0 con 0x7f063006bb20 2023-12-19T07:47:57.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f06257fa700 1 -- 172.21.15.111:0/4055532223 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f062c0180a0 con 0x7f063006bb20 2023-12-19T07:47:57.898 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:57.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 msgr2=0x7f0614062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 0x7f0614062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06300b5320 tx=0x7f0620009040 comp rx=0 tx=0).stop 2023-12-19T07:47:57.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 msgr2=0x7f06300b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:57.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.915+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c7f0 tx=0x7f062c007f70 comp rx=0 tx=0).stop 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 shutdown_connections 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0614060550 0x7f0614062a00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f063015d000 0x7f06301550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063015d3d0 0x7f06301500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 --2- 172.21.15.111:0/4055532223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f063006bb20 0x7f06300b3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 >> 172.21.15.111:0/4055532223 conn(0x7f0630067bb0 msgr2=0x7f06300b01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 shutdown_connections 2023-12-19T07:47:57.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:57.919+0000 7f0636643700 1 -- 172.21.15.111:0/4055532223 wait complete. 2023-12-19T07:47:57.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:47:58.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.103+0000 7fae8a4f0700 1 Processor -- start 2023-12-19T07:47:58.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.103+0000 7fae8a4f0700 1 -- start start 2023-12-19T07:47:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae8a4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae8a4f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae8406ced0 0x7fae840b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae8a4f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae840b0280 con 0x7fae8406d8b0 2023-12-19T07:47:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae8a4f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8406b4e0 con 0x7fae8406ced0 2023-12-19T07:47:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae8a4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8406b7c0 con 0x7fae84158fa0 2023-12-19T07:47:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae83fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae83fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:37106/0 (socket says 172.21.15.111:37106) 2023-12-19T07:47:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae83fff700 1 -- 172.21.15.111:0/3253189519 learned_addr learned my addr 172.21.15.111:0/3253189519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae837fe700 1 --1- 172.21.15.111:0/3253189519 >> v1:172.21.15.143:6789/0 conn(0x7fae8406d8b0 0x7fae8406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49310/0 (socket says 172.21.15.111:49310) 2023-12-19T07:47:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 --2- 172.21.15.111:0/3253189519 >> v2:172.21.15.143:3300/0 conn(0x7fae8406ced0 0x7fae840b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 -- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 msgr2=0x7fae8406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 -- 172.21.15.111:0/3253189519 >> v1:172.21.15.143:6789/0 conn(0x7fae8406d8b0 legacy=0x7fae8406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 -- 172.21.15.111:0/3253189519 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8406baa0 con 0x7fae8406ced0 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae82ffd700 1 -- 172.21.15.111:0/3253189519 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212311421 0 0) 0x7fae840b0280 con 0x7fae8406d8b0 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae83fff700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:47:58.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae88a8d700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae8406ced0 0x7fae840b06f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009e60 tx=0x7fae7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba2d16ace3bae86 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae82ffd700 1 -- 172.21.15.111:0/3253189519 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00e070 con 0x7fae8406ced0 2023-12-19T07:47:58.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae82ffd700 1 -- 172.21.15.111:0/3253189519 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c00bb60 con 0x7fae8406ced0 2023-12-19T07:47:58.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.107+0000 7fae82ffd700 1 -- 172.21.15.111:0/3253189519 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c012430 con 0x7fae8406ced0 2023-12-19T07:47:58.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 -- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae8406ced0 msgr2=0x7fae840b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae8406ced0 0x7fae840b06f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009e60 tx=0x7fae7c009780 comp rx=0 tx=0).stop 2023-12-19T07:47:58.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 -- 172.21.15.111:0/3253189519 shutdown_connections 2023-12-19T07:47:58.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae8406ced0 0x7fae840b06f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/3253189519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae84158fa0 0x7fae8406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 -- 172.21.15.111:0/3253189519 >> 172.21.15.111:0/3253189519 conn(0x7fae840d0b90 msgr2=0x7fae840ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:58.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 -- 172.21.15.111:0/3253189519 shutdown_connections 2023-12-19T07:47:58.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.111+0000 7fae8a4f0700 1 -- 172.21.15.111:0/3253189519 wait complete. 2023-12-19T07:47:58.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 Processor -- start 2023-12-19T07:47:58.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 -- start start 2023-12-19T07:47:58.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae8406ced0 0x7fae840caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae84158fa0 0x7fae840ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae840cb5d0 con 0x7fae8406d8b0 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae84157d70 con 0x7fae84158fa0 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.115+0000 7fae8a4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae84169790 con 0x7fae8406ced0 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42884/0 (socket says 172.21.15.111:42884) 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 -- 172.21.15.111:0/2051735909 learned_addr learned my addr 172.21.15.111:0/2051735909 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae83fff700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae8406ced0 0x7fae840caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae88a8d700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae84158fa0 0x7fae840ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 -- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae8406ced0 msgr2=0x7fae840caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae8406ced0 0x7fae840caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 -- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae84158fa0 msgr2=0x7fae840ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae84158fa0 0x7fae840ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 -- 172.21.15.111:0/2051735909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740094f0 con 0x7fae8406d8b0 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae837fe700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae780011e0 tx=0x7fae78007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78005a70 con 0x7fae8406d8b0 2023-12-19T07:47:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae78005bd0 con 0x7fae8406d8b0 2023-12-19T07:47:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c009a20 con 0x7fae8406d8b0 2023-12-19T07:47:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780070d0 con 0x7fae8406d8b0 2023-12-19T07:47:58.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.119+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae84169d10 con 0x7fae8406d8b0 2023-12-19T07:47:58.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.123+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae84071120 con 0x7fae8406d8b0 2023-12-19T07:47:58.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.123+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fae780074c0 con 0x7fae8406d8b0 2023-12-19T07:47:58.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.127+0000 7fae6bfff700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 0x7fae70062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.127+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fae7807bfd0 con 0x7fae8406d8b0 2023-12-19T07:47:58.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.127+0000 7fae83fff700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 0x7fae70062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.127+0000 7fae83fff700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 0x7fae70062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fae74006d20 tx=0x7fae74009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.131+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fae7804c230 con 0x7fae8406d8b0 2023-12-19T07:47:58.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.299+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae840cf0d0 con 0x7fae8406d8b0 2023-12-19T07:47:58.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.303+0000 7fae6bfff700 1 -- 172.21.15.111:0/2051735909 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae780170a0 con 0x7fae8406d8b0 2023-12-19T07:47:58.286 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 msgr2=0x7fae70062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 0x7fae70062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fae74006d20 tx=0x7fae74009250 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 msgr2=0x7fae84156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae780011e0 tx=0x7fae78007e10 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 shutdown_connections 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae70060550 0x7fae70062a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae84158fa0 0x7fae840ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae8406d8b0 0x7fae84156d50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 --2- 172.21.15.111:0/2051735909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae8406ced0 0x7fae840caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 >> 172.21.15.111:0/2051735909 conn(0x7fae840d0b90 msgr2=0x7fae840b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:58.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 shutdown_connections 2023-12-19T07:47:58.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.307+0000 7fae8a4f0700 1 -- 172.21.15.111:0/2051735909 wait complete. 2023-12-19T07:47:58.312 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:47:58.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 Processor -- start 2023-12-19T07:47:58.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 -- start start 2023-12-19T07:47:58.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe688153800 0x7fe68815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 0x7fe6880cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68806c850 con 0x7fe688153430 2023-12-19T07:47:58.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6880cf170 con 0x7fe688153800 2023-12-19T07:47:58.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe68d345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68815a5b0 con 0x7fe6880ce790 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe686ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe688153430 0x7fe688159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47290/0 (socket says 172.21.15.111:47290) 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe686ffd700 1 -- 172.21.15.111:0/1952290004 learned_addr learned my addr 172.21.15.111:0/1952290004 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.499+0000 7fe6867fc700 1 --2- 172.21.15.111:0/1952290004 >> v2:172.21.15.111:3300/0 conn(0x7fe688153800 0x7fe68815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6877fe700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 0x7fe6880cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe685ffb700 1 -- 172.21.15.111:0/1952290004 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126206522 0 0) 0x7fe68806c850 con 0x7fe688153430 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe685ffb700 1 -- 172.21.15.111:0/1952290004 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe670003570 con 0x7fe688153430 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6867fc700 1 -- 172.21.15.111:0/1952290004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 msgr2=0x7fe6880cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6867fc700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 0x7fe6880cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6867fc700 1 -- 172.21.15.111:0/1952290004 >> v1:172.21.15.111:6789/0 conn(0x7fe688153430 legacy=0x7fe688159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6867fc700 1 -- 172.21.15.111:0/1952290004 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68815d1c0 con 0x7fe688153800 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6877fe700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 0x7fe6880cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe6867fc700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe688153800 0x7fe68815cbe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009910 tx=0x7fe67c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d721a7ac688b2a86 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe685ffb700 1 -- 172.21.15.111:0/1952290004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00b640 con 0x7fe688153800 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe685ffb700 1 -- 172.21.15.111:0/1952290004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c00b7a0 con 0x7fe688153800 2023-12-19T07:47:58.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe685ffb700 1 -- 172.21.15.111:0/1952290004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00bb60 con 0x7fe688153800 2023-12-19T07:47:58.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 -- 172.21.15.111:0/1952290004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe688153800 msgr2=0x7fe68815cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe688153800 0x7fe68815cbe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009910 tx=0x7fe67c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 -- 172.21.15.111:0/1952290004 shutdown_connections 2023-12-19T07:47:58.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6880ce790 0x7fe6880cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 --2- 172.21.15.111:0/1952290004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe688153800 0x7fe68815cbe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 -- 172.21.15.111:0/1952290004 >> 172.21.15.111:0/1952290004 conn(0x7fe6880cd720 msgr2=0x7fe68806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:58.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 -- 172.21.15.111:0/1952290004 shutdown_connections 2023-12-19T07:47:58.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.503+0000 7fe68d345700 1 -- 172.21.15.111:0/1952290004 wait complete. 2023-12-19T07:47:58.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 Processor -- start 2023-12-19T07:47:58.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 -- start start 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe688153430 0x7fe68815b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe688153800 0x7fe688150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe688153e70 con 0x7fe6880ce790 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68815c8f0 con 0x7fe688153800 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe68d345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe688154000 con 0x7fe688153430 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42900/0 (socket says 172.21.15.111:42900) 2023-12-19T07:47:58.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 -- 172.21.15.111:0/2627976121 learned_addr learned my addr 172.21.15.111:0/2627976121 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe6867fc700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe688153430 0x7fe68815b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe6877fe700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe688153800 0x7fe688150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 -- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe688153430 msgr2=0x7fe68815b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe688153430 0x7fe68815b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 -- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe688153800 msgr2=0x7fe688150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe688153800 0x7fe688150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 -- 172.21.15.111:0/2627976121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c009420 con 0x7fe6880ce790 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.507+0000 7fe686ffd700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c7c0 tx=0x7fe678007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678005b30 con 0x7fe6880ce790 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678005c90 con 0x7fe6880ce790 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678007300 con 0x7fe6880ce790 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6880c63b0 con 0x7fe6880ce790 2023-12-19T07:47:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6880c6970 con 0x7fe6880ce790 2023-12-19T07:47:58.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688071120 con 0x7fe6880ce790 2023-12-19T07:47:58.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.511+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe67801b3b0 con 0x7fe6880ce790 2023-12-19T07:47:58.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.515+0000 7fe66effd700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.515+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fe67807c080 con 0x7fe6880ce790 2023-12-19T07:47:58.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.515+0000 7fe6867fc700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.519+0000 7fe6867fc700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009860 tx=0x7fe67c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.519+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe67804c2e0 con 0x7fe6880ce790 2023-12-19T07:47:58.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.691+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe68815d4a0 con 0x7fe6880ce790 2023-12-19T07:47:58.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.691+0000 7fe66effd700 1 -- 172.21.15.111:0/2627976121 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe6780170a0 con 0x7fe6880ce790 2023-12-19T07:47:58.675 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 msgr2=0x7fe674062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009860 tx=0x7fe67c017000 comp rx=0 tx=0).stop 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 msgr2=0x7fe6880c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c7c0 tx=0x7fe678007e00 comp rx=0 tx=0).stop 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 shutdown_connections 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe688153800 0x7fe688150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe688153430 0x7fe68815b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 --2- 172.21.15.111:0/2627976121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6880ce790 0x7fe6880c5d90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 >> 172.21.15.111:0/2627976121 conn(0x7fe6880cd720 msgr2=0x7fe6880cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 shutdown_connections 2023-12-19T07:47:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.695+0000 7fe68d345700 1 -- 172.21.15.111:0/2627976121 wait complete. 2023-12-19T07:47:58.697 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:47:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 Processor -- start 2023-12-19T07:47:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 -- start start 2023-12-19T07:47:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd160068800 0x7fd160161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16006e150 con 0x7fd160140710 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16006d540 con 0x7fd160140ae0 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd165185700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16006d820 con 0x7fd160068800 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd15ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd160140710 0x7fd1600c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47300/0 (socket says 172.21.15.111:47300) 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd15ed9d700 1 -- 172.21.15.111:0/1321774943 learned_addr learned my addr 172.21.15.111:0/1321774943 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:47:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd15f59e700 1 --2- 172.21.15.111:0/1321774943 >> v2:172.21.15.111:3300/0 conn(0x7fd160068800 0x7fd160161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.871+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255895854 0 0) 0x7fd16006e150 con 0x7fd160140710 2023-12-19T07:47:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c003570 con 0x7fd160140710 2023-12-19T07:47:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807820650 0 0) 0x7fd16006d540 con 0x7fd160140ae0 2023-12-19T07:47:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15f59e700 1 -- 172.21.15.111:0/1321774943 >> v1:172.21.15.143:6789/0 conn(0x7fd160140ae0 legacy=0x7fd1600680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15f59e700 1 -- 172.21.15.111:0/1321774943 >> v1:172.21.15.111:6789/0 conn(0x7fd160140710 legacy=0x7fd1600c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15f59e700 1 -- 172.21.15.111:0/1321774943 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16006db00 con 0x7fd160068800 2023-12-19T07:47:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15f59e700 1 --2- 172.21.15.111:0/1321774943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160068800 0x7fd160161cd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd158009e60 tx=0x7fd158009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8536796061dd89ef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15800e070 con 0x7fd160068800 2023-12-19T07:47:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15800bb60 con 0x7fd160068800 2023-12-19T07:47:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd15e59c700 1 -- 172.21.15.111:0/1321774943 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158012430 con 0x7fd160068800 2023-12-19T07:47:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 -- 172.21.15.111:0/1321774943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160068800 msgr2=0x7fd160161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 --2- 172.21.15.111:0/1321774943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160068800 0x7fd160161cd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd158009e60 tx=0x7fd158009780 comp rx=0 tx=0).stop 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 -- 172.21.15.111:0/1321774943 shutdown_connections 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 --2- 172.21.15.111:0/1321774943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160068800 0x7fd160161cd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 -- 172.21.15.111:0/1321774943 >> 172.21.15.111:0/1321774943 conn(0x7fd1600c8850 msgr2=0x7fd1600c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 -- 172.21.15.111:0/1321774943 shutdown_connections 2023-12-19T07:47:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.875+0000 7fd165185700 1 -- 172.21.15.111:0/1321774943 wait complete. 2023-12-19T07:47:58.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 Processor -- start 2023-12-19T07:47:58.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 -- start start 2023-12-19T07:47:58.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd160068800 0x7fd160156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd160140ae0 0x7fd1600ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16016a0f0 con 0x7fd160140710 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16016a990 con 0x7fd160140ae0 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd165185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd160158f60 con 0x7fd160068800 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd157fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd157fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42910/0 (socket says 172.21.15.111:42910) 2023-12-19T07:47:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd157fff700 1 -- 172.21.15.111:0/351384247 learned_addr learned my addr 172.21.15.111:0/351384247 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:58.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd15f59e700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd160140ae0 0x7fd1600ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.879+0000 7fd15ed9d700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd160068800 0x7fd160156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 -- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd160068800 msgr2=0x7fd160156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd160068800 0x7fd160156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 -- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd160140ae0 msgr2=0x7fd1600ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd160140ae0 0x7fd1600ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 -- 172.21.15.111:0/351384247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd158009a20 con 0x7fd160140710 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd157fff700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd14800c7e0 tx=0x7fd148007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148012700 con 0x7fd160140710 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd148012880 con 0x7fd160140710 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16016a340 con 0x7fd160140710 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148012c40 con 0x7fd160140710 2023-12-19T07:47:58.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd160158b50 con 0x7fd160140710 2023-12-19T07:47:58.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd160071120 con 0x7fd160140710 2023-12-19T07:47:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.883+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd14801b400 con 0x7fd160140710 2023-12-19T07:47:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.887+0000 7fd156ffd700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 0x7fd140062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.887+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd14807c670 con 0x7fd160140710 2023-12-19T07:47:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.887+0000 7fd15ed9d700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 0x7fd140062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:58.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.887+0000 7fd15ed9d700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 0x7fd140062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1600accd0 tx=0x7fd150009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:58.891+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd14804c970 con 0x7fd160140710 2023-12-19T07:47:59.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.063+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1600642a0 con 0x7fd160140710 2023-12-19T07:47:59.053 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:47:59.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.067+0000 7fd156ffd700 1 -- 172.21.15.111:0/351384247 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17540 (secure 0 0 0) 0x7fd14804ff90 con 0x7fd160140710 2023-12-19T07:47:59.053 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:47:59.054745+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 msgr2=0x7fd140062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 0x7fd140062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1600accd0 tx=0x7fd150009040 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 msgr2=0x7fd160158610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd14800c7e0 tx=0x7fd148007df0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 shutdown_connections 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd140060550 0x7fd140062a00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd160140ae0 0x7fd1600ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd160140710 0x7fd160158610 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 --2- 172.21.15.111:0/351384247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd160068800 0x7fd160156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 >> 172.21.15.111:0/351384247 conn(0x7fd1600c8850 msgr2=0x7fd1600ac250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:59.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 shutdown_connections 2023-12-19T07:47:59.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.075+0000 7fd165185700 1 -- 172.21.15.111:0/351384247 wait complete. 2023-12-19T07:47:59.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:47:59.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:47:59.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.259+0000 7f35b99c5700 1 Processor -- start 2023-12-19T07:47:59.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.259+0000 7f35b99c5700 1 -- start start 2023-12-19T07:47:59.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b99c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f35b4158c60 0x7f35b40afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b99c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b40b0060 0x7f35b40670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b99c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b40cb550 con 0x7f35b4067620 2023-12-19T07:47:59.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b99c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b4067aa0 con 0x7f35b4158c60 2023-12-19T07:47:59.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b99c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b406cf40 con 0x7f35b40b0060 2023-12-19T07:47:59.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f35b4158c60 0x7f35b40afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f35b4158c60 0x7f35b40afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42918/0 (socket says 172.21.15.111:42918) 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 -- 172.21.15.111:0/2057924130 learned_addr learned my addr 172.21.15.111:0/2057924130 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b27fc700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b40b0060 0x7f35b40670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 -- 172.21.15.111:0/2057924130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b40b0060 msgr2=0x7f35b40670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b40b0060 0x7f35b40670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 -- 172.21.15.111:0/2057924130 >> v1:172.21.15.143:6789/0 conn(0x7f35b4067620 legacy=0x7f35b406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 -- 172.21.15.111:0/2057924130 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40cb270 con 0x7f35b4158c60 2023-12-19T07:47:59.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b2ffd700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4158c60 0x7f35b40afb20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f35a40099b0 tx=0x7f35a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4737f9953ae6b3b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b1ffb700 1 -- 172.21.15.111:0/2057924130 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400b860 con 0x7f35b4158c60 2023-12-19T07:47:59.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b1ffb700 1 -- 172.21.15.111:0/2057924130 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a400b9c0 con 0x7f35b4158c60 2023-12-19T07:47:59.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.263+0000 7f35b1ffb700 1 -- 172.21.15.111:0/2057924130 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400bd60 con 0x7f35b4158c60 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 -- 172.21.15.111:0/2057924130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4158c60 msgr2=0x7f35b40afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4158c60 0x7f35b40afb20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f35a40099b0 tx=0x7f35a4009250 comp rx=0 tx=0).stop 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 -- 172.21.15.111:0/2057924130 shutdown_connections 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b40b0060 0x7f35b40670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 --2- 172.21.15.111:0/2057924130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4158c60 0x7f35b40afb20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 -- 172.21.15.111:0/2057924130 >> 172.21.15.111:0/2057924130 conn(0x7f35b40cfde0 msgr2=0x7f35b40d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:59.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 -- 172.21.15.111:0/2057924130 shutdown_connections 2023-12-19T07:47:59.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.267+0000 7f35b99c5700 1 -- 172.21.15.111:0/2057924130 wait complete. 2023-12-19T07:47:59.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 Processor -- start 2023-12-19T07:47:59.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 -- start start 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35b40b0060 0x7f35b40c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b4158c60 0x7f35b40cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b415a220 con 0x7f35b4067620 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b4159610 con 0x7f35b40b0060 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b99c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b41598f0 con 0x7f35b4158c60 2023-12-19T07:47:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42928/0 (socket says 172.21.15.111:42928) 2023-12-19T07:47:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 -- 172.21.15.111:0/3844796318 learned_addr learned my addr 172.21.15.111:0/3844796318 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b27fc700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35b40b0060 0x7f35b40c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b37fe700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b4158c60 0x7f35b40cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 -- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b4158c60 msgr2=0x7f35b40cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b4158c60 0x7f35b40cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 -- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35b40b0060 msgr2=0x7f35b40c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35b40b0060 0x7f35b40c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.271+0000 7f35b2ffd700 1 -- 172.21.15.111:0/3844796318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40094f0 con 0x7f35b4067620 2023-12-19T07:47:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f35b2ffd700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f35b406a200 tx=0x7f35a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4024440 con 0x7f35b4067620 2023-12-19T07:47:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a40245a0 con 0x7f35b4067620 2023-12-19T07:47:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4024940 con 0x7f35b4067620 2023-12-19T07:47:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b4159bd0 con 0x7f35b4067620 2023-12-19T07:47:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b40cef70 con 0x7f35b4067620 2023-12-19T07:47:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.275+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b404be40 con 0x7f35b4067620 2023-12-19T07:47:59.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.279+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f35a401a420 con 0x7f35b4067620 2023-12-19T07:47:59.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.279+0000 7f359affd700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 0x7f35a0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.279+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f35a407e810 con 0x7f35b4067620 2023-12-19T07:47:59.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.279+0000 7f35b27fc700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 0x7f35a0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.279+0000 7f35b27fc700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 0x7f35a0062a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35b416c0e0 tx=0x7f35a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.283+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f35a404eae0 con 0x7f35b4067620 2023-12-19T07:47:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35b406c8f0 con 0x7f35b4067620 2023-12-19T07:47:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f359affd700 1 -- 172.21.15.111:0/3844796318 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35a4052100 con 0x7f35b4067620 2023-12-19T07:47:59.438 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 msgr2=0x7f35a0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 0x7f35a0062a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35b416c0e0 tx=0x7f35a8008040 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 msgr2=0x7f35b4159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f35b406a200 tx=0x7f35a4017000 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 shutdown_connections 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35a0060580 0x7f35a0062a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35b40b0060 0x7f35b40c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35b4158c60 0x7f35b40cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 --2- 172.21.15.111:0/3844796318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35b4067620 0x7f35b4159030 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 >> 172.21.15.111:0/3844796318 conn(0x7f35b40cfde0 msgr2=0x7f35b41580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:59.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 shutdown_connections 2023-12-19T07:47:59.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.455+0000 7f35b99c5700 1 -- 172.21.15.111:0/3844796318 wait complete. 2023-12-19T07:47:59.461 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:47:59.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.639+0000 7fa078a43700 1 Processor -- start 2023-12-19T07:47:59.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.639+0000 7fa078a43700 1 -- start start 2023-12-19T07:47:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa078a43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 0x7fa0740caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa078a43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 0x7fa07406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa078a43700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07406c850 con 0x7fa07406aeb0 2023-12-19T07:47:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa078a43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07406b470 con 0x7fa074159770 2023-12-19T07:47:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa078a43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07406b750 con 0x7fa074158c90 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa073fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 0x7fa07406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 0x7fa0740caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa073fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 0x7fa07406cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:50494/0 (socket says 172.21.15.111:50494) 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa073fff700 1 -- 172.21.15.111:0/2663507871 learned_addr learned my addr 172.21.15.111:0/2663507871 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa072ffd700 1 --1- 172.21.15.111:0/2663507871 >> v1:172.21.15.143:6789/0 conn(0x7fa07406aeb0 0x7fa074159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49332/0 (socket says 172.21.15.111:49332) 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 -- 172.21.15.111:0/2663507871 >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 msgr2=0x7fa07406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 --2- 172.21.15.111:0/2663507871 >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 0x7fa07406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 -- 172.21.15.111:0/2663507871 >> v1:172.21.15.143:6789/0 conn(0x7fa07406aeb0 legacy=0x7fa074159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 -- 172.21.15.111:0/2663507871 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07406ba30 con 0x7fa074158c90 2023-12-19T07:47:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0737fe700 1 --2- 172.21.15.111:0/2663507871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 0x7fa0740caf40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0640098e0 tx=0x7fa0640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41d8fd7f9272cf7f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0727fc700 1 -- 172.21.15.111:0/2663507871 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06400b640 con 0x7fa074158c90 2023-12-19T07:47:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0727fc700 1 -- 172.21.15.111:0/2663507871 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06400b7a0 con 0x7fa074158c90 2023-12-19T07:47:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.643+0000 7fa0727fc700 1 -- 172.21.15.111:0/2663507871 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06400bb40 con 0x7fa074158c90 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- 172.21.15.111:0/2663507871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 msgr2=0x7fa0740caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 --2- 172.21.15.111:0/2663507871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 0x7fa0740caf40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0640098e0 tx=0x7fa0640091a0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- 172.21.15.111:0/2663507871 shutdown_connections 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 --2- 172.21.15.111:0/2663507871 >> v2:172.21.15.143:3300/0 conn(0x7fa074159770 0x7fa07406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 --2- 172.21.15.111:0/2663507871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074158c90 0x7fa0740caf40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- 172.21.15.111:0/2663507871 >> 172.21.15.111:0/2663507871 conn(0x7fa074067bb0 msgr2=0x7fa0740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- 172.21.15.111:0/2663507871 shutdown_connections 2023-12-19T07:47:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- 172.21.15.111:0/2663507871 wait complete. 2023-12-19T07:47:59.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 Processor -- start 2023-12-19T07:47:59.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.647+0000 7fa078a43700 1 -- start start 2023-12-19T07:47:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa074158c90 0x7fa074157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074159770 0x7fa0740c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740c6730 con 0x7fa07406aeb0 2023-12-19T07:47:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740c6a10 con 0x7fa074158c90 2023-12-19T07:47:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa078a43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740c6cf0 con 0x7fa074159770 2023-12-19T07:47:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42938/0 (socket says 172.21.15.111:42938) 2023-12-19T07:47:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 -- 172.21.15.111:0/982446904 learned_addr learned my addr 172.21.15.111:0/982446904 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa073fff700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074159770 0x7fa0740c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa072ffd700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa074158c90 0x7fa074157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 -- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074159770 msgr2=0x7fa0740c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074159770 0x7fa0740c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 -- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa074158c90 msgr2=0x7fa074157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa074158c90 0x7fa074157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 -- 172.21.15.111:0/982446904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064009420 con 0x7fa07406aeb0 2023-12-19T07:47:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.651+0000 7fa0737fe700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa07406e2e0 tx=0x7fa064017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06401bcf0 con 0x7fa07406aeb0 2023-12-19T07:47:59.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06401be90 con 0x7fa07406aeb0 2023-12-19T07:47:59.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0740c6fd0 con 0x7fa07406aeb0 2023-12-19T07:47:59.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064011670 con 0x7fa07406aeb0 2023-12-19T07:47:59.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa074153bb0 con 0x7fa07406aeb0 2023-12-19T07:47:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.655+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07404be40 con 0x7fa07406aeb0 2023-12-19T07:47:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.659+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa0640119e0 con 0x7fa07406aeb0 2023-12-19T07:47:59.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.659+0000 7fa05b7fe700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:47:59.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.659+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fa06407e220 con 0x7fa07406aeb0 2023-12-19T07:47:59.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.659+0000 7fa072ffd700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:47:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.659+0000 7fa072ffd700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 0x7fa060062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa068007fd0 tx=0x7fa068007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:47:59.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.663+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa06404e470 con 0x7fa07406aeb0 2023-12-19T07:47:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.843+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa07406bd10 con 0x7fa07406aeb0 2023-12-19T07:47:59.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.843+0000 7fa05b7fe700 1 -- 172.21.15.111:0/982446904 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa064051a90 con 0x7fa07406aeb0 2023-12-19T07:47:59.828 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 msgr2=0x7fa060062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 0x7fa060062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa068007fd0 tx=0x7fa068007d70 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 msgr2=0x7fa074153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa07406e2e0 tx=0x7fa064017000 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 shutdown_connections 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa060060550 0x7fa060062a00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa074158c90 0x7fa074157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa074159770 0x7fa0740c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 --2- 172.21.15.111:0/982446904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa07406aeb0 0x7fa074153670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 >> 172.21.15.111:0/982446904 conn(0x7fa074067bb0 msgr2=0x7fa0740cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 shutdown_connections 2023-12-19T07:47:59.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:47:59.847+0000 7fa078a43700 1 -- 172.21.15.111:0/982446904 wait complete. 2023-12-19T07:47:59.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:48:00.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 Processor -- start 2023-12-19T07:48:00.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 -- start start 2023-12-19T07:48:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24d00cb970 0x7f24d00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d00cbdf0 con 0x7f24d00ca2c0 2023-12-19T07:48:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d00cc0d0 con 0x7f24d00cb970 2023-12-19T07:48:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24d5c6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00cc3b0 con 0x7f24d0158e20 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24ceffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24d00cb970 0x7f24d00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24ceffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24d00cb970 0x7f24d00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42946/0 (socket says 172.21.15.111:42946) 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.039+0000 7f24ceffd700 1 -- 172.21.15.111:0/1120033683 learned_addr learned my addr 172.21.15.111:0/1120033683 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24cf7fe700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ceffd700 1 -- 172.21.15.111:0/1120033683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 msgr2=0x7f24d00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ceffd700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ceffd700 1 -- 172.21.15.111:0/1120033683 >> v1:172.21.15.143:6789/0 conn(0x7f24d00ca2c0 legacy=0x7f24d006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ceffd700 1 -- 172.21.15.111:0/1120033683 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d00cc690 con 0x7f24d00cb970 2023-12-19T07:48:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ceffd700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00cb970 0x7f24d00ceb00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24c40099b0 tx=0x7f24c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54b2bfa5f3ead10 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ce7fc700 1 -- 172.21.15.111:0/1120033683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c400b860 con 0x7f24d00cb970 2023-12-19T07:48:00.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ce7fc700 1 -- 172.21.15.111:0/1120033683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c400b9c0 con 0x7f24d00cb970 2023-12-19T07:48:00.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24ce7fc700 1 -- 172.21.15.111:0/1120033683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c400bd60 con 0x7f24d00cb970 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 -- 172.21.15.111:0/1120033683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00cb970 msgr2=0x7f24d00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00cb970 0x7f24d00ceb00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24c40099b0 tx=0x7f24c4009250 comp rx=0 tx=0).stop 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 -- 172.21.15.111:0/1120033683 shutdown_connections 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00cb970 0x7f24d00ceb00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/1120033683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 -- 172.21.15.111:0/1120033683 >> 172.21.15.111:0/1120033683 conn(0x7f24d0067bb0 msgr2=0x7f24d00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 -- 172.21.15.111:0/1120033683 shutdown_connections 2023-12-19T07:48:00.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.043+0000 7f24d5c6e700 1 -- 172.21.15.111:0/1120033683 wait complete. 2023-12-19T07:48:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 Processor -- start 2023-12-19T07:48:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 -- start start 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d014c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 0x7f24d00b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d016a630 con 0x7f24d00ca2c0 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d014cc90 con 0x7f24d00b52b0 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24d5c6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00b5c70 con 0x7f24d0158e20 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24cf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24cf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42960/0 (socket says 172.21.15.111:42960) 2023-12-19T07:48:00.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.047+0000 7f24cf7fe700 1 -- 172.21.15.111:0/3534089674 learned_addr learned my addr 172.21.15.111:0/3534089674 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24ceffd700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d014c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 -- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 msgr2=0x7f24d014c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cffff700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 0x7f24d00b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d014c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 -- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 msgr2=0x7f24d00b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 0x7f24d00b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 -- 172.21.15.111:0/3534089674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c40094f0 con 0x7f24d00ca2c0 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24ceffd700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d014c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cffff700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 0x7f24d00b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24cf7fe700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f24c000d860 tx=0x7f24c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00125f0 con 0x7f24d00ca2c0 2023-12-19T07:48:00.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0012750 con 0x7f24d00ca2c0 2023-12-19T07:48:00.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d00b3d60 con 0x7f24d00ca2c0 2023-12-19T07:48:00.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0012b10 con 0x7f24d00ca2c0 2023-12-19T07:48:00.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d00b3430 con 0x7f24d00ca2c0 2023-12-19T07:48:00.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.051+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d0071120 con 0x7f24d00ca2c0 2023-12-19T07:48:00.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.055+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f24c001e460 con 0x7f24d00ca2c0 2023-12-19T07:48:00.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.055+0000 7f24b77fe700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 0x7f24bc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.055+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f24c007d230 con 0x7f24d00ca2c0 2023-12-19T07:48:00.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.055+0000 7f24ceffd700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 0x7f24bc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.055+0000 7f24ceffd700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 0x7f24bc062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24c40099b0 tx=0x7f24c4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.063+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f24c0050f10 con 0x7f24d00ca2c0 2023-12-19T07:48:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.263+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24d006a7a0 con 0x7f24d00ca2c0 2023-12-19T07:48:00.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.263+0000 7f24b77fe700 1 -- 172.21.15.111:0/3534089674 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24d006a7a0 con 0x7f24d00ca2c0 2023-12-19T07:48:00.248 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 msgr2=0x7f24bc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 0x7f24bc062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24c40099b0 tx=0x7f24c4017000 comp rx=0 tx=0).stop 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 msgr2=0x7f24d016a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f24c000d860 tx=0x7f24c0008040 comp rx=0 tx=0).stop 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 shutdown_connections 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24bc060550 0x7f24bc062a00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24d00b52b0 0x7f24d00b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0158e20 0x7f24d014c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 --2- 172.21.15.111:0/3534089674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24d00ca2c0 0x7f24d016a0f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 >> 172.21.15.111:0/3534089674 conn(0x7f24d0067bb0 msgr2=0x7f24d00cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:00.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 shutdown_connections 2023-12-19T07:48:00.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.267+0000 7f24d5c6e700 1 -- 172.21.15.111:0/3534089674 wait complete. 2023-12-19T07:48:00.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:48:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.447+0000 7fe779363700 1 Processor -- start 2023-12-19T07:48:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.447+0000 7fe779363700 1 -- start start 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe779363700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe7740ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe779363700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe7740caa10 0x7fe77415d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe779363700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740cbbb0 con 0x7fe7740cc040 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe779363700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77415d960 con 0x7fe7740caa10 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe779363700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77406dc90 con 0x7fe7740ce2d0 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe778b62700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe7740cc040 0x7fe774158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47316/0 (socket says 172.21.15.111:47316) 2023-12-19T07:48:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe778b62700 1 -- 172.21.15.111:0/4009262484 learned_addr learned my addr 172.21.15.111:0/4009262484 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe7740ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe76bfff700 1 --2- 172.21.15.111:0/4009262484 >> v2:172.21.15.143:3300/0 conn(0x7fe7740caa10 0x7fe77415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816623850 0 0) 0x7fe7740cbbb0 con 0x7fe7740cc040 2023-12-19T07:48:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760003570 con 0x7fe7740cc040 2023-12-19T07:48:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1388838998 0 0) 0x7fe760003570 con 0x7fe7740cc040 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 msgr2=0x7fe7740ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe7740ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 >> v2:172.21.15.143:3300/0 conn(0x7fe7740caa10 msgr2=0x7fe77415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 --2- 172.21.15.111:0/4009262484 >> v2:172.21.15.143:3300/0 conn(0x7fe7740caa10 0x7fe77415d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7740cafa0 con 0x7fe7740cc040 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fe76c0017f0 con 0x7fe7740cc040 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 >> v1:172.21.15.111:6789/0 conn(0x7fe7740cc040 legacy=0x7fe774158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe760003a60 0x7fe760003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe760004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe760004d50 0x7fe7600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77415d960 con 0x7fe760004d50 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77406dc90 con 0x7fe760003a60 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76000ba60 con 0x7fe7740ce2d0 2023-12-19T07:48:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe760003a60 0x7fe760003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe778b62700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe760004d50 0x7fe7600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe76bfff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe760004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 -- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 msgr2=0x7fe760004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7740ce2d0 0x7fe760004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 -- 172.21.15.111:0/4009262484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe760004d50 msgr2=0x7fe7600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe760004d50 0x7fe7600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 -- 172.21.15.111:0/4009262484 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760003570 con 0x7fe760003a60 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe778b62700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe760004d50 0x7fe7600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe773fff700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe760003a60 0x7fe760003e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe764009c00 tx=0x7fe7640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6b113cd6c55b115 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76400e710 con 0x7fe760003a60 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76400e870 con 0x7fe760003a60 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.451+0000 7fe7737fe700 1 -- 172.21.15.111:0/4009262484 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76400ec30 con 0x7fe760003a60 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe760003a60 msgr2=0x7fe760003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 --2- 172.21.15.111:0/4009262484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe760003a60 0x7fe760003e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe764009c00 tx=0x7fe7640091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe778b62700 1 -- 172.21.15.111:0/4009262484 reap_dead start 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- 172.21.15.111:0/4009262484 shutdown_connections 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- 172.21.15.111:0/4009262484 >> 172.21.15.111:0/4009262484 conn(0x7fe774067bb0 msgr2=0x7fe7740cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- 172.21.15.111:0/4009262484 shutdown_connections 2023-12-19T07:48:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- 172.21.15.111:0/4009262484 wait complete. 2023-12-19T07:48:00.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 Processor -- start 2023-12-19T07:48:00.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.455+0000 7fe779363700 1 -- start start 2023-12-19T07:48:00.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 0x7fe774143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7740c9970 0x7fe7740b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c7d90 con 0x7fe7740c9500 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c7ab0 con 0x7fe7740c9970 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c8a20 con 0x7fe760003a60 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42992/0 (socket says 172.21.15.111:42992) 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 -- 172.21.15.111:0/1499033773 learned_addr learned my addr 172.21.15.111:0/1499033773 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe778b62700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7740c9970 0x7fe7740b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe773fff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 0x7fe774143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 -- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 msgr2=0x7fe774143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 0x7fe774143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 -- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7740c9970 msgr2=0x7fe7740b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7740c9970 0x7fe7740b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 -- 172.21.15.111:0/1499033773 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7740c8d00 con 0x7fe7740c9500 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe773fff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 0x7fe774143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe76bfff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe75c011d40 tx=0x7fe75c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c00fac0 con 0x7fe7740c9500 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c00fc20 con 0x7fe7740c9500 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c014440 con 0x7fe7740c9500 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7740c8fe0 con 0x7fe7740c9500 2023-12-19T07:48:00.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7741586d0 con 0x7fe7740c9500 2023-12-19T07:48:00.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.459+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7740c5e90 con 0x7fe7740c9500 2023-12-19T07:48:00.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.463+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe75c014a80 con 0x7fe7740c9500 2023-12-19T07:48:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.463+0000 7fe770ff9700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 0x7fe754062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.463+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fe75c018030 con 0x7fe7740c9500 2023-12-19T07:48:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.463+0000 7fe773fff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 0x7fe754062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.463+0000 7fe773fff700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 0x7fe754062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7640091e0 tx=0x7fe76401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.467+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe7740c5e90 con 0x7fe7740c9500 2023-12-19T07:48:00.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.643+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe7740c5e90 con 0x7fe7740c9500 2023-12-19T07:48:00.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.643+0000 7fe770ff9700 1 -- 172.21.15.111:0/1499033773 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe7740c5e90 con 0x7fe7740c9500 2023-12-19T07:48:00.628 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:00.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 msgr2=0x7fe754062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 0x7fe754062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7640091e0 tx=0x7fe76401a000 comp rx=0 tx=0).stop 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 msgr2=0x7fe7740b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe75c011d40 tx=0x7fe75c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 shutdown_connections 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe754060550 0x7fe754062a00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7740c9970 0x7fe7740b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7740c9500 0x7fe7740b48b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 --2- 172.21.15.111:0/1499033773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe760003a60 0x7fe774143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 >> 172.21.15.111:0/1499033773 conn(0x7fe774067bb0 msgr2=0x7fe774155340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:00.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 shutdown_connections 2023-12-19T07:48:00.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.647+0000 7fe779363700 1 -- 172.21.15.111:0/1499033773 wait complete. 2023-12-19T07:48:00.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.831+0000 7f492a22e700 1 Processor -- start 2023-12-19T07:48:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 -- start start 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 0x7f492406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49240ad750 0x7f49240c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492406e530 con 0x7f49240c8b20 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49241566d0 con 0x7f49240ad750 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f492a22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49240c9130 con 0x7f49240a7010 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49240ad750 0x7f49240c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49240ad750 0x7f49240c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42998/0 (socket says 172.21.15.111:42998) 2023-12-19T07:48:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 -- 172.21.15.111:0/1710462614 learned_addr learned my addr 172.21.15.111:0/1710462614 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f49237fe700 1 --2- 172.21.15.111:0/1710462614 >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 0x7f492406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 -- 172.21.15.111:0/1710462614 >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 msgr2=0x7f492406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 --2- 172.21.15.111:0/1710462614 >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 0x7f492406d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 -- 172.21.15.111:0/1710462614 >> v1:172.21.15.143:6789/0 conn(0x7f49240c8b20 legacy=0x7f49240ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 -- 172.21.15.111:0/1710462614 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240b02b0 con 0x7f49240ad750 2023-12-19T07:48:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f49237fe700 1 --2- 172.21.15.111:0/1710462614 >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 0x7f492406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.835+0000 7f4923fff700 1 --2- 172.21.15.111:0/1710462614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f49240c9450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f491c009f20 tx=0x7f491c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2d827a18ef8837 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f4922ffd700 1 -- 172.21.15.111:0/1710462614 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c00e070 con 0x7f49240ad750 2023-12-19T07:48:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f4922ffd700 1 -- 172.21.15.111:0/1710462614 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491c00ba40 con 0x7f49240ad750 2023-12-19T07:48:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f4922ffd700 1 -- 172.21.15.111:0/1710462614 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c0092e0 con 0x7f49240ad750 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 -- 172.21.15.111:0/1710462614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 msgr2=0x7f49240c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 --2- 172.21.15.111:0/1710462614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f49240c9450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f491c009f20 tx=0x7f491c009840 comp rx=0 tx=0).stop 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 -- 172.21.15.111:0/1710462614 shutdown_connections 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 --2- 172.21.15.111:0/1710462614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f49240c9450 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 --2- 172.21.15.111:0/1710462614 >> v2:172.21.15.143:3300/0 conn(0x7f49240a7010 0x7f492406d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 -- 172.21.15.111:0/1710462614 >> 172.21.15.111:0/1710462614 conn(0x7f49240af4f0 msgr2=0x7f49240af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 -- 172.21.15.111:0/1710462614 shutdown_connections 2023-12-19T07:48:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.839+0000 7f492a22e700 1 -- 172.21.15.111:0/1710462614 wait complete. 2023-12-19T07:48:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 Processor -- start 2023-12-19T07:48:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 -- start start 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 0x7f4924096c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49240c8b20 0x7f4924158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492415a560 con 0x7f49240ad750 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492415c0a0 con 0x7f49240c8b20 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f492a22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4924159cc0 con 0x7f49240a7010 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f491affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f491affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43010/0 (socket says 172.21.15.111:43010) 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f491affd700 1 -- 172.21.15.111:0/2024194534 learned_addr learned my addr 172.21.15.111:0/2024194534 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.843+0000 7f49237fe700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 0x7f4924096c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f4923fff700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49240c8b20 0x7f4924158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 -- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 msgr2=0x7f4924096c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 0x7f4924096c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 -- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49240c8b20 msgr2=0x7f4924158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49240c8b20 0x7f4924158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 -- 172.21.15.111:0/2024194534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c009ae0 con 0x7f49240ad750 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f49237fe700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 0x7f4924096c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491affd700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c4e0 tx=0x7f490c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c011590 con 0x7f49240ad750 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c011710 con 0x7f49240ad750 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4924159fa0 con 0x7f49240ad750 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c011ad0 con 0x7f49240ad750 2023-12-19T07:48:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492415c3c0 con 0x7f49240ad750 2023-12-19T07:48:00.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.847+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4924071120 con 0x7f49240ad750 2023-12-19T07:48:00.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.851+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f490c01b3b0 con 0x7f49240ad750 2023-12-19T07:48:00.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.851+0000 7f491bfff700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 0x7f4904062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:00.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.851+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f490c07c150 con 0x7f49240ad750 2023-12-19T07:48:00.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.851+0000 7f49237fe700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 0x7f4904062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:00.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.855+0000 7f49237fe700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 0x7f4904062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4914006f00 tx=0x7f4914008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:00.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:00.855+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f490c04c300 con 0x7f49240ad750 2023-12-19T07:48:01.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.031+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f492406f4d0 con 0x7f49240ad750 2023-12-19T07:48:01.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f491bfff700 1 -- 172.21.15.111:0/2024194534 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7f490c04f920 con 0x7f49240ad750 2023-12-19T07:48:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:01.018 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:01.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 msgr2=0x7f4904062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 0x7f4904062a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4914006f00 tx=0x7f4914008040 comp rx=0 tx=0).stop 2023-12-19T07:48:01.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 msgr2=0x7f4924097f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f490c00c4e0 tx=0x7f490c00c8a0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 shutdown_connections 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4904060550 0x7f4904062a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49240c8b20 0x7f4924158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49240ad750 0x7f4924097f90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 --2- 172.21.15.111:0/2024194534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49240a7010 0x7f4924096c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 >> 172.21.15.111:0/2024194534 conn(0x7f49240af4f0 msgr2=0x7f49240acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 shutdown_connections 2023-12-19T07:48:01.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.035+0000 7f492a22e700 1 -- 172.21.15.111:0/2024194534 wait complete. 2023-12-19T07:48:01.040 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:48:01.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:48:01.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.227+0000 7f72fefc5700 1 Processor -- start 2023-12-19T07:48:01.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.227+0000 7f72fefc5700 1 -- start start 2023-12-19T07:48:01.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.227+0000 7f72fefc5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f8158e20 0x7f72f80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.227+0000 7f72fefc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 0x7f72f80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80cbdf0 con 0x7f72f80cb970 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80cc0d0 con 0x7f72f8158e20 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80cc3b0 con 0x7f72f80ca2c0 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f8158e20 0x7f72f80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f8158e20 0x7f72f80ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43018/0 (socket says 172.21.15.111:43018) 2023-12-19T07:48:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 -- 172.21.15.111:0/794879405 learned_addr learned my addr 172.21.15.111:0/794879405 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fd562700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 0x7f72f80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 -- 172.21.15.111:0/794879405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 msgr2=0x7f72f80ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 0x7f72f80ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 -- 172.21.15.111:0/794879405 >> v1:172.21.15.111:6789/0 conn(0x7f72f80cb970 legacy=0x7f72f806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 -- 172.21.15.111:0/794879405 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80cc690 con 0x7f72f8158e20 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fd562700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 0x7f72f80ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fcd61700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80ce5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0099e0 tx=0x7f72ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff5a31daf7a4594 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72f77fe700 1 -- 172.21.15.111:0/794879405 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00b820 con 0x7f72f8158e20 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72f77fe700 1 -- 172.21.15.111:0/794879405 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec00b980 con 0x7f72f8158e20 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72f77fe700 1 -- 172.21.15.111:0/794879405 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00bd20 con 0x7f72f8158e20 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- 172.21.15.111:0/794879405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 msgr2=0x7f72f80ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80ce5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0099e0 tx=0x7f72ec009250 comp rx=0 tx=0).stop 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- 172.21.15.111:0/794879405 shutdown_connections 2023-12-19T07:48:01.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80ca2c0 0x7f72f80ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 --2- 172.21.15.111:0/794879405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80ce5c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- 172.21.15.111:0/794879405 >> 172.21.15.111:0/794879405 conn(0x7f72f8067bb0 msgr2=0x7f72f80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- 172.21.15.111:0/794879405 shutdown_connections 2023-12-19T07:48:01.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.231+0000 7f72fefc5700 1 -- 172.21.15.111:0/794879405 wait complete. 2023-12-19T07:48:01.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 Processor -- start 2023-12-19T07:48:01.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 -- start start 2023-12-19T07:48:01.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 0x7f72f8144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80b3f00 0x7f72f80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8142350 con 0x7f72f8158e20 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c7190 con 0x7f72f80ca2c0 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72fefc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c7750 con 0x7f72f80b3f00 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43034/0 (socket says 172.21.15.111:43034) 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.235+0000 7f72f7fff700 1 -- 172.21.15.111:0/3552808422 learned_addr learned my addr 172.21.15.111:0/3552808422 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fcd61700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 0x7f72f8144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fd562700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80b3f00 0x7f72f80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 -- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80b3f00 msgr2=0x7f72f80b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80b3f00 0x7f72f80b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 -- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 msgr2=0x7f72f8144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 0x7f72f8144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 -- 172.21.15.111:0/3552808422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0094f0 con 0x7f72f8158e20 2023-12-19T07:48:01.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fcd61700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 0x7f72f8144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:01.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f7fff700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c7c0 tx=0x7f72e4007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4011590 con 0x7f72f8158e20 2023-12-19T07:48:01.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e40116f0 con 0x7f72f8158e20 2023-12-19T07:48:01.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4011ab0 con 0x7f72f8158e20 2023-12-19T07:48:01.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f80b4440 con 0x7f72f8158e20 2023-12-19T07:48:01.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f80c73e0 con 0x7f72f8158e20 2023-12-19T07:48:01.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.239+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f80c78c0 con 0x7f72f8158e20 2023-12-19T07:48:01.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.243+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f72e401c440 con 0x7f72f8158e20 2023-12-19T07:48:01.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.243+0000 7f72f4ff9700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 0x7f72dc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.243+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f72e407c020 con 0x7f72f8158e20 2023-12-19T07:48:01.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.243+0000 7f72fcd61700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 0x7f72dc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.243+0000 7f72fcd61700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 0x7f72dc062a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0099e0 tx=0x7f72ec017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.247+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72e404c270 con 0x7f72f8158e20 2023-12-19T07:48:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.415+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72f8071120 con 0x7f72f8158e20 2023-12-19T07:48:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.415+0000 7f72f4ff9700 1 -- 172.21.15.111:0/3552808422 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72e40170a0 con 0x7f72f8158e20 2023-12-19T07:48:01.401 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 msgr2=0x7f72dc062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 0x7f72dc062a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0099e0 tx=0x7f72ec017000 comp rx=0 tx=0).stop 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 msgr2=0x7f72f80b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c7c0 tx=0x7f72e4007f50 comp rx=0 tx=0).stop 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 shutdown_connections 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72dc060580 0x7f72dc062a30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72f80b3f00 0x7f72f80b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f8158e20 0x7f72f80b39c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 --2- 172.21.15.111:0/3552808422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72f80ca2c0 0x7f72f8144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 >> 172.21.15.111:0/3552808422 conn(0x7f72f8067bb0 msgr2=0x7f72f80cdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 shutdown_connections 2023-12-19T07:48:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.419+0000 7f72fefc5700 1 -- 172.21.15.111:0/3552808422 wait complete. 2023-12-19T07:48:01.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:48:01.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.603+0000 7fd544ee3700 1 Processor -- start 2023-12-19T07:48:01.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.603+0000 7fd544ee3700 1 -- start start 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd544ee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd540158c90 0x7fd5400caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd544ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd544ee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54006c850 con 0x7fd5401595a0 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd544ee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54006b470 con 0x7fd540158c90 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd544ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54006b750 con 0x7fd54006aeb0 2023-12-19T07:48:01.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd53effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd53ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd5401595a0 0x7fd54015c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49356/0 (socket says 172.21.15.111:49356) 2023-12-19T07:48:01.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd53ffff700 1 -- 172.21.15.111:0/3841832053 learned_addr learned my addr 172.21.15.111:0/3841832053 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:01.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.607+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/3841832053 >> v2:172.21.15.143:3300/0 conn(0x7fd540158c90 0x7fd5400caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53e7fc700 1 -- 172.21.15.111:0/3841832053 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058809487 0 0) 0x7fd54006c850 con 0x7fd5401595a0 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53e7fc700 1 -- 172.21.15.111:0/3841832053 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd528003570 con 0x7fd5401595a0 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53f7fe700 1 -- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 msgr2=0x7fd540159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53f7fe700 1 -- 172.21.15.111:0/3841832053 >> v1:172.21.15.143:6789/0 conn(0x7fd5401595a0 legacy=0x7fd54015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53f7fe700 1 -- 172.21.15.111:0/3841832053 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54006ba30 con 0x7fd540158c90 2023-12-19T07:48:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400caf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098e0 tx=0x7fd5300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b2fcfeeb33b80c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53e7fc700 1 -- 172.21.15.111:0/3841832053 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530011630 con 0x7fd540158c90 2023-12-19T07:48:01.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53e7fc700 1 -- 172.21.15.111:0/3841832053 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530011790 con 0x7fd540158c90 2023-12-19T07:48:01.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd53e7fc700 1 -- 172.21.15.111:0/3841832053 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530011b30 con 0x7fd540158c90 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 -- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 msgr2=0x7fd5400caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 --2- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400caf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098e0 tx=0x7fd5300091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 -- 172.21.15.111:0/3841832053 shutdown_connections 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 --2- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 --2- 172.21.15.111:0/3841832053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400caf40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 -- 172.21.15.111:0/3841832053 >> 172.21.15.111:0/3841832053 conn(0x7fd540067bb0 msgr2=0x7fd5400cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 -- 172.21.15.111:0/3841832053 shutdown_connections 2023-12-19T07:48:01.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.611+0000 7fd544ee3700 1 -- 172.21.15.111:0/3841832053 wait complete. 2023-12-19T07:48:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 Processor -- start 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 -- start start 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5401562c0 con 0x7fd5401595a0 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540156b60 con 0x7fd540158c90 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd544ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400c57a0 con 0x7fd54006aeb0 2023-12-19T07:48:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43038/0 (socket says 172.21.15.111:43038) 2023-12-19T07:48:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 -- 172.21.15.111:0/437763857 learned_addr learned my addr 172.21.15.111:0/437763857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53effd700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 -- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 msgr2=0x7fd540151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 -- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 msgr2=0x7fd5400c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53ffff700 1 -- 172.21.15.111:0/437763857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530009420 con 0x7fd5401595a0 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.615+0000 7fd53effd700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd53ffff700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd53800cc40 tx=0x7fd538001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53800a830 con 0x7fd5401595a0 2023-12-19T07:48:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53800a990 con 0x7fd5401595a0 2023-12-19T07:48:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd540156510 con 0x7fd5401595a0 2023-12-19T07:48:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53800ad10 con 0x7fd5401595a0 2023-12-19T07:48:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd540155660 con 0x7fd5401595a0 2023-12-19T07:48:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54004be40 con 0x7fd5401595a0 2023-12-19T07:48:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.619+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd538007300 con 0x7fd5401595a0 2023-12-19T07:48:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.623+0000 7fd5277fe700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 0x7fd52c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.623+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fd53807cf60 con 0x7fd5401595a0 2023-12-19T07:48:01.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.623+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 0x7fd52c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.623+0000 7fd53f7fe700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 0x7fd52c062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd5300091e0 tx=0x7fd530017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.631+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd53804d100 con 0x7fd5401595a0 2023-12-19T07:48:01.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.799+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd540156ca0 con 0x7fd5401595a0 2023-12-19T07:48:01.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.799+0000 7fd5277fe700 1 -- 172.21.15.111:0/437763857 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd538050720 con 0x7fd5401595a0 2023-12-19T07:48:01.783 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 msgr2=0x7fd52c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 0x7fd52c062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd5300091e0 tx=0x7fd530017000 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 msgr2=0x7fd540156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd53800cc40 tx=0x7fd538001a40 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 shutdown_connections 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd52c060550 0x7fd52c062a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd540158c90 0x7fd5400c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5401595a0 0x7fd540156ff0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 --2- 172.21.15.111:0/437763857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd54006aeb0 0x7fd540151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 >> 172.21.15.111:0/437763857 conn(0x7fd540067bb0 msgr2=0x7fd5400afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 shutdown_connections 2023-12-19T07:48:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.803+0000 7fd544ee3700 1 -- 172.21.15.111:0/437763857 wait complete. 2023-12-19T07:48:01.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:48:01.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 Processor -- start 2023-12-19T07:48:01.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 -- start start 2023-12-19T07:48:01.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d34159a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3406db20 con 0x7f3d3406b770 2023-12-19T07:48:01.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3406dfb0 con 0x7f3d3406e530 2023-12-19T07:48:01.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d392ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340cc310 con 0x7f3d341554d0 2023-12-19T07:48:01.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d337fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3d3406b770 0x7f3d340af340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52542/0 (socket says 172.21.15.111:52542) 2023-12-19T07:48:01.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d337fe700 1 -- 172.21.15.111:0/1994330275 learned_addr learned my addr 172.21.15.111:0/1994330275 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:01.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:01.999+0000 7f3d32ffd700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d34159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157444715 0 0) 0x7f3d3406db20 con 0x7f3d3406b770 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c003570 con 0x7f3d3406b770 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521095570 0 0) 0x7f3d3406dfb0 con 0x7f3d3406e530 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3406db20 con 0x7f3d3406e530 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219951827 0 0) 0x7f3d1c003570 con 0x7f3d3406b770 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 msgr2=0x7f3d34159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d34159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 >> v1:172.21.15.143:6789/0 conn(0x7f3d3406e530 legacy=0x7f3d3406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3406e980 con 0x7f3d3406b770 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3d2c0017f0 con 0x7f3d3406b770 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 >> v1:172.21.15.111:6789/0 conn(0x7f3d3406b770 legacy=0x7f3d340af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c003c20 0x7f3d1c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340cc310 con 0x7f3d1c004810 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c00bb40 con 0x7f3d1c003c20 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c00bd00 con 0x7f3d341554d0 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d337fe700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d32ffd700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c003c20 0x7f3d1c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 -- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 msgr2=0x7f3d1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d341554d0 0x7f3d1c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 -- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c003c20 msgr2=0x7f3d1c004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c003c20 0x7f3d1c004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 -- 172.21.15.111:0/1994330275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c003570 con 0x7f3d1c004810 2023-12-19T07:48:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d1c004c80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d28001a10 tx=0x7f3d28007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c077b9796687dde3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800c5d0 con 0x7f3d1c004810 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2800c730 con 0x7f3d1c004810 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d31ffb700 1 -- 172.21.15.111:0/1994330275 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800cad0 con 0x7f3d1c004810 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d392ae700 1 -- 172.21.15.111:0/1994330275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 msgr2=0x7f3d1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1994330275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d1c004c80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d28001a10 tx=0x7f3d28007d30 comp rx=0 tx=0).stop 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.003+0000 7f3d337fe700 1 -- 172.21.15.111:0/1994330275 reap_dead start 2023-12-19T07:48:01.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 -- 172.21.15.111:0/1994330275 shutdown_connections 2023-12-19T07:48:01.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 -- 172.21.15.111:0/1994330275 >> 172.21.15.111:0/1994330275 conn(0x7f3d340cc580 msgr2=0x7f3d340cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:01.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 -- 172.21.15.111:0/1994330275 shutdown_connections 2023-12-19T07:48:01.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 -- 172.21.15.111:0/1994330275 wait complete. 2023-12-19T07:48:02.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 Processor -- start 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.007+0000 7f3d392ae700 1 -- start start 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340c3850 0x7f3d340c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d34167350 0x7f3d34158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d341572b0 con 0x7f3d1c004810 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340c2610 con 0x7f3d34167350 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34156980 con 0x7f3d340c3850 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37744/0 (socket says 172.21.15.111:37744) 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 -- 172.21.15.111:0/1440111968 learned_addr learned my addr 172.21.15.111:0/1440111968 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340c3850 0x7f3d340c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d337fe700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d34167350 0x7f3d34158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 -- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340c3850 msgr2=0x7f3d340c1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340c3850 0x7f3d340c1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 -- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d34167350 msgr2=0x7f3d34158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d34167350 0x7f3d34158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 -- 172.21.15.111:0/1440111968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34156c60 con 0x7f3d1c004810 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d32ffd700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3d240119c0 tx=0x7f3d24008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400f6d0 con 0x7f3d1c004810 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2400f830 con 0x7f3d1c004810 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d34156f40 con 0x7f3d1c004810 2023-12-19T07:48:02.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400fbf0 con 0x7f3d1c004810 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.011+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d340c23e0 con 0x7f3d1c004810 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.015+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d340aede0 con 0x7f3d1c004810 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.015+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3d24015460 con 0x7f3d1c004810 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.015+0000 7f3d1affd700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 0x7f3d20062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.015+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f3d2407d280 con 0x7f3d1c004810 2023-12-19T07:48:02.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.015+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 0x7f3d20062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.019+0000 7f3d327fc700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 0x7f3d20062a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3d3416ba60 tx=0x7f3d28015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.023+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3d340aede0 con 0x7f3d1c004810 2023-12-19T07:48:02.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.191+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d340aede0 con 0x7f3d1c004810 2023-12-19T07:48:02.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.191+0000 7f3d1affd700 1 -- 172.21.15.111:0/1440111968 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d340aede0 con 0x7f3d1c004810 2023-12-19T07:48:02.176 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 msgr2=0x7f3d20062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 0x7f3d20062a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3d3416ba60 tx=0x7f3d28015000 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 msgr2=0x7f3d34145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3d240119c0 tx=0x7f3d24008040 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 shutdown_connections 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3d20060580 0x7f3d20062a30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d34167350 0x7f3d34158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340c3850 0x7f3d340c1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 --2- 172.21.15.111:0/1440111968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d1c004810 0x7f3d34145ef0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 >> 172.21.15.111:0/1440111968 conn(0x7f3d340cc580 msgr2=0x7f3d34153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 shutdown_connections 2023-12-19T07:48:02.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.195+0000 7f3d392ae700 1 -- 172.21.15.111:0/1440111968 wait complete. 2023-12-19T07:48:02.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:48:02.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.387+0000 7f8d4baf0700 1 Processor -- start 2023-12-19T07:48:02.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.387+0000 7f8d4baf0700 1 -- start start 2023-12-19T07:48:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4baf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 0x7f8d440ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4baf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d440c7450 con 0x7f8d4415d410 2023-12-19T07:48:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4baf0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4406bea0 con 0x7f8d4406b1b0 2023-12-19T07:48:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4baf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d440c6e00 con 0x7f8d440cab10 2023-12-19T07:48:02.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4988c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8d4415d410 0x7f8d440ca400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52556/0 (socket says 172.21.15.111:52556) 2023-12-19T07:48:02.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4988c700 1 -- 172.21.15.111:0/3438610551 learned_addr learned my addr 172.21.15.111:0/3438610551 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:02.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4908b700 1 --2- 172.21.15.111:0/3438610551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 0x7f8d440ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918973655 0 0) 0x7f8d440c7450 con 0x7f8d4415d410 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d30003570 con 0x7f8d4415d410 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675761857 0 0) 0x7f8d4406bea0 con 0x7f8d4406b1b0 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d440c7450 con 0x7f8d4406b1b0 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4908b700 1 -- 172.21.15.111:0/3438610551 >> v1:172.21.15.143:6789/0 conn(0x7f8d4406b1b0 legacy=0x7f8d4406b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4908b700 1 -- 172.21.15.111:0/3438610551 >> v1:172.21.15.111:6789/0 conn(0x7f8d4415d410 legacy=0x7f8d440ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4908b700 1 -- 172.21.15.111:0/3438610551 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d440c70e0 con 0x7f8d440cab10 2023-12-19T07:48:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.391+0000 7f8d4908b700 1 --2- 172.21.15.111:0/3438610551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 0x7f8d440ab6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009910 tx=0x7f8d340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a584a974bf462 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34011630 con 0x7f8d440cab10 2023-12-19T07:48:02.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34011790 con 0x7f8d440cab10 2023-12-19T07:48:02.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4888a700 1 -- 172.21.15.111:0/3438610551 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34011b30 con 0x7f8d440cab10 2023-12-19T07:48:02.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 -- 172.21.15.111:0/3438610551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 msgr2=0x7f8d440ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/3438610551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 0x7f8d440ab6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009910 tx=0x7f8d340091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 -- 172.21.15.111:0/3438610551 shutdown_connections 2023-12-19T07:48:02.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/3438610551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d440cab10 0x7f8d440ab6c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 -- 172.21.15.111:0/3438610551 >> 172.21.15.111:0/3438610551 conn(0x7f8d44067bb0 msgr2=0x7f8d440cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:02.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 -- 172.21.15.111:0/3438610551 shutdown_connections 2023-12-19T07:48:02.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 -- 172.21.15.111:0/3438610551 wait complete. 2023-12-19T07:48:02.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.395+0000 7f8d4baf0700 1 Processor -- start 2023-12-19T07:48:02.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 -- start start 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d440cab10 0x7f8d4416dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 0x7f8d441619a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44161130 con 0x7f8d4406b1b0 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d440c4930 con 0x7f8d440cab10 2023-12-19T07:48:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4416e180 con 0x7f8d4415d410 2023-12-19T07:48:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37754/0 (socket says 172.21.15.111:37754) 2023-12-19T07:48:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 -- 172.21.15.111:0/53214157 learned_addr learned my addr 172.21.15.111:0/53214157 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4908b700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d440cab10 0x7f8d4416dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4a08d700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 0x7f8d441619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 -- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 msgr2=0x7f8d441619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 0x7f8d441619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 -- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d440cab10 msgr2=0x7f8d4416dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d440cab10 0x7f8d4416dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 -- 172.21.15.111:0/53214157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34009420 con 0x7f8d4406b1b0 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4a08d700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 0x7f8d441619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4988c700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000c7c0 tx=0x7f8d40007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40005c00 con 0x7f8d4406b1b0 2023-12-19T07:48:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d44161340 con 0x7f8d4406b1b0 2023-12-19T07:48:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d40005d60 con 0x7f8d4406b1b0 2023-12-19T07:48:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.399+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40007300 con 0x7f8d4406b1b0 2023-12-19T07:48:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.403+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d440c45c0 con 0x7f8d4406b1b0 2023-12-19T07:48:02.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.403+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d440c4420 con 0x7f8d4406b1b0 2023-12-19T07:48:02.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.403+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8d4001b3b0 con 0x7f8d4406b1b0 2023-12-19T07:48:02.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.403+0000 7f8d39ffb700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 0x7f8d28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.403+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f8d4007c140 con 0x7f8d4406b1b0 2023-12-19T07:48:02.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.407+0000 7f8d4908b700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 0x7f8d28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.407+0000 7f8d4908b700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 0x7f8d28062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009860 tx=0x7f8d34017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.411+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d440c4420 con 0x7f8d4406b1b0 2023-12-19T07:48:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.579+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8d440c4420 con 0x7f8d4406b1b0 2023-12-19T07:48:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.579+0000 7f8d39ffb700 1 -- 172.21.15.111:0/53214157 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8d40017020 con 0x7f8d4406b1b0 2023-12-19T07:48:02.564 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 msgr2=0x7f8d28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 0x7f8d28062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009860 tx=0x7f8d34017000 comp rx=0 tx=0).stop 2023-12-19T07:48:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 msgr2=0x7f8d44163260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000c7c0 tx=0x7f8d40007ea0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 shutdown_connections 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d28060550 0x7f8d28062a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d440cab10 0x7f8d4416dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d4415d410 0x7f8d441619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 --2- 172.21.15.111:0/53214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d4406b1b0 0x7f8d44163260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 >> 172.21.15.111:0/53214157 conn(0x7f8d44067bb0 msgr2=0x7f8d4406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 shutdown_connections 2023-12-19T07:48:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.583+0000 7f8d4baf0700 1 -- 172.21.15.111:0/53214157 wait complete. 2023-12-19T07:48:02.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.767+0000 7fafe77c5700 1 Processor -- start 2023-12-19T07:48:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.767+0000 7fafe77c5700 1 -- start start 2023-12-19T07:48:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe77c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe77c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00c7190 con 0x7fafe00c15b0 2023-12-19T07:48:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe77c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00c1b50 con 0x7fafe015d320 2023-12-19T07:48:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe77c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00c1e30 con 0x7fafe0068800 2023-12-19T07:48:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe4d60700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fafe00c15b0 0x7fafe00680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52566/0 (socket says 172.21.15.111:52566) 2023-12-19T07:48:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe4d60700 1 -- 172.21.15.111:0/337026877 learned_addr learned my addr 172.21.15.111:0/337026877 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe5d62700 1 --2- 172.21.15.111:0/337026877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045068029 0 0) 0x7fafe00c7190 con 0x7fafe00c15b0 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc003570 con 0x7fafe00c15b0 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593695868 0 0) 0x7fafe00c1b50 con 0x7fafe015d320 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe00c7190 con 0x7fafe015d320 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe5d62700 1 -- 172.21.15.111:0/337026877 >> v1:172.21.15.143:6789/0 conn(0x7fafe015d320 legacy=0x7fafe006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe5d62700 1 -- 172.21.15.111:0/337026877 >> v1:172.21.15.111:6789/0 conn(0x7fafe00c15b0 legacy=0x7fafe00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.771+0000 7fafe5d62700 1 -- 172.21.15.111:0/337026877 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00c6e20 con 0x7fafe0068800 2023-12-19T07:48:02.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe5d62700 1 --2- 172.21.15.111:0/337026877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe015d6f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009e90 tx=0x7fafd8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d4459156f582063 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd800e070 con 0x7fafe0068800 2023-12-19T07:48:02.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd800bb80 con 0x7fafe0068800 2023-12-19T07:48:02.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafd7fff700 1 -- 172.21.15.111:0/337026877 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8012430 con 0x7fafe0068800 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 -- 172.21.15.111:0/337026877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 msgr2=0x7fafe015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 --2- 172.21.15.111:0/337026877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe015d6f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009e90 tx=0x7fafd8009780 comp rx=0 tx=0).stop 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 -- 172.21.15.111:0/337026877 shutdown_connections 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 --2- 172.21.15.111:0/337026877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe015d6f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 -- 172.21.15.111:0/337026877 >> 172.21.15.111:0/337026877 conn(0x7fafe00c8850 msgr2=0x7fafe00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:02.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 -- 172.21.15.111:0/337026877 shutdown_connections 2023-12-19T07:48:02.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.775+0000 7fafe77c5700 1 -- 172.21.15.111:0/337026877 wait complete. 2023-12-19T07:48:02.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 Processor -- start 2023-12-19T07:48:02.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 -- start start 2023-12-19T07:48:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe01540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafe015d320 0x7fafe0163160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0171620 con 0x7fafe00c15b0 2023-12-19T07:48:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0170a10 con 0x7fafe015d320 2023-12-19T07:48:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe77c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0170cf0 con 0x7fafe0068800 2023-12-19T07:48:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe4d60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe4d60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37762/0 (socket says 172.21.15.111:37762) 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe4d60700 1 -- 172.21.15.111:0/3018280372 learned_addr learned my addr 172.21.15.111:0/3018280372 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe5561700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe01540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.779+0000 7fafe5d62700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafe015d320 0x7fafe0163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 -- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 msgr2=0x7fafe01540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe01540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 -- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafe015d320 msgr2=0x7fafe0163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafe015d320 0x7fafe0163160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 -- 172.21.15.111:0/3018280372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd8009a20 con 0x7fafe00c15b0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe4d60700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fafd000c790 tx=0x7fafd0007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0005b30 con 0x7fafe00c15b0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0005c90 con 0x7fafe00c15b0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe0170fd0 con 0x7fafe00c15b0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0007300 con 0x7fafe00c15b0 2023-12-19T07:48:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe0154630 con 0x7fafe00c15b0 2023-12-19T07:48:02.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.783+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe0171790 con 0x7fafe00c15b0 2023-12-19T07:48:02.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.787+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fafd001b3b0 con 0x7fafe00c15b0 2023-12-19T07:48:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.787+0000 7fafd57fa700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 0x7fafc4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.787+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fafd007c0e0 con 0x7fafe00c15b0 2023-12-19T07:48:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.787+0000 7fafe5561700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 0x7fafc4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.787+0000 7fafe5561700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 0x7fafc4062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fafdc007fd0 tx=0x7fafdc000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:02.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.795+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fafd004c290 con 0x7fafe00c15b0 2023-12-19T07:48:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.963+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafe0071120 con 0x7fafe00c15b0 2023-12-19T07:48:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.963+0000 7fafd57fa700 1 -- 172.21.15.111:0/3018280372 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7fafd004f8b0 con 0x7fafe00c15b0 2023-12-19T07:48:02.947 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:02.947 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 msgr2=0x7fafc4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 0x7fafc4062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fafdc007fd0 tx=0x7fafdc000bc0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 msgr2=0x7fafe0162c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fafd000c790 tx=0x7fafd0007ea0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 shutdown_connections 2023-12-19T07:48:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fafc4060550 0x7fafc4062a00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafe015d320 0x7fafe0163160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafe00c15b0 0x7fafe0162c20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 --2- 172.21.15.111:0/3018280372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafe0068800 0x7fafe01540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 >> 172.21.15.111:0/3018280372 conn(0x7fafe00c8850 msgr2=0x7fafe00b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 shutdown_connections 2023-12-19T07:48:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:02.967+0000 7fafe77c5700 1 -- 172.21.15.111:0/3018280372 wait complete. 2023-12-19T07:48:02.969 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:48:02.970 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:48:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.155+0000 7f845ecae700 1 Processor -- start 2023-12-19T07:48:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.155+0000 7f845ecae700 1 -- start start 2023-12-19T07:48:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ecae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 0x7f845806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ecae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580cbdf0 con 0x7f8458158e20 2023-12-19T07:48:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ecae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580cc0d0 con 0x7f84580cecd0 2023-12-19T07:48:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ecae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84580cc3b0 con 0x7f84580cb970 2023-12-19T07:48:03.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ca4a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8458158e20 0x7f84580ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52576/0 (socket says 172.21.15.111:52576) 2023-12-19T07:48:03.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f845ca4a700 1 -- 172.21.15.111:0/960300688 learned_addr learned my addr 172.21.15.111:0/960300688 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:03.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f8457fff700 1 --2- 172.21.15.111:0/960300688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 0x7f845806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906088892 0 0) 0x7f84580cbdf0 con 0x7f8458158e20 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8448003570 con 0x7f8458158e20 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529937152 0 0) 0x7f84580cc0d0 con 0x7f84580cecd0 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84580cbdf0 con 0x7f84580cecd0 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f8457fff700 1 -- 172.21.15.111:0/960300688 >> v1:172.21.15.143:6789/0 conn(0x7f84580cecd0 legacy=0x7f845806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f8457fff700 1 -- 172.21.15.111:0/960300688 >> v1:172.21.15.111:6789/0 conn(0x7f8458158e20 legacy=0x7f84580ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.159+0000 7f8457fff700 1 -- 172.21.15.111:0/960300688 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84580cc690 con 0x7f84580cb970 2023-12-19T07:48:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f8457fff700 1 --2- 172.21.15.111:0/960300688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 0x7f845806acc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8444009910 tx=0x7f84440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1737b240365478c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444011630 con 0x7f84580cb970 2023-12-19T07:48:03.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444011790 con 0x7f84580cb970 2023-12-19T07:48:03.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f84577fe700 1 -- 172.21.15.111:0/960300688 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444011b30 con 0x7f84580cb970 2023-12-19T07:48:03.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 -- 172.21.15.111:0/960300688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 msgr2=0x7f845806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 --2- 172.21.15.111:0/960300688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 0x7f845806acc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8444009910 tx=0x7f84440091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 -- 172.21.15.111:0/960300688 shutdown_connections 2023-12-19T07:48:03.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 --2- 172.21.15.111:0/960300688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84580cb970 0x7f845806acc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 -- 172.21.15.111:0/960300688 >> 172.21.15.111:0/960300688 conn(0x7f8458067bb0 msgr2=0x7f84580d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:03.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 -- 172.21.15.111:0/960300688 shutdown_connections 2023-12-19T07:48:03.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.163+0000 7f845ecae700 1 -- 172.21.15.111:0/960300688 wait complete. 2023-12-19T07:48:03.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 Processor -- start 2023-12-19T07:48:03.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 -- start start 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 0x7f845814ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8458151900 0x7f8458157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c9950 con 0x7f84580cecd0 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c8d40 con 0x7f8458158e20 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ecae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c9020 con 0x7f8458151900 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ca4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ca4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37766/0 (socket says 172.21.15.111:37766) 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845ca4a700 1 -- 172.21.15.111:0/438728454 learned_addr learned my addr 172.21.15.111:0/438728454 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f8457fff700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 0x7f845814ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.167+0000 7f845d24b700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8458151900 0x7f8458157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 -- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8458151900 msgr2=0x7f8458157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8458151900 0x7f8458157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 -- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 msgr2=0x7f845814ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 0x7f845814ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 -- 172.21.15.111:0/438728454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444009420 con 0x7f84580cecd0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f8457fff700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 0x7f845814ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ca4a700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f844c00c790 tx=0x7f844c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c005b30 con 0x7f84580cecd0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c005c90 con 0x7f84580cecd0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84580c9300 con 0x7f84580cecd0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c007300 con 0x7f84580cecd0 2023-12-19T07:48:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84581536c0 con 0x7f84580cecd0 2023-12-19T07:48:03.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.171+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84580c9780 con 0x7f84580cecd0 2023-12-19T07:48:03.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.175+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f844c01b3b0 con 0x7f84580cecd0 2023-12-19T07:48:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.175+0000 7f8454ff9700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 0x7f843c062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.175+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f844c07c1f0 con 0x7f84580cecd0 2023-12-19T07:48:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.175+0000 7f8457fff700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 0x7f843c062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.175+0000 7f8457fff700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 0x7f843c062a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84440093f0 tx=0x7f8444017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.179+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f844c04c3a0 con 0x7f84580cecd0 2023-12-19T07:48:03.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.347+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8458071120 con 0x7f84580cecd0 2023-12-19T07:48:03.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.351+0000 7f8454ff9700 1 -- 172.21.15.111:0/438728454 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f844c017020 con 0x7f84580cecd0 2023-12-19T07:48:03.334 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 msgr2=0x7f843c062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 0x7f843c062a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84440093f0 tx=0x7f8444017000 comp rx=0 tx=0).stop 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 msgr2=0x7f84580c8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f844c00c790 tx=0x7f844c007ea0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 shutdown_connections 2023-12-19T07:48:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f843c0605d0 0x7f843c062a80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8458158e20 0x7f845814ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8458151900 0x7f8458157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 --2- 172.21.15.111:0/438728454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84580cecd0 0x7f84580c8660 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 >> 172.21.15.111:0/438728454 conn(0x7f8458067bb0 msgr2=0x7f845806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 shutdown_connections 2023-12-19T07:48:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.355+0000 7f845ecae700 1 -- 172.21.15.111:0/438728454 wait complete. 2023-12-19T07:48:03.361 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:48:03.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:48:03.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:48:03.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 Processor -- start 2023-12-19T07:48:03.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 -- start start 2023-12-19T07:48:03.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 0x7f69040abb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040ac390 con 0x7f690406ae20 2023-12-19T07:48:03.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040ac670 con 0x7f69040ab6d0 2023-12-19T07:48:03.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6908b23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040ac950 con 0x7f69040b1440 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f69037fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f690406ae20 0x7f690406b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52582/0 (socket says 172.21.15.111:52582) 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f69037fe700 1 -- 172.21.15.111:0/80829743 learned_addr learned my addr 172.21.15.111:0/80829743 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6902ffd700 1 --2- 172.21.15.111:0/80829743 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 0x7f69040abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 --2- 172.21.15.111:0/80829743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069768940 0 0) 0x7f69040ac390 con 0x7f690406ae20 2023-12-19T07:48:03.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec003570 con 0x7f690406ae20 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705364873 0 0) 0x7f68ec003570 con 0x7f690406ae20 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 -- 172.21.15.111:0/80829743 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 msgr2=0x7f69040abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 --2- 172.21.15.111:0/80829743 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 0x7f69040abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 -- 172.21.15.111:0/80829743 >> v1:172.21.15.111:6789/0 conn(0x7f690406ae20 legacy=0x7f690406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 -- 172.21.15.111:0/80829743 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040acc30 con 0x7f69040b1440 2023-12-19T07:48:03.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6902ffd700 1 --2- 172.21.15.111:0/80829743 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 0x7f69040abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.531+0000 7f6903fff700 1 --2- 172.21.15.111:0/80829743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040b18a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009e60 tx=0x7f68fc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90d097c299f6c5cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00e070 con 0x7f69040b1440 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00b7a0 con 0x7f69040b1440 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f69027fc700 1 -- 172.21.15.111:0/80829743 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4001470 con 0x7f69040b1440 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 -- 172.21.15.111:0/80829743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 msgr2=0x7f69040b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 --2- 172.21.15.111:0/80829743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040b18a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009e60 tx=0x7f68fc009780 comp rx=0 tx=0).stop 2023-12-19T07:48:03.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 -- 172.21.15.111:0/80829743 shutdown_connections 2023-12-19T07:48:03.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 --2- 172.21.15.111:0/80829743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040b18a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 --2- 172.21.15.111:0/80829743 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab6d0 0x7f69040abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 -- 172.21.15.111:0/80829743 >> 172.21.15.111:0/80829743 conn(0x7f69040c8cd0 msgr2=0x7f69040c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:03.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 -- 172.21.15.111:0/80829743 shutdown_connections 2023-12-19T07:48:03.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.535+0000 7f6908b23700 1 -- 172.21.15.111:0/80829743 wait complete. 2023-12-19T07:48:03.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 Processor -- start 2023-12-19T07:48:03.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 -- start start 2023-12-19T07:48:03.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 0x7f690415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690416a0b0 con 0x7f69040ab6d0 2023-12-19T07:48:03.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904169790 con 0x7f690406ae20 2023-12-19T07:48:03.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6908b23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040caa70 con 0x7f69040b1440 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37774/0 (socket says 172.21.15.111:37774) 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 -- 172.21.15.111:0/3202426601 learned_addr learned my addr 172.21.15.111:0/3202426601 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6903fff700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f69037fe700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 0x7f690415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 -- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 msgr2=0x7f69040c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 -- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 msgr2=0x7f690415cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 0x7f690415cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 -- 172.21.15.111:0/3202426601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f8009420 con 0x7f69040ab6d0 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f69037fe700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 0x7f690415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.539+0000 7f6902ffd700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f68f800ec80 tx=0x7f68f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f800c8f0 con 0x7f69040ab6d0 2023-12-19T07:48:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f800ca50 con 0x7f69040ab6d0 2023-12-19T07:48:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f800cdd0 con 0x7f69040ab6d0 2023-12-19T07:48:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc009a20 con 0x7f69040ab6d0 2023-12-19T07:48:03.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6904169d40 con 0x7f69040ab6d0 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.543+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040c6690 con 0x7f69040ab6d0 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.547+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f68f8013680 con 0x7f69040ab6d0 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.547+0000 7f68eb7fe700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 0x7f68f0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.547+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f68f807d240 con 0x7f69040ab6d0 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.547+0000 7f69037fe700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 0x7f68f0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.547+0000 7f69037fe700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 0x7f68f0062a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68f40011e0 tx=0x7f68f40015e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.551+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f69040c6690 con 0x7f69040ab6d0 2023-12-19T07:48:03.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.715+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69040ad040 con 0x7f69040ab6d0 2023-12-19T07:48:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.715+0000 7f68eb7fe700 1 -- 172.21.15.111:0/3202426601 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69040ad040 con 0x7f69040ab6d0 2023-12-19T07:48:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 msgr2=0x7f68f0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 0x7f68f0062a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68f40011e0 tx=0x7f68f40015e0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 msgr2=0x7f690415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f68f800ec80 tx=0x7f68f80091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 shutdown_connections 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68f0060580 0x7f68f0062a30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69040b1440 0x7f69040c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69040ab6d0 0x7f690415d250 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 --2- 172.21.15.111:0/3202426601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f690406ae20 0x7f690415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 >> 172.21.15.111:0/3202426601 conn(0x7f69040c8cd0 msgr2=0x7f690406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:03.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 shutdown_connections 2023-12-19T07:48:03.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.719+0000 7f6908b23700 1 -- 172.21.15.111:0/3202426601 wait complete. 2023-12-19T07:48:03.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:48:03.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.907+0000 7f5178c16700 1 Processor -- start 2023-12-19T07:48:03.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 -- start start 2023-12-19T07:48:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f51740c69a0 0x7f51740c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517406d970 con 0x7f517406c7c0 2023-12-19T07:48:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517406cd60 con 0x7f51740c69a0 2023-12-19T07:48:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5178c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517406d040 con 0x7f517406de00 2023-12-19T07:48:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f51737fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f51740c69a0 0x7f51740c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:53616/0 (socket says 172.21.15.111:53616) 2023-12-19T07:48:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 -- 172.21.15.111:0/2098824463 learned_addr learned my addr 172.21.15.111:0/2098824463 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5173fff700 1 --1- 172.21.15.111:0/2098824463 >> v1:172.21.15.143:6789/0 conn(0x7f517406c7c0 0x7f517415cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42214/0 (socket says 172.21.15.111:42214) 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 -- 172.21.15.111:0/2098824463 >> v2:172.21.15.111:3300/0 conn(0x7f51740c69a0 msgr2=0x7f51740c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 --2- 172.21.15.111:0/2098824463 >> v2:172.21.15.111:3300/0 conn(0x7f51740c69a0 0x7f51740c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 -- 172.21.15.111:0/2098824463 >> v1:172.21.15.143:6789/0 conn(0x7f517406c7c0 legacy=0x7f517415cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 -- 172.21.15.111:0/2098824463 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517406d320 con 0x7f517406de00 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.911+0000 7f5172ffd700 1 --2- 172.21.15.111:0/2098824463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51680099e0 tx=0x7f5168009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3c4bc11831f0d54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f51727fc700 1 -- 172.21.15.111:0/2098824463 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516800b820 con 0x7f517406de00 2023-12-19T07:48:03.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f51727fc700 1 -- 172.21.15.111:0/2098824463 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516800b980 con 0x7f517406de00 2023-12-19T07:48:03.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f51727fc700 1 -- 172.21.15.111:0/2098824463 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516800bd20 con 0x7f517406de00 2023-12-19T07:48:03.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- 172.21.15.111:0/2098824463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 msgr2=0x7f517406b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 --2- 172.21.15.111:0/2098824463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51680099e0 tx=0x7f5168009250 comp rx=0 tx=0).stop 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- 172.21.15.111:0/2098824463 shutdown_connections 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 --2- 172.21.15.111:0/2098824463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f517406de00 0x7f517406b880 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 --2- 172.21.15.111:0/2098824463 >> v2:172.21.15.111:3300/0 conn(0x7f51740c69a0 0x7f51740c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- 172.21.15.111:0/2098824463 >> 172.21.15.111:0/2098824463 conn(0x7f5174067b50 msgr2=0x7f51740cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- 172.21.15.111:0/2098824463 shutdown_connections 2023-12-19T07:48:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- 172.21.15.111:0/2098824463 wait complete. 2023-12-19T07:48:03.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 Processor -- start 2023-12-19T07:48:03.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.915+0000 7f5178c16700 1 -- start start 2023-12-19T07:48:03.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f517406c7c0 0x7f51740c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51740c69a0 0x7f5174170a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740af9b0 con 0x7f517406de00 2023-12-19T07:48:03.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517415d720 con 0x7f517406c7c0 2023-12-19T07:48:03.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5174160490 con 0x7f51740c69a0 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37792/0 (socket says 172.21.15.111:37792) 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 -- 172.21.15.111:0/3798868836 learned_addr learned my addr 172.21.15.111:0/3798868836 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f51737fe700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f517406c7c0 0x7f51740c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5173fff700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51740c69a0 0x7f5174170a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 -- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51740c69a0 msgr2=0x7f5174170a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51740c69a0 0x7f5174170a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 -- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f517406c7c0 msgr2=0x7f51740c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f517406c7c0 0x7f51740c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 -- 172.21.15.111:0/3798868836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680094f0 con 0x7f517406de00 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5172ffd700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f517406d180 tx=0x7f5168017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168024440 con 0x7f517406de00 2023-12-19T07:48:03.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517415d9e0 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51680245c0 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168024980 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.919+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517415a2f0 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.923+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51740afb60 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.923+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f516801a420 con 0x7f517406de00 2023-12-19T07:48:03.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.923+0000 7f515b7fe700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 0x7f5160062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:03.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.923+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f516807eea0 con 0x7f517406de00 2023-12-19T07:48:03.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.923+0000 7f51737fe700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 0x7f5160062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:03.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.927+0000 7f51737fe700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 0x7f5160062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5164009ca0 tx=0x7f5164008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:03.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:03.931+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f516804f0f0 con 0x7f517406de00 2023-12-19T07:48:04.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.103+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f517404be40 con 0x7f517406de00 2023-12-19T07:48:04.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.103+0000 7f515b7fe700 1 -- 172.21.15.111:0/3798868836 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5168052710 con 0x7f517406de00 2023-12-19T07:48:04.087 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 msgr2=0x7f5160062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 0x7f5160062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5164009ca0 tx=0x7f5164008040 comp rx=0 tx=0).stop 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 msgr2=0x7f517414ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f517406d180 tx=0x7f5168017000 comp rx=0 tx=0).stop 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 shutdown_connections 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5160060550 0x7f5160062a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51740c69a0 0x7f5174170a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517406de00 0x7f517414ad20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 --2- 172.21.15.111:0/3798868836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f517406c7c0 0x7f51740c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 >> 172.21.15.111:0/3798868836 conn(0x7f5174067b50 msgr2=0x7f51740cb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:04.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 shutdown_connections 2023-12-19T07:48:04.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.107+0000 7f5178c16700 1 -- 172.21.15.111:0/3798868836 wait complete. 2023-12-19T07:48:04.112 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:48:04.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 Processor -- start 2023-12-19T07:48:04.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 -- start start 2023-12-19T07:48:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5de806baa0 0x7f5de815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5de80adab0 0x7f5de80adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de815ae30 con 0x7f5de8154780 2023-12-19T07:48:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de80abf70 con 0x7f5de80adab0 2023-12-19T07:48:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.287+0000 7f5dece03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de8154b50 con 0x7f5de806baa0 2023-12-19T07:48:04.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5de80adab0 0x7f5de80adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de6ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5de806baa0 0x7f5de815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de77fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5de8154780 0x7f5de8157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42218/0 (socket says 172.21.15.111:42218) 2023-12-19T07:48:04.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5de80adab0 0x7f5de80adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37808/0 (socket says 172.21.15.111:37808) 2023-12-19T07:48:04.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de77fe700 1 -- 172.21.15.111:0/1165505383 learned_addr learned my addr 172.21.15.111:0/1165505383 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:04.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 -- 172.21.15.111:0/1165505383 >> v2:172.21.15.143:3300/0 conn(0x7f5de806baa0 msgr2=0x7f5de815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:04.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 --2- 172.21.15.111:0/1165505383 >> v2:172.21.15.143:3300/0 conn(0x7f5de806baa0 0x7f5de815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 -- 172.21.15.111:0/1165505383 >> v1:172.21.15.143:6789/0 conn(0x7f5de8154780 legacy=0x7f5de8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 -- 172.21.15.111:0/1165505383 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de80ac210 con 0x7f5de80adab0 2023-12-19T07:48:04.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de7fff700 1 --2- 172.21.15.111:0/1165505383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de80adab0 0x7f5de80adeb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5de80cb990 tx=0x7f5de0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae8516a88d468e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de67fc700 1 -- 172.21.15.111:0/1165505383 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de000e070 con 0x7f5de80adab0 2023-12-19T07:48:04.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de67fc700 1 -- 172.21.15.111:0/1165505383 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de000bb60 con 0x7f5de80adab0 2023-12-19T07:48:04.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5de67fc700 1 -- 172.21.15.111:0/1165505383 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de0012430 con 0x7f5de80adab0 2023-12-19T07:48:04.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 -- 172.21.15.111:0/1165505383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de80adab0 msgr2=0x7f5de80adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 --2- 172.21.15.111:0/1165505383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de80adab0 0x7f5de80adeb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5de80cb990 tx=0x7f5de0009780 comp rx=0 tx=0).stop 2023-12-19T07:48:04.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 -- 172.21.15.111:0/1165505383 shutdown_connections 2023-12-19T07:48:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 --2- 172.21.15.111:0/1165505383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de80adab0 0x7f5de80adeb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 --2- 172.21.15.111:0/1165505383 >> v2:172.21.15.143:3300/0 conn(0x7f5de806baa0 0x7f5de815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 -- 172.21.15.111:0/1165505383 >> 172.21.15.111:0/1165505383 conn(0x7f5de8067b50 msgr2=0x7f5de80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.291+0000 7f5dece03700 1 -- 172.21.15.111:0/1165505383 shutdown_connections 2023-12-19T07:48:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.295+0000 7f5dece03700 1 -- 172.21.15.111:0/1165505383 wait complete. 2023-12-19T07:48:04.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.295+0000 7f5dece03700 1 Processor -- start 2023-12-19T07:48:04.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.295+0000 7f5dece03700 1 -- start start 2023-12-19T07:48:04.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5de80adab0 0x7f5de80c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5de8154780 0x7f5de80c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de815d560 con 0x7f5de806baa0 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81579b0 con 0x7f5de80adab0 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de815d280 con 0x7f5de8154780 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37816/0 (socket says 172.21.15.111:37816) 2023-12-19T07:48:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 -- 172.21.15.111:0/1853892774 learned_addr learned my addr 172.21.15.111:0/1853892774 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de6ffd700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5de80adab0 0x7f5de80c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de7fff700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5de8154780 0x7f5de80c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 -- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5de8154780 msgr2=0x7f5de80c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5de8154780 0x7f5de80c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 -- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5de80adab0 msgr2=0x7f5de80c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5de80adab0 0x7f5de80c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 -- 172.21.15.111:0/1853892774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0009a20 con 0x7f5de806baa0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5de77fe700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800c7c0 tx=0x7f5dd8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8005b30 con 0x7f5de806baa0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd8005c90 con 0x7f5de806baa0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de8157c50 con 0x7f5de806baa0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8004a80 con 0x7f5de806baa0 2023-12-19T07:48:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.299+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de80af0c0 con 0x7f5de806baa0 2023-12-19T07:48:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.303+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de80af2b0 con 0x7f5de806baa0 2023-12-19T07:48:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.303+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5dd80070d0 con 0x7f5de806baa0 2023-12-19T07:48:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.303+0000 7f5dcf7fe700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 0x7f5dd4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.303+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f5dd807bef0 con 0x7f5de806baa0 2023-12-19T07:48:04.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.303+0000 7f5de6ffd700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 0x7f5dd4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.307+0000 7f5de6ffd700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 0x7f5dd4062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e170 tx=0x7f5ddc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.311+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5dd8049be0 con 0x7f5de806baa0 2023-12-19T07:48:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.483+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5de804be40 con 0x7f5de806baa0 2023-12-19T07:48:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.483+0000 7f5dcf7fe700 1 -- 172.21.15.111:0/1853892774 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5dd804c140 con 0x7f5de806baa0 2023-12-19T07:48:04.466 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 msgr2=0x7f5dd4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 0x7f5dd4062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e170 tx=0x7f5ddc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 msgr2=0x7f5de80aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800c7c0 tx=0x7f5dd8007f70 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 shutdown_connections 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5dd4060550 0x7f5dd4062a00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5de80adab0 0x7f5de80c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5de8154780 0x7f5de80c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 --2- 172.21.15.111:0/1853892774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5de806baa0 0x7f5de80aeb80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 >> 172.21.15.111:0/1853892774 conn(0x7f5de8067b50 msgr2=0x7f5de80c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:04.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 shutdown_connections 2023-12-19T07:48:04.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.487+0000 7f5dece03700 1 -- 172.21.15.111:0/1853892774 wait complete. 2023-12-19T07:48:04.509 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-12-19T07:48:04.509 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T07:48:04.509 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:04.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 Processor -- start 2023-12-19T07:48:04.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 -- start start 2023-12-19T07:48:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f10800c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10800ca310 0x7f10800c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10800cb750 con 0x7f108015d230 2023-12-19T07:48:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800c8070 con 0x7f10800ca310 2023-12-19T07:48:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.679+0000 7f108562e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10800ca770 con 0x7f108006d8e0 2023-12-19T07:48:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f10767fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f10800c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107effd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f108015d230 0x7f10800cb040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42224/0 (socket says 172.21.15.111:42224) 2023-12-19T07:48:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107effd700 1 -- 172.21.15.111:0/1916979692 learned_addr learned my addr 172.21.15.111:0/1916979692 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10800ca310 0x7f10800c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107e7fc700 1 -- 172.21.15.111:0/1916979692 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660810836 0 0) 0x7f1070001490 con 0x7f108015d230 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107e7fc700 1 -- 172.21.15.111:0/1916979692 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c003570 con 0x7f108015d230 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 -- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 msgr2=0x7f10800c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 --2- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f10800c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 -- 172.21.15.111:0/1916979692 >> v1:172.21.15.143:6789/0 conn(0x7f108015d230 legacy=0x7f10800cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 -- 172.21.15.111:0/1916979692 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800cc5a0 con 0x7f10800ca310 2023-12-19T07:48:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107ffff700 1 --2- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10800ca310 0x7f10800c84c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10780098e0 tx=0x7f10780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bace63c19ccf809 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107e7fc700 1 -- 172.21.15.111:0/1916979692 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078011630 con 0x7f10800ca310 2023-12-19T07:48:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107e7fc700 1 -- 172.21.15.111:0/1916979692 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1078011790 con 0x7f10800ca310 2023-12-19T07:48:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f107e7fc700 1 -- 172.21.15.111:0/1916979692 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078011b30 con 0x7f10800ca310 2023-12-19T07:48:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 -- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10800ca310 msgr2=0x7f10800c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 --2- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10800ca310 0x7f10800c84c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10780098e0 tx=0x7f10780091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 -- 172.21.15.111:0/1916979692 shutdown_connections 2023-12-19T07:48:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 --2- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10800ca310 0x7f10800c84c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 --2- 172.21.15.111:0/1916979692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f10800c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 -- 172.21.15.111:0/1916979692 >> 172.21.15.111:0/1916979692 conn(0x7f1080068500 msgr2=0x7f10800688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 -- 172.21.15.111:0/1916979692 shutdown_connections 2023-12-19T07:48:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.683+0000 7f108562e700 1 -- 172.21.15.111:0/1916979692 wait complete. 2023-12-19T07:48:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 Processor -- start 2023-12-19T07:48:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 -- start start 2023-12-19T07:48:04.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f108015d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108015d230 0x7f1080170a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108015ea40 con 0x7f10800ca310 2023-12-19T07:48:04.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108015de30 con 0x7f108015d230 2023-12-19T07:48:04.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f108562e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108015e110 con 0x7f108006d8e0 2023-12-19T07:48:04.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f10767fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f10767fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37820/0 (socket says 172.21.15.111:37820) 2023-12-19T07:48:04.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.687+0000 7f10767fc700 1 -- 172.21.15.111:0/1177206726 learned_addr learned my addr 172.21.15.111:0/1177206726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:04.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f107effd700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f108015d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f107ffff700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108015d230 0x7f1080170a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 -- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 msgr2=0x7f108015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f108015d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 -- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108015d230 msgr2=0x7f1080170a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108015d230 0x7f1080170a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 -- 172.21.15.111:0/1177206726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009420 con 0x7f10800ca310 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10767fc700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f106800ec50 tx=0x7f10680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106800c850 con 0x7f10800ca310 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106800c9b0 con 0x7f10800ca310 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106800cd70 con 0x7f10800ca310 2023-12-19T07:48:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108015e3f0 con 0x7f10800ca310 2023-12-19T07:48:04.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108015db40 con 0x7f10800ca310 2023-12-19T07:48:04.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.691+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10800c78d0 con 0x7f10800ca310 2023-12-19T07:48:04.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.695+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1068013680 con 0x7f10800ca310 2023-12-19T07:48:04.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.695+0000 7f10777fe700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 0x7f1060062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:04.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.695+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f106807cd90 con 0x7f10800ca310 2023-12-19T07:48:04.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.695+0000 7f107effd700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 0x7f1060062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:04.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.699+0000 7f107effd700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 0x7f1060062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f10700015f0 tx=0x7f1070009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:04.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.703+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10800c78d0 con 0x7f10800ca310 2023-12-19T07:48:04.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.871+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10800c78d0 con 0x7f10800ca310 2023-12-19T07:48:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.871+0000 7f10777fe700 1 -- 172.21.15.111:0/1177206726 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7f10800c78d0 con 0x7f10800ca310 2023-12-19T07:48:04.856 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:04.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 msgr2=0x7f1060062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 0x7f1060062a00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f10700015f0 tx=0x7f1070009040 comp rx=0 tx=0).stop 2023-12-19T07:48:04.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 msgr2=0x7f10801704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f106800ec50 tx=0x7f10680091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 shutdown_connections 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1060060550 0x7f1060062a00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108015d230 0x7f1080170a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10800ca310 0x7f10801704e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 --2- 172.21.15.111:0/1177206726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108006d8e0 0x7f108015d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 >> 172.21.15.111:0/1177206726 conn(0x7f1080068500 msgr2=0x7f10800c2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 shutdown_connections 2023-12-19T07:48:04.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:04.875+0000 7f108562e700 1 -- 172.21.15.111:0/1177206726 wait complete. 2023-12-19T07:48:04.879 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:48:04.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:48:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.063+0000 7fe696eae700 1 Processor -- start 2023-12-19T07:48:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.063+0000 7fe696eae700 1 -- start start 2023-12-19T07:48:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe696eae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 0x7fe6900c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe696eae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe696eae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69006a7a0 con 0x7fe69006c7c0 2023-12-19T07:48:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe696eae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6900c7630 con 0x7fe69006ac30 2023-12-19T07:48:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe696eae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69006d4b0 con 0x7fe6900c6d20 2023-12-19T07:48:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe69544b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe694c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 0x7fe6900c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe69544b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:56358/0 (socket says 172.21.15.111:56358) 2023-12-19T07:48:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe69544b700 1 -- 172.21.15.111:0/3933940336 learned_addr learned my addr 172.21.15.111:0/3933940336 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.067+0000 7fe68ffff700 1 --1- 172.21.15.111:0/3933940336 >> v1:172.21.15.143:6789/0 conn(0x7fe69006c7c0 0x7fe69006cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42236/0 (socket says 172.21.15.111:42236) 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe694c4a700 1 -- 172.21.15.111:0/3933940336 >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 msgr2=0x7fe69006b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe694c4a700 1 --2- 172.21.15.111:0/3933940336 >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe694c4a700 1 -- 172.21.15.111:0/3933940336 >> v1:172.21.15.143:6789/0 conn(0x7fe69006c7c0 legacy=0x7fe69006cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe694c4a700 1 -- 172.21.15.111:0/3933940336 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69006d790 con 0x7fe6900c6d20 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe69544b700 1 --2- 172.21.15.111:0/3933940336 >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe694c4a700 1 --2- 172.21.15.111:0/3933940336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 0x7fe6900c70f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098e0 tx=0x7fe6840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac4ae41a209f871 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe68f7fe700 1 -- 172.21.15.111:0/3933940336 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400b640 con 0x7fe6900c6d20 2023-12-19T07:48:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe68f7fe700 1 -- 172.21.15.111:0/3933940336 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68400b7a0 con 0x7fe6900c6d20 2023-12-19T07:48:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe68f7fe700 1 -- 172.21.15.111:0/3933940336 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400bb40 con 0x7fe6900c6d20 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 -- 172.21.15.111:0/3933940336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 msgr2=0x7fe6900c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 --2- 172.21.15.111:0/3933940336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 0x7fe6900c70f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098e0 tx=0x7fe6840091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 -- 172.21.15.111:0/3933940336 shutdown_connections 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 --2- 172.21.15.111:0/3933940336 >> v2:172.21.15.143:3300/0 conn(0x7fe69006ac30 0x7fe69006b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 --2- 172.21.15.111:0/3933940336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6900c6d20 0x7fe6900c70f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 -- 172.21.15.111:0/3933940336 >> 172.21.15.111:0/3933940336 conn(0x7fe690067bb0 msgr2=0x7fe6900cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 -- 172.21.15.111:0/3933940336 shutdown_connections 2023-12-19T07:48:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.071+0000 7fe696eae700 1 -- 172.21.15.111:0/3933940336 wait complete. 2023-12-19T07:48:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 Processor -- start 2023-12-19T07:48:05.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 -- start start 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 0x7fe690151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6900c6d20 0x7fe690147920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690151af0 con 0x7fe69006c7c0 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900b15d0 con 0x7fe6900c6d20 2023-12-19T07:48:05.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe696eae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900b0550 con 0x7fe69006ac30 2023-12-19T07:48:05.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe68ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe68ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37828/0 (socket says 172.21.15.111:37828) 2023-12-19T07:48:05.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe694c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 0x7fe690151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe68ffff700 1 -- 172.21.15.111:0/4272702437 learned_addr learned my addr 172.21.15.111:0/4272702437 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:05.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.075+0000 7fe69544b700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6900c6d20 0x7fe690147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 -- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 msgr2=0x7fe690151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 0x7fe690151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 -- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6900c6d20 msgr2=0x7fe690147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6900c6d20 0x7fe690147920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 -- 172.21.15.111:0/4272702437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688009ae0 con 0x7fe69006c7c0 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe694c4a700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 0x7fe690151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68ffff700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00c7c0 tx=0x7fe67c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c005a70 con 0x7fe69006c7c0 2023-12-19T07:48:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c005bd0 con 0x7fe69006c7c0 2023-12-19T07:48:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe684009420 con 0x7fe69006c7c0 2023-12-19T07:48:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0070d0 con 0x7fe69006c7c0 2023-12-19T07:48:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6900b0e70 con 0x7fe69006c7c0 2023-12-19T07:48:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.079+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6900b0970 con 0x7fe69006c7c0 2023-12-19T07:48:05.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.083+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe67c007480 con 0x7fe69006c7c0 2023-12-19T07:48:05.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.083+0000 7fe68cff9700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.083+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fe67c07bf60 con 0x7fe69006c7c0 2023-12-19T07:48:05.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.083+0000 7fe694c4a700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.087+0000 7fe694c4a700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6840093f0 tx=0x7fe684017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.091+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe6900b0970 con 0x7fe69006c7c0 2023-12-19T07:48:05.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.263+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6900b0940 con 0x7fe69006c7c0 2023-12-19T07:48:05.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.263+0000 7fe68cff9700 1 -- 172.21.15.111:0/4272702437 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe67c04c1b0 con 0x7fe69006c7c0 2023-12-19T07:48:05.248 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:48:05.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 msgr2=0x7fe674062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6840093f0 tx=0x7fe684017000 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 msgr2=0x7fe690159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00c7c0 tx=0x7fe67c007f70 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 shutdown_connections 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6900c6d20 0x7fe690147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69006c7c0 0x7fe690159c80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 --2- 172.21.15.111:0/4272702437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69006ac30 0x7fe690151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 >> 172.21.15.111:0/4272702437 conn(0x7fe690067bb0 msgr2=0x7fe6900c2250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 shutdown_connections 2023-12-19T07:48:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.267+0000 7fe696eae700 1 -- 172.21.15.111:0/4272702437 wait complete. 2023-12-19T07:48:05.274 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:48:05.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.459+0000 7fc1e6145700 1 Processor -- start 2023-12-19T07:48:05.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.459+0000 7fc1e6145700 1 -- start start 2023-12-19T07:48:05.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1e6145700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1e015c520 0x7fc1e00a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1e6145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c8f0 0x7fc1e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1e6145700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e00c6ab0 con 0x7fc1e006ace0 2023-12-19T07:48:05.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1e6145700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e006b520 con 0x7fc1e015c520 2023-12-19T07:48:05.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1e6145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00cb910 con 0x7fc1e015c8f0 2023-12-19T07:48:05.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1e015c520 0x7fc1e00a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1deffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c8f0 0x7fc1e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1dffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc1e006ace0 0x7fc1e00ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42242/0 (socket says 172.21.15.111:42242) 2023-12-19T07:48:05.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1dffff700 1 -- 172.21.15.111:0/986642989 learned_addr learned my addr 172.21.15.111:0/986642989 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 -- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c8f0 msgr2=0x7fc1e006a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 --2- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c8f0 0x7fc1e006a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 -- 172.21.15.111:0/986642989 >> v1:172.21.15.143:6789/0 conn(0x7fc1e006ace0 legacy=0x7fc1e00ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 -- 172.21.15.111:0/986642989 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e006b240 con 0x7fc1e015c520 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1de7fc700 1 -- 172.21.15.111:0/986642989 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584105750 0 0) 0x7fc1e00c6ab0 con 0x7fc1e006ace0 2023-12-19T07:48:05.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.463+0000 7fc1df7fe700 1 --2- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c520 0x7fc1e00a8850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00098e0 tx=0x7fc1d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abbc65830c8b749 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1de7fc700 1 -- 172.21.15.111:0/986642989 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000b6d0 con 0x7fc1e015c520 2023-12-19T07:48:05.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1de7fc700 1 -- 172.21.15.111:0/986642989 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d000b830 con 0x7fc1e015c520 2023-12-19T07:48:05.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1de7fc700 1 -- 172.21.15.111:0/986642989 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000bbd0 con 0x7fc1e015c520 2023-12-19T07:48:05.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 -- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c520 msgr2=0x7fc1e00a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 --2- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c520 0x7fc1e00a8850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00098e0 tx=0x7fc1d00091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 -- 172.21.15.111:0/986642989 shutdown_connections 2023-12-19T07:48:05.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 --2- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c8f0 0x7fc1e006a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 --2- 172.21.15.111:0/986642989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c520 0x7fc1e00a8850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 -- 172.21.15.111:0/986642989 >> 172.21.15.111:0/986642989 conn(0x7fc1e0067bb0 msgr2=0x7fc1e00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:05.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 -- 172.21.15.111:0/986642989 shutdown_connections 2023-12-19T07:48:05.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 -- 172.21.15.111:0/986642989 wait complete. 2023-12-19T07:48:05.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.467+0000 7fc1e6145700 1 Processor -- start 2023-12-19T07:48:05.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 -- start start 2023-12-19T07:48:05.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 0x7fc1e00c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c8f0 0x7fc1e00c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00b08b0 con 0x7fc1e006ace0 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00afca0 con 0x7fc1e015c8f0 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1e6145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e00aff80 con 0x7fc1e015c520 2023-12-19T07:48:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1df7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 0x7fc1e00c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1dffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c8f0 0x7fc1e00c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1df7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37832/0 (socket says 172.21.15.111:37832) 2023-12-19T07:48:05.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1df7fe700 1 -- 172.21.15.111:0/1657742362 learned_addr learned my addr 172.21.15.111:0/1657742362 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:05.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 -- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c8f0 msgr2=0x7fc1e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c8f0 0x7fc1e00c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 -- 172.21.15.111:0/1657742362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 msgr2=0x7fc1e015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 -- 172.21.15.111:0/1657742362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d0009420 con 0x7fc1e015c520 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1df7fe700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:05.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.471+0000 7fc1deffd700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 0x7fc1e00c3940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400e8d0 tx=0x7fc1d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4009450 con 0x7fc1e015c520 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d4013680 con 0x7fc1e015c520 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4013a40 con 0x7fc1e015c520 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e00b0260 con 0x7fc1e015c520 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e015da60 con 0x7fc1e015c520 2023-12-19T07:48:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e0071120 con 0x7fc1e015c520 2023-12-19T07:48:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.475+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc1d40123d0 con 0x7fc1e015c520 2023-12-19T07:48:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.479+0000 7fc1c77fe700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 0x7fc1cc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.479+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fc1d407cb50 con 0x7fc1e015c520 2023-12-19T07:48:05.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.479+0000 7fc1df7fe700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 0x7fc1cc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.479+0000 7fc1df7fe700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 0x7fc1cc062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0003080 tx=0x7fc1d0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.487+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc1d4050830 con 0x7fc1e015c520 2023-12-19T07:48:05.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.659+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1e00b00c0 con 0x7fc1e015c520 2023-12-19T07:48:05.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.659+0000 7fc1c77fe700 1 -- 172.21.15.111:0/1657742362 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc1d4017020 con 0x7fc1e015c520 2023-12-19T07:48:05.644 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:05.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 msgr2=0x7fc1cc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 0x7fc1cc062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0003080 tx=0x7fc1d0017000 comp rx=0 tx=0).stop 2023-12-19T07:48:05.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 msgr2=0x7fc1e00c3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 0x7fc1e00c3940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400e8d0 tx=0x7fc1d40091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 shutdown_connections 2023-12-19T07:48:05.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc1cc060550 0x7fc1cc062a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1e015c8f0 0x7fc1e00c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1e015c520 0x7fc1e00c3940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 --2- 172.21.15.111:0/1657742362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1e006ace0 0x7fc1e015efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 >> 172.21.15.111:0/1657742362 conn(0x7fc1e0067bb0 msgr2=0x7fc1e00c83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:05.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 shutdown_connections 2023-12-19T07:48:05.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.663+0000 7fc1e6145700 1 -- 172.21.15.111:0/1657742362 wait complete. 2023-12-19T07:48:05.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:48:05.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.851+0000 7f026f785700 1 Processor -- start 2023-12-19T07:48:05.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026f785700 1 -- start start 2023-12-19T07:48:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026f785700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02680cb970 0x7f026806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026f785700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680cbdf0 con 0x7f02680cecd0 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026f785700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680cc0d0 con 0x7f0268158e20 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026f785700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02680cc3b0 con 0x7f02680cb970 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026cd20700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02680cb970 0x7f026806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026dd22700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f02680cecd0 0x7f026806b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52584/0 (socket says 172.21.15.111:52584) 2023-12-19T07:48:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.855+0000 7f026dd22700 1 -- 172.21.15.111:0/93786469 learned_addr learned my addr 172.21.15.111:0/93786469 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026cd20700 1 -- 172.21.15.111:0/93786469 >> v1:172.21.15.143:6789/0 conn(0x7f0268158e20 legacy=0x7f02680ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026cd20700 1 -- 172.21.15.111:0/93786469 >> v1:172.21.15.111:6789/0 conn(0x7f02680cecd0 legacy=0x7f026806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f025ffff700 1 -- 172.21.15.111:0/93786469 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415659971 0 0) 0x7f02680cbdf0 con 0x7f02680cecd0 2023-12-19T07:48:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026cd20700 1 -- 172.21.15.111:0/93786469 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680cc690 con 0x7f02680cb970 2023-12-19T07:48:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026cd20700 1 --2- 172.21.15.111:0/93786469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02680cb970 0x7f026806acc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0258009910 tx=0x7f02580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99485cfc56827f43 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f025ffff700 1 -- 172.21.15.111:0/93786469 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025800b820 con 0x7f02680cb970 2023-12-19T07:48:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f025ffff700 1 -- 172.21.15.111:0/93786469 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025800b980 con 0x7f02680cb970 2023-12-19T07:48:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f025ffff700 1 -- 172.21.15.111:0/93786469 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025800bd20 con 0x7f02680cb970 2023-12-19T07:48:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 -- 172.21.15.111:0/93786469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02680cb970 msgr2=0x7f026806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 --2- 172.21.15.111:0/93786469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02680cb970 0x7f026806acc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0258009910 tx=0x7f02580091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 -- 172.21.15.111:0/93786469 shutdown_connections 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 --2- 172.21.15.111:0/93786469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02680cb970 0x7f026806acc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 -- 172.21.15.111:0/93786469 >> 172.21.15.111:0/93786469 conn(0x7f0268067bb0 msgr2=0x7f02680d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 -- 172.21.15.111:0/93786469 shutdown_connections 2023-12-19T07:48:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.859+0000 7f026f785700 1 -- 172.21.15.111:0/93786469 wait complete. 2023-12-19T07:48:05.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 Processor -- start 2023-12-19T07:48:05.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 -- start start 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02680cecd0 0x7f02680c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 0x7f026814fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02680c8900 con 0x7f0268158e20 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268141f40 con 0x7f0268142250 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026f785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0268150d10 con 0x7f02680cecd0 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37850/0 (socket says 172.21.15.111:37850) 2023-12-19T07:48:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 -- 172.21.15.111:0/2799052334 learned_addr learned my addr 172.21.15.111:0/2799052334 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026d521700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02680cecd0 0x7f02680c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026dd22700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 0x7f026814fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 -- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02680cecd0 msgr2=0x7f02680c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02680cecd0 0x7f02680c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 -- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 msgr2=0x7f026814fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 0x7f026814fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 -- 172.21.15.111:0/2799052334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009420 con 0x7f0268158e20 2023-12-19T07:48:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026dd22700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 0x7f026814fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.863+0000 7f026cd20700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0258013000 tx=0x7f0258017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025801f070 con 0x7f0268158e20 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025800bd00 con 0x7f0268158e20 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025801a650 con 0x7f0268158e20 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0268150ff0 con 0x7f0268158e20 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02680c8590 con 0x7f0268158e20 2023-12-19T07:48:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02680c8ab0 con 0x7f0268158e20 2023-12-19T07:48:05.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.867+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f025801a9c0 con 0x7f0268158e20 2023-12-19T07:48:05.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.871+0000 7f025d7fa700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 0x7f024c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:05.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.871+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f025807dcd0 con 0x7f0268158e20 2023-12-19T07:48:05.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.871+0000 7f026d521700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 0x7f024c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:05.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.871+0000 7f026d521700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 0x7f024c062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f02640037c0 tx=0x7f0264009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:05.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:05.875+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02680c8ab0 con 0x7f0268158e20 2023-12-19T07:48:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.047+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02680c8ab0 con 0x7f0268158e20 2023-12-19T07:48:06.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.047+0000 7f025d7fa700 1 -- 172.21.15.111:0/2799052334 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02680c8ab0 con 0x7f0268158e20 2023-12-19T07:48:06.031 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:48:06.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 msgr2=0x7f024c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 0x7f024c062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f02640037c0 tx=0x7f0264009040 comp rx=0 tx=0).stop 2023-12-19T07:48:06.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 msgr2=0x7f026814f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0258013000 tx=0x7f0258017000 comp rx=0 tx=0).stop 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 shutdown_connections 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f024c060550 0x7f024c062a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268142250 0x7f026814fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0268158e20 0x7f026814f5a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 --2- 172.21.15.111:0/2799052334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02680cecd0 0x7f02680c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 >> 172.21.15.111:0/2799052334 conn(0x7f0268067bb0 msgr2=0x7f026814ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:06.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 shutdown_connections 2023-12-19T07:48:06.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.051+0000 7f026f785700 1 -- 172.21.15.111:0/2799052334 wait complete. 2023-12-19T07:48:06.057 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:48:06.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.243+0000 7f6311ef0700 1 Processor -- start 2023-12-19T07:48:06.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.243+0000 7f6311ef0700 1 -- start start 2023-12-19T07:48:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f6311ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c158e20 0x7f630c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f6311ef0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f630c0cb970 0x7f630c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f6311ef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630c0cbdf0 con 0x7f630c0ca2c0 2023-12-19T07:48:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f6311ef0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c0cc0d0 con 0x7f630c0cb970 2023-12-19T07:48:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f6311ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c0cc3b0 con 0x7f630c158e20 2023-12-19T07:48:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f630c0cb970 0x7f630c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f630c0cb970 0x7f630c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37862/0 (socket says 172.21.15.111:37862) 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 -- 172.21.15.111:0/971928425 learned_addr learned my addr 172.21.15.111:0/971928425 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630b7fe700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c158e20 0x7f630c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 -- 172.21.15.111:0/971928425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c158e20 msgr2=0x7f630c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c158e20 0x7f630c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 -- 172.21.15.111:0/971928425 >> v1:172.21.15.143:6789/0 conn(0x7f630c0ca2c0 legacy=0x7f630c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 -- 172.21.15.111:0/971928425 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c0cc690 con 0x7f630c0cb970 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.247+0000 7f630affd700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c0cb970 0x7f630c0ceb00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63000099b0 tx=0x7f6300009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b913e17ead6eb9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f630a7fc700 1 -- 172.21.15.111:0/971928425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630000b860 con 0x7f630c0cb970 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f630a7fc700 1 -- 172.21.15.111:0/971928425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630000b9c0 con 0x7f630c0cb970 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f630a7fc700 1 -- 172.21.15.111:0/971928425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630000bd60 con 0x7f630c0cb970 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- 172.21.15.111:0/971928425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c0cb970 msgr2=0x7f630c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c0cb970 0x7f630c0ceb00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63000099b0 tx=0x7f6300009250 comp rx=0 tx=0).stop 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- 172.21.15.111:0/971928425 shutdown_connections 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c0cb970 0x7f630c0ceb00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 --2- 172.21.15.111:0/971928425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c158e20 0x7f630c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- 172.21.15.111:0/971928425 >> 172.21.15.111:0/971928425 conn(0x7f630c067bb0 msgr2=0x7f630c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- 172.21.15.111:0/971928425 shutdown_connections 2023-12-19T07:48:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- 172.21.15.111:0/971928425 wait complete. 2023-12-19T07:48:06.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 Processor -- start 2023-12-19T07:48:06.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.251+0000 7f6311ef0700 1 -- start start 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f630c0ca2c0 0x7f630c146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 0x7f630c152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1509d0 con 0x7f630c158e20 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c0b48b0 con 0x7f630c0ca2c0 2023-12-19T07:48:06.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c150c40 con 0x7f630c0b4410 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37868/0 (socket says 172.21.15.111:37868) 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 -- 172.21.15.111:0/2028533632 learned_addr learned my addr 172.21.15.111:0/2028533632 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630b7fe700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f630c0ca2c0 0x7f630c146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630bfff700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 0x7f630c152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 -- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 msgr2=0x7f630c152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 0x7f630c152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 -- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f630c0ca2c0 msgr2=0x7f630c146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f630c0ca2c0 0x7f630c146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 -- 172.21.15.111:0/2028533632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63000094f0 con 0x7f630c158e20 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630bfff700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 0x7f630c152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f630affd700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f630c069a80 tx=0x7f6300017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300024440 con 0x7f630c158e20 2023-12-19T07:48:06.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63000245c0 con 0x7f630c158e20 2023-12-19T07:48:06.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c150f20 con 0x7f630c158e20 2023-12-19T07:48:06.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300024980 con 0x7f630c158e20 2023-12-19T07:48:06.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.255+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c0b4b90 con 0x7f630c158e20 2023-12-19T07:48:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.259+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c151160 con 0x7f630c158e20 2023-12-19T07:48:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.259+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f630001a420 con 0x7f630c158e20 2023-12-19T07:48:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.263+0000 7f62f37fe700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 0x7f62f8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.263+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f630007e900 con 0x7f630c158e20 2023-12-19T07:48:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.263+0000 7f630b7fe700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 0x7f62f8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.263+0000 7f630b7fe700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 0x7f62f8062a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006fd0 tx=0x7f62fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.267+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f630c151160 con 0x7f630c158e20 2023-12-19T07:48:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.443+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f630c151160 con 0x7f630c158e20 2023-12-19T07:48:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.443+0000 7f62f37fe700 1 -- 172.21.15.111:0/2028533632 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f630c151160 con 0x7f630c158e20 2023-12-19T07:48:06.428 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 msgr2=0x7f62f8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 0x7f62f8062a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62fc006fd0 tx=0x7f62fc008040 comp rx=0 tx=0).stop 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 msgr2=0x7f630c0b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f630c069a80 tx=0x7f6300017000 comp rx=0 tx=0).stop 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 shutdown_connections 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f62f8060580 0x7f62f8062a30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f630c0b4410 0x7f630c152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630c158e20 0x7f630c0b3ed0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 --2- 172.21.15.111:0/2028533632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f630c0ca2c0 0x7f630c146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 >> 172.21.15.111:0/2028533632 conn(0x7f630c067bb0 msgr2=0x7f630c157780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 shutdown_connections 2023-12-19T07:48:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.447+0000 7f6311ef0700 1 -- 172.21.15.111:0/2028533632 wait complete. 2023-12-19T07:48:06.452 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-12-19T07:48:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.631+0000 7fe9153c5700 1 Processor -- start 2023-12-19T07:48:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 -- start start 2023-12-19T07:48:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe9101552f0 0x7fe9101556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe9100ae520 0x7fe9100c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91006a9d0 con 0x7fe91006c7c0 2023-12-19T07:48:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100c1b90 con 0x7fe9100ae520 2023-12-19T07:48:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe9153c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91006acb0 con 0x7fe9101552f0 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe90e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe9100ae520 0x7fe9100c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe90e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe9100ae520 0x7fe9100c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37872/0 (socket says 172.21.15.111:37872) 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.635+0000 7fe90e7fc700 1 -- 172.21.15.111:0/2495882408 learned_addr learned my addr 172.21.15.111:0/2495882408 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90effd700 1 --2- 172.21.15.111:0/2495882408 >> v2:172.21.15.143:3300/0 conn(0x7fe9101552f0 0x7fe9101556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90e7fc700 1 -- 172.21.15.111:0/2495882408 >> v2:172.21.15.143:3300/0 conn(0x7fe9101552f0 msgr2=0x7fe9101556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90e7fc700 1 --2- 172.21.15.111:0/2495882408 >> v2:172.21.15.143:3300/0 conn(0x7fe9101552f0 0x7fe9101556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90e7fc700 1 -- 172.21.15.111:0/2495882408 >> v1:172.21.15.143:6789/0 conn(0x7fe91006c7c0 legacy=0x7fe91006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90e7fc700 1 -- 172.21.15.111:0/2495882408 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91006af90 con 0x7fe9100ae520 2023-12-19T07:48:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90e7fc700 1 --2- 172.21.15.111:0/2495882408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe9100c8000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe910069700 tx=0x7fe904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb853107fcac990 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90dffb700 1 -- 172.21.15.111:0/2495882408 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90400b860 con 0x7fe9100ae520 2023-12-19T07:48:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90dffb700 1 -- 172.21.15.111:0/2495882408 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90400b9c0 con 0x7fe9100ae520 2023-12-19T07:48:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe90dffb700 1 -- 172.21.15.111:0/2495882408 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90400bd60 con 0x7fe9100ae520 2023-12-19T07:48:06.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 -- 172.21.15.111:0/2495882408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 msgr2=0x7fe9100c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 --2- 172.21.15.111:0/2495882408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe9100c8000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe910069700 tx=0x7fe904009250 comp rx=0 tx=0).stop 2023-12-19T07:48:06.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 -- 172.21.15.111:0/2495882408 shutdown_connections 2023-12-19T07:48:06.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 --2- 172.21.15.111:0/2495882408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe9100c8000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 --2- 172.21.15.111:0/2495882408 >> v2:172.21.15.143:3300/0 conn(0x7fe9101552f0 0x7fe9101556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 -- 172.21.15.111:0/2495882408 >> 172.21.15.111:0/2495882408 conn(0x7fe9100c8850 msgr2=0x7fe9100c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:06.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 -- 172.21.15.111:0/2495882408 shutdown_connections 2023-12-19T07:48:06.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.639+0000 7fe9153c5700 1 -- 172.21.15.111:0/2495882408 wait complete. 2023-12-19T07:48:06.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.643+0000 7fe9153c5700 1 Processor -- start 2023-12-19T07:48:06.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.643+0000 7fe9153c5700 1 -- start start 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.643+0000 7fe9153c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 0x7fe9101569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9101552f0 0x7fe9100afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100aad50 con 0x7fe9100ae520 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910156fa0 con 0x7fe9101552f0 2023-12-19T07:48:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe910157560 con 0x7fe91006c7c0 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37878/0 (socket says 172.21.15.111:37878) 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 -- 172.21.15.111:0/1029247829 learned_addr learned my addr 172.21.15.111:0/1029247829 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90effd700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 0x7fe9101569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90f7fe700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9101552f0 0x7fe9100afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 -- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 msgr2=0x7fe9101569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 0x7fe9101569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 -- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9101552f0 msgr2=0x7fe9100afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9101552f0 0x7fe9100afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 -- 172.21.15.111:0/1029247829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9040094f0 con 0x7fe9100ae520 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90effd700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 0x7fe9101569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe90e7fc700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe91006a560 tx=0x7fe904017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904024440 con 0x7fe9100ae520 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9040245a0 con 0x7fe9100ae520 2023-12-19T07:48:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9100a9400 con 0x7fe9100ae520 2023-12-19T07:48:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904024960 con 0x7fe9100ae520 2023-12-19T07:48:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.647+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9101571f0 con 0x7fe9100ae520 2023-12-19T07:48:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.651+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9101576f0 con 0x7fe9100ae520 2023-12-19T07:48:06.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.651+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe904024d50 con 0x7fe9100ae520 2023-12-19T07:48:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.651+0000 7fe8f6ffd700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 0x7fe8fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.651+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fe90407e850 con 0x7fe9100ae520 2023-12-19T07:48:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.655+0000 7fe90effd700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 0x7fe8fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.655+0000 7fe90effd700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 0x7fe8fc062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe90000ae40 tx=0x7fe90000c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.659+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe90404ea00 con 0x7fe9100ae520 2023-12-19T07:48:06.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:06.891+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fe9100a98e0 con 0x7fe9100ae520 2023-12-19T07:48:07.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.131+0000 7fe8f6ffd700 1 -- 172.21.15.111:0/1029247829 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe904052020 con 0x7fe9100ae520 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 msgr2=0x7fe8fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 0x7fe8fc062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe90000ae40 tx=0x7fe90000c040 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 msgr2=0x7fe910153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe91006a560 tx=0x7fe904017000 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 shutdown_connections 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8fc060550 0x7fe8fc062a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9101552f0 0x7fe9100afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100ae520 0x7fe910153fe0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 --2- 172.21.15.111:0/1029247829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe91006c7c0 0x7fe9101569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 >> 172.21.15.111:0/1029247829 conn(0x7fe9100c8850 msgr2=0x7fe9100ca5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 shutdown_connections 2023-12-19T07:48:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.135+0000 7fe9153c5700 1 -- 172.21.15.111:0/1029247829 wait complete. 2023-12-19T07:48:07.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T07:48:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.327+0000 7ff78264c700 1 Processor -- start 2023-12-19T07:48:07.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.327+0000 7ff78264c700 1 -- start start 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78264c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c158c90 0x7ff77c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78264c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff77c159770 0x7ff77c06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78264c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c06c850 con 0x7ff77c06aeb0 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78264c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c06b470 con 0x7ff77c159770 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78264c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c06b750 con 0x7ff77c158c90 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff780e49700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff77c06aeb0 0x7ff77c159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52596/0 (socket says 172.21.15.111:52596) 2023-12-19T07:48:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff780e49700 1 -- 172.21.15.111:0/34252059 learned_addr learned my addr 172.21.15.111:0/34252059 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:07.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff781e4b700 1 --2- 172.21.15.111:0/34252059 >> v2:172.21.15.143:3300/0 conn(0x7ff77c159770 0x7ff77c06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff78164a700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c158c90 0x7ff77c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff773fff700 1 -- 172.21.15.111:0/34252059 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946528867 0 0) 0x7ff77c06c850 con 0x7ff77c06aeb0 2023-12-19T07:48:07.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.331+0000 7ff773fff700 1 -- 172.21.15.111:0/34252059 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff768003570 con 0x7ff77c06aeb0 2023-12-19T07:48:07.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff781e4b700 1 -- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c158c90 msgr2=0x7ff77c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:07.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff781e4b700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c158c90 0x7ff77c0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff781e4b700 1 -- 172.21.15.111:0/34252059 >> v1:172.21.15.111:6789/0 conn(0x7ff77c06aeb0 legacy=0x7ff77c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff781e4b700 1 -- 172.21.15.111:0/34252059 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c06ba30 con 0x7ff77c159770 2023-12-19T07:48:07.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff781e4b700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c06cce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e60 tx=0x7ff774009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=432821c772e3e401 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:07.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff773fff700 1 -- 172.21.15.111:0/34252059 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400e070 con 0x7ff77c159770 2023-12-19T07:48:07.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff773fff700 1 -- 172.21.15.111:0/34252059 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77400bab0 con 0x7ff77c159770 2023-12-19T07:48:07.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff773fff700 1 -- 172.21.15.111:0/34252059 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77400be70 con 0x7ff77c159770 2023-12-19T07:48:07.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 -- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 msgr2=0x7ff77c06cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c06cce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e60 tx=0x7ff774009780 comp rx=0 tx=0).stop 2023-12-19T07:48:07.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 -- 172.21.15.111:0/34252059 shutdown_connections 2023-12-19T07:48:07.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c06cce0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 --2- 172.21.15.111:0/34252059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c158c90 0x7ff77c0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 -- 172.21.15.111:0/34252059 >> 172.21.15.111:0/34252059 conn(0x7ff77c067bb0 msgr2=0x7ff77c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:07.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 -- 172.21.15.111:0/34252059 shutdown_connections 2023-12-19T07:48:07.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.335+0000 7ff78264c700 1 -- 172.21.15.111:0/34252059 wait complete. 2023-12-19T07:48:07.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 Processor -- start 2023-12-19T07:48:07.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 -- start start 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 0x7ff77c0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 0x7ff77c0b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c15bea0 con 0x7ff77c158c90 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c15b570 con 0x7ff77c159770 2023-12-19T07:48:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.339+0000 7ff78264c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c15b850 con 0x7ff77c06aeb0 2023-12-19T07:48:07.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff780e49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 0x7ff77c0b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff78164a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 0x7ff77c0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff78164a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 0x7ff77c0b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:53778/0 (socket says 172.21.15.111:53778) 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff78164a700 1 -- 172.21.15.111:0/1214031880 learned_addr learned my addr 172.21.15.111:0/1214031880 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 -- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 msgr2=0x7ff77c0b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 0x7ff77c0b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 -- 172.21.15.111:0/1214031880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 msgr2=0x7ff77c0b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 0x7ff77c0b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 -- 172.21.15.111:0/1214031880 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff774009a20 con 0x7ff77c159770 2023-12-19T07:48:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff780e49700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 0x7ff77c0b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff781e4b700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c155640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff774016000 tx=0x7ff774019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774012e10 con 0x7ff77c159770 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774027470 con 0x7ff77c159770 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774027830 con 0x7ff77c159770 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c15bb30 con 0x7ff77c159770 2023-12-19T07:48:07.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.343+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c155b80 con 0x7ff77c159770 2023-12-19T07:48:07.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.347+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c15c010 con 0x7ff77c159770 2023-12-19T07:48:07.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.347+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff774024070 con 0x7ff77c159770 2023-12-19T07:48:07.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.347+0000 7ff7717fa700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 0x7ff760062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:07.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.347+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7ff77407f170 con 0x7ff77c159770 2023-12-19T07:48:07.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.347+0000 7ff78164a700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 0x7ff760062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:07.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.351+0000 7ff78164a700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 0x7ff760062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff778009e70 tx=0x7ff7780091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:07.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.359+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff77404f320 con 0x7ff77c159770 2023-12-19T07:48:07.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.535+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff77c15b990 con 0x7ff77c159770 2023-12-19T07:48:07.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.547+0000 7ff7717fa700 1 -- 172.21.15.111:0/1214031880 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff77c15b990 con 0x7ff77c159770 2023-12-19T07:48:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.551+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 msgr2=0x7ff760062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.551+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 0x7ff760062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff778009e70 tx=0x7ff7780091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.551+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 msgr2=0x7ff77c155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.551+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c155640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff774016000 tx=0x7ff774019000 comp rx=0 tx=0).stop 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 shutdown_connections 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff760060550 0x7ff760062a00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff77c159770 0x7ff77c155640 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff77c158c90 0x7ff77c0b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 --2- 172.21.15.111:0/1214031880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff77c06aeb0 0x7ff77c0b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 >> 172.21.15.111:0/1214031880 conn(0x7ff77c067bb0 msgr2=0x7ff77c06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 shutdown_connections 2023-12-19T07:48:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:07.555+0000 7ff78264c700 1 -- 172.21.15.111:0/1214031880 wait complete. 2023-12-19T07:48:07.558 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T07:48:07.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T07:48:07.597 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T07:48:07.646 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T07:48:07.647 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T07:48:07.647 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:48:07.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:07.687 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T07:48:07.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:48:07.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:07.688 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T07:48:07.749 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi143.front.sepia.ceph.com... 2023-12-19T07:48:07.749 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:48:07.749 DEBUG:teuthology.orchestra.run.smithi143:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-12-19T07:48:07.790 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T07:48:07.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:48:07.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:07.791 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T07:48:07.860 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T07:48:07.861 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:48:07.861 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-12-19T07:48:07.895 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T07:48:07.895 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:48:07.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:07.895 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T07:48:07.957 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi143.front.sepia.ceph.com... 2023-12-19T07:48:07.957 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:48:07.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-12-19T07:48:07.988 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T07:48:07.988 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:48:07.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:07.988 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T07:48:08.049 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T07:48:08.049 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:08.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 Processor -- start 2023-12-19T07:48:08.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 -- start start 2023-12-19T07:48:08.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d7c15d320 0x7f8d7c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d7c0c15b0 0x7f8d7c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0c7190 con 0x7f8d7c06d690 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0c1b50 con 0x7f8d7c15d320 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d83f85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0c1e30 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.239+0000 7f8d81d21700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d7c15d320 0x7f8d7c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d82522700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8d7c06d690 0x7f8d7c0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42286/0 (socket says 172.21.15.111:42286) 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d82522700 1 -- 172.21.15.111:0/1283759435 learned_addr learned my addr 172.21.15.111:0/1283759435 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:08.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81520700 1 --2- 172.21.15.111:0/1283759435 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c0c15b0 0x7f8d7c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d80d1f700 1 -- 172.21.15.111:0/1283759435 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69102665 0 0) 0x7f8d7c0c7190 con 0x7f8d7c06d690 2023-12-19T07:48:08.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d80d1f700 1 -- 172.21.15.111:0/1283759435 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d68003570 con 0x7f8d7c06d690 2023-12-19T07:48:08.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81d21700 1 -- 172.21.15.111:0/1283759435 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c0c15b0 msgr2=0x7f8d7c0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81d21700 1 --2- 172.21.15.111:0/1283759435 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c0c15b0 0x7f8d7c0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81d21700 1 -- 172.21.15.111:0/1283759435 >> v1:172.21.15.143:6789/0 conn(0x7f8d7c06d690 legacy=0x7f8d7c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81d21700 1 -- 172.21.15.111:0/1283759435 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0c6e20 con 0x7f8d7c15d320 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d81d21700 1 --2- 172.21.15.111:0/1283759435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c15d320 0x7f8d7c15d6f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c06f530 tx=0x7f8d780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fad02eab0817b13 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d80d1f700 1 -- 172.21.15.111:0/1283759435 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78011630 con 0x7f8d7c15d320 2023-12-19T07:48:08.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d80d1f700 1 -- 172.21.15.111:0/1283759435 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d780117b0 con 0x7f8d7c15d320 2023-12-19T07:48:08.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d80d1f700 1 -- 172.21.15.111:0/1283759435 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78011b70 con 0x7f8d7c15d320 2023-12-19T07:48:08.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 -- 172.21.15.111:0/1283759435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c15d320 msgr2=0x7f8d7c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1283759435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c15d320 0x7f8d7c15d6f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c06f530 tx=0x7f8d780091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 -- 172.21.15.111:0/1283759435 shutdown_connections 2023-12-19T07:48:08.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1283759435 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c0c15b0 0x7f8d7c0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1283759435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c15d320 0x7f8d7c15d6f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 -- 172.21.15.111:0/1283759435 >> 172.21.15.111:0/1283759435 conn(0x7f8d7c0c8850 msgr2=0x7f8d7c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:08.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 -- 172.21.15.111:0/1283759435 shutdown_connections 2023-12-19T07:48:08.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.243+0000 7f8d83f85700 1 -- 172.21.15.111:0/1283759435 wait complete. 2023-12-19T07:48:08.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 Processor -- start 2023-12-19T07:48:08.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 -- start start 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c06d690 0x7f8d7c1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c15d320 0x7f8d7c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c161220 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c160930 con 0x7f8d7c15d320 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d83f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c160bd0 con 0x7f8d7c06d690 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d81520700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d81520700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37910/0 (socket says 172.21.15.111:37910) 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d81520700 1 -- 172.21.15.111:0/1090704815 learned_addr learned my addr 172.21.15.111:0/1090704815 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:08.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d81d21700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c06d690 0x7f8d7c1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.247+0000 7f8d82522700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c15d320 0x7f8d7c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 -- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c06d690 msgr2=0x7f8d7c1574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c06d690 0x7f8d7c1574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 -- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c15d320 msgr2=0x7f8d7c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c15d320 0x7f8d7c0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 -- 172.21.15.111:0/1090704815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009420 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d81520700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ec80 tx=0x7f8d6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00c9d0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c00cb30 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c160eb0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c013470 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c0c2e90 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.251+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c1613d0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.255+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8d6c013820 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.255+0000 7f8d71ffb700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 0x7f8d60062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.255+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f8d6c07cef0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.255+0000 7f8d81d21700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 0x7f8d60062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.259+0000 7f8d81d21700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 0x7f8d60062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c06f530 tx=0x7f8d78017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.263+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d7c1613d0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.471+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7c071120 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.471+0000 7f8d71ffb700 1 -- 172.21.15.111:0/1090704815 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f8d6c04d0a0 con 0x7f8d7c0c15b0 2023-12-19T07:48:08.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:08.457 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:48:08.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 msgr2=0x7f8d60062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 0x7f8d60062a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c06f530 tx=0x7f8d78017000 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 msgr2=0x7f8d7c0c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ec80 tx=0x7f8d6c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 shutdown_connections 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d60060550 0x7f8d60062a00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c15d320 0x7f8d7c0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c0c15b0 0x7f8d7c0c9ed0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.475+0000 7f8d83f85700 1 --2- 172.21.15.111:0/1090704815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c06d690 0x7f8d7c1574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.479+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 >> 172.21.15.111:0/1090704815 conn(0x7f8d7c0c8850 msgr2=0x7f8d7c0b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.479+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 shutdown_connections 2023-12-19T07:48:08.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.479+0000 7f8d83f85700 1 -- 172.21.15.111:0/1090704815 wait complete. 2023-12-19T07:48:08.464 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:48:08.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:08.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 Processor -- start 2023-12-19T07:48:08.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 -- start start 2023-12-19T07:48:08.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fddf0158e20 0x7fddf006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf006b730 con 0x7fddf015ca80 2023-12-19T07:48:08.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf006ab20 con 0x7fddf0158e20 2023-12-19T07:48:08.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5c03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf006ae00 con 0x7fddf01591f0 2023-12-19T07:48:08.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5402700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fddf015ca80 0x7fddf006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52608/0 (socket says 172.21.15.111:52608) 2023-12-19T07:48:08.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf5402700 1 -- 172.21.15.111:0/2042206017 learned_addr learned my addr 172.21.15.111:0/2042206017 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:08.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fddf4c01700 1 --2- 172.21.15.111:0/2042206017 >> v2:172.21.15.143:3300/0 conn(0x7fddf0158e20 0x7fddf006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.667+0000 7fdde7fff700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fdde77fe700 1 -- 172.21.15.111:0/2042206017 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204644170 0 0) 0x7fddf006b730 con 0x7fddf015ca80 2023-12-19T07:48:08.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fdde77fe700 1 -- 172.21.15.111:0/2042206017 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8003570 con 0x7fddf015ca80 2023-12-19T07:48:08.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf4c01700 1 -- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 msgr2=0x7fddf00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf4c01700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf4c01700 1 -- 172.21.15.111:0/2042206017 >> v1:172.21.15.111:6789/0 conn(0x7fddf015ca80 legacy=0x7fddf006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf4c01700 1 -- 172.21.15.111:0/2042206017 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf006b0e0 con 0x7fddf0158e20 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf4c01700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf006a540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddec0098e0 tx=0x7fddec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd236b1b0c352e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fdde77fe700 1 -- 172.21.15.111:0/2042206017 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec011630 con 0x7fddf0158e20 2023-12-19T07:48:08.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fdde77fe700 1 -- 172.21.15.111:0/2042206017 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec011790 con 0x7fddf0158e20 2023-12-19T07:48:08.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fdde77fe700 1 -- 172.21.15.111:0/2042206017 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec011b30 con 0x7fddf0158e20 2023-12-19T07:48:08.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 -- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 msgr2=0x7fddf006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf006a540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddec0098e0 tx=0x7fddec0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 -- 172.21.15.111:0/2042206017 shutdown_connections 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 --2- 172.21.15.111:0/2042206017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf006a540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 -- 172.21.15.111:0/2042206017 >> 172.21.15.111:0/2042206017 conn(0x7fddf0067bb0 msgr2=0x7fddf00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 -- 172.21.15.111:0/2042206017 shutdown_connections 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.671+0000 7fddf5c03700 1 -- 172.21.15.111:0/2042206017 wait complete. 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 Processor -- start 2023-12-19T07:48:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 -- start start 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf0144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0157630 con 0x7fddf015ca80 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0156a20 con 0x7fddf0158e20 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5c03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0156d00 con 0x7fddf01591f0 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37922/0 (socket says 172.21.15.111:37922) 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 -- 172.21.15.111:0/4145263147 learned_addr learned my addr 172.21.15.111:0/4145263147 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf4c01700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf0144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fdde7fff700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 -- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 msgr2=0x7fddf00c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 -- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 msgr2=0x7fddf0144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf0144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 -- 172.21.15.111:0/4145263147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec009420 con 0x7fddf015ca80 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf4c01700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf0144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.675+0000 7fddf5402700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001460 tx=0x7fdde8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde800a870 con 0x7fddf015ca80 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde800a9d0 con 0x7fddf015ca80 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0156fe0 con 0x7fddf015ca80 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde800ad50 con 0x7fddf015ca80 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf0156640 con 0x7fddf015ca80 2023-12-19T07:48:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf01577a0 con 0x7fddf015ca80 2023-12-19T07:48:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.679+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdde8007300 con 0x7fddf015ca80 2023-12-19T07:48:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.683+0000 7fdde4ff9700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 0x7fdddc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.683+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fdde807cef0 con 0x7fddf015ca80 2023-12-19T07:48:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.683+0000 7fddf4c01700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 0x7fdddc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.683+0000 7fddf4c01700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 0x7fdddc062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fddec0091e0 tx=0x7fddec017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.691+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdde804d120 con 0x7fddf015ca80 2023-12-19T07:48:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.859+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddf004be40 con 0x7fddf015ca80 2023-12-19T07:48:08.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.859+0000 7fdde4ff9700 1 -- 172.21.15.111:0/4145263147 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7fdde8050740 con 0x7fddf015ca80 2023-12-19T07:48:08.846 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:08.846 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 msgr2=0x7fdddc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 0x7fdddc062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fddec0091e0 tx=0x7fddec017000 comp rx=0 tx=0).stop 2023-12-19T07:48:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 msgr2=0x7fddf00c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001460 tx=0x7fdde8001a40 comp rx=0 tx=0).stop 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 shutdown_connections 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdddc060550 0x7fdddc062a00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf015ca80 0x7fddf00c7e80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddf01591f0 0x7fddf00c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 --2- 172.21.15.111:0/4145263147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddf0158e20 0x7fddf0144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 >> 172.21.15.111:0/4145263147 conn(0x7fddf0067bb0 msgr2=0x7fddf00b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:08.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 shutdown_connections 2023-12-19T07:48:08.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:08.863+0000 7fddf5c03700 1 -- 172.21.15.111:0/4145263147 wait complete. 2023-12-19T07:48:08.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:47:14.985715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:47:15.606888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-12-19T07:47:19.275016+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:48:08.871 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:48:09.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.059+0000 7f79f48cc700 1 Processor -- start 2023-12-19T07:48:09.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.059+0000 7f79f48cc700 1 -- start start 2023-12-19T07:48:09.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79f48cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 0x7f79f00c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79f48cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f0159b80 con 0x7f79f00c53f0 2023-12-19T07:48:09.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79f48cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f00c8220 con 0x7f79f00c7a80 2023-12-19T07:48:09.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79f48cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f00c7f40 con 0x7f79f0141e70 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79ef59e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f79f00c53f0 0x7f79f015c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52616/0 (socket says 172.21.15.111:52616) 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79ef59e700 1 -- 172.21.15.111:0/433326382 learned_addr learned my addr 172.21.15.111:0/433326382 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79eed9d700 1 --2- 172.21.15.111:0/433326382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 0x7f79f00c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79edd9b700 1 -- 172.21.15.111:0/433326382 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156580275 0 0) 0x7f79f00c8220 con 0x7f79f00c7a80 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79eed9d700 1 -- 172.21.15.111:0/433326382 >> v1:172.21.15.143:6789/0 conn(0x7f79f00c7a80 legacy=0x7f79f00c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79eed9d700 1 -- 172.21.15.111:0/433326382 >> v1:172.21.15.111:6789/0 conn(0x7f79f00c53f0 legacy=0x7f79f015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79eed9d700 1 -- 172.21.15.111:0/433326382 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f0159530 con 0x7f79f0141e70 2023-12-19T07:48:09.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79eed9d700 1 --2- 172.21.15.111:0/433326382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 0x7f79f00c7540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009910 tx=0x7f79e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52449b6be35e8c34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79edd9b700 1 -- 172.21.15.111:0/433326382 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000b760 con 0x7f79f0141e70 2023-12-19T07:48:09.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79edd9b700 1 -- 172.21.15.111:0/433326382 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000b8c0 con 0x7f79f0141e70 2023-12-19T07:48:09.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.063+0000 7f79edd9b700 1 -- 172.21.15.111:0/433326382 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000bc60 con 0x7f79f0141e70 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 -- 172.21.15.111:0/433326382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 msgr2=0x7f79f00c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 --2- 172.21.15.111:0/433326382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 0x7f79f00c7540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009910 tx=0x7f79e00091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 -- 172.21.15.111:0/433326382 shutdown_connections 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 --2- 172.21.15.111:0/433326382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f0141e70 0x7f79f00c7540 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 -- 172.21.15.111:0/433326382 >> 172.21.15.111:0/433326382 conn(0x7f79f0067b50 msgr2=0x7f79f00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 -- 172.21.15.111:0/433326382 shutdown_connections 2023-12-19T07:48:09.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.067+0000 7f79f48cc700 1 -- 172.21.15.111:0/433326382 wait complete. 2023-12-19T07:48:09.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 Processor -- start 2023-12-19T07:48:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 -- start start 2023-12-19T07:48:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 0x7f79f013a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79f00c7a80 0x7f79f013beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f016a630 con 0x7f79f0141e70 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f013fe30 con 0x7f79f00c7a80 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79f48cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f013f540 con 0x7f79f00c53f0 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79ef59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79ef59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37930/0 (socket says 172.21.15.111:37930) 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79ef59e700 1 -- 172.21.15.111:0/2046736854 learned_addr learned my addr 172.21.15.111:0/2046736854 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79eed9d700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 0x7f79f013a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.071+0000 7f79ee59c700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79f00c7a80 0x7f79f013beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 -- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79f00c7a80 msgr2=0x7f79f013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79f00c7a80 0x7f79f013beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 -- 172.21.15.111:0/2046736854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 msgr2=0x7f79f013c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 -- 172.21.15.111:0/2046736854 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009420 con 0x7f79f00c53f0 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79ef59e700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79eed9d700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 0x7f79f013a5e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79e0015000 tx=0x7f79e0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0011c70 con 0x7f79f00c53f0 2023-12-19T07:48:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e0011dd0 con 0x7f79f00c53f0 2023-12-19T07:48:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f016a350 con 0x7f79f00c53f0 2023-12-19T07:48:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e001c690 con 0x7f79f00c53f0 2023-12-19T07:48:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f013fac0 con 0x7f79f00c53f0 2023-12-19T07:48:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.075+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f004be40 con 0x7f79f00c53f0 2023-12-19T07:48:09.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.079+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f79e001cac0 con 0x7f79f00c53f0 2023-12-19T07:48:09.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.079+0000 7f79d6ffd700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 0x7f79dc062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.079+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f79e007ef90 con 0x7f79f00c53f0 2023-12-19T07:48:09.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.079+0000 7f79ee59c700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 0x7f79dc062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.083+0000 7f79ee59c700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 0x7f79dc062ad0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f79f01593d0 tx=0x7f79e4007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.087+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f79e004f1c0 con 0x7f79f00c53f0 2023-12-19T07:48:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.263+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79f004be40 con 0x7f79f00c53f0 2023-12-19T07:48:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.263+0000 7f79d6ffd700 1 -- 172.21.15.111:0/2046736854 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79e00527e0 con 0x7f79f00c53f0 2023-12-19T07:48:09.249 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 msgr2=0x7f79dc062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 0x7f79dc062ad0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f79f01593d0 tx=0x7f79e4007d30 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 msgr2=0x7f79f013a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 0x7f79f013a5e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79e0015000 tx=0x7f79e0016000 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 shutdown_connections 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f79dc060620 0x7f79dc062ad0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79f00c7a80 0x7f79f013beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79f0141e70 0x7f79f013c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 --2- 172.21.15.111:0/2046736854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79f00c53f0 0x7f79f013a5e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 >> 172.21.15.111:0/2046736854 conn(0x7f79f0067b50 msgr2=0x7f79f006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 shutdown_connections 2023-12-19T07:48:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.267+0000 7f79f48cc700 1 -- 172.21.15.111:0/2046736854 wait complete. 2023-12-19T07:48:09.270 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:48:09.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 Processor -- start 2023-12-19T07:48:09.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 -- start start 2023-12-19T07:48:09.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 0x7fb8600c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86006d970 con 0x7fb8600c69a0 2023-12-19T07:48:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86006cd60 con 0x7fb86006c7c0 2023-12-19T07:48:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb8649c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86006d040 con 0x7fb86006de00 2023-12-19T07:48:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb8600c69a0 0x7fb86006b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52628/0 (socket says 172.21.15.111:52628) 2023-12-19T07:48:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85ed9d700 1 -- 172.21.15.111:0/2095897916 learned_addr learned my addr 172.21.15.111:0/2095897916 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:09.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85e59c700 1 --2- 172.21.15.111:0/2095897916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 0x7fb8600c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993186894 0 0) 0x7fb86006d970 con 0x7fb8600c69a0 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003570 con 0x7fb8600c69a0 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165678245 0 0) 0x7fb86006cd60 con 0x7fb86006c7c0 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.443+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86006d970 con 0x7fb86006c7c0 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85e59c700 1 -- 172.21.15.111:0/2095897916 >> v1:172.21.15.143:6789/0 conn(0x7fb86006c7c0 legacy=0x7fb86015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85e59c700 1 -- 172.21.15.111:0/2095897916 >> v1:172.21.15.111:6789/0 conn(0x7fb8600c69a0 legacy=0x7fb86006b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85e59c700 1 -- 172.21.15.111:0/2095897916 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86006d320 con 0x7fb86006de00 2023-12-19T07:48:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85e59c700 1 --2- 172.21.15.111:0/2095897916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 0x7fb8600c6d70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb848009e90 tx=0x7fb848009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349377d728f46260 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84800e070 con 0x7fb86006de00 2023-12-19T07:48:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84800bb80 con 0x7fb86006de00 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb85dd9b700 1 -- 172.21.15.111:0/2095897916 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848012430 con 0x7fb86006de00 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 -- 172.21.15.111:0/2095897916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 msgr2=0x7fb8600c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 --2- 172.21.15.111:0/2095897916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 0x7fb8600c6d70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb848009e90 tx=0x7fb848009780 comp rx=0 tx=0).stop 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 -- 172.21.15.111:0/2095897916 shutdown_connections 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 --2- 172.21.15.111:0/2095897916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86006de00 0x7fb8600c6d70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 -- 172.21.15.111:0/2095897916 >> 172.21.15.111:0/2095897916 conn(0x7fb860067b50 msgr2=0x7fb8600cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 -- 172.21.15.111:0/2095897916 shutdown_connections 2023-12-19T07:48:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.447+0000 7fb8649c4700 1 -- 172.21.15.111:0/2095897916 wait complete. 2023-12-19T07:48:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 Processor -- start 2023-12-19T07:48:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 -- start start 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86006de00 0x7fb8600cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8600c69a0 0x7fb8600cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860170b70 con 0x7fb86006c7c0 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860171410 con 0x7fb86006de00 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb8649c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86016e330 con 0x7fb8600c69a0 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37936/0 (socket says 172.21.15.111:37936) 2023-12-19T07:48:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 -- 172.21.15.111:0/3855635410 learned_addr learned my addr 172.21.15.111:0/3855635410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85e59c700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86006de00 0x7fb8600cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85f59e700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8600c69a0 0x7fb8600cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 -- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8600c69a0 msgr2=0x7fb8600cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8600c69a0 0x7fb8600cbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 -- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86006de00 msgr2=0x7fb8600cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86006de00 0x7fb8600cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 -- 172.21.15.111:0/3855635410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848009a20 con 0x7fb86006c7c0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.451+0000 7fb85ed9d700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c7c0 tx=0x7fb850007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850005b30 con 0x7fb86006c7c0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb850005c90 con 0x7fb86006c7c0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb860170dc0 con 0x7fb86006c7c0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850007300 con 0x7fb86006c7c0 2023-12-19T07:48:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86015e800 con 0x7fb86006c7c0 2023-12-19T07:48:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86004be40 con 0x7fb86006c7c0 2023-12-19T07:48:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.455+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb85001b3b0 con 0x7fb86006c7c0 2023-12-19T07:48:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.459+0000 7fb856ffd700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 0x7fb840062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.459+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fb85007bfd0 con 0x7fb86006c7c0 2023-12-19T07:48:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.459+0000 7fb85e59c700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 0x7fb840062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.459+0000 7fb85e59c700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 0x7fb840062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb848009de0 tx=0x7fb848019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.463+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb85004fcb0 con 0x7fb86006c7c0 2023-12-19T07:48:09.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.627+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb86006d600 con 0x7fb86006c7c0 2023-12-19T07:48:09.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb856ffd700 1 -- 172.21.15.111:0/3855635410 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb85004f840 con 0x7fb86006c7c0 2023-12-19T07:48:09.614 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 msgr2=0x7fb840062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 0x7fb840062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb848009de0 tx=0x7fb848019000 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 msgr2=0x7fb86015e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb85000c7c0 tx=0x7fb850007ea0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 shutdown_connections 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb840060550 0x7fb840062a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb86006de00 0x7fb8600cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8600c69a0 0x7fb8600cbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 --2- 172.21.15.111:0/3855635410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86006c7c0 0x7fb86015e2c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 >> 172.21.15.111:0/3855635410 conn(0x7fb860067b50 msgr2=0x7fb8600c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 shutdown_connections 2023-12-19T07:48:09.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.631+0000 7fb8649c4700 1 -- 172.21.15.111:0/3855635410 wait complete. 2023-12-19T07:48:09.636 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:48:09.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.819+0000 7f173f59e700 1 Processor -- start 2023-12-19T07:48:09.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.819+0000 7f173f59e700 1 -- start start 2023-12-19T07:48:09.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1740155520 0x7f17400c7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400ad020 con 0x7f174006b530 2023-12-19T07:48:09.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400c81e0 con 0x7f17400ae6c0 2023-12-19T07:48:09.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400aca10 con 0x7f1740155520 2023-12-19T07:48:09.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1740155520 0x7f17400c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f1737fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f17400ae6c0 0x7f174006ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42318/0 (socket says 172.21.15.111:42318) 2023-12-19T07:48:09.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f1737fff700 1 -- 172.21.15.111:0/933368707 learned_addr learned my addr 172.21.15.111:0/933368707 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:09.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f174006b530 0x7f17400ad4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52634/0 (socket says 172.21.15.111:52634) 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865291882 0 0) 0x7f17400c81e0 con 0x7f17400ae6c0 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f172c003570 con 0x7f17400ae6c0 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173e59c700 1 -- 172.21.15.111:0/933368707 >> v1:172.21.15.143:6789/0 conn(0x7f17400ae6c0 legacy=0x7f174006ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173e59c700 1 -- 172.21.15.111:0/933368707 >> v1:172.21.15.111:6789/0 conn(0x7f174006b530 legacy=0x7f17400ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650952415 0 0) 0x7f17400ad020 con 0x7f174006b530 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.823+0000 7f173e59c700 1 -- 172.21.15.111:0/933368707 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400c7c20 con 0x7f1740155520 2023-12-19T07:48:09.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173e59c700 1 --2- 172.21.15.111:0/933368707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17400c7130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17300098e0 tx=0x7f17300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3d30c1ffa38a6e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173000b6d0 con 0x7f1740155520 2023-12-19T07:48:09.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173000b830 con 0x7f1740155520 2023-12-19T07:48:09.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173dd9b700 1 -- 172.21.15.111:0/933368707 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173000bbd0 con 0x7f1740155520 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 -- 172.21.15.111:0/933368707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 msgr2=0x7f17400c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 --2- 172.21.15.111:0/933368707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17400c7130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17300098e0 tx=0x7f17300091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 -- 172.21.15.111:0/933368707 shutdown_connections 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 --2- 172.21.15.111:0/933368707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17400c7130 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 -- 172.21.15.111:0/933368707 >> 172.21.15.111:0/933368707 conn(0x7f17400c8cd0 msgr2=0x7f17400c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 -- 172.21.15.111:0/933368707 shutdown_connections 2023-12-19T07:48:09.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 -- 172.21.15.111:0/933368707 wait complete. 2023-12-19T07:48:09.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.827+0000 7f173f59e700 1 Processor -- start 2023-12-19T07:48:09.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 -- start start 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f174006b530 0x7f17400ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17401558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c5d70 con 0x7f17400ae6c0 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c5160 con 0x7f1740155520 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400c5440 con 0x7f174006b530 2023-12-19T07:48:09.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37944/0 (socket says 172.21.15.111:37944) 2023-12-19T07:48:09.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 -- 172.21.15.111:0/2194850 learned_addr learned my addr 172.21.15.111:0/2194850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:09.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173e59c700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f174006b530 0x7f17400ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f173ed9d700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17401558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 -- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f174006b530 msgr2=0x7f17400ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f174006b530 0x7f17400ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 -- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 msgr2=0x7f17401558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17401558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 -- 172.21.15.111:0/2194850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009420 con 0x7f17400ae6c0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f1737fff700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f172800c790 tx=0x7f1728007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.831+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728005c00 con 0x7f17400ae6c0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1728005d60 con 0x7f17400ae6c0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17280072e0 con 0x7f17400ae6c0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17400c5720 con 0x7f17400ae6c0 2023-12-19T07:48:09.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17400ab580 con 0x7f17400ae6c0 2023-12-19T07:48:09.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17400c5f20 con 0x7f17400ae6c0 2023-12-19T07:48:09.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.835+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1728012400 con 0x7f17400ae6c0 2023-12-19T07:48:09.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.839+0000 7f17367fc700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 0x7f1720062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:09.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.839+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f172807bfa0 con 0x7f17400ae6c0 2023-12-19T07:48:09.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.839+0000 7f173e59c700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 0x7f1720062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:09.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.839+0000 7f173e59c700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 0x7f1720062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17300093f0 tx=0x7f1730017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:09.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:09.847+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f17400c5f20 con 0x7f17400ae6c0 2023-12-19T07:48:09.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.011+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f174004be40 con 0x7f17400ae6c0 2023-12-19T07:48:09.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.011+0000 7f17367fc700 1 -- 172.21.15.111:0/2194850 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1728017020 con 0x7f17400ae6c0 2023-12-19T07:48:09.996 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 msgr2=0x7f1720062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 0x7f1720062a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17300093f0 tx=0x7f1730017000 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 msgr2=0x7f17400c47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f172800c790 tx=0x7f1728007db0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 shutdown_connections 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1720060550 0x7f1720062a00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1740155520 0x7f17401558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17400ae6c0 0x7f17400c47f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 --2- 172.21.15.111:0/2194850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f174006b530 0x7f17400ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:09.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 >> 172.21.15.111:0/2194850 conn(0x7f17400c8cd0 msgr2=0x7f174006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:09.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 shutdown_connections 2023-12-19T07:48:09.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.015+0000 7f173f59e700 1 -- 172.21.15.111:0/2194850 wait complete. 2023-12-19T07:48:10.019 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-12-19T07:48:10.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.199+0000 7f82fb523700 1 Processor -- start 2023-12-19T07:48:10.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.199+0000 7f82fb523700 1 -- start start 2023-12-19T07:48:10.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fb523700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82f406aeb0 0x7f82f4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fb523700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82f41595a0 0x7f82f41599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fb523700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f406c850 con 0x7f82f4158c90 2023-12-19T07:48:10.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fb523700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40cb650 con 0x7f82f41595a0 2023-12-19T07:48:10.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fb523700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f406b470 con 0x7f82f406aeb0 2023-12-19T07:48:10.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fa521700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f82f4158c90 0x7f82f40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52644/0 (socket says 172.21.15.111:52644) 2023-12-19T07:48:10.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fa521700 1 -- 172.21.15.111:0/1873402261 learned_addr learned my addr 172.21.15.111:0/1873402261 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:10.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fad22700 1 --2- 172.21.15.111:0/1873402261 >> v2:172.21.15.111:3300/0 conn(0x7f82f41595a0 0x7f82f41599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82f9d20700 1 --2- 172.21.15.111:0/1873402261 >> v2:172.21.15.143:3300/0 conn(0x7f82f406aeb0 0x7f82f4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82f951f700 1 -- 172.21.15.111:0/1873402261 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441281000 0 0) 0x7f82f406c850 con 0x7f82f4158c90 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82f951f700 1 -- 172.21.15.111:0/1873402261 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e0003570 con 0x7f82f4158c90 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fad22700 1 -- 172.21.15.111:0/1873402261 >> v2:172.21.15.143:3300/0 conn(0x7f82f406aeb0 msgr2=0x7f82f4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fad22700 1 --2- 172.21.15.111:0/1873402261 >> v2:172.21.15.143:3300/0 conn(0x7f82f406aeb0 0x7f82f4159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fad22700 1 -- 172.21.15.111:0/1873402261 >> v1:172.21.15.111:6789/0 conn(0x7f82f4158c90 legacy=0x7f82f40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.203+0000 7f82fad22700 1 -- 172.21.15.111:0/1873402261 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f406b750 con 0x7f82f41595a0 2023-12-19T07:48:10.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fad22700 1 --2- 172.21.15.111:0/1873402261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f41595a0 0x7f82f41599c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f82f4068000 tx=0x7f82ec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3787cf868cac326 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82f951f700 1 -- 172.21.15.111:0/1873402261 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00e070 con 0x7f82f41595a0 2023-12-19T07:48:10.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82f951f700 1 -- 172.21.15.111:0/1873402261 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00bab0 con 0x7f82f41595a0 2023-12-19T07:48:10.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82f951f700 1 -- 172.21.15.111:0/1873402261 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00be70 con 0x7f82f41595a0 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 -- 172.21.15.111:0/1873402261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f41595a0 msgr2=0x7f82f41599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 --2- 172.21.15.111:0/1873402261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f41595a0 0x7f82f41599c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f82f4068000 tx=0x7f82ec009780 comp rx=0 tx=0).stop 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 -- 172.21.15.111:0/1873402261 shutdown_connections 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 --2- 172.21.15.111:0/1873402261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f41595a0 0x7f82f41599c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 --2- 172.21.15.111:0/1873402261 >> v2:172.21.15.143:3300/0 conn(0x7f82f406aeb0 0x7f82f4159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 -- 172.21.15.111:0/1873402261 >> 172.21.15.111:0/1873402261 conn(0x7f82f4067bb0 msgr2=0x7f82f40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 -- 172.21.15.111:0/1873402261 shutdown_connections 2023-12-19T07:48:10.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.207+0000 7f82fb523700 1 -- 172.21.15.111:0/1873402261 wait complete. 2023-12-19T07:48:10.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 Processor -- start 2023-12-19T07:48:10.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 -- start start 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f4158c90 0x7f82f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f41595a0 0x7f82f415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b5990 con 0x7f82f406aeb0 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f415dc30 con 0x7f82f41595a0 2023-12-19T07:48:10.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fb523700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f415c420 con 0x7f82f4158c90 2023-12-19T07:48:10.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37962/0 (socket says 172.21.15.111:37962) 2023-12-19T07:48:10.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 -- 172.21.15.111:0/3060831956 learned_addr learned my addr 172.21.15.111:0/3060831956 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:10.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82f9d20700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f4158c90 0x7f82f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fad22700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f41595a0 0x7f82f415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 -- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f4158c90 msgr2=0x7f82f415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f4158c90 0x7f82f415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 -- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f41595a0 msgr2=0x7f82f415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f41595a0 0x7f82f415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 -- 172.21.15.111:0/3060831956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009a20 con 0x7f82f406aeb0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.211+0000 7f82fa521700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82f000c7f0 tx=0x7f82f0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0005b30 con 0x7f82f406aeb0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f0005c90 con 0x7f82f406aeb0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0007340 con 0x7f82f406aeb0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f415d260 con 0x7f82f406aeb0 2023-12-19T07:48:10.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f415c930 con 0x7f82f406aeb0 2023-12-19T07:48:10.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f404be40 con 0x7f82f406aeb0 2023-12-19T07:48:10.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.215+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f82f001b400 con 0x7f82f406aeb0 2023-12-19T07:48:10.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.219+0000 7f82ea7fc700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 0x7f82d8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.219+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f82f007c7c0 con 0x7f82f406aeb0 2023-12-19T07:48:10.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.219+0000 7f82f9d20700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 0x7f82d8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.219+0000 7f82f9d20700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 0x7f82d8062a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f82f416c920 tx=0x7f82e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.223+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f82f004c9f0 con 0x7f82f406aeb0 2023-12-19T07:48:10.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.391+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f82f406ba30 con 0x7f82f406aeb0 2023-12-19T07:48:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.391+0000 7f82ea7fc700 1 -- 172.21.15.111:0/3060831956 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f82f00076f0 con 0x7f82f406aeb0 2023-12-19T07:48:10.376 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 msgr2=0x7f82d8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 0x7f82d8062a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f82f416c920 tx=0x7f82e4008040 comp rx=0 tx=0).stop 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 msgr2=0x7f82f415e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82f000c7f0 tx=0x7f82f0007e00 comp rx=0 tx=0).stop 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 shutdown_connections 2023-12-19T07:48:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d8060580 0x7f82d8062a30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f41595a0 0x7f82f415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f4158c90 0x7f82f415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 --2- 172.21.15.111:0/3060831956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82f406aeb0 0x7f82f415e190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 >> 172.21.15.111:0/3060831956 conn(0x7f82f4067bb0 msgr2=0x7f82f40ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 shutdown_connections 2023-12-19T07:48:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.395+0000 7f82fb523700 1 -- 172.21.15.111:0/3060831956 wait complete. 2023-12-19T07:48:10.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:10.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.575+0000 7f4582bc5700 1 Processor -- start 2023-12-19T07:48:10.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 -- start start 2023-12-19T07:48:10.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f457c158d40 0x7f457c067960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c068be0 con 0x7f457c0cf9c0 2023-12-19T07:48:10.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c067fd0 con 0x7f457c158d40 2023-12-19T07:48:10.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4582bc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0682b0 con 0x7f457c069070 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f457bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4580961700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f457c158d40 0x7f457c067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f457bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:53856/0 (socket says 172.21.15.111:53856) 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4580961700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f457c158d40 0x7f457c067960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:56488/0 (socket says 172.21.15.111:56488) 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f457bfff700 1 -- 172.21.15.111:0/3547222421 learned_addr learned my addr 172.21.15.111:0/3547222421 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:10.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.579+0000 7f4581162700 1 --1- 172.21.15.111:0/3547222421 >> v1:172.21.15.143:6789/0 conn(0x7f457c0cf9c0 0x7f457c0cb240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42334/0 (socket says 172.21.15.111:42334) 2023-12-19T07:48:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4580961700 1 -- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 msgr2=0x7f457c159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4580961700 1 --2- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4580961700 1 -- 172.21.15.111:0/3547222421 >> v1:172.21.15.143:6789/0 conn(0x7f457c0cf9c0 legacy=0x7f457c0cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4580961700 1 -- 172.21.15.111:0/3547222421 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c068590 con 0x7f457c158d40 2023-12-19T07:48:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4580961700 1 --2- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c158d40 0x7f457c067960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f45700098e0 tx=0x7f45700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a8d11ab5cd44826 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f457b7fe700 1 -- 172.21.15.111:0/3547222421 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457000b640 con 0x7f457c158d40 2023-12-19T07:48:10.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f457b7fe700 1 -- 172.21.15.111:0/3547222421 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457000b7a0 con 0x7f457c158d40 2023-12-19T07:48:10.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f457b7fe700 1 -- 172.21.15.111:0/3547222421 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457000bb40 con 0x7f457c158d40 2023-12-19T07:48:10.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 -- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c158d40 msgr2=0x7f457c067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 --2- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c158d40 0x7f457c067960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f45700098e0 tx=0x7f45700091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 -- 172.21.15.111:0/3547222421 shutdown_connections 2023-12-19T07:48:10.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 --2- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 --2- 172.21.15.111:0/3547222421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c158d40 0x7f457c067960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 -- 172.21.15.111:0/3547222421 >> 172.21.15.111:0/3547222421 conn(0x7f457c0ccc40 msgr2=0x7f457c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:10.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 -- 172.21.15.111:0/3547222421 shutdown_connections 2023-12-19T07:48:10.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.583+0000 7f4582bc5700 1 -- 172.21.15.111:0/3547222421 wait complete. 2023-12-19T07:48:10.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 Processor -- start 2023-12-19T07:48:10.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 -- start start 2023-12-19T07:48:10.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c0cf9c0 0x7f457c15e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0b5c20 con 0x7f457c158d40 2023-12-19T07:48:10.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c15cc70 con 0x7f457c0cf9c0 2023-12-19T07:48:10.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4582bc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0b30c0 con 0x7f457c069070 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4581162700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4581162700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37966/0 (socket says 172.21.15.111:37966) 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f4581162700 1 -- 172.21.15.111:0/2111128829 learned_addr learned my addr 172.21.15.111:0/2111128829 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.587+0000 7f457bfff700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c0cf9c0 0x7f457c15e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4580961700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 -- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 msgr2=0x7f457c15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 -- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c0cf9c0 msgr2=0x7f457c15e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c0cf9c0 0x7f457c15e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 -- 172.21.15.111:0/2111128829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570009420 con 0x7f457c158d40 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4581162700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f457400cd00 tx=0x7f4574001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400ce60 con 0x7f457c158d40 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574005df0 con 0x7f457c158d40 2023-12-19T07:48:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400ab00 con 0x7f457c158d40 2023-12-19T07:48:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c0b3310 con 0x7f457c158d40 2023-12-19T07:48:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c0b38d0 con 0x7f457c158d40 2023-12-19T07:48:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.591+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c071120 con 0x7f457c158d40 2023-12-19T07:48:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.595+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f45740070f0 con 0x7f457c158d40 2023-12-19T07:48:10.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.595+0000 7f4578ff9700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 0x7f4560062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.595+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f457407cb60 con 0x7f457c158d40 2023-12-19T07:48:10.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.599+0000 7f4580961700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 0x7f4560062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.599+0000 7f4580961700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 0x7f4560062a30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45700093f0 tx=0x7f4570017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.603+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f457404ce30 con 0x7f457c158d40 2023-12-19T07:48:10.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.807+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f457c068870 con 0x7f457c158d40 2023-12-19T07:48:10.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.807+0000 7f4578ff9700 1 -- 172.21.15.111:0/2111128829 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f4574050450 con 0x7f457c158d40 2023-12-19T07:48:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:10.793 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 msgr2=0x7f4560062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 0x7f4560062a30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45700093f0 tx=0x7f4570017000 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 msgr2=0x7f457c0b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f457400cd00 tx=0x7f4574001b10 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 shutdown_connections 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4560060580 0x7f4560062a30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c0cf9c0 0x7f457c15e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f457c158d40 0x7f457c0b1a40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 --2- 172.21.15.111:0/2111128829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c069070 0x7f457c15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 >> 172.21.15.111:0/2111128829 conn(0x7f457c0ccc40 msgr2=0x7f457c155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 shutdown_connections 2023-12-19T07:48:10.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.811+0000 7f4582bc5700 1 -- 172.21.15.111:0/2111128829 wait complete. 2023-12-19T07:48:10.798 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:48:10.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:10.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 Processor -- start 2023-12-19T07:48:10.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 -- start start 2023-12-19T07:48:10.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdd00cc330 0x7fbdd0068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 0x7fbdd0142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd00cbea0 con 0x7fbdd00cc330 2023-12-19T07:48:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd00cb030 con 0x7fbdd00cc780 2023-12-19T07:48:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.991+0000 7fbdd7278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00cb2d0 con 0x7fbdd00cea90 2023-12-19T07:48:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdd00cc330 0x7fbdd0068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdd00cc330 0x7fbdd0068a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37970/0 (socket says 172.21.15.111:37970) 2023-12-19T07:48:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 -- 172.21.15.111:0/4001965781 learned_addr learned my addr 172.21.15.111:0/4001965781 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd5815700 1 --2- 172.21.15.111:0/4001965781 >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 0x7fbdd0142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd5014700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 -- 172.21.15.111:0/4001965781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 msgr2=0x7fbdd00cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 -- 172.21.15.111:0/4001965781 >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 msgr2=0x7fbdd0142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 --2- 172.21.15.111:0/4001965781 >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 0x7fbdd0142720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 -- 172.21.15.111:0/4001965781 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0067e00 con 0x7fbdd00cc330 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd5815700 1 --2- 172.21.15.111:0/4001965781 >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 0x7fbdd0142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd4813700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc330 0x7fbdd0068a70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00cae00 tx=0x7fbdc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91ff67b5e6243e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdc7fff700 1 -- 172.21.15.111:0/4001965781 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000b640 con 0x7fbdd00cc330 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdc7fff700 1 -- 172.21.15.111:0/4001965781 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc000b7a0 con 0x7fbdd00cc330 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdc7fff700 1 -- 172.21.15.111:0/4001965781 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000bb60 con 0x7fbdd00cc330 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 -- 172.21.15.111:0/4001965781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc330 msgr2=0x7fbdd0068a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc330 0x7fbdd0068a70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00cae00 tx=0x7fbdc0009250 comp rx=0 tx=0).stop 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 -- 172.21.15.111:0/4001965781 shutdown_connections 2023-12-19T07:48:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 --2- 172.21.15.111:0/4001965781 >> v2:172.21.15.143:3300/0 conn(0x7fbdd00cc780 0x7fbdd0142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc330 0x7fbdd0068a70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 --2- 172.21.15.111:0/4001965781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 -- 172.21.15.111:0/4001965781 >> 172.21.15.111:0/4001965781 conn(0x7fbdd00cd650 msgr2=0x7fbdd006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:10.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 -- 172.21.15.111:0/4001965781 shutdown_connections 2023-12-19T07:48:10.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.995+0000 7fbdd7278700 1 -- 172.21.15.111:0/4001965781 wait complete. 2023-12-19T07:48:10.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 Processor -- start 2023-12-19T07:48:10.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 -- start start 2023-12-19T07:48:10.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 0x7fbdd00b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00b4000 con 0x7fbdd00cc780 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00c9cf0 con 0x7fbdd00cc330 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd7278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00c9fd0 con 0x7fbdd00cea90 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37986/0 (socket says 172.21.15.111:37986) 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 -- 172.21.15.111:0/2216037668 learned_addr learned my addr 172.21.15.111:0/2216037668 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd5815700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 -- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 msgr2=0x7fbdd00c62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd5014700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 0x7fbdd00b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00c62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 -- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 msgr2=0x7fbdd00b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 0x7fbdd00b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:10.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd4813700 1 -- 172.21.15.111:0/2216037668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8009a20 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd5815700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:10.999+0000 7fbdd5014700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 0x7fbdd00b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdd4813700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbdd006a0f0 tx=0x7fbdc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0000d10 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc000bb20 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0011630 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc00094f0 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd00ca330 con 0x7fbdd00cc780 2023-12-19T07:48:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd00b4470 con 0x7fbdd00cc780 2023-12-19T07:48:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.003+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbdc00119a0 con 0x7fbdd00cc780 2023-12-19T07:48:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.007+0000 7fbdc57fa700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 0x7fbdb4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.007+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fbdc007de20 con 0x7fbdd00cc780 2023-12-19T07:48:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.007+0000 7fbdd5014700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 0x7fbdb4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.007+0000 7fbdd5014700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 0x7fbdb4062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc006f00 tx=0x7fbdcc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:10.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.011+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbdc004e100 con 0x7fbdd00cc780 2023-12-19T07:48:11.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.187+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdd0071120 con 0x7fbdd00cc780 2023-12-19T07:48:11.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.187+0000 7fbdc57fa700 1 -- 172.21.15.111:0/2216037668 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7fbdc0051720 con 0x7fbdd00cc780 2023-12-19T07:48:11.173 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:11.173 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:11.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 msgr2=0x7fbdb4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 0x7fbdb4062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc006f00 tx=0x7fbdcc008040 comp rx=0 tx=0).stop 2023-12-19T07:48:11.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 msgr2=0x7fbdd00c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbdd006a0f0 tx=0x7fbdc0017000 comp rx=0 tx=0).stop 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 shutdown_connections 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdb4060550 0x7fbdb4062a00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00cea90 0x7fbdd00c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdd00cc780 0x7fbdd00c9420 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 --2- 172.21.15.111:0/2216037668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd00cc330 0x7fbdd00b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 >> 172.21.15.111:0/2216037668 conn(0x7fbdd00cd650 msgr2=0x7fbdd00c4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 shutdown_connections 2023-12-19T07:48:11.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.191+0000 7fbdd7278700 1 -- 172.21.15.111:0/2216037668 wait complete. 2023-12-19T07:48:11.195 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T07:48:11.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 Processor -- start 2023-12-19T07:48:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 -- start start 2023-12-19T07:48:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 0x7f5bbc0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc06dc20 con 0x7f5bbc0ae140 2023-12-19T07:48:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc06d940 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.371+0000 7f5bc24f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0c2200 con 0x7f5bbc0ae510 2023-12-19T07:48:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5bbc0ae140 0x7f5bbc06d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52648/0 (socket says 172.21.15.111:52648) 2023-12-19T07:48:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbbfff700 1 -- 172.21.15.111:0/580755797 learned_addr learned my addr 172.21.15.111:0/580755797 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bb3fff700 1 --2- 172.21.15.111:0/580755797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 0x7f5bbc0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825147811 0 0) 0x7f5bbc06dc20 con 0x7f5bbc0ae140 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba8003570 con 0x7f5bbc0ae140 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003890408 0 0) 0x7f5bbc06d940 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc06dc20 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bb3fff700 1 -- 172.21.15.111:0/580755797 >> v1:172.21.15.143:6789/0 conn(0x7f5bbc0c1d60 legacy=0x7f5bbc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bb3fff700 1 -- 172.21.15.111:0/580755797 >> v1:172.21.15.111:6789/0 conn(0x7f5bbc0ae140 legacy=0x7f5bbc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bb3fff700 1 -- 172.21.15.111:0/580755797 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0c6e20 con 0x7f5bbc0ae510 2023-12-19T07:48:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bb3fff700 1 --2- 172.21.15.111:0/580755797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 0x7f5bbc0a8850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009910 tx=0x7f5ba40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e588e183651145e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4011630 con 0x7f5bbc0ae510 2023-12-19T07:48:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba4011790 con 0x7f5bbc0ae510 2023-12-19T07:48:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bbb7fe700 1 -- 172.21.15.111:0/580755797 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4011b30 con 0x7f5bbc0ae510 2023-12-19T07:48:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bc24f0700 1 -- 172.21.15.111:0/580755797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 msgr2=0x7f5bbc0a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.375+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/580755797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 0x7f5bbc0a8850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009910 tx=0x7f5ba40091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- 172.21.15.111:0/580755797 shutdown_connections 2023-12-19T07:48:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/580755797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae510 0x7f5bbc0a8850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- 172.21.15.111:0/580755797 >> 172.21.15.111:0/580755797 conn(0x7f5bbc0c8850 msgr2=0x7f5bbc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- 172.21.15.111:0/580755797 shutdown_connections 2023-12-19T07:48:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- 172.21.15.111:0/580755797 wait complete. 2023-12-19T07:48:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 Processor -- start 2023-12-19T07:48:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- start start 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae140 0x7f5bbc1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bbc0ae510 0x7f5bbc0af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc156c10 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc1599c0 con 0x7f5bbc0ae510 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc24f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc159ca0 con 0x7f5bbc0ae140 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.379+0000 7f5bc0a8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37996/0 (socket says 172.21.15.111:37996) 2023-12-19T07:48:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 -- 172.21.15.111:0/3382686991 learned_addr learned my addr 172.21.15.111:0/3382686991 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bbbfff700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae140 0x7f5bbc1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bb3fff700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bbc0ae510 0x7f5bbc0af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 -- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae140 msgr2=0x7f5bbc1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae140 0x7f5bbc1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 -- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bbc0ae510 msgr2=0x7f5bbc0af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bbc0ae510 0x7f5bbc0af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 -- 172.21.15.111:0/3382686991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba4009420 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc0a8d700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c860 tx=0x7f5bb4002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4002030 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb40021b0 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4012630 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc159f80 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc15a540 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.383+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc15a760 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.387+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5bb4012a20 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.387+0000 7f5bb8ff9700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 0x7f5b9c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.387+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f5bb407ca30 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.387+0000 7f5bbbfff700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 0x7f5b9c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.391+0000 7f5bbbfff700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 0x7f5b9c062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bac007fd0 tx=0x7f5bac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.395+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5bb404cc80 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.599+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bbc071120 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.603+0000 7f5bb8ff9700 1 -- 172.21.15.111:0/3382686991 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f5bb40502a0 con 0x7f5bbc0c1d60 2023-12-19T07:48:11.587 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:11.587 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 msgr2=0x7f5b9c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 0x7f5b9c062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bac007fd0 tx=0x7f5bac009040 comp rx=0 tx=0).stop 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 msgr2=0x7f5bbc15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c860 tx=0x7f5bb4002700 comp rx=0 tx=0).stop 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 shutdown_connections 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5b9c060550 0x7f5b9c062a00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bbc0ae510 0x7f5bbc0af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bbc0c1d60 0x7f5bbc15aa60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 --2- 172.21.15.111:0/3382686991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bbc0ae140 0x7f5bbc1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 >> 172.21.15.111:0/3382686991 conn(0x7f5bbc0c8850 msgr2=0x7f5bbc06cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 shutdown_connections 2023-12-19T07:48:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.607+0000 7f5bc24f0700 1 -- 172.21.15.111:0/3382686991 wait complete. 2023-12-19T07:48:11.593 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:48:11.612 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.791+0000 7f65797c5700 1 Processor -- start 2023-12-19T07:48:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.791+0000 7f65797c5700 1 -- start start 2023-12-19T07:48:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65797c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65740ad150 0x7f657406a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65797c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 0x7f6574159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65797c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657406c9f0 con 0x7f65740ca010 2023-12-19T07:48:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65797c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65740ca860 con 0x7f65740ad150 2023-12-19T07:48:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65797c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740c8d20 con 0x7f65740ca3e0 2023-12-19T07:48:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f6572ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f65740ca010 0x7f65740c7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56032/0 (socket says 172.21.15.111:56032) 2023-12-19T07:48:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65740ad150 0x7f657406a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f6572ffd700 1 -- 172.21.15.111:0/3905618186 learned_addr learned my addr 172.21.15.111:0/3905618186 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f6573fff700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 0x7f6574159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 -- 172.21.15.111:0/3905618186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 msgr2=0x7f6574159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 0x7f6574159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 -- 172.21.15.111:0/3905618186 >> v1:172.21.15.111:6789/0 conn(0x7f65740ca010 legacy=0x7f65740c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 -- 172.21.15.111:0/3905618186 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740c8480 con 0x7f65740ad150 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f6571ffb700 1 -- 172.21.15.111:0/3905618186 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256989579 0 0) 0x7f657406c9f0 con 0x7f65740ca010 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f6573fff700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 0x7f6574159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.795+0000 7f65727fc700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f657406a3f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65680098e0 tx=0x7f65680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a1239c69d12078 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f6571ffb700 1 -- 172.21.15.111:0/3905618186 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568009e80 con 0x7f65740ad150 2023-12-19T07:48:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f6571ffb700 1 -- 172.21.15.111:0/3905618186 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800b860 con 0x7f65740ad150 2023-12-19T07:48:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f6571ffb700 1 -- 172.21.15.111:0/3905618186 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800bbe0 con 0x7f65740ad150 2023-12-19T07:48:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- 172.21.15.111:0/3905618186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 msgr2=0x7f657406a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f657406a3f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65680098e0 tx=0x7f65680091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- 172.21.15.111:0/3905618186 shutdown_connections 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca3e0 0x7f6574159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 --2- 172.21.15.111:0/3905618186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f657406a3f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- 172.21.15.111:0/3905618186 >> 172.21.15.111:0/3905618186 conn(0x7f6574068500 msgr2=0x7f65740688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- 172.21.15.111:0/3905618186 shutdown_connections 2023-12-19T07:48:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- 172.21.15.111:0/3905618186 wait complete. 2023-12-19T07:48:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 Processor -- start 2023-12-19T07:48:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.799+0000 7f65797c5700 1 -- start start 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 0x7f65740c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65740ca3e0 0x7f65740c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740b18d0 con 0x7f65740ad150 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657414f7c0 con 0x7f65740ca3e0 2023-12-19T07:48:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657414f4e0 con 0x7f65740ca010 2023-12-19T07:48:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41468/0 (socket says 172.21.15.111:41468) 2023-12-19T07:48:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 -- 172.21.15.111:0/2726615500 learned_addr learned my addr 172.21.15.111:0/2726615500 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65727fc700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 0x7f65740c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6573fff700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65740ca3e0 0x7f65740c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 -- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 msgr2=0x7f65740c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 0x7f65740c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 -- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65740ca3e0 msgr2=0x7f65740c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65740ca3e0 0x7f65740c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 -- 172.21.15.111:0/2726615500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568009420 con 0x7f65740ad150 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65727fc700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 0x7f65740c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f6572ffd700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f656400c7e0 tx=0x7f6564007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564005b30 con 0x7f65740ad150 2023-12-19T07:48:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6564005c90 con 0x7f65740ad150 2023-12-19T07:48:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564007300 con 0x7f65740ad150 2023-12-19T07:48:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65740b1280 con 0x7f65740ad150 2023-12-19T07:48:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.803+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65740ac320 con 0x7f65740ad150 2023-12-19T07:48:11.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.807+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6574071120 con 0x7f65740ad150 2023-12-19T07:48:11.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.807+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f656401b400 con 0x7f65740ad150 2023-12-19T07:48:11.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.811+0000 7f655affd700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 0x7f6560062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:11.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.811+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7f656407c6d0 con 0x7f65740ad150 2023-12-19T07:48:11.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.811+0000 7f65727fc700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 0x7f6560062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:11.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.811+0000 7f65727fc700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 0x7f6560062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65680091e0 tx=0x7f6568017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:11.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:11.815+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f656404c900 con 0x7f65740ad150 2023-12-19T07:48:12.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.019+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65740c8760 con 0x7f65740ad150 2023-12-19T07:48:12.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.019+0000 7f655affd700 1 -- 172.21.15.111:0/2726615500 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4605 (secure 0 0 0) 0x7f656404ff20 con 0x7f65740ad150 2023-12-19T07:48:12.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2761338786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2761338786},{"type":"v1","addr":"172.21.15.143:6837","nonce":2761338786}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2626457750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2626457750},{"type":"v1","addr":"172.21.15.111:6834","nonce":2626457750}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3509559454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":3509559454},{"type":"v1","addr":"172.21.15.111:6835","nonce":3509559454}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:47:16.604129+0000","modified":"2023-12-19T07:47:32.042674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/1214316551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1214316551},{"type":"v1","addr":"172.21.15.143:6835","nonce":1214316551}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 msgr2=0x7f6560062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 0x7f6560062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65680091e0 tx=0x7f6568017000 comp rx=0 tx=0).stop 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 msgr2=0x7f65740abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f656400c7e0 tx=0x7f6564007df0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 shutdown_connections 2023-12-19T07:48:12.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6560060550 0x7f6560062a00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65740ca3e0 0x7f65740c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65740ca010 0x7f65740c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 --2- 172.21.15.111:0/2726615500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740ad150 0x7f65740abde0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 >> 172.21.15.111:0/2726615500 conn(0x7f6574068500 msgr2=0x7f6574155430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 shutdown_connections 2023-12-19T07:48:12.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.023+0000 7f65797c5700 1 -- 172.21.15.111:0/2726615500 wait complete. 2023-12-19T07:48:12.010 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-12-19T07:48:12.028 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:12.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 Processor -- start 2023-12-19T07:48:12.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 -- start start 2023-12-19T07:48:12.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc968155960 0x7fc96815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96806d990 con 0x7fc96806de20 2023-12-19T07:48:12.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96815a570 con 0x7fc9680c7f00 2023-12-19T07:48:12.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc96d3cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96806cd80 con 0x7fc968155960 2023-12-19T07:48:12.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc95ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc96806de20 0x7fc9680c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56046/0 (socket says 172.21.15.111:56046) 2023-12-19T07:48:12.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc95ffff700 1 -- 172.21.15.111:0/2553693889 learned_addr learned my addr 172.21.15.111:0/2553693889 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:12.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc967fff700 1 --2- 172.21.15.111:0/2553693889 >> v2:172.21.15.143:3300/0 conn(0x7fc968155960 0x7fc96815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580254885 0 0) 0x7fc96806d990 con 0x7fc96806de20 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.203+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954003570 con 0x7fc96806de20 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc967fff700 1 -- 172.21.15.111:0/2553693889 >> v1:172.21.15.143:6789/0 conn(0x7fc9680c7f00 legacy=0x7fc9680aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc967fff700 1 -- 172.21.15.111:0/2553693889 >> v1:172.21.15.111:6789/0 conn(0x7fc96806de20 legacy=0x7fc9680c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240538046 0 0) 0x7fc96815a570 con 0x7fc9680c7f00 2023-12-19T07:48:12.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc967fff700 1 -- 172.21.15.111:0/2553693889 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96806d060 con 0x7fc968155960 2023-12-19T07:48:12.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc967fff700 1 --2- 172.21.15.111:0/2553693889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc96815a030 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9580098e0 tx=0x7fc9580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c875f7fe5017019 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800b640 con 0x7fc968155960 2023-12-19T07:48:12.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800b7a0 con 0x7fc968155960 2023-12-19T07:48:12.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc9677fe700 1 -- 172.21.15.111:0/2553693889 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800bb40 con 0x7fc968155960 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 -- 172.21.15.111:0/2553693889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 msgr2=0x7fc96815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/2553693889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc96815a030 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9580098e0 tx=0x7fc9580091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 -- 172.21.15.111:0/2553693889 shutdown_connections 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/2553693889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc96815a030 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 -- 172.21.15.111:0/2553693889 >> 172.21.15.111:0/2553693889 conn(0x7fc968067bb0 msgr2=0x7fc9680cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 -- 172.21.15.111:0/2553693889 shutdown_connections 2023-12-19T07:48:12.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.207+0000 7fc96d3cc700 1 -- 172.21.15.111:0/2553693889 wait complete. 2023-12-19T07:48:12.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 Processor -- start 2023-12-19T07:48:12.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 -- start start 2023-12-19T07:48:12.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc96806de20 0x7fc9680b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc968144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680c1c50 con 0x7fc9680c7f00 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680c1f30 con 0x7fc968155960 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96d3cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680c2210 con 0x7fc96806de20 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41480/0 (socket says 172.21.15.111:41480) 2023-12-19T07:48:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc95ffff700 1 -- 172.21.15.111:0/3554119447 learned_addr learned my addr 172.21.15.111:0/3554119447 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:12.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc967fff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc96806de20 0x7fc9680b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.211+0000 7fc96cbcb700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc968144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 -- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc96806de20 msgr2=0x7fc9680b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc96806de20 0x7fc9680b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 -- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 msgr2=0x7fc968144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc968144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 -- 172.21.15.111:0/3554119447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc958009420 con 0x7fc9680c7f00 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc96cbcb700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc968144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc95ffff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc9500011e0 tx=0x7fc950007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950005b30 con 0x7fc9680c7f00 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950005c90 con 0x7fc9680c7f00 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680c24f0 con 0x7fc9680c7f00 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950007320 con 0x7fc9680c7f00 2023-12-19T07:48:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9681587d0 con 0x7fc9680c7f00 2023-12-19T07:48:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.215+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9680c2070 con 0x7fc9680c7f00 2023-12-19T07:48:12.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.219+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc950007690 con 0x7fc9680c7f00 2023-12-19T07:48:12.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.219+0000 7fc964ff9700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 0x7fc948062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.219+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fc95007c080 con 0x7fc9680c7f00 2023-12-19T07:48:12.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.219+0000 7fc967fff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 0x7fc948062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.219+0000 7fc967fff700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 0x7fc948062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc9580093f0 tx=0x7fc958017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.223+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc9680c2070 con 0x7fc9680c7f00 2023-12-19T07:48:12.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.395+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9680c2070 con 0x7fc9680c7f00 2023-12-19T07:48:12.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.399+0000 7fc964ff9700 1 -- 172.21.15.111:0/3554119447 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+17540 (secure 0 0 0) 0x7fc9680c2070 con 0x7fc9680c7f00 2023-12-19T07:48:12.384 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:12.385 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:00.066751+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:47:14.985715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-19T07:47:15.606888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-12-19T07:47:19.275016+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:12.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.399+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 msgr2=0x7fc948062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.399+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 0x7fc948062a00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc9580093f0 tx=0x7fc958017000 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.399+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 msgr2=0x7fc9680c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.399+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc9500011e0 tx=0x7fc950007df0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 shutdown_connections 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc948060550 0x7fc948062a00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc968155960 0x7fc968144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9680c7f00 0x7fc9680c1270 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 --2- 172.21.15.111:0/3554119447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc96806de20 0x7fc9680b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 >> 172.21.15.111:0/3554119447 conn(0x7fc968067bb0 msgr2=0x7fc9680afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 shutdown_connections 2023-12-19T07:48:12.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.403+0000 7fc96d3cc700 1 -- 172.21.15.111:0/3554119447 wait complete. 2023-12-19T07:48:12.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T07:48:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 Processor -- start 2023-12-19T07:48:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 -- start start 2023-12-19T07:48:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe33015d050 0x7fe33006a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe330160e60 0x7fe3300af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33006ba50 con 0x7fe3300afd20 2023-12-19T07:48:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33006bee0 con 0x7fe33015d050 2023-12-19T07:48:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe33682e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3300af600 con 0x7fe330160e60 2023-12-19T07:48:12.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe32ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe33015d050 0x7fe33006a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe334dcb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe3300afd20 0x7fe33006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56060/0 (socket says 172.21.15.111:56060) 2023-12-19T07:48:12.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.595+0000 7fe334dcb700 1 -- 172.21.15.111:0/2190760330 learned_addr learned my addr 172.21.15.111:0/2190760330 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:12.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32f7fe700 1 --2- 172.21.15.111:0/2190760330 >> v2:172.21.15.143:3300/0 conn(0x7fe330160e60 0x7fe3300af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32ffff700 1 -- 172.21.15.111:0/2190760330 >> v2:172.21.15.143:3300/0 conn(0x7fe330160e60 msgr2=0x7fe3300af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:12.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32ffff700 1 --2- 172.21.15.111:0/2190760330 >> v2:172.21.15.143:3300/0 conn(0x7fe330160e60 0x7fe3300af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32ffff700 1 -- 172.21.15.111:0/2190760330 >> v1:172.21.15.111:6789/0 conn(0x7fe3300afd20 legacy=0x7fe33006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32ffff700 1 -- 172.21.15.111:0/2190760330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300b0140 con 0x7fe33015d050 2023-12-19T07:48:12.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32ffff700 1 --2- 172.21.15.111:0/2190760330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe33015d050 0x7fe33006a740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe33006a150 tx=0x7fe3200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1099d40c83fa03cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32effd700 1 -- 172.21.15.111:0/2190760330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320011630 con 0x7fe33015d050 2023-12-19T07:48:12.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32effd700 1 -- 172.21.15.111:0/2190760330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe320011790 con 0x7fe33015d050 2023-12-19T07:48:12.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe32effd700 1 -- 172.21.15.111:0/2190760330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320011b30 con 0x7fe33015d050 2023-12-19T07:48:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 -- 172.21.15.111:0/2190760330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe33015d050 msgr2=0x7fe33006a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 --2- 172.21.15.111:0/2190760330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe33015d050 0x7fe33006a740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe33006a150 tx=0x7fe3200091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 -- 172.21.15.111:0/2190760330 shutdown_connections 2023-12-19T07:48:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 --2- 172.21.15.111:0/2190760330 >> v2:172.21.15.143:3300/0 conn(0x7fe330160e60 0x7fe3300af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 --2- 172.21.15.111:0/2190760330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe33015d050 0x7fe33006a740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 -- 172.21.15.111:0/2190760330 >> 172.21.15.111:0/2190760330 conn(0x7fe330067b60 msgr2=0x7fe3300ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:12.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 -- 172.21.15.111:0/2190760330 shutdown_connections 2023-12-19T07:48:12.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.599+0000 7fe33682e700 1 -- 172.21.15.111:0/2190760330 wait complete. 2023-12-19T07:48:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.603+0000 7fe33682e700 1 Processor -- start 2023-12-19T07:48:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.603+0000 7fe33682e700 1 -- start start 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.603+0000 7fe33682e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.603+0000 7fe33682e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe33015d050 0x7fe3300c1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.603+0000 7fe33682e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe330160e60 0x7fe330159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300c9a20 con 0x7fe3300afd20 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300c90f0 con 0x7fe33015d050 2023-12-19T07:48:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300c93d0 con 0x7fe330160e60 2023-12-19T07:48:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41498/0 (socket says 172.21.15.111:41498) 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 -- 172.21.15.111:0/3824105134 learned_addr learned my addr 172.21.15.111:0/3824105134 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe334dcb700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe330160e60 0x7fe330159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32f7fe700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe33015d050 0x7fe3300c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 -- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe330160e60 msgr2=0x7fe330159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe330160e60 0x7fe330159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 -- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe33015d050 msgr2=0x7fe3300c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe33015d050 0x7fe3300c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 -- 172.21.15.111:0/3824105134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320009420 con 0x7fe3300afd20 2023-12-19T07:48:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe32ffff700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe330069d50 tx=0x7fe320017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320011b10 con 0x7fe3300afd20 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3200253f0 con 0x7fe3300afd20 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320025790 con 0x7fe3300afd20 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3300c96b0 con 0x7fe3300afd20 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3300c8db0 con 0x7fe3300afd20 2023-12-19T07:48:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.607+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3300c9840 con 0x7fe3300afd20 2023-12-19T07:48:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.611+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe320025b80 con 0x7fe3300afd20 2023-12-19T07:48:12.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.611+0000 7fe317fff700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 0x7fe31c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:12.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.611+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6638+0+0 (secure 0 0 0) 0x7fe32007e4c0 con 0x7fe3300afd20 2023-12-19T07:48:12.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.611+0000 7fe32f7fe700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 0x7fe31c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.615+0000 7fe32f7fe700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 0x7fe31c062a00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe3300ca920 tx=0x7fe324008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.619+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe32004e710 con 0x7fe3300afd20 2023-12-19T07:48:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.823+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe330071120 con 0x7fe3300afd20 2023-12-19T07:48:12.810 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T07:48:12.827+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T07:48:12.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.835+0000 7fe317fff700 1 -- 172.21.15.111:0/3824105134 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe320051d30 con 0x7fe3300afd20 2023-12-19T07:48:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 msgr2=0x7fe31c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 0x7fe31c062a00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe3300ca920 tx=0x7fe324008040 comp rx=0 tx=0).stop 2023-12-19T07:48:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 msgr2=0x7fe3300c1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe330069d50 tx=0x7fe320017000 comp rx=0 tx=0).stop 2023-12-19T07:48:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 shutdown_connections 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe31c060550 0x7fe31c062a00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe33015d050 0x7fe3300c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe330160e60 0x7fe330159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 --2- 172.21.15.111:0/3824105134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3300afd20 0x7fe3300c1010 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 >> 172.21.15.111:0/3824105134 conn(0x7fe330067b60 msgr2=0x7fe33015bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 shutdown_connections 2023-12-19T07:48:12.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:12.839+0000 7fe33682e700 1 -- 172.21.15.111:0/3824105134 wait complete. 2023-12-19T07:48:12.824 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T07:48:12.831 INFO:tasks.ceph.mds.b.smithi143.stderr: -3643> 2023-12-19T07:47:18.356+0000 7f4639d2a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T07:48:12.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T07:48:12.886 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T07:48:13.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.023+0000 7f5c4eee3700 1 Processor -- start 2023-12-19T07:48:13.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.023+0000 7f5c4eee3700 1 -- start start 2023-12-19T07:48:13.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4eee3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5c480c69a0 0x7f5c480c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:13.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4eee3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4806d970 con 0x7f5c4806c7c0 2023-12-19T07:48:13.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4eee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4806cd60 con 0x7f5c4806de00 2023-12-19T07:48:13.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4eee3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4806d040 con 0x7f5c480c69a0 2023-12-19T07:48:13.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4dee1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5c480c69a0 0x7f5c480c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:13.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4e6e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5c4806c7c0 0x7f5c4815cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56064/0 (socket says 172.21.15.111:56064) 2023-12-19T07:48:13.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4e6e2700 1 -- 172.21.15.111:0/3789016385 learned_addr learned my addr 172.21.15.111:0/3789016385 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:13.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4dee1700 1 -- 172.21.15.111:0/3789016385 >> v1:172.21.15.143:6789/0 conn(0x7f5c4806de00 legacy=0x7f5c4806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4dee1700 1 -- 172.21.15.111:0/3789016385 >> v1:172.21.15.111:6789/0 conn(0x7f5c4806c7c0 legacy=0x7f5c4815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4cedf700 1 -- 172.21.15.111:0/3789016385 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365003682 0 0) 0x7f5c4806d970 con 0x7f5c4806c7c0 2023-12-19T07:48:13.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4dee1700 1 -- 172.21.15.111:0/3789016385 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4806d320 con 0x7f5c480c69a0 2023-12-19T07:48:13.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4dee1700 1 --2- 172.21.15.111:0/3789016385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c480c69a0 0x7f5c480c6d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c440098e0 tx=0x7f5c440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39887df1f9e7fc44 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:13.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.027+0000 7f5c4cedf700 1 -- 172.21.15.111:0/3789016385 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44009e80 con 0x7f5c480c69a0 2023-12-19T07:48:13.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4cedf700 1 -- 172.21.15.111:0/3789016385 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4400b860 con 0x7f5c480c69a0 2023-12-19T07:48:13.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4cedf700 1 -- 172.21.15.111:0/3789016385 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4400bbe0 con 0x7f5c480c69a0 2023-12-19T07:48:13.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 -- 172.21.15.111:0/3789016385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c480c69a0 msgr2=0x7f5c480c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/3789016385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c480c69a0 0x7f5c480c6d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c440098e0 tx=0x7f5c440091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 -- 172.21.15.111:0/3789016385 shutdown_connections 2023-12-19T07:48:13.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/3789016385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c480c69a0 0x7f5c480c6d70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 -- 172.21.15.111:0/3789016385 >> 172.21.15.111:0/3789016385 conn(0x7f5c48067b50 msgr2=0x7f5c480cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:13.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 -- 172.21.15.111:0/3789016385 shutdown_connections 2023-12-19T07:48:13.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 -- 172.21.15.111:0/3789016385 wait complete. 2023-12-19T07:48:13.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.031+0000 7f5c4eee3700 1 Processor -- start 2023-12-19T07:48:13.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 -- start start 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c4806de00 0x7f5c4815e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c480c69a0 0x7f5c4815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4815d840 con 0x7f5c4806c7c0 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4815eef0 con 0x7f5c4806de00 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4eee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4815f410 con 0x7f5c480c69a0 2023-12-19T07:48:13.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:13.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41518/0 (socket says 172.21.15.111:41518) 2023-12-19T07:48:13.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 -- 172.21.15.111:0/264933127 learned_addr learned my addr 172.21.15.111:0/264933127 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:13.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4e6e2700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c480c69a0 0x7f5c4815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:13.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4d6e0700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c4806de00 0x7f5c4815e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:13.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 -- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c480c69a0 msgr2=0x7f5c4815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c480c69a0 0x7f5c4815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 -- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c4806de00 msgr2=0x7f5c4815e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c4806de00 0x7f5c4815e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 -- 172.21.15.111:0/264933127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44009420 con 0x7f5c4806c7c0 2023-12-19T07:48:13.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.035+0000 7f5c4dee1700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806a0d0 tx=0x7f5c44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:13.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44011420 con 0x7f5c4806c7c0 2023-12-19T07:48:13.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44011580 con 0x7f5c4806c7c0 2023-12-19T07:48:13.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44011920 con 0x7f5c4806c7c0 2023-12-19T07:48:13.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c480af0d0 con 0x7f5c4806c7c0 2023-12-19T07:48:13.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c480aeb00 con 0x7f5c4806c7c0 2023-12-19T07:48:13.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.039+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c480aee80 con 0x7f5c4806c7c0 2023-12-19T07:48:13.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.043+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5c44011c90 con 0x7f5c4806c7c0 2023-12-19T07:48:13.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.043+0000 7f5c3dffb700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 0x7f5c2c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:13.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.043+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6724+0+0 (secure 0 0 0) 0x7f5c4407e300 con 0x7f5c4806c7c0 2023-12-19T07:48:13.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.043+0000 7f5c4d6e0700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 0x7f5c2c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.047+0000 7f5c4d6e0700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 0x7f5c2c062a30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5c48170e30 tx=0x7f5c38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:13.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.051+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c4404e4d0 con 0x7f5c4806c7c0 2023-12-19T07:48:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.263+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c4804be40 con 0x7f5c4806c7c0 2023-12-19T07:48:13.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.839+0000 7f5c3dffb700 1 -- 172.21.15.111:0/264933127 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c44051af0 con 0x7f5c4806c7c0 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 msgr2=0x7f5c2c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 0x7f5c2c062a30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5c48170e30 tx=0x7f5c38009040 comp rx=0 tx=0).stop 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 msgr2=0x7f5c48153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806a0d0 tx=0x7f5c44017000 comp rx=0 tx=0).stop 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 shutdown_connections 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c2c060580 0x7f5c2c062a30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c4806de00 0x7f5c4815e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c480c69a0 0x7f5c4815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 --2- 172.21.15.111:0/264933127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c4806c7c0 0x7f5c48153240 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:13.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 >> 172.21.15.111:0/264933127 conn(0x7f5c48067b50 msgr2=0x7f5c480cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:13.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 shutdown_connections 2023-12-19T07:48:13.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:13.843+0000 7f5c4eee3700 1 -- 172.21.15.111:0/264933127 wait complete. 2023-12-19T07:48:13.851 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T07:48:14.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.047+0000 7f33414f0700 1 Processor -- start 2023-12-19T07:48:14.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 -- start start 2023-12-19T07:48:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 0x7f333c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f333c068a90 0x7f333c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c06d0d0 con 0x7f333c159220 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f333c06c7c0 con 0x7f333c068a90 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.051+0000 7f33414f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0c7b40 con 0x7f333c158e50 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333a7fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f333c159220 0x7f333c068380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:49920/0 (socket says 172.21.15.111:49920) 2023-12-19T07:48:14.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 0x7f333c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333a7fc700 1 -- 172.21.15.111:0/1949458319 learned_addr learned my addr 172.21.15.111:0/1949458319 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:14.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 --2- 172.21.15.111:0/1949458319 >> v2:172.21.15.143:3300/0 conn(0x7f333c068a90 0x7f333c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f3339ffb700 1 -- 172.21.15.111:0/1949458319 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498376982 0 0) 0x7f333c06d0d0 con 0x7f333c159220 2023-12-19T07:48:14.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f3339ffb700 1 -- 172.21.15.111:0/1949458319 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3324003570 con 0x7f333c159220 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 -- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 msgr2=0x7f333c0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 0x7f333c0c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 -- 172.21.15.111:0/1949458319 >> v1:172.21.15.143:6789/0 conn(0x7f333c159220 legacy=0x7f333c068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 -- 172.21.15.111:0/1949458319 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c06ca80 con 0x7f333c068a90 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333affd700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 0x7f333c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f333b7fe700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c068a90 0x7f333c06d560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e60 tx=0x7f3334009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa3082c005033e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f3339ffb700 1 -- 172.21.15.111:0/1949458319 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333400e070 con 0x7f333c068a90 2023-12-19T07:48:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f3339ffb700 1 -- 172.21.15.111:0/1949458319 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333400bab0 con 0x7f333c068a90 2023-12-19T07:48:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.055+0000 7f3339ffb700 1 -- 172.21.15.111:0/1949458319 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333400be50 con 0x7f333c068a90 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 -- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c068a90 msgr2=0x7f333c06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c068a90 0x7f333c06d560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3334009e60 tx=0x7f3334009780 comp rx=0 tx=0).stop 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 -- 172.21.15.111:0/1949458319 shutdown_connections 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c068a90 0x7f333c06d560 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 --2- 172.21.15.111:0/1949458319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c158e50 0x7f333c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 -- 172.21.15.111:0/1949458319 >> 172.21.15.111:0/1949458319 conn(0x7f333c0c8db0 msgr2=0x7f333c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 -- 172.21.15.111:0/1949458319 shutdown_connections 2023-12-19T07:48:14.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.059+0000 7f33414f0700 1 -- 172.21.15.111:0/1949458319 wait complete. 2023-12-19T07:48:14.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 Processor -- start 2023-12-19T07:48:14.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 -- start start 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 0x7f333c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 0x7f333c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0c03e0 con 0x7f333c159220 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0c05f0 con 0x7f333c158e50 2023-12-19T07:48:14.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f33414f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0c08d0 con 0x7f333c068a90 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333a7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 0x7f333c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 0x7f333c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333a7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:53748/0 (socket says 172.21.15.111:53748) 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333a7fc700 1 -- 172.21.15.111:0/155247521 learned_addr learned my addr 172.21.15.111:0/155247521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:14.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.063+0000 7f333affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 0x7f333c0c29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:50576/0 (socket says 172.21.15.111:50576) 2023-12-19T07:48:14.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 -- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 msgr2=0x7f333c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:14.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 0x7f333c0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:14.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 -- 172.21.15.111:0/155247521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 msgr2=0x7f333c156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:14.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 0x7f333c156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:14.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 -- 172.21.15.111:0/155247521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c009420 con 0x7f333c158e50 2023-12-19T07:48:14.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333b7fe700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 0x7f333c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:14.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f333a7fc700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f333000c560 tx=0x7f3330007dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:14.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330005b30 con 0x7f333c158e50 2023-12-19T07:48:14.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3330005c90 con 0x7f333c158e50 2023-12-19T07:48:14.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33300072e0 con 0x7f333c158e50 2023-12-19T07:48:14.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3334009a20 con 0x7f333c158e50 2023-12-19T07:48:14.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c0b04e0 con 0x7f333c158e50 2023-12-19T07:48:14.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.067+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333c0b0700 con 0x7f333c158e50 2023-12-19T07:48:14.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.071+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3330034a70 con 0x7f333c158e50 2023-12-19T07:48:14.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.071+0000 7f3322ffd700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 0x7f3328062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:14.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.071+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6724+0+0 (secure 0 0 0) 0x7f333007be90 con 0x7f333c158e50 2023-12-19T07:48:14.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.071+0000 7f333affd700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 0x7f3328062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:14.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.075+0000 7f333affd700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 0x7f3328062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f332c009f40 tx=0x7f332c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:14.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.079+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f333004c110 con 0x7f333c158e50 2023-12-19T07:48:14.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:14.251+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f333c071120 con 0x7f333c158e50 2023-12-19T07:48:15.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.187+0000 7f3322ffd700 1 -- 172.21.15.111:0/155247521 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3330001150 con 0x7f333c158e50 2023-12-19T07:48:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 msgr2=0x7f3328062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 0x7f3328062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f332c009f40 tx=0x7f332c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 msgr2=0x7f333c15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f333000c560 tx=0x7f3330007dd0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 shutdown_connections 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3328060550 0x7f3328062a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f333c158e50 0x7f333c15d090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f333c159220 0x7f333c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 --2- 172.21.15.111:0/155247521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f333c068a90 0x7f333c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 >> 172.21.15.111:0/155247521 conn(0x7f333c0c8db0 msgr2=0x7f333c06b400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 shutdown_connections 2023-12-19T07:48:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.191+0000 7f33414f0700 1 -- 172.21.15.111:0/155247521 wait complete. 2023-12-19T07:48:15.179 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T07:48:15.197 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T07:48:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.387+0000 7f0959a8e700 1 Processor -- start 2023-12-19T07:48:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.387+0000 7f0959a8e700 1 -- start start 2023-12-19T07:48:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0959a8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f095406ed80 0x7f095406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0959a8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095406b420 con 0x7f0954158da0 2023-12-19T07:48:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0959a8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095415d720 con 0x7f095406df10 2023-12-19T07:48:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0959a8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540ca290 con 0x7f095406ed80 2023-12-19T07:48:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0958a8c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0954158da0 0x7f095406e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56070/0 (socket says 172.21.15.111:56070) 2023-12-19T07:48:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0958a8c700 1 -- 172.21.15.111:0/3128259456 learned_addr learned my addr 172.21.15.111:0/3128259456 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0953fff700 1 --2- 172.21.15.111:0/3128259456 >> v2:172.21.15.143:3300/0 conn(0x7f095406ed80 0x7f095406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808956523 0 0) 0x7f095406b420 con 0x7f0954158da0 2023-12-19T07:48:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0944003570 con 0x7f0954158da0 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128201140 0 0) 0x7f095415d720 con 0x7f095406df10 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095406b420 con 0x7f095406df10 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020519376 0 0) 0x7f0944003570 con 0x7f0954158da0 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 >> v2:172.21.15.143:3300/0 conn(0x7f095406ed80 msgr2=0x7f095406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 --2- 172.21.15.111:0/3128259456 >> v2:172.21.15.143:3300/0 conn(0x7f095406ed80 0x7f095406a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 >> v1:172.21.15.143:6789/0 conn(0x7f095406df10 legacy=0x7f095406ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540ca4a0 con 0x7f0954158da0 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0948001470 con 0x7f0954158da0 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f0953fff700 1 --2- 172.21.15.111:0/3128259456 >> v2:172.21.15.143:3300/0 conn(0x7f095406ed80 0x7f095406a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 >> v1:172.21.15.111:6789/0 conn(0x7f0954158da0 legacy=0x7f095406e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f0944004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0944004810 0x7f0944004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0944005210 0x7f094400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540ca290 con 0x7f0944003c40 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094400c130 con 0x7f0944005210 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.391+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094400c2c0 con 0x7f0944004810 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f0944004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0953fff700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0944004810 0x7f0944004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f095928d700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0944005210 0x7f094400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 -- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0944004810 msgr2=0x7f0944004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0944004810 0x7f0944004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 -- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0944005210 msgr2=0x7f094400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0944005210 0x7f094400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 -- 172.21.15.111:0/3128259456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944003570 con 0x7f0944003c40 2023-12-19T07:48:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f095928d700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0944005210 0x7f094400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f095928d700 1 -- 172.21.15.111:0/3128259456 reap_dead start 2023-12-19T07:48:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0953fff700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0944004810 0x7f0944004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0958a8c700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f0944004030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f094800c210 tx=0x7f09540caa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be246dc99ad5a34 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800cad0 con 0x7f0944003c40 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800cc30 con 0x7f0944003c40 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f09537fe700 1 -- 172.21.15.111:0/3128259456 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948005920 con 0x7f0944003c40 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 -- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 msgr2=0x7f0944004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f0944004030 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f094800c210 tx=0x7f09540caa60 comp rx=0 tx=0).stop 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 -- 172.21.15.111:0/3128259456 shutdown_connections 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 --2- 172.21.15.111:0/3128259456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f0944004030 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 -- 172.21.15.111:0/3128259456 >> 172.21.15.111:0/3128259456 conn(0x7f09540cc760 msgr2=0x7f09540ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 -- 172.21.15.111:0/3128259456 shutdown_connections 2023-12-19T07:48:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.395+0000 7f0959a8e700 1 -- 172.21.15.111:0/3128259456 wait complete. 2023-12-19T07:48:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 Processor -- start 2023-12-19T07:48:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 -- start start 2023-12-19T07:48:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09541553a0 0x7f09540cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09540b3a70 0x7f09540b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540c8d80 con 0x7f0944003c40 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09540c98c0 con 0x7f09541553a0 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540c9020 con 0x7f09540b3a70 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41534/0 (socket says 172.21.15.111:41534) 2023-12-19T07:48:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 -- 172.21.15.111:0/1626374281 learned_addr learned my addr 172.21.15.111:0/1626374281 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:15.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f095928d700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09540b3a70 0x7f09540b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0953fff700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09541553a0 0x7f09540cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 -- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09540b3a70 msgr2=0x7f09540b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09540b3a70 0x7f09540b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 -- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09541553a0 msgr2=0x7f09540cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09541553a0 0x7f09540cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 -- 172.21.15.111:0/1626374281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540c9300 con 0x7f0944003c40 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0958a8c700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f094800c210 tx=0x7f0948015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800ce70 con 0x7f0944003c40 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0948001370 con 0x7f0944003c40 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.399+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09540c95e0 con 0x7f0944003c40 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948004a80 con 0x7f0944003c40 2023-12-19T07:48:15.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09540b4670 con 0x7f0944003c40 2023-12-19T07:48:15.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095404be40 con 0x7f0944003c40 2023-12-19T07:48:15.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0948019070 con 0x7f0944003c40 2023-12-19T07:48:15.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0950ff9700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 0x7f0938062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:15.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.403+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6286+0+0 (secure 0 0 0) 0x7f094807d8b0 con 0x7f0944003c40 2023-12-19T07:48:15.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.407+0000 7f0953fff700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 0x7f0938062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:15.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.407+0000 7f0953fff700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 0x7f0938062a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09400093f0 tx=0x7f0940008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.411+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f09480acbe0 con 0x7f0944003c40 2023-12-19T07:48:15.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:15.583+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f09540b4860 con 0x7f0944003c40 2023-12-19T07:48:16.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.203+0000 7f0950ff9700 1 -- 172.21.15.111:0/1626374281 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f094804dc60 con 0x7f0944003c40 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 msgr2=0x7f0938062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 0x7f0938062a00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09400093f0 tx=0x7f0940008040 comp rx=0 tx=0).stop 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 msgr2=0x7f09540cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f094800c210 tx=0x7f0948015000 comp rx=0 tx=0).stop 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 shutdown_connections 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0938060550 0x7f0938062a00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09541553a0 0x7f09540cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09540b3a70 0x7f09540b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 --2- 172.21.15.111:0/1626374281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0944003c40 0x7f09540cd820 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.207+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 >> 172.21.15.111:0/1626374281 conn(0x7f09540cc760 msgr2=0x7f09540b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:16.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.211+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 shutdown_connections 2023-12-19T07:48:16.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.211+0000 7f0959a8e700 1 -- 172.21.15.111:0/1626374281 wait complete. 2023-12-19T07:48:16.197 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T07:48:16.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-12-19T07:48:16.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.391+0000 7fa73b2e3700 1 Processor -- start 2023-12-19T07:48:16.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.391+0000 7fa73b2e3700 1 -- start start 2023-12-19T07:48:16.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73b2e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 0x7fa73406bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73b2e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa734159d60 0x7fa73406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73b2e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73415a1e0 con 0x7fa7340b2160 2023-12-19T07:48:16.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73b2e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7340cd3d0 con 0x7fa734159d60 2023-12-19T07:48:16.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73b2e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7341598d0 con 0x7fa73406b6f0 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa734159d60 0x7fa73406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa734159d60 0x7fa73406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41546/0 (socket says 172.21.15.111:41546) 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 -- 172.21.15.111:0/3088127119 learned_addr learned my addr 172.21.15.111:0/3088127119 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73a2e1700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 0x7fa73406bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa7392df700 1 -- 172.21.15.111:0/3088127119 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618585107 0 0) 0x7fa724001110 con 0x7fa7340b2160 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa7392df700 1 -- 172.21.15.111:0/3088127119 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa720003570 con 0x7fa7340b2160 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 -- 172.21.15.111:0/3088127119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 msgr2=0x7fa73406bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 0x7fa73406bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 -- 172.21.15.111:0/3088127119 >> v1:172.21.15.143:6789/0 conn(0x7fa7340b2160 legacy=0x7fa73406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 -- 172.21.15.111:0/3088127119 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7340b2590 con 0x7fa734159d60 2023-12-19T07:48:16.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73a2e1700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 0x7fa73406bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:16.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa73aae2700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa734159d60 0x7fa73406ac90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009f50 tx=0x7fa72c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56219ca418e0925a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:16.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa7392df700 1 -- 172.21.15.111:0/3088127119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c00e070 con 0x7fa734159d60 2023-12-19T07:48:16.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa7392df700 1 -- 172.21.15.111:0/3088127119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c00b910 con 0x7fa734159d60 2023-12-19T07:48:16.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.395+0000 7fa7392df700 1 -- 172.21.15.111:0/3088127119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0092e0 con 0x7fa734159d60 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- 172.21.15.111:0/3088127119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa734159d60 msgr2=0x7fa73406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa734159d60 0x7fa73406ac90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009f50 tx=0x7fa72c009840 comp rx=0 tx=0).stop 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- 172.21.15.111:0/3088127119 shutdown_connections 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa734159d60 0x7fa73406ac90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/3088127119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa73406b6f0 0x7fa73406bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- 172.21.15.111:0/3088127119 >> 172.21.15.111:0/3088127119 conn(0x7fa734067bb0 msgr2=0x7fa7340cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- 172.21.15.111:0/3088127119 shutdown_connections 2023-12-19T07:48:16.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- 172.21.15.111:0/3088127119 wait complete. 2023-12-19T07:48:16.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 Processor -- start 2023-12-19T07:48:16.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.399+0000 7fa73b2e3700 1 -- start start 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7340b2160 0x7fa73413f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 0x7fa73413f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa734154ad0 con 0x7fa73406b6f0 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa734154db0 con 0x7fa7340b2160 2023-12-19T07:48:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa734155090 con 0x7fa734159d60 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41552/0 (socket says 172.21.15.111:41552) 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 -- 172.21.15.111:0/2571789449 learned_addr learned my addr 172.21.15.111:0/2571789449 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73aae2700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 0x7fa73413f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa739ae0700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7340b2160 0x7fa73413f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 -- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 msgr2=0x7fa73413f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 0x7fa73413f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 -- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7340b2160 msgr2=0x7fa73413f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7340b2160 0x7fa73413f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 -- 172.21.15.111:0/2571789449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa730009420 con 0x7fa73406b6f0 2023-12-19T07:48:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73aae2700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 0x7fa73413f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73a2e1700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa73000ec80 tx=0x7fa7300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000c9d0 con 0x7fa73406b6f0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73000cb30 con 0x7fa73406b6f0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c009ae0 con 0x7fa73406b6f0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000ceb0 con 0x7fa73406b6f0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.403+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7341513b0 con 0x7fa73406b6f0 2023-12-19T07:48:16.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.407+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa734151880 con 0x7fa73406b6f0 2023-12-19T07:48:16.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.407+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa730013680 con 0x7fa73406b6f0 2023-12-19T07:48:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.407+0000 7fa72a7fc700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 0x7fa718062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.407+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5777+0+0 (secure 0 0 0) 0x7fa73007cb60 con 0x7fa73406b6f0 2023-12-19T07:48:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.407+0000 7fa739ae0700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 0x7fa718062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:16.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.411+0000 7fa739ae0700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 0x7fa718062a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa734155570 tx=0x7fa724009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:16.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.411+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa73004d100 con 0x7fa73406b6f0 2023-12-19T07:48:16.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:16.595+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa73404be40 con 0x7fa73406b6f0 2023-12-19T07:48:17.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.215+0000 7fa72a7fc700 1 -- 172.21.15.111:0/2571789449 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7fa730050720 con 0x7fa73406b6f0 2023-12-19T07:48:17.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 msgr2=0x7fa718062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 0x7fa718062a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa734155570 tx=0x7fa724009040 comp rx=0 tx=0).stop 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 msgr2=0x7fa734154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa73000ec80 tx=0x7fa7300091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 shutdown_connections 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa718060550 0x7fa718062a00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7340b2160 0x7fa73413f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa734159d60 0x7fa73413f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 --2- 172.21.15.111:0/2571789449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa73406b6f0 0x7fa734154250 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 >> 172.21.15.111:0/2571789449 conn(0x7fa734067bb0 msgr2=0x7fa73406ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:17.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 shutdown_connections 2023-12-19T07:48:17.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.223+0000 7fa73b2e3700 1 -- 172.21.15.111:0/2571789449 wait complete. 2023-12-19T07:48:17.208 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' removed 2023-12-19T07:48:17.226 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T07:48:17.226 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T07:48:17.226 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T07:48:17.226 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T07:48:17.226 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T07:48:17.227 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T07:48:17.227 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T07:48:17.227 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T07:48:17.227 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T07:48:17.227 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T07:48:17.227 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T07:48:17.227 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T07:48:17.228 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T07:48:17.228 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T07:48:17.228 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T07:48:17.228 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T07:48:17.247 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T07:48:17.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T07:48:17.249 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T07:48:17.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T07:48:17.250 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T07:48:17.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T07:48:17.252 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T07:48:17.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T07:48:17.299 INFO:tasks.ceph.mds.a:Started 2023-12-19T07:48:17.299 DEBUG:teuthology.parallel:result is None 2023-12-19T07:48:17.300 INFO:tasks.ceph.mds.b:Started 2023-12-19T07:48:17.300 DEBUG:teuthology.parallel:result is None 2023-12-19T07:48:17.305 INFO:tasks.ceph.mds.c:Started 2023-12-19T07:48:17.305 DEBUG:teuthology.parallel:result is None 2023-12-19T07:48:17.306 INFO:tasks.ceph.mds.d:Started 2023-12-19T07:48:17.306 DEBUG:teuthology.parallel:result is None 2023-12-19T07:48:17.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T07:48:17.404 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T07:48:17.413 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T07:48:17.413 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T07:48:17.418 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T07:48:17.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 Processor -- start 2023-12-19T07:48:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 -- start start 2023-12-19T07:48:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf7c163bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c06e160 con 0x7fdf7c159220 2023-12-19T07:48:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c06d0c0 con 0x7fdf7c158e50 2023-12-19T07:48:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.483+0000 7fdf830f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c06ced0 con 0x7fdf7c06ae50 2023-12-19T07:48:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdf7c159220 0x7fdf7c06a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56092/0 (socket says 172.21.15.111:56092) 2023-12-19T07:48:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7bfff700 1 -- 172.21.15.111:0/452337053 learned_addr learned my addr 172.21.15.111:0/452337053 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf7c163bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976714314 0 0) 0x7fdf7c06e160 con 0x7fdf7c159220 2023-12-19T07:48:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c003570 con 0x7fdf7c159220 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693101934 0 0) 0x7fdf7c06d0c0 con 0x7fdf7c158e50 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf7c06e160 con 0x7fdf7c158e50 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 696461213 0 0) 0x7fdf6c003570 con 0x7fdf7c159220 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 msgr2=0x7fdf7c163bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf7c163bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 >> v1:172.21.15.143:6789/0 conn(0x7fdf7c158e50 legacy=0x7fdf7c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c06d3a0 con 0x7fdf7c159220 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fdf68001470 con 0x7fdf7c159220 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 >> v1:172.21.15.111:6789/0 conn(0x7fdf7c159220 legacy=0x7fdf7c06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf6c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004180 0x7fdf6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf6c004d50 0x7fdf6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00bb70 con 0x7fdf6c004d50 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00bce0 con 0x7fdf6c004180 2023-12-19T07:48:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00bea0 con 0x7fdf7c06ae50 2023-12-19T07:48:17.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf6c004d50 0x7fdf6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf80e8c700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf6c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7bfff700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004180 0x7fdf6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 -- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 msgr2=0x7fdf6c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c06ae50 0x7fdf6c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 -- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004180 msgr2=0x7fdf6c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004180 0x7fdf6c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 -- 172.21.15.111:0/452337053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c003570 con 0x7fdf6c004d50 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf8168d700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf6c004d50 0x7fdf6c0051c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0aeb00 tx=0x7fdf7400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30682b754c5a656b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400ea60 con 0x7fdf6c004d50 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7400f040 con 0x7fdf6c004d50 2023-12-19T07:48:17.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.487+0000 7fdf7b7fe700 1 -- 172.21.15.111:0/452337053 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400ee10 con 0x7fdf6c004d50 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 -- 172.21.15.111:0/452337053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf6c004d50 msgr2=0x7fdf6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 --2- 172.21.15.111:0/452337053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf6c004d50 0x7fdf6c0051c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0aeb00 tx=0x7fdf7400b040 comp rx=0 tx=0).stop 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf8168d700 1 -- 172.21.15.111:0/452337053 reap_dead start 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 -- 172.21.15.111:0/452337053 shutdown_connections 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 -- 172.21.15.111:0/452337053 >> 172.21.15.111:0/452337053 conn(0x7fdf7c067b60 msgr2=0x7fdf7c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 -- 172.21.15.111:0/452337053 shutdown_connections 2023-12-19T07:48:17.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 -- 172.21.15.111:0/452337053 wait complete. 2023-12-19T07:48:17.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.491+0000 7fdf830f0700 1 Processor -- start 2023-12-19T07:48:17.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- start start 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 0x7fdf7c14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c155630 0x7fdf7c0c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c155aa0 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c1597a0 con 0x7fdf6c004d50 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0ca5f0 con 0x7fdf7c155630 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41602/0 (socket says 172.21.15.111:41602) 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 -- 172.21.15.111:0/4020881655 learned_addr learned my addr 172.21.15.111:0/4020881655 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf80e8c700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 0x7fdf7c14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf8168d700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c155630 0x7fdf7c0c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 -- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c155630 msgr2=0x7fdf7c0c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c155630 0x7fdf7c0c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 -- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 msgr2=0x7fdf7c14bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:17.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 0x7fdf7c14bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:17.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 -- 172.21.15.111:0/4020881655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0c8030 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf80e8c700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 0x7fdf7c14bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:17.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf7bfff700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800e860 tx=0x7fdf68001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800a520 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf68010040 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800a8d0 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c0ca860 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.495+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c16d380 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.499+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c159950 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.499+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdf6800ad00 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.499+0000 7fdf78ff9700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 0x7fdf60062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.499+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5259+0+0 (secure 0 0 0) 0x7fdf6807baa0 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.499+0000 7fdf80e8c700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 0x7fdf60062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:17.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.503+0000 7fdf80e8c700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 0x7fdf60062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf70003aa0 tx=0x7fdf70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:17.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.507+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdf7c159950 con 0x7fdf7c0c7bc0 2023-12-19T07:48:17.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:17.691+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdf7c159950 con 0x7fdf7c0c7bc0 2023-12-19T07:48:19.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf78ff9700 1 -- 172.21.15.111:0/4020881655 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdf7c159950 con 0x7fdf7c0c7bc0 2023-12-19T07:48:19.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 msgr2=0x7fdf60062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 0x7fdf60062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf70003aa0 tx=0x7fdf70009040 comp rx=0 tx=0).stop 2023-12-19T07:48:19.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 msgr2=0x7fdf7c1550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800e860 tx=0x7fdf68001010 comp rx=0 tx=0).stop 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 shutdown_connections 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf60060550 0x7fdf60062a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf7c155630 0x7fdf7c0c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf7c0c7bc0 0x7fdf7c1550f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 --2- 172.21.15.111:0/4020881655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf6c004d50 0x7fdf7c14bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 >> 172.21.15.111:0/4020881655 conn(0x7fdf7c067b60 msgr2=0x7fdf7c0ab670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 shutdown_connections 2023-12-19T07:48:19.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.083+0000 7fdf830f0700 1 -- 172.21.15.111:0/4020881655 wait complete. 2023-12-19T07:48:19.071 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T07:48:19.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T07:48:19.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.275+0000 7f4e782f0700 1 Processor -- start 2023-12-19T07:48:19.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.275+0000 7f4e782f0700 1 -- start start 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e782f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4e700c15b0 0x7f4e7015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e782f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700aeba0 0x7f4e700680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e782f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700c7190 con 0x7f4e7015d320 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e782f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700aefd0 con 0x7f4e700c15b0 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e782f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700c1b50 con 0x7f4e700aeba0 2023-12-19T07:48:19.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4e700c15b0 0x7f4e7015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4e700c15b0 0x7f4e7015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41618/0 (socket says 172.21.15.111:41618) 2023-12-19T07:48:19.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 -- 172.21.15.111:0/4088517110 learned_addr learned my addr 172.21.15.111:0/4088517110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:19.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7608c700 1 --1- 172.21.15.111:0/4088517110 >> v1:172.21.15.111:6789/0 conn(0x7f4e7015d320 0x7f4e7006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56108/0 (socket says 172.21.15.111:56108) 2023-12-19T07:48:19.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7688d700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700aeba0 0x7f4e700680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 -- 172.21.15.111:0/4088517110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700aeba0 msgr2=0x7f4e700680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700aeba0 0x7f4e700680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 -- 172.21.15.111:0/4088517110 >> v1:172.21.15.111:6789/0 conn(0x7f4e7015d320 legacy=0x7f4e7006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 -- 172.21.15.111:0/4088517110 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700c1e30 con 0x7f4e700c15b0 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7588b700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e700c15b0 0x7f4e7015d6f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e600099b0 tx=0x7f4e60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ab504021969e7d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7508a700 1 -- 172.21.15.111:0/4088517110 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60009ea0 con 0x7f4e700c15b0 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7508a700 1 -- 172.21.15.111:0/4088517110 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6000b860 con 0x7f4e700c15b0 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.279+0000 7f4e7508a700 1 -- 172.21.15.111:0/4088517110 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000bc00 con 0x7f4e700c15b0 2023-12-19T07:48:19.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- 172.21.15.111:0/4088517110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e700c15b0 msgr2=0x7f4e7015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e700c15b0 0x7f4e7015d6f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e600099b0 tx=0x7f4e60009250 comp rx=0 tx=0).stop 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- 172.21.15.111:0/4088517110 shutdown_connections 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700aeba0 0x7f4e700680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 --2- 172.21.15.111:0/4088517110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e700c15b0 0x7f4e7015d6f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- 172.21.15.111:0/4088517110 >> 172.21.15.111:0/4088517110 conn(0x7f4e700c8850 msgr2=0x7f4e700c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- 172.21.15.111:0/4088517110 shutdown_connections 2023-12-19T07:48:19.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- 172.21.15.111:0/4088517110 wait complete. 2023-12-19T07:48:19.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 Processor -- start 2023-12-19T07:48:19.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.283+0000 7f4e782f0700 1 -- start start 2023-12-19T07:48:19.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 0x7f4e70162450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700c15b0 0x7f4e7016d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7016e7b0 con 0x7f4e7015d320 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7016dd00 con 0x7f4e700aeba0 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7016df10 con 0x7f4e700c15b0 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41628/0 (socket says 172.21.15.111:41628) 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 -- 172.21.15.111:0/2007124422 learned_addr learned my addr 172.21.15.111:0/2007124422 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7608c700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 0x7f4e70162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7588b700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700c15b0 0x7f4e7016d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 -- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700c15b0 msgr2=0x7f4e7016d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700c15b0 0x7f4e7016d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 -- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 msgr2=0x7f4e70162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 0x7f4e70162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 -- 172.21.15.111:0/2007124422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600094f0 con 0x7f4e7015d320 2023-12-19T07:48:19.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7608c700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 0x7f4e70162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e7688d700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009df0 tx=0x7f4e6800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800cb50 con 0x7f4e7015d320 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6800f040 con 0x7f4e7015d320 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7016e1f0 con 0x7f4e7015d320 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68014480 con 0x7f4e7015d320 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.287+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7015fbc0 con 0x7f4e7015d320 2023-12-19T07:48:19.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.291+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e70071120 con 0x7f4e7015d320 2023-12-19T07:48:19.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.291+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4e68014830 con 0x7f4e7015d320 2023-12-19T07:48:19.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.291+0000 7f4e667fc700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 0x7f4e54062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.291+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f4e6807ced0 con 0x7f4e7015d320 2023-12-19T07:48:19.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.291+0000 7f4e7608c700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 0x7f4e54062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.295+0000 7f4e7608c700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 0x7f4e54062a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c007fd0 tx=0x7f4e6c007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.295+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e6804d6d0 con 0x7f4e7015d320 2023-12-19T07:48:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.531+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4e700642a0 con 0x7f4e7015d320 2023-12-19T07:48:19.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.535+0000 7f4e667fc700 1 -- 172.21.15.111:0/2007124422 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f4e68018020 con 0x7f4e7015d320 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.518 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T07:48:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.523 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.525 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T07:48:19.526 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.527 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T07:48:19.528 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-mds", 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFlaCrBDRAA2Sfsqyq0xnAxCZZPXCdJdw==", 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-mds" 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-mgr", 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFlvTXBDRAAr9/nyrNWutYT4S3aFL3HkA==", 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-mgr" 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-osd", 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFlEkDBDRAAlyteVXNRZasvOUMwzX10hw==", 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-osd" 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rbd", 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFlqUzBDRAATfbVvSN5sdgk5NSsSWFm2g==", 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rbd" 2023-12-19T07:48:19.531 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFliFbBDRAAEjo466sJ985o5ovkcsNBdQ==", 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rgw", 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBtSoFlBGHBDRAAchiRI/6HKgCsSO2chNX/JQ==", 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rgw" 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T07:48:19.533 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T07:48:19.534 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.534 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:19.534 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T07:48:19.534 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 msgr2=0x7f4e54062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 0x7f4e54062a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c007fd0 tx=0x7f4e6c007e30 comp rx=0 tx=0).stop 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 msgr2=0x7f4e70153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009df0 tx=0x7f4e6800b040 comp rx=0 tx=0).stop 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 shutdown_connections 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e54060550 0x7f4e54062a00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e7015d320 0x7f4e70153f50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e700c15b0 0x7f4e7016d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 --2- 172.21.15.111:0/2007124422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e700aeba0 0x7f4e70162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 >> 172.21.15.111:0/2007124422 conn(0x7f4e700c8850 msgr2=0x7f4e70163730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:19.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 shutdown_connections 2023-12-19T07:48:19.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.539+0000 7f4e782f0700 1 -- 172.21.15.111:0/2007124422 wait complete. 2023-12-19T07:48:19.542 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-12-19T07:48:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.711+0000 7f9d2d84c700 1 Processor -- start 2023-12-19T07:48:19.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.711+0000 7f9d2d84c700 1 -- start start 2023-12-19T07:48:19.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 0x7f9d2815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 0x7f9d280c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d280c51b0 0x7f9d2806a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d280c5610 con 0x7f9d28158d60 2023-12-19T07:48:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d2806c7c0 con 0x7f9d280c51b0 2023-12-19T07:48:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2806caa0 con 0x7f9d2806b7a0 2023-12-19T07:48:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2c84a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 0x7f9d2815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 0x7f9d280c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2c84a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 0x7f9d2815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41636/0 (socket says 172.21.15.111:41636) 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2c84a700 1 -- 172.21.15.111:0/967940769 learned_addr learned my addr 172.21.15.111:0/967940769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d2d04b700 1 --2- 172.21.15.111:0/967940769 >> v2:172.21.15.143:3300/0 conn(0x7f9d280c51b0 0x7f9d2806a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 -- 172.21.15.111:0/967940769 >> v2:172.21.15.143:3300/0 conn(0x7f9d280c51b0 msgr2=0x7f9d2806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 --2- 172.21.15.111:0/967940769 >> v2:172.21.15.143:3300/0 conn(0x7f9d280c51b0 0x7f9d2806a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 -- 172.21.15.111:0/967940769 >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 msgr2=0x7f9d2815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 --2- 172.21.15.111:0/967940769 >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 0x7f9d2815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 -- 172.21.15.111:0/967940769 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2815ce60 con 0x7f9d2806b7a0 2023-12-19T07:48:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d27fff700 1 --2- 172.21.15.111:0/967940769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 0x7f9d280c4c70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9d280c5b50 tx=0x7f9d140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d05e68068a9c8a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d277fe700 1 -- 172.21.15.111:0/967940769 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14011630 con 0x7f9d2806b7a0 2023-12-19T07:48:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d277fe700 1 -- 172.21.15.111:0/967940769 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d140117b0 con 0x7f9d2806b7a0 2023-12-19T07:48:19.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.715+0000 7f9d277fe700 1 -- 172.21.15.111:0/967940769 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14011b70 con 0x7f9d2806b7a0 2023-12-19T07:48:19.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- 172.21.15.111:0/967940769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 msgr2=0x7f9d280c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/967940769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 0x7f9d280c4c70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9d280c5b50 tx=0x7f9d140091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- 172.21.15.111:0/967940769 shutdown_connections 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/967940769 >> v2:172.21.15.143:3300/0 conn(0x7f9d280c51b0 0x7f9d2806a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/967940769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d2806b7a0 0x7f9d280c4c70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/967940769 >> v2:172.21.15.111:3300/0 conn(0x7f9d28158d60 0x7f9d2815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- 172.21.15.111:0/967940769 >> 172.21.15.111:0/967940769 conn(0x7f9d28067bb0 msgr2=0x7f9d280cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- 172.21.15.111:0/967940769 shutdown_connections 2023-12-19T07:48:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- 172.21.15.111:0/967940769 wait complete. 2023-12-19T07:48:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 Processor -- start 2023-12-19T07:48:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.719+0000 7f9d2d84c700 1 -- start start 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 0x7f9d28150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 0x7f9d280ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2806d5d0 con 0x7f9d2806b7a0 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28149940 con 0x7f9d280c51b0 2023-12-19T07:48:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28157ab0 con 0x7f9d28158d60 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41638/0 (socket says 172.21.15.111:41638) 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 -- 172.21.15.111:0/626793353 learned_addr learned my addr 172.21.15.111:0/626793353 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d27fff700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 0x7f9d28150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d04b700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 0x7f9d280ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 -- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 msgr2=0x7f9d280ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 0x7f9d280ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 -- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 msgr2=0x7f9d28150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 0x7f9d28150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 -- 172.21.15.111:0/626793353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14009420 con 0x7f9d2806b7a0 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d27fff700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 0x7f9d28150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d04b700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 0x7f9d280ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2c84a700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00e980 tx=0x7f9d1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c00c850 con 0x7f9d2806b7a0 2023-12-19T07:48:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c00c9b0 con 0x7f9d2806b7a0 2023-12-19T07:48:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c00cd30 con 0x7f9d2806b7a0 2023-12-19T07:48:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d28157d90 con 0x7f9d2806b7a0 2023-12-19T07:48:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d28158350 con 0x7f9d2806b7a0 2023-12-19T07:48:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.723+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2804be40 con 0x7f9d2806b7a0 2023-12-19T07:48:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.727+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9d1c013680 con 0x7f9d2806b7a0 2023-12-19T07:48:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.727+0000 7f9d24ff9700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 0x7f9d0c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.727+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f9d1c07c800 con 0x7f9d2806b7a0 2023-12-19T07:48:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.727+0000 7f9d27fff700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 0x7f9d0c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.731+0000 7f9d27fff700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 0x7f9d0c062a30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d280c5b50 tx=0x7f9d14017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.735+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d1c04cf60 con 0x7f9d2806b7a0 2023-12-19T07:48:19.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.959+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f9d280cc1d0 con 0x7f9d2806b7a0 2023-12-19T07:48:19.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.967+0000 7f9d24ff9700 1 -- 172.21.15.111:0/626793353 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9d1c050580 con 0x7f9d2806b7a0 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 msgr2=0x7f9d0c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 0x7f9d0c062a30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d280c5b50 tx=0x7f9d14017000 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 msgr2=0x7f9d28149400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00e980 tx=0x7f9d1c009250 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 shutdown_connections 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d0c060580 0x7f9d0c062a30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d280c51b0 0x7f9d28150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d28158d60 0x7f9d280ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 --2- 172.21.15.111:0/626793353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2806b7a0 0x7f9d28149400 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 >> 172.21.15.111:0/626793353 conn(0x7f9d28067bb0 msgr2=0x7f9d280cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 shutdown_connections 2023-12-19T07:48:19.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:19.971+0000 7f9d2d84c700 1 -- 172.21.15.111:0/626793353 wait complete. 2023-12-19T07:48:19.974 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-12-19T07:48:20.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.147+0000 7f4eb9f63700 1 Processor -- start 2023-12-19T07:48:20.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.147+0000 7f4eb9f63700 1 -- start start 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4eb4158e20 0x7f4eb406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4eb41591f0 0x7f4eb40ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb406b730 con 0x7f4eb415ca80 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb406ab20 con 0x7f4eb4158e20 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb406ae00 con 0x7f4eb41591f0 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4eb4158e20 0x7f4eb406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4eb4158e20 0x7f4eb406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41654/0 (socket says 172.21.15.111:41654) 2023-12-19T07:48:20.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 -- 172.21.15.111:0/1366385796 learned_addr learned my addr 172.21.15.111:0/1366385796 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:20.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb3fff700 1 --2- 172.21.15.111:0/1366385796 >> v2:172.21.15.143:3300/0 conn(0x7f4eb41591f0 0x7f4eb40ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 -- 172.21.15.111:0/1366385796 >> v2:172.21.15.143:3300/0 conn(0x7f4eb41591f0 msgr2=0x7f4eb40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 --2- 172.21.15.111:0/1366385796 >> v2:172.21.15.143:3300/0 conn(0x7f4eb41591f0 0x7f4eb40ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 -- 172.21.15.111:0/1366385796 >> v1:172.21.15.143:6789/0 conn(0x7f4eb415ca80 legacy=0x7f4eb406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 -- 172.21.15.111:0/1366385796 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb406b0e0 con 0x7f4eb4158e20 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb8f61700 1 --2- 172.21.15.111:0/1366385796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb4158e20 0x7f4eb406a540 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40642c0 tx=0x7f4ea8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b64f17592bd8bdd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb37fe700 1 -- 172.21.15.111:0/1366385796 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea800b860 con 0x7f4eb4158e20 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb37fe700 1 -- 172.21.15.111:0/1366385796 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea800b9c0 con 0x7f4eb4158e20 2023-12-19T07:48:20.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb37fe700 1 -- 172.21.15.111:0/1366385796 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea800bd60 con 0x7f4eb4158e20 2023-12-19T07:48:20.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- 172.21.15.111:0/1366385796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb4158e20 msgr2=0x7f4eb406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/1366385796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb4158e20 0x7f4eb406a540 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40642c0 tx=0x7f4ea8009250 comp rx=0 tx=0).stop 2023-12-19T07:48:20.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- 172.21.15.111:0/1366385796 shutdown_connections 2023-12-19T07:48:20.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/1366385796 >> v2:172.21.15.143:3300/0 conn(0x7f4eb41591f0 0x7f4eb40ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/1366385796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb4158e20 0x7f4eb406a540 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- 172.21.15.111:0/1366385796 >> 172.21.15.111:0/1366385796 conn(0x7f4eb4067bb0 msgr2=0x7f4eb40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:20.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- 172.21.15.111:0/1366385796 shutdown_connections 2023-12-19T07:48:20.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.151+0000 7f4eb9f63700 1 -- 172.21.15.111:0/1366385796 wait complete. 2023-12-19T07:48:20.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 Processor -- start 2023-12-19T07:48:20.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 -- start start 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb4158e20 0x7f4eb40c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb41591f0 0x7f4eb40c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb40c8b20 con 0x7f4eb415ca80 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb416a660 con 0x7f4eb41591f0 2023-12-19T07:48:20.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9f63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb416a900 con 0x7f4eb4158e20 2023-12-19T07:48:20.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.155+0000 7f4eb9762700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb8f61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb4158e20 0x7f4eb40c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb41591f0 0x7f4eb40c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41656/0 (socket says 172.21.15.111:41656) 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 -- 172.21.15.111:0/2829606925 learned_addr learned my addr 172.21.15.111:0/2829606925 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 -- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb4158e20 msgr2=0x7f4eb40c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb4158e20 0x7f4eb40c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 -- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb41591f0 msgr2=0x7f4eb40c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb41591f0 0x7f4eb40c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 -- 172.21.15.111:0/2829606925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea80094f0 con 0x7f4eb415ca80 2023-12-19T07:48:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9762700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00c840 tx=0x7f4eac001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac012630 con 0x7f4eb415ca80 2023-12-19T07:48:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac0127b0 con 0x7f4eb415ca80 2023-12-19T07:48:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac012b70 con 0x7f4eb415ca80 2023-12-19T07:48:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb416abe0 con 0x7f4eb415ca80 2023-12-19T07:48:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb416b1a0 con 0x7f4eb415ca80 2023-12-19T07:48:20.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.159+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb416b3c0 con 0x7f4eb415ca80 2023-12-19T07:48:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.163+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4eac01a3a0 con 0x7f4eb415ca80 2023-12-19T07:48:20.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.163+0000 7f4eb0ff9700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 0x7f4e98062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.163+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f4eac07c6e0 con 0x7f4eb415ca80 2023-12-19T07:48:20.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.163+0000 7f4eb8f61700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 0x7f4e98062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.167+0000 7f4eb8f61700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 0x7f4e98062a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40642c0 tx=0x7f4ea8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.171+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4eac04cee0 con 0x7f4eb415ca80 2023-12-19T07:48:20.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.399+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f4eb404be40 con 0x7f4eb415ca80 2023-12-19T07:48:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.407+0000 7f4eb0ff9700 1 -- 172.21.15.111:0/2829606925 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4eac050500 con 0x7f4eb415ca80 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 msgr2=0x7f4e98062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 0x7f4e98062a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40642c0 tx=0x7f4ea8017000 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 msgr2=0x7f4eb4169360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00c840 tx=0x7f4eac001670 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 shutdown_connections 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e98060550 0x7f4e98062a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb41591f0 0x7f4eb40c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eb415ca80 0x7f4eb4169360 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 --2- 172.21.15.111:0/2829606925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb4158e20 0x7f4eb40c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 >> 172.21.15.111:0/2829606925 conn(0x7f4eb4067bb0 msgr2=0x7f4eb40b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:20.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 shutdown_connections 2023-12-19T07:48:20.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.411+0000 7f4eb9f63700 1 -- 172.21.15.111:0/2829606925 wait complete. 2023-12-19T07:48:20.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-12-19T07:48:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.599+0000 7f1a21b45700 1 Processor -- start 2023-12-19T07:48:20.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.599+0000 7f1a21b45700 1 -- start start 2023-12-19T07:48:20.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a21b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a21b45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 0x7f1a1c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a21b45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c06d7d0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a21b45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1c0ae280 con 0x7f1a1c06c7c0 2023-12-19T07:48:20.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a21b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0cb8f0 con 0x7f1a1c140420 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1a1c1407f0 0x7f1a1c06a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56122/0 (socket says 172.21.15.111:56122) 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1bfff700 1 -- 172.21.15.111:0/2304986887 learned_addr learned my addr 172.21.15.111:0/2304986887 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/2304986887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1affd700 1 --2- 172.21.15.111:0/2304986887 >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 0x7f1a1c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143446692 0 0) 0x7f1a1c06d7d0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04003570 con 0x7f1a1c1407f0 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1b7fe700 1 -- 172.21.15.111:0/2304986887 >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 msgr2=0x7f1a1c0b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/2304986887 >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 0x7f1a1c0b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1b7fe700 1 -- 172.21.15.111:0/2304986887 >> v1:172.21.15.111:6789/0 conn(0x7f1a1c1407f0 legacy=0x7f1a1c06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022994325 0 0) 0x7f1a04003570 con 0x7f1a1c1407f0 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1b7fe700 1 -- 172.21.15.111:0/2304986887 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c06cc50 con 0x7f1a1c140420 2023-12-19T07:48:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.603+0000 7f1a1affd700 1 --2- 172.21.15.111:0/2304986887 >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 0x7f1a1c0b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:20.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/2304986887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c15b220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0098e0 tx=0x7f1a0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0302bb8d4881f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a140017f0 con 0x7f1a1c140420 2023-12-19T07:48:20.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c011420 con 0x7f1a1c140420 2023-12-19T07:48:20.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a1a7fc700 1 -- 172.21.15.111:0/2304986887 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0117c0 con 0x7f1a1c140420 2023-12-19T07:48:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 -- 172.21.15.111:0/2304986887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 msgr2=0x7f1a1c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 --2- 172.21.15.111:0/2304986887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c15b220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0098e0 tx=0x7f1a0c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 -- 172.21.15.111:0/2304986887 shutdown_connections 2023-12-19T07:48:20.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 --2- 172.21.15.111:0/2304986887 >> v2:172.21.15.143:3300/0 conn(0x7f1a1c06c7c0 0x7f1a1c0b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 --2- 172.21.15.111:0/2304986887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c15b220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 -- 172.21.15.111:0/2304986887 >> 172.21.15.111:0/2304986887 conn(0x7f1a1c067b60 msgr2=0x7f1a1c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:20.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 -- 172.21.15.111:0/2304986887 shutdown_connections 2023-12-19T07:48:20.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.607+0000 7f1a21b45700 1 -- 172.21.15.111:0/2304986887 wait complete. 2023-12-19T07:48:20.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 Processor -- start 2023-12-19T07:48:20.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 -- start start 2023-12-19T07:48:20.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a1c06c7c0 0x7f1a1c0c0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0c3270 con 0x7f1a1c1407f0 2023-12-19T07:48:20.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c16a180 con 0x7f1a1c06c7c0 2023-12-19T07:48:20.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a21b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c0c2940 con 0x7f1a1c140420 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41670/0 (socket says 172.21.15.111:41670) 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 -- 172.21.15.111:0/300960562 learned_addr learned my addr 172.21.15.111:0/300960562 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1affd700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a1c06c7c0 0x7f1a1c0c0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 -- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 msgr2=0x7f1a1c0c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c0c3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 -- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a1c06c7c0 msgr2=0x7f1a1c0c0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a1c06c7c0 0x7f1a1c0c0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 -- 172.21.15.111:0/300960562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10009420 con 0x7f1a1c1407f0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.611+0000 7f1a1bfff700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400cbe0 tx=0x7f1a14001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14004dc0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1400a6b0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c009420 con 0x7f1a1c1407f0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400aa70 con 0x7f1a1c1407f0 2023-12-19T07:48:20.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c0c2f00 con 0x7f1a1c1407f0 2023-12-19T07:48:20.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c0b0ad0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.615+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1a14007150 con 0x7f1a1c1407f0 2023-12-19T07:48:20.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.619+0000 7f1a037fe700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 0x7f1a08062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:20.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.619+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f1a1407bd30 con 0x7f1a1c1407f0 2023-12-19T07:48:20.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.619+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 0x7f1a08062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:20.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.619+0000 7f1a1b7fe700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 0x7f1a08062a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0030c0 tx=0x7f1a0c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:20.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.623+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a1c0b0ad0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.847+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f1a1c0b0ad0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a037fe700 1 -- 172.21.15.111:0/300960562 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1a1c0b0ad0 con 0x7f1a1c1407f0 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 msgr2=0x7f1a08062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 0x7f1a08062a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0030c0 tx=0x7f1a0c016000 comp rx=0 tx=0).stop 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 msgr2=0x7f1a1c157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400cbe0 tx=0x7f1a14001640 comp rx=0 tx=0).stop 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 shutdown_connections 2023-12-19T07:48:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a08060550 0x7f1a08062a00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a1c1407f0 0x7f1a1c157620 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a1c140420 0x7f1a1c0c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 --2- 172.21.15.111:0/300960562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a1c06c7c0 0x7f1a1c0c0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.859+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 >> 172.21.15.111:0/300960562 conn(0x7f1a1c067b60 msgr2=0x7f1a1c15d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.863+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 shutdown_connections 2023-12-19T07:48:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:20.863+0000 7f1a21b45700 1 -- 172.21.15.111:0/300960562 wait complete. 2023-12-19T07:48:20.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-12-19T07:48:21.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.043+0000 7f05b6fc3700 1 Processor -- start 2023-12-19T07:48:21.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.043+0000 7f05b6fc3700 1 -- start start 2023-12-19T07:48:21.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b6fc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05b01591f0 0x7f05b006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b6fc3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05b00ce810 0x7f05b00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b6fc3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b006b730 con 0x7f05b0158e20 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b6fc3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b006ab20 con 0x7f05b00ce810 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b6fc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b006ae00 con 0x7f05b01591f0 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b5fc1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f05b0158e20 0x7f05b00ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56126/0 (socket says 172.21.15.111:56126) 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b5fc1700 1 -- 172.21.15.111:0/557517178 learned_addr learned my addr 172.21.15.111:0/557517178 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:21.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 --2- 172.21.15.111:0/557517178 >> v2:172.21.15.111:3300/0 conn(0x7f05b00ce810 0x7f05b00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b57c0700 1 --2- 172.21.15.111:0/557517178 >> v2:172.21.15.143:3300/0 conn(0x7f05b01591f0 0x7f05b006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 -- 172.21.15.111:0/557517178 >> v2:172.21.15.143:3300/0 conn(0x7f05b01591f0 msgr2=0x7f05b006a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 --2- 172.21.15.111:0/557517178 >> v2:172.21.15.143:3300/0 conn(0x7f05b01591f0 0x7f05b006a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 -- 172.21.15.111:0/557517178 >> v1:172.21.15.111:6789/0 conn(0x7f05b0158e20 legacy=0x7f05b00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 -- 172.21.15.111:0/557517178 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b006b0e0 con 0x7f05b00ce810 2023-12-19T07:48:21.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b67c2700 1 --2- 172.21.15.111:0/557517178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b00ca950 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f05b00642c0 tx=0x7f05a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a884c8f4a939ee39 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b4fbf700 1 -- 172.21.15.111:0/557517178 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800e070 con 0x7f05b00ce810 2023-12-19T07:48:21.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b4fbf700 1 -- 172.21.15.111:0/557517178 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a800bb60 con 0x7f05b00ce810 2023-12-19T07:48:21.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.047+0000 7f05b4fbf700 1 -- 172.21.15.111:0/557517178 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8012430 con 0x7f05b00ce810 2023-12-19T07:48:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 -- 172.21.15.111:0/557517178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 msgr2=0x7f05b00ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/557517178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b00ca950 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f05b00642c0 tx=0x7f05a8009780 comp rx=0 tx=0).stop 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 -- 172.21.15.111:0/557517178 shutdown_connections 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/557517178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b00ca950 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/557517178 >> v2:172.21.15.143:3300/0 conn(0x7f05b01591f0 0x7f05b006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 -- 172.21.15.111:0/557517178 >> 172.21.15.111:0/557517178 conn(0x7f05b0067bb0 msgr2=0x7f05b00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 -- 172.21.15.111:0/557517178 shutdown_connections 2023-12-19T07:48:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.051+0000 7f05b6fc3700 1 -- 172.21.15.111:0/557517178 wait complete. 2023-12-19T07:48:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 Processor -- start 2023-12-19T07:48:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 -- start start 2023-12-19T07:48:21.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 0x7f05b0159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05b01591f0 0x7f05b0152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b016a350 con 0x7f05b00ce810 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b015c060 con 0x7f05b0158e20 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0154780 con 0x7f05b01591f0 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41682/0 (socket says 172.21.15.111:41682) 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 -- 172.21.15.111:0/2577241026 learned_addr learned my addr 172.21.15.111:0/2577241026 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b57c0700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 0x7f05b0159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b67c2700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05b01591f0 0x7f05b0152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 -- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05b01591f0 msgr2=0x7f05b0152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05b01591f0 0x7f05b0152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 -- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 msgr2=0x7f05b0159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 0x7f05b0159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 -- 172.21.15.111:0/2577241026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8009a20 con 0x7f05b00ce810 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b57c0700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 0x7f05b0159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b5fc1700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00c7c0 tx=0x7f05ac007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac005b30 con 0x7f05b00ce810 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac005cb0 con 0x7f05b00ce810 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b0154a60 con 0x7f05b00ce810 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.055+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac004a80 con 0x7f05b00ce810 2023-12-19T07:48:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.059+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b0154d40 con 0x7f05b00ce810 2023-12-19T07:48:21.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.059+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b004be40 con 0x7f05b00ce810 2023-12-19T07:48:21.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.059+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f05ac0070f0 con 0x7f05b00ce810 2023-12-19T07:48:21.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.063+0000 7f05a5ffb700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 0x7f0594062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.063+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f05ac07ba40 con 0x7f05b00ce810 2023-12-19T07:48:21.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.063+0000 7f05b57c0700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 0x7f0594062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.063+0000 7f05b57c0700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 0x7f0594062a00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05b01598a0 tx=0x7f05a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.067+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f05ac07de60 con 0x7f05b00ce810 2023-12-19T07:48:21.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.299+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f05b006b3c0 con 0x7f05b00ce810 2023-12-19T07:48:21.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.307+0000 7f05a5ffb700 1 -- 172.21.15.111:0/2577241026 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f05ac018020 con 0x7f05b00ce810 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 msgr2=0x7f0594062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 0x7f0594062a00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05b01598a0 tx=0x7f05a0008040 comp rx=0 tx=0).stop 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 msgr2=0x7f05b015ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00c7c0 tx=0x7f05ac007f70 comp rx=0 tx=0).stop 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 shutdown_connections 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0594060550 0x7f0594062a00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05b0158e20 0x7f05b0159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05b01591f0 0x7f05b0152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 --2- 172.21.15.111:0/2577241026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05b00ce810 0x7f05b015ba60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 >> 172.21.15.111:0/2577241026 conn(0x7f05b0067bb0 msgr2=0x7f05b0153ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:21.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 shutdown_connections 2023-12-19T07:48:21.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.311+0000 7f05b6fc3700 1 -- 172.21.15.111:0/2577241026 wait complete. 2023-12-19T07:48:21.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-12-19T07:48:21.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 Processor -- start 2023-12-19T07:48:21.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 -- start start 2023-12-19T07:48:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 0x7fae9c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae9c06df20 0x7fae9c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c06b2c0 con 0x7fae9c0cad10 2023-12-19T07:48:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06a6b0 con 0x7fae9c06df20 2023-12-19T07:48:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2b23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06a990 con 0x7fae9c0cb0e0 2023-12-19T07:48:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2322700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae9c06df20 0x7fae9c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2322700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae9c06df20 0x7fae9c15bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41692/0 (socket says 172.21.15.111:41692) 2023-12-19T07:48:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2322700 1 -- 172.21.15.111:0/29182384 learned_addr learned my addr 172.21.15.111:0/29182384 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea1320700 1 --2- 172.21.15.111:0/29182384 >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 0x7fae9c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.495+0000 7faea2322700 1 -- 172.21.15.111:0/29182384 >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 msgr2=0x7fae9c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2322700 1 --2- 172.21.15.111:0/29182384 >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 0x7fae9c06d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2322700 1 -- 172.21.15.111:0/29182384 >> v1:172.21.15.111:6789/0 conn(0x7fae9c0cad10 legacy=0x7fae9c06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2322700 1 -- 172.21.15.111:0/29182384 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c06ac70 con 0x7fae9c06df20 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea1320700 1 --2- 172.21.15.111:0/29182384 >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 0x7fae9c06d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2322700 1 --2- 172.21.15.111:0/29182384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c15bd00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0696f0 tx=0x7fae94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728bdbedadf9d8e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea0b1f700 1 -- 172.21.15.111:0/29182384 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9400e070 con 0x7fae9c06df20 2023-12-19T07:48:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea0b1f700 1 -- 172.21.15.111:0/29182384 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9400ba50 con 0x7fae9c06df20 2023-12-19T07:48:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea0b1f700 1 -- 172.21.15.111:0/29182384 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940092e0 con 0x7fae9c06df20 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 -- 172.21.15.111:0/29182384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 msgr2=0x7fae9c15bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 --2- 172.21.15.111:0/29182384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c15bd00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0696f0 tx=0x7fae94009840 comp rx=0 tx=0).stop 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 -- 172.21.15.111:0/29182384 shutdown_connections 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 --2- 172.21.15.111:0/29182384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c15bd00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 --2- 172.21.15.111:0/29182384 >> v2:172.21.15.143:3300/0 conn(0x7fae9c0cb0e0 0x7fae9c06d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 -- 172.21.15.111:0/29182384 >> 172.21.15.111:0/29182384 conn(0x7fae9c067bb0 msgr2=0x7fae9c0cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 -- 172.21.15.111:0/29182384 shutdown_connections 2023-12-19T07:48:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.499+0000 7faea2b23700 1 -- 172.21.15.111:0/29182384 wait complete. 2023-12-19T07:48:21.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 Processor -- start 2023-12-19T07:48:21.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 -- start start 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae9c0cad10 0x7fae9c15d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9c0cb0e0 0x7fae9c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c1690b0 con 0x7fae9c06df20 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c168d70 con 0x7fae9c0cad10 2023-12-19T07:48:21.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2b23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0b5230 con 0x7fae9c0cb0e0 2023-12-19T07:48:21.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41698/0 (socket says 172.21.15.111:41698) 2023-12-19T07:48:21.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 -- 172.21.15.111:0/185251977 learned_addr learned my addr 172.21.15.111:0/185251977 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1320700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae9c0cad10 0x7fae9c15d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea2322700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9c0cb0e0 0x7fae9c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 -- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9c0cb0e0 msgr2=0x7fae9c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9c0cb0e0 0x7fae9c15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 -- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae9c0cad10 msgr2=0x7fae9c15d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae9c0cad10 0x7fae9c15d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.503+0000 7faea1b21700 1 -- 172.21.15.111:0/185251977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94009ae0 con 0x7fae9c06df20 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7faea1b21700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c590 tx=0x7fae9800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98011590 con 0x7fae9c06df20 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae98011710 con 0x7fae9c06df20 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98011ad0 con 0x7fae9c06df20 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c0b5510 con 0x7fae9c06df20 2023-12-19T07:48:21.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c0b4e80 con 0x7fae9c06df20 2023-12-19T07:48:21.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.507+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c0b5a10 con 0x7fae9c06df20 2023-12-19T07:48:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.511+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fae9801b3b0 con 0x7fae9c06df20 2023-12-19T07:48:21.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.511+0000 7fae91ffb700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 0x7fae80062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.511+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fae9807bac0 con 0x7fae9c06df20 2023-12-19T07:48:21.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.511+0000 7faea1320700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 0x7fae80062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.511+0000 7faea1320700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 0x7fae80062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0b5c90 tx=0x7fae8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.515+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fae9804c2c0 con 0x7fae9c06df20 2023-12-19T07:48:21.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.743+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fae9c0d0d00 con 0x7fae9c06df20 2023-12-19T07:48:21.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.751+0000 7fae91ffb700 1 -- 172.21.15.111:0/185251977 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fae98017020 con 0x7fae9c06df20 2023-12-19T07:48:21.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 msgr2=0x7fae80062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 0x7fae80062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0b5c90 tx=0x7fae8c008040 comp rx=0 tx=0).stop 2023-12-19T07:48:21.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 msgr2=0x7fae9c0b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c590 tx=0x7fae9800c8a0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 shutdown_connections 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae80060550 0x7fae80062a00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae9c0cad10 0x7fae9c15d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9c0cb0e0 0x7fae9c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 --2- 172.21.15.111:0/185251977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae9c06df20 0x7fae9c0b4940 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 >> 172.21.15.111:0/185251977 conn(0x7fae9c067bb0 msgr2=0x7fae9c0cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:21.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 shutdown_connections 2023-12-19T07:48:21.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.755+0000 7faea2b23700 1 -- 172.21.15.111:0/185251977 wait complete. 2023-12-19T07:48:21.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-12-19T07:48:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.951+0000 7fde823c5700 1 Processor -- start 2023-12-19T07:48:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.951+0000 7fde823c5700 1 -- start start 2023-12-19T07:48:21.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde823c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 0x7fde7c06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde823c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c06ba50 con 0x7fde7c0afef0 2023-12-19T07:48:21.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde823c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0af600 con 0x7fde7c160e60 2023-12-19T07:48:21.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde823c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c06bee0 con 0x7fde7c15d050 2023-12-19T07:48:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde80962700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fde7c0afef0 0x7fde7c15d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33810/0 (socket says 172.21.15.111:33810) 2023-12-19T07:48:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde80962700 1 -- 172.21.15.111:0/1037711271 learned_addr learned my addr 172.21.15.111:0/1037711271 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7bfff700 1 --2- 172.21.15.111:0/1037711271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 0x7fde7c06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7b7fe700 1 -- 172.21.15.111:0/1037711271 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466999834 0 0) 0x7fde7c06ba50 con 0x7fde7c0afef0 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7b7fe700 1 -- 172.21.15.111:0/1037711271 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde68003570 con 0x7fde7c0afef0 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7bfff700 1 -- 172.21.15.111:0/1037711271 >> v1:172.21.15.143:6789/0 conn(0x7fde7c160e60 legacy=0x7fde7c0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7bfff700 1 -- 172.21.15.111:0/1037711271 >> v1:172.21.15.111:6789/0 conn(0x7fde7c0afef0 legacy=0x7fde7c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.955+0000 7fde7bfff700 1 -- 172.21.15.111:0/1037711271 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c06ae40 con 0x7fde7c15d050 2023-12-19T07:48:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde7bfff700 1 --2- 172.21.15.111:0/1037711271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 0x7fde7c06a740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009910 tx=0x7fde6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e8e9f8870ddaa03 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde7b7fe700 1 -- 172.21.15.111:0/1037711271 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c00b680 con 0x7fde7c15d050 2023-12-19T07:48:21.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde7b7fe700 1 -- 172.21.15.111:0/1037711271 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c00b7e0 con 0x7fde7c15d050 2023-12-19T07:48:21.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde7b7fe700 1 -- 172.21.15.111:0/1037711271 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c00bb60 con 0x7fde7c15d050 2023-12-19T07:48:21.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 -- 172.21.15.111:0/1037711271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 msgr2=0x7fde7c06a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 --2- 172.21.15.111:0/1037711271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 0x7fde7c06a740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009910 tx=0x7fde6c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 -- 172.21.15.111:0/1037711271 shutdown_connections 2023-12-19T07:48:21.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 --2- 172.21.15.111:0/1037711271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c15d050 0x7fde7c06a740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 -- 172.21.15.111:0/1037711271 >> 172.21.15.111:0/1037711271 conn(0x7fde7c067b60 msgr2=0x7fde7c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 -- 172.21.15.111:0/1037711271 shutdown_connections 2023-12-19T07:48:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.959+0000 7fde823c5700 1 -- 172.21.15.111:0/1037711271 wait complete. 2023-12-19T07:48:21.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 Processor -- start 2023-12-19T07:48:21.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 -- start start 2023-12-19T07:48:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde7c0afef0 0x7fde7c0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c160e60 0x7fde7c0c1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c8140 con 0x7fde7c15d050 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c7770 con 0x7fde7c0afef0 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.963+0000 7fde823c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c6b60 con 0x7fde7c160e60 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44668/0 (socket says 172.21.15.111:44668) 2023-12-19T07:48:21.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 -- 172.21.15.111:0/3923489094 learned_addr learned my addr 172.21.15.111:0/3923489094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde80962700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c160e60 0x7fde7c0c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde7bfff700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde7c0afef0 0x7fde7c0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 -- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c160e60 msgr2=0x7fde7c0c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c160e60 0x7fde7c0c1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 -- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde7c0afef0 msgr2=0x7fde7c0c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde7c0afef0 0x7fde7c0c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:21.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 -- 172.21.15.111:0/3923489094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c009420 con 0x7fde7c15d050 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde737fe700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c790 tx=0x7fde64007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011590 con 0x7fde7c15d050 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64011710 con 0x7fde7c15d050 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011ad0 con 0x7fde7c15d050 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c0c6e40 con 0x7fde7c15d050 2023-12-19T07:48:21.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.967+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c0c7400 con 0x7fde7c15d050 2023-12-19T07:48:21.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.971+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c0c78e0 con 0x7fde7c15d050 2023-12-19T07:48:21.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.971+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fde6401b3b0 con 0x7fde7c15d050 2023-12-19T07:48:21.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.971+0000 7fde78ff9700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 0x7fde5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:21.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.971+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fde6407b970 con 0x7fde7c15d050 2023-12-19T07:48:21.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.975+0000 7fde7bfff700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 0x7fde5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:21.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.975+0000 7fde7bfff700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 0x7fde5c062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0093f0 tx=0x7fde6c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:21.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:21.979+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde6404c0d0 con 0x7fde7c15d050 2023-12-19T07:48:22.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.211+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fde7c147ab0 con 0x7fde7c15d050 2023-12-19T07:48:22.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.219+0000 7fde78ff9700 1 -- 172.21.15.111:0/3923489094 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fde6404f6f0 con 0x7fde7c15d050 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 msgr2=0x7fde5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 0x7fde5c062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0093f0 tx=0x7fde6c017000 comp rx=0 tx=0).stop 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 msgr2=0x7fde7c147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c790 tx=0x7fde64007f40 comp rx=0 tx=0).stop 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 shutdown_connections 2023-12-19T07:48:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde5c060550 0x7fde5c062a00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde7c160e60 0x7fde7c0c1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde7c15d050 0x7fde7c147510 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 --2- 172.21.15.111:0/3923489094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde7c0afef0 0x7fde7c0c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 >> 172.21.15.111:0/3923489094 conn(0x7fde7c067b60 msgr2=0x7fde7c159ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 shutdown_connections 2023-12-19T07:48:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.223+0000 7fde823c5700 1 -- 172.21.15.111:0/3923489094 wait complete. 2023-12-19T07:48:22.228 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:22.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 Processor -- start 2023-12-19T07:48:22.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 -- start start 2023-12-19T07:48:22.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa308140ae0 0x7fa3080680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30806e150 con 0x7fa308068630 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308068a70 con 0x7fa308140710 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa30d478700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30806d540 con 0x7fa308140ae0 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.419+0000 7fa3077fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa308068630 0x7fa3080ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33822/0 (socket says 172.21.15.111:33822) 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3077fe700 1 -- 172.21.15.111:0/869300756 learned_addr learned my addr 172.21.15.111:0/869300756 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3067fc700 1 --2- 172.21.15.111:0/869300756 >> v2:172.21.15.143:3300/0 conn(0x7fa308140ae0 0x7fa3080680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa306ffd700 1 --1- 172.21.15.111:0/869300756 >> v1:172.21.15.143:6789/0 conn(0x7fa308140710 0x7fa3080c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:43332/0 (socket says 172.21.15.111:43332) 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533109978 0 0) 0x7fa308068a70 con 0x7fa308140710 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0003570 con 0x7fa308140710 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121211264 0 0) 0x7fa30806e150 con 0x7fa308068630 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3067fc700 1 -- 172.21.15.111:0/869300756 >> v1:172.21.15.143:6789/0 conn(0x7fa308140710 legacy=0x7fa3080c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3067fc700 1 -- 172.21.15.111:0/869300756 >> v1:172.21.15.111:6789/0 conn(0x7fa308068630 legacy=0x7fa3080ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3067fc700 1 -- 172.21.15.111:0/869300756 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30806d820 con 0x7fa308140ae0 2023-12-19T07:48:22.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa3067fc700 1 --2- 172.21.15.111:0/869300756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140ae0 0x7fa3080680f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0098e0 tx=0x7fa2fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3eb9c47f316ba25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00b640 con 0x7fa308140ae0 2023-12-19T07:48:22.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00b7a0 con 0x7fa308140ae0 2023-12-19T07:48:22.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa305ffb700 1 -- 172.21.15.111:0/869300756 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00bb40 con 0x7fa308140ae0 2023-12-19T07:48:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 -- 172.21.15.111:0/869300756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140ae0 msgr2=0x7fa3080680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 --2- 172.21.15.111:0/869300756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140ae0 0x7fa3080680f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0098e0 tx=0x7fa2fc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 -- 172.21.15.111:0/869300756 shutdown_connections 2023-12-19T07:48:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 --2- 172.21.15.111:0/869300756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140ae0 0x7fa3080680f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 -- 172.21.15.111:0/869300756 >> 172.21.15.111:0/869300756 conn(0x7fa3080c8850 msgr2=0x7fa3080c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 -- 172.21.15.111:0/869300756 shutdown_connections 2023-12-19T07:48:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.423+0000 7fa30d478700 1 -- 172.21.15.111:0/869300756 wait complete. 2023-12-19T07:48:22.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 Processor -- start 2023-12-19T07:48:22.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 -- start start 2023-12-19T07:48:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa308068630 0x7fa3080aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140710 0x7fa3080b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080aed40 con 0x7fa308140ae0 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080af020 con 0x7fa308140710 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa30d478700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080af300 con 0x7fa308068630 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa3077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa3077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44678/0 (socket says 172.21.15.111:44678) 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa3077fe700 1 -- 172.21.15.111:0/3194781654 learned_addr learned my addr 172.21.15.111:0/3194781654 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa306ffd700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa308068630 0x7fa3080aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.427+0000 7fa3067fc700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140710 0x7fa3080b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 -- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa308068630 msgr2=0x7fa3080aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa308068630 0x7fa3080aa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 -- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140710 msgr2=0x7fa3080b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140710 0x7fa3080b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 -- 172.21.15.111:0/3194781654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc009420 con 0x7fa308140ae0 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa3077fe700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa30000cc40 tx=0x7fa300001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000a8b0 con 0x7fa308140ae0 2023-12-19T07:48:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30000aa10 con 0x7fa308140ae0 2023-12-19T07:48:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000add0 con 0x7fa308140ae0 2023-12-19T07:48:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3080af5e0 con 0x7fa308140ae0 2023-12-19T07:48:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3080aae60 con 0x7fa308140ae0 2023-12-19T07:48:22.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.431+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa308071120 con 0x7fa308140ae0 2023-12-19T07:48:22.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.435+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa300007300 con 0x7fa308140ae0 2023-12-19T07:48:22.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.435+0000 7fa2eeffd700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 0x7fa2f4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.435+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fa30007c970 con 0x7fa308140ae0 2023-12-19T07:48:22.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.435+0000 7fa306ffd700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 0x7fa2f4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.435+0000 7fa306ffd700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 0x7fa2f4062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8004f70 tx=0x7fa2f8007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.439+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa30004d150 con 0x7fa308140ae0 2023-12-19T07:48:22.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.619+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3080642a0 con 0x7fa308140ae0 2023-12-19T07:48:22.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.623+0000 7fa2eeffd700 1 -- 172.21.15.111:0/3194781654 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+12387 (secure 0 0 0) 0x7fa300050770 con 0x7fa308140ae0 2023-12-19T07:48:22.606 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:22.607 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":36,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:19.077049+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 msgr2=0x7fa2f4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 0x7fa2f4062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8004f70 tx=0x7fa2f8007d40 comp rx=0 tx=0).stop 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 msgr2=0x7fa3080a9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa30000cc40 tx=0x7fa300001a00 comp rx=0 tx=0).stop 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 shutdown_connections 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa2f4060550 0x7fa2f4062a00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308140710 0x7fa3080b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa308140ae0 0x7fa3080a9b90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 --2- 172.21.15.111:0/3194781654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa308068630 0x7fa3080aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 >> 172.21.15.111:0/3194781654 conn(0x7fa3080c8850 msgr2=0x7fa30813e440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:22.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 shutdown_connections 2023-12-19T07:48:22.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.627+0000 7fa30d478700 1 -- 172.21.15.111:0/3194781654 wait complete. 2023-12-19T07:48:22.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T07:48:22.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:48:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.823+0000 7f25b359e700 1 Processor -- start 2023-12-19T07:48:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.823+0000 7f25b359e700 1 -- start start 2023-12-19T07:48:22.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25b406dcf0 0x7f25b4159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b406c850 con 0x7f25b4158c90 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b40cb650 con 0x7f25b406aeb0 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b406b470 con 0x7f25b406dcf0 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b2d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25b406dcf0 0x7f25b4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f25b4158c90 0x7f25b40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33836/0 (socket says 172.21.15.111:33836) 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b2d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25b406dcf0 0x7f25b4159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44686/0 (socket says 172.21.15.111:44686) 2023-12-19T07:48:22.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b259c700 1 -- 172.21.15.111:0/219151876 learned_addr learned my addr 172.21.15.111:0/219151876 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b2d9d700 1 -- 172.21.15.111:0/219151876 >> v1:172.21.15.143:6789/0 conn(0x7f25b406aeb0 legacy=0x7f25b4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b2d9d700 1 -- 172.21.15.111:0/219151876 >> v1:172.21.15.111:6789/0 conn(0x7f25b4158c90 legacy=0x7f25b40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b159a700 1 -- 172.21.15.111:0/219151876 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260753325 0 0) 0x7f25b406c850 con 0x7f25b4158c90 2023-12-19T07:48:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.827+0000 7f25b2d9d700 1 -- 172.21.15.111:0/219151876 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b406b750 con 0x7f25b406dcf0 2023-12-19T07:48:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b2d9d700 1 --2- 172.21.15.111:0/219151876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b406dcf0 0x7f25b4159770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009e60 tx=0x7f25ac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fd7a92db010dea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b159a700 1 -- 172.21.15.111:0/219151876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac00e070 con 0x7f25b406dcf0 2023-12-19T07:48:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b159a700 1 -- 172.21.15.111:0/219151876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac00bc30 con 0x7f25b406dcf0 2023-12-19T07:48:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b159a700 1 -- 172.21.15.111:0/219151876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac012430 con 0x7f25b406dcf0 2023-12-19T07:48:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- 172.21.15.111:0/219151876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b406dcf0 msgr2=0x7f25b4159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 --2- 172.21.15.111:0/219151876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b406dcf0 0x7f25b4159770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009e60 tx=0x7f25ac009780 comp rx=0 tx=0).stop 2023-12-19T07:48:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- 172.21.15.111:0/219151876 shutdown_connections 2023-12-19T07:48:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 --2- 172.21.15.111:0/219151876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b406dcf0 0x7f25b4159770 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- 172.21.15.111:0/219151876 >> 172.21.15.111:0/219151876 conn(0x7f25b4067bb0 msgr2=0x7f25b40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- 172.21.15.111:0/219151876 shutdown_connections 2023-12-19T07:48:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- 172.21.15.111:0/219151876 wait complete. 2023-12-19T07:48:22.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 Processor -- start 2023-12-19T07:48:22.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.831+0000 7f25b359e700 1 -- start start 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 0x7f25b40c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b406dcf0 0x7f25b40c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b41565b0 con 0x7f25b4158c90 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4156890 con 0x7f25b406aeb0 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4156b70 con 0x7f25b406dcf0 2023-12-19T07:48:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44694/0 (socket says 172.21.15.111:44694) 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 -- 172.21.15.111:0/2258630490 learned_addr learned my addr 172.21.15.111:0/2258630490 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b259c700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 0x7f25b40c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b1d9b700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b406dcf0 0x7f25b40c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 -- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b406dcf0 msgr2=0x7f25b40c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b406dcf0 0x7f25b40c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 -- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 msgr2=0x7f25b40c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 0x7f25b40c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 -- 172.21.15.111:0/2258630490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac009a20 con 0x7f25b4158c90 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b259c700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 0x7f25b40c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b2d9d700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25ac013000 tx=0x7f25ac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac01b630 con 0x7f25b4158c90 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac01b790 con 0x7f25b4158c90 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b4156e50 con 0x7f25b4158c90 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac01bb30 con 0x7f25b4158c90 2023-12-19T07:48:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.835+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b41536e0 con 0x7f25b4158c90 2023-12-19T07:48:22.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.839+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b4157350 con 0x7f25b4158c90 2023-12-19T07:48:22.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.839+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f25ac01c400 con 0x7f25b4158c90 2023-12-19T07:48:22.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.839+0000 7f25a27fc700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 0x7f2594062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:22.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.839+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f25ac07e630 con 0x7f25b4158c90 2023-12-19T07:48:22.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.839+0000 7f25b259c700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 0x7f2594062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:22.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.843+0000 7f25b259c700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 0x7f2594062a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25a4001890 tx=0x7f25a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:22.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:22.847+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f25ac04ed90 con 0x7f25b4158c90 2023-12-19T07:48:23.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.019+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25b404be40 con 0x7f25b4158c90 2023-12-19T07:48:23.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.019+0000 7f25a27fc700 1 -- 172.21.15.111:0/2258630490 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25ac02a020 con 0x7f25b4158c90 2023-12-19T07:48:23.005 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:48:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 msgr2=0x7f2594062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 0x7f2594062a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25a4001890 tx=0x7f25a4009040 comp rx=0 tx=0).stop 2023-12-19T07:48:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 msgr2=0x7f25b40c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25ac013000 tx=0x7f25ac019000 comp rx=0 tx=0).stop 2023-12-19T07:48:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 shutdown_connections 2023-12-19T07:48:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2594060550 0x7f2594062a00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25b4158c90 0x7f25b40c5860 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b406dcf0 0x7f25b40c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 --2- 172.21.15.111:0/2258630490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b406aeb0 0x7f25b40c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 >> 172.21.15.111:0/2258630490 conn(0x7f25b4067bb0 msgr2=0x7f25b4159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 shutdown_connections 2023-12-19T07:48:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.023+0000 7f25b359e700 1 -- 172.21.15.111:0/2258630490 wait complete. 2023-12-19T07:48:23.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T07:48:23.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.211+0000 7fed534cd700 1 Processor -- start 2023-12-19T07:48:23.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.211+0000 7fed534cd700 1 -- start start 2023-12-19T07:48:23.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed534cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed4c067c90 0x7fed4c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed534cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed534cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c0c6630 con 0x7fed4c14fba0 2023-12-19T07:48:23.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed534cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4c0c6910 con 0x7fed4c067c90 2023-12-19T07:48:23.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed534cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0c6bf0 con 0x7fed4c0c5900 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:55732/0 (socket says 172.21.15.111:55732) 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 -- 172.21.15.111:0/836419277 learned_addr learned my addr 172.21.15.111:0/836419277 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51269700 1 --2- 172.21.15.111:0/836419277 >> v2:172.21.15.143:3300/0 conn(0x7fed4c067c90 0x7fed4c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed50a68700 1 --1- 172.21.15.111:0/836419277 >> v1:172.21.15.143:6789/0 conn(0x7fed4c14fba0 0x7fed4c150000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:43360/0 (socket says 172.21.15.111:43360) 2023-12-19T07:48:23.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 -- 172.21.15.111:0/836419277 >> v2:172.21.15.143:3300/0 conn(0x7fed4c067c90 msgr2=0x7fed4c068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 --2- 172.21.15.111:0/836419277 >> v2:172.21.15.143:3300/0 conn(0x7fed4c067c90 0x7fed4c068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 -- 172.21.15.111:0/836419277 >> v1:172.21.15.143:6789/0 conn(0x7fed4c14fba0 legacy=0x7fed4c150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed43fff700 1 -- 172.21.15.111:0/836419277 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246263825 0 0) 0x7fed4c0c6630 con 0x7fed4c14fba0 2023-12-19T07:48:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 -- 172.21.15.111:0/836419277 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c0c6ed0 con 0x7fed4c0c5900 2023-12-19T07:48:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.215+0000 7fed51a6a700 1 --2- 172.21.15.111:0/836419277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed44009f20 tx=0x7fed44009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b48b6df18c5310 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed43fff700 1 -- 172.21.15.111:0/836419277 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4400e070 con 0x7fed4c0c5900 2023-12-19T07:48:23.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed43fff700 1 -- 172.21.15.111:0/836419277 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4400b9d0 con 0x7fed4c0c5900 2023-12-19T07:48:23.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed43fff700 1 -- 172.21.15.111:0/836419277 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed440092e0 con 0x7fed4c0c5900 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 -- 172.21.15.111:0/836419277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 msgr2=0x7fed4c0c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 --2- 172.21.15.111:0/836419277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed44009f20 tx=0x7fed44009840 comp rx=0 tx=0).stop 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 -- 172.21.15.111:0/836419277 shutdown_connections 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 --2- 172.21.15.111:0/836419277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c0c5900 0x7fed4c0c5d60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 --2- 172.21.15.111:0/836419277 >> v2:172.21.15.143:3300/0 conn(0x7fed4c067c90 0x7fed4c068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 -- 172.21.15.111:0/836419277 >> 172.21.15.111:0/836419277 conn(0x7fed4c0cd6d0 msgr2=0x7fed4c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:23.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 -- 172.21.15.111:0/836419277 shutdown_connections 2023-12-19T07:48:23.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.219+0000 7fed534cd700 1 -- 172.21.15.111:0/836419277 wait complete. 2023-12-19T07:48:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 Processor -- start 2023-12-19T07:48:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 -- start start 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed4c0c5900 0x7fed4c151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c14fba0 0x7fed4c1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b3f00 con 0x7fed4c0c5900 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b41a0 con 0x7fed4c067c90 2023-12-19T07:48:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed534cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b4480 con 0x7fed4c14fba0 2023-12-19T07:48:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed50a68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed4c0c5900 0x7fed4c151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed51269700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed51a6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c14fba0 0x7fed4c1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed51269700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:34562/0 (socket says 172.21.15.111:34562) 2023-12-19T07:48:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.223+0000 7fed51269700 1 -- 172.21.15.111:0/4195414070 learned_addr learned my addr 172.21.15.111:0/4195414070 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 -- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c14fba0 msgr2=0x7fed4c1507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c14fba0 0x7fed4c1507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 -- 172.21.15.111:0/4195414070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed4c0c5900 msgr2=0x7fed4c151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed4c0c5900 0x7fed4c151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 -- 172.21.15.111:0/4195414070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44009ae0 con 0x7fed4c067c90 2023-12-19T07:48:23.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed51269700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed4800ec80 tx=0x7fed480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800c850 con 0x7fed4c067c90 2023-12-19T07:48:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4800c9b0 con 0x7fed4c067c90 2023-12-19T07:48:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c0b4760 con 0x7fed4c067c90 2023-12-19T07:48:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800cd50 con 0x7fed4c067c90 2023-12-19T07:48:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c0b4d20 con 0x7fed4c067c90 2023-12-19T07:48:23.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.227+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c071120 con 0x7fed4c067c90 2023-12-19T07:48:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.231+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fed48013680 con 0x7fed4c067c90 2023-12-19T07:48:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.231+0000 7fed417fa700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 0x7fed30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.231+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fed4807c860 con 0x7fed4c067c90 2023-12-19T07:48:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.231+0000 7fed50a68700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 0x7fed30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.235+0000 7fed50a68700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 0x7fed30062a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fed4c1593f0 tx=0x7fed3c007ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.239+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fed4804cfc0 con 0x7fed4c067c90 2023-12-19T07:48:23.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.415+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fed4c0642a0 con 0x7fed4c067c90 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.419+0000 7fed417fa700 1 -- 172.21.15.111:0/4195414070 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) v1 ==== 84+0+65 (secure 0 0 0) 0x7fed480505e0 con 0x7fed4c067c90 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:23.403 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T07:48:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.419+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 msgr2=0x7fed30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.419+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 0x7fed30062a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fed4c1593f0 tx=0x7fed3c007ef0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.419+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 msgr2=0x7fed4c0b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.419+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed4800ec80 tx=0x7fed480091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 shutdown_connections 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fed30060550 0x7fed30062a00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed4c14fba0 0x7fed4c1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed4c0c5900 0x7fed4c151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 --2- 172.21.15.111:0/4195414070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed4c067c90 0x7fed4c0b39c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 >> 172.21.15.111:0/4195414070 conn(0x7fed4c0cd6d0 msgr2=0x7fed4c14ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 shutdown_connections 2023-12-19T07:48:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.423+0000 7fed534cd700 1 -- 172.21.15.111:0/4195414070 wait complete. 2023-12-19T07:48:23.425 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T07:48:23.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T07:48:23.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 Processor -- start 2023-12-19T07:48:23.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 -- start start 2023-12-19T07:48:23.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f348c06a580 0x7f348c06bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c0ab4b0 0x7f348c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0a9b70 con 0x7f348c06b730 2023-12-19T07:48:23.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c06b0b0 con 0x7f348c06a580 2023-12-19T07:48:23.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.607+0000 7f349140c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c06aaf0 con 0x7f348c0ab4b0 2023-12-19T07:48:23.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f348c06a580 0x7f348c06bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f3490c0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c0ab4b0 0x7f348c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f348bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f348c06b730 0x7f348c0a9460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:43366/0 (socket says 172.21.15.111:43366) 2023-12-19T07:48:23.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f348bfff700 1 -- 172.21.15.111:0/4200469208 learned_addr learned my addr 172.21.15.111:0/4200469208 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 -- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c0ab4b0 msgr2=0x7f348c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 --2- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c0ab4b0 0x7f348c06d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 -- 172.21.15.111:0/4200469208 >> v1:172.21.15.143:6789/0 conn(0x7f348c06b730 legacy=0x7f348c0a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 -- 172.21.15.111:0/4200469208 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c06add0 con 0x7f348c06a580 2023-12-19T07:48:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f34837fe700 1 --2- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06a580 0x7f348c06bb00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34740098e0 tx=0x7f34740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e99ef8856123e8dd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f348b7fe700 1 -- 172.21.15.111:0/4200469208 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400b640 con 0x7f348c06a580 2023-12-19T07:48:23.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f348b7fe700 1 -- 172.21.15.111:0/4200469208 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400b7a0 con 0x7f348c06a580 2023-12-19T07:48:23.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f348b7fe700 1 -- 172.21.15.111:0/4200469208 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347400bb40 con 0x7f348c06a580 2023-12-19T07:48:23.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f349140c700 1 -- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06a580 msgr2=0x7f348c06bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.611+0000 7f349140c700 1 --2- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06a580 0x7f348c06bb00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34740098e0 tx=0x7f34740091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 -- 172.21.15.111:0/4200469208 shutdown_connections 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 --2- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c0ab4b0 0x7f348c06d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 --2- 172.21.15.111:0/4200469208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06a580 0x7f348c06bb00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 -- 172.21.15.111:0/4200469208 >> 172.21.15.111:0/4200469208 conn(0x7f348c067bb0 msgr2=0x7f348c0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 -- 172.21.15.111:0/4200469208 shutdown_connections 2023-12-19T07:48:23.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 -- 172.21.15.111:0/4200469208 wait complete. 2023-12-19T07:48:23.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.615+0000 7f349140c700 1 Processor -- start 2023-12-19T07:48:23.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 -- start start 2023-12-19T07:48:23.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c06a580 0x7f348c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06b730 0x7f348c0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c16a990 con 0x7f348c0ab4b0 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0ca0a0 con 0x7f348c06b730 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0ca370 con 0x7f348c06a580 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44720/0 (socket says 172.21.15.111:44720) 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 -- 172.21.15.111:0/411180612 learned_addr learned my addr 172.21.15.111:0/411180612 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f348bfff700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c06a580 0x7f348c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f34837fe700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06b730 0x7f348c0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 -- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c06a580 msgr2=0x7f348c0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c06a580 0x7f348c0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 -- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06b730 msgr2=0x7f348c0c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06b730 0x7f348c0c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 -- 172.21.15.111:0/411180612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474009420 con 0x7f348c0ab4b0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3490c0b700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3484009df0 tx=0x7f348400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348400caa0 con 0x7f348c0ab4b0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348400f040 con 0x7f348c0ab4b0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348400ce70 con 0x7f348c0ab4b0 2023-12-19T07:48:23.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.619+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c0ca650 con 0x7f348c0ab4b0 2023-12-19T07:48:23.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.623+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c0cac10 con 0x7f348c0ab4b0 2023-12-19T07:48:23.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.623+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c16ab00 con 0x7f348c0ab4b0 2023-12-19T07:48:23.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.623+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3484014690 con 0x7f348c0ab4b0 2023-12-19T07:48:23.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.627+0000 7f3488ff9700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 0x7f346c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:23.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.627+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f348407cdd0 con 0x7f348c0ab4b0 2023-12-19T07:48:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.627+0000 7f348bfff700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 0x7f346c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.627+0000 7f348bfff700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 0x7f346c062a30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f348c0a91b0 tx=0x7f347c007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:23.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.635+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3484051060 con 0x7f348c0ab4b0 2023-12-19T07:48:23.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:23.815+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f348c04be40 con 0x7f348c0ab4b0 2023-12-19T07:48:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.275+0000 7f3488ff9700 1 -- 172.21.15.111:0/411180612 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v37) v1 ==== 138+0+0 (secure 0 0 0) 0x7f34840180b0 con 0x7f348c0ab4b0 2023-12-19T07:48:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 msgr2=0x7f346c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 0x7f346c062a30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f348c0a91b0 tx=0x7f347c007eb0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 msgr2=0x7f348c169390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3484009df0 tx=0x7f348400b040 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 shutdown_connections 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f346c060580 0x7f346c062a30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c06b730 0x7f348c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f348c0ab4b0 0x7f348c169390 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 --2- 172.21.15.111:0/411180612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f348c06a580 0x7f348c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 >> 172.21.15.111:0/411180612 conn(0x7f348c067bb0 msgr2=0x7f348c1567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:24.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 shutdown_connections 2023-12-19T07:48:24.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.283+0000 7f349140c700 1 -- 172.21.15.111:0/411180612 wait complete. 2023-12-19T07:48:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T07:48:24.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T07:48:24.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.487+0000 7fc4cf8a5700 1 Processor -- start 2023-12-19T07:48:24.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.487+0000 7fc4cf8a5700 1 -- start start 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cf8a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4c80cb970 0x7fc4c806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cf8a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 0x7fc4c806b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cf8a5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80cbdf0 con 0x7fc4c8158e20 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cf8a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c80cc0d0 con 0x7fc4c80cb970 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cf8a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80cc3b0 con 0x7fc4c80cecd0 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4c80cb970 0x7fc4c806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4c80cb970 0x7fc4c806acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44724/0 (socket says 172.21.15.111:44724) 2023-12-19T07:48:24.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 -- 172.21.15.111:0/2803333467 learned_addr learned my addr 172.21.15.111:0/2803333467 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:24.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cd641700 1 --1- 172.21.15.111:0/2803333467 >> v1:172.21.15.111:6789/0 conn(0x7fc4c8158e20 0x7fc4c80ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33838/0 (socket says 172.21.15.111:33838) 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cde42700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 0x7fc4c806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 -- 172.21.15.111:0/2803333467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 msgr2=0x7fc4c806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 0x7fc4c806b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 -- 172.21.15.111:0/2803333467 >> v1:172.21.15.111:6789/0 conn(0x7fc4c8158e20 legacy=0x7fc4c80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4bffff700 1 -- 172.21.15.111:0/2803333467 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911254872 0 0) 0x7fc4c80cbdf0 con 0x7fc4c8158e20 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 -- 172.21.15.111:0/2803333467 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c80cc690 con 0x7fc4c80cb970 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cde42700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 0x7fc4c806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4cce40700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cb970 0x7fc4c806acc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80099e0 tx=0x7fc4b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a570b84d80e7a5cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4bffff700 1 -- 172.21.15.111:0/2803333467 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800b820 con 0x7fc4c80cb970 2023-12-19T07:48:24.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4bffff700 1 -- 172.21.15.111:0/2803333467 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b800b980 con 0x7fc4c80cb970 2023-12-19T07:48:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.491+0000 7fc4bffff700 1 -- 172.21.15.111:0/2803333467 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800bd20 con 0x7fc4c80cb970 2023-12-19T07:48:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/2803333467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cb970 msgr2=0x7fc4c806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cb970 0x7fc4c806acc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80099e0 tx=0x7fc4b8009250 comp rx=0 tx=0).stop 2023-12-19T07:48:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/2803333467 shutdown_connections 2023-12-19T07:48:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c80cecd0 0x7fc4c806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/2803333467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cb970 0x7fc4c806acc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/2803333467 >> 172.21.15.111:0/2803333467 conn(0x7fc4c8067bb0 msgr2=0x7fc4c80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/2803333467 shutdown_connections 2023-12-19T07:48:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.495+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/2803333467 wait complete. 2023-12-19T07:48:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 Processor -- start 2023-12-19T07:48:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 -- start start 2023-12-19T07:48:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4c8158e20 0x7fc4c816a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c814e8b0 0x7fc4c80cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80b4360 con 0x7fc4c80cecd0 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c814f2f0 con 0x7fc4c8158e20 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cf8a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80b48e0 con 0x7fc4c814e8b0 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cd641700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cd641700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44728/0 (socket says 172.21.15.111:44728) 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.499+0000 7fc4cd641700 1 -- 172.21.15.111:0/4287668711 learned_addr learned my addr 172.21.15.111:0/4287668711 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cde42700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c814e8b0 0x7fc4c80cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cce40700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4c8158e20 0x7fc4c816a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 -- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c814e8b0 msgr2=0x7fc4c80cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c814e8b0 0x7fc4c80cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 -- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4c8158e20 msgr2=0x7fc4c816a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4c8158e20 0x7fc4c816a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 -- 172.21.15.111:0/4287668711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b80094f0 con 0x7fc4c80cecd0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cd641700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40011e0 tx=0x7fc4c4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4005a70 con 0x7fc4c80cecd0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c814ed30 con 0x7fc4c80cecd0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4005bd0 con 0x7fc4c80cecd0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40070d0 con 0x7fc4c80cecd0 2023-12-19T07:48:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.503+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c816a600 con 0x7fc4c80cecd0 2023-12-19T07:48:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.507+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c8071120 con 0x7fc4c80cecd0 2023-12-19T07:48:24.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.507+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc4c4007480 con 0x7fc4c80cecd0 2023-12-19T07:48:24.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.507+0000 7fc4bd7fa700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 0x7fc4ac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:24.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.507+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5543+0+0 (secure 0 0 0) 0x7fc4c407bc90 con 0x7fc4c80cecd0 2023-12-19T07:48:24.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.507+0000 7fc4cce40700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 0x7fc4ac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.511+0000 7fc4cce40700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 0x7fc4ac062a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80099e0 tx=0x7fc4b8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.515+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc4c404c320 con 0x7fc4c80cecd0 2023-12-19T07:48:24.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:24.687+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc4c806a7a0 con 0x7fc4c80cecd0 2023-12-19T07:48:25.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.287+0000 7fc4bd7fa700 1 -- 172.21.15.111:0/4287668711 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v38) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc4c40170d0 con 0x7fc4c80cecd0 2023-12-19T07:48:25.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.299+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 msgr2=0x7fc4ac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.299+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 0x7fc4ac062a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80099e0 tx=0x7fc4b8017000 comp rx=0 tx=0).stop 2023-12-19T07:48:25.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.299+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 msgr2=0x7fc4c814e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.299+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40011e0 tx=0x7fc4c4007e10 comp rx=0 tx=0).stop 2023-12-19T07:48:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 shutdown_connections 2023-12-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4ac060550 0x7fc4ac062a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4c8158e20 0x7fc4c816a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4c814e8b0 0x7fc4c80cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 --2- 172.21.15.111:0/4287668711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4c80cecd0 0x7fc4c814e370 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.303+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 >> 172.21.15.111:0/4287668711 conn(0x7fc4c8067bb0 msgr2=0x7fc4c806b000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.307+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 shutdown_connections 2023-12-19T07:48:25.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.307+0000 7fc4cf8a5700 1 -- 172.21.15.111:0/4287668711 wait complete. 2023-12-19T07:48:25.299 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T07:48:25.319 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T07:48:25.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 Processor -- start 2023-12-19T07:48:25.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 -- start start 2023-12-19T07:48:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03700c6d20 0x7f03700c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037006a7a0 con 0x7f037006ac30 2023-12-19T07:48:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700c7630 con 0x7f037006c7c0 2023-12-19T07:48:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f03768f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037006d4b0 con 0x7f03700c6d20 2023-12-19T07:48:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f0374e8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f037006ac30 0x7f037006b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33854/0 (socket says 172.21.15.111:33854) 2023-12-19T07:48:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.503+0000 7f0374e8d700 1 -- 172.21.15.111:0/3701356130 learned_addr learned my addr 172.21.15.111:0/3701356130 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036ffff700 1 --2- 172.21.15.111:0/3701356130 >> v2:172.21.15.143:3300/0 conn(0x7f03700c6d20 0x7f03700c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036f7fe700 1 -- 172.21.15.111:0/3701356130 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901706317 0 0) 0x7f037006a7a0 con 0x7f037006ac30 2023-12-19T07:48:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036f7fe700 1 -- 172.21.15.111:0/3701356130 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c003570 con 0x7f037006ac30 2023-12-19T07:48:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036ffff700 1 -- 172.21.15.111:0/3701356130 >> v1:172.21.15.143:6789/0 conn(0x7f037006c7c0 legacy=0x7f037006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036ffff700 1 -- 172.21.15.111:0/3701356130 >> v1:172.21.15.111:6789/0 conn(0x7f037006ac30 legacy=0x7f037006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036ffff700 1 -- 172.21.15.111:0/3701356130 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037006d790 con 0x7f03700c6d20 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036ffff700 1 --2- 172.21.15.111:0/3701356130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03700c6d20 0x7f03700c70f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f03600098e0 tx=0x7f03600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a09ac5f0a009faf3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036f7fe700 1 -- 172.21.15.111:0/3701356130 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036000b640 con 0x7f03700c6d20 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036f7fe700 1 -- 172.21.15.111:0/3701356130 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036000b7a0 con 0x7f03700c6d20 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f036f7fe700 1 -- 172.21.15.111:0/3701356130 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036000bb20 con 0x7f03700c6d20 2023-12-19T07:48:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 -- 172.21.15.111:0/3701356130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03700c6d20 msgr2=0x7f03700c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 --2- 172.21.15.111:0/3701356130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03700c6d20 0x7f03700c70f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f03600098e0 tx=0x7f03600091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 -- 172.21.15.111:0/3701356130 shutdown_connections 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 --2- 172.21.15.111:0/3701356130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03700c6d20 0x7f03700c70f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 -- 172.21.15.111:0/3701356130 >> 172.21.15.111:0/3701356130 conn(0x7f0370067bb0 msgr2=0x7f03700cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.507+0000 7f03768f0700 1 -- 172.21.15.111:0/3701356130 shutdown_connections 2023-12-19T07:48:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.511+0000 7f03768f0700 1 -- 172.21.15.111:0/3701356130 wait complete. 2023-12-19T07:48:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.511+0000 7f03768f0700 1 Processor -- start 2023-12-19T07:48:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.511+0000 7f03768f0700 1 -- start start 2023-12-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 0x7f0370150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 0x7f03700a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03700c6d20 0x7f03700a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03700acda0 con 0x7f037006ac30 2023-12-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03701531b0 con 0x7f037006c7c0 2023-12-19T07:48:25.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03700ac470 con 0x7f03700c6d20 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 0x7f0370150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 0x7f0370150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44734/0 (socket says 172.21.15.111:44734) 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036ffff700 1 -- 172.21.15.111:0/3984769195 learned_addr learned my addr 172.21.15.111:0/3984769195 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 0x7f03700a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f0374e8d700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03700c6d20 0x7f03700a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 -- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03700c6d20 msgr2=0x7f03700a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03700c6d20 0x7f03700a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 -- 172.21.15.111:0/3984769195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 msgr2=0x7f0370150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 0x7f0370150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 -- 172.21.15.111:0/3984769195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360009420 con 0x7f037006c7c0 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03677fe700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 0x7f03700a8ec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f035800c590 tx=0x7f0358007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358011590 con 0x7f037006c7c0 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0358011710 con 0x7f037006c7c0 2023-12-19T07:48:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03700ac750 con 0x7f037006c7c0 2023-12-19T07:48:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.515+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358011ad0 con 0x7f037006c7c0 2023-12-19T07:48:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.519+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03701510a0 con 0x7f037006c7c0 2023-12-19T07:48:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.519+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03700c2020 con 0x7f037006c7c0 2023-12-19T07:48:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.519+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f035801b3b0 con 0x7f037006c7c0 2023-12-19T07:48:25.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.523+0000 7f036cff9700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 0x7f0350062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:25.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.523+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5925+0+0 (secure 0 0 0) 0x7f035807bc70 con 0x7f037006c7c0 2023-12-19T07:48:25.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.523+0000 7f036ffff700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 0x7f0350062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:25.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.523+0000 7f036ffff700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 0x7f0350062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f03600093c0 tx=0x7f0360017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.531+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f03700c2020 con 0x7f037006c7c0 2023-12-19T07:48:25.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:25.739+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f037006ddc0 con 0x7f037006c7c0 2023-12-19T07:48:26.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.299+0000 7f036cff9700 1 -- 172.21.15.111:0/3984769195 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f037006ddc0 con 0x7f037006c7c0 2023-12-19T07:48:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.311+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 msgr2=0x7f0350062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.311+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 0x7f0350062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f03600093c0 tx=0x7f0360017000 comp rx=0 tx=0).stop 2023-12-19T07:48:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.311+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 msgr2=0x7f03700a8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.311+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 0x7f03700a8ec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f035800c590 tx=0x7f0358007ec0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 shutdown_connections 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0350060550 0x7f0350062a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f037006c7c0 0x7f03700a8ec0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03700c6d20 0x7f03700a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 --2- 172.21.15.111:0/3984769195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f037006ac30 0x7f0370150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.315+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 >> 172.21.15.111:0/3984769195 conn(0x7f0370067bb0 msgr2=0x7f0370158530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:26.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.319+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 shutdown_connections 2023-12-19T07:48:26.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.319+0000 7f03768f0700 1 -- 172.21.15.111:0/3984769195 wait complete. 2023-12-19T07:48:26.315 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 5 and data pool 6 2023-12-19T07:48:26.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:26.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 Processor -- start 2023-12-19T07:48:26.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 -- start start 2023-12-19T07:48:26.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb49c0c6cd0 0x7fb49c06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c06b470 con 0x7fb49c0c6900 2023-12-19T07:48:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c06be40 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a2323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c06e110 con 0x7fb49c15cc20 2023-12-19T07:48:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a1321700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb49c0c6900 0x7fb49c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33868/0 (socket says 172.21.15.111:33868) 2023-12-19T07:48:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a1321700 1 -- 172.21.15.111:0/1522007180 learned_addr learned my addr 172.21.15.111:0/1522007180 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/1522007180 >> v2:172.21.15.143:3300/0 conn(0x7fb49c0c6cd0 0x7fb49c06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb4a1b22700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb493fff700 1 -- 172.21.15.111:0/1522007180 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267650628 0 0) 0x7fb49c06b470 con 0x7fb49c0c6900 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.515+0000 7fb493fff700 1 -- 172.21.15.111:0/1522007180 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb488003570 con 0x7fb49c0c6900 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a0b20700 1 -- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 msgr2=0x7fb49c15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c15d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a0b20700 1 -- 172.21.15.111:0/1522007180 >> v1:172.21.15.111:6789/0 conn(0x7fb49c0c6900 legacy=0x7fb49c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a0b20700 1 -- 172.21.15.111:0/1522007180 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c161d10 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c06b900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0098e0 tx=0x7fb48c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69b4b45386848f4f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb493fff700 1 -- 172.21.15.111:0/1522007180 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011630 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb493fff700 1 -- 172.21.15.111:0/1522007180 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c011790 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb493fff700 1 -- 172.21.15.111:0/1522007180 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011b30 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 -- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 msgr2=0x7fb49c06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c06b900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0098e0 tx=0x7fb48c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 -- 172.21.15.111:0/1522007180 shutdown_connections 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c15d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 --2- 172.21.15.111:0/1522007180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c06b900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 -- 172.21.15.111:0/1522007180 >> 172.21.15.111:0/1522007180 conn(0x7fb49c067bb0 msgr2=0x7fb49c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 -- 172.21.15.111:0/1522007180 shutdown_connections 2023-12-19T07:48:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.519+0000 7fb4a2323700 1 -- 172.21.15.111:0/1522007180 wait complete. 2023-12-19T07:48:26.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 Processor -- start 2023-12-19T07:48:26.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 -- start start 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c15d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c16e550 con 0x7fb49c0c6900 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c16d940 con 0x7fb49c0c6cd0 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.523+0000 7fb4a2323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c16dc20 con 0x7fb49c15cc20 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a0b20700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44736/0 (socket says 172.21.15.111:44736) 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 -- 172.21.15.111:0/2299956522 learned_addr learned my addr 172.21.15.111:0/2299956522 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1b22700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 -- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 msgr2=0x7fb49c147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 -- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 msgr2=0x7fb49c15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c15d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 -- 172.21.15.111:0/2299956522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c009420 con 0x7fb49c0c6900 2023-12-19T07:48:26.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:26.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a1321700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c7c0 tx=0x7fb498007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498005b30 con 0x7fb49c0c6900 2023-12-19T07:48:26.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498005cb0 con 0x7fb49c0c6900 2023-12-19T07:48:26.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498007320 con 0x7fb49c0c6900 2023-12-19T07:48:26.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c16df00 con 0x7fb49c0c6900 2023-12-19T07:48:26.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c15d2d0 con 0x7fb49c0c6900 2023-12-19T07:48:26.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.527+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c04be40 con 0x7fb49c0c6900 2023-12-19T07:48:26.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.531+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb49801b3b0 con 0x7fb49c0c6900 2023-12-19T07:48:26.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.535+0000 7fb4917fa700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 0x7fb480062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.535+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fb49807be90 con 0x7fb49c0c6900 2023-12-19T07:48:26.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.535+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 0x7fb480062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.535+0000 7fb4a0b20700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 0x7fb480062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0091e0 tx=0x7fb48c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.539+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb49804c2a0 con 0x7fb49c0c6900 2023-12-19T07:48:26.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.711+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb49c06a580 con 0x7fb49c0c6900 2023-12-19T07:48:26.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4917fa700 1 -- 172.21.15.111:0/2299956522 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+15616 (secure 0 0 0) 0x7fb498017090 con 0x7fb49c0c6900 2023-12-19T07:48:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:26.698 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:25.748241+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:26.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 msgr2=0x7fb480062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 0x7fb480062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0091e0 tx=0x7fb48c017000 comp rx=0 tx=0).stop 2023-12-19T07:48:26.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 msgr2=0x7fb49c162130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb49800c7c0 tx=0x7fb498007e00 comp rx=0 tx=0).stop 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 shutdown_connections 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb480060550 0x7fb480062a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0c6cd0 0x7fb49c15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb49c15cc20 0x7fb49c147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 --2- 172.21.15.111:0/2299956522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb49c0c6900 0x7fb49c162130 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 >> 172.21.15.111:0/2299956522 conn(0x7fb49c067bb0 msgr2=0x7fb49c0abdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:26.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 shutdown_connections 2023-12-19T07:48:26.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.715+0000 7fb4a2323700 1 -- 172.21.15.111:0/2299956522 wait complete. 2023-12-19T07:48:26.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:26.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.899+0000 7f8e7a3c5700 1 Processor -- start 2023-12-19T07:48:26.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.899+0000 7f8e7a3c5700 1 -- start start 2023-12-19T07:48:26.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e7a3c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 0x7f8e7415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e7a3c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 0x7f8e740aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e7a3c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e740c7190 con 0x7f8e7406d690 2023-12-19T07:48:26.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e7a3c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e740c1b50 con 0x7f8e740c15b0 2023-12-19T07:48:26.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e7a3c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740c1e30 con 0x7f8e7415d320 2023-12-19T07:48:26.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e78962700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8e7406d690 0x7f8e740680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33872/0 (socket says 172.21.15.111:33872) 2023-12-19T07:48:26.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e78962700 1 -- 172.21.15.111:0/3991954111 learned_addr learned my addr 172.21.15.111:0/3991954111 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:26.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e73fff700 1 --2- 172.21.15.111:0/3991954111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 0x7f8e7415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e6bfff700 1 --2- 172.21.15.111:0/3991954111 >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 0x7f8e740aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e737fe700 1 -- 172.21.15.111:0/3991954111 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657009798 0 0) 0x7f8e740c7190 con 0x7f8e7406d690 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e737fe700 1 -- 172.21.15.111:0/3991954111 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e60003570 con 0x7f8e7406d690 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e73fff700 1 -- 172.21.15.111:0/3991954111 >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 msgr2=0x7f8e740aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e73fff700 1 --2- 172.21.15.111:0/3991954111 >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 0x7f8e740aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e73fff700 1 -- 172.21.15.111:0/3991954111 >> v1:172.21.15.111:6789/0 conn(0x7f8e7406d690 legacy=0x7f8e740680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e73fff700 1 -- 172.21.15.111:0/3991954111 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e740c6e20 con 0x7f8e7415d320 2023-12-19T07:48:26.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.903+0000 7f8e6bfff700 1 --2- 172.21.15.111:0/3991954111 >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 0x7f8e740aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:26.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e73fff700 1 --2- 172.21.15.111:0/3991954111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 0x7f8e7415d6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e640098e0 tx=0x7f8e640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4572ef08638b7e25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e737fe700 1 -- 172.21.15.111:0/3991954111 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6400b640 con 0x7f8e7415d320 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e737fe700 1 -- 172.21.15.111:0/3991954111 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6400b7a0 con 0x7f8e7415d320 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e737fe700 1 -- 172.21.15.111:0/3991954111 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6400bb40 con 0x7f8e7415d320 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/3991954111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 msgr2=0x7f8e7415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/3991954111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 0x7f8e7415d6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e640098e0 tx=0x7f8e640091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/3991954111 shutdown_connections 2023-12-19T07:48:26.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/3991954111 >> v2:172.21.15.143:3300/0 conn(0x7f8e740c15b0 0x7f8e740aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/3991954111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7415d320 0x7f8e7415d6f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/3991954111 >> 172.21.15.111:0/3991954111 conn(0x7f8e740c8850 msgr2=0x7f8e740c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:26.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/3991954111 shutdown_connections 2023-12-19T07:48:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/3991954111 wait complete. 2023-12-19T07:48:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 Processor -- start 2023-12-19T07:48:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.907+0000 7f8e7a3c5700 1 -- start start 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 0x7f8e7416e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 0x7f8e74067dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740c37f0 con 0x7f8e7415d320 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740c3a40 con 0x7f8e740c15b0 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740c3d20 con 0x7f8e7406d690 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44748/0 (socket says 172.21.15.111:44748) 2023-12-19T07:48:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 -- 172.21.15.111:0/1543464778 learned_addr learned my addr 172.21.15.111:0/1543464778 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e73fff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 0x7f8e7416e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e6bfff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 0x7f8e74067dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 -- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 msgr2=0x7f8e7416e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 0x7f8e7416e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 -- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 msgr2=0x7f8e74067dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 0x7f8e74067dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 -- 172.21.15.111:0/1543464778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e64009420 con 0x7f8e7415d320 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e6bfff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 0x7f8e74067dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e73fff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 0x7f8e7416e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e78962700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c001460 tx=0x7f8e6c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00ce80 con 0x7f8e7415d320 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e740c4000 con 0x7f8e7415d320 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c00ab70 con 0x7f8e7415d320 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0070f0 con 0x7f8e7415d320 2023-12-19T07:48:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.911+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7416ed60 con 0x7f8e7415d320 2023-12-19T07:48:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.915+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e74071120 con 0x7f8e7415d320 2023-12-19T07:48:26.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.915+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8e6c007520 con 0x7f8e7415d320 2023-12-19T07:48:26.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.919+0000 7f8e70ff9700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 0x7f8e54062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:26.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.919+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f8e6c07cf30 con 0x7f8e7415d320 2023-12-19T07:48:26.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.919+0000 7f8e73fff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 0x7f8e54062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:26.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.919+0000 7f8e73fff700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 0x7f8e54062a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8e640093f0 tx=0x7f8e64017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:26.923+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e6c04d2b0 con 0x7f8e7415d320 2023-12-19T07:48:27.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.087+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e740642a0 con 0x7f8e7415d320 2023-12-19T07:48:27.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.091+0000 7f8e70ff9700 1 -- 172.21.15.111:0/1543464778 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+15616 (secure 0 0 0) 0x7f8e6c0508d0 con 0x7f8e7415d320 2023-12-19T07:48:27.075 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:27.075 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:25.748241+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.091+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 msgr2=0x7f8e54062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 0x7f8e54062a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8e640093f0 tx=0x7f8e64017000 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 msgr2=0x7f8e740c4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c001460 tx=0x7f8e6c001ae0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 shutdown_connections 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e54060550 0x7f8e54062a00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e740c15b0 0x7f8e74067dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e7415d320 0x7f8e740c4800 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 --2- 172.21.15.111:0/1543464778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e7406d690 0x7f8e7416e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 >> 172.21.15.111:0/1543464778 conn(0x7f8e740c8850 msgr2=0x7f8e7415abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 shutdown_connections 2023-12-19T07:48:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.095+0000 7f8e7a3c5700 1 -- 172.21.15.111:0/1543464778 wait complete. 2023-12-19T07:48:27.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T07:48:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.279+0000 7f82be0a5700 1 Processor -- start 2023-12-19T07:48:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.279+0000 7f82be0a5700 1 -- start start 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82b815d410 0x7f82b80ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82b80ca400 0x7f82b80ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b806b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b80c7450 con 0x7f82b815d410 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b80cad50 con 0x7f82b80ca400 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82be0a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b806bea0 con 0x7f82b806b1b0 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82b815d410 0x7f82b80ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82b815d410 0x7f82b80ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44756/0 (socket says 172.21.15.111:44756) 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 -- 172.21.15.111:0/4160917875 learned_addr learned my addr 172.21.15.111:0/4160917875 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b6ffd700 1 --2- 172.21.15.111:0/4160917875 >> v2:172.21.15.143:3300/0 conn(0x7f82b80ca400 0x7f82b80ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b7fff700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b806b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 -- 172.21.15.111:0/4160917875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 msgr2=0x7f82b806b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b806b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 -- 172.21.15.111:0/4160917875 >> v2:172.21.15.143:3300/0 conn(0x7f82b80ca400 msgr2=0x7f82b80ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 --2- 172.21.15.111:0/4160917875 >> v2:172.21.15.143:3300/0 conn(0x7f82b80ca400 0x7f82b80ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 -- 172.21.15.111:0/4160917875 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80c6e00 con 0x7f82b815d410 2023-12-19T07:48:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b77fe700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80ab6c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f82b806c230 tx=0x7f82a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca966ef98a8149f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b67fc700 1 -- 172.21.15.111:0/4160917875 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8009e70 con 0x7f82b815d410 2023-12-19T07:48:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b67fc700 1 -- 172.21.15.111:0/4160917875 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a800b860 con 0x7f82b815d410 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.283+0000 7f82b67fc700 1 -- 172.21.15.111:0/4160917875 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a800bc00 con 0x7f82b815d410 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- 172.21.15.111:0/4160917875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 msgr2=0x7f82b80ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80ab6c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f82b806c230 tx=0x7f82a8009250 comp rx=0 tx=0).stop 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- 172.21.15.111:0/4160917875 shutdown_connections 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b806b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 --2- 172.21.15.111:0/4160917875 >> v2:172.21.15.143:3300/0 conn(0x7f82b80ca400 0x7f82b80ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 --2- 172.21.15.111:0/4160917875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80ab6c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- 172.21.15.111:0/4160917875 >> 172.21.15.111:0/4160917875 conn(0x7f82b8067bb0 msgr2=0x7f82b80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- 172.21.15.111:0/4160917875 shutdown_connections 2023-12-19T07:48:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- 172.21.15.111:0/4160917875 wait complete. 2023-12-19T07:48:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 Processor -- start 2023-12-19T07:48:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.287+0000 7f82be0a5700 1 -- start start 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b8159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b80ca400 0x7f82b80afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80b12f0 con 0x7f82b815d410 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b815fc90 con 0x7f82b80ca400 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80b06e0 con 0x7f82b806b1b0 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44772/0 (socket says 172.21.15.111:44772) 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 -- 172.21.15.111:0/3171324048 learned_addr learned my addr 172.21.15.111:0/3171324048 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b77fe700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b8159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b6ffd700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b80ca400 0x7f82b80afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 -- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 msgr2=0x7f82b8159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b8159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 -- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b80ca400 msgr2=0x7f82b80afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b80ca400 0x7f82b80afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 -- 172.21.15.111:0/3171324048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a80094f0 con 0x7f82b815d410 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82b7fff700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007f20 tx=0x7f82b0009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b000ebf0 con 0x7f82b815d410 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b000ed70 con 0x7f82b815d410 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0014670 con 0x7f82b815d410 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b80b09c0 con 0x7f82b815d410 2023-12-19T07:48:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.291+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b80b0f80 con 0x7f82b815d410 2023-12-19T07:48:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.295+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b80b1460 con 0x7f82b815d410 2023-12-19T07:48:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.295+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f82b00149e0 con 0x7f82b815d410 2023-12-19T07:48:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.295+0000 7f829f7fe700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 0x7f82a4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.295+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f82b007cf10 con 0x7f82b815d410 2023-12-19T07:48:27.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.299+0000 7f82b77fe700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 0x7f82a4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:27.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.299+0000 7f82b77fe700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 0x7f82a4062a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82b806c230 tx=0x7f82a8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:27.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.303+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f82b004d3c0 con 0x7f82b815d410 2023-12-19T07:48:27.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:27.511+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f82b8071120 con 0x7f82b815d410 2023-12-19T07:48:28.216 INFO:tasks.ceph.mds.b.smithi143.stderr:2023-12-19T07:48:28.235+0000 7fef473b4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T07:48:28.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.331+0000 7f829f7fe700 1 -- 172.21.15.111:0/3171324048 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82b00509e0 con 0x7f82b815d410 2023-12-19T07:48:28.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 msgr2=0x7f82a4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 0x7f82a4062a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82b806c230 tx=0x7f82a8017000 comp rx=0 tx=0).stop 2023-12-19T07:48:28.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 msgr2=0x7f82b80b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007f20 tx=0x7f82b0009860 comp rx=0 tx=0).stop 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 shutdown_connections 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82a4060550 0x7f82a4062a00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b80ca400 0x7f82b80afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82b815d410 0x7f82b80b00b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 --2- 172.21.15.111:0/3171324048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82b806b1b0 0x7f82b8159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 >> 172.21.15.111:0/3171324048 conn(0x7f82b8067bb0 msgr2=0x7f82b80cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 shutdown_connections 2023-12-19T07:48:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.339+0000 7f82be0a5700 1 -- 172.21.15.111:0/3171324048 wait complete. 2023-12-19T07:48:28.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T07:48:28.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.527+0000 7f366082e700 1 Processor -- start 2023-12-19T07:48:28.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.527+0000 7f366082e700 1 -- start start 2023-12-19T07:48:28.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f366082e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f365c0cb970 0x7f365c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:28.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f366082e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0cbdf0 con 0x7f365c0cecd0 2023-12-19T07:48:28.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f366082e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0cc0d0 con 0x7f365c158e20 2023-12-19T07:48:28.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f366082e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365c0cc3b0 con 0x7f365c0cb970 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f365c0cb970 0x7f365c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f365c0cb970 0x7f365c06acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44788/0 (socket says 172.21.15.111:44788) 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 -- 172.21.15.111:0/642220821 learned_addr learned my addr 172.21.15.111:0/642220821 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 -- 172.21.15.111:0/642220821 >> v1:172.21.15.143:6789/0 conn(0x7f365c158e20 legacy=0x7f365c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 -- 172.21.15.111:0/642220821 >> v1:172.21.15.111:6789/0 conn(0x7f365c0cecd0 legacy=0x7f365c06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.531+0000 7f3659d9b700 1 -- 172.21.15.111:0/642220821 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c0cc690 con 0x7f365c0cb970 2023-12-19T07:48:28.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f3659d9b700 1 --2- 172.21.15.111:0/642220821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c0cb970 0x7f365c06acc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f36500099e0 tx=0x7f3650009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd89e3f599ba07e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:28.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f365959a700 1 -- 172.21.15.111:0/642220821 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365000b820 con 0x7f365c0cb970 2023-12-19T07:48:28.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f365959a700 1 -- 172.21.15.111:0/642220821 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365000b980 con 0x7f365c0cb970 2023-12-19T07:48:28.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f365959a700 1 -- 172.21.15.111:0/642220821 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365000bd20 con 0x7f365c0cb970 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- 172.21.15.111:0/642220821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c0cb970 msgr2=0x7f365c06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 --2- 172.21.15.111:0/642220821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c0cb970 0x7f365c06acc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f36500099e0 tx=0x7f3650009250 comp rx=0 tx=0).stop 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- 172.21.15.111:0/642220821 shutdown_connections 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 --2- 172.21.15.111:0/642220821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c0cb970 0x7f365c06acc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- 172.21.15.111:0/642220821 >> 172.21.15.111:0/642220821 conn(0x7f365c067bb0 msgr2=0x7f365c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- 172.21.15.111:0/642220821 shutdown_connections 2023-12-19T07:48:28.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- 172.21.15.111:0/642220821 wait complete. 2023-12-19T07:48:28.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 Processor -- start 2023-12-19T07:48:28.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.535+0000 7f366082e700 1 -- start start 2023-12-19T07:48:28.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c0cecd0 0x7f365c14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:28.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:28.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f365c0c7b50 0x7f365c06b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:28.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c0b4b30 con 0x7f365c158e20 2023-12-19T07:48:28.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c14d160 con 0x7f365c0cecd0 2023-12-19T07:48:28.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c0c9b40 con 0x7f365c0c7b50 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44802/0 (socket says 172.21.15.111:44802) 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 -- 172.21.15.111:0/1275685069 learned_addr learned my addr 172.21.15.111:0/1275685069 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f365a59c700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c0cecd0 0x7f365c14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f365ad9d700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f365c0c7b50 0x7f365c06b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 -- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f365c0c7b50 msgr2=0x7f365c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f365c0c7b50 0x7f365c06b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 -- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c0cecd0 msgr2=0x7f365c14d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c0cecd0 0x7f365c14d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:28.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 -- 172.21.15.111:0/1275685069 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500094f0 con 0x7f365c158e20 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f3659d9b700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f36500094c0 tx=0x7f3650017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365001a420 con 0x7f365c158e20 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365001a5a0 con 0x7f365c158e20 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365001a960 con 0x7f365c158e20 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c0b3c40 con 0x7f365c158e20 2023-12-19T07:48:28.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.539+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c0b4200 con 0x7f365c158e20 2023-12-19T07:48:28.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.543+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c0b4ca0 con 0x7f365c158e20 2023-12-19T07:48:28.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.543+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f365001ad10 con 0x7f365c158e20 2023-12-19T07:48:28.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.543+0000 7f364a7fc700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 0x7f363c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:28.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.543+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f365007e2b0 con 0x7f365c158e20 2023-12-19T07:48:28.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.543+0000 7f365a59c700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 0x7f363c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:28.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.547+0000 7f365a59c700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 0x7f363c062a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f364c0039f0 tx=0x7f364c007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:28.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.551+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f365004e760 con 0x7f365c158e20 2023-12-19T07:48:28.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:28.755+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f365c071120 con 0x7f365c158e20 2023-12-19T07:48:29.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.347+0000 7f364a7fc700 1 -- 172.21.15.111:0/1275685069 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f36500150c0 con 0x7f365c158e20 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 msgr2=0x7f363c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 0x7f363c062a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f364c0039f0 tx=0x7f364c007ec0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 msgr2=0x7f365c14cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f36500094c0 tx=0x7f3650017000 comp rx=0 tx=0).stop 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 shutdown_connections 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f363c060550 0x7f363c062a00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f365c0c7b50 0x7f365c06b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c158e20 0x7f365c14cc20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 --2- 172.21.15.111:0/1275685069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c0cecd0 0x7f365c14d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 >> 172.21.15.111:0/1275685069 conn(0x7f365c067bb0 msgr2=0x7f365c1564c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 shutdown_connections 2023-12-19T07:48:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.351+0000 7f366082e700 1 -- 172.21.15.111:0/1275685069 wait complete. 2023-12-19T07:48:29.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.535+0000 7ff70d8e3700 1 Processor -- start 2023-12-19T07:48:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.535+0000 7ff70d8e3700 1 -- start start 2023-12-19T07:48:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d8e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7080c6900 0x7ff7080c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff70815cc20 0x7ff70815d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d8e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70806b470 con 0x7ff70806b900 2023-12-19T07:48:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d8e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70806e110 con 0x7ff7080c6900 2023-12-19T07:48:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff708161d10 con 0x7ff70815cc20 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff6fffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff70806b900 0x7ff70806da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33902/0 (socket says 172.21.15.111:33902) 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff6fffff700 1 -- 172.21.15.111:0/3939359289 learned_addr learned my addr 172.21.15.111:0/3939359289 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3939359289 >> v2:172.21.15.143:3300/0 conn(0x7ff7080c6900 0x7ff7080c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70d0e2700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff70815cc20 0x7ff70815d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff6ff7fe700 1 -- 172.21.15.111:0/3939359289 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975638827 0 0) 0x7ff70806b470 con 0x7ff70806b900 2023-12-19T07:48:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff6ff7fe700 1 -- 172.21.15.111:0/3939359289 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0003570 con 0x7ff70806b900 2023-12-19T07:48:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70c8e1700 1 -- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff70815cc20 msgr2=0x7ff70815d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.539+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff70815cc20 0x7ff70815d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70c8e1700 1 -- 172.21.15.111:0/3939359289 >> v1:172.21.15.111:6789/0 conn(0x7ff70806b900 legacy=0x7ff70806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70c8e1700 1 -- 172.21.15.111:0/3939359289 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70806bd70 con 0x7ff7080c6900 2023-12-19T07:48:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c6900 0x7ff7080c6cd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff7040098e0 tx=0x7ff7040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3af96a982da59960 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff6ff7fe700 1 -- 172.21.15.111:0/3939359289 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704011630 con 0x7ff7080c6900 2023-12-19T07:48:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff6ff7fe700 1 -- 172.21.15.111:0/3939359289 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff704011790 con 0x7ff7080c6900 2023-12-19T07:48:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff6ff7fe700 1 -- 172.21.15.111:0/3939359289 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff704011b30 con 0x7ff7080c6900 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c6900 msgr2=0x7ff7080c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c6900 0x7ff7080c6cd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff7040098e0 tx=0x7ff7040091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3939359289 shutdown_connections 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff70815cc20 0x7ff70815d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3939359289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c6900 0x7ff7080c6cd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3939359289 >> 172.21.15.111:0/3939359289 conn(0x7ff708067bb0 msgr2=0x7ff7080cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3939359289 shutdown_connections 2023-12-19T07:48:29.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.543+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3939359289 wait complete. 2023-12-19T07:48:29.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 Processor -- start 2023-12-19T07:48:29.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 -- start start 2023-12-19T07:48:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff70806b900 0x7ff708171140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c6900 0x7ff708158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70815db80 con 0x7ff70815cc20 2023-12-19T07:48:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff708171680 con 0x7ff70806b900 2023-12-19T07:48:29.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70815dd60 con 0x7ff7080c6900 2023-12-19T07:48:29.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d0e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d0e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44810/0 (socket says 172.21.15.111:44810) 2023-12-19T07:48:29.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70d0e2700 1 -- 172.21.15.111:0/3399442095 learned_addr learned my addr 172.21.15.111:0/3399442095 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:29.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff70806b900 0x7ff708171140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.547+0000 7ff6fffff700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c6900 0x7ff708158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 -- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c6900 msgr2=0x7ff708158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c6900 0x7ff708158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 -- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff70806b900 msgr2=0x7ff708171140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff70806b900 0x7ff708171140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 -- 172.21.15.111:0/3399442095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704009420 con 0x7ff70815cc20 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d0e2700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff700009df0 tx=0x7ff70000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000caa0 con 0x7ff70815cc20 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000f040 con 0x7ff70815cc20 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000ce70 con 0x7ff70815cc20 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70815e040 con 0x7ff70815cc20 2023-12-19T07:48:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70815e600 con 0x7ff70815cc20 2023-12-19T07:48:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.551+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70804be40 con 0x7ff70815cc20 2023-12-19T07:48:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.555+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff700014690 con 0x7ff70815cc20 2023-12-19T07:48:29.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.555+0000 7ff6fcff9700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 0x7ff6f4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.555+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff70007d4b0 con 0x7ff70815cc20 2023-12-19T07:48:29.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.555+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 0x7ff6f4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.559+0000 7ff70c8e1700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 0x7ff6f4062a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff7040091e0 tx=0x7ff704017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.559+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff700051350 con 0x7ff70815cc20 2023-12-19T07:48:29.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.767+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff70806a580 con 0x7ff70815cc20 2023-12-19T07:48:29.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.767+0000 7ff6fcff9700 1 -- 172.21.15.111:0/3399442095 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7ff700050ee0 con 0x7ff70815cc20 2023-12-19T07:48:29.751 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:29.751 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 msgr2=0x7ff6f4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 0x7ff6f4062a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff7040091e0 tx=0x7ff704017000 comp rx=0 tx=0).stop 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 msgr2=0x7ff70815f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff700009df0 tx=0x7ff70000b040 comp rx=0 tx=0).stop 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 shutdown_connections 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff6f4060550 0x7ff6f4062a00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff70815cc20 0x7ff70815f9f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c6900 0x7ff708158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 --2- 172.21.15.111:0/3399442095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff70806b900 0x7ff708171140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 >> 172.21.15.111:0/3399442095 conn(0x7ff708067bb0 msgr2=0x7ff708159b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:29.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 shutdown_connections 2023-12-19T07:48:29.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.771+0000 7ff70d8e3700 1 -- 172.21.15.111:0/3399442095 wait complete. 2023-12-19T07:48:29.757 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:48:29.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:48:29.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 Processor -- start 2023-12-19T07:48:29.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 -- start start 2023-12-19T07:48:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4a2c15cc20 0x7f4a2c15d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c06b470 con 0x7f4a2c0c6900 2023-12-19T07:48:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c06be40 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.963+0000 7f4a30944700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c06e110 con 0x7f4a2c15cc20 2023-12-19T07:48:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44820/0 (socket says 172.21.15.111:44820) 2023-12-19T07:48:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 -- 172.21.15.111:0/3914261118 learned_addr learned my addr 172.21.15.111:0/3914261118 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a2b59e700 1 --2- 172.21.15.111:0/3914261118 >> v2:172.21.15.143:3300/0 conn(0x7f4a2c15cc20 0x7f4a2c15d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 -- 172.21.15.111:0/3914261118 >> v2:172.21.15.143:3300/0 conn(0x7f4a2c15cc20 msgr2=0x7f4a2c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 --2- 172.21.15.111:0/3914261118 >> v2:172.21.15.143:3300/0 conn(0x7f4a2c15cc20 0x7f4a2c15d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 -- 172.21.15.111:0/3914261118 >> v1:172.21.15.143:6789/0 conn(0x7f4a2c0c6900 legacy=0x7f4a2c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 -- 172.21.15.111:0/3914261118 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c161d10 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a23fff700 1 --2- 172.21.15.111:0/3914261118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0c5c50 tx=0x7f4a14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a64d27b38ede44 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a2a59c700 1 -- 172.21.15.111:0/3914261118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14009e70 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a2a59c700 1 -- 172.21.15.111:0/3914261118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1400b860 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a2a59c700 1 -- 172.21.15.111:0/3914261118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400bc00 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 -- 172.21.15.111:0/3914261118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6cd0 msgr2=0x7f4a2c06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 --2- 172.21.15.111:0/3914261118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0c5c50 tx=0x7f4a14009250 comp rx=0 tx=0).stop 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 -- 172.21.15.111:0/3914261118 shutdown_connections 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 --2- 172.21.15.111:0/3914261118 >> v2:172.21.15.143:3300/0 conn(0x7f4a2c15cc20 0x7f4a2c15d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 --2- 172.21.15.111:0/3914261118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6cd0 0x7f4a2c06b900 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 -- 172.21.15.111:0/3914261118 >> 172.21.15.111:0/3914261118 conn(0x7f4a2c067bb0 msgr2=0x7f4a2c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 -- 172.21.15.111:0/3914261118 shutdown_connections 2023-12-19T07:48:29.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.967+0000 7f4a30944700 1 -- 172.21.15.111:0/3914261118 wait complete. 2023-12-19T07:48:29.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 Processor -- start 2023-12-19T07:48:29.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 -- start start 2023-12-19T07:48:29.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c6cd0 0x7f4a2c0c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c15cc20 0x7f4a2c15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0c0fd0 con 0x7f4a2c0c6900 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c16ed50 con 0x7f4a2c15cc20 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a30944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c16f5b0 con 0x7f4a2c0c6cd0 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a2ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a2ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44826/0 (socket says 172.21.15.111:44826) 2023-12-19T07:48:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a2ad9d700 1 -- 172.21.15.111:0/3754718844 learned_addr learned my addr 172.21.15.111:0/3754718844 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a23fff700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c6cd0 0x7f4a2c0c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.971+0000 7f4a2b59e700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c15cc20 0x7f4a2c15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 -- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c6cd0 msgr2=0x7f4a2c0c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c6cd0 0x7f4a2c0c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 -- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c15cc20 msgr2=0x7f4a2c15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c15cc20 0x7f4a2c15abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 -- 172.21.15.111:0/3754718844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a140094f0 con 0x7f4a2c0c6900 2023-12-19T07:48:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a2ad9d700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c590 tx=0x7f4a1c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c011590 con 0x7f4a2c0c6900 2023-12-19T07:48:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c011710 con 0x7f4a2c0c6900 2023-12-19T07:48:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c011ad0 con 0x7f4a2c0c6900 2023-12-19T07:48:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c0c0cf0 con 0x7f4a2c0c6900 2023-12-19T07:48:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c16f240 con 0x7f4a2c0c6900 2023-12-19T07:48:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.975+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c0c5310 con 0x7f4a2c0c6900 2023-12-19T07:48:29.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.979+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4a1c01b3b0 con 0x7f4a2c0c6900 2023-12-19T07:48:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.979+0000 7f4a22ffd700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 0x7f4a0c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.979+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4a1c07be10 con 0x7f4a2c0c6900 2023-12-19T07:48:29.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.979+0000 7f4a23fff700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 0x7f4a0c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:29.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.983+0000 7f4a23fff700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 0x7f4a0c062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0c5c50 tx=0x7f4a14017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:29.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:29.987+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4a1c07f030 con 0x7f4a2c0c6900 2023-12-19T07:48:30.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.155+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a2c04be40 con 0x7f4a2c0c6900 2023-12-19T07:48:30.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.155+0000 7f4a22ffd700 1 -- 172.21.15.111:0/3754718844 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+15603 (secure 0 0 0) 0x7f4a1c017090 con 0x7f4a2c0c6900 2023-12-19T07:48:30.141 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:30.141 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:48:27.310685+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 msgr2=0x7f4a0c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 0x7f4a0c062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0c5c50 tx=0x7f4a14017000 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 msgr2=0x7f4a2c16e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c590 tx=0x7f4a1c00c8a0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 shutdown_connections 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4a0c060550 0x7f4a0c062a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c15cc20 0x7f4a2c15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c6cd0 0x7f4a2c0c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 --2- 172.21.15.111:0/3754718844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a2c0c6900 0x7f4a2c16e270 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 >> 172.21.15.111:0/3754718844 conn(0x7f4a2c067bb0 msgr2=0x7f4a2c0ac010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 shutdown_connections 2023-12-19T07:48:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.159+0000 7f4a30944700 1 -- 172.21.15.111:0/3754718844 wait complete. 2023-12-19T07:48:30.169 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T07:48:30.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:48:30.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.343+0000 7fbb73945700 1 Processor -- start 2023-12-19T07:48:30.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 -- start start 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb6c0c15b0 0x7fbb6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c0aeba0 0x7fbb6c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0c7190 con 0x7fbb6c15d320 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c0aefd0 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb73945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0c1b50 con 0x7fbb6c0aeba0 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb70ee0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb6c0c15b0 0x7fbb6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb716e1700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbb6c15d320 0x7fbb6c06d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:43408/0 (socket says 172.21.15.111:43408) 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb716e1700 1 -- 172.21.15.111:0/3523146158 learned_addr learned my addr 172.21.15.111:0/3523146158 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.347+0000 7fbb71ee2700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c0aeba0 0x7fbb6c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb70ee0700 1 -- 172.21.15.111:0/3523146158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c0aeba0 msgr2=0x7fbb6c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:30.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb70ee0700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c0aeba0 0x7fbb6c0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb70ee0700 1 -- 172.21.15.111:0/3523146158 >> v1:172.21.15.143:6789/0 conn(0x7fbb6c15d320 legacy=0x7fbb6c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb70ee0700 1 -- 172.21.15.111:0/3523146158 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0c1e30 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb70ee0700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c15d6f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0098e0 tx=0x7fbb5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e6b81da2ea25c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb63fff700 1 -- 172.21.15.111:0/3523146158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00b640 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb63fff700 1 -- 172.21.15.111:0/3523146158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c00b7a0 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.351+0000 7fbb63fff700 1 -- 172.21.15.111:0/3523146158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00bb40 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- 172.21.15.111:0/3523146158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 msgr2=0x7fbb6c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c15d6f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0098e0 tx=0x7fbb5c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- 172.21.15.111:0/3523146158 shutdown_connections 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c0aeba0 0x7fbb6c0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 --2- 172.21.15.111:0/3523146158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c15d6f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- 172.21.15.111:0/3523146158 >> 172.21.15.111:0/3523146158 conn(0x7fbb6c0c8850 msgr2=0x7fbb6c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- 172.21.15.111:0/3523146158 shutdown_connections 2023-12-19T07:48:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- 172.21.15.111:0/3523146158 wait complete. 2023-12-19T07:48:30.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 Processor -- start 2023-12-19T07:48:30.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.355+0000 7fbb73945700 1 -- start start 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb6c0aeba0 0x7fbb6c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c0cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c158700 con 0x7fbb6c0c15b0 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0b02c0 con 0x7fbb6c0aeba0 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c0b05a0 con 0x7fbb6c15d320 2023-12-19T07:48:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb70ee0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c0cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:55914/0 (socket says 172.21.15.111:55914) 2023-12-19T07:48:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 -- 172.21.15.111:0/1141223022 learned_addr learned my addr 172.21.15.111:0/1141223022 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb716e1700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb6c0aeba0 0x7fbb6c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 -- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb6c0aeba0 msgr2=0x7fbb6c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb6c0aeba0 0x7fbb6c1581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 -- 172.21.15.111:0/1141223022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 msgr2=0x7fbb6c0cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c0cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 -- 172.21.15.111:0/1141223022 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c009420 con 0x7fbb6c15d320 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb71ee2700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009df0 tx=0x7fbb6400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6400caa0 con 0x7fbb6c15d320 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6400f040 con 0x7fbb6c15d320 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c0b0880 con 0x7fbb6c15d320 2023-12-19T07:48:30.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6400ce70 con 0x7fbb6c15d320 2023-12-19T07:48:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.359+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0b0e40 con 0x7fbb6c15d320 2023-12-19T07:48:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.363+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c0b1030 con 0x7fbb6c15d320 2023-12-19T07:48:30.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.363+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbb64014690 con 0x7fbb6c15d320 2023-12-19T07:48:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.367+0000 7fbb617fa700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 0x7fbb50062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.367+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fbb6407d1a0 con 0x7fbb6c15d320 2023-12-19T07:48:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.367+0000 7fbb716e1700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 0x7fbb50062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.367+0000 7fbb716e1700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 0x7fbb50062a30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbb68004fc0 tx=0x7fbb68007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.375+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbb64051040 con 0x7fbb6c15d320 2023-12-19T07:48:30.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.547+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb6c071120 con 0x7fbb50060580 2023-12-19T07:48:30.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.567+0000 7fbb617fa700 1 -- 172.21.15.111:0/1141223022 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286152 (secure 0 0 0) 0x7fbb6c071120 con 0x7fbb50060580 2023-12-19T07:48:30.557 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 msgr2=0x7fbb50062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 0x7fbb50062a30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbb68004fc0 tx=0x7fbb68007eb0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 msgr2=0x7fbb6c0cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009df0 tx=0x7fbb6400b040 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 shutdown_connections 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbb50060580 0x7fbb50062a30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb6c15d320 0x7fbb6c0cb360 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb6c0c15b0 0x7fbb6c0cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 --2- 172.21.15.111:0/1141223022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb6c0aeba0 0x7fbb6c1581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.571+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 >> 172.21.15.111:0/1141223022 conn(0x7fbb6c0c8850 msgr2=0x7fbb6c0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.575+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 shutdown_connections 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.575+0000 7fbb73945700 1 -- 172.21.15.111:0/1141223022 wait complete. 2023-12-19T07:48:30.560 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:48:30.578 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-12-19T07:48:28.635605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382928,"kb_used_data":4624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591792,"statfs":{"total":773094113280,"available":770653995008,"internally_reserved":0,"allocated":4734976,"data_stored":2690495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.355713"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338949+0000","last_change":"2023-12-19T07:48:26.338949+0000","last_active":"2023-12-19T07:48:26.338949+0000","last_peered":"2023-12-19T07:48:26.338949+0000","last_clean":"2023-12-19T07:48:26.338949+0000","last_became_active":"2023-12-19T07:48:26.338453+0000","last_became_peered":"2023-12-19T07:48:26.338453+0000","last_unstale":"2023-12-19T07:48:26.338949+0000","last_undegraded":"2023-12-19T07:48:26.338949+0000","last_fullsized":"2023-12-19T07:48:26.338949+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:40:37.176996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.322646+0000","last_change":"2023-12-19T07:48:26.322646+0000","last_active":"2023-12-19T07:48:26.322646+0000","last_peered":"2023-12-19T07:48:26.322646+0000","last_clean":"2023-12-19T07:48:26.322646+0000","last_became_active":"2023-12-19T07:48:26.322220+0000","last_became_peered":"2023-12-19T07:48:26.322220+0000","last_unstale":"2023-12-19T07:48:26.322646+0000","last_undegraded":"2023-12-19T07:48:26.322646+0000","last_fullsized":"2023-12-19T07:48:26.322646+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:51.265875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335408+0000","last_change":"2023-12-19T07:48:26.335408+0000","last_active":"2023-12-19T07:48:26.335408+0000","last_peered":"2023-12-19T07:48:26.335408+0000","last_clean":"2023-12-19T07:48:26.335408+0000","last_became_active":"2023-12-19T07:48:26.335070+0000","last_became_peered":"2023-12-19T07:48:26.335070+0000","last_unstale":"2023-12-19T07:48:26.335408+0000","last_undegraded":"2023-12-19T07:48:26.335408+0000","last_fullsized":"2023-12-19T07:48:26.335408+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333968+0000","last_change":"2023-12-19T07:48:26.333968+0000","last_active":"2023-12-19T07:48:26.333968+0000","last_peered":"2023-12-19T07:48:26.333968+0000","last_clean":"2023-12-19T07:48:26.333968+0000","last_became_active":"2023-12-19T07:48:26.333576+0000","last_became_peered":"2023-12-19T07:48:26.333576+0000","last_unstale":"2023-12-19T07:48:26.333968+0000","last_undegraded":"2023-12-19T07:48:26.333968+0000","last_fullsized":"2023-12-19T07:48:26.333968+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:16:25.760635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332639+0000","last_change":"2023-12-19T07:48:26.332639+0000","last_active":"2023-12-19T07:48:26.332639+0000","last_peered":"2023-12-19T07:48:26.332639+0000","last_clean":"2023-12-19T07:48:26.332639+0000","last_became_active":"2023-12-19T07:48:26.332377+0000","last_became_peered":"2023-12-19T07:48:26.332377+0000","last_unstale":"2023-12-19T07:48:26.332639+0000","last_undegraded":"2023-12-19T07:48:26.332639+0000","last_fullsized":"2023-12-19T07:48:26.332639+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:52.550119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.313912+0000","last_change":"2023-12-19T07:48:26.313912+0000","last_active":"2023-12-19T07:48:26.313912+0000","last_peered":"2023-12-19T07:48:26.313912+0000","last_clean":"2023-12-19T07:48:26.313912+0000","last_became_active":"2023-12-19T07:48:26.313587+0000","last_became_peered":"2023-12-19T07:48:26.313587+0000","last_unstale":"2023-12-19T07:48:26.313912+0000","last_undegraded":"2023-12-19T07:48:26.313912+0000","last_fullsized":"2023-12-19T07:48:26.313912+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330580+0000","last_change":"2023-12-19T07:48:26.330580+0000","last_active":"2023-12-19T07:48:26.330580+0000","last_peered":"2023-12-19T07:48:26.330580+0000","last_clean":"2023-12-19T07:48:26.330580+0000","last_became_active":"2023-12-19T07:48:26.330235+0000","last_became_peered":"2023-12-19T07:48:26.330235+0000","last_unstale":"2023-12-19T07:48:26.330580+0000","last_undegraded":"2023-12-19T07:48:26.330580+0000","last_fullsized":"2023-12-19T07:48:26.330580+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332731+0000","last_change":"2023-12-19T07:48:26.332731+0000","last_active":"2023-12-19T07:48:26.332731+0000","last_peered":"2023-12-19T07:48:26.332731+0000","last_clean":"2023-12-19T07:48:26.332731+0000","last_became_active":"2023-12-19T07:48:26.332505+0000","last_became_peered":"2023-12-19T07:48:26.332505+0000","last_unstale":"2023-12-19T07:48:26.332731+0000","last_undegraded":"2023-12-19T07:48:26.332731+0000","last_fullsized":"2023-12-19T07:48:26.332731+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:01:15.851921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336389+0000","last_change":"2023-12-19T07:48:26.336389+0000","last_active":"2023-12-19T07:48:26.336389+0000","last_peered":"2023-12-19T07:48:26.336389+0000","last_clean":"2023-12-19T07:48:26.336389+0000","last_became_active":"2023-12-19T07:48:26.335600+0000","last_became_peered":"2023-12-19T07:48:26.335600+0000","last_unstale":"2023-12-19T07:48:26.336389+0000","last_undegraded":"2023-12-19T07:48:26.336389+0000","last_fullsized":"2023-12-19T07:48:26.336389+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335726+0000","last_change":"2023-12-19T07:48:26.335726+0000","last_active":"2023-12-19T07:48:26.335726+0000","last_peered":"2023-12-19T07:48:26.335726+0000","last_clean":"2023-12-19T07:48:26.335726+0000","last_became_active":"2023-12-19T07:48:26.332605+0000","last_became_peered":"2023-12-19T07:48:26.332605+0000","last_unstale":"2023-12-19T07:48:26.335726+0000","last_undegraded":"2023-12-19T07:48:26.335726+0000","last_fullsized":"2023-12-19T07:48:26.335726+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:21:34.779152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330824+0000","last_change":"2023-12-19T07:48:26.330824+0000","last_active":"2023-12-19T07:48:26.330824+0000","last_peered":"2023-12-19T07:48:26.330824+0000","last_clean":"2023-12-19T07:48:26.330824+0000","last_became_active":"2023-12-19T07:48:26.328335+0000","last_became_peered":"2023-12-19T07:48:26.328335+0000","last_unstale":"2023-12-19T07:48:26.330824+0000","last_undegraded":"2023-12-19T07:48:26.330824+0000","last_fullsized":"2023-12-19T07:48:26.330824+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:20.894146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333234+0000","last_change":"2023-12-19T07:48:26.333234+0000","last_active":"2023-12-19T07:48:26.333234+0000","last_peered":"2023-12-19T07:48:26.333234+0000","last_clean":"2023-12-19T07:48:26.333234+0000","last_became_active":"2023-12-19T07:48:26.332992+0000","last_became_peered":"2023-12-19T07:48:26.332992+0000","last_unstale":"2023-12-19T07:48:26.333234+0000","last_undegraded":"2023-12-19T07:48:26.333234+0000","last_fullsized":"2023-12-19T07:48:26.333234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343411+0000","last_change":"2023-12-19T07:48:26.343411+0000","last_active":"2023-12-19T07:48:26.343411+0000","last_peered":"2023-12-19T07:48:26.343411+0000","last_clean":"2023-12-19T07:48:26.343411+0000","last_became_active":"2023-12-19T07:48:26.343153+0000","last_became_peered":"2023-12-19T07:48:26.343153+0000","last_unstale":"2023-12-19T07:48:26.343411+0000","last_undegraded":"2023-12-19T07:48:26.343411+0000","last_fullsized":"2023-12-19T07:48:26.343411+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:35:21.377012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339119+0000","last_change":"2023-12-19T07:48:26.339119+0000","last_active":"2023-12-19T07:48:26.339119+0000","last_peered":"2023-12-19T07:48:26.339119+0000","last_clean":"2023-12-19T07:48:26.339119+0000","last_became_active":"2023-12-19T07:48:26.338827+0000","last_became_peered":"2023-12-19T07:48:26.338827+0000","last_unstale":"2023-12-19T07:48:26.339119+0000","last_undegraded":"2023-12-19T07:48:26.339119+0000","last_fullsized":"2023-12-19T07:48:26.339119+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:21:34.779152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343632+0000","last_change":"2023-12-19T07:48:26.343632+0000","last_active":"2023-12-19T07:48:26.343632+0000","last_peered":"2023-12-19T07:48:26.343632+0000","last_clean":"2023-12-19T07:48:26.343632+0000","last_became_active":"2023-12-19T07:48:26.337642+0000","last_became_peered":"2023-12-19T07:48:26.337642+0000","last_unstale":"2023-12-19T07:48:26.343632+0000","last_undegraded":"2023-12-19T07:48:26.343632+0000","last_fullsized":"2023-12-19T07:48:26.343632+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336605+0000","last_change":"2023-12-19T07:48:26.336605+0000","last_active":"2023-12-19T07:48:26.336605+0000","last_peered":"2023-12-19T07:48:26.336605+0000","last_clean":"2023-12-19T07:48:26.336605+0000","last_became_active":"2023-12-19T07:48:26.335279+0000","last_became_peered":"2023-12-19T07:48:26.335279+0000","last_unstale":"2023-12-19T07:48:26.336605+0000","last_undegraded":"2023-12-19T07:48:26.336605+0000","last_fullsized":"2023-12-19T07:48:26.336605+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:54.963432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336300+0000","last_change":"2023-12-19T07:48:26.336300+0000","last_active":"2023-12-19T07:48:26.336300+0000","last_peered":"2023-12-19T07:48:26.336300+0000","last_clean":"2023-12-19T07:48:26.336300+0000","last_became_active":"2023-12-19T07:48:26.336039+0000","last_became_peered":"2023-12-19T07:48:26.336039+0000","last_unstale":"2023-12-19T07:48:26.336300+0000","last_undegraded":"2023-12-19T07:48:26.336300+0000","last_fullsized":"2023-12-19T07:48:26.336300+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:00.664922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338475+0000","last_change":"2023-12-19T07:48:26.338475+0000","last_active":"2023-12-19T07:48:26.338475+0000","last_peered":"2023-12-19T07:48:26.338475+0000","last_clean":"2023-12-19T07:48:26.338475+0000","last_became_active":"2023-12-19T07:48:26.338197+0000","last_became_peered":"2023-12-19T07:48:26.338197+0000","last_unstale":"2023-12-19T07:48:26.338475+0000","last_undegraded":"2023-12-19T07:48:26.338475+0000","last_fullsized":"2023-12-19T07:48:26.338475+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:53.490385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320407+0000","last_change":"2023-12-19T07:48:26.320407+0000","last_active":"2023-12-19T07:48:26.320407+0000","last_peered":"2023-12-19T07:48:26.320407+0000","last_clean":"2023-12-19T07:48:26.320407+0000","last_became_active":"2023-12-19T07:48:26.319941+0000","last_became_peered":"2023-12-19T07:48:26.319941+0000","last_unstale":"2023-12-19T07:48:26.320407+0000","last_undegraded":"2023-12-19T07:48:26.320407+0000","last_fullsized":"2023-12-19T07:48:26.320407+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:49:22.548670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343816+0000","last_change":"2023-12-19T07:48:26.343816+0000","last_active":"2023-12-19T07:48:26.343816+0000","last_peered":"2023-12-19T07:48:26.343816+0000","last_clean":"2023-12-19T07:48:26.343816+0000","last_became_active":"2023-12-19T07:48:26.338302+0000","last_became_peered":"2023-12-19T07:48:26.338302+0000","last_unstale":"2023-12-19T07:48:26.343816+0000","last_undegraded":"2023-12-19T07:48:26.343816+0000","last_fullsized":"2023-12-19T07:48:26.343816+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339543+0000","last_change":"2023-12-19T07:48:26.339543+0000","last_active":"2023-12-19T07:48:26.339543+0000","last_peered":"2023-12-19T07:48:26.339543+0000","last_clean":"2023-12-19T07:48:26.339543+0000","last_became_active":"2023-12-19T07:48:26.339266+0000","last_became_peered":"2023-12-19T07:48:26.339266+0000","last_unstale":"2023-12-19T07:48:26.339543+0000","last_undegraded":"2023-12-19T07:48:26.339543+0000","last_fullsized":"2023-12-19T07:48:26.339543+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:21.593223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.329305+0000","last_change":"2023-12-19T07:48:26.329305+0000","last_active":"2023-12-19T07:48:26.329305+0000","last_peered":"2023-12-19T07:48:26.329305+0000","last_clean":"2023-12-19T07:48:26.329305+0000","last_became_active":"2023-12-19T07:48:26.328930+0000","last_became_peered":"2023-12-19T07:48:26.328930+0000","last_unstale":"2023-12-19T07:48:26.329305+0000","last_undegraded":"2023-12-19T07:48:26.329305+0000","last_fullsized":"2023-12-19T07:48:26.329305+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320275+0000","last_change":"2023-12-19T07:48:26.320275+0000","last_active":"2023-12-19T07:48:26.320275+0000","last_peered":"2023-12-19T07:48:26.320275+0000","last_clean":"2023-12-19T07:48:26.320275+0000","last_became_active":"2023-12-19T07:48:26.319953+0000","last_became_peered":"2023-12-19T07:48:26.319953+0000","last_unstale":"2023-12-19T07:48:26.320275+0000","last_undegraded":"2023-12-19T07:48:26.320275+0000","last_fullsized":"2023-12-19T07:48:26.320275+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320639+0000","last_change":"2023-12-19T07:48:26.320639+0000","last_active":"2023-12-19T07:48:26.320639+0000","last_peered":"2023-12-19T07:48:26.320639+0000","last_clean":"2023-12-19T07:48:26.320639+0000","last_became_active":"2023-12-19T07:48:26.320337+0000","last_became_peered":"2023-12-19T07:48:26.320337+0000","last_unstale":"2023-12-19T07:48:26.320639+0000","last_undegraded":"2023-12-19T07:48:26.320639+0000","last_fullsized":"2023-12-19T07:48:26.320639+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:15.201218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343830+0000","last_change":"2023-12-19T07:48:26.343830+0000","last_active":"2023-12-19T07:48:26.343830+0000","last_peered":"2023-12-19T07:48:26.343830+0000","last_clean":"2023-12-19T07:48:26.343830+0000","last_became_active":"2023-12-19T07:48:26.343424+0000","last_became_peered":"2023-12-19T07:48:26.343424+0000","last_unstale":"2023-12-19T07:48:26.343830+0000","last_undegraded":"2023-12-19T07:48:26.343830+0000","last_fullsized":"2023-12-19T07:48:26.343830+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:32.856939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.328931+0000","last_change":"2023-12-19T07:48:26.328931+0000","last_active":"2023-12-19T07:48:26.328931+0000","last_peered":"2023-12-19T07:48:26.328931+0000","last_clean":"2023-12-19T07:48:26.328931+0000","last_became_active":"2023-12-19T07:48:26.328649+0000","last_became_peered":"2023-12-19T07:48:26.328649+0000","last_unstale":"2023-12-19T07:48:26.328931+0000","last_undegraded":"2023-12-19T07:48:26.328931+0000","last_fullsized":"2023-12-19T07:48:26.328931+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:23.623245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339633+0000","last_change":"2023-12-19T07:48:26.339633+0000","last_active":"2023-12-19T07:48:26.339633+0000","last_peered":"2023-12-19T07:48:26.339633+0000","last_clean":"2023-12-19T07:48:26.339633+0000","last_became_active":"2023-12-19T07:48:26.339354+0000","last_became_peered":"2023-12-19T07:48:26.339354+0000","last_unstale":"2023-12-19T07:48:26.339633+0000","last_undegraded":"2023-12-19T07:48:26.339633+0000","last_fullsized":"2023-12-19T07:48:26.339633+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:18.417615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333909+0000","last_change":"2023-12-19T07:48:26.333909+0000","last_active":"2023-12-19T07:48:26.333909+0000","last_peered":"2023-12-19T07:48:26.333909+0000","last_clean":"2023-12-19T07:48:26.333909+0000","last_became_active":"2023-12-19T07:48:26.332177+0000","last_became_peered":"2023-12-19T07:48:26.332177+0000","last_unstale":"2023-12-19T07:48:26.333909+0000","last_undegraded":"2023-12-19T07:48:26.333909+0000","last_fullsized":"2023-12-19T07:48:26.333909+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.340766+0000","last_change":"2023-12-19T07:48:26.340766+0000","last_active":"2023-12-19T07:48:26.340766+0000","last_peered":"2023-12-19T07:48:26.340766+0000","last_clean":"2023-12-19T07:48:26.340766+0000","last_became_active":"2023-12-19T07:48:26.340465+0000","last_became_peered":"2023-12-19T07:48:26.340465+0000","last_unstale":"2023-12-19T07:48:26.340766+0000","last_undegraded":"2023-12-19T07:48:26.340766+0000","last_fullsized":"2023-12-19T07:48:26.340766+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.342664+0000","last_change":"2023-12-19T07:48:26.342664+0000","last_active":"2023-12-19T07:48:26.342664+0000","last_peered":"2023-12-19T07:48:26.342664+0000","last_clean":"2023-12-19T07:48:26.342664+0000","last_became_active":"2023-12-19T07:48:26.341935+0000","last_became_peered":"2023-12-19T07:48:26.341935+0000","last_unstale":"2023-12-19T07:48:26.342664+0000","last_undegraded":"2023-12-19T07:48:26.342664+0000","last_fullsized":"2023-12-19T07:48:26.342664+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.323761+0000","last_change":"2023-12-19T07:48:26.323761+0000","last_active":"2023-12-19T07:48:26.323761+0000","last_peered":"2023-12-19T07:48:26.323761+0000","last_clean":"2023-12-19T07:48:26.323761+0000","last_became_active":"2023-12-19T07:48:26.323455+0000","last_became_peered":"2023-12-19T07:48:26.323455+0000","last_unstale":"2023-12-19T07:48:26.323761+0000","last_undegraded":"2023-12-19T07:48:26.323761+0000","last_fullsized":"2023-12-19T07:48:26.323761+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326617+0000","last_change":"2023-12-19T07:48:26.326617+0000","last_active":"2023-12-19T07:48:26.326617+0000","last_peered":"2023-12-19T07:48:26.326617+0000","last_clean":"2023-12-19T07:48:26.326617+0000","last_became_active":"2023-12-19T07:48:26.323631+0000","last_became_peered":"2023-12-19T07:48:26.323631+0000","last_unstale":"2023-12-19T07:48:26.326617+0000","last_undegraded":"2023-12-19T07:48:26.326617+0000","last_fullsized":"2023-12-19T07:48:26.326617+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335504+0000","last_change":"2023-12-19T07:48:26.335504+0000","last_active":"2023-12-19T07:48:26.335504+0000","last_peered":"2023-12-19T07:48:26.335504+0000","last_clean":"2023-12-19T07:48:26.335504+0000","last_became_active":"2023-12-19T07:48:26.335210+0000","last_became_peered":"2023-12-19T07:48:26.335210+0000","last_unstale":"2023-12-19T07:48:26.335504+0000","last_undegraded":"2023-12-19T07:48:26.335504+0000","last_fullsized":"2023-12-19T07:48:26.335504+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:09.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.344917+0000","last_change":"2023-12-19T07:48:25.323929+0000","last_active":"2023-12-19T07:48:26.344917+0000","last_peered":"2023-12-19T07:48:26.344917+0000","last_clean":"2023-12-19T07:48:26.344917+0000","last_became_active":"2023-12-19T07:48:25.323601+0000","last_became_peered":"2023-12-19T07:48:25.323601+0000","last_unstale":"2023-12-19T07:48:26.344917+0000","last_undegraded":"2023-12-19T07:48:26.344917+0000","last_fullsized":"2023-12-19T07:48:26.344917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:14.044929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338249+0000","last_change":"2023-12-19T07:48:26.338249+0000","last_active":"2023-12-19T07:48:26.338249+0000","last_peered":"2023-12-19T07:48:26.338249+0000","last_clean":"2023-12-19T07:48:26.338249+0000","last_became_active":"2023-12-19T07:48:26.337983+0000","last_became_peered":"2023-12-19T07:48:26.337983+0000","last_unstale":"2023-12-19T07:48:26.338249+0000","last_undegraded":"2023-12-19T07:48:26.338249+0000","last_fullsized":"2023-12-19T07:48:26.338249+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320929+0000","last_change":"2023-12-19T07:48:25.323223+0000","last_active":"2023-12-19T07:48:26.320929+0000","last_peered":"2023-12-19T07:48:26.320929+0000","last_clean":"2023-12-19T07:48:26.320929+0000","last_became_active":"2023-12-19T07:48:25.322943+0000","last_became_peered":"2023-12-19T07:48:25.322943+0000","last_unstale":"2023-12-19T07:48:26.320929+0000","last_undegraded":"2023-12-19T07:48:26.320929+0000","last_fullsized":"2023-12-19T07:48:26.320929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:50.370509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.340690+0000","last_change":"2023-12-19T07:48:26.340690+0000","last_active":"2023-12-19T07:48:26.340690+0000","last_peered":"2023-12-19T07:48:26.340690+0000","last_clean":"2023-12-19T07:48:26.340690+0000","last_became_active":"2023-12-19T07:48:26.340402+0000","last_became_peered":"2023-12-19T07:48:26.340402+0000","last_unstale":"2023-12-19T07:48:26.340690+0000","last_undegraded":"2023-12-19T07:48:26.340690+0000","last_fullsized":"2023-12-19T07:48:26.340690+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:20.894146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332430+0000","last_change":"2023-12-19T07:48:25.311015+0000","last_active":"2023-12-19T07:48:26.332430+0000","last_peered":"2023-12-19T07:48:26.332430+0000","last_clean":"2023-12-19T07:48:26.332430+0000","last_became_active":"2023-12-19T07:48:25.310667+0000","last_became_peered":"2023-12-19T07:48:25.310667+0000","last_unstale":"2023-12-19T07:48:26.332430+0000","last_undegraded":"2023-12-19T07:48:26.332430+0000","last_fullsized":"2023-12-19T07:48:26.332430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:38.234163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341252+0000","last_change":"2023-12-19T07:48:26.341252+0000","last_active":"2023-12-19T07:48:26.341252+0000","last_peered":"2023-12-19T07:48:26.341252+0000","last_clean":"2023-12-19T07:48:26.341252+0000","last_became_active":"2023-12-19T07:48:26.340914+0000","last_became_peered":"2023-12-19T07:48:26.340914+0000","last_unstale":"2023-12-19T07:48:26.341252+0000","last_undegraded":"2023-12-19T07:48:26.341252+0000","last_fullsized":"2023-12-19T07:48:26.341252+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:15:17.506583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.329182+0000","last_change":"2023-12-19T07:48:25.320384+0000","last_active":"2023-12-19T07:48:26.329182+0000","last_peered":"2023-12-19T07:48:26.329182+0000","last_clean":"2023-12-19T07:48:26.329182+0000","last_became_active":"2023-12-19T07:48:25.320115+0000","last_became_peered":"2023-12-19T07:48:25.320115+0000","last_unstale":"2023-12-19T07:48:26.329182+0000","last_undegraded":"2023-12-19T07:48:26.329182+0000","last_fullsized":"2023-12-19T07:48:26.329182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:52.474691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331874+0000","last_change":"2023-12-19T07:48:26.331874+0000","last_active":"2023-12-19T07:48:26.331874+0000","last_peered":"2023-12-19T07:48:26.331874+0000","last_clean":"2023-12-19T07:48:26.331874+0000","last_became_active":"2023-12-19T07:48:26.331460+0000","last_became_peered":"2023-12-19T07:48:26.331460+0000","last_unstale":"2023-12-19T07:48:26.331874+0000","last_undegraded":"2023-12-19T07:48:26.331874+0000","last_fullsized":"2023-12-19T07:48:26.331874+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:09.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335182+0000","last_change":"2023-12-19T07:48:25.325686+0000","last_active":"2023-12-19T07:48:26.335182+0000","last_peered":"2023-12-19T07:48:26.335182+0000","last_clean":"2023-12-19T07:48:26.335182+0000","last_became_active":"2023-12-19T07:48:25.325324+0000","last_became_peered":"2023-12-19T07:48:25.325324+0000","last_unstale":"2023-12-19T07:48:26.335182+0000","last_undegraded":"2023-12-19T07:48:26.335182+0000","last_fullsized":"2023-12-19T07:48:26.335182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:29.482918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330466+0000","last_change":"2023-12-19T07:48:26.330466+0000","last_active":"2023-12-19T07:48:26.330466+0000","last_peered":"2023-12-19T07:48:26.330466+0000","last_clean":"2023-12-19T07:48:26.330466+0000","last_became_active":"2023-12-19T07:48:26.330171+0000","last_became_peered":"2023-12-19T07:48:26.330171+0000","last_unstale":"2023-12-19T07:48:26.330466+0000","last_undegraded":"2023-12-19T07:48:26.330466+0000","last_fullsized":"2023-12-19T07:48:26.330466+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.307263+0000","last_change":"2023-12-19T07:48:25.332216+0000","last_active":"2023-12-19T07:48:26.307263+0000","last_peered":"2023-12-19T07:48:26.307263+0000","last_clean":"2023-12-19T07:48:26.307263+0000","last_became_active":"2023-12-19T07:48:25.331890+0000","last_became_peered":"2023-12-19T07:48:25.331890+0000","last_unstale":"2023-12-19T07:48:26.307263+0000","last_undegraded":"2023-12-19T07:48:26.307263+0000","last_fullsized":"2023-12-19T07:48:26.307263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:14.185525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.334980+0000","last_change":"2023-12-19T07:48:26.334980+0000","last_active":"2023-12-19T07:48:26.334980+0000","last_peered":"2023-12-19T07:48:26.334980+0000","last_clean":"2023-12-19T07:48:26.334980+0000","last_became_active":"2023-12-19T07:48:26.334709+0000","last_became_peered":"2023-12-19T07:48:26.334709+0000","last_unstale":"2023-12-19T07:48:26.334980+0000","last_undegraded":"2023-12-19T07:48:26.334980+0000","last_fullsized":"2023-12-19T07:48:26.334980+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:21.593223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.315899+0000","last_change":"2023-12-19T07:48:25.311418+0000","last_active":"2023-12-19T07:48:25.315899+0000","last_peered":"2023-12-19T07:48:25.315899+0000","last_clean":"2023-12-19T07:48:25.315899+0000","last_became_active":"2023-12-19T07:48:25.311156+0000","last_became_peered":"2023-12-19T07:48:25.311156+0000","last_unstale":"2023-12-19T07:48:25.315899+0000","last_undegraded":"2023-12-19T07:48:25.315899+0000","last_fullsized":"2023-12-19T07:48:25.315899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:17.401922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.319768+0000","last_change":"2023-12-19T07:48:26.319768+0000","last_active":"2023-12-19T07:48:26.319768+0000","last_peered":"2023-12-19T07:48:26.319768+0000","last_clean":"2023-12-19T07:48:26.319768+0000","last_became_active":"2023-12-19T07:48:26.319473+0000","last_became_peered":"2023-12-19T07:48:26.319473+0000","last_unstale":"2023-12-19T07:48:26.319768+0000","last_undegraded":"2023-12-19T07:48:26.319768+0000","last_fullsized":"2023-12-19T07:48:26.319768+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:36:16.492768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.324309+0000","last_change":"2023-12-19T07:48:25.318027+0000","last_active":"2023-12-19T07:48:25.324309+0000","last_peered":"2023-12-19T07:48:25.324309+0000","last_clean":"2023-12-19T07:48:25.324309+0000","last_became_active":"2023-12-19T07:48:25.317733+0000","last_became_peered":"2023-12-19T07:48:25.317733+0000","last_unstale":"2023-12-19T07:48:25.324309+0000","last_undegraded":"2023-12-19T07:48:25.324309+0000","last_fullsized":"2023-12-19T07:48:25.324309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:43.219229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333174+0000","last_change":"2023-12-19T07:48:26.333174+0000","last_active":"2023-12-19T07:48:26.333174+0000","last_peered":"2023-12-19T07:48:26.333174+0000","last_clean":"2023-12-19T07:48:26.333174+0000","last_became_active":"2023-12-19T07:48:26.332896+0000","last_became_peered":"2023-12-19T07:48:26.332896+0000","last_unstale":"2023-12-19T07:48:26.333174+0000","last_undegraded":"2023-12-19T07:48:26.333174+0000","last_fullsized":"2023-12-19T07:48:26.333174+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:41:30.791474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.321945+0000","last_change":"2023-12-19T07:48:25.318295+0000","last_active":"2023-12-19T07:48:25.321945+0000","last_peered":"2023-12-19T07:48:25.321945+0000","last_clean":"2023-12-19T07:48:25.321945+0000","last_became_active":"2023-12-19T07:48:25.317975+0000","last_became_peered":"2023-12-19T07:48:25.317975+0000","last_unstale":"2023-12-19T07:48:25.321945+0000","last_undegraded":"2023-12-19T07:48:25.321945+0000","last_fullsized":"2023-12-19T07:48:25.321945+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:19.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333137+0000","last_change":"2023-12-19T07:48:26.333137+0000","last_active":"2023-12-19T07:48:26.333137+0000","last_peered":"2023-12-19T07:48:26.333137+0000","last_clean":"2023-12-19T07:48:26.333137+0000","last_became_active":"2023-12-19T07:48:26.332905+0000","last_became_peered":"2023-12-19T07:48:26.332905+0000","last_unstale":"2023-12-19T07:48:26.333137+0000","last_undegraded":"2023-12-19T07:48:26.333137+0000","last_fullsized":"2023-12-19T07:48:26.333137+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.335633+0000","last_change":"2023-12-19T07:48:25.327715+0000","last_active":"2023-12-19T07:48:25.335633+0000","last_peered":"2023-12-19T07:48:25.335633+0000","last_clean":"2023-12-19T07:48:25.335633+0000","last_became_active":"2023-12-19T07:48:25.327344+0000","last_became_peered":"2023-12-19T07:48:25.327344+0000","last_unstale":"2023-12-19T07:48:25.335633+0000","last_undegraded":"2023-12-19T07:48:25.335633+0000","last_fullsized":"2023-12-19T07:48:25.335633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:34:30.931509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331347+0000","last_change":"2023-12-19T07:48:26.331347+0000","last_active":"2023-12-19T07:48:26.331347+0000","last_peered":"2023-12-19T07:48:26.331347+0000","last_clean":"2023-12-19T07:48:26.331347+0000","last_became_active":"2023-12-19T07:48:26.331063+0000","last_became_peered":"2023-12-19T07:48:26.331063+0000","last_unstale":"2023-12-19T07:48:26.331347+0000","last_undegraded":"2023-12-19T07:48:26.331347+0000","last_fullsized":"2023-12-19T07:48:26.331347+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:23.623245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.347473+0000","last_change":"2023-12-19T07:48:25.329659+0000","last_active":"2023-12-19T07:48:26.347473+0000","last_peered":"2023-12-19T07:48:26.347473+0000","last_clean":"2023-12-19T07:48:26.347473+0000","last_became_active":"2023-12-19T07:48:25.329402+0000","last_became_peered":"2023-12-19T07:48:25.329402+0000","last_unstale":"2023-12-19T07:48:26.347473+0000","last_undegraded":"2023-12-19T07:48:26.347473+0000","last_fullsized":"2023-12-19T07:48:26.347473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:03.366692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.325819+0000","last_change":"2023-12-19T07:48:26.325819+0000","last_active":"2023-12-19T07:48:26.325819+0000","last_peered":"2023-12-19T07:48:26.325819+0000","last_clean":"2023-12-19T07:48:26.325819+0000","last_became_active":"2023-12-19T07:48:26.325493+0000","last_became_peered":"2023-12-19T07:48:26.325493+0000","last_unstale":"2023-12-19T07:48:26.325819+0000","last_undegraded":"2023-12-19T07:48:26.325819+0000","last_fullsized":"2023-12-19T07:48:26.325819+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330839+0000","last_change":"2023-12-19T07:48:25.322692+0000","last_active":"2023-12-19T07:48:26.330839+0000","last_peered":"2023-12-19T07:48:26.330839+0000","last_clean":"2023-12-19T07:48:26.330839+0000","last_became_active":"2023-12-19T07:48:25.322033+0000","last_became_peered":"2023-12-19T07:48:25.322033+0000","last_unstale":"2023-12-19T07:48:26.330839+0000","last_undegraded":"2023-12-19T07:48:26.330839+0000","last_fullsized":"2023-12-19T07:48:26.330839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:53.305160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331762+0000","last_change":"2023-12-19T07:48:26.331762+0000","last_active":"2023-12-19T07:48:26.331762+0000","last_peered":"2023-12-19T07:48:26.331762+0000","last_clean":"2023-12-19T07:48:26.331762+0000","last_became_active":"2023-12-19T07:48:26.331032+0000","last_became_peered":"2023-12-19T07:48:26.331032+0000","last_unstale":"2023-12-19T07:48:26.331762+0000","last_undegraded":"2023-12-19T07:48:26.331762+0000","last_fullsized":"2023-12-19T07:48:26.331762+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:54.963432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331123+0000","last_change":"2023-12-19T07:48:25.314507+0000","last_active":"2023-12-19T07:48:26.331123+0000","last_peered":"2023-12-19T07:48:26.331123+0000","last_clean":"2023-12-19T07:48:26.331123+0000","last_became_active":"2023-12-19T07:48:25.314181+0000","last_became_peered":"2023-12-19T07:48:25.314181+0000","last_unstale":"2023-12-19T07:48:26.331123+0000","last_undegraded":"2023-12-19T07:48:26.331123+0000","last_fullsized":"2023-12-19T07:48:26.331123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:28.249753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330377+0000","last_change":"2023-12-19T07:48:26.330377+0000","last_active":"2023-12-19T07:48:26.330377+0000","last_peered":"2023-12-19T07:48:26.330377+0000","last_clean":"2023-12-19T07:48:26.330377+0000","last_became_active":"2023-12-19T07:48:26.330068+0000","last_became_peered":"2023-12-19T07:48:26.330068+0000","last_unstale":"2023-12-19T07:48:26.330377+0000","last_undegraded":"2023-12-19T07:48:26.330377+0000","last_fullsized":"2023-12-19T07:48:26.330377+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:08.964380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.316092+0000","last_change":"2023-12-19T07:48:25.311733+0000","last_active":"2023-12-19T07:48:25.316092+0000","last_peered":"2023-12-19T07:48:25.316092+0000","last_clean":"2023-12-19T07:48:25.316092+0000","last_became_active":"2023-12-19T07:48:25.311357+0000","last_became_peered":"2023-12-19T07:48:25.311357+0000","last_unstale":"2023-12-19T07:48:25.316092+0000","last_undegraded":"2023-12-19T07:48:25.316092+0000","last_fullsized":"2023-12-19T07:48:25.316092+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:31.841246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326403+0000","last_change":"2023-12-19T07:48:26.326403+0000","last_active":"2023-12-19T07:48:26.326403+0000","last_peered":"2023-12-19T07:48:26.326403+0000","last_clean":"2023-12-19T07:48:26.326403+0000","last_became_active":"2023-12-19T07:48:26.326117+0000","last_became_peered":"2023-12-19T07:48:26.326117+0000","last_unstale":"2023-12-19T07:48:26.326403+0000","last_undegraded":"2023-12-19T07:48:26.326403+0000","last_fullsized":"2023-12-19T07:48:26.326403+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.331150+0000","last_change":"2023-12-19T07:48:25.325731+0000","last_active":"2023-12-19T07:48:25.331150+0000","last_peered":"2023-12-19T07:48:25.331150+0000","last_clean":"2023-12-19T07:48:25.331150+0000","last_became_active":"2023-12-19T07:48:25.325424+0000","last_became_peered":"2023-12-19T07:48:25.325424+0000","last_unstale":"2023-12-19T07:48:25.331150+0000","last_undegraded":"2023-12-19T07:48:25.331150+0000","last_fullsized":"2023-12-19T07:48:25.331150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:14.185525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331579+0000","last_change":"2023-12-19T07:48:26.331579+0000","last_active":"2023-12-19T07:48:26.331579+0000","last_peered":"2023-12-19T07:48:26.331579+0000","last_clean":"2023-12-19T07:48:26.331579+0000","last_became_active":"2023-12-19T07:48:26.331294+0000","last_became_peered":"2023-12-19T07:48:26.331294+0000","last_unstale":"2023-12-19T07:48:26.331579+0000","last_undegraded":"2023-12-19T07:48:26.331579+0000","last_fullsized":"2023-12-19T07:48:26.331579+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:20:56.894788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331496+0000","last_change":"2023-12-19T07:48:25.332417+0000","last_active":"2023-12-19T07:48:26.331496+0000","last_peered":"2023-12-19T07:48:26.331496+0000","last_clean":"2023-12-19T07:48:26.331496+0000","last_became_active":"2023-12-19T07:48:25.332145+0000","last_became_peered":"2023-12-19T07:48:25.332145+0000","last_unstale":"2023-12-19T07:48:26.331496+0000","last_undegraded":"2023-12-19T07:48:26.331496+0000","last_fullsized":"2023-12-19T07:48:26.331496+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:14.044929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341253+0000","last_change":"2023-12-19T07:48:26.341253+0000","last_active":"2023-12-19T07:48:26.341253+0000","last_peered":"2023-12-19T07:48:26.341253+0000","last_clean":"2023-12-19T07:48:26.341253+0000","last_became_active":"2023-12-19T07:48:26.340941+0000","last_became_peered":"2023-12-19T07:48:26.340941+0000","last_unstale":"2023-12-19T07:48:26.341253+0000","last_undegraded":"2023-12-19T07:48:26.341253+0000","last_fullsized":"2023-12-19T07:48:26.341253+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:39:12.625159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339130+0000","last_change":"2023-12-19T07:48:25.324069+0000","last_active":"2023-12-19T07:48:26.339130+0000","last_peered":"2023-12-19T07:48:26.339130+0000","last_clean":"2023-12-19T07:48:26.339130+0000","last_became_active":"2023-12-19T07:48:25.323728+0000","last_became_peered":"2023-12-19T07:48:25.323728+0000","last_unstale":"2023-12-19T07:48:26.339130+0000","last_undegraded":"2023-12-19T07:48:26.339130+0000","last_fullsized":"2023-12-19T07:48:26.339130+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:19:00.406567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326949+0000","last_change":"2023-12-19T07:48:26.326949+0000","last_active":"2023-12-19T07:48:26.326949+0000","last_peered":"2023-12-19T07:48:26.326949+0000","last_clean":"2023-12-19T07:48:26.326949+0000","last_became_active":"2023-12-19T07:48:26.326654+0000","last_became_peered":"2023-12-19T07:48:26.326654+0000","last_unstale":"2023-12-19T07:48:26.326949+0000","last_undegraded":"2023-12-19T07:48:26.326949+0000","last_fullsized":"2023-12-19T07:48:26.326949+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.328671+0000","last_change":"2023-12-19T07:48:25.328973+0000","last_active":"2023-12-19T07:48:26.328671+0000","last_peered":"2023-12-19T07:48:26.328671+0000","last_clean":"2023-12-19T07:48:26.328671+0000","last_became_active":"2023-12-19T07:48:25.328625+0000","last_became_peered":"2023-12-19T07:48:25.328625+0000","last_unstale":"2023-12-19T07:48:26.328671+0000","last_undegraded":"2023-12-19T07:48:26.328671+0000","last_fullsized":"2023-12-19T07:48:26.328671+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:17.401922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324593+0000","last_change":"2023-12-19T07:48:26.324593+0000","last_active":"2023-12-19T07:48:26.324593+0000","last_peered":"2023-12-19T07:48:26.324593+0000","last_clean":"2023-12-19T07:48:26.324593+0000","last_became_active":"2023-12-19T07:48:26.324256+0000","last_became_peered":"2023-12-19T07:48:26.324256+0000","last_unstale":"2023-12-19T07:48:26.324593+0000","last_undegraded":"2023-12-19T07:48:26.324593+0000","last_fullsized":"2023-12-19T07:48:26.324593+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.316266+0000","last_change":"2023-12-19T07:48:25.324855+0000","last_active":"2023-12-19T07:48:26.316266+0000","last_peered":"2023-12-19T07:48:26.316266+0000","last_clean":"2023-12-19T07:48:26.316266+0000","last_became_active":"2023-12-19T07:48:25.324535+0000","last_became_peered":"2023-12-19T07:48:25.324535+0000","last_unstale":"2023-12-19T07:48:26.316266+0000","last_undegraded":"2023-12-19T07:48:26.316266+0000","last_fullsized":"2023-12-19T07:48:26.316266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:40.692055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326150+0000","last_change":"2023-12-19T07:48:26.326150+0000","last_active":"2023-12-19T07:48:26.326150+0000","last_peered":"2023-12-19T07:48:26.326150+0000","last_clean":"2023-12-19T07:48:26.326150+0000","last_became_active":"2023-12-19T07:48:26.325831+0000","last_became_peered":"2023-12-19T07:48:26.325831+0000","last_unstale":"2023-12-19T07:48:26.326150+0000","last_undegraded":"2023-12-19T07:48:26.326150+0000","last_fullsized":"2023-12-19T07:48:26.326150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:15.060622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.316187+0000","last_change":"2023-12-19T07:48:25.312141+0000","last_active":"2023-12-19T07:48:25.316187+0000","last_peered":"2023-12-19T07:48:25.316187+0000","last_clean":"2023-12-19T07:48:25.316187+0000","last_became_active":"2023-12-19T07:48:25.311800+0000","last_became_peered":"2023-12-19T07:48:25.311800+0000","last_unstale":"2023-12-19T07:48:25.316187+0000","last_undegraded":"2023-12-19T07:48:25.316187+0000","last_fullsized":"2023-12-19T07:48:25.316187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:52:53.274789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324130+0000","last_change":"2023-12-19T07:48:26.324130+0000","last_active":"2023-12-19T07:48:26.324130+0000","last_peered":"2023-12-19T07:48:26.324130+0000","last_clean":"2023-12-19T07:48:26.324130+0000","last_became_active":"2023-12-19T07:48:26.323625+0000","last_became_peered":"2023-12-19T07:48:26.323625+0000","last_unstale":"2023-12-19T07:48:26.324130+0000","last_undegraded":"2023-12-19T07:48:26.324130+0000","last_fullsized":"2023-12-19T07:48:26.324130+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:45:29.145398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330452+0000","last_change":"2023-12-19T07:48:25.314057+0000","last_active":"2023-12-19T07:48:26.330452+0000","last_peered":"2023-12-19T07:48:26.330452+0000","last_clean":"2023-12-19T07:48:26.330452+0000","last_became_active":"2023-12-19T07:48:25.313662+0000","last_became_peered":"2023-12-19T07:48:25.313662+0000","last_unstale":"2023-12-19T07:48:26.330452+0000","last_undegraded":"2023-12-19T07:48:26.330452+0000","last_fullsized":"2023-12-19T07:48:26.330452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:36.379068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.327832+0000","last_change":"2023-12-19T07:48:26.327832+0000","last_active":"2023-12-19T07:48:26.327832+0000","last_peered":"2023-12-19T07:48:26.327832+0000","last_clean":"2023-12-19T07:48:26.327832+0000","last_became_active":"2023-12-19T07:48:26.327563+0000","last_became_peered":"2023-12-19T07:48:26.327563+0000","last_unstale":"2023-12-19T07:48:26.327832+0000","last_undegraded":"2023-12-19T07:48:26.327832+0000","last_fullsized":"2023-12-19T07:48:26.327832+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.315784+0000","last_change":"2023-12-19T07:48:25.314368+0000","last_active":"2023-12-19T07:48:25.315784+0000","last_peered":"2023-12-19T07:48:25.315784+0000","last_clean":"2023-12-19T07:48:25.315784+0000","last_became_active":"2023-12-19T07:48:25.314037+0000","last_became_peered":"2023-12-19T07:48:25.314037+0000","last_unstale":"2023-12-19T07:48:25.315784+0000","last_undegraded":"2023-12-19T07:48:25.315784+0000","last_fullsized":"2023-12-19T07:48:25.315784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:03.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336268+0000","last_change":"2023-12-19T07:48:26.336268+0000","last_active":"2023-12-19T07:48:26.336268+0000","last_peered":"2023-12-19T07:48:26.336268+0000","last_clean":"2023-12-19T07:48:26.336268+0000","last_became_active":"2023-12-19T07:48:26.335758+0000","last_became_peered":"2023-12-19T07:48:26.335758+0000","last_unstale":"2023-12-19T07:48:26.336268+0000","last_undegraded":"2023-12-19T07:48:26.336268+0000","last_fullsized":"2023-12-19T07:48:26.336268+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:00.664922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.336353+0000","last_change":"2023-12-19T07:48:25.331900+0000","last_active":"2023-12-19T07:48:25.336353+0000","last_peered":"2023-12-19T07:48:25.336353+0000","last_clean":"2023-12-19T07:48:25.336353+0000","last_became_active":"2023-12-19T07:48:25.331580+0000","last_became_peered":"2023-12-19T07:48:25.331580+0000","last_unstale":"2023-12-19T07:48:25.336353+0000","last_undegraded":"2023-12-19T07:48:25.336353+0000","last_fullsized":"2023-12-19T07:48:25.336353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:16:24.744942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320862+0000","last_change":"2023-12-19T07:48:26.320862+0000","last_active":"2023-12-19T07:48:26.320862+0000","last_peered":"2023-12-19T07:48:26.320862+0000","last_clean":"2023-12-19T07:48:26.320862+0000","last_became_active":"2023-12-19T07:48:26.319686+0000","last_became_peered":"2023-12-19T07:48:26.319686+0000","last_unstale":"2023-12-19T07:48:26.320862+0000","last_undegraded":"2023-12-19T07:48:26.320862+0000","last_fullsized":"2023-12-19T07:48:26.320862+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:39:51.239053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320717+0000","last_change":"2023-12-19T07:48:25.315121+0000","last_active":"2023-12-19T07:48:25.320717+0000","last_peered":"2023-12-19T07:48:25.320717+0000","last_clean":"2023-12-19T07:48:25.320717+0000","last_became_active":"2023-12-19T07:48:25.314836+0000","last_became_peered":"2023-12-19T07:48:25.314836+0000","last_unstale":"2023-12-19T07:48:25.320717+0000","last_undegraded":"2023-12-19T07:48:25.320717+0000","last_fullsized":"2023-12-19T07:48:25.320717+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:52.481514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336744+0000","last_change":"2023-12-19T07:48:26.336744+0000","last_active":"2023-12-19T07:48:26.336744+0000","last_peered":"2023-12-19T07:48:26.336744+0000","last_clean":"2023-12-19T07:48:26.336744+0000","last_became_active":"2023-12-19T07:48:26.336484+0000","last_became_peered":"2023-12-19T07:48:26.336484+0000","last_unstale":"2023-12-19T07:48:26.336744+0000","last_undegraded":"2023-12-19T07:48:26.336744+0000","last_fullsized":"2023-12-19T07:48:26.336744+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.350690+0000","last_change":"2023-12-19T07:48:25.321713+0000","last_active":"2023-12-19T07:48:26.350690+0000","last_peered":"2023-12-19T07:48:26.350690+0000","last_clean":"2023-12-19T07:48:26.350690+0000","last_became_active":"2023-12-19T07:48:25.321434+0000","last_became_peered":"2023-12-19T07:48:25.321434+0000","last_unstale":"2023-12-19T07:48:26.350690+0000","last_undegraded":"2023-12-19T07:48:26.350690+0000","last_fullsized":"2023-12-19T07:48:26.350690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:50.370509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.334902+0000","last_change":"2023-12-19T07:48:26.334902+0000","last_active":"2023-12-19T07:48:26.334902+0000","last_peered":"2023-12-19T07:48:26.334902+0000","last_clean":"2023-12-19T07:48:26.334902+0000","last_became_active":"2023-12-19T07:48:26.334624+0000","last_became_peered":"2023-12-19T07:48:26.334624+0000","last_unstale":"2023-12-19T07:48:26.334902+0000","last_undegraded":"2023-12-19T07:48:26.334902+0000","last_fullsized":"2023-12-19T07:48:26.334902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.330382+0000","last_change":"2023-12-19T07:48:25.327789+0000","last_active":"2023-12-19T07:48:25.330382+0000","last_peered":"2023-12-19T07:48:25.330382+0000","last_clean":"2023-12-19T07:48:25.330382+0000","last_became_active":"2023-12-19T07:48:25.327422+0000","last_became_peered":"2023-12-19T07:48:25.327422+0000","last_unstale":"2023-12-19T07:48:25.330382+0000","last_undegraded":"2023-12-19T07:48:25.330382+0000","last_fullsized":"2023-12-19T07:48:25.330382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:46:06.770421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.325056+0000","last_change":"2023-12-19T07:48:26.325056+0000","last_active":"2023-12-19T07:48:26.325056+0000","last_peered":"2023-12-19T07:48:26.325056+0000","last_clean":"2023-12-19T07:48:26.325056+0000","last_became_active":"2023-12-19T07:48:26.324774+0000","last_became_peered":"2023-12-19T07:48:26.324774+0000","last_unstale":"2023-12-19T07:48:26.325056+0000","last_undegraded":"2023-12-19T07:48:26.325056+0000","last_fullsized":"2023-12-19T07:48:26.325056+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:58.126030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.323933+0000","last_change":"2023-12-19T07:48:25.316753+0000","last_active":"2023-12-19T07:48:25.323933+0000","last_peered":"2023-12-19T07:48:25.323933+0000","last_clean":"2023-12-19T07:48:25.323933+0000","last_became_active":"2023-12-19T07:48:25.316423+0000","last_became_peered":"2023-12-19T07:48:25.316423+0000","last_unstale":"2023-12-19T07:48:25.323933+0000","last_undegraded":"2023-12-19T07:48:25.323933+0000","last_fullsized":"2023-12-19T07:48:25.323933+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:25.350499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326058+0000","last_change":"2023-12-19T07:48:26.326058+0000","last_active":"2023-12-19T07:48:26.326058+0000","last_peered":"2023-12-19T07:48:26.326058+0000","last_clean":"2023-12-19T07:48:26.326058+0000","last_became_active":"2023-12-19T07:48:26.325479+0000","last_became_peered":"2023-12-19T07:48:26.325479+0000","last_unstale":"2023-12-19T07:48:26.326058+0000","last_undegraded":"2023-12-19T07:48:26.326058+0000","last_fullsized":"2023-12-19T07:48:26.326058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320862+0000","last_change":"2023-12-19T07:48:25.314957+0000","last_active":"2023-12-19T07:48:25.320862+0000","last_peered":"2023-12-19T07:48:25.320862+0000","last_clean":"2023-12-19T07:48:25.320862+0000","last_became_active":"2023-12-19T07:48:25.314664+0000","last_became_peered":"2023-12-19T07:48:25.314664+0000","last_unstale":"2023-12-19T07:48:25.320862+0000","last_undegraded":"2023-12-19T07:48:25.320862+0000","last_fullsized":"2023-12-19T07:48:25.320862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:03.366692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341804+0000","last_change":"2023-12-19T07:48:26.341804+0000","last_active":"2023-12-19T07:48:26.341804+0000","last_peered":"2023-12-19T07:48:26.341804+0000","last_clean":"2023-12-19T07:48:26.341804+0000","last_became_active":"2023-12-19T07:48:26.341522+0000","last_became_peered":"2023-12-19T07:48:26.341522+0000","last_unstale":"2023-12-19T07:48:26.341804+0000","last_undegraded":"2023-12-19T07:48:26.341804+0000","last_fullsized":"2023-12-19T07:48:26.341804+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.322886+0000","last_change":"2023-12-19T07:48:25.315743+0000","last_active":"2023-12-19T07:48:25.322886+0000","last_peered":"2023-12-19T07:48:25.322886+0000","last_clean":"2023-12-19T07:48:25.322886+0000","last_became_active":"2023-12-19T07:48:25.315454+0000","last_became_peered":"2023-12-19T07:48:25.315454+0000","last_unstale":"2023-12-19T07:48:25.322886+0000","last_undegraded":"2023-12-19T07:48:25.322886+0000","last_fullsized":"2023-12-19T07:48:25.322886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:45.891457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324363+0000","last_change":"2023-12-19T07:48:26.324363+0000","last_active":"2023-12-19T07:48:26.324363+0000","last_peered":"2023-12-19T07:48:26.324363+0000","last_clean":"2023-12-19T07:48:26.324363+0000","last_became_active":"2023-12-19T07:48:26.324028+0000","last_became_peered":"2023-12-19T07:48:26.324028+0000","last_unstale":"2023-12-19T07:48:26.324363+0000","last_undegraded":"2023-12-19T07:48:26.324363+0000","last_fullsized":"2023-12-19T07:48:26.324363+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.331235+0000","last_change":"2023-12-19T07:48:25.323261+0000","last_active":"2023-12-19T07:48:25.331235+0000","last_peered":"2023-12-19T07:48:25.331235+0000","last_clean":"2023-12-19T07:48:25.331235+0000","last_became_active":"2023-12-19T07:48:25.323058+0000","last_became_peered":"2023-12-19T07:48:25.323058+0000","last_unstale":"2023-12-19T07:48:25.331235+0000","last_undegraded":"2023-12-19T07:48:25.331235+0000","last_fullsized":"2023-12-19T07:48:25.331235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:31.841246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.337098+0000","last_change":"2023-12-19T07:48:26.337098+0000","last_active":"2023-12-19T07:48:26.337098+0000","last_peered":"2023-12-19T07:48:26.337098+0000","last_clean":"2023-12-19T07:48:26.337098+0000","last_became_active":"2023-12-19T07:48:26.336844+0000","last_became_peered":"2023-12-19T07:48:26.336844+0000","last_unstale":"2023-12-19T07:48:26.337098+0000","last_undegraded":"2023-12-19T07:48:26.337098+0000","last_fullsized":"2023-12-19T07:48:26.337098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.318790+0000","last_change":"2023-12-19T07:48:25.308759+0000","last_active":"2023-12-19T07:48:25.318790+0000","last_peered":"2023-12-19T07:48:25.318790+0000","last_clean":"2023-12-19T07:48:25.318790+0000","last_became_active":"2023-12-19T07:48:25.308425+0000","last_became_peered":"2023-12-19T07:48:25.308425+0000","last_unstale":"2023-12-19T07:48:25.318790+0000","last_undegraded":"2023-12-19T07:48:25.318790+0000","last_fullsized":"2023-12-19T07:48:25.318790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:20.577530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.345244+0000","last_change":"2023-12-19T07:48:26.345244+0000","last_active":"2023-12-19T07:48:26.345244+0000","last_peered":"2023-12-19T07:48:26.345244+0000","last_clean":"2023-12-19T07:48:26.345244+0000","last_became_active":"2023-12-19T07:48:26.338717+0000","last_became_peered":"2023-12-19T07:48:26.338717+0000","last_unstale":"2023-12-19T07:48:26.345244+0000","last_undegraded":"2023-12-19T07:48:26.345244+0000","last_fullsized":"2023-12-19T07:48:26.345244+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.335516+0000","last_change":"2023-12-19T07:48:25.329960+0000","last_active":"2023-12-19T07:48:25.335516+0000","last_peered":"2023-12-19T07:48:25.335516+0000","last_clean":"2023-12-19T07:48:25.335516+0000","last_became_active":"2023-12-19T07:48:25.328098+0000","last_became_peered":"2023-12-19T07:48:25.328098+0000","last_unstale":"2023-12-19T07:48:25.335516+0000","last_undegraded":"2023-12-19T07:48:25.335516+0000","last_fullsized":"2023-12-19T07:48:25.335516+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:45.891457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320511+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:48:25.320511+0000","last_peered":"2023-12-19T07:48:25.320511+0000","last_clean":"2023-12-19T07:48:25.320511+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:48:25.320511+0000","last_undegraded":"2023-12-19T07:48:25.320511+0000","last_fullsized":"2023-12-19T07:48:25.320511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298296,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073544,"statfs":{"total":96636764160,"available":96331309056,"internally_reserved":0,"allocated":1032192,"data_stored":777243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.4750000000000001}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.97599999999999998}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.359},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3779999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":1.522},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.3089999999999999}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":2.8279999999999998,"5min":2.8279999999999998,"15min":2.8279999999999998},"last":1.095},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.417}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":1.452}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.409}]},{"osd":1,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.173},{"interface":"front","average":{"1min":1.21,"5min":1.21,"15min":1.21},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.224}]},{"osd":2,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.111}]},{"osd":3,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.3129999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":1.268,"5min":1.268,"15min":1.268},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":1.276}]},{"osd":5,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.3680000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.496}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.107}]},{"osd":1,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.05},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":3.113,"5min":3.113,"15min":3.113},"last":0.91100000000000003}]},{"osd":2,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.9039999999999999,"5min":2.9039999999999999,"15min":2.9039999999999999},"last":0.95899999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":1.254},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":4.3879999999999999,"5min":4.3879999999999999,"15min":4.3879999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":5.0819999999999999,"5min":5.0819999999999999,"15min":5.0819999999999999},"last":1.151},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":4.6310000000000002,"5min":4.6310000000000002,"15min":4.6310000000000002},"last":1.458}]},{"osd":6,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.01,"5min":1.01,"15min":1.01},"max":{"1min":3.9649999999999999,"5min":3.9649999999999999,"15min":3.9649999999999999},"last":1.411},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":1.2989999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":4.4539999999999997,"5min":4.4539999999999997,"15min":4.4539999999999997},"last":1.512},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":4.1980000000000004,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":1.486}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.1879999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.7490000000000001}]},{"osd":2,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.446},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.661,"5min":2.661,"15min":2.661},"last":1.242}]},{"osd":5,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":3.161,"5min":3.161,"15min":3.161},"last":1.407},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.591}]},{"osd":6,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.649,"15min":1.649},"min":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.7170000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":466944,"data_stored":195528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.74099999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.8879999999999999,"5min":4.8879999999999999,"15min":4.8879999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":8.0830000000000002,"5min":8.0830000000000002,"15min":8.0830000000000002},"last":1.272}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.85299999999999998}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":4.7130000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"last":1.167},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":5.0220000000000002,"5min":5.0220000000000002,"15min":5.0220000000000002},"last":1.397}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.995},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":8.5299999999999994,"5min":8.5299999999999994,"15min":8.5299999999999994},"last":1.139}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"min":{"1min":1.26,"5min":1.26,"15min":1.26},"max":{"1min":7.2750000000000004,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":4.6550000000000002,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":1.349}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":16.081,"5min":16.081,"15min":16.081},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.351,"5min":2.351,"15min":2.351},"min":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":15.285,"5min":15.285,"15min":15.285},"last":1.478}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.093,"5min":4.093,"15min":4.093},"last":1.0429999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":3.9089999999999998},"last":1.131}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.96399999999999997}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":3.9239999999999999,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":4.0170000000000003,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":1.3080000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.948,"5min":3.948,"15min":3.948},"last":1.49},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":1.101}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"min":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"max":{"1min":3.9620000000000002,"5min":3.9620000000000002,"15min":3.9620000000000002},"last":1.427}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":3.9809999999999999},"last":1.579},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.452}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.61099999999999999}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2609999999999999}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.78200000000000003}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.995},{"interface":"front","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":1.026}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.24}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":1.131,"5min":1.131,"15min":1.131},"max":{"1min":3.024,"5min":3.024,"15min":3.024},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.0640000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":3.6080000000000001,"5min":3.6080000000000001,"15min":3.6080000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":1.1020000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298308,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073532,"statfs":{"total":96636764160,"available":96331296768,"internally_reserved":0,"allocated":1044480,"data_stored":783349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":1.083},{"interface":"front","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.018},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.605}]},{"osd":5,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":1.028,"5min":1.028,"15min":1.028},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.1100000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.2929999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.671},{"interface":"front","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.028,"5min":1.028,"15min":1.028},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:48:30.583 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T07:48:30.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.795+0000 7f903b59e700 1 Processor -- start 2023-12-19T07:48:30.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.795+0000 7f903b59e700 1 -- start start 2023-12-19T07:48:30.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f903c0b1410 0x7f903c0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c06a580 0x7f903c06a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c06c7c0 con 0x7f903c15a860 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c06cd80 con 0x7f903c0b1410 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c06af10 con 0x7f903c06a580 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f903c0b1410 0x7f903c0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f903c0b1410 0x7f903c0ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44862/0 (socket says 172.21.15.111:44862) 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 -- 172.21.15.111:0/2512116316 learned_addr learned my addr 172.21.15.111:0/2512116316 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:30.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f9031d9b700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c06a580 0x7f903c06a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 -- 172.21.15.111:0/2512116316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c06a580 msgr2=0x7f903c06a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:30.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c06a580 0x7f903c06a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 -- 172.21.15.111:0/2512116316 >> v1:172.21.15.143:6789/0 conn(0x7f903c15a860 legacy=0x7f903c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 -- 172.21.15.111:0/2512116316 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c06b8f0 con 0x7f903c0b1410 2023-12-19T07:48:30.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f903a59c700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c0b1410 0x7f903c0ab440 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f902c0099b0 tx=0x7f902c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab14218440faf8d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f9039d9b700 1 -- 172.21.15.111:0/2512116316 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c009ea0 con 0x7f903c0b1410 2023-12-19T07:48:30.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f9039d9b700 1 -- 172.21.15.111:0/2512116316 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c00b860 con 0x7f903c0b1410 2023-12-19T07:48:30.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.799+0000 7f9039d9b700 1 -- 172.21.15.111:0/2512116316 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c00bc00 con 0x7f903c0b1410 2023-12-19T07:48:30.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 -- 172.21.15.111:0/2512116316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c0b1410 msgr2=0x7f903c0ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c0b1410 0x7f903c0ab440 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f902c0099b0 tx=0x7f902c009250 comp rx=0 tx=0).stop 2023-12-19T07:48:30.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 -- 172.21.15.111:0/2512116316 shutdown_connections 2023-12-19T07:48:30.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c06a580 0x7f903c06a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 --2- 172.21.15.111:0/2512116316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c0b1410 0x7f903c0ab440 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 -- 172.21.15.111:0/2512116316 >> 172.21.15.111:0/2512116316 conn(0x7f903c067bb0 msgr2=0x7f903c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:30.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 -- 172.21.15.111:0/2512116316 shutdown_connections 2023-12-19T07:48:30.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 -- 172.21.15.111:0/2512116316 wait complete. 2023-12-19T07:48:30.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.803+0000 7f903b59e700 1 Processor -- start 2023-12-19T07:48:30.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- start start 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f903c06a580 0x7f903c15e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c0b1410 0x7f903c14e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c4d10 con 0x7f903c15a860 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c4ff0 con 0x7f903c06a580 2023-12-19T07:48:30.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c52d0 con 0x7f903c0b1410 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f903c06a580 0x7f903c15e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44876/0 (socket says 172.21.15.111:44876) 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f9031d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c0b1410 0x7f903c14e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 -- 172.21.15.111:0/3788387486 learned_addr learned my addr 172.21.15.111:0/3788387486 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 -- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c0b1410 msgr2=0x7f903c14e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c0b1410 0x7f903c14e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 -- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f903c06a580 msgr2=0x7f903c15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f903c06a580 0x7f903c15e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 -- 172.21.15.111:0/3788387486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0094f0 con 0x7f903c15a860 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903ad9d700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f903400c8e0 tx=0x7f9034001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034012630 con 0x7f903c15a860 2023-12-19T07:48:30.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034012790 con 0x7f903c15a860 2023-12-19T07:48:30.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034012b50 con 0x7f903c15a860 2023-12-19T07:48:30.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c0c55b0 con 0x7f903c15a860 2023-12-19T07:48:30.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.807+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c15e920 con 0x7f903c15a860 2023-12-19T07:48:30.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.811+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c0ad170 con 0x7f903c15a860 2023-12-19T07:48:30.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.811+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f903401a3a0 con 0x7f903c15a860 2023-12-19T07:48:30.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.811+0000 7f9032ffd700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 0x7f901c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:30.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.811+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f903407cb20 con 0x7f903c15a860 2023-12-19T07:48:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.815+0000 7f903a59c700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 0x7f901c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:30.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.815+0000 7f903a59c700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 0x7f901c062a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f902c0099b0 tx=0x7f902c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:30.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.819+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f903c0ad170 con 0x7f903c15a860 2023-12-19T07:48:30.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:30.987+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f903c0ad170 con 0x7f901c060550 2023-12-19T07:48:30.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.007+0000 7f9032ffd700 1 -- 172.21.15.111:0/3788387486 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286152 (secure 0 0 0) 0x7f903c0ad170 con 0x7f901c060550 2023-12-19T07:48:30.996 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 msgr2=0x7f901c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 0x7f901c062a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f902c0099b0 tx=0x7f902c017000 comp rx=0 tx=0).stop 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 msgr2=0x7f903c14eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f903400c8e0 tx=0x7f9034001710 comp rx=0 tx=0).stop 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 shutdown_connections 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f901c060550 0x7f901c062a00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f903c15a860 0x7f903c14eb00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f903c0b1410 0x7f903c14e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 --2- 172.21.15.111:0/3788387486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f903c06a580 0x7f903c15e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 >> 172.21.15.111:0/3788387486 conn(0x7f903c067bb0 msgr2=0x7f903c0ab440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 shutdown_connections 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.011+0000 7f903b59e700 1 -- 172.21.15.111:0/3788387486 wait complete. 2023-12-19T07:48:31.001 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T07:48:31.018 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-12-19T07:48:30.636216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382928,"kb_used_data":4624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591792,"statfs":{"total":773094113280,"available":770653995008,"internally_reserved":0,"allocated":4734976,"data_stored":2690495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001905"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338949+0000","last_change":"2023-12-19T07:48:26.338949+0000","last_active":"2023-12-19T07:48:26.338949+0000","last_peered":"2023-12-19T07:48:26.338949+0000","last_clean":"2023-12-19T07:48:26.338949+0000","last_became_active":"2023-12-19T07:48:26.338453+0000","last_became_peered":"2023-12-19T07:48:26.338453+0000","last_unstale":"2023-12-19T07:48:26.338949+0000","last_undegraded":"2023-12-19T07:48:26.338949+0000","last_fullsized":"2023-12-19T07:48:26.338949+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:40:37.176996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.322646+0000","last_change":"2023-12-19T07:48:26.322646+0000","last_active":"2023-12-19T07:48:26.322646+0000","last_peered":"2023-12-19T07:48:26.322646+0000","last_clean":"2023-12-19T07:48:26.322646+0000","last_became_active":"2023-12-19T07:48:26.322220+0000","last_became_peered":"2023-12-19T07:48:26.322220+0000","last_unstale":"2023-12-19T07:48:26.322646+0000","last_undegraded":"2023-12-19T07:48:26.322646+0000","last_fullsized":"2023-12-19T07:48:26.322646+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:51.265875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335408+0000","last_change":"2023-12-19T07:48:26.335408+0000","last_active":"2023-12-19T07:48:26.335408+0000","last_peered":"2023-12-19T07:48:26.335408+0000","last_clean":"2023-12-19T07:48:26.335408+0000","last_became_active":"2023-12-19T07:48:26.335070+0000","last_became_peered":"2023-12-19T07:48:26.335070+0000","last_unstale":"2023-12-19T07:48:26.335408+0000","last_undegraded":"2023-12-19T07:48:26.335408+0000","last_fullsized":"2023-12-19T07:48:26.335408+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333968+0000","last_change":"2023-12-19T07:48:26.333968+0000","last_active":"2023-12-19T07:48:26.333968+0000","last_peered":"2023-12-19T07:48:26.333968+0000","last_clean":"2023-12-19T07:48:26.333968+0000","last_became_active":"2023-12-19T07:48:26.333576+0000","last_became_peered":"2023-12-19T07:48:26.333576+0000","last_unstale":"2023-12-19T07:48:26.333968+0000","last_undegraded":"2023-12-19T07:48:26.333968+0000","last_fullsized":"2023-12-19T07:48:26.333968+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:16:25.760635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332639+0000","last_change":"2023-12-19T07:48:26.332639+0000","last_active":"2023-12-19T07:48:26.332639+0000","last_peered":"2023-12-19T07:48:26.332639+0000","last_clean":"2023-12-19T07:48:26.332639+0000","last_became_active":"2023-12-19T07:48:26.332377+0000","last_became_peered":"2023-12-19T07:48:26.332377+0000","last_unstale":"2023-12-19T07:48:26.332639+0000","last_undegraded":"2023-12-19T07:48:26.332639+0000","last_fullsized":"2023-12-19T07:48:26.332639+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:52.550119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.313912+0000","last_change":"2023-12-19T07:48:26.313912+0000","last_active":"2023-12-19T07:48:26.313912+0000","last_peered":"2023-12-19T07:48:26.313912+0000","last_clean":"2023-12-19T07:48:26.313912+0000","last_became_active":"2023-12-19T07:48:26.313587+0000","last_became_peered":"2023-12-19T07:48:26.313587+0000","last_unstale":"2023-12-19T07:48:26.313912+0000","last_undegraded":"2023-12-19T07:48:26.313912+0000","last_fullsized":"2023-12-19T07:48:26.313912+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330580+0000","last_change":"2023-12-19T07:48:26.330580+0000","last_active":"2023-12-19T07:48:26.330580+0000","last_peered":"2023-12-19T07:48:26.330580+0000","last_clean":"2023-12-19T07:48:26.330580+0000","last_became_active":"2023-12-19T07:48:26.330235+0000","last_became_peered":"2023-12-19T07:48:26.330235+0000","last_unstale":"2023-12-19T07:48:26.330580+0000","last_undegraded":"2023-12-19T07:48:26.330580+0000","last_fullsized":"2023-12-19T07:48:26.330580+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332731+0000","last_change":"2023-12-19T07:48:26.332731+0000","last_active":"2023-12-19T07:48:26.332731+0000","last_peered":"2023-12-19T07:48:26.332731+0000","last_clean":"2023-12-19T07:48:26.332731+0000","last_became_active":"2023-12-19T07:48:26.332505+0000","last_became_peered":"2023-12-19T07:48:26.332505+0000","last_unstale":"2023-12-19T07:48:26.332731+0000","last_undegraded":"2023-12-19T07:48:26.332731+0000","last_fullsized":"2023-12-19T07:48:26.332731+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:01:15.851921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336389+0000","last_change":"2023-12-19T07:48:26.336389+0000","last_active":"2023-12-19T07:48:26.336389+0000","last_peered":"2023-12-19T07:48:26.336389+0000","last_clean":"2023-12-19T07:48:26.336389+0000","last_became_active":"2023-12-19T07:48:26.335600+0000","last_became_peered":"2023-12-19T07:48:26.335600+0000","last_unstale":"2023-12-19T07:48:26.336389+0000","last_undegraded":"2023-12-19T07:48:26.336389+0000","last_fullsized":"2023-12-19T07:48:26.336389+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335726+0000","last_change":"2023-12-19T07:48:26.335726+0000","last_active":"2023-12-19T07:48:26.335726+0000","last_peered":"2023-12-19T07:48:26.335726+0000","last_clean":"2023-12-19T07:48:26.335726+0000","last_became_active":"2023-12-19T07:48:26.332605+0000","last_became_peered":"2023-12-19T07:48:26.332605+0000","last_unstale":"2023-12-19T07:48:26.335726+0000","last_undegraded":"2023-12-19T07:48:26.335726+0000","last_fullsized":"2023-12-19T07:48:26.335726+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:21:34.779152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330824+0000","last_change":"2023-12-19T07:48:26.330824+0000","last_active":"2023-12-19T07:48:26.330824+0000","last_peered":"2023-12-19T07:48:26.330824+0000","last_clean":"2023-12-19T07:48:26.330824+0000","last_became_active":"2023-12-19T07:48:26.328335+0000","last_became_peered":"2023-12-19T07:48:26.328335+0000","last_unstale":"2023-12-19T07:48:26.330824+0000","last_undegraded":"2023-12-19T07:48:26.330824+0000","last_fullsized":"2023-12-19T07:48:26.330824+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:20.894146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333234+0000","last_change":"2023-12-19T07:48:26.333234+0000","last_active":"2023-12-19T07:48:26.333234+0000","last_peered":"2023-12-19T07:48:26.333234+0000","last_clean":"2023-12-19T07:48:26.333234+0000","last_became_active":"2023-12-19T07:48:26.332992+0000","last_became_peered":"2023-12-19T07:48:26.332992+0000","last_unstale":"2023-12-19T07:48:26.333234+0000","last_undegraded":"2023-12-19T07:48:26.333234+0000","last_fullsized":"2023-12-19T07:48:26.333234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343411+0000","last_change":"2023-12-19T07:48:26.343411+0000","last_active":"2023-12-19T07:48:26.343411+0000","last_peered":"2023-12-19T07:48:26.343411+0000","last_clean":"2023-12-19T07:48:26.343411+0000","last_became_active":"2023-12-19T07:48:26.343153+0000","last_became_peered":"2023-12-19T07:48:26.343153+0000","last_unstale":"2023-12-19T07:48:26.343411+0000","last_undegraded":"2023-12-19T07:48:26.343411+0000","last_fullsized":"2023-12-19T07:48:26.343411+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:35:21.377012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339119+0000","last_change":"2023-12-19T07:48:26.339119+0000","last_active":"2023-12-19T07:48:26.339119+0000","last_peered":"2023-12-19T07:48:26.339119+0000","last_clean":"2023-12-19T07:48:26.339119+0000","last_became_active":"2023-12-19T07:48:26.338827+0000","last_became_peered":"2023-12-19T07:48:26.338827+0000","last_unstale":"2023-12-19T07:48:26.339119+0000","last_undegraded":"2023-12-19T07:48:26.339119+0000","last_fullsized":"2023-12-19T07:48:26.339119+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:21:34.779152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343632+0000","last_change":"2023-12-19T07:48:26.343632+0000","last_active":"2023-12-19T07:48:26.343632+0000","last_peered":"2023-12-19T07:48:26.343632+0000","last_clean":"2023-12-19T07:48:26.343632+0000","last_became_active":"2023-12-19T07:48:26.337642+0000","last_became_peered":"2023-12-19T07:48:26.337642+0000","last_unstale":"2023-12-19T07:48:26.343632+0000","last_undegraded":"2023-12-19T07:48:26.343632+0000","last_fullsized":"2023-12-19T07:48:26.343632+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336605+0000","last_change":"2023-12-19T07:48:26.336605+0000","last_active":"2023-12-19T07:48:26.336605+0000","last_peered":"2023-12-19T07:48:26.336605+0000","last_clean":"2023-12-19T07:48:26.336605+0000","last_became_active":"2023-12-19T07:48:26.335279+0000","last_became_peered":"2023-12-19T07:48:26.335279+0000","last_unstale":"2023-12-19T07:48:26.336605+0000","last_undegraded":"2023-12-19T07:48:26.336605+0000","last_fullsized":"2023-12-19T07:48:26.336605+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:54.963432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336300+0000","last_change":"2023-12-19T07:48:26.336300+0000","last_active":"2023-12-19T07:48:26.336300+0000","last_peered":"2023-12-19T07:48:26.336300+0000","last_clean":"2023-12-19T07:48:26.336300+0000","last_became_active":"2023-12-19T07:48:26.336039+0000","last_became_peered":"2023-12-19T07:48:26.336039+0000","last_unstale":"2023-12-19T07:48:26.336300+0000","last_undegraded":"2023-12-19T07:48:26.336300+0000","last_fullsized":"2023-12-19T07:48:26.336300+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:00.664922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338475+0000","last_change":"2023-12-19T07:48:26.338475+0000","last_active":"2023-12-19T07:48:26.338475+0000","last_peered":"2023-12-19T07:48:26.338475+0000","last_clean":"2023-12-19T07:48:26.338475+0000","last_became_active":"2023-12-19T07:48:26.338197+0000","last_became_peered":"2023-12-19T07:48:26.338197+0000","last_unstale":"2023-12-19T07:48:26.338475+0000","last_undegraded":"2023-12-19T07:48:26.338475+0000","last_fullsized":"2023-12-19T07:48:26.338475+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:53.490385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320407+0000","last_change":"2023-12-19T07:48:26.320407+0000","last_active":"2023-12-19T07:48:26.320407+0000","last_peered":"2023-12-19T07:48:26.320407+0000","last_clean":"2023-12-19T07:48:26.320407+0000","last_became_active":"2023-12-19T07:48:26.319941+0000","last_became_peered":"2023-12-19T07:48:26.319941+0000","last_unstale":"2023-12-19T07:48:26.320407+0000","last_undegraded":"2023-12-19T07:48:26.320407+0000","last_fullsized":"2023-12-19T07:48:26.320407+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:49:22.548670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343816+0000","last_change":"2023-12-19T07:48:26.343816+0000","last_active":"2023-12-19T07:48:26.343816+0000","last_peered":"2023-12-19T07:48:26.343816+0000","last_clean":"2023-12-19T07:48:26.343816+0000","last_became_active":"2023-12-19T07:48:26.338302+0000","last_became_peered":"2023-12-19T07:48:26.338302+0000","last_unstale":"2023-12-19T07:48:26.343816+0000","last_undegraded":"2023-12-19T07:48:26.343816+0000","last_fullsized":"2023-12-19T07:48:26.343816+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339543+0000","last_change":"2023-12-19T07:48:26.339543+0000","last_active":"2023-12-19T07:48:26.339543+0000","last_peered":"2023-12-19T07:48:26.339543+0000","last_clean":"2023-12-19T07:48:26.339543+0000","last_became_active":"2023-12-19T07:48:26.339266+0000","last_became_peered":"2023-12-19T07:48:26.339266+0000","last_unstale":"2023-12-19T07:48:26.339543+0000","last_undegraded":"2023-12-19T07:48:26.339543+0000","last_fullsized":"2023-12-19T07:48:26.339543+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:21.593223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.329305+0000","last_change":"2023-12-19T07:48:26.329305+0000","last_active":"2023-12-19T07:48:26.329305+0000","last_peered":"2023-12-19T07:48:26.329305+0000","last_clean":"2023-12-19T07:48:26.329305+0000","last_became_active":"2023-12-19T07:48:26.328930+0000","last_became_peered":"2023-12-19T07:48:26.328930+0000","last_unstale":"2023-12-19T07:48:26.329305+0000","last_undegraded":"2023-12-19T07:48:26.329305+0000","last_fullsized":"2023-12-19T07:48:26.329305+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320275+0000","last_change":"2023-12-19T07:48:26.320275+0000","last_active":"2023-12-19T07:48:26.320275+0000","last_peered":"2023-12-19T07:48:26.320275+0000","last_clean":"2023-12-19T07:48:26.320275+0000","last_became_active":"2023-12-19T07:48:26.319953+0000","last_became_peered":"2023-12-19T07:48:26.319953+0000","last_unstale":"2023-12-19T07:48:26.320275+0000","last_undegraded":"2023-12-19T07:48:26.320275+0000","last_fullsized":"2023-12-19T07:48:26.320275+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320639+0000","last_change":"2023-12-19T07:48:26.320639+0000","last_active":"2023-12-19T07:48:26.320639+0000","last_peered":"2023-12-19T07:48:26.320639+0000","last_clean":"2023-12-19T07:48:26.320639+0000","last_became_active":"2023-12-19T07:48:26.320337+0000","last_became_peered":"2023-12-19T07:48:26.320337+0000","last_unstale":"2023-12-19T07:48:26.320639+0000","last_undegraded":"2023-12-19T07:48:26.320639+0000","last_fullsized":"2023-12-19T07:48:26.320639+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:15.201218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.343830+0000","last_change":"2023-12-19T07:48:26.343830+0000","last_active":"2023-12-19T07:48:26.343830+0000","last_peered":"2023-12-19T07:48:26.343830+0000","last_clean":"2023-12-19T07:48:26.343830+0000","last_became_active":"2023-12-19T07:48:26.343424+0000","last_became_peered":"2023-12-19T07:48:26.343424+0000","last_unstale":"2023-12-19T07:48:26.343830+0000","last_undegraded":"2023-12-19T07:48:26.343830+0000","last_fullsized":"2023-12-19T07:48:26.343830+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:32.856939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.328931+0000","last_change":"2023-12-19T07:48:26.328931+0000","last_active":"2023-12-19T07:48:26.328931+0000","last_peered":"2023-12-19T07:48:26.328931+0000","last_clean":"2023-12-19T07:48:26.328931+0000","last_became_active":"2023-12-19T07:48:26.328649+0000","last_became_peered":"2023-12-19T07:48:26.328649+0000","last_unstale":"2023-12-19T07:48:26.328931+0000","last_undegraded":"2023-12-19T07:48:26.328931+0000","last_fullsized":"2023-12-19T07:48:26.328931+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:23.623245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339633+0000","last_change":"2023-12-19T07:48:26.339633+0000","last_active":"2023-12-19T07:48:26.339633+0000","last_peered":"2023-12-19T07:48:26.339633+0000","last_clean":"2023-12-19T07:48:26.339633+0000","last_became_active":"2023-12-19T07:48:26.339354+0000","last_became_peered":"2023-12-19T07:48:26.339354+0000","last_unstale":"2023-12-19T07:48:26.339633+0000","last_undegraded":"2023-12-19T07:48:26.339633+0000","last_fullsized":"2023-12-19T07:48:26.339633+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:18.417615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333909+0000","last_change":"2023-12-19T07:48:26.333909+0000","last_active":"2023-12-19T07:48:26.333909+0000","last_peered":"2023-12-19T07:48:26.333909+0000","last_clean":"2023-12-19T07:48:26.333909+0000","last_became_active":"2023-12-19T07:48:26.332177+0000","last_became_peered":"2023-12-19T07:48:26.332177+0000","last_unstale":"2023-12-19T07:48:26.333909+0000","last_undegraded":"2023-12-19T07:48:26.333909+0000","last_fullsized":"2023-12-19T07:48:26.333909+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.340766+0000","last_change":"2023-12-19T07:48:26.340766+0000","last_active":"2023-12-19T07:48:26.340766+0000","last_peered":"2023-12-19T07:48:26.340766+0000","last_clean":"2023-12-19T07:48:26.340766+0000","last_became_active":"2023-12-19T07:48:26.340465+0000","last_became_peered":"2023-12-19T07:48:26.340465+0000","last_unstale":"2023-12-19T07:48:26.340766+0000","last_undegraded":"2023-12-19T07:48:26.340766+0000","last_fullsized":"2023-12-19T07:48:26.340766+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.342664+0000","last_change":"2023-12-19T07:48:26.342664+0000","last_active":"2023-12-19T07:48:26.342664+0000","last_peered":"2023-12-19T07:48:26.342664+0000","last_clean":"2023-12-19T07:48:26.342664+0000","last_became_active":"2023-12-19T07:48:26.341935+0000","last_became_peered":"2023-12-19T07:48:26.341935+0000","last_unstale":"2023-12-19T07:48:26.342664+0000","last_undegraded":"2023-12-19T07:48:26.342664+0000","last_fullsized":"2023-12-19T07:48:26.342664+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.323761+0000","last_change":"2023-12-19T07:48:26.323761+0000","last_active":"2023-12-19T07:48:26.323761+0000","last_peered":"2023-12-19T07:48:26.323761+0000","last_clean":"2023-12-19T07:48:26.323761+0000","last_became_active":"2023-12-19T07:48:26.323455+0000","last_became_peered":"2023-12-19T07:48:26.323455+0000","last_unstale":"2023-12-19T07:48:26.323761+0000","last_undegraded":"2023-12-19T07:48:26.323761+0000","last_fullsized":"2023-12-19T07:48:26.323761+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326617+0000","last_change":"2023-12-19T07:48:26.326617+0000","last_active":"2023-12-19T07:48:26.326617+0000","last_peered":"2023-12-19T07:48:26.326617+0000","last_clean":"2023-12-19T07:48:26.326617+0000","last_became_active":"2023-12-19T07:48:26.323631+0000","last_became_peered":"2023-12-19T07:48:26.323631+0000","last_unstale":"2023-12-19T07:48:26.326617+0000","last_undegraded":"2023-12-19T07:48:26.326617+0000","last_fullsized":"2023-12-19T07:48:26.326617+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335504+0000","last_change":"2023-12-19T07:48:26.335504+0000","last_active":"2023-12-19T07:48:26.335504+0000","last_peered":"2023-12-19T07:48:26.335504+0000","last_clean":"2023-12-19T07:48:26.335504+0000","last_became_active":"2023-12-19T07:48:26.335210+0000","last_became_peered":"2023-12-19T07:48:26.335210+0000","last_unstale":"2023-12-19T07:48:26.335504+0000","last_undegraded":"2023-12-19T07:48:26.335504+0000","last_fullsized":"2023-12-19T07:48:26.335504+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:09.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.344917+0000","last_change":"2023-12-19T07:48:25.323929+0000","last_active":"2023-12-19T07:48:26.344917+0000","last_peered":"2023-12-19T07:48:26.344917+0000","last_clean":"2023-12-19T07:48:26.344917+0000","last_became_active":"2023-12-19T07:48:25.323601+0000","last_became_peered":"2023-12-19T07:48:25.323601+0000","last_unstale":"2023-12-19T07:48:26.344917+0000","last_undegraded":"2023-12-19T07:48:26.344917+0000","last_fullsized":"2023-12-19T07:48:26.344917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:14.044929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.338249+0000","last_change":"2023-12-19T07:48:26.338249+0000","last_active":"2023-12-19T07:48:26.338249+0000","last_peered":"2023-12-19T07:48:26.338249+0000","last_clean":"2023-12-19T07:48:26.338249+0000","last_became_active":"2023-12-19T07:48:26.337983+0000","last_became_peered":"2023-12-19T07:48:26.337983+0000","last_unstale":"2023-12-19T07:48:26.338249+0000","last_undegraded":"2023-12-19T07:48:26.338249+0000","last_fullsized":"2023-12-19T07:48:26.338249+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320929+0000","last_change":"2023-12-19T07:48:25.323223+0000","last_active":"2023-12-19T07:48:26.320929+0000","last_peered":"2023-12-19T07:48:26.320929+0000","last_clean":"2023-12-19T07:48:26.320929+0000","last_became_active":"2023-12-19T07:48:25.322943+0000","last_became_peered":"2023-12-19T07:48:25.322943+0000","last_unstale":"2023-12-19T07:48:26.320929+0000","last_undegraded":"2023-12-19T07:48:26.320929+0000","last_fullsized":"2023-12-19T07:48:26.320929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:50.370509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.340690+0000","last_change":"2023-12-19T07:48:26.340690+0000","last_active":"2023-12-19T07:48:26.340690+0000","last_peered":"2023-12-19T07:48:26.340690+0000","last_clean":"2023-12-19T07:48:26.340690+0000","last_became_active":"2023-12-19T07:48:26.340402+0000","last_became_peered":"2023-12-19T07:48:26.340402+0000","last_unstale":"2023-12-19T07:48:26.340690+0000","last_undegraded":"2023-12-19T07:48:26.340690+0000","last_fullsized":"2023-12-19T07:48:26.340690+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:20.894146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.332430+0000","last_change":"2023-12-19T07:48:25.311015+0000","last_active":"2023-12-19T07:48:26.332430+0000","last_peered":"2023-12-19T07:48:26.332430+0000","last_clean":"2023-12-19T07:48:26.332430+0000","last_became_active":"2023-12-19T07:48:25.310667+0000","last_became_peered":"2023-12-19T07:48:25.310667+0000","last_unstale":"2023-12-19T07:48:26.332430+0000","last_undegraded":"2023-12-19T07:48:26.332430+0000","last_fullsized":"2023-12-19T07:48:26.332430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:38.234163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341252+0000","last_change":"2023-12-19T07:48:26.341252+0000","last_active":"2023-12-19T07:48:26.341252+0000","last_peered":"2023-12-19T07:48:26.341252+0000","last_clean":"2023-12-19T07:48:26.341252+0000","last_became_active":"2023-12-19T07:48:26.340914+0000","last_became_peered":"2023-12-19T07:48:26.340914+0000","last_unstale":"2023-12-19T07:48:26.341252+0000","last_undegraded":"2023-12-19T07:48:26.341252+0000","last_fullsized":"2023-12-19T07:48:26.341252+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:15:17.506583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.329182+0000","last_change":"2023-12-19T07:48:25.320384+0000","last_active":"2023-12-19T07:48:26.329182+0000","last_peered":"2023-12-19T07:48:26.329182+0000","last_clean":"2023-12-19T07:48:26.329182+0000","last_became_active":"2023-12-19T07:48:25.320115+0000","last_became_peered":"2023-12-19T07:48:25.320115+0000","last_unstale":"2023-12-19T07:48:26.329182+0000","last_undegraded":"2023-12-19T07:48:26.329182+0000","last_fullsized":"2023-12-19T07:48:26.329182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:52.474691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331874+0000","last_change":"2023-12-19T07:48:26.331874+0000","last_active":"2023-12-19T07:48:26.331874+0000","last_peered":"2023-12-19T07:48:26.331874+0000","last_clean":"2023-12-19T07:48:26.331874+0000","last_became_active":"2023-12-19T07:48:26.331460+0000","last_became_peered":"2023-12-19T07:48:26.331460+0000","last_unstale":"2023-12-19T07:48:26.331874+0000","last_undegraded":"2023-12-19T07:48:26.331874+0000","last_fullsized":"2023-12-19T07:48:26.331874+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:08:09.163599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.335182+0000","last_change":"2023-12-19T07:48:25.325686+0000","last_active":"2023-12-19T07:48:26.335182+0000","last_peered":"2023-12-19T07:48:26.335182+0000","last_clean":"2023-12-19T07:48:26.335182+0000","last_became_active":"2023-12-19T07:48:25.325324+0000","last_became_peered":"2023-12-19T07:48:25.325324+0000","last_unstale":"2023-12-19T07:48:26.335182+0000","last_undegraded":"2023-12-19T07:48:26.335182+0000","last_fullsized":"2023-12-19T07:48:26.335182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:29.482918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330466+0000","last_change":"2023-12-19T07:48:26.330466+0000","last_active":"2023-12-19T07:48:26.330466+0000","last_peered":"2023-12-19T07:48:26.330466+0000","last_clean":"2023-12-19T07:48:26.330466+0000","last_became_active":"2023-12-19T07:48:26.330171+0000","last_became_peered":"2023-12-19T07:48:26.330171+0000","last_unstale":"2023-12-19T07:48:26.330466+0000","last_undegraded":"2023-12-19T07:48:26.330466+0000","last_fullsized":"2023-12-19T07:48:26.330466+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.307263+0000","last_change":"2023-12-19T07:48:25.332216+0000","last_active":"2023-12-19T07:48:26.307263+0000","last_peered":"2023-12-19T07:48:26.307263+0000","last_clean":"2023-12-19T07:48:26.307263+0000","last_became_active":"2023-12-19T07:48:25.331890+0000","last_became_peered":"2023-12-19T07:48:25.331890+0000","last_unstale":"2023-12-19T07:48:26.307263+0000","last_undegraded":"2023-12-19T07:48:26.307263+0000","last_fullsized":"2023-12-19T07:48:26.307263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:14.185525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.334980+0000","last_change":"2023-12-19T07:48:26.334980+0000","last_active":"2023-12-19T07:48:26.334980+0000","last_peered":"2023-12-19T07:48:26.334980+0000","last_clean":"2023-12-19T07:48:26.334980+0000","last_became_active":"2023-12-19T07:48:26.334709+0000","last_became_peered":"2023-12-19T07:48:26.334709+0000","last_unstale":"2023-12-19T07:48:26.334980+0000","last_undegraded":"2023-12-19T07:48:26.334980+0000","last_fullsized":"2023-12-19T07:48:26.334980+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:21.593223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.315899+0000","last_change":"2023-12-19T07:48:25.311418+0000","last_active":"2023-12-19T07:48:25.315899+0000","last_peered":"2023-12-19T07:48:25.315899+0000","last_clean":"2023-12-19T07:48:25.315899+0000","last_became_active":"2023-12-19T07:48:25.311156+0000","last_became_peered":"2023-12-19T07:48:25.311156+0000","last_unstale":"2023-12-19T07:48:25.315899+0000","last_undegraded":"2023-12-19T07:48:25.315899+0000","last_fullsized":"2023-12-19T07:48:25.315899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:17.401922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.319768+0000","last_change":"2023-12-19T07:48:26.319768+0000","last_active":"2023-12-19T07:48:26.319768+0000","last_peered":"2023-12-19T07:48:26.319768+0000","last_clean":"2023-12-19T07:48:26.319768+0000","last_became_active":"2023-12-19T07:48:26.319473+0000","last_became_peered":"2023-12-19T07:48:26.319473+0000","last_unstale":"2023-12-19T07:48:26.319768+0000","last_undegraded":"2023-12-19T07:48:26.319768+0000","last_fullsized":"2023-12-19T07:48:26.319768+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:36:16.492768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.324309+0000","last_change":"2023-12-19T07:48:25.318027+0000","last_active":"2023-12-19T07:48:25.324309+0000","last_peered":"2023-12-19T07:48:25.324309+0000","last_clean":"2023-12-19T07:48:25.324309+0000","last_became_active":"2023-12-19T07:48:25.317733+0000","last_became_peered":"2023-12-19T07:48:25.317733+0000","last_unstale":"2023-12-19T07:48:25.324309+0000","last_undegraded":"2023-12-19T07:48:25.324309+0000","last_fullsized":"2023-12-19T07:48:25.324309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:43.219229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333174+0000","last_change":"2023-12-19T07:48:26.333174+0000","last_active":"2023-12-19T07:48:26.333174+0000","last_peered":"2023-12-19T07:48:26.333174+0000","last_clean":"2023-12-19T07:48:26.333174+0000","last_became_active":"2023-12-19T07:48:26.332896+0000","last_became_peered":"2023-12-19T07:48:26.332896+0000","last_unstale":"2023-12-19T07:48:26.333174+0000","last_undegraded":"2023-12-19T07:48:26.333174+0000","last_fullsized":"2023-12-19T07:48:26.333174+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:41:30.791474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.321945+0000","last_change":"2023-12-19T07:48:25.318295+0000","last_active":"2023-12-19T07:48:25.321945+0000","last_peered":"2023-12-19T07:48:25.321945+0000","last_clean":"2023-12-19T07:48:25.321945+0000","last_became_active":"2023-12-19T07:48:25.317975+0000","last_became_peered":"2023-12-19T07:48:25.317975+0000","last_unstale":"2023-12-19T07:48:25.321945+0000","last_undegraded":"2023-12-19T07:48:25.321945+0000","last_fullsized":"2023-12-19T07:48:25.321945+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:19.878453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.333137+0000","last_change":"2023-12-19T07:48:26.333137+0000","last_active":"2023-12-19T07:48:26.333137+0000","last_peered":"2023-12-19T07:48:26.333137+0000","last_clean":"2023-12-19T07:48:26.333137+0000","last_became_active":"2023-12-19T07:48:26.332905+0000","last_became_peered":"2023-12-19T07:48:26.332905+0000","last_unstale":"2023-12-19T07:48:26.333137+0000","last_undegraded":"2023-12-19T07:48:26.333137+0000","last_fullsized":"2023-12-19T07:48:26.333137+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:05.607377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.335633+0000","last_change":"2023-12-19T07:48:25.327715+0000","last_active":"2023-12-19T07:48:25.335633+0000","last_peered":"2023-12-19T07:48:25.335633+0000","last_clean":"2023-12-19T07:48:25.335633+0000","last_became_active":"2023-12-19T07:48:25.327344+0000","last_became_peered":"2023-12-19T07:48:25.327344+0000","last_unstale":"2023-12-19T07:48:25.335633+0000","last_undegraded":"2023-12-19T07:48:25.335633+0000","last_fullsized":"2023-12-19T07:48:25.335633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:34:30.931509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331347+0000","last_change":"2023-12-19T07:48:26.331347+0000","last_active":"2023-12-19T07:48:26.331347+0000","last_peered":"2023-12-19T07:48:26.331347+0000","last_clean":"2023-12-19T07:48:26.331347+0000","last_became_active":"2023-12-19T07:48:26.331063+0000","last_became_peered":"2023-12-19T07:48:26.331063+0000","last_unstale":"2023-12-19T07:48:26.331347+0000","last_undegraded":"2023-12-19T07:48:26.331347+0000","last_fullsized":"2023-12-19T07:48:26.331347+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:23.623245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.347473+0000","last_change":"2023-12-19T07:48:25.329659+0000","last_active":"2023-12-19T07:48:26.347473+0000","last_peered":"2023-12-19T07:48:26.347473+0000","last_clean":"2023-12-19T07:48:26.347473+0000","last_became_active":"2023-12-19T07:48:25.329402+0000","last_became_peered":"2023-12-19T07:48:25.329402+0000","last_unstale":"2023-12-19T07:48:26.347473+0000","last_undegraded":"2023-12-19T07:48:26.347473+0000","last_fullsized":"2023-12-19T07:48:26.347473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:03.366692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.325819+0000","last_change":"2023-12-19T07:48:26.325819+0000","last_active":"2023-12-19T07:48:26.325819+0000","last_peered":"2023-12-19T07:48:26.325819+0000","last_clean":"2023-12-19T07:48:26.325819+0000","last_became_active":"2023-12-19T07:48:26.325493+0000","last_became_peered":"2023-12-19T07:48:26.325493+0000","last_unstale":"2023-12-19T07:48:26.325819+0000","last_undegraded":"2023-12-19T07:48:26.325819+0000","last_fullsized":"2023-12-19T07:48:26.325819+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330839+0000","last_change":"2023-12-19T07:48:25.322692+0000","last_active":"2023-12-19T07:48:26.330839+0000","last_peered":"2023-12-19T07:48:26.330839+0000","last_clean":"2023-12-19T07:48:26.330839+0000","last_became_active":"2023-12-19T07:48:25.322033+0000","last_became_peered":"2023-12-19T07:48:25.322033+0000","last_unstale":"2023-12-19T07:48:26.330839+0000","last_undegraded":"2023-12-19T07:48:26.330839+0000","last_fullsized":"2023-12-19T07:48:26.330839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:53.305160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331762+0000","last_change":"2023-12-19T07:48:26.331762+0000","last_active":"2023-12-19T07:48:26.331762+0000","last_peered":"2023-12-19T07:48:26.331762+0000","last_clean":"2023-12-19T07:48:26.331762+0000","last_became_active":"2023-12-19T07:48:26.331032+0000","last_became_peered":"2023-12-19T07:48:26.331032+0000","last_unstale":"2023-12-19T07:48:26.331762+0000","last_undegraded":"2023-12-19T07:48:26.331762+0000","last_fullsized":"2023-12-19T07:48:26.331762+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:54.963432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331123+0000","last_change":"2023-12-19T07:48:25.314507+0000","last_active":"2023-12-19T07:48:26.331123+0000","last_peered":"2023-12-19T07:48:26.331123+0000","last_clean":"2023-12-19T07:48:26.331123+0000","last_became_active":"2023-12-19T07:48:25.314181+0000","last_became_peered":"2023-12-19T07:48:25.314181+0000","last_unstale":"2023-12-19T07:48:26.331123+0000","last_undegraded":"2023-12-19T07:48:26.331123+0000","last_fullsized":"2023-12-19T07:48:26.331123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:28.249753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330377+0000","last_change":"2023-12-19T07:48:26.330377+0000","last_active":"2023-12-19T07:48:26.330377+0000","last_peered":"2023-12-19T07:48:26.330377+0000","last_clean":"2023-12-19T07:48:26.330377+0000","last_became_active":"2023-12-19T07:48:26.330068+0000","last_became_peered":"2023-12-19T07:48:26.330068+0000","last_unstale":"2023-12-19T07:48:26.330377+0000","last_undegraded":"2023-12-19T07:48:26.330377+0000","last_fullsized":"2023-12-19T07:48:26.330377+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:08.964380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.316092+0000","last_change":"2023-12-19T07:48:25.311733+0000","last_active":"2023-12-19T07:48:25.316092+0000","last_peered":"2023-12-19T07:48:25.316092+0000","last_clean":"2023-12-19T07:48:25.316092+0000","last_became_active":"2023-12-19T07:48:25.311357+0000","last_became_peered":"2023-12-19T07:48:25.311357+0000","last_unstale":"2023-12-19T07:48:25.316092+0000","last_undegraded":"2023-12-19T07:48:25.316092+0000","last_fullsized":"2023-12-19T07:48:25.316092+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:31.841246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326403+0000","last_change":"2023-12-19T07:48:26.326403+0000","last_active":"2023-12-19T07:48:26.326403+0000","last_peered":"2023-12-19T07:48:26.326403+0000","last_clean":"2023-12-19T07:48:26.326403+0000","last_became_active":"2023-12-19T07:48:26.326117+0000","last_became_peered":"2023-12-19T07:48:26.326117+0000","last_unstale":"2023-12-19T07:48:26.326403+0000","last_undegraded":"2023-12-19T07:48:26.326403+0000","last_fullsized":"2023-12-19T07:48:26.326403+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:46.637053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.331150+0000","last_change":"2023-12-19T07:48:25.325731+0000","last_active":"2023-12-19T07:48:25.331150+0000","last_peered":"2023-12-19T07:48:25.331150+0000","last_clean":"2023-12-19T07:48:25.331150+0000","last_became_active":"2023-12-19T07:48:25.325424+0000","last_became_peered":"2023-12-19T07:48:25.325424+0000","last_unstale":"2023-12-19T07:48:25.331150+0000","last_undegraded":"2023-12-19T07:48:25.331150+0000","last_fullsized":"2023-12-19T07:48:25.331150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:14.185525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331579+0000","last_change":"2023-12-19T07:48:26.331579+0000","last_active":"2023-12-19T07:48:26.331579+0000","last_peered":"2023-12-19T07:48:26.331579+0000","last_clean":"2023-12-19T07:48:26.331579+0000","last_became_active":"2023-12-19T07:48:26.331294+0000","last_became_peered":"2023-12-19T07:48:26.331294+0000","last_unstale":"2023-12-19T07:48:26.331579+0000","last_undegraded":"2023-12-19T07:48:26.331579+0000","last_fullsized":"2023-12-19T07:48:26.331579+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:20:56.894788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.331496+0000","last_change":"2023-12-19T07:48:25.332417+0000","last_active":"2023-12-19T07:48:26.331496+0000","last_peered":"2023-12-19T07:48:26.331496+0000","last_clean":"2023-12-19T07:48:26.331496+0000","last_became_active":"2023-12-19T07:48:25.332145+0000","last_became_peered":"2023-12-19T07:48:25.332145+0000","last_unstale":"2023-12-19T07:48:26.331496+0000","last_undegraded":"2023-12-19T07:48:26.331496+0000","last_fullsized":"2023-12-19T07:48:26.331496+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:14.044929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341253+0000","last_change":"2023-12-19T07:48:26.341253+0000","last_active":"2023-12-19T07:48:26.341253+0000","last_peered":"2023-12-19T07:48:26.341253+0000","last_clean":"2023-12-19T07:48:26.341253+0000","last_became_active":"2023-12-19T07:48:26.340941+0000","last_became_peered":"2023-12-19T07:48:26.340941+0000","last_unstale":"2023-12-19T07:48:26.341253+0000","last_undegraded":"2023-12-19T07:48:26.341253+0000","last_fullsized":"2023-12-19T07:48:26.341253+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:39:12.625159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.339130+0000","last_change":"2023-12-19T07:48:25.324069+0000","last_active":"2023-12-19T07:48:26.339130+0000","last_peered":"2023-12-19T07:48:26.339130+0000","last_clean":"2023-12-19T07:48:26.339130+0000","last_became_active":"2023-12-19T07:48:25.323728+0000","last_became_peered":"2023-12-19T07:48:25.323728+0000","last_unstale":"2023-12-19T07:48:26.339130+0000","last_undegraded":"2023-12-19T07:48:26.339130+0000","last_fullsized":"2023-12-19T07:48:26.339130+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:19:00.406567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326949+0000","last_change":"2023-12-19T07:48:26.326949+0000","last_active":"2023-12-19T07:48:26.326949+0000","last_peered":"2023-12-19T07:48:26.326949+0000","last_clean":"2023-12-19T07:48:26.326949+0000","last_became_active":"2023-12-19T07:48:26.326654+0000","last_became_peered":"2023-12-19T07:48:26.326654+0000","last_unstale":"2023-12-19T07:48:26.326949+0000","last_undegraded":"2023-12-19T07:48:26.326949+0000","last_fullsized":"2023-12-19T07:48:26.326949+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.328671+0000","last_change":"2023-12-19T07:48:25.328973+0000","last_active":"2023-12-19T07:48:26.328671+0000","last_peered":"2023-12-19T07:48:26.328671+0000","last_clean":"2023-12-19T07:48:26.328671+0000","last_became_active":"2023-12-19T07:48:25.328625+0000","last_became_peered":"2023-12-19T07:48:25.328625+0000","last_unstale":"2023-12-19T07:48:26.328671+0000","last_undegraded":"2023-12-19T07:48:26.328671+0000","last_fullsized":"2023-12-19T07:48:26.328671+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:17.401922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324593+0000","last_change":"2023-12-19T07:48:26.324593+0000","last_active":"2023-12-19T07:48:26.324593+0000","last_peered":"2023-12-19T07:48:26.324593+0000","last_clean":"2023-12-19T07:48:26.324593+0000","last_became_active":"2023-12-19T07:48:26.324256+0000","last_became_peered":"2023-12-19T07:48:26.324256+0000","last_unstale":"2023-12-19T07:48:26.324593+0000","last_undegraded":"2023-12-19T07:48:26.324593+0000","last_fullsized":"2023-12-19T07:48:26.324593+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:56:42.460173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.316266+0000","last_change":"2023-12-19T07:48:25.324855+0000","last_active":"2023-12-19T07:48:26.316266+0000","last_peered":"2023-12-19T07:48:26.316266+0000","last_clean":"2023-12-19T07:48:26.316266+0000","last_became_active":"2023-12-19T07:48:25.324535+0000","last_became_peered":"2023-12-19T07:48:25.324535+0000","last_unstale":"2023-12-19T07:48:26.316266+0000","last_undegraded":"2023-12-19T07:48:26.316266+0000","last_fullsized":"2023-12-19T07:48:26.316266+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:40.692055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326150+0000","last_change":"2023-12-19T07:48:26.326150+0000","last_active":"2023-12-19T07:48:26.326150+0000","last_peered":"2023-12-19T07:48:26.326150+0000","last_clean":"2023-12-19T07:48:26.326150+0000","last_became_active":"2023-12-19T07:48:26.325831+0000","last_became_peered":"2023-12-19T07:48:26.325831+0000","last_unstale":"2023-12-19T07:48:26.326150+0000","last_undegraded":"2023-12-19T07:48:26.326150+0000","last_fullsized":"2023-12-19T07:48:26.326150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:15.060622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.316187+0000","last_change":"2023-12-19T07:48:25.312141+0000","last_active":"2023-12-19T07:48:25.316187+0000","last_peered":"2023-12-19T07:48:25.316187+0000","last_clean":"2023-12-19T07:48:25.316187+0000","last_became_active":"2023-12-19T07:48:25.311800+0000","last_became_peered":"2023-12-19T07:48:25.311800+0000","last_unstale":"2023-12-19T07:48:25.316187+0000","last_undegraded":"2023-12-19T07:48:25.316187+0000","last_fullsized":"2023-12-19T07:48:25.316187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:52:53.274789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324130+0000","last_change":"2023-12-19T07:48:26.324130+0000","last_active":"2023-12-19T07:48:26.324130+0000","last_peered":"2023-12-19T07:48:26.324130+0000","last_clean":"2023-12-19T07:48:26.324130+0000","last_became_active":"2023-12-19T07:48:26.323625+0000","last_became_peered":"2023-12-19T07:48:26.323625+0000","last_unstale":"2023-12-19T07:48:26.324130+0000","last_undegraded":"2023-12-19T07:48:26.324130+0000","last_fullsized":"2023-12-19T07:48:26.324130+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:45:29.145398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.330452+0000","last_change":"2023-12-19T07:48:25.314057+0000","last_active":"2023-12-19T07:48:26.330452+0000","last_peered":"2023-12-19T07:48:26.330452+0000","last_clean":"2023-12-19T07:48:26.330452+0000","last_became_active":"2023-12-19T07:48:25.313662+0000","last_became_peered":"2023-12-19T07:48:25.313662+0000","last_unstale":"2023-12-19T07:48:26.330452+0000","last_undegraded":"2023-12-19T07:48:26.330452+0000","last_fullsized":"2023-12-19T07:48:26.330452+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:36.379068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.327832+0000","last_change":"2023-12-19T07:48:26.327832+0000","last_active":"2023-12-19T07:48:26.327832+0000","last_peered":"2023-12-19T07:48:26.327832+0000","last_clean":"2023-12-19T07:48:26.327832+0000","last_became_active":"2023-12-19T07:48:26.327563+0000","last_became_peered":"2023-12-19T07:48:26.327563+0000","last_unstale":"2023-12-19T07:48:26.327832+0000","last_undegraded":"2023-12-19T07:48:26.327832+0000","last_fullsized":"2023-12-19T07:48:26.327832+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.315784+0000","last_change":"2023-12-19T07:48:25.314368+0000","last_active":"2023-12-19T07:48:25.315784+0000","last_peered":"2023-12-19T07:48:25.315784+0000","last_clean":"2023-12-19T07:48:25.315784+0000","last_became_active":"2023-12-19T07:48:25.314037+0000","last_became_peered":"2023-12-19T07:48:25.314037+0000","last_unstale":"2023-12-19T07:48:25.315784+0000","last_undegraded":"2023-12-19T07:48:25.315784+0000","last_fullsized":"2023-12-19T07:48:25.315784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:03.429854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336268+0000","last_change":"2023-12-19T07:48:26.336268+0000","last_active":"2023-12-19T07:48:26.336268+0000","last_peered":"2023-12-19T07:48:26.336268+0000","last_clean":"2023-12-19T07:48:26.336268+0000","last_became_active":"2023-12-19T07:48:26.335758+0000","last_became_peered":"2023-12-19T07:48:26.335758+0000","last_unstale":"2023-12-19T07:48:26.336268+0000","last_undegraded":"2023-12-19T07:48:26.336268+0000","last_fullsized":"2023-12-19T07:48:26.336268+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:27:00.664922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.336353+0000","last_change":"2023-12-19T07:48:25.331900+0000","last_active":"2023-12-19T07:48:25.336353+0000","last_peered":"2023-12-19T07:48:25.336353+0000","last_clean":"2023-12-19T07:48:25.336353+0000","last_became_active":"2023-12-19T07:48:25.331580+0000","last_became_peered":"2023-12-19T07:48:25.331580+0000","last_unstale":"2023-12-19T07:48:25.336353+0000","last_undegraded":"2023-12-19T07:48:25.336353+0000","last_fullsized":"2023-12-19T07:48:25.336353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:16:24.744942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.320862+0000","last_change":"2023-12-19T07:48:26.320862+0000","last_active":"2023-12-19T07:48:26.320862+0000","last_peered":"2023-12-19T07:48:26.320862+0000","last_clean":"2023-12-19T07:48:26.320862+0000","last_became_active":"2023-12-19T07:48:26.319686+0000","last_became_peered":"2023-12-19T07:48:26.319686+0000","last_unstale":"2023-12-19T07:48:26.320862+0000","last_undegraded":"2023-12-19T07:48:26.320862+0000","last_fullsized":"2023-12-19T07:48:26.320862+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:39:51.239053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320717+0000","last_change":"2023-12-19T07:48:25.315121+0000","last_active":"2023-12-19T07:48:25.320717+0000","last_peered":"2023-12-19T07:48:25.320717+0000","last_clean":"2023-12-19T07:48:25.320717+0000","last_became_active":"2023-12-19T07:48:25.314836+0000","last_became_peered":"2023-12-19T07:48:25.314836+0000","last_unstale":"2023-12-19T07:48:25.320717+0000","last_undegraded":"2023-12-19T07:48:25.320717+0000","last_fullsized":"2023-12-19T07:48:25.320717+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:52.481514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.336744+0000","last_change":"2023-12-19T07:48:26.336744+0000","last_active":"2023-12-19T07:48:26.336744+0000","last_peered":"2023-12-19T07:48:26.336744+0000","last_clean":"2023-12-19T07:48:26.336744+0000","last_became_active":"2023-12-19T07:48:26.336484+0000","last_became_peered":"2023-12-19T07:48:26.336484+0000","last_unstale":"2023-12-19T07:48:26.336744+0000","last_undegraded":"2023-12-19T07:48:26.336744+0000","last_fullsized":"2023-12-19T07:48:26.336744+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.350690+0000","last_change":"2023-12-19T07:48:25.321713+0000","last_active":"2023-12-19T07:48:26.350690+0000","last_peered":"2023-12-19T07:48:26.350690+0000","last_clean":"2023-12-19T07:48:26.350690+0000","last_became_active":"2023-12-19T07:48:25.321434+0000","last_became_peered":"2023-12-19T07:48:25.321434+0000","last_unstale":"2023-12-19T07:48:26.350690+0000","last_undegraded":"2023-12-19T07:48:26.350690+0000","last_fullsized":"2023-12-19T07:48:26.350690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:32:50.370509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.334902+0000","last_change":"2023-12-19T07:48:26.334902+0000","last_active":"2023-12-19T07:48:26.334902+0000","last_peered":"2023-12-19T07:48:26.334902+0000","last_clean":"2023-12-19T07:48:26.334902+0000","last_became_active":"2023-12-19T07:48:26.334624+0000","last_became_peered":"2023-12-19T07:48:26.334624+0000","last_unstale":"2023-12-19T07:48:26.334902+0000","last_undegraded":"2023-12-19T07:48:26.334902+0000","last_fullsized":"2023-12-19T07:48:26.334902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:28:44.234923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.330382+0000","last_change":"2023-12-19T07:48:25.327789+0000","last_active":"2023-12-19T07:48:25.330382+0000","last_peered":"2023-12-19T07:48:25.330382+0000","last_clean":"2023-12-19T07:48:25.330382+0000","last_became_active":"2023-12-19T07:48:25.327422+0000","last_became_peered":"2023-12-19T07:48:25.327422+0000","last_unstale":"2023-12-19T07:48:25.330382+0000","last_undegraded":"2023-12-19T07:48:25.330382+0000","last_fullsized":"2023-12-19T07:48:25.330382+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:46:06.770421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.325056+0000","last_change":"2023-12-19T07:48:26.325056+0000","last_active":"2023-12-19T07:48:26.325056+0000","last_peered":"2023-12-19T07:48:26.325056+0000","last_clean":"2023-12-19T07:48:26.325056+0000","last_became_active":"2023-12-19T07:48:26.324774+0000","last_became_peered":"2023-12-19T07:48:26.324774+0000","last_unstale":"2023-12-19T07:48:26.325056+0000","last_undegraded":"2023-12-19T07:48:26.325056+0000","last_fullsized":"2023-12-19T07:48:26.325056+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:58.126030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.323933+0000","last_change":"2023-12-19T07:48:25.316753+0000","last_active":"2023-12-19T07:48:25.323933+0000","last_peered":"2023-12-19T07:48:25.323933+0000","last_clean":"2023-12-19T07:48:25.323933+0000","last_became_active":"2023-12-19T07:48:25.316423+0000","last_became_peered":"2023-12-19T07:48:25.316423+0000","last_unstale":"2023-12-19T07:48:25.323933+0000","last_undegraded":"2023-12-19T07:48:25.323933+0000","last_fullsized":"2023-12-19T07:48:25.323933+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:25.350499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.326058+0000","last_change":"2023-12-19T07:48:26.326058+0000","last_active":"2023-12-19T07:48:26.326058+0000","last_peered":"2023-12-19T07:48:26.326058+0000","last_clean":"2023-12-19T07:48:26.326058+0000","last_became_active":"2023-12-19T07:48:26.325479+0000","last_became_peered":"2023-12-19T07:48:26.325479+0000","last_unstale":"2023-12-19T07:48:26.326058+0000","last_undegraded":"2023-12-19T07:48:26.326058+0000","last_fullsized":"2023-12-19T07:48:26.326058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:25.214169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320862+0000","last_change":"2023-12-19T07:48:25.314957+0000","last_active":"2023-12-19T07:48:25.320862+0000","last_peered":"2023-12-19T07:48:25.320862+0000","last_clean":"2023-12-19T07:48:25.320862+0000","last_became_active":"2023-12-19T07:48:25.314664+0000","last_became_peered":"2023-12-19T07:48:25.314664+0000","last_unstale":"2023-12-19T07:48:25.320862+0000","last_undegraded":"2023-12-19T07:48:25.320862+0000","last_fullsized":"2023-12-19T07:48:25.320862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:03.366692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.341804+0000","last_change":"2023-12-19T07:48:26.341804+0000","last_active":"2023-12-19T07:48:26.341804+0000","last_peered":"2023-12-19T07:48:26.341804+0000","last_clean":"2023-12-19T07:48:26.341804+0000","last_became_active":"2023-12-19T07:48:26.341522+0000","last_became_peered":"2023-12-19T07:48:26.341522+0000","last_unstale":"2023-12-19T07:48:26.341804+0000","last_undegraded":"2023-12-19T07:48:26.341804+0000","last_fullsized":"2023-12-19T07:48:26.341804+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.322886+0000","last_change":"2023-12-19T07:48:25.315743+0000","last_active":"2023-12-19T07:48:25.322886+0000","last_peered":"2023-12-19T07:48:25.322886+0000","last_clean":"2023-12-19T07:48:25.322886+0000","last_became_active":"2023-12-19T07:48:25.315454+0000","last_became_peered":"2023-12-19T07:48:25.315454+0000","last_unstale":"2023-12-19T07:48:25.322886+0000","last_undegraded":"2023-12-19T07:48:25.322886+0000","last_fullsized":"2023-12-19T07:48:25.322886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:45.891457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.324363+0000","last_change":"2023-12-19T07:48:26.324363+0000","last_active":"2023-12-19T07:48:26.324363+0000","last_peered":"2023-12-19T07:48:26.324363+0000","last_clean":"2023-12-19T07:48:26.324363+0000","last_became_active":"2023-12-19T07:48:26.324028+0000","last_became_peered":"2023-12-19T07:48:26.324028+0000","last_unstale":"2023-12-19T07:48:26.324363+0000","last_undegraded":"2023-12-19T07:48:26.324363+0000","last_fullsized":"2023-12-19T07:48:26.324363+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:29.265447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.331235+0000","last_change":"2023-12-19T07:48:25.323261+0000","last_active":"2023-12-19T07:48:25.331235+0000","last_peered":"2023-12-19T07:48:25.331235+0000","last_clean":"2023-12-19T07:48:25.331235+0000","last_became_active":"2023-12-19T07:48:25.323058+0000","last_became_peered":"2023-12-19T07:48:25.323058+0000","last_unstale":"2023-12-19T07:48:25.331235+0000","last_undegraded":"2023-12-19T07:48:25.331235+0000","last_fullsized":"2023-12-19T07:48:25.331235+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:07:31.841246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.337098+0000","last_change":"2023-12-19T07:48:26.337098+0000","last_active":"2023-12-19T07:48:26.337098+0000","last_peered":"2023-12-19T07:48:26.337098+0000","last_clean":"2023-12-19T07:48:26.337098+0000","last_became_active":"2023-12-19T07:48:26.336844+0000","last_became_peered":"2023-12-19T07:48:26.336844+0000","last_unstale":"2023-12-19T07:48:26.337098+0000","last_undegraded":"2023-12-19T07:48:26.337098+0000","last_fullsized":"2023-12-19T07:48:26.337098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:29.199529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.318790+0000","last_change":"2023-12-19T07:48:25.308759+0000","last_active":"2023-12-19T07:48:25.318790+0000","last_peered":"2023-12-19T07:48:25.318790+0000","last_clean":"2023-12-19T07:48:25.318790+0000","last_became_active":"2023-12-19T07:48:25.308425+0000","last_became_peered":"2023-12-19T07:48:25.308425+0000","last_unstale":"2023-12-19T07:48:25.318790+0000","last_undegraded":"2023-12-19T07:48:25.318790+0000","last_fullsized":"2023-12-19T07:48:25.318790+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:42:20.577530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":8,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-19T07:48:26.345244+0000","last_change":"2023-12-19T07:48:26.345244+0000","last_active":"2023-12-19T07:48:26.345244+0000","last_peered":"2023-12-19T07:48:26.345244+0000","last_clean":"2023-12-19T07:48:26.345244+0000","last_became_active":"2023-12-19T07:48:26.338717+0000","last_became_peered":"2023-12-19T07:48:26.338717+0000","last_unstale":"2023-12-19T07:48:26.345244+0000","last_undegraded":"2023-12-19T07:48:26.345244+0000","last_fullsized":"2023-12-19T07:48:26.345244+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:25.277769+0000","last_clean_scrub_stamp":"2023-12-19T07:48:25.277769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:10:54.320854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.335516+0000","last_change":"2023-12-19T07:48:25.329960+0000","last_active":"2023-12-19T07:48:25.335516+0000","last_peered":"2023-12-19T07:48:25.335516+0000","last_clean":"2023-12-19T07:48:25.335516+0000","last_became_active":"2023-12-19T07:48:25.328098+0000","last_became_peered":"2023-12-19T07:48:25.328098+0000","last_unstale":"2023-12-19T07:48:25.335516+0000","last_undegraded":"2023-12-19T07:48:25.335516+0000","last_fullsized":"2023-12-19T07:48:25.335516+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:48:24.262076+0000","last_clean_scrub_stamp":"2023-12-19T07:48:24.262076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:45.891457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-19T07:48:25.320511+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T07:48:25.320511+0000","last_peered":"2023-12-19T07:48:25.320511+0000","last_clean":"2023-12-19T07:48:25.320511+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T07:48:25.320511+0000","last_undegraded":"2023-12-19T07:48:25.320511+0000","last_fullsized":"2023-12-19T07:48:25.320511+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298296,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073544,"statfs":{"total":96636764160,"available":96331309056,"internally_reserved":0,"allocated":1032192,"data_stored":777243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.4750000000000001}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.97599999999999998}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1459999999999999}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.306,"15min":1.306},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.359},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":1.3779999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.966,"5min":1.966,"15min":1.966},"last":1.522},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.3089999999999999}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":2.8279999999999998,"5min":2.8279999999999998,"15min":2.8279999999999998},"last":1.095},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.417}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.097,"5min":3.097,"15min":3.097},"last":1.452}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.409}]},{"osd":1,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.173},{"interface":"front","average":{"1min":1.21,"5min":1.21,"15min":1.21},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.224}]},{"osd":2,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.111}]},{"osd":3,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.3129999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":1.268,"5min":1.268,"15min":1.268},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":1.276}]},{"osd":5,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.3680000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2639999999999998},"last":1.496}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.107}]},{"osd":1,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.05},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":3.113,"5min":3.113,"15min":3.113},"last":0.91100000000000003}]},{"osd":2,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.9039999999999999,"5min":2.9039999999999999,"15min":2.9039999999999999},"last":0.95899999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":3.5870000000000002,"5min":3.5870000000000002,"15min":3.5870000000000002},"last":1.254},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":4.3879999999999999,"5min":4.3879999999999999,"15min":4.3879999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":5.0819999999999999,"5min":5.0819999999999999,"15min":5.0819999999999999},"last":1.151},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":4.6310000000000002,"5min":4.6310000000000002,"15min":4.6310000000000002},"last":1.458}]},{"osd":6,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.583,"15min":1.583},"min":{"1min":1.01,"5min":1.01,"15min":1.01},"max":{"1min":3.9649999999999999,"5min":3.9649999999999999,"15min":3.9649999999999999},"last":1.411},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":1.2989999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:09 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":4.4539999999999997,"5min":4.4539999999999997,"15min":4.4539999999999997},"last":1.512},{"interface":"front","average":{"1min":1.714,"5min":1.714,"15min":1.714},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":4.1980000000000004,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":1.486}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.1879999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.7490000000000001}]},{"osd":2,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":1.056,"5min":1.056,"15min":1.056},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.446},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.661,"5min":2.661,"15min":2.661},"last":1.242}]},{"osd":5,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":3.161,"5min":3.161,"15min":3.161},"last":1.407},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.591}]},{"osd":6,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"max":{"1min":2.923,"5min":2.923,"15min":2.923},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.135,"5min":1.135,"15min":1.135},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.649,"15min":1.649},"min":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.7170000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297744,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074096,"statfs":{"total":96636764160,"available":96331874304,"internally_reserved":0,"allocated":466944,"data_stored":195528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.74099999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":4.8879999999999999,"5min":4.8879999999999999,"15min":4.8879999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.609,"5min":1.609,"15min":1.609},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":8.0830000000000002,"5min":8.0830000000000002,"15min":8.0830000000000002},"last":1.272}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.85299999999999998}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":4.7130000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"last":1.167},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":5.0220000000000002,"5min":5.0220000000000002,"15min":5.0220000000000002},"last":1.397}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.995},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"min":{"1min":1.05,"5min":1.05,"15min":1.05},"max":{"1min":8.5299999999999994,"5min":8.5299999999999994,"15min":8.5299999999999994},"last":1.139}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"min":{"1min":1.26,"5min":1.26,"15min":1.26},"max":{"1min":7.2750000000000004,"5min":7.2750000000000004,"15min":7.2750000000000004},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":4.6550000000000002,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":1.349}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":16.081,"5min":16.081,"15min":16.081},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.351,"5min":2.351,"15min":2.351},"min":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":15.285,"5min":15.285,"15min":15.285},"last":1.478}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":4.093,"5min":4.093,"15min":4.093},"last":1.0429999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":3.9089999999999998},"last":1.131}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.96399999999999997}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":3.9239999999999999,"5min":3.9239999999999999,"15min":3.9239999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":4.0170000000000003,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":1.3080000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.948,"5min":3.948,"15min":3.948},"last":1.49},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":1.101}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"min":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"max":{"1min":3.9620000000000002,"5min":3.9620000000000002,"15min":3.9620000000000002},"last":1.427}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":3.9809999999999999},"last":1.579},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.998},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.452}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":186875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.61099999999999999}]},{"osd":2,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2609999999999999}]},{"osd":3,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.78200000000000003}]},{"osd":4,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.244,"15min":1.244},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.995},{"interface":"front","average":{"1min":1.379,"5min":1.379,"15min":1.379},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":1.026}]},{"osd":5,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.24}]},{"osd":6,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":1.131,"5min":1.131,"15min":1.131},"max":{"1min":3.024,"5min":3.024,"15min":3.024},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":1.0640000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:48:06 2023","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.621,"15min":1.621},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":3.6080000000000001,"5min":3.6080000000000001,"15min":3.6080000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":1.1020000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298308,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073532,"statfs":{"total":96636764160,"available":96331296768,"internally_reserved":0,"allocated":1044480,"data_stored":783349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.8129999999999997,"5min":5.8129999999999997,"15min":5.8129999999999997},"last":1.083},{"interface":"front","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.89800000000000002}]},{"osd":3,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.018},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.605}]},{"osd":5,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":1.028,"5min":1.028,"15min":1.028},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":1.1100000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.2929999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:48:07 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.671},{"interface":"front","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.028,"5min":1.028,"15min":1.028},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.7609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T07:48:31.023 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T07:48:31.024 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T07:48:31.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 Processor -- start 2023-12-19T07:48:31.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 -- start start 2023-12-19T07:48:31.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4d580c6d20 0x7f4d580c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 0x7f4d5806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5806a7a0 con 0x7f4d5815aa30 2023-12-19T07:48:31.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d580c7630 con 0x7f4d580c6d20 2023-12-19T07:48:31.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d60185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5806d4b0 con 0x7f4d5806c7c0 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.263+0000 7f4d5e722700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4d5815aa30 0x7f4d5815ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33912/0 (socket says 172.21.15.111:33912) 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5e722700 1 -- 172.21.15.111:0/809272842 learned_addr learned my addr 172.21.15.111:0/809272842 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 --2- 172.21.15.111:0/809272842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 0x7f4d5806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5df21700 1 --2- 172.21.15.111:0/809272842 >> v2:172.21.15.143:3300/0 conn(0x7f4d580c6d20 0x7f4d580c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 -- 172.21.15.111:0/809272842 >> v2:172.21.15.143:3300/0 conn(0x7f4d580c6d20 msgr2=0x7f4d580c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 --2- 172.21.15.111:0/809272842 >> v2:172.21.15.143:3300/0 conn(0x7f4d580c6d20 0x7f4d580c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 -- 172.21.15.111:0/809272842 >> v1:172.21.15.111:6789/0 conn(0x7f4d5815aa30 legacy=0x7f4d5815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 -- 172.21.15.111:0/809272842 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5806d790 con 0x7f4d5806c7c0 2023-12-19T07:48:31.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5d720700 1 --2- 172.21.15.111:0/809272842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 0x7f4d5806cc60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d480098e0 tx=0x7f4d480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca9653437c3ed63 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5cf1f700 1 -- 172.21.15.111:0/809272842 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800b640 con 0x7f4d5806c7c0 2023-12-19T07:48:31.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5cf1f700 1 -- 172.21.15.111:0/809272842 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4800b7a0 con 0x7f4d5806c7c0 2023-12-19T07:48:31.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d5cf1f700 1 -- 172.21.15.111:0/809272842 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800bb40 con 0x7f4d5806c7c0 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 -- 172.21.15.111:0/809272842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 msgr2=0x7f4d5806cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 --2- 172.21.15.111:0/809272842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 0x7f4d5806cc60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d480098e0 tx=0x7f4d480091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 -- 172.21.15.111:0/809272842 shutdown_connections 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 --2- 172.21.15.111:0/809272842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d5806c7c0 0x7f4d5806cc60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 --2- 172.21.15.111:0/809272842 >> v2:172.21.15.143:3300/0 conn(0x7f4d580c6d20 0x7f4d580c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 -- 172.21.15.111:0/809272842 >> 172.21.15.111:0/809272842 conn(0x7f4d58067bb0 msgr2=0x7f4d580cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:31.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 -- 172.21.15.111:0/809272842 shutdown_connections 2023-12-19T07:48:31.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.267+0000 7f4d60185700 1 -- 172.21.15.111:0/809272842 wait complete. 2023-12-19T07:48:31.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 Processor -- start 2023-12-19T07:48:31.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 -- start start 2023-12-19T07:48:31.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d5806c7c0 0x7f4d5816da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d580c6d20 0x7f4d58153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580ae250 con 0x7f4d5815aa30 2023-12-19T07:48:31.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580af0a0 con 0x7f4d5806c7c0 2023-12-19T07:48:31.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d60185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580b1620 con 0x7f4d580c6d20 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5d720700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d580c6d20 0x7f4d58153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44886/0 (socket says 172.21.15.111:44886) 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5df21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d5806c7c0 0x7f4d5816da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 -- 172.21.15.111:0/1603597347 learned_addr learned my addr 172.21.15.111:0/1603597347 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 -- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d580c6d20 msgr2=0x7f4d58153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d580c6d20 0x7f4d58153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 -- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d5806c7c0 msgr2=0x7f4d5816da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d5806c7c0 0x7f4d5816da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.271+0000 7f4d5e722700 1 -- 172.21.15.111:0/1603597347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48009420 con 0x7f4d5815aa30 2023-12-19T07:48:31.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d5e722700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000c560 tx=0x7f4d50007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50005b30 con 0x7f4d5815aa30 2023-12-19T07:48:31.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50005c90 con 0x7f4d5815aa30 2023-12-19T07:48:31.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d580b1900 con 0x7f4d5815aa30 2023-12-19T07:48:31.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500072e0 con 0x7f4d5815aa30 2023-12-19T07:48:31.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d58153e00 con 0x7f4d5815aa30 2023-12-19T07:48:31.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.275+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d58071120 con 0x7f4d5815aa30 2023-12-19T07:48:31.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.279+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4d50007690 con 0x7f4d5815aa30 2023-12-19T07:48:31.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.279+0000 7f4d4dffb700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 0x7f4d3c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.279+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4d5007bbd0 con 0x7f4d5815aa30 2023-12-19T07:48:31.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.279+0000 7f4d5df21700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 0x7f4d3c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.283+0000 7f4d5df21700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 0x7f4d3c062a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400f570 tx=0x7f4d5400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.287+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4d50049be0 con 0x7f4d5815aa30 2023-12-19T07:48:31.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.527+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4d5806da70 con 0x7f4d5815aa30 2023-12-19T07:48:31.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d4dffb700 1 -- 172.21.15.111:0/1603597347 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4d500170e0 con 0x7f4d5815aa30 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 msgr2=0x7f4d3c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 0x7f4d3c062a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400f570 tx=0x7f4d5400b040 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 msgr2=0x7f4d580b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000c560 tx=0x7f4d50007f60 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 shutdown_connections 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4d3c060550 0x7f4d3c062a00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d5815aa30 0x7f4d580b0700 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d580c6d20 0x7f4d58153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 --2- 172.21.15.111:0/1603597347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d5806c7c0 0x7f4d5816da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 >> 172.21.15.111:0/1603597347 conn(0x7f4d58067bb0 msgr2=0x7f4d580aad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 shutdown_connections 2023-12-19T07:48:31.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.535+0000 7f4d60185700 1 -- 172.21.15.111:0/1603597347 wait complete. 2023-12-19T07:48:31.523 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T07:48:31.550 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:48:31.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 Processor -- start 2023-12-19T07:48:31.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 -- start start 2023-12-19T07:48:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 0x7f028806a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028806c7c0 con 0x7f028806d210 2023-12-19T07:48:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028806cd80 con 0x7f02880b1410 2023-12-19T07:48:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028d4e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028806af10 con 0x7f028806a580 2023-12-19T07:48:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028cce2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f028806d210 0x7f028806d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48496/0 (socket says 172.21.15.111:48496) 2023-12-19T07:48:31.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f028cce2700 1 -- 172.21.15.111:0/44791099 learned_addr learned my addr 172.21.15.111:0/44791099 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:31.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.723+0000 7f027ffff700 1 --2- 172.21.15.111:0/44791099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 0x7f028806a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f027ffff700 1 -- 172.21.15.111:0/44791099 >> v1:172.21.15.143:6789/0 conn(0x7f02880b1410 legacy=0x7f02880ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f027ffff700 1 -- 172.21.15.111:0/44791099 >> v1:172.21.15.111:6789/0 conn(0x7f028806d210 legacy=0x7f028806d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f027ffff700 1 -- 172.21.15.111:0/44791099 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028806b8f0 con 0x7f028806a580 2023-12-19T07:48:31.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f02877fe700 1 -- 172.21.15.111:0/44791099 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179879598 0 0) 0x7f028806c7c0 con 0x7f028806d210 2023-12-19T07:48:31.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f027ffff700 1 --2- 172.21.15.111:0/44791099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 0x7f028806a9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0270009890 tx=0x7f02700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c1aa26900d9b04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f02877fe700 1 -- 172.21.15.111:0/44791099 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270011630 con 0x7f028806a580 2023-12-19T07:48:31.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f02877fe700 1 -- 172.21.15.111:0/44791099 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0270011790 con 0x7f028806a580 2023-12-19T07:48:31.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f02877fe700 1 -- 172.21.15.111:0/44791099 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270011b50 con 0x7f028806a580 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 -- 172.21.15.111:0/44791099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 msgr2=0x7f028806a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 --2- 172.21.15.111:0/44791099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 0x7f028806a9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0270009890 tx=0x7f02700091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 -- 172.21.15.111:0/44791099 shutdown_connections 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 --2- 172.21.15.111:0/44791099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806a580 0x7f028806a9d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 -- 172.21.15.111:0/44791099 >> 172.21.15.111:0/44791099 conn(0x7f0288067bb0 msgr2=0x7f02880cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 -- 172.21.15.111:0/44791099 shutdown_connections 2023-12-19T07:48:31.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.727+0000 7f028d4e3700 1 -- 172.21.15.111:0/44791099 wait complete. 2023-12-19T07:48:31.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 Processor -- start 2023-12-19T07:48:31.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 -- start start 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806d210 0x7f028816c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 0x7f0288149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02881577b0 con 0x7f028806a580 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028816d0a0 con 0x7f02880b1410 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f028d4e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028816d380 con 0x7f028806d210 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f0287fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f0287fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46700/0 (socket says 172.21.15.111:46700) 2023-12-19T07:48:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.731+0000 7f0287fff700 1 -- 172.21.15.111:0/601626105 learned_addr learned my addr 172.21.15.111:0/601626105 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f027ffff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806d210 0x7f028816c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f028cce2700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 0x7f0288149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 -- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806d210 msgr2=0x7f028816c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806d210 0x7f028816c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 -- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 msgr2=0x7f0288149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 0x7f0288149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 -- 172.21.15.111:0/601626105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270009420 con 0x7f028806a580 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f028cce2700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 0x7f0288149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0287fff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f027800c5c0 tx=0x7f027800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278011590 con 0x7f028806a580 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0278011710 con 0x7f028806a580 2023-12-19T07:48:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0288157440 con 0x7f028806a580 2023-12-19T07:48:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278011ad0 con 0x7f028806a580 2023-12-19T07:48:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028816ccb0 con 0x7f028806a580 2023-12-19T07:48:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.735+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028804be40 con 0x7f028806a580 2023-12-19T07:48:31.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.739+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f027801a360 con 0x7f028806a580 2023-12-19T07:48:31.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.739+0000 7f0284ff9700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 0x7f026c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:31.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.739+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f027807b9e0 con 0x7f028806a580 2023-12-19T07:48:31.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.739+0000 7f027ffff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 0x7f026c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:31.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.743+0000 7f027ffff700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 0x7f026c062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0270009860 tx=0x7f0270016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:31.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.743+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f027804be90 con 0x7f028806a580 2023-12-19T07:48:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.947+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02880ab220 con 0x7f028806a580 2023-12-19T07:48:31.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f0284ff9700 1 -- 172.21.15.111:0/601626105 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f027804f4b0 con 0x7f028806a580 2023-12-19T07:48:31.934 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:48:31.934 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 msgr2=0x7f026c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 0x7f026c062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0270009860 tx=0x7f0270016000 comp rx=0 tx=0).stop 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 msgr2=0x7f0288157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f027800c5c0 tx=0x7f027800c8d0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 shutdown_connections 2023-12-19T07:48:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f026c060550 0x7f026c062a00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02880b1410 0x7f0288149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f028806d210 0x7f028816c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 --2- 172.21.15.111:0/601626105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f028806a580 0x7f0288157c40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 >> 172.21.15.111:0/601626105 conn(0x7f0288067bb0 msgr2=0x7f02880c9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 shutdown_connections 2023-12-19T07:48:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:31.951+0000 7f028d4e3700 1 -- 172.21.15.111:0/601626105 wait complete. 2023-12-19T07:48:31.939 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:48:32.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T07:48:25.748981+0000', 'modified': '2023-12-19T07:48:29.337126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5110}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5110': {'gid': 5110, 'name': 'b', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.143:6835/1413455479', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1413455479}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1413455479}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T07:48:32.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T07:48:32.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T07:48:32.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.543+0000 7f1f330ae700 1 Processor -- start 2023-12-19T07:48:32.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.543+0000 7f1f330ae700 1 -- start start 2023-12-19T07:48:32.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f330ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f2c15d050 0x7f1f2c06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f330ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f330ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c06ba50 con 0x7f1f2c0afd20 2023-12-19T07:48:32.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f330ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c06bee0 con 0x7f1f2c15d050 2023-12-19T07:48:32.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f330ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0b01a0 con 0x7f1f2c160e60 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f2c15d050 0x7f1f2c06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f2c15d050 0x7f1f2c06a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46710/0 (socket says 172.21.15.111:46710) 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1610143632 learned_addr learned my addr 172.21.15.111:0/1610143632 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1610143632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 msgr2=0x7f1f2c0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c0af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1610143632 >> v1:172.21.15.143:6789/0 conn(0x7f1f2c0afd20 legacy=0x7f1f2c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1610143632 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0af600 con 0x7f1f2c15d050 2023-12-19T07:48:32.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c15d050 0x7f1f2c06a740 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1f200099b0 tx=0x7f1f20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dfb6c043300eae server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f2b7fe700 1 -- 172.21.15.111:0/1610143632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000b860 con 0x7f1f2c15d050 2023-12-19T07:48:32.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f2b7fe700 1 -- 172.21.15.111:0/1610143632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000b9c0 con 0x7f1f2c15d050 2023-12-19T07:48:32.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.547+0000 7f1f2b7fe700 1 -- 172.21.15.111:0/1610143632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000bd60 con 0x7f1f2c15d050 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- 172.21.15.111:0/1610143632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c15d050 msgr2=0x7f1f2c06a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c15d050 0x7f1f2c06a740 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1f200099b0 tx=0x7f1f20009250 comp rx=0 tx=0).stop 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- 172.21.15.111:0/1610143632 shutdown_connections 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c0af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 --2- 172.21.15.111:0/1610143632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c15d050 0x7f1f2c06a740 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- 172.21.15.111:0/1610143632 >> 172.21.15.111:0/1610143632 conn(0x7f1f2c067b60 msgr2=0x7f1f2c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- 172.21.15.111:0/1610143632 shutdown_connections 2023-12-19T07:48:32.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- 172.21.15.111:0/1610143632 wait complete. 2023-12-19T07:48:32.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 Processor -- start 2023-12-19T07:48:32.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.551+0000 7f1f330ae700 1 -- start start 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f2c15d050 0x7f1f2c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0aa040 con 0x7f1f2c0afd20 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0aa320 con 0x7f1f2c15d050 2023-12-19T07:48:32.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0aa600 con 0x7f1f2c160e60 2023-12-19T07:48:32.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46714/0 (socket says 172.21.15.111:46714) 2023-12-19T07:48:32.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 -- 172.21.15.111:0/556555834 learned_addr learned my addr 172.21.15.111:0/556555834 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f3164b700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f2c15d050 0x7f1f2c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 -- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 msgr2=0x7f1f2c15ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c15ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 -- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f2c15d050 msgr2=0x7f1f2c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f2c15d050 0x7f1f2c15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 -- 172.21.15.111:0/556555834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200094f0 con 0x7f1f2c0afd20 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009290 tx=0x7f1f20017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f28ff9700 1 -- 172.21.15.111:0/556555834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20024440 con 0x7f1f2c0afd20 2023-12-19T07:48:32.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f28ff9700 1 -- 172.21.15.111:0/556555834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f200245a0 con 0x7f1f2c0afd20 2023-12-19T07:48:32.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c0aa8e0 con 0x7f1f2c0afd20 2023-12-19T07:48:32.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f28ff9700 1 -- 172.21.15.111:0/556555834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20024960 con 0x7f1f2c0afd20 2023-12-19T07:48:32.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.555+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c1595c0 con 0x7f1f2c0afd20 2023-12-19T07:48:32.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.559+0000 7f1f28ff9700 1 -- 172.21.15.111:0/556555834 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1f2001a420 con 0x7f1f2c0afd20 2023-12-19T07:48:32.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.559+0000 7f1f28ff9700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 0x7f1f10062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.559+0000 7f1f28ff9700 1 -- 172.21.15.111:0/556555834 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1f2007e700 con 0x7f1f2c0afd20 2023-12-19T07:48:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.559+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 0x7f1f10062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 0x7f1f10062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0cb4c0 tx=0x7f1f18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 Processor -- start 2023-12-19T07:48:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 -- start start 2023-12-19T07:48:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x26e2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x260ff30 con 0x24af790 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x260bed0 con 0x2464710 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x25d20b0 con 0x2464ae0 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3164b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3164b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46728/0 (socket says 172.21.15.111:46728) 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f3164b700 1 -- 172.21.15.111:0/4110596120 learned_addr learned my addr 172.21.15.111:0/4110596120 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x26e2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.563+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 -- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 msgr2=0x26e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 -- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 msgr2=0x26e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x26e2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 -- 172.21.15.111:0/4110596120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24b09e0 con 0x24af790 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3164b700 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1f24003ce0 tx=0x7f1f24009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e29bc85d3a0234 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f15ffb700 1 -- 172.21.15.111:0/4110596120 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24004120 con 0x24af790 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f15ffb700 1 -- 172.21.15.111:0/4110596120 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2400e860 con 0x24af790 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f15ffb700 1 -- 172.21.15.111:0/4110596120 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400ec20 con 0x24af790 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- 172.21.15.111:0/4110596120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 msgr2=0x26e2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1f24003ce0 tx=0x7f1f24009040 comp rx=0 tx=0).stop 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- 172.21.15.111:0/4110596120 shutdown_connections 2023-12-19T07:48:32.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x26e2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x24af790 0x26e2980 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 --2- 172.21.15.111:0/4110596120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- 172.21.15.111:0/4110596120 >> 172.21.15.111:0/4110596120 conn(0x2464eb0 msgr2=0x269e930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- 172.21.15.111:0/4110596120 shutdown_connections 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- 172.21.15.111:0/4110596120 wait complete. 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.567+0000 7f1f3533b740 1 -- start start 2023-12-19T07:48:32.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 Processor -- start 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x25d1d50 con 0x2464710 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x25aa970 con 0x24af790 2023-12-19T07:48:32.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x265b480 con 0x2464ae0 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 10 client.0 _ll_register_callbacks cb 0x26556a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46732/0 (socket says 172.21.15.111:46732) 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1382619088 learned_addr learned my addr 172.21.15.111:0/1382619088 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3164b700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 msgr2=0x26e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 msgr2=0x26c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1382619088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2400f040 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1f2004ecd0 tx=0x7f1f2007e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704cd49bc7030488 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f0affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2007cea0 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1f20067e40 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x25aacc0 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f200597b0 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 10 client.5271 fetch_fsmap learned FSMap version 16 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.571+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x25da370 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x25aacc0 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 16) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f1f20052a70 con 0x2464710 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 10 client.5271 fetch_fsmap finished waiting for FSMap version 16 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 10 client.5271 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:48:32.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 20 client.5271 populate_metadata read hostname 'smithi111' 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x26c92d0 0x26f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 4 client.5271 mds_command: new command op to 5110 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 --> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x25ae3e0 con 0x26c92d0 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x26c92d0 0x26f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f0affd700 10 client.5271 ms_handle_connect on v2:172.21.15.143:6834/1413455479 2023-12-19T07:48:32.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.575+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x26c92d0 0x26f16b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e3aada943ded3c server_cookie=1906b49eaa5dee49 in_seq=0 out_seq=0 2023-12-19T07:48:32.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f0affd700 1 -- 172.21.15.111:0/1382619088 <== mds.0 v2:172.21.15.143:6834/1413455479 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x25ae3e0 con 0x26c92d0 2023-12-19T07:48:32.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f0affd700 10 client.5271 handle_command_reply: tid=0 2023-12-19T07:48:32.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 client.5271 shutdown 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 msgr2=0x26e6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1f2004ecd0 tx=0x7f1f2007e040 comp rx=0 tx=0).stop 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 shutdown_connections 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x26c92d0 0x26f16b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x26e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1382619088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x26e6510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f0affd700 0 client.5271 ms_handle_reset on v2:172.21.15.143:6834/1413455479 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 >> 172.21.15.111:0/1382619088 conn(0x2464eb0 msgr2=0x2696f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 shutdown_connections 2023-12-19T07:48:32.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 1 -- 172.21.15.111:0/1382619088 wait complete. 2023-12-19T07:48:32.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.579+0000 7f1f3533b740 20 client.5271 trim_cache size 0 max 16384 2023-12-19T07:48:32.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.595+0000 7f1f3533b740 1 Processor -- start 2023-12-19T07:48:32.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- start start 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x2699010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x25dff40 con 0x2464710 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2607b50 con 0x24af790 2023-12-19T07:48:32.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x25e1550 con 0x2464ae0 2023-12-19T07:48:32.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46740/0 (socket says 172.21.15.111:46740) 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1895730903 learned_addr learned my addr 172.21.15.111:0/1895730903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3164b700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x2699010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 msgr2=0x2699010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x2699010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 msgr2=0x26e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 -- 172.21.15.111:0/1895730903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25c0d80 con 0x2464710 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x2699010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3164b700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:32.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000b3e0 tx=0x7f1f20069000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb998d5c6dc0efa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f167fc700 1 -- 172.21.15.111:0/1895730903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20052f10 con 0x2464710 2023-12-19T07:48:32.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f167fc700 1 -- 172.21.15.111:0/1895730903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f200597b0 con 0x2464710 2023-12-19T07:48:32.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f167fc700 1 -- 172.21.15.111:0/1895730903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20055a60 con 0x2464710 2023-12-19T07:48:32.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- 172.21.15.111:0/1895730903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 msgr2=0x2698ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000b3e0 tx=0x7f1f20069000 comp rx=0 tx=0).stop 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- 172.21.15.111:0/1895730903 shutdown_connections 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x24af790 0x26e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2464ae0 0x2699010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 --2- 172.21.15.111:0/1895730903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464710 0x2698ad0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- 172.21.15.111:0/1895730903 >> 172.21.15.111:0/1895730903 conn(0x2464eb0 msgr2=0x26981a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.599+0000 7f1f3533b740 1 -- 172.21.15.111:0/1895730903 shutdown_connections 2023-12-19T07:48:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.603+0000 7f1f3533b740 1 -- 172.21.15.111:0/1895730903 wait complete. 2023-12-19T07:48:32.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.603+0000 7f1f3533b740 1 -- start start 2023-12-19T07:48:32.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.603+0000 7f1f3533b740 1 Processor -- start 2023-12-19T07:48:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x2697da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x24af790 0x269ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2699f00 con 0x2464ae0 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2612400 con 0x2464710 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x260c770 con 0x24af790 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 10 client.0 _ll_register_callbacks cb 0x24309d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46756/0 (socket says 172.21.15.111:46756) 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 -- 172.21.15.111:0/446134003 learned_addr learned my addr 172.21.15.111:0/446134003 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3164b700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x24af790 0x269ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x2697da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 -- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x24af790 msgr2=0x269ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x24af790 0x269ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 -- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 msgr2=0x2697da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x2697da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 -- 172.21.15.111:0/446134003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x25da370 con 0x2464ae0 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f2bfff700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800fe70 tx=0x7f1f18018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9d8e43d15a55cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18029070 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1f18014de0 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f2400e860 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f18005a00 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 10 client.5277 fetch_fsmap learned FSMap version 16 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x25b5130 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2400e860 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 16) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f1f18005b60 con 0x2464ae0 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 10 client.5277 fetch_fsmap finished waiting for FSMap version 16 2023-12-19T07:48:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 10 client.5277 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T07:48:32.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 20 client.5277 populate_metadata read hostname 'smithi111' 2023-12-19T07:48:32.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x269bf40 0x273b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 4 client.5277 mds_command: new command op to 5110 tid=0[{"prefix": "status"}] 2023-12-19T07:48:32.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.607+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 --> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] -- command(tid 0: {"prefix": "status"}) v1 -- 0x25e0410 con 0x269bf40 2023-12-19T07:48:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x269bf40 0x273b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f30e4a700 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x269bf40 0x273b530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e7733ccadf7f5b2 server_cookie=78f7b60939c33f93 in_seq=0 out_seq=0 2023-12-19T07:48:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f0affd700 10 client.5277 ms_handle_connect on v2:172.21.15.143:6834/1413455479 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f0affd700 1 -- 172.21.15.111:0/446134003 <== mds.0 v2:172.21.15.143:6834/1413455479 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x25e0410 con 0x269bf40 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f0affd700 10 client.5277 handle_command_reply: tid=0 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 client.5277 shutdown 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 msgr2=0x269b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800fe70 tx=0x7f1f18018040 comp rx=0 tx=0).stop 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 shutdown_connections 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:6834/1413455479,v1:172.21.15.143:6835/1413455479] conn(0x269bf40 0x273b530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x24af790 0x269ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2464ae0 0x269b4c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 --2- 172.21.15.111:0/446134003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2464710 0x2697da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f0affd700 0 client.5277 ms_handle_reset on v2:172.21.15.143:6834/1413455479 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 >> 172.21.15.111:0/446134003 conn(0x2464eb0 msgr2=0x2699570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 shutdown_connections 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 1 -- 172.21.15.111:0/446134003 wait complete. 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f3533b740 20 client.5277 trim_cache size 0 max 16384 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 5110, 2023-12-19T07:48:32.598 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.319155028, 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 16, 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 40, 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 40, 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 15.176164155 2023-12-19T07:48:32.599 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 msgr2=0x7f1f10062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 0x7f1f10062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0cb4c0 tx=0x7f1f18008040 comp rx=0 tx=0).stop 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 msgr2=0x7f1f2c159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.611+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009290 tx=0x7f1f20017000 comp rx=0 tx=0).stop 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 shutdown_connections 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1f10060550 0x7f1f10062a00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f2c15d050 0x7f1f2c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f2c160e60 0x7f1f2c15ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 --2- 172.21.15.111:0/556555834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f2c0afd20 0x7f1f2c159080 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 >> 172.21.15.111:0/556555834 conn(0x7f1f2c067b60 msgr2=0x7f1f2c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 shutdown_connections 2023-12-19T07:48:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.615+0000 7f1f330ae700 1 -- 172.21.15.111:0/556555834 wait complete. 2023-12-19T07:48:32.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T07:48:32.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.807+0000 7f39ca585700 1 Processor -- start 2023-12-19T07:48:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 -- start start 2023-12-19T07:48:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39c4067c90 0x7f39c4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40c6630 con 0x7f39c414fba0 2023-12-19T07:48:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40c6910 con 0x7f39c4067c90 2023-12-19T07:48:32.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.811+0000 7f39ca585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40c6bf0 con 0x7f39c40c5900 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39c414fba0 0x7f39c4150000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48498/0 (socket says 172.21.15.111:48498) 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c37fe700 1 -- 172.21.15.111:0/349849561 learned_addr learned my addr 172.21.15.111:0/349849561 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 --2- 172.21.15.111:0/349849561 >> v2:172.21.15.143:3300/0 conn(0x7f39c4067c90 0x7f39c4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c8b22700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c2ffd700 1 -- 172.21.15.111:0/349849561 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692950144 0 0) 0x7f39c40c6630 con 0x7f39c414fba0 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c2ffd700 1 -- 172.21.15.111:0/349849561 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac003570 con 0x7f39c414fba0 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 -- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 msgr2=0x7f39c40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 -- 172.21.15.111:0/349849561 >> v1:172.21.15.111:6789/0 conn(0x7f39c414fba0 legacy=0x7f39c4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 -- 172.21.15.111:0/349849561 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40c6ed0 con 0x7f39c4067c90 2023-12-19T07:48:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c3fff700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c4067c90 0x7f39c4068060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39b40098e0 tx=0x7f39b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f84874ceda962d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c2ffd700 1 -- 172.21.15.111:0/349849561 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4011630 con 0x7f39c4067c90 2023-12-19T07:48:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c2ffd700 1 -- 172.21.15.111:0/349849561 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4011790 con 0x7f39c4067c90 2023-12-19T07:48:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39c2ffd700 1 -- 172.21.15.111:0/349849561 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4011b30 con 0x7f39c4067c90 2023-12-19T07:48:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39ca585700 1 -- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c4067c90 msgr2=0x7f39c4068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39ca585700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c4067c90 0x7f39c4068060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39b40098e0 tx=0x7f39b40091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39ca585700 1 -- 172.21.15.111:0/349849561 shutdown_connections 2023-12-19T07:48:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.815+0000 7f39ca585700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.819+0000 7f39ca585700 1 --2- 172.21.15.111:0/349849561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c4067c90 0x7f39c4068060 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.819+0000 7f39ca585700 1 -- 172.21.15.111:0/349849561 >> 172.21.15.111:0/349849561 conn(0x7f39c40cd6d0 msgr2=0x7f39c406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:32.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.819+0000 7f39ca585700 1 -- 172.21.15.111:0/349849561 shutdown_connections 2023-12-19T07:48:32.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.819+0000 7f39ca585700 1 -- 172.21.15.111:0/349849561 wait complete. 2023-12-19T07:48:32.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.819+0000 7f39ca585700 1 Processor -- start 2023-12-19T07:48:32.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 -- start start 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c4152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 0x7f39c41527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40ca1c0 con 0x7f39c4067c90 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40b3870 con 0x7f39c414fba0 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39ca585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41407a0 con 0x7f39c40c5900 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46766/0 (socket says 172.21.15.111:46766) 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 -- 172.21.15.111:0/3575751092 learned_addr learned my addr 172.21.15.111:0/3575751092 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c37fe700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c4152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c8b22700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 0x7f39c41527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 -- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 msgr2=0x7f39c4152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c4152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 -- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 msgr2=0x7f39c41527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 0x7f39c41527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c3fff700 1 -- 172.21.15.111:0/3575751092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4009420 con 0x7f39c4067c90 2023-12-19T07:48:32.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.823+0000 7f39c8b22700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 0x7f39c41527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39c3fff700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009ed0 tx=0x7f39b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4011900 con 0x7f39c4067c90 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4011a80 con 0x7f39c4067c90 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c40c9c00 con 0x7f39c4067c90 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b401a680 con 0x7f39c4067c90 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c40b3500 con 0x7f39c4067c90 2023-12-19T07:48:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.827+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c40b3360 con 0x7f39c4067c90 2023-12-19T07:48:32.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.831+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f39b401aa30 con 0x7f39c4067c90 2023-12-19T07:48:32.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.831+0000 7f39abfff700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 0x7f39b0062ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:32.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.831+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f39b407e540 con 0x7f39c4067c90 2023-12-19T07:48:32.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.831+0000 7f39c37fe700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 0x7f39b0062ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:32.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.835+0000 7f39c37fe700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 0x7f39b0062ab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39c414edd0 tx=0x7f39b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:32.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:32.839+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39c40b3360 con 0x7f39c4067c90 2023-12-19T07:48:33.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.051+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39c40b3360 con 0x7f39c4067c90 2023-12-19T07:48:33.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.051+0000 7f39abfff700 1 -- 172.21.15.111:0/3575751092 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f39c40b3360 con 0x7f39c4067c90 2023-12-19T07:48:33.037 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 msgr2=0x7f39b0062ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 0x7f39b0062ab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39c414edd0 tx=0x7f39b8009040 comp rx=0 tx=0).stop 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 msgr2=0x7f39c40c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009ed0 tx=0x7f39b4017000 comp rx=0 tx=0).stop 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 shutdown_connections 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060600 0x7f39b0062ab0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c414fba0 0x7f39c41527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:33.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c40c5900 0x7f39c4152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 --2- 172.21.15.111:0/3575751092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4067c90 0x7f39c40c91f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 >> 172.21.15.111:0/3575751092 conn(0x7f39c40cd6d0 msgr2=0x7f39c40ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 shutdown_connections 2023-12-19T07:48:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:33.055+0000 7f39ca585700 1 -- 172.21.15.111:0/3575751092 wait complete. 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T07:48:33.058 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T07:48:33.059 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T07:48:33.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.059 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:48:33.077 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42344sec preferred_lft 42344sec 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T07:48:33.078 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.079 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T07:48:33.080 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T07:48:33.080 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T07:48:33.080 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T07:48:33.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.081 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T07:48:33.167 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T07:48:33.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.170 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T07:48:33.234 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T07:48:33.235 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T07:48:33.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.236 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T07:48:33.236 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T07:48:33.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T07:48:33.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T07:48:33.237 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T07:48:33.237 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T07:48:33.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.368 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T07:48:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T07:48:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T07:48:33.386 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T07:48:33.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:48:33.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T07:48:33.467 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T07:48:33.468 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:48:33.468 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:33.527 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T07:48:33.527 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T07:48:33.527 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:33.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:48:33.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:48:33.587 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:48:33.666 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:48:33.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:48:33.743 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:48:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:48:33.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:33.747 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:48:33.806 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T07:48:33.806 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T07:48:33.859 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T07:48:33.895 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T07:48:33.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T07:48:33.970 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T07:48:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T07:48:33.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:33.973 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T07:48:33.993 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T07:48:34.007+0000 7fee6b300380 -1 init, newargv = 0x556c88c61200 newargc=15 2023-12-19T07:48:33.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[43819]: starting ceph client 2023-12-19T07:48:34.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[43819]: starting fuse 2023-12-19T07:48:34.006 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T07:48:34.007 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T07:48:34.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T07:48:34.007 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T07:48:34.007 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T07:48:34.008 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T07:48:34.009 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T07:48:34.172 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.43819.asok 2023-12-19T07:48:34.357 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43819.asok 2023-12-19T07:48:34.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:34.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43819.asok status 2023-12-19T07:48:34.540 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T07:48:34.540 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T07:48:34.540 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T07:48:34.540 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T07:48:34.540 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "43819", 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 5292, 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 5292 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:48:34.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2912542223 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2912542223 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.5292 192.168.0.1:0/2912542223", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2912542223", 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 16, 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 40, 2023-12-19T07:48:34.542 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T07:48:34.543 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T07:48:34.543 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T07:48:34.543 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T07:48:34.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:34.553 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T07:48:34.553 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:34.573 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T07:48:34.573 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:34.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:34.574 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T07:48:34.657 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T07:48:34.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.895+0000 7f339b4ae700 1 Processor -- start 2023-12-19T07:48:34.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.899+0000 7f339b4ae700 1 -- start start 2023-12-19T07:48:34.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f33940b3870 0x7f33940b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339406b0d0 con 0x7f3394158cc0 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33940b4790 con 0x7f339406dc00 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33940b41d0 con 0x7f33940b3870 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339924a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3394158cc0 0x7f339406d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48506/0 (socket says 172.21.15.111:48506) 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339924a700 1 -- 172.21.15.111:0/3927484834 learned_addr learned my addr 172.21.15.111:0/3927484834 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:34.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f3399a4b700 1 --2- 172.21.15.111:0/3927484834 >> v2:172.21.15.111:3300/0 conn(0x7f33940b3870 0x7f33940b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927115450 0 0) 0x7f339406b0d0 con 0x7f3394158cc0 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380003570 con 0x7f3394158cc0 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947608151 0 0) 0x7f33940b4790 con 0x7f339406dc00 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f339406b0d0 con 0x7f339406dc00 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f3399a4b700 1 -- 172.21.15.111:0/3927484834 >> v1:172.21.15.143:6789/0 conn(0x7f339406dc00 legacy=0x7f339406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:34.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f3399a4b700 1 -- 172.21.15.111:0/3927484834 >> v1:172.21.15.111:6789/0 conn(0x7f3394158cc0 legacy=0x7f339406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:34.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f3399a4b700 1 -- 172.21.15.111:0/3927484834 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339406a7e0 con 0x7f33940b3870 2023-12-19T07:48:34.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f3399a4b700 1 --2- 172.21.15.111:0/3927484834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940b3870 0x7f33940b3c90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f338c009e60 tx=0x7f338c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f1d4a41fbac30a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:34.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c00e070 con 0x7f33940b3870 2023-12-19T07:48:34.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c00ba70 con 0x7f33940b3870 2023-12-19T07:48:34.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f338bfff700 1 -- 172.21.15.111:0/3927484834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c00bdf0 con 0x7f33940b3870 2023-12-19T07:48:34.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 -- 172.21.15.111:0/3927484834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940b3870 msgr2=0x7f33940b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:34.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.903+0000 7f339b4ae700 1 --2- 172.21.15.111:0/3927484834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940b3870 0x7f33940b3c90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f338c009e60 tx=0x7f338c009780 comp rx=0 tx=0).stop 2023-12-19T07:48:34.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 -- 172.21.15.111:0/3927484834 shutdown_connections 2023-12-19T07:48:34.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 --2- 172.21.15.111:0/3927484834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33940b3870 0x7f33940b3c90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:34.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 -- 172.21.15.111:0/3927484834 >> 172.21.15.111:0/3927484834 conn(0x7f3394067bb0 msgr2=0x7f33940cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:34.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 -- 172.21.15.111:0/3927484834 shutdown_connections 2023-12-19T07:48:34.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 -- 172.21.15.111:0/3927484834 wait complete. 2023-12-19T07:48:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 Processor -- start 2023-12-19T07:48:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.907+0000 7f339b4ae700 1 -- start start 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b3870 0x7f339416bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3394158cc0 0x7f3394153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394152be0 con 0x7f339406dc00 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394151100 con 0x7f33940b3870 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339416c7a0 con 0x7f3394158cc0 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46786/0 (socket says 172.21.15.111:46786) 2023-12-19T07:48:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 -- 172.21.15.111:0/689323870 learned_addr learned my addr 172.21.15.111:0/689323870 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f3398a49700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b3870 0x7f339416bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f3399a4b700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3394158cc0 0x7f3394153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 -- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3394158cc0 msgr2=0x7f3394153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3394158cc0 0x7f3394153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 -- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b3870 msgr2=0x7f339416bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b3870 0x7f339416bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 -- 172.21.15.111:0/689323870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c009a20 con 0x7f339406dc00 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339924a700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f339000c590 tx=0x7f3390007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390005b30 con 0x7f339406dc00 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3390005cb0 con 0x7f339406dc00 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339416ca80 con 0x7f339406dc00 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390007320 con 0x7f339406dc00 2023-12-19T07:48:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.911+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339416c490 con 0x7f339406dc00 2023-12-19T07:48:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.915+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394071120 con 0x7f339406dc00 2023-12-19T07:48:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.915+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f339001b3b0 con 0x7f339406dc00 2023-12-19T07:48:34.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.919+0000 7f33897fa700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 0x7f3378062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:34.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.919+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f339007bf40 con 0x7f339406dc00 2023-12-19T07:48:34.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.919+0000 7f3398a49700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 0x7f3378062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:34.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.919+0000 7f3398a49700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 0x7f3378062a00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3394154b90 tx=0x7f3384009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:34.923+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f339004c350 con 0x7f339406dc00 2023-12-19T07:48:35.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.115+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3394152d20 con 0x7f3378060550 2023-12-19T07:48:35.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f33897fa700 1 -- 172.21.15.111:0/689323870 <== mgr.4100 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3384003510 con 0x7f3378060550 2023-12-19T07:48:35.102 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T07:48:35.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T07:48:35.102 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T07:48:35.102 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T07:48:35.102 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T07:48:35.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 msgr2=0x7f3378062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 0x7f3378062a00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3394154b90 tx=0x7f3384009040 comp rx=0 tx=0).stop 2023-12-19T07:48:35.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 msgr2=0x7f3394150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f339000c590 tx=0x7f3390007e00 comp rx=0 tx=0).stop 2023-12-19T07:48:35.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 shutdown_connections 2023-12-19T07:48:35.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3378060550 0x7f3378062a00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33940b3870 0x7f339416bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3394158cc0 0x7f3394153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 --2- 172.21.15.111:0/689323870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339406dc00 0x7f3394150bc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 >> 172.21.15.111:0/689323870 conn(0x7f3394067bb0 msgr2=0x7f3394156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:35.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 shutdown_connections 2023-12-19T07:48:35.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.119+0000 7f339b4ae700 1 -- 172.21.15.111:0/689323870 wait complete. 2023-12-19T07:48:35.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T07:48:35.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.303+0000 7f027e8f0700 1 Processor -- start 2023-12-19T07:48:35.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.303+0000 7f027e8f0700 1 -- start start 2023-12-19T07:48:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f027e8f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780c6d20 0x7f02780c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f027e8f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f027806c7c0 0x7f027806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f027e8f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027806a7a0 con 0x7f027815aa30 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f027e8f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02780c7630 con 0x7f02780c6d20 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f027e8f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027806d4b0 con 0x7f027806c7c0 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780c6d20 0x7f02780c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780c6d20 0x7f02780c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46788/0 (socket says 172.21.15.111:46788) 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 -- 172.21.15.111:0/3227084667 learned_addr learned my addr 172.21.15.111:0/3227084667 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:35.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f026ffff700 1 --2- 172.21.15.111:0/3227084667 >> v2:172.21.15.143:3300/0 conn(0x7f027806c7c0 0x7f027806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 -- 172.21.15.111:0/3227084667 >> v2:172.21.15.143:3300/0 conn(0x7f027806c7c0 msgr2=0x7f027806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 --2- 172.21.15.111:0/3227084667 >> v2:172.21.15.143:3300/0 conn(0x7f027806c7c0 0x7f027806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 -- 172.21.15.111:0/3227084667 >> v1:172.21.15.111:6789/0 conn(0x7f027815aa30 legacy=0x7f027815ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 -- 172.21.15.111:0/3227084667 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027806d790 con 0x7f02780c6d20 2023-12-19T07:48:35.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f0277fff700 1 --2- 172.21.15.111:0/3227084667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02780c70f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f02780c86f0 tx=0x7f0268009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b484183bc6987bc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f02777fe700 1 -- 172.21.15.111:0/3227084667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0268009e70 con 0x7f02780c6d20 2023-12-19T07:48:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f02777fe700 1 -- 172.21.15.111:0/3227084667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026800b860 con 0x7f02780c6d20 2023-12-19T07:48:35.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.307+0000 7f02777fe700 1 -- 172.21.15.111:0/3227084667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800bc00 con 0x7f02780c6d20 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- 172.21.15.111:0/3227084667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 msgr2=0x7f02780c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 --2- 172.21.15.111:0/3227084667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02780c70f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f02780c86f0 tx=0x7f0268009250 comp rx=0 tx=0).stop 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- 172.21.15.111:0/3227084667 shutdown_connections 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 --2- 172.21.15.111:0/3227084667 >> v2:172.21.15.143:3300/0 conn(0x7f027806c7c0 0x7f027806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 --2- 172.21.15.111:0/3227084667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02780c70f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- 172.21.15.111:0/3227084667 >> 172.21.15.111:0/3227084667 conn(0x7f0278067bb0 msgr2=0x7f02780cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- 172.21.15.111:0/3227084667 shutdown_connections 2023-12-19T07:48:35.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- 172.21.15.111:0/3227084667 wait complete. 2023-12-19T07:48:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 Processor -- start 2023-12-19T07:48:35.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.311+0000 7f027e8f0700 1 -- start start 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027806c7c0 0x7f0278153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 0x7f02780ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02780aeba0 con 0x7f02780c6d20 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0278157730 con 0x7f027815aa30 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027816a350 con 0x7f027806c7c0 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46798/0 (socket says 172.21.15.111:46798) 2023-12-19T07:48:35.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 -- 172.21.15.111:0/2865727467 learned_addr learned my addr 172.21.15.111:0/2865727467 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f0277fff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027806c7c0 0x7f0278153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027ce8d700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 0x7f02780ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 -- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027806c7c0 msgr2=0x7f0278153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027806c7c0 0x7f0278153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 -- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 msgr2=0x7f02780ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 0x7f02780ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 -- 172.21.15.111:0/2865727467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680094f0 con 0x7f02780c6d20 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027ce8d700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 0x7f02780ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f026ffff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f026000ec80 tx=0x7f02600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026000c9d0 con 0x7f02780c6d20 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026000cb30 con 0x7f02780c6d20 2023-12-19T07:48:35.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027816a070 con 0x7f02780c6d20 2023-12-19T07:48:35.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260013470 con 0x7f02780c6d20 2023-12-19T07:48:35.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.315+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02780ae230 con 0x7f02780c6d20 2023-12-19T07:48:35.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02780aed50 con 0x7f02780c6d20 2023-12-19T07:48:35.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0260013820 con 0x7f02780c6d20 2023-12-19T07:48:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f0274ff9700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 0x7f0258062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:35.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f026007ccc0 con 0x7f02780c6d20 2023-12-19T07:48:35.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f0277fff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 0x7f0258062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:35.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.319+0000 7f0277fff700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 0x7f0258062a00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02780c86f0 tx=0x7f0268017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:35.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.327+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02780aed50 con 0x7f02780c6d20 2023-12-19T07:48:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:35.531+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0278071120 con 0x7f02780c6d20 2023-12-19T07:48:36.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.071+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f026004d540 con 0x7f02780c6d20 2023-12-19T07:48:36.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.071+0000 7f0274ff9700 1 -- 172.21.15.111:0/2865727467 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f026004d170 con 0x7f02780c6d20 2023-12-19T07:48:36.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 msgr2=0x7f0258062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 0x7f0258062a00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02780c86f0 tx=0x7f0268017000 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 msgr2=0x7f02781488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f026000ec80 tx=0x7f02600091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 shutdown_connections 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0258060550 0x7f0258062a00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027815aa30 0x7f02780ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780c6d20 0x7f02781488c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 --2- 172.21.15.111:0/2865727467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027806c7c0 0x7f0278153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 >> 172.21.15.111:0/2865727467 conn(0x7f0278067bb0 msgr2=0x7f02780c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 shutdown_connections 2023-12-19T07:48:36.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.075+0000 7f027e8f0700 1 -- 172.21.15.111:0/2865727467 wait complete. 2023-12-19T07:48:36.079 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T07:48:36.239 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:36.255+0000 7f783375a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:48:36.251 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:36.271+0000 7fd5f09f9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:48:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.275+0000 7f4f70985700 1 Processor -- start 2023-12-19T07:48:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.275+0000 7f4f70985700 1 -- start start 2023-12-19T07:48:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f70985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f70985700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0cbdf0 con 0x7f4f6c0cecd0 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f70985700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0cc0d0 con 0x7f4f6c158e20 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f70985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0cc3b0 con 0x7f4f6c0cb970 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f6ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4f6c0cecd0 0x7f4f6c06b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48542/0 (socket says 172.21.15.111:48542) 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f6a59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4f6c158e20 0x7f4f6c0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:55736/0 (socket says 172.21.15.111:55736) 2023-12-19T07:48:36.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:33698/0 (socket says 172.21.15.111:33698) 2023-12-19T07:48:36.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f6ad9d700 1 -- 172.21.15.111:0/3556349733 learned_addr learned my addr 172.21.15.111:0/3556349733 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:36.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 -- 172.21.15.111:0/3556349733 >> v1:172.21.15.143:6789/0 conn(0x7f4f6c158e20 legacy=0x7f4f6c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 -- 172.21.15.111:0/3556349733 >> v1:172.21.15.111:6789/0 conn(0x7f4f6c0cecd0 legacy=0x7f4f6c06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 -- 172.21.15.111:0/3556349733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0cc690 con 0x7f4f6c0cb970 2023-12-19T07:48:36.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.279+0000 7f4f69d9b700 1 --2- 172.21.15.111:0/3556349733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009910 tx=0x7f4f600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af33ee74b80b4c14 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:36.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f6959a700 1 -- 172.21.15.111:0/3556349733 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000b860 con 0x7f4f6c0cb970 2023-12-19T07:48:36.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f6959a700 1 -- 172.21.15.111:0/3556349733 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6000b9c0 con 0x7f4f6c0cb970 2023-12-19T07:48:36.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f6959a700 1 -- 172.21.15.111:0/3556349733 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000bd60 con 0x7f4f6c0cb970 2023-12-19T07:48:36.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 -- 172.21.15.111:0/3556349733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 msgr2=0x7f4f6c06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 --2- 172.21.15.111:0/3556349733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009910 tx=0x7f4f600091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 -- 172.21.15.111:0/3556349733 shutdown_connections 2023-12-19T07:48:36.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 --2- 172.21.15.111:0/3556349733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c0cb970 0x7f4f6c06acc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 -- 172.21.15.111:0/3556349733 >> 172.21.15.111:0/3556349733 conn(0x7f4f6c067bb0 msgr2=0x7f4f6c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:36.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 -- 172.21.15.111:0/3556349733 shutdown_connections 2023-12-19T07:48:36.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.283+0000 7f4f70985700 1 -- 172.21.15.111:0/3556349733 wait complete. 2023-12-19T07:48:36.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 Processor -- start 2023-12-19T07:48:36.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 -- start start 2023-12-19T07:48:36.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f6c0cecd0 0x7f4f6c1553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c158e20 0x7f4f6c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c14b4a0 con 0x7f4f6c153ec0 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c155930 con 0x7f4f6c0cecd0 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f70985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c1561d0 con 0x7f4f6c158e20 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46810/0 (socket says 172.21.15.111:46810) 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 -- 172.21.15.111:0/2809023426 learned_addr learned my addr 172.21.15.111:0/2809023426 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f69d9b700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c158e20 0x7f4f6c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6a59c700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f6c0cecd0 0x7f4f6c1553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c158e20 msgr2=0x7f4f6c155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c158e20 0x7f4f6c155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f6c0cecd0 msgr2=0x7f4f6c1553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f6c0cecd0 0x7f4f6c1553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 -- 172.21.15.111:0/2809023426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f60009420 con 0x7f4f6c153ec0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f6ad9d700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400cc70 tx=0x7f4f640019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.287+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400a7d0 con 0x7f4f6c153ec0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6400a950 con 0x7f4f6c153ec0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400ad10 con 0x7f4f6c153ec0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c154350 con 0x7f4f6c153ec0 2023-12-19T07:48:36.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c154bf0 con 0x7f4f6c153ec0 2023-12-19T07:48:36.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c154750 con 0x7f4f6c153ec0 2023-12-19T07:48:36.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.291+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f4f64007300 con 0x7f4f6c153ec0 2023-12-19T07:48:36.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.295+0000 7f4f5a7fc700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 0x7f4f4c062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.295+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4f6407cc70 con 0x7f4f6c153ec0 2023-12-19T07:48:36.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.295+0000 7f4f6a59c700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 msgr2=0x7f4f4c062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:36.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.295+0000 7f4f6a59c700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 0x7f4f4c062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:48:36.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.299+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4f6404d100 con 0x7f4f6c153ec0 2023-12-19T07:48:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.487+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f4f6c071120 con 0x7f4f6c153ec0 2023-12-19T07:48:36.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.495+0000 7f4f5a7fc700 1 -- 172.21.15.111:0/2809023426 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4f64050720 con 0x7f4f6c153ec0 2023-12-19T07:48:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f6a59c700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 msgr2=0x7f4f4c062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 msgr2=0x7f4f4c062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-19T07:48:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f6a59c700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 0x7f4f4c062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:48:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 0x7f4f4c062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 msgr2=0x7f4f6c14d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400cc70 tx=0x7f4f640019f0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 shutdown_connections 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4f4c0605a0 0x7f4f4c062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f6c153ec0 0x7f4f6c14d3a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f6c158e20 0x7f4f6c155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 --2- 172.21.15.111:0/2809023426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f6c0cecd0 0x7f4f6c1553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 >> 172.21.15.111:0/2809023426 conn(0x7f4f6c067bb0 msgr2=0x7f4f6c0cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 shutdown_connections 2023-12-19T07:48:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.499+0000 7f4f70985700 1 -- 172.21.15.111:0/2809023426 wait complete. 2023-12-19T07:48:36.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T07:48:36.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.695+0000 7f2bd08ae700 1 Processor -- start 2023-12-19T07:48:36.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.695+0000 7f2bd08ae700 1 -- start start 2023-12-19T07:48:36.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bd08ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2bcc1514a0 0x7f2bcc157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bd08ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0cbee0 con 0x7f2bcc0cae60 2023-12-19T07:48:36.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bd08ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc158530 con 0x7f2bcc06b7a0 2023-12-19T07:48:36.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bd08ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc0cc370 con 0x7f2bcc1514a0 2023-12-19T07:48:36.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bc9d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2bcc0cae60 0x7f2bcc06b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48552/0 (socket says 172.21.15.111:48552) 2023-12-19T07:48:36.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bc9d9b700 1 -- 172.21.15.111:0/1094632727 learned_addr learned my addr 172.21.15.111:0/1094632727 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:36.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bca59c700 1 --2- 172.21.15.111:0/1094632727 >> v2:172.21.15.111:3300/0 conn(0x7f2bcc1514a0 0x7f2bcc157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bca59c700 1 -- 172.21.15.111:0/1094632727 >> v1:172.21.15.143:6789/0 conn(0x7f2bcc06b7a0 legacy=0x7f2bcc158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bca59c700 1 -- 172.21.15.111:0/1094632727 >> v1:172.21.15.111:6789/0 conn(0x7f2bcc0cae60 legacy=0x7f2bcc06b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bca59c700 1 -- 172.21.15.111:0/1094632727 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc0cb2d0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bca59c700 1 --2- 172.21.15.111:0/1094632727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc157ff0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0098e0 tx=0x7f2bbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a85ab18f1329344 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bc959a700 1 -- 172.21.15.111:0/1094632727 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc009e80 con 0x7f2bcc1514a0 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bc959a700 1 -- 172.21.15.111:0/1094632727 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc00b860 con 0x7f2bcc1514a0 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.699+0000 7f2bc959a700 1 -- 172.21.15.111:0/1094632727 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc00bbe0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- 172.21.15.111:0/1094632727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 msgr2=0x7f2bcc157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/1094632727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc157ff0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0098e0 tx=0x7f2bbc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- 172.21.15.111:0/1094632727 shutdown_connections 2023-12-19T07:48:36.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/1094632727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc157ff0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- 172.21.15.111:0/1094632727 >> 172.21.15.111:0/1094632727 conn(0x7f2bcc0cc580 msgr2=0x7f2bcc0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:36.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- 172.21.15.111:0/1094632727 shutdown_connections 2023-12-19T07:48:36.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- 172.21.15.111:0/1094632727 wait complete. 2023-12-19T07:48:36.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 Processor -- start 2023-12-19T07:48:36.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- start start 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bcc06b7a0 0x7f2bcc15e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 0x7f2bcc16a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0b0d30 con 0x7f2bcc1514a0 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc16ac90 con 0x7f2bcc0cae60 2023-12-19T07:48:36.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.703+0000 7f2bd08ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc15e4b0 con 0x7f2bcc06b7a0 2023-12-19T07:48:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46836/0 (socket says 172.21.15.111:46836) 2023-12-19T07:48:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 -- 172.21.15.111:0/3943705372 learned_addr learned my addr 172.21.15.111:0/3943705372 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bca59c700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bcc06b7a0 0x7f2bcc15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bc9d9b700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 0x7f2bcc16a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bcc06b7a0 msgr2=0x7f2bcc15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bcc06b7a0 0x7f2bcc15e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 msgr2=0x7f2bcc16a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 0x7f2bcc16a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 -- 172.21.15.111:0/3943705372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc009420 con 0x7f2bcc1514a0 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bc9d9b700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 0x7f2bcc16a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:36.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bcad9d700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c910 tx=0x7f2bc40016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4012630 con 0x7f2bcc1514a0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc40127b0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc15d900 con 0x7f2bcc1514a0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4012b70 con 0x7f2bcc1514a0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc15e140 con 0x7f2bcc1514a0 2023-12-19T07:48:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.707+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc0c85a0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.711+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f2bc400a3b0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.711+0000 7f2bba7fc700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 0x7f2bac062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.711+0000 7f2bca59c700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 msgr2=0x7f2bac062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.711+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f2bc407cd00 con 0x7f2bcc1514a0 2023-12-19T07:48:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.711+0000 7f2bca59c700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 0x7f2bac062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:48:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.719+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2bcc0c85a0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.790 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:36.807+0000 7fd5f09f9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:48:36.833 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:36.847+0000 7f783375a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:48:36.858 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:36.875+0000 7fd5f09f9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:48:36.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.907+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f2bcc0c85a0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.911+0000 7f2bba7fc700 1 -- 172.21.15.111:0/3943705372 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2bcc0c85a0 con 0x7f2bcc1514a0 2023-12-19T07:48:36.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bca59c700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 msgr2=0x7f2bac062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:36.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bca59c700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 0x7f2bac062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 msgr2=0x7f2bac062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 0x7f2bac062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 msgr2=0x7f2bcc0b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400c910 tx=0x7f2bc40016f0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 shutdown_connections 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2bac0605a0 0x7f2bac062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc0cae60 0x7f2bcc16a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bcc1514a0 0x7f2bcc0b1480 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 --2- 172.21.15.111:0/3943705372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bcc06b7a0 0x7f2bcc15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 >> 172.21.15.111:0/3943705372 conn(0x7f2bcc0cc580 msgr2=0x7f2bcc0ae880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:36.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 shutdown_connections 2023-12-19T07:48:36.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:36.915+0000 7f2bd08ae700 1 -- 172.21.15.111:0/3943705372 wait complete. 2023-12-19T07:48:36.911 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:36.927+0000 7f783375a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:48:36.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:36.917 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T07:48:36.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:36.948 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-12-19T07:48:36.993 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.011+0000 7fd5f09f9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:48:37.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T07:48:37.054 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.071+0000 7f783375a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:48:37.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.235+0000 7faa79e78700 1 Processor -- start 2023-12-19T07:48:37.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 -- start start 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faa740cb500 0x7faa7415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa7415c760 0x7faa740b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b0ac0 con 0x7faa740cb130 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7415cbb0 con 0x7faa740cb500 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.239+0000 7faa79e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7406b210 con 0x7faa7415c760 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faa740cb500 0x7faa7415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faa740cb500 0x7faa7415c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46844/0 (socket says 172.21.15.111:46844) 2023-12-19T07:48:37.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 -- 172.21.15.111:0/4123649632 learned_addr learned my addr 172.21.15.111:0/4123649632 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa73fff700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa7415c760 0x7faa740b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 -- 172.21.15.111:0/4123649632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa7415c760 msgr2=0x7faa740b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa7415c760 0x7faa740b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 -- 172.21.15.111:0/4123649632 >> v1:172.21.15.143:6789/0 conn(0x7faa740cb130 legacy=0x7faa7406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 -- 172.21.15.111:0/4123649632 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740ca810 con 0x7faa740cb500 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa72ffd700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa740cb500 0x7faa7415c220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa680099b0 tx=0x7faa68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7dea410148bd7b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:37.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa727fc700 1 -- 172.21.15.111:0/4123649632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800b860 con 0x7faa740cb500 2023-12-19T07:48:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa727fc700 1 -- 172.21.15.111:0/4123649632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6800b9c0 con 0x7faa740cb500 2023-12-19T07:48:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa727fc700 1 -- 172.21.15.111:0/4123649632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800bd60 con 0x7faa740cb500 2023-12-19T07:48:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 -- 172.21.15.111:0/4123649632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa740cb500 msgr2=0x7faa7415c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa740cb500 0x7faa7415c220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa680099b0 tx=0x7faa68009250 comp rx=0 tx=0).stop 2023-12-19T07:48:37.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 -- 172.21.15.111:0/4123649632 shutdown_connections 2023-12-19T07:48:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa7415c760 0x7faa740b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 --2- 172.21.15.111:0/4123649632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa740cb500 0x7faa7415c220 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 -- 172.21.15.111:0/4123649632 >> 172.21.15.111:0/4123649632 conn(0x7faa74067bb0 msgr2=0x7faa740cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 -- 172.21.15.111:0/4123649632 shutdown_connections 2023-12-19T07:48:37.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.243+0000 7faa79e78700 1 -- 172.21.15.111:0/4123649632 wait complete. 2023-12-19T07:48:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.247+0000 7faa79e78700 1 Processor -- start 2023-12-19T07:48:37.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.247+0000 7faa79e78700 1 -- start start 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.247+0000 7faa79e78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa740cb130 0x7faa74155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.247+0000 7faa79e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa740cb500 0x7faa741691c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74159160 con 0x7faa7415c760 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74158550 con 0x7faa740cb130 2023-12-19T07:48:37.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa74158830 con 0x7faa740cb500 2023-12-19T07:48:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46848/0 (socket says 172.21.15.111:46848) 2023-12-19T07:48:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 -- 172.21.15.111:0/3783753709 learned_addr learned my addr 172.21.15.111:0/3783753709 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa72ffd700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa740cb500 0x7faa741691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:37.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa740cb130 0x7faa74155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa740cb500 msgr2=0x7faa741691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa740cb500 0x7faa741691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa740cb130 msgr2=0x7faa74155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa740cb130 0x7faa74155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680094f0 con 0x7faa7415c760 2023-12-19T07:48:37.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa73fff700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009fa0 tx=0x7faa6c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00ebc0 con 0x7faa7415c760 2023-12-19T07:48:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c00ed20 con 0x7faa7415c760 2023-12-19T07:48:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa74158b10 con 0x7faa7415c760 2023-12-19T07:48:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c014670 con 0x7faa7415c760 2023-12-19T07:48:37.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.251+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7415ac90 con 0x7faa7415c760 2023-12-19T07:48:37.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa741592d0 con 0x7faa7415c760 2023-12-19T07:48:37.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 6) v1 ==== 79978+0+0 (secure 0 0 0) 0x7faa6c0149e0 con 0x7faa7415c760 2023-12-19T07:48:37.240 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.259+0000 7fd5f09f9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:48:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa5b7fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6025+0+0 (secure 0 0 0) 0x7faa6c07cfb0 con 0x7faa7415c760 2023-12-19T07:48:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa737fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:37.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.255+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:48:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.263+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faa6c0acbe0 con 0x7faa7415c760 2023-12-19T07:48:37.318 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.335+0000 7fd5f09f9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:48:37.349 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.363+0000 7f783375a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:48:37.423 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.439+0000 7f783375a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:48:37.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.459+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7faa74071120 con 0x7faa600605a0 2023-12-19T07:48:37.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.459+0000 7faa737fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:37.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.459+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:48:37.528 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.547+0000 7fd5f09f9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:48:37.605 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.623+0000 7fd5f09f9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:48:37.644 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.659+0000 7f783375a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:48:37.691 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:37.707+0000 7fd5f09f9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:48:37.727 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.743+0000 7f783375a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:48:37.814 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:37.831+0000 7f783375a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:48:37.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.863+0000 7faa737fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:37.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:37.863+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T07:48:38.063 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.082+0000 7fd5f09f9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:48:38.213 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:38.227+0000 7f783375a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:48:38.231 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.250+0000 7fd5f09f9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:48:38.381 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:38.395+0000 7f783375a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:48:38.575 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.590+0000 7fd5f09f9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:48:38.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:38.667+0000 7faa737fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:38.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:38.667+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T07:48:38.751 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:38.767+0000 7f783375a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:48:38.788 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.806+0000 7fd5f09f9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:48:38.885 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.902+0000 7fd5f09f9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:48:38.962 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:38.979+0000 7f783375a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:48:38.965 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:38.982+0000 7fd5f09f9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:48:39.049 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.063+0000 7f783375a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:48:39.051 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.070+0000 7fd5f09f9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:48:39.127 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.143+0000 7f783375a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:48:39.206 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.223+0000 7f783375a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:48:39.291 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.306+0000 7fd5f09f9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:48:39.375 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.394+0000 7fd5f09f9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:48:39.424 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.439+0000 7f783375a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:48:39.470 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.486+0000 7fd5f09f9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:48:39.501 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.515+0000 7f783375a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:48:39.588 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.603+0000 7f783375a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:48:39.673 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.690+0000 7fd5f09f9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:48:39.765 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:39.782+0000 7fd5f09f9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:48:39.775 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.791+0000 7f783375a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:48:39.860 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:39.875+0000 7f783375a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:48:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:40.270+0000 7faa737fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:48:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:40.270+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T07:48:40.503 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.522+0000 7fd5f09f9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:48:40.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:40.538+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 7) v1 ==== 79689+0+0 (secure 0 0 0) 0x7faa6c04d830 con 0x7faa7415c760 2023-12-19T07:48:40.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:40.538+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:48:40.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:40.538+0000 7faa5b7fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:40.592 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:48:40.606+0000 7f783375a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:48:40.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b10c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b10c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b10c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b10c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b10c9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b38ce700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b38ce700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b38ce700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b38ce700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:40.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:48:40.750+0000 7fd5b38ce700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:48:41.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.550+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 8) v1 ==== 79816+0+0 (secure 0 0 0) 0x7faa6c0513c0 con 0x7faa7415c760 2023-12-19T07:48:41.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.550+0000 7faa5b7fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.550+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7faa74071120 con 0x7faa600605a0 2023-12-19T07:48:41.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.554+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.558+0000 7faa737fe700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa64001200 tx=0x7faa64007ec0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:41.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.694+0000 7faa5b7fe700 1 -- 172.21.15.111:0/3783753709 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7faa74071120 con 0x7faa600605a0 2023-12-19T07:48:41.678 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir0 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 msgr2=0x7faa60062a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa64001200 tx=0x7faa64007ec0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 msgr2=0x7faa740b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009fa0 tx=0x7faa6c009900 comp rx=0 tx=0).stop 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 shutdown_connections 2023-12-19T07:48:41.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faa600605a0 0x7faa60062a50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa7415c760 0x7faa740b37f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa740cb500 0x7faa741691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 --2- 172.21.15.111:0/3783753709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa740cb130 0x7faa74155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 >> 172.21.15.111:0/3783753709 conn(0x7faa74067bb0 msgr2=0x7faa740cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 shutdown_connections 2023-12-19T07:48:41.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.698+0000 7faa79e78700 1 -- 172.21.15.111:0/3783753709 wait complete. 2023-12-19T07:48:41.705 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-12-19T07:48:41.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:41.705 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-12-19T07:48:41.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T07:48:41.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.962+0000 7f73d6e25700 1 Processor -- start 2023-12-19T07:48:41.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.962+0000 7f73d6e25700 1 -- start start 2023-12-19T07:48:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d6e25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d00c86e0 0x7f73d006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d6e25700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f73d00c9ac0 0x7f73d0162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d6e25700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d006d570 con 0x7f73d006df40 2023-12-19T07:48:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d6e25700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d006c960 con 0x7f73d00c9ac0 2023-12-19T07:48:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d6e25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d006cc40 con 0x7f73d00c86e0 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73cffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f73d006df40 0x7f73d00672f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:45520/0 (socket says 172.21.15.111:45520) 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73cffff700 1 -- 172.21.15.111:0/3986440418 learned_addr learned my addr 172.21.15.111:0/3986440418 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 --2- 172.21.15.111:0/3986440418 >> v2:172.21.15.111:3300/0 conn(0x7f73d00c9ac0 0x7f73d0162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d4bc1700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d00c86e0 0x7f73d006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73cf7fe700 1 -- 172.21.15.111:0/3986440418 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665197251 0 0) 0x7f73d006d570 con 0x7f73d006df40 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73cf7fe700 1 -- 172.21.15.111:0/3986440418 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c0003570 con 0x7f73d006df40 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 -- 172.21.15.111:0/3986440418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d00c86e0 msgr2=0x7f73d006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d00c86e0 0x7f73d006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 -- 172.21.15.111:0/3986440418 >> v1:172.21.15.143:6789/0 conn(0x7f73d006df40 legacy=0x7f73d00672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 -- 172.21.15.111:0/3986440418 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d006cf20 con 0x7f73d00c9ac0 2023-12-19T07:48:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.966+0000 7f73d53c2700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d0162ae0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009e60 tx=0x7f73c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbfdcac5b63782d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73cf7fe700 1 -- 172.21.15.111:0/3986440418 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800e070 con 0x7f73d00c9ac0 2023-12-19T07:48:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73cf7fe700 1 -- 172.21.15.111:0/3986440418 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800bab0 con 0x7f73d00c9ac0 2023-12-19T07:48:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73cf7fe700 1 -- 172.21.15.111:0/3986440418 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800be70 con 0x7f73d00c9ac0 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 -- 172.21.15.111:0/3986440418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 msgr2=0x7f73d0162ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d0162ae0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f73c8009e60 tx=0x7f73c8009780 comp rx=0 tx=0).stop 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 -- 172.21.15.111:0/3986440418 shutdown_connections 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d0162ae0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 --2- 172.21.15.111:0/3986440418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d00c86e0 0x7f73d006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 -- 172.21.15.111:0/3986440418 >> 172.21.15.111:0/3986440418 conn(0x7f73d00cdce0 msgr2=0x7f73d00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 -- 172.21.15.111:0/3986440418 shutdown_connections 2023-12-19T07:48:41.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.970+0000 7f73d6e25700 1 -- 172.21.15.111:0/3986440418 wait complete. 2023-12-19T07:48:41.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 Processor -- start 2023-12-19T07:48:41.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 -- start start 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 0x7f73d015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 0x7f73d0156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0159c80 con 0x7f73d00c9ac0 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d016daa0 con 0x7f73d00c86e0 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d6e25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d015a7e0 con 0x7f73d006df40 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d53c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d53c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47500/0 (socket says 172.21.15.111:47500) 2023-12-19T07:48:41.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d53c2700 1 -- 172.21.15.111:0/1880591004 learned_addr learned my addr 172.21.15.111:0/1880591004 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:41.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73d4bc1700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 0x7f73d015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.974+0000 7f73cffff700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 0x7f73d0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 -- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 msgr2=0x7f73d015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 0x7f73d015ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 -- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 msgr2=0x7f73d0156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 0x7f73d0156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 -- 172.21.15.111:0/1880591004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8009a20 con 0x7f73d00c9ac0 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d4bc1700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 0x7f73d015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73cffff700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 0x7f73d0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d53c2700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73d006baf0 tx=0x7f73c8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8027470 con 0x7f73d00c9ac0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c80275f0 con 0x7f73d00c9ac0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d015b1b0 con 0x7f73d00c9ac0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80279b0 con 0x7f73d00c9ac0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d015a190 con 0x7f73d00c9ac0 2023-12-19T07:48:41.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.978+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d015a950 con 0x7f73d00c9ac0 2023-12-19T07:48:41.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.982+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f73c8024070 con 0x7f73d00c9ac0 2023-12-19T07:48:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.982+0000 7f73ccff9700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 0x7f73b40628d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.982+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f73c807f1a0 con 0x7f73d00c9ac0 2023-12-19T07:48:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.982+0000 7f73d4bc1700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 0x7f73b40628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.986+0000 7f73d4bc1700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 0x7f73b40628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009d80 tx=0x7f73c4008040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:41.990+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f73c804f800 con 0x7f73d00c9ac0 2023-12-19T07:48:42.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.178+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f73d0071120 con 0x7f73b4060420 2023-12-19T07:48:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.294+0000 7f73ccff9700 1 -- 172.21.15.111:0/1880591004 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f73d0071120 con 0x7f73b4060420 2023-12-19T07:48:42.281 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir1 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 msgr2=0x7f73b40628d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 0x7f73b40628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009d80 tx=0x7f73c4008040 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 msgr2=0x7f73d015b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73d006baf0 tx=0x7f73c8019000 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 shutdown_connections 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f73b4060420 0x7f73b40628d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73d00c86e0 0x7f73d0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d00c9ac0 0x7f73d015b620 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 --2- 172.21.15.111:0/1880591004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73d006df40 0x7f73d015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 >> 172.21.15.111:0/1880591004 conn(0x7f73d00cdce0 msgr2=0x7f73d0067bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:42.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.298+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 shutdown_connections 2023-12-19T07:48:42.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.302+0000 7f73d6e25700 1 -- 172.21.15.111:0/1880591004 wait complete. 2023-12-19T07:48:42.308 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-12-19T07:48:42.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:42.309 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-12-19T07:48:42.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T07:48:42.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.562+0000 7f56428f0700 1 Processor -- start 2023-12-19T07:48:42.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.562+0000 7f56428f0700 1 -- start start 2023-12-19T07:48:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f56428f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 0x7f563c15c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f56428f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0b0ac0 con 0x7f563c0cb130 2023-12-19T07:48:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f56428f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c06b210 con 0x7f563c15c760 2023-12-19T07:48:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f56428f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15cbe0 con 0x7f563c0cb500 2023-12-19T07:48:42.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f563bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f563c0cb130 0x7f563c06a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48184/0 (socket says 172.21.15.111:48184) 2023-12-19T07:48:42.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f563bfff700 1 -- 172.21.15.111:0/3586119087 learned_addr learned my addr 172.21.15.111:0/3586119087 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:42.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f5633fff700 1 --2- 172.21.15.111:0/3586119087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 0x7f563c15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:42.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886456196 0 0) 0x7f563c0b0ac0 con 0x7f563c0cb130 2023-12-19T07:48:42.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5628003570 con 0x7f563c0cb130 2023-12-19T07:48:42.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f5633fff700 1 -- 172.21.15.111:0/3586119087 >> v1:172.21.15.143:6789/0 conn(0x7f563c15c760 legacy=0x7f563c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086571370 0 0) 0x7f563c06b210 con 0x7f563c15c760 2023-12-19T07:48:42.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f5633fff700 1 -- 172.21.15.111:0/3586119087 >> v1:172.21.15.111:6789/0 conn(0x7f563c0cb130 legacy=0x7f563c06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.566+0000 7f5633fff700 1 -- 172.21.15.111:0/3586119087 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0ca810 con 0x7f563c0cb500 2023-12-19T07:48:42.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f5633fff700 1 --2- 172.21.15.111:0/3586119087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 0x7f563c15c220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5624009910 tx=0x7f56240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa52f5132f664c64 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:42.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c001470 con 0x7f563c0cb500 2023-12-19T07:48:42.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5624011420 con 0x7f563c0cb500 2023-12-19T07:48:42.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f563b7fe700 1 -- 172.21.15.111:0/3586119087 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240117c0 con 0x7f563c0cb500 2023-12-19T07:48:42.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 -- 172.21.15.111:0/3586119087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 msgr2=0x7f563c15c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 --2- 172.21.15.111:0/3586119087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 0x7f563c15c220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5624009910 tx=0x7f56240091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 -- 172.21.15.111:0/3586119087 shutdown_connections 2023-12-19T07:48:42.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 --2- 172.21.15.111:0/3586119087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c0cb500 0x7f563c15c220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 -- 172.21.15.111:0/3586119087 >> 172.21.15.111:0/3586119087 conn(0x7f563c067bb0 msgr2=0x7f563c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:42.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 -- 172.21.15.111:0/3586119087 shutdown_connections 2023-12-19T07:48:42.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.570+0000 7f56428f0700 1 -- 172.21.15.111:0/3586119087 wait complete. 2023-12-19T07:48:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 Processor -- start 2023-12-19T07:48:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 -- start start 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f563c0cb130 0x7f563c14dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c15c760 0x7f563c14fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c1508a0 con 0x7f563c0cb500 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0b4500 con 0x7f563c0cb130 2023-12-19T07:48:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f56428f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0b47e0 con 0x7f563c15c760 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47502/0 (socket says 172.21.15.111:47502) 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 -- 172.21.15.111:0/2399922114 learned_addr learned my addr 172.21.15.111:0/2399922114 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5640e8d700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c15c760 0x7f563c14fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f563bfff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f563c0cb130 0x7f563c14dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 -- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c15c760 msgr2=0x7f563c14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c15c760 0x7f563c14fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 -- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f563c0cb130 msgr2=0x7f563c14dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f563c0cb130 0x7f563c14dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.574+0000 7f5633fff700 1 -- 172.21.15.111:0/2399922114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624009420 con 0x7f563c0cb500 2023-12-19T07:48:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f5633fff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f563c06f300 tx=0x7f5624017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5624010ac0 con 0x7f563c0cb500 2023-12-19T07:48:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5624010c40 con 0x7f563c0cb500 2023-12-19T07:48:42.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562401d430 con 0x7f563c0cb500 2023-12-19T07:48:42.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c0b4ac0 con 0x7f563c0cb500 2023-12-19T07:48:42.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c14e4e0 con 0x7f563c0cb500 2023-12-19T07:48:42.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.578+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c071120 con 0x7f563c0cb500 2023-12-19T07:48:42.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.582+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f562401d7a0 con 0x7f563c0cb500 2023-12-19T07:48:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.582+0000 7f5638ff9700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 0x7f561c0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.582+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f562407df30 con 0x7f563c0cb500 2023-12-19T07:48:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.582+0000 7f563bfff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 0x7f561c0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.586+0000 7f563bfff700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 0x7f561c0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f562c0015a0 tx=0x7f562c000bc0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:42.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.590+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f562404e4c0 con 0x7f563c0cb500 2023-12-19T07:48:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.606+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f562404e890 con 0x7f563c0cb500 2023-12-19T07:48:42.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.786+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f563c0cbe90 con 0x7f561c0603f0 2023-12-19T07:48:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.906+0000 7f5638ff9700 1 -- 172.21.15.111:0/2399922114 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f563c0cbe90 con 0x7f561c0603f0 2023-12-19T07:48:42.893 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir2 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 msgr2=0x7f561c0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 0x7f561c0628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f562c0015a0 tx=0x7f562c000bc0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 msgr2=0x7f563c152850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f563c06f300 tx=0x7f5624017000 comp rx=0 tx=0).stop 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 shutdown_connections 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f561c0603f0 0x7f561c0628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563c15c760 0x7f563c14fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f563c0cb500 0x7f563c152850 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 --2- 172.21.15.111:0/2399922114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f563c0cb130 0x7f563c14dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 >> 172.21.15.111:0/2399922114 conn(0x7f563c067bb0 msgr2=0x7f563c14e930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 shutdown_connections 2023-12-19T07:48:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:42.914+0000 7f56428f0700 1 -- 172.21.15.111:0/2399922114 wait complete. 2023-12-19T07:48:42.924 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-12-19T07:48:42.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:42.925 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-12-19T07:48:42.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T07:48:43.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.190+0000 7f735759e700 1 Processor -- start 2023-12-19T07:48:43.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.190+0000 7f735759e700 1 -- start start 2023-12-19T07:48:43.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f735759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f735759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73580cac50 0x7f73580af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f735759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580cb3f0 con 0x7f73580aedc0 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f735759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73580cc140 con 0x7f73580cac50 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f735759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73580af5e0 con 0x7f73580af190 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f7355d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f7355d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52078/0 (socket says 172.21.15.111:52078) 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.194+0000 7f7355d9b700 1 -- 172.21.15.111:0/1982125576 learned_addr learned my addr 172.21.15.111:0/1982125576 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:43.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7356d9d700 1 --2- 172.21.15.111:0/1982125576 >> v2:172.21.15.143:3300/0 conn(0x7f73580cac50 0x7f73580af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7355d9b700 1 -- 172.21.15.111:0/1982125576 >> v2:172.21.15.143:3300/0 conn(0x7f73580cac50 msgr2=0x7f73580af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:43.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7355d9b700 1 --2- 172.21.15.111:0/1982125576 >> v2:172.21.15.143:3300/0 conn(0x7f73580cac50 0x7f73580af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7355d9b700 1 -- 172.21.15.111:0/1982125576 >> v1:172.21.15.143:6789/0 conn(0x7f73580aedc0 legacy=0x7f735806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7355d9b700 1 -- 172.21.15.111:0/1982125576 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735806d220 con 0x7f73580af190 2023-12-19T07:48:43.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f7355d9b700 1 --2- 172.21.15.111:0/1982125576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f734c0099b0 tx=0x7f734c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef2c93acf2b234b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735559a700 1 -- 172.21.15.111:0/1982125576 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c009ea0 con 0x7f73580af190 2023-12-19T07:48:43.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735559a700 1 -- 172.21.15.111:0/1982125576 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c00b860 con 0x7f73580af190 2023-12-19T07:48:43.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735559a700 1 -- 172.21.15.111:0/1982125576 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c00bc00 con 0x7f73580af190 2023-12-19T07:48:43.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 -- 172.21.15.111:0/1982125576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 msgr2=0x7f735806af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 --2- 172.21.15.111:0/1982125576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f734c0099b0 tx=0x7f734c009250 comp rx=0 tx=0).stop 2023-12-19T07:48:43.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 -- 172.21.15.111:0/1982125576 shutdown_connections 2023-12-19T07:48:43.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 --2- 172.21.15.111:0/1982125576 >> v2:172.21.15.143:3300/0 conn(0x7f73580cac50 0x7f73580af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 --2- 172.21.15.111:0/1982125576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580af190 0x7f735806af50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.198+0000 7f735759e700 1 -- 172.21.15.111:0/1982125576 >> 172.21.15.111:0/1982125576 conn(0x7f73580cc760 msgr2=0x7f73580ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:43.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- 172.21.15.111:0/1982125576 shutdown_connections 2023-12-19T07:48:43.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- 172.21.15.111:0/1982125576 wait complete. 2023-12-19T07:48:43.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 Processor -- start 2023-12-19T07:48:43.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- start start 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 0x7f73581692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580cac50 0x7f735815a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735816a2b0 con 0x7f73580aedc0 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73580c4650 con 0x7f73580af190 2023-12-19T07:48:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7358169800 con 0x7f73580cac50 2023-12-19T07:48:43.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47516/0 (socket says 172.21.15.111:47516) 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f735659c700 1 -- 172.21.15.111:0/3717772994 learned_addr learned my addr 172.21.15.111:0/3717772994 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f7356d9d700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580cac50 0x7f735815a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.202+0000 7f7355d9b700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 0x7f73581692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 -- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580cac50 msgr2=0x7f735815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580cac50 0x7f735815a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 -- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 msgr2=0x7f73581692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 0x7f73581692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 -- 172.21.15.111:0/3717772994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0094f0 con 0x7f73580aedc0 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f7355d9b700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 0x7f73581692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735659c700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f734800c590 tx=0x7f734800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348011590 con 0x7f73580aedc0 2023-12-19T07:48:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7348011710 con 0x7f73580aedc0 2023-12-19T07:48:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348011ad0 con 0x7f73580aedc0 2023-12-19T07:48:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73580c5ce0 con 0x7f73580aedc0 2023-12-19T07:48:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358169fd0 con 0x7f73580aedc0 2023-12-19T07:48:43.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.206+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735804be40 con 0x7f73580aedc0 2023-12-19T07:48:43.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.210+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f734801b3b0 con 0x7f73580aedc0 2023-12-19T07:48:43.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.210+0000 7f73467fc700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 0x7f7338062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.210+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f734807c330 con 0x7f73580aedc0 2023-12-19T07:48:43.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.210+0000 7f7355d9b700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 0x7f7338062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.214+0000 7f7355d9b700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 0x7f7338062c30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f734c0099b0 tx=0x7f734c017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.218+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f734804c3c0 con 0x7f73580aedc0 2023-12-19T07:48:43.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.406+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f735806c8f0 con 0x7f7338060780 2023-12-19T07:48:43.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.526+0000 7f73467fc700 1 -- 172.21.15.111:0/3717772994 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f735806c8f0 con 0x7f7338060780 2023-12-19T07:48:43.514 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir3 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 msgr2=0x7f7338062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 0x7f7338062c30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f734c0099b0 tx=0x7f734c017000 comp rx=0 tx=0).stop 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 msgr2=0x7f73580c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f734800c590 tx=0x7f734800c8a0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 shutdown_connections 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7338060780 0x7f7338062c30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73580af190 0x7f73581692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73580cac50 0x7f735815a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 --2- 172.21.15.111:0/3717772994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580aedc0 0x7f73580c4110 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 >> 172.21.15.111:0/3717772994 conn(0x7f73580cc760 msgr2=0x7f735815c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:43.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 shutdown_connections 2023-12-19T07:48:43.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.530+0000 7f735759e700 1 -- 172.21.15.111:0/3717772994 wait complete. 2023-12-19T07:48:43.540 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-12-19T07:48:43.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:43.540 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-12-19T07:48:43.570 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T07:48:43.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.802+0000 7fc70d46e700 1 Processor -- start 2023-12-19T07:48:43.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.802+0000 7fc70d46e700 1 -- start start 2023-12-19T07:48:43.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc70d46e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7080cb500 0x7fc70815c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc70d46e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc70815c760 0x7fc7080b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc70d46e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080b0ac0 con 0x7fc7080cb130 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc70d46e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70815cbe0 con 0x7fc70815c760 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc70d46e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70806b210 con 0x7fc7080cb500 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc70815c760 0x7fc7080b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc70815c760 0x7fc7080b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47528/0 (socket says 172.21.15.111:47528) 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 -- 172.21.15.111:0/2904033801 learned_addr learned my addr 172.21.15.111:0/2904033801 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7067fc700 1 --2- 172.21.15.111:0/2904033801 >> v2:172.21.15.143:3300/0 conn(0x7fc7080cb500 0x7fc70815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 -- 172.21.15.111:0/2904033801 >> v2:172.21.15.143:3300/0 conn(0x7fc7080cb500 msgr2=0x7fc70815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 --2- 172.21.15.111:0/2904033801 >> v2:172.21.15.143:3300/0 conn(0x7fc7080cb500 0x7fc70815c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 -- 172.21.15.111:0/2904033801 >> v1:172.21.15.143:6789/0 conn(0x7fc7080cb130 legacy=0x7fc70806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 -- 172.21.15.111:0/2904033801 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7080ca810 con 0x7fc70815c760 2023-12-19T07:48:43.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc7077fe700 1 --2- 172.21.15.111:0/2904033801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc70815c760 0x7fc7080b0f50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc7080cc930 tx=0x7fc700009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6713d708f2f0bf0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc705ffb700 1 -- 172.21.15.111:0/2904033801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70000e070 con 0x7fc70815c760 2023-12-19T07:48:43.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc705ffb700 1 -- 172.21.15.111:0/2904033801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70000ba20 con 0x7fc70815c760 2023-12-19T07:48:43.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.806+0000 7fc705ffb700 1 -- 172.21.15.111:0/2904033801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7000092e0 con 0x7fc70815c760 2023-12-19T07:48:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- 172.21.15.111:0/2904033801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc70815c760 msgr2=0x7fc7080b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 --2- 172.21.15.111:0/2904033801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc70815c760 0x7fc7080b0f50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc7080cc930 tx=0x7fc700009840 comp rx=0 tx=0).stop 2023-12-19T07:48:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- 172.21.15.111:0/2904033801 shutdown_connections 2023-12-19T07:48:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 --2- 172.21.15.111:0/2904033801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc70815c760 0x7fc7080b0f50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 --2- 172.21.15.111:0/2904033801 >> v2:172.21.15.143:3300/0 conn(0x7fc7080cb500 0x7fc70815c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- 172.21.15.111:0/2904033801 >> 172.21.15.111:0/2904033801 conn(0x7fc708067bb0 msgr2=0x7fc7080cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- 172.21.15.111:0/2904033801 shutdown_connections 2023-12-19T07:48:43.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- 172.21.15.111:0/2904033801 wait complete. 2023-12-19T07:48:43.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 Processor -- start 2023-12-19T07:48:43.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.810+0000 7fc70d46e700 1 -- start start 2023-12-19T07:48:43.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 0x7fc70814de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc70815c760 0x7fc708151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70814e3c0 con 0x7fc7080cb500 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70815b9e0 con 0x7fc70815c760 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7080b41c0 con 0x7fc7080cb130 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47532/0 (socket says 172.21.15.111:47532) 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 -- 172.21.15.111:0/1395631541 learned_addr learned my addr 172.21.15.111:0/1395631541 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7077fe700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc70815c760 0x7fc708151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc706ffd700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 0x7fc70814de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 -- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 msgr2=0x7fc70814de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 0x7fc70814de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 -- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc70815c760 msgr2=0x7fc708151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc70815c760 0x7fc708151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:43.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 -- 172.21.15.111:0/1395631541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700009ae0 con 0x7fc7080cb500 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc706ffd700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 0x7fc70814de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc7067fc700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00ec80 tx=0x7fc6fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00c850 con 0x7fc7080cb500 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00c9b0 con 0x7fc7080cb500 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00cd50 con 0x7fc7080cb500 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.814+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70815b420 con 0x7fc7080cb500 2023-12-19T07:48:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.818+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7080b3e50 con 0x7fc7080cb500 2023-12-19T07:48:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.818+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7080b4370 con 0x7fc7080cb500 2023-12-19T07:48:43.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.818+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fc6fc013680 con 0x7fc7080cb500 2023-12-19T07:48:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.818+0000 7fc6eeffd700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 0x7fc6f4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.818+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fc6fc07d090 con 0x7fc7080cb500 2023-12-19T07:48:43.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.822+0000 7fc706ffd700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 0x7fc6f4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:43.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.822+0000 7fc706ffd700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 0x7fc6f4062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8007fd0 tx=0x7fc6f8009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:43.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:43.826+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc6fc04d120 con 0x7fc7080cb500 2023-12-19T07:48:43.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.010+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fc708071120 con 0x7fc6f4060750 2023-12-19T07:48:44.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.130+0000 7fc6eeffd700 1 -- 172.21.15.111:0/1395631541 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc708071120 con 0x7fc6f4060750 2023-12-19T07:48:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir4 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 msgr2=0x7fc6f4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 0x7fc6f4062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8007fd0 tx=0x7fc6f8009040 comp rx=0 tx=0).stop 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 msgr2=0x7fc70815a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00ec80 tx=0x7fc6fc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 shutdown_connections 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc6f4060750 0x7fc6f4062c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc70815c760 0x7fc708151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7080cb500 0x7fc70815a910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 --2- 172.21.15.111:0/1395631541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7080cb130 0x7fc70814de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 >> 172.21.15.111:0/1395631541 conn(0x7fc708067bb0 msgr2=0x7fc7080cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 shutdown_connections 2023-12-19T07:48:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.134+0000 7fc70d46e700 1 -- 172.21.15.111:0/1395631541 wait complete. 2023-12-19T07:48:44.146 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-12-19T07:48:44.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:44.146 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-12-19T07:48:44.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T07:48:44.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.406+0000 7f739759e700 1 Processor -- start 2023-12-19T07:48:44.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.406+0000 7f739759e700 1 -- start start 2023-12-19T07:48:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f739759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 0x7f7398068050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f739759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f739759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980ad6b0 con 0x7f73980c0d30 2023-12-19T07:48:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f739759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7398068730 con 0x7f73980c1100 2023-12-19T07:48:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f739759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7398068a10 con 0x7f73980aeb50 2023-12-19T07:48:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52108/0 (socket says 172.21.15.111:52108) 2023-12-19T07:48:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 -- 172.21.15.111:0/827510819 learned_addr learned my addr 172.21.15.111:0/827510819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7395d9b700 1 --2- 172.21.15.111:0/827510819 >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 0x7f7398068050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 -- 172.21.15.111:0/827510819 >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 msgr2=0x7f7398068050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 --2- 172.21.15.111:0/827510819 >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 0x7f7398068050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 -- 172.21.15.111:0/827510819 >> v1:172.21.15.143:6789/0 conn(0x7f73980c0d30 legacy=0x7f73980c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 -- 172.21.15.111:0/827510819 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7398068cf0 con 0x7f73980aeb50 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7395d9b700 1 --2- 172.21.15.111:0/827510819 >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 0x7f7398068050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.410+0000 7f7396d9d700 1 --2- 172.21.15.111:0/827510819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7390009f20 tx=0x7f7390009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a242432acfa0337 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739559a700 1 -- 172.21.15.111:0/827510819 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739000e070 con 0x7f73980aeb50 2023-12-19T07:48:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739559a700 1 -- 172.21.15.111:0/827510819 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739000ba80 con 0x7f73980aeb50 2023-12-19T07:48:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739559a700 1 -- 172.21.15.111:0/827510819 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73900092e0 con 0x7f73980aeb50 2023-12-19T07:48:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 -- 172.21.15.111:0/827510819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 msgr2=0x7f73980ac5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 --2- 172.21.15.111:0/827510819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7390009f20 tx=0x7f7390009840 comp rx=0 tx=0).stop 2023-12-19T07:48:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 -- 172.21.15.111:0/827510819 shutdown_connections 2023-12-19T07:48:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 --2- 172.21.15.111:0/827510819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980ac5b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 --2- 172.21.15.111:0/827510819 >> v2:172.21.15.143:3300/0 conn(0x7f73980c1100 0x7f7398068050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 -- 172.21.15.111:0/827510819 >> 172.21.15.111:0/827510819 conn(0x7f73980ce5e0 msgr2=0x7f73980ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:44.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 -- 172.21.15.111:0/827510819 shutdown_connections 2023-12-19T07:48:44.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.414+0000 7f739759e700 1 -- 172.21.15.111:0/827510819 wait complete. 2023-12-19T07:48:44.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 Processor -- start 2023-12-19T07:48:44.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 -- start start 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 0x7f739815de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980b2af0 con 0x7f73980c0d30 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980b1ee0 con 0x7f73980c1100 2023-12-19T07:48:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f739759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980b21c0 con 0x7f73980aeb50 2023-12-19T07:48:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.418+0000 7f7395d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47542/0 (socket says 172.21.15.111:47542) 2023-12-19T07:48:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 -- 172.21.15.111:0/229505845 learned_addr learned my addr 172.21.15.111:0/229505845 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f739659c700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7396d9d700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 0x7f739815de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 -- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 msgr2=0x7f73980b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 -- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 msgr2=0x7f739815de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 0x7f739815de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 -- 172.21.15.111:0/229505845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390009ae0 con 0x7f73980c0d30 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7396d9d700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 0x7f739815de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f739659c700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f7395d9b700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e8d0 tx=0x7f738c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c00ee70 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c00c5e0 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c00c980 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73980b24a0 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739815e390 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.422+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73980b2ca0 con 0x7f73980c0d30 2023-12-19T07:48:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.426+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f738c013410 con 0x7f73980c0d30 2023-12-19T07:48:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.426+0000 7f73867fc700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 0x7f7378062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.426+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f738c07ccd0 con 0x7f73980c0d30 2023-12-19T07:48:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.426+0000 7f739659c700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 0x7f7378062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.426+0000 7f739659c700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 0x7f7378062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7388007fd0 tx=0x7f7388009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:44.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.434+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f73980b2ca0 con 0x7f73980c0d30 2023-12-19T07:48:44.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.622+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f739804be40 con 0x7f7378060750 2023-12-19T07:48:44.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.746+0000 7f73867fc700 1 -- 172.21.15.111:0/229505845 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f739804be40 con 0x7f7378060750 2023-12-19T07:48:44.730 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir5 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 msgr2=0x7f7378062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 0x7f7378062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7388007fd0 tx=0x7f7388009040 comp rx=0 tx=0).stop 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 msgr2=0x7f739815cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e8d0 tx=0x7f738c008040 comp rx=0 tx=0).stop 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 shutdown_connections 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7378060750 0x7f7378062c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73980c1100 0x7f739815de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73980c0d30 0x7f739815cd60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 --2- 172.21.15.111:0/229505845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73980aeb50 0x7f73980b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:44.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 >> 172.21.15.111:0/229505845 conn(0x7f73980ce5e0 msgr2=0x7f7398069980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:44.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 shutdown_connections 2023-12-19T07:48:44.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:44.750+0000 7f739759e700 1 -- 172.21.15.111:0/229505845 wait complete. 2023-12-19T07:48:44.759 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-12-19T07:48:44.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:44.759 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-12-19T07:48:44.786 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T07:48:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.014+0000 7f9ffb22e700 1 Processor -- start 2023-12-19T07:48:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 -- start start 2023-12-19T07:48:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 0x7f9ff415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ff415c760 0x7f9ff40b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40b0ac0 con 0x7f9ff40cb130 2023-12-19T07:48:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff415cbe0 con 0x7f9ff415c760 2023-12-19T07:48:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.018+0000 7f9ffb22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff406b210 con 0x7f9ff40cb500 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ff415c760 0x7f9ff40b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ff415c760 0x7f9ff40b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45826/0 (socket says 172.21.15.111:45826) 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 -- 172.21.15.111:0/1501930929 learned_addr learned my addr 172.21.15.111:0/1501930929 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff3fff700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 0x7f9ff415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff8fca700 1 --1- 172.21.15.111:0/1501930929 >> v1:172.21.15.143:6789/0 conn(0x7f9ff40cb130 0x7f9ff406a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:45564/0 (socket says 172.21.15.111:45564) 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 -- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 msgr2=0x7f9ff415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 0x7f9ff415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 -- 172.21.15.111:0/1501930929 >> v1:172.21.15.143:6789/0 conn(0x7f9ff40cb130 legacy=0x7f9ff406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 -- 172.21.15.111:0/1501930929 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff40ca810 con 0x7f9ff415c760 2023-12-19T07:48:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff3fff700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 0x7f9ff415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff97cb700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff415c760 0x7f9ff40b0f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009f20 tx=0x7f9fec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=672653ddee008a39 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff37fe700 1 -- 172.21.15.111:0/1501930929 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00e070 con 0x7f9ff415c760 2023-12-19T07:48:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff37fe700 1 -- 172.21.15.111:0/1501930929 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec00b9d0 con 0x7f9ff415c760 2023-12-19T07:48:45.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.022+0000 7f9ff37fe700 1 -- 172.21.15.111:0/1501930929 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0092e0 con 0x7f9ff415c760 2023-12-19T07:48:45.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff415c760 msgr2=0x7f9ff40b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff415c760 0x7f9ff40b0f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009f20 tx=0x7f9fec009840 comp rx=0 tx=0).stop 2023-12-19T07:48:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1501930929 shutdown_connections 2023-12-19T07:48:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff415c760 0x7f9ff40b0f50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1501930929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff40cb500 0x7f9ff415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1501930929 >> 172.21.15.111:0/1501930929 conn(0x7f9ff4067bb0 msgr2=0x7f9ff40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:45.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1501930929 shutdown_connections 2023-12-19T07:48:45.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1501930929 wait complete. 2023-12-19T07:48:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.026+0000 7f9ffb22e700 1 Processor -- start 2023-12-19T07:48:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 -- start start 2023-12-19T07:48:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 0x7f9ff414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff415c760 0x7f9ff415a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b5d60 con 0x7f9ff40cb500 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b3810 con 0x7f9ff40cb130 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ffb22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b2ee0 con 0x7f9ff415c760 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47546/0 (socket says 172.21.15.111:47546) 2023-12-19T07:48:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 -- 172.21.15.111:0/1597092105 learned_addr learned my addr 172.21.15.111:0/1597092105 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff8fca700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 0x7f9ff414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff97cb700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff415c760 0x7f9ff415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 -- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff415c760 msgr2=0x7f9ff415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff415c760 0x7f9ff415a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 -- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 msgr2=0x7f9ff414fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 0x7f9ff414fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff3fff700 1 -- 172.21.15.111:0/1597092105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0009420 con 0x7f9ff40cb500 2023-12-19T07:48:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.030+0000 7f9ff8fca700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 0x7f9ff414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:45.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ff3fff700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ec80 tx=0x7f9fe00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000c920 con 0x7f9ff40cb500 2023-12-19T07:48:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe000ca80 con 0x7f9ff40cb500 2023-12-19T07:48:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec009ae0 con 0x7f9ff40cb500 2023-12-19T07:48:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000ce00 con 0x7f9ff40cb500 2023-12-19T07:48:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff40b34a0 con 0x7f9ff40cb500 2023-12-19T07:48:45.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.034+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40b39a0 con 0x7f9ff40cb500 2023-12-19T07:48:45.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.038+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f9fe0013680 con 0x7f9ff40cb500 2023-12-19T07:48:45.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.038+0000 7f9ff0ff9700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 0x7f9fd8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.038+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f9fe007d0c0 con 0x7f9ff40cb500 2023-12-19T07:48:45.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.038+0000 7f9ff8fca700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 0x7f9fd8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.038+0000 7f9ff8fca700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 0x7f9fd8062c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8001710 tx=0x7f9fe8007eb0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.046+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9fe004d1f0 con 0x7f9ff40cb500 2023-12-19T07:48:45.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.238+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ff4071120 con 0x7f9fd8060750 2023-12-19T07:48:45.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.362+0000 7f9ff0ff9700 1 -- 172.21.15.111:0/1597092105 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f9ff4071120 con 0x7f9fd8060750 2023-12-19T07:48:45.347 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir6 2023-12-19T07:48:45.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 msgr2=0x7f9fd8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 0x7f9fd8062c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8001710 tx=0x7f9fe8007eb0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 msgr2=0x7f9ff414dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ec80 tx=0x7f9fe00091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 shutdown_connections 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9fd8060750 0x7f9fd8062c00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ff415c760 0x7f9ff415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff40cb500 0x7f9ff414dd00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 --2- 172.21.15.111:0/1597092105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ff40cb130 0x7f9ff414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 >> 172.21.15.111:0/1597092105 conn(0x7f9ff4067bb0 msgr2=0x7f9ff40cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 shutdown_connections 2023-12-19T07:48:45.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.366+0000 7f9ffb22e700 1 -- 172.21.15.111:0/1597092105 wait complete. 2023-12-19T07:48:45.374 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-12-19T07:48:45.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:45.375 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-12-19T07:48:45.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T07:48:45.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 Processor -- start 2023-12-19T07:48:45.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 -- start start 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 0x7f8e200cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e2006a6a0 0x7f8e2006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 0x7f8e2015c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2006b030 con 0x7f8e2006a6a0 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e200b0ac0 con 0x7f8e200cb130 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e272ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2015cbb0 con 0x7f8e2015c220 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e2006a6a0 0x7f8e2006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e2006a6a0 0x7f8e2006aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47552/0 (socket says 172.21.15.111:47552) 2023-12-19T07:48:45.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 -- 172.21.15.111:0/2408644485 learned_addr learned my addr 172.21.15.111:0/2408644485 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e2504a700 1 --2- 172.21.15.111:0/2408644485 >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 0x7f8e200cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 -- 172.21.15.111:0/2408644485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 msgr2=0x7f8e2015c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e2584b700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 0x7f8e2015c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 0x7f8e2015c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 -- 172.21.15.111:0/2408644485 >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 msgr2=0x7f8e200cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 --2- 172.21.15.111:0/2408644485 >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 0x7f8e200cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e24849700 1 -- 172.21.15.111:0/2408644485 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2006b210 con 0x7f8e2006a6a0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e2504a700 1 --2- 172.21.15.111:0/2408644485 >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 0x7f8e200cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.634+0000 7f8e2584b700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 0x7f8e2015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e24849700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2006aaf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8e200694e0 tx=0x7f8e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203f8da5d7db46b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e17fff700 1 -- 172.21.15.111:0/2408644485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000b860 con 0x7f8e2006a6a0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e17fff700 1 -- 172.21.15.111:0/2408644485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1000b9c0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e17fff700 1 -- 172.21.15.111:0/2408644485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000bd60 con 0x7f8e2006a6a0 2023-12-19T07:48:45.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 -- 172.21.15.111:0/2408644485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 msgr2=0x7f8e2006aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2006aaf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8e200694e0 tx=0x7f8e10009250 comp rx=0 tx=0).stop 2023-12-19T07:48:45.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 -- 172.21.15.111:0/2408644485 shutdown_connections 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e2015c220 0x7f8e2015c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 --2- 172.21.15.111:0/2408644485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2006aaf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 --2- 172.21.15.111:0/2408644485 >> v2:172.21.15.143:3300/0 conn(0x7f8e200cb130 0x7f8e200cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 -- 172.21.15.111:0/2408644485 >> 172.21.15.111:0/2408644485 conn(0x7f8e20067bb0 msgr2=0x7f8e200cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 -- 172.21.15.111:0/2408644485 shutdown_connections 2023-12-19T07:48:45.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.638+0000 7f8e272ae700 1 -- 172.21.15.111:0/2408644485 wait complete. 2023-12-19T07:48:45.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 Processor -- start 2023-12-19T07:48:45.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- start start 2023-12-19T07:48:45.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e200cb130 0x7f8e20143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e2015c220 0x7f8e2015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2015a110 con 0x7f8e2006a6a0 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2015bb70 con 0x7f8e2015c220 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20159500 con 0x7f8e200cb130 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47556/0 (socket says 172.21.15.111:47556) 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 -- 172.21.15.111:0/1782532412 learned_addr learned my addr 172.21.15.111:0/1782532412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2584b700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e2015c220 0x7f8e2015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e24849700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e200cb130 0x7f8e20143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 -- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e200cb130 msgr2=0x7f8e20143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e200cb130 0x7f8e20143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 -- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e2015c220 msgr2=0x7f8e2015a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e2015c220 0x7f8e2015a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 -- 172.21.15.111:0/1782532412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100094f0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e2504a700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c880 tx=0x7f8e1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c011600 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e201597e0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c011760 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c011ae0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.642+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e20159da0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.646+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e20071120 con 0x7f8e2006a6a0 2023-12-19T07:48:45.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.646+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f8e1c010350 con 0x7f8e2006a6a0 2023-12-19T07:48:45.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.650+0000 7f8e157fa700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 0x7f8e04062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:45.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.650+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f8e1c07cf20 con 0x7f8e2006a6a0 2023-12-19T07:48:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.650+0000 7f8e24849700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 0x7f8e04062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.650+0000 7f8e24849700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 0x7f8e04062c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8e200694e0 tx=0x7f8e10017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:45.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.654+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e1c0509d0 con 0x7f8e2006a6a0 2023-12-19T07:48:45.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.850+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e2015a250 con 0x7f8e04060750 2023-12-19T07:48:45.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.970+0000 7f8e157fa700 1 -- 172.21.15.111:0/1782532412 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8e1000bb30 con 0x7f8e04060750 2023-12-19T07:48:45.954 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir7 2023-12-19T07:48:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 msgr2=0x7f8e04062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 0x7f8e04062c00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8e200694e0 tx=0x7f8e10017000 comp rx=0 tx=0).stop 2023-12-19T07:48:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 msgr2=0x7f8e2015b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c880 tx=0x7f8e1c008040 comp rx=0 tx=0).stop 2023-12-19T07:48:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 shutdown_connections 2023-12-19T07:48:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e04060750 0x7f8e04062c00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e2015c220 0x7f8e2015a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e200cb130 0x7f8e20143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 --2- 172.21.15.111:0/1782532412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e2006a6a0 0x7f8e2015b630 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 >> 172.21.15.111:0/1782532412 conn(0x7f8e20067bb0 msgr2=0x7f8e200b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 shutdown_connections 2023-12-19T07:48:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:45.974+0000 7f8e272ae700 1 -- 172.21.15.111:0/1782532412 wait complete. 2023-12-19T07:48:45.980 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-12-19T07:48:45.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:45.980 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-12-19T07:48:46.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T07:48:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.254+0000 7f7644545700 1 Processor -- start 2023-12-19T07:48:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.254+0000 7f7644545700 1 -- start start 2023-12-19T07:48:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7644545700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f763c159220 0x7f763c068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7644545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763c06d0d0 con 0x7f763c158e50 2023-12-19T07:48:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7644545700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763c06c7c0 con 0x7f763c069310 2023-12-19T07:48:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7644545700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763c0c7b40 con 0x7f763c159220 2023-12-19T07:48:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76422e1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f763c158e50 0x7f763c0c72c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48196/0 (socket says 172.21.15.111:48196) 2023-12-19T07:48:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76422e1700 1 -- 172.21.15.111:0/2118294279 learned_addr learned my addr 172.21.15.111:0/2118294279 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7641ae0700 1 --2- 172.21.15.111:0/2118294279 >> v2:172.21.15.143:3300/0 conn(0x7f763c159220 0x7f763c068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682505422 0 0) 0x7f763c06d0d0 con 0x7f763c158e50 2023-12-19T07:48:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7628003570 con 0x7f763c158e50 2023-12-19T07:48:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805644469 0 0) 0x7f763c06c7c0 con 0x7f763c069310 2023-12-19T07:48:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f763c06d0d0 con 0x7f763c069310 2023-12-19T07:48:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7641ae0700 1 -- 172.21.15.111:0/2118294279 >> v1:172.21.15.143:6789/0 conn(0x7f763c069310 legacy=0x7f763c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7641ae0700 1 -- 172.21.15.111:0/2118294279 >> v1:172.21.15.111:6789/0 conn(0x7f763c158e50 legacy=0x7f763c0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7641ae0700 1 -- 172.21.15.111:0/2118294279 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c06ca80 con 0x7f763c159220 2023-12-19T07:48:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f7641ae0700 1 --2- 172.21.15.111:0/2118294279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c159220 0x7f763c068380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f762c0098e0 tx=0x7f762c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a137063f421cbdc4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c011630 con 0x7f763c159220 2023-12-19T07:48:46.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c011790 con 0x7f763c159220 2023-12-19T07:48:46.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.258+0000 7f76412df700 1 -- 172.21.15.111:0/2118294279 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c011b30 con 0x7f763c159220 2023-12-19T07:48:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 -- 172.21.15.111:0/2118294279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c159220 msgr2=0x7f763c068380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 --2- 172.21.15.111:0/2118294279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c159220 0x7f763c068380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f762c0098e0 tx=0x7f762c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 -- 172.21.15.111:0/2118294279 shutdown_connections 2023-12-19T07:48:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 --2- 172.21.15.111:0/2118294279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c159220 0x7f763c068380 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 -- 172.21.15.111:0/2118294279 >> 172.21.15.111:0/2118294279 conn(0x7f763c0c8db0 msgr2=0x7f763c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:46.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 -- 172.21.15.111:0/2118294279 shutdown_connections 2023-12-19T07:48:46.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.262+0000 7f7644545700 1 -- 172.21.15.111:0/2118294279 wait complete. 2023-12-19T07:48:46.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 Processor -- start 2023-12-19T07:48:46.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 -- start start 2023-12-19T07:48:46.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c069310 0x7f763c0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 0x7f763c0c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c0b06e0 con 0x7f763c158e50 2023-12-19T07:48:46.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c0afad0 con 0x7f763c069310 2023-12-19T07:48:46.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7644545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763c0afdb0 con 0x7f763c159220 2023-12-19T07:48:46.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47562/0 (socket says 172.21.15.111:47562) 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 -- 172.21.15.111:0/3552063969 learned_addr learned my addr 172.21.15.111:0/3552063969 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f76422e1700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c069310 0x7f763c0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7642ae2700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 0x7f763c0c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 -- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 msgr2=0x7f763c0c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 0x7f763c0c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 -- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c069310 msgr2=0x7f763c0af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c069310 0x7f763c0af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7641ae0700 1 -- 172.21.15.111:0/3552063969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c009420 con 0x7f763c158e50 2023-12-19T07:48:46.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.266+0000 7f7642ae2700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 0x7f763c0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:46.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f7641ae0700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f763c0c7eb0 tx=0x7f762c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c01f070 con 0x7f763c158e50 2023-12-19T07:48:46.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c000bc0 con 0x7f763c158e50 2023-12-19T07:48:46.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763c0b0090 con 0x7f763c158e50 2023-12-19T07:48:46.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c011b30 con 0x7f763c158e50 2023-12-19T07:48:46.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763c0b1ba0 con 0x7f763c158e50 2023-12-19T07:48:46.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.270+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763c071120 con 0x7f763c158e50 2023-12-19T07:48:46.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.274+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f762c023070 con 0x7f763c158e50 2023-12-19T07:48:46.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.274+0000 7f76327fc700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 0x7f7620062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.274+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f762c07de90 con 0x7f763c158e50 2023-12-19T07:48:46.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.274+0000 7f76422e1700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 0x7f7620062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.274+0000 7f76422e1700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 0x7f7620062c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7638007fd0 tx=0x7f7638009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.282+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f762c051a60 con 0x7f763c158e50 2023-12-19T07:48:46.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.470+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f763c06cd60 con 0x7f7620060750 2023-12-19T07:48:46.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.594+0000 7f76327fc700 1 -- 172.21.15.111:0/3552063969 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f763c06cd60 con 0x7f7620060750 2023-12-19T07:48:46.578 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir8 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.594+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 msgr2=0x7f7620062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.594+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 0x7f7620062c00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7638007fd0 tx=0x7f7638009040 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.594+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 msgr2=0x7f763c0c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f763c0c7eb0 tx=0x7f762c017000 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 shutdown_connections 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7620060750 0x7f7620062c00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f763c159220 0x7f763c0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f763c158e50 0x7f763c0c5600 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 --2- 172.21.15.111:0/3552063969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f763c069310 0x7f763c0af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 >> 172.21.15.111:0/3552063969 conn(0x7f763c0c8db0 msgr2=0x7f763c0ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 shutdown_connections 2023-12-19T07:48:46.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.598+0000 7f7644545700 1 -- 172.21.15.111:0/3552063969 wait complete. 2023-12-19T07:48:46.606 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-12-19T07:48:46.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:48:46.607 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-12-19T07:48:46.629 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T07:48:46.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 Processor -- start 2023-12-19T07:48:46.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 -- start start 2023-12-19T07:48:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e8813eca0 0x7f6e88068300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e880cb970 con 0x7f6e880cb510 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e88068c90 con 0x7f6e8813eca0 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.870+0000 7f6e87fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e88159750 con 0x7f6e88068840 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e8813eca0 0x7f6e88068300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e8813eca0 0x7f6e88068300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47572/0 (socket says 172.21.15.111:47572) 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 -- 172.21.15.111:0/1430347736 learned_addr learned my addr 172.21.15.111:0/1430347736 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e867fc700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 -- 172.21.15.111:0/1430347736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 msgr2=0x7f6e880cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 -- 172.21.15.111:0/1430347736 >> v1:172.21.15.143:6789/0 conn(0x7f6e880cb510 legacy=0x7f6e880c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 -- 172.21.15.111:0/1430347736 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8806df60 con 0x7f6e8813eca0 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e867fc700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:46.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e86ffd700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88068300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6e780099b0 tx=0x7f6e78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2618bbe62b4dacdc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e85ffb700 1 -- 172.21.15.111:0/1430347736 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800b860 con 0x7f6e8813eca0 2023-12-19T07:48:46.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e85ffb700 1 -- 172.21.15.111:0/1430347736 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7800b9c0 con 0x7f6e8813eca0 2023-12-19T07:48:46.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e85ffb700 1 -- 172.21.15.111:0/1430347736 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800bd60 con 0x7f6e8813eca0 2023-12-19T07:48:46.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 -- 172.21.15.111:0/1430347736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 msgr2=0x7f6e88068300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88068300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6e780099b0 tx=0x7f6e78009250 comp rx=0 tx=0).stop 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 -- 172.21.15.111:0/1430347736 shutdown_connections 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 --2- 172.21.15.111:0/1430347736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88068300 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 -- 172.21.15.111:0/1430347736 >> 172.21.15.111:0/1430347736 conn(0x7f6e880c8c50 msgr2=0x7f6e880c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 -- 172.21.15.111:0/1430347736 shutdown_connections 2023-12-19T07:48:46.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.874+0000 7f6e87fff700 1 -- 172.21.15.111:0/1430347736 wait complete. 2023-12-19T07:48:46.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.878+0000 7f6e87fff700 1 Processor -- start 2023-12-19T07:48:46.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.878+0000 7f6e87fff700 1 -- start start 2023-12-19T07:48:46.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 0x7f6e880c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e880ade40 con 0x7f6e8813eca0 2023-12-19T07:48:46.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e880ad230 con 0x7f6e880cb510 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e880ad510 con 0x7f6e88068840 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 0x7f6e880c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47584/0 (socket says 172.21.15.111:47584) 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e877fe700 1 -- 172.21.15.111:0/3531322898 learned_addr learned my addr 172.21.15.111:0/3531322898 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:46.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e86ffd700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 -- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 msgr2=0x7f6e880acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 -- 172.21.15.111:0/3531322898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 msgr2=0x7f6e88154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 -- 172.21.15.111:0/3531322898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780094f0 con 0x7f6e880cb510 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e877fe700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e867fc700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 0x7f6e880c54b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c006fd0 tx=0x7f6e7c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0125f0 con 0x7f6e880cb510 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c012750 con 0x7f6e880cb510 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c012b10 con 0x7f6e880cb510 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.882+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e880ad7f0 con 0x7f6e880cb510 2023-12-19T07:48:46.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.886+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e880c65e0 con 0x7f6e880cb510 2023-12-19T07:48:46.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.886+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e880adff0 con 0x7f6e880cb510 2023-12-19T07:48:46.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.886+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f6e7c013480 con 0x7f6e880cb510 2023-12-19T07:48:46.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.890+0000 7f6e6effd700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 0x7f6e74062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:46.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.890+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f6e7c07cf00 con 0x7f6e880cb510 2023-12-19T07:48:46.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.890+0000 7f6e86ffd700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 0x7f6e74062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:46.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.890+0000 7f6e86ffd700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 0x7f6e74062c00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e780099b0 tx=0x7f6e78017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:46.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:46.898+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6e880adff0 con 0x7f6e880cb510 2023-12-19T07:48:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.090+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e880adfb0 con 0x7f6e74060750 2023-12-19T07:48:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.202+0000 7f6e6effd700 1 -- 172.21.15.111:0/3531322898 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6e880adfb0 con 0x7f6e74060750 2023-12-19T07:48:47.188 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/dir9 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 msgr2=0x7f6e74062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 0x7f6e74062c00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e780099b0 tx=0x7f6e78017000 comp rx=0 tx=0).stop 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 msgr2=0x7f6e880c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 0x7f6e880c54b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c006fd0 tx=0x7f6e7c008040 comp rx=0 tx=0).stop 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 shutdown_connections 2023-12-19T07:48:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e74060750 0x7f6e74062c00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e880cb510 0x7f6e880c54b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e8813eca0 0x7f6e88154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 --2- 172.21.15.111:0/3531322898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e88068840 0x7f6e880acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 >> 172.21.15.111:0/3531322898 conn(0x7f6e880c8c50 msgr2=0x7f6e8806b290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 shutdown_connections 2023-12-19T07:48:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.206+0000 7f6e87fff700 1 -- 172.21.15.111:0/3531322898 wait complete. 2023-12-19T07:48:47.214 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-12-19T07:48:47.214 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T07:48:47.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.402+0000 7fa39f8f0700 1 Processor -- start 2023-12-19T07:48:47.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 -- start start 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa398068a90 0x7fa39806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39806d0d0 con 0x7fa398159220 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39806c7c0 con 0x7fa398068a90 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.406+0000 7fa39f8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980c7b40 con 0x7fa398158e50 2023-12-19T07:48:47.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39ce8b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa398159220 0x7fa398068380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48206/0 (socket says 172.21.15.111:48206) 2023-12-19T07:48:47.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39ce8b700 1 -- 172.21.15.111:0/2921002112 learned_addr learned my addr 172.21.15.111:0/2921002112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:47.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 --2- 172.21.15.111:0/2921002112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39de8d700 1 --2- 172.21.15.111:0/2921002112 >> v2:172.21.15.111:3300/0 conn(0x7fa398068a90 0x7fa39806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa38ffff700 1 -- 172.21.15.111:0/2921002112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681192932 0 0) 0x7fa39806d0d0 con 0x7fa398159220 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa38ffff700 1 -- 172.21.15.111:0/2921002112 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa384003570 con 0x7fa398159220 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 -- 172.21.15.111:0/2921002112 >> v2:172.21.15.111:3300/0 conn(0x7fa398068a90 msgr2=0x7fa39806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 --2- 172.21.15.111:0/2921002112 >> v2:172.21.15.111:3300/0 conn(0x7fa398068a90 0x7fa39806d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 -- 172.21.15.111:0/2921002112 >> v1:172.21.15.111:6789/0 conn(0x7fa398159220 legacy=0x7fa398068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 -- 172.21.15.111:0/2921002112 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39806ca80 con 0x7fa398158e50 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39d68c700 1 --2- 172.21.15.111:0/2921002112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c72c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa3940098e0 tx=0x7fa3940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47fd24b359284ba server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa38ffff700 1 -- 172.21.15.111:0/2921002112 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa394011630 con 0x7fa398158e50 2023-12-19T07:48:47.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa38ffff700 1 -- 172.21.15.111:0/2921002112 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa394011790 con 0x7fa398158e50 2023-12-19T07:48:47.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa38ffff700 1 -- 172.21.15.111:0/2921002112 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa394011b30 con 0x7fa398158e50 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 -- 172.21.15.111:0/2921002112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 msgr2=0x7fa3980c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/2921002112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c72c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa3940098e0 tx=0x7fa3940091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 -- 172.21.15.111:0/2921002112 shutdown_connections 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/2921002112 >> v2:172.21.15.111:3300/0 conn(0x7fa398068a90 0x7fa39806d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/2921002112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c72c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.410+0000 7fa39f8f0700 1 -- 172.21.15.111:0/2921002112 >> 172.21.15.111:0/2921002112 conn(0x7fa3980c8db0 msgr2=0x7fa3980c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:47.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- 172.21.15.111:0/2921002112 shutdown_connections 2023-12-19T07:48:47.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- 172.21.15.111:0/2921002112 wait complete. 2023-12-19T07:48:47.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 Processor -- start 2023-12-19T07:48:47.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- start start 2023-12-19T07:48:47.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa398068a90 0x7fa3980a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980a97b0 con 0x7fa398068a90 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980b0770 con 0x7fa398159220 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.414+0000 7fa39f8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980aa660 con 0x7fa398158e50 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39d68c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa398068a90 0x7fa3980a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45884/0 (socket says 172.21.15.111:45884) 2023-12-19T07:48:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 -- 172.21.15.111:0/985527251 learned_addr learned my addr 172.21.15.111:0/985527251 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39ce8b700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 -- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 msgr2=0x7fa3980c0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 -- 172.21.15.111:0/985527251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa398068a90 msgr2=0x7fa3980a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa398068a90 0x7fa3980a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 -- 172.21.15.111:0/985527251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394009420 con 0x7fa398159220 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39de8d700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa390009ef0 tx=0x7fa39000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39000cb70 con 0x7fa398159220 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39000f040 con 0x7fa398159220 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa390014480 con 0x7fa398159220 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39816c970 con 0x7fa398159220 2023-12-19T07:48:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39816cf30 con 0x7fa398159220 2023-12-19T07:48:47.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.418+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3980b0900 con 0x7fa398159220 2023-12-19T07:48:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.422+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fa3900147f0 con 0x7fa398159220 2023-12-19T07:48:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.422+0000 7fa38d7fa700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 0x7fa37c062c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.422+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fa39007db00 con 0x7fa398159220 2023-12-19T07:48:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.422+0000 7fa39d68c700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 0x7fa37c062c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.422+0000 7fa39d68c700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 0x7fa37c062c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa3940098e0 tx=0x7fa394017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.430+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa39004da80 con 0x7fa398159220 2023-12-19T07:48:47.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.626+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa39806cd60 con 0x7fa37c0607d0 2023-12-19T07:48:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.722+0000 7fa38d7fa700 1 -- 172.21.15.111:0/985527251 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa39806cd60 con 0x7fa37c0607d0 2023-12-19T07:48:47.708 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-12-19T07:48:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 msgr2=0x7fa37c062c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 0x7fa37c062c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa3940098e0 tx=0x7fa394017000 comp rx=0 tx=0).stop 2023-12-19T07:48:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 msgr2=0x7fa3981547e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa390009ef0 tx=0x7fa39000b040 comp rx=0 tx=0).stop 2023-12-19T07:48:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 shutdown_connections 2023-12-19T07:48:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa37c0607d0 0x7fa37c062c80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa398159220 0x7fa3981547e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa398158e50 0x7fa3980c0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 --2- 172.21.15.111:0/985527251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa398068a90 0x7fa3980a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 >> 172.21.15.111:0/985527251 conn(0x7fa3980c8db0 msgr2=0x7fa3980ad4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 shutdown_connections 2023-12-19T07:48:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.726+0000 7fa39f8f0700 1 -- 172.21.15.111:0/985527251 wait complete. 2023-12-19T07:48:47.733 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-12-19T07:48:47.734 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T07:48:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.918+0000 7fb346d45700 1 Processor -- start 2023-12-19T07:48:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.918+0000 7fb346d45700 1 -- start start 2023-12-19T07:48:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb346d45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb3400cb130 0x7fb3400cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb346d45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3400b0ac0 con 0x7fb34006a6a0 2023-12-19T07:48:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb346d45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34006b210 con 0x7fb34015c930 2023-12-19T07:48:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb346d45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400ca810 con 0x7fb3400cb130 2023-12-19T07:48:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb33ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb34006a6a0 0x7fb34015c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48214/0 (socket says 172.21.15.111:48214) 2023-12-19T07:48:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb33ffff700 1 -- 172.21.15.111:0/4272371687 learned_addr learned my addr 172.21.15.111:0/4272371687 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.922+0000 7fb344ae1700 1 --2- 172.21.15.111:0/4272371687 >> v2:172.21.15.143:3300/0 conn(0x7fb3400cb130 0x7fb3400cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb33f7fe700 1 -- 172.21.15.111:0/4272371687 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926182511 0 0) 0x7fb3400b0ac0 con 0x7fb34006a6a0 2023-12-19T07:48:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb33f7fe700 1 -- 172.21.15.111:0/4272371687 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb330003570 con 0x7fb34006a6a0 2023-12-19T07:48:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb344ae1700 1 -- 172.21.15.111:0/4272371687 >> v1:172.21.15.143:6789/0 conn(0x7fb34015c930 legacy=0x7fb3400b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb344ae1700 1 -- 172.21.15.111:0/4272371687 >> v1:172.21.15.111:6789/0 conn(0x7fb34006a6a0 legacy=0x7fb34015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb344ae1700 1 -- 172.21.15.111:0/4272371687 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400b0190 con 0x7fb3400cb130 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb344ae1700 1 --2- 172.21.15.111:0/4272371687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400cb500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb3340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1614672252accfc6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb33f7fe700 1 -- 172.21.15.111:0/4272371687 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400b640 con 0x7fb3400cb130 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb33f7fe700 1 -- 172.21.15.111:0/4272371687 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33400b7a0 con 0x7fb3400cb130 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb33f7fe700 1 -- 172.21.15.111:0/4272371687 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400bb40 con 0x7fb3400cb130 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 -- 172.21.15.111:0/4272371687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 msgr2=0x7fb3400cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 --2- 172.21.15.111:0/4272371687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400cb500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb3340091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 -- 172.21.15.111:0/4272371687 shutdown_connections 2023-12-19T07:48:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 --2- 172.21.15.111:0/4272371687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400cb500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 -- 172.21.15.111:0/4272371687 >> 172.21.15.111:0/4272371687 conn(0x7fb340067bb0 msgr2=0x7fb3400cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 -- 172.21.15.111:0/4272371687 shutdown_connections 2023-12-19T07:48:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.926+0000 7fb346d45700 1 -- 172.21.15.111:0/4272371687 wait complete. 2023-12-19T07:48:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 Processor -- start 2023-12-19T07:48:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 -- start start 2023-12-19T07:48:47.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34006a6a0 0x7fb34015b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3401435b0 con 0x7fb34015c930 2023-12-19T07:48:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340158940 con 0x7fb3400cb130 2023-12-19T07:48:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb346d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb340158c20 con 0x7fb34006a6a0 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47602/0 (socket says 172.21.15.111:47602) 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb33ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 -- 172.21.15.111:0/6200854 learned_addr learned my addr 172.21.15.111:0/6200854 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb344ae1700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34006a6a0 0x7fb34015b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 -- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34006a6a0 msgr2=0x7fb34015b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34006a6a0 0x7fb34015b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 -- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 msgr2=0x7fb3400b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400b4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb3452e2700 1 -- 172.21.15.111:0/6200854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334009420 con 0x7fb34015c930 2023-12-19T07:48:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.930+0000 7fb33ffff700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb3452e2700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb33800c810 tx=0x7fb338001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338012630 con 0x7fb34015c930 2023-12-19T07:48:47.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380127b0 con 0x7fb34015c930 2023-12-19T07:48:47.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338012b70 con 0x7fb34015c930 2023-12-19T07:48:47.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb340158f00 con 0x7fb34015c930 2023-12-19T07:48:47.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3401594c0 con 0x7fb34015c930 2023-12-19T07:48:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.934+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb340071120 con 0x7fb34015c930 2023-12-19T07:48:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.938+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fb338003030 con 0x7fb34015c930 2023-12-19T07:48:47.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.938+0000 7fb33cff9700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 0x7fb324062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:47.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.938+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fb33807cc30 con 0x7fb34015c930 2023-12-19T07:48:47.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.938+0000 7fb344ae1700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 0x7fb324062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:47.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.938+0000 7fb344ae1700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 0x7fb324062c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3340093f0 tx=0x7fb334017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:47.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:47.946+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb33804cd60 con 0x7fb34015c930 2023-12-19T07:48:48.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.138+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3400cbe90 con 0x7fb324060750 2023-12-19T07:48:48.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.242+0000 7fb33cff9700 1 -- 172.21.15.111:0/6200854 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb3400cbe90 con 0x7fb324060750 2023-12-19T07:48:48.226 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-12-19T07:48:48.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 msgr2=0x7fb324062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 0x7fb324062c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3340093f0 tx=0x7fb334017000 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 msgr2=0x7fb3400b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb33800c810 tx=0x7fb338001650 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 shutdown_connections 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb324060750 0x7fb324062c00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3400cb130 0x7fb3400b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34015c930 0x7fb3400b4b80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 --2- 172.21.15.111:0/6200854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34006a6a0 0x7fb34015b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 >> 172.21.15.111:0/6200854 conn(0x7fb340067bb0 msgr2=0x7fb3400cdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:48.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 shutdown_connections 2023-12-19T07:48:48.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.246+0000 7fb346d45700 1 -- 172.21.15.111:0/6200854 wait complete. 2023-12-19T07:48:48.252 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-12-19T07:48:48.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T07:48:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.438+0000 7f5966185700 1 Processor -- start 2023-12-19T07:48:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.438+0000 7f5966185700 1 -- start start 2023-12-19T07:48:48.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.438+0000 7f5966185700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f59600cb500 0x7f596015c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f5966185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59600b0ac0 con 0x7f596015c760 2023-12-19T07:48:48.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f5966185700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596006b210 con 0x7f59600cb130 2023-12-19T07:48:48.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f5966185700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596015cbe0 con 0x7f59600cb500 2023-12-19T07:48:48.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f596015c760 0x7f59600b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48216/0 (socket says 172.21.15.111:48216) 2023-12-19T07:48:48.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595ffff700 1 -- 172.21.15.111:0/2332431088 learned_addr learned my addr 172.21.15.111:0/2332431088 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:48.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595effd700 1 --2- 172.21.15.111:0/2332431088 >> v2:172.21.15.111:3300/0 conn(0x7f59600cb500 0x7f596015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792066607 0 0) 0x7f59600b0ac0 con 0x7f596015c760 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948003570 con 0x7f596015c760 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206123329 0 0) 0x7f596006b210 con 0x7f59600cb130 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59600b0ac0 con 0x7f59600cb130 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595effd700 1 -- 172.21.15.111:0/2332431088 >> v1:172.21.15.143:6789/0 conn(0x7f59600cb130 legacy=0x7f596006a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410417473 0 0) 0x7f5948003570 con 0x7f596015c760 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595effd700 1 -- 172.21.15.111:0/2332431088 >> v1:172.21.15.111:6789/0 conn(0x7f596015c760 legacy=0x7f59600b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595effd700 1 -- 172.21.15.111:0/2332431088 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600ca810 con 0x7f59600cb500 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595effd700 1 --2- 172.21.15.111:0/2332431088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb500 0x7f596015c220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59540098e0 tx=0x7f59540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab14139eb9ff2fc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59580017f0 con 0x7f59600cb500 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954011420 con 0x7f59600cb500 2023-12-19T07:48:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.442+0000 7f595e7fc700 1 -- 172.21.15.111:0/2332431088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540117c0 con 0x7f59600cb500 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 -- 172.21.15.111:0/2332431088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb500 msgr2=0x7f596015c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 --2- 172.21.15.111:0/2332431088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb500 0x7f596015c220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59540098e0 tx=0x7f59540091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 -- 172.21.15.111:0/2332431088 shutdown_connections 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 --2- 172.21.15.111:0/2332431088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb500 0x7f596015c220 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 -- 172.21.15.111:0/2332431088 >> 172.21.15.111:0/2332431088 conn(0x7f5960067bb0 msgr2=0x7f59600cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 -- 172.21.15.111:0/2332431088 shutdown_connections 2023-12-19T07:48:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 -- 172.21.15.111:0/2332431088 wait complete. 2023-12-19T07:48:48.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.446+0000 7f5966185700 1 Processor -- start 2023-12-19T07:48:48.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 -- start start 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600cb500 0x7f59600b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f596015c760 0x7f59601521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b5350 con 0x7f59600cb130 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b4740 con 0x7f59600cb500 2023-12-19T07:48:48.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600b4a20 con 0x7f596015c760 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600cb500 0x7f59600b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47626/0 (socket says 172.21.15.111:47626) 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 -- 172.21.15.111:0/884488532 learned_addr learned my addr 172.21.15.111:0/884488532 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595ffff700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f596015c760 0x7f59601521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 -- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f596015c760 msgr2=0x7f59601521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f596015c760 0x7f59601521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 -- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600cb500 msgr2=0x7f59600b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600cb500 0x7f59600b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 -- 172.21.15.111:0/884488532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5954009420 con 0x7f59600cb130 2023-12-19T07:48:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f595f7fe700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f595000c790 tx=0x7f5950007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950010510 con 0x7f59600cb130 2023-12-19T07:48:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5950010670 con 0x7f59600cb130 2023-12-19T07:48:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950010a10 con 0x7f59600cb130 2023-12-19T07:48:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.450+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59600b4d00 con 0x7f59600cb130 2023-12-19T07:48:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.454+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59601526f0 con 0x7f59600cb130 2023-12-19T07:48:48.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.454+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59600b5d20 con 0x7f59600cb130 2023-12-19T07:48:48.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.454+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f595001b3b0 con 0x7f59600cb130 2023-12-19T07:48:48.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.458+0000 7f59477fe700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 0x7f594c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.458+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f595007c5a0 con 0x7f59600cb130 2023-12-19T07:48:48.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.458+0000 7f595effd700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 0x7f594c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.458+0000 7f595effd700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 0x7f594c062c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59540093f0 tx=0x7f5954016000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.462+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f595004c6d0 con 0x7f59600cb130 2023-12-19T07:48:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.658+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f59600b5490 con 0x7f594c060780 2023-12-19T07:48:48.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.758+0000 7f59477fe700 1 -- 172.21.15.111:0/884488532 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5954011790 con 0x7f594c060780 2023-12-19T07:48:48.744 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-12-19T07:48:48.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 msgr2=0x7f594c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 0x7f594c062c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59540093f0 tx=0x7f5954016000 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 msgr2=0x7f59600b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f595000c790 tx=0x7f5950007e10 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 shutdown_connections 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f594c060780 0x7f594c062c30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59600cb500 0x7f59600b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f596015c760 0x7f59601521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 --2- 172.21.15.111:0/884488532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59600cb130 0x7f59600b4160 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 >> 172.21.15.111:0/884488532 conn(0x7f5960067bb0 msgr2=0x7f596014ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:48.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 shutdown_connections 2023-12-19T07:48:48.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.762+0000 7f5966185700 1 -- 172.21.15.111:0/884488532 wait complete. 2023-12-19T07:48:48.772 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-12-19T07:48:48.773 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T07:48:48.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.966+0000 7f8b7e203700 1 Processor -- start 2023-12-19T07:48:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 -- start start 2023-12-19T07:48:48.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8b780c9440 0x7f8b780c9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b780cbbc0 0x7f8b780cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b780cb730 con 0x7f8b780ac600 2023-12-19T07:48:48.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b78161c20 con 0x7f8b780c9440 2023-12-19T07:48:48.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7e203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780cb0e0 con 0x7f8b780cbbc0 2023-12-19T07:48:48.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7da02700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8b780ac600 0x7f8b780aca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48224/0 (socket says 172.21.15.111:48224) 2023-12-19T07:48:48.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7da02700 1 -- 172.21.15.111:0/1480107846 learned_addr learned my addr 172.21.15.111:0/1480107846 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:48.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7ca00700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b780cbbc0 0x7f8b780cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.970+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> v2:172.21.15.111:3300/0 conn(0x7f8b780c9440 0x7f8b780c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459655980 0 0) 0x7f8b780cb730 con 0x7f8b780ac600 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b64003570 con 0x7f8b780ac600 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753750205 0 0) 0x7f8b64003570 con 0x7f8b780ac600 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b780cbbc0 msgr2=0x7f8b780cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b780cbbc0 0x7f8b780cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 >> v2:172.21.15.111:3300/0 conn(0x7f8b780c9440 msgr2=0x7f8b780c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 --2- 172.21.15.111:0/1480107846 >> v2:172.21.15.111:3300/0 conn(0x7f8b780c9440 0x7f8b780c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780cb3c0 con 0x7f8b780ac600 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f8b700017f0 con 0x7f8b780ac600 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> v2:172.21.15.111:3300/0 conn(0x7f8b780c9440 0x7f8b780c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 >> v1:172.21.15.111:6789/0 conn(0x7f8b780ac600 legacy=0x7f8b780aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b64003ae0 0x7f8b64003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64004810 0x7f8b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b640051c0 0x7f8b6400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b78161c20 con 0x7f8b640051c0 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780cb0e0 con 0x7f8b64004810 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6400c080 con 0x7f8b64003ae0 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7ca00700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b780cbbc0 0x7f8b780cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b64003ae0 0x7f8b64003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7da02700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b640051c0 0x7f8b6400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7ca00700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64004810 0x7f8b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 -- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64004810 msgr2=0x7f8b64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64004810 0x7f8b64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 -- 172.21.15.111:0/1480107846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b640051c0 msgr2=0x7f8b6400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b640051c0 0x7f8b6400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.974+0000 7f8b7d201700 1 -- 172.21.15.111:0/1480107846 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64003570 con 0x7f8b64003ae0 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7d201700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b64003ae0 0x7f8b64003ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009df0 tx=0x7f8b740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc48fedf07a2e0c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400e920 con 0x7f8b64003ae0 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7400ea80 con 0x7f8b64003ae0 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b6ffff700 1 -- 172.21.15.111:0/1480107846 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400ee20 con 0x7f8b64003ae0 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 -- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b64003ae0 msgr2=0x7f8b64003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 --2- 172.21.15.111:0/1480107846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b64003ae0 0x7f8b64003ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009df0 tx=0x7f8b740091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7da02700 1 -- 172.21.15.111:0/1480107846 reap_dead start 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 -- 172.21.15.111:0/1480107846 shutdown_connections 2023-12-19T07:48:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 -- 172.21.15.111:0/1480107846 >> 172.21.15.111:0/1480107846 conn(0x7f8b7806b5b0 msgr2=0x7f8b7806b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 -- 172.21.15.111:0/1480107846 shutdown_connections 2023-12-19T07:48:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.978+0000 7f8b7e203700 1 -- 172.21.15.111:0/1480107846 wait complete. 2023-12-19T07:48:48.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 Processor -- start 2023-12-19T07:48:48.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 -- start start 2023-12-19T07:48:48.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64003ae0 0x7f8b78159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b7815eb90 0x7f8b780c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c60b0 con 0x7f8b780c6a80 2023-12-19T07:48:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c54e0 con 0x7f8b64003ae0 2023-12-19T07:48:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7e203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b780c5780 con 0x7f8b7815eb90 2023-12-19T07:48:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7da02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7da02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47656/0 (socket says 172.21.15.111:47656) 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7da02700 1 -- 172.21.15.111:0/2800691063 learned_addr learned my addr 172.21.15.111:0/2800691063 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7d201700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64003ae0 0x7f8b78159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.982+0000 7f8b7ca00700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b7815eb90 0x7f8b780c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 -- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b7815eb90 msgr2=0x7f8b780c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b7815eb90 0x7f8b780c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 -- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64003ae0 msgr2=0x7f8b78159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64003ae0 0x7f8b78159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 -- 172.21.15.111:0/2800691063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780c5a60 con 0x7f8b780c6a80 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7da02700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b700013d0 tx=0x7f8b70002140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000ac70 con 0x7f8b780c6a80 2023-12-19T07:48:48.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7000adf0 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70013640 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b780c5d40 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7815e6b0 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7815e8a0 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.986+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f8b700139b0 con 0x7f8b780c6a80 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.990+0000 7f8b6d7fa700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 0x7f8b5c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:48.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.990+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f8b7007d6b0 con 0x7f8b780c6a80 2023-12-19T07:48:48.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.990+0000 7f8b7d201700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 0x7f8b5c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:48.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.990+0000 7f8b7d201700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 0x7f8b5c062c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b740091e0 tx=0x7f8b7401a000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:48.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:48.994+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8b7815e8a0 con 0x7f8b780c6a80 2023-12-19T07:48:49.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.186+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b7815e8a0 con 0x7f8b5c060780 2023-12-19T07:48:49.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.290+0000 7f8b6d7fa700 1 -- 172.21.15.111:0/2800691063 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f8b7815e8a0 con 0x7f8b5c060780 2023-12-19T07:48:49.274 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-12-19T07:48:49.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.290+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 msgr2=0x7f8b5c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.290+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 0x7f8b5c062c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8b740091e0 tx=0x7f8b7401a000 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.290+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 msgr2=0x7f8b7815c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.290+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b700013d0 tx=0x7f8b70002140 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 shutdown_connections 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8b5c060780 0x7f8b5c062c30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b780c6a80 0x7f8b7815c960 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b7815eb90 0x7f8b780c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 --2- 172.21.15.111:0/2800691063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b64003ae0 0x7f8b78159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 >> 172.21.15.111:0/2800691063 conn(0x7f8b7806b5b0 msgr2=0x7f8b780ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 shutdown_connections 2023-12-19T07:48:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.294+0000 7f8b7e203700 1 -- 172.21.15.111:0/2800691063 wait complete. 2023-12-19T07:48:49.301 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-12-19T07:48:49.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T07:48:49.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.490+0000 7fe16546e700 1 Processor -- start 2023-12-19T07:48:49.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.490+0000 7fe16546e700 1 -- start start 2023-12-19T07:48:49.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe16546e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 0x7fe1600bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:49.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe16546e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600c7e70 con 0x7fe160156b90 2023-12-19T07:48:49.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe16546e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600c72a0 con 0x7fe1600c0300 2023-12-19T07:48:49.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe16546e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1600c7540 con 0x7fe160156f60 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe15effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe160156b90 0x7fe1600ae280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48240/0 (socket says 172.21.15.111:48240) 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe15effd700 1 -- 172.21.15.111:0/3686509048 learned_addr learned my addr 172.21.15.111:0/3686509048 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.494+0000 7fe15e7fc700 1 --2- 172.21.15.111:0/3686509048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 0x7fe1600bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112990997 0 0) 0x7fe1600c7e70 con 0x7fe160156b90 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003570 con 0x7fe160156b90 2023-12-19T07:48:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442650059 0 0) 0x7fe1600c72a0 con 0x7fe1600c0300 2023-12-19T07:48:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1600c7e70 con 0x7fe1600c0300 2023-12-19T07:48:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15e7fc700 1 -- 172.21.15.111:0/3686509048 >> v1:172.21.15.143:6789/0 conn(0x7fe1600c0300 legacy=0x7fe1600c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095857730 0 0) 0x7fe148003570 con 0x7fe160156b90 2023-12-19T07:48:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15e7fc700 1 -- 172.21.15.111:0/3686509048 >> v1:172.21.15.111:6789/0 conn(0x7fe160156b90 legacy=0x7fe1600ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15e7fc700 1 -- 172.21.15.111:0/3686509048 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600c7820 con 0x7fe160156f60 2023-12-19T07:48:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15e7fc700 1 --2- 172.21.15.111:0/3686509048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 0x7fe1600bfdc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe154009910 tx=0x7fe1540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df0cd95f8cf66b09 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150001470 con 0x7fe160156f60 2023-12-19T07:48:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe154011420 con 0x7fe160156f60 2023-12-19T07:48:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe15dffb700 1 -- 172.21.15.111:0/3686509048 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1540117c0 con 0x7fe160156f60 2023-12-19T07:48:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 -- 172.21.15.111:0/3686509048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 msgr2=0x7fe1600bfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 --2- 172.21.15.111:0/3686509048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 0x7fe1600bfdc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe154009910 tx=0x7fe1540091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 -- 172.21.15.111:0/3686509048 shutdown_connections 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 --2- 172.21.15.111:0/3686509048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156f60 0x7fe1600bfdc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 -- 172.21.15.111:0/3686509048 >> 172.21.15.111:0/3686509048 conn(0x7fe1600c8db0 msgr2=0x7fe1600c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 -- 172.21.15.111:0/3686509048 shutdown_connections 2023-12-19T07:48:49.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.498+0000 7fe16546e700 1 -- 172.21.15.111:0/3686509048 wait complete. 2023-12-19T07:48:49.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.502+0000 7fe16546e700 1 Processor -- start 2023-12-19T07:48:49.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.502+0000 7fe16546e700 1 -- start start 2023-12-19T07:48:49.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1600c0300 0x7fe1601537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:49.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156b90 0x7fe16006de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:49.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:49.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16006d6b0 con 0x7fe160156f60 2023-12-19T07:48:49.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16015c9a0 con 0x7fe1600c0300 2023-12-19T07:48:49.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16015bd90 con 0x7fe160156b90 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47660/0 (socket says 172.21.15.111:47660) 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 -- 172.21.15.111:0/2027444977 learned_addr learned my addr 172.21.15.111:0/2027444977 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15effd700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1600c0300 0x7fe1601537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15e7fc700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156b90 0x7fe16006de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 -- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156b90 msgr2=0x7fe16006de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156b90 0x7fe16006de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 -- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1600c0300 msgr2=0x7fe1601537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1600c0300 0x7fe1601537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 -- 172.21.15.111:0/2027444977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009420 con 0x7fe160156f60 2023-12-19T07:48:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe15f7fe700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c910 tx=0x7fe1580027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158012630 con 0x7fe160156f60 2023-12-19T07:48:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1580127b0 con 0x7fe160156f60 2023-12-19T07:48:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16015c070 con 0x7fe160156f60 2023-12-19T07:48:49.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158012b70 con 0x7fe160156f60 2023-12-19T07:48:49.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.506+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16015c630 con 0x7fe160156f60 2023-12-19T07:48:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.510+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe160070540 con 0x7fe160156f60 2023-12-19T07:48:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.510+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fe15801b400 con 0x7fe160156f60 2023-12-19T07:48:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.510+0000 7fe146ffd700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 0x7fe14c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.510+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fe15807d000 con 0x7fe160156f60 2023-12-19T07:48:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.514+0000 7fe15effd700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 0x7fe14c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.514+0000 7fe15effd700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 0x7fe14c062c30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe150004dd0 tx=0x7fe150001a20 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:49.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.518+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe15804d130 con 0x7fe160156f60 2023-12-19T07:48:49.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.710+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1600a9f90 con 0x7fe14c060780 2023-12-19T07:48:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.818+0000 7fe146ffd700 1 -- 172.21.15.111:0/2027444977 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe1600a9f90 con 0x7fe14c060780 2023-12-19T07:48:49.803 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-12-19T07:48:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 msgr2=0x7fe14c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 0x7fe14c062c30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe150004dd0 tx=0x7fe150001a20 comp rx=0 tx=0).stop 2023-12-19T07:48:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 msgr2=0x7fe1600c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe15800c910 tx=0x7fe1580027a0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 shutdown_connections 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe14c060780 0x7fe14c062c30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe160156f60 0x7fe1600c17b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe160156b90 0x7fe16006de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 --2- 172.21.15.111:0/2027444977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1600c0300 0x7fe1601537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 >> 172.21.15.111:0/2027444977 conn(0x7fe1600c8db0 msgr2=0x7fe160068380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 shutdown_connections 2023-12-19T07:48:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:49.822+0000 7fe16546e700 1 -- 172.21.15.111:0/2027444977 wait complete. 2023-12-19T07:48:49.829 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-12-19T07:48:49.829 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T07:48:50.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.026+0000 7fca4c8a5700 1 Processor -- start 2023-12-19T07:48:50.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.026+0000 7fca4c8a5700 1 -- start start 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4c8a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca480c8300 0x7fca480c8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4c8a5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480c7e70 con 0x7fca48156f60 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4c8a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480c72a0 con 0x7fca48156b90 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4c8a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480c8c50 con 0x7fca480c8300 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca45d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fca48156f60 0x7fca480bfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48244/0 (socket says 172.21.15.111:48244) 2023-12-19T07:48:50.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca45d9b700 1 -- 172.21.15.111:0/3163375010 learned_addr learned my addr 172.21.15.111:0/3163375010 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:50.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca46d9d700 1 --2- 172.21.15.111:0/3163375010 >> v2:172.21.15.111:3300/0 conn(0x7fca480c8300 0x7fca480c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121188458 0 0) 0x7fca480c7e70 con 0x7fca48156f60 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.030+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30003570 con 0x7fca48156f60 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107495034 0 0) 0x7fca480c72a0 con 0x7fca48156b90 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca480c7e70 con 0x7fca48156b90 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3606781938 0 0) 0x7fca30003570 con 0x7fca48156f60 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 >> v2:172.21.15.111:3300/0 conn(0x7fca480c8300 msgr2=0x7fca480c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 --2- 172.21.15.111:0/3163375010 >> v2:172.21.15.111:3300/0 conn(0x7fca480c8300 0x7fca480c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 >> v1:172.21.15.143:6789/0 conn(0x7fca48156b90 legacy=0x7fca480ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480c7540 con 0x7fca48156f60 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fca3c001470 con 0x7fca48156f60 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 >> v1:172.21.15.111:6789/0 conn(0x7fca48156f60 legacy=0x7fca480bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 0x7fca30004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30004810 0x7fca30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30005210 0x7fca3000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480c8c50 con 0x7fca30003c40 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3000c130 con 0x7fca30005210 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3000c2c0 con 0x7fca30004810 2023-12-19T07:48:50.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 0x7fca30004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca46d9d700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30005210 0x7fca3000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca45d9b700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30004810 0x7fca30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 -- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30004810 msgr2=0x7fca30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30004810 0x7fca30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 -- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30005210 msgr2=0x7fca3000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30005210 0x7fca3000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 -- 172.21.15.111:0/3163375010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30003570 con 0x7fca30003c40 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca46d9d700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30005210 0x7fca3000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca46d9d700 1 -- 172.21.15.111:0/3163375010 reap_dead start 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4659c700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 0x7fca30004030 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fca38001a40 tx=0x7fca38007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd428d1f038056c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3800c520 con 0x7fca30003c40 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3800c680 con 0x7fca30003c40 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4559a700 1 -- 172.21.15.111:0/3163375010 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3800ca20 con 0x7fca30003c40 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4c8a5700 1 -- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 msgr2=0x7fca30004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.034+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 0x7fca30004030 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fca38001a40 tx=0x7fca38007d30 comp rx=0 tx=0).stop 2023-12-19T07:48:50.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.038+0000 7fca4c8a5700 1 -- 172.21.15.111:0/3163375010 shutdown_connections 2023-12-19T07:48:50.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.038+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/3163375010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca30003c40 0x7fca30004030 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.038+0000 7fca4c8a5700 1 -- 172.21.15.111:0/3163375010 >> 172.21.15.111:0/3163375010 conn(0x7fca480c8db0 msgr2=0x7fca480c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:50.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.038+0000 7fca4c8a5700 1 -- 172.21.15.111:0/3163375010 shutdown_connections 2023-12-19T07:48:50.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.038+0000 7fca4c8a5700 1 -- 172.21.15.111:0/3163375010 wait complete. 2023-12-19T07:48:50.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 Processor -- start 2023-12-19T07:48:50.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- start start 2023-12-19T07:48:50.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30003c40 0x7fca4814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca480b07e0 0x7fca480b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48157dd0 con 0x7fca48159de0 2023-12-19T07:48:50.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480b1190 con 0x7fca480b07e0 2023-12-19T07:48:50.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4806cac0 con 0x7fca30003c40 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47684/0 (socket says 172.21.15.111:47684) 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 -- 172.21.15.111:0/784967989 learned_addr learned my addr 172.21.15.111:0/784967989 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca46d9d700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca480b07e0 0x7fca480b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4659c700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30003c40 0x7fca4814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 -- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30003c40 msgr2=0x7fca4814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30003c40 0x7fca4814c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 -- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca480b07e0 msgr2=0x7fca480b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca480b07e0 0x7fca480b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 -- 172.21.15.111:0/784967989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4814cc70 con 0x7fca48159de0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca45d9b700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fca3c001050 tx=0x7fca480c7de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c010520 con 0x7fca48159de0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c010680 con 0x7fca48159de0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48157fb0 con 0x7fca48159de0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c010a40 con 0x7fca48159de0 2023-12-19T07:48:50.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.042+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca48158570 con 0x7fca48159de0 2023-12-19T07:48:50.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.046+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48071120 con 0x7fca48159de0 2023-12-19T07:48:50.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.046+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fca3c0133e0 con 0x7fca48159de0 2023-12-19T07:48:50.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.050+0000 7fca367fc700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 0x7fca28062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.050+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fca3c07c7e0 con 0x7fca48159de0 2023-12-19T07:48:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.050+0000 7fca4659c700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 0x7fca28062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.050+0000 7fca4659c700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 0x7fca28062c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca38001a40 tx=0x7fca38015000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.054+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca3c04c870 con 0x7fca48159de0 2023-12-19T07:48:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.246+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca4806cc00 con 0x7fca28060750 2023-12-19T07:48:50.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.342+0000 7fca367fc700 1 -- 172.21.15.111:0/784967989 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fca4806cc00 con 0x7fca28060750 2023-12-19T07:48:50.327 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-12-19T07:48:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 msgr2=0x7fca28062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 0x7fca28062c00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca38001a40 tx=0x7fca38015000 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 msgr2=0x7fca4815a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fca3c001050 tx=0x7fca480c7de0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 shutdown_connections 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca28060750 0x7fca28062c00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca480b07e0 0x7fca480b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca48159de0 0x7fca4815a250 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 --2- 172.21.15.111:0/784967989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca30003c40 0x7fca4814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 >> 172.21.15.111:0/784967989 conn(0x7fca480c8db0 msgr2=0x7fca480b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 shutdown_connections 2023-12-19T07:48:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.346+0000 7fca4c8a5700 1 -- 172.21.15.111:0/784967989 wait complete. 2023-12-19T07:48:50.352 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-12-19T07:48:50.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T07:48:50.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 Processor -- start 2023-12-19T07:48:50.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 -- start start 2023-12-19T07:48:50.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f712c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f712c0ac650 0x7f712c0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712c06d7b0 con 0x7f712c06b060 2023-12-19T07:48:50.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712c06e180 con 0x7f712c0ac650 2023-12-19T07:48:50.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7134585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712c0ae570 con 0x7f712c156e70 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7131b20700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f712c06b060 0x7f712c0abf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48246/0 (socket says 172.21.15.111:48246) 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7131b20700 1 -- 172.21.15.111:0/3193554405 learned_addr learned my addr 172.21.15.111:0/3193554405 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f712c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7132b22700 1 --2- 172.21.15.111:0/3193554405 >> v2:172.21.15.143:3300/0 conn(0x7f712c0ac650 0x7f712c0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496044154 0 0) 0x7f712c06d7b0 con 0x7f712c06b060 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7118003570 con 0x7f712c06b060 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3755523932 0 0) 0x7f7118003570 con 0x7f712c06b060 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 msgr2=0x7f712c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f712c06dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 >> v2:172.21.15.143:3300/0 conn(0x7f712c0ac650 msgr2=0x7f712c0aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 --2- 172.21.15.111:0/3193554405 >> v2:172.21.15.143:3300/0 conn(0x7f712c0ac650 0x7f712c0aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c06cba0 con 0x7f712c06b060 2023-12-19T07:48:50.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f711c001470 con 0x7f712c06b060 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f712c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.538+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 >> v1:172.21.15.111:6789/0 conn(0x7f712c06b060 legacy=0x7f712c0abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f7118003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7118004810 0x7f7118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f71180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c06e180 con 0x7f7118003a60 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c0ae570 con 0x7f7118004810 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f711800ba60 con 0x7f712c156e70 2023-12-19T07:48:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f7118003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7131b20700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7118004810 0x7f7118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132b22700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f71180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 -- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 msgr2=0x7f71180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c156e70 0x7f71180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 -- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7118004810 msgr2=0x7f7118004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7118004810 0x7f7118004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 -- 172.21.15.111:0/3193554405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128009420 con 0x7f7118003a60 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132321700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f7118003e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7128009df0 tx=0x7f71280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b071f26f525c46 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128009740 con 0x7f7118003a60 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712800e920 con 0x7f7118003a60 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f713131f700 1 -- 172.21.15.111:0/3193554405 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712800ece0 con 0x7f7118003a60 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 -- 172.21.15.111:0/3193554405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 msgr2=0x7f7118003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 --2- 172.21.15.111:0/3193554405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f7118003e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7128009df0 tx=0x7f71280091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7132b22700 1 -- 172.21.15.111:0/3193554405 reap_dead start 2023-12-19T07:48:50.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 -- 172.21.15.111:0/3193554405 shutdown_connections 2023-12-19T07:48:50.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 -- 172.21.15.111:0/3193554405 >> 172.21.15.111:0/3193554405 conn(0x7f712c067bb0 msgr2=0x7f712c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:50.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 -- 172.21.15.111:0/3193554405 shutdown_connections 2023-12-19T07:48:50.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.542+0000 7f7134585700 1 -- 172.21.15.111:0/3193554405 wait complete. 2023-12-19T07:48:50.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 Processor -- start 2023-12-19T07:48:50.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 -- start start 2023-12-19T07:48:50.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 0x7f712c150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f712c0cb7e0 0x7f712c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c1669c0 con 0x7f7118003a60 2023-12-19T07:48:50.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c16aec0 con 0x7f712c0cb7e0 2023-12-19T07:48:50.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7134585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712c15cb60 con 0x7f712c167670 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.546+0000 7f7132321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47706/0 (socket says 172.21.15.111:47706) 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 -- 172.21.15.111:0/3920734100 learned_addr learned my addr 172.21.15.111:0/3920734100 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132b22700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f712c0cb7e0 0x7f712c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7131b20700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 0x7f712c150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 -- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 msgr2=0x7f712c150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 0x7f712c150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 -- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f712c0cb7e0 msgr2=0x7f712c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f712c0cb7e0 0x7f712c0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 -- 172.21.15.111:0/3920734100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c1671e0 con 0x7f7118003a60 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7131b20700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 0x7f712c150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:48:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7132321700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f71280091e0 tx=0x7f712801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128022070 con 0x7f7118003a60 2023-12-19T07:48:50.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7128013ab0 con 0x7f7118003a60 2023-12-19T07:48:50.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c0cc170 con 0x7f7118003a60 2023-12-19T07:48:50.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128013e70 con 0x7f7118003a60 2023-12-19T07:48:50.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c15c880 con 0x7f7118003a60 2023-12-19T07:48:50.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.550+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c071120 con 0x7f7118003a60 2023-12-19T07:48:50.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.554+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f7128026070 con 0x7f7118003a60 2023-12-19T07:48:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.554+0000 7f71227fc700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 0x7f7110062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.554+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f712807e710 con 0x7f7118003a60 2023-12-19T07:48:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.554+0000 7f7131b20700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 0x7f7110062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:50.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.554+0000 7f7131b20700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 0x7f7110062c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f711c001200 tx=0x7f711c009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:50.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.558+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7128052230 con 0x7f7118003a60 2023-12-19T07:48:50.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.754+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f712c06ce80 con 0x7f7110060750 2023-12-19T07:48:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.854+0000 7f71227fc700 1 -- 172.21.15.111:0/3920734100 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f712c06ce80 con 0x7f7110060750 2023-12-19T07:48:50.841 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 msgr2=0x7f7110062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 0x7f7110062c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f711c001200 tx=0x7f711c009040 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 msgr2=0x7f712c16a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f71280091e0 tx=0x7f712801a000 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 shutdown_connections 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7110060750 0x7f7110062c00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f712c0cb7e0 0x7f712c0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f712c167670 0x7f712c150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 --2- 172.21.15.111:0/3920734100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7118003a60 0x7f712c16a980 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 >> 172.21.15.111:0/3920734100 conn(0x7f712c067bb0 msgr2=0x7f712c157240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 shutdown_connections 2023-12-19T07:48:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:50.862+0000 7f7134585700 1 -- 172.21.15.111:0/3920734100 wait complete. 2023-12-19T07:48:50.866 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-12-19T07:48:50.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T07:48:51.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.066+0000 7fec75a8e700 1 Processor -- start 2023-12-19T07:48:51.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.066+0000 7fec75a8e700 1 -- start start 2023-12-19T07:48:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 0x7fec700af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7006d6b0 0x7fec7006db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec700cc140 con 0x7fec7006d6b0 2023-12-19T07:48:51.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec700cb3f0 con 0x7fec700aedc0 2023-12-19T07:48:51.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec75a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7006d220 con 0x7fec7006af50 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7006d6b0 0x7fec7006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7006d6b0 0x7fec7006db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47714/0 (socket says 172.21.15.111:47714) 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 -- 172.21.15.111:0/4019271822 learned_addr learned my addr 172.21.15.111:0/4019271822 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec6ffff700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec74a8c700 1 --2- 172.21.15.111:0/4019271822 >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 0x7fec700af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 -- 172.21.15.111:0/4019271822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 msgr2=0x7fec700cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 -- 172.21.15.111:0/4019271822 >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 msgr2=0x7fec700af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 --2- 172.21.15.111:0/4019271822 >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 0x7fec700af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 -- 172.21.15.111:0/4019271822 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700cb190 con 0x7fec7006d6b0 2023-12-19T07:48:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec74a8c700 1 --2- 172.21.15.111:0/4019271822 >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 0x7fec700af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:48:51.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.070+0000 7fec7528d700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7006db30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fec700cc6c0 tx=0x7fec68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a55d87f8a9c3d35 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec6f7fe700 1 -- 172.21.15.111:0/4019271822 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800e070 con 0x7fec7006d6b0 2023-12-19T07:48:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec6f7fe700 1 -- 172.21.15.111:0/4019271822 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6800ba50 con 0x7fec7006d6b0 2023-12-19T07:48:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec6f7fe700 1 -- 172.21.15.111:0/4019271822 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680092e0 con 0x7fec7006d6b0 2023-12-19T07:48:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 -- 172.21.15.111:0/4019271822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 msgr2=0x7fec7006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7006db30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fec700cc6c0 tx=0x7fec68009840 comp rx=0 tx=0).stop 2023-12-19T07:48:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 -- 172.21.15.111:0/4019271822 shutdown_connections 2023-12-19T07:48:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7006db30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 --2- 172.21.15.111:0/4019271822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 --2- 172.21.15.111:0/4019271822 >> v2:172.21.15.143:3300/0 conn(0x7fec700aedc0 0x7fec700af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 -- 172.21.15.111:0/4019271822 >> 172.21.15.111:0/4019271822 conn(0x7fec700cc760 msgr2=0x7fec700ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 -- 172.21.15.111:0/4019271822 shutdown_connections 2023-12-19T07:48:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.074+0000 7fec75a8e700 1 -- 172.21.15.111:0/4019271822 wait complete. 2023-12-19T07:48:51.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 Processor -- start 2023-12-19T07:48:51.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 -- start start 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec700aedc0 0x7fec700c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7016cef0 con 0x7fec7006d6b0 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec700b2aa0 con 0x7fec700aedc0 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec75a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700b2d80 con 0x7fec7006af50 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47728/0 (socket says 172.21.15.111:47728) 2023-12-19T07:48:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec6ffff700 1 -- 172.21.15.111:0/1541138230 learned_addr learned my addr 172.21.15.111:0/1541138230 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec74a8c700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.078+0000 7fec7528d700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec700aedc0 0x7fec700c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 -- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 msgr2=0x7fec700c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 -- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec700aedc0 msgr2=0x7fec700c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec700aedc0 0x7fec700c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 -- 172.21.15.111:0/1541138230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec68009ae0 con 0x7fec7006d6b0 2023-12-19T07:48:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6ffff700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00d860 tx=0x7fec5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0125f0 con 0x7fec7006d6b0 2023-12-19T07:48:51.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c012770 con 0x7fec7006d6b0 2023-12-19T07:48:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c012b30 con 0x7fec7006d6b0 2023-12-19T07:48:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec700b3060 con 0x7fec7006d6b0 2023-12-19T07:48:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.082+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec700b3620 con 0x7fec7006d6b0 2023-12-19T07:48:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.086+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec700b3810 con 0x7fec7006d6b0 2023-12-19T07:48:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.086+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fec5c013480 con 0x7fec7006d6b0 2023-12-19T07:48:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.086+0000 7fec6cff9700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 0x7fec54062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.086+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fec5c07d0d0 con 0x7fec7006d6b0 2023-12-19T07:48:51.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.086+0000 7fec74a8c700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 0x7fec54062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.090+0000 7fec74a8c700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 0x7fec54062c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec6400adc0 tx=0x7fec64006f90 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.094+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec5c04d200 con 0x7fec7006d6b0 2023-12-19T07:48:51.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.294+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fec7004be40 con 0x7fec54060750 2023-12-19T07:48:51.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.398+0000 7fec6cff9700 1 -- 172.21.15.111:0/1541138230 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fec7004be40 con 0x7fec54060750 2023-12-19T07:48:51.383 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 msgr2=0x7fec54062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 0x7fec54062c00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec6400adc0 tx=0x7fec64006f90 comp rx=0 tx=0).stop 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 msgr2=0x7fec7016c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00d860 tx=0x7fec5c008040 comp rx=0 tx=0).stop 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 shutdown_connections 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec54060750 0x7fec54062c00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec700aedc0 0x7fec700c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7006d6b0 0x7fec7016c2c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 --2- 172.21.15.111:0/1541138230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec7006af50 0x7fec700c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 >> 172.21.15.111:0/1541138230 conn(0x7fec700cc760 msgr2=0x7fec700cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:51.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 shutdown_connections 2023-12-19T07:48:51.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.402+0000 7fec75a8e700 1 -- 172.21.15.111:0/1541138230 wait complete. 2023-12-19T07:48:51.411 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-12-19T07:48:51.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T07:48:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.606+0000 7f37f759e700 1 Processor -- start 2023-12-19T07:48:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.606+0000 7f37f759e700 1 -- start start 2023-12-19T07:48:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f37f80aedc0 0x7f37f80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f806b3d0 con 0x7f37f80cac50 2023-12-19T07:48:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80cb3f0 con 0x7f37f806af50 2023-12-19T07:48:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f80cc140 con 0x7f37f80aedc0 2023-12-19T07:48:51.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f6d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f37f80cac50 0x7f37f80af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48252/0 (socket says 172.21.15.111:48252) 2023-12-19T07:48:51.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f6d9d700 1 -- 172.21.15.111:0/2610077101 learned_addr learned my addr 172.21.15.111:0/2610077101 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:51.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f659c700 1 --2- 172.21.15.111:0/2610077101 >> v2:172.21.15.111:3300/0 conn(0x7f37f80aedc0 0x7f37f80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164549595 0 0) 0x7f37f0001490 con 0x7f37f80cac50 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e0003570 con 0x7f37f80cac50 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922356652 0 0) 0x7f37f80cb3f0 con 0x7f37f806af50 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f659c700 1 -- 172.21.15.111:0/2610077101 >> v1:172.21.15.143:6789/0 conn(0x7f37f806af50 legacy=0x7f37f806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f659c700 1 -- 172.21.15.111:0/2610077101 >> v1:172.21.15.111:6789/0 conn(0x7f37f80cac50 legacy=0x7f37f80af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f659c700 1 -- 172.21.15.111:0/2610077101 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f806d220 con 0x7f37f80aedc0 2023-12-19T07:48:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f659c700 1 --2- 172.21.15.111:0/2610077101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80aedc0 0x7f37f80af190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f37e80098e0 tx=0x7f37e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a164de7f48738a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e800b640 con 0x7f37f80aedc0 2023-12-19T07:48:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e800b7a0 con 0x7f37f80aedc0 2023-12-19T07:48:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.610+0000 7f37f559a700 1 -- 172.21.15.111:0/2610077101 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e800bb40 con 0x7f37f80aedc0 2023-12-19T07:48:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- 172.21.15.111:0/2610077101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80aedc0 msgr2=0x7f37f80af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 --2- 172.21.15.111:0/2610077101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80aedc0 0x7f37f80af190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f37e80098e0 tx=0x7f37e80091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- 172.21.15.111:0/2610077101 shutdown_connections 2023-12-19T07:48:51.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 --2- 172.21.15.111:0/2610077101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80aedc0 0x7f37f80af190 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- 172.21.15.111:0/2610077101 >> 172.21.15.111:0/2610077101 conn(0x7f37f80cc760 msgr2=0x7f37f80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:51.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- 172.21.15.111:0/2610077101 shutdown_connections 2023-12-19T07:48:51.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- 172.21.15.111:0/2610077101 wait complete. 2023-12-19T07:48:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 Processor -- start 2023-12-19T07:48:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.614+0000 7f37f759e700 1 -- start start 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37f806af50 0x7f37f814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37f80aedc0 0x7f37f80b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80c5010 con 0x7f37f80cac50 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80b1b60 con 0x7f37f80aedc0 2023-12-19T07:48:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80c46e0 con 0x7f37f806af50 2023-12-19T07:48:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47754/0 (socket says 172.21.15.111:47754) 2023-12-19T07:48:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 -- 172.21.15.111:0/1166780997 learned_addr learned my addr 172.21.15.111:0/1166780997 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f659c700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37f806af50 0x7f37f814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f5d9b700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37f80aedc0 0x7f37f80b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 -- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37f806af50 msgr2=0x7f37f814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37f806af50 0x7f37f814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 -- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37f80aedc0 msgr2=0x7f37f80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37f80aedc0 0x7f37f80b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 -- 172.21.15.111:0/1166780997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8009420 con 0x7f37f80cac50 2023-12-19T07:48:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f6d9d700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f37f000cd00 tx=0x7f37f0001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f000ce00 con 0x7f37f80cac50 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f000aac0 con 0x7f37f80cac50 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f000ae40 con 0x7f37f80cac50 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f80c49c0 con 0x7f37f80cac50 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.618+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f80b2cd0 con 0x7f37f80cac50 2023-12-19T07:48:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.622+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f804be40 con 0x7f37f80cac50 2023-12-19T07:48:51.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.622+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f37f0007300 con 0x7f37f80cac50 2023-12-19T07:48:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.622+0000 7f37e67fc700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 0x7f37d8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.626+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7f37f007cfe0 con 0x7f37f80cac50 2023-12-19T07:48:51.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.626+0000 7f37f659c700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 0x7f37d8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:51.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.626+0000 7f37f659c700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 0x7f37d8062c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37e80093f0 tx=0x7f37e8017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:51.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.630+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f37f004d0c0 con 0x7f37f80cac50 2023-12-19T07:48:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.818+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f806b3d0 con 0x7f37d8060750 2023-12-19T07:48:51.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.918+0000 7f37e67fc700 1 -- 172.21.15.111:0/1166780997 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f37f806b3d0 con 0x7f37d8060750 2023-12-19T07:48:51.905 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 msgr2=0x7f37d8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 0x7f37d8062c00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37e80093f0 tx=0x7f37e8017000 comp rx=0 tx=0).stop 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 msgr2=0x7f37f806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f37f000cd00 tx=0x7f37f0001ad0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 shutdown_connections 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37d8060750 0x7f37d8062c00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37f80aedc0 0x7f37f80b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37f80cac50 0x7f37f806d6b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 --2- 172.21.15.111:0/1166780997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37f806af50 0x7f37f814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:51.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 >> 172.21.15.111:0/1166780997 conn(0x7f37f80cc760 msgr2=0x7f37f80cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:51.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 shutdown_connections 2023-12-19T07:48:51.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:51.922+0000 7f37f759e700 1 -- 172.21.15.111:0/1166780997 wait complete. 2023-12-19T07:48:51.932 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-12-19T07:48:51.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T07:48:52.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.118+0000 7fd815bc3700 1 Processor -- start 2023-12-19T07:48:52.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.118+0000 7fd815bc3700 1 -- start start 2023-12-19T07:48:52.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd815bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 0x7fd81006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd815bc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 0x7fd81006ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd815bc3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81015d720 con 0x7fd810158da0 2023-12-19T07:48:52.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd815bc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8100ca290 con 0x7fd81006df10 2023-12-19T07:48:52.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd815bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100ca4a0 con 0x7fd81006ed80 2023-12-19T07:48:52.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 0x7fd81006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd814bc1700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd810158da0 0x7fd81006e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36232/0 (socket says 172.21.15.111:36232) 2023-12-19T07:48:52.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd814bc1700 1 -- 172.21.15.111:0/2535921343 learned_addr learned my addr 172.21.15.111:0/2535921343 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:48:52.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd8153c2700 1 --2- 172.21.15.111:0/2535921343 >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 0x7fd81006ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80f7fe700 1 -- 172.21.15.111:0/2535921343 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878566023 0 0) 0x7fd81015d720 con 0x7fd810158da0 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80f7fe700 1 -- 172.21.15.111:0/2535921343 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd800003570 con 0x7fd810158da0 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 -- 172.21.15.111:0/2535921343 >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 msgr2=0x7fd81006ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 --2- 172.21.15.111:0/2535921343 >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 0x7fd81006ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 -- 172.21.15.111:0/2535921343 >> v1:172.21.15.143:6789/0 conn(0x7fd810158da0 legacy=0x7fd81006e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 -- 172.21.15.111:0/2535921343 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8100ca780 con 0x7fd81006ed80 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd8153c2700 1 --2- 172.21.15.111:0/2535921343 >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 0x7fd81006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:48:52.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80ffff700 1 --2- 172.21.15.111:0/2535921343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 0x7fd81006a7d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0098e0 tx=0x7fd7fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138367a8e207c785 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:52.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.122+0000 7fd80f7fe700 1 -- 172.21.15.111:0/2535921343 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011630 con 0x7fd81006ed80 2023-12-19T07:48:52.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd80f7fe700 1 -- 172.21.15.111:0/2535921343 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc011790 con 0x7fd81006ed80 2023-12-19T07:48:52.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd80f7fe700 1 -- 172.21.15.111:0/2535921343 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011b30 con 0x7fd81006ed80 2023-12-19T07:48:52.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 -- 172.21.15.111:0/2535921343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 msgr2=0x7fd81006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 --2- 172.21.15.111:0/2535921343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 0x7fd81006a7d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0098e0 tx=0x7fd7fc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 -- 172.21.15.111:0/2535921343 shutdown_connections 2023-12-19T07:48:52.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 --2- 172.21.15.111:0/2535921343 >> v2:172.21.15.143:3300/0 conn(0x7fd81006df10 0x7fd81006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 --2- 172.21.15.111:0/2535921343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006ed80 0x7fd81006a7d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 -- 172.21.15.111:0/2535921343 >> 172.21.15.111:0/2535921343 conn(0x7fd8100cc760 msgr2=0x7fd8100ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:52.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 -- 172.21.15.111:0/2535921343 shutdown_connections 2023-12-19T07:48:52.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 -- 172.21.15.111:0/2535921343 wait complete. 2023-12-19T07:48:52.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.126+0000 7fd815bc3700 1 Processor -- start 2023-12-19T07:48:52.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 -- start start 2023-12-19T07:48:52.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 0x7fd8100b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 0x7fd8101505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 0x7fd8100b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100c9520 con 0x7fd810158da0 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810157540 con 0x7fd81006ed80 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd815bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd810157730 con 0x7fd81006df10 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd814bc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 0x7fd8100b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd8153c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 0x7fd8100b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 0x7fd8101505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd8153c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 0x7fd8100b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44812/0 (socket says 172.21.15.111:44812) 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd814bc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 0x7fd8100b56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:34982/0 (socket says 172.21.15.111:34982) 2023-12-19T07:48:52.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd8153c2700 1 -- 172.21.15.111:0/369282166 learned_addr learned my addr 172.21.15.111:0/369282166 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 -- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 msgr2=0x7fd8100b56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 0x7fd8100b56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 -- 172.21.15.111:0/369282166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 msgr2=0x7fd8100b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 0x7fd8100b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 -- 172.21.15.111:0/369282166 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808009420 con 0x7fd81006ed80 2023-12-19T07:48:52.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.130+0000 7fd80ffff700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 0x7fd8101505a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd81006a310 tx=0x7fd7fc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:52.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011900 con 0x7fd81006ed80 2023-12-19T07:48:52.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc011a60 con 0x7fd81006ed80 2023-12-19T07:48:52.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc025620 con 0x7fd81006ed80 2023-12-19T07:48:52.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc009420 con 0x7fd81006ed80 2023-12-19T07:48:52.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd810157cf0 con 0x7fd81006ed80 2023-12-19T07:48:52.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81004be40 con 0x7fd81006ed80 2023-12-19T07:48:52.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.134+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fd7fc025a10 con 0x7fd81006ed80 2023-12-19T07:48:52.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.138+0000 7fd80cff9700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 0x7fd7f4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:48:52.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.138+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6283+0+0 (secure 0 0 0) 0x7fd7fc07e480 con 0x7fd81006ed80 2023-12-19T07:48:52.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.138+0000 7fd814bc1700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 0x7fd7f4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:48:52.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.138+0000 7fd814bc1700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 0x7fd7f4062c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd804005c50 tx=0x7fd804009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:48:52.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.150+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd7fc04e510 con 0x7fd81006ed80 2023-12-19T07:48:52.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.338+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81006f360 con 0x7fd7f4060750 2023-12-19T07:48:52.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.442+0000 7fd80cff9700 1 -- 172.21.15.111:0/369282166 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd81006f360 con 0x7fd7f4060750 2023-12-19T07:48:52.427 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 msgr2=0x7fd7f4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 0x7fd7f4062c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd804005c50 tx=0x7fd804009040 comp rx=0 tx=0).stop 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 msgr2=0x7fd8101505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 0x7fd8101505a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd81006a310 tx=0x7fd7fc017000 comp rx=0 tx=0).stop 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 shutdown_connections 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd7f4060750 0x7fd7f4062c00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd81006ed80 0x7fd8101505a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd810158da0 0x7fd8100b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 --2- 172.21.15.111:0/369282166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd81006df10 0x7fd8100b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:48:52.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 >> 172.21.15.111:0/369282166 conn(0x7fd8100cc760 msgr2=0x7fd8100ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:48:52.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 shutdown_connections 2023-12-19T07:48:52.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:48:52.450+0000 7fd815bc3700 1 -- 172.21.15.111:0/369282166 wait complete. 2023-12-19T07:48:52.455 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-12-19T07:49:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:49:00.262+0000 7fd5a38ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:49:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:49:00.262+0000 7fd5a38ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:49:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:49:00.262+0000 7fd5a38ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:49:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:49:00.262+0000 7fd5a38ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:49:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:49:00.262+0000 7fd5a38ae700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:49:03.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:49:03.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:49:33.736 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:49:33.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:50:03.834 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:50:03.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:50:33.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:50:33.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:51:04.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:51:04.051 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:51:34.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:51:34.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:52:04.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:52:04.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:52:34.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:52:34.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:53:04.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:53:04.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:53:34.453 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:53:34.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:54:00.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:54:00.451+0000 7fd59a09b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:54:00.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:54:00.451+0000 7fd59a09b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:54:00.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:54:00.451+0000 7fd59a09b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:54:00.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:54:00.451+0000 7fd59a09b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:54:00.438 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:54:00.451+0000 7fd59a09b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:54:04.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:54:04.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:54:34.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:54:34.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:55:04.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:55:04.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:55:34.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:55:34.822 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:56:04.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:56:04.905 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:56:34.979 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:56:34.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:57:05.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:57:05.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:57:35.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:57:35.173 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:58:05.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:58:05.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:58:35.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:58:35.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:59:05.410 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:59:05.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:59:20.456 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T07:59:20.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.625+0000 7f02290e3700 1 Processor -- start 2023-12-19T07:59:20.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 -- start start 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 0x7f022406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f022415d360 0x7f02240ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022406b550 con 0x7f022406dde0 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022406af40 con 0x7f022415d360 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.629+0000 7f02290e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240af5a0 con 0x7f022415cf90 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f022415d360 0x7f02240ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f022415d360 0x7f02240ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46952/0 (socket says 172.21.15.111:46952) 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 -- 172.21.15.111:0/1480111019 learned_addr learned my addr 172.21.15.111:0/1480111019 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:20.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02288e2700 1 --1- 172.21.15.111:0/1480111019 >> v1:172.21.15.111:6789/0 conn(0x7f022406dde0 0x7f02240aee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33916/0 (socket says 172.21.15.111:33916) 2023-12-19T07:59:20.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0223fff700 1 --2- 172.21.15.111:0/1480111019 >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 0x7f022406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 -- 172.21.15.111:0/1480111019 >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 msgr2=0x7f022406b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:20.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 --2- 172.21.15.111:0/1480111019 >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 0x7f022406b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 -- 172.21.15.111:0/1480111019 >> v1:172.21.15.111:6789/0 conn(0x7f022406dde0 legacy=0x7f02240aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0222ffd700 1 -- 172.21.15.111:0/1480111019 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222740358 0 0) 0x7f022406b550 con 0x7f022406dde0 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 -- 172.21.15.111:0/1480111019 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02240ae0c0 con 0x7f022415d360 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0223fff700 1 --2- 172.21.15.111:0/1480111019 >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 0x7f022406b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02237fe700 1 --2- 172.21.15.111:0/1480111019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415d360 0x7f02240ae950 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0224069830 tx=0x7f0218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0ae940ae85284a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0222ffd700 1 -- 172.21.15.111:0/1480111019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021800b860 con 0x7f022415d360 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0222ffd700 1 -- 172.21.15.111:0/1480111019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021800b9c0 con 0x7f022415d360 2023-12-19T07:59:20.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f0222ffd700 1 -- 172.21.15.111:0/1480111019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021800bd60 con 0x7f022415d360 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 -- 172.21.15.111:0/1480111019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415d360 msgr2=0x7f02240ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 --2- 172.21.15.111:0/1480111019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415d360 0x7f02240ae950 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0224069830 tx=0x7f0218009250 comp rx=0 tx=0).stop 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 -- 172.21.15.111:0/1480111019 shutdown_connections 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 --2- 172.21.15.111:0/1480111019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415d360 0x7f02240ae950 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 --2- 172.21.15.111:0/1480111019 >> v2:172.21.15.143:3300/0 conn(0x7f022415cf90 0x7f022406b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 -- 172.21.15.111:0/1480111019 >> 172.21.15.111:0/1480111019 conn(0x7f0224067bb0 msgr2=0x7f02240c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 -- 172.21.15.111:0/1480111019 shutdown_connections 2023-12-19T07:59:20.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.633+0000 7f02290e3700 1 -- 172.21.15.111:0/1480111019 wait complete. 2023-12-19T07:59:20.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.637+0000 7f02290e3700 1 Processor -- start 2023-12-19T07:59:20.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.637+0000 7f02290e3700 1 -- start start 2023-12-19T07:59:20.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f022406dde0 0x7f022416d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f022415d360 0x7f022415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022416e5e0 con 0x7f022415cf90 2023-12-19T07:59:20.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022416dd40 con 0x7f022415d360 2023-12-19T07:59:20.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022416e020 con 0x7f022406dde0 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46962/0 (socket says 172.21.15.111:46962) 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 -- 172.21.15.111:0/1326902963 learned_addr learned my addr 172.21.15.111:0/1326902963 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02288e2700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f022415d360 0x7f022415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f0223fff700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f022406dde0 0x7f022416d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 -- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f022406dde0 msgr2=0x7f022416d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f022406dde0 0x7f022416d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 -- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f022415d360 msgr2=0x7f022415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f022415d360 0x7f022415b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 -- 172.21.15.111:0/1326902963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180094f0 con 0x7f022415cf90 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02237fe700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f02180090b0 tx=0x7f0218017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218024440 con 0x7f022415cf90 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022416e300 con 0x7f022415cf90 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02180245a0 con 0x7f022415cf90 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218024960 con 0x7f022415cf90 2023-12-19T07:59:20.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.641+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022415b730 con 0x7f022415cf90 2023-12-19T07:59:20.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.645+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0224158b80 con 0x7f022415cf90 2023-12-19T07:59:20.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.645+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f021801a420 con 0x7f022415cf90 2023-12-19T07:59:20.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.649+0000 7f020bfff700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 0x7f0210062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:20.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.649+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f021807ed00 con 0x7f022415cf90 2023-12-19T07:59:20.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.649+0000 7f0223fff700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 0x7f0210062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:20.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.649+0000 7f0223fff700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 0x7f0210062c00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0214006f00 tx=0x7f0214008040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:20.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.657+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f021804ee10 con 0x7f022415cf90 2023-12-19T07:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.853+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f022415b920 con 0x7f0210060750 2023-12-19T07:59:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.985+0000 7f020bfff700 1 -- 172.21.15.111:0/1326902963 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f022415b920 con 0x7f0210060750 2023-12-19T07:59:20.983 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 msgr2=0x7f0210062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 0x7f0210062c00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0214006f00 tx=0x7f0214008040 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 msgr2=0x7f0224158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f02180090b0 tx=0x7f0218017000 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 shutdown_connections 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0210060750 0x7f0210062c00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f022415d360 0x7f022415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f022415cf90 0x7f0224158490 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 --2- 172.21.15.111:0/1326902963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f022406dde0 0x7f022416d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:20.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 >> 172.21.15.111:0/1326902963 conn(0x7f0224067bb0 msgr2=0x7f02241577d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:20.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 shutdown_connections 2023-12-19T07:59:20.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:20.989+0000 7f02290e3700 1 -- 172.21.15.111:0/1326902963 wait complete. 2023-12-19T07:59:21.006 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-12-19T07:59:21.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T07:59:21.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.181+0000 7f6c6b59e700 1 Processor -- start 2023-12-19T07:59:21.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 -- start start 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6c6c1568e0 0x7f6c6c0c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 0x7f6c6c0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0afa50 con 0x7f6c6c156cb0 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c92a0 con 0x7f6c6c1568e0 2023-12-19T07:59:21.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.185+0000 7f6c6b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0af280 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46964/0 (socket says 172.21.15.111:46964) 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6ad9d700 1 -- 172.21.15.111:0/707346905 learned_addr learned my addr 172.21.15.111:0/707346905 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/707346905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 0x7f6c6c0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6a59c700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.143:3300/0 conn(0x7f6c6c1568e0 0x7f6c6c0c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 -- 172.21.15.111:0/707346905 >> v2:172.21.15.143:3300/0 conn(0x7f6c6c1568e0 msgr2=0x7f6c6c0c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.143:3300/0 conn(0x7f6c6c1568e0 0x7f6c6c0c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 -- 172.21.15.111:0/707346905 >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 msgr2=0x7f6c6c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 -- 172.21.15.111:0/707346905 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c0c8a00 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6ad9d700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/707346905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 0x7f6c6c0ae800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c06ad00 tx=0x7f6c600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b826abcf851ba290 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6959a700 1 -- 172.21.15.111:0/707346905 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60011630 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6959a700 1 -- 172.21.15.111:0/707346905 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c600117b0 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6959a700 1 -- 172.21.15.111:0/707346905 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60011b70 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6b59e700 1 -- 172.21.15.111:0/707346905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 msgr2=0x7f6c6c0ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.189+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/707346905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 0x7f6c6c0ae800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c06ad00 tx=0x7f6c600091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- 172.21.15.111:0/707346905 shutdown_connections 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.111:3300/0 conn(0x7f6c6c156cb0 0x7f6c6c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/707346905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c0c84a0 0x7f6c6c0ae800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/707346905 >> v2:172.21.15.143:3300/0 conn(0x7f6c6c1568e0 0x7f6c6c0c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- 172.21.15.111:0/707346905 >> 172.21.15.111:0/707346905 conn(0x7f6c6c06b5b0 msgr2=0x7f6c6c06b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- 172.21.15.111:0/707346905 shutdown_connections 2023-12-19T07:59:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- 172.21.15.111:0/707346905 wait complete. 2023-12-19T07:59:21.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 Processor -- start 2023-12-19T07:59:21.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- start start 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c1568e0 0x7f6c6c16a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c6c156cb0 0x7f6c6c16a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c2c90 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c26d0 con 0x7f6c6c156cb0 2023-12-19T07:59:21.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0c29b0 con 0x7f6c6c1568e0 2023-12-19T07:59:21.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.193+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46974/0 (socket says 172.21.15.111:46974) 2023-12-19T07:59:21.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 -- 172.21.15.111:0/1025874533 learned_addr learned my addr 172.21.15.111:0/1025874533 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:21.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6ad9d700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c6c156cb0 0x7f6c6c16a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c1568e0 0x7f6c6c16a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 -- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c1568e0 msgr2=0x7f6c6c16a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c1568e0 0x7f6c6c16a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 -- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c6c156cb0 msgr2=0x7f6c6c16a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c6c156cb0 0x7f6c6c16a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 -- 172.21.15.111:0/1025874533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64009ae0 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6a59c700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00ec80 tx=0x7f6c5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c009690 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5c00c850 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00cc10 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c60009420 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.197+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c0c4b90 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.201+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c04be40 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.201+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f6c5c013470 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.201+0000 7f6c5a7fc700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 0x7f6c4c062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.201+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f6c5c07cf10 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.205+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 0x7f6c4c062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.205+0000 7f6c69d9b700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 0x7f6c4c062c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c06ad00 tx=0x7f6c60017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.209+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c5c050ab0 con 0x7f6c6c0c84a0 2023-12-19T07:59:21.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.405+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c6c0c8ce0 con 0x7f6c4c060750 2023-12-19T07:59:21.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.537+0000 7f6c5a7fc700 1 -- 172.21.15.111:0/1025874533 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6c6c0c8ce0 con 0x7f6c4c060750 2023-12-19T07:59:21.537 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-12-19T07:59:21.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 msgr2=0x7f6c4c062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 0x7f6c4c062c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c06ad00 tx=0x7f6c60017000 comp rx=0 tx=0).stop 2023-12-19T07:59:21.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 msgr2=0x7f6c6c15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00ec80 tx=0x7f6c5c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 shutdown_connections 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c4c060750 0x7f6c4c062c00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c6c156cb0 0x7f6c6c16a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c6c1568e0 0x7f6c6c16a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 --2- 172.21.15.111:0/1025874533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c6c0c84a0 0x7f6c6c15a8b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 >> 172.21.15.111:0/1025874533 conn(0x7f6c6c06b5b0 msgr2=0x7f6c6c068570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 shutdown_connections 2023-12-19T07:59:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.541+0000 7f6c6b59e700 1 -- 172.21.15.111:0/1025874533 wait complete. 2023-12-19T07:59:21.564 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-12-19T07:59:21.564 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T07:59:21.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.741+0000 7ff81e02e700 1 Processor -- start 2023-12-19T07:59:21.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 -- start start 2023-12-19T07:59:21.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8180c86e0 0x7ff81806da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff81806df40 0x7ff8180672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81806d570 con 0x7ff818067830 2023-12-19T07:59:21.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff818067cb0 con 0x7ff81806df40 2023-12-19T07:59:21.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff81e02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81806c960 con 0x7ff8180c86e0 2023-12-19T07:59:21.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff8177fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8180c86e0 0x7ff81806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff817fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff818067830 0x7ff8180c8ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:34454/0 (socket says 172.21.15.111:34454) 2023-12-19T07:59:21.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff817fff700 1 -- 172.21.15.111:0/600751946 learned_addr learned my addr 172.21.15.111:0/600751946 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.745+0000 7ff80ffff700 1 --2- 172.21.15.111:0/600751946 >> v2:172.21.15.143:3300/0 conn(0x7ff81806df40 0x7ff8180672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff80ffff700 1 -- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8180c86e0 msgr2=0x7ff81806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff80ffff700 1 --2- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8180c86e0 0x7ff81806da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff80ffff700 1 -- 172.21.15.111:0/600751946 >> v1:172.21.15.143:6789/0 conn(0x7ff818067830 legacy=0x7ff8180c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff80ffff700 1 -- 172.21.15.111:0/600751946 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81806cc40 con 0x7ff81806df40 2023-12-19T07:59:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff816ffd700 1 -- 172.21.15.111:0/600751946 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244809229 0 0) 0x7ff81806d570 con 0x7ff818067830 2023-12-19T07:59:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff80ffff700 1 --2- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff81806df40 0x7ff8180672f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff8000098e0 tx=0x7ff8000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fce7f9e1d00caa9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff816ffd700 1 -- 172.21.15.111:0/600751946 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000b640 con 0x7ff81806df40 2023-12-19T07:59:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff816ffd700 1 -- 172.21.15.111:0/600751946 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000b7a0 con 0x7ff81806df40 2023-12-19T07:59:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff816ffd700 1 -- 172.21.15.111:0/600751946 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000bb40 con 0x7ff81806df40 2023-12-19T07:59:21.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 -- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff81806df40 msgr2=0x7ff8180672f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 --2- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff81806df40 0x7ff8180672f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff8000098e0 tx=0x7ff8000091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 -- 172.21.15.111:0/600751946 shutdown_connections 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 --2- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff81806df40 0x7ff8180672f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 --2- 172.21.15.111:0/600751946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8180c86e0 0x7ff81806da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 -- 172.21.15.111:0/600751946 >> 172.21.15.111:0/600751946 conn(0x7ff8180cdce0 msgr2=0x7ff8180ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 -- 172.21.15.111:0/600751946 shutdown_connections 2023-12-19T07:59:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.749+0000 7ff81e02e700 1 -- 172.21.15.111:0/600751946 wait complete. 2023-12-19T07:59:21.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.753+0000 7ff81e02e700 1 Processor -- start 2023-12-19T07:59:21.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.753+0000 7ff81e02e700 1 -- start start 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff818067830 0x7ff8180b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8180c86e0 0x7ff81815a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818159980 con 0x7ff81806df40 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818158d70 con 0x7ff8180c86e0 2023-12-19T07:59:21.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff81e02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff818159050 con 0x7ff818067830 2023-12-19T07:59:21.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff8177fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff818067830 0x7ff8180b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48692/0 (socket says 172.21.15.111:48692) 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 -- 172.21.15.111:0/2063770197 learned_addr learned my addr 172.21.15.111:0/2063770197 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff817fff700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8180c86e0 0x7ff81815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 -- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff818067830 msgr2=0x7ff8180b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff818067830 0x7ff8180b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 -- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8180c86e0 msgr2=0x7ff81815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8180c86e0 0x7ff81815a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:21.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 -- 172.21.15.111:0/2063770197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800009420 con 0x7ff81806df40 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.757+0000 7ff80ffff700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8180c82d0 tx=0x7ff800017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80000bb20 con 0x7ff81806df40 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800000c90 con 0x7ff81806df40 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff818159330 con 0x7ff81806df40 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800011630 con 0x7ff81806df40 2023-12-19T07:59:21.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818158a80 con 0x7ff81806df40 2023-12-19T07:59:21.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818071120 con 0x7ff81806df40 2023-12-19T07:59:21.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.761+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7ff8000119a0 con 0x7ff81806df40 2023-12-19T07:59:21.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.765+0000 7ff80f7fe700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 0x7ff7f8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:21.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.765+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7ff80007e170 con 0x7ff81806df40 2023-12-19T07:59:21.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.765+0000 7ff8177fe700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 0x7ff7f8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:21.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.765+0000 7ff8177fe700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 0x7ff7f8062c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff81815c130 tx=0x7ff8080091a0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:21.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.769+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff80004e1e0 con 0x7ff81806df40 2023-12-19T07:59:21.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:21.969+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff81806cf20 con 0x7ff7f8060750 2023-12-19T07:59:22.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.105+0000 7ff80f7fe700 1 -- 172.21.15.111:0/2063770197 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff81806cf20 con 0x7ff7f8060750 2023-12-19T07:59:22.105 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-12-19T07:59:22.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 msgr2=0x7ff7f8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 0x7ff7f8062c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff81815c130 tx=0x7ff8080091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 msgr2=0x7ff818158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8180c82d0 tx=0x7ff800017000 comp rx=0 tx=0).stop 2023-12-19T07:59:22.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 shutdown_connections 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7f8060750 0x7ff7f8062c00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8180c86e0 0x7ff81815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81806df40 0x7ff818158540 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 --2- 172.21.15.111:0/2063770197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff818067830 0x7ff8180b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 >> 172.21.15.111:0/2063770197 conn(0x7ff8180cdce0 msgr2=0x7ff8180c3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 shutdown_connections 2023-12-19T07:59:22.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.109+0000 7ff81e02e700 1 -- 172.21.15.111:0/2063770197 wait complete. 2023-12-19T07:59:22.131 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-12-19T07:59:22.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T07:59:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.317+0000 7f36dad23700 1 Processor -- start 2023-12-19T07:59:22.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.317+0000 7f36dad23700 1 -- start start 2023-12-19T07:59:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36dad23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36dad23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f36d40cac50 0x7f36d40af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36dad23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40cb3f0 con 0x7f36d40aedc0 2023-12-19T07:59:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36dad23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d40cc140 con 0x7f36d40cac50 2023-12-19T07:59:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36dad23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40af5e0 con 0x7f36d40af190 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f36d40cac50 0x7f36d40af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f36d40cac50 0x7f36d40af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48700/0 (socket says 172.21.15.111:48700) 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 -- 172.21.15.111:0/2161771919 learned_addr learned my addr 172.21.15.111:0/2161771919 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36d9520700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 -- 172.21.15.111:0/2161771919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 msgr2=0x7f36d406af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:22.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d406af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 -- 172.21.15.111:0/2161771919 >> v1:172.21.15.143:6789/0 conn(0x7f36d40aedc0 legacy=0x7f36d406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 -- 172.21.15.111:0/2161771919 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d406d220 con 0x7f36d40cac50 2023-12-19T07:59:22.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36da522700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d40af930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009f20 tx=0x7f36cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87852ef45388c515 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.321+0000 7f36d8d1f700 1 -- 172.21.15.111:0/2161771919 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00e070 con 0x7f36d40cac50 2023-12-19T07:59:22.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36d8d1f700 1 -- 172.21.15.111:0/2161771919 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00ba80 con 0x7f36d40cac50 2023-12-19T07:59:22.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36d8d1f700 1 -- 172.21.15.111:0/2161771919 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0092e0 con 0x7f36d40cac50 2023-12-19T07:59:22.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 -- 172.21.15.111:0/2161771919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 msgr2=0x7f36d40af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d40af930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009f20 tx=0x7f36cc009840 comp rx=0 tx=0).stop 2023-12-19T07:59:22.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 -- 172.21.15.111:0/2161771919 shutdown_connections 2023-12-19T07:59:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d40af930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 --2- 172.21.15.111:0/2161771919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d406af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 -- 172.21.15.111:0/2161771919 >> 172.21.15.111:0/2161771919 conn(0x7f36d40cc760 msgr2=0x7f36d40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 -- 172.21.15.111:0/2161771919 shutdown_connections 2023-12-19T07:59:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.325+0000 7f36dad23700 1 -- 172.21.15.111:0/2161771919 wait complete. 2023-12-19T07:59:22.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 Processor -- start 2023-12-19T07:59:22.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 -- start start 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 0x7f36d40c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40c65a0 con 0x7f36d40cac50 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40c3c90 con 0x7f36d40aedc0 2023-12-19T07:59:22.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36dad23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40c9b70 con 0x7f36d40af190 2023-12-19T07:59:22.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36da522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36da522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48714/0 (socket says 172.21.15.111:48714) 2023-12-19T07:59:22.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36d9d21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 0x7f36d40c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36da522700 1 -- 172.21.15.111:0/2597005859 learned_addr learned my addr 172.21.15.111:0/2597005859 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:22.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.329+0000 7f36d9520700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 -- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 msgr2=0x7f36d414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d414e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 -- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 msgr2=0x7f36d40c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 0x7f36d40c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 -- 172.21.15.111:0/2597005859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc009ae0 con 0x7f36d40cac50 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36d9d21700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 0x7f36d40c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:22.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36da522700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00b6c0 tx=0x7f36cc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0094e0 con 0x7f36d40cac50 2023-12-19T07:59:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc0092e0 con 0x7f36d40cac50 2023-12-19T07:59:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d40c5c00 con 0x7f36d40cac50 2023-12-19T07:59:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc026460 con 0x7f36d40cac50 2023-12-19T07:59:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d40c61c0 con 0x7f36d40cac50 2023-12-19T07:59:22.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.333+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d40c66e0 con 0x7f36d40cac50 2023-12-19T07:59:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.337+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f36cc003040 con 0x7f36d40cac50 2023-12-19T07:59:22.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.337+0000 7f36c9ffb700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 0x7f36b8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.337+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f36cc07ecf0 con 0x7f36d40cac50 2023-12-19T07:59:22.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.337+0000 7f36d9d21700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 0x7f36b8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.341+0000 7f36d9d21700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 0x7f36b8062c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36d0001dc0 tx=0x7f36d0009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.345+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f36d40c66e0 con 0x7f36d40cac50 2023-12-19T07:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.541+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f36d40c66e0 con 0x7f36b8060750 2023-12-19T07:59:22.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.677+0000 7f36c9ffb700 1 -- 172.21.15.111:0/2597005859 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f36d40c66e0 con 0x7f36b8060750 2023-12-19T07:59:22.677 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 msgr2=0x7f36b8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 0x7f36b8062c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36d0001dc0 tx=0x7f36d0009040 comp rx=0 tx=0).stop 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 msgr2=0x7f36d4169100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00b6c0 tx=0x7f36cc018000 comp rx=0 tx=0).stop 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 shutdown_connections 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36b8060750 0x7f36b8062c00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d40cac50 0x7f36d4169100 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36d40af190 0x7f36d414e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 --2- 172.21.15.111:0/2597005859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36d40aedc0 0x7f36d40c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 >> 172.21.15.111:0/2597005859 conn(0x7f36d40cc760 msgr2=0x7f36d40b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:22.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 shutdown_connections 2023-12-19T07:59:22.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.681+0000 7f36dad23700 1 -- 172.21.15.111:0/2597005859 wait complete. 2023-12-19T07:59:22.701 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-12-19T07:59:22.701 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T07:59:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 Processor -- start 2023-12-19T07:59:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 -- start start 2023-12-19T07:59:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb434158b30 0x7fb4340ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb4340c9880 0x7fb43415d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43415d870 con 0x7fb4340c7a00 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340cacc0 con 0x7fb4340c9880 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.877+0000 7fb439bcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340c7f70 con 0x7fb434158b30 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb4340c9880 0x7fb43415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb4340c9880 0x7fb43415d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48722/0 (socket says 172.21.15.111:48722) 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 -- 172.21.15.111:0/1474836773 learned_addr learned my addr 172.21.15.111:0/1474836773 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:22.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb438bca700 1 --2- 172.21.15.111:0/1474836773 >> v2:172.21.15.143:3300/0 conn(0x7fb434158b30 0x7fb4340ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 -- 172.21.15.111:0/1474836773 >> v2:172.21.15.143:3300/0 conn(0x7fb434158b30 msgr2=0x7fb4340ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:22.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 --2- 172.21.15.111:0/1474836773 >> v2:172.21.15.143:3300/0 conn(0x7fb434158b30 0x7fb4340ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 -- 172.21.15.111:0/1474836773 >> v1:172.21.15.143:6789/0 conn(0x7fb4340c7a00 legacy=0x7fb4340c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 -- 172.21.15.111:0/1474836773 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340caf10 con 0x7fb4340c9880 2023-12-19T07:59:22.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4393cb700 1 --2- 172.21.15.111:0/1474836773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4340c9880 0x7fb43415d330 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf568b60987c1b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4337fe700 1 -- 172.21.15.111:0/1474836773 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00e070 con 0x7fb4340c9880 2023-12-19T07:59:22.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4337fe700 1 -- 172.21.15.111:0/1474836773 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00b990 con 0x7fb4340c9880 2023-12-19T07:59:22.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb4337fe700 1 -- 172.21.15.111:0/1474836773 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0092e0 con 0x7fb4340c9880 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 -- 172.21.15.111:0/1474836773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4340c9880 msgr2=0x7fb43415d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1474836773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4340c9880 0x7fb43415d330 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).stop 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 -- 172.21.15.111:0/1474836773 shutdown_connections 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1474836773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4340c9880 0x7fb43415d330 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1474836773 >> v2:172.21.15.143:3300/0 conn(0x7fb434158b30 0x7fb4340ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 -- 172.21.15.111:0/1474836773 >> 172.21.15.111:0/1474836773 conn(0x7fb43406b5b0 msgr2=0x7fb43406b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 -- 172.21.15.111:0/1474836773 shutdown_connections 2023-12-19T07:59:22.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.881+0000 7fb439bcc700 1 -- 172.21.15.111:0/1474836773 wait complete. 2023-12-19T07:59:22.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.885+0000 7fb439bcc700 1 Processor -- start 2023-12-19T07:59:22.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.885+0000 7fb439bcc700 1 -- start start 2023-12-19T07:59:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4340c7a00 0x7fb4340af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4340c9880 0x7fb4340c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43416e4e0 con 0x7fb434158b30 2023-12-19T07:59:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43416d8d0 con 0x7fb4340c9880 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43416dbb0 con 0x7fb4340c7a00 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48736/0 (socket says 172.21.15.111:48736) 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 -- 172.21.15.111:0/1104056035 learned_addr learned my addr 172.21.15.111:0/1104056035 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb433fff700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4340c9880 0x7fb4340c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb438bca700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4340c7a00 0x7fb4340af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 -- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4340c7a00 msgr2=0x7fb4340af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4340c7a00 0x7fb4340af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 -- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4340c9880 msgr2=0x7fb4340c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4340c9880 0x7fb4340c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 -- 172.21.15.111:0/1104056035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009ae0 con 0x7fb434158b30 2023-12-19T07:59:22.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb4393cb700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009880 tx=0x7fb42c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c008ad0 con 0x7fb434158b30 2023-12-19T07:59:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c008c30 con 0x7fb434158b30 2023-12-19T07:59:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43416de90 con 0x7fb434158b30 2023-12-19T07:59:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c026460 con 0x7fb434158b30 2023-12-19T07:59:22.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.889+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4340afd40 con 0x7fb434158b30 2023-12-19T07:59:22.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.893+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb434063410 con 0x7fb434158b30 2023-12-19T07:59:22.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.893+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fb42c003040 con 0x7fb434158b30 2023-12-19T07:59:22.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.893+0000 7fb430ff9700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 0x7fb418062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:22.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.897+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fb42c07ecb0 con 0x7fb434158b30 2023-12-19T07:59:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.897+0000 7fb438bca700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 0x7fb418062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.897+0000 7fb438bca700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 0x7fb418062c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb43416e890 tx=0x7fb4280091a0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:22.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:22.901+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb434063410 con 0x7fb434158b30 2023-12-19T07:59:23.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.093+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4340c8970 con 0x7fb418060780 2023-12-19T07:59:23.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.229+0000 7fb430ff9700 1 -- 172.21.15.111:0/1104056035 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb4340c8970 con 0x7fb418060780 2023-12-19T07:59:23.226 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-12-19T07:59:23.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.229+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 msgr2=0x7fb418062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.229+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 0x7fb418062c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb43416e890 tx=0x7fb4280091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.229+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 msgr2=0x7fb4340b1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.229+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009880 tx=0x7fb42c018000 comp rx=0 tx=0).stop 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 shutdown_connections 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb418060780 0x7fb418062c30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4340c9880 0x7fb4340c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434158b30 0x7fb4340b1420 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 --2- 172.21.15.111:0/1104056035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4340c7a00 0x7fb4340af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 >> 172.21.15.111:0/1104056035 conn(0x7fb43406b5b0 msgr2=0x7fb434068920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 shutdown_connections 2023-12-19T07:59:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.233+0000 7fb439bcc700 1 -- 172.21.15.111:0/1104056035 wait complete. 2023-12-19T07:59:23.252 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-12-19T07:59:23.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T07:59:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.417+0000 7fe7dd1cc700 1 Processor -- start 2023-12-19T07:59:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.417+0000 7fe7dd1cc700 1 -- start start 2023-12-19T07:59:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7dd1cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe7d80c8f30 0x7fe7d806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7dd1cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806c7c0 con 0x7fe7d80c8ae0 2023-12-19T07:59:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7dd1cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806ca10 con 0x7fe7d8156640 2023-12-19T07:59:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7dd1cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d806ccf0 con 0x7fe7d80c8f30 2023-12-19T07:59:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7cf7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe7d80c8ae0 0x7fe7d8157210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46992/0 (socket says 172.21.15.111:46992) 2023-12-19T07:59:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7cf7fe700 1 -- 172.21.15.111:0/490511507 learned_addr learned my addr 172.21.15.111:0/490511507 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.421+0000 7fe7dc9cb700 1 --2- 172.21.15.111:0/490511507 >> v2:172.21.15.111:3300/0 conn(0x7fe7d80c8f30 0x7fe7d806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373708347 0 0) 0x7fe7d806c7c0 con 0x7fe7d80c8ae0 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4003570 con 0x7fe7d80c8ae0 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184702471 0 0) 0x7fe7d806ca10 con 0x7fe7d8156640 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d806c7c0 con 0x7fe7d8156640 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/490511507 >> v1:172.21.15.143:6789/0 conn(0x7fe7d8156640 legacy=0x7fe7d8156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/490511507 >> v1:172.21.15.111:6789/0 conn(0x7fe7d80c8ae0 legacy=0x7fe7d8157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/490511507 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d806cfd0 con 0x7fe7d80c8f30 2023-12-19T07:59:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dc9cb700 1 --2- 172.21.15.111:0/490511507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d80c8f30 0x7fe7d806d7d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e60 tx=0x7fe7d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53241fb104aa180b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000e070 con 0x7fe7d80c8f30 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d000bab0 con 0x7fe7d80c8f30 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7d77fe700 1 -- 172.21.15.111:0/490511507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000be70 con 0x7fe7d80c8f30 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/490511507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d80c8f30 msgr2=0x7fe7d806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/490511507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d80c8f30 0x7fe7d806d7d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e60 tx=0x7fe7d0009780 comp rx=0 tx=0).stop 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/490511507 shutdown_connections 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/490511507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d80c8f30 0x7fe7d806d7d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/490511507 >> 172.21.15.111:0/490511507 conn(0x7fe7d8067bb0 msgr2=0x7fe7d80c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/490511507 shutdown_connections 2023-12-19T07:59:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.425+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/490511507 wait complete. 2023-12-19T07:59:23.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 Processor -- start 2023-12-19T07:59:23.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 -- start start 2023-12-19T07:59:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7d80c8ae0 0x7fe7d813b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 0x7fe7d8154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80b0fc0 con 0x7fe7d8156640 2023-12-19T07:59:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d8159550 con 0x7fe7d80c8f30 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dd1cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d8169db0 con 0x7fe7d80c8ae0 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dc9cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dc9cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48756/0 (socket says 172.21.15.111:48756) 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/3669258833 learned_addr learned my addr 172.21.15.111:0/3669258833 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7cf7fe700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 0x7fe7d8154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.429+0000 7fe7d7fff700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7d80c8ae0 0x7fe7d813b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7d80c8ae0 msgr2=0x7fe7d813b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7d80c8ae0 0x7fe7d813b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 msgr2=0x7fe7d8154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 0x7fe7d8154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 -- 172.21.15.111:0/3669258833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d0009a20 con 0x7fe7d8156640 2023-12-19T07:59:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7cf7fe700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 0x7fe7d8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dc9cb700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80699a0 tx=0x7fe7d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0012db0 con 0x7fe7d8156640 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d0026480 con 0x7fe7d8156640 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0026840 con 0x7fe7d8156640 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d81597f0 con 0x7fe7d8156640 2023-12-19T07:59:23.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d816a090 con 0x7fe7d8156640 2023-12-19T07:59:23.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.433+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d804be40 con 0x7fe7d8156640 2023-12-19T07:59:23.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.437+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fe7d001c400 con 0x7fe7d8156640 2023-12-19T07:59:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.437+0000 7fe7d4ff9700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 0x7fe7b8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.437+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fe7d007ee40 con 0x7fe7d8156640 2023-12-19T07:59:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.437+0000 7fe7d7fff700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 0x7fe7b8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.437+0000 7fe7d7fff700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 0x7fe7b8062c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8007fd0 tx=0x7fe7c8007d50 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.445+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe7d004eda0 con 0x7fe7d8156640 2023-12-19T07:59:23.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.641+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7d8064040 con 0x7fe7b8060750 2023-12-19T07:59:23.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.765+0000 7fe7d4ff9700 1 -- 172.21.15.111:0/3669258833 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe7d8064040 con 0x7fe7b8060750 2023-12-19T07:59:23.765 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-12-19T07:59:23.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 msgr2=0x7fe7b8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 0x7fe7b8062c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8007fd0 tx=0x7fe7c8007d50 comp rx=0 tx=0).stop 2023-12-19T07:59:23.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 msgr2=0x7fe7d8155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80699a0 tx=0x7fe7d0019000 comp rx=0 tx=0).stop 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 shutdown_connections 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe7b8060750 0x7fe7b8062c00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80c8f30 0x7fe7d8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7d8156640 0x7fe7d8155040 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 --2- 172.21.15.111:0/3669258833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7d80c8ae0 0x7fe7d813b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 >> 172.21.15.111:0/3669258833 conn(0x7fe7d8067bb0 msgr2=0x7fe7d81585e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:23.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 shutdown_connections 2023-12-19T07:59:23.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.769+0000 7fe7dd1cc700 1 -- 172.21.15.111:0/3669258833 wait complete. 2023-12-19T07:59:23.790 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-12-19T07:59:23.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T07:59:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.969+0000 7f8438e8e700 1 Processor -- start 2023-12-19T07:59:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.969+0000 7f8438e8e700 1 -- start start 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8438e8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f84340af190 0x7f843406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8438e8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f84340cac50 0x7f84340af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8438e8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84340cb3f0 con 0x7f84340aedc0 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8438e8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84340cc140 con 0x7f84340cac50 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8438e8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84340af610 con 0x7f84340af190 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f84337fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f84340aedc0 0x7f843406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47004/0 (socket says 172.21.15.111:47004) 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f84337fe700 1 -- 172.21.15.111:0/3148584029 learned_addr learned my addr 172.21.15.111:0/3148584029 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:23.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8433fff700 1 --2- 172.21.15.111:0/3148584029 >> v2:172.21.15.111:3300/0 conn(0x7f84340cac50 0x7f84340af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f8432ffd700 1 --2- 172.21.15.111:0/3148584029 >> v2:172.21.15.143:3300/0 conn(0x7f84340af190 0x7f843406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f84327fc700 1 -- 172.21.15.111:0/3148584029 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711056287 0 0) 0x7f84340cb3f0 con 0x7f84340aedc0 2023-12-19T07:59:23.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.973+0000 7f84327fc700 1 -- 172.21.15.111:0/3148584029 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f841c003570 con 0x7f84340aedc0 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8433fff700 1 -- 172.21.15.111:0/3148584029 >> v2:172.21.15.143:3300/0 conn(0x7f84340af190 msgr2=0x7f843406af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8433fff700 1 --2- 172.21.15.111:0/3148584029 >> v2:172.21.15.143:3300/0 conn(0x7f84340af190 0x7f843406af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8433fff700 1 -- 172.21.15.111:0/3148584029 >> v1:172.21.15.111:6789/0 conn(0x7f84340aedc0 legacy=0x7f843406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8433fff700 1 -- 172.21.15.111:0/3148584029 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843406d220 con 0x7f84340cac50 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8433fff700 1 --2- 172.21.15.111:0/3148584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340cac50 0x7f84340af930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f84340cfc50 tx=0x7f842c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f3dbd37ffb20e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f84327fc700 1 -- 172.21.15.111:0/3148584029 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c00e070 con 0x7f84340cac50 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f84327fc700 1 -- 172.21.15.111:0/3148584029 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c00bb60 con 0x7f84340cac50 2023-12-19T07:59:23.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f84327fc700 1 -- 172.21.15.111:0/3148584029 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c012430 con 0x7f84340cac50 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 -- 172.21.15.111:0/3148584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340cac50 msgr2=0x7f84340af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 --2- 172.21.15.111:0/3148584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340cac50 0x7f84340af930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f84340cfc50 tx=0x7f842c009780 comp rx=0 tx=0).stop 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 -- 172.21.15.111:0/3148584029 shutdown_connections 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 --2- 172.21.15.111:0/3148584029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340cac50 0x7f84340af930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 --2- 172.21.15.111:0/3148584029 >> v2:172.21.15.143:3300/0 conn(0x7f84340af190 0x7f843406af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 -- 172.21.15.111:0/3148584029 >> 172.21.15.111:0/3148584029 conn(0x7f84340cc760 msgr2=0x7f84340ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 -- 172.21.15.111:0/3148584029 shutdown_connections 2023-12-19T07:59:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.977+0000 7f8438e8e700 1 -- 172.21.15.111:0/3148584029 wait complete. 2023-12-19T07:59:23.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 Processor -- start 2023-12-19T07:59:23.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 -- start start 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84340af190 0x7f84340c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84340cac50 0x7f8434170290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84340c3750 con 0x7f84340aedc0 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84340c42d0 con 0x7f84340cac50 2023-12-19T07:59:23.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f8438e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84340c39a0 con 0x7f84340af190 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f84337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f84337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48780/0 (socket says 172.21.15.111:48780) 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.981+0000 7f84337fe700 1 -- 172.21.15.111:0/862093540 learned_addr learned my addr 172.21.15.111:0/862093540 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f8433fff700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84340cac50 0x7f8434170290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f8432ffd700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84340af190 0x7f84340c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 -- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84340af190 msgr2=0x7f84340c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84340af190 0x7f84340c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 -- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84340cac50 msgr2=0x7f8434170290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:23.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84340cac50 0x7f8434170290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:23.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 -- 172.21.15.111:0/862093540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c009a20 con 0x7f84340aedc0 2023-12-19T07:59:23.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f84337fe700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f842400c7f0 tx=0x7f8424007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424005cd0 con 0x7f84340aedc0 2023-12-19T07:59:23.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424005e50 con 0x7f84340aedc0 2023-12-19T07:59:23.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84340c3c80 con 0x7f84340aedc0 2023-12-19T07:59:23.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424007320 con 0x7f84340aedc0 2023-12-19T07:59:23.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84340c99d0 con 0x7f84340aedc0 2023-12-19T07:59:23.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.985+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84340c9bf0 con 0x7f84340aedc0 2023-12-19T07:59:23.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.989+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f842401b3b0 con 0x7f84340aedc0 2023-12-19T07:59:23.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.989+0000 7f841b7fe700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 0x7f8420062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:23.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.989+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f842407c4a0 con 0x7f84340aedc0 2023-12-19T07:59:23.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.989+0000 7f8432ffd700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 0x7f8420062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:23.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.993+0000 7f8432ffd700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 0x7f8420062c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f843408b350 tx=0x7f8428008040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:23.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:23.997+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f842404c510 con 0x7f84340aedc0 2023-12-19T07:59:24.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.189+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f84340c4410 con 0x7f8420060750 2023-12-19T07:59:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.321+0000 7f841b7fe700 1 -- 172.21.15.111:0/862093540 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f84280108b0 con 0x7f8420060750 2023-12-19T07:59:24.319 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 msgr2=0x7f8420062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 0x7f8420062c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f843408b350 tx=0x7f8428008040 comp rx=0 tx=0).stop 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 msgr2=0x7f84340c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f842400c7f0 tx=0x7f8424007e00 comp rx=0 tx=0).stop 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 shutdown_connections 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8420060750 0x7f8420062c00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84340cac50 0x7f8434170290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84340af190 0x7f84340c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 --2- 172.21.15.111:0/862093540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84340aedc0 0x7f84340c4760 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 >> 172.21.15.111:0/862093540 conn(0x7f84340cc760 msgr2=0x7f84340b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 shutdown_connections 2023-12-19T07:59:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.325+0000 7f8438e8e700 1 -- 172.21.15.111:0/862093540 wait complete. 2023-12-19T07:59:24.345 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-12-19T07:59:24.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T07:59:24.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.529+0000 7efd2dc8e700 1 Processor -- start 2023-12-19T07:59:24.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.529+0000 7efd2dc8e700 1 -- start start 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2dc8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd281558d0 0x7efd281569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2dc8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd280abe10 0x7efd280ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2dc8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280cbfb0 con 0x7efd280adc80 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2dc8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd28156fa0 con 0x7efd280abe10 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2dc8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd280abc60 con 0x7efd281558d0 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd280abe10 0x7efd280ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2d48d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efd280adc80 0x7efd280c6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47018/0 (socket says 172.21.15.111:47018) 2023-12-19T07:59:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2d48d700 1 -- 172.21.15.111:0/1187404399 learned_addr learned my addr 172.21.15.111:0/1187404399 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd2cc8c700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd281558d0 0x7efd281569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1f7fe700 1 -- 172.21.15.111:0/1187404399 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036816514 0 0) 0x7efd280cbfb0 con 0x7efd280adc80 2023-12-19T07:59:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1f7fe700 1 -- 172.21.15.111:0/1187404399 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10003570 con 0x7efd280adc80 2023-12-19T07:59:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 -- 172.21.15.111:0/1187404399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd281558d0 msgr2=0x7efd281569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd281558d0 0x7efd281569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 -- 172.21.15.111:0/1187404399 >> v1:172.21.15.111:6789/0 conn(0x7efd280adc80 legacy=0x7efd280c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 -- 172.21.15.111:0/1187404399 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280cc290 con 0x7efd280abe10 2023-12-19T07:59:24.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.533+0000 7efd1ffff700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280abe10 0x7efd280ad740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efd180098e0 tx=0x7efd180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc5ad45b8ec628 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:24.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd1f7fe700 1 -- 172.21.15.111:0/1187404399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18011630 con 0x7efd280abe10 2023-12-19T07:59:24.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd1f7fe700 1 -- 172.21.15.111:0/1187404399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd18011790 con 0x7efd280abe10 2023-12-19T07:59:24.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd1f7fe700 1 -- 172.21.15.111:0/1187404399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18011b30 con 0x7efd280abe10 2023-12-19T07:59:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1187404399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280abe10 msgr2=0x7efd280ad740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280abe10 0x7efd280ad740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efd180098e0 tx=0x7efd180091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1187404399 shutdown_connections 2023-12-19T07:59:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280abe10 0x7efd280ad740 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1187404399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd281558d0 0x7efd281569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1187404399 >> 172.21.15.111:0/1187404399 conn(0x7efd2806b5b0 msgr2=0x7efd2806b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1187404399 shutdown_connections 2023-12-19T07:59:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1187404399 wait complete. 2023-12-19T07:59:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.537+0000 7efd2dc8e700 1 Processor -- start 2023-12-19T07:59:24.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- start start 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd280abe10 0x7efd28143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd281558d0 0x7efd280c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd280b0720 con 0x7efd280adc80 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd28144360 con 0x7efd281558d0 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd28154c00 con 0x7efd280abe10 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48804/0 (socket says 172.21.15.111:48804) 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 -- 172.21.15.111:0/1324817995 learned_addr learned my addr 172.21.15.111:0/1324817995 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2cc8c700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd280abe10 0x7efd28143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2d48d700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd281558d0 0x7efd280c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 -- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd280abe10 msgr2=0x7efd28143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd280abe10 0x7efd28143e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 -- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd281558d0 msgr2=0x7efd280c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd281558d0 0x7efd280c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 -- 172.21.15.111:0/1324817995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18009420 con 0x7efd280adc80 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1ffff700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efd280c37f0 tx=0x7efd18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18011b10 con 0x7efd280adc80 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd180233f0 con 0x7efd280adc80 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18023770 con 0x7efd280adc80 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd280c2a00 con 0x7efd280adc80 2023-12-19T07:59:24.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.541+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd280b00d0 con 0x7efd280adc80 2023-12-19T07:59:24.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.545+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2804be40 con 0x7efd280adc80 2023-12-19T07:59:24.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.545+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7efd1802c3b0 con 0x7efd280adc80 2023-12-19T07:59:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.545+0000 7efd1cff9700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 0x7efd14062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.545+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7efd1807e290 con 0x7efd280adc80 2023-12-19T07:59:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.549+0000 7efd2cc8c700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 0x7efd14062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:24.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.549+0000 7efd2cc8c700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 0x7efd14062c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efd24009c10 tx=0x7efd24008040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.553+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd1804e380 con 0x7efd280adc80 2023-12-19T07:59:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.749+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7efd280c3820 con 0x7efd14060780 2023-12-19T07:59:24.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.881+0000 7efd1cff9700 1 -- 172.21.15.111:0/1324817995 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7efd280c3820 con 0x7efd14060780 2023-12-19T07:59:24.880 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-12-19T07:59:24.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 msgr2=0x7efd14062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 0x7efd14062c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efd24009c10 tx=0x7efd24008040 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 msgr2=0x7efd280c1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efd280c37f0 tx=0x7efd18017000 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 shutdown_connections 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd14060780 0x7efd14062c30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd281558d0 0x7efd280c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd280adc80 0x7efd280c1e40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 --2- 172.21.15.111:0/1324817995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd280abe10 0x7efd28143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 >> 172.21.15.111:0/1324817995 conn(0x7efd2806b5b0 msgr2=0x7efd28155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:24.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 shutdown_connections 2023-12-19T07:59:24.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:24.885+0000 7efd2dc8e700 1 -- 172.21.15.111:0/1324817995 wait complete. 2023-12-19T07:59:24.903 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-12-19T07:59:24.903 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T07:59:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 Processor -- start 2023-12-19T07:59:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 -- start start 2023-12-19T07:59:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f950815c930 0x7f9508163e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080b0ac0 con 0x7f95080cb500 2023-12-19T07:59:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950806b210 con 0x7f95080cb130 2023-12-19T07:59:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.085+0000 7f950df45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95080ca810 con 0x7f950815c930 2023-12-19T07:59:25.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9506ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95080cb500 0x7f950815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47022/0 (socket says 172.21.15.111:47022) 2023-12-19T07:59:25.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95077fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f95080cb130 0x7f950806a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:34504/0 (socket says 172.21.15.111:34504) 2023-12-19T07:59:25.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9506ffd700 1 -- 172.21.15.111:0/2983939588 learned_addr learned my addr 172.21.15.111:0/2983939588 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:25.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9507fff700 1 --2- 172.21.15.111:0/2983939588 >> v2:172.21.15.143:3300/0 conn(0x7f950815c930 0x7f9508163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521406202 0 0) 0x7f95080b0ac0 con 0x7f95080cb500 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f0003570 con 0x7f95080cb500 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921525980 0 0) 0x7f950806b210 con 0x7f95080cb130 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95080b0ac0 con 0x7f95080cb130 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9507fff700 1 -- 172.21.15.111:0/2983939588 >> v1:172.21.15.143:6789/0 conn(0x7f95080cb130 legacy=0x7f950806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9507fff700 1 -- 172.21.15.111:0/2983939588 >> v1:172.21.15.111:6789/0 conn(0x7f95080cb500 legacy=0x7f950815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190663224 0 0) 0x7f94f0003570 con 0x7f95080cb500 2023-12-19T07:59:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9507fff700 1 -- 172.21.15.111:0/2983939588 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080b0190 con 0x7f950815c930 2023-12-19T07:59:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f9507fff700 1 --2- 172.21.15.111:0/2983939588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f950815c930 0x7f9508163e10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9500009e60 tx=0x7f9500009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f580b5e7d44e1b06 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000e070 con 0x7f950815c930 2023-12-19T07:59:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950000b7a0 con 0x7f950815c930 2023-12-19T07:59:25.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.089+0000 7f95067fc700 1 -- 172.21.15.111:0/2983939588 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0013a0 con 0x7f950815c930 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 -- 172.21.15.111:0/2983939588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f950815c930 msgr2=0x7f9508163e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 --2- 172.21.15.111:0/2983939588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f950815c930 0x7f9508163e10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9500009e60 tx=0x7f9500009780 comp rx=0 tx=0).stop 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 -- 172.21.15.111:0/2983939588 shutdown_connections 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 --2- 172.21.15.111:0/2983939588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f950815c930 0x7f9508163e10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 -- 172.21.15.111:0/2983939588 >> 172.21.15.111:0/2983939588 conn(0x7f9508067bb0 msgr2=0x7f95080cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 -- 172.21.15.111:0/2983939588 shutdown_connections 2023-12-19T07:59:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 -- 172.21.15.111:0/2983939588 wait complete. 2023-12-19T07:59:25.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.093+0000 7f950df45700 1 Processor -- start 2023-12-19T07:59:25.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 -- start start 2023-12-19T07:59:25.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95080cb500 0x7f95081569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f950815c930 0x7f9508156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950814c780 con 0x7f95080cb130 2023-12-19T07:59:25.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080b35c0 con 0x7f95080cb500 2023-12-19T07:59:25.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f950df45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950814da90 con 0x7f950815c930 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f9507fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f950815c930 0x7f9508156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48816/0 (socket says 172.21.15.111:48816) 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f9506ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95080cb500 0x7f95081569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 -- 172.21.15.111:0/1051399757 learned_addr learned my addr 172.21.15.111:0/1051399757 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 -- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f950815c930 msgr2=0x7f9508156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f950815c930 0x7f9508156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 -- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95080cb500 msgr2=0x7f95081569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95080cb500 0x7f95081569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.097+0000 7f95077fe700 1 -- 172.21.15.111:0/1051399757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500009a20 con 0x7f95080cb130 2023-12-19T07:59:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f95077fe700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c7c0 tx=0x7f94f8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8005c10 con 0x7f95080cb130 2023-12-19T07:59:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f8005d70 con 0x7f95080cb130 2023-12-19T07:59:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9508157550 con 0x7f95080cb130 2023-12-19T07:59:25.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f8007320 con 0x7f95080cb130 2023-12-19T07:59:25.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95080b3250 con 0x7f95080cb130 2023-12-19T07:59:25.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.101+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95080b30b0 con 0x7f95080cb130 2023-12-19T07:59:25.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.105+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f94f801b3b0 con 0x7f95080cb130 2023-12-19T07:59:25.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.105+0000 7f94ef7fe700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 0x7f94f4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.105+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f94f807c4d0 con 0x7f95080cb130 2023-12-19T07:59:25.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.105+0000 7f9506ffd700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 0x7f94f4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.105+0000 7f9506ffd700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 0x7f94f4062c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f950816c7f0 tx=0x7f94fc001a00 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.109+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f95080b30b0 con 0x7f95080cb130 2023-12-19T07:59:25.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.297+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f95080b3700 con 0x7f94f4060750 2023-12-19T07:59:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.425+0000 7f94ef7fe700 1 -- 172.21.15.111:0/1051399757 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f94fc00f9b0 con 0x7f94f4060750 2023-12-19T07:59:25.424 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-12-19T07:59:25.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 msgr2=0x7f94f4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 0x7f94f4062c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f950816c7f0 tx=0x7f94fc001a00 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 msgr2=0x7f950814c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c7c0 tx=0x7f94f8007e00 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 shutdown_connections 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f94f4060750 0x7f94f4062c00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95080cb500 0x7f95081569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f950815c930 0x7f9508156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 --2- 172.21.15.111:0/1051399757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95080cb130 0x7f950814c240 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 >> 172.21.15.111:0/1051399757 conn(0x7f9508067bb0 msgr2=0x7f950806a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 shutdown_connections 2023-12-19T07:59:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.429+0000 7f950df45700 1 -- 172.21.15.111:0/1051399757 wait complete. 2023-12-19T07:59:25.452 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-12-19T07:59:25.452 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T07:59:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.629+0000 7fdc8ffff700 1 Processor -- start 2023-12-19T07:59:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.629+0000 7fdc8ffff700 1 -- start start 2023-12-19T07:59:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc900cac50 0x7fdc900af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc900cb3f0 con 0x7fdc9006af50 2023-12-19T07:59:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc900cc140 con 0x7fdc900cac50 2023-12-19T07:59:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9006d220 con 0x7fdc900aedc0 2023-12-19T07:59:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:34298/0 (socket says 172.21.15.111:34298) 2023-12-19T07:59:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8effd700 1 -- 172.21.15.111:0/1151109047 learned_addr learned my addr 172.21.15.111:0/1151109047 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.633+0000 7fdc8f7fe700 1 --2- 172.21.15.111:0/1151109047 >> v2:172.21.15.143:3300/0 conn(0x7fdc900cac50 0x7fdc900af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8dffb700 1 -- 172.21.15.111:0/1151109047 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014098330 0 0) 0x7fdc900cb3f0 con 0x7fdc9006af50 2023-12-19T07:59:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8dffb700 1 -- 172.21.15.111:0/1151109047 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc78003570 con 0x7fdc9006af50 2023-12-19T07:59:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8effd700 1 -- 172.21.15.111:0/1151109047 >> v2:172.21.15.143:3300/0 conn(0x7fdc900cac50 msgr2=0x7fdc900af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8effd700 1 --2- 172.21.15.111:0/1151109047 >> v2:172.21.15.143:3300/0 conn(0x7fdc900cac50 0x7fdc900af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8effd700 1 -- 172.21.15.111:0/1151109047 >> v1:172.21.15.143:6789/0 conn(0x7fdc9006af50 legacy=0x7fdc9006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8effd700 1 -- 172.21.15.111:0/1151109047 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9006b3a0 con 0x7fdc900aedc0 2023-12-19T07:59:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8effd700 1 --2- 172.21.15.111:0/1151109047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099b0 tx=0x7fdc80009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f12dddc82fc485a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8dffb700 1 -- 172.21.15.111:0/1151109047 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000b680 con 0x7fdc900aedc0 2023-12-19T07:59:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8dffb700 1 -- 172.21.15.111:0/1151109047 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8000b7e0 con 0x7fdc900aedc0 2023-12-19T07:59:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8dffb700 1 -- 172.21.15.111:0/1151109047 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000bb80 con 0x7fdc900aedc0 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 -- 172.21.15.111:0/1151109047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 msgr2=0x7fdc900af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/1151109047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099b0 tx=0x7fdc80009250 comp rx=0 tx=0).stop 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 -- 172.21.15.111:0/1151109047 shutdown_connections 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/1151109047 >> v2:172.21.15.143:3300/0 conn(0x7fdc900cac50 0x7fdc900af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/1151109047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900af190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 -- 172.21.15.111:0/1151109047 >> 172.21.15.111:0/1151109047 conn(0x7fdc900cc760 msgr2=0x7fdc900ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 -- 172.21.15.111:0/1151109047 shutdown_connections 2023-12-19T07:59:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.637+0000 7fdc8ffff700 1 -- 172.21.15.111:0/1151109047 wait complete. 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 Processor -- start 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 -- start start 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9006af50 0x7fdc90170580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9016d800 con 0x7fdc900cac50 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9016dae0 con 0x7fdc9006af50 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8ffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9016ddc0 con 0x7fdc900aedc0 2023-12-19T07:59:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48822/0 (socket says 172.21.15.111:48822) 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 -- 172.21.15.111:0/3723181392 learned_addr learned my addr 172.21.15.111:0/3723181392 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8effd700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9006af50 0x7fdc90170580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8e7fc700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 -- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 msgr2=0x7fdc900b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 -- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9006af50 msgr2=0x7fdc90170580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.641+0000 7fdc8f7fe700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9006af50 0x7fdc90170580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8f7fe700 1 -- 172.21.15.111:0/3723181392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc800094f0 con 0x7fdc900cac50 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8e7fc700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8f7fe700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009df0 tx=0x7fdc8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800cb50 con 0x7fdc900cac50 2023-12-19T07:59:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8800f040 con 0x7fdc900cac50 2023-12-19T07:59:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9016e0a0 con 0x7fdc900cac50 2023-12-19T07:59:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88014480 con 0x7fdc900cac50 2023-12-19T07:59:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc900c40a0 con 0x7fdc900cac50 2023-12-19T07:59:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.645+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9004be40 con 0x7fdc900cac50 2023-12-19T07:59:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.649+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fdc88014830 con 0x7fdc900cac50 2023-12-19T07:59:25.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.649+0000 7fdc76ffd700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 0x7fdc7c062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:25.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.649+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fdc8807d800 con 0x7fdc900cac50 2023-12-19T07:59:25.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.649+0000 7fdc8effd700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 0x7fdc7c062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:25.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.649+0000 7fdc8effd700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 0x7fdc7c062c00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099b0 tx=0x7fdc80017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.657+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc8804d760 con 0x7fdc900cac50 2023-12-19T07:59:25.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.849+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc9006c8f0 con 0x7fdc7c060750 2023-12-19T07:59:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.985+0000 7fdc76ffd700 1 -- 172.21.15.111:0/3723181392 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdc9006c8f0 con 0x7fdc7c060750 2023-12-19T07:59:25.986 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 msgr2=0x7fdc7c062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 0x7fdc7c062c00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099b0 tx=0x7fdc80017000 comp rx=0 tx=0).stop 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 msgr2=0x7fdc900b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009df0 tx=0x7fdc8800b040 comp rx=0 tx=0).stop 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 shutdown_connections 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc7c060750 0x7fdc7c062c00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc900cac50 0x7fdc900b4a60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900aedc0 0x7fdc900b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.989+0000 7fdc8ffff700 1 --2- 172.21.15.111:0/3723181392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9006af50 0x7fdc90170580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.993+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 >> 172.21.15.111:0/3723181392 conn(0x7fdc900cc760 msgr2=0x7fdc9006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.993+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 shutdown_connections 2023-12-19T07:59:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:25.993+0000 7fdc8ffff700 1 -- 172.21.15.111:0/3723181392 wait complete. 2023-12-19T07:59:26.012 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-12-19T07:59:26.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:26.013 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-12-19T07:59:26.054 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:26.054 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:26.054 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:26.055 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:26.055 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_01_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_01_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:26.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir0 2023-12-19T07:59:26.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.281+0000 7fadccd85700 1 Processor -- start 2023-12-19T07:59:26.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.281+0000 7fadccd85700 1 -- start start 2023-12-19T07:59:26.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.285+0000 7fadccd85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadc80c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.285+0000 7fadccd85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc806dcd0 con 0x7fadc80c86e0 2023-12-19T07:59:26.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.285+0000 7fadccd85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc806e160 con 0x7fadc80c8ff0 2023-12-19T07:59:26.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.285+0000 7fadccd85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc806ced0 con 0x7fadc80cb920 2023-12-19T07:59:26.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fadc80c86e0 0x7fadc806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47034/0 (socket says 172.21.15.111:47034) 2023-12-19T07:59:26.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc659c700 1 -- 172.21.15.111:0/1099189077 learned_addr learned my addr 172.21.15.111:0/1099189077 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadc80c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843477024 0 0) 0x7fadc806dcd0 con 0x7fadc80c86e0 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb0003570 con 0x7fadc80c86e0 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206448533 0 0) 0x7fadc806e160 con 0x7fadc80c8ff0 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc806dcd0 con 0x7fadc80c8ff0 2023-12-19T07:59:26.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782660108 0 0) 0x7fadb0003570 con 0x7fadc80c86e0 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 msgr2=0x7fadc80c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadc80c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 >> v1:172.21.15.143:6789/0 conn(0x7fadc80c8ff0 legacy=0x7fadc8166ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc806d0c0 con 0x7fadc80c86e0 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fadb8001470 con 0x7fadc80c86e0 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 >> v1:172.21.15.111:6789/0 conn(0x7fadc80c86e0 legacy=0x7fadc806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb0003c20 0x7fadb0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadb0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb000bbf0 con 0x7fadb0004810 2023-12-19T07:59:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb000bd30 con 0x7fadb0003c20 2023-12-19T07:59:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb000bef0 con 0x7fadc80cb920 2023-12-19T07:59:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc659c700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb0003c20 0x7fadb0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc6d9d700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadb0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 -- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 msgr2=0x7fadb0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadb0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 -- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb0003c20 msgr2=0x7fadb0004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb0003c20 0x7fadb0004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 -- 172.21.15.111:0/1099189077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb0003570 con 0x7fadb0004810 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc6d9d700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80cb920 0x7fadb0005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadb0004c80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009c00 tx=0x7fadbc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67166e58c6f4e111 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.289+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00e710 con 0x7fadb0004810 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc00e870 con 0x7fadb0004810 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadc559a700 1 -- 172.21.15.111:0/1099189077 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00ec30 con 0x7fadb0004810 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- 172.21.15.111:0/1099189077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 msgr2=0x7fadb0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 --2- 172.21.15.111:0/1099189077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadb0004c80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009c00 tx=0x7fadbc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadc6d9d700 1 -- 172.21.15.111:0/1099189077 reap_dead start 2023-12-19T07:59:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- 172.21.15.111:0/1099189077 shutdown_connections 2023-12-19T07:59:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- 172.21.15.111:0/1099189077 >> 172.21.15.111:0/1099189077 conn(0x7fadc80cdce0 msgr2=0x7fadc80ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:26.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- 172.21.15.111:0/1099189077 shutdown_connections 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- 172.21.15.111:0/1099189077 wait complete. 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 Processor -- start 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- start start 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc80b0b80 0x7fadc8157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80b2f10 0x7fadc80b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc8160e50 con 0x7fadb0004810 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc815b380 con 0x7fadc80b0b80 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.293+0000 7fadccd85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80b1990 con 0x7fadc80b2f10 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48834/0 (socket says 172.21.15.111:48834) 2023-12-19T07:59:26.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 -- 172.21.15.111:0/2473937720 learned_addr learned my addr 172.21.15.111:0/2473937720 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc6d9d700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80b2f10 0x7fadc80b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc80b0b80 0x7fadc8157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 -- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80b2f10 msgr2=0x7fadc80b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80b2f10 0x7fadc80b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 -- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc80b0b80 msgr2=0x7fadc8157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc80b0b80 0x7fadc8157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 -- 172.21.15.111:0/2473937720 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc815a770 con 0x7fadb0004810 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadc659c700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fadb800ebf0 tx=0x7fadb8001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800c7e0 con 0x7fadb0004810 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb800c940 con 0x7fadb0004810 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc815aa50 con 0x7fadb0004810 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800cd00 con 0x7fadb0004810 2023-12-19T07:59:26.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc815b010 con 0x7fadb0004810 2023-12-19T07:59:26.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.297+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc8071120 con 0x7fadb0004810 2023-12-19T07:59:26.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.301+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fadb8013630 con 0x7fadb0004810 2023-12-19T07:59:26.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.301+0000 7fadb67fc700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 0x7fada8062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.301+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fadb807ce20 con 0x7fadb0004810 2023-12-19T07:59:26.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.301+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 0x7fada8062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.305+0000 7fadc5d9b700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 0x7fada8062c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0091e0 tx=0x7fadbc01a000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.309+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fadb804ce90 con 0x7fadb0004810 2023-12-19T07:59:26.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.497+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fadc815b1d0 con 0x7fada8060780 2023-12-19T07:59:26.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.513+0000 7fadb67fc700 1 -- 172.21.15.111:0/2473937720 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fadc815b1d0 con 0x7fada8060780 2023-12-19T07:59:26.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:26.512 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:41", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:26.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 msgr2=0x7fada8062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 0x7fada8062c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0091e0 tx=0x7fadbc01a000 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 msgr2=0x7fadc80b1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fadb800ebf0 tx=0x7fadb8001010 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 shutdown_connections 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fada8060780 0x7fada8062c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc80b0b80 0x7fadc8157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadc80b2f10 0x7fadc80b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 --2- 172.21.15.111:0/2473937720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadb0004810 0x7fadc80b1450 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 >> 172.21.15.111:0/2473937720 conn(0x7fadc80cdce0 msgr2=0x7fadc8067bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 shutdown_connections 2023-12-19T07:59:26.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.517+0000 7fadccd85700 1 -- 172.21.15.111:0/2473937720 wait complete. 2023-12-19T07:59:26.536 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:41", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:26.537 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:41', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:26.537 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:26.537 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:26.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:26.537 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-12-19T07:59:26.572 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:26.572 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:26.572 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:26.574 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_01_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_01_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:26.574 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir1 2023-12-19T07:59:26.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.789+0000 7f5590ce3700 1 Processor -- start 2023-12-19T07:59:26.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.789+0000 7f5590ce3700 1 -- start start 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f5590ce3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f558c0aef60 0x7f558c156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f5590ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f558c0ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f5590ce3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c06beb0 con 0x7f558c1569e0 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f5590ce3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c06cb80 con 0x7f558c0aef60 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f5590ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c06ce60 con 0x7f558c0adf40 2023-12-19T07:59:26.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f558c1569e0 0x7f558c0b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47038/0 (socket says 172.21.15.111:47038) 2023-12-19T07:59:26.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558b7fe700 1 -- 172.21.15.111:0/233637514 learned_addr learned my addr 172.21.15.111:0/233637514 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:26.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 --2- 172.21.15.111:0/233637514 >> v2:172.21.15.143:3300/0 conn(0x7f558c0aef60 0x7f558c156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558bfff700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f558c0ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683529740 0 0) 0x7f558c06beb0 con 0x7f558c1569e0 2023-12-19T07:59:26.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574003570 con 0x7f558c1569e0 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558906868 0 0) 0x7f5574003570 con 0x7f558c1569e0 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 msgr2=0x7f558c0ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f558c0ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 >> v2:172.21.15.143:3300/0 conn(0x7f558c0aef60 msgr2=0x7f558c156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 --2- 172.21.15.111:0/233637514 >> v2:172.21.15.143:3300/0 conn(0x7f558c0aef60 0x7f558c156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c06d140 con 0x7f558c1569e0 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f557c001470 con 0x7f558c1569e0 2023-12-19T07:59:26.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 >> v1:172.21.15.111:6789/0 conn(0x7f558c1569e0 legacy=0x7f558c0b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f5574003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f5574004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5574004d50 0x7f55740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c06cb80 con 0x7f5574003fe0 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c06ce60 con 0x7f5574004d50 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557400ba90 con 0x7f558c0adf40 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f5574004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558b7fe700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f5574003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558bfff700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5574004d50 0x7f55740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 -- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 msgr2=0x7f5574003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c0adf40 0x7f5574003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 -- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5574004d50 msgr2=0x7f55740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5574004d50 0x7f55740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558affd700 1 -- 172.21.15.111:0/233637514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5574003570 con 0x7f5574003fe0 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.793+0000 7f558bfff700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5574004d50 0x7f55740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f558affd700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f5574004810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5580009c00 tx=0x7f55800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed269f6350f4a37 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558000e6b0 con 0x7f5574003fe0 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000e810 con 0x7f5574003fe0 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f558a7fc700 1 -- 172.21.15.111:0/233637514 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558000ebd0 con 0x7f5574003fe0 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 -- 172.21.15.111:0/233637514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 msgr2=0x7f5574004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 --2- 172.21.15.111:0/233637514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f5574004810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5580009c00 tx=0x7f55800091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f558bfff700 1 -- 172.21.15.111:0/233637514 reap_dead start 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 -- 172.21.15.111:0/233637514 shutdown_connections 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 -- 172.21.15.111:0/233637514 >> 172.21.15.111:0/233637514 conn(0x7f558c0c8c00 msgr2=0x7f558c0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:26.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 -- 172.21.15.111:0/233637514 shutdown_connections 2023-12-19T07:59:26.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 -- 172.21.15.111:0/233637514 wait complete. 2023-12-19T07:59:26.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.797+0000 7f5590ce3700 1 Processor -- start 2023-12-19T07:59:26.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- start start 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f558c16d170 0x7f558c0b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c155850 0x7f558c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c16e1b0 con 0x7f5574003fe0 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c1561e0 con 0x7f558c16d170 2023-12-19T07:59:26.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c153300 con 0x7f558c155850 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48850/0 (socket says 172.21.15.111:48850) 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 -- 172.21.15.111:0/1570149949 learned_addr learned my addr 172.21.15.111:0/1570149949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558affd700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f558c16d170 0x7f558c0b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558bfff700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c155850 0x7f558c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 -- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c155850 msgr2=0x7f558c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c155850 0x7f558c155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 -- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f558c16d170 msgr2=0x7f558c0b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f558c16d170 0x7f558c0b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 -- 172.21.15.111:0/1570149949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0c4790 con 0x7f5574003fe0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f558b7fe700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f557c00e960 tx=0x7f557c001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c012590 con 0x7f5574003fe0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c0126f0 con 0x7f5574003fe0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c16d630 con 0x7f5574003fe0 2023-12-19T07:59:26.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c012ab0 con 0x7f5574003fe0 2023-12-19T07:59:26.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.801+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c16dc70 con 0x7f5574003fe0 2023-12-19T07:59:26.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.805+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c04be40 con 0x7f5574003fe0 2023-12-19T07:59:26.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.805+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f557c01d460 con 0x7f5574003fe0 2023-12-19T07:59:26.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.805+0000 7f55737fe700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 0x7f5578062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:26.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.809+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f557c07cdc0 con 0x7f5574003fe0 2023-12-19T07:59:26.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.809+0000 7f558affd700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 0x7f5578062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:26.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.809+0000 7f558affd700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 0x7f5578062c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55800091e0 tx=0x7f558001a000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:26.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:26.813+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f557c04ce30 con 0x7f5574003fe0 2023-12-19T07:59:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.005+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f558c16df90 con 0x7f5578060750 2023-12-19T07:59:27.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.021+0000 7f55737fe700 1 -- 172.21.15.111:0/1570149949 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f558c16df90 con 0x7f5578060750 2023-12-19T07:59:27.021 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:27.021 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:42", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 msgr2=0x7f5578062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 0x7f5578062c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55800091e0 tx=0x7f558001a000 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 msgr2=0x7f558c152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f557c00e960 tx=0x7f557c001df0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 shutdown_connections 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5578060750 0x7f5578062c00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f558c16d170 0x7f558c0b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f558c155850 0x7f558c155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 --2- 172.21.15.111:0/1570149949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5574003fe0 0x7f558c152dc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 >> 172.21.15.111:0/1570149949 conn(0x7f558c0c8c00 msgr2=0x7f558c1541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 shutdown_connections 2023-12-19T07:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.025+0000 7f5590ce3700 1 -- 172.21.15.111:0/1570149949 wait complete. 2023-12-19T07:59:27.046 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:42", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:27.046 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:42', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:27.046 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:27.046 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:27.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:27.046 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-12-19T07:59:27.079 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:27.079 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:27.079 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:27.079 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:27.079 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:27.080 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:27.081 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_01_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:27.081 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir2 2023-12-19T07:59:27.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.289+0000 7fd69bfff700 1 Processor -- start 2023-12-19T07:59:27.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.289+0000 7fd69bfff700 1 -- start start 2023-12-19T07:59:27.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd69c0c8ae0 0x7fd69c157210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c157750 0x7fd69c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c06c7c0 con 0x7fd69c156640 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c06ca10 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c06ccf0 con 0x7fd69c157750 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd69c0c8ae0 0x7fd69c157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd69c0c8ae0 0x7fd69c157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48866/0 (socket says 172.21.15.111:48866) 2023-12-19T07:59:27.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 -- 172.21.15.111:0/463506626 learned_addr learned my addr 172.21.15.111:0/463506626 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:27.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69b7fe700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c157750 0x7fd69c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 -- 172.21.15.111:0/463506626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c157750 msgr2=0x7fd69c06d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:27.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c157750 0x7fd69c06d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 -- 172.21.15.111:0/463506626 >> v1:172.21.15.111:6789/0 conn(0x7fd69c156640 legacy=0x7fd69c156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 -- 172.21.15.111:0/463506626 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c06cfd0 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.293+0000 7fd69a7fc700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0c8ae0 0x7fd69c157210 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd6900099b0 tx=0x7fd690009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f031ade3da5bb86c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd699ffb700 1 -- 172.21.15.111:0/463506626 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000b860 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd699ffb700 1 -- 172.21.15.111:0/463506626 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69000b9c0 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd699ffb700 1 -- 172.21.15.111:0/463506626 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000bd60 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 -- 172.21.15.111:0/463506626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0c8ae0 msgr2=0x7fd69c157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0c8ae0 0x7fd69c157210 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd6900099b0 tx=0x7fd690009250 comp rx=0 tx=0).stop 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 -- 172.21.15.111:0/463506626 shutdown_connections 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c157750 0x7fd69c06d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 --2- 172.21.15.111:0/463506626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0c8ae0 0x7fd69c157210 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 -- 172.21.15.111:0/463506626 >> 172.21.15.111:0/463506626 conn(0x7fd69c067bb0 msgr2=0x7fd69c0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 -- 172.21.15.111:0/463506626 shutdown_connections 2023-12-19T07:59:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.297+0000 7fd69bfff700 1 -- 172.21.15.111:0/463506626 wait complete. 2023-12-19T07:59:27.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 Processor -- start 2023-12-19T07:59:27.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 -- start start 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c0c8ae0 0x7fd69c15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c157750 0x7fd69c14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c15de80 con 0x7fd69c156640 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c15bf00 con 0x7fd69c157750 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69bfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c14ffd0 con 0x7fd69c0c8ae0 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48872/0 (socket says 172.21.15.111:48872) 2023-12-19T07:59:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69a7fc700 1 -- 172.21.15.111:0/4127913563 learned_addr learned my addr 172.21.15.111:0/4127913563 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69b7fe700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c157750 0x7fd69c14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.301+0000 7fd69affd700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c0c8ae0 0x7fd69c15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 -- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c0c8ae0 msgr2=0x7fd69c15b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c0c8ae0 0x7fd69c15b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 -- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c157750 msgr2=0x7fd69c14d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c157750 0x7fd69c14d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 -- 172.21.15.111:0/4127913563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6900094f0 con 0x7fd69c156640 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69a7fc700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd69c14d110 tx=0x7fd690017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690024440 con 0x7fd69c156640 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6900245c0 con 0x7fd69c156640 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690024980 con 0x7fd69c156640 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c15d830 con 0x7fd69c156640 2023-12-19T07:59:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c142260 con 0x7fd69c156640 2023-12-19T07:59:27.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.305+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c064040 con 0x7fd69c156640 2023-12-19T07:59:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.309+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fd69001a420 con 0x7fd69c156640 2023-12-19T07:59:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.309+0000 7fd682ffd700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 0x7fd688062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.309+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fd69007ec40 con 0x7fd69c156640 2023-12-19T07:59:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.309+0000 7fd69affd700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 0x7fd688062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.313+0000 7fd69affd700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 0x7fd688062c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd69c06b640 tx=0x7fd68c009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.317+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd69004ecb0 con 0x7fd69c156640 2023-12-19T07:59:27.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.509+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd69c04be40 con 0x7fd688060750 2023-12-19T07:59:27.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.525+0000 7fd682ffd700 1 -- 172.21.15.111:0/4127913563 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fd69c04be40 con 0x7fd688060750 2023-12-19T07:59:27.526 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:27.526 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:42", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 msgr2=0x7fd688062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 0x7fd688062c00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd69c06b640 tx=0x7fd68c009040 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 msgr2=0x7fd69c141d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd69c14d110 tx=0x7fd690017000 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 shutdown_connections 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd688060750 0x7fd688062c00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c157750 0x7fd69c14d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c156640 0x7fd69c141d20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 --2- 172.21.15.111:0/4127913563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c0c8ae0 0x7fd69c15b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.529+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 >> 172.21.15.111:0/4127913563 conn(0x7fd69c067bb0 msgr2=0x7fd69c06a840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.533+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 shutdown_connections 2023-12-19T07:59:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.533+0000 7fd69bfff700 1 -- 172.21.15.111:0/4127913563 wait complete. 2023-12-19T07:59:27.549 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:42", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:27.550 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:42', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:27.550 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:27.550 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:27.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:27.550 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-12-19T07:59:27.586 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:27.587 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:27.587 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:27.587 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:27.587 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:27.587 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:27.588 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:27.590 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_01_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_01_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_01_UTC'] 2023-12-19T07:59:27.590 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir3 2023-12-19T07:59:27.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.817+0000 7fcf9e878700 1 Processor -- start 2023-12-19T07:59:27.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.817+0000 7fcf9e878700 1 -- start start 2023-12-19T07:59:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9e878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcf980cb130 0x7fcf980cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9e878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 0x7fcf9806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9e878700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980b0ac0 con 0x7fcf9806a6a0 2023-12-19T07:59:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9e878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9806b030 con 0x7fcf980cb130 2023-12-19T07:59:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9e878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9806b210 con 0x7fcf9815c930 2023-12-19T07:59:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf97fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcf980cb130 0x7fcf980cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf977fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcf9806a6a0 0x7fcf9815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:34520/0 (socket says 172.21.15.111:34520) 2023-12-19T07:59:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf9ce15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 0x7fcf9806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.821+0000 7fcf977fe700 1 -- 172.21.15.111:0/49218988 learned_addr learned my addr 172.21.15.111:0/49218988 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf96ffd700 1 -- 172.21.15.111:0/49218988 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095808282 0 0) 0x7fcf980b0ac0 con 0x7fcf9806a6a0 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf96ffd700 1 -- 172.21.15.111:0/49218988 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80003570 con 0x7fcf9806a6a0 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf97fff700 1 -- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 msgr2=0x7fcf9806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf97fff700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 0x7fcf9806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf97fff700 1 -- 172.21.15.111:0/49218988 >> v1:172.21.15.143:6789/0 conn(0x7fcf9806a6a0 legacy=0x7fcf9815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf97fff700 1 -- 172.21.15.111:0/49218988 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980ca810 con 0x7fcf980cb130 2023-12-19T07:59:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9ce15700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 0x7fcf9806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:27.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf97fff700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf980cb500 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcf880098e0 tx=0x7fcf880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9934e4f85ab239e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf96ffd700 1 -- 172.21.15.111:0/49218988 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88011630 con 0x7fcf980cb130 2023-12-19T07:59:27.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf96ffd700 1 -- 172.21.15.111:0/49218988 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88011790 con 0x7fcf980cb130 2023-12-19T07:59:27.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf96ffd700 1 -- 172.21.15.111:0/49218988 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88011b30 con 0x7fcf980cb130 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 -- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 msgr2=0x7fcf980cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf980cb500 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcf880098e0 tx=0x7fcf880091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 -- 172.21.15.111:0/49218988 shutdown_connections 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9815c930 0x7fcf9806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 --2- 172.21.15.111:0/49218988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf980cb500 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 -- 172.21.15.111:0/49218988 >> 172.21.15.111:0/49218988 conn(0x7fcf98067bb0 msgr2=0x7fcf980cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 -- 172.21.15.111:0/49218988 shutdown_connections 2023-12-19T07:59:27.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.825+0000 7fcf9e878700 1 -- 172.21.15.111:0/49218988 wait complete. 2023-12-19T07:59:27.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 Processor -- start 2023-12-19T07:59:27.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 -- start start 2023-12-19T07:59:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9806a6a0 0x7fcf980b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf9815b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9816a800 con 0x7fcf9815c930 2023-12-19T07:59:27.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9816aa50 con 0x7fcf980cb130 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9e878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9816ad30 con 0x7fcf9806a6a0 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9ce15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9ce15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48874/0 (socket says 172.21.15.111:48874) 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf9ce15700 1 -- 172.21.15.111:0/3189884187 learned_addr learned my addr 172.21.15.111:0/3189884187 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf977fe700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf9815b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.829+0000 7fcf97fff700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9806a6a0 0x7fcf980b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 -- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9806a6a0 msgr2=0x7fcf980b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9806a6a0 0x7fcf980b54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 -- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 msgr2=0x7fcf9815b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf9815b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 -- 172.21.15.111:0/3189884187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009a20 con 0x7fcf9815c930 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9ce15700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009f20 tx=0x7fcf9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000cb70 con 0x7fcf9815c930 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9000f040 con 0x7fcf9815c930 2023-12-19T07:59:27.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf88009420 con 0x7fcf9815c930 2023-12-19T07:59:27.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90014480 con 0x7fcf9815c930 2023-12-19T07:59:27.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9816b2f0 con 0x7fcf9815c930 2023-12-19T07:59:27.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.833+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98071120 con 0x7fcf9815c930 2023-12-19T07:59:27.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.837+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fcf90003020 con 0x7fcf9815c930 2023-12-19T07:59:27.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.837+0000 7fcf7ffff700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 0x7fcf84062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:27.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.841+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7fcf9007d780 con 0x7fcf9815c930 2023-12-19T07:59:27.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.841+0000 7fcf97fff700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 0x7fcf84062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:27.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.841+0000 7fcf97fff700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 0x7fcf84062c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf880091e0 tx=0x7fcf88017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:27.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:27.845+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcf9004d6e0 con 0x7fcf9815c930 2023-12-19T07:59:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.037+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf9816b4e0 con 0x7fcf84060750 2023-12-19T07:59:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.057+0000 7fcf7ffff700 1 -- 172.21.15.111:0/3189884187 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fcf9816b4e0 con 0x7fcf84060750 2023-12-19T07:59:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:28.055 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:43", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 msgr2=0x7fcf84062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 0x7fcf84062c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf880091e0 tx=0x7fcf88017000 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 msgr2=0x7fcf9815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009f20 tx=0x7fcf9000b040 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 shutdown_connections 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcf84060750 0x7fcf84062c00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf980cb130 0x7fcf9815b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf9815c930 0x7fcf9815b5f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 --2- 172.21.15.111:0/3189884187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf9806a6a0 0x7fcf980b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 >> 172.21.15.111:0/3189884187 conn(0x7fcf98067bb0 msgr2=0x7fcf980cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 shutdown_connections 2023-12-19T07:59:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.061+0000 7fcf9e878700 1 -- 172.21.15.111:0/3189884187 wait complete. 2023-12-19T07:59:28.081 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:43", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:28.081 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:43', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:28.081 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:28.081 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:28.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:28.082 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-12-19T07:59:28.113 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:28.114 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:28.115 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:28.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir4 2023-12-19T07:59:28.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.329+0000 7f4e96723700 1 Processor -- start 2023-12-19T07:59:28.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.329+0000 7f4e96723700 1 -- start start 2023-12-19T07:59:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e96723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 0x7f4e9006af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e96723700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e900cb3f0 con 0x7f4e900af190 2023-12-19T07:59:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e96723700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9006d220 con 0x7f4e900aedc0 2023-12-19T07:59:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e96723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900cc140 con 0x7f4e900b0040 2023-12-19T07:59:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e94f20700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4e900af190 0x7f4e900af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47064/0 (socket says 172.21.15.111:47064) 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e94f20700 1 -- 172.21.15.111:0/777638156 learned_addr learned my addr 172.21.15.111:0/777638156 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e95721700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4e900aedc0 0x7f4e9006d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:34528/0 (socket says 172.21.15.111:34528) 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.333+0000 7f4e95f22700 1 --2- 172.21.15.111:0/777638156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 0x7f4e9006af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e87fff700 1 -- 172.21.15.111:0/777638156 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171111087 0 0) 0x7f4e900cb3f0 con 0x7f4e900af190 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e87fff700 1 -- 172.21.15.111:0/777638156 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c003570 con 0x7f4e900af190 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e95f22700 1 -- 172.21.15.111:0/777638156 >> v1:172.21.15.143:6789/0 conn(0x7f4e900aedc0 legacy=0x7f4e9006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e95f22700 1 -- 172.21.15.111:0/777638156 >> v1:172.21.15.111:6789/0 conn(0x7f4e900af190 legacy=0x7f4e900af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e95f22700 1 -- 172.21.15.111:0/777638156 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9006c8f0 con 0x7f4e900b0040 2023-12-19T07:59:28.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e95f22700 1 --2- 172.21.15.111:0/777638156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 0x7f4e9006af50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009e90 tx=0x7f4e88009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f48cd410c15a43e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e87fff700 1 -- 172.21.15.111:0/777638156 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8800e070 con 0x7f4e900b0040 2023-12-19T07:59:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e87fff700 1 -- 172.21.15.111:0/777638156 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8800bc30 con 0x7f4e900b0040 2023-12-19T07:59:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.337+0000 7f4e87fff700 1 -- 172.21.15.111:0/777638156 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88012430 con 0x7f4e900b0040 2023-12-19T07:59:28.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 -- 172.21.15.111:0/777638156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 msgr2=0x7f4e9006af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 --2- 172.21.15.111:0/777638156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 0x7f4e9006af50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009e90 tx=0x7f4e88009780 comp rx=0 tx=0).stop 2023-12-19T07:59:28.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 -- 172.21.15.111:0/777638156 shutdown_connections 2023-12-19T07:59:28.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 --2- 172.21.15.111:0/777638156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900b0040 0x7f4e9006af50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 -- 172.21.15.111:0/777638156 >> 172.21.15.111:0/777638156 conn(0x7f4e900cc760 msgr2=0x7f4e900ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:28.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 -- 172.21.15.111:0/777638156 shutdown_connections 2023-12-19T07:59:28.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 -- 172.21.15.111:0/777638156 wait complete. 2023-12-19T07:59:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.341+0000 7f4e96723700 1 Processor -- start 2023-12-19T07:59:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 -- start start 2023-12-19T07:59:28.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900af190 0x7f4e9014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e900b0040 0x7f4e900c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900af610 con 0x7f4e900aedc0 2023-12-19T07:59:28.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9015b560 con 0x7f4e900b0040 2023-12-19T07:59:28.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e96723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900c4d20 con 0x7f4e900af190 2023-12-19T07:59:28.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48890/0 (socket says 172.21.15.111:48890) 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95f22700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e900b0040 0x7f4e900c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 -- 172.21.15.111:0/1417180702 learned_addr learned my addr 172.21.15.111:0/1417180702 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e94f20700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900af190 0x7f4e9014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 -- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900af190 msgr2=0x7f4e9014b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900af190 0x7f4e9014b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 -- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e900b0040 msgr2=0x7f4e900c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e900b0040 0x7f4e900c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 -- 172.21.15.111:0/1417180702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88009a20 con 0x7f4e900aedc0 2023-12-19T07:59:28.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.345+0000 7f4e95721700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00c560 tx=0x7f4e8c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c005a70 con 0x7f4e900aedc0 2023-12-19T07:59:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c005bd0 con 0x7f4e900aedc0 2023-12-19T07:59:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e900c5000 con 0x7f4e900aedc0 2023-12-19T07:59:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0070d0 con 0x7f4e900aedc0 2023-12-19T07:59:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e900c55c0 con 0x7f4e900aedc0 2023-12-19T07:59:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.349+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9004be40 con 0x7f4e900aedc0 2023-12-19T07:59:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.353+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f4e8c007480 con 0x7f4e900aedc0 2023-12-19T07:59:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.353+0000 7f4e857fa700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 0x7f4e74062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.353+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f4e8c07c1c0 con 0x7f4e900aedc0 2023-12-19T07:59:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.353+0000 7f4e94f20700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 0x7f4e74062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.353+0000 7f4e94f20700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 0x7f4e74062c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e9014e220 tx=0x7f4e80009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.357+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e8c04c230 con 0x7f4e900aedc0 2023-12-19T07:59:28.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.549+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e9006cbd0 con 0x7f4e74060750 2023-12-19T07:59:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.569+0000 7f4e857fa700 1 -- 172.21.15.111:0/1417180702 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f4e9006cbd0 con 0x7f4e74060750 2023-12-19T07:59:28.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:28.567 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:44", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 msgr2=0x7f4e74062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 0x7f4e74062c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e9014e220 tx=0x7f4e80009040 comp rx=0 tx=0).stop 2023-12-19T07:59:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 msgr2=0x7f4e9014a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00c560 tx=0x7f4e8c007f70 comp rx=0 tx=0).stop 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 shutdown_connections 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e74060750 0x7f4e74062c00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e900b0040 0x7f4e900c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e900af190 0x7f4e9014b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 --2- 172.21.15.111:0/1417180702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e900aedc0 0x7f4e9014a860 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 >> 172.21.15.111:0/1417180702 conn(0x7f4e900cc760 msgr2=0x7f4e9015c210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 shutdown_connections 2023-12-19T07:59:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.573+0000 7f4e96723700 1 -- 172.21.15.111:0/1417180702 wait complete. 2023-12-19T07:59:28.590 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:44", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:28.591 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:44', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:28.591 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:28.591 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:28.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:28.591 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-12-19T07:59:28.620 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:28.621 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:28.622 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:28.622 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:28.623 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_01_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_01_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:28.623 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir5 2023-12-19T07:59:28.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.837+0000 7f39d964c700 1 Processor -- start 2023-12-19T07:59:28.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.837+0000 7f39d964c700 1 -- start start 2023-12-19T07:59:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d964c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39d40b02b0 0x7f39d40687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d964c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d4159280 con 0x7f39d4158de0 2023-12-19T07:59:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d964c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d40ad3b0 con 0x7f39d4158a10 2023-12-19T07:59:28.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d964c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d4069120 con 0x7f39d40b02b0 2023-12-19T07:59:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d8e4b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39d4158de0 0x7f39d406d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47080/0 (socket says 172.21.15.111:47080) 2023-12-19T07:59:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d8e4b700 1 -- 172.21.15.111:0/2875932629 learned_addr learned my addr 172.21.15.111:0/2875932629 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/2875932629 >> v2:172.21.15.111:3300/0 conn(0x7f39d40b02b0 0x7f39d40687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054014434 0 0) 0x7f39d4159280 con 0x7f39d4158de0 2023-12-19T07:59:28.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0003570 con 0x7f39d4158de0 2023-12-19T07:59:28.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203724084 0 0) 0x7f39d40ad3b0 con 0x7f39d4158a10 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39cb7fe700 1 -- 172.21.15.111:0/2875932629 >> v1:172.21.15.143:6789/0 conn(0x7f39d4158a10 legacy=0x7f39d40680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39cb7fe700 1 -- 172.21.15.111:0/2875932629 >> v1:172.21.15.111:6789/0 conn(0x7f39d4158de0 legacy=0x7f39d406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.841+0000 7f39cb7fe700 1 -- 172.21.15.111:0/2875932629 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4068e40 con 0x7f39d40b02b0 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/2875932629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d40b02b0 0x7f39d40687f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009e60 tx=0x7f39bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a336d4543ffd6d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00e070 con 0x7f39d40b02b0 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00b850 con 0x7f39d40b02b0 2023-12-19T07:59:28.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d37fe700 1 -- 172.21.15.111:0/2875932629 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc001470 con 0x7f39d40b02b0 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 -- 172.21.15.111:0/2875932629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d40b02b0 msgr2=0x7f39d40687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 --2- 172.21.15.111:0/2875932629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d40b02b0 0x7f39d40687f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009e60 tx=0x7f39bc009780 comp rx=0 tx=0).stop 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 -- 172.21.15.111:0/2875932629 shutdown_connections 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 --2- 172.21.15.111:0/2875932629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d40b02b0 0x7f39d40687f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 -- 172.21.15.111:0/2875932629 >> 172.21.15.111:0/2875932629 conn(0x7f39d40ce5e0 msgr2=0x7f39d40ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 -- 172.21.15.111:0/2875932629 shutdown_connections 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.845+0000 7f39d964c700 1 -- 172.21.15.111:0/2875932629 wait complete. 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 Processor -- start 2023-12-19T07:59:28.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 -- start start 2023-12-19T07:59:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39d4158a10 0x7f39d40b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39d40c1f50 0x7f39d41570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40b1ab0 con 0x7f39d4158de0 2023-12-19T07:59:28.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d415add0 con 0x7f39d4158a10 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d964c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40b1810 con 0x7f39d40c1f50 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48914/0 (socket says 172.21.15.111:48914) 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 -- 172.21.15.111:0/1411044519 learned_addr learned my addr 172.21.15.111:0/1411044519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d3fff700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39d4158a10 0x7f39d40b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39d8e4b700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39d40c1f50 0x7f39d41570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 -- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39d40c1f50 msgr2=0x7f39d41570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39d40c1f50 0x7f39d41570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 -- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39d4158a10 msgr2=0x7f39d40b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39d4158a10 0x7f39d40b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.849+0000 7f39cb7fe700 1 -- 172.21.15.111:0/1411044519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc009a20 con 0x7f39d4158de0 2023-12-19T07:59:28.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f39bc003000 tx=0x7f39bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc000f40 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc01f410 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc01f7d0 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d415a4a0 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d415aa60 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d415ac20 con 0x7f39d4158de0 2023-12-19T07:59:28.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.853+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f39bc01fbc0 con 0x7f39d4158de0 2023-12-19T07:59:28.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.857+0000 7f39d0ff9700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 0x7f39b4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:28.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.857+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f39bc07f120 con 0x7f39d4158de0 2023-12-19T07:59:28.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.857+0000 7f39d3fff700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 0x7f39b4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:28.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.857+0000 7f39d3fff700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 0x7f39b4062c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39c4007fd0 tx=0x7f39c4007d30 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:28.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:28.861+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39bc04f080 con 0x7f39d4158de0 2023-12-19T07:59:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.053+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39d404beb0 con 0x7f39b4060750 2023-12-19T07:59:29.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.073+0000 7f39d0ff9700 1 -- 172.21.15.111:0/1411044519 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f39d404beb0 con 0x7f39b4060750 2023-12-19T07:59:29.071 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:29.071 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:44", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.073+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 msgr2=0x7f39b4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.073+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 0x7f39b4062c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39c4007fd0 tx=0x7f39c4007d30 comp rx=0 tx=0).stop 2023-12-19T07:59:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.073+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 msgr2=0x7f39d40b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.073+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f39bc003000 tx=0x7f39bc017000 comp rx=0 tx=0).stop 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 shutdown_connections 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b4060750 0x7f39b4062c00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39d40c1f50 0x7f39d41570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d4158de0 0x7f39d40b12d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 --2- 172.21.15.111:0/1411044519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39d4158a10 0x7f39d40b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 >> 172.21.15.111:0/1411044519 conn(0x7f39d40ce5e0 msgr2=0x7f39d41542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 shutdown_connections 2023-12-19T07:59:29.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.077+0000 7f39d964c700 1 -- 172.21.15.111:0/1411044519 wait complete. 2023-12-19T07:59:29.098 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:44", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:29.098 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:44', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:29.098 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:29.098 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:29.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:29.099 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-12-19T07:59:29.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:29.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:29.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:29.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_01_UTC 2023-12-19T07:59:29.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:29.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:29.137 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_01_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:29.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir6 2023-12-19T07:59:29.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.361+0000 7f9671763700 1 Processor -- start 2023-12-19T07:59:29.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.361+0000 7f9671763700 1 -- start start 2023-12-19T07:59:29.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f966c0aedc0 0x7f966c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c0cac50 0x7f966c0af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c06b3d0 con 0x7f966c06af50 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966c0cc140 con 0x7f966c0aedc0 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c0cb3f0 con 0x7f966c0cac50 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f966c0aedc0 0x7f966c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f96637fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f966c06af50 0x7f966c06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47090/0 (socket says 172.21.15.111:47090) 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f96637fe700 1 -- 172.21.15.111:0/1315435627 learned_addr learned my addr 172.21.15.111:0/1315435627 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:29.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9670f62700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c0cac50 0x7f966c0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 -- 172.21.15.111:0/1315435627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c0cac50 msgr2=0x7f966c0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c0cac50 0x7f966c0af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 -- 172.21.15.111:0/1315435627 >> v1:172.21.15.111:6789/0 conn(0x7f966c06af50 legacy=0x7f966c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966b7fe700 1 -- 172.21.15.111:0/1315435627 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607928084 0 0) 0x7f9654001110 con 0x7f966c06af50 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 -- 172.21.15.111:0/1315435627 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c06d220 con 0x7f966c0aedc0 2023-12-19T07:59:29.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966bfff700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0aedc0 0x7f966c0af190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f965c0098e0 tx=0x7f965c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b137e16cd3f67d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966b7fe700 1 -- 172.21.15.111:0/1315435627 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c009e80 con 0x7f966c0aedc0 2023-12-19T07:59:29.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966b7fe700 1 -- 172.21.15.111:0/1315435627 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c00b860 con 0x7f966c0aedc0 2023-12-19T07:59:29.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f966b7fe700 1 -- 172.21.15.111:0/1315435627 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c00bbe0 con 0x7f966c0aedc0 2023-12-19T07:59:29.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 -- 172.21.15.111:0/1315435627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0aedc0 msgr2=0x7f966c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.365+0000 7f9671763700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0aedc0 0x7f966c0af190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f965c0098e0 tx=0x7f965c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- 172.21.15.111:0/1315435627 shutdown_connections 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c0cac50 0x7f966c0af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 --2- 172.21.15.111:0/1315435627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0aedc0 0x7f966c0af190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- 172.21.15.111:0/1315435627 >> 172.21.15.111:0/1315435627 conn(0x7f966c0cc760 msgr2=0x7f966c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- 172.21.15.111:0/1315435627 shutdown_connections 2023-12-19T07:59:29.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- 172.21.15.111:0/1315435627 wait complete. 2023-12-19T07:59:29.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 Processor -- start 2023-12-19T07:59:29.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- start start 2023-12-19T07:59:29.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 0x7f966c1479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f966c0aedc0 0x7f966c0c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c15a6f0 con 0x7f966c0cac50 2023-12-19T07:59:29.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c0b3ed0 con 0x7f966c0aedc0 2023-12-19T07:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9671763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c147f10 con 0x7f966c06af50 2023-12-19T07:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.369+0000 7f9670f62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48930/0 (socket says 172.21.15.111:48930) 2023-12-19T07:59:29.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 -- 172.21.15.111:0/1985993323 learned_addr learned my addr 172.21.15.111:0/1985993323 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f96637fe700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f966c0aedc0 0x7f966c0c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f966bfff700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 0x7f966c1479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 -- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 msgr2=0x7f966c1479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 0x7f966c1479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 -- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f966c0aedc0 msgr2=0x7f966c0c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f966c0aedc0 0x7f966c0c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 -- 172.21.15.111:0/1985993323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c009420 con 0x7f966c0cac50 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f966bfff700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 0x7f966c1479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9670f62700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9664009fa0 tx=0x7f9664009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966400ed20 con 0x7f966c0cac50 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966400ee80 con 0x7f966c0cac50 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664014670 con 0x7f966c0cac50 2023-12-19T07:59:29.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c0b32c0 con 0x7f966c0cac50 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c0b3880 con 0x7f966c0cac50 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.373+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c0b4060 con 0x7f966c0cac50 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.377+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f96640149e0 con 0x7f966c0cac50 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.377+0000 7f9668ff9700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 0x7f964c062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.377+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f966407d600 con 0x7f966c0cac50 2023-12-19T07:59:29.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.377+0000 7f966bfff700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 0x7f964c062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.377+0000 7f966bfff700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 0x7f964c062c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f965c0091e0 tx=0x7f965c017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.385+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f966404d670 con 0x7f966c0cac50 2023-12-19T07:59:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.573+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f966c04be40 con 0x7f964c060750 2023-12-19T07:59:29.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9668ff9700 1 -- 172.21.15.111:0/1985993323 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f966c04be40 con 0x7f964c060750 2023-12-19T07:59:29.591 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:29.591 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:45", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 msgr2=0x7f964c062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 0x7f964c062c00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f965c0091e0 tx=0x7f965c017000 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 msgr2=0x7f966c0c3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9664009fa0 tx=0x7f9664009900 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 shutdown_connections 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f964c060750 0x7f964c062c00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f966c0aedc0 0x7f966c0c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f966c0cac50 0x7f966c0c3c90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 --2- 172.21.15.111:0/1985993323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f966c06af50 0x7f966c1479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.593+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 >> 172.21.15.111:0/1985993323 conn(0x7f966c0cc760 msgr2=0x7f966c15c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.597+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 shutdown_connections 2023-12-19T07:59:29.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.597+0000 7f9671763700 1 -- 172.21.15.111:0/1985993323 wait complete. 2023-12-19T07:59:29.613 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:45", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:29.613 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:45', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:29.613 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:29.614 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:29.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:29.614 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-12-19T07:59:29.647 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:29.647 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:29.647 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:29.648 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:29.650 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_01_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_01_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:29.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir7 2023-12-19T07:59:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.861+0000 7f98db523700 1 Processor -- start 2023-12-19T07:59:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.861+0000 7f98db523700 1 -- start start 2023-12-19T07:59:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98db523700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d4158da0 0x7f98d406e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98db523700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98d40caca0 0x7f98d406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98db523700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d415d720 con 0x7f98d406df10 2023-12-19T07:59:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98db523700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d40ca290 con 0x7f98d40caca0 2023-12-19T07:59:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98db523700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40ca4a0 con 0x7f98d4158da0 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98d40caca0 0x7f98d406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98d40caca0 0x7f98d406ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48938/0 (socket says 172.21.15.111:48938) 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 -- 172.21.15.111:0/214185931 learned_addr learned my addr 172.21.15.111:0/214185931 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98d9d20700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f98d406df10 0x7f98d406a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47092/0 (socket says 172.21.15.111:47092) 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98da521700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d4158da0 0x7f98d406e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 -- 172.21.15.111:0/214185931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d4158da0 msgr2=0x7f98d406e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:29.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d4158da0 0x7f98d406e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 -- 172.21.15.111:0/214185931 >> v1:172.21.15.111:6789/0 conn(0x7f98d406df10 legacy=0x7f98d406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 -- 172.21.15.111:0/214185931 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d40ca780 con 0x7f98d40caca0 2023-12-19T07:59:29.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98dad22700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d406ebb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f20 tx=0x7f98cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133f9b179cc030b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98d951f700 1 -- 172.21.15.111:0/214185931 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00e070 con 0x7f98d40caca0 2023-12-19T07:59:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98d951f700 1 -- 172.21.15.111:0/214185931 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc00b9d0 con 0x7f98d40caca0 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.865+0000 7f98d951f700 1 -- 172.21.15.111:0/214185931 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0092e0 con 0x7f98d40caca0 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- 172.21.15.111:0/214185931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 msgr2=0x7f98d406ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d406ebb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009f20 tx=0x7f98cc009840 comp rx=0 tx=0).stop 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- 172.21.15.111:0/214185931 shutdown_connections 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d406ebb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 --2- 172.21.15.111:0/214185931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d4158da0 0x7f98d406e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- 172.21.15.111:0/214185931 >> 172.21.15.111:0/214185931 conn(0x7f98d40cc760 msgr2=0x7f98d40ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- 172.21.15.111:0/214185931 shutdown_connections 2023-12-19T07:59:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- 172.21.15.111:0/214185931 wait complete. 2023-12-19T07:59:29.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 Processor -- start 2023-12-19T07:59:29.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 -- start start 2023-12-19T07:59:29.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.869+0000 7f98db523700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d406df10 0x7f98d41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98d4158da0 0x7f98d415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d4155f20 con 0x7f98d40caca0 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40b3f60 con 0x7f98d4158da0 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d40c7030 con 0x7f98d406df10 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48950/0 (socket says 172.21.15.111:48950) 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 -- 172.21.15.111:0/3393086383 learned_addr learned my addr 172.21.15.111:0/3393086383 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98dad22700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98d4158da0 0x7f98d415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98da521700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d406df10 0x7f98d41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 -- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d406df10 msgr2=0x7f98d41559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d406df10 0x7f98d41559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 -- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98d4158da0 msgr2=0x7f98d415c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:29.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98d4158da0 0x7f98d415c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:29.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 -- 172.21.15.111:0/3393086383 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc009ae0 con 0x7f98d40caca0 2023-12-19T07:59:29.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98d9d20700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c7c0 tx=0x7f98c4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005a70 con 0x7f98d40caca0 2023-12-19T07:59:29.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4005bd0 con 0x7f98d40caca0 2023-12-19T07:59:29.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d40c7310 con 0x7f98d40caca0 2023-12-19T07:59:29.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40070d0 con 0x7f98d40caca0 2023-12-19T07:59:29.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.873+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d415d230 con 0x7f98d40caca0 2023-12-19T07:59:29.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.877+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d40b4110 con 0x7f98d40caca0 2023-12-19T07:59:29.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.877+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f98c4007480 con 0x7f98d40caca0 2023-12-19T07:59:29.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.877+0000 7f98ca7fc700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 0x7f98b8062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:29.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.877+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f98c407c170 con 0x7f98d40caca0 2023-12-19T07:59:29.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.881+0000 7f98da521700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 0x7f98b8062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:29.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.881+0000 7f98da521700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 0x7f98b8062c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d10 tx=0x7f98d00091a0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:29.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:29.885+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f98d40b4110 con 0x7f98d40caca0 2023-12-19T07:59:30.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.073+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98d40b3a70 con 0x7f98b8060780 2023-12-19T07:59:30.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.093+0000 7f98ca7fc700 1 -- 172.21.15.111:0/3393086383 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f98d0009950 con 0x7f98b8060780 2023-12-19T07:59:30.092 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:30.092 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:45", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 msgr2=0x7f98b8062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 0x7f98b8062c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d10 tx=0x7f98d00091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 msgr2=0x7f98d415bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c7c0 tx=0x7f98c4007f70 comp rx=0 tx=0).stop 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 shutdown_connections 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98b8060780 0x7f98b8062c30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98d4158da0 0x7f98d415c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40caca0 0x7f98d415bf60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 --2- 172.21.15.111:0/3393086383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98d406df10 0x7f98d41559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 >> 172.21.15.111:0/3393086383 conn(0x7f98d40cc760 msgr2=0x7f98d40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 shutdown_connections 2023-12-19T07:59:30.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.097+0000 7f98db523700 1 -- 172.21.15.111:0/3393086383 wait complete. 2023-12-19T07:59:30.116 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:45", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:30.116 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:45', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:30.116 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:30.116 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:30.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:30.117 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-12-19T07:59:30.147 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_01_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:30.148 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:30.149 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_01_UTC', 'scheduled-2023-12-19-07_50_00_UTC', 'scheduled-2023-12-19-07_51_01_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_00_UTC', 'scheduled-2023-12-19-07_54_01_UTC', 'scheduled-2023-12-19-07_55_00_UTC', 'scheduled-2023-12-19-07_56_01_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_00_UTC'] 2023-12-19T07:59:30.149 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir8 2023-12-19T07:59:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 Processor -- start 2023-12-19T07:59:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 -- start start 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f10158a10 0x7f5f10158de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f100687f0 0x7f5f1006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f100ad3b0 con 0x7f5f100b02b0 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f10069120 con 0x7f5f10158a10 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.361+0000 7f5f0f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f10068e40 con 0x7f5f100687f0 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f10158a10 0x7f5f10158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f10158a10 0x7f5f10158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48958/0 (socket says 172.21.15.111:48958) 2023-12-19T07:59:30.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 -- 172.21.15.111:0/4218825744 learned_addr learned my addr 172.21.15.111:0/4218825744 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0ed9d700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f100687f0 0x7f5f1006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 -- 172.21.15.111:0/4218825744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f100687f0 msgr2=0x7f5f1006d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f100687f0 0x7f5f1006d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 -- 172.21.15.111:0/4218825744 >> v1:172.21.15.143:6789/0 conn(0x7f5f100b02b0 legacy=0x7f5f100680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 -- 172.21.15.111:0/4218825744 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f100ad7c0 con 0x7f5f10158a10 2023-12-19T07:59:30.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0e59c700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f10158a10 0x7f5f10158de0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5f000099b0 tx=0x7f5f00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561231afb0b0cbc3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0d59a700 1 -- 172.21.15.111:0/4218825744 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0000b860 con 0x7f5f10158a10 2023-12-19T07:59:30.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0d59a700 1 -- 172.21.15.111:0/4218825744 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0000b9c0 con 0x7f5f10158a10 2023-12-19T07:59:30.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0d59a700 1 -- 172.21.15.111:0/4218825744 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0000bd60 con 0x7f5f10158a10 2023-12-19T07:59:30.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 -- 172.21.15.111:0/4218825744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f10158a10 msgr2=0x7f5f10158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f10158a10 0x7f5f10158de0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5f000099b0 tx=0x7f5f00009250 comp rx=0 tx=0).stop 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 -- 172.21.15.111:0/4218825744 shutdown_connections 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f100687f0 0x7f5f1006d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/4218825744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f10158a10 0x7f5f10158de0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 -- 172.21.15.111:0/4218825744 >> 172.21.15.111:0/4218825744 conn(0x7f5f100ce5e0 msgr2=0x7f5f100ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 -- 172.21.15.111:0/4218825744 shutdown_connections 2023-12-19T07:59:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.365+0000 7f5f0f59e700 1 -- 172.21.15.111:0/4218825744 wait complete. 2023-12-19T07:59:30.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 Processor -- start 2023-12-19T07:59:30.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 -- start start 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f10158a10 0x7f5f1015b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f1015b980 0x7f5f101698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1016cc80 con 0x7f5f100687f0 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f10169140 con 0x7f5f1015b980 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.369+0000 7f5f0f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1016ce70 con 0x7f5f10158a10 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48972/0 (socket says 172.21.15.111:48972) 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 -- 172.21.15.111:0/3200041874 learned_addr learned my addr 172.21.15.111:0/3200041874 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0dd9b700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f10158a10 0x7f5f1015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0ed9d700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f1015b980 0x7f5f101698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 -- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f10158a10 msgr2=0x7f5f1015b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f10158a10 0x7f5f1015b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 -- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f1015b980 msgr2=0x7f5f101698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f1015b980 0x7f5f101698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 -- 172.21.15.111:0/3200041874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000094f0 con 0x7f5f100687f0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0e59c700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5f00006fd0 tx=0x7f5f00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00024440 con 0x7f5f100687f0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f000245c0 con 0x7f5f100687f0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00024980 con 0x7f5f100687f0 2023-12-19T07:59:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1016d150 con 0x7f5f100687f0 2023-12-19T07:59:30.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f101693c0 con 0x7f5f100687f0 2023-12-19T07:59:30.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.373+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1004be40 con 0x7f5f100687f0 2023-12-19T07:59:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.377+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f5f0001a420 con 0x7f5f100687f0 2023-12-19T07:59:30.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.377+0000 7f5efe7fc700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 0x7f5ef0062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.377+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f5f0007eb10 con 0x7f5f100687f0 2023-12-19T07:59:30.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.377+0000 7f5f0dd9b700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 0x7f5ef0062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.381+0000 7f5f0dd9b700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 0x7f5ef0062c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f100b38e0 tx=0x7f5f04009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.381+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5f0004eb80 con 0x7f5f100687f0 2023-12-19T07:59:30.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.569+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f1016d5d0 con 0x7f5ef0060750 2023-12-19T07:59:30.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5efe7fc700 1 -- 172.21.15.111:0/3200041874 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f5f1016d5d0 con 0x7f5ef0060750 2023-12-19T07:59:30.586 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:30.587 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:46", "first": "2023-12-19T07:49:01", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:30.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 msgr2=0x7f5ef0062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 0x7f5ef0062c00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f100b38e0 tx=0x7f5f04009040 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 msgr2=0x7f5f1016c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5f00006fd0 tx=0x7f5f00017000 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 shutdown_connections 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5ef0060750 0x7f5ef0062c00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f1015b980 0x7f5f101698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f10158a10 0x7f5f1015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 --2- 172.21.15.111:0/3200041874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100687f0 0x7f5f1016c740 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 >> 172.21.15.111:0/3200041874 conn(0x7f5f100ce5e0 msgr2=0x7f5f100ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:30.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 shutdown_connections 2023-12-19T07:59:30.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.589+0000 7f5f0f59e700 1 -- 172.21.15.111:0/3200041874 wait complete. 2023-12-19T07:59:30.611 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:46", "first": "2023-12-19T07:49:01", "last": "2023-12-19T07:59:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:30.611 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:46', 'first': '2023-12-19T07:49:01', 'last': '2023-12-19T07:59:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:30.611 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:30.612 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:30.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:30.612 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-12-19T07:59:30.643 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_01_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:30.644 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:30.645 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:30.645 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:30.645 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:30.646 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-07_49_00_UTC', 'scheduled-2023-12-19-07_50_01_UTC', 'scheduled-2023-12-19-07_51_00_UTC', 'scheduled-2023-12-19-07_52_00_UTC', 'scheduled-2023-12-19-07_53_01_UTC', 'scheduled-2023-12-19-07_54_00_UTC', 'scheduled-2023-12-19-07_55_01_UTC', 'scheduled-2023-12-19-07_56_00_UTC', 'scheduled-2023-12-19-07_57_00_UTC', 'scheduled-2023-12-19-07_58_00_UTC', 'scheduled-2023-12-19-07_59_01_UTC'] 2023-12-19T07:59:30.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/dir9 2023-12-19T07:59:30.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.869+0000 7ff8c9443700 1 Processor -- start 2023-12-19T07:59:30.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.869+0000 7ff8c9443700 1 -- start start 2023-12-19T07:59:30.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c9443700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8c406b730 0x7ff8c406bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c9443700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 0x7ff8c406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c9443700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c40a9b70 con 0x7ff8c406a580 2023-12-19T07:59:30.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c9443700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c406b0b0 con 0x7ff8c406b730 2023-12-19T07:59:30.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c9443700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c406aaf0 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8bbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8c406a580 0x7ff8c40a9460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47100/0 (socket says 172.21.15.111:47100) 2023-12-19T07:59:30.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8bbfff700 1 -- 172.21.15.111:0/4061345881 learned_addr learned my addr 172.21.15.111:0/4061345881 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:30.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/4061345881 >> v2:172.21.15.111:3300/0 conn(0x7ff8c406b730 0x7ff8c406bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c8c42700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 0x7ff8c406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.873+0000 7ff8c37fe700 1 -- 172.21.15.111:0/4061345881 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374060081 0 0) 0x7ff8c40a9b70 con 0x7ff8c406a580 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c37fe700 1 -- 172.21.15.111:0/4061345881 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b0003570 con 0x7ff8c406a580 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c3fff700 1 -- 172.21.15.111:0/4061345881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 msgr2=0x7ff8c406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 0x7ff8c406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c3fff700 1 -- 172.21.15.111:0/4061345881 >> v1:172.21.15.111:6789/0 conn(0x7ff8c406a580 legacy=0x7ff8c40a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c3fff700 1 -- 172.21.15.111:0/4061345881 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c406add0 con 0x7ff8c406b730 2023-12-19T07:59:30.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c8c42700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 0x7ff8c406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:30.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c406b730 0x7ff8c406bb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40098e0 tx=0x7ff8b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33f9e7d3e81825e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c37fe700 1 -- 172.21.15.111:0/4061345881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4011630 con 0x7ff8c406b730 2023-12-19T07:59:30.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c37fe700 1 -- 172.21.15.111:0/4061345881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4011790 con 0x7ff8c406b730 2023-12-19T07:59:30.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c37fe700 1 -- 172.21.15.111:0/4061345881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4011b30 con 0x7ff8c406b730 2023-12-19T07:59:30.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 -- 172.21.15.111:0/4061345881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c406b730 msgr2=0x7ff8c406bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c406b730 0x7ff8c406bb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40098e0 tx=0x7ff8b40091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 -- 172.21.15.111:0/4061345881 shutdown_connections 2023-12-19T07:59:30.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40ab4b0 0x7ff8c406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 --2- 172.21.15.111:0/4061345881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c406b730 0x7ff8c406bb00 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 -- 172.21.15.111:0/4061345881 >> 172.21.15.111:0/4061345881 conn(0x7ff8c4067bb0 msgr2=0x7ff8c40c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:30.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 -- 172.21.15.111:0/4061345881 shutdown_connections 2023-12-19T07:59:30.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.877+0000 7ff8c9443700 1 -- 172.21.15.111:0/4061345881 wait complete. 2023-12-19T07:59:30.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 Processor -- start 2023-12-19T07:59:30.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 -- start start 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c406a580 0x7ff8c4149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c406b730 0x7ff8c41543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41557c0 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4154910 con 0x7ff8c406b730 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c9443700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4154bb0 con 0x7ff8c406a580 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c8c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c8c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48990/0 (socket says 172.21.15.111:48990) 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.881+0000 7ff8c8c42700 1 -- 172.21.15.111:0/1340880730 learned_addr learned my addr 172.21.15.111:0/1340880730 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:30.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c406a580 0x7ff8c4149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8bbfff700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c406b730 0x7ff8c41543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 -- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c406a580 msgr2=0x7ff8c4149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c406a580 0x7ff8c4149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 -- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c406b730 msgr2=0x7ff8c41543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c406b730 0x7ff8c41543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 -- 172.21.15.111:0/1340880730 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc009a20 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c8c42700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009f20 tx=0x7ff8bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00cb70 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc00f040 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc014480 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b4009420 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c4155170 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.885+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c41499d0 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.889+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7ff8bc003020 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.889+0000 7ff8c0ff9700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 0x7ff8a4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:30.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.889+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7ff8bc07d780 con 0x7ff8c40ab4b0 2023-12-19T07:59:30.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.889+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 0x7ff8a4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:30.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.893+0000 7ff8c3fff700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 0x7ff8a4062c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40091e0 tx=0x7ff8b4017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:30.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:30.893+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff8bc04d6e0 con 0x7ff8c40ab4b0 2023-12-19T07:59:31.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.085+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8c4155990 con 0x7ff8a4060750 2023-12-19T07:59:31.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.101+0000 7ff8c0ff9700 1 -- 172.21.15.111:0/1340880730 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7ff8b4000d00 con 0x7ff8a4060750 2023-12-19T07:59:31.101 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:31.101 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:47", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 msgr2=0x7ff8a4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 0x7ff8a4062c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40091e0 tx=0x7ff8b4017000 comp rx=0 tx=0).stop 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 msgr2=0x7ff8c4155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009f20 tx=0x7ff8bc00b040 comp rx=0 tx=0).stop 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 shutdown_connections 2023-12-19T07:59:31.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8a4060750 0x7ff8a4062c00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c406b730 0x7ff8c41543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8c40ab4b0 0x7ff8c4155c50 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 --2- 172.21.15.111:0/1340880730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c406a580 0x7ff8c4149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 >> 172.21.15.111:0/1340880730 conn(0x7ff8c4067bb0 msgr2=0x7ff8c40a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 shutdown_connections 2023-12-19T07:59:31.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.105+0000 7ff8c9443700 1 -- 172.21.15.111:0/1340880730 wait complete. 2023-12-19T07:59:31.126 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T07:48:47", "first": "2023-12-19T07:49:00", "last": "2023-12-19T07:59:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-12-19T07:59:31.126 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T07:48:47', 'first': '2023-12-19T07:49:00', 'last': '2023-12-19T07:59:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-12-19T07:59:31.126 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T07:59:31.126 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T07:59:31.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir0 1M 2023-12-19T07:59:31.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.297+0000 7f811e678700 1 Processor -- start 2023-12-19T07:59:31.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.297+0000 7f811e678700 1 -- start start 2023-12-19T07:59:31.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f81180cb130 0x7f81180cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811806a6a0 0x7f811806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81180b0ac0 con 0x7f811815c220 2023-12-19T07:59:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811806b030 con 0x7f81180cb130 2023-12-19T07:59:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811806b210 con 0x7f811806a6a0 2023-12-19T07:59:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811cc15700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f811815c220 0x7f811806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47102/0 (socket says 172.21.15.111:47102) 2023-12-19T07:59:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f81180cb130 0x7f81180cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811cc15700 1 -- 172.21.15.111:0/4188307706 learned_addr learned my addr 172.21.15.111:0/4188307706 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f81177fe700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811806a6a0 0x7f811806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8116ffd700 1 -- 172.21.15.111:0/4188307706 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502596858 0 0) 0x7f81180b0ac0 con 0x7f811815c220 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8116ffd700 1 -- 172.21.15.111:0/4188307706 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8100003570 con 0x7f811815c220 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 -- 172.21.15.111:0/4188307706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811806a6a0 msgr2=0x7f811806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811806a6a0 0x7f811806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 -- 172.21.15.111:0/4188307706 >> v1:172.21.15.111:6789/0 conn(0x7f811815c220 legacy=0x7f811806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 -- 172.21.15.111:0/4188307706 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180ca810 con 0x7f81180cb130 2023-12-19T07:59:31.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8117fff700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81180cb130 0x7f81180cb500 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f81080098e0 tx=0x7f81080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd057b4e71cc091 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:31.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8116ffd700 1 -- 172.21.15.111:0/4188307706 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108011630 con 0x7f81180cb130 2023-12-19T07:59:31.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8116ffd700 1 -- 172.21.15.111:0/4188307706 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108011790 con 0x7f81180cb130 2023-12-19T07:59:31.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f8116ffd700 1 -- 172.21.15.111:0/4188307706 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108011b30 con 0x7f81180cb130 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 -- 172.21.15.111:0/4188307706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81180cb130 msgr2=0x7f81180cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.301+0000 7f811e678700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81180cb130 0x7f81180cb500 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f81080098e0 tx=0x7f81080091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- 172.21.15.111:0/4188307706 shutdown_connections 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811806a6a0 0x7f811806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 --2- 172.21.15.111:0/4188307706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81180cb130 0x7f81180cb500 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- 172.21.15.111:0/4188307706 >> 172.21.15.111:0/4188307706 conn(0x7f8118067bb0 msgr2=0x7f81180cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- 172.21.15.111:0/4188307706 shutdown_connections 2023-12-19T07:59:31.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- 172.21.15.111:0/4188307706 wait complete. 2023-12-19T07:59:31.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 Processor -- start 2023-12-19T07:59:31.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- start start 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 0x7f81181561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 0x7f8118142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8118143760 con 0x7f811806a6a0 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8118153c00 con 0x7f81180cb130 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.305+0000 7f811e678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8118153ee0 con 0x7f811815c220 2023-12-19T07:59:31.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49008/0 (socket says 172.21.15.111:49008) 2023-12-19T07:59:31.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 -- 172.21.15.111:0/3482465166 learned_addr learned my addr 172.21.15.111:0/3482465166 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:31.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f811cc15700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 0x7f8118142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f81177fe700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 0x7f81181561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 -- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 msgr2=0x7f8118142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 0x7f8118142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 -- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 msgr2=0x7f81181561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 0x7f81181561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 -- 172.21.15.111:0/3482465166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108009420 con 0x7f811806a6a0 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f811cc15700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 0x7f8118142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f81177fe700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 0x7f81181561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:31.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f8117fff700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f811806f300 tx=0x7f8108017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108011b10 con 0x7f811806a6a0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81080253f0 con 0x7f811806a6a0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81181541c0 con 0x7f811806a6a0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080257b0 con 0x7f811806a6a0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8118154780 con 0x7f811806a6a0 2023-12-19T07:59:31.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.309+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8118071120 con 0x7f811806a6a0 2023-12-19T07:59:31.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.313+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f8108025ba0 con 0x7f811806a6a0 2023-12-19T07:59:31.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.313+0000 7f80fffff700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 0x7f8104062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:31.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.313+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6312+0+0 (secure 0 0 0) 0x7f810807e6e0 con 0x7f811806a6a0 2023-12-19T07:59:31.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.313+0000 7f81177fe700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 0x7f8104062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:31.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.317+0000 7f81177fe700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 0x7f8104062c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f810c009ce0 tx=0x7f810c008040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:31.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.317+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f810804e7f0 con 0x7f811806a6a0 2023-12-19T07:59:31.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.513+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir0", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f81180b4a80 con 0x7f8104060750 2023-12-19T07:59:31.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.661+0000 7f80fffff700 1 -- 172.21.15.111:0/3482465166 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f81180b4a80 con 0x7f8104060750 2023-12-19T07:59:31.659 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-12-19T07:59:31.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 msgr2=0x7f8104062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 0x7f8104062c00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f810c009ce0 tx=0x7f810c008040 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 msgr2=0x7f81180b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f811806f300 tx=0x7f8108017000 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 shutdown_connections 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8104060750 0x7f8104062c00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81180cb130 0x7f81181561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f811815c220 0x7f8118142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 --2- 172.21.15.111:0/3482465166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f811806a6a0 0x7f81180b44e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 >> 172.21.15.111:0/3482465166 conn(0x7f8118067bb0 msgr2=0x7f81180b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 shutdown_connections 2023-12-19T07:59:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:31.665+0000 7f811e678700 1 -- 172.21.15.111:0/3482465166 wait complete. 2023-12-19T07:59:31.689 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-12-19T07:59:31.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:31.689 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-12-19T07:59:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:31.724 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:31.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:31.724 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:31.809 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:31.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:31.809 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:31.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:31.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:31.896 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:31.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:31.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:31.981 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:32.062 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:32.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.062 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_53_01_UTC) 2023-12-19T07:59:32.144 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:32.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.144 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:32.227 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:32.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.228 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:32.310 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:32.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.310 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:32.396 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:32.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.397 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:32.481 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:32.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.481 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_58_01_UTC) 2023-12-19T07:59:32.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:32.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.561 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:32.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:32.646 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-12-19T07:59:32.721 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir1 1M 2023-12-19T07:59:32.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 Processor -- start 2023-12-19T07:59:32.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 -- start start 2023-12-19T07:59:32.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec15d250 0x7f09ec0c8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec0ca560 0x7f09ec06cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec1619f0 con 0x7f09ec0c8870 2023-12-19T07:59:32.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0c8cc0 con 0x7f09ec15d250 2023-12-19T07:59:32.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f2e6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0cab60 con 0x7f09ec0ca560 2023-12-19T07:59:32.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f1e69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec15d250 0x7f09ec0c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f1e69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec15d250 0x7f09ec0c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41000/0 (socket says 172.21.15.111:41000) 2023-12-19T07:59:32.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.945+0000 7f09f1e69700 1 -- 172.21.15.111:0/1349692720 learned_addr learned my addr 172.21.15.111:0/1349692720 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f266a700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec0ca560 0x7f09ec06cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f1e69700 1 -- 172.21.15.111:0/1349692720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec0ca560 msgr2=0x7f09ec06cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f1e69700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec0ca560 0x7f09ec06cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f1e69700 1 -- 172.21.15.111:0/1349692720 >> v1:172.21.15.143:6789/0 conn(0x7f09ec0c8870 legacy=0x7f09ec0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f1e69700 1 -- 172.21.15.111:0/1349692720 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec1610c0 con 0x7f09ec15d250 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f1e69700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec15d250 0x7f09ec0c8330 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f09e80099b0 tx=0x7f09e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc3de86e980de0a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f0e67700 1 -- 172.21.15.111:0/1349692720 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8009ea0 con 0x7f09ec15d250 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f0e67700 1 -- 172.21.15.111:0/1349692720 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e800b760 con 0x7f09ec15d250 2023-12-19T07:59:32.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f0e67700 1 -- 172.21.15.111:0/1349692720 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e800bb00 con 0x7f09ec15d250 2023-12-19T07:59:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 -- 172.21.15.111:0/1349692720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec15d250 msgr2=0x7f09ec0c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec15d250 0x7f09ec0c8330 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f09e80099b0 tx=0x7f09e8009250 comp rx=0 tx=0).stop 2023-12-19T07:59:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 -- 172.21.15.111:0/1349692720 shutdown_connections 2023-12-19T07:59:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec0ca560 0x7f09ec06cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/1349692720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec15d250 0x7f09ec0c8330 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 -- 172.21.15.111:0/1349692720 >> 172.21.15.111:0/1349692720 conn(0x7f09ec06e800 msgr2=0x7f09ec06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 -- 172.21.15.111:0/1349692720 shutdown_connections 2023-12-19T07:59:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.949+0000 7f09f2e6b700 1 -- 172.21.15.111:0/1349692720 wait complete. 2023-12-19T07:59:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 Processor -- start 2023-12-19T07:59:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- start start 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec0ca560 0x7f09ec0c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 0x7f09ec0c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0af4b0 con 0x7f09ec0c8870 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0b1860 con 0x7f09ec0ca560 2023-12-19T07:59:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec151330 con 0x7f09ec15d250 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41006/0 (socket says 172.21.15.111:41006) 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 -- 172.21.15.111:0/2252023425 learned_addr learned my addr 172.21.15.111:0/2252023425 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f266a700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 0x7f09ec0c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1668700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec0ca560 0x7f09ec0c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 -- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 msgr2=0x7f09ec0c1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 0x7f09ec0c1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 -- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec0ca560 msgr2=0x7f09ec0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec0ca560 0x7f09ec0c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 -- 172.21.15.111:0/2252023425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80094f0 con 0x7f09ec0c8870 2023-12-19T07:59:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f266a700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 0x7f09ec0c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f1e69700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f09ec069f00 tx=0x7f09e8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8011e30 con 0x7f09ec0c8870 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e801c460 con 0x7f09ec0c8870 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec155730 con 0x7f09ec0c8870 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e801c820 con 0x7f09ec0c8870 2023-12-19T07:59:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.953+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec1550e0 con 0x7f09ec0c8870 2023-12-19T07:59:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.957+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec04be40 con 0x7f09ec0c8870 2023-12-19T07:59:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.957+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f09e801cc50 con 0x7f09ec0c8870 2023-12-19T07:59:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.957+0000 7f09e1ffb700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 0x7f09d0062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.957+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6592+0+0 (secure 0 0 0) 0x7f09e807ec40 con 0x7f09ec0c8870 2023-12-19T07:59:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.957+0000 7f09f1668700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 0x7f09d0062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.961+0000 7f09f1668700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 0x7f09d0062c00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f09ec157980 tx=0x7f09dc007ea0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:32.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:32.961+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f09e804ec40 con 0x7f09ec0c8870 2023-12-19T07:59:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.149+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir1", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f09ec1613a0 con 0x7f09d0060750 2023-12-19T07:59:33.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.313+0000 7f09e1ffb700 1 -- 172.21.15.111:0/2252023425 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f09ec1613a0 con 0x7f09d0060750 2023-12-19T07:59:33.311 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 msgr2=0x7f09d0062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 0x7f09d0062c00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f09ec157980 tx=0x7f09dc007ea0 comp rx=0 tx=0).stop 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 msgr2=0x7f09ec150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f09ec069f00 tx=0x7f09e8016000 comp rx=0 tx=0).stop 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 shutdown_connections 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d0060750 0x7f09d0062c00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec0ca560 0x7f09ec0c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec15d250 0x7f09ec0c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:33.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 --2- 172.21.15.111:0/2252023425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0c8870 0x7f09ec150df0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:33.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 >> 172.21.15.111:0/2252023425 conn(0x7f09ec06e800 msgr2=0x7f09ec0c6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:33.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 shutdown_connections 2023-12-19T07:59:33.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:33.317+0000 7f09f2e6b700 1 -- 172.21.15.111:0/2252023425 wait complete. 2023-12-19T07:59:33.339 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-12-19T07:59:33.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.339 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-12-19T07:59:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:33.385 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:33.386 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:33.387 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:33.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.387 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:33.473 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:33.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.474 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:33.559 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:33.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.559 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:33.650 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:33.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.650 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:33.735 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:33.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.736 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:33.818 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:33.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.819 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:33.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:33.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:33.908 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_55_01_UTC) 2023-12-19T07:59:34.000 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:34.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:34.000 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:34.085 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:34.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:34.085 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_57_01_UTC) 2023-12-19T07:59:34.167 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:34.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:34.167 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:34.244 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:34.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:34.244 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:34.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:34.326 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-12-19T07:59:34.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir2 1M 2023-12-19T07:59:34.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.629+0000 7fca73363700 1 Processor -- start 2023-12-19T07:59:34.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.629+0000 7fca73363700 1 -- start start 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca6c06df10 0x7fca6c06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca6c0caca0 0x7fca6c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c0ca290 con 0x7fca6c06df10 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6c15d720 con 0x7fca6c0caca0 2023-12-19T07:59:34.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca73363700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c0ca4a0 con 0x7fca6c158da0 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca6c06df10 0x7fca6c06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca6c06df10 0x7fca6c06ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41016/0 (socket says 172.21.15.111:41016) 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 -- 172.21.15.111:0/1486770615 learned_addr learned my addr 172.21.15.111:0/1486770615 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca72361700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca72b62700 1 --2- 172.21.15.111:0/1486770615 >> v2:172.21.15.143:3300/0 conn(0x7fca6c0caca0 0x7fca6c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 -- 172.21.15.111:0/1486770615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 msgr2=0x7fca6c06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 -- 172.21.15.111:0/1486770615 >> v2:172.21.15.143:3300/0 conn(0x7fca6c0caca0 msgr2=0x7fca6c06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 --2- 172.21.15.111:0/1486770615 >> v2:172.21.15.143:3300/0 conn(0x7fca6c0caca0 0x7fca6c06a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 -- 172.21.15.111:0/1486770615 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0ca780 con 0x7fca6c06df10 2023-12-19T07:59:34.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca72361700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.633+0000 7fca71b60700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c06ebb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af650 tx=0x7fca5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c97d506961816a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca7135f700 1 -- 172.21.15.111:0/1486770615 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00b860 con 0x7fca6c06df10 2023-12-19T07:59:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca7135f700 1 -- 172.21.15.111:0/1486770615 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c00b9c0 con 0x7fca6c06df10 2023-12-19T07:59:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca7135f700 1 -- 172.21.15.111:0/1486770615 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00bd60 con 0x7fca6c06df10 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- 172.21.15.111:0/1486770615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 msgr2=0x7fca6c06ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c06ebb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af650 tx=0x7fca5c009250 comp rx=0 tx=0).stop 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- 172.21.15.111:0/1486770615 shutdown_connections 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 --2- 172.21.15.111:0/1486770615 >> v2:172.21.15.143:3300/0 conn(0x7fca6c0caca0 0x7fca6c06a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c06ebb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 --2- 172.21.15.111:0/1486770615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- 172.21.15.111:0/1486770615 >> 172.21.15.111:0/1486770615 conn(0x7fca6c0cc760 msgr2=0x7fca6c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- 172.21.15.111:0/1486770615 shutdown_connections 2023-12-19T07:59:34.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- 172.21.15.111:0/1486770615 wait complete. 2023-12-19T07:59:34.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 Processor -- start 2023-12-19T07:59:34.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.637+0000 7fca73363700 1 -- start start 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c0caca0 0x7fca6c0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c0b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c15be90 con 0x7fca6c06df10 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c15b310 con 0x7fca6c0caca0 2023-12-19T07:59:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c15b5f0 con 0x7fca6c158da0 2023-12-19T07:59:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41032/0 (socket says 172.21.15.111:41032) 2023-12-19T07:59:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 -- 172.21.15.111:0/666507500 learned_addr learned my addr 172.21.15.111:0/666507500 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca71b60700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c0caca0 0x7fca6c0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72b62700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c0b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 -- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 msgr2=0x7fca6c0b5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c0b5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 -- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c0caca0 msgr2=0x7fca6c0b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c0caca0 0x7fca6c0b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 -- 172.21.15.111:0/666507500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0094f0 con 0x7fca6c06df10 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca72361700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca6800d860 tx=0x7fca68008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca680125f0 con 0x7fca6c06df10 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca68012750 con 0x7fca6c06df10 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68012af0 con 0x7fca6c06df10 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6c15b8d0 con 0x7fca6c06df10 2023-12-19T07:59:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.641+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c144a30 con 0x7fca6c06df10 2023-12-19T07:59:34.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.645+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6c04be40 con 0x7fca6c06df10 2023-12-19T07:59:34.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.645+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fca68014430 con 0x7fca6c06df10 2023-12-19T07:59:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.645+0000 7fca627fc700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 0x7fca50062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.645+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6608+0+0 (secure 0 0 0) 0x7fca6807d290 con 0x7fca6c06df10 2023-12-19T07:59:34.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.649+0000 7fca71b60700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 0x7fca50062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:34.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.649+0000 7fca71b60700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 0x7fca50062c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af650 tx=0x7fca5c017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:34.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.653+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca6804d280 con 0x7fca6c06df10 2023-12-19T07:59:34.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.853+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir2", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fca6c0c70f0 con 0x7fca50060750 2023-12-19T07:59:34.989 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-12-19T07:59:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.989+0000 7fca627fc700 1 -- 172.21.15.111:0/666507500 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fca6c0c70f0 con 0x7fca50060750 2023-12-19T07:59:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.993+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 msgr2=0x7fca50062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.993+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 0x7fca50062c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af650 tx=0x7fca5c017000 comp rx=0 tx=0).stop 2023-12-19T07:59:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.993+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 msgr2=0x7fca6c0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.993+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fca6800d860 tx=0x7fca68008040 comp rx=0 tx=0).stop 2023-12-19T07:59:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 shutdown_connections 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca50060750 0x7fca50062c00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c0caca0 0x7fca6c0b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c158da0 0x7fca6c0b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 --2- 172.21.15.111:0/666507500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca6c06df10 0x7fca6c0b46d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 >> 172.21.15.111:0/666507500 conn(0x7fca6c0cc760 msgr2=0x7fca6c06e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 shutdown_connections 2023-12-19T07:59:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:34.997+0000 7fca73363700 1 -- 172.21.15.111:0/666507500 wait complete. 2023-12-19T07:59:35.016 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-12-19T07:59:35.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.016 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:35.066 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:35.067 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:35.068 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:35.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.068 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:35.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:35.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.101 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:35.173 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:35.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.174 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:35.263 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:35.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.264 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:35.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:35.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.348 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:35.433 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:35.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.433 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:35.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:59:35.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:35.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.519 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:35.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T07:59:35.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:35.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.603 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_56_01_UTC) 2023-12-19T07:59:35.712 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:35.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.712 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:35.795 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:35.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.795 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:35.878 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:35.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.878 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:35.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:35.992 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-12-19T07:59:36.011 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir3 1M 2023-12-19T07:59:36.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.233+0000 7f7001c4c700 1 Processor -- start 2023-12-19T07:59:36.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.233+0000 7f7001c4c700 1 -- start start 2023-12-19T07:59:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7001c4c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 0x7f6ffc06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7001c4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7001c4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc15d720 con 0x7f6ffc06df10 2023-12-19T07:59:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7001c4c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0ca290 con 0x7f6ffc158da0 2023-12-19T07:59:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7001c4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0ca4a0 con 0x7f6ffc0caca0 2023-12-19T07:59:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f6ffbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6ffc06df10 0x7f6ffc06a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52168/0 (socket says 172.21.15.111:52168) 2023-12-19T07:59:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f6ffbfff700 1 -- 172.21.15.111:0/1080386112 learned_addr learned my addr 172.21.15.111:0/1080386112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 --2- 172.21.15.111:0/1080386112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7000c4a700 1 --2- 172.21.15.111:0/1080386112 >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 0x7f6ffc06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f6ffb7fe700 1 -- 172.21.15.111:0/1080386112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741137081 0 0) 0x7f6ffc15d720 con 0x7f6ffc06df10 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 -- 172.21.15.111:0/1080386112 >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 msgr2=0x7f6ffc06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 --2- 172.21.15.111:0/1080386112 >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 0x7f6ffc06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 -- 172.21.15.111:0/1080386112 >> v1:172.21.15.111:6789/0 conn(0x7f6ffc06df10 legacy=0x7f6ffc06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 -- 172.21.15.111:0/1080386112 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0ca780 con 0x7f6ffc0caca0 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f7000c4a700 1 --2- 172.21.15.111:0/1080386112 >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 0x7f6ffc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.237+0000 7f700144b700 1 --2- 172.21.15.111:0/1080386112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc06ebb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40098e0 tx=0x7f6ff40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a8b1cf3dc2fda4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f6ffb7fe700 1 -- 172.21.15.111:0/1080386112 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400b640 con 0x7f6ffc0caca0 2023-12-19T07:59:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f6ffb7fe700 1 -- 172.21.15.111:0/1080386112 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff400b7a0 con 0x7f6ffc0caca0 2023-12-19T07:59:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f6ffb7fe700 1 -- 172.21.15.111:0/1080386112 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400bb40 con 0x7f6ffc0caca0 2023-12-19T07:59:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 -- 172.21.15.111:0/1080386112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 msgr2=0x7f6ffc06ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1080386112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc06ebb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40098e0 tx=0x7f6ff40091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 -- 172.21.15.111:0/1080386112 shutdown_connections 2023-12-19T07:59:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1080386112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc06ebb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1080386112 >> v2:172.21.15.143:3300/0 conn(0x7f6ffc158da0 0x7f6ffc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 -- 172.21.15.111:0/1080386112 >> 172.21.15.111:0/1080386112 conn(0x7f6ffc0cc760 msgr2=0x7f6ffc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.241+0000 7f7001c4c700 1 -- 172.21.15.111:0/1080386112 shutdown_connections 2023-12-19T07:59:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.245+0000 7f7001c4c700 1 -- 172.21.15.111:0/1080386112 wait complete. 2023-12-19T07:59:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.245+0000 7f7001c4c700 1 Processor -- start 2023-12-19T07:59:36.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 -- start start 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ffc158da0 0x7f6ffc0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0b30f0 con 0x7f6ffc06df10 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c9770 con 0x7f6ffc158da0 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7001c4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c9a50 con 0x7f6ffc0caca0 2023-12-19T07:59:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41046/0 (socket says 172.21.15.111:41046) 2023-12-19T07:59:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 -- 172.21.15.111:0/1592549426 learned_addr learned my addr 172.21.15.111:0/1592549426 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f6ffbfff700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f700144b700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ffc158da0 0x7f6ffc0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 -- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 msgr2=0x7f6ffc0c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc0c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 -- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ffc158da0 msgr2=0x7f6ffc0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ffc158da0 0x7f6ffc0ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.249+0000 7f7000c4a700 1 -- 172.21.15.111:0/1592549426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0009a20 con 0x7f6ffc06df10 2023-12-19T07:59:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f7000c4a700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c0c0 tx=0x7f6ff000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000ccb0 con 0x7f6ffc06df10 2023-12-19T07:59:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff000f040 con 0x7f6ffc06df10 2023-12-19T07:59:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0014480 con 0x7f6ffc06df10 2023-12-19T07:59:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff4009420 con 0x7f6ffc06df10 2023-12-19T07:59:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc0b2d80 con 0x7f6ffc06df10 2023-12-19T07:59:36.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc04be40 con 0x7f6ffc06df10 2023-12-19T07:59:36.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.253+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f6ff00147f0 con 0x7f6ffc06df10 2023-12-19T07:59:36.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.257+0000 7f6ff8ff9700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 0x7f6fe0062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:36.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.257+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f6ff007de50 con 0x7f6ffc06df10 2023-12-19T07:59:36.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.257+0000 7f6ffbfff700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 0x7f6fe0062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:36.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.257+0000 7f6ffbfff700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 0x7f6fe0062c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc08b350 tx=0x7f6fe8007d30 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.261+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ff004dc80 con 0x7f6ffc06df10 2023-12-19T07:59:36.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.461+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir3", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ffc0b3230 con 0x7f6fe0060750 2023-12-19T07:59:36.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.613+0000 7f6ff8ff9700 1 -- 172.21.15.111:0/1592549426 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f6fe8001420 con 0x7f6fe0060750 2023-12-19T07:59:36.614 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 msgr2=0x7f6fe0062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 0x7f6fe0062c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc08b350 tx=0x7f6fe8007d30 comp rx=0 tx=0).stop 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 msgr2=0x7f6ffc174d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c0c0 tx=0x7f6ff000b040 comp rx=0 tx=0).stop 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 shutdown_connections 2023-12-19T07:59:36.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fe0060750 0x7f6fe0062c00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ffc158da0 0x7f6ffc0ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ffc0caca0 0x7f6ffc0c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 --2- 172.21.15.111:0/1592549426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ffc06df10 0x7f6ffc174d40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 >> 172.21.15.111:0/1592549426 conn(0x7f6ffc0cc760 msgr2=0x7f6ffc0af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 shutdown_connections 2023-12-19T07:59:36.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:36.617+0000 7f7001c4c700 1 -- 172.21.15.111:0/1592549426 wait complete. 2023-12-19T07:59:36.637 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-12-19T07:59:36.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:36.637 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-12-19T07:59:36.678 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:36.678 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:36.678 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:36.678 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:36.678 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:36.679 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:36.680 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:36.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:36.680 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:36.765 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:36.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:36.766 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:36.850 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:36.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:36.850 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_51_01_UTC) 2023-12-19T07:59:36.938 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:36.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:36.938 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:37.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:37.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.030 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:37.112 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:37.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.112 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:37.194 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:37.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.194 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:37.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:37.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.281 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:37.365 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_57_01_UTC 2023-12-19T07:59:37.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.365 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_57_01_UTC) 2023-12-19T07:59:37.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:37.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.454 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:37.538 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:37.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.539 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-12-19-07_59_01_UTC) 2023-12-19T07:59:37.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:37.620 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-12-19T07:59:37.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir4 1M 2023-12-19T07:59:37.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 Processor -- start 2023-12-19T07:59:37.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 -- start start 2023-12-19T07:59:37.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5000addc0 0x7fa5000c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 0x7fa5000b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50006da90 con 0x7fa5000ad9f0 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50006a620 con 0x7fa5000addc0 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa50624c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50006dd30 con 0x7fa50015b140 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa504a49700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5000addc0 0x7fa5000c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa504a49700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5000addc0 0x7fa5000c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41054/0 (socket says 172.21.15.111:41054) 2023-12-19T07:59:37.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.921+0000 7fa504a49700 1 -- 172.21.15.111:0/1847046395 learned_addr learned my addr 172.21.15.111:0/1847046395 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:37.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa505a4b700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 0x7fa5000b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa504a49700 1 -- 172.21.15.111:0/1847046395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 msgr2=0x7fa5000b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:37.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa504a49700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 0x7fa5000b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa504a49700 1 -- 172.21.15.111:0/1847046395 >> v1:172.21.15.111:6789/0 conn(0x7fa5000ad9f0 legacy=0x7fa50006ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa504a49700 1 -- 172.21.15.111:0/1847046395 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50006a8c0 con 0x7fa5000addc0 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa505a4b700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 0x7fa5000b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa504a49700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa5000c8db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00099b0 tx=0x7fa4f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164e38fe79c70d99 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa4f7fff700 1 -- 172.21.15.111:0/1847046395 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000b860 con 0x7fa5000addc0 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa4f7fff700 1 -- 172.21.15.111:0/1847046395 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f000b9c0 con 0x7fa5000addc0 2023-12-19T07:59:37.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa4f7fff700 1 -- 172.21.15.111:0/1847046395 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000bd60 con 0x7fa5000addc0 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 -- 172.21.15.111:0/1847046395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 msgr2=0x7fa5000c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa5000c8db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00099b0 tx=0x7fa4f0009250 comp rx=0 tx=0).stop 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 -- 172.21.15.111:0/1847046395 shutdown_connections 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa50015b140 0x7fa5000b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 --2- 172.21.15.111:0/1847046395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa5000c8db0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 -- 172.21.15.111:0/1847046395 >> 172.21.15.111:0/1847046395 conn(0x7fa500067bb0 msgr2=0x7fa5000c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:37.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 -- 172.21.15.111:0/1847046395 shutdown_connections 2023-12-19T07:59:37.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.925+0000 7fa50624c700 1 -- 172.21.15.111:0/1847046395 wait complete. 2023-12-19T07:59:37.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 Processor -- start 2023-12-19T07:59:37.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 -- start start 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5000ad9f0 0x7fa50013b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa50015b140 0x7fa50013e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c5a50 con 0x7fa5000addc0 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c1d70 con 0x7fa50015b140 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.929+0000 7fa50624c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5000c2050 con 0x7fa5000ad9f0 2023-12-19T07:59:37.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41068/0 (socket says 172.21.15.111:41068) 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 -- 172.21.15.111:0/85153886 learned_addr learned my addr 172.21.15.111:0/85153886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa50524a700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5000ad9f0 0x7fa50013b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa505a4b700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa50015b140 0x7fa50013e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 -- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5000ad9f0 msgr2=0x7fa50013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5000ad9f0 0x7fa50013b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 -- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa50015b140 msgr2=0x7fa50013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa50015b140 0x7fa50013e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 -- 172.21.15.111:0/85153886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f00094f0 con 0x7fa5000addc0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa504a49700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0009290 tx=0x7fa4f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0024440 con 0x7fa5000addc0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f00245a0 con 0x7fa5000addc0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5000c2330 con 0x7fa5000addc0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0024940 con 0x7fa5000addc0 2023-12-19T07:59:37.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.933+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5000c10d0 con 0x7fa5000addc0 2023-12-19T07:59:37.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.937+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5000c5c50 con 0x7fa5000addc0 2023-12-19T07:59:37.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.937+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fa4f001a420 con 0x7fa5000addc0 2023-12-19T07:59:37.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.937+0000 7fa4f57fa700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 0x7fa4e4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:37.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.937+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6944+0+0 (secure 0 0 0) 0x7fa4f007ee50 con 0x7fa5000addc0 2023-12-19T07:59:37.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.937+0000 7fa50524a700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 0x7fa4e4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:37.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.941+0000 7fa50524a700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 0x7fa4e4062c00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc003a80 tx=0x7fa4fc007ec0 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:37.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:37.945+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa5000c5c50 con 0x7fa5000addc0 2023-12-19T07:59:38.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.193+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir4", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5000c5c50 con 0x7fa4e4060750 2023-12-19T07:59:38.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.397+0000 7fa4f57fa700 1 -- 172.21.15.111:0/85153886 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fa5000c5c50 con 0x7fa4e4060750 2023-12-19T07:59:38.395 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-12-19T07:59:38.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 msgr2=0x7fa4e4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 0x7fa4e4062c00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc003a80 tx=0x7fa4fc007ec0 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 msgr2=0x7fa50013e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0009290 tx=0x7fa4f0017000 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 shutdown_connections 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4e4060750 0x7fa4e4062c00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa50015b140 0x7fa50013e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5000addc0 0x7fa50013e190 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 --2- 172.21.15.111:0/85153886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5000ad9f0 0x7fa50013b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 >> 172.21.15.111:0/85153886 conn(0x7fa500067bb0 msgr2=0x7fa5000b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 shutdown_connections 2023-12-19T07:59:38.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:38.401+0000 7fa50624c700 1 -- 172.21.15.111:0/85153886 wait complete. 2023-12-19T07:59:38.418 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-12-19T07:59:38.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.419 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-12-19T07:59:38.459 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:38.459 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:38.460 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:38.461 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:38.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.461 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:38.543 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:38.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.543 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:38.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:38.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.629 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:38.722 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:38.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.722 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:38.807 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:38.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.808 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:38.893 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:38.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.893 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:38.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:38.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:38.976 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:39.058 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:39.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:39.058 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:39.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:39.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:39.141 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:39.229 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:39.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:39.229 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:39.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:39.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:39.313 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:39.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:39.397 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-12-19T07:59:39.470 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir5 1M 2023-12-19T07:59:39.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.693+0000 7fd910ca5700 1 Processor -- start 2023-12-19T07:59:39.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.693+0000 7fd910ca5700 1 -- start start 2023-12-19T07:59:39.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd910ca5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd90c0cb130 0x7fd90c0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd910ca5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0b0ac0 con 0x7fd90c06a6a0 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd910ca5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c06b210 con 0x7fd90c15c930 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd910ca5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c0ca810 con 0x7fd90c0cb130 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd903fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd90c06a6a0 0x7fd90c15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52196/0 (socket says 172.21.15.111:52196) 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd903fff700 1 -- 172.21.15.111:0/1594490829 learned_addr learned my addr 172.21.15.111:0/1594490829 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:39.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd90a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd90c0cb130 0x7fd90c0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:39.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd90a59c700 1 -- 172.21.15.111:0/1594490829 >> v1:172.21.15.143:6789/0 conn(0x7fd90c15c930 legacy=0x7fd90c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:39.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd909d9b700 1 -- 172.21.15.111:0/1594490829 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984199891 0 0) 0x7fd90c0b0ac0 con 0x7fd90c06a6a0 2023-12-19T07:59:39.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd90a59c700 1 -- 172.21.15.111:0/1594490829 >> v1:172.21.15.111:6789/0 conn(0x7fd90c06a6a0 legacy=0x7fd90c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:39.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd90a59c700 1 -- 172.21.15.111:0/1594490829 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0b0190 con 0x7fd90c0cb130 2023-12-19T07:59:39.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd90a59c700 1 --2- 172.21.15.111:0/1594490829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0cb500 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0098e0 tx=0x7fd8fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9042b3e9d58882f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:39.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd909d9b700 1 -- 172.21.15.111:0/1594490829 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc009e80 con 0x7fd90c0cb130 2023-12-19T07:59:39.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd909d9b700 1 -- 172.21.15.111:0/1594490829 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc00b860 con 0x7fd90c0cb130 2023-12-19T07:59:39.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.697+0000 7fd909d9b700 1 -- 172.21.15.111:0/1594490829 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00bbe0 con 0x7fd90c0cb130 2023-12-19T07:59:39.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 -- 172.21.15.111:0/1594490829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 msgr2=0x7fd90c0cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:39.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 --2- 172.21.15.111:0/1594490829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0cb500 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0098e0 tx=0x7fd8fc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:39.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 -- 172.21.15.111:0/1594490829 shutdown_connections 2023-12-19T07:59:39.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 --2- 172.21.15.111:0/1594490829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0cb500 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:39.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 -- 172.21.15.111:0/1594490829 >> 172.21.15.111:0/1594490829 conn(0x7fd90c067bb0 msgr2=0x7fd90c0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:39.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 -- 172.21.15.111:0/1594490829 shutdown_connections 2023-12-19T07:59:39.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.701+0000 7fd910ca5700 1 -- 172.21.15.111:0/1594490829 wait complete. 2023-12-19T07:59:39.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 Processor -- start 2023-12-19T07:59:39.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 -- start start 2023-12-19T07:59:39.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06a6a0 0x7fd90c16c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c15c930 0x7fd90c0b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c1696e0 con 0x7fd90c0cb130 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c16a500 con 0x7fd90c15c930 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd910ca5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c1698f0 con 0x7fd90c06a6a0 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41094/0 (socket says 172.21.15.111:41094) 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 -- 172.21.15.111:0/107242536 learned_addr learned my addr 172.21.15.111:0/107242536 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd90a59c700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06a6a0 0x7fd90c16c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd90ad9d700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c15c930 0x7fd90c0b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 -- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06a6a0 msgr2=0x7fd90c16c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06a6a0 0x7fd90c16c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 -- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c15c930 msgr2=0x7fd90c0b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c15c930 0x7fd90c0b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:39.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 -- 172.21.15.111:0/107242536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009420 con 0x7fd90c0cb130 2023-12-19T07:59:39.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd903fff700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c810 tx=0x7fd8f4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:39.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.705+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4005c00 con 0x7fd90c0cb130 2023-12-19T07:59:39.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4005d60 con 0x7fd90c0cb130 2023-12-19T07:59:39.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c169bd0 con 0x7fd90c0cb130 2023-12-19T07:59:39.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4007340 con 0x7fd90c0cb130 2023-12-19T07:59:39.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c16a190 con 0x7fd90c0cb130 2023-12-19T07:59:39.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c16a690 con 0x7fd90c0cb130 2023-12-19T07:59:39.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.709+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fd8f401b400 con 0x7fd90c0cb130 2023-12-19T07:59:39.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.713+0000 7fd9027fc700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 0x7fd8ec062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:39.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.713+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7116+0+0 (secure 0 0 0) 0x7fd8f407cc20 con 0x7fd90c0cb130 2023-12-19T07:59:39.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.713+0000 7fd90a59c700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 0x7fd8ec062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:39.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.713+0000 7fd90a59c700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 0x7fd8ec062c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0093f0 tx=0x7fd8fc017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:39.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.717+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8f404c9f0 con 0x7fd90c0cb130 2023-12-19T07:59:39.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:39.913+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir5", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd90c15ac40 con 0x7fd8ec060750 2023-12-19T07:59:40.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.065+0000 7fd9027fc700 1 -- 172.21.15.111:0/107242536 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd8fc00baf0 con 0x7fd8ec060750 2023-12-19T07:59:40.064 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-12-19T07:59:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 msgr2=0x7fd8ec062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 0x7fd8ec062c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0093f0 tx=0x7fd8fc017000 comp rx=0 tx=0).stop 2023-12-19T07:59:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 msgr2=0x7fd90c0b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c810 tx=0x7fd8f4007d90 comp rx=0 tx=0).stop 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 shutdown_connections 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8ec060750 0x7fd8ec062c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c15c930 0x7fd90c0b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd90c0cb130 0x7fd90c0b50b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 --2- 172.21.15.111:0/107242536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06a6a0 0x7fd90c16c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 >> 172.21.15.111:0/107242536 conn(0x7fd90c067bb0 msgr2=0x7fd90c0cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 shutdown_connections 2023-12-19T07:59:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:40.069+0000 7fd910ca5700 1 -- 172.21.15.111:0/107242536 wait complete. 2023-12-19T07:59:40.095 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-12-19T07:59:40.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.095 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:40.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:40.137 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:40.137 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:40.138 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:40.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.138 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:40.227 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:40.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.228 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:40.315 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:40.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.316 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:40.403 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:40.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.403 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:40.487 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:40.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.487 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:40.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:40.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.573 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_54_01_UTC) 2023-12-19T07:59:40.652 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:40.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.652 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:40.737 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:40.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.737 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_56_01_UTC) 2023-12-19T07:59:40.819 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:40.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.819 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:40.902 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:40.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.902 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:40.987 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:40.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:40.988 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:41.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:41.074 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-12-19T07:59:41.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir6 1M 2023-12-19T07:59:41.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.369+0000 7f4de23cc700 1 Processor -- start 2023-12-19T07:59:41.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.369+0000 7f4de23cc700 1 -- start start 2023-12-19T07:59:41.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de23cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ddc0af190 0x7f4ddc06af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:41.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de23cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc0cb3f0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de23cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc06d220 con 0x7f4ddc0aedc0 2023-12-19T07:59:41.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de23cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cc140 con 0x7f4ddc0af190 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ddc0af190 0x7f4ddc06af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ddc0af190 0x7f4ddc06af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41098/0 (socket says 172.21.15.111:41098) 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 -- 172.21.15.111:0/1256800461 learned_addr learned my addr 172.21.15.111:0/1256800461 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de1bcb700 1 --1- 172.21.15.111:0/1256800461 >> v1:172.21.15.111:6789/0 conn(0x7f4ddc0cac50 0x7f4ddc0af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52208/0 (socket says 172.21.15.111:52208) 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 -- 172.21.15.111:0/1256800461 >> v1:172.21.15.143:6789/0 conn(0x7f4ddc0aedc0 legacy=0x7f4ddc06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 -- 172.21.15.111:0/1256800461 >> v1:172.21.15.111:6789/0 conn(0x7f4ddc0cac50 legacy=0x7f4ddc0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 -- 172.21.15.111:0/1256800461 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc06c8f0 con 0x7f4ddc0af190 2023-12-19T07:59:41.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4de0bc9700 1 --2- 172.21.15.111:0/1256800461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0af190 0x7f4ddc06af50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0099b0 tx=0x7f4dcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc80fdec9f3cce1f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:41.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4dd3fff700 1 -- 172.21.15.111:0/1256800461 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00b820 con 0x7f4ddc0af190 2023-12-19T07:59:41.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4dd3fff700 1 -- 172.21.15.111:0/1256800461 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc00b980 con 0x7f4ddc0af190 2023-12-19T07:59:41.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.373+0000 7f4dd3fff700 1 -- 172.21.15.111:0/1256800461 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00bd20 con 0x7f4ddc0af190 2023-12-19T07:59:41.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- 172.21.15.111:0/1256800461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0af190 msgr2=0x7f4ddc06af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 --2- 172.21.15.111:0/1256800461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0af190 0x7f4ddc06af50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0099b0 tx=0x7f4dcc009250 comp rx=0 tx=0).stop 2023-12-19T07:59:41.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- 172.21.15.111:0/1256800461 shutdown_connections 2023-12-19T07:59:41.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 --2- 172.21.15.111:0/1256800461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0af190 0x7f4ddc06af50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- 172.21.15.111:0/1256800461 >> 172.21.15.111:0/1256800461 conn(0x7f4ddc0cc760 msgr2=0x7f4ddc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:41.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- 172.21.15.111:0/1256800461 shutdown_connections 2023-12-19T07:59:41.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- 172.21.15.111:0/1256800461 wait complete. 2023-12-19T07:59:41.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 Processor -- start 2023-12-19T07:59:41.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.377+0000 7f4de23cc700 1 -- start start 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ddc0aedc0 0x7f4ddc0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ddc0af190 0x7f4ddc0c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0af610 con 0x7f4ddc0cac50 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0c4f20 con 0x7f4ddc0af190 2023-12-19T07:59:41.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0c58f0 con 0x7f4ddc0aedc0 2023-12-19T07:59:41.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:41.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41108/0 (socket says 172.21.15.111:41108) 2023-12-19T07:59:41.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 -- 172.21.15.111:0/3678034051 learned_addr learned my addr 172.21.15.111:0/3678034051 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de13ca700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ddc0aedc0 0x7f4ddc0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de0bc9700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ddc0af190 0x7f4ddc0c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 -- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ddc0aedc0 msgr2=0x7f4ddc0c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ddc0aedc0 0x7f4ddc0c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 -- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ddc0af190 msgr2=0x7f4ddc0c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ddc0af190 0x7f4ddc0c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 -- 172.21.15.111:0/3678034051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0094f0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de1bcb700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400cd00 tx=0x7f4dd4001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd400ce60 con 0x7f4ddc0cac50 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd4005df0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc0c6da0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd400ab00 con 0x7f4ddc0cac50 2023-12-19T07:59:41.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.381+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc0c70c0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.385+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc0c5090 con 0x7f4ddc0cac50 2023-12-19T07:59:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.385+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f4dd40070f0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.385+0000 7f4dd17fa700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 0x7f4dc0062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:41.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.389+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7116+0+0 (secure 0 0 0) 0x7f4dd407d210 con 0x7f4ddc0cac50 2023-12-19T07:59:41.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.389+0000 7f4de13ca700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 0x7f4dc0062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.389+0000 7f4de13ca700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 0x7f4dc0062c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009af0 tx=0x7f4dd8009060 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:41.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.393+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4dd404cfe0 con 0x7f4ddc0cac50 2023-12-19T07:59:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.581+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir6", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ddc04be40 con 0x7f4dc0060750 2023-12-19T07:59:41.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.729+0000 7f4dd17fa700 1 -- 172.21.15.111:0/3678034051 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f4ddc04be40 con 0x7f4dc0060750 2023-12-19T07:59:41.730 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-12-19T07:59:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 msgr2=0x7f4dc0062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 0x7f4dc0062c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009af0 tx=0x7f4dd8009060 comp rx=0 tx=0).stop 2023-12-19T07:59:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 msgr2=0x7f4ddc0b4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400cd00 tx=0x7f4dd4001b10 comp rx=0 tx=0).stop 2023-12-19T07:59:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 shutdown_connections 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4dc0060750 0x7f4dc0062c00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ddc0af190 0x7f4ddc0c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ddc0cac50 0x7f4ddc0b4d10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 --2- 172.21.15.111:0/3678034051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ddc0aedc0 0x7f4ddc0c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 >> 172.21.15.111:0/3678034051 conn(0x7f4ddc0cc760 msgr2=0x7f4ddc0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 shutdown_connections 2023-12-19T07:59:41.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:41.733+0000 7f4de23cc700 1 -- 172.21.15.111:0/3678034051 wait complete. 2023-12-19T07:59:41.756 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-12-19T07:59:41.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:41.756 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_01_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:41.796 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:41.797 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:41.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:41.798 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:41.884 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:41.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:41.885 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:41.968 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:41.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:41.968 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:42.046 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_52_01_UTC 2023-12-19T07:59:42.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.047 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_52_01_UTC) 2023-12-19T07:59:42.129 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:42.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.129 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:42.214 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:42.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.214 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:42.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:42.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.299 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:42.382 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:42.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.382 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:42.473 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:42.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.473 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:42.554 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:42.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.554 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:42.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:42.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.642 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:42.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:42.729 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-12-19T07:59:42.801 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir7 1M 2023-12-19T07:59:43.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 Processor -- start 2023-12-19T07:59:43.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 -- start start 2023-12-19T07:59:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f006af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69f00cac50 0x7f69f00af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00cb3f0 con 0x7f69f00aedc0 2023-12-19T07:59:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00cc140 con 0x7f69f00cac50 2023-12-19T07:59:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f528e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00af5e0 con 0x7f69f00af190 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69effff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f69f00aedc0 0x7f69f006d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41456/0 (socket says 172.21.15.111:41456) 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69effff700 1 -- 172.21.15.111:0/1715598166 learned_addr learned my addr 172.21.15.111:0/1715598166 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69f4a8d700 1 --2- 172.21.15.111:0/1715598166 >> v2:172.21.15.143:3300/0 conn(0x7f69f00cac50 0x7f69f00af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.025+0000 7f69ef7fe700 1 --2- 172.21.15.111:0/1715598166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f006af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69eeffd700 1 -- 172.21.15.111:0/1715598166 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152604328 0 0) 0x7f69f00cb3f0 con 0x7f69f00aedc0 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69eeffd700 1 -- 172.21.15.111:0/1715598166 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d8003570 con 0x7f69f00aedc0 2023-12-19T07:59:43.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69ef7fe700 1 -- 172.21.15.111:0/1715598166 >> v2:172.21.15.143:3300/0 conn(0x7f69f00cac50 msgr2=0x7f69f00af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69ef7fe700 1 --2- 172.21.15.111:0/1715598166 >> v2:172.21.15.143:3300/0 conn(0x7f69f00cac50 0x7f69f00af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69ef7fe700 1 -- 172.21.15.111:0/1715598166 >> v1:172.21.15.111:6789/0 conn(0x7f69f00aedc0 legacy=0x7f69f006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69ef7fe700 1 -- 172.21.15.111:0/1715598166 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f006d220 con 0x7f69f00af190 2023-12-19T07:59:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69ef7fe700 1 --2- 172.21.15.111:0/1715598166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f006af50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f69e40098e0 tx=0x7f69e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73331d98f4ed97a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69eeffd700 1 -- 172.21.15.111:0/1715598166 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011630 con 0x7f69f00af190 2023-12-19T07:59:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69eeffd700 1 -- 172.21.15.111:0/1715598166 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4011790 con 0x7f69f00af190 2023-12-19T07:59:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69eeffd700 1 -- 172.21.15.111:0/1715598166 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4011b30 con 0x7f69f00af190 2023-12-19T07:59:43.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 -- 172.21.15.111:0/1715598166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 msgr2=0x7f69f006af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 --2- 172.21.15.111:0/1715598166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f006af50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f69e40098e0 tx=0x7f69e40091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 -- 172.21.15.111:0/1715598166 shutdown_connections 2023-12-19T07:59:43.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 --2- 172.21.15.111:0/1715598166 >> v2:172.21.15.143:3300/0 conn(0x7f69f00cac50 0x7f69f00af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 --2- 172.21.15.111:0/1715598166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f006af50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.029+0000 7f69f528e700 1 -- 172.21.15.111:0/1715598166 >> 172.21.15.111:0/1715598166 conn(0x7f69f00cc760 msgr2=0x7f69f00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:43.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.033+0000 7f69f528e700 1 -- 172.21.15.111:0/1715598166 shutdown_connections 2023-12-19T07:59:43.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.033+0000 7f69f528e700 1 -- 172.21.15.111:0/1715598166 wait complete. 2023-12-19T07:59:43.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.033+0000 7f69f528e700 1 Processor -- start 2023-12-19T07:59:43.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 -- start start 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 0x7f69f0170210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f00b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 0x7f69f00c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00c45f0 con 0x7f69f00cac50 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00c48d0 con 0x7f69f00aedc0 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f528e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00c4bb0 con 0x7f69f00af190 2023-12-19T07:59:43.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 0x7f69f0170210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f00b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 0x7f69f0170210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:59378/0 (socket says 172.21.15.111:59378) 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69effff700 1 -- 172.21.15.111:0/1741399622 learned_addr learned my addr 172.21.15.111:0/1741399622 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 0x7f69f00c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 -- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 msgr2=0x7f69f00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f00b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 -- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 msgr2=0x7f69f0170210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 0x7f69f0170210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 -- 172.21.15.111:0/1741399622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4009420 con 0x7f69f00cac50 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.037+0000 7f69f4a8d700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 0x7f69f00c8e50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f69e800ecc0 tx=0x7f69e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e800eed0 con 0x7f69f00cac50 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e800f040 con 0x7f69f00cac50 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e800cd80 con 0x7f69f00cac50 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00c4e90 con 0x7f69f00cac50 2023-12-19T07:59:43.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f0170750 con 0x7f69f00cac50 2023-12-19T07:59:43.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f00c5370 con 0x7f69f00cac50 2023-12-19T07:59:43.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.041+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f69e8014690 con 0x7f69f00cac50 2023-12-19T07:59:43.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.045+0000 7f69d7fff700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 0x7f69dc062d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:43.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.045+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7168+0+0 (secure 0 0 0) 0x7f69e807dd70 con 0x7f69f00cac50 2023-12-19T07:59:43.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.045+0000 7f69effff700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 0x7f69dc062d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:43.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.045+0000 7f69effff700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 0x7f69dc062d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69e0001910 tx=0x7f69e0009040 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:43.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.053+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f69e804da20 con 0x7f69f00cac50 2023-12-19T07:59:43.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.373+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir7", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f69f004be40 con 0x7f69dc060850 2023-12-19T07:59:43.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.517+0000 7f69d7fff700 1 -- 172.21.15.111:0/1741399622 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f69f004be40 con 0x7f69dc060850 2023-12-19T07:59:43.516 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-12-19T07:59:43.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 msgr2=0x7f69dc062d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 0x7f69dc062d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69e0001910 tx=0x7f69e0009040 comp rx=0 tx=0).stop 2023-12-19T07:59:43.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 msgr2=0x7f69f00c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 0x7f69f00c8e50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f69e800ecc0 tx=0x7f69e800b040 comp rx=0 tx=0).stop 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 shutdown_connections 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69dc060850 0x7f69dc062d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f00cac50 0x7f69f00c8e50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69f00af190 0x7f69f00b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 --2- 172.21.15.111:0/1741399622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69f00aedc0 0x7f69f0170210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 >> 172.21.15.111:0/1741399622 conn(0x7f69f00cc760 msgr2=0x7f69f006e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 shutdown_connections 2023-12-19T07:59:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:43.521+0000 7f69f528e700 1 -- 172.21.15.111:0/1741399622 wait complete. 2023-12-19T07:59:43.542 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-12-19T07:59:43.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.542 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-12-19T07:59:43.580 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:43.580 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:43.581 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:43.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:43.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.582 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:43.667 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:43.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.667 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:43.747 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:43.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.747 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:43.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:43.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.829 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:43.915 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:43.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:43.916 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:44.011 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:44.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.011 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:44.041 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:44.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.041 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:44.123 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:44.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.124 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_56_01_UTC) 2023-12-19T07:59:44.208 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:44.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.209 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:44.293 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_58_01_UTC 2023-12-19T07:59:44.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.293 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_58_01_UTC) 2023-12-19T07:59:44.378 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:44.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.378 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:44.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:44.462 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-12-19T07:59:44.538 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir8 1M 2023-12-19T07:59:44.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.753+0000 7f06f1b85700 1 Processor -- start 2023-12-19T07:59:44.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.753+0000 7f06f1b85700 1 -- start start 2023-12-19T07:59:44.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06f1b85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06ec0cb130 0x7f06ec0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06f1b85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06ec15c930 0x7f06ec06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06f1b85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec0b0ac0 con 0x7f06ec06a6a0 2023-12-19T07:59:44.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06f1b85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec06b030 con 0x7f06ec15c930 2023-12-19T07:59:44.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06f1b85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec06b210 con 0x7f06ec0cb130 2023-12-19T07:59:44.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eaffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f06ec06a6a0 0x7f06ec15c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41460/0 (socket says 172.21.15.111:41460) 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eaffd700 1 -- 172.21.15.111:0/2984783215 learned_addr learned my addr 172.21.15.111:0/2984783215 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/2984783215 >> v2:172.21.15.143:3300/0 conn(0x7f06ec0cb130 0x7f06ec0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06ebfff700 1 --2- 172.21.15.111:0/2984783215 >> v2:172.21.15.111:3300/0 conn(0x7f06ec15c930 0x7f06ec06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06ea7fc700 1 -- 172.21.15.111:0/2984783215 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694624599 0 0) 0x7f06ec0b0ac0 con 0x7f06ec06a6a0 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06ea7fc700 1 -- 172.21.15.111:0/2984783215 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d4003570 con 0x7f06ec06a6a0 2023-12-19T07:59:44.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 -- 172.21.15.111:0/2984783215 >> v2:172.21.15.111:3300/0 conn(0x7f06ec15c930 msgr2=0x7f06ec06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:44.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/2984783215 >> v2:172.21.15.111:3300/0 conn(0x7f06ec15c930 0x7f06ec06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 -- 172.21.15.111:0/2984783215 >> v1:172.21.15.111:6789/0 conn(0x7f06ec06a6a0 legacy=0x7f06ec15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:44.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 -- 172.21.15.111:0/2984783215 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0ca810 con 0x7f06ec0cb130 2023-12-19T07:59:44.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.757+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/2984783215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec0cb130 0x7f06ec0cb500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0694e0 tx=0x7f06dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30973e8bf0ba6d9c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:44.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06ea7fc700 1 -- 172.21.15.111:0/2984783215 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc011630 con 0x7f06ec0cb130 2023-12-19T07:59:44.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06ea7fc700 1 -- 172.21.15.111:0/2984783215 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc0117b0 con 0x7f06ec0cb130 2023-12-19T07:59:44.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06ea7fc700 1 -- 172.21.15.111:0/2984783215 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc011b70 con 0x7f06ec0cb130 2023-12-19T07:59:44.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 -- 172.21.15.111:0/2984783215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec0cb130 msgr2=0x7f06ec0cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:44.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 --2- 172.21.15.111:0/2984783215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec0cb130 0x7f06ec0cb500 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0694e0 tx=0x7f06dc0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 -- 172.21.15.111:0/2984783215 shutdown_connections 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 --2- 172.21.15.111:0/2984783215 >> v2:172.21.15.111:3300/0 conn(0x7f06ec15c930 0x7f06ec06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 --2- 172.21.15.111:0/2984783215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec0cb130 0x7f06ec0cb500 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 -- 172.21.15.111:0/2984783215 >> 172.21.15.111:0/2984783215 conn(0x7f06ec067bb0 msgr2=0x7f06ec0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 -- 172.21.15.111:0/2984783215 shutdown_connections 2023-12-19T07:59:44.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.761+0000 7f06f1b85700 1 -- 172.21.15.111:0/2984783215 wait complete. 2023-12-19T07:59:44.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 Processor -- start 2023-12-19T07:59:44.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 -- start start 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06ec06a6a0 0x7f06ec1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec15c930 0x7f06ec0b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec169440 con 0x7f06ec0cb130 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec169710 con 0x7f06ec15c930 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06f1b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec159640 con 0x7f06ec06a6a0 2023-12-19T07:59:44.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06eaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06eaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37782/0 (socket says 172.21.15.111:37782) 2023-12-19T07:59:44.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06eaffd700 1 -- 172.21.15.111:0/1161238025 learned_addr learned my addr 172.21.15.111:0/1161238025 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:44.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06ec06a6a0 0x7f06ec1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.765+0000 7f06ebfff700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec15c930 0x7f06ec0b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 -- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06ec06a6a0 msgr2=0x7f06ec1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06ec06a6a0 0x7f06ec1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 -- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec15c930 msgr2=0x7f06ec0b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec15c930 0x7f06ec0b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 -- 172.21.15.111:0/1161238025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc009420 con 0x7f06ec0cb130 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06eaffd700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06e000c590 tx=0x7f06e0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0005cd0 con 0x7f06ec0cb130 2023-12-19T07:59:44.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e0005e50 con 0x7f06ec0cb130 2023-12-19T07:59:44.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec159920 con 0x7f06ec0cb130 2023-12-19T07:59:44.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0007320 con 0x7f06ec0cb130 2023-12-19T07:59:44.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec155e00 con 0x7f06ec0cb130 2023-12-19T07:59:44.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f06e001b3b0 con 0x7f06ec0cb130 2023-12-19T07:59:44.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.769+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec159e20 con 0x7f06ec0cb130 2023-12-19T07:59:44.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.773+0000 7f06d37fe700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 0x7f06d8062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:44.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.773+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7196+0+0 (secure 0 0 0) 0x7f06e007c7d0 con 0x7f06ec0cb130 2023-12-19T07:59:44.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.773+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 0x7f06d8062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:44.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.773+0000 7f06eb7fe700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 0x7f06d8062c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0694e0 tx=0x7f06dc017000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:44.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.781+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f06e004c570 con 0x7f06ec0cb130 2023-12-19T07:59:44.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:44.969+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir8", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f06ec071120 con 0x7f06d8060750 2023-12-19T07:59:45.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.125+0000 7f06d37fe700 1 -- 172.21.15.111:0/1161238025 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f06ec071120 con 0x7f06d8060750 2023-12-19T07:59:45.125 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 msgr2=0x7f06d8062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 0x7f06d8062c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0694e0 tx=0x7f06dc017000 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 msgr2=0x7f06ec0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06e000c590 tx=0x7f06e0007e00 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 shutdown_connections 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d8060750 0x7f06d8062c00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06ec15c930 0x7f06ec0b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ec0cb130 0x7f06ec0b4e50 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 --2- 172.21.15.111:0/1161238025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06ec06a6a0 0x7f06ec1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 >> 172.21.15.111:0/1161238025 conn(0x7f06ec067bb0 msgr2=0x7f06ec0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 shutdown_connections 2023-12-19T07:59:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:45.129+0000 7f06f1b85700 1 -- 172.21.15.111:0/1161238025 wait complete. 2023-12-19T07:59:45.150 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-12-19T07:59:45.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.150 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-12-19T07:59:45.194 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_01_UTC 2023-12-19T07:59:45.194 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:45.195 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:45.196 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_49_01_UTC 2023-12-19T07:59:45.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.196 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_49_01_UTC) 2023-12-19T07:59:45.279 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_50_00_UTC 2023-12-19T07:59:45.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.279 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_50_00_UTC) 2023-12-19T07:59:45.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_51_01_UTC 2023-12-19T07:59:45.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.362 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_51_01_UTC) 2023-12-19T07:59:45.450 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:45.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.450 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:45.537 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_53_00_UTC 2023-12-19T07:59:45.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.537 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_53_00_UTC) 2023-12-19T07:59:45.622 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_54_01_UTC 2023-12-19T07:59:45.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.622 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_54_01_UTC) 2023-12-19T07:59:45.705 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_55_00_UTC 2023-12-19T07:59:45.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.706 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_55_00_UTC) 2023-12-19T07:59:45.797 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_56_01_UTC 2023-12-19T07:59:45.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.797 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_56_01_UTC) 2023-12-19T07:59:45.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:45.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.877 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:45.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:45.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:45.965 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:46.050 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_59_00_UTC 2023-12-19T07:59:46.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:46.050 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-12-19-07_59_00_UTC) 2023-12-19T07:59:46.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:46.133 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-12-19T07:59:46.203 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/dir9 1M 2023-12-19T07:59:46.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.429+0000 7fdc4c22e700 1 Processor -- start 2023-12-19T07:59:46.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.433+0000 7fdc4c22e700 1 -- start start 2023-12-19T07:59:46.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.433+0000 7fdc4c22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc440cb500 0x7fdc4415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.433+0000 7fdc4c22e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc440b0ac0 con 0x7fdc440cb130 2023-12-19T07:59:46.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc4c22e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4406b210 con 0x7fdc4415c760 2023-12-19T07:59:46.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc4c22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4415cbe0 con 0x7fdc440cb500 2023-12-19T07:59:46.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc440cb130 0x7fdc4406a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41472/0 (socket says 172.21.15.111:41472) 2023-12-19T07:59:46.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 -- 172.21.15.111:0/3105867683 learned_addr learned my addr 172.21.15.111:0/3105867683 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:46.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc497c9700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc440cb500 0x7fdc4415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529964888 0 0) 0x7fdc440b0ac0 con 0x7fdc440cb130 2023-12-19T07:59:46.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30003570 con 0x7fdc440cb130 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839034490 0 0) 0x7fdc4406b210 con 0x7fdc4415c760 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc440b0ac0 con 0x7fdc4415c760 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3150335242 0 0) 0x7fdc30003570 con 0x7fdc440cb130 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc440cb500 msgr2=0x7fdc4415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc440cb500 0x7fdc4415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 >> v1:172.21.15.143:6789/0 conn(0x7fdc4415c760 legacy=0x7fdc440b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc440ca810 con 0x7fdc440cb130 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fdc40001470 con 0x7fdc440cb130 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 >> v1:172.21.15.111:6789/0 conn(0x7fdc440cb130 legacy=0x7fdc4406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc30003c20 0x7fdc30004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc30004810 0x7fdc30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc300051c0 0x7fdc3000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4415cbe0 con 0x7fdc30003c20 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3000c020 con 0x7fdc30004810 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3000c1e0 con 0x7fdc300051c0 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc497c9700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc440cb500 0x7fdc4415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc30003c20 0x7fdc30004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc4a7cb700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc300051c0 0x7fdc3000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc497c9700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc30004810 0x7fdc30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 -- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc300051c0 msgr2=0x7fdc3000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc300051c0 0x7fdc3000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 -- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc30004810 msgr2=0x7fdc30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc30004810 0x7fdc30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 -- 172.21.15.111:0/3105867683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30003570 con 0x7fdc30003c20 2023-12-19T07:59:46.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc49fca700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc30003c20 0x7fdc30004010 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000c2d0 tx=0x7fdc40001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b397730969a7e2a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:46.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4000cc00 con 0x7fdc30003c20 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4000cd60 con 0x7fdc30003c20 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.437+0000 7fdc48fc8700 1 -- 172.21.15.111:0/3105867683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40005b70 con 0x7fdc30003c20 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- 172.21.15.111:0/3105867683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc30003c20 msgr2=0x7fdc30004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/3105867683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc30003c20 0x7fdc30004010 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000c2d0 tx=0x7fdc40001e10 comp rx=0 tx=0).stop 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4a7cb700 1 -- 172.21.15.111:0/3105867683 reap_dead start 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- 172.21.15.111:0/3105867683 shutdown_connections 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- 172.21.15.111:0/3105867683 >> 172.21.15.111:0/3105867683 conn(0x7fdc44067bb0 msgr2=0x7fdc440cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:46.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- 172.21.15.111:0/3105867683 shutdown_connections 2023-12-19T07:59:46.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- 172.21.15.111:0/3105867683 wait complete. 2023-12-19T07:59:46.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 Processor -- start 2023-12-19T07:59:46.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- start start 2023-12-19T07:59:46.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc44156ea0 0x7fdc440c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4406aa30 0x7fdc440b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4416aed0 con 0x7fdc4416a900 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440c50f0 con 0x7fdc4406aa30 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc4c22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4406aea0 con 0x7fdc44156ea0 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37794/0 (socket says 172.21.15.111:37794) 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 -- 172.21.15.111:0/614040821 learned_addr learned my addr 172.21.15.111:0/614040821 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc49fca700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc44156ea0 0x7fdc440c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 -- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc44156ea0 msgr2=0x7fdc440c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc44156ea0 0x7fdc440c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 -- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4406aa30 msgr2=0x7fdc440b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4406aa30 0x7fdc440b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.441+0000 7fdc497c9700 1 -- 172.21.15.111:0/614040821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc440b3ab0 con 0x7fdc4416a900 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc497c9700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdc34011910 tx=0x7fdc34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34011e50 con 0x7fdc4416a900 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3400f880 con 0x7fdc4416a900 2023-12-19T07:59:46.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc440c5970 con 0x7fdc4416a900 2023-12-19T07:59:46.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3400fc40 con 0x7fdc4416a900 2023-12-19T07:59:46.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc440b3d00 con 0x7fdc4416a900 2023-12-19T07:59:46.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4406cc80 con 0x7fdc4416a900 2023-12-19T07:59:46.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.445+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7fdc34015460 con 0x7fdc4416a900 2023-12-19T07:59:46.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.449+0000 7fdc39ffb700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 0x7fdc28062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:46.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.449+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7196+0+0 (secure 0 0 0) 0x7fdc3407db10 con 0x7fdc4416a900 2023-12-19T07:59:46.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.449+0000 7fdc49fca700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 0x7fdc28062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:46.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.449+0000 7fdc49fca700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 0x7fdc28062c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc44063490 tx=0x7fdc40015000 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:46.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.453+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc4406cc80 con 0x7fdc4416a900 2023-12-19T07:59:46.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.649+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/dir9", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc4406cc80 con 0x7fdc28060780 2023-12-19T07:59:46.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.797+0000 7fdc39ffb700 1 -- 172.21.15.111:0/614040821 <== mgr.5308 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fdc4406cc80 con 0x7fdc28060780 2023-12-19T07:59:46.795 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 msgr2=0x7fdc28062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 0x7fdc28062c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc44063490 tx=0x7fdc40015000 comp rx=0 tx=0).stop 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 msgr2=0x7fdc4406a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdc34011910 tx=0x7fdc34008040 comp rx=0 tx=0).stop 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 shutdown_connections 2023-12-19T07:59:46.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc28060780 0x7fdc28062c30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4406aa30 0x7fdc440b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc4416a900 0x7fdc4406a4f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 --2- 172.21.15.111:0/614040821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc44156ea0 0x7fdc440c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 >> 172.21.15.111:0/614040821 conn(0x7fdc44067bb0 msgr2=0x7fdc440b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 shutdown_connections 2023-12-19T07:59:46.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:46.801+0000 7fdc4c22e700 1 -- 172.21.15.111:0/614040821 wait complete. 2023-12-19T07:59:46.820 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-12-19T07:59:46.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:46.820 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-12-19T07:59:46.861 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:46.861 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_50_01_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:46.862 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:46.863 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_49_00_UTC 2023-12-19T07:59:46.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:46.863 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_49_00_UTC) 2023-12-19T07:59:46.955 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_50_01_UTC 2023-12-19T07:59:46.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:46.956 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_50_01_UTC) 2023-12-19T07:59:47.042 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_51_00_UTC 2023-12-19T07:59:47.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.042 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_51_00_UTC) 2023-12-19T07:59:47.128 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_52_00_UTC 2023-12-19T07:59:47.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.128 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_52_00_UTC) 2023-12-19T07:59:47.218 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_53_01_UTC 2023-12-19T07:59:47.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.218 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_53_01_UTC) 2023-12-19T07:59:47.305 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_54_00_UTC 2023-12-19T07:59:47.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.306 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_54_00_UTC) 2023-12-19T07:59:47.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_55_01_UTC 2023-12-19T07:59:47.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.395 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_55_01_UTC) 2023-12-19T07:59:47.476 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_56_00_UTC 2023-12-19T07:59:47.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.476 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_56_00_UTC) 2023-12-19T07:59:47.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_57_00_UTC 2023-12-19T07:59:47.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.561 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_57_00_UTC) 2023-12-19T07:59:47.648 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_58_00_UTC 2023-12-19T07:59:47.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.648 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_58_00_UTC) 2023-12-19T07:59:47.732 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_59_01_UTC 2023-12-19T07:59:47.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.732 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-12-19-07_59_01_UTC) 2023-12-19T07:59:47.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T07:59:47.814 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-12-19T07:59:47.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T07:59:48.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.133+0000 7f2994c2e700 1 Processor -- start 2023-12-19T07:59:48.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 -- start start 2023-12-19T07:59:48.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2990154b90 0x7f2990155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29900c7f80 con 0x7f299006c9f0 2023-12-19T07:59:48.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29900cac60 con 0x7f2990154b90 2023-12-19T07:59:48.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f2994c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2990155570 con 0x7f29900c6e90 2023-12-19T07:59:48.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f298dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.137+0000 7f298dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:48554/0 (socket says 172.21.15.111:48554) 2023-12-19T07:59:48.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 -- 172.21.15.111:0/2272028548 learned_addr learned my addr 172.21.15.111:0/2272028548 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:48.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298ed9d700 1 --2- 172.21.15.111:0/2272028548 >> v2:172.21.15.111:3300/0 conn(0x7f2990154b90 0x7f2990155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 -- 172.21.15.111:0/2272028548 >> v2:172.21.15.111:3300/0 conn(0x7f2990154b90 msgr2=0x7f2990155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:48.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 --2- 172.21.15.111:0/2272028548 >> v2:172.21.15.111:3300/0 conn(0x7f2990154b90 0x7f2990155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 -- 172.21.15.111:0/2272028548 >> v1:172.21.15.111:6789/0 conn(0x7f299006c9f0 legacy=0x7f299006b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:48.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 -- 172.21.15.111:0/2272028548 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299006b970 con 0x7f29900c6e90 2023-12-19T07:59:48.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298dd9b700 1 --2- 172.21.15.111:0/2272028548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2980009320 tx=0x7f2980009ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2d0d4092ce13dec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:48.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298d59a700 1 -- 172.21.15.111:0/2272028548 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298000b820 con 0x7f29900c6e90 2023-12-19T07:59:48.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298d59a700 1 -- 172.21.15.111:0/2272028548 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298000b980 con 0x7f29900c6e90 2023-12-19T07:59:48.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.141+0000 7f298d59a700 1 -- 172.21.15.111:0/2272028548 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298000bd20 con 0x7f29900c6e90 2023-12-19T07:59:48.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 -- 172.21.15.111:0/2272028548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 msgr2=0x7f299006cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:48.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 --2- 172.21.15.111:0/2272028548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2980009320 tx=0x7f2980009ae0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 -- 172.21.15.111:0/2272028548 shutdown_connections 2023-12-19T07:59:48.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 --2- 172.21.15.111:0/2272028548 >> v2:172.21.15.111:3300/0 conn(0x7f2990154b90 0x7f2990155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 --2- 172.21.15.111:0/2272028548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29900c6e90 0x7f299006cdc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.145+0000 7f2994c2e700 1 -- 172.21.15.111:0/2272028548 >> 172.21.15.111:0/2272028548 conn(0x7f2990067bb0 msgr2=0x7f29900cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:48.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.149+0000 7f2994c2e700 1 -- 172.21.15.111:0/2272028548 shutdown_connections 2023-12-19T07:59:48.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.149+0000 7f2994c2e700 1 -- 172.21.15.111:0/2272028548 wait complete. 2023-12-19T07:59:48.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 Processor -- start 2023-12-19T07:59:48.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 -- start start 2023-12-19T07:59:48.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 0x7f2990156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 0x7f29900ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299016e390 con 0x7f29900c6e90 2023-12-19T07:59:48.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299016d780 con 0x7f2990154b90 2023-12-19T07:59:48.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f2994c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299016da60 con 0x7f299006c9f0 2023-12-19T07:59:48.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 0x7f29900ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37822/0 (socket says 172.21.15.111:37822) 2023-12-19T07:59:48.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298dd9b700 1 -- 172.21.15.111:0/236591393 learned_addr learned my addr 172.21.15.111:0/236591393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:48.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.153+0000 7f298e59c700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 0x7f2990156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 -- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 msgr2=0x7f2990156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 0x7f2990156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 -- 172.21.15.111:0/236591393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 msgr2=0x7f29900adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 -- 172.21.15.111:0/236591393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980009d80 con 0x7f2990154b90 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298e59c700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 0x7f2990156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:48.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298dd9b700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:48.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f298ed9d700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 0x7f29900ab6c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f297800d890 tx=0x7f2978008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:48.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.157+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978018070 con 0x7f2990154b90 2023-12-19T07:59:48.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.161+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29780125f0 con 0x7f2990154b90 2023-12-19T07:59:48.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.161+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978012990 con 0x7f2990154b90 2023-12-19T07:59:48.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.161+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299016dd40 con 0x7f2990154b90 2023-12-19T07:59:48.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.161+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299016d450 con 0x7f2990154b90 2023-12-19T07:59:48.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.161+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2990071120 con 0x7f2990154b90 2023-12-19T07:59:48.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.165+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 80149+0+0 (secure 0 0 0) 0x7f2978014430 con 0x7f2990154b90 2023-12-19T07:59:48.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.165+0000 7f29867fc700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 0x7f2970062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:48.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.165+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7184+0+0 (secure 0 0 0) 0x7f297807d0c0 con 0x7f2990154b90 2023-12-19T07:59:48.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.165+0000 7f298e59c700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 0x7f2970062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:48.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.169+0000 7f298e59c700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 0x7f2970062cd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29900b1c80 tx=0x7f2988009780 comp rx=0 tx=0).ready entity=mgr.5308 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:48.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.169+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f297804cef0 con 0x7f2990154b90 2023-12-19T07:59:48.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:48.485+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f29900c7370 con 0x7f2990154b90 2023-12-19T07:59:49.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.381+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f2978012d80 con 0x7f2990154b90 2023-12-19T07:59:49.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f29867fc700 1 -- 172.21.15.111:0/236591393 <== mon.1 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v13) v1 ==== 93+0+0 (secure 0 0 0) 0x7f297804cef0 con 0x7f2990154b90 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 msgr2=0x7f2970062cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 0x7f2970062cd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29900b1c80 tx=0x7f2988009780 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 msgr2=0x7f29900ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 0x7f29900ab6c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f297800d890 tx=0x7f2978008040 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 shutdown_connections 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2970060820 0x7f2970062cd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2990154b90 0x7f29900ab6c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29900c6e90 0x7f29900adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 --2- 172.21.15.111:0/236591393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299006c9f0 0x7f2990156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 >> 172.21.15.111:0/236591393 conn(0x7f2990067bb0 msgr2=0x7f299014fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 shutdown_connections 2023-12-19T07:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.385+0000 7f2994c2e700 1 -- 172.21.15.111:0/236591393 wait complete. 2023-12-19T07:59:49.406 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T07:59:49.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T07:59:49.446 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T07:59:49.489 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T07:59:49.489 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T07:59:49.489 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T07:59:49.489 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T07:59:49.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[43819]: fuse finished with error 0 and tester_r 0 2023-12-19T07:59:49.522 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:59:49.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:59:49.523 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T07:59:49.576 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:49.577+0000 7f8c99549000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:59:49.579 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:49.582+0000 7f65b81ab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T07:59:49.709 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:59:49.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:59:49.709 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T07:59:49.728 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T07:59:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:59:49.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T07:59:49.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:59:49.730 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T07:59:49.770 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T07:59:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:59:49.772 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T07:59:49.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T07:59:49.772 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T07:59:49.792 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T07:59:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T07:59:49.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:59:49.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.981+0000 7ff847a6e700 1 Processor -- start 2023-12-19T07:59:49.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 -- start start 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 0x7ff840067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 0x7ff8400cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff840159340 0x7ff8400cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff840068c70 con 0x7ff840159340 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff840068060 con 0x7ff840158f70 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff847a6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff840068340 con 0x7ff840069100 2023-12-19T07:59:49.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff840159340 0x7ff8400cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff840159340 0x7ff8400cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37862/0 (socket says 172.21.15.111:37862) 2023-12-19T07:59:49.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 -- 172.21.15.111:0/3007044797 learned_addr learned my addr 172.21.15.111:0/3007044797 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84580a700 1 --2- 172.21.15.111:0/3007044797 >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 0x7ff840067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff845009700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 0x7ff8400cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 -- 172.21.15.111:0/3007044797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 msgr2=0x7ff8400cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 0x7ff8400cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 -- 172.21.15.111:0/3007044797 >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 msgr2=0x7ff840067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 --2- 172.21.15.111:0/3007044797 >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 0x7ff840067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84600b700 1 -- 172.21.15.111:0/3007044797 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840068620 con 0x7ff840159340 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff84580a700 1 --2- 172.21.15.111:0/3007044797 >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 0x7ff840067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.985+0000 7ff845009700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 0x7ff8400cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:49.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff84600b700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840159340 0x7ff8400cb3e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff8400cdec0 tx=0x7ff838009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b82527b84099a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:49.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff844808700 1 -- 172.21.15.111:0/3007044797 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83800e070 con 0x7ff840159340 2023-12-19T07:59:49.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff844808700 1 -- 172.21.15.111:0/3007044797 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83800ba50 con 0x7ff840159340 2023-12-19T07:59:49.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff844808700 1 -- 172.21.15.111:0/3007044797 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380092e0 con 0x7ff840159340 2023-12-19T07:59:49.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 -- 172.21.15.111:0/3007044797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840159340 msgr2=0x7ff8400cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:49.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840159340 0x7ff8400cb3e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff8400cdec0 tx=0x7ff838009840 comp rx=0 tx=0).stop 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 -- 172.21.15.111:0/3007044797 shutdown_connections 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840159340 0x7ff8400cb3e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3007044797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840069100 0x7ff8400cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3007044797 >> v2:172.21.15.143:3300/0 conn(0x7ff840158f70 0x7ff840067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 -- 172.21.15.111:0/3007044797 >> 172.21.15.111:0/3007044797 conn(0x7ff8400ccc40 msgr2=0x7ff8400cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 -- 172.21.15.111:0/3007044797 shutdown_connections 2023-12-19T07:59:49.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.989+0000 7ff847a6e700 1 -- 172.21.15.111:0/3007044797 wait complete. 2023-12-19T07:59:49.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 Processor -- start 2023-12-19T07:59:49.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 -- start start 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff840069100 0x7ff8400c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840159340 0x7ff8400b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400c9e40 con 0x7ff840158f70 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400b23f0 con 0x7ff840069100 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff847a6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400b17e0 con 0x7ff840159340 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff845009700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff845009700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37872/0 (socket says 172.21.15.111:37872) 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff845009700 1 -- 172.21.15.111:0/3753911813 learned_addr learned my addr 172.21.15.111:0/3753911813 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.993+0000 7ff84580a700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff840069100 0x7ff8400c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff84600b700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840159340 0x7ff8400b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840159340 msgr2=0x7ff8400b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840159340 0x7ff8400b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff840069100 msgr2=0x7ff8400c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff840069100 0x7ff8400c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 -- 172.21.15.111:0/3753911813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838009ae0 con 0x7ff840158f70 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff845009700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff83000c880 tx=0x7ff830008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830011600 con 0x7ff840158f70 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830011760 con 0x7ff840158f70 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8400b1ac0 con 0x7ff840158f70 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff830011ae0 con 0x7ff840158f70 2023-12-19T07:59:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8400b2080 con 0x7ff840158f70 2023-12-19T07:59:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840071120 con 0x7ff840158f70 2023-12-19T07:59:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:49.997+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7ff830018350 con 0x7ff840158f70 2023-12-19T07:59:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.001+0000 7ff835ffb700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 0x7ff824062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.001+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7ff83007d090 con 0x7ff840158f70 2023-12-19T07:59:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.001+0000 7ff84580a700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 msgr2=0x7ff824062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.001+0000 7ff84580a700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 0x7ff824062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:50.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.005+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff830050740 con 0x7ff840158f70 2023-12-19T07:59:50.137 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.137+0000 7f8c99549000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:59:50.142 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.146+0000 7f65b81ab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T07:59:50.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.205+0000 7ff84580a700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 msgr2=0x7ff824062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:50.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.205+0000 7ff84580a700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 0x7ff824062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:59:50.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.213+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff840068900 con 0x7ff840158f70 2023-12-19T07:59:50.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.213+0000 7ff835ffb700 1 -- 172.21.15.111:0/3753911813 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7ff8300502d0 con 0x7ff840158f70 2023-12-19T07:59:50.213 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:50.214 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:59:50.215 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.217+0000 7f8c99549000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:59:50.216 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.222+0000 7f65b81ab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 msgr2=0x7ff824062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 0x7ff824062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 msgr2=0x7ff8400b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff83000c880 tx=0x7ff830008040 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 shutdown_connections 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff824060700 0x7ff824062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff840159340 0x7ff8400b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff840158f70 0x7ff8400b5840 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 --2- 172.21.15.111:0/3753911813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff840069100 0x7ff8400c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 >> 172.21.15.111:0/3753911813 conn(0x7ff8400ccc40 msgr2=0x7ff840154360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 shutdown_connections 2023-12-19T07:59:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.217+0000 7ff847a6e700 1 -- 172.21.15.111:0/3753911813 wait complete. 2023-12-19T07:59:50.220 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:59:50.247 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:59:50.362 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.366+0000 7f65b81ab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:59:50.370 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.373+0000 7f8c99549000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:59:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.409+0000 7f6fdd2f0700 1 Processor -- start 2023-12-19T07:59:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.409+0000 7f6fdd2f0700 1 -- start start 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.409+0000 7f6fdd2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fdd2f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd806c9f0 con 0x7f6fd80c2980 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fdd2f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80c7f20 con 0x7f6fd813f890 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fdd2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80c1c50 con 0x7f6fd8068800 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fcffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6fd80c2980 0x7f6fd80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41500/0 (socket says 172.21.15.111:41500) 2023-12-19T07:59:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fcffff700 1 -- 172.21.15.111:0/1503800364 learned_addr learned my addr 172.21.15.111:0/1503800364 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd77fe700 1 --2- 172.21.15.111:0/1503800364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531012901 0 0) 0x7f6fd806c9f0 con 0x7f6fd80c2980 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003570 con 0x7f6fd80c2980 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698751122 0 0) 0x7f6fd80c7f20 con 0x7f6fd813f890 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd806c9f0 con 0x7f6fd813f890 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865903308 0 0) 0x7f6fc4003570 con 0x7f6fd80c2980 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd77fe700 1 -- 172.21.15.111:0/1503800364 >> v1:172.21.15.143:6789/0 conn(0x7f6fd813f890 legacy=0x7f6fd80c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd77fe700 1 -- 172.21.15.111:0/1503800364 >> v1:172.21.15.111:6789/0 conn(0x7f6fd80c2980 legacy=0x7f6fd80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd77fe700 1 -- 172.21.15.111:0/1503800364 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80c8190 con 0x7f6fd8068800 2023-12-19T07:59:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd77fe700 1 --2- 172.21.15.111:0/1503800364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd80aeba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009e90 tx=0x7f6fd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d18b36804f24ea13 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000e070 con 0x7f6fd8068800 2023-12-19T07:59:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000b870 con 0x7f6fd8068800 2023-12-19T07:59:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.413+0000 7f6fd67fc700 1 -- 172.21.15.111:0/1503800364 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0001470 con 0x7f6fd8068800 2023-12-19T07:59:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/1503800364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 msgr2=0x7f6fd80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/1503800364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd80aeba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009e90 tx=0x7f6fd0009780 comp rx=0 tx=0).stop 2023-12-19T07:59:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/1503800364 shutdown_connections 2023-12-19T07:59:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/1503800364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd80aeba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/1503800364 >> 172.21.15.111:0/1503800364 conn(0x7f6fd80c8850 msgr2=0x7f6fd80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/1503800364 shutdown_connections 2023-12-19T07:59:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/1503800364 wait complete. 2023-12-19T07:59:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 Processor -- start 2023-12-19T07:59:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.417+0000 7f6fdd2f0700 1 -- start start 2023-12-19T07:59:50.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80c2980 0x7f6fd80ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80aa860 con 0x7f6fd813f890 2023-12-19T07:59:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd815b420 con 0x7f6fd80c2980 2023-12-19T07:59:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd813dc30 con 0x7f6fd8068800 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80c2980 0x7f6fd80ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37880/0 (socket says 172.21.15.111:37880) 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 -- 172.21.15.111:0/2198156357 learned_addr learned my addr 172.21.15.111:0/2198156357 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd6ffd700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 -- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 msgr2=0x7f6fd815aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd815aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 -- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80c2980 msgr2=0x7f6fd80ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80c2980 0x7f6fd80ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 -- 172.21.15.111:0/2198156357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0009a20 con 0x7f6fd813f890 2023-12-19T07:59:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fd77fe700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009de0 tx=0x7f6fd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0012c20 con 0x7f6fd813f890 2023-12-19T07:59:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0012d80 con 0x7f6fd813f890 2023-12-19T07:59:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd001e6a0 con 0x7f6fd813f890 2023-12-19T07:59:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd80aaa70 con 0x7f6fd813f890 2023-12-19T07:59:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.421+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd80ab030 con 0x7f6fd813f890 2023-12-19T07:59:50.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.425+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd813ed60 con 0x7f6fd813f890 2023-12-19T07:59:50.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.425+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f6fd001ead0 con 0x7f6fd813f890 2023-12-19T07:59:50.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.429+0000 7f6fceffd700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 0x7f6fb8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.429+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7f6fd007f140 con 0x7f6fd813f890 2023-12-19T07:59:50.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.429+0000 7f6fd6ffd700 1 -- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 msgr2=0x7f6fb8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:50.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.429+0000 7f6fd6ffd700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 0x7f6fb8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.437+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6fd813ed60 con 0x7f6fd813f890 2023-12-19T07:59:50.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.601+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd813ed60 con 0x7f6fd813f890 2023-12-19T07:59:50.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fceffd700 1 -- 172.21.15.111:0/2198156357 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17556 (secure 0 0 0) 0x7f6fd813ed60 con 0x7f6fd813f890 2023-12-19T07:59:50.604 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":54,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:59:48.348041+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3058403037":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1676361505":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/4055025414":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6833/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6832/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1940071424":"2023-12-20T07:48:40.528791+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":188,"length":33}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":56,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":216,"length":5}]}],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":9,"length":2},{"begin":14,"length":2},{"begin":19,"length":1},{"begin":22,"length":1},{"begin":27,"length":2},{"begin":34,"length":1},{"begin":40,"length":2},{"begin":44,"length":2},{"begin":47,"length":1},{"begin":58,"length":2},{"begin":61,"length":2},{"begin":64,"length":1},{"begin":67,"length":1},{"begin":72,"length":1},{"begin":77,"length":1},{"begin":80,"length":1},{"begin":82,"length":1},{"begin":84,"length":1},{"begin":87,"length":1},{"begin":92,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":111,"length":1},{"begin":155,"length":33}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 msgr2=0x7f6fb8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 0x7f6fb8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 msgr2=0x7f6fd8168ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009de0 tx=0x7f6fd0017000 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 shutdown_connections 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fb8060700 0x7f6fb8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80c2980 0x7f6fd80ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fd813f890 0x7f6fd8168ca0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 --2- 172.21.15.111:0/2198156357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd8068800 0x7f6fd815aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 >> 172.21.15.111:0/2198156357 conn(0x7f6fd80c8850 msgr2=0x7f6fd813e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 shutdown_connections 2023-12-19T07:59:50.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.605+0000 7f6fdd2f0700 1 -- 172.21.15.111:0/2198156357 wait complete. 2023-12-19T07:59:50.624 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T07:48:23.875106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T07:48:24.748962+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T07:59:50.625 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:59:50.642 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.646+0000 7f65b81ab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:59:50.654 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.657+0000 7f8c99549000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T07:59:50.718 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.722+0000 7f65b81ab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:59:50.729 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.729+0000 7f8c99549000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T07:59:50.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.789+0000 7fc61842e700 1 Processor -- start 2023-12-19T07:59:50.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.789+0000 7fc61842e700 1 -- start start 2023-12-19T07:59:50.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc61842e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc61006db40 0x7fc6100cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc61842e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6100c9db0 con 0x7fc6100c63f0 2023-12-19T07:59:50.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc61842e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6100cb2e0 con 0x7fc61006df10 2023-12-19T07:59:50.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc61842e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6100ca950 con 0x7fc61006db40 2023-12-19T07:59:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6159c9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc6100c63f0 0x7fc6100ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41510/0 (socket says 172.21.15.111:41510) 2023-12-19T07:59:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6159c9700 1 -- 172.21.15.111:0/3713121274 learned_addr learned my addr 172.21.15.111:0/3713121274 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6161ca700 1 --2- 172.21.15.111:0/3713121274 >> v2:172.21.15.143:3300/0 conn(0x7fc61006db40 0x7fc6100cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227556627 0 0) 0x7fc6100c9db0 con 0x7fc6100c63f0 2023-12-19T07:59:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5fc003570 con 0x7fc6100c63f0 2023-12-19T07:59:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380410147 0 0) 0x7fc5fc003570 con 0x7fc6100c63f0 2023-12-19T07:59:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 >> v2:172.21.15.143:3300/0 conn(0x7fc61006db40 msgr2=0x7fc6100cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 --2- 172.21.15.111:0/3713121274 >> v2:172.21.15.143:3300/0 conn(0x7fc61006db40 0x7fc6100cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 >> v1:172.21.15.143:6789/0 conn(0x7fc61006df10 legacy=0x7fc610159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6100cb5c0 con 0x7fc6100c63f0 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fc600001470 con 0x7fc6100c63f0 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 >> v1:172.21.15.111:6789/0 conn(0x7fc6100c63f0 legacy=0x7fc6100ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 0x7fc5fc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc004810 0x7fc5fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5fc0051c0 0x7fc5fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100ca950 con 0x7fc5fc003a60 2023-12-19T07:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc00c0d0 con 0x7fc5fc0051c0 2023-12-19T07:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc00c290 con 0x7fc5fc004810 2023-12-19T07:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6161ca700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 0x7fc5fc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6159c9700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc004810 0x7fc5fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.793+0000 7fc6169cb700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5fc0051c0 0x7fc5fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 -- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc004810 msgr2=0x7fc5fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc004810 0x7fc5fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 -- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5fc0051c0 msgr2=0x7fc5fc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5fc0051c0 0x7fc5fc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6169cb700 1 -- 172.21.15.111:0/3713121274 reap_dead start 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 -- 172.21.15.111:0/3713121274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc003570 con 0x7fc5fc003a60 2023-12-19T07:59:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6161ca700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 0x7fc5fc003e70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009c00 tx=0x7fc60c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53eb106b8b5b1e40 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c00e710 con 0x7fc5fc003a60 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c00e870 con 0x7fc5fc003a60 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc6151c8700 1 -- 172.21.15.111:0/3713121274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c00ec30 con 0x7fc5fc003a60 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 -- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 msgr2=0x7fc5fc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 0x7fc5fc003e70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009c00 tx=0x7fc60c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 -- 172.21.15.111:0/3713121274 shutdown_connections 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 --2- 172.21.15.111:0/3713121274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5fc003a60 0x7fc5fc003e70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 -- 172.21.15.111:0/3713121274 >> 172.21.15.111:0/3713121274 conn(0x7fc610067bb0 msgr2=0x7fc6100cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 -- 172.21.15.111:0/3713121274 shutdown_connections 2023-12-19T07:59:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 -- 172.21.15.111:0/3713121274 wait complete. 2023-12-19T07:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.797+0000 7fc61842e700 1 Processor -- start 2023-12-19T07:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- start start 2023-12-19T07:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc003a60 0x7fc6101550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc610169660 0x7fc610156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc610154c20 con 0x7fc6101691f0 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6101555f0 con 0x7fc610169660 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc610154010 con 0x7fc5fc003a60 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37894/0 (socket says 172.21.15.111:37894) 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 -- 172.21.15.111:0/2172239288 learned_addr learned my addr 172.21.15.111:0/2172239288 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6161ca700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc003a60 0x7fc6101550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6169cb700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc610169660 0x7fc610156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 -- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc003a60 msgr2=0x7fc6101550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc003a60 0x7fc6101550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 -- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc610169660 msgr2=0x7fc610156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc610169660 0x7fc610156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 -- 172.21.15.111:0/2172239288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6101542f0 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6159c9700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc60000eb90 tx=0x7fc600000d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600005e90 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000c7e0 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6101545d0 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60000cba0 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.801+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61015bc60 con 0x7fc6101691f0 2023-12-19T07:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc610154a50 con 0x7fc6101691f0 2023-12-19T07:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fc60001d460 con 0x7fc6101691f0 2023-12-19T07:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc6067fc700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 0x7fc5f4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7fc60007d070 con 0x7fc6101691f0 2023-12-19T07:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc6161ca700 1 -- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 msgr2=0x7fc5f4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.805+0000 7fc6161ca700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 0x7fc5f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.813+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc60004ce60 con 0x7fc6101691f0 2023-12-19T07:59:50.934 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:50.938+0000 7f65b81ab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:59:50.960 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:50.961+0000 7f8c99549000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T07:59:50.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.985+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc610071120 con 0x7fc6101691f0 2023-12-19T07:59:50.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.985+0000 7fc6067fc700 1 -- 172.21.15.111:0/2172239288 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc600050480 con 0x7fc6101691f0 2023-12-19T07:59:50.984 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T07:59:50.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 msgr2=0x7fc5f4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 0x7fc5f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 msgr2=0x7fc610156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc60000eb90 tx=0x7fc600000d00 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 shutdown_connections 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5f4060700 0x7fc5f4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc610169660 0x7fc610156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6101691f0 0x7fc610156040 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 --2- 172.21.15.111:0/2172239288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5fc003a60 0x7fc6101550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 >> 172.21.15.111:0/2172239288 conn(0x7fc610067bb0 msgr2=0x7fc610158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 shutdown_connections 2023-12-19T07:59:50.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:50.989+0000 7fc61842e700 1 -- 172.21.15.111:0/2172239288 wait complete. 2023-12-19T07:59:51.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T07:59:51.021 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:51.026+0000 7f65b81ab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:59:51.038 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:51.037+0000 7f8c99549000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T07:59:51.114 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:51.118+0000 7f65b81ab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:59:51.129 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:51.129+0000 7f8c99549000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T07:59:51.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 Processor -- start 2023-12-19T07:59:51.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 -- start start 2023-12-19T07:59:51.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f091006df10 0x7f09100cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100c9db0 con 0x7f09100c63f0 2023-12-19T07:59:51.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100cb2e0 con 0x7f091006db40 2023-12-19T07:59:51.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f091866e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09100ca950 con 0x7f091006df10 2023-12-19T07:59:51.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f0915c09700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09100c63f0 0x7f0910159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41512/0 (socket says 172.21.15.111:41512) 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f0915c09700 1 -- 172.21.15.111:0/1320842799 learned_addr learned my addr 172.21.15.111:0/1320842799 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f0916c0b700 1 --2- 172.21.15.111:0/1320842799 >> v2:172.21.15.143:3300/0 conn(0x7f091006df10 0x7f09100cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f0915408700 1 -- 172.21.15.111:0/1320842799 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991436813 0 0) 0x7f09100c9db0 con 0x7f09100c63f0 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.197+0000 7f0915408700 1 -- 172.21.15.111:0/1320842799 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08fc003570 con 0x7f09100c63f0 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0916c0b700 1 -- 172.21.15.111:0/1320842799 >> v1:172.21.15.143:6789/0 conn(0x7f091006db40 legacy=0x7f09100ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0916c0b700 1 -- 172.21.15.111:0/1320842799 >> v1:172.21.15.111:6789/0 conn(0x7f09100c63f0 legacy=0x7f0910159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0916c0b700 1 -- 172.21.15.111:0/1320842799 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09100cb5c0 con 0x7f091006df10 2023-12-19T07:59:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0916c0b700 1 --2- 172.21.15.111:0/1320842799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006df10 0x7f09100cc3e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0908009e60 tx=0x7f0908009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23a8fff449d48068 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0915408700 1 -- 172.21.15.111:0/1320842799 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090800e070 con 0x7f091006df10 2023-12-19T07:59:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0915408700 1 -- 172.21.15.111:0/1320842799 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090800bb60 con 0x7f091006df10 2023-12-19T07:59:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f0915408700 1 -- 172.21.15.111:0/1320842799 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908012430 con 0x7f091006df10 2023-12-19T07:59:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 -- 172.21.15.111:0/1320842799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006df10 msgr2=0x7f09100cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 --2- 172.21.15.111:0/1320842799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006df10 0x7f09100cc3e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0908009e60 tx=0x7f0908009780 comp rx=0 tx=0).stop 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 -- 172.21.15.111:0/1320842799 shutdown_connections 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 --2- 172.21.15.111:0/1320842799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006df10 0x7f09100cc3e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 -- 172.21.15.111:0/1320842799 >> 172.21.15.111:0/1320842799 conn(0x7f0910067bb0 msgr2=0x7f09100cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 -- 172.21.15.111:0/1320842799 shutdown_connections 2023-12-19T07:59:51.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.201+0000 7f091866e700 1 -- 172.21.15.111:0/1320842799 wait complete. 2023-12-19T07:59:51.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 Processor -- start 2023-12-19T07:59:51.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 -- start start 2023-12-19T07:59:51.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006db40 0x7f091014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09100c63f0 0x7f09100c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910155b00 con 0x7f091006df10 2023-12-19T07:59:51.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910154ef0 con 0x7f091006db40 2023-12-19T07:59:51.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091866e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09101551d0 con 0x7f09100c63f0 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37906/0 (socket says 172.21.15.111:37906) 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 -- 172.21.15.111:0/2949149348 learned_addr learned my addr 172.21.15.111:0/2949149348 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0916c0b700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09100c63f0 0x7f09100c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f091640a700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006db40 0x7f091014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 -- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09100c63f0 msgr2=0x7f09100c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09100c63f0 0x7f09100c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 -- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006db40 msgr2=0x7f091014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006db40 0x7f091014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.205+0000 7f0915c09700 1 -- 172.21.15.111:0/2949149348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908009a20 con 0x7f091006df10 2023-12-19T07:59:51.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f0915c09700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f090000c560 tx=0x7f0900007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900005b30 con 0x7f091006df10 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0900005cb0 con 0x7f091006df10 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09101554b0 con 0x7f091006df10 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900007320 con 0x7f091006df10 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091014fdd0 con 0x7f091006df10 2023-12-19T07:59:51.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.209+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0910155930 con 0x7f091006df10 2023-12-19T07:59:51.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.213+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f090001b3b0 con 0x7f091006df10 2023-12-19T07:59:51.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.213+0000 7f09067fc700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 0x7f08f4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.213+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7f090007c540 con 0x7f091006df10 2023-12-19T07:59:51.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.213+0000 7f091640a700 1 -- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 msgr2=0x7f08f4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:51.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.213+0000 7f091640a700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 0x7f08f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:51.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.217+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f090004c1d0 con 0x7f091006df10 2023-12-19T07:59:51.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.393+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f09100cb8a0 con 0x7f091006df10 2023-12-19T07:59:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.393+0000 7f09067fc700 1 -- 172.21.15.111:0/2949149348 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0900053e60 con 0x7f091006df10 2023-12-19T07:59:51.392 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 msgr2=0x7f08f4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 0x7f08f4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 msgr2=0x7f0910156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f090000c560 tx=0x7f0900007dd0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 shutdown_connections 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f08f4060700 0x7f08f4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09100c63f0 0x7f09100c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f091006df10 0x7f0910156a30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 --2- 172.21.15.111:0/2949149348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f091006db40 0x7f091014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 >> 172.21.15.111:0/2949149348 conn(0x7f0910067bb0 msgr2=0x7f09100c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 shutdown_connections 2023-12-19T07:59:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.397+0000 7f091866e700 1 -- 172.21.15.111:0/2949149348 wait complete. 2023-12-19T07:59:51.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T07:59:51.517 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:51.522+0000 7f65b81ab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:59:51.564 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:51.565+0000 7f8c99549000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T07:59:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.589+0000 7fd2c1c16700 1 Processor -- start 2023-12-19T07:59:51.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.589+0000 7fd2c1c16700 1 -- start start 2023-12-19T07:59:51.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1c16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2bc13f2d0 0x7fd2bc0ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 0x7fd2bc0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1c16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0adc60 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1c16700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc06d7d0 con 0x7fd2bc13f2d0 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0c8140 con 0x7fd2bc0c7780 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c0c14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2bc13f2d0 0x7fd2bc0ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd2bc13f6a0 0x7fd2bc15a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41520/0 (socket says 172.21.15.111:41520) 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bbfff700 1 -- 172.21.15.111:0/1448369416 learned_addr learned my addr 172.21.15.111:0/1448369416 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1448369416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 0x7fd2bc0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 -- 172.21.15.111:0/1448369416 >> v2:172.21.15.111:3300/0 conn(0x7fd2bc13f2d0 msgr2=0x7fd2bc0ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:51.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1448369416 >> v2:172.21.15.111:3300/0 conn(0x7fd2bc13f2d0 0x7fd2bc0ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 -- 172.21.15.111:0/1448369416 >> v1:172.21.15.111:6789/0 conn(0x7fd2bc13f6a0 legacy=0x7fd2bc15a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 -- 172.21.15.111:0/1448369416 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0ad050 con 0x7fd2bc0c7780 2023-12-19T07:59:51.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bb7fe700 1 -- 172.21.15.111:0/1448369416 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798406114 0 0) 0x7fd2bc0adc60 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1448369416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 0x7fd2bc0c7c00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009e60 tx=0x7fd2b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51162ce04961342b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bb7fe700 1 -- 172.21.15.111:0/1448369416 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b400bb60 con 0x7fd2bc0c7780 2023-12-19T07:59:51.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bb7fe700 1 -- 172.21.15.111:0/1448369416 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b400bcc0 con 0x7fd2bc0c7780 2023-12-19T07:59:51.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.593+0000 7fd2bb7fe700 1 -- 172.21.15.111:0/1448369416 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40116b0 con 0x7fd2bc0c7780 2023-12-19T07:59:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1448369416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 msgr2=0x7fd2bc0c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1448369416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 0x7fd2bc0c7c00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009e60 tx=0x7fd2b4009780 comp rx=0 tx=0).stop 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1448369416 shutdown_connections 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1448369416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc0c7780 0x7fd2bc0c7c00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1448369416 >> v2:172.21.15.111:3300/0 conn(0x7fd2bc13f2d0 0x7fd2bc0ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1448369416 >> 172.21.15.111:0/1448369416 conn(0x7fd2bc067b50 msgr2=0x7fd2bc0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1448369416 shutdown_connections 2023-12-19T07:59:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1448369416 wait complete. 2023-12-19T07:59:51.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.597+0000 7fd2c1c16700 1 Processor -- start 2023-12-19T07:59:51.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 -- start start 2023-12-19T07:59:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc0c7780 0x7fd2bc169680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc13f2d0 0x7fd2bc16a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc169bc0 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc16bee0 con 0x7fd2bc0c7780 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc16ac20 con 0x7fd2bc13f2d0 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37924/0 (socket says 172.21.15.111:37924) 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 -- 172.21.15.111:0/1552419097 learned_addr learned my addr 172.21.15.111:0/1552419097 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c0c14700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc0c7780 0x7fd2bc169680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2bbfff700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc13f2d0 0x7fd2bc16a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 -- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc13f2d0 msgr2=0x7fd2bc16a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc13f2d0 0x7fd2bc16a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 -- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc0c7780 msgr2=0x7fd2bc169680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc0c7780 0x7fd2bc169680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 -- 172.21.15.111:0/1552419097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4009a20 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1415700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40094c0 tx=0x7fd2b401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40196e0 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4019860 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4019c20 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.601+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc16ae40 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.605+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc16b400 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.605+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc04be40 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.605+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fd2b402c3b0 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.605+0000 7fd2b8ff9700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 0x7fd2a0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.605+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7fd2b407f170 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.609+0000 7fd2c0c14700 1 -- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 msgr2=0x7fd2a0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.609+0000 7fd2c0c14700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 0x7fd2a0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:51.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.613+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd2b404ed90 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.694 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:51.698+0000 7f65b81ab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:59:51.748 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:51.749+0000 7f8c99549000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T07:59:51.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.793+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2bc06ba60 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.797+0000 7fd2b8ff9700 1 -- 172.21.15.111:0/1552419097 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd2b40523b0 con 0x7fd2bc13f6a0 2023-12-19T07:59:51.794 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T07:59:51.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 msgr2=0x7fd2a0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 0x7fd2a0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 msgr2=0x7fd2bc13d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40094c0 tx=0x7fd2b401b000 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 shutdown_connections 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2a0060700 0x7fd2a0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2bc13f6a0 0x7fd2bc13d920 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2bc13f2d0 0x7fd2bc16a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 --2- 172.21.15.111:0/1552419097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2bc0c7780 0x7fd2bc169680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 >> 172.21.15.111:0/1552419097 conn(0x7fd2bc067b50 msgr2=0x7fd2bc0c4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 shutdown_connections 2023-12-19T07:59:51.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.801+0000 7fd2c1c16700 1 -- 172.21.15.111:0/1552419097 wait complete. 2023-12-19T07:59:51.817 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:59:51.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.977+0000 7f668bf85700 1 Processor -- start 2023-12-19T07:59:51.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.977+0000 7f668bf85700 1 -- start start 2023-12-19T07:59:51.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668bf85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 0x7f668415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668bf85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f66840aeba0 0x7f66840680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668bf85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66840c7190 con 0x7f668415d320 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668bf85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66840c1b50 con 0x7f66840aeba0 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668bf85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66840c1e30 con 0x7f66840c15b0 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6689520700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 0x7f668415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6689d21700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f668415d320 0x7f668406d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38840/0 (socket says 172.21.15.111:38840) 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6689d21700 1 -- 172.21.15.111:0/340760984 learned_addr learned my addr 172.21.15.111:0/340760984 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:51.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 --2- 172.21.15.111:0/340760984 >> v2:172.21.15.143:3300/0 conn(0x7f66840aeba0 0x7f66840680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 -- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 msgr2=0x7f668415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:51.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 0x7f668415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 -- 172.21.15.111:0/340760984 >> v1:172.21.15.143:6789/0 conn(0x7f668415d320 legacy=0x7f668406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6688d1f700 1 -- 172.21.15.111:0/340760984 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660191711 0 0) 0x7f66840c7190 con 0x7f668415d320 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 -- 172.21.15.111:0/340760984 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66840c6e20 con 0x7f66840aeba0 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6689520700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 0x7f668415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f668a522700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840aeba0 0x7f66840680f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f667c009e60 tx=0x7f667c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f24c98a93f391a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.981+0000 7f6688d1f700 1 -- 172.21.15.111:0/340760984 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c00e070 con 0x7f66840aeba0 2023-12-19T07:59:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f6688d1f700 1 -- 172.21.15.111:0/340760984 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c00bb60 con 0x7f66840aeba0 2023-12-19T07:59:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f6688d1f700 1 -- 172.21.15.111:0/340760984 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c012430 con 0x7f66840aeba0 2023-12-19T07:59:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 -- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840aeba0 msgr2=0x7f66840680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840aeba0 0x7f66840680f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f667c009e60 tx=0x7f667c009780 comp rx=0 tx=0).stop 2023-12-19T07:59:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 -- 172.21.15.111:0/340760984 shutdown_connections 2023-12-19T07:59:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840aeba0 0x7f66840680f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 --2- 172.21.15.111:0/340760984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66840c15b0 0x7f668415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 -- 172.21.15.111:0/340760984 >> 172.21.15.111:0/340760984 conn(0x7f66840c8850 msgr2=0x7f66840c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 -- 172.21.15.111:0/340760984 shutdown_connections 2023-12-19T07:59:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 -- 172.21.15.111:0/340760984 wait complete. 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.985+0000 7f668bf85700 1 Processor -- start 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 -- start start 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840c15b0 0x7f6684156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 0x7f6684155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668415d890 con 0x7f66840aeba0 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840b0500 con 0x7f66840c15b0 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668bf85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6684156520 con 0x7f668415d320 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f6689d21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f6689d21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54560/0 (socket says 172.21.15.111:54560) 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 0x7f6684155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f6689d21700 1 -- 172.21.15.111:0/2769715110 learned_addr learned my addr 172.21.15.111:0/2769715110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f6689520700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840c15b0 0x7f6684156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840c15b0 msgr2=0x7f6684156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840c15b0 0x7f6684156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 msgr2=0x7f66840b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 -- 172.21.15.111:0/2769715110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6674009420 con 0x7f668415d320 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f6689d21700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.989+0000 7f668a522700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 0x7f6684155fe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f667c016000 tx=0x7f667c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c027470 con 0x7f668415d320 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c0275d0 con 0x7f668415d320 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c009a20 con 0x7f668415d320 2023-12-19T07:59:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c027990 con 0x7f668415d320 2023-12-19T07:59:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66840afc80 con 0x7f668415d320 2023-12-19T07:59:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6684071120 con 0x7f668415d320 2023-12-19T07:59:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.993+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f667c024070 con 0x7f668415d320 2023-12-19T07:59:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.997+0000 7f6679ffb700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 0x7f6668062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.997+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7f667c07f830 con 0x7f668415d320 2023-12-19T07:59:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.997+0000 7f6689d21700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 msgr2=0x7f6668062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:51.997+0000 7f6689d21700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 0x7f6668062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:52.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.005+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f667c04f540 con 0x7f668415d320 2023-12-19T07:59:52.074 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.078+0000 7f65b81ab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:59:52.134 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.133+0000 7f8c99549000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T07:59:52.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.201+0000 7f6689d21700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 msgr2=0x7f6668062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:52.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.201+0000 7f6689d21700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 0x7f6668062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:59:52.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.213+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66840b0640 con 0x7f668415d320 2023-12-19T07:59:52.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.217+0000 7f6679ffb700 1 -- 172.21.15.111:0/2769715110 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f667c052b60 con 0x7f668415d320 2023-12-19T07:59:52.215 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:52.215 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:59:52.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 msgr2=0x7f6668062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:52.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 0x7f6668062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 msgr2=0x7f6684155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 0x7f6684155fe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f667c016000 tx=0x7f667c019000 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 shutdown_connections 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6668060700 0x7f6668062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66840c15b0 0x7f6684156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f668415d320 0x7f6684155fe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 --2- 172.21.15.111:0/2769715110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66840aeba0 0x7f66840b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 >> 172.21.15.111:0/2769715110 conn(0x7f66840c8850 msgr2=0x7f668406b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 shutdown_connections 2023-12-19T07:59:52.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.221+0000 7f668bf85700 1 -- 172.21.15.111:0/2769715110 wait complete. 2023-12-19T07:59:52.221 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:59:52.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:59:52.284 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.290+0000 7f65b81ab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:59:52.362 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.365+0000 7f8c99549000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T07:59:52.374 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.378+0000 7f65b81ab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:59:52.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.389+0000 7fa1692e3700 1 Processor -- start 2023-12-19T07:59:52.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.389+0000 7fa1692e3700 1 -- start start 2023-12-19T07:59:52.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.389+0000 7fa1692e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa16406a540 0x7fa16406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640ca5a0 con 0x7fa1640cc470 2023-12-19T07:59:52.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16406dac0 con 0x7fa164158e70 2023-12-19T07:59:52.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16406d220 con 0x7fa16406a540 2023-12-19T07:59:52.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa168ae2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa1640cc470 0x7fa1640cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41790/0 (socket says 172.21.15.111:41790) 2023-12-19T07:59:52.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa168ae2700 1 -- 172.21.15.111:0/4281809568 learned_addr learned my addr 172.21.15.111:0/4281809568 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa15bfff700 1 --2- 172.21.15.111:0/4281809568 >> v2:172.21.15.111:3300/0 conn(0x7fa16406a540 0x7fa16406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55337108 0 0) 0x7fa1640ca5a0 con 0x7fa1640cc470 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150003570 con 0x7fa1640cc470 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855005978 0 0) 0x7fa16406dac0 con 0x7fa164158e70 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1640ca5a0 con 0x7fa164158e70 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa15bfff700 1 -- 172.21.15.111:0/4281809568 >> v1:172.21.15.143:6789/0 conn(0x7fa164158e70 legacy=0x7fa1640afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa15bfff700 1 -- 172.21.15.111:0/4281809568 >> v1:172.21.15.111:6789/0 conn(0x7fa1640cc470 legacy=0x7fa1640cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa15bfff700 1 -- 172.21.15.111:0/4281809568 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16406d500 con 0x7fa16406a540 2023-12-19T07:59:52.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa15bfff700 1 --2- 172.21.15.111:0/4281809568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa16406c9d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0098e0 tx=0x7fa14c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b22399e9b62cb83 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:52.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c011630 con 0x7fa16406a540 2023-12-19T07:59:52.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c011790 con 0x7fa16406a540 2023-12-19T07:59:52.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1637fe700 1 -- 172.21.15.111:0/4281809568 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c011b30 con 0x7fa16406a540 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- 172.21.15.111:0/4281809568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 msgr2=0x7fa16406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 --2- 172.21.15.111:0/4281809568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa16406c9d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0098e0 tx=0x7fa14c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- 172.21.15.111:0/4281809568 shutdown_connections 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 --2- 172.21.15.111:0/4281809568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa16406c9d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- 172.21.15.111:0/4281809568 >> 172.21.15.111:0/4281809568 conn(0x7fa1640670e0 msgr2=0x7fa1640674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.393+0000 7fa1692e3700 1 -- 172.21.15.111:0/4281809568 shutdown_connections 2023-12-19T07:59:52.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.397+0000 7fa1692e3700 1 -- 172.21.15.111:0/4281809568 wait complete. 2023-12-19T07:59:52.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.397+0000 7fa1692e3700 1 Processor -- start 2023-12-19T07:59:52.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.397+0000 7fa1692e3700 1 -- start start 2023-12-19T07:59:52.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 0x7fa1640c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164158e70 0x7fa1640c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640b3bd0 con 0x7fa16406a540 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640b2fc0 con 0x7fa164158e70 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640b32a0 con 0x7fa1640cc470 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54580/0 (socket says 172.21.15.111:54580) 2023-12-19T07:59:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 -- 172.21.15.111:0/2446310667 learned_addr learned my addr 172.21.15.111:0/2446310667 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa168ae2700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164158e70 0x7fa1640c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 0x7fa1640c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 -- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 msgr2=0x7fa1640c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 0x7fa1640c5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 -- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164158e70 msgr2=0x7fa1640c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164158e70 0x7fa1640c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 -- 172.21.15.111:0/2446310667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c009420 con 0x7fa16406a540 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 0x7fa1640c5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa163fff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c580 tx=0x7fa154007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154010510 con 0x7fa16406a540 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa154010670 con 0x7fa16406a540 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154010a30 con 0x7fa16406a540 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1640b3580 con 0x7fa16406a540 2023-12-19T07:59:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.401+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1640b2c70 con 0x7fa16406a540 2023-12-19T07:59:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16404be40 con 0x7fa16406a540 2023-12-19T07:59:52.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fa15401b400 con 0x7fa16406a540 2023-12-19T07:59:52.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa160ff9700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 0x7fa144062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7fa15407ce50 con 0x7fa16406a540 2023-12-19T07:59:52.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa15bfff700 1 -- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 msgr2=0x7fa144062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:52.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.405+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 0x7fa144062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:52.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.413+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa15404cae0 con 0x7fa16406a540 2023-12-19T07:59:52.446 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.450+0000 7f65b81ab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:59:52.456 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.457+0000 7f8c99549000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T07:59:52.528 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.534+0000 7f65b81ab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:59:52.536 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.537+0000 7f8c99549000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T07:59:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.585+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa164156c30 con 0x7fa16406a540 2023-12-19T07:59:52.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa160ff9700 1 -- 172.21.15.111:0/2446310667 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+17556 (secure 0 0 0) 0x7fa154050100 con 0x7fa16406a540 2023-12-19T07:59:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:52.587 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":54,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:59:48.348041+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3058403037":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1676361505":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/4055025414":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6833/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6832/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1940071424":"2023-12-20T07:48:40.528791+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":188,"length":33}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":56,"length":1},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":216,"length":5}]}],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":1},{"begin":9,"length":2},{"begin":14,"length":2},{"begin":19,"length":1},{"begin":22,"length":1},{"begin":27,"length":2},{"begin":34,"length":1},{"begin":40,"length":2},{"begin":44,"length":2},{"begin":47,"length":1},{"begin":58,"length":2},{"begin":61,"length":2},{"begin":64,"length":1},{"begin":67,"length":1},{"begin":72,"length":1},{"begin":77,"length":1},{"begin":80,"length":1},{"begin":82,"length":1},{"begin":84,"length":1},{"begin":87,"length":1},{"begin":92,"length":2},{"begin":99,"length":1},{"begin":104,"length":2},{"begin":111,"length":1},{"begin":155,"length":33}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 msgr2=0x7fa144062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 0x7fa144062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 msgr2=0x7fa1640b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c580 tx=0x7fa154007e10 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 shutdown_connections 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa144060700 0x7fa144062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164158e70 0x7fa1640c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1640cc470 0x7fa1640c5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 --2- 172.21.15.111:0/2446310667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa16406a540 0x7fa1640b4060 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 >> 172.21.15.111:0/2446310667 conn(0x7fa1640670e0 msgr2=0x7fa1640b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 shutdown_connections 2023-12-19T07:59:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.589+0000 7fa1692e3700 1 -- 172.21.15.111:0/2446310667 wait complete. 2023-12-19T07:59:52.608 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T07:59:52.609 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:59:52.624 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.625+0000 7f8c99549000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T07:59:52.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.754+0000 7f65b81ab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:59:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.765+0000 7fe832ef0700 1 Processor -- start 2023-12-19T07:59:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.765+0000 7fe832ef0700 1 -- start start 2023-12-19T07:59:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe832ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0b0110 0x7fe82c0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe832ef0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe82c0ccfd0 0x7fe82c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe832ef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c0ccb40 con 0x7fe82c153290 2023-12-19T07:59:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe832ef0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0cbf30 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe832ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0cc210 con 0x7fe82c0b0110 2023-12-19T07:59:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe830c8c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0b0110 0x7fe82c0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe82c0ccfd0 0x7fe82c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe83148d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe82c153290 0x7fe82c1536d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38864/0 (socket says 172.21.15.111:38864) 2023-12-19T07:59:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe82c0ccfd0 0x7fe82c15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:34792/0 (socket says 172.21.15.111:34792) 2023-12-19T07:59:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe83148d700 1 -- 172.21.15.111:0/2139675396 learned_addr learned my addr 172.21.15.111:0/2139675396 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 -- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0b0110 msgr2=0x7fe82c0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 --2- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0b0110 0x7fe82c0b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 -- 172.21.15.111:0/2139675396 >> v1:172.21.15.143:6789/0 conn(0x7fe82c153290 legacy=0x7fe82c1536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.769+0000 7fe82bfff700 1 -- 172.21.15.111:0/2139675396 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c0cc4f0 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe82bfff700 1 --2- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c15a170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098e0 tx=0x7fe8180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7a439825ee9d4e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe82b7fe700 1 -- 172.21.15.111:0/2139675396 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81800b640 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe82b7fe700 1 -- 172.21.15.111:0/2139675396 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81800b7a0 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe82b7fe700 1 -- 172.21.15.111:0/2139675396 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81800bb40 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 -- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 msgr2=0x7fe82c15a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 --2- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c15a170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098e0 tx=0x7fe8180091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 -- 172.21.15.111:0/2139675396 shutdown_connections 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 --2- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c15a170 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 --2- 172.21.15.111:0/2139675396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0b0110 0x7fe82c0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 -- 172.21.15.111:0/2139675396 >> 172.21.15.111:0/2139675396 conn(0x7fe82c06c7c0 msgr2=0x7fe82c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 -- 172.21.15.111:0/2139675396 shutdown_connections 2023-12-19T07:59:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.773+0000 7fe832ef0700 1 -- 172.21.15.111:0/2139675396 wait complete. 2023-12-19T07:59:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 Processor -- start 2023-12-19T07:59:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- start start 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c0a9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c153290 0x7fe82c0a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0c7b20 con 0x7fe82c0b0110 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c15a590 con 0x7fe82c0ccfd0 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c15a870 con 0x7fe82c153290 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54596/0 (socket says 172.21.15.111:54596) 2023-12-19T07:59:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 -- 172.21.15.111:0/3550403537 learned_addr learned my addr 172.21.15.111:0/3550403537 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe83148d700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c153290 0x7fe82c0a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe82bfff700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c0a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c153290 msgr2=0x7fe82c0a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c153290 0x7fe82c0a9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 msgr2=0x7fe82c0a9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c0a9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 -- 172.21.15.111:0/3550403537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818009420 con 0x7fe82c0b0110 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe82bfff700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c0a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:52.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe830c8c700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe82000ec80 tx=0x7fe8200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82000c9d0 con 0x7fe82c0b0110 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82000cb30 con 0x7fe82c0b0110 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c15ab50 con 0x7fe82c0b0110 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820013470 con 0x7fe82c0b0110 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.777+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c152070 con 0x7fe82c0b0110 2023-12-19T07:59:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.781+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c0b3ea0 con 0x7fe82c0b0110 2023-12-19T07:59:52.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.781+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7fe820013820 con 0x7fe82c0b0110 2023-12-19T07:59:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.785+0000 7fe828ff9700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 0x7fe810062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.785+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7308+0+0 (secure 0 0 0) 0x7fe82007d520 con 0x7fe82c0b0110 2023-12-19T07:59:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.785+0000 7fe82bfff700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 msgr2=0x7fe810062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.785+0000 7fe82bfff700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 0x7fe810062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:52.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.789+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe82c0b3ea0 con 0x7fe82c0b0110 2023-12-19T07:59:52.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.826+0000 7f65b81ab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:59:52.862 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.861+0000 7f8c99549000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T07:59:52.909 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:52.910+0000 7f65b81ab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:59:52.947 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:52.949+0000 7f8c99549000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T07:59:52.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.989+0000 7fe82bfff700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 msgr2=0x7fe810062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:52.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.989+0000 7fe82bfff700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 0x7fe810062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:59:52.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:52.997+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe82c06bfd0 con 0x7fe82c0b0110 2023-12-19T07:59:52.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.001+0000 7fe828ff9700 1 -- 172.21.15.111:0/3550403537 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fe82c06bfd0 con 0x7fe82c0b0110 2023-12-19T07:59:53.000 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:53.000 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 msgr2=0x7fe810062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 0x7fe810062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 msgr2=0x7fe82c151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe82000ec80 tx=0x7fe8200091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 shutdown_connections 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe810060700 0x7fe810062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c0ccfd0 0x7fe82c0a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c153290 0x7fe82c0a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 --2- 172.21.15.111:0/3550403537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe82c0b0110 0x7fe82c151b30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 >> 172.21.15.111:0/3550403537 conn(0x7fe82c06c7c0 msgr2=0x7fe82c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:53.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 shutdown_connections 2023-12-19T07:59:53.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.005+0000 7fe832ef0700 1 -- 172.21.15.111:0/3550403537 wait complete. 2023-12-19T07:59:53.007 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:59:53.023 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T07:59:53.039 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:53.041+0000 7f8c99549000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T07:59:53.100 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:53.106+0000 7f65b81ab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:59:53.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:53.190+0000 7f65b81ab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:59:53.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.201+0000 7f0a6a66e700 1 Processor -- start 2023-12-19T07:59:53.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.201+0000 7f0a6a66e700 1 -- start start 2023-12-19T07:59:53.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a6a66e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a6a66e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0a6406de30 0x7f0a640c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a6a66e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6406d9a0 con 0x7f0a640cc5e0 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a6a66e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640cce80 con 0x7f0a6406de30 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a6a66e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640ad250 con 0x7f0a640c8ad0 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a5bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0a6406de30 0x7f0a640c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:49458/0 (socket says 172.21.15.111:49458) 2023-12-19T07:59:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 -- 172.21.15.111:0/1637588743 learned_addr learned my addr 172.21.15.111:0/1637588743 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:53.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 -- 172.21.15.111:0/1637588743 >> v2:172.21.15.111:3300/0 conn(0x7f0a6406de30 msgr2=0x7f0a640c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:53.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 --2- 172.21.15.111:0/1637588743 >> v2:172.21.15.111:3300/0 conn(0x7f0a6406de30 0x7f0a640c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 -- 172.21.15.111:0/1637588743 >> v1:172.21.15.111:6789/0 conn(0x7f0a640cc5e0 legacy=0x7f0a640670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 -- 172.21.15.111:0/1637588743 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640ccba0 con 0x7f0a640c8ad0 2023-12-19T07:59:53.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a63fff700 1 --2- 172.21.15.111:0/1637588743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a540099b0 tx=0x7f0a54009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8252d56c0f670d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:53.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.205+0000 7f0a637fe700 1 -- 172.21.15.111:0/1637588743 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54009ea0 con 0x7f0a640c8ad0 2023-12-19T07:59:53.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a637fe700 1 -- 172.21.15.111:0/1637588743 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5400b860 con 0x7f0a640c8ad0 2023-12-19T07:59:53.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a637fe700 1 -- 172.21.15.111:0/1637588743 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400bc00 con 0x7f0a640c8ad0 2023-12-19T07:59:53.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- 172.21.15.111:0/1637588743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 msgr2=0x7f0a640cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/1637588743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a540099b0 tx=0x7f0a54009250 comp rx=0 tx=0).stop 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- 172.21.15.111:0/1637588743 shutdown_connections 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/1637588743 >> v2:172.21.15.111:3300/0 conn(0x7f0a6406de30 0x7f0a640c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/1637588743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640c8ad0 0x7f0a640cbbd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- 172.21.15.111:0/1637588743 >> 172.21.15.111:0/1637588743 conn(0x7f0a640ce000 msgr2=0x7f0a640ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- 172.21.15.111:0/1637588743 shutdown_connections 2023-12-19T07:59:53.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- 172.21.15.111:0/1637588743 wait complete. 2023-12-19T07:59:53.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 Processor -- start 2023-12-19T07:59:53.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.209+0000 7f0a6a66e700 1 -- start start 2023-12-19T07:59:53.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a640c8ad0 0x7f0a641496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640cc5e0 0x7f0a64149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640c5910 con 0x7f0a6406de30 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6415bf00 con 0x7f0a640c8ad0 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a640c52c0 con 0x7f0a640cc5e0 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54608/0 (socket says 172.21.15.111:54608) 2023-12-19T07:59:53.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 -- 172.21.15.111:0/2719280846 learned_addr learned my addr 172.21.15.111:0/2719280846 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640cc5e0 msgr2=0x7f0a64149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640cc5e0 0x7f0a64149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a640c8ad0 msgr2=0x7f0a641496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a640c8ad0 0x7f0a641496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 -- 172.21.15.111:0/2719280846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540094f0 con 0x7f0a6406de30 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a63fff700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0a640ad390 tx=0x7f0a54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400bbe0 con 0x7f0a6406de30 2023-12-19T07:59:53.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a54011420 con 0x7f0a6406de30 2023-12-19T07:59:53.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a640c55a0 con 0x7f0a6406de30 2023-12-19T07:59:53.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a540117c0 con 0x7f0a6406de30 2023-12-19T07:59:53.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.213+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6415bcb0 con 0x7f0a6406de30 2023-12-19T07:59:53.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.217+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a640c5120 con 0x7f0a6406de30 2023-12-19T07:59:53.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.217+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f0a54011bb0 con 0x7f0a6406de30 2023-12-19T07:59:53.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.217+0000 7f0a60ff9700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 0x7f0a44062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.217+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6848+0+0 (secure 0 0 0) 0x7f0a5407e8f0 con 0x7f0a6406de30 2023-12-19T07:59:53.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.221+0000 7f0a5bfff700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 msgr2=0x7f0a44062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:53.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.221+0000 7f0a5bfff700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 0x7f0a44062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:53.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.225+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a640c5120 con 0x7f0a6406de30 2023-12-19T07:59:53.266 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:53.269+0000 7f8c99549000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T07:59:53.361 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:53.361+0000 7f8c99549000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T07:59:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.425+0000 7f0a5bfff700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 msgr2=0x7f0a44062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.425+0000 7f0a5bfff700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 0x7f0a44062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T07:59:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.433+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a640c5120 con 0x7f0a6406de30 2023-12-19T07:59:53.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a60ff9700 1 -- 172.21.15.111:0/2719280846 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f0a640c5120 con 0x7f0a6406de30 2023-12-19T07:59:53.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:53.435 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T07:48:25.748981+0000","modified":"2023-12-19T07:48:29.337126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5110":{"gid":5110,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.143:6835/1413455479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1413455479},{"type":"v1","addr":"172.21.15.143:6835","nonce":1413455479}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-12-19T07:59:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 msgr2=0x7f0a44062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 0x7f0a44062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 msgr2=0x7f0a640b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0a640ad390 tx=0x7f0a54017000 comp rx=0 tx=0).stop 2023-12-19T07:59:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 shutdown_connections 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a44060700 0x7f0a44062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a640c8ad0 0x7f0a641496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a640cc5e0 0x7f0a64149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 --2- 172.21.15.111:0/2719280846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6406de30 0x7f0a640b2c10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 >> 172.21.15.111:0/2719280846 conn(0x7f0a640ce000 msgr2=0x7f0a640c9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 shutdown_connections 2023-12-19T07:59:53.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.437+0000 7f0a6a66e700 1 -- 172.21.15.111:0/2719280846 wait complete. 2023-12-19T07:59:53.440 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 16 2023-12-19T07:59:53.456 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:59:53.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.617+0000 7f970759e700 1 Processor -- start 2023-12-19T07:59:53.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.617+0000 7f970759e700 1 -- start start 2023-12-19T07:59:53.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f97080ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806a9d0 0x7f970815af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970806c7c0 con 0x7f970806a580 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970806cd80 con 0x7f970806a9d0 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970806b8f0 con 0x7f97080b1410 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806a9d0 0x7f970815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806a9d0 0x7f970815af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54610/0 (socket says 172.21.15.111:54610) 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 -- 172.21.15.111:0/111047344 learned_addr learned my addr 172.21.15.111:0/111047344 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f970659c700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f97080ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 -- 172.21.15.111:0/111047344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 msgr2=0x7f97080ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:53.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f97080ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 -- 172.21.15.111:0/111047344 >> v1:172.21.15.111:6789/0 conn(0x7f970806a580 legacy=0x7f970815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.621+0000 7f9706d9d700 1 -- 172.21.15.111:0/111047344 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97080ac1e0 con 0x7f970806a9d0 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f9706d9d700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f970815af70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9700009f20 tx=0x7f9700009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bfe36b18d83b70 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970559a700 1 -- 172.21.15.111:0/111047344 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000e070 con 0x7f970806a9d0 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970559a700 1 -- 172.21.15.111:0/111047344 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000b9d0 con 0x7f970806a9d0 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970559a700 1 -- 172.21.15.111:0/111047344 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97000092e0 con 0x7f970806a9d0 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- 172.21.15.111:0/111047344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 msgr2=0x7f970815af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f970815af70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9700009f20 tx=0x7f9700009840 comp rx=0 tx=0).stop 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- 172.21.15.111:0/111047344 shutdown_connections 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f970815af70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 --2- 172.21.15.111:0/111047344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f97080ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- 172.21.15.111:0/111047344 >> 172.21.15.111:0/111047344 conn(0x7f9708067bb0 msgr2=0x7f97080cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- 172.21.15.111:0/111047344 shutdown_connections 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- 172.21.15.111:0/111047344 wait complete. 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 Processor -- start 2023-12-19T07:59:53.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.625+0000 7f970759e700 1 -- start start 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970806a580 0x7f97080c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f970816d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970816dde0 con 0x7f970806a9d0 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970816e060 con 0x7f970806a580 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970816e340 con 0x7f97080b1410 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54612/0 (socket says 172.21.15.111:54612) 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 -- 172.21.15.111:0/2929159285 learned_addr learned my addr 172.21.15.111:0/2929159285 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970659c700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970806a580 0x7f97080c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9706d9d700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f970816d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 -- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 msgr2=0x7f970816d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f970816d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 -- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970806a580 msgr2=0x7f97080c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970806a580 0x7f97080c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 -- 172.21.15.111:0/2929159285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700009ae0 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f9705d9b700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c590 tx=0x7f96fc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011590 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc0116f0 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970816e620 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc011ab0 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970816ea50 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.629+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970816e860 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.633+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 80132+0+0 (secure 0 0 0) 0x7f96fc01b3b0 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.633+0000 7f96f67fc700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 0x7f96e8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.633+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6848+0+0 (secure 0 0 0) 0x7f96fc07c5a0 con 0x7f970806a9d0 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.633+0000 7f970659c700 1 -- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 msgr2=0x7f96e8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.633+0000 7f970659c700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 0x7f96e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T07:59:53.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.641+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f970816e860 con 0x7f970806a9d0 2023-12-19T07:59:53.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.813+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9708157320 con 0x7f970806a9d0 2023-12-19T07:59:53.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.813+0000 7f96f67fc700 1 -- 172.21.15.111:0/2929159285 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16864 (secure 0 0 0) 0x7f9708157320 con 0x7f970806a9d0 2023-12-19T07:59:53.813 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:53.813 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:59:53.062039+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-12-19T07:48:23.875106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-12-19T07:48:24.748962+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.38","mappings":[{"from":5,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3058403037":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1676361505":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/4055025414":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6833/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6832/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/1940071424":"2023-12-20T07:48:40.528791+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":110,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:59:53.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 msgr2=0x7f96e8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 0x7f96e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 msgr2=0x7f9708159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c590 tx=0x7f96fc00c950 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 shutdown_connections 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96e8060700 0x7f96e8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080b1410 0x7f970816d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806a9d0 0x7f9708159400 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 --2- 172.21.15.111:0/2929159285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970806a580 0x7f97080c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 >> 172.21.15.111:0/2929159285 conn(0x7f9708067bb0 msgr2=0x7f97080c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 shutdown_connections 2023-12-19T07:59:53.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.817+0000 7f970759e700 1 -- 172.21.15.111:0/2929159285 wait complete. 2023-12-19T07:59:53.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T07:59:53.940 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:53.942+0000 7f65b81ab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:59:53.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 Processor -- start 2023-12-19T07:59:53.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 -- start start 2023-12-19T07:59:53.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01f806a580 0x7f01f806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f01f806d210 0x7f01f806d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:53.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f806c7c0 con 0x7f01f80b1410 2023-12-19T07:59:53.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f806af40 con 0x7f01f806d210 2023-12-19T07:59:53.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01fc8ad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f806cd80 con 0x7f01f806a580 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f6d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f01f80b1410 0x7f01f80ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41830/0 (socket says 172.21.15.111:41830) 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01f806a580 0x7f01f806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f6d9d700 1 -- 172.21.15.111:0/348764058 learned_addr learned my addr 172.21.15.111:0/348764058 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f759e700 1 --2- 172.21.15.111:0/348764058 >> v2:172.21.15.111:3300/0 conn(0x7f01f806d210 0x7f01f806d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 -- 172.21.15.111:0/348764058 >> v2:172.21.15.111:3300/0 conn(0x7f01f806d210 msgr2=0x7f01f806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 --2- 172.21.15.111:0/348764058 >> v2:172.21.15.111:3300/0 conn(0x7f01f806d210 0x7f01f806d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 -- 172.21.15.111:0/348764058 >> v1:172.21.15.111:6789/0 conn(0x7f01f80b1410 legacy=0x7f01f80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 -- 172.21.15.111:0/348764058 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f806b8f0 con 0x7f01f806a580 2023-12-19T07:59:53.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f659c700 1 --2- 172.21.15.111:0/348764058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f806aa00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f01f8068040 tx=0x7f01ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff9646b0c7a0da78 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:53.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:53.997+0000 7f01f5d9b700 1 -- 172.21.15.111:0/348764058 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec011630 con 0x7f01f806a580 2023-12-19T07:59:53.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01f5d9b700 1 -- 172.21.15.111:0/348764058 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec011790 con 0x7f01f806a580 2023-12-19T07:59:53.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01f5d9b700 1 -- 172.21.15.111:0/348764058 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec011b30 con 0x7f01f806a580 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- 172.21.15.111:0/348764058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 msgr2=0x7f01f806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/348764058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f806aa00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f01f8068040 tx=0x7f01ec0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- 172.21.15.111:0/348764058 shutdown_connections 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/348764058 >> v2:172.21.15.111:3300/0 conn(0x7f01f806d210 0x7f01f806d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/348764058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f806aa00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:53.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- 172.21.15.111:0/348764058 >> 172.21.15.111:0/348764058 conn(0x7f01f8067bb0 msgr2=0x7f01f80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:54.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- 172.21.15.111:0/348764058 shutdown_connections 2023-12-19T07:59:54.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- 172.21.15.111:0/348764058 wait complete. 2023-12-19T07:59:54.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 Processor -- start 2023-12-19T07:59:54.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.001+0000 7f01fc8ad700 1 -- start start 2023-12-19T07:59:54.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f81483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01f80b1410 0x7f01f80c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f8155d10 con 0x7f01f806d210 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f81588b0 con 0x7f01f806a580 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f8158010 con 0x7f01f80b1410 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54628/0 (socket says 172.21.15.111:54628) 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 -- 172.21.15.111:0/2642642346 learned_addr learned my addr 172.21.15.111:0/2642642346 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f759e700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01f80b1410 0x7f01f80c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f6d9d700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f81483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 -- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01f80b1410 msgr2=0x7f01f80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01f80b1410 0x7f01f80c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 -- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 msgr2=0x7f01f81483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f81483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 -- 172.21.15.111:0/2642642346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec009420 con 0x7f01f806d210 2023-12-19T07:59:54.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01f659c700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f01f806cec0 tx=0x7f01ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:54.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec000d10 con 0x7f01f806d210 2023-12-19T07:59:54.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec011b10 con 0x7f01f806d210 2023-12-19T07:59:54.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec025600 con 0x7f01f806d210 2023-12-19T07:59:54.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f81582f0 con 0x7f01f806d210 2023-12-19T07:59:54.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.005+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f8158cb0 con 0x7f01f806d210 2023-12-19T07:59:54.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.009+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f8158b00 con 0x7f01f806d210 2023-12-19T07:59:54.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.009+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 14) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f01ec0259f0 con 0x7f01f806d210 2023-12-19T07:59:54.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.009+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f01ec07e180 con 0x7f01f806d210 2023-12-19T07:59:54.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.017+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f01ec04e4c0 con 0x7f01f806d210 2023-12-19T07:59:54.056 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658310f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658310f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658310f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658310f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658310f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658190c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658190c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658190c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.058 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658190c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.058 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T07:59:54.062+0000 7f658190c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T07:59:54.164 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T07:59:54.165+0000 7f8c99549000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T07:59:54.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.232+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f01f804be40 con 0x7f01f806d210 2023-12-19T07:59:54.234 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T07:59:54.236+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T07:59:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01deffd700 1 -- 172.21.15.111:0/2642642346 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f01ec015090 con 0x7f01f806d210 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 msgr2=0x7f01f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f01f806cec0 tx=0x7f01ec017000 comp rx=0 tx=0).stop 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 shutdown_connections 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01f80b1410 0x7f01f80c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f806d210 0x7f01f80a4de0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.244+0000 7f01fc8ad700 1 --2- 172.21.15.111:0/2642642346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01f806a580 0x7f01f81483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.248+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 >> 172.21.15.111:0/2642642346 conn(0x7f01f8067bb0 msgr2=0x7f01f80ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:54.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.248+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 shutdown_connections 2023-12-19T07:59:54.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.248+0000 7f01fc8ad700 1 -- 172.21.15.111:0/2642642346 wait complete. 2023-12-19T07:59:54.248 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T07:59:54.265 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T07:59:54.311 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T07:59:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.432+0000 7ff4c51c5700 1 Processor -- start 2023-12-19T07:59:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.432+0000 7ff4c51c5700 1 -- start start 2023-12-19T07:59:54.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4c51c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c0158e20 0x7ff4c00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4c51c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4c00cb970 0x7ff4c00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4c51c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00cbdf0 con 0x7ff4c00ca2c0 2023-12-19T07:59:54.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4c51c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00cc0d0 con 0x7ff4c00cb970 2023-12-19T07:59:54.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4c51c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00cc3b0 con 0x7ff4c0158e20 2023-12-19T07:59:54.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4c00cb970 0x7ff4c00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4c00cb970 0x7ff4c00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:34836/0 (socket says 172.21.15.111:34836) 2023-12-19T07:59:54.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 -- 172.21.15.111:0/4020017503 learned_addr learned my addr 172.21.15.111:0/4020017503 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4bed9d700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c0158e20 0x7ff4c00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 -- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c0158e20 msgr2=0x7ff4c00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c0158e20 0x7ff4c00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 -- 172.21.15.111:0/4020017503 >> v1:172.21.15.143:6789/0 conn(0x7ff4c00ca2c0 legacy=0x7ff4c006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 -- 172.21.15.111:0/4020017503 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00cc690 con 0x7ff4c00cb970 2023-12-19T07:59:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4be59c700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c00cb970 0x7ff4c00ceb00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40099b0 tx=0x7ff4b4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eef3599b9956907 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4bdd9b700 1 -- 172.21.15.111:0/4020017503 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400b860 con 0x7ff4c00cb970 2023-12-19T07:59:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4bdd9b700 1 -- 172.21.15.111:0/4020017503 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b400b9c0 con 0x7ff4c00cb970 2023-12-19T07:59:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.436+0000 7ff4bdd9b700 1 -- 172.21.15.111:0/4020017503 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400bd60 con 0x7ff4c00cb970 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c00cb970 msgr2=0x7ff4c00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c00cb970 0x7ff4c00ceb00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40099b0 tx=0x7ff4b4009250 comp rx=0 tx=0).stop 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- 172.21.15.111:0/4020017503 shutdown_connections 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c00cb970 0x7ff4c00ceb00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/4020017503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c0158e20 0x7ff4c00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- 172.21.15.111:0/4020017503 >> 172.21.15.111:0/4020017503 conn(0x7ff4c0067bb0 msgr2=0x7ff4c00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- 172.21.15.111:0/4020017503 shutdown_connections 2023-12-19T07:59:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- 172.21.15.111:0/4020017503 wait complete. 2023-12-19T07:59:54.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 Processor -- start 2023-12-19T07:59:54.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.440+0000 7ff4c51c5700 1 -- start start 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c00ca2c0 0x7ff4c0155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c0158e20 0x7ff4c014bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00b3040 con 0x7ff4c00c6a10 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c0156870 con 0x7ff4c0158e20 2023-12-19T07:59:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4c51c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c0156b50 con 0x7ff4c00ca2c0 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c0158e20 0x7ff4c014bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54642/0 (socket says 172.21.15.111:54642) 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 -- 172.21.15.111:0/536673606 learned_addr learned my addr 172.21.15.111:0/536673606 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bed9d700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c00ca2c0 0x7ff4c0155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 -- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c00ca2c0 msgr2=0x7ff4c0155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c00ca2c0 0x7ff4c0155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 -- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c0158e20 msgr2=0x7ff4c014bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:54.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c0158e20 0x7ff4c014bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:54.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 -- 172.21.15.111:0/536673606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40094f0 con 0x7ff4c00c6a10 2023-12-19T07:59:54.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4bf59e700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c910 tx=0x7ff4b8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:54.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.444+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8006ba0 con 0x7ff4c00c6a10 2023-12-19T07:59:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.448+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8006d20 con 0x7ff4c00c6a10 2023-12-19T07:59:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.448+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8012880 con 0x7ff4c00c6a10 2023-12-19T07:59:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.448+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c0156e30 con 0x7ff4c00c6a10 2023-12-19T07:59:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.448+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c0156460 con 0x7ff4c00c6a10 2023-12-19T07:59:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.448+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c0071120 con 0x7ff4c00c6a10 2023-12-19T07:59:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.452+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 14) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ff4b8012c30 con 0x7ff4c00c6a10 2023-12-19T07:59:54.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.452+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7191+0+0 (secure 0 0 0) 0x7ff4b807c700 con 0x7ff4c00c6a10 2023-12-19T07:59:54.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.460+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff4b804c820 con 0x7ff4c00c6a10 2023-12-19T07:59:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.664+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff4c006d190 con 0x7ff4c00c6a10 2023-12-19T07:59:54.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.968+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 15) v1 ==== 79799+0+0 (secure 0 0 0) 0x7ff4b804c4a0 con 0x7ff4c00c6a10 2023-12-19T07:59:54.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.968+0000 7ff4a6ffd700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 0x7ff4ac068c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:54.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.976+0000 7ff4bed9d700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 0x7ff4ac068c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:54.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:54.976+0000 7ff4bed9d700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 0x7ff4ac068c10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0006fd0 tx=0x7ff4b0008040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:55.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.248+0000 7ff4a6ffd700 1 -- 172.21.15.111:0/536673606 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff4b8069150 con 0x7ff4c00c6a10 2023-12-19T07:59:55.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 msgr2=0x7ff4ac068c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 0x7ff4ac068c10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0006fd0 tx=0x7ff4b0008040 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 msgr2=0x7ff4c00c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c910 tx=0x7ff4b8001710 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 shutdown_connections 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff4ac066830 0x7ff4ac068c10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c0158e20 0x7ff4c014bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c00c6a10 0x7ff4c00c92e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 --2- 172.21.15.111:0/536673606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c00ca2c0 0x7ff4c0155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 >> 172.21.15.111:0/536673606 conn(0x7ff4c0067bb0 msgr2=0x7ff4c00cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 shutdown_connections 2023-12-19T07:59:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.252+0000 7ff4c51c5700 1 -- 172.21.15.111:0/536673606 wait complete. 2023-12-19T07:59:55.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T07:59:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.444+0000 7fe24766e700 1 Processor -- start 2023-12-19T07:59:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.444+0000 7fe24766e700 1 -- start start 2023-12-19T07:59:55.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cb500 0x7fe24015c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 0x7fe2400b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2400b0ac0 con 0x7fe2400cb130 2023-12-19T07:59:55.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24015cbe0 con 0x7fe2400cb500 2023-12-19T07:59:55.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24006b210 con 0x7fe24015c760 2023-12-19T07:59:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cb500 0x7fe24015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cb500 0x7fe24015c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54646/0 (socket says 172.21.15.111:54646) 2023-12-19T07:59:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 -- 172.21.15.111:0/2430612572 learned_addr learned my addr 172.21.15.111:0/2430612572 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe245c0b700 1 --2- 172.21.15.111:0/2430612572 >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 0x7fe2400b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 -- 172.21.15.111:0/2430612572 >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 msgr2=0x7fe2400b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 --2- 172.21.15.111:0/2430612572 >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 0x7fe2400b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 -- 172.21.15.111:0/2430612572 >> v1:172.21.15.143:6789/0 conn(0x7fe2400cb130 legacy=0x7fe24006a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 -- 172.21.15.111:0/2430612572 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2400ca810 con 0x7fe2400cb500 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe245c0b700 1 --2- 172.21.15.111:0/2430612572 >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 0x7fe2400b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe244c09700 1 --2- 172.21.15.111:0/2430612572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb500 0x7fe24015c220 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2400cfc60 tx=0x7fe230009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46a143b0f07641 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe237fff700 1 -- 172.21.15.111:0/2430612572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23000b820 con 0x7fe2400cb500 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe237fff700 1 -- 172.21.15.111:0/2430612572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23000b980 con 0x7fe2400cb500 2023-12-19T07:59:55.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe237fff700 1 -- 172.21.15.111:0/2430612572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23000bd20 con 0x7fe2400cb500 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- 172.21.15.111:0/2430612572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb500 msgr2=0x7fe24015c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 --2- 172.21.15.111:0/2430612572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb500 0x7fe24015c220 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2400cfc60 tx=0x7fe230009250 comp rx=0 tx=0).stop 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- 172.21.15.111:0/2430612572 shutdown_connections 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 --2- 172.21.15.111:0/2430612572 >> v2:172.21.15.143:3300/0 conn(0x7fe24015c760 0x7fe2400b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 --2- 172.21.15.111:0/2430612572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb500 0x7fe24015c220 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- 172.21.15.111:0/2430612572 >> 172.21.15.111:0/2430612572 conn(0x7fe240067bb0 msgr2=0x7fe2400cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- 172.21.15.111:0/2430612572 shutdown_connections 2023-12-19T07:59:55.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.448+0000 7fe24766e700 1 -- 172.21.15.111:0/2430612572 wait complete. 2023-12-19T07:59:55.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 Processor -- start 2023-12-19T07:59:55.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 -- start start 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2400cb500 0x7fe240152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe24015c760 0x7fe240153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe240151730 con 0x7fe2400cb130 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2401535c0 con 0x7fe24015c760 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24766e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400b3960 con 0x7fe2400cb500 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24540a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24540a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54654/0 (socket says 172.21.15.111:54654) 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe24540a700 1 -- 172.21.15.111:0/40552849 learned_addr learned my addr 172.21.15.111:0/40552849 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe244c09700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2400cb500 0x7fe240152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.452+0000 7fe245c0b700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe24015c760 0x7fe240153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 -- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2400cb500 msgr2=0x7fe240152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2400cb500 0x7fe240152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 -- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe24015c760 msgr2=0x7fe240153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe24015c760 0x7fe240153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 -- 172.21.15.111:0/40552849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2300094f0 con 0x7fe2400cb130 2023-12-19T07:59:55.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24540a700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe23c00b600 tx=0x7fe23c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:55.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c00f510 con 0x7fe2400cb130 2023-12-19T07:59:55.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c00f690 con 0x7fe2400cb130 2023-12-19T07:59:55.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c00fa50 con 0x7fe2400cb130 2023-12-19T07:59:55.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2400b3c40 con 0x7fe2400cb130 2023-12-19T07:59:55.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe240144080 con 0x7fe2400cb130 2023-12-19T07:59:55.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.456+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe240071120 con 0x7fe2400cb130 2023-12-19T07:59:55.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.460+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fe23c01a400 con 0x7fe2400cb130 2023-12-19T07:59:55.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.460+0000 7fe2357fa700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 0x7fe224062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:55.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.460+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7191+0+0 (secure 0 0 0) 0x7fe23c07c3e0 con 0x7fe2400cb130 2023-12-19T07:59:55.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.460+0000 7fe244c09700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 0x7fe224062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:55.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.464+0000 7fe244c09700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 0x7fe224062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2400cfc60 tx=0x7fe230016000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:55.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.468+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe23c04c730 con 0x7fe2400cb130 2023-12-19T07:59:55.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:55.636+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe2400cbe90 con 0x7fe2400cb130 2023-12-19T07:59:56.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.256+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe23c04cb00 con 0x7fe2400cb130 2023-12-19T07:59:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.264+0000 7fe2357fa700 1 -- 172.21.15.111:0/40552849 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe23c04c730 con 0x7fe2400cb130 2023-12-19T07:59:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.264+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 msgr2=0x7fe224062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.264+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 0x7fe224062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe2400cfc60 tx=0x7fe230016000 comp rx=0 tx=0).stop 2023-12-19T07:59:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 msgr2=0x7fe240143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe23c00b600 tx=0x7fe23c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 shutdown_connections 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2240603a0 0x7fe224062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe24015c760 0x7fe240153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2400cb500 0x7fe240152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 --2- 172.21.15.111:0/40552849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cb130 0x7fe240143b40 secure :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe23c00b600 tx=0x7fe23c0091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 >> 172.21.15.111:0/40552849 conn(0x7fe240067bb0 msgr2=0x7fe24006db30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 shutdown_connections 2023-12-19T07:59:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.268+0000 7fe24766e700 1 -- 172.21.15.111:0/40552849 wait complete. 2023-12-19T07:59:56.269 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T07:59:56.288 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T07:59:56.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.472+0000 7f103759e700 1 Processor -- start 2023-12-19T07:59:56.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.472+0000 7f103759e700 1 -- start start 2023-12-19T07:59:56.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.472+0000 7f103759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10380aedc0 0x7f10380af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.472+0000 7f103759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f103806af50 0x7f10380cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103806b3d0 con 0x7f103806d6b0 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10380cc140 con 0x7f10380aedc0 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10380cb3f0 con 0x7f103806af50 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10380aedc0 0x7f10380af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10380aedc0 0x7f10380af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54658/0 (socket says 172.21.15.111:54658) 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 -- 172.21.15.111:0/419932304 learned_addr learned my addr 172.21.15.111:0/419932304 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f1035d9b700 1 --2- 172.21.15.111:0/419932304 >> v2:172.21.15.143:3300/0 conn(0x7f103806af50 0x7f10380cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 -- 172.21.15.111:0/419932304 >> v2:172.21.15.143:3300/0 conn(0x7f103806af50 msgr2=0x7f10380cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 --2- 172.21.15.111:0/419932304 >> v2:172.21.15.143:3300/0 conn(0x7f103806af50 0x7f10380cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 -- 172.21.15.111:0/419932304 >> v1:172.21.15.143:6789/0 conn(0x7f103806d6b0 legacy=0x7f10380af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 -- 172.21.15.111:0/419932304 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103806d220 con 0x7f10380aedc0 2023-12-19T07:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103659c700 1 --2- 172.21.15.111:0/419932304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380aedc0 0x7f10380af190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f10380cfc50 tx=0x7f1028009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fdb8d412eca0e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103559a700 1 -- 172.21.15.111:0/419932304 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102800b860 con 0x7f10380aedc0 2023-12-19T07:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103559a700 1 -- 172.21.15.111:0/419932304 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102800b9c0 con 0x7f10380aedc0 2023-12-19T07:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103559a700 1 -- 172.21.15.111:0/419932304 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102800bd60 con 0x7f10380aedc0 2023-12-19T07:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103759e700 1 -- 172.21.15.111:0/419932304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380aedc0 msgr2=0x7f10380af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.476+0000 7f103759e700 1 --2- 172.21.15.111:0/419932304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380aedc0 0x7f10380af190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f10380cfc50 tx=0x7f1028009250 comp rx=0 tx=0).stop 2023-12-19T07:59:56.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 -- 172.21.15.111:0/419932304 shutdown_connections 2023-12-19T07:59:56.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 --2- 172.21.15.111:0/419932304 >> v2:172.21.15.143:3300/0 conn(0x7f103806af50 0x7f10380cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 --2- 172.21.15.111:0/419932304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380aedc0 0x7f10380af190 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 -- 172.21.15.111:0/419932304 >> 172.21.15.111:0/419932304 conn(0x7f10380cc760 msgr2=0x7f10380ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:56.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 -- 172.21.15.111:0/419932304 shutdown_connections 2023-12-19T07:59:56.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 -- 172.21.15.111:0/419932304 wait complete. 2023-12-19T07:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 Processor -- start 2023-12-19T07:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.480+0000 7f103759e700 1 -- start start 2023-12-19T07:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 0x7f1038159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10380aedc0 0x7f10380c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10380b4250 con 0x7f103806d6b0 2023-12-19T07:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10380b4530 con 0x7f10380aedc0 2023-12-19T07:59:56.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10380b4810 con 0x7f103806af50 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54666/0 (socket says 172.21.15.111:54666) 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 -- 172.21.15.111:0/661359301 learned_addr learned my addr 172.21.15.111:0/661359301 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103659c700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 0x7f1038159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1036d9d700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10380aedc0 0x7f10380c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 -- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 msgr2=0x7f1038159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 0x7f1038159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 -- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10380aedc0 msgr2=0x7f10380c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10380aedc0 0x7f10380c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 -- 172.21.15.111:0/661359301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280094f0 con 0x7f103806d6b0 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103659c700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 0x7f1038159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f1035d9b700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d860 tx=0x7f102c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:56.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c009610 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c009770 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10380b4af0 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c009b30 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.484+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10380c5d10 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.488+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103804be40 con 0x7f103806d6b0 2023-12-19T07:59:56.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.488+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f102c01e460 con 0x7f103806d6b0 2023-12-19T07:59:56.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.492+0000 7f10267fc700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 0x7f1018062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:56.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.492+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6753+0+0 (secure 0 0 0) 0x7f102c07d420 con 0x7f103806d6b0 2023-12-19T07:59:56.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.492+0000 7f103659c700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 0x7f1018062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:56.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.492+0000 7f103659c700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 0x7f1018062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10380cfc50 tx=0x7f1028017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:56.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.500+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f102c04d440 con 0x7f103806d6b0 2023-12-19T07:59:56.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:56.668+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10380c6330 con 0x7f103806d6b0 2023-12-19T07:59:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.264+0000 7f10267fc700 1 -- 172.21.15.111:0/661359301 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f102c050a60 con 0x7f103806d6b0 2023-12-19T07:59:57.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 msgr2=0x7f1018062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 0x7f1018062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f10380cfc50 tx=0x7f1028017000 comp rx=0 tx=0).stop 2023-12-19T07:59:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 msgr2=0x7f10380b3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d860 tx=0x7f102c008040 comp rx=0 tx=0).stop 2023-12-19T07:59:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 shutdown_connections 2023-12-19T07:59:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1018060670 0x7f1018062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10380aedc0 0x7f10380c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103806d6b0 0x7f10380b3790 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 --2- 172.21.15.111:0/661359301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f103806af50 0x7f1038159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 >> 172.21.15.111:0/661359301 conn(0x7f10380cc760 msgr2=0x7f103815bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 shutdown_connections 2023-12-19T07:59:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.272+0000 7f103759e700 1 -- 172.21.15.111:0/661359301 wait complete. 2023-12-19T07:59:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T07:59:57.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T07:59:57.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.456+0000 7f6975e0c700 1 Processor -- start 2023-12-19T07:59:57.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.456+0000 7f6975e0c700 1 -- start start 2023-12-19T07:59:57.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6975e0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6970158c90 0x7f69700caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6975e0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6970159770 0x7f697006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6975e0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697006c850 con 0x7f697006aeb0 2023-12-19T07:59:57.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6975e0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697006b470 con 0x7f6970158c90 2023-12-19T07:59:57.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6975e0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697006b750 con 0x7f6970159770 2023-12-19T07:59:57.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6970158c90 0x7f69700caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6970158c90 0x7f69700caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54678/0 (socket says 172.21.15.111:54678) 2023-12-19T07:59:57.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 -- 172.21.15.111:0/1447830092 learned_addr learned my addr 172.21.15.111:0/1447830092 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:57.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f697560b700 1 --2- 172.21.15.111:0/1447830092 >> v2:172.21.15.143:3300/0 conn(0x7f6970159770 0x7f697006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 -- 172.21.15.111:0/1447830092 >> v2:172.21.15.143:3300/0 conn(0x7f6970159770 msgr2=0x7f697006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:57.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 --2- 172.21.15.111:0/1447830092 >> v2:172.21.15.143:3300/0 conn(0x7f6970159770 0x7f697006cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 -- 172.21.15.111:0/1447830092 >> v1:172.21.15.143:6789/0 conn(0x7f697006aeb0 legacy=0x7f6970159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 -- 172.21.15.111:0/1447830092 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697006ba30 con 0x7f6970158c90 2023-12-19T07:59:57.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f6974e0a700 1 --2- 172.21.15.111:0/1447830092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f69700caf40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f69700cc7f0 tx=0x7f6964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97f40f13d082c5e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.460+0000 7f696f7fe700 1 -- 172.21.15.111:0/1447830092 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964009e70 con 0x7f6970158c90 2023-12-19T07:59:57.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f696f7fe700 1 -- 172.21.15.111:0/1447830092 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400b860 con 0x7f6970158c90 2023-12-19T07:59:57.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f696f7fe700 1 -- 172.21.15.111:0/1447830092 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400bc00 con 0x7f6970158c90 2023-12-19T07:59:57.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- 172.21.15.111:0/1447830092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 msgr2=0x7f69700caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1447830092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f69700caf40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f69700cc7f0 tx=0x7f6964009250 comp rx=0 tx=0).stop 2023-12-19T07:59:57.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- 172.21.15.111:0/1447830092 shutdown_connections 2023-12-19T07:59:57.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1447830092 >> v2:172.21.15.143:3300/0 conn(0x7f6970159770 0x7f697006cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1447830092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f69700caf40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- 172.21.15.111:0/1447830092 >> 172.21.15.111:0/1447830092 conn(0x7f6970067bb0 msgr2=0x7f69700cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:57.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- 172.21.15.111:0/1447830092 shutdown_connections 2023-12-19T07:59:57.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- 172.21.15.111:0/1447830092 wait complete. 2023-12-19T07:59:57.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 Processor -- start 2023-12-19T07:59:57.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.464+0000 7f6975e0c700 1 -- start start 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f697006aeb0 0x7f697014fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6970159770 0x7f6970156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970155f40 con 0x7f6970158c90 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970150bc0 con 0x7f6970159770 2023-12-19T07:59:57.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700ca030 con 0x7f697006aeb0 2023-12-19T07:59:57.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54686/0 (socket says 172.21.15.111:54686) 2023-12-19T07:59:57.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 -- 172.21.15.111:0/1066455836 learned_addr learned my addr 172.21.15.111:0/1066455836 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:57.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6974e0a700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f697006aeb0 0x7f697014fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f697560b700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6970159770 0x7f6970156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 -- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f697006aeb0 msgr2=0x7f697014fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f697006aeb0 0x7f697014fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 -- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6970159770 msgr2=0x7f6970156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6970159770 0x7f6970156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 -- 172.21.15.111:0/1066455836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640094f0 con 0x7f6970158c90 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696ffff700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c7c0 tx=0x7f695c007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c011590 con 0x7f6970158c90 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c0116f0 con 0x7f6970158c90 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69701556a0 con 0x7f6970158c90 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c011ab0 con 0x7f6970158c90 2023-12-19T07:59:57.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.468+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6970155c60 con 0x7f6970158c90 2023-12-19T07:59:57.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.472+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697004be40 con 0x7f6970158c90 2023-12-19T07:59:57.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.472+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f695c01b3b0 con 0x7f6970158c90 2023-12-19T07:59:57.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.472+0000 7f696cff9700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 0x7f6954062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.476+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f695c07c0a0 con 0x7f6970158c90 2023-12-19T07:59:57.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.476+0000 7f6974e0a700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 0x7f6954062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.476+0000 7f6974e0a700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 0x7f6954062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69700cc7f0 tx=0x7f6964017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.480+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f695c04c280 con 0x7f6970158c90 2023-12-19T07:59:57.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.652+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f6970150d00 con 0x7f6970158c90 2023-12-19T07:59:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.660+0000 7f696cff9700 1 -- 172.21.15.111:0/1066455836 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v13) v1 ==== 120+0+0 (secure 0 0 0) 0x7f695c04f8a0 con 0x7f6970158c90 2023-12-19T07:59:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 msgr2=0x7f6954062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 0x7f6954062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69700cc7f0 tx=0x7f6964017000 comp rx=0 tx=0).stop 2023-12-19T07:59:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 msgr2=0x7f6970153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c7c0 tx=0x7f695c007f40 comp rx=0 tx=0).stop 2023-12-19T07:59:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 shutdown_connections 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6954060670 0x7f6954062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6970159770 0x7f6970156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970158c90 0x7f6970153600 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 --2- 172.21.15.111:0/1066455836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f697006aeb0 0x7f697014fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 >> 172.21.15.111:0/1066455836 conn(0x7f6970067bb0 msgr2=0x7f69700ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 shutdown_connections 2023-12-19T07:59:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.664+0000 7f6975e0c700 1 -- 172.21.15.111:0/1066455836 wait complete. 2023-12-19T07:59:57.686 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T07:59:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.856+0000 7f5351a4c700 1 Processor -- start 2023-12-19T07:59:57.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.856+0000 7f5351a4c700 1 -- start start 2023-12-19T07:59:57.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.856+0000 7f5351a4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f534c06ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.856+0000 7f5351a4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c0c7820 con 0x7f534c155520 2023-12-19T07:59:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f5351a4c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c155d60 con 0x7f534c06b360 2023-12-19T07:59:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f5351a4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c06b7e0 con 0x7f534c1558f0 2023-12-19T07:59:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f5350a4a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f534c155520 0x7f534c0c7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41846/0 (socket says 172.21.15.111:41846) 2023-12-19T07:59:57.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f5350a4a700 1 -- 172.21.15.111:0/2681035716 learned_addr learned my addr 172.21.15.111:0/2681035716 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f534c06ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464049773 0 0) 0x7f534c0c7820 con 0x7f534c155520 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c004680 con 0x7f534c155520 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289755876 0 0) 0x7f534c155d60 con 0x7f534c06b360 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c0c7820 con 0x7f534c06b360 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520082047 0 0) 0x7f533c004680 con 0x7f534c155520 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 msgr2=0x7f534c06ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:57.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f534c06ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 >> v1:172.21.15.143:6789/0 conn(0x7f534c06b360 legacy=0x7f534c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0abe10 con 0x7f534c155520 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f5340001470 con 0x7f534c155520 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 >> v1:172.21.15.111:6789/0 conn(0x7f534c155520 legacy=0x7f534c0c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f533c004d30 0x7f533c005140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f533c005920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f533c005e60 0x7f533c0062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c06b7e0 con 0x7f533c004d30 2023-12-19T07:59:57.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c003670 con 0x7f533c005e60 2023-12-19T07:59:57.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c003830 con 0x7f534c1558f0 2023-12-19T07:59:57.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f5350a4a700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f533c004d30 0x7f533c005140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f533c005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f535124b700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f533c005e60 0x7f533c0062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 -- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f533c005e60 msgr2=0x7f533c0062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f533c005e60 0x7f533c0062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 -- 172.21.15.111:0/2681035716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f533c004d30 msgr2=0x7f533c005140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f533c004d30 0x7f533c005140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.860+0000 7f534bfff700 1 -- 172.21.15.111:0/2681035716 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c004680 con 0x7f534c1558f0 2023-12-19T07:59:57.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5350a4a700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f533c004d30 0x7f533c005140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:57.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f534bfff700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f533c005920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5338009c00 tx=0x7f53380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44abca5c4236dfa1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800e710 con 0x7f534c1558f0 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533800e870 con 0x7f534c1558f0 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f534b7fe700 1 -- 172.21.15.111:0/2681035716 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800ec30 con 0x7f534c1558f0 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 -- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 msgr2=0x7f533c005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 --2- 172.21.15.111:0/2681035716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c1558f0 0x7f533c005920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5338009c00 tx=0x7f53380091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f535124b700 1 -- 172.21.15.111:0/2681035716 reap_dead start 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 -- 172.21.15.111:0/2681035716 shutdown_connections 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 -- 172.21.15.111:0/2681035716 >> 172.21.15.111:0/2681035716 conn(0x7f534c0c8cd0 msgr2=0x7f534c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 -- 172.21.15.111:0/2681035716 shutdown_connections 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.864+0000 7f5351a4c700 1 -- 172.21.15.111:0/2681035716 wait complete. 2023-12-19T07:59:57.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 Processor -- start 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 -- start start 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f534c1558f0 0x7f534c0c2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c144d60 0x7f534c154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c154ab0 con 0x7f534c154640 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c1694e0 con 0x7f534c1558f0 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0c26b0 con 0x7f534c144d60 2023-12-19T07:59:57.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54694/0 (socket says 172.21.15.111:54694) 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 -- 172.21.15.111:0/3062520193 learned_addr learned my addr 172.21.15.111:0/3062520193 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5350a4a700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f534c1558f0 0x7f534c0c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f534bfff700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c144d60 0x7f534c154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 -- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c144d60 msgr2=0x7f534c154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c144d60 0x7f534c154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 -- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f534c1558f0 msgr2=0x7f534c0c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f534c1558f0 0x7f534c0c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 -- 172.21.15.111:0/3062520193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c14de20 con 0x7f534c154640 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f535124b700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f534c0702d0 tx=0x7f53440017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344006e80 con 0x7f534c154640 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534400c540 con 0x7f534c154640 2023-12-19T07:59:57.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c169760 con 0x7f534c154640 2023-12-19T07:59:57.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.868+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534400c900 con 0x7f534c154640 2023-12-19T07:59:57.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.872+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c169d20 con 0x7f534c154640 2023-12-19T07:59:57.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.872+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534c169f40 con 0x7f534c154640 2023-12-19T07:59:57.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.872+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f534400ccb0 con 0x7f534c154640 2023-12-19T07:59:57.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.876+0000 7f5348ff9700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 0x7f5330062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:57.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.876+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f534407c5d0 con 0x7f534c154640 2023-12-19T07:59:57.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.876+0000 7f5350a4a700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 0x7f5330062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:57.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.876+0000 7f5350a4a700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 0x7f5330062af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5340007c60 tx=0x7f534c0c7790 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:57.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:57.880+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f534404c850 con 0x7f534c154640 2023-12-19T07:59:58.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.048+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f534c04be40 con 0x7f534c154640 2023-12-19T07:59:58.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.060+0000 7f5348ff9700 1 -- 172.21.15.111:0/3062520193 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f534404fe70 con 0x7f534c154640 2023-12-19T07:59:58.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 msgr2=0x7f5330062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 0x7f5330062af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5340007c60 tx=0x7f534c0c7790 comp rx=0 tx=0).stop 2023-12-19T07:59:58.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 msgr2=0x7f534c0bfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f534c0702d0 tx=0x7f53440017a0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 shutdown_connections 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5330060640 0x7f5330062af0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534c154640 0x7f534c0bfd60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f534c144d60 0x7f534c154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 --2- 172.21.15.111:0/3062520193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f534c1558f0 0x7f534c0c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 >> 172.21.15.111:0/3062520193 conn(0x7f534c0c8cd0 msgr2=0x7f534c155ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:58.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 shutdown_connections 2023-12-19T07:59:58.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.064+0000 7f5351a4c700 1 -- 172.21.15.111:0/3062520193 wait complete. 2023-12-19T07:59:58.083 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T07:59:58.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.248+0000 7f006cee3700 1 Processor -- start 2023-12-19T07:59:58.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 -- start start 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0068158c90 0x7f00680caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0068159770 0x7f006806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006806c850 con 0x7f006806aeb0 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006806b470 con 0x7f0068158c90 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f006cee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006806b750 con 0x7f0068159770 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0068158c90 0x7f00680caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0068158c90 0x7f00680caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54708/0 (socket says 172.21.15.111:54708) 2023-12-19T07:59:58.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 -- 172.21.15.111:0/605919404 learned_addr learned my addr 172.21.15.111:0/605919404 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:58.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f0067fff700 1 --2- 172.21.15.111:0/605919404 >> v2:172.21.15.143:3300/0 conn(0x7f0068159770 0x7f006806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 -- 172.21.15.111:0/605919404 >> v2:172.21.15.143:3300/0 conn(0x7f0068159770 msgr2=0x7f006806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:58.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 --2- 172.21.15.111:0/605919404 >> v2:172.21.15.143:3300/0 conn(0x7f0068159770 0x7f006806cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 -- 172.21.15.111:0/605919404 >> v1:172.21.15.143:6789/0 conn(0x7f006806aeb0 legacy=0x7f0068159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.252+0000 7f00677fe700 1 -- 172.21.15.111:0/605919404 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006806ba30 con 0x7f0068158c90 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f00677fe700 1 --2- 172.21.15.111:0/605919404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068158c90 0x7f00680caf40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f00680cc7f0 tx=0x7f0058009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba960a1e368f54e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f00667fc700 1 -- 172.21.15.111:0/605919404 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800b820 con 0x7f0068158c90 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f00667fc700 1 -- 172.21.15.111:0/605919404 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005800b980 con 0x7f0068158c90 2023-12-19T07:59:58.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f00667fc700 1 -- 172.21.15.111:0/605919404 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800bd20 con 0x7f0068158c90 2023-12-19T07:59:58.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 -- 172.21.15.111:0/605919404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068158c90 msgr2=0x7f00680caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 --2- 172.21.15.111:0/605919404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068158c90 0x7f00680caf40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f00680cc7f0 tx=0x7f0058009250 comp rx=0 tx=0).stop 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 -- 172.21.15.111:0/605919404 shutdown_connections 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 --2- 172.21.15.111:0/605919404 >> v2:172.21.15.143:3300/0 conn(0x7f0068159770 0x7f006806cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 --2- 172.21.15.111:0/605919404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0068158c90 0x7f00680caf40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 -- 172.21.15.111:0/605919404 >> 172.21.15.111:0/605919404 conn(0x7f0068067bb0 msgr2=0x7f00680cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:58.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 -- 172.21.15.111:0/605919404 shutdown_connections 2023-12-19T07:59:58.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.256+0000 7f006cee3700 1 -- 172.21.15.111:0/605919404 wait complete. 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 Processor -- start 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 -- start start 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0068158c90 0x7f00681556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0068159770 0x7f0068152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006814df90 con 0x7f006806aeb0 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006814cf90 con 0x7f0068159770 2023-12-19T07:59:58.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f006cee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006814d660 con 0x7f0068158c90 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54716/0 (socket says 172.21.15.111:54716) 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 -- 172.21.15.111:0/4235765177 learned_addr learned my addr 172.21.15.111:0/4235765177 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 -- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0068158c90 msgr2=0x7f00681556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0068158c90 0x7f00681556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 -- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0068159770 msgr2=0x7f0068152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0068159770 0x7f0068152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.260+0000 7f00677fe700 1 -- 172.21.15.111:0/4235765177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580094f0 con 0x7f006806aeb0 2023-12-19T07:59:58.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f00677fe700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0068069b70 tx=0x7f0058016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058011de0 con 0x7f006806aeb0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006814d940 con 0x7f006806aeb0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0068155c10 con 0x7f006806aeb0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058020070 con 0x7f006806aeb0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005801b660 con 0x7f006806aeb0 2023-12-19T07:59:58.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.264+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00480052b0 con 0x7f006806aeb0 2023-12-19T07:59:58.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.268+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0058022030 con 0x7f006806aeb0 2023-12-19T07:59:58.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.268+0000 7f004f7fe700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 0x7f0054062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.268+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f005807ea30 con 0x7f006806aeb0 2023-12-19T07:59:58.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.268+0000 7f0066ffd700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 0x7f0054062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.268+0000 7f0066ffd700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 0x7f0054062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f005c007fd0 tx=0x7f005c007b20 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.276+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f005804ec10 con 0x7f006806aeb0 2023-12-19T07:59:58.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.448+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f0048005830 con 0x7f006806aeb0 2023-12-19T07:59:58.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.456+0000 7f004f7fe700 1 -- 172.21.15.111:0/4235765177 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v15) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0058052230 con 0x7f006806aeb0 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 msgr2=0x7f0054062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 0x7f0054062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f005c007fd0 tx=0x7f005c007b20 comp rx=0 tx=0).stop 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 msgr2=0x7f006814ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0068069b70 tx=0x7f0058016000 comp rx=0 tx=0).stop 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 shutdown_connections 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0054060670 0x7f0054062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0068159770 0x7f0068152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0068158c90 0x7f00681556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 --2- 172.21.15.111:0/4235765177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f006806aeb0 0x7f006814ca50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 >> 172.21.15.111:0/4235765177 conn(0x7f0068067bb0 msgr2=0x7f00680affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 shutdown_connections 2023-12-19T07:59:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.460+0000 7f006cee3700 1 -- 172.21.15.111:0/4235765177 wait complete. 2023-12-19T07:59:58.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-12-19T07:59:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 Processor -- start 2023-12-19T07:59:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 -- start start 2023-12-19T07:59:58.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac7013f890 0x7fac700aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7006c9f0 con 0x7fac700c1a80 2023-12-19T07:59:58.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac700c7f20 con 0x7fac7013f890 2023-12-19T07:59:58.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.644+0000 7fac76e2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700c8190 con 0x7fac700c2980 2023-12-19T07:59:58.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac7013f890 0x7fac700aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac7013f890 0x7fac700aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:34952/0 (socket says 172.21.15.111:34952) 2023-12-19T07:59:58.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 -- 172.21.15.111:0/107707732 learned_addr learned my addr 172.21.15.111:0/107707732 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:58.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac67fff700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 -- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 msgr2=0x7fac700c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 -- 172.21.15.111:0/107707732 >> v1:172.21.15.143:6789/0 conn(0x7fac700c1a80 legacy=0x7fac700680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 -- 172.21.15.111:0/107707732 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac700c8470 con 0x7fac7013f890 2023-12-19T07:59:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac74bca700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7013f890 0x7fac700aeba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00d1a0 tx=0x7fac6c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9ad96b2604da256 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac677fe700 1 -- 172.21.15.111:0/107707732 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c010070 con 0x7fac7013f890 2023-12-19T07:59:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac677fe700 1 -- 172.21.15.111:0/107707732 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6c00de10 con 0x7fac7013f890 2023-12-19T07:59:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac677fe700 1 -- 172.21.15.111:0/107707732 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00b320 con 0x7fac7013f890 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 -- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7013f890 msgr2=0x7fac700aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7013f890 0x7fac700aeba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00d1a0 tx=0x7fac6c00ba50 comp rx=0 tx=0).stop 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 -- 172.21.15.111:0/107707732 shutdown_connections 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 --2- 172.21.15.111:0/107707732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7013f890 0x7fac700aeba0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 -- 172.21.15.111:0/107707732 >> 172.21.15.111:0/107707732 conn(0x7fac700c8850 msgr2=0x7fac700c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 -- 172.21.15.111:0/107707732 shutdown_connections 2023-12-19T07:59:58.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.648+0000 7fac76e2e700 1 -- 172.21.15.111:0/107707732 wait complete. 2023-12-19T07:59:58.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.652+0000 7fac76e2e700 1 Processor -- start 2023-12-19T07:59:58.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.652+0000 7fac76e2e700 1 -- start start 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.652+0000 7fac76e2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.652+0000 7fac76e2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.652+0000 7fac76e2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac700c38a0 0x7fac7015b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac76e2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac70157700 con 0x7fac7013f890 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac76e2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac70157f80 con 0x7fac700c38a0 2023-12-19T07:59:58.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac76e2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700c5e60 con 0x7fac700c2980 2023-12-19T07:59:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54728/0 (socket says 172.21.15.111:54728) 2023-12-19T07:59:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 -- 172.21.15.111:0/3785433164 learned_addr learned my addr 172.21.15.111:0/3785433164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac74bca700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac753cb700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac700c38a0 0x7fac7015b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 -- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 msgr2=0x7fac700c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 -- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac700c38a0 msgr2=0x7fac7015b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac700c38a0 0x7fac7015b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 -- 172.21.15.111:0/3785433164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c00bcf0 con 0x7fac7013f890 2023-12-19T07:59:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac67fff700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fac6000ec50 tx=0x7fac600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000c7f0 con 0x7fac7013f890 2023-12-19T07:59:58.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6000c950 con 0x7fac7013f890 2023-12-19T07:59:58.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000cd10 con 0x7fac7013f890 2023-12-19T07:59:58.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac70157c50 con 0x7fac7013f890 2023-12-19T07:59:58.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.656+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac700c5b80 con 0x7fac7013f890 2023-12-19T07:59:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.660+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac70071120 con 0x7fac7013f890 2023-12-19T07:59:58.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.660+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fac60013680 con 0x7fac7013f890 2023-12-19T07:59:58.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.664+0000 7fac64ff9700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:58.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.664+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fac6007d140 con 0x7fac7013f890 2023-12-19T07:59:58.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.664+0000 7fac74bca700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:58.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.664+0000 7fac74bca700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 0x7fac5c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00ba90 tx=0x7fac6c01a000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:58.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.668+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac6004d320 con 0x7fac7013f890 2023-12-19T07:59:58.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:58.900+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fac701580c0 con 0x7fac7013f890 2023-12-19T07:59:59.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.364+0000 7fac64ff9700 1 -- 172.21.15.111:0/3785433164 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v463) v1 ==== 158+0+0 (secure 0 0 0) 0x7fac60050940 con 0x7fac7013f890 2023-12-19T07:59:59.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 msgr2=0x7fac5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 0x7fac5c062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00ba90 tx=0x7fac6c01a000 comp rx=0 tx=0).stop 2023-12-19T07:59:59.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 msgr2=0x7fac7013e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fac6000ec50 tx=0x7fac600091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 shutdown_connections 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fac5c060670 0x7fac5c062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac700c38a0 0x7fac7015b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac7013f890 0x7fac7013e9b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 --2- 172.21.15.111:0/3785433164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac700c2980 0x7fac700c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 >> 172.21.15.111:0/3785433164 conn(0x7fac700c8850 msgr2=0x7fac7015aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 shutdown_connections 2023-12-19T07:59:59.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.368+0000 7fac76e2e700 1 -- 172.21.15.111:0/3785433164 wait complete. 2023-12-19T07:59:59.393 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-12-19T07:59:59.393 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-12-19T07:59:59.393 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T07:59:59.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T07:59:59.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.556+0000 7f22bf0e3700 1 Processor -- start 2023-12-19T07:59:59.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.556+0000 7f22bf0e3700 1 -- start start 2023-12-19T07:59:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bf0e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f22b806a580 0x7f22b806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bf0e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b806d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bf0e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b806c7c0 con 0x7f22b80b1410 2023-12-19T07:59:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bf0e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b806af40 con 0x7f22b806a580 2023-12-19T07:59:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bf0e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b806cd80 con 0x7f22b806d210 2023-12-19T07:59:59.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f22b806a580 0x7f22b806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f22b806a580 0x7f22b806aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:34970/0 (socket says 172.21.15.111:34970) 2023-12-19T07:59:59.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 -- 172.21.15.111:0/2791669763 learned_addr learned my addr 172.21.15.111:0/2791669763 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:59.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22be8e2700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b806d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 -- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 msgr2=0x7f22b806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T07:59:59.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b806d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 -- 172.21.15.111:0/2791669763 >> v1:172.21.15.143:6789/0 conn(0x7f22b80b1410 legacy=0x7f22b80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.560+0000 7f22bd8e0700 1 -- 172.21.15.111:0/2791669763 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b806b8f0 con 0x7f22b806a580 2023-12-19T07:59:59.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b806a580 0x7f22b806aa00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22a80099b0 tx=0x7f22a8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe60502e511bffa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bd0df700 1 -- 172.21.15.111:0/2791669763 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8009ea0 con 0x7f22b806a580 2023-12-19T07:59:59.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bd0df700 1 -- 172.21.15.111:0/2791669763 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a800b860 con 0x7f22b806a580 2023-12-19T07:59:59.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bd0df700 1 -- 172.21.15.111:0/2791669763 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a800bc00 con 0x7f22b806a580 2023-12-19T07:59:59.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 -- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b806a580 msgr2=0x7f22b806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b806a580 0x7f22b806aa00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22a80099b0 tx=0x7f22a8009250 comp rx=0 tx=0).stop 2023-12-19T07:59:59.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 -- 172.21.15.111:0/2791669763 shutdown_connections 2023-12-19T07:59:59.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b806d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/2791669763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b806a580 0x7f22b806aa00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 -- 172.21.15.111:0/2791669763 >> 172.21.15.111:0/2791669763 conn(0x7f22b8067bb0 msgr2=0x7f22b80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:59.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 -- 172.21.15.111:0/2791669763 shutdown_connections 2023-12-19T07:59:59.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.564+0000 7f22bf0e3700 1 -- 172.21.15.111:0/2791669763 wait complete. 2023-12-19T07:59:59.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 Processor -- start 2023-12-19T07:59:59.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 -- start start 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 0x7f22b80c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80c5fd0 con 0x7f22b806a580 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80c6df0 con 0x7f22b80b1410 2023-12-19T07:59:59.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bf0e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b815c710 con 0x7f22b806d210 2023-12-19T07:59:59.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54740/0 (socket says 172.21.15.111:54740) 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 -- 172.21.15.111:0/3578745399 learned_addr learned my addr 172.21.15.111:0/3578745399 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be8e2700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 0x7f22b80c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 -- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 msgr2=0x7f22b80c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b80c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 -- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 msgr2=0x7f22b80c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 0x7f22b80c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 -- 172.21.15.111:0/3578745399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80094f0 con 0x7f22b806a580 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be8e2700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 0x7f22b80c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T07:59:59.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T07:59:59.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.568+0000 7f22be0e1700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c4e0 tx=0x7f22b400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4011590 con 0x7f22b806a580 2023-12-19T07:59:59.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b4011710 con 0x7f22b806a580 2023-12-19T07:59:59.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b80c61e0 con 0x7f22b806a580 2023-12-19T07:59:59.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4011ad0 con 0x7f22b806a580 2023-12-19T07:59:59.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b80c67a0 con 0x7f22b806a580 2023-12-19T07:59:59.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b804be40 con 0x7f22b806a580 2023-12-19T07:59:59.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.572+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f22b401b3b0 con 0x7f22b806a580 2023-12-19T07:59:59.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.576+0000 7f22ae7fc700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 0x7f229c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.576+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f22b407bfe0 con 0x7f22b806a580 2023-12-19T07:59:59.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.576+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 0x7f229c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.576+0000 7f22bd8e0700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 0x7f229c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22a80099b0 tx=0x7f22a8017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.580+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f22b404c260 con 0x7f22b806a580 2023-12-19T07:59:59.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.752+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b80cc340 con 0x7f22b806a580 2023-12-19T07:59:59.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22ae7fc700 1 -- 172.21.15.111:0/3578745399 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+14073 (secure 0 0 0) 0x7f22b4017090 con 0x7f22b806a580 2023-12-19T07:59:59.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T07:59:59.755 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":59,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T07:59:57.264017+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6834/1413455479":"2023-12-20T07:59:54.238292+0000","172.21.15.143:0/975373013":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/321012843":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/135156546":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/3415340599":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/390373130":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/2749223424":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/1940071424":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6832/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6833/33796":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/4055025414":"2023-12-20T07:48:40.528791+0000","172.21.15.143:6835/1413455479":"2023-12-20T07:59:54.238292+0000","172.21.15.143:0/778258468":"2023-12-20T07:59:53.947853+0000","172.21.15.143:0/1676361505":"2023-12-20T07:48:40.528791+0000","172.21.15.143:0/3058403037":"2023-12-20T07:48:40.528791+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 msgr2=0x7f229c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 0x7f229c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22a80099b0 tx=0x7f22a8017000 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 msgr2=0x7f22b81484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c4e0 tx=0x7f22b400c7f0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 shutdown_connections 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f229c060670 0x7f229c062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22b80b1410 0x7f22b80c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f22b806d210 0x7f22b80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 --2- 172.21.15.111:0/3578745399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22b806a580 0x7f22b81484f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 >> 172.21.15.111:0/3578745399 conn(0x7f22b8067bb0 msgr2=0x7f22b80adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 shutdown_connections 2023-12-19T07:59:59.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.756+0000 7f22bf0e3700 1 -- 172.21.15.111:0/3578745399 wait complete. 2023-12-19T07:59:59.782 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T07:59:59.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T07:59:59.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.940+0000 7f25270f0700 1 Processor -- start 2023-12-19T07:59:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f25270f0700 1 -- start start 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f25270f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f25200c6d20 0x7f25200c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f25270f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252006a7a0 con 0x7f252006ac30 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f25270f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25200c7630 con 0x7f252006c7c0 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f25270f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252006d4b0 con 0x7f25200c6d20 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f252568d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f252006ac30 0x7f252006b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41850/0 (socket says 172.21.15.111:41850) 2023-12-19T07:59:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f252568d700 1 -- 172.21.15.111:0/1098883438 learned_addr learned my addr 172.21.15.111:0/1098883438 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T07:59:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.944+0000 7f2524e8c700 1 --2- 172.21.15.111:0/1098883438 >> v2:172.21.15.143:3300/0 conn(0x7f25200c6d20 0x7f25200c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f251f7fe700 1 -- 172.21.15.111:0/1098883438 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591426162 0 0) 0x7f252006a7a0 con 0x7f252006ac30 2023-12-19T07:59:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f2524e8c700 1 -- 172.21.15.111:0/1098883438 >> v1:172.21.15.143:6789/0 conn(0x7f252006c7c0 legacy=0x7f252006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f2524e8c700 1 -- 172.21.15.111:0/1098883438 >> v1:172.21.15.111:6789/0 conn(0x7f252006ac30 legacy=0x7f252006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f2524e8c700 1 -- 172.21.15.111:0/1098883438 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252006d790 con 0x7f25200c6d20 2023-12-19T07:59:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f2524e8c700 1 --2- 172.21.15.111:0/1098883438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25200c6d20 0x7f25200c70f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25140098e0 tx=0x7f25140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33da18da17c37580 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f251f7fe700 1 -- 172.21.15.111:0/1098883438 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514009e80 con 0x7f25200c6d20 2023-12-19T07:59:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f251f7fe700 1 -- 172.21.15.111:0/1098883438 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251400b860 con 0x7f25200c6d20 2023-12-19T07:59:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f251f7fe700 1 -- 172.21.15.111:0/1098883438 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251400bbe0 con 0x7f25200c6d20 2023-12-19T07:59:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 -- 172.21.15.111:0/1098883438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25200c6d20 msgr2=0x7f25200c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 --2- 172.21.15.111:0/1098883438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25200c6d20 0x7f25200c70f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f25140098e0 tx=0x7f25140091a0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 -- 172.21.15.111:0/1098883438 shutdown_connections 2023-12-19T07:59:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 --2- 172.21.15.111:0/1098883438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25200c6d20 0x7f25200c70f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 -- 172.21.15.111:0/1098883438 >> 172.21.15.111:0/1098883438 conn(0x7f2520067bb0 msgr2=0x7f25200cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T07:59:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 -- 172.21.15.111:0/1098883438 shutdown_connections 2023-12-19T07:59:59.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.948+0000 7f25270f0700 1 -- 172.21.15.111:0/1098883438 wait complete. 2023-12-19T07:59:59.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 Processor -- start 2023-12-19T07:59:59.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 -- start start 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 0x7f2520156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 0x7f25201523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.952+0000 7f25270f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520156f10 con 0x7f252006ac30 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f25270f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520169f70 con 0x7f252006c7c0 2023-12-19T07:59:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f25270f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252016a400 con 0x7f25200c6d20 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54744/0 (socket says 172.21.15.111:54744) 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 -- 172.21.15.111:0/3143983088 learned_addr learned my addr 172.21.15.111:0/3143983088 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f252568d700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 0x7f25201523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 -- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 msgr2=0x7f25201523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 0x7f25201523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f251ffff700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 0x7f2520156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 -- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 msgr2=0x7f2520156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 0x7f2520156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 -- 172.21.15.111:0/3143983088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2514009420 con 0x7f252006ac30 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f252568d700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 0x7f25201523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f251ffff700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 0x7f2520156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T07:59:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f2524e8c700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f25200cc660 tx=0x7f2514017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251401f070 con 0x7f252006ac30 2023-12-19T07:59:59.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251400b9d0 con 0x7f252006ac30 2023-12-19T07:59:59.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2520169640 con 0x7f252006ac30 2023-12-19T07:59:59.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514025670 con 0x7f252006ac30 2023-12-19T07:59:59.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2520169c00 con 0x7f252006ac30 2023-12-19T07:59:59.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.956+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2520071120 con 0x7f252006ac30 2023-12-19T07:59:59.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.960+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2514023070 con 0x7f252006ac30 2023-12-19T07:59:59.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.960+0000 7f251cff9700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 0x7f2504062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T07:59:59.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.960+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f251407df10 con 0x7f252006ac30 2023-12-19T07:59:59.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.964+0000 7f251ffff700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 0x7f2504062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T07:59:59.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.964+0000 7f251ffff700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 0x7f2504062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2520152f50 tx=0x7f250c009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T07:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T07:59:59.968+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f251404e130 con 0x7f252006ac30 2023-12-19T08:00:00.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f252006da70 con 0x7f252006ac30 2023-12-19T08:00:00.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f251cff9700 1 -- 172.21.15.111:0/3143983088 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2514015020 con 0x7f252006ac30 2023-12-19T08:00:00.143 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:00:00.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 msgr2=0x7f2504062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 0x7f2504062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2520152f50 tx=0x7f250c009040 comp rx=0 tx=0).stop 2023-12-19T08:00:00.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 msgr2=0x7f25201508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f25200cc660 tx=0x7f2514017000 comp rx=0 tx=0).stop 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 shutdown_connections 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f25040606a0 0x7f2504062b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f252006c7c0 0x7f2520156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25200c6d20 0x7f25201523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 --2- 172.21.15.111:0/3143983088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252006ac30 0x7f25201508b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.144+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 >> 172.21.15.111:0/3143983088 conn(0x7f2520067bb0 msgr2=0x7f2520158530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.148+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 shutdown_connections 2023-12-19T08:00:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.148+0000 7f25270f0700 1 -- 172.21.15.111:0/3143983088 wait complete. 2023-12-19T08:00:00.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-12-19T08:00:00.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.324+0000 7fd797cf0700 1 Processor -- start 2023-12-19T08:00:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd797cf0700 1 -- start start 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd797cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd797cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd790068c70 con 0x7fd790158f70 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd797cf0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd790068060 con 0x7fd790159340 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd797cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd790068340 con 0x7fd790069100 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd795a8c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd790158f70 0x7fd7900cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41864/0 (socket says 172.21.15.111:41864) 2023-12-19T08:00:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd795a8c700 1 -- 172.21.15.111:0/2325548468 learned_addr learned my addr 172.21.15.111:0/2325548468 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd79528b700 1 --2- 172.21.15.111:0/2325548468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.328+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699352559 0 0) 0x7fd790068c70 con 0x7fd790158f70 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c003570 con 0x7fd790158f70 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063303505 0 0) 0x7fd790068060 con 0x7fd790159340 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd79528b700 1 -- 172.21.15.111:0/2325548468 >> v1:172.21.15.143:6789/0 conn(0x7fd790159340 legacy=0x7fd7900cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd79528b700 1 -- 172.21.15.111:0/2325548468 >> v1:172.21.15.111:6789/0 conn(0x7fd790158f70 legacy=0x7fd7900cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd79528b700 1 -- 172.21.15.111:0/2325548468 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790068620 con 0x7fd790069100 2023-12-19T08:00:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd79528b700 1 --2- 172.21.15.111:0/2325548468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790067980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd780009910 tx=0x7fd7800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c016cd9534250d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000b680 con 0x7fd790069100 2023-12-19T08:00:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78000b7e0 con 0x7fd790069100 2023-12-19T08:00:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd794a8a700 1 -- 172.21.15.111:0/2325548468 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000bba0 con 0x7fd790069100 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 -- 172.21.15.111:0/2325548468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 msgr2=0x7fd790067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 --2- 172.21.15.111:0/2325548468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790067980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd780009910 tx=0x7fd7800091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 -- 172.21.15.111:0/2325548468 shutdown_connections 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 --2- 172.21.15.111:0/2325548468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790067980 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 -- 172.21.15.111:0/2325548468 >> 172.21.15.111:0/2325548468 conn(0x7fd7900ccc40 msgr2=0x7fd7900cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 -- 172.21.15.111:0/2325548468 shutdown_connections 2023-12-19T08:00:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.332+0000 7fd797cf0700 1 -- 172.21.15.111:0/2325548468 wait complete. 2023-12-19T08:00:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 Processor -- start 2023-12-19T08:00:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 -- start start 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790169ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd790159340 0x7fd790149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79016a3e0 con 0x7fd790158f70 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7900b2ac0 con 0x7fd790159340 2023-12-19T08:00:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd797cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79016cfa0 con 0x7fd790069100 2023-12-19T08:00:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54746/0 (socket says 172.21.15.111:54746) 2023-12-19T08:00:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 -- 172.21.15.111:0/4004694965 learned_addr learned my addr 172.21.15.111:0/4004694965 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79628d700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd790159340 0x7fd790149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd795a8c700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790169ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 -- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 msgr2=0x7fd790169ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790169ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 -- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd790159340 msgr2=0x7fd790149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd790159340 0x7fd790149ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd79528b700 1 -- 172.21.15.111:0/4004694965 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780009420 con 0x7fd790158f70 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.336+0000 7fd795a8c700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790169ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd79528b700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd780009fb0 tx=0x7fd780017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000bb60 con 0x7fd790158f70 2023-12-19T08:00:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd780027450 con 0x7fd790158f70 2023-12-19T08:00:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7800277f0 con 0x7fd790158f70 2023-12-19T08:00:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79016d280 con 0x7fd790158f70 2023-12-19T08:00:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79016d840 con 0x7fd790158f70 2023-12-19T08:00:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd790071120 con 0x7fd790158f70 2023-12-19T08:00:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.340+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd780011420 con 0x7fd790158f70 2023-12-19T08:00:00.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.344+0000 7fd785ffb700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 0x7fd774062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:00.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.344+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fd78007ec30 con 0x7fd790158f70 2023-12-19T08:00:00.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.344+0000 7fd795a8c700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 0x7fd774062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:00.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.344+0000 7fd795a8c700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 0x7fd774062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0058d0 tx=0x7fd78c009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:00.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.348+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd78004ee10 con 0x7fd790158f70 2023-12-19T08:00:00.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:00.576+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fd790068900 con 0x7fd790158f70 2023-12-19T08:00:01.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.376+0000 7fd785ffb700 1 -- 172.21.15.111:0/4004694965 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v465) v1 ==== 157+0+0 (secure 0 0 0) 0x7fd7800150e0 con 0x7fd790158f70 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 msgr2=0x7fd774062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 0x7fd774062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0058d0 tx=0x7fd78c009040 comp rx=0 tx=0).stop 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 msgr2=0x7fd7900c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd780009fb0 tx=0x7fd780017000 comp rx=0 tx=0).stop 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 shutdown_connections 2023-12-19T08:00:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd774060670 0x7fd774062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd790159340 0x7fd790149ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd790158f70 0x7fd7900c8560 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 --2- 172.21.15.111:0/4004694965 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd790069100 0x7fd790169ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.380+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 >> 172.21.15.111:0/4004694965 conn(0x7fd7900ccc40 msgr2=0x7fd790157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.384+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 shutdown_connections 2023-12-19T08:00:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.384+0000 7fd797cf0700 1 -- 172.21.15.111:0/4004694965 wait complete. 2023-12-19T08:00:01.401 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:00:01.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 Processor -- start 2023-12-19T08:00:01.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 -- start start 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3006c7c0 0x7fef3006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fef3006ac30 0x7fef3006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3006a7a0 con 0x7fef300c6d20 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3006d4b0 con 0x7fef3006ac30 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef35b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3006d790 con 0x7fef3006c7c0 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fef3006ac30 0x7fef3006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fef3006ac30 0x7fef3006b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54758/0 (socket says 172.21.15.111:54758) 2023-12-19T08:00:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 -- 172.21.15.111:0/3591042972 learned_addr learned my addr 172.21.15.111:0/3591042972 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef27fff700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3006c7c0 0x7fef3006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 -- 172.21.15.111:0/3591042972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3006c7c0 msgr2=0x7fef3006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3006c7c0 0x7fef3006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 -- 172.21.15.111:0/3591042972 >> v1:172.21.15.143:6789/0 conn(0x7fef300c6d20 legacy=0x7fef300c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.572+0000 7fef2ffff700 1 -- 172.21.15.111:0/3591042972 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3006da70 con 0x7fef3006ac30 2023-12-19T08:00:01.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef2ffff700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006ac30 0x7fef3006b050 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fef28009f20 tx=0x7fef28009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e659b8af7f74efa6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:01.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef2effd700 1 -- 172.21.15.111:0/3591042972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2800e070 con 0x7fef3006ac30 2023-12-19T08:00:01.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef2effd700 1 -- 172.21.15.111:0/3591042972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2800ba80 con 0x7fef3006ac30 2023-12-19T08:00:01.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef2effd700 1 -- 172.21.15.111:0/3591042972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef280092e0 con 0x7fef3006ac30 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 -- 172.21.15.111:0/3591042972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006ac30 msgr2=0x7fef3006b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006ac30 0x7fef3006b050 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fef28009f20 tx=0x7fef28009840 comp rx=0 tx=0).stop 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 -- 172.21.15.111:0/3591042972 shutdown_connections 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006ac30 0x7fef3006b050 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 --2- 172.21.15.111:0/3591042972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef3006c7c0 0x7fef3006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 -- 172.21.15.111:0/3591042972 >> 172.21.15.111:0/3591042972 conn(0x7fef30067bb0 msgr2=0x7fef300cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 -- 172.21.15.111:0/3591042972 shutdown_connections 2023-12-19T08:00:01.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.576+0000 7fef35b85700 1 -- 172.21.15.111:0/3591042972 wait complete. 2023-12-19T08:00:01.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 Processor -- start 2023-12-19T08:00:01.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 -- start start 2023-12-19T08:00:01.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 0x7fef301501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef300c6d20 0x7fef30152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef300af370 con 0x7fef3006c7c0 2023-12-19T08:00:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3014d4b0 con 0x7fef3006ac30 2023-12-19T08:00:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef35b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef30155a70 con 0x7fef300c6d20 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54760/0 (socket says 172.21.15.111:54760) 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 -- 172.21.15.111:0/2878997656 learned_addr learned my addr 172.21.15.111:0/2878997656 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef2ffff700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef300c6d20 0x7fef30152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef2f7fe700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 0x7fef301501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 -- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef300c6d20 msgr2=0x7fef30152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef300c6d20 0x7fef30152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 -- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 msgr2=0x7fef301501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 0x7fef301501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef27fff700 1 -- 172.21.15.111:0/2878997656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef28009ae0 con 0x7fef3006c7c0 2023-12-19T08:00:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.580+0000 7fef2f7fe700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 0x7fef301501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef27fff700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fef1800d890 tx=0x7fef18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef180125f0 con 0x7fef3006c7c0 2023-12-19T08:00:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18012770 con 0x7fef3006c7c0 2023-12-19T08:00:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef300aed20 con 0x7fef3006c7c0 2023-12-19T08:00:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18012b30 con 0x7fef3006c7c0 2023-12-19T08:00:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef30155670 con 0x7fef3006c7c0 2023-12-19T08:00:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.584+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef30071120 con 0x7fef3006c7c0 2023-12-19T08:00:01.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.588+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fef1801e460 con 0x7fef3006c7c0 2023-12-19T08:00:01.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.588+0000 7fef277fe700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 0x7fef10062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:01.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.588+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fef1807d2a0 con 0x7fef3006c7c0 2023-12-19T08:00:01.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.588+0000 7fef2f7fe700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 0x7fef10062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:01.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.592+0000 7fef2f7fe700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 0x7fef10062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef20007fd0 tx=0x7fef20009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:01.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.592+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fef1804d480 con 0x7fef3006c7c0 2023-12-19T08:00:01.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.764+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fef3006dd50 con 0x7fef3006c7c0 2023-12-19T08:00:01.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.772+0000 7fef277fe700 1 -- 172.21.15.111:0/2878997656 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef18050aa0 con 0x7fef3006c7c0 2023-12-19T08:00:01.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 msgr2=0x7fef10062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 0x7fef10062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fef20007fd0 tx=0x7fef20009040 comp rx=0 tx=0).stop 2023-12-19T08:00:01.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 msgr2=0x7fef3014cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:01.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fef1800d890 tx=0x7fef18008040 comp rx=0 tx=0).stop 2023-12-19T08:00:01.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 shutdown_connections 2023-12-19T08:00:01.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fef10060670 0x7fef10062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef300c6d20 0x7fef30152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef3006c7c0 0x7fef3014cf70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 --2- 172.21.15.111:0/2878997656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef3006ac30 0x7fef301501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:01.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 >> 172.21.15.111:0/2878997656 conn(0x7fef30067bb0 msgr2=0x7fef300c0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:01.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 shutdown_connections 2023-12-19T08:00:01.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:01.776+0000 7fef35b85700 1 -- 172.21.15.111:0/2878997656 wait complete. 2023-12-19T08:00:01.799 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:00:01.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:00:01.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:00:01.889 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:00:01.889 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:00:01.889 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:00:02.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.056+0000 7f5301945700 1 Processor -- start 2023-12-19T08:00:02.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.056+0000 7f5301945700 1 -- start start 2023-12-19T08:00:02.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f5301945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52fc15c5b0 0x7f52fc0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f5301945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52fc06cfb0 con 0x7f52fc0ccd80 2023-12-19T08:00:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f5301945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52fc0cb660 con 0x7f52fc0cc9b0 2023-12-19T08:00:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f5301945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52fc0cb380 con 0x7f52fc15c5b0 2023-12-19T08:00:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f52fc0ccd80 0x7f52fc15bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37754/0 (socket says 172.21.15.111:37754) 2023-12-19T08:00:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 -- 172.21.15.111:0/873249019 learned_addr learned my addr 172.21.15.111:0/873249019 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:02.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fb7fe700 1 --2- 172.21.15.111:0/873249019 >> v2:172.21.15.143:3300/0 conn(0x7f52fc15c5b0 0x7f52fc0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284108772 0 0) 0x7f52fc06cfb0 con 0x7f52fc0ccd80 2023-12-19T08:00:02.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e4003570 con 0x7f52fc0ccd80 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792518067 0 0) 0x7f52e4003570 con 0x7f52fc0ccd80 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 >> v2:172.21.15.143:3300/0 conn(0x7f52fc15c5b0 msgr2=0x7f52fc0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 --2- 172.21.15.111:0/873249019 >> v2:172.21.15.143:3300/0 conn(0x7f52fc15c5b0 0x7f52fc0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 >> v1:172.21.15.143:6789/0 conn(0x7f52fc0cc9b0 legacy=0x7f52fc0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc06c960 con 0x7f52fc0ccd80 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f52f0001470 con 0x7f52fc0ccd80 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 >> v1:172.21.15.111:6789/0 conn(0x7f52fc0ccd80 legacy=0x7f52fc15bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e4003a60 0x7f52e4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 0x7f52e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40051c0 0x7f52e400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0cb380 con 0x7f52e4004810 2023-12-19T08:00:02.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e400c0d0 con 0x7f52e4003a60 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e400c290 con 0x7f52e40051c0 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 0x7f52e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52faffd700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e4003a60 0x7f52e4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fb7fe700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40051c0 0x7f52e400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 -- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40051c0 msgr2=0x7f52e400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52e40051c0 0x7f52e400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 -- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e4003a60 msgr2=0x7f52e4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52e4003a60 0x7f52e4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fa7fc700 1 -- 172.21.15.111:0/873249019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e4003570 con 0x7f52e4004810 2023-12-19T08:00:02.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.060+0000 7f52fb7fe700 1 -- 172.21.15.111:0/873249019 reap_dead start 2023-12-19T08:00:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 0x7f52e4004c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f52f000c3c0 tx=0x7f52f0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ace7d4a69cebac7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f000cc10 con 0x7f52e4004810 2023-12-19T08:00:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f000cd70 con 0x7f52e4004810 2023-12-19T08:00:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f52f9ffb700 1 -- 172.21.15.111:0/873249019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0005b50 con 0x7f52e4004810 2023-12-19T08:00:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 msgr2=0x7f52e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 0x7f52e4004c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f52f000c3c0 tx=0x7f52f0001e50 comp rx=0 tx=0).stop 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- 172.21.15.111:0/873249019 shutdown_connections 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 --2- 172.21.15.111:0/873249019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e4004810 0x7f52e4004c80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- 172.21.15.111:0/873249019 >> 172.21.15.111:0/873249019 conn(0x7f52fc0670e0 msgr2=0x7f52fc0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- 172.21.15.111:0/873249019 shutdown_connections 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- 172.21.15.111:0/873249019 wait complete. 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 Processor -- start 2023-12-19T08:00:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.064+0000 7f5301945700 1 -- start start 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 0x7f52fc15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc0c47b0 0x7f52fc0c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc15d400 con 0x7f52fc1481c0 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc16c100 con 0x7f52fc15ca50 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc16c350 con 0x7f52fc0c47b0 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46104/0 (socket says 172.21.15.111:46104) 2023-12-19T08:00:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 -- 172.21.15.111:0/3441098537 learned_addr learned my addr 172.21.15.111:0/3441098537 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 0x7f52fc15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52fb7fe700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc0c47b0 0x7f52fc0c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 -- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc0c47b0 msgr2=0x7f52fc0c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc0c47b0 0x7f52fc0c4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 -- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 msgr2=0x7f52fc15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 0x7f52fc15cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 -- 172.21.15.111:0/3441098537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc16c630 con 0x7f52fc1481c0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 0x7f52fc15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52faffd700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f52ec002950 tx=0x7f52ec007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec002e90 con 0x7f52fc1481c0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec00f8d0 con 0x7f52fc1481c0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc16c910 con 0x7f52fc1481c0 2023-12-19T08:00:02.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec00fc70 con 0x7f52fc1481c0 2023-12-19T08:00:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc16cf60 con 0x7f52fc1481c0 2023-12-19T08:00:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.068+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc04be40 con 0x7f52fc1481c0 2023-12-19T08:00:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.072+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f52ec006820 con 0x7f52fc1481c0 2023-12-19T08:00:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.072+0000 7f52e2ffd700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 0x7f52e8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.072+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f52ec07ce50 con 0x7f52fc1481c0 2023-12-19T08:00:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.072+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 0x7f52e8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.076+0000 7f52fa7fc700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 0x7f52e8062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52fc04c4e0 tx=0x7f52f0015000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.080+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f52ec04d030 con 0x7f52fc1481c0 2023-12-19T08:00:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.296+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52fc06cc40 con 0x7f52fc1481c0 2023-12-19T08:00:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f52e2ffd700 1 -- 172.21.15.111:0/3441098537 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f52ec050650 con 0x7f52fc1481c0 2023-12-19T08:00:02.299 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:02.299 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/289168718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":289168718},{"type":"v1","addr":"172.21.15.143:6837","nonce":289168718}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5091,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/454642445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":454642445},{"type":"v1","addr":"172.21.15.111:6833","nonce":454642445}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5100,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1174777214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1174777214},{"type":"v1","addr":"172.21.15.111:6835","nonce":1174777214}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5785,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1445448202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1445448202},{"type":"v1","addr":"172.21.15.143:6835","nonce":1445448202}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 msgr2=0x7f52e8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 0x7f52e8062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52fc04c4e0 tx=0x7f52f0015000 comp rx=0 tx=0).stop 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 msgr2=0x7f52fc0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f52ec002950 tx=0x7f52ec007d30 comp rx=0 tx=0).stop 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 shutdown_connections 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52e8060670 0x7f52e8062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc15ca50 0x7f52fc15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc0c47b0 0x7f52fc0c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 --2- 172.21.15.111:0/3441098537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52fc1481c0 0x7f52fc0b4840 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 >> 172.21.15.111:0/3441098537 conn(0x7f52fc0670e0 msgr2=0x7f52fc0b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 shutdown_connections 2023-12-19T08:00:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.300+0000 7f5301945700 1 -- 172.21.15.111:0/3441098537 wait complete. 2023-12-19T08:00:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 18 2023-12-19T08:00:02.323 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:00:02.323 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:00:02.323 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:00:02.323 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:00:02.324 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:00:02.324 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:00:02.324 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:00:02.324 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:00:02.324 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:00:02.324 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:00:02.324 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:00:02.325 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:00:02.325 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:00:02.325 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:00:02.325 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:00:02.325 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:00:02.337 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:00:02.338 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:00:02.342 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:00:02.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:00:02.344 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:00:02.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:00:02.347 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:00:02.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:00:02.390 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:00:02.390 DEBUG:teuthology.parallel:result is None 2023-12-19T08:00:02.393 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:00:02.393 DEBUG:teuthology.parallel:result is None 2023-12-19T08:00:02.393 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:00:02.394 DEBUG:teuthology.parallel:result is None 2023-12-19T08:00:02.399 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:00:02.399 DEBUG:teuthology.parallel:result is None 2023-12-19T08:00:02.400 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:00:02.488 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:00:02.493 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:00:02.512 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:00:02.513 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:00:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 Processor -- start 2023-12-19T08:00:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 -- start start 2023-12-19T08:00:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d4156880 0x7fc7d40c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7d40c6dc0 0x7fc7d40ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 0x7fc7d40adc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d40c7300 con 0x7fc7d4156880 2023-12-19T08:00:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d406d1d0 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d986e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d406dd10 con 0x7fc7d4156c50 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d4156880 0x7fc7d40c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d4156880 0x7fc7d40c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46160/0 (socket says 172.21.15.111:46160) 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/1478170039 learned_addr learned my addr 172.21.15.111:0/1478170039 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d27fc700 1 --2- 172.21.15.111:0/1478170039 >> v2:172.21.15.143:3300/0 conn(0x7fc7d40c6dc0 0x7fc7d40ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d37fe700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 0x7fc7d40adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/1478170039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 msgr2=0x7fc7d40adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 0x7fc7d40adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/1478170039 >> v2:172.21.15.143:3300/0 conn(0x7fc7d40c6dc0 msgr2=0x7fc7d40ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/1478170039 >> v2:172.21.15.143:3300/0 conn(0x7fc7d40c6dc0 0x7fc7d40ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/1478170039 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d406d3e0 con 0x7fc7d4156880 2023-12-19T08:00:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d37fe700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 0x7fc7d40adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.556+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d4156880 0x7fc7d40c6880 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc7d406a380 tx=0x7fc7c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edba9e7a370e0340 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d1ffb700 1 -- 172.21.15.111:0/1478170039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400b860 con 0x7fc7d4156880 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d1ffb700 1 -- 172.21.15.111:0/1478170039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c400b9c0 con 0x7fc7d4156880 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d1ffb700 1 -- 172.21.15.111:0/1478170039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400bd60 con 0x7fc7d4156880 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- 172.21.15.111:0/1478170039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d4156880 msgr2=0x7fc7d40c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d4156880 0x7fc7d40c6880 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc7d406a380 tx=0x7fc7c4009250 comp rx=0 tx=0).stop 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- 172.21.15.111:0/1478170039 shutdown_connections 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156c50 0x7fc7d40adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 --2- 172.21.15.111:0/1478170039 >> v2:172.21.15.143:3300/0 conn(0x7fc7d40c6dc0 0x7fc7d40ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 --2- 172.21.15.111:0/1478170039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d4156880 0x7fc7d40c6880 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- 172.21.15.111:0/1478170039 >> 172.21.15.111:0/1478170039 conn(0x7fc7d4067bb0 msgr2=0x7fc7d40cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- 172.21.15.111:0/1478170039 shutdown_connections 2023-12-19T08:00:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- 172.21.15.111:0/1478170039 wait complete. 2023-12-19T08:00:02.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 Processor -- start 2023-12-19T08:00:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.560+0000 7fc7d986e700 1 -- start start 2023-12-19T08:00:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156880 0x7fc7d4157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 0x7fc7d40c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4159cd0 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4158260 con 0x7fc7d4156c50 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4159eb0 con 0x7fc7d4156880 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46164/0 (socket says 172.21.15.111:46164) 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/2193840551 learned_addr learned my addr 172.21.15.111:0/2193840551 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d37fe700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 0x7fc7d40c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156880 msgr2=0x7fc7d4157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156880 0x7fc7d4157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 msgr2=0x7fc7d40c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 0x7fc7d40c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 -- 172.21.15.111:0/2193840551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c40094f0 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d37fe700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 0x7fc7d40c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d2ffd700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40cb8c0 tx=0x7fc7c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4024440 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c40245a0 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d40c4340 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4024960 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.564+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d40c5360 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.568+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d40c4550 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.568+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc7c401a420 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.568+0000 7fc7baffd700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 0x7fc7c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.568+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fc7c407e9b0 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.568+0000 7fc7d27fc700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 0x7fc7c0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.572+0000 7fc7d27fc700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 0x7fc7c0062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40cc740 tx=0x7fc7c8008040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.572+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc7c404ec30 con 0x7fc7d40c6dc0 2023-12-19T08:00:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:02.752+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc7d404be40 con 0x7fc7d40c6dc0 2023-12-19T08:00:03.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.804+0000 7fc7baffd700 1 -- 172.21.15.111:0/2193840551 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc7c4015080 con 0x7fc7d40c6dc0 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 msgr2=0x7fc7c0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 0x7fc7c0062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40cc740 tx=0x7fc7c8008040 comp rx=0 tx=0).stop 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 msgr2=0x7fc7d4159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40cb8c0 tx=0x7fc7c4017000 comp rx=0 tx=0).stop 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 shutdown_connections 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc7c0060670 0x7fc7c0062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d4156c50 0x7fc7d40c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d4156880 0x7fc7d4157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:03.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 --2- 172.21.15.111:0/2193840551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d40c6dc0 0x7fc7d4159790 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:03.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 >> 172.21.15.111:0/2193840551 conn(0x7fc7d4067bb0 msgr2=0x7fc7d406c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:03.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 shutdown_connections 2023-12-19T08:00:03.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:03.808+0000 7fc7d986e700 1 -- 172.21.15.111:0/2193840551 wait complete. 2023-12-19T08:00:03.812 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:00:03.831 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:00:04.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.008+0000 7f128d4ae700 1 Processor -- start 2023-12-19T08:00:04.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.008+0000 7f128d4ae700 1 -- start start 2023-12-19T08:00:04.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12880ad150 0x7f128806a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f1288159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128806c9f0 con 0x7f12880ca010 2023-12-19T08:00:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12880ca860 con 0x7f12880ad150 2023-12-19T08:00:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12880c8d20 con 0x7f12880ca3e0 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12880ad150 0x7f128806a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12880ad150 0x7f128806a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46180/0 (socket says 172.21.15.111:46180) 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 -- 172.21.15.111:0/1423147146 learned_addr learned my addr 172.21.15.111:0/1423147146 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f1287fff700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f1288159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 -- 172.21.15.111:0/1423147146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 msgr2=0x7f1288159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f1288159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 -- 172.21.15.111:0/1423147146 >> v1:172.21.15.143:6789/0 conn(0x7f12880ca010 legacy=0x7f12880c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 -- 172.21.15.111:0/1423147146 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12880c8480 con 0x7f12880ad150 2023-12-19T08:00:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f1287fff700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f1288159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f12867fc700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ad150 0x7f128806a3f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f127c0099b0 tx=0x7f127c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2662551b6af044 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f1285ffb700 1 -- 172.21.15.111:0/1423147146 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c00b820 con 0x7f12880ad150 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f1285ffb700 1 -- 172.21.15.111:0/1423147146 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127c00b980 con 0x7f12880ad150 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f1285ffb700 1 -- 172.21.15.111:0/1423147146 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c00bd20 con 0x7f12880ad150 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 -- 172.21.15.111:0/1423147146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ad150 msgr2=0x7f128806a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.012+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ad150 0x7f128806a3f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f127c0099b0 tx=0x7f127c009250 comp rx=0 tx=0).stop 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- 172.21.15.111:0/1423147146 shutdown_connections 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f1288159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1423147146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ad150 0x7f128806a3f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- 172.21.15.111:0/1423147146 >> 172.21.15.111:0/1423147146 conn(0x7f1288068500 msgr2=0x7f12880688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- 172.21.15.111:0/1423147146 shutdown_connections 2023-12-19T08:00:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- 172.21.15.111:0/1423147146 wait complete. 2023-12-19T08:00:04.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 Processor -- start 2023-12-19T08:00:04.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- start start 2023-12-19T08:00:04.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12880ad150 0x7f128814c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f12880b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128815b210 con 0x7f12880ca010 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128815a600 con 0x7f12880ad150 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f128d4ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128815a8e0 con 0x7f12880ca3e0 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f12867fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f12867fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46190/0 (socket says 172.21.15.111:46190) 2023-12-19T08:00:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.016+0000 7f12867fc700 1 -- 172.21.15.111:0/1063079009 learned_addr learned my addr 172.21.15.111:0/1063079009 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f1287fff700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f12880b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f1286ffd700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12880ad150 0x7f128814c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 -- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 msgr2=0x7f12880b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f12880b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 -- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12880ad150 msgr2=0x7f128814c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12880ad150 0x7f128814c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 -- 172.21.15.111:0/1063079009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c0094f0 con 0x7f12880ca010 2023-12-19T08:00:04.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f12867fc700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f127c009290 tx=0x7f127c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c011c10 con 0x7f12880ca010 2023-12-19T08:00:04.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127c020070 con 0x7f12880ca010 2023-12-19T08:00:04.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128815abc0 con 0x7f12880ca010 2023-12-19T08:00:04.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127c01b450 con 0x7f12880ca010 2023-12-19T08:00:04.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12880b1370 con 0x7f12880ca010 2023-12-19T08:00:04.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.020+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1288071120 con 0x7f12880ca010 2023-12-19T08:00:04.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.024+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f127c022030 con 0x7f12880ca010 2023-12-19T08:00:04.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.024+0000 7f126effd700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 0x7f1274062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.024+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f127c07e840 con 0x7f12880ca010 2023-12-19T08:00:04.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.024+0000 7f1286ffd700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 0x7f1274062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.024+0000 7f1286ffd700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 0x7f1274062ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1278003a30 tx=0x7f1278007ea0 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.032+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f127c04eca0 con 0x7f12880ca010 2023-12-19T08:00:04.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.256+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1288071120 con 0x7f12880ca010 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.260+0000 7f126effd700 1 -- 172.21.15.111:0/1063079009 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f127c0522c0 con 0x7f12880ca010 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:00:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:00:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:00:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:00:04.272 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:00:04.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 msgr2=0x7f1274062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 0x7f1274062ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1278003a30 tx=0x7f1278007ea0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 msgr2=0x7f12880b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f127c009290 tx=0x7f127c016000 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 shutdown_connections 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12740606f0 0x7f1274062ba0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12880ca3e0 0x7f12880b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12880ca010 0x7f12880b0210 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 --2- 172.21.15.111:0/1063079009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12880ad150 0x7f128814c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 >> 172.21.15.111:0/1063079009 conn(0x7f1288068500 msgr2=0x7f12880c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 shutdown_connections 2023-12-19T08:00:04.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.264+0000 7f128d4ae700 1 -- 172.21.15.111:0/1063079009 wait complete. 2023-12-19T08:00:04.285 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:00:04.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.448+0000 7fdbe600c700 1 Processor -- start 2023-12-19T08:00:04.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.448+0000 7fdbe600c700 1 -- start start 2023-12-19T08:00:04.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdbe015d330 0x7fdbe006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe006b470 con 0x7fdbe00c6cd0 2023-12-19T08:00:04.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe006e110 con 0x7fdbe00c6900 2023-12-19T08:00:04.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe006be40 con 0x7fdbe015d330 2023-12-19T08:00:04.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe4809700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdbe00c6cd0 0x7fdbe015cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37786/0 (socket says 172.21.15.111:37786) 2023-12-19T08:00:04.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe4809700 1 -- 172.21.15.111:0/2499533707 learned_addr learned my addr 172.21.15.111:0/2499533707 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:04.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe580b700 1 --2- 172.21.15.111:0/2499533707 >> v2:172.21.15.111:3300/0 conn(0x7fdbe015d330 0x7fdbe006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340083345 0 0) 0x7fdbe006b470 con 0x7fdbe00c6cd0 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc003570 con 0x7fdbe00c6cd0 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe580b700 1 -- 172.21.15.111:0/2499533707 >> v1:172.21.15.143:6789/0 conn(0x7fdbe00c6900 legacy=0x7fdbe006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe580b700 1 -- 172.21.15.111:0/2499533707 >> v1:172.21.15.111:6789/0 conn(0x7fdbe00c6cd0 legacy=0x7fdbe015cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 230792300 0 0) 0x7fdbcc003570 con 0x7fdbe00c6cd0 2023-12-19T08:00:04.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe580b700 1 -- 172.21.15.111:0/2499533707 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0161d10 con 0x7fdbe015d330 2023-12-19T08:00:04.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe580b700 1 --2- 172.21.15.111:0/2499533707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe015d330 0x7fdbe006b900 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e60 tx=0x7fdbd8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70fb5ad54ad49 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd800e070 con 0x7fdbe015d330 2023-12-19T08:00:04.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd800b850 con 0x7fdbe015d330 2023-12-19T08:00:04.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbd7fff700 1 -- 172.21.15.111:0/2499533707 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0001470 con 0x7fdbe015d330 2023-12-19T08:00:04.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- 172.21.15.111:0/2499533707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe015d330 msgr2=0x7fdbe006b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 --2- 172.21.15.111:0/2499533707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe015d330 0x7fdbe006b900 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e60 tx=0x7fdbd8009780 comp rx=0 tx=0).stop 2023-12-19T08:00:04.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- 172.21.15.111:0/2499533707 shutdown_connections 2023-12-19T08:00:04.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 --2- 172.21.15.111:0/2499533707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe015d330 0x7fdbe006b900 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.452+0000 7fdbe600c700 1 -- 172.21.15.111:0/2499533707 >> 172.21.15.111:0/2499533707 conn(0x7fdbe0067bb0 msgr2=0x7fdbe00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:04.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.456+0000 7fdbe600c700 1 -- 172.21.15.111:0/2499533707 shutdown_connections 2023-12-19T08:00:04.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.456+0000 7fdbe600c700 1 -- 172.21.15.111:0/2499533707 wait complete. 2023-12-19T08:00:04.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.456+0000 7fdbe600c700 1 Processor -- start 2023-12-19T08:00:04.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.456+0000 7fdbe600c700 1 -- start start 2023-12-19T08:00:04.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 0x7fdbe00c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 0x7fdbe014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe016f350 con 0x7fdbe00c6900 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe016f630 con 0x7fdbe00c6cd0 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe600c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe016f910 con 0x7fdbe015d330 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46214/0 (socket says 172.21.15.111:46214) 2023-12-19T08:00:04.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 -- 172.21.15.111:0/3605021866 learned_addr learned my addr 172.21.15.111:0/3605021866 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:04.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe4809700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 0x7fdbe00c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe580b700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 0x7fdbe014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 -- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 msgr2=0x7fdbe014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 0x7fdbe014ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 -- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 msgr2=0x7fdbe00c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 0x7fdbe00c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 -- 172.21.15.111:0/3605021866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009a20 con 0x7fdbe00c6900 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe4809700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 0x7fdbe00c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe580b700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 0x7fdbe014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.460+0000 7fdbe500a700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00c4e0 tx=0x7fdbdc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc011590 con 0x7fdbe00c6900 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc011710 con 0x7fdbe00c6900 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc011ad0 con 0x7fdbe00c6900 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe016fbf0 con 0x7fdbe00c6900 2023-12-19T08:00:04.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00c17b0 con 0x7fdbe00c6900 2023-12-19T08:00:04.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe004be40 con 0x7fdbe00c6900 2023-12-19T08:00:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.464+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdbdc01b3b0 con 0x7fdbe00c6900 2023-12-19T08:00:04.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.468+0000 7fdbd57fa700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 0x7fdbc4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.468+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5665+0+0 (secure 0 0 0) 0x7fdbdc07be20 con 0x7fdbe00c6900 2023-12-19T08:00:04.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.468+0000 7fdbe4809700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 0x7fdbc4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.468+0000 7fdbe4809700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 0x7fdbc4062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbe015c2f0 tx=0x7fdbd0000bc0 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.472+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdbdc04c320 con 0x7fdbe00c6900 2023-12-19T08:00:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.644+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe00c9fd0 con 0x7fdbe00c6900 2023-12-19T08:00:04.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbd57fa700 1 -- 172.21.15.111:0/3605021866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13132 (secure 0 0 0) 0x7fdbdc017020 con 0x7fdbe00c6900 2023-12-19T08:00:04.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:04.647 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":64,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:00:03.799305+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:00:04.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 msgr2=0x7fdbc4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 0x7fdbc4062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbe015c2f0 tx=0x7fdbd0000bc0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 msgr2=0x7fdbe01703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00c4e0 tx=0x7fdbdc00c8a0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 shutdown_connections 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdbc4060670 0x7fdbc4062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe00c6cd0 0x7fdbe00c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe015d330 0x7fdbe014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 --2- 172.21.15.111:0/3605021866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbe00c6900 0x7fdbe01703f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 >> 172.21.15.111:0/3605021866 conn(0x7fdbe0067bb0 msgr2=0x7fdbe015ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 shutdown_connections 2023-12-19T08:00:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.648+0000 7fdbe600c700 1 -- 172.21.15.111:0/3605021866 wait complete. 2023-12-19T08:00:04.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:00:04.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:00:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 Processor -- start 2023-12-19T08:00:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 -- start start 2023-12-19T08:00:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 0x7feb5006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feb500cecd0 0x7feb5006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500cbdf0 con 0x7feb50158e20 2023-12-19T08:00:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500cc0d0 con 0x7feb500cecd0 2023-12-19T08:00:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb5522e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500cc3b0 con 0x7feb500cb970 2023-12-19T08:00:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb4ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7feb50158e20 0x7feb500ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37790/0 (socket says 172.21.15.111:37790) 2023-12-19T08:00:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb4ed9d700 1 -- 172.21.15.111:0/3179303851 learned_addr learned my addr 172.21.15.111:0/3179303851 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.840+0000 7feb4f59e700 1 --2- 172.21.15.111:0/3179303851 >> v2:172.21.15.111:3300/0 conn(0x7feb500cecd0 0x7feb5006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 --2- 172.21.15.111:0/3179303851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 0x7feb5006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb4e59c700 1 -- 172.21.15.111:0/3179303851 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810508632 0 0) 0x7feb500cbdf0 con 0x7feb50158e20 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 -- 172.21.15.111:0/3179303851 >> v2:172.21.15.111:3300/0 conn(0x7feb500cecd0 msgr2=0x7feb5006b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 --2- 172.21.15.111:0/3179303851 >> v2:172.21.15.111:3300/0 conn(0x7feb500cecd0 0x7feb5006b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 -- 172.21.15.111:0/3179303851 >> v1:172.21.15.111:6789/0 conn(0x7feb50158e20 legacy=0x7feb500ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 -- 172.21.15.111:0/3179303851 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500cc690 con 0x7feb500cb970 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb47fff700 1 --2- 172.21.15.111:0/3179303851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 0x7feb5006acc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb380098e0 tx=0x7feb380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726dc9bc41f9839c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb4e59c700 1 -- 172.21.15.111:0/3179303851 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800b640 con 0x7feb500cb970 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb4e59c700 1 -- 172.21.15.111:0/3179303851 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800b7a0 con 0x7feb500cb970 2023-12-19T08:00:04.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb4e59c700 1 -- 172.21.15.111:0/3179303851 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800bb40 con 0x7feb500cb970 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 -- 172.21.15.111:0/3179303851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 msgr2=0x7feb5006acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 --2- 172.21.15.111:0/3179303851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 0x7feb5006acc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb380098e0 tx=0x7feb380091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 -- 172.21.15.111:0/3179303851 shutdown_connections 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 --2- 172.21.15.111:0/3179303851 >> v2:172.21.15.111:3300/0 conn(0x7feb500cecd0 0x7feb5006b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 --2- 172.21.15.111:0/3179303851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cb970 0x7feb5006acc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 -- 172.21.15.111:0/3179303851 >> 172.21.15.111:0/3179303851 conn(0x7feb50067bb0 msgr2=0x7feb500d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 -- 172.21.15.111:0/3179303851 shutdown_connections 2023-12-19T08:00:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.844+0000 7feb5522e700 1 -- 172.21.15.111:0/3179303851 wait complete. 2023-12-19T08:00:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 Processor -- start 2023-12-19T08:00:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 -- start start 2023-12-19T08:00:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 0x7feb500cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb50158e20 0x7feb5014f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5014c3a0 con 0x7feb500b4180 2023-12-19T08:00:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50150930 con 0x7feb50158e20 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.848+0000 7feb5522e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5014f8b0 con 0x7feb500cecd0 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 0x7feb500cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46234/0 (socket says 172.21.15.111:46234) 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4f59e700 1 -- 172.21.15.111:0/1338958302 learned_addr learned my addr 172.21.15.111:0/1338958302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb47fff700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb50158e20 0x7feb5014f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 -- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb50158e20 msgr2=0x7feb5014f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb50158e20 0x7feb5014f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 -- 172.21.15.111:0/1338958302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 msgr2=0x7feb500b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 -- 172.21.15.111:0/1338958302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38009420 con 0x7feb500cecd0 2023-12-19T08:00:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4f59e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb4ed9d700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 0x7feb500cd440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb4000c4f0 tx=0x7feb40007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4000cea0 con 0x7feb500cecd0 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb40005b30 con 0x7feb500cecd0 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5014fb90 con 0x7feb500cecd0 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400070d0 con 0x7feb500cecd0 2023-12-19T08:00:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb50150150 con 0x7feb500cecd0 2023-12-19T08:00:04.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.852+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb50071120 con 0x7feb500cecd0 2023-12-19T08:00:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.856+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7feb400074c0 con 0x7feb500cecd0 2023-12-19T08:00:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.856+0000 7feb46ffd700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 0x7feb34062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.856+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5665+0+0 (secure 0 0 0) 0x7feb4007bbd0 con 0x7feb500cecd0 2023-12-19T08:00:04.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.856+0000 7feb47fff700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 0x7feb34062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:04.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.856+0000 7feb47fff700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 0x7feb34062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb380093f0 tx=0x7feb38017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:04.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:04.864+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feb4004c0d0 con 0x7feb500cecd0 2023-12-19T08:00:05.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.032+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb5006a7a0 con 0x7feb500cecd0 2023-12-19T08:00:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.036+0000 7feb46ffd700 1 -- 172.21.15.111:0/1338958302 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb4004f6f0 con 0x7feb500cecd0 2023-12-19T08:00:05.035 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 msgr2=0x7feb34062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 0x7feb34062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb380093f0 tx=0x7feb38017000 comp rx=0 tx=0).stop 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 msgr2=0x7feb500cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 0x7feb500cd440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb4000c4f0 tx=0x7feb40007f50 comp rx=0 tx=0).stop 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 shutdown_connections 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feb34060670 0x7feb34062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb50158e20 0x7feb5014f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb500b4180 0x7feb500b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 --2- 172.21.15.111:0/1338958302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb500cecd0 0x7feb500cd440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 >> 172.21.15.111:0/1338958302 conn(0x7feb50067bb0 msgr2=0x7feb5014cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 shutdown_connections 2023-12-19T08:00:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.040+0000 7feb5522e700 1 -- 172.21.15.111:0/1338958302 wait complete. 2023-12-19T08:00:05.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:00:05.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.232+0000 7f531166e700 1 Processor -- start 2023-12-19T08:00:05.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.232+0000 7f531166e700 1 -- start start 2023-12-19T08:00:05.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f530c0c8680 0x7f530c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0c9320 con 0x7f530c159f90 2023-12-19T08:00:05.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0c8ad0 con 0x7f530c0c8680 2023-12-19T08:00:05.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c0c3930 con 0x7f530c0cc140 2023-12-19T08:00:05.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:38216/0 (socket says 172.21.15.111:38216) 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 -- 172.21.15.111:0/2152282675 learned_addr learned my addr 172.21.15.111:0/2152282675 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530a7fc700 1 --2- 172.21.15.111:0/2152282675 >> v2:172.21.15.143:3300/0 conn(0x7f530c0c8680 0x7f530c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 -- 172.21.15.111:0/2152282675 >> v2:172.21.15.143:3300/0 conn(0x7f530c0c8680 msgr2=0x7f530c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 --2- 172.21.15.111:0/2152282675 >> v2:172.21.15.143:3300/0 conn(0x7f530c0c8680 0x7f530c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 -- 172.21.15.111:0/2152282675 >> v1:172.21.15.143:6789/0 conn(0x7f530c159f90 legacy=0x7f530c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 -- 172.21.15.111:0/2152282675 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0c8d60 con 0x7f530c0cc140 2023-12-19T08:00:05.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f530affd700 1 --2- 172.21.15.111:0/2152282675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0099b0 tx=0x7f52fc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ccc3e5a36f63b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f5309ffb700 1 -- 172.21.15.111:0/2152282675 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc009ea0 con 0x7f530c0cc140 2023-12-19T08:00:05.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f5309ffb700 1 -- 172.21.15.111:0/2152282675 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc00b860 con 0x7f530c0cc140 2023-12-19T08:00:05.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f5309ffb700 1 -- 172.21.15.111:0/2152282675 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00bc00 con 0x7f530c0cc140 2023-12-19T08:00:05.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- 172.21.15.111:0/2152282675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 msgr2=0x7f530c0cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 --2- 172.21.15.111:0/2152282675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0099b0 tx=0x7f52fc009250 comp rx=0 tx=0).stop 2023-12-19T08:00:05.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- 172.21.15.111:0/2152282675 shutdown_connections 2023-12-19T08:00:05.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 --2- 172.21.15.111:0/2152282675 >> v2:172.21.15.143:3300/0 conn(0x7f530c0c8680 0x7f530c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 --2- 172.21.15.111:0/2152282675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0cc510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.236+0000 7f531166e700 1 -- 172.21.15.111:0/2152282675 >> 172.21.15.111:0/2152282675 conn(0x7f530c06b670 msgr2=0x7f530c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:05.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- 172.21.15.111:0/2152282675 shutdown_connections 2023-12-19T08:00:05.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- 172.21.15.111:0/2152282675 wait complete. 2023-12-19T08:00:05.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 Processor -- start 2023-12-19T08:00:05.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- start start 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f530c0c8680 0x7f530c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c14a800 con 0x7f530c159f90 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c148590 con 0x7f530c0c8680 2023-12-19T08:00:05.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f531166e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c149ed0 con 0x7f530c0cc140 2023-12-19T08:00:05.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f530b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f530b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46238/0 (socket says 172.21.15.111:46238) 2023-12-19T08:00:05.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f530b7fe700 1 -- 172.21.15.111:0/1640856091 learned_addr learned my addr 172.21.15.111:0/1640856091 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:05.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f530a7fc700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.240+0000 7f530affd700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f530c0c8680 0x7f530c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 -- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 msgr2=0x7f530c0c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 -- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f530c0c8680 msgr2=0x7f530c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f530c0c8680 0x7f530c14a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 -- 172.21.15.111:0/1640856091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0094f0 con 0x7f530c159f90 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f530b7fe700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f530400c840 tx=0x7f5304001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304012630 con 0x7f530c159f90 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53040127b0 con 0x7f530c159f90 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c16cfe0 con 0x7f530c159f90 2023-12-19T08:00:05.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304012b70 con 0x7f530c159f90 2023-12-19T08:00:05.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c16d5a0 con 0x7f530c159f90 2023-12-19T08:00:05.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.244+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c148720 con 0x7f530c159f90 2023-12-19T08:00:05.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.248+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f530401a3a0 con 0x7f530c159f90 2023-12-19T08:00:05.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.248+0000 7f52f2ffd700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 0x7f52f8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.248+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f530407cb20 con 0x7f530c159f90 2023-12-19T08:00:05.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.248+0000 7f530affd700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 0x7f52f8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.248+0000 7f530affd700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 0x7f52f8062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0099b0 tx=0x7f52fc017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.256+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f530404cf80 con 0x7f530c159f90 2023-12-19T08:00:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.428+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f530c04be40 con 0x7f530c159f90 2023-12-19T08:00:05.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.428+0000 7f52f2ffd700 1 -- 172.21.15.111:0/1640856091 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7f53040505a0 con 0x7f530c159f90 2023-12-19T08:00:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:05.429 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 msgr2=0x7f52f8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 0x7f52f8062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0099b0 tx=0x7f52fc017000 comp rx=0 tx=0).stop 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 msgr2=0x7f530c144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f530400c840 tx=0x7f5304001670 comp rx=0 tx=0).stop 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 shutdown_connections 2023-12-19T08:00:05.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52f8060670 0x7f52f8062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f530c159f90 0x7f530c144480 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f530c0cc140 0x7f530c0c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 --2- 172.21.15.111:0/1640856091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f530c0c8680 0x7f530c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 >> 172.21.15.111:0/1640856091 conn(0x7f530c06b670 msgr2=0x7f530c14b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 shutdown_connections 2023-12-19T08:00:05.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.432+0000 7f531166e700 1 -- 172.21.15.111:0/1640856091 wait complete. 2023-12-19T08:00:05.453 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:00:05.454 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:00:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.616+0000 7f146e723700 1 Processor -- start 2023-12-19T08:00:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.616+0000 7f146e723700 1 -- start start 2023-12-19T08:00:05.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146e723700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1468158da0 0x7f146806e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146e723700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f14680caca0 0x7f146806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146e723700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146815d720 con 0x7f146806df10 2023-12-19T08:00:05.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146e723700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14680ca290 con 0x7f1468158da0 2023-12-19T08:00:05.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146e723700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14680ca4a0 con 0x7f14680caca0 2023-12-19T08:00:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1468158da0 0x7f146806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1468158da0 0x7f146806e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46254/0 (socket says 172.21.15.111:46254) 2023-12-19T08:00:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146df22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f14680caca0 0x7f146806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 -- 172.21.15.111:0/962762358 learned_addr learned my addr 172.21.15.111:0/962762358 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:05.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 -- 172.21.15.111:0/962762358 >> v2:172.21.15.143:3300/0 conn(0x7f14680caca0 msgr2=0x7f146806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:05.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 --2- 172.21.15.111:0/962762358 >> v2:172.21.15.143:3300/0 conn(0x7f14680caca0 0x7f146806ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 -- 172.21.15.111:0/962762358 >> v1:172.21.15.143:6789/0 conn(0x7f146806df10 legacy=0x7f146806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.620+0000 7f146d721700 1 -- 172.21.15.111:0/962762358 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680ca780 con 0x7f1468158da0 2023-12-19T08:00:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146d721700 1 --2- 172.21.15.111:0/962762358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1468158da0 0x7f146806e670 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1468069c20 tx=0x7f1464009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9c4ffeb82270f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f145ffff700 1 -- 172.21.15.111:0/962762358 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464009e70 con 0x7f1468158da0 2023-12-19T08:00:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f145ffff700 1 -- 172.21.15.111:0/962762358 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146400b860 con 0x7f1468158da0 2023-12-19T08:00:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f145ffff700 1 -- 172.21.15.111:0/962762358 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400bc00 con 0x7f1468158da0 2023-12-19T08:00:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 -- 172.21.15.111:0/962762358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1468158da0 msgr2=0x7f146806e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 --2- 172.21.15.111:0/962762358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1468158da0 0x7f146806e670 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1468069c20 tx=0x7f1464009250 comp rx=0 tx=0).stop 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 -- 172.21.15.111:0/962762358 shutdown_connections 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 --2- 172.21.15.111:0/962762358 >> v2:172.21.15.143:3300/0 conn(0x7f14680caca0 0x7f146806ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 --2- 172.21.15.111:0/962762358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1468158da0 0x7f146806e670 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 -- 172.21.15.111:0/962762358 >> 172.21.15.111:0/962762358 conn(0x7f14680cc760 msgr2=0x7f14680ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 -- 172.21.15.111:0/962762358 shutdown_connections 2023-12-19T08:00:05.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.624+0000 7f146e723700 1 -- 172.21.15.111:0/962762358 wait complete. 2023-12-19T08:00:05.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 Processor -- start 2023-12-19T08:00:05.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 -- start start 2023-12-19T08:00:05.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f146806df10 0x7f1468156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1468158da0 0x7f14680c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c38d0 con 0x7f14680caca0 2023-12-19T08:00:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c9460 con 0x7f146806df10 2023-12-19T08:00:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146e723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14680c9740 con 0x7f1468158da0 2023-12-19T08:00:05.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46256/0 (socket says 172.21.15.111:46256) 2023-12-19T08:00:05.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 -- 172.21.15.111:0/815528591 learned_addr learned my addr 172.21.15.111:0/815528591 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146d721700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f146806df10 0x7f1468156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146df22700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1468158da0 0x7f14680c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 -- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1468158da0 msgr2=0x7f14680c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1468158da0 0x7f14680c3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 -- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f146806df10 msgr2=0x7f1468156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f146806df10 0x7f1468156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.628+0000 7f146cf20700 1 -- 172.21.15.111:0/815528591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640094f0 con 0x7f14680caca0 2023-12-19T08:00:05.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f146cf20700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f145800c790 tx=0x7f1458007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458011590 con 0x7f14680caca0 2023-12-19T08:00:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458011710 con 0x7f14680caca0 2023-12-19T08:00:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458011ad0 con 0x7f14680caca0 2023-12-19T08:00:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14680c9a20 con 0x7f14680caca0 2023-12-19T08:00:05.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146815caf0 con 0x7f14680caca0 2023-12-19T08:00:05.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146804be40 con 0x7f14680caca0 2023-12-19T08:00:05.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.632+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f145801b3b0 con 0x7f14680caca0 2023-12-19T08:00:05.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.636+0000 7f145d7fa700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 0x7f144c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:05.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.636+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f145807bd50 con 0x7f14680caca0 2023-12-19T08:00:05.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.636+0000 7f146d721700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 0x7f144c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:05.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.636+0000 7f146d721700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 0x7f144c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1468069c20 tx=0x7f1464017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:05.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.640+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f145804c1b0 con 0x7f14680caca0 2023-12-19T08:00:05.657 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:00:05.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:00:05.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:05.816+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f146806f360 con 0x7f14680caca0 2023-12-19T08:00:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.412+0000 7f145d7fa700 1 -- 172.21.15.111:0/815528591 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f145804f7d0 con 0x7f14680caca0 2023-12-19T08:00:06.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.416+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 msgr2=0x7f144c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.416+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 0x7f144c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1468069c20 tx=0x7f1464017000 comp rx=0 tx=0).stop 2023-12-19T08:00:06.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.416+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 msgr2=0x7f146815c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.416+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f145800c790 tx=0x7f1458007f40 comp rx=0 tx=0).stop 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 shutdown_connections 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f144c060670 0x7f144c062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1468158da0 0x7f14680c3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14680caca0 0x7f146815c5b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 --2- 172.21.15.111:0/815528591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f146806df10 0x7f1468156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 >> 172.21.15.111:0/815528591 conn(0x7f14680cc760 msgr2=0x7f14680ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 shutdown_connections 2023-12-19T08:00:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.420+0000 7f146e723700 1 -- 172.21.15.111:0/815528591 wait complete. 2023-12-19T08:00:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:00:06.440 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:00:06.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 Processor -- start 2023-12-19T08:00:06.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 -- start start 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 0x7ff8980ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8980caa10 0x7ff89815d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980cbbb0 con 0x7ff8980cc040 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89815d960 con 0x7ff8980caa10 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.608+0000 7ff89e86b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89806dc90 con 0x7ff8980ce2d0 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89e06a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8980cc040 0x7ff898158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37798/0 (socket says 172.21.15.111:37798) 2023-12-19T08:00:06.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89e06a700 1 -- 172.21.15.111:0/2217821993 learned_addr learned my addr 172.21.15.111:0/2217821993 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d869700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 0x7ff8980ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 --2- 172.21.15.111:0/2217821993 >> v2:172.21.15.143:3300/0 conn(0x7ff8980caa10 0x7ff89815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89c867700 1 -- 172.21.15.111:0/2217821993 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212010196 0 0) 0x7ff8980cbbb0 con 0x7ff8980cc040 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89c867700 1 -- 172.21.15.111:0/2217821993 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff884003570 con 0x7ff8980cc040 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 -- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 msgr2=0x7ff8980ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 0x7ff8980ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 -- 172.21.15.111:0/2217821993 >> v1:172.21.15.111:6789/0 conn(0x7ff8980cc040 legacy=0x7ff898158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 -- 172.21.15.111:0/2217821993 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980cafa0 con 0x7ff8980caa10 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d869700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 0x7ff8980ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89d068700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980caa10 0x7ff89815d420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff8880098e0 tx=0x7ff8880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a417f5edb1d2d84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89c867700 1 -- 172.21.15.111:0/2217821993 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888011630 con 0x7ff8980caa10 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89c867700 1 -- 172.21.15.111:0/2217821993 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888011790 con 0x7ff8980caa10 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89c867700 1 -- 172.21.15.111:0/2217821993 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888011b30 con 0x7ff8980caa10 2023-12-19T08:00:06.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89e86b700 1 -- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980caa10 msgr2=0x7ff89815d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.612+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980caa10 0x7ff89815d420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff8880098e0 tx=0x7ff8880091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 -- 172.21.15.111:0/2217821993 shutdown_connections 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980caa10 0x7ff89815d420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2217821993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980ce2d0 0x7ff8980ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 -- 172.21.15.111:0/2217821993 >> 172.21.15.111:0/2217821993 conn(0x7ff898067bb0 msgr2=0x7ff8980cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 -- 172.21.15.111:0/2217821993 shutdown_connections 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 -- 172.21.15.111:0/2217821993 wait complete. 2023-12-19T08:00:06.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 Processor -- start 2023-12-19T08:00:06.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.616+0000 7ff89e86b700 1 -- start start 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 0x7ff8980c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980ce2d0 0x7ff898157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c7f30 con 0x7ff8980cc040 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c73b0 con 0x7ff8980ce2d0 2023-12-19T08:00:06.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c7690 con 0x7ff8980caa10 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d068700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 0x7ff8980c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d068700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 0x7ff8980c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46260/0 (socket says 172.21.15.111:46260) 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:38262/0 (socket says 172.21.15.111:38262) 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d068700 1 -- 172.21.15.111:0/2797652521 learned_addr learned my addr 172.21.15.111:0/2797652521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:06.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e06a700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980ce2d0 0x7ff898157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 -- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980ce2d0 msgr2=0x7ff898157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980ce2d0 0x7ff898157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 -- 172.21.15.111:0/2797652521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 msgr2=0x7ff8980c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 0x7ff8980c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 -- 172.21.15.111:0/2797652521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894009420 con 0x7ff8980caa10 2023-12-19T08:00:06.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89d869700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff89400ea00 tx=0x7ff8940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894009660 con 0x7ff8980caa10 2023-12-19T08:00:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89400c6a0 con 0x7ff8980caa10 2023-12-19T08:00:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff888009420 con 0x7ff8980caa10 2023-12-19T08:00:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89400ca40 con 0x7ff8980caa10 2023-12-19T08:00:06.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.620+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8980c7c50 con 0x7ff8980caa10 2023-12-19T08:00:06.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.624+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89804be40 con 0x7ff8980caa10 2023-12-19T08:00:06.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.624+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff894013470 con 0x7ff8980caa10 2023-12-19T08:00:06.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.628+0000 7ff88dffb700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 0x7ff87c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:06.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.628+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6035+0+0 (secure 0 0 0) 0x7ff89407cc40 con 0x7ff8980caa10 2023-12-19T08:00:06.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.628+0000 7ff89d068700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 0x7ff87c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.628+0000 7ff89d068700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 0x7ff87c062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff888011fd0 tx=0x7ff888017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:06.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.636+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff89404cf30 con 0x7ff8980caa10 2023-12-19T08:00:06.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:06.804+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff8980cb280 con 0x7ff8980caa10 2023-12-19T08:00:07.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.432+0000 7ff88dffb700 1 -- 172.21.15.111:0/2797652521 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff894050550 con 0x7ff8980caa10 2023-12-19T08:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 msgr2=0x7ff87c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 0x7ff87c062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff888011fd0 tx=0x7ff888017000 comp rx=0 tx=0).stop 2023-12-19T08:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 msgr2=0x7ff8981695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff89400ea00 tx=0x7ff8940091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 shutdown_connections 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff87c060670 0x7ff87c062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8980ce2d0 0x7ff898157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8980cc040 0x7ff8980c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 --2- 172.21.15.111:0/2797652521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8980caa10 0x7ff8981695c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.448+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 >> 172.21.15.111:0/2797652521 conn(0x7ff898067bb0 msgr2=0x7ff8980cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.452+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 shutdown_connections 2023-12-19T08:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.452+0000 7ff89e86b700 1 -- 172.21.15.111:0/2797652521 wait complete. 2023-12-19T08:00:07.457 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:00:07.470 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:00:07.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.632+0000 7f364b59e700 1 Processor -- start 2023-12-19T08:00:07.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.632+0000 7f364b59e700 1 -- start start 2023-12-19T08:00:07.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f364c06dcf0 0x7f364c159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:07.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c06c850 con 0x7f364c158c90 2023-12-19T08:00:07.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0cb650 con 0x7f364c06aeb0 2023-12-19T08:00:07.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364c06b470 con 0x7f364c06dcf0 2023-12-19T08:00:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364a59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f364c158c90 0x7f364c0caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37808/0 (socket says 172.21.15.111:37808) 2023-12-19T08:00:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364a59c700 1 -- 172.21.15.111:0/269451352 learned_addr learned my addr 172.21.15.111:0/269451352 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364ad9d700 1 --2- 172.21.15.111:0/269451352 >> v2:172.21.15.143:3300/0 conn(0x7f364c06dcf0 0x7f364c159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364959a700 1 -- 172.21.15.111:0/269451352 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131228820 0 0) 0x7f364c06c850 con 0x7f364c158c90 2023-12-19T08:00:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364959a700 1 -- 172.21.15.111:0/269451352 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3634003570 con 0x7f364c158c90 2023-12-19T08:00:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364ad9d700 1 -- 172.21.15.111:0/269451352 >> v1:172.21.15.143:6789/0 conn(0x7f364c06aeb0 legacy=0x7f364c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364ad9d700 1 -- 172.21.15.111:0/269451352 >> v1:172.21.15.111:6789/0 conn(0x7f364c158c90 legacy=0x7f364c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.636+0000 7f364ad9d700 1 -- 172.21.15.111:0/269451352 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c06b750 con 0x7f364c06dcf0 2023-12-19T08:00:07.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364ad9d700 1 --2- 172.21.15.111:0/269451352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c06dcf0 0x7f364c159770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3644009e60 tx=0x7f3644009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b11ad1c96b1d839a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:07.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364959a700 1 -- 172.21.15.111:0/269451352 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364400e070 con 0x7f364c06dcf0 2023-12-19T08:00:07.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364959a700 1 -- 172.21.15.111:0/269451352 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364400bb60 con 0x7f364c06dcf0 2023-12-19T08:00:07.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364959a700 1 -- 172.21.15.111:0/269451352 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644012430 con 0x7f364c06dcf0 2023-12-19T08:00:07.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 -- 172.21.15.111:0/269451352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c06dcf0 msgr2=0x7f364c159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 --2- 172.21.15.111:0/269451352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c06dcf0 0x7f364c159770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3644009e60 tx=0x7f3644009780 comp rx=0 tx=0).stop 2023-12-19T08:00:07.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 -- 172.21.15.111:0/269451352 shutdown_connections 2023-12-19T08:00:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 --2- 172.21.15.111:0/269451352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c06dcf0 0x7f364c159770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 -- 172.21.15.111:0/269451352 >> 172.21.15.111:0/269451352 conn(0x7f364c067bb0 msgr2=0x7f364c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 -- 172.21.15.111:0/269451352 shutdown_connections 2023-12-19T08:00:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.640+0000 7f364b59e700 1 -- 172.21.15.111:0/269451352 wait complete. 2023-12-19T08:00:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 Processor -- start 2023-12-19T08:00:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 -- start start 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c06dcf0 0x7f364c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c158c90 0x7f364c152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c155ce0 con 0x7f364c06aeb0 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c154e30 con 0x7f364c158c90 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1550d0 con 0x7f364c06dcf0 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46270/0 (socket says 172.21.15.111:46270) 2023-12-19T08:00:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 -- 172.21.15.111:0/1859335804 learned_addr learned my addr 172.21.15.111:0/1859335804 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:07.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364ad9d700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c158c90 0x7f364c152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:07.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f3649d9b700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c06dcf0 0x7f364c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:07.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 -- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c06dcf0 msgr2=0x7f364c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c06dcf0 0x7f364c1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 -- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c158c90 msgr2=0x7f364c152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c158c90 0x7f364c152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 -- 172.21.15.111:0/1859335804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644009a20 con 0x7f364c06aeb0 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364a59c700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f363c00c590 tx=0x7f363c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c005cd0 con 0x7f364c06aeb0 2023-12-19T08:00:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c005e50 con 0x7f364c06aeb0 2023-12-19T08:00:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c1553b0 con 0x7f364c06aeb0 2023-12-19T08:00:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.644+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c007320 con 0x7f364c06aeb0 2023-12-19T08:00:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.648+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c155970 con 0x7f364c06aeb0 2023-12-19T08:00:07.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.648+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c04be40 con 0x7f364c06aeb0 2023-12-19T08:00:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.648+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f363c01b3b0 con 0x7f364c06aeb0 2023-12-19T08:00:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.652+0000 7f363a7fc700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 0x7f362c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.652+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f363c07c350 con 0x7f364c06aeb0 2023-12-19T08:00:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.652+0000 7f3649d9b700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 0x7f362c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.652+0000 7f3649d9b700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 0x7f362c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f364c0b2540 tx=0x7f3640009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:07.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.656+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f363c04c4c0 con 0x7f364c06aeb0 2023-12-19T08:00:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:07.864+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f364c06ba30 con 0x7f364c06aeb0 2023-12-19T08:00:08.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.452+0000 7f363a7fc700 1 -- 172.21.15.111:0/1859335804 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7f363c04fae0 con 0x7f364c06aeb0 2023-12-19T08:00:08.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 msgr2=0x7f362c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 0x7f362c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f364c0b2540 tx=0x7f3640009040 comp rx=0 tx=0).stop 2023-12-19T08:00:08.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 msgr2=0x7f364c0c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f363c00c590 tx=0x7f363c007e00 comp rx=0 tx=0).stop 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 shutdown_connections 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c060670 0x7f362c062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c158c90 0x7f364c152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.456+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c06dcf0 0x7f364c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.460+0000 7f364b59e700 1 --2- 172.21.15.111:0/1859335804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06aeb0 0x7f364c0c9430 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.460+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 >> 172.21.15.111:0/1859335804 conn(0x7f364c067bb0 msgr2=0x7f364c0cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.460+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 shutdown_connections 2023-12-19T08:00:08.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.460+0000 7f364b59e700 1 -- 172.21.15.111:0/1859335804 wait complete. 2023-12-19T08:00:08.466 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 7 and data pool 8 2023-12-19T08:00:08.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:00:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.660+0000 7fa4460f0700 1 Processor -- start 2023-12-19T08:00:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa4460f0700 1 -- start start 2023-12-19T08:00:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa4460f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4400c92d0 0x7fa4400abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa4460f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400c8730 con 0x7fa4400cb750 2023-12-19T08:00:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa4460f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400c7e70 con 0x7fa4400aba90 2023-12-19T08:00:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa4460f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4400c23a0 con 0x7fa4400c92d0 2023-12-19T08:00:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa4400cb750 0x7fa4400c1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37824/0 (socket says 172.21.15.111:37824) 2023-12-19T08:00:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43f7fe700 1 -- 172.21.15.111:0/1890774878 learned_addr learned my addr 172.21.15.111:0/1890774878 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43ffff700 1 --2- 172.21.15.111:0/1890774878 >> v2:172.21.15.143:3300/0 conn(0x7fa4400c92d0 0x7fa4400abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585800526 0 0) 0x7fa4400c7e70 con 0x7fa4400aba90 2023-12-19T08:00:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa428003570 con 0x7fa4400aba90 2023-12-19T08:00:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620925640 0 0) 0x7fa4400c8730 con 0x7fa4400cb750 2023-12-19T08:00:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4400c7e70 con 0x7fa4400cb750 2023-12-19T08:00:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.664+0000 7fa43ffff700 1 -- 172.21.15.111:0/1890774878 >> v1:172.21.15.143:6789/0 conn(0x7fa4400aba90 legacy=0x7fa4400c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635017238 0 0) 0x7fa428003570 con 0x7fa4400aba90 2023-12-19T08:00:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43ffff700 1 -- 172.21.15.111:0/1890774878 >> v1:172.21.15.111:6789/0 conn(0x7fa4400cb750 legacy=0x7fa4400c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43ffff700 1 -- 172.21.15.111:0/1890774878 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400ca290 con 0x7fa4400c92d0 2023-12-19T08:00:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43ffff700 1 --2- 172.21.15.111:0/1890774878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa4400abff0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa438009e60 tx=0x7fa438009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d53f2f62f9dbeb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:08.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800e070 con 0x7fa4400c92d0 2023-12-19T08:00:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43800b7a0 con 0x7fa4400c92d0 2023-12-19T08:00:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa43e7fc700 1 -- 172.21.15.111:0/1890774878 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa430001370 con 0x7fa4400c92d0 2023-12-19T08:00:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 -- 172.21.15.111:0/1890774878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 msgr2=0x7fa4400abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 --2- 172.21.15.111:0/1890774878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa4400abff0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa438009e60 tx=0x7fa438009780 comp rx=0 tx=0).stop 2023-12-19T08:00:08.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 -- 172.21.15.111:0/1890774878 shutdown_connections 2023-12-19T08:00:08.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 --2- 172.21.15.111:0/1890774878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa4400abff0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 -- 172.21.15.111:0/1890774878 >> 172.21.15.111:0/1890774878 conn(0x7fa4400cd9c0 msgr2=0x7fa4400c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:08.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 -- 172.21.15.111:0/1890774878 shutdown_connections 2023-12-19T08:00:08.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.668+0000 7fa4460f0700 1 -- 172.21.15.111:0/1890774878 wait complete. 2023-12-19T08:00:08.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 Processor -- start 2023-12-19T08:00:08.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 -- start start 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4400aba90 0x7fa44014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa44015db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa440169720 con 0x7fa4400cb750 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4401699e0 con 0x7fa4400c92d0 2023-12-19T08:00:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa440169cc0 con 0x7fa4400aba90 2023-12-19T08:00:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46286/0 (socket says 172.21.15.111:46286) 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 -- 172.21.15.111:0/3340163458 learned_addr learned my addr 172.21.15.111:0/3340163458 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43effd700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa44015db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43f7fe700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4400aba90 0x7fa44014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 -- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4400aba90 msgr2=0x7fa44014d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4400aba90 0x7fa44014d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 -- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 msgr2=0x7fa44015db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa44015db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 -- 172.21.15.111:0/3340163458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438009a20 con 0x7fa4400cb750 2023-12-19T08:00:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa43ffff700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa44006cfa0 tx=0x7fa438017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434001470 con 0x7fa4400cb750 2023-12-19T08:00:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa440169fa0 con 0x7fa4400cb750 2023-12-19T08:00:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa438026400 con 0x7fa4400cb750 2023-12-19T08:00:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.672+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4380267e0 con 0x7fa4400cb750 2023-12-19T08:00:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.676+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4400ad2b0 con 0x7fa4400cb750 2023-12-19T08:00:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.676+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44016a4a0 con 0x7fa4400cb750 2023-12-19T08:00:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.676+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa438019030 con 0x7fa4400cb750 2023-12-19T08:00:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.680+0000 7fa4277fe700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 0x7fa42c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.680+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa43807e8a0 con 0x7fa4400cb750 2023-12-19T08:00:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.680+0000 7fa43f7fe700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 0x7fa42c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:08.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.680+0000 7fa43f7fe700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 0x7fa42c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa430001160 tx=0x7fa430001950 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:08.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.688+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa43804e940 con 0x7fa4400cb750 2023-12-19T08:00:08.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.860+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa440071120 con 0x7fa4400cb750 2023-12-19T08:00:08.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.860+0000 7fa4277fe700 1 -- 172.21.15.111:0/3340163458 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16361 (secure 0 0 0) 0x7fa438051f60 con 0x7fa4400cb750 2023-12-19T08:00:08.861 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:08.862 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":67,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:00:07.868936+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 msgr2=0x7fa42c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 0x7fa42c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa430001160 tx=0x7fa430001950 comp rx=0 tx=0).stop 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 msgr2=0x7fa44015e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa44006cfa0 tx=0x7fa438017000 comp rx=0 tx=0).stop 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 shutdown_connections 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa42c060670 0x7fa42c062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4400c92d0 0x7fa44015db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4400cb750 0x7fa44015e060 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 --2- 172.21.15.111:0/3340163458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4400aba90 0x7fa44014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:08.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 >> 172.21.15.111:0/3340163458 conn(0x7fa4400cd9c0 msgr2=0x7fa4400af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:08.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 shutdown_connections 2023-12-19T08:00:08.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:08.864+0000 7fa4460f0700 1 -- 172.21.15.111:0/3340163458 wait complete. 2023-12-19T08:00:08.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:00:09.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 Processor -- start 2023-12-19T08:00:09.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 -- start start 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c15d320 0x7f6c4c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c4c0aeba0 0x7f6c4c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c0c7190 con 0x7f6c4c0c15b0 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4c0c1b50 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c5286e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c0c1e30 con 0x7f6c4c15d320 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c50e0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c4c0aeba0 0x7f6c4c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c43fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6c4c0c15b0 0x7f6c4c06d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37836/0 (socket says 172.21.15.111:37836) 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c43fff700 1 -- 172.21.15.111:0/205247328 learned_addr learned my addr 172.21.15.111:0/205247328 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:09.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c4bfff700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c15d320 0x7f6c4c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c50e0b700 1 -- 172.21.15.111:0/205247328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c15d320 msgr2=0x7f6c4c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:09.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.052+0000 7f6c50e0b700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c15d320 0x7f6c4c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c50e0b700 1 -- 172.21.15.111:0/205247328 >> v1:172.21.15.111:6789/0 conn(0x7f6c4c0c15b0 legacy=0x7f6c4c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c50e0b700 1 -- 172.21.15.111:0/205247328 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c0c6e20 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c4b7fe700 1 -- 172.21.15.111:0/205247328 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512719480 0 0) 0x7f6c4c0c7190 con 0x7f6c4c0c15b0 2023-12-19T08:00:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c50e0b700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c0680f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009e60 tx=0x7f6c44009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95b7a63a415fc5a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c4b7fe700 1 -- 172.21.15.111:0/205247328 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400e070 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c4b7fe700 1 -- 172.21.15.111:0/205247328 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4400bb60 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c4b7fe700 1 -- 172.21.15.111:0/205247328 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44012430 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 -- 172.21.15.111:0/205247328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c0aeba0 msgr2=0x7f6c4c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c0680f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009e60 tx=0x7f6c44009780 comp rx=0 tx=0).stop 2023-12-19T08:00:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 -- 172.21.15.111:0/205247328 shutdown_connections 2023-12-19T08:00:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c0680f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 --2- 172.21.15.111:0/205247328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c15d320 0x7f6c4c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 -- 172.21.15.111:0/205247328 >> 172.21.15.111:0/205247328 conn(0x7f6c4c0c8850 msgr2=0x7f6c4c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 -- 172.21.15.111:0/205247328 shutdown_connections 2023-12-19T08:00:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.056+0000 7f6c5286e700 1 -- 172.21.15.111:0/205247328 wait complete. 2023-12-19T08:00:09.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 Processor -- start 2023-12-19T08:00:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- start start 2023-12-19T08:00:09.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c15f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c0c15b0 0x7f6c4c0c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c0c2e90 con 0x7f6c4c15d320 2023-12-19T08:00:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c15ff10 con 0x7f6c4c0aeba0 2023-12-19T08:00:09.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c1601f0 con 0x7f6c4c0c15b0 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46306/0 (socket says 172.21.15.111:46306) 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 -- 172.21.15.111:0/1390505002 learned_addr learned my addr 172.21.15.111:0/1390505002 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c4bfff700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c15f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c43fff700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c0c15b0 0x7f6c4c0c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 -- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c0c15b0 msgr2=0x7f6c4c0c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c0c15b0 0x7f6c4c0c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 -- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c4c0aeba0 msgr2=0x7f6c4c15f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c15f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 -- 172.21.15.111:0/1390505002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c44009a20 con 0x7f6c4c15d320 2023-12-19T08:00:09.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c50e0b700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400b6c0 tx=0x7f6c44019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44027470 con 0x7f6c4c15d320 2023-12-19T08:00:09.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c440275d0 con 0x7f6c4c15d320 2023-12-19T08:00:09.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c0af1c0 con 0x7f6c4c15d320 2023-12-19T08:00:09.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44027990 con 0x7f6c4c15d320 2023-12-19T08:00:09.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.060+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c160550 con 0x7f6c4c15d320 2023-12-19T08:00:09.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.064+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c0c5d20 con 0x7f6c4c15d320 2023-12-19T08:00:09.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.064+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6c44024070 con 0x7f6c4c15d320 2023-12-19T08:00:09.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.068+0000 7f6c48ff9700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 0x7f6c2c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.068+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6c4407f390 con 0x7f6c4c15d320 2023-12-19T08:00:09.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.068+0000 7f6c4bfff700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 0x7f6c2c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.068+0000 7f6c4bfff700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 0x7f6c2c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009d10 tx=0x7f6c3c0091a0 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.072+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c4c0c5d20 con 0x7f6c4c15d320 2023-12-19T08:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.248+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c4c0c5d20 con 0x7f6c4c15d320 2023-12-19T08:00:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.248+0000 7f6c48ff9700 1 -- 172.21.15.111:0/1390505002 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16361 (secure 0 0 0) 0x7f6c4c0c5d20 con 0x7f6c4c15d320 2023-12-19T08:00:09.250 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:09.250 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":67,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:00:07.868936+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 msgr2=0x7f6c2c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 0x7f6c2c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009d10 tx=0x7f6c3c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 msgr2=0x7f6c4c16d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400b6c0 tx=0x7f6c44019000 comp rx=0 tx=0).stop 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 shutdown_connections 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c2c060670 0x7f6c2c062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c4c15d320 0x7f6c4c16d4c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c4c0c15b0 0x7f6c4c0c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 --2- 172.21.15.111:0/1390505002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c4c0aeba0 0x7f6c4c15f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 >> 172.21.15.111:0/1390505002 conn(0x7f6c4c0c8850 msgr2=0x7f6c4c160930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 shutdown_connections 2023-12-19T08:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.252+0000 7f6c5286e700 1 -- 172.21.15.111:0/1390505002 wait complete. 2023-12-19T08:00:09.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:00:09.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.428+0000 7f1c2e816700 1 Processor -- start 2023-12-19T08:00:09.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.428+0000 7f1c2e816700 1 -- start start 2023-12-19T08:00:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e816700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 0x7f1c280caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e816700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2806c850 con 0x7f1c28159770 2023-12-19T08:00:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e816700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2806b470 con 0x7f1c2806aeb0 2023-12-19T08:00:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e816700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2806b750 con 0x7f1c28158c90 2023-12-19T08:00:09.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e015700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1c28159770 0x7f1c2806cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37850/0 (socket says 172.21.15.111:37850) 2023-12-19T08:00:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2e015700 1 -- 172.21.15.111:0/618006604 learned_addr learned my addr 172.21.15.111:0/618006604 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:09.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2d814700 1 --2- 172.21.15.111:0/618006604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 0x7f1c280caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892295169 0 0) 0x7f1c2806b470 con 0x7f1c2806aeb0 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c14003570 con 0x7f1c2806aeb0 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940215911 0 0) 0x7f1c2806c850 con 0x7f1c28159770 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2d814700 1 -- 172.21.15.111:0/618006604 >> v1:172.21.15.143:6789/0 conn(0x7f1c2806aeb0 legacy=0x7f1c28159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2d814700 1 -- 172.21.15.111:0/618006604 >> v1:172.21.15.111:6789/0 conn(0x7f1c28159770 legacy=0x7f1c2806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.432+0000 7f1c2d814700 1 -- 172.21.15.111:0/618006604 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2806ba30 con 0x7f1c28158c90 2023-12-19T08:00:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2d814700 1 --2- 172.21.15.111:0/618006604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 0x7f1c280caf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009910 tx=0x7f1c240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6f33ab5f95d772c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400b680 con 0x7f1c28158c90 2023-12-19T08:00:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2400b7e0 con 0x7f1c28158c90 2023-12-19T08:00:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2c812700 1 -- 172.21.15.111:0/618006604 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400bba0 con 0x7f1c28158c90 2023-12-19T08:00:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 -- 172.21.15.111:0/618006604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 msgr2=0x7f1c280caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 --2- 172.21.15.111:0/618006604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 0x7f1c280caf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009910 tx=0x7f1c240091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 -- 172.21.15.111:0/618006604 shutdown_connections 2023-12-19T08:00:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 --2- 172.21.15.111:0/618006604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c28158c90 0x7f1c280caf40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 -- 172.21.15.111:0/618006604 >> 172.21.15.111:0/618006604 conn(0x7f1c28067bb0 msgr2=0x7f1c280cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 -- 172.21.15.111:0/618006604 shutdown_connections 2023-12-19T08:00:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.436+0000 7f1c2e816700 1 -- 172.21.15.111:0/618006604 wait complete. 2023-12-19T08:00:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 Processor -- start 2023-12-19T08:00:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 -- start start 2023-12-19T08:00:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c2806aeb0 0x7f1c28156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c28158c90 0x7f1c28153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28168fb0 con 0x7f1c28159770 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28157090 con 0x7f1c28158c90 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e816700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281535d0 con 0x7f1c2806aeb0 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46308/0 (socket says 172.21.15.111:46308) 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2d814700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c2806aeb0 0x7f1c28156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 -- 172.21.15.111:0/764615211 learned_addr learned my addr 172.21.15.111:0/764615211 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2d013700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c28158c90 0x7f1c28153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 -- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c2806aeb0 msgr2=0x7f1c28156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c2806aeb0 0x7f1c28156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 -- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c28158c90 msgr2=0x7f1c28153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c28158c90 0x7f1c28153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.440+0000 7f1c2e015700 1 -- 172.21.15.111:0/764615211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24009420 con 0x7f1c28159770 2023-12-19T08:00:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c2e015700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000cc50 tx=0x7f1c200019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000a830 con 0x7f1c28159770 2023-12-19T08:00:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2000a990 con 0x7f1c28159770 2023-12-19T08:00:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000ad50 con 0x7f1c28159770 2023-12-19T08:00:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c280c52a0 con 0x7f1c28159770 2023-12-19T08:00:09.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280c5860 con 0x7f1c28159770 2023-12-19T08:00:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c28169120 con 0x7f1c28159770 2023-12-19T08:00:09.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.444+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1c20007300 con 0x7f1c28159770 2023-12-19T08:00:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.448+0000 7f1c1dffb700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 0x7f1c0c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.448+0000 7f1c2d814700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 0x7f1c0c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:09.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.448+0000 7f1c2d814700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 0x7f1c0c062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c240093f0 tx=0x7f1c24017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:09.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.448+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f1c20051510 con 0x7f1c28159770 2023-12-19T08:00:09.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.456+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1c2004d110 con 0x7f1c28159770 2023-12-19T08:00:09.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:09.656+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1c2804be40 con 0x7f1c28159770 2023-12-19T08:00:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.476+0000 7f1c1dffb700 1 -- 172.21.15.111:0/764615211 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1c20050730 con 0x7f1c28159770 2023-12-19T08:00:10.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 msgr2=0x7f1c0c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 0x7f1c0c062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c240093f0 tx=0x7f1c24017000 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 msgr2=0x7f1c280c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000cc50 tx=0x7f1c200019b0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 shutdown_connections 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c0c0606a0 0x7f1c0c062b50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c28158c90 0x7f1c28153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c28159770 0x7f1c280c7d80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 --2- 172.21.15.111:0/764615211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c2806aeb0 0x7f1c28156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 >> 172.21.15.111:0/764615211 conn(0x7f1c28067bb0 msgr2=0x7f1c28159bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 shutdown_connections 2023-12-19T08:00:10.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.480+0000 7f1c2e816700 1 -- 172.21.15.111:0/764615211 wait complete. 2023-12-19T08:00:10.504 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:00:10.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.672+0000 7fe2b4c2e700 1 Processor -- start 2023-12-19T08:00:10.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 -- start start 2023-12-19T08:00:10.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b013cf20 0x7fe2b00cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b013d2f0 0x7fe2b00c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 0x7fe2b006b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00cc0e0 con 0x7fe2b013cf20 2023-12-19T08:00:10.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b006b9e0 con 0x7fe2b006b060 2023-12-19T08:00:10.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2b4c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00cb4d0 con 0x7fe2b013d2f0 2023-12-19T08:00:10.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b013cf20 0x7fe2b00cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b013cf20 0x7fe2b00cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46322/0 (socket says 172.21.15.111:46322) 2023-12-19T08:00:10.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 -- 172.21.15.111:0/2191206524 learned_addr learned my addr 172.21.15.111:0/2191206524 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:10.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b013d2f0 0x7fe2b00c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2aed9d700 1 --2- 172.21.15.111:0/2191206524 >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 0x7fe2b006b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 -- 172.21.15.111:0/2191206524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b013d2f0 msgr2=0x7fe2b00c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b013d2f0 0x7fe2b00c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 -- 172.21.15.111:0/2191206524 >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 msgr2=0x7fe2b006b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/2191206524 >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 0x7fe2b006b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 -- 172.21.15.111:0/2191206524 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00cb7b0 con 0x7fe2b013cf20 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2aed9d700 1 --2- 172.21.15.111:0/2191206524 >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 0x7fe2b006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.676+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b013cf20 0x7fe2b00cadf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00cd1d0 tx=0x7fe2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3f4e17fb4544b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2add9b700 1 -- 172.21.15.111:0/2191206524 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000b860 con 0x7fe2b013cf20 2023-12-19T08:00:10.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2add9b700 1 -- 172.21.15.111:0/2191206524 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000b9c0 con 0x7fe2b013cf20 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2add9b700 1 -- 172.21.15.111:0/2191206524 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000bd60 con 0x7fe2b013cf20 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/2191206524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b013cf20 msgr2=0x7fe2b00cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b013cf20 0x7fe2b00cadf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00cd1d0 tx=0x7fe2a0009250 comp rx=0 tx=0).stop 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/2191206524 shutdown_connections 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/2191206524 >> v2:172.21.15.143:3300/0 conn(0x7fe2b006b060 0x7fe2b006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b013d2f0 0x7fe2b00c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/2191206524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b013cf20 0x7fe2b00cadf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/2191206524 >> 172.21.15.111:0/2191206524 conn(0x7fe2b00cc580 msgr2=0x7fe2b00cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/2191206524 shutdown_connections 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/2191206524 wait complete. 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 Processor -- start 2023-12-19T08:00:10.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.680+0000 7fe2b4c2e700 1 -- start start 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b013d2f0 0x7fe2b00ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 0x7fe2b0153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00ab770 con 0x7fe2b006b060 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00ab540 con 0x7fe2b013d2f0 2023-12-19T08:00:10.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0155aa0 con 0x7fe2b0156ac0 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46326/0 (socket says 172.21.15.111:46326) 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 -- 172.21.15.111:0/14657843 learned_addr learned my addr 172.21.15.111:0/14657843 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2aed9d700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 0x7fe2b0153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b013d2f0 0x7fe2b00ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 -- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 msgr2=0x7fe2b0153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 0x7fe2b0153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 -- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b013d2f0 msgr2=0x7fe2b00ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b013d2f0 0x7fe2b00ab000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:10.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 -- 172.21.15.111:0/14657843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00094f0 con 0x7fe2b006b060 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2aed9d700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 0x7fe2b0153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2ae59c700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe2b006e9d0 tx=0x7fe2a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0011e50 con 0x7fe2b006b060 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a0024440 con 0x7fe2b006b060 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b0155d40 con 0x7fe2b006b060 2023-12-19T08:00:10.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0024800 con 0x7fe2b006b060 2023-12-19T08:00:10.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b0156340 con 0x7fe2b006b060 2023-12-19T08:00:10.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.684+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b0071120 con 0x7fe2b006b060 2023-12-19T08:00:10.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.688+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe2a001a420 con 0x7fe2b006b060 2023-12-19T08:00:10.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.688+0000 7fe2a67fc700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 0x7fe290062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:10.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.688+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe2a007ea10 con 0x7fe2b006b060 2023-12-19T08:00:10.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.692+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 0x7fe290062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:10.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.692+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 0x7fe290062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00aad00 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:10.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.696+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2a004eb20 con 0x7fe2b006b060 2023-12-19T08:00:10.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:10.900+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe2b006f090 con 0x7fe2b006b060 2023-12-19T08:00:11.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.484+0000 7fe2a67fc700 1 -- 172.21.15.111:0/14657843 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe2a0052140 con 0x7fe2b006b060 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 msgr2=0x7fe290062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 0x7fe290062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00aad00 tx=0x7fe298008040 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 msgr2=0x7fe2b016db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe2b006e9d0 tx=0x7fe2a0017000 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 shutdown_connections 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe290060670 0x7fe290062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b013d2f0 0x7fe2b00ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0156ac0 0x7fe2b0153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 --2- 172.21.15.111:0/14657843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b006b060 0x7fe2b016db40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 >> 172.21.15.111:0/14657843 conn(0x7fe2b00cc580 msgr2=0x7fe2b00b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 shutdown_connections 2023-12-19T08:00:11.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.488+0000 7fe2b4c2e700 1 -- 172.21.15.111:0/14657843 wait complete. 2023-12-19T08:00:11.511 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:00:11.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.672+0000 7f72d06cd700 1 Processor -- start 2023-12-19T08:00:11.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.672+0000 7f72d06cd700 1 -- start start 2023-12-19T08:00:11.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.676+0000 7f72d06cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f72c80b0110 0x7f72c80b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.676+0000 7f72d06cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.676+0000 7f72d06cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c80ccb40 con 0x7f72c8153290 2023-12-19T08:00:11.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.676+0000 7f72d06cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c80cbf30 con 0x7f72c80b0110 2023-12-19T08:00:11.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.676+0000 7f72d06cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80cc210 con 0x7f72c80ccfd0 2023-12-19T08:00:11.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cec6a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f72c8153290 0x7f72c81536d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37864/0 (socket says 172.21.15.111:37864) 2023-12-19T08:00:11.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cec6a700 1 -- 172.21.15.111:0/2464547904 learned_addr learned my addr 172.21.15.111:0/2464547904 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:11.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 --2- 172.21.15.111:0/2464547904 >> v2:172.21.15.143:3300/0 conn(0x7f72c80b0110 0x7f72c80b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cdc68700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cd467700 1 -- 172.21.15.111:0/2464547904 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557900500 0 0) 0x7f72c80ccb40 con 0x7f72c8153290 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cd467700 1 -- 172.21.15.111:0/2464547904 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b4003570 con 0x7f72c8153290 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 -- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 msgr2=0x7f72c815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 -- 172.21.15.111:0/2464547904 >> v1:172.21.15.111:6789/0 conn(0x7f72c8153290 legacy=0x7f72c81536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 -- 172.21.15.111:0/2464547904 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c80cc4f0 con 0x7f72c80b0110 2023-12-19T08:00:11.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72ce469700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c80b0110 0x7f72c80b04e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f72c40098e0 tx=0x7f72c40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b95ecaaa171e5e79 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:11.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cd467700 1 -- 172.21.15.111:0/2464547904 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011630 con 0x7f72c80b0110 2023-12-19T08:00:11.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cd467700 1 -- 172.21.15.111:0/2464547904 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c4011790 con 0x7f72c80b0110 2023-12-19T08:00:11.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72cd467700 1 -- 172.21.15.111:0/2464547904 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011b30 con 0x7f72c80b0110 2023-12-19T08:00:11.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72d06cd700 1 -- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c80b0110 msgr2=0x7f72c80b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.680+0000 7f72d06cd700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c80b0110 0x7f72c80b04e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f72c40098e0 tx=0x7f72c40091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 -- 172.21.15.111:0/2464547904 shutdown_connections 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 --2- 172.21.15.111:0/2464547904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c80b0110 0x7f72c80b04e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 -- 172.21.15.111:0/2464547904 >> 172.21.15.111:0/2464547904 conn(0x7f72c806c7c0 msgr2=0x7f72c806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 -- 172.21.15.111:0/2464547904 shutdown_connections 2023-12-19T08:00:11.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 -- 172.21.15.111:0/2464547904 wait complete. 2023-12-19T08:00:11.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 Processor -- start 2023-12-19T08:00:11.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.684+0000 7f72d06cd700 1 -- start start 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c806a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c8153290 0x7f72c80687f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80682b0 con 0x7f72c80b0110 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c806ae90 con 0x7f72c8153290 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c80676a0 con 0x7f72c80ccfd0 2023-12-19T08:00:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46338/0 (socket says 172.21.15.111:46338) 2023-12-19T08:00:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 -- 172.21.15.111:0/3338609881 learned_addr learned my addr 172.21.15.111:0/3338609881 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72cdc68700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c806a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72cec6a700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c8153290 0x7f72c80687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 -- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 msgr2=0x7f72c806a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c806a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 -- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c8153290 msgr2=0x7f72c80687f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c8153290 0x7f72c80687f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 -- 172.21.15.111:0/3338609881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c4009420 con 0x7f72c80b0110 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72ce469700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f72c80649b0 tx=0x7f72c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011b10 con 0x7f72c80b0110 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c40253f0 con 0x7f72c80b0110 2023-12-19T08:00:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c8067980 con 0x7f72c80b0110 2023-12-19T08:00:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c40257b0 con 0x7f72c80b0110 2023-12-19T08:00:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.688+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c8067f40 con 0x7f72c80b0110 2023-12-19T08:00:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.692+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c8071120 con 0x7f72c80b0110 2023-12-19T08:00:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.692+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f72c4025b60 con 0x7f72c80b0110 2023-12-19T08:00:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.692+0000 7f72be7fc700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 0x7f72ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.692+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f72c407e360 con 0x7f72c80b0110 2023-12-19T08:00:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.692+0000 7f72cdc68700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 0x7f72ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.696+0000 7f72cdc68700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 0x7f72ac062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72c81527e0 tx=0x7f72b8008040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:11.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.700+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72c404e510 con 0x7f72c80b0110 2023-12-19T08:00:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.912+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72c80c8700 con 0x7f72c80b0110 2023-12-19T08:00:11.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.912+0000 7f72be7fc700 1 -- 172.21.15.111:0/3338609881 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7f72c4051b30 con 0x7f72c80b0110 2023-12-19T08:00:11.914 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:11.914 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 msgr2=0x7f72ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 0x7f72ac062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72c81527e0 tx=0x7f72b8008040 comp rx=0 tx=0).stop 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 msgr2=0x7f72c8151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f72c80649b0 tx=0x7f72c4017000 comp rx=0 tx=0).stop 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 shutdown_connections 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f72ac060670 0x7f72ac062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c8153290 0x7f72c80687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c80ccfd0 0x7f72c806a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 --2- 172.21.15.111:0/3338609881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72c80b0110 0x7f72c8151870 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 >> 172.21.15.111:0/3338609881 conn(0x7f72c806c7c0 msgr2=0x7f72c8159790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 shutdown_connections 2023-12-19T08:00:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:11.916+0000 7f72d06cd700 1 -- 172.21.15.111:0/3338609881 wait complete. 2023-12-19T08:00:11.918 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:00:11.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:00:12.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 Processor -- start 2023-12-19T08:00:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 -- start start 2023-12-19T08:00:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f111806ab50 0x7f11180cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f111806a690 0x7f11180cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11180ca030 con 0x7f11181553b0 2023-12-19T08:00:12.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11180b0750 con 0x7f111806ab50 2023-12-19T08:00:12.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f111d66e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11180ca4c0 con 0x7f111806a690 2023-12-19T08:00:12.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f11167fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f111806ab50 0x7f11180cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f11167fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f111806ab50 0x7f11180cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46144/0 (socket says 172.21.15.111:46144) 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f11167fc700 1 -- 172.21.15.111:0/1346171671 learned_addr learned my addr 172.21.15.111:0/1346171671 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.096+0000 7f1117fff700 1 --2- 172.21.15.111:0/1346171671 >> v2:172.21.15.143:3300/0 conn(0x7f111806a690 0x7f11180cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f11167fc700 1 -- 172.21.15.111:0/1346171671 >> v2:172.21.15.143:3300/0 conn(0x7f111806a690 msgr2=0x7f11180cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f11167fc700 1 --2- 172.21.15.111:0/1346171671 >> v2:172.21.15.143:3300/0 conn(0x7f111806a690 0x7f11180cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f11167fc700 1 -- 172.21.15.111:0/1346171671 >> v1:172.21.15.143:6789/0 conn(0x7f11181553b0 legacy=0x7f1118069f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f11167fc700 1 -- 172.21.15.111:0/1346171671 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11180ce6f0 con 0x7f111806ab50 2023-12-19T08:00:12.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f11167fc700 1 --2- 172.21.15.111:0/1346171671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f111806ab50 0x7f11180cb910 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f11180c8310 tx=0x7f110c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7209034995d09 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f1115ffb700 1 -- 172.21.15.111:0/1346171671 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c00be60 con 0x7f111806ab50 2023-12-19T08:00:12.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f1115ffb700 1 -- 172.21.15.111:0/1346171671 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c00ddc0 con 0x7f111806ab50 2023-12-19T08:00:12.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f1115ffb700 1 -- 172.21.15.111:0/1346171671 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c0196b0 con 0x7f111806ab50 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 -- 172.21.15.111:0/1346171671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f111806ab50 msgr2=0x7f11180cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 --2- 172.21.15.111:0/1346171671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f111806ab50 0x7f11180cb910 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f11180c8310 tx=0x7f110c00b4f0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 -- 172.21.15.111:0/1346171671 shutdown_connections 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 --2- 172.21.15.111:0/1346171671 >> v2:172.21.15.143:3300/0 conn(0x7f111806a690 0x7f11180cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 --2- 172.21.15.111:0/1346171671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f111806ab50 0x7f11180cb910 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 -- 172.21.15.111:0/1346171671 >> 172.21.15.111:0/1346171671 conn(0x7f1118069440 msgr2=0x7f11180cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:12.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 -- 172.21.15.111:0/1346171671 shutdown_connections 2023-12-19T08:00:12.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.100+0000 7f111d66e700 1 -- 172.21.15.111:0/1346171671 wait complete. 2023-12-19T08:00:12.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 Processor -- start 2023-12-19T08:00:12.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 -- start start 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f111806a690 0x7f11180af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f111806ab50 0x7f111814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180c6c40 con 0x7f11181553b0 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180c6f20 con 0x7f111806ab50 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f111d66e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11180c7200 con 0x7f111806a690 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f1117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f1117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46156/0 (socket says 172.21.15.111:46156) 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f1117fff700 1 -- 172.21.15.111:0/1421954869 learned_addr learned my addr 172.21.15.111:0/1421954869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f1116ffd700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f111806a690 0x7f11180af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.104+0000 7f11167fc700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f111806ab50 0x7f111814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 -- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f111806a690 msgr2=0x7f11180af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f111806a690 0x7f11180af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 -- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f111806ab50 msgr2=0x7f111814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f111806ab50 0x7f111814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 -- 172.21.15.111:0/1421954869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c00b790 con 0x7f11181553b0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f1117fff700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1110007f20 tx=0x7f11100098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111000eb90 con 0x7f11181553b0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111000ecf0 con 0x7f11181553b0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1110014670 con 0x7f11181553b0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11180c74e0 con 0x7f11181553b0 2023-12-19T08:00:12.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111814bc30 con 0x7f11181553b0 2023-12-19T08:00:12.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.108+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111814c380 con 0x7f11181553b0 2023-12-19T08:00:12.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.112+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1110014a60 con 0x7f11181553b0 2023-12-19T08:00:12.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.112+0000 7f10feffd700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 0x7f1104062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.112+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f111007d250 con 0x7f11181553b0 2023-12-19T08:00:12.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.112+0000 7f1116ffd700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 0x7f1104062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.112+0000 7f1116ffd700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 0x7f1104062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f11080039e0 tx=0x7f1108007ec0 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.120+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f111814c380 con 0x7f11181553b0 2023-12-19T08:00:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.292+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f111814c380 con 0x7f11181553b0 2023-12-19T08:00:12.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.292+0000 7f10feffd700 1 -- 172.21.15.111:0/1421954869 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16348 (secure 0 0 0) 0x7f111814c380 con 0x7f11181553b0 2023-12-19T08:00:12.295 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:12.295 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":68,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:00:09.364208+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.296+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 msgr2=0x7f1104062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.296+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 0x7f1104062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f11080039e0 tx=0x7f1108007ec0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 msgr2=0x7f111814eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1110007f20 tx=0x7f11100098a0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 shutdown_connections 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f11040606a0 0x7f1104062b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f111806ab50 0x7f111814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11181553b0 0x7f111814eb60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 --2- 172.21.15.111:0/1421954869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f111806a690 0x7f11180af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 >> 172.21.15.111:0/1421954869 conn(0x7f1118069440 msgr2=0x7f1118159160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 shutdown_connections 2023-12-19T08:00:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.300+0000 7f111d66e700 1 -- 172.21.15.111:0/1421954869 wait complete. 2023-12-19T08:00:12.320 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:00:12.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:00:12.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.480+0000 7fd2156ae700 1 Processor -- start 2023-12-19T08:00:12.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.480+0000 7fd2156ae700 1 -- start start 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2156ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd210158a70 0x7fd21006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2156ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 0x7fd21006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2156ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21006e100 con 0x7fd21006dd10 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2156ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21006a9d0 con 0x7fd210158a70 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2156ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21006acb0 con 0x7fd210158e40 2023-12-19T08:00:12.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd210158a70 0x7fd21006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd210158a70 0x7fd21006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46172/0 (socket says 172.21.15.111:46172) 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 -- 172.21.15.111:0/2853862275 learned_addr learned my addr 172.21.15.111:0/2853862275 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2067fc700 1 --2- 172.21.15.111:0/2853862275 >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 0x7fd21006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 -- 172.21.15.111:0/2853862275 >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 msgr2=0x7fd21006d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 --2- 172.21.15.111:0/2853862275 >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 0x7fd21006d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 -- 172.21.15.111:0/2853862275 >> v1:172.21.15.143:6789/0 conn(0x7fd21006dd10 legacy=0x7fd2100c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 -- 172.21.15.111:0/2853862275 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21006af90 con 0x7fd210158a70 2023-12-19T08:00:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd2067fc700 1 --2- 172.21.15.111:0/2853862275 >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 0x7fd21006d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.484+0000 7fd20effd700 1 --2- 172.21.15.111:0/2853862275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd210158a70 0x7fd21006bd50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd2100682f0 tx=0x7fd200009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d0af3f6404c88 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd20e7fc700 1 -- 172.21.15.111:0/2853862275 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000b860 con 0x7fd210158a70 2023-12-19T08:00:12.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd20e7fc700 1 -- 172.21.15.111:0/2853862275 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000b9c0 con 0x7fd210158a70 2023-12-19T08:00:12.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd20e7fc700 1 -- 172.21.15.111:0/2853862275 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000bd60 con 0x7fd210158a70 2023-12-19T08:00:12.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 -- 172.21.15.111:0/2853862275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd210158a70 msgr2=0x7fd21006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 --2- 172.21.15.111:0/2853862275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd210158a70 0x7fd21006bd50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd2100682f0 tx=0x7fd200009250 comp rx=0 tx=0).stop 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 -- 172.21.15.111:0/2853862275 shutdown_connections 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 --2- 172.21.15.111:0/2853862275 >> v2:172.21.15.143:3300/0 conn(0x7fd210158e40 0x7fd21006d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 --2- 172.21.15.111:0/2853862275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd210158a70 0x7fd21006bd50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 -- 172.21.15.111:0/2853862275 >> 172.21.15.111:0/2853862275 conn(0x7fd210067bb0 msgr2=0x7fd2100cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:12.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 -- 172.21.15.111:0/2853862275 shutdown_connections 2023-12-19T08:00:12.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.488+0000 7fd2156ae700 1 -- 172.21.15.111:0/2853862275 wait complete. 2023-12-19T08:00:12.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.492+0000 7fd2156ae700 1 Processor -- start 2023-12-19T08:00:12.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.492+0000 7fd2156ae700 1 -- start start 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd210158a70 0x7fd21014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd210158e40 0x7fd21014d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2100af650 con 0x7fd21006dd10 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2100ae760 con 0x7fd210158e40 2023-12-19T08:00:12.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2100aea40 con 0x7fd210158a70 2023-12-19T08:00:12.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46182/0 (socket says 172.21.15.111:46182) 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2067fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd210158a70 0x7fd21014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 -- 172.21.15.111:0/3676363948 learned_addr learned my addr 172.21.15.111:0/3676363948 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20f7fe700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd210158e40 0x7fd21014d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 -- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd210158a70 msgr2=0x7fd21014cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd210158a70 0x7fd21014cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 -- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd210158e40 msgr2=0x7fd21014d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd210158e40 0x7fd21014d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 -- 172.21.15.111:0/3676363948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000094f0 con 0x7fd21006dd10 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd20effd700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd200009290 tx=0x7fd200017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200026480 con 0x7fd21006dd10 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20001f070 con 0x7fd21006dd10 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200026850 con 0x7fd21006dd10 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2100aed20 con 0x7fd21006dd10 2023-12-19T08:00:12.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.496+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2100af2e0 con 0x7fd21006dd10 2023-12-19T08:00:12.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.500+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21004be40 con 0x7fd21006dd10 2023-12-19T08:00:12.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.500+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd20001a430 con 0x7fd21006dd10 2023-12-19T08:00:12.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.500+0000 7fd2077fe700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 0x7fd1f0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.500+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd200021070 con 0x7fd21006dd10 2023-12-19T08:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.500+0000 7fd2067fc700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 0x7fd1f0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.504+0000 7fd2067fc700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 0x7fd1f0062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2100ad9e0 tx=0x7fd1f8008040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.508+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd20004ebe0 con 0x7fd21006dd10 2023-12-19T08:00:12.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.676+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd21006b270 con 0x7fd1f00606a0 2023-12-19T08:00:12.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.696+0000 7fd2077fe700 1 -- 172.21.15.111:0/3676363948 <== mgr.5674 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286445 (secure 0 0 0) 0x7fd21006b270 con 0x7fd1f00606a0 2023-12-19T08:00:12.701 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:12.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 msgr2=0x7fd1f0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 0x7fd1f0062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd2100ad9e0 tx=0x7fd1f8008040 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 msgr2=0x7fd210137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd200009290 tx=0x7fd200017000 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 shutdown_connections 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd1f00606a0 0x7fd1f0062b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd210158e40 0x7fd21014d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd210158a70 0x7fd21014cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 --2- 172.21.15.111:0/3676363948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd21006dd10 0x7fd210137790 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 >> 172.21.15.111:0/3676363948 conn(0x7fd210067bb0 msgr2=0x7fd2100c7960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 shutdown_connections 2023-12-19T08:00:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.704+0000 7fd2156ae700 1 -- 172.21.15.111:0/3676363948 wait complete. 2023-12-19T08:00:12.706 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:00:12.726 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:00:11.989292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2422672,"kb_used_data":7504,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752552048,"statfs":{"total":773094113280,"available":770613297152,"internally_reserved":0,"allocated":7684096,"data_stored":4367581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567931"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.388311+0000","last_change":"2023-12-19T08:00:08.388311+0000","last_active":"2023-12-19T08:00:08.388311+0000","last_peered":"2023-12-19T08:00:08.388311+0000","last_clean":"2023-12-19T08:00:08.388311+0000","last_became_active":"2023-12-19T08:00:08.387985+0000","last_became_peered":"2023-12-19T08:00:08.387985+0000","last_unstale":"2023-12-19T08:00:08.388311+0000","last_undegraded":"2023-12-19T08:00:08.388311+0000","last_fullsized":"2023-12-19T08:00:08.388311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:41.875185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.404045+0000","last_change":"2023-12-19T08:00:08.404045+0000","last_active":"2023-12-19T08:00:08.404045+0000","last_peered":"2023-12-19T08:00:08.404045+0000","last_clean":"2023-12-19T08:00:08.404045+0000","last_became_active":"2023-12-19T08:00:08.401693+0000","last_became_peered":"2023-12-19T08:00:08.401693+0000","last_unstale":"2023-12-19T08:00:08.404045+0000","last_undegraded":"2023-12-19T08:00:08.404045+0000","last_fullsized":"2023-12-19T08:00:08.404045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394434+0000","last_change":"2023-12-19T08:00:08.394434+0000","last_active":"2023-12-19T08:00:08.394434+0000","last_peered":"2023-12-19T08:00:08.394434+0000","last_clean":"2023-12-19T08:00:08.394434+0000","last_became_active":"2023-12-19T08:00:08.394039+0000","last_became_peered":"2023-12-19T08:00:08.394039+0000","last_unstale":"2023-12-19T08:00:08.394434+0000","last_undegraded":"2023-12-19T08:00:08.394434+0000","last_fullsized":"2023-12-19T08:00:08.394434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:25.172819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.408429+0000","last_change":"2023-12-19T08:00:08.408429+0000","last_active":"2023-12-19T08:00:08.408429+0000","last_peered":"2023-12-19T08:00:08.408429+0000","last_clean":"2023-12-19T08:00:08.408429+0000","last_became_active":"2023-12-19T08:00:08.408160+0000","last_became_peered":"2023-12-19T08:00:08.408160+0000","last_unstale":"2023-12-19T08:00:08.408429+0000","last_undegraded":"2023-12-19T08:00:08.408429+0000","last_fullsized":"2023-12-19T08:00:08.408429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393332+0000","last_change":"2023-12-19T08:00:08.390410+0000","last_active":"2023-12-19T08:00:08.393332+0000","last_peered":"2023-12-19T08:00:08.393332+0000","last_clean":"2023-12-19T08:00:08.393332+0000","last_became_active":"2023-12-19T08:00:08.389866+0000","last_became_peered":"2023-12-19T08:00:08.389866+0000","last_unstale":"2023-12-19T08:00:08.393332+0000","last_undegraded":"2023-12-19T08:00:08.393332+0000","last_fullsized":"2023-12-19T08:00:08.393332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389924+0000","last_change":"2023-12-19T08:00:08.389924+0000","last_active":"2023-12-19T08:00:08.389924+0000","last_peered":"2023-12-19T08:00:08.389924+0000","last_clean":"2023-12-19T08:00:08.389924+0000","last_became_active":"2023-12-19T08:00:08.389634+0000","last_became_peered":"2023-12-19T08:00:08.389634+0000","last_unstale":"2023-12-19T08:00:08.389924+0000","last_undegraded":"2023-12-19T08:00:08.389924+0000","last_fullsized":"2023-12-19T08:00:08.389924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:05.302338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394117+0000","last_change":"2023-12-19T08:00:08.394117+0000","last_active":"2023-12-19T08:00:08.394117+0000","last_peered":"2023-12-19T08:00:08.394117+0000","last_clean":"2023-12-19T08:00:08.394117+0000","last_became_active":"2023-12-19T08:00:08.392982+0000","last_became_peered":"2023-12-19T08:00:08.392982+0000","last_unstale":"2023-12-19T08:00:08.394117+0000","last_undegraded":"2023-12-19T08:00:08.394117+0000","last_fullsized":"2023-12-19T08:00:08.394117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:38:46.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395325+0000","last_change":"2023-12-19T08:00:08.395325+0000","last_active":"2023-12-19T08:00:08.395325+0000","last_peered":"2023-12-19T08:00:08.395325+0000","last_clean":"2023-12-19T08:00:08.395325+0000","last_became_active":"2023-12-19T08:00:08.390961+0000","last_became_peered":"2023-12-19T08:00:08.390961+0000","last_unstale":"2023-12-19T08:00:08.395325+0000","last_undegraded":"2023-12-19T08:00:08.395325+0000","last_fullsized":"2023-12-19T08:00:08.395325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:30.885015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.396464+0000","last_change":"2023-12-19T08:00:08.396464+0000","last_active":"2023-12-19T08:00:08.396464+0000","last_peered":"2023-12-19T08:00:08.396464+0000","last_clean":"2023-12-19T08:00:08.396464+0000","last_became_active":"2023-12-19T08:00:08.396125+0000","last_became_peered":"2023-12-19T08:00:08.396125+0000","last_unstale":"2023-12-19T08:00:08.396464+0000","last_undegraded":"2023-12-19T08:00:08.396464+0000","last_fullsized":"2023-12-19T08:00:08.396464+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:16:45.597315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377184+0000","last_change":"2023-12-19T08:00:08.377184+0000","last_active":"2023-12-19T08:00:08.377184+0000","last_peered":"2023-12-19T08:00:08.377184+0000","last_clean":"2023-12-19T08:00:08.377184+0000","last_became_active":"2023-12-19T08:00:08.376726+0000","last_became_peered":"2023-12-19T08:00:08.376726+0000","last_unstale":"2023-12-19T08:00:08.377184+0000","last_undegraded":"2023-12-19T08:00:08.377184+0000","last_fullsized":"2023-12-19T08:00:08.377184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:55:47.021391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.403820+0000","last_change":"2023-12-19T08:00:08.399160+0000","last_active":"2023-12-19T08:00:08.403820+0000","last_peered":"2023-12-19T08:00:08.403820+0000","last_clean":"2023-12-19T08:00:08.403820+0000","last_became_active":"2023-12-19T08:00:08.398670+0000","last_became_peered":"2023-12-19T08:00:08.398670+0000","last_unstale":"2023-12-19T08:00:08.403820+0000","last_undegraded":"2023-12-19T08:00:08.403820+0000","last_fullsized":"2023-12-19T08:00:08.403820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:02:14.672353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390943+0000","last_change":"2023-12-19T08:00:08.390943+0000","last_active":"2023-12-19T08:00:08.390943+0000","last_peered":"2023-12-19T08:00:08.390943+0000","last_clean":"2023-12-19T08:00:08.390943+0000","last_became_active":"2023-12-19T08:00:08.390582+0000","last_became_peered":"2023-12-19T08:00:08.390582+0000","last_unstale":"2023-12-19T08:00:08.390943+0000","last_undegraded":"2023-12-19T08:00:08.390943+0000","last_fullsized":"2023-12-19T08:00:08.390943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:58.048582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377134+0000","last_change":"2023-12-19T08:00:08.377134+0000","last_active":"2023-12-19T08:00:08.377134+0000","last_peered":"2023-12-19T08:00:08.377134+0000","last_clean":"2023-12-19T08:00:08.377134+0000","last_became_active":"2023-12-19T08:00:08.376735+0000","last_became_peered":"2023-12-19T08:00:08.376735+0000","last_unstale":"2023-12-19T08:00:08.377134+0000","last_undegraded":"2023-12-19T08:00:08.377134+0000","last_fullsized":"2023-12-19T08:00:08.377134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:51:27.280347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398364+0000","last_change":"2023-12-19T08:00:08.398364+0000","last_active":"2023-12-19T08:00:08.398364+0000","last_peered":"2023-12-19T08:00:08.398364+0000","last_clean":"2023-12-19T08:00:08.398364+0000","last_became_active":"2023-12-19T08:00:08.394030+0000","last_became_peered":"2023-12-19T08:00:08.394030+0000","last_unstale":"2023-12-19T08:00:08.398364+0000","last_undegraded":"2023-12-19T08:00:08.398364+0000","last_fullsized":"2023-12-19T08:00:08.398364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:55:47.021391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.399597+0000","last_change":"2023-12-19T08:00:08.399597+0000","last_active":"2023-12-19T08:00:08.399597+0000","last_peered":"2023-12-19T08:00:08.399597+0000","last_clean":"2023-12-19T08:00:08.399597+0000","last_became_active":"2023-12-19T08:00:08.399001+0000","last_became_peered":"2023-12-19T08:00:08.399001+0000","last_unstale":"2023-12-19T08:00:08.399597+0000","last_undegraded":"2023-12-19T08:00:08.399597+0000","last_fullsized":"2023-12-19T08:00:08.399597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:49.428946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.376188+0000","last_change":"2023-12-19T08:00:08.376188+0000","last_active":"2023-12-19T08:00:08.376188+0000","last_peered":"2023-12-19T08:00:08.376188+0000","last_clean":"2023-12-19T08:00:08.376188+0000","last_became_active":"2023-12-19T08:00:08.375718+0000","last_became_peered":"2023-12-19T08:00:08.375718+0000","last_unstale":"2023-12-19T08:00:08.376188+0000","last_undegraded":"2023-12-19T08:00:08.376188+0000","last_fullsized":"2023-12-19T08:00:08.376188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:14.869430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394536+0000","last_change":"2023-12-19T08:00:08.394536+0000","last_active":"2023-12-19T08:00:08.394536+0000","last_peered":"2023-12-19T08:00:08.394536+0000","last_clean":"2023-12-19T08:00:08.394536+0000","last_became_active":"2023-12-19T08:00:08.391293+0000","last_became_peered":"2023-12-19T08:00:08.391293+0000","last_unstale":"2023-12-19T08:00:08.394536+0000","last_undegraded":"2023-12-19T08:00:08.394536+0000","last_fullsized":"2023-12-19T08:00:08.394536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:08:16.612655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.403854+0000","last_change":"2023-12-19T08:00:08.403854+0000","last_active":"2023-12-19T08:00:08.403854+0000","last_peered":"2023-12-19T08:00:08.403854+0000","last_clean":"2023-12-19T08:00:08.403854+0000","last_became_active":"2023-12-19T08:00:08.403317+0000","last_became_peered":"2023-12-19T08:00:08.403317+0000","last_unstale":"2023-12-19T08:00:08.403854+0000","last_undegraded":"2023-12-19T08:00:08.403854+0000","last_fullsized":"2023-12-19T08:00:08.403854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393129+0000","last_change":"2023-12-19T08:00:08.393129+0000","last_active":"2023-12-19T08:00:08.393129+0000","last_peered":"2023-12-19T08:00:08.393129+0000","last_clean":"2023-12-19T08:00:08.393129+0000","last_became_active":"2023-12-19T08:00:08.389863+0000","last_became_peered":"2023-12-19T08:00:08.389863+0000","last_unstale":"2023-12-19T08:00:08.393129+0000","last_undegraded":"2023-12-19T08:00:08.393129+0000","last_fullsized":"2023-12-19T08:00:08.393129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:51:27.280347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400390+0000","last_change":"2023-12-19T08:00:08.396015+0000","last_active":"2023-12-19T08:00:08.400390+0000","last_peered":"2023-12-19T08:00:08.400390+0000","last_clean":"2023-12-19T08:00:08.400390+0000","last_became_active":"2023-12-19T08:00:08.395329+0000","last_became_peered":"2023-12-19T08:00:08.395329+0000","last_unstale":"2023-12-19T08:00:08.400390+0000","last_undegraded":"2023-12-19T08:00:08.400390+0000","last_fullsized":"2023-12-19T08:00:08.400390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:20.013637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.407016+0000","last_change":"2023-12-19T08:00:08.403756+0000","last_active":"2023-12-19T08:00:08.407016+0000","last_peered":"2023-12-19T08:00:08.407016+0000","last_clean":"2023-12-19T08:00:08.407016+0000","last_became_active":"2023-12-19T08:00:08.403391+0000","last_became_peered":"2023-12-19T08:00:08.403391+0000","last_unstale":"2023-12-19T08:00:08.407016+0000","last_undegraded":"2023-12-19T08:00:08.407016+0000","last_fullsized":"2023-12-19T08:00:08.407016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:49:08.695041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390825+0000","last_change":"2023-12-19T08:00:08.390825+0000","last_active":"2023-12-19T08:00:08.390825+0000","last_peered":"2023-12-19T08:00:08.390825+0000","last_clean":"2023-12-19T08:00:08.390825+0000","last_became_active":"2023-12-19T08:00:08.390492+0000","last_became_peered":"2023-12-19T08:00:08.390492+0000","last_unstale":"2023-12-19T08:00:08.390825+0000","last_undegraded":"2023-12-19T08:00:08.390825+0000","last_fullsized":"2023-12-19T08:00:08.390825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:33.299420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398340+0000","last_change":"2023-12-19T08:00:08.395787+0000","last_active":"2023-12-19T08:00:08.398340+0000","last_peered":"2023-12-19T08:00:08.398340+0000","last_clean":"2023-12-19T08:00:08.398340+0000","last_became_active":"2023-12-19T08:00:08.395507+0000","last_became_peered":"2023-12-19T08:00:08.395507+0000","last_unstale":"2023-12-19T08:00:08.398340+0000","last_undegraded":"2023-12-19T08:00:08.398340+0000","last_fullsized":"2023-12-19T08:00:08.398340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:38:48.145515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389950+0000","last_change":"2023-12-19T08:00:08.389950+0000","last_active":"2023-12-19T08:00:08.389950+0000","last_peered":"2023-12-19T08:00:08.389950+0000","last_clean":"2023-12-19T08:00:08.389950+0000","last_became_active":"2023-12-19T08:00:08.388921+0000","last_became_peered":"2023-12-19T08:00:08.388921+0000","last_unstale":"2023-12-19T08:00:08.389950+0000","last_undegraded":"2023-12-19T08:00:08.389950+0000","last_fullsized":"2023-12-19T08:00:08.389950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:15:04.997205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394867+0000","last_change":"2023-12-19T08:00:08.394867+0000","last_active":"2023-12-19T08:00:08.394867+0000","last_peered":"2023-12-19T08:00:08.394867+0000","last_clean":"2023-12-19T08:00:08.394867+0000","last_became_active":"2023-12-19T08:00:08.394538+0000","last_became_peered":"2023-12-19T08:00:08.394538+0000","last_unstale":"2023-12-19T08:00:08.394867+0000","last_undegraded":"2023-12-19T08:00:08.394867+0000","last_fullsized":"2023-12-19T08:00:08.394867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:05:31.537513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.399154+0000","last_change":"2023-12-19T08:00:08.399154+0000","last_active":"2023-12-19T08:00:08.399154+0000","last_peered":"2023-12-19T08:00:08.399154+0000","last_clean":"2023-12-19T08:00:08.399154+0000","last_became_active":"2023-12-19T08:00:08.398881+0000","last_became_peered":"2023-12-19T08:00:08.398881+0000","last_unstale":"2023-12-19T08:00:08.399154+0000","last_undegraded":"2023-12-19T08:00:08.399154+0000","last_fullsized":"2023-12-19T08:00:08.399154+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:02:40.287354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.388407+0000","last_change":"2023-12-19T08:00:08.388407+0000","last_active":"2023-12-19T08:00:08.388407+0000","last_peered":"2023-12-19T08:00:08.388407+0000","last_clean":"2023-12-19T08:00:08.388407+0000","last_became_active":"2023-12-19T08:00:08.388092+0000","last_became_peered":"2023-12-19T08:00:08.388092+0000","last_unstale":"2023-12-19T08:00:08.388407+0000","last_undegraded":"2023-12-19T08:00:08.388407+0000","last_fullsized":"2023-12-19T08:00:08.388407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400410+0000","last_change":"2023-12-19T08:00:08.396837+0000","last_active":"2023-12-19T08:00:08.400410+0000","last_peered":"2023-12-19T08:00:08.400410+0000","last_clean":"2023-12-19T08:00:08.400410+0000","last_became_active":"2023-12-19T08:00:08.396520+0000","last_became_peered":"2023-12-19T08:00:08.396520+0000","last_unstale":"2023-12-19T08:00:08.400410+0000","last_undegraded":"2023-12-19T08:00:08.400410+0000","last_fullsized":"2023-12-19T08:00:08.400410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:03.721761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.385516+0000","last_change":"2023-12-19T08:00:08.385516+0000","last_active":"2023-12-19T08:00:08.385516+0000","last_peered":"2023-12-19T08:00:08.385516+0000","last_clean":"2023-12-19T08:00:08.385516+0000","last_became_active":"2023-12-19T08:00:08.385232+0000","last_became_peered":"2023-12-19T08:00:08.385232+0000","last_unstale":"2023-12-19T08:00:08.385516+0000","last_undegraded":"2023-12-19T08:00:08.385516+0000","last_fullsized":"2023-12-19T08:00:08.385516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:05.302338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405529+0000","last_change":"2023-12-19T08:00:08.405529+0000","last_active":"2023-12-19T08:00:08.405529+0000","last_peered":"2023-12-19T08:00:08.405529+0000","last_clean":"2023-12-19T08:00:08.405529+0000","last_became_active":"2023-12-19T08:00:08.404993+0000","last_became_peered":"2023-12-19T08:00:08.404993+0000","last_unstale":"2023-12-19T08:00:08.405529+0000","last_undegraded":"2023-12-19T08:00:08.405529+0000","last_fullsized":"2023-12-19T08:00:08.405529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:52.813620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.392726+0000","last_change":"2023-12-19T08:00:08.392726+0000","last_active":"2023-12-19T08:00:08.392726+0000","last_peered":"2023-12-19T08:00:08.392726+0000","last_clean":"2023-12-19T08:00:08.392726+0000","last_became_active":"2023-12-19T08:00:08.391785+0000","last_became_peered":"2023-12-19T08:00:08.391785+0000","last_unstale":"2023-12-19T08:00:08.392726+0000","last_undegraded":"2023-12-19T08:00:08.392726+0000","last_fullsized":"2023-12-19T08:00:08.392726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:02.360075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400602+0000","last_change":"2023-12-19T08:00:08.400602+0000","last_active":"2023-12-19T08:00:08.400602+0000","last_peered":"2023-12-19T08:00:08.400602+0000","last_clean":"2023-12-19T08:00:08.400602+0000","last_became_active":"2023-12-19T08:00:08.400174+0000","last_became_peered":"2023-12-19T08:00:08.400174+0000","last_unstale":"2023-12-19T08:00:08.400602+0000","last_undegraded":"2023-12-19T08:00:08.400602+0000","last_fullsized":"2023-12-19T08:00:08.400602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402323+0000","last_change":"2023-12-19T08:00:08.402323+0000","last_active":"2023-12-19T08:00:08.402323+0000","last_peered":"2023-12-19T08:00:08.402323+0000","last_clean":"2023-12-19T08:00:08.402323+0000","last_became_active":"2023-12-19T08:00:08.402027+0000","last_became_peered":"2023-12-19T08:00:08.402027+0000","last_unstale":"2023-12-19T08:00:08.402323+0000","last_undegraded":"2023-12-19T08:00:08.402323+0000","last_fullsized":"2023-12-19T08:00:08.402323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477181+0000","last_change":"2023-12-19T08:00:07.447816+0000","last_active":"2023-12-19T08:00:08.477181+0000","last_peered":"2023-12-19T08:00:08.477181+0000","last_clean":"2023-12-19T08:00:08.477181+0000","last_became_active":"2023-12-19T08:00:07.447496+0000","last_became_peered":"2023-12-19T08:00:07.447496+0000","last_unstale":"2023-12-19T08:00:08.477181+0000","last_undegraded":"2023-12-19T08:00:08.477181+0000","last_fullsized":"2023-12-19T08:00:08.477181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:20.994355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393292+0000","last_change":"2023-12-19T08:00:08.393292+0000","last_active":"2023-12-19T08:00:08.393292+0000","last_peered":"2023-12-19T08:00:08.393292+0000","last_clean":"2023-12-19T08:00:08.393292+0000","last_became_active":"2023-12-19T08:00:08.392373+0000","last_became_peered":"2023-12-19T08:00:08.392373+0000","last_unstale":"2023-12-19T08:00:08.393292+0000","last_undegraded":"2023-12-19T08:00:08.393292+0000","last_fullsized":"2023-12-19T08:00:08.393292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.371137+0000","last_change":"2023-12-19T08:00:07.455865+0000","last_active":"2023-12-19T08:00:08.371137+0000","last_peered":"2023-12-19T08:00:08.371137+0000","last_clean":"2023-12-19T08:00:08.371137+0000","last_became_active":"2023-12-19T08:00:07.451695+0000","last_became_peered":"2023-12-19T08:00:07.451695+0000","last_unstale":"2023-12-19T08:00:08.371137+0000","last_undegraded":"2023-12-19T08:00:08.371137+0000","last_fullsized":"2023-12-19T08:00:08.371137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:56:40.442903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405333+0000","last_change":"2023-12-19T08:00:08.403132+0000","last_active":"2023-12-19T08:00:08.405333+0000","last_peered":"2023-12-19T08:00:08.405333+0000","last_clean":"2023-12-19T08:00:08.405333+0000","last_became_active":"2023-12-19T08:00:08.401187+0000","last_became_peered":"2023-12-19T08:00:08.401187+0000","last_unstale":"2023-12-19T08:00:08.405333+0000","last_undegraded":"2023-12-19T08:00:08.405333+0000","last_fullsized":"2023-12-19T08:00:08.405333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.545605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.471647+0000","last_change":"2023-12-19T08:00:07.450829+0000","last_active":"2023-12-19T08:00:08.471647+0000","last_peered":"2023-12-19T08:00:08.471647+0000","last_clean":"2023-12-19T08:00:08.471647+0000","last_became_active":"2023-12-19T08:00:07.450439+0000","last_became_peered":"2023-12-19T08:00:07.450439+0000","last_unstale":"2023-12-19T08:00:08.471647+0000","last_undegraded":"2023-12-19T08:00:08.471647+0000","last_fullsized":"2023-12-19T08:00:08.471647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.001564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.397835+0000","last_change":"2023-12-19T08:00:08.397835+0000","last_active":"2023-12-19T08:00:08.397835+0000","last_peered":"2023-12-19T08:00:08.397835+0000","last_clean":"2023-12-19T08:00:08.397835+0000","last_became_active":"2023-12-19T08:00:08.397535+0000","last_became_peered":"2023-12-19T08:00:08.397535+0000","last_unstale":"2023-12-19T08:00:08.397835+0000","last_undegraded":"2023-12-19T08:00:08.397835+0000","last_fullsized":"2023-12-19T08:00:08.397835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:06:40.086080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.478851+0000","last_change":"2023-12-19T08:00:07.458733+0000","last_active":"2023-12-19T08:00:08.478851+0000","last_peered":"2023-12-19T08:00:08.478851+0000","last_clean":"2023-12-19T08:00:08.478851+0000","last_became_active":"2023-12-19T08:00:07.458394+0000","last_became_peered":"2023-12-19T08:00:07.458394+0000","last_unstale":"2023-12-19T08:00:08.478851+0000","last_undegraded":"2023-12-19T08:00:08.478851+0000","last_fullsized":"2023-12-19T08:00:08.478851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:02.709687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.401617+0000","last_change":"2023-12-19T08:00:08.390987+0000","last_active":"2023-12-19T08:00:08.401617+0000","last_peered":"2023-12-19T08:00:08.401617+0000","last_clean":"2023-12-19T08:00:08.401617+0000","last_became_active":"2023-12-19T08:00:08.390683+0000","last_became_peered":"2023-12-19T08:00:08.390683+0000","last_unstale":"2023-12-19T08:00:08.401617+0000","last_undegraded":"2023-12-19T08:00:08.401617+0000","last_fullsized":"2023-12-19T08:00:08.401617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.383144+0000","last_change":"2023-12-19T08:00:07.437859+0000","last_active":"2023-12-19T08:00:08.383144+0000","last_peered":"2023-12-19T08:00:08.383144+0000","last_clean":"2023-12-19T08:00:08.383144+0000","last_became_active":"2023-12-19T08:00:07.437563+0000","last_became_peered":"2023-12-19T08:00:07.437563+0000","last_unstale":"2023-12-19T08:00:08.383144+0000","last_undegraded":"2023-12-19T08:00:08.383144+0000","last_fullsized":"2023-12-19T08:00:08.383144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:18.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405691+0000","last_change":"2023-12-19T08:00:08.405691+0000","last_active":"2023-12-19T08:00:08.405691+0000","last_peered":"2023-12-19T08:00:08.405691+0000","last_clean":"2023-12-19T08:00:08.405691+0000","last_became_active":"2023-12-19T08:00:08.405231+0000","last_became_peered":"2023-12-19T08:00:08.405231+0000","last_unstale":"2023-12-19T08:00:08.405691+0000","last_undegraded":"2023-12-19T08:00:08.405691+0000","last_fullsized":"2023-12-19T08:00:08.405691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:06.401570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.476894+0000","last_change":"2023-12-19T08:00:07.449396+0000","last_active":"2023-12-19T08:00:08.476894+0000","last_peered":"2023-12-19T08:00:08.476894+0000","last_clean":"2023-12-19T08:00:08.476894+0000","last_became_active":"2023-12-19T08:00:07.448960+0000","last_became_peered":"2023-12-19T08:00:07.448960+0000","last_unstale":"2023-12-19T08:00:08.476894+0000","last_undegraded":"2023-12-19T08:00:08.476894+0000","last_fullsized":"2023-12-19T08:00:08.476894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:18.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400841+0000","last_change":"2023-12-19T08:00:08.400841+0000","last_active":"2023-12-19T08:00:08.400841+0000","last_peered":"2023-12-19T08:00:08.400841+0000","last_clean":"2023-12-19T08:00:08.400841+0000","last_became_active":"2023-12-19T08:00:08.396840+0000","last_became_peered":"2023-12-19T08:00:08.396840+0000","last_unstale":"2023-12-19T08:00:08.400841+0000","last_undegraded":"2023-12-19T08:00:08.400841+0000","last_fullsized":"2023-12-19T08:00:08.400841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:47.277168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.383629+0000","last_change":"2023-12-19T08:00:07.453804+0000","last_active":"2023-12-19T08:00:08.383629+0000","last_peered":"2023-12-19T08:00:08.383629+0000","last_clean":"2023-12-19T08:00:08.383629+0000","last_became_active":"2023-12-19T08:00:07.453231+0000","last_became_peered":"2023-12-19T08:00:07.453231+0000","last_unstale":"2023-12-19T08:00:08.383629+0000","last_undegraded":"2023-12-19T08:00:08.383629+0000","last_fullsized":"2023-12-19T08:00:08.383629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:57:55.773668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402693+0000","last_change":"2023-12-19T08:00:08.402693+0000","last_active":"2023-12-19T08:00:08.402693+0000","last_peered":"2023-12-19T08:00:08.402693+0000","last_clean":"2023-12-19T08:00:08.402693+0000","last_became_active":"2023-12-19T08:00:08.402193+0000","last_became_peered":"2023-12-19T08:00:08.402193+0000","last_unstale":"2023-12-19T08:00:08.402693+0000","last_undegraded":"2023-12-19T08:00:08.402693+0000","last_fullsized":"2023-12-19T08:00:08.402693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.452753+0000","last_change":"2023-12-19T08:00:07.443577+0000","last_active":"2023-12-19T08:00:07.452753+0000","last_peered":"2023-12-19T08:00:07.452753+0000","last_clean":"2023-12-19T08:00:07.452753+0000","last_became_active":"2023-12-19T08:00:07.443280+0000","last_became_peered":"2023-12-19T08:00:07.443280+0000","last_unstale":"2023-12-19T08:00:07.452753+0000","last_undegraded":"2023-12-19T08:00:07.452753+0000","last_fullsized":"2023-12-19T08:00:07.452753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:48:30.544946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405040+0000","last_change":"2023-12-19T08:00:08.396794+0000","last_active":"2023-12-19T08:00:08.405040+0000","last_peered":"2023-12-19T08:00:08.405040+0000","last_clean":"2023-12-19T08:00:08.405040+0000","last_became_active":"2023-12-19T08:00:08.396420+0000","last_became_peered":"2023-12-19T08:00:08.396420+0000","last_unstale":"2023-12-19T08:00:08.405040+0000","last_undegraded":"2023-12-19T08:00:08.405040+0000","last_fullsized":"2023-12-19T08:00:08.405040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.464450+0000","last_change":"2023-12-19T08:00:07.461448+0000","last_active":"2023-12-19T08:00:07.464450+0000","last_peered":"2023-12-19T08:00:07.464450+0000","last_clean":"2023-12-19T08:00:07.464450+0000","last_became_active":"2023-12-19T08:00:07.460406+0000","last_became_peered":"2023-12-19T08:00:07.460406+0000","last_unstale":"2023-12-19T08:00:07.464450+0000","last_undegraded":"2023-12-19T08:00:07.464450+0000","last_fullsized":"2023-12-19T08:00:07.464450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:48:30.544946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394867+0000","last_change":"2023-12-19T08:00:08.394867+0000","last_active":"2023-12-19T08:00:08.394867+0000","last_peered":"2023-12-19T08:00:08.394867+0000","last_clean":"2023-12-19T08:00:08.394867+0000","last_became_active":"2023-12-19T08:00:08.394354+0000","last_became_peered":"2023-12-19T08:00:08.394354+0000","last_unstale":"2023-12-19T08:00:08.394867+0000","last_undegraded":"2023-12-19T08:00:08.394867+0000","last_fullsized":"2023-12-19T08:00:08.394867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:06.401570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377982+0000","last_change":"2023-12-19T08:00:07.455332+0000","last_active":"2023-12-19T08:00:08.377982+0000","last_peered":"2023-12-19T08:00:08.377982+0000","last_clean":"2023-12-19T08:00:08.377982+0000","last_became_active":"2023-12-19T08:00:07.455016+0000","last_became_peered":"2023-12-19T08:00:07.455016+0000","last_unstale":"2023-12-19T08:00:08.377982+0000","last_undegraded":"2023-12-19T08:00:08.377982+0000","last_fullsized":"2023-12-19T08:00:08.377982+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.001564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.385131+0000","last_change":"2023-12-19T08:00:08.385131+0000","last_active":"2023-12-19T08:00:08.385131+0000","last_peered":"2023-12-19T08:00:08.385131+0000","last_clean":"2023-12-19T08:00:08.385131+0000","last_became_active":"2023-12-19T08:00:08.383224+0000","last_became_peered":"2023-12-19T08:00:08.383224+0000","last_unstale":"2023-12-19T08:00:08.385131+0000","last_undegraded":"2023-12-19T08:00:08.385131+0000","last_fullsized":"2023-12-19T08:00:08.385131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477302+0000","last_change":"2023-12-19T08:00:07.464164+0000","last_active":"2023-12-19T08:00:08.477302+0000","last_peered":"2023-12-19T08:00:08.477302+0000","last_clean":"2023-12-19T08:00:08.477302+0000","last_became_active":"2023-12-19T08:00:07.463904+0000","last_became_peered":"2023-12-19T08:00:07.463904+0000","last_unstale":"2023-12-19T08:00:08.477302+0000","last_undegraded":"2023-12-19T08:00:08.477302+0000","last_fullsized":"2023-12-19T08:00:08.477302+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:14.938405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405006+0000","last_change":"2023-12-19T08:00:08.395867+0000","last_active":"2023-12-19T08:00:08.405006+0000","last_peered":"2023-12-19T08:00:08.405006+0000","last_clean":"2023-12-19T08:00:08.405006+0000","last_became_active":"2023-12-19T08:00:08.395449+0000","last_became_peered":"2023-12-19T08:00:08.395449+0000","last_unstale":"2023-12-19T08:00:08.405006+0000","last_undegraded":"2023-12-19T08:00:08.405006+0000","last_fullsized":"2023-12-19T08:00:08.405006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.475523+0000","last_change":"2023-12-19T08:00:07.448669+0000","last_active":"2023-12-19T08:00:08.475523+0000","last_peered":"2023-12-19T08:00:08.475523+0000","last_clean":"2023-12-19T08:00:08.475523+0000","last_became_active":"2023-12-19T08:00:07.448390+0000","last_became_peered":"2023-12-19T08:00:07.448390+0000","last_unstale":"2023-12-19T08:00:08.475523+0000","last_undegraded":"2023-12-19T08:00:08.475523+0000","last_fullsized":"2023-12-19T08:00:08.475523+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:18:47.869632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394490+0000","last_change":"2023-12-19T08:00:08.394490+0000","last_active":"2023-12-19T08:00:08.394490+0000","last_peered":"2023-12-19T08:00:08.394490+0000","last_clean":"2023-12-19T08:00:08.394490+0000","last_became_active":"2023-12-19T08:00:08.392607+0000","last_became_peered":"2023-12-19T08:00:08.392607+0000","last_unstale":"2023-12-19T08:00:08.394490+0000","last_undegraded":"2023-12-19T08:00:08.394490+0000","last_fullsized":"2023-12-19T08:00:08.394490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.474618+0000","last_change":"2023-12-19T08:00:07.451368+0000","last_active":"2023-12-19T08:00:08.474618+0000","last_peered":"2023-12-19T08:00:08.474618+0000","last_clean":"2023-12-19T08:00:08.474618+0000","last_became_active":"2023-12-19T08:00:07.451053+0000","last_became_peered":"2023-12-19T08:00:07.451053+0000","last_unstale":"2023-12-19T08:00:08.474618+0000","last_undegraded":"2023-12-19T08:00:08.474618+0000","last_fullsized":"2023-12-19T08:00:08.474618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:26.770230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393692+0000","last_change":"2023-12-19T08:00:08.393692+0000","last_active":"2023-12-19T08:00:08.393692+0000","last_peered":"2023-12-19T08:00:08.393692+0000","last_clean":"2023-12-19T08:00:08.393692+0000","last_became_active":"2023-12-19T08:00:08.393410+0000","last_became_peered":"2023-12-19T08:00:08.393410+0000","last_unstale":"2023-12-19T08:00:08.393692+0000","last_undegraded":"2023-12-19T08:00:08.393692+0000","last_fullsized":"2023-12-19T08:00:08.393692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:59:53.453893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384011+0000","last_change":"2023-12-19T08:00:07.445415+0000","last_active":"2023-12-19T08:00:08.384011+0000","last_peered":"2023-12-19T08:00:08.384011+0000","last_clean":"2023-12-19T08:00:08.384011+0000","last_became_active":"2023-12-19T08:00:07.445093+0000","last_became_peered":"2023-12-19T08:00:07.445093+0000","last_unstale":"2023-12-19T08:00:08.384011+0000","last_undegraded":"2023-12-19T08:00:08.384011+0000","last_fullsized":"2023-12-19T08:00:08.384011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:08.608453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.396901+0000","last_change":"2023-12-19T08:00:08.396901+0000","last_active":"2023-12-19T08:00:08.396901+0000","last_peered":"2023-12-19T08:00:08.396901+0000","last_clean":"2023-12-19T08:00:08.396901+0000","last_became_active":"2023-12-19T08:00:08.395649+0000","last_became_peered":"2023-12-19T08:00:08.395649+0000","last_unstale":"2023-12-19T08:00:08.396901+0000","last_undegraded":"2023-12-19T08:00:08.396901+0000","last_fullsized":"2023-12-19T08:00:08.396901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:53.028508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.462268+0000","last_change":"2023-12-19T08:00:07.459719+0000","last_active":"2023-12-19T08:00:07.462268+0000","last_peered":"2023-12-19T08:00:07.462268+0000","last_clean":"2023-12-19T08:00:07.462268+0000","last_became_active":"2023-12-19T08:00:07.459427+0000","last_became_peered":"2023-12-19T08:00:07.459427+0000","last_unstale":"2023-12-19T08:00:07.462268+0000","last_undegraded":"2023-12-19T08:00:07.462268+0000","last_fullsized":"2023-12-19T08:00:07.462268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:18.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.391559+0000","last_change":"2023-12-19T08:00:08.391559+0000","last_active":"2023-12-19T08:00:08.391559+0000","last_peered":"2023-12-19T08:00:08.391559+0000","last_clean":"2023-12-19T08:00:08.391559+0000","last_became_active":"2023-12-19T08:00:08.390919+0000","last_became_peered":"2023-12-19T08:00:08.390919+0000","last_unstale":"2023-12-19T08:00:08.391559+0000","last_undegraded":"2023-12-19T08:00:08.391559+0000","last_fullsized":"2023-12-19T08:00:08.391559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:56:54.670456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.475696+0000","last_change":"2023-12-19T08:00:07.456186+0000","last_active":"2023-12-19T08:00:08.475696+0000","last_peered":"2023-12-19T08:00:08.475696+0000","last_clean":"2023-12-19T08:00:08.475696+0000","last_became_active":"2023-12-19T08:00:07.455879+0000","last_became_peered":"2023-12-19T08:00:07.455879+0000","last_unstale":"2023-12-19T08:00:08.475696+0000","last_undegraded":"2023-12-19T08:00:08.475696+0000","last_fullsized":"2023-12-19T08:00:08.475696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:51:05.852256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395276+0000","last_change":"2023-12-19T08:00:08.395276+0000","last_active":"2023-12-19T08:00:08.395276+0000","last_peered":"2023-12-19T08:00:08.395276+0000","last_clean":"2023-12-19T08:00:08.395276+0000","last_became_active":"2023-12-19T08:00:08.390538+0000","last_became_peered":"2023-12-19T08:00:08.390538+0000","last_unstale":"2023-12-19T08:00:08.395276+0000","last_undegraded":"2023-12-19T08:00:08.395276+0000","last_fullsized":"2023-12-19T08:00:08.395276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:47.296674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384004+0000","last_change":"2023-12-19T08:00:07.456833+0000","last_active":"2023-12-19T08:00:08.384004+0000","last_peered":"2023-12-19T08:00:08.384004+0000","last_clean":"2023-12-19T08:00:08.384004+0000","last_became_active":"2023-12-19T08:00:07.456547+0000","last_became_peered":"2023-12-19T08:00:07.456547+0000","last_unstale":"2023-12-19T08:00:08.384004+0000","last_undegraded":"2023-12-19T08:00:08.384004+0000","last_fullsized":"2023-12-19T08:00:08.384004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:36.901880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386947+0000","last_change":"2023-12-19T08:00:08.386947+0000","last_active":"2023-12-19T08:00:08.386947+0000","last_peered":"2023-12-19T08:00:08.386947+0000","last_clean":"2023-12-19T08:00:08.386947+0000","last_became_active":"2023-12-19T08:00:08.386615+0000","last_became_peered":"2023-12-19T08:00:08.386615+0000","last_unstale":"2023-12-19T08:00:08.386947+0000","last_undegraded":"2023-12-19T08:00:08.386947+0000","last_fullsized":"2023-12-19T08:00:08.386947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:42.828044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.474519+0000","last_change":"2023-12-19T08:00:07.450210+0000","last_active":"2023-12-19T08:00:08.474519+0000","last_peered":"2023-12-19T08:00:08.474519+0000","last_clean":"2023-12-19T08:00:08.474519+0000","last_became_active":"2023-12-19T08:00:07.449951+0000","last_became_peered":"2023-12-19T08:00:07.449951+0000","last_unstale":"2023-12-19T08:00:08.474519+0000","last_undegraded":"2023-12-19T08:00:08.474519+0000","last_fullsized":"2023-12-19T08:00:08.474519+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:25:22.223342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402214+0000","last_change":"2023-12-19T08:00:08.391873+0000","last_active":"2023-12-19T08:00:08.402214+0000","last_peered":"2023-12-19T08:00:08.402214+0000","last_clean":"2023-12-19T08:00:08.402214+0000","last_became_active":"2023-12-19T08:00:08.391578+0000","last_became_peered":"2023-12-19T08:00:08.391578+0000","last_unstale":"2023-12-19T08:00:08.402214+0000","last_undegraded":"2023-12-19T08:00:08.402214+0000","last_fullsized":"2023-12-19T08:00:08.402214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477997+0000","last_change":"2023-12-19T08:00:07.458199+0000","last_active":"2023-12-19T08:00:08.477997+0000","last_peered":"2023-12-19T08:00:08.477997+0000","last_clean":"2023-12-19T08:00:08.477997+0000","last_became_active":"2023-12-19T08:00:07.457542+0000","last_became_peered":"2023-12-19T08:00:07.457542+0000","last_unstale":"2023-12-19T08:00:08.477997+0000","last_undegraded":"2023-12-19T08:00:08.477997+0000","last_fullsized":"2023-12-19T08:00:08.477997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:42:20.373572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394895+0000","last_change":"2023-12-19T08:00:08.394895+0000","last_active":"2023-12-19T08:00:08.394895+0000","last_peered":"2023-12-19T08:00:08.394895+0000","last_clean":"2023-12-19T08:00:08.394895+0000","last_became_active":"2023-12-19T08:00:08.394511+0000","last_became_peered":"2023-12-19T08:00:08.394511+0000","last_unstale":"2023-12-19T08:00:08.394895+0000","last_undegraded":"2023-12-19T08:00:08.394895+0000","last_fullsized":"2023-12-19T08:00:08.394895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:02.360075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386973+0000","last_change":"2023-12-19T08:00:07.456049+0000","last_active":"2023-12-19T08:00:08.386973+0000","last_peered":"2023-12-19T08:00:08.386973+0000","last_clean":"2023-12-19T08:00:08.386973+0000","last_became_active":"2023-12-19T08:00:07.455706+0000","last_became_peered":"2023-12-19T08:00:07.455706+0000","last_unstale":"2023-12-19T08:00:08.386973+0000","last_undegraded":"2023-12-19T08:00:08.386973+0000","last_fullsized":"2023-12-19T08:00:08.386973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:03.457275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394844+0000","last_change":"2023-12-19T08:00:08.394844+0000","last_active":"2023-12-19T08:00:08.394844+0000","last_peered":"2023-12-19T08:00:08.394844+0000","last_clean":"2023-12-19T08:00:08.394844+0000","last_became_active":"2023-12-19T08:00:08.394453+0000","last_became_peered":"2023-12-19T08:00:08.394453+0000","last_unstale":"2023-12-19T08:00:08.394844+0000","last_undegraded":"2023-12-19T08:00:08.394844+0000","last_fullsized":"2023-12-19T08:00:08.394844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:52.813620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.473661+0000","last_change":"2023-12-19T08:00:07.447562+0000","last_active":"2023-12-19T08:00:08.473661+0000","last_peered":"2023-12-19T08:00:08.473661+0000","last_clean":"2023-12-19T08:00:08.473661+0000","last_became_active":"2023-12-19T08:00:07.447236+0000","last_became_peered":"2023-12-19T08:00:07.447236+0000","last_unstale":"2023-12-19T08:00:08.473661+0000","last_undegraded":"2023-12-19T08:00:08.473661+0000","last_fullsized":"2023-12-19T08:00:08.473661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:18.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395131+0000","last_change":"2023-12-19T08:00:08.393818+0000","last_active":"2023-12-19T08:00:08.395131+0000","last_peered":"2023-12-19T08:00:08.395131+0000","last_clean":"2023-12-19T08:00:08.395131+0000","last_became_active":"2023-12-19T08:00:08.393451+0000","last_became_peered":"2023-12-19T08:00:08.393451+0000","last_unstale":"2023-12-19T08:00:08.395131+0000","last_undegraded":"2023-12-19T08:00:08.395131+0000","last_fullsized":"2023-12-19T08:00:08.395131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:37.913953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.455550+0000","last_change":"2023-12-19T08:00:07.447387+0000","last_active":"2023-12-19T08:00:07.455550+0000","last_peered":"2023-12-19T08:00:07.455550+0000","last_clean":"2023-12-19T08:00:07.455550+0000","last_became_active":"2023-12-19T08:00:07.447067+0000","last_became_peered":"2023-12-19T08:00:07.447067+0000","last_unstale":"2023-12-19T08:00:07.455550+0000","last_undegraded":"2023-12-19T08:00:07.455550+0000","last_fullsized":"2023-12-19T08:00:07.455550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:24.995268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384694+0000","last_change":"2023-12-19T08:00:08.384694+0000","last_active":"2023-12-19T08:00:08.384694+0000","last_peered":"2023-12-19T08:00:08.384694+0000","last_clean":"2023-12-19T08:00:08.384694+0000","last_became_active":"2023-12-19T08:00:08.384193+0000","last_became_peered":"2023-12-19T08:00:08.384193+0000","last_unstale":"2023-12-19T08:00:08.384694+0000","last_undegraded":"2023-12-19T08:00:08.384694+0000","last_fullsized":"2023-12-19T08:00:08.384694+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.459646+0000","last_change":"2023-12-19T08:00:07.457517+0000","last_active":"2023-12-19T08:00:07.459646+0000","last_peered":"2023-12-19T08:00:07.459646+0000","last_clean":"2023-12-19T08:00:07.459646+0000","last_became_active":"2023-12-19T08:00:07.457244+0000","last_became_peered":"2023-12-19T08:00:07.457244+0000","last_unstale":"2023-12-19T08:00:07.459646+0000","last_undegraded":"2023-12-19T08:00:07.459646+0000","last_fullsized":"2023-12-19T08:00:07.459646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:55:50.171010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.401203+0000","last_change":"2023-12-19T08:00:08.401203+0000","last_active":"2023-12-19T08:00:08.401203+0000","last_peered":"2023-12-19T08:00:08.401203+0000","last_clean":"2023-12-19T08:00:08.401203+0000","last_became_active":"2023-12-19T08:00:08.400859+0000","last_became_peered":"2023-12-19T08:00:08.400859+0000","last_unstale":"2023-12-19T08:00:08.401203+0000","last_undegraded":"2023-12-19T08:00:08.401203+0000","last_fullsized":"2023-12-19T08:00:08.401203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:38:46.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386639+0000","last_change":"2023-12-19T08:00:07.449435+0000","last_active":"2023-12-19T08:00:08.386639+0000","last_peered":"2023-12-19T08:00:08.386639+0000","last_clean":"2023-12-19T08:00:08.386639+0000","last_became_active":"2023-12-19T08:00:07.449153+0000","last_became_peered":"2023-12-19T08:00:07.449153+0000","last_unstale":"2023-12-19T08:00:08.386639+0000","last_undegraded":"2023-12-19T08:00:08.386639+0000","last_fullsized":"2023-12-19T08:00:08.386639+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:26:48.499202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398312+0000","last_change":"2023-12-19T08:00:08.398312+0000","last_active":"2023-12-19T08:00:08.398312+0000","last_peered":"2023-12-19T08:00:08.398312+0000","last_clean":"2023-12-19T08:00:08.398312+0000","last_became_active":"2023-12-19T08:00:08.398013+0000","last_became_peered":"2023-12-19T08:00:08.398013+0000","last_unstale":"2023-12-19T08:00:08.398312+0000","last_undegraded":"2023-12-19T08:00:08.398312+0000","last_fullsized":"2023-12-19T08:00:08.398312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477345+0000","last_change":"2023-12-19T08:00:07.458069+0000","last_active":"2023-12-19T08:00:08.477345+0000","last_peered":"2023-12-19T08:00:08.477345+0000","last_clean":"2023-12-19T08:00:08.477345+0000","last_became_active":"2023-12-19T08:00:07.457730+0000","last_became_peered":"2023-12-19T08:00:07.457730+0000","last_unstale":"2023-12-19T08:00:08.477345+0000","last_undegraded":"2023-12-19T08:00:08.477345+0000","last_fullsized":"2023-12-19T08:00:08.477345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:22.088551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393375+0000","last_change":"2023-12-19T08:00:08.393375+0000","last_active":"2023-12-19T08:00:08.393375+0000","last_peered":"2023-12-19T08:00:08.393375+0000","last_clean":"2023-12-19T08:00:08.393375+0000","last_became_active":"2023-12-19T08:00:08.392652+0000","last_became_peered":"2023-12-19T08:00:08.392652+0000","last_unstale":"2023-12-19T08:00:08.393375+0000","last_undegraded":"2023-12-19T08:00:08.393375+0000","last_fullsized":"2023-12-19T08:00:08.393375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:59:53.453893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.459266+0000","last_change":"2023-12-19T08:00:07.454277+0000","last_active":"2023-12-19T08:00:07.459266+0000","last_peered":"2023-12-19T08:00:07.459266+0000","last_clean":"2023-12-19T08:00:07.459266+0000","last_became_active":"2023-12-19T08:00:07.453986+0000","last_became_peered":"2023-12-19T08:00:07.453986+0000","last_unstale":"2023-12-19T08:00:07.459266+0000","last_undegraded":"2023-12-19T08:00:07.459266+0000","last_fullsized":"2023-12-19T08:00:07.459266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:26.770230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.392245+0000","last_change":"2023-12-19T08:00:08.392245+0000","last_active":"2023-12-19T08:00:08.392245+0000","last_peered":"2023-12-19T08:00:08.392245+0000","last_clean":"2023-12-19T08:00:08.392245+0000","last_became_active":"2023-12-19T08:00:08.391847+0000","last_became_peered":"2023-12-19T08:00:08.391847+0000","last_unstale":"2023-12-19T08:00:08.392245+0000","last_undegraded":"2023-12-19T08:00:08.392245+0000","last_fullsized":"2023-12-19T08:00:08.392245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:29:11.403091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456226+0000","last_change":"2023-12-19T08:00:07.452077+0000","last_active":"2023-12-19T08:00:07.456226+0000","last_peered":"2023-12-19T08:00:07.456226+0000","last_clean":"2023-12-19T08:00:07.456226+0000","last_became_active":"2023-12-19T08:00:07.451759+0000","last_became_peered":"2023-12-19T08:00:07.451759+0000","last_unstale":"2023-12-19T08:00:07.456226+0000","last_undegraded":"2023-12-19T08:00:07.456226+0000","last_fullsized":"2023-12-19T08:00:07.456226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:47:56.527292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400660+0000","last_change":"2023-12-19T08:00:08.400660+0000","last_active":"2023-12-19T08:00:08.400660+0000","last_peered":"2023-12-19T08:00:08.400660+0000","last_clean":"2023-12-19T08:00:08.400660+0000","last_became_active":"2023-12-19T08:00:08.400296+0000","last_became_peered":"2023-12-19T08:00:08.400296+0000","last_unstale":"2023-12-19T08:00:08.400660+0000","last_undegraded":"2023-12-19T08:00:08.400660+0000","last_fullsized":"2023-12-19T08:00:08.400660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:14.869430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456013+0000","last_change":"2023-12-19T08:00:07.451665+0000","last_active":"2023-12-19T08:00:07.456013+0000","last_peered":"2023-12-19T08:00:07.456013+0000","last_clean":"2023-12-19T08:00:07.456013+0000","last_became_active":"2023-12-19T08:00:07.451357+0000","last_became_peered":"2023-12-19T08:00:07.451357+0000","last_unstale":"2023-12-19T08:00:07.456013+0000","last_undegraded":"2023-12-19T08:00:07.456013+0000","last_fullsized":"2023-12-19T08:00:07.456013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:57:57.248649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390216+0000","last_change":"2023-12-19T08:00:08.390216+0000","last_active":"2023-12-19T08:00:08.390216+0000","last_peered":"2023-12-19T08:00:08.390216+0000","last_clean":"2023-12-19T08:00:08.390216+0000","last_became_active":"2023-12-19T08:00:08.389934+0000","last_became_peered":"2023-12-19T08:00:08.389934+0000","last_unstale":"2023-12-19T08:00:08.390216+0000","last_undegraded":"2023-12-19T08:00:08.390216+0000","last_fullsized":"2023-12-19T08:00:08.390216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456066+0000","last_change":"2023-12-19T08:00:07.451675+0000","last_active":"2023-12-19T08:00:07.456066+0000","last_peered":"2023-12-19T08:00:07.456066+0000","last_clean":"2023-12-19T08:00:07.456066+0000","last_became_active":"2023-12-19T08:00:07.451394+0000","last_became_peered":"2023-12-19T08:00:07.451394+0000","last_unstale":"2023-12-19T08:00:07.456066+0000","last_undegraded":"2023-12-19T08:00:07.456066+0000","last_fullsized":"2023-12-19T08:00:07.456066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:22.910802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389507+0000","last_change":"2023-12-19T08:00:08.389507+0000","last_active":"2023-12-19T08:00:08.389507+0000","last_peered":"2023-12-19T08:00:08.389507+0000","last_clean":"2023-12-19T08:00:08.389507+0000","last_became_active":"2023-12-19T08:00:08.389198+0000","last_became_peered":"2023-12-19T08:00:08.389198+0000","last_unstale":"2023-12-19T08:00:08.389507+0000","last_undegraded":"2023-12-19T08:00:08.389507+0000","last_fullsized":"2023-12-19T08:00:08.389507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:07.392084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377713+0000","last_change":"2023-12-19T08:00:07.462259+0000","last_active":"2023-12-19T08:00:08.377713+0000","last_peered":"2023-12-19T08:00:08.377713+0000","last_clean":"2023-12-19T08:00:08.377713+0000","last_became_active":"2023-12-19T08:00:07.461447+0000","last_became_peered":"2023-12-19T08:00:07.461447+0000","last_unstale":"2023-12-19T08:00:08.377713+0000","last_undegraded":"2023-12-19T08:00:08.377713+0000","last_fullsized":"2023-12-19T08:00:08.377713+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:49:07.682968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402323+0000","last_change":"2023-12-19T08:00:08.402323+0000","last_active":"2023-12-19T08:00:08.402323+0000","last_peered":"2023-12-19T08:00:08.402323+0000","last_clean":"2023-12-19T08:00:08.402323+0000","last_became_active":"2023-12-19T08:00:08.401981+0000","last_became_peered":"2023-12-19T08:00:08.401981+0000","last_unstale":"2023-12-19T08:00:08.402323+0000","last_undegraded":"2023-12-19T08:00:08.402323+0000","last_fullsized":"2023-12-19T08:00:08.402323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:31:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.461135+0000","last_change":"2023-12-19T08:00:07.457828+0000","last_active":"2023-12-19T08:00:07.461135+0000","last_peered":"2023-12-19T08:00:07.461135+0000","last_clean":"2023-12-19T08:00:07.461135+0000","last_became_active":"2023-12-19T08:00:07.457521+0000","last_became_peered":"2023-12-19T08:00:07.457521+0000","last_unstale":"2023-12-19T08:00:07.461135+0000","last_undegraded":"2023-12-19T08:00:07.461135+0000","last_fullsized":"2023-12-19T08:00:07.461135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:36.901880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405631+0000","last_change":"2023-12-19T08:00:08.400553+0000","last_active":"2023-12-19T08:00:08.405631+0000","last_peered":"2023-12-19T08:00:08.405631+0000","last_clean":"2023-12-19T08:00:08.405631+0000","last_became_active":"2023-12-19T08:00:08.396748+0000","last_became_peered":"2023-12-19T08:00:08.396748+0000","last_unstale":"2023-12-19T08:00:08.405631+0000","last_undegraded":"2023-12-19T08:00:08.405631+0000","last_fullsized":"2023-12-19T08:00:08.405631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:56:41.454976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.455225+0000","last_change":"2023-12-19T08:00:07.453202+0000","last_active":"2023-12-19T08:00:07.455225+0000","last_peered":"2023-12-19T08:00:07.455225+0000","last_clean":"2023-12-19T08:00:07.455225+0000","last_became_active":"2023-12-19T08:00:07.452929+0000","last_became_peered":"2023-12-19T08:00:07.452929+0000","last_unstale":"2023-12-19T08:00:07.455225+0000","last_undegraded":"2023-12-19T08:00:07.455225+0000","last_fullsized":"2023-12-19T08:00:07.455225+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:14.938405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":281,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.449109+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:00:07.449109+0000","last_peered":"2023-12-19T08:00:07.449109+0000","last_clean":"2023-12-19T08:00:07.449109+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:00:07.449109+0000","last_undegraded":"2023-12-19T08:00:07.449109+0000","last_fullsized":"2023-12-19T08:00:07.449109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575007,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310948,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060892,"statfs":{"total":96636764160,"available":96318353408,"internally_reserved":0,"allocated":1404928,"data_stored":987510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2729999999999999,"15min":1.2310000000000001},"min":{"1min":1.0509999999999999,"5min":0.67600000000000005,"15min":0.59599999999999997},"max":{"1min":1.647,"5min":1.671,"15min":1.792},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.361,"5min":1.2949999999999999,"15min":1.2589999999999999},"min":{"1min":1.1919999999999999,"5min":0.70799999999999996,"15min":0.67200000000000004},"max":{"1min":1.5229999999999999,"5min":1.6850000000000001,"15min":1.845},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.1679999999999999,"15min":1.1459999999999999},"min":{"1min":0.96799999999999997,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":1.492,"5min":1.6259999999999999,"15min":1.944},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.1519999999999999,"15min":1.1399999999999999},"min":{"1min":0.81599999999999995,"5min":0.78600000000000003,"15min":0.75800000000000001},"max":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.8},"last":0.85699999999999998}]},{"osd":2,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.88500000000000001,"5min":0.79300000000000004,"15min":0.751},"max":{"1min":1.52,"5min":1.52,"15min":2.375},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2190000000000001,"15min":1.2370000000000001},"min":{"1min":0.94399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.532,"5min":1.8049999999999999,"15min":1.9059999999999999},"last":1.3080000000000001}]},{"osd":3,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2929999999999999,"15min":1.2949999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.73199999999999998},"max":{"1min":1.5169999999999999,"5min":1.6359999999999999,"15min":2.0579999999999998},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.341,"15min":1.365},"min":{"1min":1.1279999999999999,"5min":0.91500000000000004,"15min":0.90700000000000003},"max":{"1min":1.569,"5min":1.7110000000000001,"15min":2.6720000000000002},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.157,"15min":1.21},"min":{"1min":0.67900000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.554,"5min":1.6859999999999999,"15min":1.966},"last":1.431},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.208,"15min":1.2350000000000001},"min":{"1min":0.76500000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5609999999999999,"5min":1.6950000000000001,"15min":1.9490000000000001},"last":1.671}]},{"osd":5,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.208,"15min":1.2569999999999999},"min":{"1min":0.84899999999999998,"5min":0.79700000000000004,"15min":0.78000000000000003},"max":{"1min":1.653,"5min":1.675,"15min":2.8279999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3149999999999999,"15min":1.363},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.5940000000000001,"5min":1.74,"15min":2.3010000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.391,"15min":1.423},"min":{"1min":0.96599999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.9220000000000002},"last":1.726},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.343,"15min":1.4019999999999999},"min":{"1min":0.90200000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.607,"5min":1.9079999999999999,"15min":3.097},"last":1.5620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302176,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069664,"statfs":{"total":96636764160,"available":96327335936,"internally_reserved":0,"allocated":811008,"data_stored":397642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.48299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4159999999999999,"5min":1.613,"15min":1.665},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1479999999999999,"15min":1.1259999999999999},"min":{"1min":0.76400000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.665,"15min":1.6970000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1339999999999999,"15min":1.1879999999999999},"min":{"1min":0.71099999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.607,"5min":1.607,"15min":1.881},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.1859999999999999,"15min":1.177},"min":{"1min":0.78500000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.542,"5min":1.7470000000000001,"15min":1.7989999999999999},"last":0.92600000000000005}]},{"osd":2,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.234,"15min":1.2330000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7410000000000001,"5min":1.847,"15min":1.847},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2,"15min":1.228},"min":{"1min":0.71199999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.556,"5min":1.7749999999999999,"15min":1.7929999999999999},"last":1.222}]},{"osd":3,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.27,"15min":1.28},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.623,"5min":1.958,"15min":2.0659999999999998},"last":1.127},{"interface":"front","average":{"1min":1.335,"5min":1.29,"15min":1.323},"min":{"1min":0.91500000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6930000000000001,"5min":1.7430000000000001,"15min":2.1760000000000002},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1719999999999999,"15min":1.2090000000000001},"min":{"1min":0.84299999999999997,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5920000000000001,"5min":1.8480000000000001,"15min":2.137},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4079999999999999,"15min":1.4299999999999999},"min":{"1min":0.748,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.746,"5min":1.899,"15min":2.218},"last":1.9119999999999999}]},{"osd":5,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.254,"15min":1.296},"min":{"1min":0.86299999999999999,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.802,"5min":1.9019999999999999,"15min":2.1989999999999998},"last":1.857},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.284,"15min":1.3280000000000001},"min":{"1min":0.997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.639,"5min":1.712,"15min":2.3050000000000002},"last":1.589}]},{"osd":7,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.345,"15min":1.405},"min":{"1min":0.878,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.6990000000000001,"5min":1.956,"15min":2.2029999999999998},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.3799999999999999,"15min":1.4350000000000001},"min":{"1min":1.131,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.2639999999999998},"last":1.7649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":397018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2549999999999999,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.72499999999999998,"15min":0.53000000000000003},"max":{"1min":1.591,"5min":1.653,"15min":2.137},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.302,"15min":1.2809999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.50700000000000001},"max":{"1min":1.4850000000000001,"5min":1.954,"15min":2.3140000000000001},"last":1.534}]},{"osd":1,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.147,"15min":1.1699999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.61699999999999999},"max":{"1min":1.4890000000000001,"5min":1.629,"15min":3.5409999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.78400000000000003,"5min":0.77300000000000002,"15min":0.68899999999999995},"max":{"1min":1.534,"5min":1.6399999999999999,"15min":3.113},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1870000000000001,"15min":1.2350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.66000000000000003},"max":{"1min":1.4330000000000001,"5min":1.702,"15min":3.339},"last":1.264},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.206,"15min":1.2290000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.5189999999999999,"5min":1.6459999999999999,"15min":2.9039999999999999},"last":1.2410000000000001}]},{"osd":3,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.256,"15min":1.3},"min":{"1min":0.94399999999999995,"5min":0.88300000000000001,"15min":0.748},"max":{"1min":1.3799999999999999,"5min":1.778,"15min":3.5870000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.294,"15min":1.3540000000000001},"min":{"1min":0.95499999999999996,"5min":0.76600000000000001,"15min":0.58199999999999996},"max":{"1min":1.5469999999999999,"5min":1.881,"15min":4.3879999999999999},"last":1.474}]},{"osd":4,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.202,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.79800000000000004,"15min":0.52600000000000002},"max":{"1min":1.5920000000000001,"5min":1.6739999999999999,"15min":5.0819999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.28,"15min":1.3480000000000001},"min":{"1min":0.70299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.7070000000000001,"5min":1.8580000000000001,"15min":4.6310000000000002},"last":1.1910000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2969999999999999,"15min":1.377},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.78800000000000003},"max":{"1min":1.4279999999999999,"5min":2.0099999999999998,"15min":3.9649999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.343,"15min":1.4079999999999999},"min":{"1min":0.91200000000000003,"5min":0.81899999999999995,"15min":0.70599999999999996},"max":{"1min":1.571,"5min":1.843,"15min":3.7559999999999998},"last":1.379}]},{"osd":7,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.4059999999999999,"15min":1.47},"min":{"1min":1.0029999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.673,"5min":1.9359999999999999,"15min":4.4539999999999997},"last":1.302},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.456,"15min":1.5129999999999999},"min":{"1min":1.1319999999999999,"5min":0.96899999999999997,"15min":0.67800000000000005},"max":{"1min":1.6020000000000001,"5min":2.0590000000000002,"15min":4.1980000000000004},"last":1.413}]}]},{"osd":4,"up_from":13,"seq":55834575006,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302172,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069668,"statfs":{"total":96636764160,"available":96327340032,"internally_reserved":0,"allocated":806912,"data_stored":397572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.218,"15min":1.218},"min":{"1min":0.83299999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.3180000000000001,"15min":1.304},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.532,"5min":1.7909999999999999,"15min":2.738},"last":1.417}]},{"osd":1,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.165,"15min":1.2030000000000001},"min":{"1min":0.78400000000000003,"5min":0.71899999999999997,"15min":0.70699999999999996},"max":{"1min":1.637,"5min":1.7310000000000001,"15min":2.589},"last":1.175},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.139,"15min":1.145},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54300000000000004},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":2.2589999999999999},"last":1.0429999999999999}]},{"osd":2,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1319999999999999,"15min":1.167},"min":{"1min":0.78300000000000003,"5min":0.76000000000000001,"15min":0.628},"max":{"1min":1.3979999999999999,"5min":1.601,"15min":2.4569999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.224,"15min":1.2430000000000001},"min":{"1min":0.78700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5289999999999999,"5min":1.6299999999999999,"15min":2.8050000000000002},"last":0.95699999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2609999999999999,"15min":1.3100000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4259999999999999,"5min":1.716,"15min":3.048},"last":1.24},{"interface":"front","average":{"1min":1.27,"5min":1.3089999999999999,"15min":1.3720000000000001},"min":{"1min":0.83699999999999997,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.5620000000000001,"5min":1.637,"15min":2.661},"last":1.544}]},{"osd":5,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1839999999999999,"15min":1.2509999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.659,"5min":1.7869999999999999,"15min":3.556},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.1679999999999999,"15min":1.2529999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.625,"15min":3.2599999999999998},"last":1.377}]},{"osd":6,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1659999999999999,"15min":1.296},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.595,"5min":1.595,"15min":2.645},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3500000000000001,"15min":1.409},"min":{"1min":0.70699999999999996,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":1.8360000000000001,"15min":2.923},"last":1.5780000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3140000000000001,"15min":1.411},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.514},{"interface":"front","average":{"1min":1.329,"5min":1.355,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8,"5min":1.8,"15min":2.8460000000000001},"last":1.6619999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302184,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069656,"statfs":{"total":96636764160,"available":96327327744,"internally_reserved":0,"allocated":819200,"data_stored":398576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.85999999999999999,"15min":0.94299999999999995},"min":{"1min":0.51400000000000001,"5min":0.497,"15min":0.45500000000000002},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":5.1079999999999997},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97599999999999998,"15min":1.038},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.39200000000000002},"max":{"1min":1.2909999999999999,"5min":1.835,"15min":6.1900000000000004},"last":0.86299999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93700000000000006,"15min":1.0009999999999999},"min":{"1min":0.628,"5min":0.56499999999999995,"15min":0.45600000000000002},"max":{"1min":1.431,"5min":1.4810000000000001,"15min":4.8879999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.071,"15min":1.1839999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.56999999999999995},"max":{"1min":1.494,"5min":1.673,"15min":8.0830000000000002},"last":1.492}]},{"osd":2,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.1319999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.52800000000000002},"max":{"1min":1.4730000000000001,"5min":1.786,"15min":4.7519999999999998},"last":1.276},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1559999999999999,"15min":1.26},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.65200000000000002},"max":{"1min":1.4710000000000001,"5min":1.704,"15min":5.3330000000000002},"last":0.92900000000000005}]},{"osd":4,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.286,"15min":1.341},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":4.7130000000000001},"last":1.53},{"interface":"front","average":{"1min":1.22,"5min":1.258,"15min":1.3520000000000001},"min":{"1min":0.92800000000000005,"5min":0.91000000000000003,"15min":0.87},"max":{"1min":1.534,"5min":1.8120000000000001,"15min":5.0220000000000002},"last":0.995}]},{"osd":5,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.2829999999999999,"15min":1.393},"min":{"1min":0.95699999999999996,"5min":0.85799999999999998,"15min":0.81799999999999995},"max":{"1min":1.48,"5min":1.609,"15min":4.9889999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.355,"15min":1.5069999999999999},"min":{"1min":1.0620000000000001,"5min":0.99399999999999999,"15min":0.65800000000000003},"max":{"1min":1.5549999999999999,"5min":1.8700000000000001,"15min":8.5299999999999994},"last":1.3700000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4059999999999999,"15min":1.546},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":0.88700000000000001},"max":{"1min":1.621,"5min":2.1280000000000001,"15min":7.2750000000000004},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3879999999999999,"15min":1.448},"min":{"1min":1.087,"5min":0.82499999999999996,"15min":0.79100000000000004},"max":{"1min":1.696,"5min":2.1440000000000001,"15min":4.6550000000000002},"last":1.605}]},{"osd":7,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.6799999999999999},"min":{"1min":1.119,"5min":1.0660000000000001,"15min":0.84799999999999998},"max":{"1min":1.8400000000000001,"5min":2.2509999999999999,"15min":16.081},"last":1.665},{"interface":"front","average":{"1min":1.468,"5min":1.452,"15min":1.6619999999999999},"min":{"1min":1.1599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.833,"5min":1.9159999999999999,"15min":15.285},"last":1.085}]}]},{"osd":2,"up_from":13,"seq":55834575006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":405654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.90400000000000003,"15min":0.95199999999999996},"min":{"1min":0.57999999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.3129999999999999,"5min":1.4039999999999999,"15min":4.1479999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.054,"15min":1.0900000000000001},"min":{"1min":0.748,"5min":0.66200000000000003,"15min":0.52200000000000002},"max":{"1min":1.26,"5min":1.5029999999999999,"15min":4.093},"last":0.98399999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.94999999999999996,"15min":1},"min":{"1min":0.63700000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":3.8700000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0109999999999999,"15min":1.0489999999999999},"min":{"1min":0.66800000000000004,"5min":0.626,"15min":0.59499999999999997},"max":{"1min":1.546,"5min":1.546,"15min":3.9089999999999998},"last":0.79600000000000004}]},{"osd":3,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98299999999999998,"15min":0.95799999999999996},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.395,"5min":1.667,"15min":1.667},"last":1.141},{"interface":"front","average":{"1min":1.125,"5min":1.131,"15min":1.129},"min":{"1min":0.71499999999999997,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.4910000000000001,"5min":1.5920000000000001,"15min":1.7250000000000001},"last":0.95299999999999996}]},{"osd":4,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.321,"15min":1.3620000000000001},"min":{"1min":0.93700000000000006,"5min":0.93000000000000005,"15min":0.79300000000000004},"max":{"1min":1.603,"5min":1.603,"15min":3.9239999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2969999999999999,"15min":1.3600000000000001},"min":{"1min":0.98499999999999999,"5min":0.94999999999999996,"15min":0.82099999999999995},"max":{"1min":1.575,"5min":1.736,"15min":4.0170000000000003},"last":1.1970000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3520000000000001,"15min":1.4039999999999999},"min":{"1min":1.05,"5min":1.0049999999999999,"15min":0.93799999999999994},"max":{"1min":1.6639999999999999,"5min":1.8560000000000001,"15min":3.948},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.26,"15min":1.2689999999999999},"min":{"1min":1.05,"5min":0.84899999999999998,"15min":0.80400000000000005},"max":{"1min":1.583,"5min":1.6930000000000001,"15min":1.879},"last":1.371}]},{"osd":6,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3009999999999999,"15min":1.3120000000000001},"min":{"1min":1.018,"5min":0.92700000000000005,"15min":0.752},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.365,"15min":1.4399999999999999},"min":{"1min":1.1739999999999999,"5min":1.077,"15min":0.85399999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":3.9620000000000002},"last":1.323}]},{"osd":7,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.379,"15min":1.4630000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":0.83699999999999997},"max":{"1min":1.6699999999999999,"5min":1.8109999999999999,"15min":3.9809999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3720000000000001,"15min":1.401},"min":{"1min":0.97399999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.206},"last":1.49}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302176,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069664,"statfs":{"total":96636764160,"available":96327335936,"internally_reserved":0,"allocated":811008,"data_stored":397178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.84599999999999997},"min":{"1min":0.65500000000000003,"5min":0.56399999999999995,"15min":0.40100000000000002},"max":{"1min":1.355,"5min":1.522,"15min":1.532},"last":0.999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.95599999999999996,"15min":0.95499999999999996},"min":{"1min":0.628,"5min":0.59999999999999998,"15min":0.41199999999999998},"max":{"1min":1.224,"5min":1.4950000000000001,"15min":1.9019999999999999},"last":1.157}]},{"osd":2,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.91100000000000003,"15min":0.90900000000000003},"min":{"1min":0.67400000000000004,"5min":0.53100000000000003,"15min":0.50700000000000001},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.5509999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0529999999999999,"15min":1.0509999999999999},"min":{"1min":0.77900000000000003,"5min":0.70299999999999996,"15min":0.46300000000000002},"max":{"1min":1.4730000000000001,"5min":1.6160000000000001,"15min":1.694},"last":0.72999999999999998}]},{"osd":3,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1459999999999999,"15min":1.145},"min":{"1min":0.79100000000000004,"5min":0.69899999999999995,"15min":0.66100000000000003},"max":{"1min":1.6120000000000001,"5min":1.6899999999999999,"15min":1.7909999999999999},"last":1.331},{"interface":"front","average":{"1min":1.087,"5min":1.0509999999999999,"15min":1.0820000000000001},"min":{"1min":0.71499999999999997,"5min":0.51200000000000001,"15min":0.46800000000000003},"max":{"1min":1.4330000000000001,"5min":1.5229999999999999,"15min":1.835},"last":1.2190000000000001}]},{"osd":4,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.179,"15min":1.2030000000000001},"min":{"1min":0.84199999999999997,"5min":0.69899999999999995,"15min":0.67200000000000004},"max":{"1min":1.4359999999999999,"5min":1.7150000000000001,"15min":1.778},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.296,"15min":1.3109999999999999},"min":{"1min":1.0640000000000001,"5min":0.84199999999999997,"15min":0.64400000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":2.2480000000000002},"last":1.1080000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2669999999999999,"15min":1.2909999999999999},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.65100000000000002},"max":{"1min":1.569,"5min":1.77,"15min":2.2200000000000002},"last":1.387},{"interface":"front","average":{"1min":1.276,"5min":1.2729999999999999,"15min":1.3089999999999999},"min":{"1min":1.048,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5029999999999999,"5min":1.671,"15min":1.905},"last":1.4359999999999999}]},{"osd":6,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3740000000000001,"15min":1.403},"min":{"1min":1.129,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.901,"5min":1.901,"15min":3.024},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.375,"15min":1.397},"min":{"1min":1.073,"5min":0.747,"15min":0.747},"max":{"1min":1.706,"5min":2.012,"15min":2.6230000000000002},"last":1.286}]},{"osd":7,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.429,"15min":1.466},"min":{"1min":0.999,"5min":0.88500000000000001,"15min":0.83199999999999996},"max":{"1min":1.8160000000000001,"5min":2.0790000000000002,"15min":3.6080000000000001},"last":1.254},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.365,"15min":1.4419999999999999},"min":{"1min":1.0509999999999999,"5min":0.93799999999999994,"15min":0.88100000000000001},"max":{"1min":1.6659999999999999,"5min":1.829,"15min":4.101},"last":1.5700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":986431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.93000000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.45800000000000002},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91100000000000003,"15min":1.004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.379},"max":{"1min":1.2250000000000001,"5min":1.454,"15min":6.0309999999999997},"last":1.2450000000000001}]},{"osd":2,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.629},"max":{"1min":1.286,"5min":1.335,"15min":1.5800000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.995,"5min":1.028,"15min":1.026},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.61399999999999999},"max":{"1min":1.2509999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.73199999999999998}]},{"osd":3,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.96899999999999997,"15min":0.95199999999999996},"min":{"1min":0.60399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2150000000000001,"5min":1.671,"15min":1.671},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.78600000000000003,"5min":0.77800000000000002,"15min":0.61899999999999999},"max":{"1min":1.4319999999999999,"5min":1.54,"15min":1.6719999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2969999999999999,"15min":1.3200000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.526,"5min":1.671,"15min":2.1669999999999998},"last":1.337},{"interface":"front","average":{"1min":1.306,"5min":1.3129999999999999,"15min":1.3049999999999999},"min":{"1min":0.90100000000000002,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":2.141},"last":1.0700000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.365,"15min":1.3799999999999999},"min":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.77800000000000002},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":2.117},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.363,"15min":1.381},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.80300000000000005},"max":{"1min":1.591,"5min":1.7789999999999999,"15min":2.0859999999999999},"last":1.163}]},{"osd":6,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.276,"15min":1.274},"min":{"1min":0.95499999999999996,"5min":0.91100000000000003,"15min":0.77400000000000002},"max":{"1min":1.903,"5min":1.903,"15min":1.9059999999999999},"last":0.996},{"interface":"front","average":{"1min":1.268,"5min":1.3069999999999999,"15min":1.343},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.649,"5min":1.649,"15min":1.9530000000000001},"last":1.2909999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3149999999999999,"15min":1.3320000000000001},"min":{"1min":1.004,"5min":0.98399999999999999,"15min":0.84799999999999998},"max":{"1min":1.841,"5min":1.841,"15min":2.0539999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":1.095,"5min":0.996,"15min":0.67700000000000005},"max":{"1min":1.716,"5min":1.716,"15min":2.016},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:00:12.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:00:12.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 Processor -- start 2023-12-19T08:00:12.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 -- start start 2023-12-19T08:00:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 0x7fd97015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9700aeba0 0x7fd9700680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700c7190 con 0x7fd9700c15b0 2023-12-19T08:00:12.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700c1b50 con 0x7fd9700aeba0 2023-12-19T08:00:12.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd977f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700c1e30 con 0x7fd97015d320 2023-12-19T08:00:12.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd9754e0700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd9700c15b0 0x7fd97006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:33068/0 (socket says 172.21.15.111:33068) 2023-12-19T08:00:12.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd9754e0700 1 -- 172.21.15.111:0/164043914 learned_addr learned my addr 172.21.15.111:0/164043914 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:12.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.928+0000 7fd975ce1700 1 --2- 172.21.15.111:0/164043914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 0x7fd97015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd9764e2700 1 --2- 172.21.15.111:0/164043914 >> v2:172.21.15.143:3300/0 conn(0x7fd9700aeba0 0x7fd9700680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd974cdf700 1 -- 172.21.15.111:0/164043914 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039212689 0 0) 0x7fd9700c7190 con 0x7fd9700c15b0 2023-12-19T08:00:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd975ce1700 1 -- 172.21.15.111:0/164043914 >> v2:172.21.15.143:3300/0 conn(0x7fd9700aeba0 msgr2=0x7fd9700680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd975ce1700 1 --2- 172.21.15.111:0/164043914 >> v2:172.21.15.143:3300/0 conn(0x7fd9700aeba0 0x7fd9700680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd975ce1700 1 -- 172.21.15.111:0/164043914 >> v1:172.21.15.143:6789/0 conn(0x7fd9700c15b0 legacy=0x7fd97006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd975ce1700 1 -- 172.21.15.111:0/164043914 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700c6e20 con 0x7fd97015d320 2023-12-19T08:00:12.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd975ce1700 1 --2- 172.21.15.111:0/164043914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 0x7fd97015d6f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0098e0 tx=0x7fd96c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf04fceec516084 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd974cdf700 1 -- 172.21.15.111:0/164043914 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00b640 con 0x7fd97015d320 2023-12-19T08:00:12.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd974cdf700 1 -- 172.21.15.111:0/164043914 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00b7a0 con 0x7fd97015d320 2023-12-19T08:00:12.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd974cdf700 1 -- 172.21.15.111:0/164043914 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00bb40 con 0x7fd97015d320 2023-12-19T08:00:12.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 -- 172.21.15.111:0/164043914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 msgr2=0x7fd97015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 --2- 172.21.15.111:0/164043914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 0x7fd97015d6f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0098e0 tx=0x7fd96c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 -- 172.21.15.111:0/164043914 shutdown_connections 2023-12-19T08:00:12.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 --2- 172.21.15.111:0/164043914 >> v2:172.21.15.143:3300/0 conn(0x7fd9700aeba0 0x7fd9700680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 --2- 172.21.15.111:0/164043914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd97015d320 0x7fd97015d6f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 -- 172.21.15.111:0/164043914 >> 172.21.15.111:0/164043914 conn(0x7fd9700c8850 msgr2=0x7fd9700c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:12.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 -- 172.21.15.111:0/164043914 shutdown_connections 2023-12-19T08:00:12.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.932+0000 7fd977f45700 1 -- 172.21.15.111:0/164043914 wait complete. 2023-12-19T08:00:12.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 Processor -- start 2023-12-19T08:00:12.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 -- start start 2023-12-19T08:00:12.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9700aeba0 0x7fd9700c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97015d320 0x7fd9700c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97015d6f0 con 0x7fd9700c15b0 2023-12-19T08:00:12.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97016dc80 con 0x7fd97015d320 2023-12-19T08:00:12.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.936+0000 7fd977f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97016df60 con 0x7fd9700aeba0 2023-12-19T08:00:12.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46186/0 (socket says 172.21.15.111:46186) 2023-12-19T08:00:12.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 -- 172.21.15.111:0/3680953869 learned_addr learned my addr 172.21.15.111:0/3680953869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:12.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9764e2700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97015d320 0x7fd9700c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd975ce1700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9700aeba0 0x7fd9700c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 -- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9700aeba0 msgr2=0x7fd9700c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9700aeba0 0x7fd9700c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 -- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97015d320 msgr2=0x7fd9700c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97015d320 0x7fd9700c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 -- 172.21.15.111:0/3680953869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c009420 con 0x7fd9700c15b0 2023-12-19T08:00:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd9754e0700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c7e0 tx=0x7fd960007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960005b30 con 0x7fd9700c15b0 2023-12-19T08:00:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd960005c90 con 0x7fd9700c15b0 2023-12-19T08:00:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97016e240 con 0x7fd9700c15b0 2023-12-19T08:00:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960007300 con 0x7fd9700c15b0 2023-12-19T08:00:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9700c5ce0 con 0x7fd9700c15b0 2023-12-19T08:00:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.940+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97015d860 con 0x7fd9700c15b0 2023-12-19T08:00:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.944+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd96001b3b0 con 0x7fd9700c15b0 2023-12-19T08:00:12.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.944+0000 7fd965ffb700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 0x7fd954062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:12.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.944+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd96007c1d0 con 0x7fd9700c15b0 2023-12-19T08:00:12.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.944+0000 7fd975ce1700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 0x7fd954062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:12.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.948+0000 7fd975ce1700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 0x7fd954062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0093f0 tx=0x7fd96c017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:12.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:12.948+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd96004c2e0 con 0x7fd9700c15b0 2023-12-19T08:00:13.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.120+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd970071120 con 0x7fd954060670 2023-12-19T08:00:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.136+0000 7fd965ffb700 1 -- 172.21.15.111:0/3680953869 <== mgr.5674 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286445 (secure 0 0 0) 0x7fd970071120 con 0x7fd954060670 2023-12-19T08:00:13.143 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:13.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 msgr2=0x7fd954062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 0x7fd954062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0093f0 tx=0x7fd96c017000 comp rx=0 tx=0).stop 2023-12-19T08:00:13.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 msgr2=0x7fd9700c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c7e0 tx=0x7fd960007df0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 shutdown_connections 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd954060670 0x7fd954062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97015d320 0x7fd9700c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9700c15b0 0x7fd9700c4010 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 --2- 172.21.15.111:0/3680953869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9700aeba0 0x7fd9700c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 >> 172.21.15.111:0/3680953869 conn(0x7fd9700c8850 msgr2=0x7fd97015a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.144+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 shutdown_connections 2023-12-19T08:00:13.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.148+0000 7fd977f45700 1 -- 172.21.15.111:0/3680953869 wait complete. 2023-12-19T08:00:13.148 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:00:13.167 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:00:11.989292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2422672,"kb_used_data":7504,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752552048,"statfs":{"total":773094113280,"available":770613297152,"internally_reserved":0,"allocated":7684096,"data_stored":4367581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567931"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.388311+0000","last_change":"2023-12-19T08:00:08.388311+0000","last_active":"2023-12-19T08:00:08.388311+0000","last_peered":"2023-12-19T08:00:08.388311+0000","last_clean":"2023-12-19T08:00:08.388311+0000","last_became_active":"2023-12-19T08:00:08.387985+0000","last_became_peered":"2023-12-19T08:00:08.387985+0000","last_unstale":"2023-12-19T08:00:08.388311+0000","last_undegraded":"2023-12-19T08:00:08.388311+0000","last_fullsized":"2023-12-19T08:00:08.388311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:41.875185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.404045+0000","last_change":"2023-12-19T08:00:08.404045+0000","last_active":"2023-12-19T08:00:08.404045+0000","last_peered":"2023-12-19T08:00:08.404045+0000","last_clean":"2023-12-19T08:00:08.404045+0000","last_became_active":"2023-12-19T08:00:08.401693+0000","last_became_peered":"2023-12-19T08:00:08.401693+0000","last_unstale":"2023-12-19T08:00:08.404045+0000","last_undegraded":"2023-12-19T08:00:08.404045+0000","last_fullsized":"2023-12-19T08:00:08.404045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394434+0000","last_change":"2023-12-19T08:00:08.394434+0000","last_active":"2023-12-19T08:00:08.394434+0000","last_peered":"2023-12-19T08:00:08.394434+0000","last_clean":"2023-12-19T08:00:08.394434+0000","last_became_active":"2023-12-19T08:00:08.394039+0000","last_became_peered":"2023-12-19T08:00:08.394039+0000","last_unstale":"2023-12-19T08:00:08.394434+0000","last_undegraded":"2023-12-19T08:00:08.394434+0000","last_fullsized":"2023-12-19T08:00:08.394434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:25.172819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.408429+0000","last_change":"2023-12-19T08:00:08.408429+0000","last_active":"2023-12-19T08:00:08.408429+0000","last_peered":"2023-12-19T08:00:08.408429+0000","last_clean":"2023-12-19T08:00:08.408429+0000","last_became_active":"2023-12-19T08:00:08.408160+0000","last_became_peered":"2023-12-19T08:00:08.408160+0000","last_unstale":"2023-12-19T08:00:08.408429+0000","last_undegraded":"2023-12-19T08:00:08.408429+0000","last_fullsized":"2023-12-19T08:00:08.408429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393332+0000","last_change":"2023-12-19T08:00:08.390410+0000","last_active":"2023-12-19T08:00:08.393332+0000","last_peered":"2023-12-19T08:00:08.393332+0000","last_clean":"2023-12-19T08:00:08.393332+0000","last_became_active":"2023-12-19T08:00:08.389866+0000","last_became_peered":"2023-12-19T08:00:08.389866+0000","last_unstale":"2023-12-19T08:00:08.393332+0000","last_undegraded":"2023-12-19T08:00:08.393332+0000","last_fullsized":"2023-12-19T08:00:08.393332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389924+0000","last_change":"2023-12-19T08:00:08.389924+0000","last_active":"2023-12-19T08:00:08.389924+0000","last_peered":"2023-12-19T08:00:08.389924+0000","last_clean":"2023-12-19T08:00:08.389924+0000","last_became_active":"2023-12-19T08:00:08.389634+0000","last_became_peered":"2023-12-19T08:00:08.389634+0000","last_unstale":"2023-12-19T08:00:08.389924+0000","last_undegraded":"2023-12-19T08:00:08.389924+0000","last_fullsized":"2023-12-19T08:00:08.389924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:05.302338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394117+0000","last_change":"2023-12-19T08:00:08.394117+0000","last_active":"2023-12-19T08:00:08.394117+0000","last_peered":"2023-12-19T08:00:08.394117+0000","last_clean":"2023-12-19T08:00:08.394117+0000","last_became_active":"2023-12-19T08:00:08.392982+0000","last_became_peered":"2023-12-19T08:00:08.392982+0000","last_unstale":"2023-12-19T08:00:08.394117+0000","last_undegraded":"2023-12-19T08:00:08.394117+0000","last_fullsized":"2023-12-19T08:00:08.394117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:38:46.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395325+0000","last_change":"2023-12-19T08:00:08.395325+0000","last_active":"2023-12-19T08:00:08.395325+0000","last_peered":"2023-12-19T08:00:08.395325+0000","last_clean":"2023-12-19T08:00:08.395325+0000","last_became_active":"2023-12-19T08:00:08.390961+0000","last_became_peered":"2023-12-19T08:00:08.390961+0000","last_unstale":"2023-12-19T08:00:08.395325+0000","last_undegraded":"2023-12-19T08:00:08.395325+0000","last_fullsized":"2023-12-19T08:00:08.395325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:30.885015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.396464+0000","last_change":"2023-12-19T08:00:08.396464+0000","last_active":"2023-12-19T08:00:08.396464+0000","last_peered":"2023-12-19T08:00:08.396464+0000","last_clean":"2023-12-19T08:00:08.396464+0000","last_became_active":"2023-12-19T08:00:08.396125+0000","last_became_peered":"2023-12-19T08:00:08.396125+0000","last_unstale":"2023-12-19T08:00:08.396464+0000","last_undegraded":"2023-12-19T08:00:08.396464+0000","last_fullsized":"2023-12-19T08:00:08.396464+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:16:45.597315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377184+0000","last_change":"2023-12-19T08:00:08.377184+0000","last_active":"2023-12-19T08:00:08.377184+0000","last_peered":"2023-12-19T08:00:08.377184+0000","last_clean":"2023-12-19T08:00:08.377184+0000","last_became_active":"2023-12-19T08:00:08.376726+0000","last_became_peered":"2023-12-19T08:00:08.376726+0000","last_unstale":"2023-12-19T08:00:08.377184+0000","last_undegraded":"2023-12-19T08:00:08.377184+0000","last_fullsized":"2023-12-19T08:00:08.377184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:55:47.021391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.403820+0000","last_change":"2023-12-19T08:00:08.399160+0000","last_active":"2023-12-19T08:00:08.403820+0000","last_peered":"2023-12-19T08:00:08.403820+0000","last_clean":"2023-12-19T08:00:08.403820+0000","last_became_active":"2023-12-19T08:00:08.398670+0000","last_became_peered":"2023-12-19T08:00:08.398670+0000","last_unstale":"2023-12-19T08:00:08.403820+0000","last_undegraded":"2023-12-19T08:00:08.403820+0000","last_fullsized":"2023-12-19T08:00:08.403820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:02:14.672353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390943+0000","last_change":"2023-12-19T08:00:08.390943+0000","last_active":"2023-12-19T08:00:08.390943+0000","last_peered":"2023-12-19T08:00:08.390943+0000","last_clean":"2023-12-19T08:00:08.390943+0000","last_became_active":"2023-12-19T08:00:08.390582+0000","last_became_peered":"2023-12-19T08:00:08.390582+0000","last_unstale":"2023-12-19T08:00:08.390943+0000","last_undegraded":"2023-12-19T08:00:08.390943+0000","last_fullsized":"2023-12-19T08:00:08.390943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:58.048582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377134+0000","last_change":"2023-12-19T08:00:08.377134+0000","last_active":"2023-12-19T08:00:08.377134+0000","last_peered":"2023-12-19T08:00:08.377134+0000","last_clean":"2023-12-19T08:00:08.377134+0000","last_became_active":"2023-12-19T08:00:08.376735+0000","last_became_peered":"2023-12-19T08:00:08.376735+0000","last_unstale":"2023-12-19T08:00:08.377134+0000","last_undegraded":"2023-12-19T08:00:08.377134+0000","last_fullsized":"2023-12-19T08:00:08.377134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:51:27.280347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398364+0000","last_change":"2023-12-19T08:00:08.398364+0000","last_active":"2023-12-19T08:00:08.398364+0000","last_peered":"2023-12-19T08:00:08.398364+0000","last_clean":"2023-12-19T08:00:08.398364+0000","last_became_active":"2023-12-19T08:00:08.394030+0000","last_became_peered":"2023-12-19T08:00:08.394030+0000","last_unstale":"2023-12-19T08:00:08.398364+0000","last_undegraded":"2023-12-19T08:00:08.398364+0000","last_fullsized":"2023-12-19T08:00:08.398364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:55:47.021391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.399597+0000","last_change":"2023-12-19T08:00:08.399597+0000","last_active":"2023-12-19T08:00:08.399597+0000","last_peered":"2023-12-19T08:00:08.399597+0000","last_clean":"2023-12-19T08:00:08.399597+0000","last_became_active":"2023-12-19T08:00:08.399001+0000","last_became_peered":"2023-12-19T08:00:08.399001+0000","last_unstale":"2023-12-19T08:00:08.399597+0000","last_undegraded":"2023-12-19T08:00:08.399597+0000","last_fullsized":"2023-12-19T08:00:08.399597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:49.428946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.376188+0000","last_change":"2023-12-19T08:00:08.376188+0000","last_active":"2023-12-19T08:00:08.376188+0000","last_peered":"2023-12-19T08:00:08.376188+0000","last_clean":"2023-12-19T08:00:08.376188+0000","last_became_active":"2023-12-19T08:00:08.375718+0000","last_became_peered":"2023-12-19T08:00:08.375718+0000","last_unstale":"2023-12-19T08:00:08.376188+0000","last_undegraded":"2023-12-19T08:00:08.376188+0000","last_fullsized":"2023-12-19T08:00:08.376188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:14.869430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394536+0000","last_change":"2023-12-19T08:00:08.394536+0000","last_active":"2023-12-19T08:00:08.394536+0000","last_peered":"2023-12-19T08:00:08.394536+0000","last_clean":"2023-12-19T08:00:08.394536+0000","last_became_active":"2023-12-19T08:00:08.391293+0000","last_became_peered":"2023-12-19T08:00:08.391293+0000","last_unstale":"2023-12-19T08:00:08.394536+0000","last_undegraded":"2023-12-19T08:00:08.394536+0000","last_fullsized":"2023-12-19T08:00:08.394536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:08:16.612655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.403854+0000","last_change":"2023-12-19T08:00:08.403854+0000","last_active":"2023-12-19T08:00:08.403854+0000","last_peered":"2023-12-19T08:00:08.403854+0000","last_clean":"2023-12-19T08:00:08.403854+0000","last_became_active":"2023-12-19T08:00:08.403317+0000","last_became_peered":"2023-12-19T08:00:08.403317+0000","last_unstale":"2023-12-19T08:00:08.403854+0000","last_undegraded":"2023-12-19T08:00:08.403854+0000","last_fullsized":"2023-12-19T08:00:08.403854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393129+0000","last_change":"2023-12-19T08:00:08.393129+0000","last_active":"2023-12-19T08:00:08.393129+0000","last_peered":"2023-12-19T08:00:08.393129+0000","last_clean":"2023-12-19T08:00:08.393129+0000","last_became_active":"2023-12-19T08:00:08.389863+0000","last_became_peered":"2023-12-19T08:00:08.389863+0000","last_unstale":"2023-12-19T08:00:08.393129+0000","last_undegraded":"2023-12-19T08:00:08.393129+0000","last_fullsized":"2023-12-19T08:00:08.393129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:51:27.280347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400390+0000","last_change":"2023-12-19T08:00:08.396015+0000","last_active":"2023-12-19T08:00:08.400390+0000","last_peered":"2023-12-19T08:00:08.400390+0000","last_clean":"2023-12-19T08:00:08.400390+0000","last_became_active":"2023-12-19T08:00:08.395329+0000","last_became_peered":"2023-12-19T08:00:08.395329+0000","last_unstale":"2023-12-19T08:00:08.400390+0000","last_undegraded":"2023-12-19T08:00:08.400390+0000","last_fullsized":"2023-12-19T08:00:08.400390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:20.013637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.407016+0000","last_change":"2023-12-19T08:00:08.403756+0000","last_active":"2023-12-19T08:00:08.407016+0000","last_peered":"2023-12-19T08:00:08.407016+0000","last_clean":"2023-12-19T08:00:08.407016+0000","last_became_active":"2023-12-19T08:00:08.403391+0000","last_became_peered":"2023-12-19T08:00:08.403391+0000","last_unstale":"2023-12-19T08:00:08.407016+0000","last_undegraded":"2023-12-19T08:00:08.407016+0000","last_fullsized":"2023-12-19T08:00:08.407016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:49:08.695041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390825+0000","last_change":"2023-12-19T08:00:08.390825+0000","last_active":"2023-12-19T08:00:08.390825+0000","last_peered":"2023-12-19T08:00:08.390825+0000","last_clean":"2023-12-19T08:00:08.390825+0000","last_became_active":"2023-12-19T08:00:08.390492+0000","last_became_peered":"2023-12-19T08:00:08.390492+0000","last_unstale":"2023-12-19T08:00:08.390825+0000","last_undegraded":"2023-12-19T08:00:08.390825+0000","last_fullsized":"2023-12-19T08:00:08.390825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:33.299420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398340+0000","last_change":"2023-12-19T08:00:08.395787+0000","last_active":"2023-12-19T08:00:08.398340+0000","last_peered":"2023-12-19T08:00:08.398340+0000","last_clean":"2023-12-19T08:00:08.398340+0000","last_became_active":"2023-12-19T08:00:08.395507+0000","last_became_peered":"2023-12-19T08:00:08.395507+0000","last_unstale":"2023-12-19T08:00:08.398340+0000","last_undegraded":"2023-12-19T08:00:08.398340+0000","last_fullsized":"2023-12-19T08:00:08.398340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:38:48.145515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389950+0000","last_change":"2023-12-19T08:00:08.389950+0000","last_active":"2023-12-19T08:00:08.389950+0000","last_peered":"2023-12-19T08:00:08.389950+0000","last_clean":"2023-12-19T08:00:08.389950+0000","last_became_active":"2023-12-19T08:00:08.388921+0000","last_became_peered":"2023-12-19T08:00:08.388921+0000","last_unstale":"2023-12-19T08:00:08.389950+0000","last_undegraded":"2023-12-19T08:00:08.389950+0000","last_fullsized":"2023-12-19T08:00:08.389950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:15:04.997205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394867+0000","last_change":"2023-12-19T08:00:08.394867+0000","last_active":"2023-12-19T08:00:08.394867+0000","last_peered":"2023-12-19T08:00:08.394867+0000","last_clean":"2023-12-19T08:00:08.394867+0000","last_became_active":"2023-12-19T08:00:08.394538+0000","last_became_peered":"2023-12-19T08:00:08.394538+0000","last_unstale":"2023-12-19T08:00:08.394867+0000","last_undegraded":"2023-12-19T08:00:08.394867+0000","last_fullsized":"2023-12-19T08:00:08.394867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:05:31.537513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.399154+0000","last_change":"2023-12-19T08:00:08.399154+0000","last_active":"2023-12-19T08:00:08.399154+0000","last_peered":"2023-12-19T08:00:08.399154+0000","last_clean":"2023-12-19T08:00:08.399154+0000","last_became_active":"2023-12-19T08:00:08.398881+0000","last_became_peered":"2023-12-19T08:00:08.398881+0000","last_unstale":"2023-12-19T08:00:08.399154+0000","last_undegraded":"2023-12-19T08:00:08.399154+0000","last_fullsized":"2023-12-19T08:00:08.399154+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:02:40.287354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.388407+0000","last_change":"2023-12-19T08:00:08.388407+0000","last_active":"2023-12-19T08:00:08.388407+0000","last_peered":"2023-12-19T08:00:08.388407+0000","last_clean":"2023-12-19T08:00:08.388407+0000","last_became_active":"2023-12-19T08:00:08.388092+0000","last_became_peered":"2023-12-19T08:00:08.388092+0000","last_unstale":"2023-12-19T08:00:08.388407+0000","last_undegraded":"2023-12-19T08:00:08.388407+0000","last_fullsized":"2023-12-19T08:00:08.388407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400410+0000","last_change":"2023-12-19T08:00:08.396837+0000","last_active":"2023-12-19T08:00:08.400410+0000","last_peered":"2023-12-19T08:00:08.400410+0000","last_clean":"2023-12-19T08:00:08.400410+0000","last_became_active":"2023-12-19T08:00:08.396520+0000","last_became_peered":"2023-12-19T08:00:08.396520+0000","last_unstale":"2023-12-19T08:00:08.400410+0000","last_undegraded":"2023-12-19T08:00:08.400410+0000","last_fullsized":"2023-12-19T08:00:08.400410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:03.721761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.385516+0000","last_change":"2023-12-19T08:00:08.385516+0000","last_active":"2023-12-19T08:00:08.385516+0000","last_peered":"2023-12-19T08:00:08.385516+0000","last_clean":"2023-12-19T08:00:08.385516+0000","last_became_active":"2023-12-19T08:00:08.385232+0000","last_became_peered":"2023-12-19T08:00:08.385232+0000","last_unstale":"2023-12-19T08:00:08.385516+0000","last_undegraded":"2023-12-19T08:00:08.385516+0000","last_fullsized":"2023-12-19T08:00:08.385516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:05.302338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405529+0000","last_change":"2023-12-19T08:00:08.405529+0000","last_active":"2023-12-19T08:00:08.405529+0000","last_peered":"2023-12-19T08:00:08.405529+0000","last_clean":"2023-12-19T08:00:08.405529+0000","last_became_active":"2023-12-19T08:00:08.404993+0000","last_became_peered":"2023-12-19T08:00:08.404993+0000","last_unstale":"2023-12-19T08:00:08.405529+0000","last_undegraded":"2023-12-19T08:00:08.405529+0000","last_fullsized":"2023-12-19T08:00:08.405529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:52.813620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.392726+0000","last_change":"2023-12-19T08:00:08.392726+0000","last_active":"2023-12-19T08:00:08.392726+0000","last_peered":"2023-12-19T08:00:08.392726+0000","last_clean":"2023-12-19T08:00:08.392726+0000","last_became_active":"2023-12-19T08:00:08.391785+0000","last_became_peered":"2023-12-19T08:00:08.391785+0000","last_unstale":"2023-12-19T08:00:08.392726+0000","last_undegraded":"2023-12-19T08:00:08.392726+0000","last_fullsized":"2023-12-19T08:00:08.392726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:02.360075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400602+0000","last_change":"2023-12-19T08:00:08.400602+0000","last_active":"2023-12-19T08:00:08.400602+0000","last_peered":"2023-12-19T08:00:08.400602+0000","last_clean":"2023-12-19T08:00:08.400602+0000","last_became_active":"2023-12-19T08:00:08.400174+0000","last_became_peered":"2023-12-19T08:00:08.400174+0000","last_unstale":"2023-12-19T08:00:08.400602+0000","last_undegraded":"2023-12-19T08:00:08.400602+0000","last_fullsized":"2023-12-19T08:00:08.400602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402323+0000","last_change":"2023-12-19T08:00:08.402323+0000","last_active":"2023-12-19T08:00:08.402323+0000","last_peered":"2023-12-19T08:00:08.402323+0000","last_clean":"2023-12-19T08:00:08.402323+0000","last_became_active":"2023-12-19T08:00:08.402027+0000","last_became_peered":"2023-12-19T08:00:08.402027+0000","last_unstale":"2023-12-19T08:00:08.402323+0000","last_undegraded":"2023-12-19T08:00:08.402323+0000","last_fullsized":"2023-12-19T08:00:08.402323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477181+0000","last_change":"2023-12-19T08:00:07.447816+0000","last_active":"2023-12-19T08:00:08.477181+0000","last_peered":"2023-12-19T08:00:08.477181+0000","last_clean":"2023-12-19T08:00:08.477181+0000","last_became_active":"2023-12-19T08:00:07.447496+0000","last_became_peered":"2023-12-19T08:00:07.447496+0000","last_unstale":"2023-12-19T08:00:08.477181+0000","last_undegraded":"2023-12-19T08:00:08.477181+0000","last_fullsized":"2023-12-19T08:00:08.477181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:20.994355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393292+0000","last_change":"2023-12-19T08:00:08.393292+0000","last_active":"2023-12-19T08:00:08.393292+0000","last_peered":"2023-12-19T08:00:08.393292+0000","last_clean":"2023-12-19T08:00:08.393292+0000","last_became_active":"2023-12-19T08:00:08.392373+0000","last_became_peered":"2023-12-19T08:00:08.392373+0000","last_unstale":"2023-12-19T08:00:08.393292+0000","last_undegraded":"2023-12-19T08:00:08.393292+0000","last_fullsized":"2023-12-19T08:00:08.393292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.371137+0000","last_change":"2023-12-19T08:00:07.455865+0000","last_active":"2023-12-19T08:00:08.371137+0000","last_peered":"2023-12-19T08:00:08.371137+0000","last_clean":"2023-12-19T08:00:08.371137+0000","last_became_active":"2023-12-19T08:00:07.451695+0000","last_became_peered":"2023-12-19T08:00:07.451695+0000","last_unstale":"2023-12-19T08:00:08.371137+0000","last_undegraded":"2023-12-19T08:00:08.371137+0000","last_fullsized":"2023-12-19T08:00:08.371137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:56:40.442903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405333+0000","last_change":"2023-12-19T08:00:08.403132+0000","last_active":"2023-12-19T08:00:08.405333+0000","last_peered":"2023-12-19T08:00:08.405333+0000","last_clean":"2023-12-19T08:00:08.405333+0000","last_became_active":"2023-12-19T08:00:08.401187+0000","last_became_peered":"2023-12-19T08:00:08.401187+0000","last_unstale":"2023-12-19T08:00:08.405333+0000","last_undegraded":"2023-12-19T08:00:08.405333+0000","last_fullsized":"2023-12-19T08:00:08.405333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.545605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.471647+0000","last_change":"2023-12-19T08:00:07.450829+0000","last_active":"2023-12-19T08:00:08.471647+0000","last_peered":"2023-12-19T08:00:08.471647+0000","last_clean":"2023-12-19T08:00:08.471647+0000","last_became_active":"2023-12-19T08:00:07.450439+0000","last_became_peered":"2023-12-19T08:00:07.450439+0000","last_unstale":"2023-12-19T08:00:08.471647+0000","last_undegraded":"2023-12-19T08:00:08.471647+0000","last_fullsized":"2023-12-19T08:00:08.471647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.001564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.397835+0000","last_change":"2023-12-19T08:00:08.397835+0000","last_active":"2023-12-19T08:00:08.397835+0000","last_peered":"2023-12-19T08:00:08.397835+0000","last_clean":"2023-12-19T08:00:08.397835+0000","last_became_active":"2023-12-19T08:00:08.397535+0000","last_became_peered":"2023-12-19T08:00:08.397535+0000","last_unstale":"2023-12-19T08:00:08.397835+0000","last_undegraded":"2023-12-19T08:00:08.397835+0000","last_fullsized":"2023-12-19T08:00:08.397835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:06:40.086080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.478851+0000","last_change":"2023-12-19T08:00:07.458733+0000","last_active":"2023-12-19T08:00:08.478851+0000","last_peered":"2023-12-19T08:00:08.478851+0000","last_clean":"2023-12-19T08:00:08.478851+0000","last_became_active":"2023-12-19T08:00:07.458394+0000","last_became_peered":"2023-12-19T08:00:07.458394+0000","last_unstale":"2023-12-19T08:00:08.478851+0000","last_undegraded":"2023-12-19T08:00:08.478851+0000","last_fullsized":"2023-12-19T08:00:08.478851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:02.709687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.401617+0000","last_change":"2023-12-19T08:00:08.390987+0000","last_active":"2023-12-19T08:00:08.401617+0000","last_peered":"2023-12-19T08:00:08.401617+0000","last_clean":"2023-12-19T08:00:08.401617+0000","last_became_active":"2023-12-19T08:00:08.390683+0000","last_became_peered":"2023-12-19T08:00:08.390683+0000","last_unstale":"2023-12-19T08:00:08.401617+0000","last_undegraded":"2023-12-19T08:00:08.401617+0000","last_fullsized":"2023-12-19T08:00:08.401617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.383144+0000","last_change":"2023-12-19T08:00:07.437859+0000","last_active":"2023-12-19T08:00:08.383144+0000","last_peered":"2023-12-19T08:00:08.383144+0000","last_clean":"2023-12-19T08:00:08.383144+0000","last_became_active":"2023-12-19T08:00:07.437563+0000","last_became_peered":"2023-12-19T08:00:07.437563+0000","last_unstale":"2023-12-19T08:00:08.383144+0000","last_undegraded":"2023-12-19T08:00:08.383144+0000","last_fullsized":"2023-12-19T08:00:08.383144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:18.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405691+0000","last_change":"2023-12-19T08:00:08.405691+0000","last_active":"2023-12-19T08:00:08.405691+0000","last_peered":"2023-12-19T08:00:08.405691+0000","last_clean":"2023-12-19T08:00:08.405691+0000","last_became_active":"2023-12-19T08:00:08.405231+0000","last_became_peered":"2023-12-19T08:00:08.405231+0000","last_unstale":"2023-12-19T08:00:08.405691+0000","last_undegraded":"2023-12-19T08:00:08.405691+0000","last_fullsized":"2023-12-19T08:00:08.405691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:06.401570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.476894+0000","last_change":"2023-12-19T08:00:07.449396+0000","last_active":"2023-12-19T08:00:08.476894+0000","last_peered":"2023-12-19T08:00:08.476894+0000","last_clean":"2023-12-19T08:00:08.476894+0000","last_became_active":"2023-12-19T08:00:07.448960+0000","last_became_peered":"2023-12-19T08:00:07.448960+0000","last_unstale":"2023-12-19T08:00:08.476894+0000","last_undegraded":"2023-12-19T08:00:08.476894+0000","last_fullsized":"2023-12-19T08:00:08.476894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:18.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400841+0000","last_change":"2023-12-19T08:00:08.400841+0000","last_active":"2023-12-19T08:00:08.400841+0000","last_peered":"2023-12-19T08:00:08.400841+0000","last_clean":"2023-12-19T08:00:08.400841+0000","last_became_active":"2023-12-19T08:00:08.396840+0000","last_became_peered":"2023-12-19T08:00:08.396840+0000","last_unstale":"2023-12-19T08:00:08.400841+0000","last_undegraded":"2023-12-19T08:00:08.400841+0000","last_fullsized":"2023-12-19T08:00:08.400841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:23:47.277168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.383629+0000","last_change":"2023-12-19T08:00:07.453804+0000","last_active":"2023-12-19T08:00:08.383629+0000","last_peered":"2023-12-19T08:00:08.383629+0000","last_clean":"2023-12-19T08:00:08.383629+0000","last_became_active":"2023-12-19T08:00:07.453231+0000","last_became_peered":"2023-12-19T08:00:07.453231+0000","last_unstale":"2023-12-19T08:00:08.383629+0000","last_undegraded":"2023-12-19T08:00:08.383629+0000","last_fullsized":"2023-12-19T08:00:08.383629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:57:55.773668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402693+0000","last_change":"2023-12-19T08:00:08.402693+0000","last_active":"2023-12-19T08:00:08.402693+0000","last_peered":"2023-12-19T08:00:08.402693+0000","last_clean":"2023-12-19T08:00:08.402693+0000","last_became_active":"2023-12-19T08:00:08.402193+0000","last_became_peered":"2023-12-19T08:00:08.402193+0000","last_unstale":"2023-12-19T08:00:08.402693+0000","last_undegraded":"2023-12-19T08:00:08.402693+0000","last_fullsized":"2023-12-19T08:00:08.402693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.452753+0000","last_change":"2023-12-19T08:00:07.443577+0000","last_active":"2023-12-19T08:00:07.452753+0000","last_peered":"2023-12-19T08:00:07.452753+0000","last_clean":"2023-12-19T08:00:07.452753+0000","last_became_active":"2023-12-19T08:00:07.443280+0000","last_became_peered":"2023-12-19T08:00:07.443280+0000","last_unstale":"2023-12-19T08:00:07.452753+0000","last_undegraded":"2023-12-19T08:00:07.452753+0000","last_fullsized":"2023-12-19T08:00:07.452753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:48:30.544946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405040+0000","last_change":"2023-12-19T08:00:08.396794+0000","last_active":"2023-12-19T08:00:08.405040+0000","last_peered":"2023-12-19T08:00:08.405040+0000","last_clean":"2023-12-19T08:00:08.405040+0000","last_became_active":"2023-12-19T08:00:08.396420+0000","last_became_peered":"2023-12-19T08:00:08.396420+0000","last_unstale":"2023-12-19T08:00:08.405040+0000","last_undegraded":"2023-12-19T08:00:08.405040+0000","last_fullsized":"2023-12-19T08:00:08.405040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.464450+0000","last_change":"2023-12-19T08:00:07.461448+0000","last_active":"2023-12-19T08:00:07.464450+0000","last_peered":"2023-12-19T08:00:07.464450+0000","last_clean":"2023-12-19T08:00:07.464450+0000","last_became_active":"2023-12-19T08:00:07.460406+0000","last_became_peered":"2023-12-19T08:00:07.460406+0000","last_unstale":"2023-12-19T08:00:07.464450+0000","last_undegraded":"2023-12-19T08:00:07.464450+0000","last_fullsized":"2023-12-19T08:00:07.464450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:48:30.544946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394867+0000","last_change":"2023-12-19T08:00:08.394867+0000","last_active":"2023-12-19T08:00:08.394867+0000","last_peered":"2023-12-19T08:00:08.394867+0000","last_clean":"2023-12-19T08:00:08.394867+0000","last_became_active":"2023-12-19T08:00:08.394354+0000","last_became_peered":"2023-12-19T08:00:08.394354+0000","last_unstale":"2023-12-19T08:00:08.394867+0000","last_undegraded":"2023-12-19T08:00:08.394867+0000","last_fullsized":"2023-12-19T08:00:08.394867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:04:06.401570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377982+0000","last_change":"2023-12-19T08:00:07.455332+0000","last_active":"2023-12-19T08:00:08.377982+0000","last_peered":"2023-12-19T08:00:08.377982+0000","last_clean":"2023-12-19T08:00:08.377982+0000","last_became_active":"2023-12-19T08:00:07.455016+0000","last_became_peered":"2023-12-19T08:00:07.455016+0000","last_unstale":"2023-12-19T08:00:08.377982+0000","last_undegraded":"2023-12-19T08:00:08.377982+0000","last_fullsized":"2023-12-19T08:00:08.377982+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:19.001564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.385131+0000","last_change":"2023-12-19T08:00:08.385131+0000","last_active":"2023-12-19T08:00:08.385131+0000","last_peered":"2023-12-19T08:00:08.385131+0000","last_clean":"2023-12-19T08:00:08.385131+0000","last_became_active":"2023-12-19T08:00:08.383224+0000","last_became_peered":"2023-12-19T08:00:08.383224+0000","last_unstale":"2023-12-19T08:00:08.385131+0000","last_undegraded":"2023-12-19T08:00:08.385131+0000","last_fullsized":"2023-12-19T08:00:08.385131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:51.328284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477302+0000","last_change":"2023-12-19T08:00:07.464164+0000","last_active":"2023-12-19T08:00:08.477302+0000","last_peered":"2023-12-19T08:00:08.477302+0000","last_clean":"2023-12-19T08:00:08.477302+0000","last_became_active":"2023-12-19T08:00:07.463904+0000","last_became_peered":"2023-12-19T08:00:07.463904+0000","last_unstale":"2023-12-19T08:00:08.477302+0000","last_undegraded":"2023-12-19T08:00:08.477302+0000","last_fullsized":"2023-12-19T08:00:08.477302+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:14.938405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405006+0000","last_change":"2023-12-19T08:00:08.395867+0000","last_active":"2023-12-19T08:00:08.405006+0000","last_peered":"2023-12-19T08:00:08.405006+0000","last_clean":"2023-12-19T08:00:08.405006+0000","last_became_active":"2023-12-19T08:00:08.395449+0000","last_became_peered":"2023-12-19T08:00:08.395449+0000","last_unstale":"2023-12-19T08:00:08.405006+0000","last_undegraded":"2023-12-19T08:00:08.405006+0000","last_fullsized":"2023-12-19T08:00:08.405006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:43.799061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.475523+0000","last_change":"2023-12-19T08:00:07.448669+0000","last_active":"2023-12-19T08:00:08.475523+0000","last_peered":"2023-12-19T08:00:08.475523+0000","last_clean":"2023-12-19T08:00:08.475523+0000","last_became_active":"2023-12-19T08:00:07.448390+0000","last_became_peered":"2023-12-19T08:00:07.448390+0000","last_unstale":"2023-12-19T08:00:08.475523+0000","last_undegraded":"2023-12-19T08:00:08.475523+0000","last_fullsized":"2023-12-19T08:00:08.475523+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:18:47.869632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394490+0000","last_change":"2023-12-19T08:00:08.394490+0000","last_active":"2023-12-19T08:00:08.394490+0000","last_peered":"2023-12-19T08:00:08.394490+0000","last_clean":"2023-12-19T08:00:08.394490+0000","last_became_active":"2023-12-19T08:00:08.392607+0000","last_became_peered":"2023-12-19T08:00:08.392607+0000","last_unstale":"2023-12-19T08:00:08.394490+0000","last_undegraded":"2023-12-19T08:00:08.394490+0000","last_fullsized":"2023-12-19T08:00:08.394490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.474618+0000","last_change":"2023-12-19T08:00:07.451368+0000","last_active":"2023-12-19T08:00:08.474618+0000","last_peered":"2023-12-19T08:00:08.474618+0000","last_clean":"2023-12-19T08:00:08.474618+0000","last_became_active":"2023-12-19T08:00:07.451053+0000","last_became_peered":"2023-12-19T08:00:07.451053+0000","last_unstale":"2023-12-19T08:00:08.474618+0000","last_undegraded":"2023-12-19T08:00:08.474618+0000","last_fullsized":"2023-12-19T08:00:08.474618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:26.770230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393692+0000","last_change":"2023-12-19T08:00:08.393692+0000","last_active":"2023-12-19T08:00:08.393692+0000","last_peered":"2023-12-19T08:00:08.393692+0000","last_clean":"2023-12-19T08:00:08.393692+0000","last_became_active":"2023-12-19T08:00:08.393410+0000","last_became_peered":"2023-12-19T08:00:08.393410+0000","last_unstale":"2023-12-19T08:00:08.393692+0000","last_undegraded":"2023-12-19T08:00:08.393692+0000","last_fullsized":"2023-12-19T08:00:08.393692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:59:53.453893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384011+0000","last_change":"2023-12-19T08:00:07.445415+0000","last_active":"2023-12-19T08:00:08.384011+0000","last_peered":"2023-12-19T08:00:08.384011+0000","last_clean":"2023-12-19T08:00:08.384011+0000","last_became_active":"2023-12-19T08:00:07.445093+0000","last_became_peered":"2023-12-19T08:00:07.445093+0000","last_unstale":"2023-12-19T08:00:08.384011+0000","last_undegraded":"2023-12-19T08:00:08.384011+0000","last_fullsized":"2023-12-19T08:00:08.384011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:08.608453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.396901+0000","last_change":"2023-12-19T08:00:08.396901+0000","last_active":"2023-12-19T08:00:08.396901+0000","last_peered":"2023-12-19T08:00:08.396901+0000","last_clean":"2023-12-19T08:00:08.396901+0000","last_became_active":"2023-12-19T08:00:08.395649+0000","last_became_peered":"2023-12-19T08:00:08.395649+0000","last_unstale":"2023-12-19T08:00:08.396901+0000","last_undegraded":"2023-12-19T08:00:08.396901+0000","last_fullsized":"2023-12-19T08:00:08.396901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:05:53.028508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.462268+0000","last_change":"2023-12-19T08:00:07.459719+0000","last_active":"2023-12-19T08:00:07.462268+0000","last_peered":"2023-12-19T08:00:07.462268+0000","last_clean":"2023-12-19T08:00:07.462268+0000","last_became_active":"2023-12-19T08:00:07.459427+0000","last_became_peered":"2023-12-19T08:00:07.459427+0000","last_unstale":"2023-12-19T08:00:07.462268+0000","last_undegraded":"2023-12-19T08:00:07.462268+0000","last_fullsized":"2023-12-19T08:00:07.462268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:18.014987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.391559+0000","last_change":"2023-12-19T08:00:08.391559+0000","last_active":"2023-12-19T08:00:08.391559+0000","last_peered":"2023-12-19T08:00:08.391559+0000","last_clean":"2023-12-19T08:00:08.391559+0000","last_became_active":"2023-12-19T08:00:08.390919+0000","last_became_peered":"2023-12-19T08:00:08.390919+0000","last_unstale":"2023-12-19T08:00:08.391559+0000","last_undegraded":"2023-12-19T08:00:08.391559+0000","last_fullsized":"2023-12-19T08:00:08.391559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:56:54.670456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.475696+0000","last_change":"2023-12-19T08:00:07.456186+0000","last_active":"2023-12-19T08:00:08.475696+0000","last_peered":"2023-12-19T08:00:08.475696+0000","last_clean":"2023-12-19T08:00:08.475696+0000","last_became_active":"2023-12-19T08:00:07.455879+0000","last_became_peered":"2023-12-19T08:00:07.455879+0000","last_unstale":"2023-12-19T08:00:08.475696+0000","last_undegraded":"2023-12-19T08:00:08.475696+0000","last_fullsized":"2023-12-19T08:00:08.475696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:51:05.852256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395276+0000","last_change":"2023-12-19T08:00:08.395276+0000","last_active":"2023-12-19T08:00:08.395276+0000","last_peered":"2023-12-19T08:00:08.395276+0000","last_clean":"2023-12-19T08:00:08.395276+0000","last_became_active":"2023-12-19T08:00:08.390538+0000","last_became_peered":"2023-12-19T08:00:08.390538+0000","last_unstale":"2023-12-19T08:00:08.395276+0000","last_undegraded":"2023-12-19T08:00:08.395276+0000","last_fullsized":"2023-12-19T08:00:08.395276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:06:47.296674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384004+0000","last_change":"2023-12-19T08:00:07.456833+0000","last_active":"2023-12-19T08:00:08.384004+0000","last_peered":"2023-12-19T08:00:08.384004+0000","last_clean":"2023-12-19T08:00:08.384004+0000","last_became_active":"2023-12-19T08:00:07.456547+0000","last_became_peered":"2023-12-19T08:00:07.456547+0000","last_unstale":"2023-12-19T08:00:08.384004+0000","last_undegraded":"2023-12-19T08:00:08.384004+0000","last_fullsized":"2023-12-19T08:00:08.384004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:36.901880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386947+0000","last_change":"2023-12-19T08:00:08.386947+0000","last_active":"2023-12-19T08:00:08.386947+0000","last_peered":"2023-12-19T08:00:08.386947+0000","last_clean":"2023-12-19T08:00:08.386947+0000","last_became_active":"2023-12-19T08:00:08.386615+0000","last_became_peered":"2023-12-19T08:00:08.386615+0000","last_unstale":"2023-12-19T08:00:08.386947+0000","last_undegraded":"2023-12-19T08:00:08.386947+0000","last_fullsized":"2023-12-19T08:00:08.386947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:57:42.828044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.474519+0000","last_change":"2023-12-19T08:00:07.450210+0000","last_active":"2023-12-19T08:00:08.474519+0000","last_peered":"2023-12-19T08:00:08.474519+0000","last_clean":"2023-12-19T08:00:08.474519+0000","last_became_active":"2023-12-19T08:00:07.449951+0000","last_became_peered":"2023-12-19T08:00:07.449951+0000","last_unstale":"2023-12-19T08:00:08.474519+0000","last_undegraded":"2023-12-19T08:00:08.474519+0000","last_fullsized":"2023-12-19T08:00:08.474519+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:25:22.223342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402214+0000","last_change":"2023-12-19T08:00:08.391873+0000","last_active":"2023-12-19T08:00:08.402214+0000","last_peered":"2023-12-19T08:00:08.402214+0000","last_clean":"2023-12-19T08:00:08.402214+0000","last_became_active":"2023-12-19T08:00:08.391578+0000","last_became_peered":"2023-12-19T08:00:08.391578+0000","last_unstale":"2023-12-19T08:00:08.402214+0000","last_undegraded":"2023-12-19T08:00:08.402214+0000","last_fullsized":"2023-12-19T08:00:08.402214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:35.639270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477997+0000","last_change":"2023-12-19T08:00:07.458199+0000","last_active":"2023-12-19T08:00:08.477997+0000","last_peered":"2023-12-19T08:00:08.477997+0000","last_clean":"2023-12-19T08:00:08.477997+0000","last_became_active":"2023-12-19T08:00:07.457542+0000","last_became_peered":"2023-12-19T08:00:07.457542+0000","last_unstale":"2023-12-19T08:00:08.477997+0000","last_undegraded":"2023-12-19T08:00:08.477997+0000","last_fullsized":"2023-12-19T08:00:08.477997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:42:20.373572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394895+0000","last_change":"2023-12-19T08:00:08.394895+0000","last_active":"2023-12-19T08:00:08.394895+0000","last_peered":"2023-12-19T08:00:08.394895+0000","last_clean":"2023-12-19T08:00:08.394895+0000","last_became_active":"2023-12-19T08:00:08.394511+0000","last_became_peered":"2023-12-19T08:00:08.394511+0000","last_unstale":"2023-12-19T08:00:08.394895+0000","last_undegraded":"2023-12-19T08:00:08.394895+0000","last_fullsized":"2023-12-19T08:00:08.394895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:43:02.360075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386973+0000","last_change":"2023-12-19T08:00:07.456049+0000","last_active":"2023-12-19T08:00:08.386973+0000","last_peered":"2023-12-19T08:00:08.386973+0000","last_clean":"2023-12-19T08:00:08.386973+0000","last_became_active":"2023-12-19T08:00:07.455706+0000","last_became_peered":"2023-12-19T08:00:07.455706+0000","last_unstale":"2023-12-19T08:00:08.386973+0000","last_undegraded":"2023-12-19T08:00:08.386973+0000","last_fullsized":"2023-12-19T08:00:08.386973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:00:03.457275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.394844+0000","last_change":"2023-12-19T08:00:08.394844+0000","last_active":"2023-12-19T08:00:08.394844+0000","last_peered":"2023-12-19T08:00:08.394844+0000","last_clean":"2023-12-19T08:00:08.394844+0000","last_became_active":"2023-12-19T08:00:08.394453+0000","last_became_peered":"2023-12-19T08:00:08.394453+0000","last_unstale":"2023-12-19T08:00:08.394844+0000","last_undegraded":"2023-12-19T08:00:08.394844+0000","last_fullsized":"2023-12-19T08:00:08.394844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:06:52.813620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.473661+0000","last_change":"2023-12-19T08:00:07.447562+0000","last_active":"2023-12-19T08:00:08.473661+0000","last_peered":"2023-12-19T08:00:08.473661+0000","last_clean":"2023-12-19T08:00:08.473661+0000","last_became_active":"2023-12-19T08:00:07.447236+0000","last_became_peered":"2023-12-19T08:00:07.447236+0000","last_unstale":"2023-12-19T08:00:08.473661+0000","last_undegraded":"2023-12-19T08:00:08.473661+0000","last_fullsized":"2023-12-19T08:00:08.473661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:27:18.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.395131+0000","last_change":"2023-12-19T08:00:08.393818+0000","last_active":"2023-12-19T08:00:08.395131+0000","last_peered":"2023-12-19T08:00:08.395131+0000","last_clean":"2023-12-19T08:00:08.395131+0000","last_became_active":"2023-12-19T08:00:08.393451+0000","last_became_peered":"2023-12-19T08:00:08.393451+0000","last_unstale":"2023-12-19T08:00:08.395131+0000","last_undegraded":"2023-12-19T08:00:08.395131+0000","last_fullsized":"2023-12-19T08:00:08.395131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:37.913953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.455550+0000","last_change":"2023-12-19T08:00:07.447387+0000","last_active":"2023-12-19T08:00:07.455550+0000","last_peered":"2023-12-19T08:00:07.455550+0000","last_clean":"2023-12-19T08:00:07.455550+0000","last_became_active":"2023-12-19T08:00:07.447067+0000","last_became_peered":"2023-12-19T08:00:07.447067+0000","last_unstale":"2023-12-19T08:00:07.455550+0000","last_undegraded":"2023-12-19T08:00:07.455550+0000","last_fullsized":"2023-12-19T08:00:07.455550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:33:24.995268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.384694+0000","last_change":"2023-12-19T08:00:08.384694+0000","last_active":"2023-12-19T08:00:08.384694+0000","last_peered":"2023-12-19T08:00:08.384694+0000","last_clean":"2023-12-19T08:00:08.384694+0000","last_became_active":"2023-12-19T08:00:08.384193+0000","last_became_peered":"2023-12-19T08:00:08.384193+0000","last_unstale":"2023-12-19T08:00:08.384694+0000","last_undegraded":"2023-12-19T08:00:08.384694+0000","last_fullsized":"2023-12-19T08:00:08.384694+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.459646+0000","last_change":"2023-12-19T08:00:07.457517+0000","last_active":"2023-12-19T08:00:07.459646+0000","last_peered":"2023-12-19T08:00:07.459646+0000","last_clean":"2023-12-19T08:00:07.459646+0000","last_became_active":"2023-12-19T08:00:07.457244+0000","last_became_peered":"2023-12-19T08:00:07.457244+0000","last_unstale":"2023-12-19T08:00:07.459646+0000","last_undegraded":"2023-12-19T08:00:07.459646+0000","last_fullsized":"2023-12-19T08:00:07.459646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:55:50.171010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.401203+0000","last_change":"2023-12-19T08:00:08.401203+0000","last_active":"2023-12-19T08:00:08.401203+0000","last_peered":"2023-12-19T08:00:08.401203+0000","last_clean":"2023-12-19T08:00:08.401203+0000","last_became_active":"2023-12-19T08:00:08.400859+0000","last_became_peered":"2023-12-19T08:00:08.400859+0000","last_unstale":"2023-12-19T08:00:08.401203+0000","last_undegraded":"2023-12-19T08:00:08.401203+0000","last_fullsized":"2023-12-19T08:00:08.401203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:38:46.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.386639+0000","last_change":"2023-12-19T08:00:07.449435+0000","last_active":"2023-12-19T08:00:08.386639+0000","last_peered":"2023-12-19T08:00:08.386639+0000","last_clean":"2023-12-19T08:00:08.386639+0000","last_became_active":"2023-12-19T08:00:07.449153+0000","last_became_peered":"2023-12-19T08:00:07.449153+0000","last_unstale":"2023-12-19T08:00:08.386639+0000","last_undegraded":"2023-12-19T08:00:08.386639+0000","last_fullsized":"2023-12-19T08:00:08.386639+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:26:48.499202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.398312+0000","last_change":"2023-12-19T08:00:08.398312+0000","last_active":"2023-12-19T08:00:08.398312+0000","last_peered":"2023-12-19T08:00:08.398312+0000","last_clean":"2023-12-19T08:00:08.398312+0000","last_became_active":"2023-12-19T08:00:08.398013+0000","last_became_peered":"2023-12-19T08:00:08.398013+0000","last_unstale":"2023-12-19T08:00:08.398312+0000","last_undegraded":"2023-12-19T08:00:08.398312+0000","last_fullsized":"2023-12-19T08:00:08.398312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:51.244957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.477345+0000","last_change":"2023-12-19T08:00:07.458069+0000","last_active":"2023-12-19T08:00:08.477345+0000","last_peered":"2023-12-19T08:00:08.477345+0000","last_clean":"2023-12-19T08:00:08.477345+0000","last_became_active":"2023-12-19T08:00:07.457730+0000","last_became_peered":"2023-12-19T08:00:07.457730+0000","last_unstale":"2023-12-19T08:00:08.477345+0000","last_undegraded":"2023-12-19T08:00:08.477345+0000","last_fullsized":"2023-12-19T08:00:08.477345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:22.088551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.393375+0000","last_change":"2023-12-19T08:00:08.393375+0000","last_active":"2023-12-19T08:00:08.393375+0000","last_peered":"2023-12-19T08:00:08.393375+0000","last_clean":"2023-12-19T08:00:08.393375+0000","last_became_active":"2023-12-19T08:00:08.392652+0000","last_became_peered":"2023-12-19T08:00:08.392652+0000","last_unstale":"2023-12-19T08:00:08.393375+0000","last_undegraded":"2023-12-19T08:00:08.393375+0000","last_fullsized":"2023-12-19T08:00:08.393375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:59:53.453893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.459266+0000","last_change":"2023-12-19T08:00:07.454277+0000","last_active":"2023-12-19T08:00:07.459266+0000","last_peered":"2023-12-19T08:00:07.459266+0000","last_clean":"2023-12-19T08:00:07.459266+0000","last_became_active":"2023-12-19T08:00:07.453986+0000","last_became_peered":"2023-12-19T08:00:07.453986+0000","last_unstale":"2023-12-19T08:00:07.459266+0000","last_undegraded":"2023-12-19T08:00:07.459266+0000","last_fullsized":"2023-12-19T08:00:07.459266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:26.770230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.392245+0000","last_change":"2023-12-19T08:00:08.392245+0000","last_active":"2023-12-19T08:00:08.392245+0000","last_peered":"2023-12-19T08:00:08.392245+0000","last_clean":"2023-12-19T08:00:08.392245+0000","last_became_active":"2023-12-19T08:00:08.391847+0000","last_became_peered":"2023-12-19T08:00:08.391847+0000","last_unstale":"2023-12-19T08:00:08.392245+0000","last_undegraded":"2023-12-19T08:00:08.392245+0000","last_fullsized":"2023-12-19T08:00:08.392245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:29:11.403091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456226+0000","last_change":"2023-12-19T08:00:07.452077+0000","last_active":"2023-12-19T08:00:07.456226+0000","last_peered":"2023-12-19T08:00:07.456226+0000","last_clean":"2023-12-19T08:00:07.456226+0000","last_became_active":"2023-12-19T08:00:07.451759+0000","last_became_peered":"2023-12-19T08:00:07.451759+0000","last_unstale":"2023-12-19T08:00:07.456226+0000","last_undegraded":"2023-12-19T08:00:07.456226+0000","last_fullsized":"2023-12-19T08:00:07.456226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:47:56.527292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.400660+0000","last_change":"2023-12-19T08:00:08.400660+0000","last_active":"2023-12-19T08:00:08.400660+0000","last_peered":"2023-12-19T08:00:08.400660+0000","last_clean":"2023-12-19T08:00:08.400660+0000","last_became_active":"2023-12-19T08:00:08.400296+0000","last_became_peered":"2023-12-19T08:00:08.400296+0000","last_unstale":"2023-12-19T08:00:08.400660+0000","last_undegraded":"2023-12-19T08:00:08.400660+0000","last_fullsized":"2023-12-19T08:00:08.400660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:14.869430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456013+0000","last_change":"2023-12-19T08:00:07.451665+0000","last_active":"2023-12-19T08:00:07.456013+0000","last_peered":"2023-12-19T08:00:07.456013+0000","last_clean":"2023-12-19T08:00:07.456013+0000","last_became_active":"2023-12-19T08:00:07.451357+0000","last_became_peered":"2023-12-19T08:00:07.451357+0000","last_unstale":"2023-12-19T08:00:07.456013+0000","last_undegraded":"2023-12-19T08:00:07.456013+0000","last_fullsized":"2023-12-19T08:00:07.456013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:57:57.248649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.390216+0000","last_change":"2023-12-19T08:00:08.390216+0000","last_active":"2023-12-19T08:00:08.390216+0000","last_peered":"2023-12-19T08:00:08.390216+0000","last_clean":"2023-12-19T08:00:08.390216+0000","last_became_active":"2023-12-19T08:00:08.389934+0000","last_became_peered":"2023-12-19T08:00:08.389934+0000","last_unstale":"2023-12-19T08:00:08.390216+0000","last_undegraded":"2023-12-19T08:00:08.390216+0000","last_fullsized":"2023-12-19T08:00:08.390216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:18:51.049175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.456066+0000","last_change":"2023-12-19T08:00:07.451675+0000","last_active":"2023-12-19T08:00:07.456066+0000","last_peered":"2023-12-19T08:00:07.456066+0000","last_clean":"2023-12-19T08:00:07.456066+0000","last_became_active":"2023-12-19T08:00:07.451394+0000","last_became_peered":"2023-12-19T08:00:07.451394+0000","last_unstale":"2023-12-19T08:00:07.456066+0000","last_undegraded":"2023-12-19T08:00:07.456066+0000","last_fullsized":"2023-12-19T08:00:07.456066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:22.910802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.389507+0000","last_change":"2023-12-19T08:00:08.389507+0000","last_active":"2023-12-19T08:00:08.389507+0000","last_peered":"2023-12-19T08:00:08.389507+0000","last_clean":"2023-12-19T08:00:08.389507+0000","last_became_active":"2023-12-19T08:00:08.389198+0000","last_became_peered":"2023-12-19T08:00:08.389198+0000","last_unstale":"2023-12-19T08:00:08.389507+0000","last_undegraded":"2023-12-19T08:00:08.389507+0000","last_fullsized":"2023-12-19T08:00:08.389507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:07.392084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.377713+0000","last_change":"2023-12-19T08:00:07.462259+0000","last_active":"2023-12-19T08:00:08.377713+0000","last_peered":"2023-12-19T08:00:08.377713+0000","last_clean":"2023-12-19T08:00:08.377713+0000","last_became_active":"2023-12-19T08:00:07.461447+0000","last_became_peered":"2023-12-19T08:00:07.461447+0000","last_unstale":"2023-12-19T08:00:08.377713+0000","last_undegraded":"2023-12-19T08:00:08.377713+0000","last_fullsized":"2023-12-19T08:00:08.377713+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:49:07.682968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":8,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.402323+0000","last_change":"2023-12-19T08:00:08.402323+0000","last_active":"2023-12-19T08:00:08.402323+0000","last_peered":"2023-12-19T08:00:08.402323+0000","last_clean":"2023-12-19T08:00:08.402323+0000","last_became_active":"2023-12-19T08:00:08.401981+0000","last_became_peered":"2023-12-19T08:00:08.401981+0000","last_unstale":"2023-12-19T08:00:08.402323+0000","last_undegraded":"2023-12-19T08:00:08.402323+0000","last_fullsized":"2023-12-19T08:00:08.402323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:31:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.461135+0000","last_change":"2023-12-19T08:00:07.457828+0000","last_active":"2023-12-19T08:00:07.461135+0000","last_peered":"2023-12-19T08:00:07.461135+0000","last_clean":"2023-12-19T08:00:07.461135+0000","last_became_active":"2023-12-19T08:00:07.457521+0000","last_became_peered":"2023-12-19T08:00:07.457521+0000","last_unstale":"2023-12-19T08:00:07.461135+0000","last_undegraded":"2023-12-19T08:00:07.461135+0000","last_fullsized":"2023-12-19T08:00:07.461135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:36.901880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-12-19T08:00:08.405631+0000","last_change":"2023-12-19T08:00:08.400553+0000","last_active":"2023-12-19T08:00:08.405631+0000","last_peered":"2023-12-19T08:00:08.405631+0000","last_clean":"2023-12-19T08:00:08.405631+0000","last_became_active":"2023-12-19T08:00:08.396748+0000","last_became_peered":"2023-12-19T08:00:08.396748+0000","last_unstale":"2023-12-19T08:00:08.405631+0000","last_undegraded":"2023-12-19T08:00:08.405631+0000","last_fullsized":"2023-12-19T08:00:08.405631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:07.411590+0000","last_clean_scrub_stamp":"2023-12-19T08:00:07.411590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:56:41.454976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.455225+0000","last_change":"2023-12-19T08:00:07.453202+0000","last_active":"2023-12-19T08:00:07.455225+0000","last_peered":"2023-12-19T08:00:07.455225+0000","last_clean":"2023-12-19T08:00:07.455225+0000","last_became_active":"2023-12-19T08:00:07.452929+0000","last_became_peered":"2023-12-19T08:00:07.452929+0000","last_unstale":"2023-12-19T08:00:07.455225+0000","last_undegraded":"2023-12-19T08:00:07.455225+0000","last_fullsized":"2023-12-19T08:00:07.455225+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:00:06.399517+0000","last_clean_scrub_stamp":"2023-12-19T08:00:06.399517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:14.938405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"56'139","reported_seq":281,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-12-19T08:00:07.449109+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:00:07.449109+0000","last_peered":"2023-12-19T08:00:07.449109+0000","last_clean":"2023-12-19T08:00:07.449109+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:00:07.449109+0000","last_undegraded":"2023-12-19T08:00:07.449109+0000","last_fullsized":"2023-12-19T08:00:07.449109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575007,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310948,"kb_used_data":1372,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060892,"statfs":{"total":96636764160,"available":96318353408,"internally_reserved":0,"allocated":1404928,"data_stored":987510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2729999999999999,"15min":1.2310000000000001},"min":{"1min":1.0509999999999999,"5min":0.67600000000000005,"15min":0.59599999999999997},"max":{"1min":1.647,"5min":1.671,"15min":1.792},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.361,"5min":1.2949999999999999,"15min":1.2589999999999999},"min":{"1min":1.1919999999999999,"5min":0.70799999999999996,"15min":0.67200000000000004},"max":{"1min":1.5229999999999999,"5min":1.6850000000000001,"15min":1.845},"last":1.2210000000000001}]},{"osd":1,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.1679999999999999,"15min":1.1459999999999999},"min":{"1min":0.96799999999999997,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":1.492,"5min":1.6259999999999999,"15min":1.944},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.1519999999999999,"15min":1.1399999999999999},"min":{"1min":0.81599999999999995,"5min":0.78600000000000003,"15min":0.75800000000000001},"max":{"1min":1.4470000000000001,"5min":1.4970000000000001,"15min":1.8},"last":0.85699999999999998}]},{"osd":2,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1699999999999999,"15min":1.1739999999999999},"min":{"1min":0.88500000000000001,"5min":0.79300000000000004,"15min":0.751},"max":{"1min":1.52,"5min":1.52,"15min":2.375},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2190000000000001,"15min":1.2370000000000001},"min":{"1min":0.94399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.532,"5min":1.8049999999999999,"15min":1.9059999999999999},"last":1.3080000000000001}]},{"osd":3,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.2929999999999999,"15min":1.2949999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.73199999999999998},"max":{"1min":1.5169999999999999,"5min":1.6359999999999999,"15min":2.0579999999999998},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.341,"15min":1.365},"min":{"1min":1.1279999999999999,"5min":0.91500000000000004,"15min":0.90700000000000003},"max":{"1min":1.569,"5min":1.7110000000000001,"15min":2.6720000000000002},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.157,"15min":1.21},"min":{"1min":0.67900000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.554,"5min":1.6859999999999999,"15min":1.966},"last":1.431},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.208,"15min":1.2350000000000001},"min":{"1min":0.76500000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5609999999999999,"5min":1.6950000000000001,"15min":1.9490000000000001},"last":1.671}]},{"osd":5,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.208,"15min":1.2569999999999999},"min":{"1min":0.84899999999999998,"5min":0.79700000000000004,"15min":0.78000000000000003},"max":{"1min":1.653,"5min":1.675,"15min":2.8279999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3149999999999999,"15min":1.363},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.5940000000000001,"5min":1.74,"15min":2.3010000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.391,"15min":1.423},"min":{"1min":0.96599999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.9220000000000002},"last":1.726},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.343,"15min":1.4019999999999999},"min":{"1min":0.90200000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.607,"5min":1.9079999999999999,"15min":3.097},"last":1.5620000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302176,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069664,"statfs":{"total":96636764160,"available":96327335936,"internally_reserved":0,"allocated":811008,"data_stored":397642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.48299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4159999999999999,"5min":1.613,"15min":1.665},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1479999999999999,"15min":1.1259999999999999},"min":{"1min":0.76400000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.665,"15min":1.6970000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1339999999999999,"15min":1.1879999999999999},"min":{"1min":0.71099999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.607,"5min":1.607,"15min":1.881},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.1859999999999999,"15min":1.177},"min":{"1min":0.78500000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.542,"5min":1.7470000000000001,"15min":1.7989999999999999},"last":0.92600000000000005}]},{"osd":2,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.234,"15min":1.2330000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7410000000000001,"5min":1.847,"15min":1.847},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2,"15min":1.228},"min":{"1min":0.71199999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.556,"5min":1.7749999999999999,"15min":1.7929999999999999},"last":1.222}]},{"osd":3,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.27,"15min":1.28},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.623,"5min":1.958,"15min":2.0659999999999998},"last":1.127},{"interface":"front","average":{"1min":1.335,"5min":1.29,"15min":1.323},"min":{"1min":0.91500000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6930000000000001,"5min":1.7430000000000001,"15min":2.1760000000000002},"last":1.5189999999999999}]},{"osd":4,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1719999999999999,"15min":1.2090000000000001},"min":{"1min":0.84299999999999997,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5920000000000001,"5min":1.8480000000000001,"15min":2.137},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4079999999999999,"15min":1.4299999999999999},"min":{"1min":0.748,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.746,"5min":1.899,"15min":2.218},"last":1.9119999999999999}]},{"osd":5,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.254,"15min":1.296},"min":{"1min":0.86299999999999999,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.802,"5min":1.9019999999999999,"15min":2.1989999999999998},"last":1.857},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.284,"15min":1.3280000000000001},"min":{"1min":0.997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.639,"5min":1.712,"15min":2.3050000000000002},"last":1.589}]},{"osd":7,"last update":"Tue Dec 19 07:59:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.345,"15min":1.405},"min":{"1min":0.878,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.6990000000000001,"5min":1.956,"15min":2.2029999999999998},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.3799999999999999,"15min":1.4350000000000001},"min":{"1min":1.131,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8640000000000001,"5min":2.0289999999999999,"15min":2.2639999999999998},"last":1.7649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298072,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073768,"statfs":{"total":96636764160,"available":96331538432,"internally_reserved":0,"allocated":802816,"data_stored":397018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2549999999999999,"15min":1.24},"min":{"1min":0.78000000000000003,"5min":0.72499999999999998,"15min":0.53000000000000003},"max":{"1min":1.591,"5min":1.653,"15min":2.137},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.302,"15min":1.2809999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.50700000000000001},"max":{"1min":1.4850000000000001,"5min":1.954,"15min":2.3140000000000001},"last":1.534}]},{"osd":1,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.147,"15min":1.1699999999999999},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.61699999999999999},"max":{"1min":1.4890000000000001,"5min":1.629,"15min":3.5409999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.177,"15min":1.2130000000000001},"min":{"1min":0.78400000000000003,"5min":0.77300000000000002,"15min":0.68899999999999995},"max":{"1min":1.534,"5min":1.6399999999999999,"15min":3.113},"last":1.099}]},{"osd":2,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1870000000000001,"15min":1.2350000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.66000000000000003},"max":{"1min":1.4330000000000001,"5min":1.702,"15min":3.339},"last":1.264},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.206,"15min":1.2290000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.5189999999999999,"5min":1.6459999999999999,"15min":2.9039999999999999},"last":1.2410000000000001}]},{"osd":3,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.256,"15min":1.3},"min":{"1min":0.94399999999999995,"5min":0.88300000000000001,"15min":0.748},"max":{"1min":1.3799999999999999,"5min":1.778,"15min":3.5870000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.294,"15min":1.3540000000000001},"min":{"1min":0.95499999999999996,"5min":0.76600000000000001,"15min":0.58199999999999996},"max":{"1min":1.5469999999999999,"5min":1.881,"15min":4.3879999999999999},"last":1.474}]},{"osd":4,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.202,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.79800000000000004,"15min":0.52600000000000002},"max":{"1min":1.5920000000000001,"5min":1.6739999999999999,"15min":5.0819999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.28,"15min":1.3480000000000001},"min":{"1min":0.70299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.7070000000000001,"5min":1.8580000000000001,"15min":4.6310000000000002},"last":1.1910000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2969999999999999,"15min":1.377},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.78800000000000003},"max":{"1min":1.4279999999999999,"5min":2.0099999999999998,"15min":3.9649999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.343,"15min":1.4079999999999999},"min":{"1min":0.91200000000000003,"5min":0.81899999999999995,"15min":0.70599999999999996},"max":{"1min":1.571,"5min":1.843,"15min":3.7559999999999998},"last":1.379}]},{"osd":7,"last update":"Tue Dec 19 07:59:37 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.4059999999999999,"15min":1.47},"min":{"1min":1.0029999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.673,"5min":1.9359999999999999,"15min":4.4539999999999997},"last":1.302},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.456,"15min":1.5129999999999999},"min":{"1min":1.1319999999999999,"5min":0.96899999999999997,"15min":0.67800000000000005},"max":{"1min":1.6020000000000001,"5min":2.0590000000000002,"15min":4.1980000000000004},"last":1.413}]}]},{"osd":4,"up_from":13,"seq":55834575006,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302172,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069668,"statfs":{"total":96636764160,"available":96327340032,"internally_reserved":0,"allocated":806912,"data_stored":397572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.218,"15min":1.218},"min":{"1min":0.83299999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.3180000000000001,"15min":1.304},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.532,"5min":1.7909999999999999,"15min":2.738},"last":1.417}]},{"osd":1,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.165,"15min":1.2030000000000001},"min":{"1min":0.78400000000000003,"5min":0.71899999999999997,"15min":0.70699999999999996},"max":{"1min":1.637,"5min":1.7310000000000001,"15min":2.589},"last":1.175},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.139,"15min":1.145},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.54300000000000004},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":2.2589999999999999},"last":1.0429999999999999}]},{"osd":2,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1319999999999999,"15min":1.167},"min":{"1min":0.78300000000000003,"5min":0.76000000000000001,"15min":0.628},"max":{"1min":1.3979999999999999,"5min":1.601,"15min":2.4569999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.224,"15min":1.2430000000000001},"min":{"1min":0.78700000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5289999999999999,"5min":1.6299999999999999,"15min":2.8050000000000002},"last":0.95699999999999996}]},{"osd":3,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2609999999999999,"15min":1.3100000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4259999999999999,"5min":1.716,"15min":3.048},"last":1.24},{"interface":"front","average":{"1min":1.27,"5min":1.3089999999999999,"15min":1.3720000000000001},"min":{"1min":0.83699999999999997,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.5620000000000001,"5min":1.637,"15min":2.661},"last":1.544}]},{"osd":5,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1839999999999999,"15min":1.2509999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.659,"5min":1.7869999999999999,"15min":3.556},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.1679999999999999,"15min":1.2529999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.625,"15min":3.2599999999999998},"last":1.377}]},{"osd":6,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1659999999999999,"15min":1.296},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.595,"5min":1.595,"15min":2.645},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3500000000000001,"15min":1.409},"min":{"1min":0.70699999999999996,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.718,"5min":1.8360000000000001,"15min":2.923},"last":1.5780000000000001}]},{"osd":7,"last update":"Tue Dec 19 07:59:28 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3140000000000001,"15min":1.411},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.514},{"interface":"front","average":{"1min":1.329,"5min":1.355,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.8,"5min":1.8,"15min":2.8460000000000001},"last":1.6619999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302184,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069656,"statfs":{"total":96636764160,"available":96327327744,"internally_reserved":0,"allocated":819200,"data_stored":398576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.85999999999999999,"15min":0.94299999999999995},"min":{"1min":0.51400000000000001,"5min":0.497,"15min":0.45500000000000002},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":5.1079999999999997},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97599999999999998,"15min":1.038},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.39200000000000002},"max":{"1min":1.2909999999999999,"5min":1.835,"15min":6.1900000000000004},"last":0.86299999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.93700000000000006,"15min":1.0009999999999999},"min":{"1min":0.628,"5min":0.56499999999999995,"15min":0.45600000000000002},"max":{"1min":1.431,"5min":1.4810000000000001,"15min":4.8879999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.071,"15min":1.1839999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.56999999999999995},"max":{"1min":1.494,"5min":1.673,"15min":8.0830000000000002},"last":1.492}]},{"osd":2,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.1319999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.52800000000000002},"max":{"1min":1.4730000000000001,"5min":1.786,"15min":4.7519999999999998},"last":1.276},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1559999999999999,"15min":1.26},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.65200000000000002},"max":{"1min":1.4710000000000001,"5min":1.704,"15min":5.3330000000000002},"last":0.92900000000000005}]},{"osd":4,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.286,"15min":1.341},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":4.7130000000000001},"last":1.53},{"interface":"front","average":{"1min":1.22,"5min":1.258,"15min":1.3520000000000001},"min":{"1min":0.92800000000000005,"5min":0.91000000000000003,"15min":0.87},"max":{"1min":1.534,"5min":1.8120000000000001,"15min":5.0220000000000002},"last":0.995}]},{"osd":5,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.2829999999999999,"15min":1.393},"min":{"1min":0.95699999999999996,"5min":0.85799999999999998,"15min":0.81799999999999995},"max":{"1min":1.48,"5min":1.609,"15min":4.9889999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.355,"15min":1.5069999999999999},"min":{"1min":1.0620000000000001,"5min":0.99399999999999999,"15min":0.65800000000000003},"max":{"1min":1.5549999999999999,"5min":1.8700000000000001,"15min":8.5299999999999994},"last":1.3700000000000001}]},{"osd":6,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.4059999999999999,"15min":1.546},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":0.88700000000000001},"max":{"1min":1.621,"5min":2.1280000000000001,"15min":7.2750000000000004},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3879999999999999,"15min":1.448},"min":{"1min":1.087,"5min":0.82499999999999996,"15min":0.79100000000000004},"max":{"1min":1.696,"5min":2.1440000000000001,"15min":4.6550000000000002},"last":1.605}]},{"osd":7,"last update":"Tue Dec 19 07:59:27 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.4670000000000001,"15min":1.6799999999999999},"min":{"1min":1.119,"5min":1.0660000000000001,"15min":0.84799999999999998},"max":{"1min":1.8400000000000001,"5min":2.2509999999999999,"15min":16.081},"last":1.665},{"interface":"front","average":{"1min":1.468,"5min":1.452,"15min":1.6619999999999999},"min":{"1min":1.1599999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.833,"5min":1.9159999999999999,"15min":15.285},"last":1.085}]}]},{"osd":2,"up_from":13,"seq":55834575006,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":827392,"data_stored":405654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.90400000000000003,"15min":0.95199999999999996},"min":{"1min":0.57999999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.3129999999999999,"5min":1.4039999999999999,"15min":4.1479999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.054,"15min":1.0900000000000001},"min":{"1min":0.748,"5min":0.66200000000000003,"15min":0.52200000000000002},"max":{"1min":1.26,"5min":1.5029999999999999,"15min":4.093},"last":0.98399999999999999}]},{"osd":1,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.94999999999999996,"15min":1},"min":{"1min":0.63700000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":3.8700000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0109999999999999,"15min":1.0489999999999999},"min":{"1min":0.66800000000000004,"5min":0.626,"15min":0.59499999999999997},"max":{"1min":1.546,"5min":1.546,"15min":3.9089999999999998},"last":0.79600000000000004}]},{"osd":3,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98299999999999998,"15min":0.95799999999999996},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.395,"5min":1.667,"15min":1.667},"last":1.141},{"interface":"front","average":{"1min":1.125,"5min":1.131,"15min":1.129},"min":{"1min":0.71499999999999997,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.4910000000000001,"5min":1.5920000000000001,"15min":1.7250000000000001},"last":0.95299999999999996}]},{"osd":4,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.321,"15min":1.3620000000000001},"min":{"1min":0.93700000000000006,"5min":0.93000000000000005,"15min":0.79300000000000004},"max":{"1min":1.603,"5min":1.603,"15min":3.9239999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2969999999999999,"15min":1.3600000000000001},"min":{"1min":0.98499999999999999,"5min":0.94999999999999996,"15min":0.82099999999999995},"max":{"1min":1.575,"5min":1.736,"15min":4.0170000000000003},"last":1.1970000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3520000000000001,"15min":1.4039999999999999},"min":{"1min":1.05,"5min":1.0049999999999999,"15min":0.93799999999999994},"max":{"1min":1.6639999999999999,"5min":1.8560000000000001,"15min":3.948},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.26,"15min":1.2689999999999999},"min":{"1min":1.05,"5min":0.84899999999999998,"15min":0.80400000000000005},"max":{"1min":1.583,"5min":1.6930000000000001,"15min":1.879},"last":1.371}]},{"osd":6,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3009999999999999,"15min":1.3120000000000001},"min":{"1min":1.018,"5min":0.92700000000000005,"15min":0.752},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.365,"15min":1.4399999999999999},"min":{"1min":1.1739999999999999,"5min":1.077,"15min":0.85399999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":3.9620000000000002},"last":1.323}]},{"osd":7,"last update":"Tue Dec 19 07:59:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.379,"15min":1.4630000000000001},"min":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":0.83699999999999997},"max":{"1min":1.6699999999999999,"5min":1.8109999999999999,"15min":3.9809999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3720000000000001,"15min":1.401},"min":{"1min":0.97399999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":2.206},"last":1.49}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302176,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069664,"statfs":{"total":96636764160,"available":96327335936,"internally_reserved":0,"allocated":811008,"data_stored":397178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.84599999999999997},"min":{"1min":0.65500000000000003,"5min":0.56399999999999995,"15min":0.40100000000000002},"max":{"1min":1.355,"5min":1.522,"15min":1.532},"last":0.999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.95599999999999996,"15min":0.95499999999999996},"min":{"1min":0.628,"5min":0.59999999999999998,"15min":0.41199999999999998},"max":{"1min":1.224,"5min":1.4950000000000001,"15min":1.9019999999999999},"last":1.157}]},{"osd":2,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.91100000000000003,"15min":0.90900000000000003},"min":{"1min":0.67400000000000004,"5min":0.53100000000000003,"15min":0.50700000000000001},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.5509999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0529999999999999,"15min":1.0509999999999999},"min":{"1min":0.77900000000000003,"5min":0.70299999999999996,"15min":0.46300000000000002},"max":{"1min":1.4730000000000001,"5min":1.6160000000000001,"15min":1.694},"last":0.72999999999999998}]},{"osd":3,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1459999999999999,"15min":1.145},"min":{"1min":0.79100000000000004,"5min":0.69899999999999995,"15min":0.66100000000000003},"max":{"1min":1.6120000000000001,"5min":1.6899999999999999,"15min":1.7909999999999999},"last":1.331},{"interface":"front","average":{"1min":1.087,"5min":1.0509999999999999,"15min":1.0820000000000001},"min":{"1min":0.71499999999999997,"5min":0.51200000000000001,"15min":0.46800000000000003},"max":{"1min":1.4330000000000001,"5min":1.5229999999999999,"15min":1.835},"last":1.2190000000000001}]},{"osd":4,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.179,"15min":1.2030000000000001},"min":{"1min":0.84199999999999997,"5min":0.69899999999999995,"15min":0.67200000000000004},"max":{"1min":1.4359999999999999,"5min":1.7150000000000001,"15min":1.778},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.296,"15min":1.3109999999999999},"min":{"1min":1.0640000000000001,"5min":0.84199999999999997,"15min":0.64400000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":2.2480000000000002},"last":1.1080000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2669999999999999,"15min":1.2909999999999999},"min":{"1min":1.0980000000000001,"5min":0.80500000000000005,"15min":0.65100000000000002},"max":{"1min":1.569,"5min":1.77,"15min":2.2200000000000002},"last":1.387},{"interface":"front","average":{"1min":1.276,"5min":1.2729999999999999,"15min":1.3089999999999999},"min":{"1min":1.048,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5029999999999999,"5min":1.671,"15min":1.905},"last":1.4359999999999999}]},{"osd":6,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3740000000000001,"15min":1.403},"min":{"1min":1.129,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.901,"5min":1.901,"15min":3.024},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.375,"15min":1.397},"min":{"1min":1.073,"5min":0.747,"15min":0.747},"max":{"1min":1.706,"5min":2.012,"15min":2.6230000000000002},"last":1.286}]},{"osd":7,"last update":"Tue Dec 19 07:59:24 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.429,"15min":1.466},"min":{"1min":0.999,"5min":0.88500000000000001,"15min":0.83199999999999996},"max":{"1min":1.8160000000000001,"5min":2.0790000000000002,"15min":3.6080000000000001},"last":1.254},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.365,"15min":1.4419999999999999},"min":{"1min":1.0509999999999999,"5min":0.93799999999999994,"15min":0.88100000000000001},"max":{"1min":1.6659999999999999,"5min":1.829,"15min":4.101},"last":1.5700000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":986431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.93000000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.45800000000000002},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91100000000000003,"15min":1.004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.379},"max":{"1min":1.2250000000000001,"5min":1.454,"15min":6.0309999999999997},"last":1.2450000000000001}]},{"osd":2,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.629},"max":{"1min":1.286,"5min":1.335,"15min":1.5800000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.995,"5min":1.028,"15min":1.026},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.61399999999999999},"max":{"1min":1.2509999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.73199999999999998}]},{"osd":3,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.96899999999999997,"15min":0.95199999999999996},"min":{"1min":0.60399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2150000000000001,"5min":1.671,"15min":1.671},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.133,"15min":1.1339999999999999},"min":{"1min":0.78600000000000003,"5min":0.77800000000000002,"15min":0.61899999999999999},"max":{"1min":1.4319999999999999,"5min":1.54,"15min":1.6719999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2969999999999999,"15min":1.3200000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.526,"5min":1.671,"15min":2.1669999999999998},"last":1.337},{"interface":"front","average":{"1min":1.306,"5min":1.3129999999999999,"15min":1.3049999999999999},"min":{"1min":0.90100000000000002,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":2.141},"last":1.0700000000000001}]},{"osd":5,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.365,"15min":1.3799999999999999},"min":{"1min":0.96799999999999997,"5min":0.95699999999999996,"15min":0.77800000000000002},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":2.117},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.363,"15min":1.381},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.80300000000000005},"max":{"1min":1.591,"5min":1.7789999999999999,"15min":2.0859999999999999},"last":1.163}]},{"osd":6,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.276,"15min":1.274},"min":{"1min":0.95499999999999996,"5min":0.91100000000000003,"15min":0.77400000000000002},"max":{"1min":1.903,"5min":1.903,"15min":1.9059999999999999},"last":0.996},{"interface":"front","average":{"1min":1.268,"5min":1.3069999999999999,"15min":1.343},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.649,"5min":1.649,"15min":1.9530000000000001},"last":1.2909999999999999}]},{"osd":7,"last update":"Tue Dec 19 07:59:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3149999999999999,"15min":1.3320000000000001},"min":{"1min":1.004,"5min":0.98399999999999999,"15min":0.84799999999999998},"max":{"1min":1.841,"5min":1.841,"15min":2.0539999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.371,"15min":1.4039999999999999},"min":{"1min":1.095,"5min":0.996,"15min":0.67700000000000005},"max":{"1min":1.716,"5min":1.716,"15min":2.016},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:00:13.172 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:00:13.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:00:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.384+0000 7f60baa16700 1 Processor -- start 2023-12-19T08:00:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.384+0000 7f60baa16700 1 -- start start 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60baa16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60b40ce3c0 0x7f60b40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60baa16700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f60b406b8e0 0x7f60b415d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60baa16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b406b450 con 0x7f60b40ca960 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60baa16700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b406bd40 con 0x7f60b40ce3c0 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60baa16700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b415db40 con 0x7f60b406b8e0 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60b40ce3c0 0x7f60b40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60b40ce3c0 0x7f60b40ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46192/0 (socket says 172.21.15.111:46192) 2023-12-19T08:00:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 -- 172.21.15.111:0/2442344822 learned_addr learned my addr 172.21.15.111:0/2442344822 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9213700 1 --2- 172.21.15.111:0/2442344822 >> v2:172.21.15.143:3300/0 conn(0x7f60b406b8e0 0x7f60b415d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 -- 172.21.15.111:0/2442344822 >> v2:172.21.15.143:3300/0 conn(0x7f60b406b8e0 msgr2=0x7f60b415d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 --2- 172.21.15.111:0/2442344822 >> v2:172.21.15.143:3300/0 conn(0x7f60b406b8e0 0x7f60b415d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 -- 172.21.15.111:0/2442344822 >> v1:172.21.15.111:6789/0 conn(0x7f60b40ca960 legacy=0x7f60b40cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 -- 172.21.15.111:0/2442344822 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b406a840 con 0x7f60b40ce3c0 2023-12-19T08:00:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b8a12700 1 -- 172.21.15.111:0/2442344822 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636376400 0 0) 0x7f60b406b450 con 0x7f60b40ca960 2023-12-19T08:00:13.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b9a14700 1 --2- 172.21.15.111:0/2442344822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ce3c0 0x7f60b40ce790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a380 tx=0x7f60b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aa413eba9a6cfc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b8a12700 1 -- 172.21.15.111:0/2442344822 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0009e70 con 0x7f60b40ce3c0 2023-12-19T08:00:13.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b8a12700 1 -- 172.21.15.111:0/2442344822 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b000b860 con 0x7f60b40ce3c0 2023-12-19T08:00:13.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.388+0000 7f60b8a12700 1 -- 172.21.15.111:0/2442344822 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b000bc00 con 0x7f60b40ce3c0 2023-12-19T08:00:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- 172.21.15.111:0/2442344822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ce3c0 msgr2=0x7f60b40ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 --2- 172.21.15.111:0/2442344822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ce3c0 0x7f60b40ce790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a380 tx=0x7f60b0009250 comp rx=0 tx=0).stop 2023-12-19T08:00:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- 172.21.15.111:0/2442344822 shutdown_connections 2023-12-19T08:00:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 --2- 172.21.15.111:0/2442344822 >> v2:172.21.15.143:3300/0 conn(0x7f60b406b8e0 0x7f60b415d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 --2- 172.21.15.111:0/2442344822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ce3c0 0x7f60b40ce790 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- 172.21.15.111:0/2442344822 >> 172.21.15.111:0/2442344822 conn(0x7f60b4067bb0 msgr2=0x7f60b40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:13.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- 172.21.15.111:0/2442344822 shutdown_connections 2023-12-19T08:00:13.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- 172.21.15.111:0/2442344822 wait complete. 2023-12-19T08:00:13.393 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T08:00:13.392+0000 7fcf701f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:00:13.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 Processor -- start 2023-12-19T08:00:13.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.392+0000 7f60baa16700 1 -- start start 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60b406b8e0 0x7f60b4150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60b40ce3c0 0x7f60b416d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b416cb60 con 0x7f60b40ca960 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b416ce40 con 0x7f60b40ce3c0 2023-12-19T08:00:13.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40c8c90 con 0x7f60b406b8e0 2023-12-19T08:00:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46208/0 (socket says 172.21.15.111:46208) 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 -- 172.21.15.111:0/3971650374 learned_addr learned my addr 172.21.15.111:0/3971650374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9a14700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60b406b8e0 0x7f60b4150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60ba215700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60b40ce3c0 0x7f60b416d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 -- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60b406b8e0 msgr2=0x7f60b4150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60b406b8e0 0x7f60b4150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 -- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60b40ce3c0 msgr2=0x7f60b416d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60b40ce3c0 0x7f60b416d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 -- 172.21.15.111:0/3971650374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00094f0 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60b9213700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e960 tx=0x7f60a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4013680 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a40137e0 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4013b80 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b40c8f70 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.396+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b4151440 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.400+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b404be40 con 0x7f60b40ca960 2023-12-19T08:00:13.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.400+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f60a40123d0 con 0x7f60b40ca960 2023-12-19T08:00:13.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.400+0000 7f60a9ffb700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 0x7f6098062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.400+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f60a407cca0 con 0x7f60b40ca960 2023-12-19T08:00:13.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.404+0000 7f60b9a14700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 0x7f6098062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.404+0000 7f60b9a14700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 0x7f6098062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a380 tx=0x7f60b0017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.408+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f60a404ce50 con 0x7f60b40ca960 2023-12-19T08:00:13.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.628+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f60b406ab20 con 0x7f60b40ca960 2023-12-19T08:00:13.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.640+0000 7f60a9ffb700 1 -- 172.21.15.111:0/3971650374 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f60a40170e0 con 0x7f60b40ca960 2023-12-19T08:00:13.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 msgr2=0x7f6098062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 0x7f6098062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60b406a380 tx=0x7f60b0017000 comp rx=0 tx=0).stop 2023-12-19T08:00:13.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 msgr2=0x7f60b414d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e960 tx=0x7f60a40091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 shutdown_connections 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6098060670 0x7f6098062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60b40ce3c0 0x7f60b416d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40ca960 0x7f60b414d720 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 --2- 172.21.15.111:0/3971650374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60b406b8e0 0x7f60b4150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 >> 172.21.15.111:0/3971650374 conn(0x7f60b4067bb0 msgr2=0x7f60b40c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 shutdown_connections 2023-12-19T08:00:13.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.644+0000 7f60baa16700 1 -- 172.21.15.111:0/3971650374 wait complete. 2023-12-19T08:00:13.646 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:00:13.665 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:00:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.832+0000 7fa16ae78700 1 Processor -- start 2023-12-19T08:00:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.832+0000 7fa16ae78700 1 -- start start 2023-12-19T08:00:13.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.832+0000 7fa16ae78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa164140710 0x7fa1640c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa16ae78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164068800 0x7fa1640ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa16ae78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16406e150 con 0x7fa164140ae0 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa16ae78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16406d540 con 0x7fa164140710 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa16ae78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16406d820 con 0x7fa164068800 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa164140710 0x7fa1640c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa169415700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164068800 0x7fa1640ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa164140ae0 0x7fa1640680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:33072/0 (socket says 172.21.15.111:33072) 2023-12-19T08:00:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15bfff700 1 -- 172.21.15.111:0/309358023 learned_addr learned my addr 172.21.15.111:0/309358023 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 -- 172.21.15.111:0/309358023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164068800 msgr2=0x7fa1640ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 --2- 172.21.15.111:0/309358023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164068800 0x7fa1640ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15b7fe700 1 -- 172.21.15.111:0/309358023 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885153137 0 0) 0x7fa16406e150 con 0x7fa164140ae0 2023-12-19T08:00:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 -- 172.21.15.111:0/309358023 >> v1:172.21.15.143:6789/0 conn(0x7fa164140ae0 legacy=0x7fa1640680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 -- 172.21.15.111:0/309358023 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16406db00 con 0x7fa164140710 2023-12-19T08:00:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa168c14700 1 --2- 172.21.15.111:0/309358023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1640c70a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098e0 tx=0x7fa1600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508542c477b1337e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15b7fe700 1 -- 172.21.15.111:0/309358023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000b640 con 0x7fa164140710 2023-12-19T08:00:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15b7fe700 1 -- 172.21.15.111:0/309358023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16000b7a0 con 0x7fa164140710 2023-12-19T08:00:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.836+0000 7fa15b7fe700 1 -- 172.21.15.111:0/309358023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16000bb40 con 0x7fa164140710 2023-12-19T08:00:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- 172.21.15.111:0/309358023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 msgr2=0x7fa1640c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 --2- 172.21.15.111:0/309358023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1640c70a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098e0 tx=0x7fa1600091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- 172.21.15.111:0/309358023 shutdown_connections 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 --2- 172.21.15.111:0/309358023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164068800 0x7fa1640ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 --2- 172.21.15.111:0/309358023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1640c70a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- 172.21.15.111:0/309358023 >> 172.21.15.111:0/309358023 conn(0x7fa1640c8850 msgr2=0x7fa1640c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- 172.21.15.111:0/309358023 shutdown_connections 2023-12-19T08:00:13.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- 172.21.15.111:0/309358023 wait complete. 2023-12-19T08:00:13.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 Processor -- start 2023-12-19T08:00:13.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.840+0000 7fa16ae78700 1 -- start start 2023-12-19T08:00:13.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 0x7fa16415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164140ae0 0x7fa1640ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16415d300 con 0x7fa164140710 2023-12-19T08:00:13.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640c2f30 con 0x7fa164068800 2023-12-19T08:00:13.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640c31d0 con 0x7fa164140ae0 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa168c14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 0x7fa16415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46226/0 (socket says 172.21.15.111:46226) 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 -- 172.21.15.111:0/2954330911 learned_addr learned my addr 172.21.15.111:0/2954330911 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa169415700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164140ae0 0x7fa1640ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 -- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164140ae0 msgr2=0x7fa1640ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164140ae0 0x7fa1640ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 -- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 msgr2=0x7fa16415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 0x7fa16415cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 -- 172.21.15.111:0/2954330911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160009420 con 0x7fa164140710 2023-12-19T08:00:13.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa168c14700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 0x7fa16415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa15bfff700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c560 tx=0x7fa154000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154005b30 con 0x7fa164140710 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa154005c90 con 0x7fa164140710 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1640c34b0 con 0x7fa164140710 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154007320 con 0x7fa164140710 2023-12-19T08:00:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.844+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1640c3840 con 0x7fa164140710 2023-12-19T08:00:13.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.848+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1640c3c90 con 0x7fa164140710 2023-12-19T08:00:13.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.848+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa15401b3b0 con 0x7fa164140710 2023-12-19T08:00:13.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.848+0000 7fa158ff9700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 0x7fa150062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:13.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.848+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa15407c010 con 0x7fa164140710 2023-12-19T08:00:13.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.848+0000 7fa168c14700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 0x7fa150062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:13.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.852+0000 7fa168c14700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 0x7fa150062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1600093f0 tx=0x7fa160017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:13.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:13.856+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa15404c1c0 con 0x7fa164140710 2023-12-19T08:00:14.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.064+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa164071120 con 0x7fa164140710 2023-12-19T08:00:14.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.064+0000 7fa158ff9700 1 -- 172.21.15.111:0/2954330911 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7fa15404f7e0 con 0x7fa164140710 2023-12-19T08:00:14.066 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:14.066 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 msgr2=0x7fa150062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 0x7fa150062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa1600093f0 tx=0x7fa160017000 comp rx=0 tx=0).stop 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 msgr2=0x7fa1641306a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa15400c560 tx=0x7fa154000fe0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 shutdown_connections 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa150060670 0x7fa150062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa164140ae0 0x7fa1640ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164140710 0x7fa1641306a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 --2- 172.21.15.111:0/2954330911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa164068800 0x7fa16415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 >> 172.21.15.111:0/2954330911 conn(0x7fa1640c8850 msgr2=0x7fa16406b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 shutdown_connections 2023-12-19T08:00:14.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.068+0000 7fa16ae78700 1 -- 172.21.15.111:0/2954330911 wait complete. 2023-12-19T08:00:14.071 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:00:14.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:00:07.870107+0000', 'modified': '2023-12-19T08:00:11.482481+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5943}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5943': {'gid': 5943, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/2938693255', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2938693255}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2938693255}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:00:14.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:00:14.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T08:00:14.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 Processor -- start 2023-12-19T08:00:14.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 -- start start 2023-12-19T08:00:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0bf80b1b90 0x7f0bf8156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf80b3a10 con 0x7f0bf80ad170 2023-12-19T08:00:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf8068b90 con 0x7f0bf80b1b90 2023-12-19T08:00:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.660+0000 7f0bff665700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80b1fd0 con 0x7f0bf8067aa0 2023-12-19T08:00:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:44460/0 (socket says 172.21.15.111:44460) 2023-12-19T08:00:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 -- 172.21.15.111:0/966397770 learned_addr learned my addr 172.21.15.111:0/966397770 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/966397770 >> v2:172.21.15.111:3300/0 conn(0x7f0bf80b1b90 0x7f0bf8156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 -- 172.21.15.111:0/966397770 >> v2:172.21.15.111:3300/0 conn(0x7f0bf80b1b90 msgr2=0x7f0bf8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/966397770 >> v2:172.21.15.111:3300/0 conn(0x7f0bf80b1b90 0x7f0bf8156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 -- 172.21.15.111:0/966397770 >> v1:172.21.15.111:6789/0 conn(0x7f0bf80ad170 legacy=0x7f0bf8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 -- 172.21.15.111:0/966397770 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80ad540 con 0x7f0bf8067aa0 2023-12-19T08:00:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/966397770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0be80099b0 tx=0x7f0be8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f12f328cded548 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0beffff700 1 -- 172.21.15.111:0/966397770 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be800b860 con 0x7f0bf8067aa0 2023-12-19T08:00:14.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0beffff700 1 -- 172.21.15.111:0/966397770 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be800b9c0 con 0x7f0bf8067aa0 2023-12-19T08:00:14.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0beffff700 1 -- 172.21.15.111:0/966397770 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be800bd60 con 0x7f0bf8067aa0 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 -- 172.21.15.111:0/966397770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 msgr2=0x7f0bf80b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 --2- 172.21.15.111:0/966397770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0be80099b0 tx=0x7f0be8009250 comp rx=0 tx=0).stop 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 -- 172.21.15.111:0/966397770 shutdown_connections 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 --2- 172.21.15.111:0/966397770 >> v2:172.21.15.111:3300/0 conn(0x7f0bf80b1b90 0x7f0bf8156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 --2- 172.21.15.111:0/966397770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf8067aa0 0x7f0bf80b1650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.664+0000 7f0bff665700 1 -- 172.21.15.111:0/966397770 >> 172.21.15.111:0/966397770 conn(0x7f0bf80cdda0 msgr2=0x7f0bf80ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- 172.21.15.111:0/966397770 shutdown_connections 2023-12-19T08:00:14.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- 172.21.15.111:0/966397770 wait complete. 2023-12-19T08:00:14.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 Processor -- start 2023-12-19T08:00:14.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- start start 2023-12-19T08:00:14.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 0x7f0bf815dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 0x7f0bf815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815ebe0 con 0x7f0bf80ad170 2023-12-19T08:00:14.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815e500 con 0x7f0bf8067aa0 2023-12-19T08:00:14.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.668+0000 7f0bff665700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf815cd30 con 0x7f0bf80b1b90 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 0x7f0bf815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:44462/0 (socket says 172.21.15.111:44462) 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 -- 172.21.15.111:0/220322489 learned_addr learned my addr 172.21.15.111:0/220322489 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 0x7f0bf815b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46258/0 (socket says 172.21.15.111:46258) 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfd401700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 0x7f0bf815dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 -- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 msgr2=0x7f0bf815dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 0x7f0bf815dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 -- 172.21.15.111:0/220322489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 msgr2=0x7f0bf815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 0x7f0bf815b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 -- 172.21.15.111:0/220322489 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80094f0 con 0x7f0bf80b1b90 2023-12-19T08:00:14.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfd401700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 0x7f0bf815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0007f20 tx=0x7f0bf0009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bed7fa700 1 -- 172.21.15.111:0/220322489 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000ebf0 con 0x7f0bf80b1b90 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bed7fa700 1 -- 172.21.15.111:0/220322489 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf000ed70 con 0x7f0bf80b1b90 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bed7fa700 1 -- 172.21.15.111:0/220322489 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0014670 con 0x7f0bf80b1b90 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf815d010 con 0x7f0bf80b1b90 2023-12-19T08:00:14.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.672+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf815d5d0 con 0x7f0bf80b1b90 2023-12-19T08:00:14.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.676+0000 7f0bed7fa700 1 -- 172.21.15.111:0/220322489 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0bf00149e0 con 0x7f0bf80b1b90 2023-12-19T08:00:14.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.676+0000 7f0bed7fa700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 0x7f0bdc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.676+0000 7f0bed7fa700 1 -- 172.21.15.111:0/220322489 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0bf007d2e0 con 0x7f0bf80b1b90 2023-12-19T08:00:14.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.676+0000 7f0bfd401700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 0x7f0bdc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfd401700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 0x7f0bdc062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 Processor -- start 2023-12-19T08:00:14.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 -- start start 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c41f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c42440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b6ff30 con 0x1a0f790 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b6bed0 con 0x19c4ae0 2023-12-19T08:00:14.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b320b0 con 0x19c4710 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46260/0 (socket says 172.21.15.111:46260) 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 -- 172.21.15.111:0/4095682882 learned_addr learned my addr 172.21.15.111:0/4095682882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfd401700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c41f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c42440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 -- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 msgr2=0x1c42440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c42440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 -- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 msgr2=0x1c41f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c41f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfdc02700 1 -- 172.21.15.111:0/4095682882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a109e0 con 0x1a0f790 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfd401700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c41f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.680+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c42440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:14.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0bf007b440 tx=0x7f0bf0050bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fd73db398e8f2c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0be27fc700 1 -- 172.21.15.111:0/4095682882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0066070 con 0x1a0f790 2023-12-19T08:00:14.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0be27fc700 1 -- 172.21.15.111:0/4095682882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf0058130 con 0x1a0f790 2023-12-19T08:00:14.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0be27fc700 1 -- 172.21.15.111:0/4095682882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf007d0f0 con 0x1a0f790 2023-12-19T08:00:14.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- 172.21.15.111:0/4095682882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 msgr2=0x1c42980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0bf007b440 tx=0x7f0bf0050bb0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- 172.21.15.111:0/4095682882 shutdown_connections 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c42980 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c42440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 --2- 172.21.15.111:0/4095682882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c41f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- 172.21.15.111:0/4095682882 >> 172.21.15.111:0/4095682882 conn(0x19c4eb0 msgr2=0x1bfe930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- 172.21.15.111:0/4095682882 shutdown_connections 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- 172.21.15.111:0/4095682882 wait complete. 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 -- start start 2023-12-19T08:00:14.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.684+0000 7f0c018f1740 1 Processor -- start 2023-12-19T08:00:14.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4710 0x1c46300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4ae0 0x1c46840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b31d50 con 0x1a0f790 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b0a970 con 0x19c4710 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1aa8430 con 0x19c4ae0 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 10 client.0 _ll_register_callbacks cb 0x1bb56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46274/0 (socket says 172.21.15.111:46274) 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 -- 172.21.15.111:0/3425297391 learned_addr learned my addr 172.21.15.111:0/3425297391 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4ae0 0x1c46840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfd401700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4710 0x1c46300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 -- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4ae0 msgr2=0x1c46840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4ae0 0x1c46840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 -- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4710 msgr2=0x1c46300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4710 0x1c46300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 -- 172.21.15.111:0/3425297391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf815d5d0 con 0x1a0f790 2023-12-19T08:00:14.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0054350 tx=0x7f0bf0068000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6df9a9668c10f7d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf004f4f0 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1b0acc0 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0bf006b160 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bf006b320 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 10 client.6084 fetch_fsmap learned FSMap version 24 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x1b3a370 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1b0acc0 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0bf0050640 con 0x1a0f790 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 10 client.6084 fetch_fsmap finished waiting for FSMap version 24 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 10 client.6084 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 20 client.6084 populate_metadata read hostname 'smithi111' 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1c29400 0x1c514a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 4 client.6084 mds_command: new command op to 5943 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:00:14.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 --> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1b0e3e0 con 0x1c29400 2023-12-19T08:00:14.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.688+0000 7f0bfd401700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1c29400 0x1c514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0bd77fe700 10 client.6084 ms_handle_connect on v2:172.21.15.111:6834/2938693255 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0bfd401700 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1c29400 0x1c514a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f134c195b8533f44 server_cookie=79f6bc6383bb249c in_seq=0 out_seq=0 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0bd77fe700 1 -- 172.21.15.111:0/3425297391 <== mds.0 v2:172.21.15.111:6834/2938693255 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x1b0e3e0 con 0x1c29400 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0bd77fe700 10 client.6084 handle_command_reply: tid=0 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 client.6084 shutdown 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 msgr2=0x1c27cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0054350 tx=0x7f0bf0068000 comp rx=0 tx=0).stop 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 shutdown_connections 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1c29400 0x1c514a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c27cb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4ae0 0x1c46840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 --2- 172.21.15.111:0/3425297391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4710 0x1c46300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0bd77fe700 0 client.6084 ms_handle_reset on v2:172.21.15.111:6834/2938693255 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 >> 172.21.15.111:0/3425297391 conn(0x19c4eb0 msgr2=0x1bf6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 shutdown_connections 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 1 -- 172.21.15.111:0/3425297391 wait complete. 2023-12-19T08:00:14.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.692+0000 7f0c018f1740 20 client.6084 trim_cache size 0 max 16384 2023-12-19T08:00:14.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 Processor -- start 2023-12-19T08:00:14.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 -- start start 2023-12-19T08:00:14.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c26ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c27220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a3d7c0 con 0x1a0f790 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b67b50 con 0x19c4ae0 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b6f9d0 con 0x19c4710 2023-12-19T08:00:14.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0bfdc02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46280/0 (socket says 172.21.15.111:46280) 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.712+0000 7f0bfdc02700 1 -- 172.21.15.111:0/1445466606 learned_addr learned my addr 172.21.15.111:0/1445466606 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c27220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfd401700 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c26ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 -- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 msgr2=0x1c26ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c26ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 -- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 msgr2=0x1c27220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c27220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 -- 172.21.15.111:0/1445466606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1b20d80 con 0x1a0f790 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00510e0 tx=0x7f0bf0076000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9589c4b0397b9174 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0be2ffd700 1 -- 172.21.15.111:0/1445466606 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0051890 con 0x1a0f790 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0be2ffd700 1 -- 172.21.15.111:0/1445466606 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00099c0 con 0x1a0f790 2023-12-19T08:00:14.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0be2ffd700 1 -- 172.21.15.111:0/1445466606 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf004f4f0 con 0x1a0f790 2023-12-19T08:00:14.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 -- 172.21.15.111:0/1445466606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 msgr2=0x1c25780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00510e0 tx=0x7f0bf0076000 comp rx=0 tx=0).stop 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 -- 172.21.15.111:0/1445466606 shutdown_connections 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c27220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1a0f790 0x1c25780 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 --2- 172.21.15.111:0/1445466606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x19c4710 0x1c26ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 -- 172.21.15.111:0/1445466606 >> 172.21.15.111:0/1445466606 conn(0x19c4eb0 msgr2=0x1c457b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 -- 172.21.15.111:0/1445466606 shutdown_connections 2023-12-19T08:00:14.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.716+0000 7f0c018f1740 1 -- 172.21.15.111:0/1445466606 wait complete. 2023-12-19T08:00:14.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 -- start start 2023-12-19T08:00:14.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 Processor -- start 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c9bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 0x1bfb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c289e0 con 0x19c4710 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bc26d0 con 0x19c4ae0 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bff070 con 0x1a0f790 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0c018f1740 10 client.0 _ll_register_callbacks cb 0x19909d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0bfd401700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0bfd401700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46282/0 (socket says 172.21.15.111:46282) 2023-12-19T08:00:14.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.720+0000 7f0bfd401700 1 -- 172.21.15.111:0/80703529 learned_addr learned my addr 172.21.15.111:0/80703529 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c9bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 0x1bfb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 -- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 msgr2=0x1bfb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 0x1bfb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 -- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 msgr2=0x1c9bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c9bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 -- 172.21.15.111:0/80703529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1b3a370 con 0x19c4710 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c9bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfdc02700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 0x1bfb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfd401700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009010 tx=0x7f0bf4026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c54d5daf9f63733 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:00:14.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4024810 con 0x19c4710 2023-12-19T08:00:14.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0bf402ee00 con 0x19c4710 2023-12-19T08:00:14.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1b40410 con 0x19c4710 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bf4024b80 con 0x19c4710 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 10 client.6090 fetch_fsmap learned FSMap version 24 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x1b15130 con 0x19c4710 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1b40410 con 0x19c4710 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 24) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0bf4024ce0 con 0x19c4710 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 10 client.6090 fetch_fsmap finished waiting for FSMap version 24 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 10 client.6090 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 20 client.6090 populate_metadata read hostname 'smithi111' 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1ca2920 0x1ca4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 4 client.6090 mds_command: new command op to 5943 tid=0[{"prefix": "status"}] 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 --> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1a1c4e0 con 0x1ca2920 2023-12-19T08:00:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1ca2920 0x1ca4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bfcc00700 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1ca2920 0x1ca4d00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7c8596145a9d674 server_cookie=b1cc4b316f8ad460 in_seq=0 out_seq=0 2023-12-19T08:00:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.724+0000 7f0bd77fe700 10 client.6090 ms_handle_connect on v2:172.21.15.111:6834/2938693255 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bd77fe700 1 -- 172.21.15.111:0/80703529 <== mds.0 v2:172.21.15.111:6834/2938693255 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x1a1c4e0 con 0x1ca2920 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bd77fe700 10 client.6090 handle_command_reply: tid=0 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 client.6090 shutdown 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 msgr2=0x1c9ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009010 tx=0x7f0bf4026040 comp rx=0 tx=0).stop 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 shutdown_connections 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:6834/2938693255,v1:172.21.15.111:6835/2938693255] conn(0x1ca2920 0x1ca4d00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x19c4ae0 0x1c9bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1a0f790 0x1bfb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 --2- 172.21.15.111:0/80703529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x19c4710 0x1c9ba90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 >> 172.21.15.111:0/80703529 conn(0x19c4eb0 msgr2=0x1c457b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bd77fe700 0 client.6090 ms_handle_reset on v2:172.21.15.111:6834/2938693255 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 shutdown_connections 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 1 -- 172.21.15.111:0/80703529 wait complete. 2023-12-19T08:00:14.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0c018f1740 20 client.6090 trim_cache size 0 max 16384 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 5943, 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:00:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2752448300000001, 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 24, 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 68, 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 68, 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.227382907000001 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:00:14.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 msgr2=0x7f0bdc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 0x7f0bdc062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4009040 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 msgr2=0x7f0bf816be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0007f20 tx=0x7f0bf0009860 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 shutdown_connections 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bdc060670 0x7f0bdc062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bf80b1b90 0x7f0bf816be20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf80ad170 0x7f0bf815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 --2- 172.21.15.111:0/220322489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bf8067aa0 0x7f0bf815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 >> 172.21.15.111:0/220322489 conn(0x7f0bf80cdda0 msgr2=0x7f0bf8062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 shutdown_connections 2023-12-19T08:00:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.728+0000 7f0bff665700 1 -- 172.21.15.111:0/220322489 wait complete. 2023-12-19T08:00:14.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:00:14.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.908+0000 7fac1c22e700 1 Processor -- start 2023-12-19T08:00:14.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.908+0000 7fac1c22e700 1 -- start start 2023-12-19T08:00:14.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1c22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac1415d320 0x7fac1415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1c22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 0x7fac140680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1c22e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140c7190 con 0x7fac140c15b0 2023-12-19T08:00:14.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1c22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac140aefd0 con 0x7fac1415d320 2023-12-19T08:00:14.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1c22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140c1b50 con 0x7fac140aeba0 2023-12-19T08:00:14.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac197c9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fac140c15b0 0x7fac1406d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42612/0 (socket says 172.21.15.111:42612) 2023-12-19T08:00:14.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac197c9700 1 -- 172.21.15.111:0/2857603509 learned_addr learned my addr 172.21.15.111:0/2857603509 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac19fca700 1 --2- 172.21.15.111:0/2857603509 >> v2:172.21.15.143:3300/0 conn(0x7fac1415d320 0x7fac1415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1a7cb700 1 --2- 172.21.15.111:0/2857603509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 0x7fac140680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac18fc8700 1 -- 172.21.15.111:0/2857603509 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499948365 0 0) 0x7fac140c7190 con 0x7fac140c15b0 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac18fc8700 1 -- 172.21.15.111:0/2857603509 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac00003570 con 0x7fac140c15b0 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1a7cb700 1 -- 172.21.15.111:0/2857603509 >> v2:172.21.15.143:3300/0 conn(0x7fac1415d320 msgr2=0x7fac1415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1a7cb700 1 --2- 172.21.15.111:0/2857603509 >> v2:172.21.15.143:3300/0 conn(0x7fac1415d320 0x7fac1415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1a7cb700 1 -- 172.21.15.111:0/2857603509 >> v1:172.21.15.111:6789/0 conn(0x7fac140c15b0 legacy=0x7fac1406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.912+0000 7fac1a7cb700 1 -- 172.21.15.111:0/2857603509 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140c1e30 con 0x7fac140aeba0 2023-12-19T08:00:14.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1a7cb700 1 --2- 172.21.15.111:0/2857603509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 0x7fac140680f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e60 tx=0x7fac0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a577ae6b12dbe2d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac18fc8700 1 -- 172.21.15.111:0/2857603509 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00e070 con 0x7fac140aeba0 2023-12-19T08:00:14.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac18fc8700 1 -- 172.21.15.111:0/2857603509 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c00bab0 con 0x7fac140aeba0 2023-12-19T08:00:14.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac18fc8700 1 -- 172.21.15.111:0/2857603509 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c00be70 con 0x7fac140aeba0 2023-12-19T08:00:14.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 -- 172.21.15.111:0/2857603509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 msgr2=0x7fac140680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 --2- 172.21.15.111:0/2857603509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 0x7fac140680f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e60 tx=0x7fac0c009780 comp rx=0 tx=0).stop 2023-12-19T08:00:14.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 -- 172.21.15.111:0/2857603509 shutdown_connections 2023-12-19T08:00:14.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 --2- 172.21.15.111:0/2857603509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac140aeba0 0x7fac140680f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 --2- 172.21.15.111:0/2857603509 >> v2:172.21.15.143:3300/0 conn(0x7fac1415d320 0x7fac1415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 -- 172.21.15.111:0/2857603509 >> 172.21.15.111:0/2857603509 conn(0x7fac140c8850 msgr2=0x7fac140c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 -- 172.21.15.111:0/2857603509 shutdown_connections 2023-12-19T08:00:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.916+0000 7fac1c22e700 1 -- 172.21.15.111:0/2857603509 wait complete. 2023-12-19T08:00:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 Processor -- start 2023-12-19T08:00:14.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 -- start start 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 0x7fac14152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac1415d320 0x7fac14151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14152940 con 0x7fac140aeba0 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14152bc0 con 0x7fac140c15b0 2023-12-19T08:00:14.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1c22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac14152ea0 con 0x7fac1415d320 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46294/0 (socket says 172.21.15.111:46294) 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 -- 172.21.15.111:0/169135716 learned_addr learned my addr 172.21.15.111:0/169135716 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac197c9700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 0x7fac14152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac1a7cb700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac1415d320 0x7fac14151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 -- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac1415d320 msgr2=0x7fac14151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac1415d320 0x7fac14151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 -- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 msgr2=0x7fac14152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 0x7fac14152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 -- 172.21.15.111:0/169135716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009a20 con 0x7fac140aeba0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac197c9700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 0x7fac14152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.920+0000 7fac19fca700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fac1000ec80 tx=0x7fac100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000c850 con 0x7fac140aeba0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1000c9b0 con 0x7fac140aeba0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000cd50 con 0x7fac140aeba0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14153180 con 0x7fac140aeba0 2023-12-19T08:00:14.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac14153740 con 0x7fac140aeba0 2023-12-19T08:00:14.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac14071120 con 0x7fac140aeba0 2023-12-19T08:00:14.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.924+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fac10013680 con 0x7fac140aeba0 2023-12-19T08:00:14.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.928+0000 7fac09ffb700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 0x7fabf8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:14.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.928+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fac1007cf80 con 0x7fac140aeba0 2023-12-19T08:00:14.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.928+0000 7fac197c9700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 0x7fabf8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:14.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.928+0000 7fac197c9700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 0x7fabf8062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac14068af0 tx=0x7fac04009040 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:14.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:14.932+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac100acbe0 con 0x7fac140aeba0 2023-12-19T08:00:15.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.132+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac14155cf0 con 0x7fac140aeba0 2023-12-19T08:00:15.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.132+0000 7fac09ffb700 1 -- 172.21.15.111:0/169135716 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7fac1004d130 con 0x7fac140aeba0 2023-12-19T08:00:15.134 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 msgr2=0x7fabf8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 0x7fabf8062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac14068af0 tx=0x7fac04009040 comp rx=0 tx=0).stop 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 msgr2=0x7fac140c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fac1000ec80 tx=0x7fac100091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 shutdown_connections 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fabf8060670 0x7fabf8062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac140c15b0 0x7fac14152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:15.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac1415d320 0x7fac14151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 --2- 172.21.15.111:0/169135716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac140aeba0 0x7fac140c3f80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 >> 172.21.15.111:0/169135716 conn(0x7fac140c8850 msgr2=0x7fac14154160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 shutdown_connections 2023-12-19T08:00:15.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:15.136+0000 7fac1c22e700 1 -- 172.21.15.111:0/169135716 wait complete. 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:00:15.156 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:00:15.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.157 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:00:15.179 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41642sec preferred_lft 41642sec 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.180 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:00:15.181 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:00:15.182 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:00:15.182 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:00:15.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.183 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:00:15.262 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:00:15.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.265 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:00:15.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:00:15.327 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:00:15.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.457 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:00:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:00:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:00:15.472 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:00:15.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:00:15.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:00:15.554 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:00:15.554 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:00:15.554 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:15.615 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:00:15.616 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:00:15.616 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:15.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:00:15.760 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:00:15.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:00:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:00:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:00:15.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:15.844 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:00:15.909 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:00:15.909 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:00:15.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:00:15.994 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:00:15.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:00:16.074 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:00:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:00:16.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:16.077 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:00:16.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:00:16.084+0000 7fcaea9c9380 -1 init, newargv = 0x56228517bcf0 newargc=15 2023-12-19T08:00:16.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[47111]: starting ceph client 2023-12-19T08:00:16.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[47111]: starting fuse 2023-12-19T08:00:17.199 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:00:17.240 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:00:17.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:00:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:00:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:00:17.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:17.317 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:00:17.380 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:00:17.380 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:00:17.380 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:00:17.380 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:00:17.381 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:00:17.382 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:00:17.543 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.47111.asok 2023-12-19T08:00:17.724 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47111.asok 2023-12-19T08:00:17.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:17.725 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47111.asok status 2023-12-19T08:00:17.904 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "47111", 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:00:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 6102, 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 6102 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2497009652 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:17.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2497009652 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.6102 192.168.0.1:0/2497009652", 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2497009652", 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 24, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 68, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:00:17.907 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:00:17.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:17.917 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:00:17.917 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:17.938 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:00:17.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:17.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:17.938 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:00:18.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:00:18.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.224+0000 7ff547ecd700 1 Processor -- start 2023-12-19T08:00:18.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.224+0000 7ff547ecd700 1 -- start start 2023-12-19T08:00:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff547ecd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff5400cc670 0x7ff540069d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff547ecd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5400cd660 con 0x7ff540157c20 2023-12-19T08:00:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff547ecd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff540158070 con 0x7ff5400ad4b0 2023-12-19T08:00:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff547ecd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5400c8ee0 con 0x7ff5400cc670 2023-12-19T08:00:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545468700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff540157c20 0x7ff5400c9330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42618/0 (socket says 172.21.15.111:42618) 2023-12-19T08:00:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545468700 1 -- 172.21.15.111:0/2071937736 learned_addr learned my addr 172.21.15.111:0/2071937736 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff54646a700 1 --2- 172.21.15.111:0/2071937736 >> v2:172.21.15.143:3300/0 conn(0x7ff5400cc670 0x7ff540069d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107867241 0 0) 0x7ff540158070 con 0x7ff5400ad4b0 2023-12-19T08:00:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff52c003570 con 0x7ff5400ad4b0 2023-12-19T08:00:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597498909 0 0) 0x7ff5400cd660 con 0x7ff540157c20 2023-12-19T08:00:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540158070 con 0x7ff540157c20 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707694735 0 0) 0x7ff52c003570 con 0x7ff5400ad4b0 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 >> v2:172.21.15.143:3300/0 conn(0x7ff5400cc670 msgr2=0x7ff540069d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 --2- 172.21.15.111:0/2071937736 >> v2:172.21.15.143:3300/0 conn(0x7ff5400cc670 0x7ff540069d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 >> v1:172.21.15.111:6789/0 conn(0x7ff540157c20 legacy=0x7ff5400c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400ad880 con 0x7ff5400ad4b0 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7ff53c001470 con 0x7ff5400ad4b0 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 >> v1:172.21.15.143:6789/0 conn(0x7ff5400ad4b0 legacy=0x7ff540157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 0x7ff52c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c004810 0x7ff52c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff52c0051c0 0x7ff52c00fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400c8ee0 con 0x7ff52c003bf0 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c0104c0 con 0x7ff52c004810 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c010680 con 0x7ff52c0051c0 2023-12-19T08:00:18.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545c69700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 0x7ff52c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff54646a700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff52c0051c0 0x7ff52c00fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545c69700 1 -- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff52c0051c0 msgr2=0x7ff52c00fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545c69700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff52c0051c0 0x7ff52c00fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545c69700 1 -- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c004810 msgr2=0x7ff52c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.228+0000 7ff545468700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c004810 0x7ff52c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff545c69700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c004810 0x7ff52c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff54646a700 1 -- 172.21.15.111:0/2071937736 reap_dead start 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff545c69700 1 -- 172.21.15.111:0/2071937736 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c003570 con 0x7ff52c003bf0 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff545468700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c004810 0x7ff52c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:18.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff545c69700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 0x7ff52c003fe0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff53c007fd0 tx=0x7ff5400c8e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0d61586ab3476 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00cae0 con 0x7ff52c003bf0 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c00cc40 con 0x7ff52c003bf0 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff544c67700 1 -- 172.21.15.111:0/2071937736 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c006820 con 0x7ff52c003bf0 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 -- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 msgr2=0x7ff52c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 0x7ff52c003fe0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff53c007fd0 tx=0x7ff5400c8e50 comp rx=0 tx=0).stop 2023-12-19T08:00:18.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 -- 172.21.15.111:0/2071937736 shutdown_connections 2023-12-19T08:00:18.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2071937736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c003bf0 0x7ff52c003fe0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 -- 172.21.15.111:0/2071937736 >> 172.21.15.111:0/2071937736 conn(0x7ff54006c7c0 msgr2=0x7ff54006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:18.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 -- 172.21.15.111:0/2071937736 shutdown_connections 2023-12-19T08:00:18.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.232+0000 7ff547ecd700 1 -- 172.21.15.111:0/2071937736 wait complete. 2023-12-19T08:00:18.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 Processor -- start 2023-12-19T08:00:18.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 -- start start 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c003bf0 0x7ff540164ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5400b1cf0 0x7ff5400c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400b3440 con 0x7ff540149a20 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400c6c20 con 0x7ff52c003bf0 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff547ecd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400c68d0 con 0x7ff5400b1cf0 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46324/0 (socket says 172.21.15.111:46324) 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 -- 172.21.15.111:0/2944354744 learned_addr learned my addr 172.21.15.111:0/2944354744 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545c69700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c003bf0 0x7ff540164ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff54646a700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5400b1cf0 0x7ff5400c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 -- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5400b1cf0 msgr2=0x7ff5400c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5400b1cf0 0x7ff5400c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 -- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c003bf0 msgr2=0x7ff540164ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.236+0000 7ff545468700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c003bf0 0x7ff540164ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff545468700 1 -- 172.21.15.111:0/2944354744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400b3670 con 0x7ff540149a20 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff545468700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff53000d830 tx=0x7ff530009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53000b070 con 0x7ff540149a20 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff530005bc0 con 0x7ff540149a20 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5400b3950 con 0x7ff540149a20 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5300070d0 con 0x7ff540149a20 2023-12-19T08:00:18.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540156310 con 0x7ff540149a20 2023-12-19T08:00:18.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5400b3dd0 con 0x7ff540149a20 2023-12-19T08:00:18.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.240+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff5300074c0 con 0x7ff540149a20 2023-12-19T08:00:18.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.244+0000 7ff535ffb700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 0x7ff524062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.244+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ff53007c420 con 0x7ff540149a20 2023-12-19T08:00:18.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.244+0000 7ff545c69700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 0x7ff524062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.244+0000 7ff545c69700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 0x7ff524062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0010e0 tx=0x7ff53c016000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.248+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff53004abe0 con 0x7ff540149a20 2023-12-19T08:00:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.444+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5400c6a90 con 0x7ff524060670 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.448+0000 7ff535ffb700 1 -- 172.21.15.111:0/2944354744 <== mgr.5674 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff5400c6a90 con 0x7ff524060670 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:00:18.448 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 msgr2=0x7ff524062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 0x7ff524062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0010e0 tx=0x7ff53c016000 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 msgr2=0x7ff5400b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff53000d830 tx=0x7ff530009040 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 shutdown_connections 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff524060670 0x7ff524062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5400b1cf0 0x7ff5400c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff540149a20 0x7ff5400b17b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 --2- 172.21.15.111:0/2944354744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff52c003bf0 0x7ff540164ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 >> 172.21.15.111:0/2944354744 conn(0x7ff54006c7c0 msgr2=0x7ff5400c4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 shutdown_connections 2023-12-19T08:00:18.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.452+0000 7ff547ecd700 1 -- 172.21.15.111:0/2944354744 wait complete. 2023-12-19T08:00:18.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:00:18.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.636+0000 7fc3011cc700 1 Processor -- start 2023-12-19T08:00:18.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.636+0000 7fc3011cc700 1 -- start start 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3011cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2fc141e70 0x7fc2fc0c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3011cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c53f0 0x7fc2fc15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3011cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc159b80 con 0x7fc2fc0c7a80 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3011cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2fc0c8220 con 0x7fc2fc141e70 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3011cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0c7f40 con 0x7fc2fc0c53f0 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2fc141e70 0x7fc2fc0c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2fc141e70 0x7fc2fc0c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46340/0 (socket says 172.21.15.111:46340) 2023-12-19T08:00:18.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 -- 172.21.15.111:0/1191626485 learned_addr learned my addr 172.21.15.111:0/1191626485 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:18.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc3009cb700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c53f0 0x7fc2fc15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 -- 172.21.15.111:0/1191626485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c53f0 msgr2=0x7fc2fc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:18.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c53f0 0x7fc2fc15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 -- 172.21.15.111:0/1191626485 >> v1:172.21.15.143:6789/0 conn(0x7fc2fc0c7a80 legacy=0x7fc2fc0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 -- 172.21.15.111:0/1191626485 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc159530 con 0x7fc2fc141e70 2023-12-19T08:00:18.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc0c7540 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0099b0 tx=0x7fc2ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4d7975e98fdf23 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2faffd700 1 -- 172.21.15.111:0/1191626485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00b860 con 0x7fc2fc141e70 2023-12-19T08:00:18.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2faffd700 1 -- 172.21.15.111:0/1191626485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec00b9c0 con 0x7fc2fc141e70 2023-12-19T08:00:18.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.640+0000 7fc2faffd700 1 -- 172.21.15.111:0/1191626485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00bd60 con 0x7fc2fc141e70 2023-12-19T08:00:18.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- 172.21.15.111:0/1191626485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 msgr2=0x7fc2fc0c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc0c7540 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0099b0 tx=0x7fc2ec009250 comp rx=0 tx=0).stop 2023-12-19T08:00:18.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- 172.21.15.111:0/1191626485 shutdown_connections 2023-12-19T08:00:18.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c53f0 0x7fc2fc15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1191626485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc0c7540 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- 172.21.15.111:0/1191626485 >> 172.21.15.111:0/1191626485 conn(0x7fc2fc067b50 msgr2=0x7fc2fc0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:18.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- 172.21.15.111:0/1191626485 shutdown_connections 2023-12-19T08:00:18.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- 172.21.15.111:0/1191626485 wait complete. 2023-12-19T08:00:18.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 Processor -- start 2023-12-19T08:00:18.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.644+0000 7fc3011cc700 1 -- start start 2023-12-19T08:00:18.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 0x7fc2fc16bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 0x7fc2fc13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc13f7b0 con 0x7fc2fc141e70 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc13eba0 con 0x7fc2fc0c53f0 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc13ee80 con 0x7fc2fc0c7a80 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46344/0 (socket says 172.21.15.111:46344) 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 -- 172.21.15.111:0/1245935855 learned_addr learned my addr 172.21.15.111:0/1245935855 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:18.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 0x7fc2fc16bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2fb7fe700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 0x7fc2fc13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 -- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 msgr2=0x7fc2fc13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 0x7fc2fc13e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 -- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 msgr2=0x7fc2fc16bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 0x7fc2fc16bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 -- 172.21.15.111:0/1245935855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0094f0 con 0x7fc2fc141e70 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 0x7fc2fc16bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2fb7fe700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 0x7fc2fc13e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3009cb700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4007f20 tx=0x7fc2f40098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f400eca0 con 0x7fc2fc141e70 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc13f160 con 0x7fc2fc141e70 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f400ee20 con 0x7fc2fc141e70 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4014670 con 0x7fc2fc141e70 2023-12-19T08:00:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.648+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc16c4b0 con 0x7fc2fc141e70 2023-12-19T08:00:18.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.652+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc04be40 con 0x7fc2fc141e70 2023-12-19T08:00:18.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.652+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc2f4014a60 con 0x7fc2fc141e70 2023-12-19T08:00:18.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.652+0000 7fc2e3fff700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 0x7fc2e8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:18.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.656+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fc2f407d3f0 con 0x7fc2fc141e70 2023-12-19T08:00:18.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.656+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 0x7fc2e8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:18.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.656+0000 7fc2fbfff700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 0x7fc2e8062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0099b0 tx=0x7fc2ec017000 comp rx=0 tx=0).ready entity=mgr.5674 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:18.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.664+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2f404d5a0 con 0x7fc2fc141e70 2023-12-19T08:00:18.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:18.876+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc2fc159810 con 0x7fc2fc141e70 2023-12-19T08:00:19.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.400+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fc2f404d970 con 0x7fc2fc141e70 2023-12-19T08:00:19.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.400+0000 7fc2e3fff700 1 -- 172.21.15.111:0/1245935855 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc2f4049f90 con 0x7fc2fc141e70 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 msgr2=0x7fc2e8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 0x7fc2e8062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0099b0 tx=0x7fc2ec017000 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 msgr2=0x7fc2fc140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4007f20 tx=0x7fc2f40098a0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 shutdown_connections 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc2e8060670 0x7fc2e8062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2fc141e70 0x7fc2fc140130 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2fc0c7a80 0x7fc2fc13e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 --2- 172.21.15.111:0/1245935855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2fc0c53f0 0x7fc2fc16bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 >> 172.21.15.111:0/1245935855 conn(0x7fc2fc067b50 msgr2=0x7fc2fc13b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 shutdown_connections 2023-12-19T08:00:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.404+0000 7fc3011cc700 1 -- 172.21.15.111:0/1245935855 wait complete. 2023-12-19T08:00:19.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:00:19.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.580+0000 7f1b9902e700 1 Processor -- start 2023-12-19T08:00:19.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.580+0000 7f1b9902e700 1 -- start start 2023-12-19T08:00:19.581 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:19.580+0000 7f0b2a35d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:00:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b94159790 0x7f1b940684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b94067790 con 0x7f1b940b1ed0 2023-12-19T08:00:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b94068030 con 0x7f1b940b0cb0 2023-12-19T08:00:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b940b1200 con 0x7f1b94159790 2023-12-19T08:00:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b92d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b94159790 0x7f1b940684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9359e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1b940b1ed0 0x7f1b94159b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42628/0 (socket says 172.21.15.111:42628) 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9359e700 1 -- 172.21.15.111:0/1860650576 learned_addr learned my addr 172.21.15.111:0/1860650576 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b91d9b700 1 -- 172.21.15.111:0/1860650576 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598749380 0 0) 0x7f1b94067790 con 0x7f1b940b1ed0 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b91d9b700 1 -- 172.21.15.111:0/1860650576 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c003570 con 0x7f1b940b1ed0 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1860650576 >> v1:172.21.15.143:6789/0 conn(0x7f1b940b0cb0 legacy=0x7f1b940b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1860650576 >> v1:172.21.15.111:6789/0 conn(0x7f1b940b1ed0 legacy=0x7f1b94159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1860650576 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940679e0 con 0x7f1b94159790 2023-12-19T08:00:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b92d9d700 1 --2- 172.21.15.111:0/1860650576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b94159790 0x7f1b940684c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1b840098e0 tx=0x7f1b840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4466c65b5d373410 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b91d9b700 1 -- 172.21.15.111:0/1860650576 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400b640 con 0x7f1b94159790 2023-12-19T08:00:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b91d9b700 1 -- 172.21.15.111:0/1860650576 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400b7a0 con 0x7f1b94159790 2023-12-19T08:00:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b91d9b700 1 -- 172.21.15.111:0/1860650576 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400bb40 con 0x7f1b94159790 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 -- 172.21.15.111:0/1860650576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b94159790 msgr2=0x7f1b940684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.584+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1860650576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b94159790 0x7f1b940684c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1b840098e0 tx=0x7f1b840091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 -- 172.21.15.111:0/1860650576 shutdown_connections 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1860650576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b94159790 0x7f1b940684c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 -- 172.21.15.111:0/1860650576 >> 172.21.15.111:0/1860650576 conn(0x7f1b940cdef0 msgr2=0x7f1b940ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 -- 172.21.15.111:0/1860650576 shutdown_connections 2023-12-19T08:00:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 -- 172.21.15.111:0/1860650576 wait complete. 2023-12-19T08:00:19.588 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:19.589+0000 7fd1c5161000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:00:19.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 Processor -- start 2023-12-19T08:00:19.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.588+0000 7f1b9902e700 1 -- start start 2023-12-19T08:00:19.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 0x7f1b9416d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b94159790 0x7f1b940a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940a6bc0 con 0x7f1b940b0cb0 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940a6dd0 con 0x7f1b940b1ed0 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940a5c70 con 0x7f1b94159790 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46388/0 (socket says 172.21.15.111:46388) 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1511910506 learned_addr learned my addr 172.21.15.111:0/1511910506 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9359e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b94159790 0x7f1b940a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9259c700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 0x7f1b9416d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b94159790 msgr2=0x7f1b940a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b94159790 0x7f1b940a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 msgr2=0x7f1b9416d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 0x7f1b9416d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 -- 172.21.15.111:0/1511910506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84009420 con 0x7f1b940b0cb0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9259c700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 0x7f1b9416d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b92d9d700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1b9406bce0 tx=0x7f1b84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84000d10 con 0x7f1b940b0cb0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400bb20 con 0x7f1b940b0cb0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b940a5380 con 0x7f1b940b0cb0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84011650 con 0x7f1b940b0cb0 2023-12-19T08:00:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.592+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b940a5900 con 0x7f1b940b0cb0 2023-12-19T08:00:19.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.596+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b94067d30 con 0x7f1b940b0cb0 2023-12-19T08:00:19.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.596+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f1b84011a00 con 0x7f1b940b0cb0 2023-12-19T08:00:19.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.600+0000 7f1b7affd700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 0x7f1b80062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.600+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f1b8407e300 con 0x7f1b940b0cb0 2023-12-19T08:00:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.600+0000 7f1b9259c700 1 -- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 msgr2=0x7f1b80062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.600+0000 7f1b9259c700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 0x7f1b80062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:00:19.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.608+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1b94067d30 con 0x7f1b940b0cb0 2023-12-19T08:00:19.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.780+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1b94067d30 con 0x7f1b940b0cb0 2023-12-19T08:00:19.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.788+0000 7f1b7affd700 1 -- 172.21.15.111:0/1511910506 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b94067d30 con 0x7f1b940b0cb0 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 msgr2=0x7f1b80062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 0x7f1b80062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 msgr2=0x7f1b940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1b9406bce0 tx=0x7f1b84017000 comp rx=0 tx=0).stop 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 shutdown_connections 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1b800606c0 0x7f1b80062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b940b1ed0 0x7f1b9416d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b94159790 0x7f1b940a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 --2- 172.21.15.111:0/1511910506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b940b0cb0 0x7f1b940a6100 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 >> 172.21.15.111:0/1511910506 conn(0x7f1b940cdef0 msgr2=0x7f1b94158800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:19.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 shutdown_connections 2023-12-19T08:00:19.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.792+0000 7f1b9902e700 1 -- 172.21.15.111:0/1511910506 wait complete. 2023-12-19T08:00:19.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:00:19.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.988+0000 7f92a2f45700 1 Processor -- start 2023-12-19T08:00:19.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.988+0000 7f92a2f45700 1 -- start start 2023-12-19T08:00:19.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a2f45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f929c158e20 0x7f929c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a2f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a2f45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c0cbdf0 con 0x7f929c0ca2c0 2023-12-19T08:00:19.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a2f45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c0cc0d0 con 0x7f929c158e20 2023-12-19T08:00:19.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a2f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0cc3b0 con 0x7f929c0cb970 2023-12-19T08:00:19.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:44580/0 (socket says 172.21.15.111:44580) 2023-12-19T08:00:19.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 -- 172.21.15.111:0/400327487 learned_addr learned my addr 172.21.15.111:0/400327487 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:19.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f92a0ce1700 1 --2- 172.21.15.111:0/400327487 >> v2:172.21.15.143:3300/0 conn(0x7f929c158e20 0x7f929c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 -- 172.21.15.111:0/400327487 >> v2:172.21.15.143:3300/0 conn(0x7f929c158e20 msgr2=0x7f929c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 --2- 172.21.15.111:0/400327487 >> v2:172.21.15.143:3300/0 conn(0x7f929c158e20 0x7f929c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 -- 172.21.15.111:0/400327487 >> v1:172.21.15.143:6789/0 conn(0x7f929c0ca2c0 legacy=0x7f929c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 -- 172.21.15.111:0/400327487 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c0cc690 con 0x7f929c0cb970 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929bfff700 1 --2- 172.21.15.111:0/400327487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f92880099b0 tx=0x7f9288009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a78f37a6b8f63e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929b7fe700 1 -- 172.21.15.111:0/400327487 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928800b860 con 0x7f929c0cb970 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929b7fe700 1 -- 172.21.15.111:0/400327487 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928800b9c0 con 0x7f929c0cb970 2023-12-19T08:00:19.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.992+0000 7f929b7fe700 1 -- 172.21.15.111:0/400327487 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928800bd60 con 0x7f929c0cb970 2023-12-19T08:00:19.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- 172.21.15.111:0/400327487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 msgr2=0x7f929c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:19.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- 172.21.15.111:0/400327487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f92880099b0 tx=0x7f9288009250 comp rx=0 tx=0).stop 2023-12-19T08:00:19.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- 172.21.15.111:0/400327487 shutdown_connections 2023-12-19T08:00:19.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- 172.21.15.111:0/400327487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c0cb970 0x7f929c0ceb00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- 172.21.15.111:0/400327487 >> v2:172.21.15.143:3300/0 conn(0x7f929c158e20 0x7f929c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:19.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- 172.21.15.111:0/400327487 >> 172.21.15.111:0/400327487 conn(0x7f929c067bb0 msgr2=0x7f929c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:19.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- 172.21.15.111:0/400327487 shutdown_connections 2023-12-19T08:00:19.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- 172.21.15.111:0/400327487 wait complete. 2023-12-19T08:00:19.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 Processor -- start 2023-12-19T08:00:19.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- start start 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c158e20 0x7f929c0c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 0x7f929c14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c153f60 con 0x7f929c0b5090 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0b5590 con 0x7f929c0ca2c0 2023-12-19T08:00:19.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:19.996+0000 7f92a2f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0b5870 con 0x7f929c158e20 2023-12-19T08:00:19.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:19.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:53378/0 (socket says 172.21.15.111:53378) 2023-12-19T08:00:19.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 -- 172.21.15.111:0/2190726164 learned_addr learned my addr 172.21.15.111:0/2190726164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a14e2700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 0x7f929c14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f929bfff700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c158e20 0x7f929c0c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 -- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c158e20 msgr2=0x7f929c0c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c158e20 0x7f929c0c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 -- 172.21.15.111:0/2190726164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 msgr2=0x7f929c14f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 0x7f929c14f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 -- 172.21.15.111:0/2190726164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880094f0 con 0x7f929c0ca2c0 2023-12-19T08:00:20.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a14e2700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 0x7f929c14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a0ce1700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f929000d860 tx=0x7f9290008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92900125f0 con 0x7f929c0ca2c0 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9290012750 con 0x7f929c0ca2c0 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c0b5b50 con 0x7f929c0ca2c0 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290012b10 con 0x7f929c0ca2c0 2023-12-19T08:00:20.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c0b5e30 con 0x7f929c0ca2c0 2023-12-19T08:00:20.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.000+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c0ca110 con 0x7f929c0ca2c0 2023-12-19T08:00:20.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.004+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f929001e460 con 0x7f929c0ca2c0 2023-12-19T08:00:20.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.004+0000 7f9298ff9700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 0x7f9280062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.004+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f929007d260 con 0x7f929c0ca2c0 2023-12-19T08:00:20.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.004+0000 7f929bfff700 1 -- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 msgr2=0x7f9280062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:20.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.004+0000 7f929bfff700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 0x7f9280062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:00:20.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.012+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9290050ea0 con 0x7f929c0ca2c0 2023-12-19T08:00:20.115 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.117+0000 7fd1c5161000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:00:20.144 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.144+0000 7f0b2a35d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:00:20.183 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.185+0000 7fd1c5161000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:00:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.188+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f929c06a7a0 con 0x7f929c0ca2c0 2023-12-19T08:00:20.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.200+0000 7f9298ff9700 1 -- 172.21.15.111:0/2190726164 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9290050a30 con 0x7f929c0ca2c0 2023-12-19T08:00:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 msgr2=0x7f9280062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:00:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 0x7f9280062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 msgr2=0x7f929c1691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f929000d860 tx=0x7f9290008040 comp rx=0 tx=0).stop 2023-12-19T08:00:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 shutdown_connections 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f92800606c0 0x7f9280062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929c0b5090 0x7f929c14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f929c158e20 0x7f929c0c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 --2- 172.21.15.111:0/2190726164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929c0ca2c0 0x7f929c1691f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 >> 172.21.15.111:0/2190726164 conn(0x7f929c067bb0 msgr2=0x7f929c0c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 shutdown_connections 2023-12-19T08:00:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.204+0000 7f92a2f45700 1 -- 172.21.15.111:0/2190726164 wait complete. 2023-12-19T08:00:20.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:20.227 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T08:00:20.228 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.228+0000 7f0b2a35d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:00:20.256 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T08:00:20.316 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.321+0000 7fd1c5161000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:00:20.385 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.384+0000 7f0b2a35d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:00:20.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.476+0000 7f182c265700 1 Processor -- start 2023-12-19T08:00:20.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.476+0000 7f182c265700 1 -- start start 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182c265700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f182415c930 0x7f1824163e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182c265700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240b0ac0 con 0x7f18240cb500 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182c265700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182406b210 con 0x7f18240cb130 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182c265700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240ca810 con 0x7f182415c930 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1829800700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f18240cb500 0x7f182415c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42634/0 (socket says 172.21.15.111:42634) 2023-12-19T08:00:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1829800700 1 -- 172.21.15.111:0/3527525916 learned_addr learned my addr 172.21.15.111:0/3527525916 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:00:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 --2- 172.21.15.111:0/3527525916 >> v2:172.21.15.143:3300/0 conn(0x7f182415c930 0x7f1824163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295473722 0 0) 0x7f18240b0ac0 con 0x7f18240cb500 2023-12-19T08:00:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1810003570 con 0x7f18240cb500 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740173878 0 0) 0x7f1810003570 con 0x7f18240cb500 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 >> v2:172.21.15.143:3300/0 conn(0x7f182415c930 msgr2=0x7f1824163e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 --2- 172.21.15.111:0/3527525916 >> v2:172.21.15.143:3300/0 conn(0x7f182415c930 0x7f1824163e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 >> v1:172.21.15.143:6789/0 conn(0x7f18240cb130 legacy=0x7f182406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240b0190 con 0x7f18240cb500 2023-12-19T08:00:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f1814001470 con 0x7f18240cb500 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 >> v1:172.21.15.111:6789/0 conn(0x7f18240cb500 legacy=0x7f182415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1810003a60 0x7f1810003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1810004810 0x7f1810004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 0x7f181000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240ca810 con 0x7f18100051c0 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181000c050 con 0x7f1810003a60 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181000c210 con 0x7f1810004810 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 0x7f181000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1829800700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1810004810 0x7f1810004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a001700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1810003a60 0x7f1810003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 -- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1810004810 msgr2=0x7f1810004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1810004810 0x7f1810004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 -- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1810003a60 msgr2=0x7f1810003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1810003a60 0x7f1810003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 -- 172.21.15.111:0/3527525916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1810003570 con 0x7f18100051c0 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a001700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1810003a60 0x7f1810003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f1829800700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1810004810 0x7f1810004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 -- 172.21.15.111:0/3527525916 reap_dead start 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.480+0000 7f182a802700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 0x7f181000ba60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f182415cd30 tx=0x7f181c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2833b14e12ad5a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00ea90 con 0x7f18100051c0 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c00f040 con 0x7f18100051c0 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f1828fff700 1 -- 172.21.15.111:0/3527525916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00ee40 con 0x7f18100051c0 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 -- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 msgr2=0x7f181000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 0x7f181000ba60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f182415cd30 tx=0x7f181c00b040 comp rx=0 tx=0).stop 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 -- 172.21.15.111:0/3527525916 shutdown_connections 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 --2- 172.21.15.111:0/3527525916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18100051c0 0x7f181000ba60 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 -- 172.21.15.111:0/3527525916 >> 172.21.15.111:0/3527525916 conn(0x7f1824067bb0 msgr2=0x7f18240cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 -- 172.21.15.111:0/3527525916 shutdown_connections 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 -- 172.21.15.111:0/3527525916 wait complete. 2023-12-19T08:00:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.484+0000 7f182c265700 1 Processor -- start 2023-12-19T08:00:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 -- start start 2023-12-19T08:00:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18100051c0 0x7f18240c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18241570a0 0x7f18240c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240c7c20 con 0x7f18240c8590 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182415dae0 con 0x7f18100051c0 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240c5cc0 con 0x7f18241570a0 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a001700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18100051c0 0x7f18240c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46420/0 (socket says 172.21.15.111:46420) 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 -- 172.21.15.111:0/2096867211 learned_addr learned my addr 172.21.15.111:0/2096867211 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f1829800700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18241570a0 0x7f18240c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18241570a0 msgr2=0x7f18240c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18241570a0 0x7f18240c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18100051c0 msgr2=0x7f18240c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18100051c0 0x7f18240c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240ca190 con 0x7f18240c8590 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182a802700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f182406e320 tx=0x7f181c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c01d650 con 0x7f18240c8590 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c01d7b0 con 0x7f18240c8590 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182415dd30 con 0x7f18240c8590 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.488+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c01db70 con 0x7f18240c8590 2023-12-19T08:00:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.492+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1824157ae0 con 0x7f18240c8590 2023-12-19T08:00:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.492+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18240c7d70 con 0x7f18240c8590 2023-12-19T08:00:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.492+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f181c003020 con 0x7f18240c8590 2023-12-19T08:00:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.496+0000 7f1819ffb700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.496+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f181c07e880 con 0x7f18240c8590 2023-12-19T08:00:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.496+0000 7f182a001700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.496+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:00:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.500+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18240c7d70 con 0x7f18240c8590 2023-12-19T08:00:20.570 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.573+0000 7fd1c5161000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:00:20.641 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.645+0000 7fd1c5161000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:00:20.666 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.664+0000 7f0b2a35d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:00:20.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.688+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f18240c7d70 con 0x7f18080606f0 2023-12-19T08:00:20.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.700+0000 7f182a001700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:20.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:20.700+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:00:20.744 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.744+0000 7f0b2a35d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:00:20.861 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.865+0000 7fd1c5161000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:00:20.944 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:20.949+0000 7fd1c5161000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:00:20.960 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:20.960+0000 7f0b2a35d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:00:21.034 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:21.037+0000 7fd1c5161000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:00:21.039 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:21.040+0000 7f0b2a35d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:00:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:21.104+0000 7f182a001700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:21.104+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:00:21.126 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:21.124+0000 7f0b2a35d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:00:21.440 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:21.441+0000 7fd1c5161000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:00:21.527 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:21.528+0000 7f0b2a35d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:00:21.604 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:21.605+0000 7fd1c5161000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:00:21.693 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:21.692+0000 7f0b2a35d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:00:21.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:21.908+0000 7f182a001700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:21.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:21.908+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:00:21.973 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:21.977+0000 7fd1c5161000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:00:22.068 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.068+0000 7f0b2a35d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:00:22.191 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.193+0000 7fd1c5161000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:00:22.283 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.285+0000 7fd1c5161000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:00:22.284 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.284+0000 7f0b2a35d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:00:22.360 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.361+0000 7fd1c5161000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:00:22.373 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.372+0000 7f0b2a35d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:00:22.449 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.448+0000 7f0b2a35d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:00:22.450 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.453+0000 7fd1c5161000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:00:22.531 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.532+0000 7f0b2a35d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:00:22.671 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.673+0000 7fd1c5161000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:00:22.744 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.745+0000 7fd1c5161000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:00:22.747 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.748+0000 7f0b2a35d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:00:22.824 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.824+0000 7f0b2a35d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:00:22.834 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:22.837+0000 7fd1c5161000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:00:22.910 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:22.908+0000 7f0b2a35d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:00:23.028 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:23.029+0000 7fd1c5161000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:00:23.093 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:23.092+0000 7f0b2a35d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:00:23.116 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:23.117+0000 7fd1c5161000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:00:23.179 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:23.180+0000 7f0b2a35d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:00:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:23.512+0000 7f182a001700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:00:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:23.512+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:00:23.825 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:23.829+0000 7fd1c5161000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:00:23.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:23.840+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 18) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f181c07ca60 con 0x7f18240c8590 2023-12-19T08:00:23.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:23.844+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:00:23.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:23.844+0000 7f1819ffb700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:23.891 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:00:23.892+0000 7f0b2a35d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:00:24.021 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd186c3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.021 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd186c3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.021 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd186c3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd186c3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd186c3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd182c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd182c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd182c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd182c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:00:24.025+0000 7fd182c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:00:24.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.852+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f181c04ecf0 con 0x7f18240c8590 2023-12-19T08:00:24.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.852+0000 7f1819ffb700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:24.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.852+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f18240c7d70 con 0x7f18080606f0 2023-12-19T08:00:24.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.856+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:24.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.860+0000 7f182a001700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1820009010 tx=0x7f1820009040 comp rx=0 tx=0).ready entity=mgr.5591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.960+0000 7f1819ffb700 1 -- 172.21.15.111:0/2096867211 <== mgr.5591 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f18240c7d70 con 0x7f18080606f0 2023-12-19T08:00:24.961 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 msgr2=0x7f1808062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1820009010 tx=0x7f1820009040 comp rx=0 tx=0).stop 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 msgr2=0x7f18240c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f182406e320 tx=0x7f181c019000 comp rx=0 tx=0).stop 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 shutdown_connections 2023-12-19T08:00:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f18080606f0 0x7f1808062ba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18240c8590 0x7f18240c8a00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18241570a0 0x7f18240c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 --2- 172.21.15.111:0/2096867211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18100051c0 0x7f18240c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 >> 172.21.15.111:0/2096867211 conn(0x7f1824067bb0 msgr2=0x7f18240cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 shutdown_connections 2023-12-19T08:00:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:24.964+0000 7f182c265700 1 -- 172.21.15.111:0/2096867211 wait complete. 2023-12-19T08:00:24.989 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T08:00:24.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 2M 2023-12-19T08:00:25.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.164+0000 7f8ec1c16700 1 Processor -- start 2023-12-19T08:00:25.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.164+0000 7f8ec1c16700 1 -- start start 2023-12-19T08:00:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1c16700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ebc158da0 0x7f8ebc06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1c16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc15d720 con 0x7f8ebc06df10 2023-12-19T08:00:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1c16700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0ca290 con 0x7f8ebc158da0 2023-12-19T08:00:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0ca4a0 con 0x7f8ebc0caca0 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec0c14700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ebc158da0 0x7f8ebc06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:33540/0 (socket says 172.21.15.111:33540) 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 -- 172.21.15.111:0/1618283436 learned_addr learned my addr 172.21.15.111:0/1618283436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 -- 172.21.15.111:0/1618283436 >> v2:172.21.15.143:3300/0 conn(0x7f8ebc158da0 msgr2=0x7f8ebc06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 --2- 172.21.15.111:0/1618283436 >> v2:172.21.15.143:3300/0 conn(0x7f8ebc158da0 0x7f8ebc06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 -- 172.21.15.111:0/1618283436 >> v1:172.21.15.143:6789/0 conn(0x7f8ebc06df10 legacy=0x7f8ebc06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 -- 172.21.15.111:0/1618283436 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0ca780 con 0x7f8ebc0caca0 2023-12-19T08:00:25.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ec1415700 1 --2- 172.21.15.111:0/1618283436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009f20 tx=0x7f8eb4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17bf8213dcec2fcb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ebb7fe700 1 -- 172.21.15.111:0/1618283436 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400e070 con 0x7f8ebc0caca0 2023-12-19T08:00:25.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ebb7fe700 1 -- 172.21.15.111:0/1618283436 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400b9d0 con 0x7f8ebc0caca0 2023-12-19T08:00:25.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.168+0000 7f8ebb7fe700 1 -- 172.21.15.111:0/1618283436 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40092e0 con 0x7f8ebc0caca0 2023-12-19T08:00:25.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 -- 172.21.15.111:0/1618283436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 msgr2=0x7f8ebc06ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/1618283436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009f20 tx=0x7f8eb4009840 comp rx=0 tx=0).stop 2023-12-19T08:00:25.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 -- 172.21.15.111:0/1618283436 shutdown_connections 2023-12-19T08:00:25.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/1618283436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc06ebb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/1618283436 >> v2:172.21.15.143:3300/0 conn(0x7f8ebc158da0 0x7f8ebc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 -- 172.21.15.111:0/1618283436 >> 172.21.15.111:0/1618283436 conn(0x7f8ebc0cc760 msgr2=0x7f8ebc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:25.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 -- 172.21.15.111:0/1618283436 shutdown_connections 2023-12-19T08:00:25.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.172+0000 7f8ec1c16700 1 -- 172.21.15.111:0/1618283436 wait complete. 2023-12-19T08:00:25.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 Processor -- start 2023-12-19T08:00:25.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 -- start start 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ebc158da0 0x7f8ebc152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b3870 con 0x7f8ebc06df10 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b3b50 con 0x7f8ebc158da0 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b3e30 con 0x7f8ebc0caca0 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50700/0 (socket says 172.21.15.111:50700) 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 -- 172.21.15.111:0/2038587425 learned_addr learned my addr 172.21.15.111:0/2038587425 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec1415700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ebc158da0 0x7f8ebc152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ebbfff700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 -- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 msgr2=0x7f8ebc152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 -- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ebc158da0 msgr2=0x7f8ebc152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ebc158da0 0x7f8ebc152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.176+0000 7f8ec0c14700 1 -- 172.21.15.111:0/2038587425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4009ae0 con 0x7f8ebc06df10 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8ec0c14700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000e930 tx=0x7f8eb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0009510 con 0x7f8ebc06df10 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb000c850 con 0x7f8ebc06df10 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000cc10 con 0x7f8ebc06df10 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc0b4110 con 0x7f8ebc06df10 2023-12-19T08:00:25.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc0b3450 con 0x7f8ebc06df10 2023-12-19T08:00:25.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f8eb0013470 con 0x7f8ebc06df10 2023-12-19T08:00:25.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.180+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc04be40 con 0x7f8ebc06df10 2023-12-19T08:00:25.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.184+0000 7f8eb8ff9700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 0x7f8ea00628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.184+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f8eb007cb10 con 0x7f8ebc06df10 2023-12-19T08:00:25.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.184+0000 7f8ebbfff700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 0x7f8ea00628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.184+0000 7f8ebbfff700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 0x7f8ea00628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0b0af0 tx=0x7f8ea8009040 comp rx=0 tx=0).ready entity=mgr.5591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.188+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8eb004cf30 con 0x7f8ebc06df10 2023-12-19T08:00:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.380+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "2M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ebc06f360 con 0x7f8ea00603f0 2023-12-19T08:00:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.484+0000 7f8eb8ff9700 1 -- 172.21.15.111:0/2038587425 <== mgr.5591 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ebc06f360 con 0x7f8ea00603f0 2023-12-19T08:00:25.484 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 msgr2=0x7f8ea00628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 0x7f8ea00628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0b0af0 tx=0x7f8ea8009040 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 msgr2=0x7f8ebc150050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000e930 tx=0x7f8eb00091a0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 shutdown_connections 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8ea00603f0 0x7f8ea00628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ebc158da0 0x7f8ebc152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ebc0caca0 0x7f8ebc152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 --2- 172.21.15.111:0/2038587425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ebc06df10 0x7f8ebc150050 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 >> 172.21.15.111:0/2038587425 conn(0x7f8ebc0cc760 msgr2=0x7f8ebc157580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 shutdown_connections 2023-12-19T08:00:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.488+0000 7f8ec1c16700 1 -- 172.21.15.111:0/2038587425 wait complete. 2023-12-19T08:00:25.513 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T08:00:25.513 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-08_01_00 in ~35s... 2023-12-19T08:00:25.513 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-08_02_00 in ~95s... 2023-12-19T08:00:25.514 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M', '2M'], expected_retention: [] 2023-12-19T08:00:25.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T08:00:25.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.688+0000 7f9068f63700 1 Processor -- start 2023-12-19T08:00:25.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.688+0000 7f9068f63700 1 -- start start 2023-12-19T08:00:25.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9068f63700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f90640c8ae0 0x7f9064157210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9068f63700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906406c7c0 con 0x7f9064157750 2023-12-19T08:00:25.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9068f63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906406ca10 con 0x7f9064156640 2023-12-19T08:00:25.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9068f63700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906406ccf0 con 0x7f90640c8ae0 2023-12-19T08:00:25.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f90640c8ae0 0x7f9064157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f90640c8ae0 0x7f9064157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50704/0 (socket says 172.21.15.111:50704) 2023-12-19T08:00:25.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 -- 172.21.15.111:0/3355859819 learned_addr learned my addr 172.21.15.111:0/3355859819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 -- 172.21.15.111:0/3355859819 >> v1:172.21.15.143:6789/0 conn(0x7f9064156640 legacy=0x7f9064156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 -- 172.21.15.111:0/3355859819 >> v1:172.21.15.111:6789/0 conn(0x7f9064157750 legacy=0x7f906406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.692+0000 7f9062ffd700 1 -- 172.21.15.111:0/3355859819 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906406cfd0 con 0x7f90640c8ae0 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9062ffd700 1 --2- 172.21.15.111:0/3355859819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90640c8ae0 0x7f9064157210 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f90580099b0 tx=0x7f9058009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d48447652147c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f90627fc700 1 -- 172.21.15.111:0/3355859819 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800b860 con 0x7f90640c8ae0 2023-12-19T08:00:25.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f90627fc700 1 -- 172.21.15.111:0/3355859819 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905800b9c0 con 0x7f90640c8ae0 2023-12-19T08:00:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f90627fc700 1 -- 172.21.15.111:0/3355859819 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800bd60 con 0x7f90640c8ae0 2023-12-19T08:00:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 -- 172.21.15.111:0/3355859819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90640c8ae0 msgr2=0x7f9064157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 --2- 172.21.15.111:0/3355859819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90640c8ae0 0x7f9064157210 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f90580099b0 tx=0x7f9058009250 comp rx=0 tx=0).stop 2023-12-19T08:00:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 -- 172.21.15.111:0/3355859819 shutdown_connections 2023-12-19T08:00:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 --2- 172.21.15.111:0/3355859819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90640c8ae0 0x7f9064157210 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 -- 172.21.15.111:0/3355859819 >> 172.21.15.111:0/3355859819 conn(0x7f9064067bb0 msgr2=0x7f90640c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:25.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 -- 172.21.15.111:0/3355859819 shutdown_connections 2023-12-19T08:00:25.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.696+0000 7f9068f63700 1 -- 172.21.15.111:0/3355859819 wait complete. 2023-12-19T08:00:25.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 Processor -- start 2023-12-19T08:00:25.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 -- start start 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90640c8ae0 0x7f9064148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9064157750 0x7f90640ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9064148fb0 con 0x7f9064156640 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90641443a0 con 0x7f90640c8ae0 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.700+0000 7f9068f63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9064146e10 con 0x7f9064157750 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50714/0 (socket says 172.21.15.111:50714) 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 -- 172.21.15.111:0/1047163792 learned_addr learned my addr 172.21.15.111:0/1047163792 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f90637fe700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90640c8ae0 0x7f9064148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9063fff700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9064157750 0x7f90640ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 -- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9064157750 msgr2=0x7f90640ac7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9064157750 0x7f90640ac7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 -- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90640c8ae0 msgr2=0x7f9064148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90640c8ae0 0x7f9064148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 -- 172.21.15.111:0/1047163792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580094f0 con 0x7f9064156640 2023-12-19T08:00:25.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9062ffd700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f90640699a0 tx=0x7f9058017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058011ea0 con 0x7f9064156640 2023-12-19T08:00:25.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905801a630 con 0x7f9064156640 2023-12-19T08:00:25.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905801a9b0 con 0x7f9064156640 2023-12-19T08:00:25.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90641470f0 con 0x7f9064156640 2023-12-19T08:00:25.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90641445f0 con 0x7f9064156640 2023-12-19T08:00:25.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.704+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064064040 con 0x7f9064156640 2023-12-19T08:00:25.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.708+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f905802d440 con 0x7f9064156640 2023-12-19T08:00:25.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.708+0000 7f904b7fe700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 0x7f90500628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:00:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.708+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6861+0+0 (secure 0 0 0) 0x7f905807e5f0 con 0x7f9064156640 2023-12-19T08:00:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.708+0000 7f90637fe700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 0x7f90500628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:00:25.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.712+0000 7f90637fe700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 0x7f90500628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90540037c0 tx=0x7f9054009040 comp rx=0 tx=0).ready entity=mgr.5591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:00:25.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.716+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f905804e9a0 con 0x7f9064156640 2023-12-19T08:00:25.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.860+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 20) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f905804ed70 con 0x7f9064156640 2023-12-19T08:00:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.924+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9064143ec0 con 0x7f90500603f0 2023-12-19T08:00:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.940+0000 7f904b7fe700 1 -- 172.21.15.111:0/1047163792 <== mgr.5591 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f9064143ec0 con 0x7f90500603f0 2023-12-19T08:00:25.942 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:00:25.942 INFO:teuthology.orchestra.run.smithi111.stdout:{"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-12-19T08:00:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 msgr2=0x7f90500628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 0x7f90500628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90540037c0 tx=0x7f9054009040 comp rx=0 tx=0).stop 2023-12-19T08:00:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 msgr2=0x7f9064143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f90640699a0 tx=0x7f9058017000 comp rx=0 tx=0).stop 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 shutdown_connections 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90500603f0 0x7f90500628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9064157750 0x7f90640ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9064156640 0x7f9064143920 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 --2- 172.21.15.111:0/1047163792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90640c8ae0 0x7f9064148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 >> 172.21.15.111:0/1047163792 conn(0x7f9064067bb0 msgr2=0x7f9064156a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 shutdown_connections 2023-12-19T08:00:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:00:25.944+0000 7f9068f63700 1 -- 172.21.15.111:0/1047163792 wait complete. 2023-12-19T08:00:25.968 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M", "2M"], "retention": [{}, {}]} 2023-12-19T08:00:25.969 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M', '2M'], 'retention': [{}, {}]} 2023-12-19T08:00:25.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:25.969 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:26.000 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:26.000 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:26.000 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:27.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:27.001 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:27.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:27.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:27.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:28.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:28.035 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:28.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:28.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:28.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:29.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:29.071 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:29.104 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:29.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:29.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:30.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:30.105 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:30.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:30.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:30.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:31.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:31.140 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:31.175 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:31.175 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:31.175 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:32.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:32.177 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:32.211 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:32.211 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:32.211 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:33.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:33.212 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:33.244 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:33.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:33.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:34.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:34.245 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:34.279 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:34.280 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:34.280 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:35.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:35.280 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:35.317 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:35.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:35.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:35.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:00:35.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:00:36.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:36.317 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:36.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:36.350 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:36.350 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:37.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:37.351 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:37.387 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:37.387 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:37.387 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:38.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:38.389 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:38.420 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:38.420 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:38.420 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:39.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:39.421 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:39.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:39.457 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:39.457 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:40.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:40.457 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:40.490 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:40.490 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:40.490 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:41.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:41.492 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:41.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:41.522 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:41.522 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:42.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:42.523 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:42.556 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:42.557 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:42.557 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:43.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:43.557 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:43.591 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:43.591 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:43.591 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:44.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:44.592 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:44.625 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:44.625 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:44.625 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:45.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:45.626 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:45.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:45.661 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:45.661 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:46.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:46.662 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:46.698 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:46.698 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:46.698 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:47.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:47.699 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:47.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:47.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:47.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:48.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:48.735 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:48.769 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:48.769 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:48.769 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:49.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:49.770 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:49.806 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:49.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:49.807 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:50.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:50.807 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:50.844 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:50.844 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:50.844 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:51.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:51.846 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:51.875 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:51.875 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:51.875 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:52.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:52.876 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:52.914 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:52.914 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:52.914 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:53.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:53.915 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:53.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:53.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:53.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:54.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:54.951 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:54.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:54.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:54.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:55.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:55.988 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:56.019 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:56.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:56.020 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:57.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:57.020 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:57.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:57.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:57.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:58.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:58.051 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:58.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:58.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:58.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:00:59.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:00:59.089 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:00:59.124 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:00:59.124 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:00:59.124 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:00.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:00.125 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:00.157 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:01:00.157 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:00.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:00.505 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:01:00.512+0000 7fd175c1b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:01:00.505 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:01:00.512+0000 7fd175c1b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:01:00.506 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:01:00.512+0000 7fd175c1b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:01:00.506 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:01:00.512+0000 7fd175c1b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:01:00.506 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:01:00.512+0000 7fd175c1b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:01:01.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:01.158 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:01.191 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:01.193 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:01.193 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:01.193 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:01.193 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-08_01_00_UTC'] 2023-12-19T08:01:01.193 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702972825.5137787, now = 1702972861.1937246, timo = 35 2023-12-19T08:01:02.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:02.195 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:02.229 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:02.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:02.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:02.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:03.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:03.232 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:03.265 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:03.266 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:03.266 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:03.267 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:04.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:04.268 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:04.300 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:04.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:04.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:04.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:05.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:05.303 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:05.336 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:05.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:05.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:05.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:05.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:01:05.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:01:06.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:06.415 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:06.449 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:06.450 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:06.451 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:06.451 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:07.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:07.452 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:07.486 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:07.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:07.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:07.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:08.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:08.488 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:08.518 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:08.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:08.519 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:08.519 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:09.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:09.520 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:09.551 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:09.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:09.552 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:09.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:10.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:10.554 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:10.585 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:10.586 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:10.586 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:10.587 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:11.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:11.589 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:11.623 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:11.625 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:11.625 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:11.625 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:12.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:12.626 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:12.660 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:12.662 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:12.662 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:12.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:13.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:13.663 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:13.697 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:13.699 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:13.699 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:13.699 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:14.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:14.700 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:14.736 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:14.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:14.737 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:14.737 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:15.738 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:15.739 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:15.769 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:15.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:15.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:15.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:16.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:16.773 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:16.804 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:16.806 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:16.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:16.806 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:17.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:17.807 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:17.840 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:17.841 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:17.841 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:17.842 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:18.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:18.842 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:18.876 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:18.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:18.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:18.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:19.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:19.880 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:19.909 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:19.911 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:19.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:19.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:20.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:20.913 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:20.946 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:20.947 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:20.948 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:20.948 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:21.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:21.949 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:21.983 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:21.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:21.985 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:21.985 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:22.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:22.986 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:23.020 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:23.022 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:23.022 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:23.022 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:24.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:24.024 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:24.056 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:24.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:24.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:24.058 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:25.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:25.060 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:25.094 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:25.095 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:25.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:25.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:26.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:26.097 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:26.128 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:26.129 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:26.129 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:26.129 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:27.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:27.130 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:27.164 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:27.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:27.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:27.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:28.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:28.167 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:28.204 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:28.205 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:28.205 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:28.205 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:29.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:29.206 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:29.237 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:29.238 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:29.238 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:29.238 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:30.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:30.240 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:30.273 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:30.275 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:30.275 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:30.275 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:31.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:31.276 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:31.311 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:31.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:31.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:31.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:32.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:32.314 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:32.347 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:32.348 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:32.348 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:32.348 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:33.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:33.349 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:33.379 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:33.381 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:33.381 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:33.381 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:34.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:34.382 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:34.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:34.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:34.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:35.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:35.416 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:35.446 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:35.448 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:35.448 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:35.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:35.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:01:35.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:01:36.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:36.449 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:36.481 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:36.482 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:36.482 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:36.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:37.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:37.484 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:37.518 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:37.518 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:37.518 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:38.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:38.519 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:38.549 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:38.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:38.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:38.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:39.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:39.552 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:39.582 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:39.584 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:39.584 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:39.584 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:40.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:40.585 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:40.616 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:40.617 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:40.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:40.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:41.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:41.618 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:41.652 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:41.653 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:41.653 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:41.653 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:42.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:42.655 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:42.688 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:42.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:42.690 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:42.690 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:43.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:43.691 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:43.725 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:43.727 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:43.727 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:43.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:44.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:44.727 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:44.762 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:44.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:44.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:44.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:45.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:45.765 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:45.795 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:45.796 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:45.796 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:45.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:46.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:46.797 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:46.829 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:46.830 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:46.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:46.831 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:47.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:47.832 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:47.864 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:47.865 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:47.865 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:47.865 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:48.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:48.867 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:48.896 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:48.898 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:48.898 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:48.898 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:49.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:49.899 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:49.930 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:49.930 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:49.930 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:49.930 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:50.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:50.932 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:50.963 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:50.965 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:50.965 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:50.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:51.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:51.966 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:51.997 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:51.999 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:51.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:51.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:53.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:53.000 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:53.034 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:53.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:53.036 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:53.036 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:54.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:54.037 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:54.070 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:54.071 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:54.071 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:54.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:55.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:55.072 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:55.104 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:55.105 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:55.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:55.106 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:56.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:56.107 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:56.139 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:56.141 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:56.141 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:56.141 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:57.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:57.142 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:57.173 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:57.175 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:57.175 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:57.175 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:58.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:58.177 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:58.208 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:58.210 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:58.210 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:58.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:01:59.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:01:59.211 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:01:59.238 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:01:59.240 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:01:59.240 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:01:59.240 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:02:00.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:00.241 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:02:00.272 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:02:00.273 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:02:00.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:02:00.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:02:01.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:01.275 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:02:01.307 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:02:01.307 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_02_00_UTC 2023-12-19T08:02:01.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_02_00_UTC', 'scheduled-2023-12-19-08_01_00_UTC'} 2023-12-19T08:02:01.309 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-08_02_00_UTC'} 2023-12-19T08:02:01.309 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:02:01.309 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-08_02_00_UTC'] 2023-12-19T08:02:01.309 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702972825.5137787, now = 1702972921.3094816, timo = 95 2023-12-19T08:02:02.310 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T08:02:02.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.478+0000 7fbf57fc5700 1 Processor -- start 2023-12-19T08:02:02.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 -- start start 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf5015d350 0x7fbf500cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf50161da0 0x7fbf5006a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500c6560 con 0x7fbf5015d720 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500c70e0 con 0x7fbf5015d350 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf57fc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c67b0 con 0x7fbf50161da0 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf5015d350 0x7fbf500cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf56562700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbf5015d720 0x7fbf5006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50502/0 (socket says 172.21.15.111:50502) 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf5015d350 0x7fbf500cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46266/0 (socket says 172.21.15.111:46266) 2023-12-19T08:02:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf56562700 1 -- 172.21.15.111:0/392415385 learned_addr learned my addr 172.21.15.111:0/392415385 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:02.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55560700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf50161da0 0x7fbf5006a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 -- 172.21.15.111:0/392415385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf50161da0 msgr2=0x7fbf5006a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf50161da0 0x7fbf5006a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 -- 172.21.15.111:0/392415385 >> v1:172.21.15.111:6789/0 conn(0x7fbf5015d720 legacy=0x7fbf5006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 -- 172.21.15.111:0/392415385 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500c6a90 con 0x7fbf5015d350 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.482+0000 7fbf55d61700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf5015d350 0x7fbf500cc1f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0098e0 tx=0x7fbf4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a331169fcac6dd30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf54d5f700 1 -- 172.21.15.111:0/392415385 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c009e80 con 0x7fbf5015d350 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf54d5f700 1 -- 172.21.15.111:0/392415385 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c00b860 con 0x7fbf5015d350 2023-12-19T08:02:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf54d5f700 1 -- 172.21.15.111:0/392415385 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c00bbe0 con 0x7fbf5015d350 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 -- 172.21.15.111:0/392415385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf5015d350 msgr2=0x7fbf500cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf5015d350 0x7fbf500cc1f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0098e0 tx=0x7fbf4c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 -- 172.21.15.111:0/392415385 shutdown_connections 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf50161da0 0x7fbf5006a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/392415385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf5015d350 0x7fbf500cc1f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 -- 172.21.15.111:0/392415385 >> 172.21.15.111:0/392415385 conn(0x7fbf50067bb0 msgr2=0x7fbf500cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:02.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 -- 172.21.15.111:0/392415385 shutdown_connections 2023-12-19T08:02:02.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.486+0000 7fbf57fc5700 1 -- 172.21.15.111:0/392415385 wait complete. 2023-12-19T08:02:02.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 Processor -- start 2023-12-19T08:02:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 -- start start 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 0x7fbf50160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 0x7fbf500b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50154e10 con 0x7fbf50161da0 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5015e270 con 0x7fbf5015d720 2023-12-19T08:02:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf57fc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5015e550 con 0x7fbf5015d350 2023-12-19T08:02:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf55d61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 0x7fbf50160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46276/0 (socket says 172.21.15.111:46276) 2023-12-19T08:02:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 -- 172.21.15.111:0/3568275347 learned_addr learned my addr 172.21.15.111:0/3568275347 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf55560700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 0x7fbf500b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 -- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 msgr2=0x7fbf50160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 0x7fbf50160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 -- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 msgr2=0x7fbf500b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 0x7fbf500b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 -- 172.21.15.111:0/3568275347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009420 con 0x7fbf50161da0 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf55d61700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 0x7fbf50160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf55560700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 0x7fbf500b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.490+0000 7fbf56562700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800cc40 tx=0x7fbf48001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800a910 con 0x7fbf50161da0 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4800aa70 con 0x7fbf50161da0 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5015e830 con 0x7fbf50161da0 2023-12-19T08:02:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800ae30 con 0x7fbf50161da0 2023-12-19T08:02:02.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5015ebc0 con 0x7fbf50161da0 2023-12-19T08:02:02.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5015f030 con 0x7fbf50161da0 2023-12-19T08:02:02.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.494+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) v1 ==== 80092+0+0 (secure 0 0 0) 0x7fbf48007300 con 0x7fbf50161da0 2023-12-19T08:02:02.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.498+0000 7fbf45ffb700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 0x7fbf34062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:02.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.498+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7fbf4807d2b0 con 0x7fbf50161da0 2023-12-19T08:02:02.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.498+0000 7fbf55d61700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 0x7fbf34062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:02.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.498+0000 7fbf55d61700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 0x7fbf34062b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0091e0 tx=0x7fbf4c017000 comp rx=0 tx=0).ready entity=mgr.5591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:02.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.506+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf5015f030 con 0x7fbf50161da0 2023-12-19T08:02:02.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.694+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf50160d20 con 0x7fbf340606c0 2023-12-19T08:02:02.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.822+0000 7fbf45ffb700 1 -- 172.21.15.111:0/3568275347 <== mgr.5591 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbf4c00b9d0 con 0x7fbf340606c0 2023-12-19T08:02:02.821 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path snap_test_dir1 2023-12-19T08:02:02.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 msgr2=0x7fbf34062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 0x7fbf34062b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0091e0 tx=0x7fbf4c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 msgr2=0x7fbf5016dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800cc40 tx=0x7fbf48001a50 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 shutdown_connections 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf340606c0 0x7fbf34062b70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf5015d720 0x7fbf500b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf50161da0 0x7fbf5016dc10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 --2- 172.21.15.111:0/3568275347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf5015d350 0x7fbf50160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 >> 172.21.15.111:0/3568275347 conn(0x7fbf50067bb0 msgr2=0x7fbf50161060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 shutdown_connections 2023-12-19T08:02:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:02.826+0000 7fbf57fc5700 1 -- 172.21.15.111:0/3568275347 wait complete. 2023-12-19T08:02:02.848 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T08:02:02.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:02.848 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:02:02.877 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:02:02.877 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_02_00_UTC 2023-12-19T08:02:02.878 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_01_00_UTC 2023-12-19T08:02:02.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:02.878 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_01_00_UTC) 2023-12-19T08:02:02.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_02_00_UTC 2023-12-19T08:02:02.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:02.962 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_02_00_UTC) 2023-12-19T08:02:03.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:03.039 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T08:02:03.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T08:02:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.322+0000 7fbf7ad45700 1 Processor -- start 2023-12-19T08:02:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.322+0000 7fbf7ad45700 1 -- start start 2023-12-19T08:02:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf7ad45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbf7406db40 0x7fbf740cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf7ad45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf7406df10 0x7fbf74159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf7ad45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf740c9db0 con 0x7fbf740c63f0 2023-12-19T08:02:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf7ad45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740ca950 con 0x7fbf7406df10 2023-12-19T08:02:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf7ad45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740cb2e0 con 0x7fbf7406db40 2023-12-19T08:02:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf7406df10 0x7fbf74159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf7406df10 0x7fbf74159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46282/0 (socket says 172.21.15.111:46282) 2023-12-19T08:02:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 -- 172.21.15.111:0/357732065 learned_addr learned my addr 172.21.15.111:0/357732065 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf78ae1700 1 --2- 172.21.15.111:0/357732065 >> v2:172.21.15.143:3300/0 conn(0x7fbf7406db40 0x7fbf740cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 -- 172.21.15.111:0/357732065 >> v2:172.21.15.143:3300/0 conn(0x7fbf7406db40 msgr2=0x7fbf740cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 --2- 172.21.15.111:0/357732065 >> v2:172.21.15.143:3300/0 conn(0x7fbf7406db40 0x7fbf740cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 -- 172.21.15.111:0/357732065 >> v1:172.21.15.143:6789/0 conn(0x7fbf740c63f0 legacy=0x7fbf740ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 -- 172.21.15.111:0/357732065 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740cb5c0 con 0x7fbf7406df10 2023-12-19T08:02:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf792e2700 1 --2- 172.21.15.111:0/357732065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406df10 0x7fbf74159900 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbf74067fe0 tx=0x7fbf6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f52d1becf82f99 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.326+0000 7fbf737fe700 1 -- 172.21.15.111:0/357732065 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00e070 con 0x7fbf7406df10 2023-12-19T08:02:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf737fe700 1 -- 172.21.15.111:0/357732065 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00ba20 con 0x7fbf7406df10 2023-12-19T08:02:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf737fe700 1 -- 172.21.15.111:0/357732065 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0092e0 con 0x7fbf7406df10 2023-12-19T08:02:03.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 -- 172.21.15.111:0/357732065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406df10 msgr2=0x7fbf74159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:03.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/357732065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406df10 0x7fbf74159900 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbf74067fe0 tx=0x7fbf6c009840 comp rx=0 tx=0).stop 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 -- 172.21.15.111:0/357732065 shutdown_connections 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/357732065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406df10 0x7fbf74159900 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/357732065 >> v2:172.21.15.143:3300/0 conn(0x7fbf7406db40 0x7fbf740cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 -- 172.21.15.111:0/357732065 >> 172.21.15.111:0/357732065 conn(0x7fbf74067bb0 msgr2=0x7fbf740cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 -- 172.21.15.111:0/357732065 shutdown_connections 2023-12-19T08:02:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 -- 172.21.15.111:0/357732065 wait complete. 2023-12-19T08:02:03.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.330+0000 7fbf7ad45700 1 Processor -- start 2023-12-19T08:02:03.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 -- start start 2023-12-19T08:02:03.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf7406db40 0x7fbf741513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406df10 0x7fbf7414f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740c3b80 con 0x7fbf740c63f0 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741522f0 con 0x7fbf7406db40 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf7ad45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741540c0 con 0x7fbf7406df10 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46294/0 (socket says 172.21.15.111:46294) 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 -- 172.21.15.111:0/3174440933 learned_addr learned my addr 172.21.15.111:0/3174440933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf78ae1700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf7406db40 0x7fbf741513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf73fff700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406df10 0x7fbf7414f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 -- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406df10 msgr2=0x7fbf7414f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406df10 0x7fbf7414f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 -- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf7406db40 msgr2=0x7fbf741513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf7406db40 0x7fbf741513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 -- 172.21.15.111:0/3174440933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c009ae0 con 0x7fbf740c63f0 2023-12-19T08:02:03.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.334+0000 7fbf792e2700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c012000 tx=0x7fbf6c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c008a80 con 0x7fbf740c63f0 2023-12-19T08:02:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c008c00 con 0x7fbf740c63f0 2023-12-19T08:02:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf74151a50 con 0x7fbf740c63f0 2023-12-19T08:02:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c026460 con 0x7fbf740c63f0 2023-12-19T08:02:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf74152010 con 0x7fbf740c63f0 2023-12-19T08:02:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf741596d0 con 0x7fbf740c63f0 2023-12-19T08:02:03.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.338+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) v1 ==== 80092+0+0 (secure 0 0 0) 0x7fbf6c003040 con 0x7fbf740c63f0 2023-12-19T08:02:03.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.342+0000 7fbf70ff9700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 0x7fbf58062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:03.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.342+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7fbf6c07eec0 con 0x7fbf740c63f0 2023-12-19T08:02:03.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.342+0000 7fbf78ae1700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 0x7fbf58062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:03.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.342+0000 7fbf78ae1700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 0x7fbf58062b70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf68005e30 tx=0x7fbf6800b780 comp rx=0 tx=0).ready entity=mgr.5591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:03.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.346+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf6c04ede0 con 0x7fbf740c63f0 2023-12-19T08:02:03.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:03.554+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fbf74154250 con 0x7fbf740c63f0 2023-12-19T08:02:04.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.426+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fbf6c083070 con 0x7fbf740c63f0 2023-12-19T08:02:04.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3174440933 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v23) v1 ==== 93+0+0 (secure 0 0 0) 0x7fbf6c052870 con 0x7fbf740c63f0 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 msgr2=0x7fbf58062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 0x7fbf58062b70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf68005e30 tx=0x7fbf6800b780 comp rx=0 tx=0).stop 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 msgr2=0x7fbf740c2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c012000 tx=0x7fbf6c018000 comp rx=0 tx=0).stop 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 shutdown_connections 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf580606c0 0x7fbf58062b70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740c63f0 0x7fbf740c2d80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:04.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406df10 0x7fbf7414f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:04.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 --2- 172.21.15.111:0/3174440933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf7406db40 0x7fbf741513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:04.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.430+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 >> 172.21.15.111:0/3174440933 conn(0x7fbf74067bb0 msgr2=0x7fbf740adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:04.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.434+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 shutdown_connections 2023-12-19T08:02:04.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:04.434+0000 7fbf7ad45700 1 -- 172.21.15.111:0/3174440933 wait complete. 2023-12-19T08:02:04.454 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:02:04.454 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:04.498 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:04.538 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:02:04.539 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T08:02:04.539 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T08:02:04.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:04.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[47111]: fuse finished with error 0 and tester_r 0 2023-12-19T08:02:04.577 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:02:04.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:04.577 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T08:02:04.616 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:04.618+0000 7f09abf5e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:04.617 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:04.618+0000 7f2277dc1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:04.763 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:02:04.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:04.763 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T08:02:04.784 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T08:02:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:04.786 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:02:04.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:04.786 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T08:02:04.824 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T08:02:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:04.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:02:04.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:04.826 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T08:02:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:04.850 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T08:02:04.851 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:05.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.030+0000 7fae9d48e700 1 Processor -- start 2023-12-19T08:02:05.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.030+0000 7fae9d48e700 1 -- start start 2023-12-19T08:02:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9d48e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae98158e20 0x7fae9806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9d48e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 0x7fae980ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9d48e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9806b730 con 0x7fae9815ca80 2023-12-19T08:02:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9d48e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9806ab20 con 0x7fae98158e20 2023-12-19T08:02:05.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9d48e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9806ae00 con 0x7fae981591f0 2023-12-19T08:02:05.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae977fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 0x7fae980ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae97fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae98158e20 0x7fae9806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9cc8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fae9815ca80 0x7fae9806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50514/0 (socket says 172.21.15.111:50514) 2023-12-19T08:02:05.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae9cc8d700 1 -- 172.21.15.111:0/1108433977 learned_addr learned my addr 172.21.15.111:0/1108433977 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:05.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.034+0000 7fae96ffd700 1 -- 172.21.15.111:0/1108433977 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811165257 0 0) 0x7fae9806b730 con 0x7fae9815ca80 2023-12-19T08:02:05.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae96ffd700 1 -- 172.21.15.111:0/1108433977 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae80003570 con 0x7fae9815ca80 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae97fff700 1 -- 172.21.15.111:0/1108433977 >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 msgr2=0x7fae980ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae97fff700 1 --2- 172.21.15.111:0/1108433977 >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 0x7fae980ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae97fff700 1 -- 172.21.15.111:0/1108433977 >> v1:172.21.15.111:6789/0 conn(0x7fae9815ca80 legacy=0x7fae9806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae97fff700 1 -- 172.21.15.111:0/1108433977 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9806b0e0 con 0x7fae98158e20 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae977fe700 1 --2- 172.21.15.111:0/1108433977 >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 0x7fae980ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae97fff700 1 --2- 172.21.15.111:0/1108433977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae9806a540 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fae980642c0 tx=0x7fae880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3bcec3a2187259 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae96ffd700 1 -- 172.21.15.111:0/1108433977 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88011630 con 0x7fae98158e20 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae96ffd700 1 -- 172.21.15.111:0/1108433977 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae880117b0 con 0x7fae98158e20 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae96ffd700 1 -- 172.21.15.111:0/1108433977 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88011b70 con 0x7fae98158e20 2023-12-19T08:02:05.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 -- 172.21.15.111:0/1108433977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 msgr2=0x7fae9806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 --2- 172.21.15.111:0/1108433977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae9806a540 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fae980642c0 tx=0x7fae880091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 -- 172.21.15.111:0/1108433977 shutdown_connections 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 --2- 172.21.15.111:0/1108433977 >> v2:172.21.15.143:3300/0 conn(0x7fae981591f0 0x7fae980ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 --2- 172.21.15.111:0/1108433977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae9806a540 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 -- 172.21.15.111:0/1108433977 >> 172.21.15.111:0/1108433977 conn(0x7fae98067bb0 msgr2=0x7fae980cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:05.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 -- 172.21.15.111:0/1108433977 shutdown_connections 2023-12-19T08:02:05.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.038+0000 7fae9d48e700 1 -- 172.21.15.111:0/1108433977 wait complete. 2023-12-19T08:02:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 Processor -- start 2023-12-19T08:02:05.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 -- start start 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae981591f0 0x7fae981546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9815ca80 0x7fae981693c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980c8800 con 0x7fae98158e20 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980c9640 con 0x7fae981591f0 2023-12-19T08:02:05.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae9d48e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae980c9ad0 con 0x7fae9815ca80 2023-12-19T08:02:05.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46338/0 (socket says 172.21.15.111:46338) 2023-12-19T08:02:05.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 -- 172.21.15.111:0/4122383287 learned_addr learned my addr 172.21.15.111:0/4122383287 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae977fe700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae981591f0 0x7fae981546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9815ca80 msgr2=0x7fae981693c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9815ca80 0x7fae981693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae981591f0 msgr2=0x7fae981546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae981591f0 0x7fae981546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 -- 172.21.15.111:0/4122383287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c009420 con 0x7fae98158e20 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.042+0000 7fae97fff700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fae88013000 tx=0x7fae88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8801f070 con 0x7fae98158e20 2023-12-19T08:02:05.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae88011920 con 0x7fae98158e20 2023-12-19T08:02:05.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae88009420 con 0x7fae98158e20 2023-12-19T08:02:05.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88023410 con 0x7fae98158e20 2023-12-19T08:02:05.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae980c8d90 con 0x7fae98158e20 2023-12-19T08:02:05.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9804be40 con 0x7fae98158e20 2023-12-19T08:02:05.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.046+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fae8802c3b0 con 0x7fae98158e20 2023-12-19T08:02:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.050+0000 7fae7ffff700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 0x7fae84062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.050+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7fae8807e140 con 0x7fae98158e20 2023-12-19T08:02:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.050+0000 7fae977fe700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 msgr2=0x7fae84062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.050+0000 7fae977fe700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 0x7fae84062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.058+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fae8804e1c0 con 0x7fae98158e20 2023-12-19T08:02:05.167 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.170+0000 7f09abf5e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:05.178 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:05.178+0000 7f2277dc1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:05.241 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.242+0000 7f09abf5e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.254+0000 7fae977fe700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 msgr2=0x7fae84062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:05.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.254+0000 7fae977fe700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 0x7fae84062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:05.260 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:05.258+0000 7f2277dc1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:05.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.262+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae980ced70 con 0x7fae98158e20 2023-12-19T08:02:05.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.262+0000 7fae7ffff700 1 -- 172.21.15.111:0/4122383287 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7fae880516e0 con 0x7fae98158e20 2023-12-19T08:02:05.264 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:05.264 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:02:05.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 msgr2=0x7fae84062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 0x7fae84062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 msgr2=0x7fae98157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fae88013000 tx=0x7fae88017000 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 shutdown_connections 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae840606a0 0x7fae84062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae981591f0 0x7fae981546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae9815ca80 0x7fae981693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 --2- 172.21.15.111:0/4122383287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae98158e20 0x7fae98157b10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.266+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 >> 172.21.15.111:0/4122383287 conn(0x7fae98067bb0 msgr2=0x7fae980c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:05.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.270+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 shutdown_connections 2023-12-19T08:02:05.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.270+0000 7fae9d48e700 1 -- 172.21.15.111:0/4122383287 wait complete. 2023-12-19T08:02:05.270 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:02:05.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:05.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.386+0000 7f09abf5e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:05.413 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:05.414+0000 7f2277dc1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:05.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.454+0000 7f2a9266e700 1 Processor -- start 2023-12-19T08:02:05.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.454+0000 7f2a9266e700 1 -- start start 2023-12-19T08:02:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a9266e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a8c0ae520 0x7f2a8c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a9266e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a9266e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c06d140 con 0x7f2a8c1552f0 2023-12-19T08:02:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a9266e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c8710 con 0x7f2a8c0ae520 2023-12-19T08:02:05.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a9266e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c1b90 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a83fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a8c0ae520 0x7f2a8c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:49986/0 (socket says 172.21.15.111:49986) 2023-12-19T08:02:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1796660282 learned_addr learned my addr 172.21.15.111:0/1796660282 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:05.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a8bfff700 1 --1- 172.21.15.111:0/1796660282 >> v1:172.21.15.143:6789/0 conn(0x7f2a8c1552f0 0x7f2a8c0c8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36646/0 (socket says 172.21.15.111:36646) 2023-12-19T08:02:05.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1796660282 >> v2:172.21.15.143:3300/0 conn(0x7f2a8c0ae520 msgr2=0x7f2a8c1556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:05.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 --2- 172.21.15.111:0/1796660282 >> v2:172.21.15.143:3300/0 conn(0x7f2a8c0ae520 0x7f2a8c1556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1796660282 >> v1:172.21.15.143:6789/0 conn(0x7f2a8c1552f0 legacy=0x7f2a8c0c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1796660282 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c06a9d0 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a90c0b700 1 --2- 172.21.15.111:0/1796660282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400d1a0 tx=0x7f2a8400ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6feb8fd364b288af server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a8b7fe700 1 -- 172.21.15.111:0/1796660282 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84010070 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a8b7fe700 1 -- 172.21.15.111:0/1796660282 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8400de10 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.458+0000 7f2a8b7fe700 1 -- 172.21.15.111:0/1796660282 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400b320 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- 172.21.15.111:0/1796660282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 msgr2=0x7f2a8c06cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1796660282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400d1a0 tx=0x7f2a8400ba50 comp rx=0 tx=0).stop 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- 172.21.15.111:0/1796660282 shutdown_connections 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1796660282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c06cc00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1796660282 >> v2:172.21.15.143:3300/0 conn(0x7f2a8c0ae520 0x7f2a8c1556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- 172.21.15.111:0/1796660282 >> 172.21.15.111:0/1796660282 conn(0x7f2a8c0c8850 msgr2=0x7f2a8c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:05.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- 172.21.15.111:0/1796660282 shutdown_connections 2023-12-19T08:02:05.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- 172.21.15.111:0/1796660282 wait complete. 2023-12-19T08:02:05.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 Processor -- start 2023-12-19T08:02:05.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.462+0000 7f2a9266e700 1 -- start start 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 0x7f2a8c158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c155d50 con 0x7f2a8c1552f0 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c155fc0 con 0x7f2a8c0ae520 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c1562a0 con 0x7f2a8c06c7c0 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46344/0 (socket says 172.21.15.111:46344) 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1979012329 learned_addr learned my addr 172.21.15.111:0/1979012329 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:05.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a8bfff700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a83fff700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 0x7f2a8c158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 msgr2=0x7f2a8c0c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c0c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 msgr2=0x7f2a8c158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 0x7f2a8c158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 -- 172.21.15.111:0/1979012329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8400bcf0 con 0x7f2a8c1552f0 2023-12-19T08:02:05.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a83fff700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 0x7f2a8c158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a90c0b700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c06a560 tx=0x7f2a8401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400acf0 con 0x7f2a8c1552f0 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8400ae50 con 0x7f2a8c1552f0 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c156580 con 0x7f2a8c1552f0 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840276b0 con 0x7f2a8c1552f0 2023-12-19T08:02:05.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.466+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c150d80 con 0x7f2a8c1552f0 2023-12-19T08:02:05.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.470+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c071120 con 0x7f2a8c1552f0 2023-12-19T08:02:05.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.470+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f2a84003040 con 0x7f2a8c1552f0 2023-12-19T08:02:05.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.470+0000 7f2a88ff9700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 0x7f2a6c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.470+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f2a84080cd0 con 0x7f2a8c1552f0 2023-12-19T08:02:05.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.470+0000 7f2a8bfff700 1 -- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 msgr2=0x7f2a6c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:05.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.474+0000 7f2a8bfff700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 0x7f2a6c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:05.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.478+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2a84050b40 con 0x7f2a8c1552f0 2023-12-19T08:02:05.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.646+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a8c0642a0 con 0x7f2a8c1552f0 2023-12-19T08:02:05.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.650+0000 7f2a88ff9700 1 -- 172.21.15.111:0/1979012329 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16939 (secure 0 0 0) 0x7f2a84054160 con 0x7f2a8c1552f0 2023-12-19T08:02:05.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:05.651 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":71,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:01:25.451374+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/134362188":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3495942537":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/189999424":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3297717311":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/934963750":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/1624932641":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6833/33796":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6832/33796":"2023-12-20T08:00:23.832713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 msgr2=0x7f2a6c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 0x7f2a6c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 msgr2=0x7f2a8c0afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c06a560 tx=0x7f2a8401a000 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 shutdown_connections 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2a6c060670 0x7f2a6c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a8c0ae520 0x7f2a8c158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a8c1552f0 0x7f2a8c0afb70 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 --2- 172.21.15.111:0/1979012329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a8c06c7c0 0x7f2a8c0c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 >> 172.21.15.111:0/1979012329 conn(0x7f2a8c0c8850 msgr2=0x7f2a8c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:05.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 shutdown_connections 2023-12-19T08:02:05.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.654+0000 7f2a9266e700 1 -- 172.21.15.111:0/1979012329 wait complete. 2023-12-19T08:02:05.669 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.670+0000 7f09abf5e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:05.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:00:05.879580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:00:06.863124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T08:02:05.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:05.704 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:05.702+0000 7f2277dc1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:05.749 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.750+0000 7f09abf5e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:05.783 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:05.782+0000 7f2277dc1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:05.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 Processor -- start 2023-12-19T08:02:05.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 -- start start 2023-12-19T08:02:05.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35e406df10 0x7f35e40cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40c9db0 con 0x7f35e40c63f0 2023-12-19T08:02:05.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40cb2e0 con 0x7f35e406db40 2023-12-19T08:02:05.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35eaacd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e40ca950 con 0x7f35e406df10 2023-12-19T08:02:05.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35dbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f35e40c63f0 0x7f35e4159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50528/0 (socket says 172.21.15.111:50528) 2023-12-19T08:02:05.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35dbfff700 1 -- 172.21.15.111:0/4118670318 learned_addr learned my addr 172.21.15.111:0/4118670318 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:05.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35e906a700 1 --2- 172.21.15.111:0/4118670318 >> v2:172.21.15.143:3300/0 conn(0x7f35e406df10 0x7f35e40cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878659867 0 0) 0x7f35e40c9db0 con 0x7f35e40c63f0 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003570 con 0x7f35e40c63f0 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2653164320 0 0) 0x7f35cc003570 con 0x7f35e40c63f0 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 >> v2:172.21.15.143:3300/0 conn(0x7f35e406df10 msgr2=0x7f35e40cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 --2- 172.21.15.111:0/4118670318 >> v2:172.21.15.143:3300/0 conn(0x7f35e406df10 0x7f35e40cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 >> v1:172.21.15.143:6789/0 conn(0x7f35e406db40 legacy=0x7f35e40ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40cb5c0 con 0x7f35e40c63f0 2023-12-19T08:02:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f35d4001470 con 0x7f35e40c63f0 2023-12-19T08:02:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 >> v1:172.21.15.111:6789/0 conn(0x7f35e40c63f0 legacy=0x7f35e4159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc003a60 0x7f35cc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40ca950 con 0x7f35cc004810 2023-12-19T08:02:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00c050 con 0x7f35cc0051c0 2023-12-19T08:02:05.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00e420 con 0x7f35cc003a60 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.838+0000 7f35dbfff700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35e8869700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc003a60 0x7f35cc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35e906a700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 -- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc003a60 msgr2=0x7f35cc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35cc003a60 0x7f35cc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 -- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0051c0 msgr2=0x7f35cc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 -- 172.21.15.111:0/4118670318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc003570 con 0x7f35cc004810 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35e906a700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35e906a700 1 -- 172.21.15.111:0/4118670318 reap_dead start 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35dbfff700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c390 tx=0x7f35d4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c58c380ddcef2f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d400cc60 con 0x7f35cc004810 2023-12-19T08:02:05.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400cdc0 con 0x7f35cc004810 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35db7fe700 1 -- 172.21.15.111:0/4118670318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4005b70 con 0x7f35cc004810 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 msgr2=0x7f35cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c390 tx=0x7f35d4001e70 comp rx=0 tx=0).stop 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- 172.21.15.111:0/4118670318 shutdown_connections 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 --2- 172.21.15.111:0/4118670318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- 172.21.15.111:0/4118670318 >> 172.21.15.111:0/4118670318 conn(0x7f35e4067bb0 msgr2=0x7f35e40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- 172.21.15.111:0/4118670318 shutdown_connections 2023-12-19T08:02:05.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- 172.21.15.111:0/4118670318 wait complete. 2023-12-19T08:02:05.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 Processor -- start 2023-12-19T08:02:05.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.842+0000 7f35eaacd700 1 -- start start 2023-12-19T08:02:05.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35e414c7e0 0x7f35e41509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35e414cbb0 0x7f35e4150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40b06d0 con 0x7f35e414f2b0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e414fc60 con 0x7f35e414cbb0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e41521a0 con 0x7f35e414c7e0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46360/0 (socket says 172.21.15.111:46360) 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 -- 172.21.15.111:0/1294218359 learned_addr learned my addr 172.21.15.111:0/1294218359 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e8869700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35e414c7e0 0x7f35e41509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35dbfff700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35e414cbb0 0x7f35e4150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 -- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35e414c7e0 msgr2=0x7f35e41509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35e414c7e0 0x7f35e41509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 -- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35e414cbb0 msgr2=0x7f35e4150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35e414cbb0 0x7f35e4150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 -- 172.21.15.111:0/1294218359 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4152630 con 0x7f35e414f2b0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35e906a700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35e40b1e70 tx=0x7f35dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:05.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00fc20 con 0x7f35e414f2b0 2023-12-19T08:02:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc00fd80 con 0x7f35e414f2b0 2023-12-19T08:02:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc016630 con 0x7f35e414f2b0 2023-12-19T08:02:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e4151590 con 0x7f35e414f2b0 2023-12-19T08:02:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.846+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e4151b50 con 0x7f35e414f2b0 2023-12-19T08:02:05.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e4152310 con 0x7f35e414f2b0 2023-12-19T08:02:05.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f35dc016a20 con 0x7f35e414f2b0 2023-12-19T08:02:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35d8ff9700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 0x7f35d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f35dc07dcd0 con 0x7f35e414f2b0 2023-12-19T08:02:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35e8869700 1 -- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 msgr2=0x7f35d0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:05.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.850+0000 7f35e8869700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 0x7f35d0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:05.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:05.854+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f35dc04db40 con 0x7f35e414f2b0 2023-12-19T08:02:05.971 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:05.974+0000 7f09abf5e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:06.012 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:06.010+0000 7f2277dc1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.022+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35e4071120 con 0x7f35e414f2b0 2023-12-19T08:02:06.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.026+0000 7f35d8ff9700 1 -- 172.21.15.111:0/1294218359 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35dc051160 con 0x7f35e414f2b0 2023-12-19T08:02:06.025 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:06.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 msgr2=0x7f35d0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:06.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 0x7f35d0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 msgr2=0x7f35e414f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35e40b1e70 tx=0x7f35dc00b040 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 shutdown_connections 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35d0060670 0x7f35d0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35e414cbb0 0x7f35e4150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e414f2b0 0x7f35e414f720 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 --2- 172.21.15.111:0/1294218359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35e414c7e0 0x7f35e41509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 >> 172.21.15.111:0/1294218359 conn(0x7f35e4067bb0 msgr2=0x7f35e406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:06.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 shutdown_connections 2023-12-19T08:02:06.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.030+0000 7f35eaacd700 1 -- 172.21.15.111:0/1294218359 wait complete. 2023-12-19T08:02:06.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T08:02:06.058 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:02:06.058 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:06.062+0000 7f09abf5e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:06.107 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:06.102+0000 7f2277dc1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:06.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:02:06.160 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:06.162+0000 7f09abf5e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:06.185 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:06.186+0000 7f2277dc1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.254+0000 7f3fd926b700 1 Processor -- start 2023-12-19T08:02:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.254+0000 7f3fd926b700 1 -- start start 2023-12-19T08:02:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd926b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3fd406aeb0 0x7f3fd4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd926b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3fd41595a0 0x7f3fd41599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd926b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd406c850 con 0x7f3fd4158c90 2023-12-19T08:02:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd926b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd40cb650 con 0x7f3fd406aeb0 2023-12-19T08:02:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd926b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd406b470 con 0x7f3fd41595a0 2023-12-19T08:02:06.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3fd406aeb0 0x7f3fd4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3fd406aeb0 0x7f3fd4159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46368/0 (socket says 172.21.15.111:46368) 2023-12-19T08:02:06.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 -- 172.21.15.111:0/2641337773 learned_addr learned my addr 172.21.15.111:0/2641337773 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:06.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd8a6a700 1 --2- 172.21.15.111:0/2641337773 >> v2:172.21.15.143:3300/0 conn(0x7f3fd41595a0 0x7f3fd41599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 -- 172.21.15.111:0/2641337773 >> v2:172.21.15.143:3300/0 conn(0x7f3fd41595a0 msgr2=0x7f3fd41599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 --2- 172.21.15.111:0/2641337773 >> v2:172.21.15.143:3300/0 conn(0x7f3fd41595a0 0x7f3fd41599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 -- 172.21.15.111:0/2641337773 >> v1:172.21.15.111:6789/0 conn(0x7f3fd4158c90 legacy=0x7f3fd40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 -- 172.21.15.111:0/2641337773 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd406b750 con 0x7f3fd406aeb0 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd37fe700 1 --2- 172.21.15.111:0/2641337773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd4159060 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40cc7f0 tx=0x7f3fc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc734bfd79397e45 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd2ffd700 1 -- 172.21.15.111:0/2641337773 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc800b860 con 0x7f3fd406aeb0 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd2ffd700 1 -- 172.21.15.111:0/2641337773 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc800b9c0 con 0x7f3fd406aeb0 2023-12-19T08:02:06.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.258+0000 7f3fd2ffd700 1 -- 172.21.15.111:0/2641337773 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc800bd60 con 0x7f3fd406aeb0 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- 172.21.15.111:0/2641337773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 msgr2=0x7f3fd4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 --2- 172.21.15.111:0/2641337773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd4159060 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40cc7f0 tx=0x7f3fc8009250 comp rx=0 tx=0).stop 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- 172.21.15.111:0/2641337773 shutdown_connections 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 --2- 172.21.15.111:0/2641337773 >> v2:172.21.15.143:3300/0 conn(0x7f3fd41595a0 0x7f3fd41599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 --2- 172.21.15.111:0/2641337773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd4159060 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- 172.21.15.111:0/2641337773 >> 172.21.15.111:0/2641337773 conn(0x7f3fd4067bb0 msgr2=0x7f3fd40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- 172.21.15.111:0/2641337773 shutdown_connections 2023-12-19T08:02:06.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- 172.21.15.111:0/2641337773 wait complete. 2023-12-19T08:02:06.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 Processor -- start 2023-12-19T08:02:06.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.262+0000 7f3fd926b700 1 -- start start 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 0x7f3fd406d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 0x7f3fd4088c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd4089c80 con 0x7f3fd406aeb0 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40896c0 con 0x7f3fd41595a0 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd926b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40899a0 con 0x7f3fd4158c90 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46370/0 (socket says 172.21.15.111:46370) 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 -- 172.21.15.111:0/3183356521 learned_addr learned my addr 172.21.15.111:0/3183356521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd8a6a700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 0x7f3fd4088c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd37fe700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 0x7f3fd406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 -- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 msgr2=0x7f3fd406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 0x7f3fd406d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 -- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 msgr2=0x7f3fd4088c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 0x7f3fd4088c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 -- 172.21.15.111:0/3183356521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80094f0 con 0x7f3fd406aeb0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd8a6a700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 0x7f3fd4088c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd37fe700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 0x7f3fd406d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fd3fff700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c590 tx=0x7f3fc400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.266+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4011590 con 0x7f3fd406aeb0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc40116f0 con 0x7f3fd406aeb0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4011ab0 con 0x7f3fd406aeb0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd4087010 con 0x7f3fd406aeb0 2023-12-19T08:02:06.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd4158060 con 0x7f3fd406aeb0 2023-12-19T08:02:06.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd404be40 con 0x7f3fd406aeb0 2023-12-19T08:02:06.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.270+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f3fc401b3b0 con 0x7f3fd406aeb0 2023-12-19T08:02:06.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.274+0000 7f3fbbfff700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 0x7f3fc0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.274+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f3fc407c490 con 0x7f3fd406aeb0 2023-12-19T08:02:06.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.274+0000 7f3fd37fe700 1 -- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 msgr2=0x7f3fc0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:06.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.274+0000 7f3fd37fe700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 0x7f3fc0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:06.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.282+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3fc404c410 con 0x7f3fd406aeb0 2023-12-19T08:02:06.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.450+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3fd406ba30 con 0x7f3fd406aeb0 2023-12-19T08:02:06.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.450+0000 7f3fbbfff700 1 -- 172.21.15.111:0/3183356521 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3fc404fa30 con 0x7f3fd406aeb0 2023-12-19T08:02:06.451 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T08:02:06.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 msgr2=0x7f3fc0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 0x7f3fc0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 msgr2=0x7f3fd416dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c590 tx=0x7f3fc400c950 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 shutdown_connections 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3fc0060670 0x7f3fc0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fd41595a0 0x7f3fd4088c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fd4158c90 0x7f3fd406d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 --2- 172.21.15.111:0/3183356521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fd406aeb0 0x7f3fd416dcd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 >> 172.21.15.111:0/3183356521 conn(0x7f3fd4067bb0 msgr2=0x7f3fd40afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:06.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 shutdown_connections 2023-12-19T08:02:06.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.454+0000 7f3fd926b700 1 -- 172.21.15.111:0/3183356521 wait complete. 2023-12-19T08:02:06.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T08:02:06.571 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:06.574+0000 7f09abf5e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:06.597 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:06.598+0000 7f2277dc1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:06.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.686+0000 7fc5be2f0700 1 Processor -- start 2023-12-19T08:02:06.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.686+0000 7fc5be2f0700 1 -- start start 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b806b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cbdf0 con 0x7fc5b8158e20 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cc0d0 con 0x7fc5b80cb970 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80cc3b0 con 0x7fc5b80cecd0 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc5b8158e20 0x7fc5b80ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50552/0 (socket says 172.21.15.111:50552) 2023-12-19T08:02:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b7fff700 1 -- 172.21.15.111:0/3354148547 learned_addr learned my addr 172.21.15.111:0/3354148547 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5bc88d700 1 --2- 172.21.15.111:0/3354148547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b806b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986221255 0 0) 0x7fc5b80cbdf0 con 0x7fc5b8158e20 2023-12-19T08:02:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a0003570 con 0x7fc5b8158e20 2023-12-19T08:02:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355765702 0 0) 0x7fc5b80cc0d0 con 0x7fc5b80cb970 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b80cbdf0 con 0x7fc5b80cb970 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5bc88d700 1 -- 172.21.15.111:0/3354148547 >> v1:172.21.15.143:6789/0 conn(0x7fc5b80cb970 legacy=0x7fc5b806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394984508 0 0) 0x7fc5a0003570 con 0x7fc5b8158e20 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5bc88d700 1 -- 172.21.15.111:0/3354148547 >> v1:172.21.15.111:6789/0 conn(0x7fc5b8158e20 legacy=0x7fc5b80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5bc88d700 1 -- 172.21.15.111:0/3354148547 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80cc690 con 0x7fc5b80cecd0 2023-12-19T08:02:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5bc88d700 1 --2- 172.21.15.111:0/3354148547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b806b3d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009ec0 tx=0x7fc5b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d368be7db61a9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:06.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000e070 con 0x7fc5b80cecd0 2023-12-19T08:02:06.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000b870 con 0x7fc5b80cecd0 2023-12-19T08:02:06.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5b6ffd700 1 -- 172.21.15.111:0/3354148547 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8001470 con 0x7fc5b80cecd0 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 -- 172.21.15.111:0/3354148547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 msgr2=0x7fc5b806b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.690+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/3354148547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b806b3d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009ec0 tx=0x7fc5b0009780 comp rx=0 tx=0).stop 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 -- 172.21.15.111:0/3354148547 shutdown_connections 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/3354148547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b806b3d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 -- 172.21.15.111:0/3354148547 >> 172.21.15.111:0/3354148547 conn(0x7fc5b8067bb0 msgr2=0x7fc5b80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 -- 172.21.15.111:0/3354148547 shutdown_connections 2023-12-19T08:02:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 -- 172.21.15.111:0/3354148547 wait complete. 2023-12-19T08:02:06.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 Processor -- start 2023-12-19T08:02:06.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.694+0000 7fc5be2f0700 1 -- start start 2023-12-19T08:02:06.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b8168c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b80b4d90 0x7fc5b816cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b814d940 con 0x7fc5b8158e20 2023-12-19T08:02:06.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80b42b0 con 0x7fc5b80b4d90 2023-12-19T08:02:06.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80b4590 con 0x7fc5b80cecd0 2023-12-19T08:02:06.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46378/0 (socket says 172.21.15.111:46378) 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 -- 172.21.15.111:0/2173909767 learned_addr learned my addr 172.21.15.111:0/2173909767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5bc88d700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b80b4d90 0x7fc5b816cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b7fff700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b8168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 -- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 msgr2=0x7fc5b8168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b8168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 -- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b80b4d90 msgr2=0x7fc5b816cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b80b4d90 0x7fc5b816cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 -- 172.21.15.111:0/2173909767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0009a20 con 0x7fc5b8158e20 2023-12-19T08:02:06.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5b77fe700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c7c0 tx=0x7fc5ac007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:06.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac010510 con 0x7fc5b8158e20 2023-12-19T08:02:06.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac002e90 con 0x7fc5b8158e20 2023-12-19T08:02:06.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80b4870 con 0x7fc5b8158e20 2023-12-19T08:02:06.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0108e0 con 0x7fc5b8158e20 2023-12-19T08:02:06.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.698+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80b4040 con 0x7fc5b8158e20 2023-12-19T08:02:06.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.702+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b80b4ab0 con 0x7fc5b8158e20 2023-12-19T08:02:06.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.702+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fc5ac010d10 con 0x7fc5b8158e20 2023-12-19T08:02:06.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.706+0000 7fc59ffff700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 0x7fc5a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:06.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.706+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7fc5ac07c2e0 con 0x7fc5b8158e20 2023-12-19T08:02:06.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.706+0000 7fc5b7fff700 1 -- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 msgr2=0x7fc5a4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:06.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.706+0000 7fc5b7fff700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 0x7fc5a4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:06.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.710+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc5ac049be0 con 0x7fc5b8158e20 2023-12-19T08:02:06.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:06.734+0000 7f09abf5e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:06.769 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:06.770+0000 7f2277dc1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:06.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.874+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5b806a7a0 con 0x7fc5b8158e20 2023-12-19T08:02:06.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.874+0000 7fc59ffff700 1 -- 172.21.15.111:0/2173909767 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5ac017020 con 0x7fc5b8158e20 2023-12-19T08:02:06.876 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T08:02:06.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 msgr2=0x7fc5a4062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 0x7fc5a4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 msgr2=0x7fc5b8169160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c7c0 tx=0x7fc5ac007de0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 shutdown_connections 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc5a4060670 0x7fc5a4062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b80b4d90 0x7fc5b816cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5b8158e20 0x7fc5b8169160 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 --2- 172.21.15.111:0/2173909767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80cecd0 0x7fc5b8168c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 >> 172.21.15.111:0/2173909767 conn(0x7fc5b8067bb0 msgr2=0x7fc5b814cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 shutdown_connections 2023-12-19T08:02:06.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:06.878+0000 7fc5be2f0700 1 -- 172.21.15.111:0/2173909767 wait complete. 2023-12-19T08:02:06.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 Processor -- start 2023-12-19T08:02:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 -- start start 2023-12-19T08:02:07.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 0x7f6fcc0670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0cd540 con 0x7f6fcc158df0 2023-12-19T08:02:07.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0c7ed0 con 0x7f6fcc15d140 2023-12-19T08:02:07.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.058+0000 7f6fcb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc15d610 con 0x7f6fcc06d760 2023-12-19T08:02:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fca59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6fcc158df0 0x7f6fcc15ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50560/0 (socket says 172.21.15.111:50560) 2023-12-19T08:02:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fca59c700 1 -- 172.21.15.111:0/2877240670 learned_addr learned my addr 172.21.15.111:0/2877240670 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcad9d700 1 --2- 172.21.15.111:0/2877240670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 0x7f6fcc0670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123515409 0 0) 0x7f6fcc0cd540 con 0x7f6fcc158df0 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4003570 con 0x7f6fcc158df0 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995850877 0 0) 0x7f6fcc0c7ed0 con 0x7f6fcc15d140 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc0cd540 con 0x7f6fcc15d140 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcad9d700 1 -- 172.21.15.111:0/2877240670 >> v1:172.21.15.143:6789/0 conn(0x7f6fcc15d140 legacy=0x7f6fcc06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795675700 0 0) 0x7f6fb4003570 con 0x7f6fcc158df0 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcad9d700 1 -- 172.21.15.111:0/2877240670 >> v1:172.21.15.111:6789/0 conn(0x7f6fcc158df0 legacy=0x7f6fcc15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcad9d700 1 -- 172.21.15.111:0/2877240670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0c7880 con 0x7f6fcc06d760 2023-12-19T08:02:07.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcad9d700 1 --2- 172.21.15.111:0/2877240670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 0x7f6fcc0670e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009e90 tx=0x7f6fc4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a5fae8495f7a319 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400e070 con 0x7f6fcc06d760 2023-12-19T08:02:07.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc400b870 con 0x7f6fcc06d760 2023-12-19T08:02:07.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fc959a700 1 -- 172.21.15.111:0/2877240670 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc001470 con 0x7f6fcc06d760 2023-12-19T08:02:07.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2877240670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 msgr2=0x7f6fcc0670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2877240670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 0x7f6fcc0670e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4009e90 tx=0x7f6fc4009780 comp rx=0 tx=0).stop 2023-12-19T08:02:07.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2877240670 shutdown_connections 2023-12-19T08:02:07.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2877240670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc06d760 0x7f6fcc0670e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.062+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2877240670 >> 172.21.15.111:0/2877240670 conn(0x7f6fcc06c080 msgr2=0x7f6fcc0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:07.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.066+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2877240670 shutdown_connections 2023-12-19T08:02:07.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.066+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2877240670 wait complete. 2023-12-19T08:02:07.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.066+0000 7f6fcb59e700 1 Processor -- start 2023-12-19T08:02:07.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.066+0000 7f6fcb59e700 1 -- start start 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 0x7f6fcc0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc15d140 0x7f6fcc0c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0b20a0 con 0x7f6fcc06d760 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0aff70 con 0x7f6fcc158df0 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc14bf40 con 0x7f6fcc15d140 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46384/0 (socket says 172.21.15.111:46384) 2023-12-19T08:02:07.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 -- 172.21.15.111:0/2001347510 learned_addr learned my addr 172.21.15.111:0/2001347510 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:07.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcad9d700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc15d140 0x7f6fcc0c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fc9d9b700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 0x7f6fcc0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc15d140 msgr2=0x7f6fcc0c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc15d140 0x7f6fcc0c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 msgr2=0x7f6fcc0c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 0x7f6fcc0c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 -- 172.21.15.111:0/2001347510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4009a20 con 0x7f6fcc06d760 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fc9d9b700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 0x7f6fcc0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fca59c700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00cb70 tx=0x7f6fbc001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc007e90 con 0x7f6fcc06d760 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc007360 con 0x7f6fcc06d760 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fcc0c5180 con 0x7f6fcc06d760 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc004830 con 0x7f6fcc06d760 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.070+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc0af920 con 0x7f6fcc06d760 2023-12-19T08:02:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fcc04be40 con 0x7f6fcc06d760 2023-12-19T08:02:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f6fbc004ba0 con 0x7f6fcc06d760 2023-12-19T08:02:07.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fba7fc700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 0x7f6fac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f6fbc07c5b0 con 0x7f6fcc06d760 2023-12-19T08:02:07.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fc9d9b700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 msgr2=0x7f6fac062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:07.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.074+0000 7f6fc9d9b700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 0x7f6fac062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:07.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.078+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6fbc04c650 con 0x7f6fcc06d760 2023-12-19T08:02:07.093 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.094+0000 7f09abf5e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:07.150 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.150+0000 7f2277dc1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:07.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.278+0000 7f6fc9d9b700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 msgr2=0x7f6fac062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:07.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.278+0000 7f6fc9d9b700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 0x7f6fac062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:07.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.286+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fcc0cda00 con 0x7f6fcc06d760 2023-12-19T08:02:07.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.286+0000 7f6fba7fc700 1 -- 172.21.15.111:0/2001347510 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7f6fbc04fc70 con 0x7f6fcc06d760 2023-12-19T08:02:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:07.288 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 msgr2=0x7f6fac062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 0x7f6fac062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 msgr2=0x7f6fcc14ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00cb70 tx=0x7f6fbc001b50 comp rx=0 tx=0).stop 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 shutdown_connections 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6fac060670 0x7f6fac062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fcc158df0 0x7f6fcc0c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fcc15d140 0x7f6fcc0c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 --2- 172.21.15.111:0/2001347510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fcc06d760 0x7f6fcc14ba00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 >> 172.21.15.111:0/2001347510 conn(0x7f6fcc06c080 msgr2=0x7f6fcc1580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 shutdown_connections 2023-12-19T08:02:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.290+0000 7f6fcb59e700 1 -- 172.21.15.111:0/2001347510 wait complete. 2023-12-19T08:02:07.294 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:02:07.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:07.312 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.314+0000 7f09abf5e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:07.383 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.382+0000 7f2277dc1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:07.401 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.402+0000 7f09abf5e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:07.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 Processor -- start 2023-12-19T08:02:07.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 -- start start 2023-12-19T08:02:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f007c1591f0 0x7f007c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c06b730 con 0x7f007c15ca80 2023-12-19T08:02:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c06ab20 con 0x7f007c1591f0 2023-12-19T08:02:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008328e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06ae00 con 0x7f007c158e20 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0081a8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f007c1591f0 0x7f007c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0082a8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f007c15ca80 0x7f007c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50564/0 (socket says 172.21.15.111:50564) 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0082a8d700 1 -- 172.21.15.111:0/633046524 learned_addr learned my addr 172.21.15.111:0/633046524 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008228c700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008128a700 1 -- 172.21.15.111:0/633046524 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31979960 0 0) 0x7f007c06b730 con 0x7f007c15ca80 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f008128a700 1 -- 172.21.15.111:0/633046524 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0068003570 con 0x7f007c15ca80 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0081a8b700 1 -- 172.21.15.111:0/633046524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 msgr2=0x7f007c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0081a8b700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0081a8b700 1 -- 172.21.15.111:0/633046524 >> v1:172.21.15.111:6789/0 conn(0x7f007c15ca80 legacy=0x7f007c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.466+0000 7f0081a8b700 1 -- 172.21.15.111:0/633046524 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c06b0e0 con 0x7f007c1591f0 2023-12-19T08:02:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f0081a8b700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0ce100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098e0 tx=0x7f006c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95806d51ef50c1e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008128a700 1 -- 172.21.15.111:0/633046524 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c011630 con 0x7f007c1591f0 2023-12-19T08:02:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008128a700 1 -- 172.21.15.111:0/633046524 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c011790 con 0x7f007c1591f0 2023-12-19T08:02:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008128a700 1 -- 172.21.15.111:0/633046524 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c011b30 con 0x7f007c1591f0 2023-12-19T08:02:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 -- 172.21.15.111:0/633046524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 msgr2=0x7f007c0ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0ce100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098e0 tx=0x7f006c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 -- 172.21.15.111:0/633046524 shutdown_connections 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0ce100 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 --2- 172.21.15.111:0/633046524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 -- 172.21.15.111:0/633046524 >> 172.21.15.111:0/633046524 conn(0x7f007c067bb0 msgr2=0x7f007c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 -- 172.21.15.111:0/633046524 shutdown_connections 2023-12-19T08:02:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.470+0000 7f008328e700 1 -- 172.21.15.111:0/633046524 wait complete. 2023-12-19T08:02:07.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 Processor -- start 2023-12-19T08:02:07.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 -- start start 2023-12-19T08:02:07.474 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.478+0000 7f09abf5e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 0x7f007c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0c77b0 con 0x7f007c1591f0 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0c9d90 con 0x7f007c15ca80 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008328e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0ca070 con 0x7f007c158e20 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46402/0 (socket says 172.21.15.111:46402) 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 -- 172.21.15.111:0/472994433 learned_addr learned my addr 172.21.15.111:0/472994433 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:07.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008228c700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0082a8d700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 0x7f007c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 -- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 msgr2=0x7f007c152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 -- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 msgr2=0x7f007c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 0x7f007c0c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 -- 172.21.15.111:0/472994433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c009420 con 0x7f007c1591f0 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0082a8d700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 0x7f007c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f008228c700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:07.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.474+0000 7f0081a8b700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f006c013000 tx=0x7f006c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c011900 con 0x7f007c1591f0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c011a80 con 0x7f007c1591f0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c026680 con 0x7f007c1591f0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c0c7160 con 0x7f007c1591f0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c152da0 con 0x7f007c1591f0 2023-12-19T08:02:07.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c0b5b80 con 0x7f007c1591f0 2023-12-19T08:02:07.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f006c023030 con 0x7f007c1591f0 2023-12-19T08:02:07.479 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f2277dc1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:07.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 0x7f0060062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.478+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f006c07e5c0 con 0x7f007c1591f0 2023-12-19T08:02:07.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.482+0000 7f008228c700 1 -- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 msgr2=0x7f0060062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:07.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.482+0000 7f008228c700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 0x7f0060062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:07.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.486+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f006c04e740 con 0x7f007c1591f0 2023-12-19T08:02:07.554 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.554+0000 7f09abf5e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:07.562 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.562+0000 7f2277dc1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:07.645 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.646+0000 7f2277dc1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:07.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.658+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f007c06bdb0 con 0x7f007c1591f0 2023-12-19T08:02:07.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.658+0000 7f00727fc700 1 -- 172.21.15.111:0/472994433 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16939 (secure 0 0 0) 0x7f007c06bdb0 con 0x7f007c1591f0 2023-12-19T08:02:07.659 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:07.660 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":71,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:01:25.451374+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/134362188":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3495942537":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/189999424":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3297717311":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/934963750":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/1624932641":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6833/33796":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6832/33796":"2023-12-20T08:00:23.832713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 msgr2=0x7f0060062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 0x7f0060062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 msgr2=0x7f007c0c7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f006c013000 tx=0x7f006c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 shutdown_connections 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0060060670 0x7f0060062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f007c15ca80 0x7f007c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f007c1591f0 0x7f007c0c7c40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 --2- 172.21.15.111:0/472994433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f007c158e20 0x7f007c152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 >> 172.21.15.111:0/472994433 conn(0x7f007c067bb0 msgr2=0x7f007c0b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 shutdown_connections 2023-12-19T08:02:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.662+0000 7f008328e700 1 -- 172.21.15.111:0/472994433 wait complete. 2023-12-19T08:02:07.688 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T08:02:07.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:07.774 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.774+0000 7f09abf5e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 Processor -- start 2023-12-19T08:02:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 -- start start 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9bf0158d60 0x7f9bf015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf00c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 0x7f9bf006a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00c5610 con 0x7f9bf0158d60 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf006c7c0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf506b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf006caa0 con 0x7f9bf006b7a0 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9bf0158d60 0x7f9bf015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9bf0158d60 0x7f9bf015c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46416/0 (socket says 172.21.15.111:46416) 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 -- 172.21.15.111:0/643728112 learned_addr learned my addr 172.21.15.111:0/643728112 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9be7fff700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf00c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf486a700 1 --2- 172.21.15.111:0/643728112 >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 0x7f9bf006a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 -- 172.21.15.111:0/643728112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 msgr2=0x7f9bf00c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf00c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 -- 172.21.15.111:0/643728112 >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 msgr2=0x7f9bf006a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 --2- 172.21.15.111:0/643728112 >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 0x7f9bf006a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 -- 172.21.15.111:0/643728112 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf015ce60 con 0x7f9bf0158d60 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9bf486a700 1 --2- 172.21.15.111:0/643728112 >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 0x7f9bf006a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.850+0000 7f9beffff700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf0158d60 0x7f9bf015c920 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00c5b50 tx=0x7f9be0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d874ebd2679fe4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bef7fe700 1 -- 172.21.15.111:0/643728112 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0009e70 con 0x7f9bf0158d60 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bef7fe700 1 -- 172.21.15.111:0/643728112 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000b860 con 0x7f9bf0158d60 2023-12-19T08:02:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bef7fe700 1 -- 172.21.15.111:0/643728112 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000bc20 con 0x7f9bf0158d60 2023-12-19T08:02:07.855 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.854+0000 7f09abf5e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 -- 172.21.15.111:0/643728112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf0158d60 msgr2=0x7f9bf015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf0158d60 0x7f9bf015c920 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00c5b50 tx=0x7f9be0009250 comp rx=0 tx=0).stop 2023-12-19T08:02:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 -- 172.21.15.111:0/643728112 shutdown_connections 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 --2- 172.21.15.111:0/643728112 >> v2:172.21.15.143:3300/0 conn(0x7f9bf00c51b0 0x7f9bf006a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf00c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 --2- 172.21.15.111:0/643728112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf0158d60 0x7f9bf015c920 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 -- 172.21.15.111:0/643728112 >> 172.21.15.111:0/643728112 conn(0x7f9bf0067bb0 msgr2=0x7f9bf00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 -- 172.21.15.111:0/643728112 shutdown_connections 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 -- 172.21.15.111:0/643728112 wait complete. 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.854+0000 7f9bf506b700 1 Processor -- start 2023-12-19T08:02:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 -- start start 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf0137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf0158d60 0x7f9bf0151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014a740 con 0x7f9bf00c51b0 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014a110 con 0x7f9bf0158d60 2023-12-19T08:02:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf016d400 con 0x7f9bf006b7a0 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46432/0 (socket says 172.21.15.111:46432) 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 -- 172.21.15.111:0/396260517 learned_addr learned my addr 172.21.15.111:0/396260517 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf486a700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf0158d60 0x7f9bf0151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9beffff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf0137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 msgr2=0x7f9bf0137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf0137aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf0158d60 msgr2=0x7f9bf0151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf0158d60 0x7f9bf0151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 -- 172.21.15.111:0/396260517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00094f0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9be7fff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800ec80 tx=0x7f9bd80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800c850 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd800c9d0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800cd90 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.858+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf014a3d0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.862+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf014bb90 con 0x7f9bf00c51b0 2023-12-19T08:02:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.862+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf014a8f0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.862+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f9bd8013680 con 0x7f9bf00c51b0 2023-12-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.862+0000 7f9becff9700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 0x7f9bd0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.866+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f9bd807d150 con 0x7f9bf00c51b0 2023-12-19T08:02:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.866+0000 7f9beffff700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 msgr2=0x7f9bd0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.866+0000 7f9beffff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 0x7f9bd0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:07.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:07.870+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9bf014a8f0 con 0x7f9bf00c51b0 2023-12-19T08:02:07.907 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:07.906+0000 7f2277dc1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:07.942 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:07.942+0000 7f09abf5e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:08.032 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:08.030+0000 7f2277dc1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:08.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.070+0000 7f9beffff700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 msgr2=0x7f9bd0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:08.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.070+0000 7f9beffff700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 0x7f9bd0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:08.127 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:08.130+0000 7f09abf5e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:08.147 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:08.146+0000 7f2277dc1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:08.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.154+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bf004be40 con 0x7f9bf00c51b0 2023-12-19T08:02:08.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.154+0000 7f9becff9700 1 -- 172.21.15.111:0/396260517 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7f9bd804d0d0 con 0x7f9bf00c51b0 2023-12-19T08:02:08.154 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:08.154 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 msgr2=0x7f9bd0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 0x7f9bd0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 msgr2=0x7f9bf0151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800ec80 tx=0x7f9bd80091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 shutdown_connections 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bd0060670 0x7f9bd0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf0158d60 0x7f9bf0151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf00c51b0 0x7f9bf0151a80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 --2- 172.21.15.111:0/396260517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf006b7a0 0x7f9bf0137aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 >> 172.21.15.111:0/396260517 conn(0x7f9bf0067bb0 msgr2=0x7f9bf00c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 shutdown_connections 2023-12-19T08:02:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.158+0000 7f9bf506b700 1 -- 172.21.15.111:0/396260517 wait complete. 2023-12-19T08:02:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:02:08.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:08.209 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:08.210+0000 7f09abf5e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.338+0000 7f9dbf59e700 1 Processor -- start 2023-12-19T08:02:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.338+0000 7f9dbf59e700 1 -- start start 2023-12-19T08:02:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbf59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9dc0155960 0x7f9dc015a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbf59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc00c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbf59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc006d990 con 0x7f9dc006c7c0 2023-12-19T08:02:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbf59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc015a570 con 0x7f9dc0155960 2023-12-19T08:02:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbf59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc006cd80 con 0x7f9dc006de20 2023-12-19T08:02:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9dc0155960 0x7f9dc015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbdd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc00c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9dc0155960 0x7f9dc015a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:57746/0 (socket says 172.21.15.111:57746) 2023-12-19T08:02:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 -- 172.21.15.111:0/3055983366 learned_addr learned my addr 172.21.15.111:0/3055983366 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 -- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 msgr2=0x7f9dc00c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc00c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 -- 172.21.15.111:0/3055983366 >> v1:172.21.15.143:6789/0 conn(0x7f9dc006c7c0 legacy=0x7f9dc00c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 -- 172.21.15.111:0/3055983366 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc006d060 con 0x7f9dc0155960 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbdd9b700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc00c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbd59a700 1 -- 172.21.15.111:0/3055983366 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652694698 0 0) 0x7f9dc006d990 con 0x7f9dc006c7c0 2023-12-19T08:02:08.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbe59c700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc0155960 0x7f9dc015a030 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9db00099b0 tx=0x7f9db0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35ae215a9f67ae05 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:08.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbd59a700 1 -- 172.21.15.111:0/3055983366 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0009ea0 con 0x7f9dc0155960 2023-12-19T08:02:08.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbd59a700 1 -- 172.21.15.111:0/3055983366 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db000b860 con 0x7f9dc0155960 2023-12-19T08:02:08.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.342+0000 7f9dbd59a700 1 -- 172.21.15.111:0/3055983366 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db000bc00 con 0x7f9dc0155960 2023-12-19T08:02:08.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc0155960 msgr2=0x7f9dc015a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc0155960 0x7f9dc015a030 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9db00099b0 tx=0x7f9db0009250 comp rx=0 tx=0).stop 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- 172.21.15.111:0/3055983366 shutdown_connections 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc00c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/3055983366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc0155960 0x7f9dc015a030 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- 172.21.15.111:0/3055983366 >> 172.21.15.111:0/3055983366 conn(0x7f9dc0067bb0 msgr2=0x7f9dc00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- 172.21.15.111:0/3055983366 shutdown_connections 2023-12-19T08:02:08.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- 172.21.15.111:0/3055983366 wait complete. 2023-12-19T08:02:08.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 Processor -- start 2023-12-19T08:02:08.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.346+0000 7f9dbf59e700 1 -- start start 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc0152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dc0155960 0x7f9dc01483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc01539a0 con 0x7f9dc0155960 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0153c80 con 0x7f9dc006c7c0 2023-12-19T08:02:08.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0153f60 con 0x7f9dc006de20 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:57756/0 (socket says 172.21.15.111:57756) 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dc0155960 0x7f9dc01483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 -- 172.21.15.111:0/1336327938 learned_addr learned my addr 172.21.15.111:0/1336327938 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbdd9b700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc0152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 msgr2=0x7f9dc0152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc0152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dc0155960 msgr2=0x7f9dc01483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dc0155960 0x7f9dc01483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 -- 172.21.15.111:0/1336327938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db4009420 con 0x7f9dc006c7c0 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbe59c700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0064040 tx=0x7f9db0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0000d10 con 0x7f9dc006c7c0 2023-12-19T08:02:08.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db000bbe0 con 0x7f9dc006c7c0 2023-12-19T08:02:08.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0011690 con 0x7f9dc006c7c0 2023-12-19T08:02:08.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db00094f0 con 0x7f9dc006c7c0 2023-12-19T08:02:08.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.350+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc014b6a0 con 0x7f9dc006c7c0 2023-12-19T08:02:08.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc004be40 con 0x7f9dc006c7c0 2023-12-19T08:02:08.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f9db0011ac0 con 0x7f9dc006c7c0 2023-12-19T08:02:08.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dae7fc700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 0x7f9da0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f9db007e690 con 0x7f9dc006c7c0 2023-12-19T08:02:08.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dbdd9b700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 msgr2=0x7f9da0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:08.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.354+0000 7f9dbdd9b700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 0x7f9da0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:08.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.362+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9db004e610 con 0x7f9dc006c7c0 2023-12-19T08:02:08.365 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:08.366+0000 7f2277dc1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:08.463 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:08.462+0000 7f2277dc1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:08.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.558+0000 7f9dbdd9b700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 msgr2=0x7f9da0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:08.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.558+0000 7f9dbdd9b700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 0x7f9da0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:08.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.586+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dc00cc350 con 0x7f9dc006c7c0 2023-12-19T08:02:08.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.586+0000 7f9dae7fc700 1 -- 172.21.15.111:0/1336327938 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4609 (secure 0 0 0) 0x7f9db0051c30 con 0x7f9dc006c7c0 2023-12-19T08:02:08.587 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:08.587 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:00:07.870107+0000","modified":"2023-12-19T08:00:11.482481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2938693255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2938693255},{"type":"v1","addr":"172.21.15.111:6835","nonce":2938693255}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 msgr2=0x7f9da0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 0x7f9da0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 msgr2=0x7f9dc00c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0064040 tx=0x7f9db0017000 comp rx=0 tx=0).stop 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 shutdown_connections 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9da0060670 0x7f9da0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dc0155960 0x7f9dc01483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc006de20 0x7f9dc0152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 --2- 172.21.15.111:0/1336327938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc006c7c0 0x7f9dc00c9730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 >> 172.21.15.111:0/1336327938 conn(0x7f9dc0067bb0 msgr2=0x7f9dc0150a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 shutdown_connections 2023-12-19T08:02:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.590+0000 7f9dbf59e700 1 -- 172.21.15.111:0/1336327938 wait complete. 2023-12-19T08:02:08.593 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-12-19T08:02:08.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:08.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 Processor -- start 2023-12-19T08:02:08.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 -- start start 2023-12-19T08:02:08.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 0x7f7ce00ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 0x7f7ce006a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce006c7c0 con 0x7f7ce015a860 2023-12-19T08:02:08.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce006cd80 con 0x7f7ce00b1410 2023-12-19T08:02:08.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce5dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce006af10 con 0x7f7ce006a580 2023-12-19T08:02:08.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7cd7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 0x7f7ce006a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce4dca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 0x7f7ce00ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce55cb700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7ce015a860 0x7f7ce015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36700/0 (socket says 172.21.15.111:36700) 2023-12-19T08:02:08.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.770+0000 7f7ce55cb700 1 -- 172.21.15.111:0/2162538313 learned_addr learned my addr 172.21.15.111:0/2162538313 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:08.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd77fe700 1 -- 172.21.15.111:0/2162538313 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692870301 0 0) 0x7f7ce006c7c0 con 0x7f7ce015a860 2023-12-19T08:02:08.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd77fe700 1 -- 172.21.15.111:0/2162538313 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc8003570 con 0x7f7ce015a860 2023-12-19T08:02:08.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd7fff700 1 -- 172.21.15.111:0/2162538313 >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 msgr2=0x7f7ce00ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:08.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd7fff700 1 --2- 172.21.15.111:0/2162538313 >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 0x7f7ce00ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd7fff700 1 -- 172.21.15.111:0/2162538313 >> v1:172.21.15.143:6789/0 conn(0x7f7ce015a860 legacy=0x7f7ce015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd7fff700 1 -- 172.21.15.111:0/2162538313 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce006b8f0 con 0x7f7ce006a580 2023-12-19T08:02:08.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce4dca700 1 --2- 172.21.15.111:0/2162538313 >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 0x7f7ce00ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:08.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd7fff700 1 --2- 172.21.15.111:0/2162538313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 0x7f7ce006a9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00098e0 tx=0x7f7cd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3662f32edb0c13ac server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:08.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd77fe700 1 -- 172.21.15.111:0/2162538313 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0011630 con 0x7f7ce006a580 2023-12-19T08:02:08.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd77fe700 1 -- 172.21.15.111:0/2162538313 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd0011790 con 0x7f7ce006a580 2023-12-19T08:02:08.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7cd77fe700 1 -- 172.21.15.111:0/2162538313 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0011b30 con 0x7f7ce006a580 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/2162538313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 msgr2=0x7f7ce006a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/2162538313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 0x7f7ce006a9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00098e0 tx=0x7f7cd00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/2162538313 shutdown_connections 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/2162538313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce006a580 0x7f7ce006a9d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/2162538313 >> v2:172.21.15.143:3300/0 conn(0x7f7ce00b1410 0x7f7ce00ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/2162538313 >> 172.21.15.111:0/2162538313 conn(0x7f7ce0067bb0 msgr2=0x7f7ce00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/2162538313 shutdown_connections 2023-12-19T08:02:08.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.774+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/2162538313 wait complete. 2023-12-19T08:02:08.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 Processor -- start 2023-12-19T08:02:08.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 -- start start 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce00b1410 0x7f7ce015e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce015a860 0x7f7ce015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce015e760 con 0x7f7ce006a580 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce015cee0 con 0x7f7ce00b1410 2023-12-19T08:02:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce5dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce015d180 con 0x7f7ce015a860 2023-12-19T08:02:08.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7cd7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce00b1410 0x7f7ce015e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46448/0 (socket says 172.21.15.111:46448) 2023-12-19T08:02:08.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 -- 172.21.15.111:0/3192562260 learned_addr learned my addr 172.21.15.111:0/3192562260 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce55cb700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce015a860 0x7f7ce015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 -- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce015a860 msgr2=0x7f7ce015c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce015a860 0x7f7ce015c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 -- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce00b1410 msgr2=0x7f7ce015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce00b1410 0x7f7ce015e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.778+0000 7f7ce4dca700 1 -- 172.21.15.111:0/3192562260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc009a20 con 0x7f7ce006a580 2023-12-19T08:02:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7ce4dca700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00c170 tx=0x7f7cdc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc00cd60 con 0x7f7ce006a580 2023-12-19T08:02:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc00f040 con 0x7f7ce006a580 2023-12-19T08:02:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd0009420 con 0x7f7ce006a580 2023-12-19T08:02:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc014690 con 0x7f7ce006a580 2023-12-19T08:02:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce015d740 con 0x7f7ce006a580 2023-12-19T08:02:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce004be40 con 0x7f7ce006a580 2023-12-19T08:02:08.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.782+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f7cdc014a00 con 0x7f7ce006a580 2023-12-19T08:02:08.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.786+0000 7f7cd4ff9700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 0x7f7ccc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:08.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.786+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6975+0+0 (secure 0 0 0) 0x7f7cdc07de40 con 0x7f7ce006a580 2023-12-19T08:02:08.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.786+0000 7f7cd7fff700 1 -- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 msgr2=0x7f7ccc062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:08.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.786+0000 7f7cd7fff700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 0x7f7ccc062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:08.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.794+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7cdc04dc80 con 0x7f7ce006a580 2023-12-19T08:02:08.948 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:08.950+0000 7f09abf5e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:08.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.970+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 24) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f7cdc04e050 con 0x7f7ce006a580 2023-12-19T08:02:08.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.970+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 msgr2=0x7f7ccc062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:08.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.970+0000 7f7cd4ff9700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 0x7f7ccc062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.990+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce00aad60 con 0x7f7ce006a580 2023-12-19T08:02:08.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.994+0000 7f7cd4ff9700 1 -- 172.21.15.111:0/3192562260 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+17383 (secure 0 0 0) 0x7f7cdc068fc0 con 0x7f7ce006a580 2023-12-19T08:02:08.994 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:08.994 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":73,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:08.957567+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:00:05.879580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-19T08:00:06.863124+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.e","mappings":[{"from":0,"to":6}]},{"pgid":"8.27","mappings":[{"from":0,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3260056484":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2460533822":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/1248091335":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2820673595":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2810180880":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/134362188":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3567649431":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/3495942537":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/189999424":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3297717311":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/934963750":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/1624932641":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6833/33796":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6832/33796":"2023-12-20T08:00:23.832713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 msgr2=0x7f7ce01581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00c170 tx=0x7f7cdc00b040 comp rx=0 tx=0).stop 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 shutdown_connections 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7ccc060670 0x7f7ccc062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce00b1410 0x7f7ce015e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce015a860 0x7f7ce015c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 --2- 172.21.15.111:0/3192562260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce006a580 0x7f7ce01581c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:08.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 >> 172.21.15.111:0/3192562260 conn(0x7f7ce0067bb0 msgr2=0x7f7ce00adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:08.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 shutdown_connections 2023-12-19T08:02:08.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:08.998+0000 7f7ce5dcc700 1 -- 172.21.15.111:0/3192562260 wait complete. 2023-12-19T08:02:09.020 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T08:02:09.058 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0976aa1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0976aa1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0976aa1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0976aa1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0976aa1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0973a9b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0973a9b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0973a9b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0973a9b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:09.062+0000 7f0973a9b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.166+0000 7f3842145700 1 Processor -- start 2023-12-19T08:02:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 -- start start 2023-12-19T08:02:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f383c140420 0x7f383c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f383c06c7c0 0x7f383c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c06d7d0 con 0x7f383c1407f0 2023-12-19T08:02:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c0ae280 con 0x7f383c06c7c0 2023-12-19T08:02:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3842145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cb8f0 con 0x7f383c140420 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f383c1407f0 0x7f383c06a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50578/0 (socket says 172.21.15.111:50578) 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383bfff700 1 -- 172.21.15.111:0/1641508343 learned_addr learned my addr 172.21.15.111:0/1641508343 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383b7fe700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f383c140420 0x7f383c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3832ffd700 1 --2- 172.21.15.111:0/1641508343 >> v2:172.21.15.143:3300/0 conn(0x7f383c06c7c0 0x7f383c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254796063 0 0) 0x7f383c06d7d0 con 0x7f383c1407f0 2023-12-19T08:02:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003570 con 0x7f383c1407f0 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2571082255 0 0) 0x7f3828003570 con 0x7f383c1407f0 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f383c140420 msgr2=0x7f383c15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f383c140420 0x7f383c15b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 >> v2:172.21.15.143:3300/0 conn(0x7f383c06c7c0 msgr2=0x7f383c0b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 --2- 172.21.15.111:0/1641508343 >> v2:172.21.15.143:3300/0 conn(0x7f383c06c7c0 0x7f383c0b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c06cc50 con 0x7f383c1407f0 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3834001470 con 0x7f383c1407f0 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 >> v1:172.21.15.111:6789/0 conn(0x7f383c1407f0 legacy=0x7f383c06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828003a60 0x7f3828003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3828004810 0x7f3828004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38280051c0 0x7f382800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ae280 con 0x7f38280051c0 2023-12-19T08:02:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cb8f0 con 0x7f3828004810 2023-12-19T08:02:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280103b0 con 0x7f3828003a60 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f3832ffd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3828004810 0x7f3828004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.170+0000 7f383b7fe700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828003a60 0x7f3828003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f383bfff700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38280051c0 0x7f382800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 -- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828003a60 msgr2=0x7f3828003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828003a60 0x7f3828003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 -- 172.21.15.111:0/1641508343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38280051c0 msgr2=0x7f382800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38280051c0 0x7f382800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 -- 172.21.15.111:0/1641508343 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828003570 con 0x7f3828004810 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3832ffd700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3828004810 0x7f3828004c80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3824009c00 tx=0x7f38240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf93124342df376c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400e710 con 0x7f3828004810 2023-12-19T08:02:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382400e870 con 0x7f3828004810 2023-12-19T08:02:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f383affd700 1 -- 172.21.15.111:0/1641508343 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400ec30 con 0x7f3828004810 2023-12-19T08:02:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 -- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3828004810 msgr2=0x7f3828004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 --2- 172.21.15.111:0/1641508343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3828004810 0x7f3828004c80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3824009c00 tx=0x7f38240091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f383bfff700 1 -- 172.21.15.111:0/1641508343 reap_dead start 2023-12-19T08:02:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 -- 172.21.15.111:0/1641508343 shutdown_connections 2023-12-19T08:02:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 -- 172.21.15.111:0/1641508343 >> 172.21.15.111:0/1641508343 conn(0x7f383c067b60 msgr2=0x7f383c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 -- 172.21.15.111:0/1641508343 shutdown_connections 2023-12-19T08:02:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.174+0000 7f3842145700 1 -- 172.21.15.111:0/1641508343 wait complete. 2023-12-19T08:02:09.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 Processor -- start 2023-12-19T08:02:09.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 -- start start 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828004810 0x7f383c14a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 0x7f383c167f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0b09c0 con 0x7f383c0affe0 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c13ae60 con 0x7f383c156610 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3842145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c168830 con 0x7f3828004810 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46480/0 (socket says 172.21.15.111:46480) 2023-12-19T08:02:09.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 -- 172.21.15.111:0/753095515 learned_addr learned my addr 172.21.15.111:0/753095515 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:09.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 -- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828004810 msgr2=0x7f383c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828004810 0x7f383c14a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f383bfff700 1 -- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 msgr2=0x7f383c167f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.178+0000 7f3832ffd700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 0x7f383c167f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f383bfff700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 0x7f383c167f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f383bfff700 1 -- 172.21.15.111:0/753095515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c168b10 con 0x7f383c0affe0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3832ffd700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 0x7f383c167f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f383bfff700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3834000c00 tx=0x7f3834001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400f910 con 0x7f383c0affe0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383400fa70 con 0x7f383c0affe0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400fe30 con 0x7f383c0affe0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c0c1f20 con 0x7f383c0affe0 2023-12-19T08:02:09.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c168470 con 0x7f383c0affe0 2023-12-19T08:02:09.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.182+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c071120 con 0x7f383c0affe0 2023-12-19T08:02:09.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.186+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f3834013600 con 0x7f383c0affe0 2023-12-19T08:02:09.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.186+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7205+0+0 (secure 0 0 0) 0x7f383407c9a0 con 0x7f383c0affe0 2023-12-19T08:02:09.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.194+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f383404cd70 con 0x7f383c0affe0 2023-12-19T08:02:09.277 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:09.278+0000 7f2277dc1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:09.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.398+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f383c06cf30 con 0x7f383c0affe0 2023-12-19T08:02:09.401 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T08:02:09.402+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T08:02:09.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.410+0000 7f3833fff700 1 -- 172.21.15.111:0/753095515 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3834050390 con 0x7f383c0affe0 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 msgr2=0x7f383c0b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3834000c00 tx=0x7f3834001e90 comp rx=0 tx=0).stop 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 shutdown_connections 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f383c156610 0x7f383c167f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0affe0 0x7f383c0b0480 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 --2- 172.21.15.111:0/753095515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3828004810 0x7f383c14a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 >> 172.21.15.111:0/753095515 conn(0x7f383c067b60 msgr2=0x7f383c13c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 shutdown_connections 2023-12-19T08:02:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.414+0000 7f3842145700 1 -- 172.21.15.111:0/753095515 wait complete. 2023-12-19T08:02:09.416 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T08:02:09.434 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T08:02:09.476 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:02:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.590+0000 7fa36daf0700 1 Processor -- start 2023-12-19T08:02:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.590+0000 7fa36daf0700 1 -- start start 2023-12-19T08:02:09.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa36daf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa36813f7e0 0x7fa3681564a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa36daf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 0x7fa3680af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa36daf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680af780 con 0x7fa3680cba00 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa36daf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa368157950 con 0x7fa36813fbb0 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa36daf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa368156af0 con 0x7fa36813f7e0 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa366ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 0x7fa3680af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa367fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa3680cba00 0x7fa368159a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50594/0 (socket says 172.21.15.111:50594) 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa367fff700 1 -- 172.21.15.111:0/3412898944 learned_addr learned my addr 172.21.15.111:0/3412898944 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:09.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3677fe700 1 --2- 172.21.15.111:0/3412898944 >> v2:172.21.15.143:3300/0 conn(0x7fa36813f7e0 0x7fa3681564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3667fc700 1 -- 172.21.15.111:0/3412898944 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341373798 0 0) 0x7fa3680af780 con 0x7fa3680cba00 2023-12-19T08:02:09.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3667fc700 1 -- 172.21.15.111:0/3412898944 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa350003570 con 0x7fa3680cba00 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3677fe700 1 -- 172.21.15.111:0/3412898944 >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 msgr2=0x7fa3680af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3677fe700 1 --2- 172.21.15.111:0/3412898944 >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 0x7fa3680af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3677fe700 1 -- 172.21.15.111:0/3412898944 >> v1:172.21.15.111:6789/0 conn(0x7fa3680cba00 legacy=0x7fa368159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa3677fe700 1 -- 172.21.15.111:0/3412898944 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368156dd0 con 0x7fa36813f7e0 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.594+0000 7fa366ffd700 1 --2- 172.21.15.111:0/3412898944 >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 0x7fa3680af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa3677fe700 1 --2- 172.21.15.111:0/3412898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa3681564a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa3680cde80 tx=0x7fa3580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce47d97a651232b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:09.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa3667fc700 1 -- 172.21.15.111:0/3412898944 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358011630 con 0x7fa36813f7e0 2023-12-19T08:02:09.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa3667fc700 1 -- 172.21.15.111:0/3412898944 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3580117b0 con 0x7fa36813f7e0 2023-12-19T08:02:09.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa3667fc700 1 -- 172.21.15.111:0/3412898944 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358011b70 con 0x7fa36813f7e0 2023-12-19T08:02:09.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 -- 172.21.15.111:0/3412898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 msgr2=0x7fa3681564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 --2- 172.21.15.111:0/3412898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa3681564a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa3680cde80 tx=0x7fa3580091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 -- 172.21.15.111:0/3412898944 shutdown_connections 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 --2- 172.21.15.111:0/3412898944 >> v2:172.21.15.111:3300/0 conn(0x7fa36813fbb0 0x7fa3680af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 --2- 172.21.15.111:0/3412898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa3681564a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 -- 172.21.15.111:0/3412898944 >> 172.21.15.111:0/3412898944 conn(0x7fa3680cd8f0 msgr2=0x7fa3680c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 -- 172.21.15.111:0/3412898944 shutdown_connections 2023-12-19T08:02:09.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.598+0000 7fa36daf0700 1 -- 172.21.15.111:0/3412898944 wait complete. 2023-12-19T08:02:09.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.602+0000 7fa36daf0700 1 Processor -- start 2023-12-19T08:02:09.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.602+0000 7fa36daf0700 1 -- start start 2023-12-19T08:02:09.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa368165420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa36813fbb0 0x7fa368165960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36813b910 con 0x7fa3680cba00 2023-12-19T08:02:09.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa368166330 con 0x7fa36813f7e0 2023-12-19T08:02:09.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa368166050 con 0x7fa36813fbb0 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46520/0 (socket says 172.21.15.111:46520) 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 -- 172.21.15.111:0/4072176939 learned_addr learned my addr 172.21.15.111:0/4072176939 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 -- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa36813fbb0 msgr2=0x7fa368165960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa36813fbb0 0x7fa368165960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa366ffd700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa368165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 -- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 msgr2=0x7fa368165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa368165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:09.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 -- 172.21.15.111:0/4072176939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009420 con 0x7fa3680cba00 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa366ffd700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa368165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa3677fe700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa358013000 tx=0x7fa358017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35801f070 con 0x7fa3680cba00 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa358011920 con 0x7fa3680cba00 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358023410 con 0x7fa3680cba00 2023-12-19T08:02:09.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358009420 con 0x7fa3680cba00 2023-12-19T08:02:09.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa368168e10 con 0x7fa3680cba00 2023-12-19T08:02:09.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.606+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368071120 con 0x7fa3680cba00 2023-12-19T08:02:09.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.610+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fa35800b400 con 0x7fa3680cba00 2023-12-19T08:02:09.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.614+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7291+0+0 (secure 0 0 0) 0x7fa35807dd30 con 0x7fa3680cba00 2023-12-19T08:02:09.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.618+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa35804dfc0 con 0x7fa3680cba00 2023-12-19T08:02:09.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.826+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3681570b0 con 0x7fa3680cba00 2023-12-19T08:02:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.982+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 25) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fa35804da70 con 0x7fa3680cba00 2023-12-19T08:02:09.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.982+0000 7fa34f7fe700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 0x7fa354068d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:09.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.986+0000 7fa366ffd700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 0x7fa354068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:09.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:09.990+0000 7fa366ffd700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 0x7fa354068d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0093f0 tx=0x7fa35c008040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.466+0000 7fa34f7fe700 1 -- 172.21.15.111:0/4072176939 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa358053c40 con 0x7fa3680cba00 2023-12-19T08:02:10.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 msgr2=0x7fa354068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 0x7fa354068d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0093f0 tx=0x7fa35c008040 comp rx=0 tx=0).stop 2023-12-19T08:02:10.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 msgr2=0x7fa368157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa358013000 tx=0x7fa358017000 comp rx=0 tx=0).stop 2023-12-19T08:02:10.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 shutdown_connections 2023-12-19T08:02:10.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa354066930 0x7fa354068d10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa36813f7e0 0x7fa368165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa36813fbb0 0x7fa368165960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 --2- 172.21.15.111:0/4072176939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3680cba00 0x7fa368157fd0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 >> 172.21.15.111:0/4072176939 conn(0x7fa3680cd8f0 msgr2=0x7fa3680cb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 shutdown_connections 2023-12-19T08:02:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.474+0000 7fa36daf0700 1 -- 172.21.15.111:0/4072176939 wait complete. 2023-12-19T08:02:10.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T08:02:10.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.710+0000 7f0e889c4700 1 Processor -- start 2023-12-19T08:02:10.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.710+0000 7f0e889c4700 1 -- start start 2023-12-19T08:02:10.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0e840af190 0x7f0e8406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:10.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e840cb3f0 con 0x7f0e840aedc0 2023-12-19T08:02:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8406d220 con 0x7f0e840cac50 2023-12-19T08:02:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e840cc140 con 0x7f0e840af190 2023-12-19T08:02:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e8259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0e840af190 0x7f0e8406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e82d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0e840aedc0 0x7f0e8406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50596/0 (socket says 172.21.15.111:50596) 2023-12-19T08:02:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e82d9d700 1 -- 172.21.15.111:0/682036902 learned_addr learned my addr 172.21.15.111:0/682036902 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e81d9b700 1 -- 172.21.15.111:0/682036902 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132462365 0 0) 0x7f0e840cb3f0 con 0x7f0e840aedc0 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e81d9b700 1 -- 172.21.15.111:0/682036902 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e6c003570 con 0x7f0e840aedc0 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e8259c700 1 -- 172.21.15.111:0/682036902 >> v1:172.21.15.143:6789/0 conn(0x7f0e840cac50 legacy=0x7f0e840af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e8259c700 1 -- 172.21.15.111:0/682036902 >> v1:172.21.15.111:6789/0 conn(0x7f0e840aedc0 legacy=0x7f0e8406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e8259c700 1 -- 172.21.15.111:0/682036902 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8406c8f0 con 0x7f0e840af190 2023-12-19T08:02:10.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e8259c700 1 --2- 172.21.15.111:0/682036902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840af190 0x7f0e8406af50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0e780098e0 tx=0x7f0e780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706f9d701dd58df3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:10.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e81d9b700 1 -- 172.21.15.111:0/682036902 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800b640 con 0x7f0e840af190 2023-12-19T08:02:10.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e81d9b700 1 -- 172.21.15.111:0/682036902 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7800b7a0 con 0x7f0e840af190 2023-12-19T08:02:10.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e81d9b700 1 -- 172.21.15.111:0/682036902 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800bb40 con 0x7f0e840af190 2023-12-19T08:02:10.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 -- 172.21.15.111:0/682036902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840af190 msgr2=0x7f0e8406af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.714+0000 7f0e889c4700 1 --2- 172.21.15.111:0/682036902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840af190 0x7f0e8406af50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0e780098e0 tx=0x7f0e780091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- 172.21.15.111:0/682036902 shutdown_connections 2023-12-19T08:02:10.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 --2- 172.21.15.111:0/682036902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840af190 0x7f0e8406af50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- 172.21.15.111:0/682036902 >> 172.21.15.111:0/682036902 conn(0x7f0e840cc760 msgr2=0x7f0e840ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:10.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- 172.21.15.111:0/682036902 shutdown_connections 2023-12-19T08:02:10.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- 172.21.15.111:0/682036902 wait complete. 2023-12-19T08:02:10.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 Processor -- start 2023-12-19T08:02:10.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- start start 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e840af190 0x7f0e840b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e840cac50 0x7f0e840b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840af610 con 0x7f0e840aedc0 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840c51b0 con 0x7f0e840cac50 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.718+0000 7f0e889c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e84168f40 con 0x7f0e840af190 2023-12-19T08:02:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:10.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46550/0 (socket says 172.21.15.111:46550) 2023-12-19T08:02:10.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 -- 172.21.15.111:0/706537124 learned_addr learned my addr 172.21.15.111:0/706537124 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:10.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e8259c700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e840af190 0x7f0e840b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:10.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e8359e700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e840cac50 0x7f0e840b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 -- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e840af190 msgr2=0x7f0e840b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e840af190 0x7f0e840b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 -- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e840cac50 msgr2=0x7f0e840b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e840cac50 0x7f0e840b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 -- 172.21.15.111:0/706537124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78009420 con 0x7f0e840aedc0 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e82d9d700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c590 tx=0x7f0e74007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74005b30 con 0x7f0e840aedc0 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74005c90 con 0x7f0e840aedc0 2023-12-19T08:02:10.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e840b5140 con 0x7f0e840aedc0 2023-12-19T08:02:10.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74007320 con 0x7f0e840aedc0 2023-12-19T08:02:10.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.722+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e840b5700 con 0x7f0e840aedc0 2023-12-19T08:02:10.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.726+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8406ae40 con 0x7f0e840aedc0 2023-12-19T08:02:10.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.726+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f0e7401b3b0 con 0x7f0e840aedc0 2023-12-19T08:02:10.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.726+0000 7f0e6affd700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 0x7f0e70062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:10.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.726+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7291+0+0 (secure 0 0 0) 0x7f0e7407bef0 con 0x7f0e840aedc0 2023-12-19T08:02:10.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.726+0000 7f0e8259c700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 0x7f0e70062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:10.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.730+0000 7f0e8259c700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 0x7f0e70062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e780093f0 tx=0x7f0e78017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.734+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e7404c090 con 0x7f0e840aedc0 2023-12-19T08:02:10.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:10.902+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e8404be40 con 0x7f0e840aedc0 2023-12-19T08:02:11.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.486+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0e7404c460 con 0x7f0e840aedc0 2023-12-19T08:02:11.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.494+0000 7f0e6affd700 1 -- 172.21.15.111:0/706537124 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v75) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0e7404f890 con 0x7f0e840aedc0 2023-12-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.498+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 msgr2=0x7f0e70062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.498+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 0x7f0e70062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e780093f0 tx=0x7f0e78017000 comp rx=0 tx=0).stop 2023-12-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 msgr2=0x7f0e8415a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c590 tx=0x7f0e74007e00 comp rx=0 tx=0).stop 2023-12-19T08:02:11.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 shutdown_connections 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e700603a0 0x7f0e70062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e840cac50 0x7f0e840b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e840af190 0x7f0e840b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 --2- 172.21.15.111:0/706537124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e840aedc0 0x7f0e8415a1b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 >> 172.21.15.111:0/706537124 conn(0x7f0e840cc760 msgr2=0x7f0e840ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 shutdown_connections 2023-12-19T08:02:11.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.502+0000 7f0e889c4700 1 -- 172.21.15.111:0/706537124 wait complete. 2023-12-19T08:02:11.504 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T08:02:11.522 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T08:02:11.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.686+0000 7f914b385700 1 Processor -- start 2023-12-19T08:02:11.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.686+0000 7f914b385700 1 -- start start 2023-12-19T08:02:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f914406dc40 0x7f91440abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914406d7b0 con 0x7f91440ac480 2023-12-19T08:02:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440ae570 con 0x7f9144156e70 2023-12-19T08:02:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91440ac8a0 con 0x7f914406dc40 2023-12-19T08:02:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9148920700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f914406dc40 0x7f91440abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9149922700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f91440ac480 0x7f91440aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50598/0 (socket says 172.21.15.111:50598) 2023-12-19T08:02:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9149922700 1 -- 172.21.15.111:0/3891894457 learned_addr learned my addr 172.21.15.111:0/3891894457 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9149121700 1 --1- 172.21.15.111:0/3891894457 >> v1:172.21.15.143:6789/0 conn(0x7f9144156e70 0x7f914406b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36712/0 (socket says 172.21.15.111:36712) 2023-12-19T08:02:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f913bfff700 1 -- 172.21.15.111:0/3891894457 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816317835 0 0) 0x7f91440ae570 con 0x7f9144156e70 2023-12-19T08:02:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f913bfff700 1 -- 172.21.15.111:0/3891894457 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130003570 con 0x7f9144156e70 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9148920700 1 -- 172.21.15.111:0/3891894457 >> v1:172.21.15.143:6789/0 conn(0x7f9144156e70 legacy=0x7f914406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9148920700 1 -- 172.21.15.111:0/3891894457 >> v1:172.21.15.111:6789/0 conn(0x7f91440ac480 legacy=0x7f91440aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9148920700 1 -- 172.21.15.111:0/3891894457 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914406e0e0 con 0x7f914406dc40 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f9148920700 1 --2- 172.21.15.111:0/3891894457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f914406dc40 0x7f91440abf40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f91340098e0 tx=0x7f91340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35de45f1500b86dd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f913bfff700 1 -- 172.21.15.111:0/3891894457 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400b640 con 0x7f914406dc40 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f913bfff700 1 -- 172.21.15.111:0/3891894457 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400b7a0 con 0x7f914406dc40 2023-12-19T08:02:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f913bfff700 1 -- 172.21.15.111:0/3891894457 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400bb40 con 0x7f914406dc40 2023-12-19T08:02:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 -- 172.21.15.111:0/3891894457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f914406dc40 msgr2=0x7f91440abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.690+0000 7f914b385700 1 --2- 172.21.15.111:0/3891894457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f914406dc40 0x7f91440abf40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f91340098e0 tx=0x7f91340091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 -- 172.21.15.111:0/3891894457 shutdown_connections 2023-12-19T08:02:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 --2- 172.21.15.111:0/3891894457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f914406dc40 0x7f91440abf40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 -- 172.21.15.111:0/3891894457 >> 172.21.15.111:0/3891894457 conn(0x7f9144067bb0 msgr2=0x7f91440cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 -- 172.21.15.111:0/3891894457 shutdown_connections 2023-12-19T08:02:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 -- 172.21.15.111:0/3891894457 wait complete. 2023-12-19T08:02:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 Processor -- start 2023-12-19T08:02:11.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.694+0000 7f914b385700 1 -- start start 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 0x7f91440c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9144156e70 0x7f91440c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9144153630 con 0x7f91440ac480 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9144153910 con 0x7f9144156e70 2023-12-19T08:02:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9144153bf0 con 0x7f914406dc40 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46566/0 (socket says 172.21.15.111:46566) 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 -- 172.21.15.111:0/1710404286 learned_addr learned my addr 172.21.15.111:0/1710404286 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9149922700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9144156e70 0x7f91440c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9149121700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 0x7f91440c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 -- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 msgr2=0x7f91440c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 0x7f91440c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 -- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9144156e70 msgr2=0x7f91440c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:11.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9144156e70 0x7f91440c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:11.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 -- 172.21.15.111:0/1710404286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9134009420 con 0x7f91440ac480 2023-12-19T08:02:11.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9149121700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 0x7f91440c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:11.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f9148920700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9144068360 tx=0x7f9134017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:11.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134000d10 con 0x7f91440ac480 2023-12-19T08:02:11.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400bb20 con 0x7f91440ac480 2023-12-19T08:02:11.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134011630 con 0x7f91440ac480 2023-12-19T08:02:11.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9144153ed0 con 0x7f91440ac480 2023-12-19T08:02:11.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.698+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91441532e0 con 0x7f91440ac480 2023-12-19T08:02:11.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.702+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9144156a70 con 0x7f91440ac480 2023-12-19T08:02:11.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.702+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f91340119a0 con 0x7f91440ac480 2023-12-19T08:02:11.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.706+0000 7f91397fa700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 0x7f9128062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:11.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.706+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6853+0+0 (secure 0 0 0) 0x7f913407e1c0 con 0x7f91440ac480 2023-12-19T08:02:11.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.706+0000 7f9149121700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 0x7f9128062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:11.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.706+0000 7f9149121700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 0x7f9128062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9140005f00 tx=0x7f9140007c60 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:11.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.710+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9144156a70 con 0x7f91440ac480 2023-12-19T08:02:11.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:11.882+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f914406cef0 con 0x7f91440ac480 2023-12-19T08:02:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.506+0000 7f91397fa700 1 -- 172.21.15.111:0/1710404286 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v76) v1 ==== 169+0+0 (secure 0 0 0) 0x7f914406cef0 con 0x7f91440ac480 2023-12-19T08:02:12.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 msgr2=0x7f9128062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 0x7f9128062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9140005f00 tx=0x7f9140007c60 comp rx=0 tx=0).stop 2023-12-19T08:02:12.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 msgr2=0x7f91440c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9144068360 tx=0x7f9134017000 comp rx=0 tx=0).stop 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 shutdown_connections 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9128060670 0x7f9128062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9144156e70 0x7f91440c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91440ac480 0x7f91440c4dc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 --2- 172.21.15.111:0/1710404286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f914406dc40 0x7f91440c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 >> 172.21.15.111:0/1710404286 conn(0x7f9144067bb0 msgr2=0x7f91440b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 shutdown_connections 2023-12-19T08:02:12.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.514+0000 7f914b385700 1 -- 172.21.15.111:0/1710404286 wait complete. 2023-12-19T08:02:12.517 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T08:02:12.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T08:02:12.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.702+0000 7f4333f85700 1 Processor -- start 2023-12-19T08:02:12.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.702+0000 7f4333f85700 1 -- start start 2023-12-19T08:02:12.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4333f85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f432c0cab10 0x7f432c0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4333f85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f432c06b1b0 0x7f432c06b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4333f85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c0c7450 con 0x7f432c15d410 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4333f85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c06bea0 con 0x7f432c0cab10 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4333f85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c0c6e00 con 0x7f432c06b1b0 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f432c0cab10 0x7f432c0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f432c0cab10 0x7f432c0ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36642/0 (socket says 172.21.15.111:36642) 2023-12-19T08:02:12.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 -- 172.21.15.111:0/3332114224 learned_addr learned my addr 172.21.15.111:0/3332114224 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:12.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 -- 172.21.15.111:0/3332114224 >> v2:172.21.15.143:3300/0 conn(0x7f432c06b1b0 msgr2=0x7f432c06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:12.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 --2- 172.21.15.111:0/3332114224 >> v2:172.21.15.143:3300/0 conn(0x7f432c06b1b0 0x7f432c06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 -- 172.21.15.111:0/3332114224 >> v1:172.21.15.111:6789/0 conn(0x7f432c15d410 legacy=0x7f432c0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:12.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 -- 172.21.15.111:0/3332114224 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c0c70e0 con 0x7f432c0cab10 2023-12-19T08:02:12.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.706+0000 7f4331520700 1 --2- 172.21.15.111:0/3332114224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c0cab10 0x7f432c0ab6c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f432c06a5a0 tx=0x7f431c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34127bf402c6693e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4330d1f700 1 -- 172.21.15.111:0/3332114224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00b860 con 0x7f432c0cab10 2023-12-19T08:02:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4330d1f700 1 -- 172.21.15.111:0/3332114224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00b9c0 con 0x7f432c0cab10 2023-12-19T08:02:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4330d1f700 1 -- 172.21.15.111:0/3332114224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00bd60 con 0x7f432c0cab10 2023-12-19T08:02:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 -- 172.21.15.111:0/3332114224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c0cab10 msgr2=0x7f432c0ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 --2- 172.21.15.111:0/3332114224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c0cab10 0x7f432c0ab6c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f432c06a5a0 tx=0x7f431c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 -- 172.21.15.111:0/3332114224 shutdown_connections 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 --2- 172.21.15.111:0/3332114224 >> v2:172.21.15.143:3300/0 conn(0x7f432c06b1b0 0x7f432c06b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 --2- 172.21.15.111:0/3332114224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c0cab10 0x7f432c0ab6c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 -- 172.21.15.111:0/3332114224 >> 172.21.15.111:0/3332114224 conn(0x7f432c067bb0 msgr2=0x7f432c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 -- 172.21.15.111:0/3332114224 shutdown_connections 2023-12-19T08:02:12.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.710+0000 7f4333f85700 1 -- 172.21.15.111:0/3332114224 wait complete. 2023-12-19T08:02:12.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 Processor -- start 2023-12-19T08:02:12.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 -- start start 2023-12-19T08:02:12.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f432c0cab10 0x7f432c1718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f432c15d410 0x7f432c0b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0afd90 con 0x7f432c06b1b0 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0b0760 con 0x7f432c0cab10 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4333f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0af180 con 0x7f432c15d410 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36658/0 (socket says 172.21.15.111:36658) 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 -- 172.21.15.111:0/614985345 learned_addr learned my addr 172.21.15.111:0/614985345 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4332522700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f432c15d410 0x7f432c0b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331520700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f432c0cab10 0x7f432c1718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 -- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f432c15d410 msgr2=0x7f432c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f432c15d410 0x7f432c0b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 -- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f432c0cab10 msgr2=0x7f432c1718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f432c0cab10 0x7f432c1718e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.714+0000 7f4331d21700 1 -- 172.21.15.111:0/614985345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0094f0 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4331d21700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f432800c590 tx=0x7f432800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328011590 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43280116f0 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c0af460 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328011ab0 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c0afa20 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.718+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c071120 con 0x7f432c06b1b0 2023-12-19T08:02:12.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.722+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f432801b3b0 con 0x7f432c06b1b0 2023-12-19T08:02:12.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.722+0000 7f4321ffb700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 0x7f4310062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:12.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.722+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f432807c190 con 0x7f432c06b1b0 2023-12-19T08:02:12.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.722+0000 7f4331520700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 0x7f4310062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:12.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.722+0000 7f4331520700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 0x7f4310062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f432c06a5a0 tx=0x7f431c017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:12.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.730+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f432804c3d0 con 0x7f432c06b1b0 2023-12-19T08:02:12.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.922+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f432c0ae770 con 0x7f432c06b1b0 2023-12-19T08:02:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.926+0000 7f4321ffb700 1 -- 172.21.15.111:0/614985345 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v19) v1 ==== 120+0+0 (secure 0 0 0) 0x7f432804f9f0 con 0x7f432c06b1b0 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 msgr2=0x7f4310062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 0x7f4310062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f432c06a5a0 tx=0x7f431c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 msgr2=0x7f432c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f432800c590 tx=0x7f432800c950 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 shutdown_connections 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4310060670 0x7f4310062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f432c0cab10 0x7f432c1718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f432c15d410 0x7f432c0b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 --2- 172.21.15.111:0/614985345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f432c06b1b0 0x7f432c0aeba0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 >> 172.21.15.111:0/614985345 conn(0x7f432c067bb0 msgr2=0x7f432c0add30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 shutdown_connections 2023-12-19T08:02:12.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:12.930+0000 7f4333f85700 1 -- 172.21.15.111:0/614985345 wait complete. 2023-12-19T08:02:12.954 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T08:02:13.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.122+0000 7f1fc0bc3700 1 Processor -- start 2023-12-19T08:02:13.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.122+0000 7f1fc0bc3700 1 -- start start 2023-12-19T08:02:13.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fc0bc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fbc158c90 0x7f1fbc0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fc0bc3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1fbc06aeb0 0x7f1fbc159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fc0bc3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc06c850 con 0x7f1fbc1595a0 2023-12-19T08:02:13.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fc0bc3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc06b470 con 0x7f1fbc06aeb0 2023-12-19T08:02:13.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fc0bc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc06b750 con 0x7f1fbc158c90 2023-12-19T08:02:13.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1fbc1595a0 0x7f1fbc15c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50564/0 (socket says 172.21.15.111:50564) 2023-12-19T08:02:13.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbbfff700 1 -- 172.21.15.111:0/2067321939 learned_addr learned my addr 172.21.15.111:0/2067321939 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:13.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 --2- 172.21.15.111:0/2067321939 >> v2:172.21.15.111:3300/0 conn(0x7f1fbc06aeb0 0x7f1fbc159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbb7fe700 1 --2- 172.21.15.111:0/2067321939 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc158c90 0x7f1fbc0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 -- 172.21.15.111:0/2067321939 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc158c90 msgr2=0x7f1fbc0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 --2- 172.21.15.111:0/2067321939 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc158c90 0x7f1fbc0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 -- 172.21.15.111:0/2067321939 >> v1:172.21.15.111:6789/0 conn(0x7f1fbc1595a0 legacy=0x7f1fbc15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 -- 172.21.15.111:0/2067321939 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc06ba30 con 0x7f1fbc06aeb0 2023-12-19T08:02:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.126+0000 7f1fbaffd700 1 --2- 172.21.15.111:0/2067321939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc06aeb0 0x7f1fbc159060 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0698f0 tx=0x7f1fb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb76dcfa838a751 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fba7fc700 1 -- 172.21.15.111:0/2067321939 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0011630 con 0x7f1fbc06aeb0 2023-12-19T08:02:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fba7fc700 1 -- 172.21.15.111:0/2067321939 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0011790 con 0x7f1fbc06aeb0 2023-12-19T08:02:13.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fba7fc700 1 -- 172.21.15.111:0/2067321939 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0011b30 con 0x7f1fbc06aeb0 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2067321939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc06aeb0 msgr2=0x7f1fbc159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2067321939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc06aeb0 0x7f1fbc159060 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0698f0 tx=0x7f1fb00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2067321939 shutdown_connections 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2067321939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc06aeb0 0x7f1fbc159060 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2067321939 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc158c90 0x7f1fbc0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2067321939 >> 172.21.15.111:0/2067321939 conn(0x7f1fbc067bb0 msgr2=0x7f1fbc0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:13.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2067321939 shutdown_connections 2023-12-19T08:02:13.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2067321939 wait complete. 2023-12-19T08:02:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 Processor -- start 2023-12-19T08:02:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.130+0000 7f1fc0bc3700 1 -- start start 2023-12-19T08:02:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc158c90 0x7f1fbc1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc1595a0 0x7f1fbc1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c6040 con 0x7f1fbc154d60 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc157b30 con 0x7f1fbc158c90 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc155220 con 0x7f1fbc1595a0 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36682/0 (socket says 172.21.15.111:36682) 2023-12-19T08:02:13.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 -- 172.21.15.111:0/2717262374 learned_addr learned my addr 172.21.15.111:0/2717262374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbb7fe700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc158c90 0x7f1fbc1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbaffd700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc1595a0 0x7f1fbc1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 -- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc1595a0 msgr2=0x7f1fbc1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc1595a0 0x7f1fbc1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 -- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc158c90 msgr2=0x7f1fbc1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc158c90 0x7f1fbc1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 -- 172.21.15.111:0/2717262374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0009420 con 0x7f1fbc154d60 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fbbfff700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400cbc0 tx=0x7f1fb40019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb400a7e0 con 0x7f1fbc154d60 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb400a940 con 0x7f1fbc154d60 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc155500 con 0x7f1fbc154d60 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb400acc0 con 0x7f1fbc154d60 2023-12-19T08:02:13.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.134+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc155ac0 con 0x7f1fbc154d60 2023-12-19T08:02:13.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.138+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc04be40 con 0x7f1fbc154d60 2023-12-19T08:02:13.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.138+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1fb4007300 con 0x7f1fbc154d60 2023-12-19T08:02:13.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.138+0000 7f1fa37fe700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 0x7f1fa8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.138+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f1fb407cfc0 con 0x7f1fbc154d60 2023-12-19T08:02:13.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.142+0000 7f1fbb7fe700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 0x7f1fa8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.142+0000 7f1fbb7fe700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 0x7f1fa8062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009db0 tx=0x7f1fac0091a0 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.146+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1fb404d150 con 0x7f1fbc154d60 2023-12-19T08:02:13.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.318+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1fbc06bd10 con 0x7f1fbc154d60 2023-12-19T08:02:13.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.330+0000 7f1fa37fe700 1 -- 172.21.15.111:0/2717262374 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1fb4050770 con 0x7f1fbc154d60 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.330+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 msgr2=0x7f1fa8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.330+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 0x7f1fa8062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009db0 tx=0x7f1fac0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.330+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 msgr2=0x7f1fbc0c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.330+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400cbc0 tx=0x7f1fb40019c0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 shutdown_connections 2023-12-19T08:02:13.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fa8060670 0x7f1fa8062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fbc154d60 0x7f1fbc0c9160 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc1595a0 0x7f1fbc1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 --2- 172.21.15.111:0/2717262374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc158c90 0x7f1fbc1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 >> 172.21.15.111:0/2717262374 conn(0x7f1fbc067bb0 msgr2=0x7f1fbc06a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 shutdown_connections 2023-12-19T08:02:13.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.334+0000 7f1fc0bc3700 1 -- 172.21.15.111:0/2717262374 wait complete. 2023-12-19T08:02:13.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T08:02:13.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.522+0000 7fc3e9fcc700 1 Processor -- start 2023-12-19T08:02:13.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.522+0000 7fc3e9fcc700 1 -- start start 2023-12-19T08:02:13.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e9fcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3e4141e70 0x7fc3e40c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e9fcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e40c53f0 0x7fc3e415c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e9fcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e4159b80 con 0x7fc3e40c7a80 2023-12-19T08:02:13.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e9fcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e40c8220 con 0x7fc3e4141e70 2023-12-19T08:02:13.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e9fcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40c7f40 con 0x7fc3e40c53f0 2023-12-19T08:02:13.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3e4141e70 0x7fc3e40c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e97cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e40c53f0 0x7fc3e415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3e4141e70 0x7fc3e40c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38676/0 (socket says 172.21.15.111:38676) 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 -- 172.21.15.111:0/1564907434 learned_addr learned my addr 172.21.15.111:0/1564907434 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 -- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e40c53f0 msgr2=0x7fc3e415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 --2- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e40c53f0 0x7fc3e415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 -- 172.21.15.111:0/1564907434 >> v1:172.21.15.143:6789/0 conn(0x7fc3e40c7a80 legacy=0x7fc3e40c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 -- 172.21.15.111:0/1564907434 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4159530 con 0x7fc3e4141e70 2023-12-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e37fe700 1 -- 172.21.15.111:0/1564907434 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036373080 0 0) 0x7fc3e4159b80 con 0x7fc3e40c7a80 2023-12-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e8fca700 1 --2- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4141e70 0x7fc3e40c7540 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80099b0 tx=0x7fc3d8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23721b34a2e86255 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e37fe700 1 -- 172.21.15.111:0/1564907434 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8009ea0 con 0x7fc3e4141e70 2023-12-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e37fe700 1 -- 172.21.15.111:0/1564907434 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800b860 con 0x7fc3e4141e70 2023-12-19T08:02:13.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.526+0000 7fc3e37fe700 1 -- 172.21.15.111:0/1564907434 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800bc00 con 0x7fc3e4141e70 2023-12-19T08:02:13.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4141e70 msgr2=0x7fc3e40c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4141e70 0x7fc3e40c7540 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80099b0 tx=0x7fc3d8009250 comp rx=0 tx=0).stop 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/1564907434 shutdown_connections 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e40c53f0 0x7fc3e415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/1564907434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e4141e70 0x7fc3e40c7540 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/1564907434 >> 172.21.15.111:0/1564907434 conn(0x7fc3e4067b50 msgr2=0x7fc3e40cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/1564907434 shutdown_connections 2023-12-19T08:02:13.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/1564907434 wait complete. 2023-12-19T08:02:13.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 Processor -- start 2023-12-19T08:02:13.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.530+0000 7fc3e9fcc700 1 -- start start 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4141e70 0x7fc3e413a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40b19c0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e413ed60 con 0x7fc3e40c53f0 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e413e430 con 0x7fc3e4141e70 2023-12-19T08:02:13.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e8fca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e97cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4141e70 0x7fc3e413a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e8fca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38682/0 (socket says 172.21.15.111:38682) 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36686/0 (socket says 172.21.15.111:36686) 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e8fca700 1 -- 172.21.15.111:0/680579393 learned_addr learned my addr 172.21.15.111:0/680579393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:13.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 -- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4141e70 msgr2=0x7fc3e413a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4141e70 0x7fc3e413a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 -- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 msgr2=0x7fc3e4157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 -- 172.21.15.111:0/680579393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80094f0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e8fca700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:13.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e3fff700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00011e0 tx=0x7fc3d0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011590 con 0x7fc3e40c7a80 2023-12-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0011710 con 0x7fc3e40c7a80 2023-12-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011ad0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.534+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e413e710 con 0x7fc3e40c7a80 2023-12-19T08:02:13.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.538+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e413aa10 con 0x7fc3e40c7a80 2023-12-19T08:02:13.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.538+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e404be40 con 0x7fc3e40c7a80 2023-12-19T08:02:13.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.538+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc3d001b3b0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.542+0000 7fc3e0ff9700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 0x7fc3c8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.542+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fc3d007bf70 con 0x7fc3e40c7a80 2023-12-19T08:02:13.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.542+0000 7fc3e8fca700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 0x7fc3c8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.542+0000 7fc3e8fca700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 0x7fc3c8062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009470 tx=0x7fc3d8017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.546+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc3d004c1b0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.718+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc3e4159810 con 0x7fc3e40c7a80 2023-12-19T08:02:13.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e0ff9700 1 -- 172.21.15.111:0/680579393 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc3d004f7d0 con 0x7fc3e40c7a80 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 msgr2=0x7fc3c8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 0x7fc3c8062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009470 tx=0x7fc3d8017000 comp rx=0 tx=0).stop 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 msgr2=0x7fc3e413f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00011e0 tx=0x7fc3d0007d30 comp rx=0 tx=0).stop 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 shutdown_connections 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc3c80606a0 0x7fc3c8062b50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4141e70 0x7fc3e413a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3e40c7a80 0x7fc3e413f1f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 --2- 172.21.15.111:0/680579393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c53f0 0x7fc3e4157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 >> 172.21.15.111:0/680579393 conn(0x7fc3e4067b50 msgr2=0x7fc3e413b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:13.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 shutdown_connections 2023-12-19T08:02:13.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.726+0000 7fc3e9fcc700 1 -- 172.21.15.111:0/680579393 wait complete. 2023-12-19T08:02:13.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-12-19T08:02:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.918+0000 7f8bfec8e700 1 Processor -- start 2023-12-19T08:02:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.918+0000 7f8bfec8e700 1 -- start start 2023-12-19T08:02:13.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfec8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf80b1410 0x7f8bf80ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfec8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8bf806a580 0x7f8bf806a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfec8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf806c7c0 con 0x7f8bf815a860 2023-12-19T08:02:13.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfec8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf806cd80 con 0x7f8bf806a580 2023-12-19T08:02:13.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfec8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf806af10 con 0x7f8bf80b1410 2023-12-19T08:02:13.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8bf806a580 0x7f8bf806a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8bf806a580 0x7f8bf806a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38692/0 (socket says 172.21.15.111:38692) 2023-12-19T08:02:13.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 -- 172.21.15.111:0/685783299 learned_addr learned my addr 172.21.15.111:0/685783299 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:13.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfdc8c700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf80b1410 0x7f8bf80ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 -- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf80b1410 msgr2=0x7f8bf80ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:13.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf80b1410 0x7f8bf80ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 -- 172.21.15.111:0/685783299 >> v1:172.21.15.143:6789/0 conn(0x7f8bf815a860 legacy=0x7f8bf815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 -- 172.21.15.111:0/685783299 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf806b8f0 con 0x7f8bf806a580 2023-12-19T08:02:13.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.922+0000 7f8bfd48b700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf806a580 0x7f8bf806a9d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8be80099b0 tx=0x7f8be8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8689433ea118c11c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfcc8a700 1 -- 172.21.15.111:0/685783299 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be800b860 con 0x7f8bf806a580 2023-12-19T08:02:13.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfcc8a700 1 -- 172.21.15.111:0/685783299 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be800b9c0 con 0x7f8bf806a580 2023-12-19T08:02:13.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfcc8a700 1 -- 172.21.15.111:0/685783299 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be800bd60 con 0x7f8bf806a580 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 -- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf806a580 msgr2=0x7f8bf806a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf806a580 0x7f8bf806a9d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8be80099b0 tx=0x7f8be8009250 comp rx=0 tx=0).stop 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 -- 172.21.15.111:0/685783299 shutdown_connections 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf806a580 0x7f8bf806a9d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/685783299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf80b1410 0x7f8bf80ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 -- 172.21.15.111:0/685783299 >> 172.21.15.111:0/685783299 conn(0x7f8bf8067bb0 msgr2=0x7f8bf80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 -- 172.21.15.111:0/685783299 shutdown_connections 2023-12-19T08:02:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.926+0000 7f8bfec8e700 1 -- 172.21.15.111:0/685783299 wait complete. 2023-12-19T08:02:13.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 Processor -- start 2023-12-19T08:02:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 -- start start 2023-12-19T08:02:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf80b1410 0x7f8bf814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 0x7f8bf8165fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf8159080 con 0x7f8bf806a580 2023-12-19T08:02:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf814bbb0 con 0x7f8bf80b1410 2023-12-19T08:02:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfec8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf814d9b0 con 0x7f8bf815a860 2023-12-19T08:02:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfdc8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfdc8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36694/0 (socket says 172.21.15.111:36694) 2023-12-19T08:02:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.930+0000 7f8bfdc8c700 1 -- 172.21.15.111:0/135798012 learned_addr learned my addr 172.21.15.111:0/135798012 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfd48b700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf80b1410 0x7f8bf814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfe48d700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 0x7f8bf8165fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 -- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 msgr2=0x7f8bf8165fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 0x7f8bf8165fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 -- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf80b1410 msgr2=0x7f8bf814cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf80b1410 0x7f8bf814cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 -- 172.21.15.111:0/135798012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80094f0 con 0x7f8bf806a580 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfe48d700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 0x7f8bf8165fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:13.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfdc8c700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400e190 tx=0x7f8bf4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400ed80 con 0x7f8bf806a580 2023-12-19T08:02:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf4014420 con 0x7f8bf806a580 2023-12-19T08:02:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400eee0 con 0x7f8bf806a580 2023-12-19T08:02:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf814d150 con 0x7f8bf806a580 2023-12-19T08:02:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf814d6d0 con 0x7f8bf806a580 2023-12-19T08:02:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.934+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf804be40 con 0x7f8bf806a580 2023-12-19T08:02:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.938+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8bf4014a70 con 0x7f8bf806a580 2023-12-19T08:02:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.938+0000 7f8bedffb700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 0x7f8bdc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.938+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f8bf407d410 con 0x7f8bf806a580 2023-12-19T08:02:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.938+0000 7f8bfd48b700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 0x7f8bdc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.942+0000 7f8bfd48b700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 0x7f8bdc062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8be80099b0 tx=0x7f8be8017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:13.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:13.942+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8bf404d5b0 con 0x7f8bf806a580 2023-12-19T08:02:14.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.174+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f8bf814b4f0 con 0x7f8bf806a580 2023-12-19T08:02:14.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.518+0000 7f8bedffb700 1 -- 172.21.15.111:0/135798012 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v545) v1 ==== 154+0+0 (secure 0 0 0) 0x7f8bf4050bd0 con 0x7f8bf806a580 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 msgr2=0x7f8bdc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 0x7f8bdc062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8be80099b0 tx=0x7f8be8017000 comp rx=0 tx=0).stop 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 msgr2=0x7f8bf8144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400e190 tx=0x7f8bf4009920 comp rx=0 tx=0).stop 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 shutdown_connections 2023-12-19T08:02:14.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bdc060670 0x7f8bdc062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bf80b1410 0x7f8bf814cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bf815a860 0x7f8bf8165fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 --2- 172.21.15.111:0/135798012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bf806a580 0x7f8bf8144650 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 >> 172.21.15.111:0/135798012 conn(0x7f8bf8067bb0 msgr2=0x7f8bf80adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 shutdown_connections 2023-12-19T08:02:14.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.522+0000 7f8bfec8e700 1 -- 172.21.15.111:0/135798012 wait complete. 2023-12-19T08:02:14.551 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-12-19T08:02:14.552 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-12-19T08:02:14.552 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T08:02:14.552 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.722+0000 7f9cbf26e700 1 Processor -- start 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.722+0000 7f9cbf26e700 1 -- start start 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9cb80ae140 0x7f9cb80a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb80ae510 0x7f9cb80c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb806dc20 con 0x7f9cb806d180 2023-12-19T08:02:14.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb806d940 con 0x7f9cb80ae140 2023-12-19T08:02:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80c6e20 con 0x7f9cb80ae510 2023-12-19T08:02:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9cb80ae140 0x7f9cb80a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd80b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9cb806d180 0x7f9cb80680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50580/0 (socket says 172.21.15.111:50580) 2023-12-19T08:02:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd80b700 1 -- 172.21.15.111:0/1958569577 learned_addr learned my addr 172.21.15.111:0/1958569577 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:14.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbc809700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb80ae510 0x7f9cb80c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 -- 172.21.15.111:0/1958569577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb80ae510 msgr2=0x7f9cb80c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:14.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb80ae510 0x7f9cb80c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 -- 172.21.15.111:0/1958569577 >> v1:172.21.15.111:6789/0 conn(0x7f9cb806d180 legacy=0x7f9cb80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 -- 172.21.15.111:0/1958569577 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80c7100 con 0x7f9cb80ae140 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbd00a700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae140 0x7f9cb80a8850 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9cb806de10 tx=0x7f9cb4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6979ab2cbfc4f4cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9caffff700 1 -- 172.21.15.111:0/1958569577 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4016070 con 0x7f9cb80ae140 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9caffff700 1 -- 172.21.15.111:0/1958569577 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb400bb60 con 0x7f9cb80ae140 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9caffff700 1 -- 172.21.15.111:0/1958569577 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400e070 con 0x7f9cb80ae140 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- 172.21.15.111:0/1958569577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae140 msgr2=0x7f9cb80a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae140 0x7f9cb80a8850 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9cb806de10 tx=0x7f9cb4009780 comp rx=0 tx=0).stop 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- 172.21.15.111:0/1958569577 shutdown_connections 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb80ae510 0x7f9cb80c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/1958569577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae140 0x7f9cb80a8850 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.726+0000 7f9cbf26e700 1 -- 172.21.15.111:0/1958569577 >> 172.21.15.111:0/1958569577 conn(0x7f9cb80c8850 msgr2=0x7f9cb80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.730+0000 7f9cbf26e700 1 -- 172.21.15.111:0/1958569577 shutdown_connections 2023-12-19T08:02:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.730+0000 7f9cbf26e700 1 -- 172.21.15.111:0/1958569577 wait complete. 2023-12-19T08:02:14.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.730+0000 7f9cbf26e700 1 Processor -- start 2023-12-19T08:02:14.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.730+0000 7f9cbf26e700 1 -- start start 2023-12-19T08:02:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb806d180 0x7f9cb8154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 0x7f9cb815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8155310 con 0x7f9cb80ae510 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb815ea20 con 0x7f9cb80ae140 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb815ed00 con 0x7f9cb806d180 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36708/0 (socket says 172.21.15.111:36708) 2023-12-19T08:02:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 -- 172.21.15.111:0/4206526180 learned_addr learned my addr 172.21.15.111:0/4206526180 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbc809700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 0x7f9cb815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd00a700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb806d180 0x7f9cb8154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 -- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb806d180 msgr2=0x7f9cb8154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb806d180 0x7f9cb8154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 -- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 msgr2=0x7f9cb815e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 0x7f9cb815e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 -- 172.21.15.111:0/4206526180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4009a20 con 0x7f9cb80ae510 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbc809700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 0x7f9cb815e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbd80b700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c590 tx=0x7f9cb0007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0005b30 con 0x7f9cb80ae510 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0005cb0 con 0x7f9cb80ae510 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb815efe0 con 0x7f9cb80ae510 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0007320 con 0x7f9cb80ae510 2023-12-19T08:02:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.734+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb815e6f0 con 0x7f9cb80ae510 2023-12-19T08:02:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.738+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb80af1f0 con 0x7f9cb80ae510 2023-12-19T08:02:14.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.738+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9cb001b3b0 con 0x7f9cb80ae510 2023-12-19T08:02:14.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.742+0000 7f9cad7fa700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 0x7f9c9c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:14.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.742+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f9cb007c190 con 0x7f9cb80ae510 2023-12-19T08:02:14.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.742+0000 7f9cbd00a700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 0x7f9c9c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:14.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.742+0000 7f9cbd00a700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 0x7f9c9c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40097c0 tx=0x7f9cb4018000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:14.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.746+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9cb80af1f0 con 0x7f9cb80ae510 2023-12-19T08:02:14.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.914+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cb80af1f0 con 0x7f9cb80ae510 2023-12-19T08:02:14.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.914+0000 7f9cad7fa700 1 -- 172.21.15.111:0/4206526180 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+14182 (secure 0 0 0) 0x7f9cb80af1f0 con 0x7f9cb80ae510 2023-12-19T08:02:14.915 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:14.915 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":76,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:12.494511+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6835/2938693255":"2023-12-20T08:02:09.402696+0000","172.21.15.143:0/3260056484":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2460533822":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/1248091335":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2820673595":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/2810180880":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/134362188":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3567649431":"2023-12-20T08:02:08.957486+0000","172.21.15.143:0/3495942537":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/189999424":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/3297717311":"2023-12-20T08:00:23.832713+0000","172.21.15.143:0/934963750":"2023-12-20T08:00:23.832713+0000","172.21.15.111:6834/2938693255":"2023-12-20T08:02:09.402696+0000","172.21.15.143:0/1624932641":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6833/33796":"2023-12-20T08:00:23.832713+0000","172.21.15.143:6832/33796":"2023-12-20T08:00:23.832713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 msgr2=0x7f9c9c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 0x7f9c9c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40097c0 tx=0x7f9cb4018000 comp rx=0 tx=0).stop 2023-12-19T08:02:14.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 msgr2=0x7f9cb80cae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c590 tx=0x7f9cb0007ef0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 shutdown_connections 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c9c060670 0x7f9c9c062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb80ae140 0x7f9cb815e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cb80ae510 0x7f9cb80cae60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 --2- 172.21.15.111:0/4206526180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb806d180 0x7f9cb8154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:14.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 >> 172.21.15.111:0/4206526180 conn(0x7f9cb80c8850 msgr2=0x7f9cb80a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:14.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 shutdown_connections 2023-12-19T08:02:14.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:14.918+0000 7f9cbf26e700 1 -- 172.21.15.111:0/4206526180 wait complete. 2023-12-19T08:02:14.939 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:02:14.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:15.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 Processor -- start 2023-12-19T08:02:15.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 -- start start 2023-12-19T08:02:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0cb970 0x7f7c3c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3c0cbdf0 con 0x7f7c3c0ca2c0 2023-12-19T08:02:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3c0cc0d0 con 0x7f7c3c158e20 2023-12-19T08:02:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c4282e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0cc3b0 con 0x7f7c3c0cb970 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c3bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c3bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36724/0 (socket says 172.21.15.111:36724) 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c3bfff700 1 -- 172.21.15.111:0/1290729133 learned_addr learned my addr 172.21.15.111:0/1290729133 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c40dcb700 1 --1- 172.21.15.111:0/1290729133 >> v1:172.21.15.111:6789/0 conn(0x7f7c3c0ca2c0 0x7f7c3c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50590/0 (socket says 172.21.15.111:50590) 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c33fff700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0cb970 0x7f7c3c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c3bfff700 1 -- 172.21.15.111:0/1290729133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0cb970 msgr2=0x7f7c3c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.110+0000 7f7c3bfff700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0cb970 0x7f7c3c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3bfff700 1 -- 172.21.15.111:0/1290729133 >> v1:172.21.15.111:6789/0 conn(0x7f7c3c0ca2c0 legacy=0x7f7c3c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3bfff700 1 -- 172.21.15.111:0/1290729133 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c0cc690 con 0x7f7c3c158e20 2023-12-19T08:02:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3bfff700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0099b0 tx=0x7f7c2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ebfadd6f963090 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3b7fe700 1 -- 172.21.15.111:0/1290729133 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c00b860 con 0x7f7c3c158e20 2023-12-19T08:02:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3b7fe700 1 -- 172.21.15.111:0/1290729133 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c00b9c0 con 0x7f7c3c158e20 2023-12-19T08:02:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c3b7fe700 1 -- 172.21.15.111:0/1290729133 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c00bd60 con 0x7f7c3c158e20 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 -- 172.21.15.111:0/1290729133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c158e20 msgr2=0x7f7c3c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0099b0 tx=0x7f7c2c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 -- 172.21.15.111:0/1290729133 shutdown_connections 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0cb970 0x7f7c3c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1290729133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c158e20 0x7f7c3c0ce5c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 -- 172.21.15.111:0/1290729133 >> 172.21.15.111:0/1290729133 conn(0x7f7c3c067bb0 msgr2=0x7f7c3c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 -- 172.21.15.111:0/1290729133 shutdown_connections 2023-12-19T08:02:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.114+0000 7f7c4282e700 1 -- 172.21.15.111:0/1290729133 wait complete. 2023-12-19T08:02:15.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 Processor -- start 2023-12-19T08:02:15.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 -- start start 2023-12-19T08:02:15.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0ca2c0 0x7f7c3c14cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3c158e20 0x7f7c3c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c150360 con 0x7f7c3c14fec0 2023-12-19T08:02:15.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0ca070 con 0x7f7c3c158e20 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c4282e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0c94a0 con 0x7f7c3c0ca2c0 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36730/0 (socket says 172.21.15.111:36730) 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 -- 172.21.15.111:0/1827353275 learned_addr learned my addr 172.21.15.111:0/1827353275 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c33fff700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3c158e20 0x7f7c3c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c3bfff700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0ca2c0 0x7f7c3c14cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 -- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0ca2c0 msgr2=0x7f7c3c14cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0ca2c0 0x7f7c3c14cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 -- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3c158e20 msgr2=0x7f7c3c14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3c158e20 0x7f7c3c14f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.118+0000 7f7c40dcb700 1 -- 172.21.15.111:0/1827353275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0094f0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c40dcb700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400cd30 tx=0x7f7c34001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400ce90 con 0x7f7c3c14fec0 2023-12-19T08:02:15.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34005df0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c14d0f0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400aaa0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c0c9a20 con 0x7f7c3c14fec0 2023-12-19T08:02:15.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c071120 con 0x7f7c3c14fec0 2023-12-19T08:02:15.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.122+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7c340070f0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.126+0000 7f7c38ff9700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 0x7f7c1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.126+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f7c3407cbc0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.126+0000 7f7c3bfff700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 0x7f7c1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.126+0000 7f7c3bfff700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 0x7f7c1c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0099b0 tx=0x7f7c2c017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.130+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c3404ce80 con 0x7f7c3c14fec0 2023-12-19T08:02:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.302+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c3c064270 con 0x7f7c3c14fec0 2023-12-19T08:02:15.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.302+0000 7f7c38ff9700 1 -- 172.21.15.111:0/1827353275 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c340504a0 con 0x7f7c3c14fec0 2023-12-19T08:02:15.304 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:15.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 msgr2=0x7f7c1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 0x7f7c1c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0099b0 tx=0x7f7c2c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:15.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 msgr2=0x7f7c3c15a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400cd30 tx=0x7f7c34001b00 comp rx=0 tx=0).stop 2023-12-19T08:02:15.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 shutdown_connections 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c1c060670 0x7f7c1c062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3c158e20 0x7f7c3c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c3c14fec0 0x7f7c3c15a0a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 --2- 172.21.15.111:0/1827353275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c3c0ca2c0 0x7f7c3c14cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 >> 172.21.15.111:0/1827353275 conn(0x7f7c3c067bb0 msgr2=0x7f7c3c14eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 shutdown_connections 2023-12-19T08:02:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.310+0000 7f7c4282e700 1 -- 172.21.15.111:0/1827353275 wait complete. 2023-12-19T08:02:15.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-12-19T08:02:15.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.494+0000 7fcc15ce3700 1 Processor -- start 2023-12-19T08:02:15.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc15ce3700 1 -- start start 2023-12-19T08:02:15.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc15ce3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc1006aa00 0x7fcc1006d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc15ce3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1006c7c0 con 0x7fcc1006a580 2023-12-19T08:02:15.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc15ce3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1006cd80 con 0x7fcc100b1410 2023-12-19T08:02:15.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc15ce3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1006b8f0 con 0x7fcc1006aa00 2023-12-19T08:02:15.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc07fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcc1006a580 0x7fcc1006d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50598/0 (socket says 172.21.15.111:50598) 2023-12-19T08:02:15.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc07fff700 1 -- 172.21.15.111:0/199674736 learned_addr learned my addr 172.21.15.111:0/199674736 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:15.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc154e2700 1 --2- 172.21.15.111:0/199674736 >> v2:172.21.15.143:3300/0 conn(0x7fcc1006aa00 0x7fcc1006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc077fe700 1 -- 172.21.15.111:0/199674736 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013923528 0 0) 0x7fcc1006c7c0 con 0x7fcc1006a580 2023-12-19T08:02:15.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.498+0000 7fcc077fe700 1 -- 172.21.15.111:0/199674736 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf8003570 con 0x7fcc1006a580 2023-12-19T08:02:15.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc154e2700 1 -- 172.21.15.111:0/199674736 >> v1:172.21.15.143:6789/0 conn(0x7fcc100b1410 legacy=0x7fcc100ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc154e2700 1 -- 172.21.15.111:0/199674736 >> v1:172.21.15.111:6789/0 conn(0x7fcc1006a580 legacy=0x7fcc1006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc154e2700 1 -- 172.21.15.111:0/199674736 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100ac1e0 con 0x7fcc1006aa00 2023-12-19T08:02:15.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc154e2700 1 --2- 172.21.15.111:0/199674736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006aa00 0x7fcc1006d920 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcc080098e0 tx=0x7fcc080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e83dc16abe3669 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc077fe700 1 -- 172.21.15.111:0/199674736 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800b640 con 0x7fcc1006aa00 2023-12-19T08:02:15.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc077fe700 1 -- 172.21.15.111:0/199674736 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0800b7a0 con 0x7fcc1006aa00 2023-12-19T08:02:15.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc077fe700 1 -- 172.21.15.111:0/199674736 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800bb40 con 0x7fcc1006aa00 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 -- 172.21.15.111:0/199674736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006aa00 msgr2=0x7fcc1006d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/199674736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006aa00 0x7fcc1006d920 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcc080098e0 tx=0x7fcc080091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 -- 172.21.15.111:0/199674736 shutdown_connections 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/199674736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006aa00 0x7fcc1006d920 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 -- 172.21.15.111:0/199674736 >> 172.21.15.111:0/199674736 conn(0x7fcc10067bb0 msgr2=0x7fcc100cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 -- 172.21.15.111:0/199674736 shutdown_connections 2023-12-19T08:02:15.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.502+0000 7fcc15ce3700 1 -- 172.21.15.111:0/199674736 wait complete. 2023-12-19T08:02:15.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 Processor -- start 2023-12-19T08:02:15.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 -- start start 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006a580 0x7fcc10166e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc100b1410 0x7fcc100ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10159f40 con 0x7fcc1006aa00 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100c0d20 con 0x7fcc1006a580 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100c57a0 con 0x7fcc100b1410 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36734/0 (socket says 172.21.15.111:36734) 2023-12-19T08:02:15.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 -- 172.21.15.111:0/3020691333 learned_addr learned my addr 172.21.15.111:0/3020691333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc14ce1700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006a580 0x7fcc10166e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc154e2700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc100b1410 0x7fcc100ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 -- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc100b1410 msgr2=0x7fcc100ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc100b1410 0x7fcc100ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 -- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006a580 msgr2=0x7fcc10166e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006a580 0x7fcc10166e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 -- 172.21.15.111:0/3020691333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc08009420 con 0x7fcc1006aa00 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc07fff700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c7c0 tx=0x7fcc00007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00005b30 con 0x7fcc1006aa00 2023-12-19T08:02:15.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00005c90 con 0x7fcc1006aa00 2023-12-19T08:02:15.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00007340 con 0x7fcc1006aa00 2023-12-19T08:02:15.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.506+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc100c5a80 con 0x7fcc1006aa00 2023-12-19T08:02:15.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.510+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc100c09b0 con 0x7fcc1006aa00 2023-12-19T08:02:15.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.510+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc100c0eb0 con 0x7fcc1006aa00 2023-12-19T08:02:15.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.510+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcc0001b400 con 0x7fcc1006aa00 2023-12-19T08:02:15.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.510+0000 7fcc04ff9700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 0x7fcbfc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:15.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.510+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7fcc0007c710 con 0x7fcc1006aa00 2023-12-19T08:02:15.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.514+0000 7fcc14ce1700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 0x7fcbfc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:15.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.514+0000 7fcc14ce1700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 0x7fcbfc062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c007fd0 tx=0x7fcc0c009040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:15.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.518+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc0004c9d0 con 0x7fcc1006aa00 2023-12-19T08:02:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:15.746+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fcc1004be40 con 0x7fcc1006aa00 2023-12-19T08:02:16.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.538+0000 7fcc04ff9700 1 -- 172.21.15.111:0/3020691333 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v547) v1 ==== 159+0+0 (secure 0 0 0) 0x7fcc0004fff0 con 0x7fcc1006aa00 2023-12-19T08:02:16.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 msgr2=0x7fcbfc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 0x7fcbfc062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c007fd0 tx=0x7fcc0c009040 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 msgr2=0x7fcc10169b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c7c0 tx=0x7fcc00007e00 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 shutdown_connections 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbfc060670 0x7fcbfc062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc100b1410 0x7fcc100ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc1006aa00 0x7fcc10169b80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 --2- 172.21.15.111:0/3020691333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc1006a580 0x7fcc10166e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 >> 172.21.15.111:0/3020691333 conn(0x7fcc10067bb0 msgr2=0x7fcc101563a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:16.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 shutdown_connections 2023-12-19T08:02:16.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.542+0000 7fcc15ce3700 1 -- 172.21.15.111:0/3020691333 wait complete. 2023-12-19T08:02:16.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:02:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.742+0000 7f429d0cd700 1 Processor -- start 2023-12-19T08:02:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.742+0000 7f429d0cd700 1 -- start start 2023-12-19T08:02:16.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429d0cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f42981554b0 0x7f42980ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429d0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f42981597d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429d0cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4298159340 con 0x7f4298159d10 2023-12-19T08:02:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429d0cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429806d790 con 0x7f42981554b0 2023-12-19T08:02:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429d0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429806cef0 con 0x7f429806c7c0 2023-12-19T08:02:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429759e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4298159d10 0x7f429806a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50608/0 (socket says 172.21.15.111:50608) 2023-12-19T08:02:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429759e700 1 -- 172.21.15.111:0/160202569 learned_addr learned my addr 172.21.15.111:0/160202569 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4296d9d700 1 --2- 172.21.15.111:0/160202569 >> v2:172.21.15.143:3300/0 conn(0x7f42981554b0 0x7f42980ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429659c700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f42981597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290687967 0 0) 0x7f4298159340 con 0x7f4298159d10 2023-12-19T08:02:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4280003570 con 0x7f4298159d10 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403881283 0 0) 0x7f4280003570 con 0x7f4298159d10 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 msgr2=0x7f42981597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f42981597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 >> v2:172.21.15.143:3300/0 conn(0x7f42981554b0 msgr2=0x7f42980ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 --2- 172.21.15.111:0/160202569 >> v2:172.21.15.143:3300/0 conn(0x7f42981554b0 0x7f42980ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429806d1d0 con 0x7f4298159d10 2023-12-19T08:02:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f42900017f0 con 0x7f4298159d10 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4296d9d700 1 --2- 172.21.15.111:0/160202569 >> v2:172.21.15.143:3300/0 conn(0x7f42981554b0 0x7f42980ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 >> v1:172.21.15.111:6789/0 conn(0x7f4298159d10 legacy=0x7f429806a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f4280003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4280003fe0 0x7f4280004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429806d790 con 0x7f4280004d50 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429806cef0 con 0x7f4280003fe0 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428000ba90 con 0x7f429806c7c0 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429759e700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f4296d9d700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f4280003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.746+0000 7f429659c700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4280003fe0 0x7f4280004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 -- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 msgr2=0x7f4280003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:16.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f429806c7c0 0x7f4280003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 -- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4280003fe0 msgr2=0x7f4280004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:16.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4280003fe0 0x7f4280004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 -- 172.21.15.111:0/160202569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280003570 con 0x7f4280004d50 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42800051c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f429000a970 tx=0x7f4290002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67eb13d75546c40 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42900070d0 con 0x7f4280004d50 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4290007270 con 0x7f4280004d50 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f4295d9b700 1 -- 172.21.15.111:0/160202569 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290007630 con 0x7f4280004d50 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 -- 172.21.15.111:0/160202569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 msgr2=0x7f42800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 --2- 172.21.15.111:0/160202569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42800051c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f429000a970 tx=0x7f4290002210 comp rx=0 tx=0).stop 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429759e700 1 -- 172.21.15.111:0/160202569 reap_dead start 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 -- 172.21.15.111:0/160202569 shutdown_connections 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 -- 172.21.15.111:0/160202569 >> 172.21.15.111:0/160202569 conn(0x7f4298067bb0 msgr2=0x7f42980cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 -- 172.21.15.111:0/160202569 shutdown_connections 2023-12-19T08:02:16.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.750+0000 7f429d0cd700 1 -- 172.21.15.111:0/160202569 wait complete. 2023-12-19T08:02:16.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 Processor -- start 2023-12-19T08:02:16.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 -- start start 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429814f990 0x7f42980ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 0x7f429814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429814e770 con 0x7f4280004d50 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429814fe90 con 0x7f429814f990 2023-12-19T08:02:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f429d0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4298150170 con 0x7f42980c2d80 2023-12-19T08:02:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f4296d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f4296d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36754/0 (socket says 172.21.15.111:36754) 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.754+0000 7f4296d9d700 1 -- 172.21.15.111:0/21714767 learned_addr learned my addr 172.21.15.111:0/21714767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429759e700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 0x7f429814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429659c700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429814f990 0x7f42980ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 -- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 msgr2=0x7f429814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 0x7f429814ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 -- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429814f990 msgr2=0x7f42980ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429814f990 0x7f42980ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 -- 172.21.15.111:0/21714767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298150450 con 0x7f4280004d50 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429759e700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 0x7f429814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f4296d9d700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4288012c00 tx=0x7f4288008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288010a20 con 0x7f4280004d50 2023-12-19T08:02:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4288017070 con 0x7f4280004d50 2023-12-19T08:02:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288010df0 con 0x7f4280004d50 2023-12-19T08:02:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4298150730 con 0x7f4280004d50 2023-12-19T08:02:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42980c31f0 con 0x7f4280004d50 2023-12-19T08:02:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.758+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298071120 con 0x7f4280004d50 2023-12-19T08:02:16.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.762+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4288019030 con 0x7f4280004d50 2023-12-19T08:02:16.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.762+0000 7f427effd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 0x7f4284062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:16.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.762+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f428807d560 con 0x7f4280004d50 2023-12-19T08:02:16.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.762+0000 7f429659c700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 0x7f4284062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:16.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.762+0000 7f429659c700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 0x7f4284062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42981509f0 tx=0x7f428c00b430 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.766+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f428804d700 con 0x7f4280004d50 2023-12-19T08:02:16.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.946+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4298152130 con 0x7f4280004d50 2023-12-19T08:02:16.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.954+0000 7f427effd700 1 -- 172.21.15.111:0/21714767 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4288050d20 con 0x7f4280004d50 2023-12-19T08:02:16.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 msgr2=0x7f4284062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 0x7f4284062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42981509f0 tx=0x7f428c00b430 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 msgr2=0x7f42980c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4288012c00 tx=0x7f4288008040 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 shutdown_connections 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f42840606a0 0x7f4284062b50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429814f990 0x7f42980ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42980c2d80 0x7f429814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 --2- 172.21.15.111:0/21714767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4280004d50 0x7f42980c5160 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 >> 172.21.15.111:0/21714767 conn(0x7f4298067bb0 msgr2=0x7f42980c8740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 shutdown_connections 2023-12-19T08:02:16.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:16.958+0000 7f429d0cd700 1 -- 172.21.15.111:0/21714767 wait complete. 2023-12-19T08:02:16.979 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:02:16.980 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:17.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:17.065 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:02:17.065 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:02:17.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 Processor -- start 2023-12-19T08:02:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 -- start start 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f34fc156940 0x7f34fc15a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 0x7f34fc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc06cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc06d160 con 0x7f34fc156940 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc0cfec0 con 0x7f34fc156d10 2023-12-19T08:02:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f350126e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0d01a0 con 0x7f34fc06c7c0 2023-12-19T08:02:17.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.226+0000 7f34faffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f34fc156940 0x7f34fc15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f34fc156940 0x7f34fc15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36762/0 (socket says 172.21.15.111:36762) 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 -- 172.21.15.111:0/4025563818 learned_addr learned my addr 172.21.15.111:0/4025563818 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34f27fc700 1 --2- 172.21.15.111:0/4025563818 >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 0x7f34fc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34fbfff700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc06cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 -- 172.21.15.111:0/4025563818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 msgr2=0x7f34fc06cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc06cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 -- 172.21.15.111:0/4025563818 >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 msgr2=0x7f34fc0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 --2- 172.21.15.111:0/4025563818 >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 0x7f34fc0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 -- 172.21.15.111:0/4025563818 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0d0480 con 0x7f34fc156940 2023-12-19T08:02:17.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34f27fc700 1 --2- 172.21.15.111:0/4025563818 >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 0x7f34fc0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:17.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34faffd700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156940 0x7f34fc15a180 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f34fc063700 tx=0x7f34ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144dceb4af44b5cd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34fa7fc700 1 -- 172.21.15.111:0/4025563818 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00e070 con 0x7f34fc156940 2023-12-19T08:02:17.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34fa7fc700 1 -- 172.21.15.111:0/4025563818 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec00ba50 con 0x7f34fc156940 2023-12-19T08:02:17.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f34fa7fc700 1 -- 172.21.15.111:0/4025563818 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec0092e0 con 0x7f34fc156940 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 -- 172.21.15.111:0/4025563818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156940 msgr2=0x7f34fc15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156940 0x7f34fc15a180 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f34fc063700 tx=0x7f34ec009840 comp rx=0 tx=0).stop 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 -- 172.21.15.111:0/4025563818 shutdown_connections 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc06cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 --2- 172.21.15.111:0/4025563818 >> v2:172.21.15.143:3300/0 conn(0x7f34fc156d10 0x7f34fc0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 --2- 172.21.15.111:0/4025563818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156940 0x7f34fc15a180 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 -- 172.21.15.111:0/4025563818 >> 172.21.15.111:0/4025563818 conn(0x7f34fc0ce320 msgr2=0x7f34fc0ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 -- 172.21.15.111:0/4025563818 shutdown_connections 2023-12-19T08:02:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.230+0000 7f350126e700 1 -- 172.21.15.111:0/4025563818 wait complete. 2023-12-19T08:02:17.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.234+0000 7f350126e700 1 Processor -- start 2023-12-19T08:02:17.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.234+0000 7f350126e700 1 -- start start 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34fc156940 0x7f34fc0b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc15def0 con 0x7f34fc156d10 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc15e1d0 con 0x7f34fc156940 2023-12-19T08:02:17.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc15e4b0 con 0x7f34fc06c7c0 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36776/0 (socket says 172.21.15.111:36776) 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 -- 172.21.15.111:0/3638131106 learned_addr learned my addr 172.21.15.111:0/3638131106 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34f27fc700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34fc156940 0x7f34fc0b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34faffd700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 -- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 msgr2=0x7f34fc159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 -- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34fc156940 msgr2=0x7f34fc0b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34fc156940 0x7f34fc0b59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 -- 172.21.15.111:0/3638131106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec009ae0 con 0x7f34fc156d10 2023-12-19T08:02:17.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34fbfff700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f34f400d890 tx=0x7f34f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40125f0 con 0x7f34fc156d10 2023-12-19T08:02:17.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f4012770 con 0x7f34fc156d10 2023-12-19T08:02:17.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc15e790 con 0x7f34fc156d10 2023-12-19T08:02:17.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4012b30 con 0x7f34fc156d10 2023-12-19T08:02:17.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.238+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc159b60 con 0x7f34fc156d10 2023-12-19T08:02:17.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.242+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34fc15ec70 con 0x7f34fc156d10 2023-12-19T08:02:17.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.242+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f34f401e460 con 0x7f34fc156d10 2023-12-19T08:02:17.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.242+0000 7f34f37fe700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 0x7f34dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.242+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f34f407d150 con 0x7f34fc156d10 2023-12-19T08:02:17.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.242+0000 7f34faffd700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 0x7f34dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.246+0000 7f34faffd700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 0x7f34dc062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009880 tx=0x7f34ec018000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.250+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f34f4050e20 con 0x7f34fc156d10 2023-12-19T08:02:17.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.462+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34fc071120 con 0x7f34fc156d10 2023-12-19T08:02:17.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.462+0000 7f34f37fe700 1 -- 172.21.15.111:0/3638131106 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f34f40509b0 con 0x7f34fc156d10 2023-12-19T08:02:17.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:17.465 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5474,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1816768797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1816768797},{"type":"v1","addr":"172.21.15.143:6837","nonce":1816768797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/3880298243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3880298243},{"type":"v1","addr":"172.21.15.143:6836","nonce":3880298243}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1353885123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1353885123},{"type":"v1","addr":"172.21.15.111:6833","nonce":1353885123}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6336,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3418504071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3418504071},{"type":"v1","addr":"172.21.15.111:6835","nonce":3418504071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 msgr2=0x7f34dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 0x7f34dc062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009880 tx=0x7f34ec018000 comp rx=0 tx=0).stop 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 msgr2=0x7f34fc14eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f34f400d890 tx=0x7f34f4008040 comp rx=0 tx=0).stop 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 shutdown_connections 2023-12-19T08:02:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f34dc060670 0x7f34dc062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34fc156940 0x7f34fc0b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34fc156d10 0x7f34fc14eb80 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 --2- 172.21.15.111:0/3638131106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34fc06c7c0 0x7f34fc159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 >> 172.21.15.111:0/3638131106 conn(0x7f34fc0ce320 msgr2=0x7f34fc0c7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 shutdown_connections 2023-12-19T08:02:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.466+0000 7f350126e700 1 -- 172.21.15.111:0/3638131106 wait complete. 2023-12-19T08:02:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-12-19T08:02:17.489 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:02:17.489 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:02:17.490 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:02:17.490 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:17.490 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:02:17.490 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:02:17.491 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:02:17.491 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:17.491 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:02:17.491 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:02:17.492 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:02:17.492 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:17.492 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:02:17.492 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:02:17.493 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:02:17.493 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:17.508 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:02:17.508 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:02:17.512 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:02:17.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:02:17.520 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:02:17.520 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:02:17.521 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:02:17.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:02:17.567 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:02:17.567 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:17.572 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:02:17.572 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:17.573 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:02:17.573 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:17.577 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:02:17.577 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:17.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:02:17.673 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:02:17.674 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:02:17.689 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:02:17.690 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:02:17.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.742+0000 7f0c58a03700 1 Processor -- start 2023-12-19T08:02:17.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.742+0000 7f0c58a03700 1 -- start start 2023-12-19T08:02:17.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c58a03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c541591f0 0x7f0c5406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c58a03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c58a03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5406b730 con 0x7f0c54158e20 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c58a03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5406ab20 con 0x7f0c541591f0 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c58a03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5406ae00 con 0x7f0c540ce810 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c52ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c541591f0 0x7f0c5406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c52ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c541591f0 0x7f0c5406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38788/0 (socket says 172.21.15.111:38788) 2023-12-19T08:02:17.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.746+0000 7f0c52ffd700 1 -- 172.21.15.111:0/3476294037 learned_addr learned my addr 172.21.15.111:0/3476294037 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:17.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c52ffd700 1 -- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 msgr2=0x7f0c540ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:17.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c52ffd700 1 --2- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c52ffd700 1 -- 172.21.15.111:0/3476294037 >> v1:172.21.15.143:6789/0 conn(0x7f0c54158e20 legacy=0x7f0c540ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c52ffd700 1 -- 172.21.15.111:0/3476294037 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5406b0e0 con 0x7f0c541591f0 2023-12-19T08:02:17.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c52ffd700 1 --2- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c541591f0 0x7f0c5406a540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0c480099b0 tx=0x7f0c48009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c2f6f736bc0bda3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c527fc700 1 -- 172.21.15.111:0/3476294037 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48009ea0 con 0x7f0c541591f0 2023-12-19T08:02:17.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c527fc700 1 -- 172.21.15.111:0/3476294037 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4800b860 con 0x7f0c541591f0 2023-12-19T08:02:17.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c527fc700 1 -- 172.21.15.111:0/3476294037 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800bc00 con 0x7f0c541591f0 2023-12-19T08:02:17.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 -- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c541591f0 msgr2=0x7f0c5406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 --2- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c541591f0 0x7f0c5406a540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0c480099b0 tx=0x7f0c48009250 comp rx=0 tx=0).stop 2023-12-19T08:02:17.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 -- 172.21.15.111:0/3476294037 shutdown_connections 2023-12-19T08:02:17.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 --2- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 --2- 172.21.15.111:0/3476294037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c541591f0 0x7f0c5406a540 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 -- 172.21.15.111:0/3476294037 >> 172.21.15.111:0/3476294037 conn(0x7f0c54067bb0 msgr2=0x7f0c540cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 -- 172.21.15.111:0/3476294037 shutdown_connections 2023-12-19T08:02:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.750+0000 7f0c58a03700 1 -- 172.21.15.111:0/3476294037 wait complete. 2023-12-19T08:02:17.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 Processor -- start 2023-12-19T08:02:17.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 -- start start 2023-12-19T08:02:17.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c54158e20 0x7f0c5416a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c9820 con 0x7f0c541591f0 2023-12-19T08:02:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5414d5e0 con 0x7f0c54158e20 2023-12-19T08:02:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c58a03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5416a740 con 0x7f0c540ce810 2023-12-19T08:02:17.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36816/0 (socket says 172.21.15.111:36816) 2023-12-19T08:02:17.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c53fff700 1 -- 172.21.15.111:0/630465237 learned_addr learned my addr 172.21.15.111:0/630465237 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c52ffd700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c54158e20 0x7f0c5416a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.754+0000 7f0c537fe700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 -- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 msgr2=0x7f0c540c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 -- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c54158e20 msgr2=0x7f0c5416a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c54158e20 0x7f0c5416a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 -- 172.21.15.111:0/630465237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c480094f0 con 0x7f0c541591f0 2023-12-19T08:02:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c53fff700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009ec0 tx=0x7f0c4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c00cb70 con 0x7f0c541591f0 2023-12-19T08:02:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c00f040 con 0x7f0c541591f0 2023-12-19T08:02:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5414c9d0 con 0x7f0c541591f0 2023-12-19T08:02:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c014480 con 0x7f0c541591f0 2023-12-19T08:02:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5414cf90 con 0x7f0c541591f0 2023-12-19T08:02:17.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.758+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5414d770 con 0x7f0c541591f0 2023-12-19T08:02:17.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.762+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0c4c014830 con 0x7f0c541591f0 2023-12-19T08:02:17.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.762+0000 7f0c3b7fe700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 0x7f0c40062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:17.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.762+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6381+0+0 (secure 0 0 0) 0x7f0c4c07d670 con 0x7f0c541591f0 2023-12-19T08:02:17.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.766+0000 7f0c537fe700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 0x7f0c40062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:17.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.766+0000 7f0c537fe700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 0x7f0c40062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0c540b2a80 tx=0x7f0c44009040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:17.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.770+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0c4c04d700 con 0x7f0c541591f0 2023-12-19T08:02:17.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:17.950+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0c5404be40 con 0x7f0c541591f0 2023-12-19T08:02:18.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.990+0000 7f0c3b7fe700 1 -- 172.21.15.111:0/630465237 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v81) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0c4c050d20 con 0x7f0c541591f0 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 msgr2=0x7f0c40062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 0x7f0c40062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0c540b2a80 tx=0x7f0c44009040 comp rx=0 tx=0).stop 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 msgr2=0x7f0c5415bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009ec0 tx=0x7f0c4c00b040 comp rx=0 tx=0).stop 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 shutdown_connections 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c40060670 0x7f0c40062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c54158e20 0x7f0c5416a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c541591f0 0x7f0c5415bbc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:18.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 --2- 172.21.15.111:0/630465237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c540ce810 0x7f0c540c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:18.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 >> 172.21.15.111:0/630465237 conn(0x7f0c54067bb0 msgr2=0x7f0c540cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:18.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 shutdown_connections 2023-12-19T08:02:18.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:18.994+0000 7f0c58a03700 1 -- 172.21.15.111:0/630465237 wait complete. 2023-12-19T08:02:18.998 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:02:19.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:02:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 Processor -- start 2023-12-19T08:02:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 -- start start 2023-12-19T08:02:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 0x7f6d3406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3406b730 con 0x7f6d34158e20 2023-12-19T08:02:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3406ab20 con 0x7f6d340ce810 2023-12-19T08:02:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d38ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3406ae00 con 0x7f6d341591f0 2023-12-19T08:02:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d337fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6d34158e20 0x7f6d340ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50620/0 (socket says 172.21.15.111:50620) 2023-12-19T08:02:19.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d337fe700 1 -- 172.21.15.111:0/3205022439 learned_addr learned my addr 172.21.15.111:0/3205022439 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:19.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.170+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3205022439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 0x7f6d3406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412028080 0 0) 0x7f6d3406ab20 con 0x7f6d340ce810 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d1c003570 con 0x7f6d340ce810 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667777099 0 0) 0x7f6d3406b730 con 0x7f6d34158e20 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d3406ab20 con 0x7f6d34158e20 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d32ffd700 1 -- 172.21.15.111:0/3205022439 >> v1:172.21.15.143:6789/0 conn(0x7f6d340ce810 legacy=0x7f6d340ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d32ffd700 1 -- 172.21.15.111:0/3205022439 >> v1:172.21.15.111:6789/0 conn(0x7f6d34158e20 legacy=0x7f6d340ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d32ffd700 1 -- 172.21.15.111:0/3205022439 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3406b0e0 con 0x7f6d341591f0 2023-12-19T08:02:19.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3205022439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 0x7f6d3406a540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d28009910 tx=0x7f6d280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3782ec120adb031 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28011630 con 0x7f6d341591f0 2023-12-19T08:02:19.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d28011790 con 0x7f6d341591f0 2023-12-19T08:02:19.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d327fc700 1 -- 172.21.15.111:0/3205022439 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28011b30 con 0x7f6d341591f0 2023-12-19T08:02:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3205022439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 msgr2=0x7f6d3406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3205022439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 0x7f6d3406a540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d28009910 tx=0x7f6d280091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3205022439 shutdown_connections 2023-12-19T08:02:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3205022439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d341591f0 0x7f6d3406a540 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3205022439 >> 172.21.15.111:0/3205022439 conn(0x7f6d34067bb0 msgr2=0x7f6d340cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.174+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3205022439 shutdown_connections 2023-12-19T08:02:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3205022439 wait complete. 2023-12-19T08:02:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 Processor -- start 2023-12-19T08:02:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 -- start start 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 0x7f6d34149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 0x7f6d3414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34149ec0 con 0x7f6d340ce810 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3414a590 con 0x7f6d341591f0 2023-12-19T08:02:19.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d38ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d340c6fa0 con 0x7f6d34158e20 2023-12-19T08:02:19.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.178+0000 7f6d337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36824/0 (socket says 172.21.15.111:36824) 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 -- 172.21.15.111:0/3426615885 learned_addr learned my addr 172.21.15.111:0/3426615885 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 0x7f6d34149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d33fff700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 0x7f6d3414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 -- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 msgr2=0x7f6d34149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 0x7f6d34149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 -- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 msgr2=0x7f6d3414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 0x7f6d3414a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 -- 172.21.15.111:0/3426615885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28009420 con 0x7f6d340ce810 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 0x7f6d34149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d33fff700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 0x7f6d3414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:19.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d337fe700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400c7c0 tx=0x7f6d24007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24005b30 con 0x7f6d340ce810 2023-12-19T08:02:19.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24005c90 con 0x7f6d340ce810 2023-12-19T08:02:19.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3414a7c0 con 0x7f6d340ce810 2023-12-19T08:02:19.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24007320 con 0x7f6d340ce810 2023-12-19T08:02:19.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3414ad80 con 0x7f6d340ce810 2023-12-19T08:02:19.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.182+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3404be40 con 0x7f6d340ce810 2023-12-19T08:02:19.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.186+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6d2401b400 con 0x7f6d340ce810 2023-12-19T08:02:19.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.186+0000 7f6d1b7fe700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 0x7f6d20062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.186+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f6d2407c440 con 0x7f6d340ce810 2023-12-19T08:02:19.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.186+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 0x7f6d20062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.190+0000 7f6d32ffd700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 0x7f6d20062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d341595c0 tx=0x7f6d28017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.194+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6d2404c910 con 0x7f6d340ce810 2023-12-19T08:02:19.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.422+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6d3406b3c0 con 0x7f6d340ce810 2023-12-19T08:02:19.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.426+0000 7f6d1b7fe700 1 -- 172.21.15.111:0/3426615885 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6d2404ff30 con 0x7f6d340ce810 2023-12-19T08:02:19.425 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:19.425 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:02:19.425 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:02:19.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:19.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.427 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.430 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:02:19.431 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.432 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.433 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.433 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:02:19.434 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:02:19.435 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:19.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:02:19.437 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.438 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:19.438 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:19.438 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:19.438 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:19.444 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:02:19.445 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 msgr2=0x7f6d20062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 0x7f6d20062b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d341595c0 tx=0x7f6d28017000 comp rx=0 tx=0).stop 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 msgr2=0x7f6d34150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400c7c0 tx=0x7f6d24007e00 comp rx=0 tx=0).stop 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 shutdown_connections 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6d200606a0 0x7f6d20062b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341591f0 0x7f6d3414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d34158e20 0x7f6d34149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 --2- 172.21.15.111:0/3426615885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d340ce810 0x7f6d34150530 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 >> 172.21.15.111:0/3426615885 conn(0x7f6d34067bb0 msgr2=0x7f6d34153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:19.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 shutdown_connections 2023-12-19T08:02:19.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.430+0000 7f6d38ce3700 1 -- 172.21.15.111:0/3426615885 wait complete. 2023-12-19T08:02:19.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:19.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 Processor -- start 2023-12-19T08:02:19.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 -- start start 2023-12-19T08:02:19.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7febf8158f70 0x7febf8067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febf8069100 0x7febf80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 0x7febf80cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf8068c70 con 0x7febf8069100 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf8068060 con 0x7febf8158f70 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.618+0000 7febfdcf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf8068340 con 0x7febf8159340 2023-12-19T08:02:19.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febf8069100 0x7febf80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febf8069100 0x7febf80cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36834/0 (socket says 172.21.15.111:36834) 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 -- 172.21.15.111:0/3476747324 learned_addr learned my addr 172.21.15.111:0/3476747324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf77fe700 1 --2- 172.21.15.111:0/3476747324 >> v2:172.21.15.143:3300/0 conn(0x7febf8158f70 0x7febf8067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf7fff700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 0x7febf80cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 -- 172.21.15.111:0/3476747324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 msgr2=0x7febf80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:19.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 0x7febf80cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 -- 172.21.15.111:0/3476747324 >> v2:172.21.15.143:3300/0 conn(0x7febf8158f70 msgr2=0x7febf8067980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3476747324 >> v2:172.21.15.143:3300/0 conn(0x7febf8158f70 0x7febf8067980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 -- 172.21.15.111:0/3476747324 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8068620 con 0x7febf8069100 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf7fff700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 0x7febf80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80cf7e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7febf80cdec0 tx=0x7febec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf409ea6ba1b0d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf67fc700 1 -- 172.21.15.111:0/3476747324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec009e70 con 0x7febf8069100 2023-12-19T08:02:19.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf67fc700 1 -- 172.21.15.111:0/3476747324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec00b860 con 0x7febf8069100 2023-12-19T08:02:19.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febf67fc700 1 -- 172.21.15.111:0/3476747324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00bc00 con 0x7febf8069100 2023-12-19T08:02:19.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febfdcf0700 1 -- 172.21.15.111:0/3476747324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 msgr2=0x7febf80cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.622+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80cf7e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7febf80cdec0 tx=0x7febec009250 comp rx=0 tx=0).stop 2023-12-19T08:02:19.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 -- 172.21.15.111:0/3476747324 shutdown_connections 2023-12-19T08:02:19.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8159340 0x7febf80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3476747324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80cf7e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3476747324 >> v2:172.21.15.143:3300/0 conn(0x7febf8158f70 0x7febf8067980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 -- 172.21.15.111:0/3476747324 >> 172.21.15.111:0/3476747324 conn(0x7febf80ccc40 msgr2=0x7febf80cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:19.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 -- 172.21.15.111:0/3476747324 shutdown_connections 2023-12-19T08:02:19.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 -- 172.21.15.111:0/3476747324 wait complete. 2023-12-19T08:02:19.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.626+0000 7febfdcf0700 1 Processor -- start 2023-12-19T08:02:19.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 -- start start 2023-12-19T08:02:19.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8158f70 0x7febf80afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 0x7febf814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b3590 con 0x7febf8069100 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b3870 con 0x7febf8159340 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b03b0 con 0x7febf8158f70 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36846/0 (socket says 172.21.15.111:36846) 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 -- 172.21.15.111:0/3564736215 learned_addr learned my addr 172.21.15.111:0/3564736215 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8158f70 0x7febf80afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf7fff700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 0x7febf814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 -- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8158f70 msgr2=0x7febf80afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8158f70 0x7febf80afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 -- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 msgr2=0x7febf814c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 0x7febf814c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 -- 172.21.15.111:0/3564736215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0094f0 con 0x7febf8069100 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf7fff700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 0x7febf814c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:19.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febf77fe700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7febe800ecb0 tx=0x7febe80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe800c9d0 con 0x7febf8069100 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe800cb30 con 0x7febf8069100 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf80b0690 con 0x7febf8069100 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.630+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe800ced0 con 0x7febf8069100 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.634+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf80b0010 con 0x7febf8069100 2023-12-19T08:02:19.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.634+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8071120 con 0x7febf8069100 2023-12-19T08:02:19.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.634+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7febe8013680 con 0x7febf8069100 2023-12-19T08:02:19.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.638+0000 7febdf7fe700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 0x7febe4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:19.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.638+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5693+0+0 (secure 0 0 0) 0x7febe807cc20 con 0x7febf8069100 2023-12-19T08:02:19.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.638+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 0x7febe4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:19.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.638+0000 7febf6ffd700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 0x7febe4062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febf80cdec0 tx=0x7febec017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:19.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.646+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7febe804d110 con 0x7febf8069100 2023-12-19T08:02:19.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.814+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf8068900 con 0x7febf8069100 2023-12-19T08:02:19.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.814+0000 7febdf7fe700 1 -- 172.21.15.111:0/3564736215 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+13176 (secure 0 0 0) 0x7febe8050730 con 0x7febf8069100 2023-12-19T08:02:19.817 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:19.817 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":81,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:18.982943+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 msgr2=0x7febe4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 0x7febe4062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febf80cdec0 tx=0x7febec017000 comp rx=0 tx=0).stop 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 msgr2=0x7febf80b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7febe800ecb0 tx=0x7febe80091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 shutdown_connections 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7febe40606a0 0x7febe4062b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febf8159340 0x7febf814c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febf8158f70 0x7febf80afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 --2- 172.21.15.111:0/3564736215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf8069100 0x7febf80b11b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 >> 172.21.15.111:0/3564736215 conn(0x7febf80ccc40 msgr2=0x7febf80cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 shutdown_connections 2023-12-19T08:02:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:19.818+0000 7febfdcf0700 1 -- 172.21.15.111:0/3564736215 wait complete. 2023-12-19T08:02:19.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:02:19.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:20.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 Processor -- start 2023-12-19T08:02:20.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 -- start start 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f75180ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75180cb970 0x7f75180ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f75180ca2c0 0x7f751806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75180cbdf0 con 0x7f75180cb970 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75180cc0d0 con 0x7f75180ca2c0 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751d22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180cc3b0 con 0x7f7518158e20 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75180cb970 0x7f75180ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75180cb970 0x7f75180ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36856/0 (socket says 172.21.15.111:36856) 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 -- 172.21.15.111:0/3921112531 learned_addr learned my addr 172.21.15.111:0/3921112531 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f751759e700 1 --2- 172.21.15.111:0/3921112531 >> v2:172.21.15.143:3300/0 conn(0x7f75180ca2c0 0x7f751806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f7516d9d700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f75180ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 -- 172.21.15.111:0/3921112531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 msgr2=0x7f75180ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:20.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f75180ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 -- 172.21.15.111:0/3921112531 >> v2:172.21.15.143:3300/0 conn(0x7f75180ca2c0 msgr2=0x7f751806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 --2- 172.21.15.111:0/3921112531 >> v2:172.21.15.143:3300/0 conn(0x7f75180ca2c0 0x7f751806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f750ffff700 1 -- 172.21.15.111:0/3921112531 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180cc690 con 0x7f75180cb970 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.010+0000 7f7516d9d700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f75180ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f750ffff700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75180cb970 0x7f75180ceb00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f75180cfd10 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8679bf992d33e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751659c700 1 -- 172.21.15.111:0/3921112531 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500009e70 con 0x7f75180cb970 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751659c700 1 -- 172.21.15.111:0/3921112531 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000b860 con 0x7f75180cb970 2023-12-19T08:02:20.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751659c700 1 -- 172.21.15.111:0/3921112531 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000bc00 con 0x7f75180cb970 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 -- 172.21.15.111:0/3921112531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75180cb970 msgr2=0x7f75180ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75180cb970 0x7f75180ceb00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f75180cfd10 tx=0x7f7500009250 comp rx=0 tx=0).stop 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 -- 172.21.15.111:0/3921112531 shutdown_connections 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 --2- 172.21.15.111:0/3921112531 >> v2:172.21.15.143:3300/0 conn(0x7f75180ca2c0 0x7f751806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75180cb970 0x7f75180ceb00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 --2- 172.21.15.111:0/3921112531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f75180ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 -- 172.21.15.111:0/3921112531 >> 172.21.15.111:0/3921112531 conn(0x7f7518067bb0 msgr2=0x7f75180d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:20.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 -- 172.21.15.111:0/3921112531 shutdown_connections 2023-12-19T08:02:20.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 -- 172.21.15.111:0/3921112531 wait complete. 2023-12-19T08:02:20.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.014+0000 7f751d22e700 1 Processor -- start 2023-12-19T08:02:20.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- start start 2023-12-19T08:02:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 0x7f751814a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f7518169fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751816a500 con 0x7f751816cb40 2023-12-19T08:02:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751815d4a0 con 0x7f75180ca2c0 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751814b160 con 0x7f7518158e20 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36864/0 (socket says 172.21.15.111:36864) 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f7516d9d700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 0x7f751814a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 -- 172.21.15.111:0/2276332590 learned_addr learned my addr 172.21.15.111:0/2276332590 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:20.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f750ffff700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f7518169fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 -- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 msgr2=0x7f7518169fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f7518169fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 -- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 msgr2=0x7f751814a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 0x7f751814a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 -- 172.21.15.111:0/2276332590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000094f0 con 0x7f751816cb40 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f7516d9d700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 0x7f751814a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751759e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7510009df0 tx=0x7f751000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751000caa0 con 0x7f751816cb40 2023-12-19T08:02:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751000f040 con 0x7f751816cb40 2023-12-19T08:02:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751816d070 con 0x7f751816cb40 2023-12-19T08:02:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751000ce70 con 0x7f751816cb40 2023-12-19T08:02:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.018+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751816d630 con 0x7f751816cb40 2023-12-19T08:02:20.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.022+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518071120 con 0x7f751816cb40 2023-12-19T08:02:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.022+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7510014690 con 0x7f751816cb40 2023-12-19T08:02:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.026+0000 7f750effd700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 0x7f74f8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.026+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f751007d360 con 0x7f751816cb40 2023-12-19T08:02:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.026+0000 7f7516d9d700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 0x7f74f8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.026+0000 7f7516d9d700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 0x7f74f8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75180c9e60 tx=0x7f7508008040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.030+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7510051130 con 0x7f751816cb40 2023-12-19T08:02:20.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f751806a7a0 con 0x7f751816cb40 2023-12-19T08:02:20.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f750effd700 1 -- 172.21.15.111:0/2276332590 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7510050cc0 con 0x7f751816cb40 2023-12-19T08:02:20.202 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 msgr2=0x7f74f8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 0x7f74f8062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75180c9e60 tx=0x7f7508008040 comp rx=0 tx=0).stop 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 msgr2=0x7f75180ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7510009df0 tx=0x7f751000b040 comp rx=0 tx=0).stop 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 shutdown_connections 2023-12-19T08:02:20.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f74f8060670 0x7f74f8062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f751816cb40 0x7f75180ce3b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7518158e20 0x7f7518169fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 --2- 172.21.15.111:0/2276332590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75180ca2c0 0x7f751814a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 >> 172.21.15.111:0/2276332590 conn(0x7f7518067bb0 msgr2=0x7f751815b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 shutdown_connections 2023-12-19T08:02:20.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.202+0000 7f751d22e700 1 -- 172.21.15.111:0/2276332590 wait complete. 2023-12-19T08:02:20.228 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:02:20.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 Processor -- start 2023-12-19T08:02:20.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 -- start start 2023-12-19T08:02:20.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77ac160f30 0x7f77ac161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f77ac15a730 0x7f77ac15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac15b0f0 con 0x7f77ac06ca60 2023-12-19T08:02:20.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac06d5b0 con 0x7f77ac15a730 2023-12-19T08:02:20.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77b1345700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac06d790 con 0x7f77ac160f30 2023-12-19T08:02:20.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77aa7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f77ac15a730 0x7f77ac15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77ab7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f77ac06ca60 0x7f77ac06cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50636/0 (socket says 172.21.15.111:50636) 2023-12-19T08:02:20.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77ab7fe700 1 -- 172.21.15.111:0/2020111466 learned_addr learned my addr 172.21.15.111:0/2020111466 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:20.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77aaffd700 1 --2- 172.21.15.111:0/2020111466 >> v2:172.21.15.143:3300/0 conn(0x7f77ac160f30 0x7f77ac161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77a9ffb700 1 -- 172.21.15.111:0/2020111466 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610400789 0 0) 0x7f77ac15b0f0 con 0x7f77ac06ca60 2023-12-19T08:02:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.386+0000 7f77aa7fc700 1 -- 172.21.15.111:0/2020111466 >> v2:172.21.15.143:3300/0 conn(0x7f77ac160f30 msgr2=0x7f77ac161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77aa7fc700 1 --2- 172.21.15.111:0/2020111466 >> v2:172.21.15.143:3300/0 conn(0x7f77ac160f30 0x7f77ac161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77aa7fc700 1 -- 172.21.15.111:0/2020111466 >> v1:172.21.15.111:6789/0 conn(0x7f77ac06ca60 legacy=0x7f77ac06cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77aa7fc700 1 -- 172.21.15.111:0/2020111466 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac06da70 con 0x7f77ac15a730 2023-12-19T08:02:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77aa7fc700 1 --2- 172.21.15.111:0/2020111466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac15a730 0x7f77ac15abb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f77ac068260 tx=0x7f77a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcedb075be0539b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77a9ffb700 1 -- 172.21.15.111:0/2020111466 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0011630 con 0x7f77ac15a730 2023-12-19T08:02:20.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77a9ffb700 1 -- 172.21.15.111:0/2020111466 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a0011790 con 0x7f77ac15a730 2023-12-19T08:02:20.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77a9ffb700 1 -- 172.21.15.111:0/2020111466 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0011b30 con 0x7f77ac15a730 2023-12-19T08:02:20.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 -- 172.21.15.111:0/2020111466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac15a730 msgr2=0x7f77ac15abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 --2- 172.21.15.111:0/2020111466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac15a730 0x7f77ac15abb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f77ac068260 tx=0x7f77a00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 -- 172.21.15.111:0/2020111466 shutdown_connections 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 --2- 172.21.15.111:0/2020111466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac15a730 0x7f77ac15abb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 --2- 172.21.15.111:0/2020111466 >> v2:172.21.15.143:3300/0 conn(0x7f77ac160f30 0x7f77ac161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 -- 172.21.15.111:0/2020111466 >> 172.21.15.111:0/2020111466 conn(0x7f77ac067bb0 msgr2=0x7f77ac0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 -- 172.21.15.111:0/2020111466 shutdown_connections 2023-12-19T08:02:20.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.390+0000 7f77b1345700 1 -- 172.21.15.111:0/2020111466 wait complete. 2023-12-19T08:02:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 Processor -- start 2023-12-19T08:02:20.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 -- start start 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac06ca60 0x7f77ac1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac15a730 0x7f77ac0c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac16e120 con 0x7f77ac160f30 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0c59e0 con 0x7f77ac15a730 2023-12-19T08:02:20.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77b1345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac16d7f0 con 0x7f77ac06ca60 2023-12-19T08:02:20.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36880/0 (socket says 172.21.15.111:36880) 2023-12-19T08:02:20.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.394+0000 7f77ab7fe700 1 -- 172.21.15.111:0/1772001219 learned_addr learned my addr 172.21.15.111:0/1772001219 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:20.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77aa7fc700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac15a730 0x7f77ac0c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77aaffd700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac06ca60 0x7f77ac1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 -- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac06ca60 msgr2=0x7f77ac1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac06ca60 0x7f77ac1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 -- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac15a730 msgr2=0x7f77ac0c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac15a730 0x7f77ac0c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 -- 172.21.15.111:0/1772001219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0009420 con 0x7f77ac160f30 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77ab7fe700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f77a400cc90 tx=0x7f77a4001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400a960 con 0x7f77ac160f30 2023-12-19T08:02:20.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a400aac0 con 0x7f77ac160f30 2023-12-19T08:02:20.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac16dad0 con 0x7f77ac160f30 2023-12-19T08:02:20.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400ae80 con 0x7f77ac160f30 2023-12-19T08:02:20.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac1585f0 con 0x7f77ac160f30 2023-12-19T08:02:20.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.398+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac16e290 con 0x7f77ac160f30 2023-12-19T08:02:20.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.402+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f77a4007300 con 0x7f77ac160f30 2023-12-19T08:02:20.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.402+0000 7f7792ffd700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 0x7f7798062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.402+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f77a407cd70 con 0x7f77ac160f30 2023-12-19T08:02:20.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.402+0000 7f77aaffd700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 0x7f7798062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.406+0000 7f77aaffd700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 0x7f7798062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f779c009ca0 tx=0x7f779c008040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.406+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f77a404d240 con 0x7f77ac160f30 2023-12-19T08:02:20.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.578+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f77ac04be40 con 0x7f77ac160f30 2023-12-19T08:02:20.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.578+0000 7f7792ffd700 1 -- 172.21.15.111:0/1772001219 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f77a4050860 con 0x7f77ac160f30 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:20.581 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:02:20.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 msgr2=0x7f7798062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 0x7f7798062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f779c009ca0 tx=0x7f779c008040 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 msgr2=0x7f77ac0c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f77a400cc90 tx=0x7f77a4001a80 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 shutdown_connections 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f77980606a0 0x7f7798062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77ac15a730 0x7f77ac0c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77ac160f30 0x7f77ac0c6a00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 --2- 172.21.15.111:0/1772001219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77ac06ca60 0x7f77ac1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 >> 172.21.15.111:0/1772001219 conn(0x7f77ac067bb0 msgr2=0x7f77ac158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 shutdown_connections 2023-12-19T08:02:20.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.582+0000 7f77b1345700 1 -- 172.21.15.111:0/1772001219 wait complete. 2023-12-19T08:02:20.603 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:02:20.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:02:20.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.770+0000 7fcbc7fff700 1 Processor -- start 2023-12-19T08:02:20.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.770+0000 7fcbc7fff700 1 -- start start 2023-12-19T08:02:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbc8156d60 0x7fcbc80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 0x7fcbc80c9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc80ae230 con 0x7fcbc8156d60 2023-12-19T08:02:20.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc80b0480 con 0x7fcbc80addf0 2023-12-19T08:02:20.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc806c7c0 con 0x7fcbc8156990 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbc8156d60 0x7fcbc80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbc8156d60 0x7fcbc80ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36896/0 (socket says 172.21.15.111:36896) 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 -- 172.21.15.111:0/2118926075 learned_addr learned my addr 172.21.15.111:0/2118926075 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc77fe700 1 --2- 172.21.15.111:0/2118926075 >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 0x7fcbc80c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc6ffd700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 -- 172.21.15.111:0/2118926075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 msgr2=0x7fcbc806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc806dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 -- 172.21.15.111:0/2118926075 >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 msgr2=0x7fcbc80c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/2118926075 >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 0x7fcbc80c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 -- 172.21.15.111:0/2118926075 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc806d620 con 0x7fcbc8156d60 2023-12-19T08:02:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc77fe700 1 --2- 172.21.15.111:0/2118926075 >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 0x7fcbc80c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc8156d60 0x7fcbc80ad8b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c8040 tx=0x7fcbbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32aa8c386b0e291a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc5ffb700 1 -- 172.21.15.111:0/2118926075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc00b820 con 0x7fcbc8156d60 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc5ffb700 1 -- 172.21.15.111:0/2118926075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc00b980 con 0x7fcbc8156d60 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc5ffb700 1 -- 172.21.15.111:0/2118926075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc00bd20 con 0x7fcbc8156d60 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- 172.21.15.111:0/2118926075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc8156d60 msgr2=0x7fcbc80ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc8156d60 0x7fcbc80ad8b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c8040 tx=0x7fcbbc009250 comp rx=0 tx=0).stop 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- 172.21.15.111:0/2118926075 shutdown_connections 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/2118926075 >> v2:172.21.15.143:3300/0 conn(0x7fcbc80addf0 0x7fcbc80c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc8156d60 0x7fcbc80ad8b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/2118926075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc806dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- 172.21.15.111:0/2118926075 >> 172.21.15.111:0/2118926075 conn(0x7fcbc80ce5e0 msgr2=0x7fcbc80ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- 172.21.15.111:0/2118926075 shutdown_connections 2023-12-19T08:02:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.774+0000 7fcbc7fff700 1 -- 172.21.15.111:0/2118926075 wait complete. 2023-12-19T08:02:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 Processor -- start 2023-12-19T08:02:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 -- start start 2023-12-19T08:02:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc80c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbc8156d60 0x7fcbc8148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc815cce0 con 0x7fcbc80addf0 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc815ac60 con 0x7fcbc8156d60 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc815c690 con 0x7fcbc8156990 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36912/0 (socket says 172.21.15.111:36912) 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc6ffd700 1 -- 172.21.15.111:0/3864653582 learned_addr learned my addr 172.21.15.111:0/3864653582 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc80c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.778+0000 7fcbc77fe700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbc8156d60 0x7fcbc8148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 -- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 msgr2=0x7fcbc80c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc80c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 -- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbc8156d60 msgr2=0x7fcbc8148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbc8156d60 0x7fcbc8148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 -- 172.21.15.111:0/3864653582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0094f0 con 0x7fcbc80addf0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc6ffd700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800d890 tx=0x7fcbb8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb80125f0 con 0x7fcbc80addf0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb8012770 con 0x7fcbc80addf0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8012b30 con 0x7fcbc80addf0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc815c970 con 0x7fcbc80addf0 2023-12-19T08:02:20.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc81494f0 con 0x7fcbc80addf0 2023-12-19T08:02:20.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.782+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc804be40 con 0x7fcbc80addf0 2023-12-19T08:02:20.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.786+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcbb801e460 con 0x7fcbc80addf0 2023-12-19T08:02:20.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.786+0000 7fcbaeffd700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 0x7fcbb4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:20.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.786+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fcbb807cee0 con 0x7fcbc80addf0 2023-12-19T08:02:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.786+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 0x7fcbb4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.786+0000 7fcbc67fc700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 0x7fcbb4062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c8040 tx=0x7fcbbc017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.794+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcbb804d3d0 con 0x7fcbc80addf0 2023-12-19T08:02:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:20.958+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcbc806ca10 con 0x7fcbc80addf0 2023-12-19T08:02:21.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.578+0000 7fcbaeffd700 1 -- 172.21.15.111:0/3864653582 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcbb80509f0 con 0x7fcbc80addf0 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 msgr2=0x7fcbb4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 0x7fcbb4062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c8040 tx=0x7fcbbc017000 comp rx=0 tx=0).stop 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 msgr2=0x7fcbc80c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800d890 tx=0x7fcbb8008040 comp rx=0 tx=0).stop 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 shutdown_connections 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcbb4060670 0x7fcbb4062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbc8156d60 0x7fcbc8148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbc8156990 0x7fcbc80c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 --2- 172.21.15.111:0/3864653582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbc80addf0 0x7fcbc80c5320 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 >> 172.21.15.111:0/3864653582 conn(0x7fcbc80ce5e0 msgr2=0x7fcbc81557f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 shutdown_connections 2023-12-19T08:02:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.582+0000 7fcbc7fff700 1 -- 172.21.15.111:0/3864653582 wait complete. 2023-12-19T08:02:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:02:21.607 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:02:21.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.790+0000 7f1537878700 1 Processor -- start 2023-12-19T08:02:21.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.790+0000 7f1537878700 1 -- start start 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1537878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1530158e20 0x7f15300ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1537878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f15300cb970 0x7f15300ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1537878700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300cbdf0 con 0x7f15300ca2c0 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1537878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300cc0d0 con 0x7f15300cb970 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1537878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300cc3b0 con 0x7f1530158e20 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1535e15700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f15300ca2c0 0x7f153006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41922/0 (socket says 172.21.15.111:41922) 2023-12-19T08:02:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1535e15700 1 -- 172.21.15.111:0/2762031404 learned_addr learned my addr 172.21.15.111:0/2762031404 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 --2- 172.21.15.111:0/2762031404 >> v2:172.21.15.143:3300/0 conn(0x7f15300cb970 0x7f15300ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1535614700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1530158e20 0x7f15300ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1527fff700 1 -- 172.21.15.111:0/2762031404 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543572111 0 0) 0x7f15300cbdf0 con 0x7f15300ca2c0 2023-12-19T08:02:21.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1527fff700 1 -- 172.21.15.111:0/2762031404 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003570 con 0x7f15300ca2c0 2023-12-19T08:02:21.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 -- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1530158e20 msgr2=0x7f15300ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:21.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1530158e20 0x7f15300ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 -- 172.21.15.111:0/2762031404 >> v1:172.21.15.111:6789/0 conn(0x7f15300ca2c0 legacy=0x7f153006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 -- 172.21.15.111:0/2762031404 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300cc690 con 0x7f15300cb970 2023-12-19T08:02:21.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1534e13700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15300cb970 0x7f15300ceb00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f15200098e0 tx=0x7f15200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c85413866e4ed65b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:21.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1527fff700 1 -- 172.21.15.111:0/2762031404 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520011630 con 0x7f15300cb970 2023-12-19T08:02:21.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1527fff700 1 -- 172.21.15.111:0/2762031404 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520011790 con 0x7f15300cb970 2023-12-19T08:02:21.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.794+0000 7f1527fff700 1 -- 172.21.15.111:0/2762031404 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520011b30 con 0x7f15300cb970 2023-12-19T08:02:21.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 -- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15300cb970 msgr2=0x7f15300ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15300cb970 0x7f15300ceb00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f15200098e0 tx=0x7f15200091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 -- 172.21.15.111:0/2762031404 shutdown_connections 2023-12-19T08:02:21.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15300cb970 0x7f15300ceb00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 --2- 172.21.15.111:0/2762031404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1530158e20 0x7f15300ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 -- 172.21.15.111:0/2762031404 >> 172.21.15.111:0/2762031404 conn(0x7f1530067bb0 msgr2=0x7f15300d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:21.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 -- 172.21.15.111:0/2762031404 shutdown_connections 2023-12-19T08:02:21.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.798+0000 7f1537878700 1 -- 172.21.15.111:0/2762031404 wait complete. 2023-12-19T08:02:21.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 Processor -- start 2023-12-19T08:02:21.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 -- start start 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1530158e20 0x7f15300b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15300b49e0 0x7f15300c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300b4e80 con 0x7f15300ca2c0 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530169620 con 0x7f1530158e20 2023-12-19T08:02:21.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1537878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153016a440 con 0x7f15300b49e0 2023-12-19T08:02:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1535614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1535614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57924/0 (socket says 172.21.15.111:57924) 2023-12-19T08:02:21.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.802+0000 7f1535614700 1 -- 172.21.15.111:0/1428944260 learned_addr learned my addr 172.21.15.111:0/1428944260 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1534e13700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1530158e20 0x7f15300b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535e15700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15300b49e0 0x7f15300c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 -- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15300b49e0 msgr2=0x7f15300c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15300b49e0 0x7f15300c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 -- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1530158e20 msgr2=0x7f15300b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1530158e20 0x7f15300b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:21.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 -- 172.21.15.111:0/1428944260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520009420 con 0x7f15300ca2c0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1535614700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f152c00e930 tx=0x7f152c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c013680 con 0x7f15300ca2c0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c0137e0 con 0x7f15300ca2c0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c013ba0 con 0x7f15300ca2c0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530169830 con 0x7f15300ca2c0 2023-12-19T08:02:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1530169df0 con 0x7f15300ca2c0 2023-12-19T08:02:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.806+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530071120 con 0x7f15300ca2c0 2023-12-19T08:02:21.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.810+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f152c00c3e0 con 0x7f15300ca2c0 2023-12-19T08:02:21.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.810+0000 7f15257fa700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 0x7f1514062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:21.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.810+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6063+0+0 (secure 0 0 0) 0x7f152c07d260 con 0x7f15300ca2c0 2023-12-19T08:02:21.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.814+0000 7f1534e13700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 0x7f1514062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:21.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.814+0000 7f1534e13700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 0x7f1514062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f15200091e0 tx=0x7f1520017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:21.818+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f152c04d660 con 0x7f15300ca2c0 2023-12-19T08:02:22.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.002+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f153006a7a0 con 0x7f15300ca2c0 2023-12-19T08:02:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.586+0000 7f15257fa700 1 -- 172.21.15.111:0/1428944260 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f152c050c80 con 0x7f15300ca2c0 2023-12-19T08:02:22.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 msgr2=0x7f1514062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 0x7f1514062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f15200091e0 tx=0x7f1520017000 comp rx=0 tx=0).stop 2023-12-19T08:02:22.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 msgr2=0x7f1530148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f152c00e930 tx=0x7f152c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 shutdown_connections 2023-12-19T08:02:22.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1514060670 0x7f1514062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1530158e20 0x7f15300b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15300b49e0 0x7f15300c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 --2- 172.21.15.111:0/1428944260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15300ca2c0 0x7f1530148b70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 >> 172.21.15.111:0/1428944260 conn(0x7f1530067bb0 msgr2=0x7f15301493f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:22.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 shutdown_connections 2023-12-19T08:02:22.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.598+0000 7f1537878700 1 -- 172.21.15.111:0/1428944260 wait complete. 2023-12-19T08:02:22.610 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:02:22.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:02:22.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.790+0000 7f8d3a505700 1 Processor -- start 2023-12-19T08:02:22.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.790+0000 7f8d3a505700 1 -- start start 2023-12-19T08:02:22.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d3a505700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d340cecd0 0x7f8d3406b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:22.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d3a505700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340cbdf0 con 0x7f8d34158e20 2023-12-19T08:02:22.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d3a505700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340cc0d0 con 0x7f8d340cb970 2023-12-19T08:02:22.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d3a505700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d340cc3b0 con 0x7f8d340cecd0 2023-12-19T08:02:22.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d33fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8d34158e20 0x7f8d340ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41926/0 (socket says 172.21.15.111:41926) 2023-12-19T08:02:22.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d33fff700 1 -- 172.21.15.111:0/406561177 learned_addr learned my addr 172.21.15.111:0/406561177 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:22.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d38aa2700 1 --2- 172.21.15.111:0/406561177 >> v2:172.21.15.143:3300/0 conn(0x7f8d340cecd0 0x7f8d3406b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d32ffd700 1 -- 172.21.15.111:0/406561177 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006563987 0 0) 0x7f8d340cbdf0 con 0x7f8d34158e20 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d32ffd700 1 -- 172.21.15.111:0/406561177 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c003570 con 0x7f8d34158e20 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d38aa2700 1 -- 172.21.15.111:0/406561177 >> v1:172.21.15.143:6789/0 conn(0x7f8d340cb970 legacy=0x7f8d3406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d38aa2700 1 -- 172.21.15.111:0/406561177 >> v1:172.21.15.111:6789/0 conn(0x7f8d34158e20 legacy=0x7f8d340ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.794+0000 7f8d38aa2700 1 -- 172.21.15.111:0/406561177 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d340cc690 con 0x7f8d340cecd0 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d38aa2700 1 --2- 172.21.15.111:0/406561177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d3406b3d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009e90 tx=0x7f8d2c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db75b9a5ae0b0d96 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:22.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d32ffd700 1 -- 172.21.15.111:0/406561177 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00e070 con 0x7f8d340cecd0 2023-12-19T08:02:22.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d32ffd700 1 -- 172.21.15.111:0/406561177 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c00bc40 con 0x7f8d340cecd0 2023-12-19T08:02:22.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d32ffd700 1 -- 172.21.15.111:0/406561177 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c012430 con 0x7f8d340cecd0 2023-12-19T08:02:22.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 -- 172.21.15.111:0/406561177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 msgr2=0x7f8d3406b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 --2- 172.21.15.111:0/406561177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d3406b3d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009e90 tx=0x7f8d2c009780 comp rx=0 tx=0).stop 2023-12-19T08:02:22.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 -- 172.21.15.111:0/406561177 shutdown_connections 2023-12-19T08:02:22.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 --2- 172.21.15.111:0/406561177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d3406b3d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 -- 172.21.15.111:0/406561177 >> 172.21.15.111:0/406561177 conn(0x7f8d34067bb0 msgr2=0x7f8d340d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:22.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 -- 172.21.15.111:0/406561177 shutdown_connections 2023-12-19T08:02:22.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.798+0000 7f8d3a505700 1 -- 172.21.15.111:0/406561177 wait complete. 2023-12-19T08:02:22.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 Processor -- start 2023-12-19T08:02:22.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 -- start start 2023-12-19T08:02:22.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d340c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:22.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:22.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3416dd90 0x7f8d340cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:22.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340b1060 con 0x7f8d34158e20 2023-12-19T08:02:22.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340b1300 con 0x7f8d340cecd0 2023-12-19T08:02:22.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d3a505700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340b15e0 con 0x7f8d3416dd90 2023-12-19T08:02:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.802+0000 7f8d337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57934/0 (socket says 172.21.15.111:57934) 2023-12-19T08:02:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 -- 172.21.15.111:0/3003343757 learned_addr learned my addr 172.21.15.111:0/3003343757 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d33fff700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d340c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:22.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d38aa2700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3416dd90 0x7f8d340cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 -- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3416dd90 msgr2=0x7f8d340cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3416dd90 0x7f8d340cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 -- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 msgr2=0x7f8d340c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d340c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 -- 172.21.15.111:0/3003343757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c009a20 con 0x7f8d34158e20 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d337fe700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800c590 tx=0x7f8d28007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:22.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28011590 con 0x7f8d34158e20 2023-12-19T08:02:22.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28011710 con 0x7f8d34158e20 2023-12-19T08:02:22.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28011ad0 con 0x7f8d34158e20 2023-12-19T08:02:22.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d340b18c0 con 0x7f8d34158e20 2023-12-19T08:02:22.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d340c9e70 con 0x7f8d34158e20 2023-12-19T08:02:22.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.806+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d34071120 con 0x7f8d34158e20 2023-12-19T08:02:22.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.810+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8d28006820 con 0x7f8d34158e20 2023-12-19T08:02:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.814+0000 7f8d1bfff700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:22.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.814+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f8d2807bcb0 con 0x7f8d34158e20 2023-12-19T08:02:22.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.814+0000 7f8d33fff700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:22.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.814+0000 7f8d33fff700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d24007fd0 tx=0x7f8d24009040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:22.818+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d2804be90 con 0x7f8d34158e20 2023-12-19T08:02:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.018+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8d3406a7a0 con 0x7f8d34158e20 2023-12-19T08:02:23.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.602+0000 7f8d1bfff700 1 -- 172.21.15.111:0/3003343757 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8d28017070 con 0x7f8d34158e20 2023-12-19T08:02:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 msgr2=0x7f8d20062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d24007fd0 tx=0x7f8d24009040 comp rx=0 tx=0).stop 2023-12-19T08:02:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 msgr2=0x7f8d3416d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800c590 tx=0x7f8d28007f50 comp rx=0 tx=0).stop 2023-12-19T08:02:23.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 shutdown_connections 2023-12-19T08:02:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3416dd90 0x7f8d340cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d34158e20 0x7f8d3416d850 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 --2- 172.21.15.111:0/3003343757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d340cecd0 0x7f8d340c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 >> 172.21.15.111:0/3003343757 conn(0x7f8d34067bb0 msgr2=0x7f8d3414ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:23.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 shutdown_connections 2023-12-19T08:02:23.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.610+0000 7f8d3a505700 1 -- 172.21.15.111:0/3003343757 wait complete. 2023-12-19T08:02:23.616 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 9 and data pool 10 2023-12-19T08:02:23.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:23.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 Processor -- start 2023-12-19T08:02:23.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 -- start start 2023-12-19T08:02:23.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0d0880 con 0x7efd9c06c7c0 2023-12-19T08:02:23.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0cfc70 con 0x7efd9c156940 2023-12-19T08:02:23.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.794+0000 7efda1a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0cff50 con 0x7efd9c156d10 2023-12-19T08:02:23.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7efd9c06c7c0 0x7efd9c15cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36468/0 (socket says 172.21.15.111:36468) 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:54612/0 (socket says 172.21.15.111:54612) 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9bfff700 1 -- 172.21.15.111:0/3626775601 learned_addr learned my addr 172.21.15.111:0/3626775601 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:40256/0 (socket says 172.21.15.111:40256) 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd99ffb700 1 -- 172.21.15.111:0/3626775601 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669212347 0 0) 0x7efd9c0d0880 con 0x7efd9c06c7c0 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd99ffb700 1 -- 172.21.15.111:0/3626775601 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd84003570 con 0x7efd9c06c7c0 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 -- 172.21.15.111:0/3626775601 >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 msgr2=0x7efd9c0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/3626775601 >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 -- 172.21.15.111:0/3626775601 >> v1:172.21.15.143:6789/0 conn(0x7efd9c06c7c0 legacy=0x7efd9c15cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 -- 172.21.15.111:0/3626775601 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0d0230 con 0x7efd9c156d10 2023-12-19T08:02:23.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9affd700 1 --2- 172.21.15.111:0/3626775601 >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:23.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/3626775601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd900098e0 tx=0x7efd900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff3a30c98dcd5054 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:23.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd99ffb700 1 -- 172.21.15.111:0/3626775601 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011630 con 0x7efd9c156d10 2023-12-19T08:02:23.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd99ffb700 1 -- 172.21.15.111:0/3626775601 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90011790 con 0x7efd9c156d10 2023-12-19T08:02:23.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.798+0000 7efd99ffb700 1 -- 172.21.15.111:0/3626775601 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011b30 con 0x7efd9c156d10 2023-12-19T08:02:23.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 -- 172.21.15.111:0/3626775601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 msgr2=0x7efd9c159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 --2- 172.21.15.111:0/3626775601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd900098e0 tx=0x7efd900091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 -- 172.21.15.111:0/3626775601 shutdown_connections 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 --2- 172.21.15.111:0/3626775601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c159f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 --2- 172.21.15.111:0/3626775601 >> v2:172.21.15.143:3300/0 conn(0x7efd9c156940 0x7efd9c0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 -- 172.21.15.111:0/3626775601 >> 172.21.15.111:0/3626775601 conn(0x7efd9c0ce320 msgr2=0x7efd9c0ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 -- 172.21.15.111:0/3626775601 shutdown_connections 2023-12-19T08:02:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.802+0000 7efda1a2e700 1 -- 172.21.15.111:0/3626775601 wait complete. 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 Processor -- start 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 -- start start 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 0x7efd9c0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0b5f20 con 0x7efd9c06c7c0 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0b5310 con 0x7efd9c156940 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0b55f0 con 0x7efd9c156d10 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57938/0 (socket says 172.21.15.111:57938) 2023-12-19T08:02:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 -- 172.21.15.111:0/2027687908 learned_addr learned my addr 172.21.15.111:0/2027687908 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 0x7efd9c0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9bfff700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 -- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 msgr2=0x7efd9c146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 -- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 msgr2=0x7efd9c0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 0x7efd9c0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 -- 172.21.15.111:0/2027687908 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c009420 con 0x7efd9c06c7c0 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 0x7efd9c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9bfff700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd9affd700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00ec80 tx=0x7efd8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:23.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00c9d0 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c00cb30 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.806+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd90009420 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.810+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00ceb0 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.810+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c0b5bb0 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.810+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c071120 con 0x7efd9c06c7c0 2023-12-19T08:02:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.810+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7efd8c013680 con 0x7efd9c06c7c0 2023-12-19T08:02:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.814+0000 7efd82ffd700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 0x7efd88062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.814+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6545+0+0 (secure 0 0 0) 0x7efd8c07d120 con 0x7efd9c06c7c0 2023-12-19T08:02:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.814+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 0x7efd88062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:23.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.814+0000 7efd9a7fc700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 0x7efd88062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0690d0 tx=0x7efd90017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:23.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.818+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd8c04d210 con 0x7efd9c06c7c0 2023-12-19T08:02:23.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.990+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd9c06d440 con 0x7efd9c06c7c0 2023-12-19T08:02:23.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.994+0000 7efd82ffd700 1 -- 172.21.15.111:0/2027687908 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16407 (secure 0 0 0) 0x7efd8c050830 con 0x7efd9c06c7c0 2023-12-19T08:02:23.995 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:23.995 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":84,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:23.021919+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:23.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 msgr2=0x7efd88062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 0x7efd88062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0690d0 tx=0x7efd90017000 comp rx=0 tx=0).stop 2023-12-19T08:02:23.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 msgr2=0x7efd9c0b63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:23.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00ec80 tx=0x7efd8c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 shutdown_connections 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd88060670 0x7efd88062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd9c156940 0x7efd9c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9c156d10 0x7efd9c146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 --2- 172.21.15.111:0/2027687908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd9c06c7c0 0x7efd9c0b63b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 >> 172.21.15.111:0/2027687908 conn(0x7efd9c0ce320 msgr2=0x7efd9c06cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 shutdown_connections 2023-12-19T08:02:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:23.998+0000 7efda1a2e700 1 -- 172.21.15.111:0/2027687908 wait complete. 2023-12-19T08:02:24.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:24.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.182+0000 7f24e6fcc700 1 Processor -- start 2023-12-19T08:02:24.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.182+0000 7f24e6fcc700 1 -- start start 2023-12-19T08:02:24.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e6fcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 0x7f24e00c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e6fcc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e006b470 con 0x7f24e006b900 2023-12-19T08:02:24.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e6fcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e006e110 con 0x7f24e015cc20 2023-12-19T08:02:24.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e6fcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e015d7b0 con 0x7f24e00c6900 2023-12-19T08:02:24.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e57c9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f24e006b900 0x7f24e006da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41938/0 (socket says 172.21.15.111:41938) 2023-12-19T08:02:24.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e57c9700 1 -- 172.21.15.111:0/2427280086 learned_addr learned my addr 172.21.15.111:0/2427280086 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e5fca700 1 --2- 172.21.15.111:0/2427280086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 0x7f24e00c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025320040 0 0) 0x7f24e006b470 con 0x7f24e006b900 2023-12-19T08:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc003570 con 0x7f24e006b900 2023-12-19T08:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e5fca700 1 -- 172.21.15.111:0/2427280086 >> v1:172.21.15.143:6789/0 conn(0x7f24e015cc20 legacy=0x7f24e015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211323742 0 0) 0x7f24e006e110 con 0x7f24e015cc20 2023-12-19T08:02:24.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e5fca700 1 -- 172.21.15.111:0/2427280086 >> v1:172.21.15.111:6789/0 conn(0x7f24e006b900 legacy=0x7f24e006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.186+0000 7f24e5fca700 1 -- 172.21.15.111:0/2427280086 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0161d10 con 0x7f24e00c6900 2023-12-19T08:02:24.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e5fca700 1 --2- 172.21.15.111:0/2427280086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 0x7f24e00c6cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009940 tx=0x7f24dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be8c40dad9d3aec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00b670 con 0x7f24e00c6900 2023-12-19T08:02:24.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc00b7d0 con 0x7f24e00c6900 2023-12-19T08:02:24.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e4fc8700 1 -- 172.21.15.111:0/2427280086 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00bb90 con 0x7f24e00c6900 2023-12-19T08:02:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 -- 172.21.15.111:0/2427280086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 msgr2=0x7f24e00c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/2427280086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 0x7f24e00c6cd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009940 tx=0x7f24dc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 -- 172.21.15.111:0/2427280086 shutdown_connections 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/2427280086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e00c6900 0x7f24e00c6cd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 -- 172.21.15.111:0/2427280086 >> 172.21.15.111:0/2427280086 conn(0x7f24e0067bb0 msgr2=0x7f24e00cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 -- 172.21.15.111:0/2427280086 shutdown_connections 2023-12-19T08:02:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.190+0000 7f24e6fcc700 1 -- 172.21.15.111:0/2427280086 wait complete. 2023-12-19T08:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.194+0000 7f24e6fcc700 1 Processor -- start 2023-12-19T08:02:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.194+0000 7f24e6fcc700 1 -- start start 2023-12-19T08:02:24.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e006b900 0x7f24e01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24e00c6900 0x7f24e014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e01605f0 con 0x7f24e015cc20 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00b0620 con 0x7f24e00c6900 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00afa10 con 0x7f24e006b900 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57946/0 (socket says 172.21.15.111:57946) 2023-12-19T08:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 -- 172.21.15.111:0/371089465 learned_addr learned my addr 172.21.15.111:0/371089465 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e57c9700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24e00c6900 0x7f24e014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e5fca700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e006b900 0x7f24e01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 -- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e006b900 msgr2=0x7f24e01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e006b900 0x7f24e01539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 -- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24e00c6900 msgr2=0x7f24e014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24e00c6900 0x7f24e014fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 -- 172.21.15.111:0/371089465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc009420 con 0x7f24e015cc20 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e67cb700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c890 tx=0x7f24d8001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8012630 con 0x7f24e015cc20 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d8012790 con 0x7f24e015cc20 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e00afcf0 con 0x7f24e015cc20 2023-12-19T08:02:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8012b50 con 0x7f24e015cc20 2023-12-19T08:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.198+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e00b02b0 con 0x7f24e015cc20 2023-12-19T08:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.202+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e00b07b0 con 0x7f24e015cc20 2023-12-19T08:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.202+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f24d801a3a0 con 0x7f24e015cc20 2023-12-19T08:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.202+0000 7f24d5ffb700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 0x7f24c4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.202+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f24d807cec0 con 0x7f24e015cc20 2023-12-19T08:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.202+0000 7f24e5fca700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 0x7f24c4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.206+0000 7f24e5fca700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 0x7f24c4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009910 tx=0x7f24dc017000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.210+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f24d804cfb0 con 0x7f24e015cc20 2023-12-19T08:02:24.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.382+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24e004be40 con 0x7f24e015cc20 2023-12-19T08:02:24.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24d5ffb700 1 -- 172.21.15.111:0/371089465 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16407 (secure 0 0 0) 0x7f24d80505d0 con 0x7f24e015cc20 2023-12-19T08:02:24.385 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:24.385 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":84,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:23.021919+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 msgr2=0x7f24c4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 0x7f24c4062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009910 tx=0x7f24dc017000 comp rx=0 tx=0).stop 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 msgr2=0x7f24e015e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c890 tx=0x7f24d8001670 comp rx=0 tx=0).stop 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 shutdown_connections 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24c4060670 0x7f24c4062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24e00c6900 0x7f24e014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24e015cc20 0x7f24e015e300 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 --2- 172.21.15.111:0/371089465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24e006b900 0x7f24e01539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 >> 172.21.15.111:0/371089465 conn(0x7f24e0067bb0 msgr2=0x7f24e0159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:24.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 shutdown_connections 2023-12-19T08:02:24.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.386+0000 7f24e6fcc700 1 -- 172.21.15.111:0/371089465 wait complete. 2023-12-19T08:02:24.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:02:24.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.574+0000 7fda244a5700 1 Processor -- start 2023-12-19T08:02:24.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 -- start start 2023-12-19T08:02:24.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda1c0c6d20 0x7fda1c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c06c7c0 0x7fda1c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fda1c15aa30 0x7fda1c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda1c06a7a0 con 0x7fda1c15aa30 2023-12-19T08:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda1c0c7630 con 0x7fda1c0c6d20 2023-12-19T08:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda244a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c06d4b0 con 0x7fda1c06c7c0 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda22a42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fda1c15aa30 0x7fda1c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda22a42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fda1c15aa30 0x7fda1c15ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57958/0 (socket says 172.21.15.111:57958) 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda22a42700 1 -- 172.21.15.111:0/1205987193 learned_addr learned my addr 172.21.15.111:0/1205987193 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda21a40700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c06c7c0 0x7fda1c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.578+0000 7fda22241700 1 --2- 172.21.15.111:0/1205987193 >> v2:172.21.15.143:3300/0 conn(0x7fda1c0c6d20 0x7fda1c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 -- 172.21.15.111:0/1205987193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c06c7c0 msgr2=0x7fda1c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c06c7c0 0x7fda1c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 -- 172.21.15.111:0/1205987193 >> v2:172.21.15.143:3300/0 conn(0x7fda1c0c6d20 msgr2=0x7fda1c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 --2- 172.21.15.111:0/1205987193 >> v2:172.21.15.143:3300/0 conn(0x7fda1c0c6d20 0x7fda1c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 -- 172.21.15.111:0/1205987193 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c06d790 con 0x7fda1c15aa30 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda22a42700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c15aa30 0x7fda1c15ae50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fda1c068030 tx=0x7fda14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a67373af31b91fa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda2123f700 1 -- 172.21.15.111:0/1205987193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1400e070 con 0x7fda1c15aa30 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda2123f700 1 -- 172.21.15.111:0/1205987193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1400ba20 con 0x7fda1c15aa30 2023-12-19T08:02:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda2123f700 1 -- 172.21.15.111:0/1205987193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda140092e0 con 0x7fda1c15aa30 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 -- 172.21.15.111:0/1205987193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c15aa30 msgr2=0x7fda1c15ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c15aa30 0x7fda1c15ae50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fda1c068030 tx=0x7fda14009840 comp rx=0 tx=0).stop 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 -- 172.21.15.111:0/1205987193 shutdown_connections 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c15aa30 0x7fda1c15ae50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 --2- 172.21.15.111:0/1205987193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c06c7c0 0x7fda1c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 --2- 172.21.15.111:0/1205987193 >> v2:172.21.15.143:3300/0 conn(0x7fda1c0c6d20 0x7fda1c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 -- 172.21.15.111:0/1205987193 >> 172.21.15.111:0/1205987193 conn(0x7fda1c067bb0 msgr2=0x7fda1c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 -- 172.21.15.111:0/1205987193 shutdown_connections 2023-12-19T08:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.582+0000 7fda244a5700 1 -- 172.21.15.111:0/1205987193 wait complete. 2023-12-19T08:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 Processor -- start 2023-12-19T08:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 -- start start 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 0x7fda1c1527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c15aa30 0x7fda1c1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c167260 con 0x7fda1c0c6d20 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c166c50 con 0x7fda1c06c7c0 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda244a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c1561e0 con 0x7fda1c15aa30 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57972/0 (socket says 172.21.15.111:57972) 2023-12-19T08:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 -- 172.21.15.111:0/2043060568 learned_addr learned my addr 172.21.15.111:0/2043060568 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda22241700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 0x7fda1c1527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda22a42700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c15aa30 0x7fda1c1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 -- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c15aa30 msgr2=0x7fda1c1535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c15aa30 0x7fda1c1535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 -- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 msgr2=0x7fda1c1527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 0x7fda1c1527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 -- 172.21.15.111:0/2043060568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14009ae0 con 0x7fda1c0c6d20 2023-12-19T08:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda22241700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 0x7fda1c1527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.586+0000 7fda21a40700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00ec80 tx=0x7fda0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c00c9d0 con 0x7fda1c0c6d20 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c00cb30 con 0x7fda1c0c6d20 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1c166ef0 con 0x7fda1c0c6d20 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c013470 con 0x7fda1c0c6d20 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1c152d10 con 0x7fda1c0c6d20 2023-12-19T08:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1c1673d0 con 0x7fda1c0c6d20 2023-12-19T08:02:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.590+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fda0c013820 con 0x7fda1c0c6d20 2023-12-19T08:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.594+0000 7fda127fc700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 0x7fda00062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.594+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fda0c07d070 con 0x7fda1c0c6d20 2023-12-19T08:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.594+0000 7fda22241700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 0x7fda00062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.594+0000 7fda22241700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 0x7fda00062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda18006fd0 tx=0x7fda18008040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.598+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fda0c0acbe0 con 0x7fda1c0c6d20 2023-12-19T08:02:24.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:24.802+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fda1c06da70 con 0x7fda1c0c6d20 2023-12-19T08:02:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.634+0000 7fda127fc700 1 -- 172.21.15.111:0/2043060568 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7fda0c04d160 con 0x7fda1c0c6d20 2023-12-19T08:02:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 msgr2=0x7fda00062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 0x7fda00062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda18006fd0 tx=0x7fda18008040 comp rx=0 tx=0).stop 2023-12-19T08:02:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 msgr2=0x7fda1c153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00ec80 tx=0x7fda0c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 shutdown_connections 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fda00060670 0x7fda00062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda1c15aa30 0x7fda1c1535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda1c0c6d20 0x7fda1c153090 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 --2- 172.21.15.111:0/2043060568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c06c7c0 0x7fda1c1527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 >> 172.21.15.111:0/2043060568 conn(0x7fda1c067bb0 msgr2=0x7fda1c159e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 shutdown_connections 2023-12-19T08:02:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.638+0000 7fda244a5700 1 -- 172.21.15.111:0/2043060568 wait complete. 2023-12-19T08:02:25.658 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:02:25.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 Processor -- start 2023-12-19T08:02:25.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 -- start start 2023-12-19T08:02:25.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 0x7fa78c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 0x7fa78c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c15ae30 con 0x7fa78c154780 2023-12-19T08:02:25.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78c0abf70 con 0x7fa78c06baa0 2023-12-19T08:02:25.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa791123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c154b50 con 0x7fa78c0adab0 2023-12-19T08:02:25.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa7837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 0x7fa78c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa790922700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 0x7fa78c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa7837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 0x7fa78c15b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57974/0 (socket says 172.21.15.111:57974) 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa7837fe700 1 -- 172.21.15.111:0/2083568586 learned_addr learned my addr 172.21.15.111:0/2083568586 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.834+0000 7fa78bfff700 1 --1- 172.21.15.111:0/2083568586 >> v1:172.21.15.143:6789/0 conn(0x7fa78c154780 0x7fa78c157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36482/0 (socket says 172.21.15.111:36482) 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa790922700 1 -- 172.21.15.111:0/2083568586 >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 msgr2=0x7fa78c15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:25.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa790922700 1 --2- 172.21.15.111:0/2083568586 >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 0x7fa78c15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa790922700 1 -- 172.21.15.111:0/2083568586 >> v1:172.21.15.143:6789/0 conn(0x7fa78c154780 legacy=0x7fa78c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa790922700 1 -- 172.21.15.111:0/2083568586 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c0ac210 con 0x7fa78c0adab0 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa790922700 1 --2- 172.21.15.111:0/2083568586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 0x7fa78c0adef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa784009e60 tx=0x7fa784009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd22e923096822e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa78b7fe700 1 -- 172.21.15.111:0/2083568586 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78400e070 con 0x7fa78c0adab0 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa78b7fe700 1 -- 172.21.15.111:0/2083568586 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78400bb60 con 0x7fa78c0adab0 2023-12-19T08:02:25.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa78b7fe700 1 -- 172.21.15.111:0/2083568586 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784012430 con 0x7fa78c0adab0 2023-12-19T08:02:25.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 -- 172.21.15.111:0/2083568586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 msgr2=0x7fa78c0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 --2- 172.21.15.111:0/2083568586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 0x7fa78c0adef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa784009e60 tx=0x7fa784009780 comp rx=0 tx=0).stop 2023-12-19T08:02:25.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 -- 172.21.15.111:0/2083568586 shutdown_connections 2023-12-19T08:02:25.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 --2- 172.21.15.111:0/2083568586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c0adab0 0x7fa78c0adef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 --2- 172.21.15.111:0/2083568586 >> v2:172.21.15.111:3300/0 conn(0x7fa78c06baa0 0x7fa78c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 -- 172.21.15.111:0/2083568586 >> 172.21.15.111:0/2083568586 conn(0x7fa78c067b50 msgr2=0x7fa78c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:25.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 -- 172.21.15.111:0/2083568586 shutdown_connections 2023-12-19T08:02:25.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.838+0000 7fa791123700 1 -- 172.21.15.111:0/2083568586 wait complete. 2023-12-19T08:02:25.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 Processor -- start 2023-12-19T08:02:25.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 -- start start 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c06baa0 0x7fa78c1405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa78c0adab0 0x7fa78c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c140120 con 0x7fa78c154780 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c13f510 con 0x7fa78c0adab0 2023-12-19T08:02:25.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c13f7f0 con 0x7fa78c06baa0 2023-12-19T08:02:25.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57990/0 (socket says 172.21.15.111:57990) 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 -- 172.21.15.111:0/443128673 learned_addr learned my addr 172.21.15.111:0/443128673 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa78bfff700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c06baa0 0x7fa78c1405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa7837fe700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa78c0adab0 0x7fa78c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 -- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c06baa0 msgr2=0x7fa78c1405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c06baa0 0x7fa78c1405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 -- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa78c0adab0 msgr2=0x7fa78c0ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa78c0adab0 0x7fa78c0ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 -- 172.21.15.111:0/443128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784009a20 con 0x7fa78c154780 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa790922700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa78c154c90 tx=0x7fa784019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784027470 con 0x7fa78c154780 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7840275f0 con 0x7fa78c154780 2023-12-19T08:02:25.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c13fad0 con 0x7fa78c154780 2023-12-19T08:02:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.842+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7840279b0 con 0x7fa78c154780 2023-12-19T08:02:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.846+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c13ddb0 con 0x7fa78c154780 2023-12-19T08:02:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.846+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c1402d0 con 0x7fa78c154780 2023-12-19T08:02:25.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.846+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa784024070 con 0x7fa78c154780 2023-12-19T08:02:25.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.846+0000 7fa788ff9700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 0x7fa76c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:25.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.846+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fa78407f710 con 0x7fa78c154780 2023-12-19T08:02:25.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.850+0000 7fa78bfff700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 0x7fa76c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:25.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.850+0000 7fa78bfff700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 0x7fa76c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa77c007fd0 tx=0x7fa77c007d70 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:25.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:25.850+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa78c1402d0 con 0x7fa78c154780 2023-12-19T08:02:26.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.062+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa78c04be40 con 0x7fa78c154780 2023-12-19T08:02:26.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.638+0000 7fa788ff9700 1 -- 172.21.15.111:0/443128673 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa78404f8a0 con 0x7fa78c154780 2023-12-19T08:02:26.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 msgr2=0x7fa76c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:26.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 0x7fa76c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa77c007fd0 tx=0x7fa77c007d70 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 msgr2=0x7fa78c0aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa78c154c90 tx=0x7fa784019000 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 shutdown_connections 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa76c060670 0x7fa76c062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa78c0adab0 0x7fa78c0ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa78c154780 0x7fa78c0aee90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 --2- 172.21.15.111:0/443128673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa78c06baa0 0x7fa78c1405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 >> 172.21.15.111:0/443128673 conn(0x7fa78c067b50 msgr2=0x7fa78c14bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 shutdown_connections 2023-12-19T08:02:26.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.642+0000 7fa791123700 1 -- 172.21.15.111:0/443128673 wait complete. 2023-12-19T08:02:26.668 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:26.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 Processor -- start 2023-12-19T08:02:26.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 -- start start 2023-12-19T08:02:26.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f388415d320 0x7f388415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 0x7f38840aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840c7190 con 0x7f388406d690 2023-12-19T08:02:26.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c1b50 con 0x7f388415d320 2023-12-19T08:02:26.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388b8f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c1e30 con 0x7f38840c15b0 2023-12-19T08:02:26.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f388415d320 0x7f388415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f388415d320 0x7f388415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58006/0 (socket says 172.21.15.111:58006) 2023-12-19T08:02:26.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 -- 172.21.15.111:0/3581933282 learned_addr learned my addr 172.21.15.111:0/3581933282 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:26.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f3888e8b700 1 --2- 172.21.15.111:0/3581933282 >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 0x7f38840aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 -- 172.21.15.111:0/3581933282 >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 msgr2=0x7f38840aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:26.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 --2- 172.21.15.111:0/3581933282 >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 0x7f38840aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 -- 172.21.15.111:0/3581933282 >> v1:172.21.15.143:6789/0 conn(0x7f388406d690 legacy=0x7f38840680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:26.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 -- 172.21.15.111:0/3581933282 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840c6e20 con 0x7f388415d320 2023-12-19T08:02:26.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f3888e8b700 1 --2- 172.21.15.111:0/3581933282 >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 0x7f38840aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:26.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.846+0000 7f388968c700 1 --2- 172.21.15.111:0/3581933282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388415d320 0x7f388415d6f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f388406f530 tx=0x7f3880009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66966b6aa0b24cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f387bfff700 1 -- 172.21.15.111:0/3581933282 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388000b820 con 0x7f388415d320 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f387bfff700 1 -- 172.21.15.111:0/3581933282 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388000b980 con 0x7f388415d320 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f387bfff700 1 -- 172.21.15.111:0/3581933282 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388000bd20 con 0x7f388415d320 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- 172.21.15.111:0/3581933282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388415d320 msgr2=0x7f388415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 --2- 172.21.15.111:0/3581933282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388415d320 0x7f388415d6f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f388406f530 tx=0x7f3880009250 comp rx=0 tx=0).stop 2023-12-19T08:02:26.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- 172.21.15.111:0/3581933282 shutdown_connections 2023-12-19T08:02:26.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 --2- 172.21.15.111:0/3581933282 >> v2:172.21.15.143:3300/0 conn(0x7f38840c15b0 0x7f38840aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 --2- 172.21.15.111:0/3581933282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388415d320 0x7f388415d6f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- 172.21.15.111:0/3581933282 >> 172.21.15.111:0/3581933282 conn(0x7f38840c8850 msgr2=0x7f38840c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:26.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- 172.21.15.111:0/3581933282 shutdown_connections 2023-12-19T08:02:26.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- 172.21.15.111:0/3581933282 wait complete. 2023-12-19T08:02:26.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 Processor -- start 2023-12-19T08:02:26.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.850+0000 7f388b8f0700 1 -- start start 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f388406d690 0x7f3884171500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f388415d320 0x7f3884157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c4280 con 0x7f38840c15b0 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c33d0 con 0x7f388406d690 2023-12-19T08:02:26.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388b8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c3670 con 0x7f388415d320 2023-12-19T08:02:26.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58008/0 (socket says 172.21.15.111:58008) 2023-12-19T08:02:26.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 -- 172.21.15.111:0/772894074 learned_addr learned my addr 172.21.15.111:0/772894074 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:26.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f388968c700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f388406d690 0x7f3884171500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3889e8d700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f388415d320 0x7f3884157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 -- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f388415d320 msgr2=0x7f3884157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f388415d320 0x7f3884157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 -- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f388406d690 msgr2=0x7f3884171500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f388406d690 0x7f3884171500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 -- 172.21.15.111:0/772894074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800094f0 con 0x7f38840c15b0 2023-12-19T08:02:26.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.854+0000 7f3888e8b700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f387400c830 tx=0x7f3874008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:26.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874011600 con 0x7f38840c15b0 2023-12-19T08:02:26.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874011760 con 0x7f38840c15b0 2023-12-19T08:02:26.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874011b20 con 0x7f38840c15b0 2023-12-19T08:02:26.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840c3950 con 0x7f38840c15b0 2023-12-19T08:02:26.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840c3f10 con 0x7f38840c15b0 2023-12-19T08:02:26.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38840c4430 con 0x7f38840c15b0 2023-12-19T08:02:26.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.858+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3874018360 con 0x7f38840c15b0 2023-12-19T08:02:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.862+0000 7f38797fa700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 0x7f3868062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.862+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f387407cd50 con 0x7f38840c15b0 2023-12-19T08:02:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.862+0000 7f388968c700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 0x7f3868062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:26.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.862+0000 7f388968c700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 0x7f3868062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f388406f530 tx=0x7f3880016000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:26.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:26.870+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38840c4430 con 0x7f38840c15b0 2023-12-19T08:02:27.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.074+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3884071120 con 0x7f38840c15b0 2023-12-19T08:02:27.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.074+0000 7f38797fa700 1 -- 172.21.15.111:0/772894074 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f387404cd30 con 0x7f38840c15b0 2023-12-19T08:02:27.076 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:27.076 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 msgr2=0x7f3868062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 0x7f3868062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f388406f530 tx=0x7f3880016000 comp rx=0 tx=0).stop 2023-12-19T08:02:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 msgr2=0x7f38840c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f387400c830 tx=0x7f3874008040 comp rx=0 tx=0).stop 2023-12-19T08:02:27.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 shutdown_connections 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3868060670 0x7f3868062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f388415d320 0x7f3884157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38840c15b0 0x7f38840c2e90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 --2- 172.21.15.111:0/772894074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f388406d690 0x7f3884171500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 >> 172.21.15.111:0/772894074 conn(0x7f38840c8850 msgr2=0x7f388406c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 shutdown_connections 2023-12-19T08:02:27.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.078+0000 7f388b8f0700 1 -- 172.21.15.111:0/772894074 wait complete. 2023-12-19T08:02:27.082 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:27.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 Processor -- start 2023-12-19T08:02:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 -- start start 2023-12-19T08:02:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faaf4140ae0 0x7faaf40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf406e150 con 0x7faaf4140710 2023-12-19T08:02:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf4068a70 con 0x7faaf4068630 2023-12-19T08:02:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faafb42e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf406d540 con 0x7faaf4140ae0 2023-12-19T08:02:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faaf91ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7faaf4140710 0x7faaf40c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41954/0 (socket says 172.21.15.111:41954) 2023-12-19T08:02:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faaf91ca700 1 -- 172.21.15.111:0/276710439 learned_addr learned my addr 172.21.15.111:0/276710439 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69260347 0 0) 0x7faaf406e150 con 0x7faaf4140710 2023-12-19T08:02:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faaf89c9700 1 --2- 172.21.15.111:0/276710439 >> v2:172.21.15.143:3300/0 conn(0x7faaf4140ae0 0x7faaf40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.258+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae0003570 con 0x7faaf4140710 2023-12-19T08:02:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886872968 0 0) 0x7faae0003570 con 0x7faaf4140710 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 >> v2:172.21.15.143:3300/0 conn(0x7faaf4140ae0 msgr2=0x7faaf40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 --2- 172.21.15.111:0/276710439 >> v2:172.21.15.143:3300/0 conn(0x7faaf4140ae0 0x7faaf40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 >> v1:172.21.15.143:6789/0 conn(0x7faaf4068630 legacy=0x7faaf40ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf406d820 con 0x7faaf4140710 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7faaf0001470 con 0x7faaf4140710 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 >> v1:172.21.15.111:6789/0 conn(0x7faaf4140710 legacy=0x7faaf40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0003a60 0x7faae0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faae0004810 0x7faae0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 0x7faae000fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf406d540 con 0x7faae0005230 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae0010570 con 0x7faae0003a60 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae0010730 con 0x7faae0004810 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 0x7faae000fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf91ca700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0003a60 0x7faae0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf89c9700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faae0004810 0x7faae0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 -- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faae0004810 msgr2=0x7faae0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faae0004810 0x7faae0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 -- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0003a60 msgr2=0x7faae0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0003a60 0x7faae0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 -- 172.21.15.111:0/276710439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0003570 con 0x7faae0005230 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 -- 172.21.15.111:0/276710439 reap_dead start 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaf99cb700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 0x7faae000fe70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faaec002f00 tx=0x7faaec002530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c06358347302e19 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec002840 con 0x7faae0005230 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaec006ba0 con 0x7faae0005230 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faaebfff700 1 -- 172.21.15.111:0/276710439 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec00c680 con 0x7faae0005230 2023-12-19T08:02:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 -- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 msgr2=0x7faae000fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 0x7faae000fe70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faaec002f00 tx=0x7faaec002530 comp rx=0 tx=0).stop 2023-12-19T08:02:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 -- 172.21.15.111:0/276710439 shutdown_connections 2023-12-19T08:02:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 --2- 172.21.15.111:0/276710439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae0005230 0x7faae000fe70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 -- 172.21.15.111:0/276710439 >> 172.21.15.111:0/276710439 conn(0x7faaf40c8850 msgr2=0x7faaf40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:27.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 -- 172.21.15.111:0/276710439 shutdown_connections 2023-12-19T08:02:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.262+0000 7faafb42e700 1 -- 172.21.15.111:0/276710439 wait complete. 2023-12-19T08:02:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 Processor -- start 2023-12-19T08:02:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 -- start start 2023-12-19T08:02:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0005230 0x7faaf40c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 0x7faaf413a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40c2150 con 0x7faaf413ad90 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40c14d0 con 0x7faae0005230 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faafb42e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf40c17b0 con 0x7faaf4139a90 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faaf99cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faaf99cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58032/0 (socket says 172.21.15.111:58032) 2023-12-19T08:02:27.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faaf99cb700 1 -- 172.21.15.111:0/3630100448 learned_addr learned my addr 172.21.15.111:0/3630100448 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faaf89c9700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 0x7faaf413a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.266+0000 7faaf91ca700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0005230 0x7faaf40c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 -- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0005230 msgr2=0x7faaf40c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0005230 0x7faaf40c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 -- 172.21.15.111:0/3630100448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 msgr2=0x7faaf413b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 -- 172.21.15.111:0/3630100448 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf40c1a90 con 0x7faaf4139a90 2023-12-19T08:02:27.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf99cb700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faaf89c9700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 0x7faaf413a850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae400f590 tx=0x7faae4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae400fec0 con 0x7faaf4139a90 2023-12-19T08:02:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4014670 con 0x7faaf4139a90 2023-12-19T08:02:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf40c1d70 con 0x7faaf4139a90 2023-12-19T08:02:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae40149f0 con 0x7faaf4139a90 2023-12-19T08:02:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf40c10c0 con 0x7faaf4139a90 2023-12-19T08:02:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.270+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf40c2290 con 0x7faaf4139a90 2023-12-19T08:02:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.274+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7faae4015410 con 0x7faaf4139a90 2023-12-19T08:02:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.274+0000 7faae97fa700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 0x7faad8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.274+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7faae407d1c0 con 0x7faaf4139a90 2023-12-19T08:02:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.274+0000 7faaf91ca700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 0x7faad8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.278+0000 7faaf91ca700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 0x7faad8062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faaf0001050 tx=0x7faaf0001e70 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.286+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faaf40c2290 con 0x7faaf4139a90 2023-12-19T08:02:27.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.458+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaf40c2290 con 0x7faaf4139a90 2023-12-19T08:02:27.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.458+0000 7faae97fa700 1 -- 172.21.15.111:0/3630100448 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16394 (secure 0 0 0) 0x7faaf40c2290 con 0x7faaf4139a90 2023-12-19T08:02:27.462 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:27.463 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":85,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:24.425191+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 msgr2=0x7faad8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 0x7faad8062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faaf0001050 tx=0x7faaf0001e70 comp rx=0 tx=0).stop 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 msgr2=0x7faaf413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 0x7faaf413a850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae400f590 tx=0x7faae4008040 comp rx=0 tx=0).stop 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 shutdown_connections 2023-12-19T08:02:27.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faad80606a0 0x7faad8062b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaf413ad90 0x7faaf413b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faaf4139a90 0x7faaf413a850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 --2- 172.21.15.111:0/3630100448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae0005230 0x7faaf40c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 >> 172.21.15.111:0/3630100448 conn(0x7faaf40c8850 msgr2=0x7faaf413c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 shutdown_connections 2023-12-19T08:02:27.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.462+0000 7faafb42e700 1 -- 172.21.15.111:0/3630100448 wait complete. 2023-12-19T08:02:27.486 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:02:27.486 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:02:27.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 Processor -- start 2023-12-19T08:02:27.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 -- start start 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc40159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc40158200 0x7fcc40158680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc400c6400 con 0x7fcc400c5fd0 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4006a7b0 con 0x7fcc40158200 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc44a0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400ad550 con 0x7fcc40159160 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc3effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcc400c5fd0 0x7fcc400adcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41970/0 (socket says 172.21.15.111:41970) 2023-12-19T08:02:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.646+0000 7fcc3effd700 1 -- 172.21.15.111:0/3552467135 learned_addr learned my addr 172.21.15.111:0/3552467135 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3ffff700 1 --2- 172.21.15.111:0/3552467135 >> v2:172.21.15.143:3300/0 conn(0x7fcc40158200 0x7fcc40158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3f7fe700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc40159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943780590 0 0) 0x7fcc400c6400 con 0x7fcc400c5fd0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28003570 con 0x7fcc400c5fd0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604454136 0 0) 0x7fcc28003570 con 0x7fcc400c5fd0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 msgr2=0x7fcc40159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc40159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 >> v2:172.21.15.143:3300/0 conn(0x7fcc40158200 msgr2=0x7fcc40158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 --2- 172.21.15.111:0/3552467135 >> v2:172.21.15.143:3300/0 conn(0x7fcc40158200 0x7fcc40158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400accb0 con 0x7fcc400c5fd0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fcc34001490 con 0x7fcc400c5fd0 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 >> v1:172.21.15.111:6789/0 conn(0x7fcc400c5fd0 legacy=0x7fcc400adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28003a60 0x7fcc28003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc28004810 0x7fcc28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4006a7b0 con 0x7fcc28004810 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400ad550 con 0x7fcc28003a60 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2800dc70 con 0x7fcc40159160 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc28004810 0x7fcc28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3f7fe700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28003a60 0x7fcc28003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3ffff700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 -- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 msgr2=0x7fcc280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc40159160 0x7fcc280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 -- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28003a60 msgr2=0x7fcc28003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28003a60 0x7fcc28003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 -- 172.21.15.111:0/3552467135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc28003570 con 0x7fcc28004810 2023-12-19T08:02:27.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.650+0000 7fcc3effd700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc28004810 0x7fcc28004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c390 tx=0x7fcc34007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70240ef0de135ff0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400cc20 con 0x7fcc28004810 2023-12-19T08:02:27.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3400cda0 con 0x7fcc28004810 2023-12-19T08:02:27.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc3e7fc700 1 -- 172.21.15.111:0/3552467135 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34004b40 con 0x7fcc28004810 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 -- 172.21.15.111:0/3552467135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc28004810 msgr2=0x7fcc28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/3552467135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc28004810 0x7fcc28004c80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c390 tx=0x7fcc34007940 comp rx=0 tx=0).stop 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc3ffff700 1 -- 172.21.15.111:0/3552467135 reap_dead start 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 -- 172.21.15.111:0/3552467135 shutdown_connections 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 -- 172.21.15.111:0/3552467135 >> 172.21.15.111:0/3552467135 conn(0x7fcc40067bb0 msgr2=0x7fcc400cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 -- 172.21.15.111:0/3552467135 shutdown_connections 2023-12-19T08:02:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.654+0000 7fcc44a0c700 1 -- 172.21.15.111:0/3552467135 wait complete. 2023-12-19T08:02:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 Processor -- start 2023-12-19T08:02:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 -- start start 2023-12-19T08:02:27.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 0x7fcc400ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc400b0070 0x7fcc401529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400af9b0 con 0x7fcc400c3690 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400b0b80 con 0x7fcc28004810 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc44a0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc400af110 con 0x7fcc400b0070 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 0x7fcc400ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 0x7fcc400ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:54746/0 (socket says 172.21.15.111:54746) 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58050/0 (socket says 172.21.15.111:58050) 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3f7fe700 1 -- 172.21.15.111:0/132451926 learned_addr learned my addr 172.21.15.111:0/132451926 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.658+0000 7fcc3effd700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc400b0070 0x7fcc401529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 -- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc400b0070 msgr2=0x7fcc401529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc400b0070 0x7fcc401529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 -- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 msgr2=0x7fcc400ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 0x7fcc400ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 -- 172.21.15.111:0/132451926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400af3f0 con 0x7fcc400c3690 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc3ffff700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcc4006a380 tx=0x7fcc3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800e040 con 0x7fcc400c3690 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3800f040 con 0x7fcc400c3690 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc400af6d0 con 0x7fcc400c3690 2023-12-19T08:02:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800fd70 con 0x7fcc400c3690 2023-12-19T08:02:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc40152f30 con 0x7fcc400c3690 2023-12-19T08:02:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.662+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc400b0d10 con 0x7fcc400c3690 2023-12-19T08:02:27.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.666+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcc38015640 con 0x7fcc400c3690 2023-12-19T08:02:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.666+0000 7fcc277fe700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 0x7fcc2c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.666+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fcc3807d700 con 0x7fcc400c3690 2023-12-19T08:02:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.666+0000 7fcc3f7fe700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 0x7fcc2c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.666+0000 7fcc3f7fe700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 0x7fcc2c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcc30006ac0 tx=0x7fcc300091a0 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.670+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc3804d6e0 con 0x7fcc400c3690 2023-12-19T08:02:27.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.834+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc4004be40 con 0x7fcc2c060670 2023-12-19T08:02:27.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.854+0000 7fcc277fe700 1 -- 172.21.15.111:0/132451926 <== mgr.6151 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286587 (secure 0 0 0) 0x7fcc4004be40 con 0x7fcc2c060670 2023-12-19T08:02:27.861 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 msgr2=0x7fcc2c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 0x7fcc2c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcc30006ac0 tx=0x7fcc300091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 msgr2=0x7fcc400c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcc4006a380 tx=0x7fcc3800b040 comp rx=0 tx=0).stop 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 shutdown_connections 2023-12-19T08:02:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc2c060670 0x7fcc2c062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc400c3690 0x7fcc400c3b00 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc400b0070 0x7fcc401529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 --2- 172.21.15.111:0/132451926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc28004810 0x7fcc400ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 >> 172.21.15.111:0/132451926 conn(0x7fcc40067bb0 msgr2=0x7fcc400c72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 shutdown_connections 2023-12-19T08:02:27.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:27.862+0000 7fcc44a0c700 1 -- 172.21.15.111:0/132451926 wait complete. 2023-12-19T08:02:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:02:27.884 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:02:26.996186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2424368,"kb_used_data":9200,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752550352,"statfs":{"total":773094113280,"available":770611560448,"internally_reserved":0,"allocated":9420800,"data_stored":5329857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412558"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454885+0000","last_change":"2023-12-19T08:02:23.452136+0000","last_active":"2023-12-19T08:02:23.454885+0000","last_peered":"2023-12-19T08:02:23.454885+0000","last_clean":"2023-12-19T08:02:23.454885+0000","last_became_active":"2023-12-19T08:02:23.451206+0000","last_became_peered":"2023-12-19T08:02:23.451206+0000","last_unstale":"2023-12-19T08:02:23.454885+0000","last_undegraded":"2023-12-19T08:02:23.454885+0000","last_fullsized":"2023-12-19T08:02:23.454885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459877+0000","last_change":"2023-12-19T08:02:23.459088+0000","last_active":"2023-12-19T08:02:23.459877+0000","last_peered":"2023-12-19T08:02:23.459877+0000","last_clean":"2023-12-19T08:02:23.459877+0000","last_became_active":"2023-12-19T08:02:23.458164+0000","last_became_peered":"2023-12-19T08:02:23.458164+0000","last_unstale":"2023-12-19T08:02:23.459877+0000","last_undegraded":"2023-12-19T08:02:23.459877+0000","last_fullsized":"2023-12-19T08:02:23.459877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468433+0000","last_change":"2023-12-19T08:02:23.457387+0000","last_active":"2023-12-19T08:02:23.468433+0000","last_peered":"2023-12-19T08:02:23.468433+0000","last_clean":"2023-12-19T08:02:23.468433+0000","last_became_active":"2023-12-19T08:02:23.456686+0000","last_became_peered":"2023-12-19T08:02:23.456686+0000","last_unstale":"2023-12-19T08:02:23.468433+0000","last_undegraded":"2023-12-19T08:02:23.468433+0000","last_fullsized":"2023-12-19T08:02:23.468433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454711+0000","last_change":"2023-12-19T08:02:23.454711+0000","last_active":"2023-12-19T08:02:23.454711+0000","last_peered":"2023-12-19T08:02:23.454711+0000","last_clean":"2023-12-19T08:02:23.454711+0000","last_became_active":"2023-12-19T08:02:23.454414+0000","last_became_peered":"2023-12-19T08:02:23.454414+0000","last_unstale":"2023-12-19T08:02:23.454711+0000","last_undegraded":"2023-12-19T08:02:23.454711+0000","last_fullsized":"2023-12-19T08:02:23.454711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461224+0000","last_change":"2023-12-19T08:02:23.461224+0000","last_active":"2023-12-19T08:02:23.461224+0000","last_peered":"2023-12-19T08:02:23.461224+0000","last_clean":"2023-12-19T08:02:23.461224+0000","last_became_active":"2023-12-19T08:02:23.460885+0000","last_became_peered":"2023-12-19T08:02:23.460885+0000","last_unstale":"2023-12-19T08:02:23.461224+0000","last_undegraded":"2023-12-19T08:02:23.461224+0000","last_fullsized":"2023-12-19T08:02:23.461224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:47.858020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457714+0000","last_change":"2023-12-19T08:02:23.457714+0000","last_active":"2023-12-19T08:02:23.457714+0000","last_peered":"2023-12-19T08:02:23.457714+0000","last_clean":"2023-12-19T08:02:23.457714+0000","last_became_active":"2023-12-19T08:02:23.457418+0000","last_became_peered":"2023-12-19T08:02:23.457418+0000","last_unstale":"2023-12-19T08:02:23.457714+0000","last_undegraded":"2023-12-19T08:02:23.457714+0000","last_fullsized":"2023-12-19T08:02:23.457714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:19.019055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.450134+0000","last_change":"2023-12-19T08:02:23.450134+0000","last_active":"2023-12-19T08:02:23.450134+0000","last_peered":"2023-12-19T08:02:23.450134+0000","last_clean":"2023-12-19T08:02:23.450134+0000","last_became_active":"2023-12-19T08:02:23.449553+0000","last_became_peered":"2023-12-19T08:02:23.449553+0000","last_unstale":"2023-12-19T08:02:23.450134+0000","last_undegraded":"2023-12-19T08:02:23.450134+0000","last_fullsized":"2023-12-19T08:02:23.450134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:26.223491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459854+0000","last_change":"2023-12-19T08:02:23.459854+0000","last_active":"2023-12-19T08:02:23.459854+0000","last_peered":"2023-12-19T08:02:23.459854+0000","last_clean":"2023-12-19T08:02:23.459854+0000","last_became_active":"2023-12-19T08:02:23.459560+0000","last_became_peered":"2023-12-19T08:02:23.459560+0000","last_unstale":"2023-12-19T08:02:23.459854+0000","last_undegraded":"2023-12-19T08:02:23.459854+0000","last_fullsized":"2023-12-19T08:02:23.459854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468443+0000","last_change":"2023-12-19T08:02:23.462244+0000","last_active":"2023-12-19T08:02:23.468443+0000","last_peered":"2023-12-19T08:02:23.468443+0000","last_clean":"2023-12-19T08:02:23.468443+0000","last_became_active":"2023-12-19T08:02:23.461841+0000","last_became_peered":"2023-12-19T08:02:23.461841+0000","last_unstale":"2023-12-19T08:02:23.468443+0000","last_undegraded":"2023-12-19T08:02:23.468443+0000","last_fullsized":"2023-12-19T08:02:23.468443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468025+0000","last_change":"2023-12-19T08:02:23.461171+0000","last_active":"2023-12-19T08:02:23.468025+0000","last_peered":"2023-12-19T08:02:23.468025+0000","last_clean":"2023-12-19T08:02:23.468025+0000","last_became_active":"2023-12-19T08:02:23.460881+0000","last_became_peered":"2023-12-19T08:02:23.460881+0000","last_unstale":"2023-12-19T08:02:23.468025+0000","last_undegraded":"2023-12-19T08:02:23.468025+0000","last_fullsized":"2023-12-19T08:02:23.468025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459692+0000","last_change":"2023-12-19T08:02:23.459692+0000","last_active":"2023-12-19T08:02:23.459692+0000","last_peered":"2023-12-19T08:02:23.459692+0000","last_clean":"2023-12-19T08:02:23.459692+0000","last_became_active":"2023-12-19T08:02:23.457808+0000","last_became_peered":"2023-12-19T08:02:23.457808+0000","last_unstale":"2023-12-19T08:02:23.459692+0000","last_undegraded":"2023-12-19T08:02:23.459692+0000","last_fullsized":"2023-12-19T08:02:23.459692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.460684+0000","last_change":"2023-12-19T08:02:23.460684+0000","last_active":"2023-12-19T08:02:23.460684+0000","last_peered":"2023-12-19T08:02:23.460684+0000","last_clean":"2023-12-19T08:02:23.460684+0000","last_became_active":"2023-12-19T08:02:23.460367+0000","last_became_peered":"2023-12-19T08:02:23.460367+0000","last_unstale":"2023-12-19T08:02:23.460684+0000","last_undegraded":"2023-12-19T08:02:23.460684+0000","last_fullsized":"2023-12-19T08:02:23.460684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461829+0000","last_change":"2023-12-19T08:02:23.461829+0000","last_active":"2023-12-19T08:02:23.461829+0000","last_peered":"2023-12-19T08:02:23.461829+0000","last_clean":"2023-12-19T08:02:23.461829+0000","last_became_active":"2023-12-19T08:02:23.461558+0000","last_became_peered":"2023-12-19T08:02:23.461558+0000","last_unstale":"2023-12-19T08:02:23.461829+0000","last_undegraded":"2023-12-19T08:02:23.461829+0000","last_fullsized":"2023-12-19T08:02:23.461829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458067+0000","last_change":"2023-12-19T08:02:23.454455+0000","last_active":"2023-12-19T08:02:23.458067+0000","last_peered":"2023-12-19T08:02:23.458067+0000","last_clean":"2023-12-19T08:02:23.458067+0000","last_became_active":"2023-12-19T08:02:23.454127+0000","last_became_peered":"2023-12-19T08:02:23.454127+0000","last_unstale":"2023-12-19T08:02:23.458067+0000","last_undegraded":"2023-12-19T08:02:23.458067+0000","last_fullsized":"2023-12-19T08:02:23.458067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.462635+0000","last_change":"2023-12-19T08:02:23.456889+0000","last_active":"2023-12-19T08:02:23.462635+0000","last_peered":"2023-12-19T08:02:23.462635+0000","last_clean":"2023-12-19T08:02:23.462635+0000","last_became_active":"2023-12-19T08:02:23.455525+0000","last_became_peered":"2023-12-19T08:02:23.455525+0000","last_unstale":"2023-12-19T08:02:23.462635+0000","last_undegraded":"2023-12-19T08:02:23.462635+0000","last_fullsized":"2023-12-19T08:02:23.462635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461414+0000","last_change":"2023-12-19T08:02:23.461414+0000","last_active":"2023-12-19T08:02:23.461414+0000","last_peered":"2023-12-19T08:02:23.461414+0000","last_clean":"2023-12-19T08:02:23.461414+0000","last_became_active":"2023-12-19T08:02:23.461091+0000","last_became_peered":"2023-12-19T08:02:23.461091+0000","last_unstale":"2023-12-19T08:02:23.461414+0000","last_undegraded":"2023-12-19T08:02:23.461414+0000","last_fullsized":"2023-12-19T08:02:23.461414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:23.567372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457788+0000","last_change":"2023-12-19T08:02:23.456210+0000","last_active":"2023-12-19T08:02:23.457788+0000","last_peered":"2023-12-19T08:02:23.457788+0000","last_clean":"2023-12-19T08:02:23.457788+0000","last_became_active":"2023-12-19T08:02:23.455906+0000","last_became_peered":"2023-12-19T08:02:23.455906+0000","last_unstale":"2023-12-19T08:02:23.457788+0000","last_undegraded":"2023-12-19T08:02:23.457788+0000","last_fullsized":"2023-12-19T08:02:23.457788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455930+0000","last_change":"2023-12-19T08:02:23.455930+0000","last_active":"2023-12-19T08:02:23.455930+0000","last_peered":"2023-12-19T08:02:23.455930+0000","last_clean":"2023-12-19T08:02:23.455930+0000","last_became_active":"2023-12-19T08:02:23.455660+0000","last_became_peered":"2023-12-19T08:02:23.455660+0000","last_unstale":"2023-12-19T08:02:23.455930+0000","last_undegraded":"2023-12-19T08:02:23.455930+0000","last_fullsized":"2023-12-19T08:02:23.455930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452042+0000","last_change":"2023-12-19T08:02:23.452042+0000","last_active":"2023-12-19T08:02:23.452042+0000","last_peered":"2023-12-19T08:02:23.452042+0000","last_clean":"2023-12-19T08:02:23.452042+0000","last_became_active":"2023-12-19T08:02:23.450799+0000","last_became_peered":"2023-12-19T08:02:23.450799+0000","last_unstale":"2023-12-19T08:02:23.452042+0000","last_undegraded":"2023-12-19T08:02:23.452042+0000","last_fullsized":"2023-12-19T08:02:23.452042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:31.625501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455639+0000","last_change":"2023-12-19T08:02:23.451442+0000","last_active":"2023-12-19T08:02:23.455639+0000","last_peered":"2023-12-19T08:02:23.455639+0000","last_clean":"2023-12-19T08:02:23.455639+0000","last_became_active":"2023-12-19T08:02:23.450876+0000","last_became_peered":"2023-12-19T08:02:23.450876+0000","last_unstale":"2023-12-19T08:02:23.455639+0000","last_undegraded":"2023-12-19T08:02:23.455639+0000","last_fullsized":"2023-12-19T08:02:23.455639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:28:39.841583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456168+0000","last_change":"2023-12-19T08:02:23.456168+0000","last_active":"2023-12-19T08:02:23.456168+0000","last_peered":"2023-12-19T08:02:23.456168+0000","last_clean":"2023-12-19T08:02:23.456168+0000","last_became_active":"2023-12-19T08:02:23.455881+0000","last_became_peered":"2023-12-19T08:02:23.455881+0000","last_unstale":"2023-12-19T08:02:23.456168+0000","last_undegraded":"2023-12-19T08:02:23.456168+0000","last_fullsized":"2023-12-19T08:02:23.456168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457631+0000","last_change":"2023-12-19T08:02:23.455226+0000","last_active":"2023-12-19T08:02:23.457631+0000","last_peered":"2023-12-19T08:02:23.457631+0000","last_clean":"2023-12-19T08:02:23.457631+0000","last_became_active":"2023-12-19T08:02:23.454878+0000","last_became_peered":"2023-12-19T08:02:23.454878+0000","last_unstale":"2023-12-19T08:02:23.457631+0000","last_undegraded":"2023-12-19T08:02:23.457631+0000","last_fullsized":"2023-12-19T08:02:23.457631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:06.062668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457663+0000","last_change":"2023-12-19T08:02:23.455332+0000","last_active":"2023-12-19T08:02:23.457663+0000","last_peered":"2023-12-19T08:02:23.457663+0000","last_clean":"2023-12-19T08:02:23.457663+0000","last_became_active":"2023-12-19T08:02:23.454961+0000","last_became_peered":"2023-12-19T08:02:23.454961+0000","last_unstale":"2023-12-19T08:02:23.457663+0000","last_undegraded":"2023-12-19T08:02:23.457663+0000","last_fullsized":"2023-12-19T08:02:23.457663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:01.898282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.470428+0000","last_change":"2023-12-19T08:02:23.464930+0000","last_active":"2023-12-19T08:02:23.470428+0000","last_peered":"2023-12-19T08:02:23.470428+0000","last_clean":"2023-12-19T08:02:23.470428+0000","last_became_active":"2023-12-19T08:02:23.464624+0000","last_became_peered":"2023-12-19T08:02:23.464624+0000","last_unstale":"2023-12-19T08:02:23.470428+0000","last_undegraded":"2023-12-19T08:02:23.470428+0000","last_fullsized":"2023-12-19T08:02:23.470428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.465808+0000","last_change":"2023-12-19T08:02:23.465808+0000","last_active":"2023-12-19T08:02:23.465808+0000","last_peered":"2023-12-19T08:02:23.465808+0000","last_clean":"2023-12-19T08:02:23.465808+0000","last_became_active":"2023-12-19T08:02:23.465517+0000","last_became_peered":"2023-12-19T08:02:23.465517+0000","last_unstale":"2023-12-19T08:02:23.465808+0000","last_undegraded":"2023-12-19T08:02:23.465808+0000","last_fullsized":"2023-12-19T08:02:23.465808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:19.019055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.467585+0000","last_change":"2023-12-19T08:02:23.467585+0000","last_active":"2023-12-19T08:02:23.467585+0000","last_peered":"2023-12-19T08:02:23.467585+0000","last_clean":"2023-12-19T08:02:23.467585+0000","last_became_active":"2023-12-19T08:02:23.466929+0000","last_became_peered":"2023-12-19T08:02:23.466929+0000","last_unstale":"2023-12-19T08:02:23.467585+0000","last_undegraded":"2023-12-19T08:02:23.467585+0000","last_fullsized":"2023-12-19T08:02:23.467585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458749+0000","last_change":"2023-12-19T08:02:23.458749+0000","last_active":"2023-12-19T08:02:23.458749+0000","last_peered":"2023-12-19T08:02:23.458749+0000","last_clean":"2023-12-19T08:02:23.458749+0000","last_became_active":"2023-12-19T08:02:23.457376+0000","last_became_peered":"2023-12-19T08:02:23.457376+0000","last_unstale":"2023-12-19T08:02:23.458749+0000","last_undegraded":"2023-12-19T08:02:23.458749+0000","last_fullsized":"2023-12-19T08:02:23.458749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458983+0000","last_change":"2023-12-19T08:02:23.457115+0000","last_active":"2023-12-19T08:02:23.458983+0000","last_peered":"2023-12-19T08:02:23.458983+0000","last_clean":"2023-12-19T08:02:23.458983+0000","last_became_active":"2023-12-19T08:02:23.456788+0000","last_became_peered":"2023-12-19T08:02:23.456788+0000","last_unstale":"2023-12-19T08:02:23.458983+0000","last_undegraded":"2023-12-19T08:02:23.458983+0000","last_fullsized":"2023-12-19T08:02:23.458983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468245+0000","last_change":"2023-12-19T08:02:23.463312+0000","last_active":"2023-12-19T08:02:23.468245+0000","last_peered":"2023-12-19T08:02:23.468245+0000","last_clean":"2023-12-19T08:02:23.468245+0000","last_became_active":"2023-12-19T08:02:23.461048+0000","last_became_peered":"2023-12-19T08:02:23.461048+0000","last_unstale":"2023-12-19T08:02:23.468245+0000","last_undegraded":"2023-12-19T08:02:23.468245+0000","last_fullsized":"2023-12-19T08:02:23.468245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.465256+0000","last_change":"2023-12-19T08:02:23.465256+0000","last_active":"2023-12-19T08:02:23.465256+0000","last_peered":"2023-12-19T08:02:23.465256+0000","last_clean":"2023-12-19T08:02:23.465256+0000","last_became_active":"2023-12-19T08:02:23.464984+0000","last_became_peered":"2023-12-19T08:02:23.464984+0000","last_unstale":"2023-12-19T08:02:23.465256+0000","last_undegraded":"2023-12-19T08:02:23.465256+0000","last_fullsized":"2023-12-19T08:02:23.465256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.450979+0000","last_change":"2023-12-19T08:02:23.450979+0000","last_active":"2023-12-19T08:02:23.450979+0000","last_peered":"2023-12-19T08:02:23.450979+0000","last_clean":"2023-12-19T08:02:23.450979+0000","last_became_active":"2023-12-19T08:02:23.450515+0000","last_became_peered":"2023-12-19T08:02:23.450515+0000","last_unstale":"2023-12-19T08:02:23.450979+0000","last_undegraded":"2023-12-19T08:02:23.450979+0000","last_fullsized":"2023-12-19T08:02:23.450979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.449461+0000","last_change":"2023-12-19T08:02:23.449461+0000","last_active":"2023-12-19T08:02:23.449461+0000","last_peered":"2023-12-19T08:02:23.449461+0000","last_clean":"2023-12-19T08:02:23.449461+0000","last_became_active":"2023-12-19T08:02:23.448800+0000","last_became_peered":"2023-12-19T08:02:23.448800+0000","last_unstale":"2023-12-19T08:02:23.449461+0000","last_undegraded":"2023-12-19T08:02:23.449461+0000","last_fullsized":"2023-12-19T08:02:23.449461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:03.323862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.453241+0000","last_change":"2023-12-19T08:02:23.453241+0000","last_active":"2023-12-19T08:02:23.453241+0000","last_peered":"2023-12-19T08:02:23.453241+0000","last_clean":"2023-12-19T08:02:23.453241+0000","last_became_active":"2023-12-19T08:02:23.452954+0000","last_became_peered":"2023-12-19T08:02:23.452954+0000","last_unstale":"2023-12-19T08:02:23.453241+0000","last_undegraded":"2023-12-19T08:02:23.453241+0000","last_fullsized":"2023-12-19T08:02:23.453241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.623024+0000","last_change":"2023-12-19T08:02:22.610102+0000","last_active":"2023-12-19T08:02:23.623024+0000","last_peered":"2023-12-19T08:02:23.623024+0000","last_clean":"2023-12-19T08:02:23.623024+0000","last_became_active":"2023-12-19T08:02:22.609712+0000","last_became_peered":"2023-12-19T08:02:22.609712+0000","last_unstale":"2023-12-19T08:02:23.623024+0000","last_undegraded":"2023-12-19T08:02:23.623024+0000","last_fullsized":"2023-12-19T08:02:23.623024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:53.974542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461324+0000","last_change":"2023-12-19T08:02:23.461324+0000","last_active":"2023-12-19T08:02:23.461324+0000","last_peered":"2023-12-19T08:02:23.461324+0000","last_clean":"2023-12-19T08:02:23.461324+0000","last_became_active":"2023-12-19T08:02:23.461045+0000","last_became_peered":"2023-12-19T08:02:23.461045+0000","last_unstale":"2023-12-19T08:02:23.461324+0000","last_undegraded":"2023-12-19T08:02:23.461324+0000","last_fullsized":"2023-12-19T08:02:23.461324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.446326+0000","last_change":"2023-12-19T08:02:22.618628+0000","last_active":"2023-12-19T08:02:23.446326+0000","last_peered":"2023-12-19T08:02:23.446326+0000","last_clean":"2023-12-19T08:02:23.446326+0000","last_became_active":"2023-12-19T08:02:22.618329+0000","last_became_peered":"2023-12-19T08:02:22.618329+0000","last_unstale":"2023-12-19T08:02:23.446326+0000","last_undegraded":"2023-12-19T08:02:23.446326+0000","last_fullsized":"2023-12-19T08:02:23.446326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:24.214042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.453986+0000","last_change":"2023-12-19T08:02:23.453986+0000","last_active":"2023-12-19T08:02:23.453986+0000","last_peered":"2023-12-19T08:02:23.453986+0000","last_clean":"2023-12-19T08:02:23.453986+0000","last_became_active":"2023-12-19T08:02:23.453648+0000","last_became_peered":"2023-12-19T08:02:23.453648+0000","last_unstale":"2023-12-19T08:02:23.453986+0000","last_undegraded":"2023-12-19T08:02:23.453986+0000","last_fullsized":"2023-12-19T08:02:23.453986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.620248+0000","last_change":"2023-12-19T08:02:22.617820+0000","last_active":"2023-12-19T08:02:23.620248+0000","last_peered":"2023-12-19T08:02:23.620248+0000","last_clean":"2023-12-19T08:02:23.620248+0000","last_became_active":"2023-12-19T08:02:22.617539+0000","last_became_peered":"2023-12-19T08:02:22.617539+0000","last_unstale":"2023-12-19T08:02:23.620248+0000","last_undegraded":"2023-12-19T08:02:23.620248+0000","last_fullsized":"2023-12-19T08:02:23.620248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:45.405609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.462092+0000","last_change":"2023-12-19T08:02:23.454793+0000","last_active":"2023-12-19T08:02:23.462092+0000","last_peered":"2023-12-19T08:02:23.462092+0000","last_clean":"2023-12-19T08:02:23.462092+0000","last_became_active":"2023-12-19T08:02:23.454469+0000","last_became_peered":"2023-12-19T08:02:23.454469+0000","last_unstale":"2023-12-19T08:02:23.462092+0000","last_undegraded":"2023-12-19T08:02:23.462092+0000","last_fullsized":"2023-12-19T08:02:23.462092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.618755+0000","last_change":"2023-12-19T08:02:22.613357+0000","last_active":"2023-12-19T08:02:23.618755+0000","last_peered":"2023-12-19T08:02:23.618755+0000","last_clean":"2023-12-19T08:02:23.618755+0000","last_became_active":"2023-12-19T08:02:22.613084+0000","last_became_peered":"2023-12-19T08:02:22.613084+0000","last_unstale":"2023-12-19T08:02:23.618755+0000","last_undegraded":"2023-12-19T08:02:23.618755+0000","last_fullsized":"2023-12-19T08:02:23.618755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:40:18.821196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.442025+0000","last_change":"2023-12-19T08:02:23.442025+0000","last_active":"2023-12-19T08:02:23.442025+0000","last_peered":"2023-12-19T08:02:23.442025+0000","last_clean":"2023-12-19T08:02:23.442025+0000","last_became_active":"2023-12-19T08:02:23.441461+0000","last_became_peered":"2023-12-19T08:02:23.441461+0000","last_unstale":"2023-12-19T08:02:23.442025+0000","last_undegraded":"2023-12-19T08:02:23.442025+0000","last_fullsized":"2023-12-19T08:02:23.442025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:57:44.207797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.617535+0000","last_change":"2023-12-19T08:02:22.610175+0000","last_active":"2023-12-19T08:02:23.617535+0000","last_peered":"2023-12-19T08:02:23.617535+0000","last_clean":"2023-12-19T08:02:23.617535+0000","last_became_active":"2023-12-19T08:02:22.609848+0000","last_became_peered":"2023-12-19T08:02:22.609848+0000","last_unstale":"2023-12-19T08:02:23.617535+0000","last_undegraded":"2023-12-19T08:02:23.617535+0000","last_fullsized":"2023-12-19T08:02:23.617535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:46.149837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458288+0000","last_change":"2023-12-19T08:02:23.455486+0000","last_active":"2023-12-19T08:02:23.458288+0000","last_peered":"2023-12-19T08:02:23.458288+0000","last_clean":"2023-12-19T08:02:23.458288+0000","last_became_active":"2023-12-19T08:02:23.455180+0000","last_became_peered":"2023-12-19T08:02:23.455180+0000","last_unstale":"2023-12-19T08:02:23.458288+0000","last_undegraded":"2023-12-19T08:02:23.458288+0000","last_fullsized":"2023-12-19T08:02:23.458288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.616438+0000","last_change":"2023-12-19T08:02:22.611027+0000","last_active":"2023-12-19T08:02:23.616438+0000","last_peered":"2023-12-19T08:02:23.616438+0000","last_clean":"2023-12-19T08:02:23.616438+0000","last_became_active":"2023-12-19T08:02:22.610745+0000","last_became_peered":"2023-12-19T08:02:22.610745+0000","last_unstale":"2023-12-19T08:02:23.616438+0000","last_undegraded":"2023-12-19T08:02:23.616438+0000","last_fullsized":"2023-12-19T08:02:23.616438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:32:05.206647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457263+0000","last_change":"2023-12-19T08:02:23.457263+0000","last_active":"2023-12-19T08:02:23.457263+0000","last_peered":"2023-12-19T08:02:23.457263+0000","last_clean":"2023-12-19T08:02:23.457263+0000","last_became_active":"2023-12-19T08:02:23.456955+0000","last_became_peered":"2023-12-19T08:02:23.456955+0000","last_unstale":"2023-12-19T08:02:23.457263+0000","last_undegraded":"2023-12-19T08:02:23.457263+0000","last_fullsized":"2023-12-19T08:02:23.457263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:40.250504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436531+0000","last_change":"2023-12-19T08:02:22.615364+0000","last_active":"2023-12-19T08:02:23.436531+0000","last_peered":"2023-12-19T08:02:23.436531+0000","last_clean":"2023-12-19T08:02:23.436531+0000","last_became_active":"2023-12-19T08:02:22.615005+0000","last_became_peered":"2023-12-19T08:02:22.615005+0000","last_unstale":"2023-12-19T08:02:23.436531+0000","last_undegraded":"2023-12-19T08:02:23.436531+0000","last_fullsized":"2023-12-19T08:02:23.436531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:42.507228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454313+0000","last_change":"2023-12-19T08:02:23.452573+0000","last_active":"2023-12-19T08:02:23.454313+0000","last_peered":"2023-12-19T08:02:23.454313+0000","last_clean":"2023-12-19T08:02:23.454313+0000","last_became_active":"2023-12-19T08:02:23.452295+0000","last_became_peered":"2023-12-19T08:02:23.452295+0000","last_unstale":"2023-12-19T08:02:23.454313+0000","last_undegraded":"2023-12-19T08:02:23.454313+0000","last_fullsized":"2023-12-19T08:02:23.454313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:44.372844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.628894+0000","last_change":"2023-12-19T08:02:22.621199+0000","last_active":"2023-12-19T08:02:22.628894+0000","last_peered":"2023-12-19T08:02:22.628894+0000","last_clean":"2023-12-19T08:02:22.628894+0000","last_became_active":"2023-12-19T08:02:22.620894+0000","last_became_peered":"2023-12-19T08:02:22.620894+0000","last_unstale":"2023-12-19T08:02:22.628894+0000","last_undegraded":"2023-12-19T08:02:22.628894+0000","last_fullsized":"2023-12-19T08:02:22.628894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:44.389604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.447084+0000","last_change":"2023-12-19T08:02:23.443563+0000","last_active":"2023-12-19T08:02:23.447084+0000","last_peered":"2023-12-19T08:02:23.447084+0000","last_clean":"2023-12-19T08:02:23.447084+0000","last_became_active":"2023-12-19T08:02:23.443262+0000","last_became_peered":"2023-12-19T08:02:23.443262+0000","last_unstale":"2023-12-19T08:02:23.447084+0000","last_undegraded":"2023-12-19T08:02:23.447084+0000","last_fullsized":"2023-12-19T08:02:23.447084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.445672+0000","last_change":"2023-12-19T08:02:22.618977+0000","last_active":"2023-12-19T08:02:23.445672+0000","last_peered":"2023-12-19T08:02:23.445672+0000","last_clean":"2023-12-19T08:02:23.445672+0000","last_became_active":"2023-12-19T08:02:22.618674+0000","last_became_peered":"2023-12-19T08:02:22.618674+0000","last_unstale":"2023-12-19T08:02:23.445672+0000","last_undegraded":"2023-12-19T08:02:23.445672+0000","last_fullsized":"2023-12-19T08:02:23.445672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:49.973599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456468+0000","last_change":"2023-12-19T08:02:23.453623+0000","last_active":"2023-12-19T08:02:23.456468+0000","last_peered":"2023-12-19T08:02:23.456468+0000","last_clean":"2023-12-19T08:02:23.456468+0000","last_became_active":"2023-12-19T08:02:23.453338+0000","last_became_peered":"2023-12-19T08:02:23.453338+0000","last_unstale":"2023-12-19T08:02:23.456468+0000","last_undegraded":"2023-12-19T08:02:23.456468+0000","last_fullsized":"2023-12-19T08:02:23.456468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:22:01.249592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618487+0000","last_change":"2023-12-19T08:02:22.615446+0000","last_active":"2023-12-19T08:02:22.618487+0000","last_peered":"2023-12-19T08:02:22.618487+0000","last_clean":"2023-12-19T08:02:22.618487+0000","last_became_active":"2023-12-19T08:02:22.615148+0000","last_became_peered":"2023-12-19T08:02:22.615148+0000","last_unstale":"2023-12-19T08:02:22.618487+0000","last_undegraded":"2023-12-19T08:02:22.618487+0000","last_fullsized":"2023-12-19T08:02:22.618487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:49.973599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458053+0000","last_change":"2023-12-19T08:02:23.458053+0000","last_active":"2023-12-19T08:02:23.458053+0000","last_peered":"2023-12-19T08:02:23.458053+0000","last_clean":"2023-12-19T08:02:23.458053+0000","last_became_active":"2023-12-19T08:02:23.457772+0000","last_became_peered":"2023-12-19T08:02:23.457772+0000","last_unstale":"2023-12-19T08:02:23.458053+0000","last_undegraded":"2023-12-19T08:02:23.458053+0000","last_fullsized":"2023-12-19T08:02:23.458053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:33:55.460893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627425+0000","last_change":"2023-12-19T08:02:22.616100+0000","last_active":"2023-12-19T08:02:23.627425+0000","last_peered":"2023-12-19T08:02:23.627425+0000","last_clean":"2023-12-19T08:02:23.627425+0000","last_became_active":"2023-12-19T08:02:22.615783+0000","last_became_peered":"2023-12-19T08:02:22.615783+0000","last_unstale":"2023-12-19T08:02:23.627425+0000","last_undegraded":"2023-12-19T08:02:23.627425+0000","last_fullsized":"2023-12-19T08:02:23.627425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:28:46.325113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454261+0000","last_change":"2023-12-19T08:02:23.454261+0000","last_active":"2023-12-19T08:02:23.454261+0000","last_peered":"2023-12-19T08:02:23.454261+0000","last_clean":"2023-12-19T08:02:23.454261+0000","last_became_active":"2023-12-19T08:02:23.453833+0000","last_became_peered":"2023-12-19T08:02:23.453833+0000","last_unstale":"2023-12-19T08:02:23.454261+0000","last_undegraded":"2023-12-19T08:02:23.454261+0000","last_fullsized":"2023-12-19T08:02:23.454261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:47.858020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.624842+0000","last_change":"2023-12-19T08:02:22.617875+0000","last_active":"2023-12-19T08:02:23.624842+0000","last_peered":"2023-12-19T08:02:23.624842+0000","last_clean":"2023-12-19T08:02:23.624842+0000","last_became_active":"2023-12-19T08:02:22.617579+0000","last_became_peered":"2023-12-19T08:02:22.617579+0000","last_unstale":"2023-12-19T08:02:23.624842+0000","last_undegraded":"2023-12-19T08:02:23.624842+0000","last_fullsized":"2023-12-19T08:02:23.624842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:33:54.444879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.463631+0000","last_change":"2023-12-19T08:02:23.463631+0000","last_active":"2023-12-19T08:02:23.463631+0000","last_peered":"2023-12-19T08:02:23.463631+0000","last_clean":"2023-12-19T08:02:23.463631+0000","last_became_active":"2023-12-19T08:02:23.463355+0000","last_became_peered":"2023-12-19T08:02:23.463355+0000","last_unstale":"2023-12-19T08:02:23.463631+0000","last_undegraded":"2023-12-19T08:02:23.463631+0000","last_fullsized":"2023-12-19T08:02:23.463631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:46.421623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627310+0000","last_change":"2023-12-19T08:02:22.620685+0000","last_active":"2023-12-19T08:02:23.627310+0000","last_peered":"2023-12-19T08:02:23.627310+0000","last_clean":"2023-12-19T08:02:23.627310+0000","last_became_active":"2023-12-19T08:02:22.620353+0000","last_became_peered":"2023-12-19T08:02:22.620353+0000","last_unstale":"2023-12-19T08:02:23.627310+0000","last_undegraded":"2023-12-19T08:02:23.627310+0000","last_fullsized":"2023-12-19T08:02:23.627310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:59.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.466084+0000","last_change":"2023-12-19T08:02:23.466084+0000","last_active":"2023-12-19T08:02:23.466084+0000","last_peered":"2023-12-19T08:02:23.466084+0000","last_clean":"2023-12-19T08:02:23.466084+0000","last_became_active":"2023-12-19T08:02:23.465699+0000","last_became_peered":"2023-12-19T08:02:23.465699+0000","last_unstale":"2023-12-19T08:02:23.466084+0000","last_undegraded":"2023-12-19T08:02:23.466084+0000","last_fullsized":"2023-12-19T08:02:23.466084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618546+0000","last_change":"2023-12-19T08:02:22.615110+0000","last_active":"2023-12-19T08:02:22.618546+0000","last_peered":"2023-12-19T08:02:22.618546+0000","last_clean":"2023-12-19T08:02:22.618546+0000","last_became_active":"2023-12-19T08:02:22.614598+0000","last_became_peered":"2023-12-19T08:02:22.614598+0000","last_unstale":"2023-12-19T08:02:22.618546+0000","last_undegraded":"2023-12-19T08:02:22.618546+0000","last_fullsized":"2023-12-19T08:02:22.618546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:45:11.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456440+0000","last_change":"2023-12-19T08:02:23.456440+0000","last_active":"2023-12-19T08:02:23.456440+0000","last_peered":"2023-12-19T08:02:23.456440+0000","last_clean":"2023-12-19T08:02:23.456440+0000","last_became_active":"2023-12-19T08:02:23.453483+0000","last_became_peered":"2023-12-19T08:02:23.453483+0000","last_unstale":"2023-12-19T08:02:23.456440+0000","last_undegraded":"2023-12-19T08:02:23.456440+0000","last_fullsized":"2023-12-19T08:02:23.456440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:49:43.200638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.445655+0000","last_change":"2023-12-19T08:02:22.611908+0000","last_active":"2023-12-19T08:02:23.445655+0000","last_peered":"2023-12-19T08:02:23.445655+0000","last_clean":"2023-12-19T08:02:23.445655+0000","last_became_active":"2023-12-19T08:02:22.611329+0000","last_became_peered":"2023-12-19T08:02:22.611329+0000","last_unstale":"2023-12-19T08:02:23.445655+0000","last_undegraded":"2023-12-19T08:02:23.445655+0000","last_fullsized":"2023-12-19T08:02:23.445655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:35.394367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455703+0000","last_change":"2023-12-19T08:02:23.454566+0000","last_active":"2023-12-19T08:02:23.455703+0000","last_peered":"2023-12-19T08:02:23.455703+0000","last_clean":"2023-12-19T08:02:23.455703+0000","last_became_active":"2023-12-19T08:02:23.454274+0000","last_became_peered":"2023-12-19T08:02:23.454274+0000","last_unstale":"2023-12-19T08:02:23.455703+0000","last_undegraded":"2023-12-19T08:02:23.455703+0000","last_fullsized":"2023-12-19T08:02:23.455703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.628041+0000","last_change":"2023-12-19T08:02:22.615290+0000","last_active":"2023-12-19T08:02:23.628041+0000","last_peered":"2023-12-19T08:02:23.628041+0000","last_clean":"2023-12-19T08:02:23.628041+0000","last_became_active":"2023-12-19T08:02:22.615007+0000","last_became_peered":"2023-12-19T08:02:22.615007+0000","last_unstale":"2023-12-19T08:02:23.628041+0000","last_undegraded":"2023-12-19T08:02:23.628041+0000","last_fullsized":"2023-12-19T08:02:23.628041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.472599+0000","last_change":"2023-12-19T08:02:23.467532+0000","last_active":"2023-12-19T08:02:23.472599+0000","last_peered":"2023-12-19T08:02:23.472599+0000","last_clean":"2023-12-19T08:02:23.472599+0000","last_became_active":"2023-12-19T08:02:23.462282+0000","last_became_peered":"2023-12-19T08:02:23.462282+0000","last_unstale":"2023-12-19T08:02:23.472599+0000","last_undegraded":"2023-12-19T08:02:23.472599+0000","last_fullsized":"2023-12-19T08:02:23.472599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451691+0000","last_change":"2023-12-19T08:02:22.614966+0000","last_active":"2023-12-19T08:02:23.451691+0000","last_peered":"2023-12-19T08:02:23.451691+0000","last_clean":"2023-12-19T08:02:23.451691+0000","last_became_active":"2023-12-19T08:02:22.614510+0000","last_became_peered":"2023-12-19T08:02:22.614510+0000","last_unstale":"2023-12-19T08:02:23.451691+0000","last_undegraded":"2023-12-19T08:02:23.451691+0000","last_fullsized":"2023-12-19T08:02:23.451691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:59.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.469161+0000","last_change":"2023-12-19T08:02:23.469161+0000","last_active":"2023-12-19T08:02:23.469161+0000","last_peered":"2023-12-19T08:02:23.469161+0000","last_clean":"2023-12-19T08:02:23.469161+0000","last_became_active":"2023-12-19T08:02:23.466870+0000","last_became_peered":"2023-12-19T08:02:23.466870+0000","last_unstale":"2023-12-19T08:02:23.469161+0000","last_undegraded":"2023-12-19T08:02:23.469161+0000","last_fullsized":"2023-12-19T08:02:23.469161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:06.813090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.626793+0000","last_change":"2023-12-19T08:02:22.619397+0000","last_active":"2023-12-19T08:02:23.626793+0000","last_peered":"2023-12-19T08:02:23.626793+0000","last_clean":"2023-12-19T08:02:23.626793+0000","last_became_active":"2023-12-19T08:02:22.618886+0000","last_became_peered":"2023-12-19T08:02:22.618886+0000","last_unstale":"2023-12-19T08:02:23.626793+0000","last_undegraded":"2023-12-19T08:02:23.626793+0000","last_fullsized":"2023-12-19T08:02:23.626793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.460303+0000","last_change":"2023-12-19T08:02:23.460303+0000","last_active":"2023-12-19T08:02:23.460303+0000","last_peered":"2023-12-19T08:02:23.460303+0000","last_clean":"2023-12-19T08:02:23.460303+0000","last_became_active":"2023-12-19T08:02:23.459947+0000","last_became_peered":"2023-12-19T08:02:23.459947+0000","last_unstale":"2023-12-19T08:02:23.460303+0000","last_undegraded":"2023-12-19T08:02:23.460303+0000","last_fullsized":"2023-12-19T08:02:23.460303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627304+0000","last_change":"2023-12-19T08:02:22.618977+0000","last_active":"2023-12-19T08:02:23.627304+0000","last_peered":"2023-12-19T08:02:23.627304+0000","last_clean":"2023-12-19T08:02:23.627304+0000","last_became_active":"2023-12-19T08:02:22.618683+0000","last_became_peered":"2023-12-19T08:02:22.618683+0000","last_unstale":"2023-12-19T08:02:23.627304+0000","last_undegraded":"2023-12-19T08:02:23.627304+0000","last_fullsized":"2023-12-19T08:02:23.627304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:02.059302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451854+0000","last_change":"2023-12-19T08:02:23.451854+0000","last_active":"2023-12-19T08:02:23.451854+0000","last_peered":"2023-12-19T08:02:23.451854+0000","last_clean":"2023-12-19T08:02:23.451854+0000","last_became_active":"2023-12-19T08:02:23.451229+0000","last_became_peered":"2023-12-19T08:02:23.451229+0000","last_unstale":"2023-12-19T08:02:23.451854+0000","last_undegraded":"2023-12-19T08:02:23.451854+0000","last_fullsized":"2023-12-19T08:02:23.451854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:15.260396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436319+0000","last_change":"2023-12-19T08:02:22.608641+0000","last_active":"2023-12-19T08:02:23.436319+0000","last_peered":"2023-12-19T08:02:23.436319+0000","last_clean":"2023-12-19T08:02:23.436319+0000","last_became_active":"2023-12-19T08:02:22.608369+0000","last_became_peered":"2023-12-19T08:02:22.608369+0000","last_unstale":"2023-12-19T08:02:23.436319+0000","last_undegraded":"2023-12-19T08:02:23.436319+0000","last_fullsized":"2023-12-19T08:02:23.436319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:53.317303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.449846+0000","last_change":"2023-12-19T08:02:23.449846+0000","last_active":"2023-12-19T08:02:23.449846+0000","last_peered":"2023-12-19T08:02:23.449846+0000","last_clean":"2023-12-19T08:02:23.449846+0000","last_became_active":"2023-12-19T08:02:23.448518+0000","last_became_peered":"2023-12-19T08:02:23.448518+0000","last_unstale":"2023-12-19T08:02:23.449846+0000","last_undegraded":"2023-12-19T08:02:23.449846+0000","last_fullsized":"2023-12-19T08:02:23.449846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.620034+0000","last_change":"2023-12-19T08:02:22.616425+0000","last_active":"2023-12-19T08:02:23.620034+0000","last_peered":"2023-12-19T08:02:23.620034+0000","last_clean":"2023-12-19T08:02:23.620034+0000","last_became_active":"2023-12-19T08:02:22.616111+0000","last_became_peered":"2023-12-19T08:02:22.616111+0000","last_unstale":"2023-12-19T08:02:23.620034+0000","last_undegraded":"2023-12-19T08:02:23.620034+0000","last_fullsized":"2023-12-19T08:02:23.620034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:46.842007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456902+0000","last_change":"2023-12-19T08:02:23.456902+0000","last_active":"2023-12-19T08:02:23.456902+0000","last_peered":"2023-12-19T08:02:23.456902+0000","last_clean":"2023-12-19T08:02:23.456902+0000","last_became_active":"2023-12-19T08:02:23.456540+0000","last_became_peered":"2023-12-19T08:02:23.456540+0000","last_unstale":"2023-12-19T08:02:23.456902+0000","last_undegraded":"2023-12-19T08:02:23.456902+0000","last_fullsized":"2023-12-19T08:02:23.456902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618530+0000","last_change":"2023-12-19T08:02:22.614364+0000","last_active":"2023-12-19T08:02:22.618530+0000","last_peered":"2023-12-19T08:02:22.618530+0000","last_clean":"2023-12-19T08:02:22.618530+0000","last_became_active":"2023-12-19T08:02:22.613813+0000","last_became_peered":"2023-12-19T08:02:22.613813+0000","last_unstale":"2023-12-19T08:02:22.618530+0000","last_undegraded":"2023-12-19T08:02:22.618530+0000","last_fullsized":"2023-12-19T08:02:22.618530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457049+0000","last_change":"2023-12-19T08:02:23.457049+0000","last_active":"2023-12-19T08:02:23.457049+0000","last_peered":"2023-12-19T08:02:23.457049+0000","last_clean":"2023-12-19T08:02:23.457049+0000","last_became_active":"2023-12-19T08:02:23.456690+0000","last_became_peered":"2023-12-19T08:02:23.456690+0000","last_unstale":"2023-12-19T08:02:23.457049+0000","last_undegraded":"2023-12-19T08:02:23.457049+0000","last_fullsized":"2023-12-19T08:02:23.457049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:27.513419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618977+0000","last_change":"2023-12-19T08:02:22.617291+0000","last_active":"2023-12-19T08:02:22.618977+0000","last_peered":"2023-12-19T08:02:22.618977+0000","last_clean":"2023-12-19T08:02:22.618977+0000","last_became_active":"2023-12-19T08:02:22.616986+0000","last_became_peered":"2023-12-19T08:02:22.616986+0000","last_unstale":"2023-12-19T08:02:22.618977+0000","last_undegraded":"2023-12-19T08:02:22.618977+0000","last_fullsized":"2023-12-19T08:02:22.618977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:54.821396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458437+0000","last_change":"2023-12-19T08:02:23.458437+0000","last_active":"2023-12-19T08:02:23.458437+0000","last_peered":"2023-12-19T08:02:23.458437+0000","last_clean":"2023-12-19T08:02:23.458437+0000","last_became_active":"2023-12-19T08:02:23.457776+0000","last_became_peered":"2023-12-19T08:02:23.457776+0000","last_unstale":"2023-12-19T08:02:23.458437+0000","last_undegraded":"2023-12-19T08:02:23.458437+0000","last_fullsized":"2023-12-19T08:02:23.458437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:36.772610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.625030+0000","last_change":"2023-12-19T08:02:22.614778+0000","last_active":"2023-12-19T08:02:23.625030+0000","last_peered":"2023-12-19T08:02:23.625030+0000","last_clean":"2023-12-19T08:02:23.625030+0000","last_became_active":"2023-12-19T08:02:22.614194+0000","last_became_peered":"2023-12-19T08:02:22.614194+0000","last_unstale":"2023-12-19T08:02:23.625030+0000","last_undegraded":"2023-12-19T08:02:23.625030+0000","last_fullsized":"2023-12-19T08:02:23.625030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:26.497406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452176+0000","last_change":"2023-12-19T08:02:23.452176+0000","last_active":"2023-12-19T08:02:23.452176+0000","last_peered":"2023-12-19T08:02:23.452176+0000","last_clean":"2023-12-19T08:02:23.452176+0000","last_became_active":"2023-12-19T08:02:23.451877+0000","last_became_peered":"2023-12-19T08:02:23.451877+0000","last_unstale":"2023-12-19T08:02:23.452176+0000","last_undegraded":"2023-12-19T08:02:23.452176+0000","last_fullsized":"2023-12-19T08:02:23.452176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:50.240450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618573+0000","last_change":"2023-12-19T08:02:22.617222+0000","last_active":"2023-12-19T08:02:22.618573+0000","last_peered":"2023-12-19T08:02:22.618573+0000","last_clean":"2023-12-19T08:02:22.618573+0000","last_became_active":"2023-12-19T08:02:22.616938+0000","last_became_peered":"2023-12-19T08:02:22.616938+0000","last_unstale":"2023-12-19T08:02:22.618573+0000","last_undegraded":"2023-12-19T08:02:22.618573+0000","last_fullsized":"2023-12-19T08:02:22.618573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:35.394367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.467785+0000","last_change":"2023-12-19T08:02:23.467785+0000","last_active":"2023-12-19T08:02:23.467785+0000","last_peered":"2023-12-19T08:02:23.467785+0000","last_clean":"2023-12-19T08:02:23.467785+0000","last_became_active":"2023-12-19T08:02:23.467315+0000","last_became_peered":"2023-12-19T08:02:23.467315+0000","last_unstale":"2023-12-19T08:02:23.467785+0000","last_undegraded":"2023-12-19T08:02:23.467785+0000","last_fullsized":"2023-12-19T08:02:23.467785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.619088+0000","last_change":"2023-12-19T08:02:22.617508+0000","last_active":"2023-12-19T08:02:22.619088+0000","last_peered":"2023-12-19T08:02:22.619088+0000","last_clean":"2023-12-19T08:02:22.619088+0000","last_became_active":"2023-12-19T08:02:22.617162+0000","last_became_peered":"2023-12-19T08:02:22.617162+0000","last_unstale":"2023-12-19T08:02:22.619088+0000","last_undegraded":"2023-12-19T08:02:22.619088+0000","last_fullsized":"2023-12-19T08:02:22.619088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:05.797077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451906+0000","last_change":"2023-12-19T08:02:23.451906+0000","last_active":"2023-12-19T08:02:23.451906+0000","last_peered":"2023-12-19T08:02:23.451906+0000","last_clean":"2023-12-19T08:02:23.451906+0000","last_became_active":"2023-12-19T08:02:23.451602+0000","last_became_peered":"2023-12-19T08:02:23.451602+0000","last_unstale":"2023-12-19T08:02:23.451906+0000","last_undegraded":"2023-12-19T08:02:23.451906+0000","last_fullsized":"2023-12-19T08:02:23.451906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:54.990555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.440840+0000","last_change":"2023-12-19T08:02:22.615931+0000","last_active":"2023-12-19T08:02:23.440840+0000","last_peered":"2023-12-19T08:02:23.440840+0000","last_clean":"2023-12-19T08:02:23.440840+0000","last_became_active":"2023-12-19T08:02:22.615334+0000","last_became_peered":"2023-12-19T08:02:22.615334+0000","last_unstale":"2023-12-19T08:02:23.440840+0000","last_undegraded":"2023-12-19T08:02:23.440840+0000","last_fullsized":"2023-12-19T08:02:23.440840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:39.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452312+0000","last_change":"2023-12-19T08:02:23.452312+0000","last_active":"2023-12-19T08:02:23.452312+0000","last_peered":"2023-12-19T08:02:23.452312+0000","last_clean":"2023-12-19T08:02:23.452312+0000","last_became_active":"2023-12-19T08:02:23.451950+0000","last_became_peered":"2023-12-19T08:02:23.451950+0000","last_unstale":"2023-12-19T08:02:23.452312+0000","last_undegraded":"2023-12-19T08:02:23.452312+0000","last_fullsized":"2023-12-19T08:02:23.452312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:46.421623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.446142+0000","last_change":"2023-12-19T08:02:22.614483+0000","last_active":"2023-12-19T08:02:23.446142+0000","last_peered":"2023-12-19T08:02:23.446142+0000","last_clean":"2023-12-19T08:02:23.446142+0000","last_became_active":"2023-12-19T08:02:22.614210+0000","last_became_peered":"2023-12-19T08:02:22.614210+0000","last_unstale":"2023-12-19T08:02:23.446142+0000","last_undegraded":"2023-12-19T08:02:23.446142+0000","last_fullsized":"2023-12-19T08:02:23.446142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:45:11.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.463998+0000","last_change":"2023-12-19T08:02:23.463998+0000","last_active":"2023-12-19T08:02:23.463998+0000","last_peered":"2023-12-19T08:02:23.463998+0000","last_clean":"2023-12-19T08:02:23.463998+0000","last_became_active":"2023-12-19T08:02:23.463721+0000","last_became_peered":"2023-12-19T08:02:23.463721+0000","last_unstale":"2023-12-19T08:02:23.463998+0000","last_undegraded":"2023-12-19T08:02:23.463998+0000","last_fullsized":"2023-12-19T08:02:23.463998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:06.813090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436205+0000","last_change":"2023-12-19T08:02:22.607009+0000","last_active":"2023-12-19T08:02:23.436205+0000","last_peered":"2023-12-19T08:02:23.436205+0000","last_clean":"2023-12-19T08:02:23.436205+0000","last_became_active":"2023-12-19T08:02:22.605969+0000","last_became_peered":"2023-12-19T08:02:22.605969+0000","last_unstale":"2023-12-19T08:02:23.436205+0000","last_undegraded":"2023-12-19T08:02:23.436205+0000","last_fullsized":"2023-12-19T08:02:23.436205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:29:08.762596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454802+0000","last_change":"2023-12-19T08:02:23.451105+0000","last_active":"2023-12-19T08:02:23.454802+0000","last_peered":"2023-12-19T08:02:23.454802+0000","last_clean":"2023-12-19T08:02:23.454802+0000","last_became_active":"2023-12-19T08:02:23.450756+0000","last_became_peered":"2023-12-19T08:02:23.450756+0000","last_unstale":"2023-12-19T08:02:23.454802+0000","last_undegraded":"2023-12-19T08:02:23.454802+0000","last_fullsized":"2023-12-19T08:02:23.454802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436648+0000","last_change":"2023-12-19T08:02:22.609723+0000","last_active":"2023-12-19T08:02:23.436648+0000","last_peered":"2023-12-19T08:02:23.436648+0000","last_clean":"2023-12-19T08:02:23.436648+0000","last_became_active":"2023-12-19T08:02:22.607744+0000","last_became_peered":"2023-12-19T08:02:22.607744+0000","last_unstale":"2023-12-19T08:02:23.436648+0000","last_undegraded":"2023-12-19T08:02:23.436648+0000","last_fullsized":"2023-12-19T08:02:23.436648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:27:53.075744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452635+0000","last_change":"2023-12-19T08:02:23.452635+0000","last_active":"2023-12-19T08:02:23.452635+0000","last_peered":"2023-12-19T08:02:23.452635+0000","last_clean":"2023-12-19T08:02:23.452635+0000","last_became_active":"2023-12-19T08:02:23.452298+0000","last_became_peered":"2023-12-19T08:02:23.452298+0000","last_unstale":"2023-12-19T08:02:23.452635+0000","last_undegraded":"2023-12-19T08:02:23.452635+0000","last_fullsized":"2023-12-19T08:02:23.452635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:09:25.923471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.440208+0000","last_change":"2023-12-19T08:02:22.617096+0000","last_active":"2023-12-19T08:02:23.440208+0000","last_peered":"2023-12-19T08:02:23.440208+0000","last_clean":"2023-12-19T08:02:23.440208+0000","last_became_active":"2023-12-19T08:02:22.616720+0000","last_became_peered":"2023-12-19T08:02:22.616720+0000","last_unstale":"2023-12-19T08:02:23.440208+0000","last_undegraded":"2023-12-19T08:02:23.440208+0000","last_fullsized":"2023-12-19T08:02:23.440208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:26.497406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451143+0000","last_change":"2023-12-19T08:02:23.451143+0000","last_active":"2023-12-19T08:02:23.451143+0000","last_peered":"2023-12-19T08:02:23.451143+0000","last_clean":"2023-12-19T08:02:23.451143+0000","last_became_active":"2023-12-19T08:02:23.450850+0000","last_became_peered":"2023-12-19T08:02:23.450850+0000","last_unstale":"2023-12-19T08:02:23.451143+0000","last_undegraded":"2023-12-19T08:02:23.451143+0000","last_fullsized":"2023-12-19T08:02:23.451143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:40:19.837209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.435261+0000","last_change":"2023-12-19T08:02:22.612387+0000","last_active":"2023-12-19T08:02:23.435261+0000","last_peered":"2023-12-19T08:02:23.435261+0000","last_clean":"2023-12-19T08:02:23.435261+0000","last_became_active":"2023-12-19T08:02:22.612088+0000","last_became_peered":"2023-12-19T08:02:22.612088+0000","last_unstale":"2023-12-19T08:02:23.435261+0000","last_undegraded":"2023-12-19T08:02:23.435261+0000","last_fullsized":"2023-12-19T08:02:23.435261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:30:27.779394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":373,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.444450+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:02:23.444450+0000","last_peered":"2023-12-19T08:02:23.444450+0000","last_clean":"2023-12-19T08:02:23.444450+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:02:23.444450+0000","last_undegraded":"2023-12-19T08:02:23.444450+0000","last_fullsized":"2023-12-19T08:02:23.444450+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575032,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":1115543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2430000000000001,"15min":1.2410000000000001},"min":{"1min":0.69699999999999995,"5min":0.67600000000000005,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.2649999999999999,"15min":1.256},"min":{"1min":1.0029999999999999,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":1.522,"5min":1.591,"15min":1.845},"last":4.6210000000000004}]},{"osd":1,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.147,"15min":1.147},"min":{"1min":0.81699999999999995,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":1.486,"5min":1.6259999999999999,"15min":1.944},"last":4.9160000000000004},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1499999999999999,"15min":1.153},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.577,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":3.8969999999999998}]},{"osd":2,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.1479999999999999,"15min":1.1639999999999999},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.532,"5min":1.532,"15min":2.375},"last":4.9409999999999998},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.2549999999999999,"15min":1.242},"min":{"1min":0.91000000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.663,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.27,"15min":1.2849999999999999},"min":{"1min":0.95299999999999996,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":1.627,"5min":1.627,"15min":2.0579999999999998},"last":4.8220000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3320000000000001,"15min":1.349},"min":{"1min":0.96899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.593,"5min":1.631,"15min":2.6720000000000002},"last":4.4980000000000002}]},{"osd":4,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.92000000000000004,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":1.736,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":6.2439999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.85899999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.554,"5min":1.671,"15min":1.9490000000000001},"last":6.8360000000000003}]},{"osd":5,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2310000000000001,"15min":1.2410000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.651,"5min":1.653,"15min":2.8279999999999998},"last":4.5679999999999996},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.365,"15min":1.359},"min":{"1min":0.85099999999999998,"5min":0.751,"15min":0.751},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.3010000000000002},"last":2.004}]},{"osd":6,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.383,"15min":1.403},"min":{"1min":1.0900000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.7470000000000001,"5min":2.0070000000000001,"15min":2.9220000000000002},"last":8.4760000000000009},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.363,"15min":1.3879999999999999},"min":{"1min":0.97499999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6879999999999999,"5min":1.867,"15min":3.097},"last":4.9039999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302384,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069456,"statfs":{"total":96636764160,"available":96327122944,"internally_reserved":0,"allocated":1024000,"data_stored":517333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0669999999999999,"15min":1.1040000000000001},"min":{"1min":0.82399999999999995,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.665},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.135,"15min":1.129},"min":{"1min":0.872,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.92800000000000005}]},{"osd":1,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1299999999999999,"15min":1.171},"min":{"1min":0.81999999999999995,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.675,"15min":1.881},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.165,"15min":1.171},"min":{"1min":0.82099999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5349999999999999,"5min":1.542,"15min":1.7989999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.226,"15min":1.23},"min":{"1min":0.83999999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.746,"5min":1.847,"15min":1.847},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.21,"15min":1.2230000000000001},"min":{"1min":0.92100000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.694,"5min":1.694,"15min":1.7929999999999999},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.25,"15min":1.2709999999999999},"min":{"1min":0.81999999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5860000000000001,"5min":1.73,"15min":2.0659999999999998},"last":1.2},{"interface":"front","average":{"1min":1.284,"5min":1.3149999999999999,"15min":1.3169999999999999},"min":{"1min":0.93899999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6120000000000001,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.2809999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1899999999999999,"15min":1.1930000000000001},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6339999999999999,"5min":1.7130000000000001,"15min":2.137},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.403,"15min":1.417},"min":{"1min":0.78300000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.7070000000000001,"5min":1.974,"15min":2.218},"last":0.81000000000000005}]},{"osd":5,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.2969999999999999},"min":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.6899999999999999,"5min":1.9039999999999999,"15min":2.1989999999999998},"last":1.165},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3140000000000001,"15min":1.3169999999999999},"min":{"1min":0.75700000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7250000000000001,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.151}]},{"osd":7,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.375,"15min":1.391},"min":{"1min":0.73599999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.7490000000000001,"5min":1.835,"15min":2.2029999999999998},"last":1.141},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4259999999999999,"15min":1.431},"min":{"1min":0.78300000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.738,"5min":2.137,"15min":2.2639999999999998},"last":1.2390000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298288,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073552,"statfs":{"total":96636764160,"available":96331317248,"internally_reserved":0,"allocated":1024000,"data_stored":517321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.252,"15min":1.2430000000000001},"min":{"1min":0.98199999999999998,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.4750000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.284,"15min":1.278},"min":{"1min":1.111,"5min":0.71499999999999997,"15min":0.50700000000000001},"max":{"1min":1.8129999999999999,"5min":1.954,"15min":2.3140000000000001},"last":1.2909999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.80100000000000005,"5min":0.78200000000000003,"15min":0.61699999999999999},"max":{"1min":1.585,"5min":1.629,"15min":3.5409999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1739999999999999,"15min":1.196},"min":{"1min":0.73899999999999999,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.615,"5min":2.7679999999999998,"15min":3.113},"last":1.361}]},{"osd":2,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.1919999999999999,"15min":1.216},"min":{"1min":0.80700000000000005,"5min":0.74199999999999999,"15min":0.66000000000000003},"max":{"1min":1.546,"5min":1.821,"15min":3.339},"last":1.089},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2070000000000001,"15min":1.224},"min":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.2509999999999999,"15min":1.278},"min":{"1min":0.90100000000000002,"5min":0.80800000000000005,"15min":0.748},"max":{"1min":1.5429999999999999,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.139},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.304,"15min":1.329},"min":{"1min":0.97899999999999998,"5min":0.76600000000000001,"15min":0.58199999999999996},"max":{"1min":1.744,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.256}]},{"osd":4,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.194,"15min":1.2370000000000001},"min":{"1min":0.69199999999999995,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.177},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.25,"15min":1.3100000000000001},"min":{"1min":0.84599999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.637,"5min":2.843,"15min":4.6310000000000002},"last":1.304}]},{"osd":6,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.29,"15min":1.3400000000000001},"min":{"1min":0.90200000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.869,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.389,"5min":1.363,"15min":1.3859999999999999},"min":{"1min":1.0740000000000001,"5min":0.81899999999999995,"15min":0.70599999999999996},"max":{"1min":1.845,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.0429999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.391,"15min":1.444},"min":{"1min":0.84899999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.96,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4570000000000001,"15min":1.4810000000000001},"min":{"1min":1.0920000000000001,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.929,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.2110000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302392,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069448,"statfs":{"total":96636764160,"available":96327114752,"internally_reserved":0,"allocated":1032192,"data_stored":518835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2190000000000001,"15min":1.218},"min":{"1min":0.92100000000000004,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.6000000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.333,"5min":1.3109999999999999,"15min":1.3049999999999999},"min":{"1min":0.94699999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6910000000000001,"5min":1.7909999999999999,"15min":2.738},"last":1.4670000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.77100000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5640000000000001,"5min":1.7589999999999999,"15min":2.589},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.115,"15min":1.1419999999999999},"min":{"1min":0.84599999999999997,"5min":0.60099999999999998,"15min":0.54300000000000004},"max":{"1min":1.528,"5min":1.528,"15min":2.2589999999999999},"last":1.163}]},{"osd":2,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.105,"15min":1.147},"min":{"1min":0.83999999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5409999999999999,"15min":2.4569999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.214,"5min":1.2170000000000001,"15min":1.238},"min":{"1min":0.80100000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.589,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.2170000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.252,"15min":1.294},"min":{"1min":0.85199999999999998,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4770000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3140000000000001,"15min":1.351},"min":{"1min":0.87,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.0620000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1870000000000001,"15min":1.226},"min":{"1min":0.58299999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.526,"5min":1.8149999999999999,"15min":3.556},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1659999999999999,"15min":1.2250000000000001},"min":{"1min":0.64700000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.538,"5min":1.764,"15min":3.2599999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.238,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.748,"5min":1.748,"15min":2.645},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3799999999999999,"15min":1.3939999999999999},"min":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.8069999999999999,"5min":2.0049999999999999,"15min":2.923},"last":1.546}]},{"osd":7,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.323,"15min":1.3779999999999999},"min":{"1min":0.58799999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6930000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.268},{"interface":"front","average":{"1min":1.417,"5min":1.3979999999999999,"15min":1.431},"min":{"1min":1.0700000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.966,"5min":1.966,"15min":2.8460000000000001},"last":1.405}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302380,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069460,"statfs":{"total":96636764160,"available":96327127040,"internally_reserved":0,"allocated":1019904,"data_stored":517299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.84599999999999997,"15min":0.89500000000000002},"min":{"1min":0.58799999999999997,"5min":0.497,"15min":0.45500000000000002},"max":{"1min":1.2070000000000001,"5min":1.617,"15min":5.1079999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.95099999999999996,"15min":0.996},"min":{"1min":0.70199999999999996,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":6.1900000000000004},"last":1.123}]},{"osd":1,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90300000000000002,"15min":0.95599999999999996},"min":{"1min":0.66900000000000004,"5min":0.54500000000000004,"15min":0.45600000000000002},"max":{"1min":1.1539999999999999,"5min":1.431,"15min":4.8879999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.022,"15min":1.1000000000000001},"min":{"1min":0.79100000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.3660000000000001,"5min":1.5129999999999999,"15min":8.0830000000000002},"last":0.49399999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.069,"15min":1.103},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.52800000000000002},"max":{"1min":1.49,"5min":1.49,"15min":4.7519999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1359999999999999,"15min":1.202},"min":{"1min":0.95599999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3140000000000001,"5min":1.8300000000000001,"15min":5.3330000000000002},"last":0.57299999999999995}]},{"osd":4,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2589999999999999,"15min":1.2989999999999999},"min":{"1min":0.82399999999999995,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.5349999999999999,"5min":1.78,"15min":4.7130000000000001},"last":1.23},{"interface":"front","average":{"1min":1.224,"5min":1.222,"15min":1.294},"min":{"1min":0.94299999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.524,"5min":1.6240000000000001,"15min":5.0220000000000002},"last":1.145}]},{"osd":5,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2470000000000001,"15min":1.325},"min":{"1min":1.028,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.627,"5min":1.7090000000000001,"15min":4.9889999999999999},"last":1.409},{"interface":"front","average":{"1min":1.371,"5min":1.333,"15min":1.417},"min":{"1min":1.125,"5min":0.90500000000000003,"15min":0.65800000000000003},"max":{"1min":1.5860000000000001,"5min":1.7350000000000001,"15min":8.5299999999999994},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.379,"15min":1.458},"min":{"1min":1.2130000000000001,"5min":0.88800000000000001,"15min":0.88700000000000001},"max":{"1min":1.7250000000000001,"5min":1.746,"15min":7.2750000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.399,"5min":1.3720000000000001,"15min":1.409},"min":{"1min":1.1100000000000001,"5min":0.82499999999999996,"15min":0.79100000000000004},"max":{"1min":1.7350000000000001,"5min":1.7789999999999999,"15min":4.6550000000000002},"last":1.26}]},{"osd":7,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.462,"15min":1.5669999999999999},"min":{"1min":1.1000000000000001,"5min":1.0629999999999999,"15min":0.84799999999999998},"max":{"1min":1.865,"5min":1.8720000000000001,"15min":16.081},"last":1.621},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4390000000000001,"15min":1.536},"min":{"1min":1.135,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.7190000000000001,"5min":1.833,"15min":15.285},"last":1.5489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302380,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069460,"statfs":{"total":96636764160,"available":96327127040,"internally_reserved":0,"allocated":1019904,"data_stored":517299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89600000000000002,"15min":0.92900000000000005},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.437},"max":{"1min":1.2749999999999999,"5min":1.776,"15min":4.1479999999999997},"last":1.137},{"interface":"front","average":{"1min":1.052,"5min":1.046,"15min":1.0740000000000001},"min":{"1min":0.82199999999999995,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.3220000000000001,"5min":1.8460000000000001,"15min":4.093},"last":0.98199999999999998}]},{"osd":1,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.94599999999999995,"15min":0.97199999999999998},"min":{"1min":0.63700000000000001,"5min":0.59299999999999997,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.5620000000000001,"15min":3.8700000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1,"15min":1.0269999999999999},"min":{"1min":0.65000000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":3.9089999999999998},"last":1.2150000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.97899999999999998,"15min":0.96499999999999997},"min":{"1min":0.68200000000000005,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.343,"5min":1.619,"15min":1.667},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.78800000000000003,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.5449999999999999,"5min":1.629,"15min":1.7250000000000001},"last":0.999}]},{"osd":4,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3320000000000001,"15min":1.345},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.79300000000000004},"max":{"1min":1.639,"5min":1.7669999999999999,"15min":3.9239999999999999},"last":1.26},{"interface":"front","average":{"1min":1.224,"5min":1.2789999999999999,"15min":1.3240000000000001},"min":{"1min":0.97299999999999998,"5min":0.89400000000000002,"15min":0.82099999999999995},"max":{"1min":1.6160000000000001,"5min":1.736,"15min":4.0170000000000003},"last":1.3779999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0920000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.6040000000000001,"5min":1.901,"15min":3.948},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.2709999999999999,"15min":1.2609999999999999},"min":{"1min":1.0589999999999999,"5min":0.81699999999999995,"15min":0.80400000000000005},"max":{"1min":1.5720000000000001,"5min":1.6930000000000001,"15min":1.879},"last":0.95399999999999996}]},{"osd":6,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.302,"15min":1.306},"min":{"1min":1.0660000000000001,"5min":0.92700000000000005,"15min":0.752},"max":{"1min":1.583,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.097},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3620000000000001,"15min":1.403},"min":{"1min":1.0369999999999999,"5min":0.97899999999999998,"15min":0.85399999999999998},"max":{"1min":1.7010000000000001,"5min":1.867,"15min":3.9620000000000002},"last":1.3400000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.369,"15min":1.423},"min":{"1min":1.107,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.607,"5min":1.8109999999999999,"15min":3.9809999999999999},"last":1.298},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.355,"15min":1.3819999999999999},"min":{"1min":1.0029999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.609,"5min":1.8620000000000001,"15min":2.206},"last":1.242}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302408,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069432,"statfs":{"total":96636764160,"available":96327098368,"internally_reserved":0,"allocated":1048576,"data_stored":519571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88300000000000001,"15min":0.85299999999999998},"min":{"1min":0.69799999999999995,"5min":0.60899999999999999,"15min":0.40100000000000002},"max":{"1min":1.534,"5min":1.661,"15min":1.661},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95299999999999996,"15min":0.95099999999999996},"min":{"1min":0.67700000000000005,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":1.4359999999999999,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.97899999999999998}]},{"osd":2,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93600000000000005,"15min":0.92600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4610000000000001,"5min":2.96,"15min":2.96},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0880000000000001,"15min":1.0580000000000001},"min":{"1min":0.91200000000000003,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":1.29,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1879999999999999,"15min":1.157},"min":{"1min":0.82299999999999995,"5min":0.72799999999999998,"15min":0.66100000000000003},"max":{"1min":1.5900000000000001,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.079},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0820000000000001,"15min":1.0780000000000001},"min":{"1min":0.81599999999999995,"5min":0.51200000000000001,"15min":0.46800000000000003},"max":{"1min":1.6020000000000001,"5min":2.407,"15min":2.407},"last":1.212}]},{"osd":4,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.1799999999999999,"15min":1.2},"min":{"1min":0.92800000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.544,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.139},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.341,"15min":1.3140000000000001},"min":{"1min":1.147,"5min":0.95999999999999996,"15min":0.64400000000000002},"max":{"1min":1.6339999999999999,"5min":3.04,"15min":3.04},"last":1.24}]},{"osd":5,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2869999999999999,"15min":1.284},"min":{"1min":0.96299999999999997,"5min":0.87,"15min":0.65100000000000002},"max":{"1min":1.724,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3029999999999999,"15min":1.302},"min":{"1min":0.96499999999999997,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.6919999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.4530000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4019999999999999,"15min":1.3939999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.72799999999999998},"max":{"1min":1.72,"5min":3.089,"15min":3.089},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3700000000000001,"15min":1.375},"min":{"1min":1.079,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.7410000000000001,"5min":2.46,"15min":2.6230000000000002},"last":1.589}]},{"osd":7,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.423,"15min":1.4390000000000001},"min":{"1min":1.1100000000000001,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":1.806,"5min":2.504,"15min":3.6080000000000001},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3779999999999999,"15min":1.413},"min":{"1min":1.0660000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7929999999999999,"5min":2.4590000000000001,"15min":4.101},"last":1.5329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302956,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068884,"statfs":{"total":96636764160,"available":96326537216,"internally_reserved":0,"allocated":1609728,"data_stored":1106656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.61799999999999999,"5min":0.48699999999999999,"15min":0.45800000000000002},"max":{"1min":1.1659999999999999,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91900000000000004,"15min":0.95799999999999996},"min":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.232,"5min":1.3260000000000001,"15min":6.0309999999999997},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.64200000000000002,"15min":0.629},"max":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.5800000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0489999999999999,"15min":1.03},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.504,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.0449999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.96099999999999997,"15min":0.95899999999999996},"min":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.671},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1279999999999999,"15min":1.1319999999999999},"min":{"1min":0.84199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.663,"5min":1.663,"15min":1.6719999999999999},"last":1.181}]},{"osd":4,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3140000000000001,"15min":1.3109999999999999},"min":{"1min":1.0149999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6479999999999999,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.3220000000000001,"15min":1.3140000000000001},"min":{"1min":1.004,"5min":0.85499999999999998,"15min":0.76100000000000001},"max":{"1min":1.7969999999999999,"5min":1.855,"15min":2.141},"last":1.3009999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.369},"min":{"1min":0.98599999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.9099999999999999,"5min":2.085,"15min":2.117},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.365,"15min":1.369},"min":{"1min":1.079,"5min":0.92300000000000004,"15min":0.80300000000000005},"max":{"1min":1.8,"5min":2.125,"15min":2.125},"last":1.2410000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.2929999999999999,"15min":1.274},"min":{"1min":0.93600000000000005,"5min":0.92000000000000004,"15min":0.77400000000000002},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.403,"5min":1.3100000000000001,"15min":1.329},"min":{"1min":1.0820000000000001,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.871,"5min":1.871,"15min":1.9530000000000001},"last":1.1200000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.345,"15min":1.331},"min":{"1min":0.99299999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.788,"5min":1.887,"15min":2.0539999999999998},"last":1.212},{"interface":"front","average":{"1min":1.419,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.0349999999999999,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.903,"5min":2.0059999999999998,"15min":2.016},"last":1.2729999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:02:27.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:02:28.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.090+0000 7fb2d31cc700 1 Processor -- start 2023-12-19T08:02:28.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.090+0000 7fb2d31cc700 1 -- start start 2023-12-19T08:02:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d31cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2cc06a580 0x7fb2cc06aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d31cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc06d210 0x7fb2cc06d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d31cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc06c7c0 con 0x7fb2cc0b1410 2023-12-19T08:02:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d31cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc06af40 con 0x7fb2cc06d210 2023-12-19T08:02:28.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d31cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc06cd80 con 0x7fb2cc06a580 2023-12-19T08:02:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc06d210 0x7fb2cc06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc06d210 0x7fb2cc06d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58062/0 (socket says 172.21.15.111:58062) 2023-12-19T08:02:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 -- 172.21.15.111:0/824947792 learned_addr learned my addr 172.21.15.111:0/824947792 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d21ca700 1 --1- 172.21.15.111:0/824947792 >> v1:172.21.15.111:6789/0 conn(0x7fb2cc0b1410 0x7fb2cc0ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41984/0 (socket says 172.21.15.111:41984) 2023-12-19T08:02:28.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d19c9700 1 --2- 172.21.15.111:0/824947792 >> v2:172.21.15.143:3300/0 conn(0x7fb2cc06a580 0x7fb2cc06aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 -- 172.21.15.111:0/824947792 >> v2:172.21.15.143:3300/0 conn(0x7fb2cc06a580 msgr2=0x7fb2cc06aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 --2- 172.21.15.111:0/824947792 >> v2:172.21.15.143:3300/0 conn(0x7fb2cc06a580 0x7fb2cc06aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 -- 172.21.15.111:0/824947792 >> v1:172.21.15.111:6789/0 conn(0x7fb2cc0b1410 legacy=0x7fb2cc0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d29cb700 1 -- 172.21.15.111:0/824947792 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc06b8f0 con 0x7fb2cc06d210 2023-12-19T08:02:28.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.094+0000 7fb2d11c8700 1 -- 172.21.15.111:0/824947792 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085336629 0 0) 0x7fb2cc06c7c0 con 0x7fb2cc0b1410 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d29cb700 1 --2- 172.21.15.111:0/824947792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc06d620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc068040 tx=0x7fb2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f99de009f2932d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d11c8700 1 -- 172.21.15.111:0/824947792 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4009e70 con 0x7fb2cc06d210 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d11c8700 1 -- 172.21.15.111:0/824947792 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c400b860 con 0x7fb2cc06d210 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d11c8700 1 -- 172.21.15.111:0/824947792 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400bc00 con 0x7fb2cc06d210 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 -- 172.21.15.111:0/824947792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 msgr2=0x7fb2cc06d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/824947792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc06d620 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc068040 tx=0x7fb2c4009250 comp rx=0 tx=0).stop 2023-12-19T08:02:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 -- 172.21.15.111:0/824947792 shutdown_connections 2023-12-19T08:02:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/824947792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc06d620 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/824947792 >> v2:172.21.15.143:3300/0 conn(0x7fb2cc06a580 0x7fb2cc06aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 -- 172.21.15.111:0/824947792 >> 172.21.15.111:0/824947792 conn(0x7fb2cc067bb0 msgr2=0x7fb2cc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 -- 172.21.15.111:0/824947792 shutdown_connections 2023-12-19T08:02:28.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.098+0000 7fb2d31cc700 1 -- 172.21.15.111:0/824947792 wait complete. 2023-12-19T08:02:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 Processor -- start 2023-12-19T08:02:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 -- start start 2023-12-19T08:02:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2cc06a580 0x7fb2cc0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2cc0b1410 0x7fb2cc16c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0c5290 con 0x7fb2cc06d210 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc169b20 con 0x7fb2cc06a580 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d31cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc15bfe0 con 0x7fb2cc0b1410 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d19c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d19c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58076/0 (socket says 172.21.15.111:58076) 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d19c9700 1 -- 172.21.15.111:0/1266514446 learned_addr learned my addr 172.21.15.111:0/1266514446 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d21ca700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2cc06a580 0x7fb2cc0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.102+0000 7fb2d29cb700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2cc0b1410 0x7fb2cc16c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 -- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2cc0b1410 msgr2=0x7fb2cc16c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2cc0b1410 0x7fb2cc16c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 -- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2cc06a580 msgr2=0x7fb2cc0c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2cc06a580 0x7fb2cc0c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 -- 172.21.15.111:0/1266514446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40094f0 con 0x7fb2cc06d210 2023-12-19T08:02:28.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d19c9700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00ec80 tx=0x7fb2bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00c850 con 0x7fb2cc06d210 2023-12-19T08:02:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc00c9b0 con 0x7fb2cc06d210 2023-12-19T08:02:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00cd50 con 0x7fb2cc06d210 2023-12-19T08:02:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc15bc90 con 0x7fb2cc06d210 2023-12-19T08:02:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0c7bc0 con 0x7fb2cc06d210 2023-12-19T08:02:28.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.106+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc15c150 con 0x7fb2cc06d210 2023-12-19T08:02:28.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.110+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb2bc013680 con 0x7fb2cc06d210 2023-12-19T08:02:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.110+0000 7fb2c27fc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 0x7fb2b0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.110+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fb2bc07cf20 con 0x7fb2cc06d210 2023-12-19T08:02:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.110+0000 7fb2d21ca700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 0x7fb2b0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.114+0000 7fb2d21ca700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 0x7fb2b0062b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc156b40 tx=0x7fb2c8001a30 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.118+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb2bc04d010 con 0x7fb2cc06d210 2023-12-19T08:02:28.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.281+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2cc04be40 con 0x7fb2b00606a0 2023-12-19T08:02:28.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.301+0000 7fb2c27fc700 1 -- 172.21.15.111:0/1266514446 <== mgr.6151 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286587 (secure 0 0 0) 0x7fb2cc04be40 con 0x7fb2b00606a0 2023-12-19T08:02:28.306 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 msgr2=0x7fb2b0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 0x7fb2b0062b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc156b40 tx=0x7fb2c8001a30 comp rx=0 tx=0).stop 2023-12-19T08:02:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 msgr2=0x7fb2cc0c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00ec80 tx=0x7fb2bc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 shutdown_connections 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb2b00606a0 0x7fb2b0062b50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2cc0b1410 0x7fb2cc16c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc06d210 0x7fb2cc0c4d50 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 --2- 172.21.15.111:0/1266514446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2cc06a580 0x7fb2cc0c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 >> 172.21.15.111:0/1266514446 conn(0x7fb2cc067bb0 msgr2=0x7fb2cc0caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 shutdown_connections 2023-12-19T08:02:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.309+0000 7fb2d31cc700 1 -- 172.21.15.111:0/1266514446 wait complete. 2023-12-19T08:02:28.312 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:02:28.331 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:02:26.996186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2424368,"kb_used_data":9200,"kb_used_omap":0,"kb_used_meta":2415104,"kb_avail":752550352,"statfs":{"total":773094113280,"available":770611560448,"internally_reserved":0,"allocated":9420800,"data_stored":5329857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2473066496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412558"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454885+0000","last_change":"2023-12-19T08:02:23.452136+0000","last_active":"2023-12-19T08:02:23.454885+0000","last_peered":"2023-12-19T08:02:23.454885+0000","last_clean":"2023-12-19T08:02:23.454885+0000","last_became_active":"2023-12-19T08:02:23.451206+0000","last_became_peered":"2023-12-19T08:02:23.451206+0000","last_unstale":"2023-12-19T08:02:23.454885+0000","last_undegraded":"2023-12-19T08:02:23.454885+0000","last_fullsized":"2023-12-19T08:02:23.454885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459877+0000","last_change":"2023-12-19T08:02:23.459088+0000","last_active":"2023-12-19T08:02:23.459877+0000","last_peered":"2023-12-19T08:02:23.459877+0000","last_clean":"2023-12-19T08:02:23.459877+0000","last_became_active":"2023-12-19T08:02:23.458164+0000","last_became_peered":"2023-12-19T08:02:23.458164+0000","last_unstale":"2023-12-19T08:02:23.459877+0000","last_undegraded":"2023-12-19T08:02:23.459877+0000","last_fullsized":"2023-12-19T08:02:23.459877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468433+0000","last_change":"2023-12-19T08:02:23.457387+0000","last_active":"2023-12-19T08:02:23.468433+0000","last_peered":"2023-12-19T08:02:23.468433+0000","last_clean":"2023-12-19T08:02:23.468433+0000","last_became_active":"2023-12-19T08:02:23.456686+0000","last_became_peered":"2023-12-19T08:02:23.456686+0000","last_unstale":"2023-12-19T08:02:23.468433+0000","last_undegraded":"2023-12-19T08:02:23.468433+0000","last_fullsized":"2023-12-19T08:02:23.468433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454711+0000","last_change":"2023-12-19T08:02:23.454711+0000","last_active":"2023-12-19T08:02:23.454711+0000","last_peered":"2023-12-19T08:02:23.454711+0000","last_clean":"2023-12-19T08:02:23.454711+0000","last_became_active":"2023-12-19T08:02:23.454414+0000","last_became_peered":"2023-12-19T08:02:23.454414+0000","last_unstale":"2023-12-19T08:02:23.454711+0000","last_undegraded":"2023-12-19T08:02:23.454711+0000","last_fullsized":"2023-12-19T08:02:23.454711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461224+0000","last_change":"2023-12-19T08:02:23.461224+0000","last_active":"2023-12-19T08:02:23.461224+0000","last_peered":"2023-12-19T08:02:23.461224+0000","last_clean":"2023-12-19T08:02:23.461224+0000","last_became_active":"2023-12-19T08:02:23.460885+0000","last_became_peered":"2023-12-19T08:02:23.460885+0000","last_unstale":"2023-12-19T08:02:23.461224+0000","last_undegraded":"2023-12-19T08:02:23.461224+0000","last_fullsized":"2023-12-19T08:02:23.461224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:47.858020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457714+0000","last_change":"2023-12-19T08:02:23.457714+0000","last_active":"2023-12-19T08:02:23.457714+0000","last_peered":"2023-12-19T08:02:23.457714+0000","last_clean":"2023-12-19T08:02:23.457714+0000","last_became_active":"2023-12-19T08:02:23.457418+0000","last_became_peered":"2023-12-19T08:02:23.457418+0000","last_unstale":"2023-12-19T08:02:23.457714+0000","last_undegraded":"2023-12-19T08:02:23.457714+0000","last_fullsized":"2023-12-19T08:02:23.457714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:19.019055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.450134+0000","last_change":"2023-12-19T08:02:23.450134+0000","last_active":"2023-12-19T08:02:23.450134+0000","last_peered":"2023-12-19T08:02:23.450134+0000","last_clean":"2023-12-19T08:02:23.450134+0000","last_became_active":"2023-12-19T08:02:23.449553+0000","last_became_peered":"2023-12-19T08:02:23.449553+0000","last_unstale":"2023-12-19T08:02:23.450134+0000","last_undegraded":"2023-12-19T08:02:23.450134+0000","last_fullsized":"2023-12-19T08:02:23.450134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:26.223491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459854+0000","last_change":"2023-12-19T08:02:23.459854+0000","last_active":"2023-12-19T08:02:23.459854+0000","last_peered":"2023-12-19T08:02:23.459854+0000","last_clean":"2023-12-19T08:02:23.459854+0000","last_became_active":"2023-12-19T08:02:23.459560+0000","last_became_peered":"2023-12-19T08:02:23.459560+0000","last_unstale":"2023-12-19T08:02:23.459854+0000","last_undegraded":"2023-12-19T08:02:23.459854+0000","last_fullsized":"2023-12-19T08:02:23.459854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468443+0000","last_change":"2023-12-19T08:02:23.462244+0000","last_active":"2023-12-19T08:02:23.468443+0000","last_peered":"2023-12-19T08:02:23.468443+0000","last_clean":"2023-12-19T08:02:23.468443+0000","last_became_active":"2023-12-19T08:02:23.461841+0000","last_became_peered":"2023-12-19T08:02:23.461841+0000","last_unstale":"2023-12-19T08:02:23.468443+0000","last_undegraded":"2023-12-19T08:02:23.468443+0000","last_fullsized":"2023-12-19T08:02:23.468443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468025+0000","last_change":"2023-12-19T08:02:23.461171+0000","last_active":"2023-12-19T08:02:23.468025+0000","last_peered":"2023-12-19T08:02:23.468025+0000","last_clean":"2023-12-19T08:02:23.468025+0000","last_became_active":"2023-12-19T08:02:23.460881+0000","last_became_peered":"2023-12-19T08:02:23.460881+0000","last_unstale":"2023-12-19T08:02:23.468025+0000","last_undegraded":"2023-12-19T08:02:23.468025+0000","last_fullsized":"2023-12-19T08:02:23.468025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.459692+0000","last_change":"2023-12-19T08:02:23.459692+0000","last_active":"2023-12-19T08:02:23.459692+0000","last_peered":"2023-12-19T08:02:23.459692+0000","last_clean":"2023-12-19T08:02:23.459692+0000","last_became_active":"2023-12-19T08:02:23.457808+0000","last_became_peered":"2023-12-19T08:02:23.457808+0000","last_unstale":"2023-12-19T08:02:23.459692+0000","last_undegraded":"2023-12-19T08:02:23.459692+0000","last_fullsized":"2023-12-19T08:02:23.459692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.460684+0000","last_change":"2023-12-19T08:02:23.460684+0000","last_active":"2023-12-19T08:02:23.460684+0000","last_peered":"2023-12-19T08:02:23.460684+0000","last_clean":"2023-12-19T08:02:23.460684+0000","last_became_active":"2023-12-19T08:02:23.460367+0000","last_became_peered":"2023-12-19T08:02:23.460367+0000","last_unstale":"2023-12-19T08:02:23.460684+0000","last_undegraded":"2023-12-19T08:02:23.460684+0000","last_fullsized":"2023-12-19T08:02:23.460684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461829+0000","last_change":"2023-12-19T08:02:23.461829+0000","last_active":"2023-12-19T08:02:23.461829+0000","last_peered":"2023-12-19T08:02:23.461829+0000","last_clean":"2023-12-19T08:02:23.461829+0000","last_became_active":"2023-12-19T08:02:23.461558+0000","last_became_peered":"2023-12-19T08:02:23.461558+0000","last_unstale":"2023-12-19T08:02:23.461829+0000","last_undegraded":"2023-12-19T08:02:23.461829+0000","last_fullsized":"2023-12-19T08:02:23.461829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458067+0000","last_change":"2023-12-19T08:02:23.454455+0000","last_active":"2023-12-19T08:02:23.458067+0000","last_peered":"2023-12-19T08:02:23.458067+0000","last_clean":"2023-12-19T08:02:23.458067+0000","last_became_active":"2023-12-19T08:02:23.454127+0000","last_became_peered":"2023-12-19T08:02:23.454127+0000","last_unstale":"2023-12-19T08:02:23.458067+0000","last_undegraded":"2023-12-19T08:02:23.458067+0000","last_fullsized":"2023-12-19T08:02:23.458067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.462635+0000","last_change":"2023-12-19T08:02:23.456889+0000","last_active":"2023-12-19T08:02:23.462635+0000","last_peered":"2023-12-19T08:02:23.462635+0000","last_clean":"2023-12-19T08:02:23.462635+0000","last_became_active":"2023-12-19T08:02:23.455525+0000","last_became_peered":"2023-12-19T08:02:23.455525+0000","last_unstale":"2023-12-19T08:02:23.462635+0000","last_undegraded":"2023-12-19T08:02:23.462635+0000","last_fullsized":"2023-12-19T08:02:23.462635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461414+0000","last_change":"2023-12-19T08:02:23.461414+0000","last_active":"2023-12-19T08:02:23.461414+0000","last_peered":"2023-12-19T08:02:23.461414+0000","last_clean":"2023-12-19T08:02:23.461414+0000","last_became_active":"2023-12-19T08:02:23.461091+0000","last_became_peered":"2023-12-19T08:02:23.461091+0000","last_unstale":"2023-12-19T08:02:23.461414+0000","last_undegraded":"2023-12-19T08:02:23.461414+0000","last_fullsized":"2023-12-19T08:02:23.461414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:23.567372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457788+0000","last_change":"2023-12-19T08:02:23.456210+0000","last_active":"2023-12-19T08:02:23.457788+0000","last_peered":"2023-12-19T08:02:23.457788+0000","last_clean":"2023-12-19T08:02:23.457788+0000","last_became_active":"2023-12-19T08:02:23.455906+0000","last_became_peered":"2023-12-19T08:02:23.455906+0000","last_unstale":"2023-12-19T08:02:23.457788+0000","last_undegraded":"2023-12-19T08:02:23.457788+0000","last_fullsized":"2023-12-19T08:02:23.457788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455930+0000","last_change":"2023-12-19T08:02:23.455930+0000","last_active":"2023-12-19T08:02:23.455930+0000","last_peered":"2023-12-19T08:02:23.455930+0000","last_clean":"2023-12-19T08:02:23.455930+0000","last_became_active":"2023-12-19T08:02:23.455660+0000","last_became_peered":"2023-12-19T08:02:23.455660+0000","last_unstale":"2023-12-19T08:02:23.455930+0000","last_undegraded":"2023-12-19T08:02:23.455930+0000","last_fullsized":"2023-12-19T08:02:23.455930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452042+0000","last_change":"2023-12-19T08:02:23.452042+0000","last_active":"2023-12-19T08:02:23.452042+0000","last_peered":"2023-12-19T08:02:23.452042+0000","last_clean":"2023-12-19T08:02:23.452042+0000","last_became_active":"2023-12-19T08:02:23.450799+0000","last_became_peered":"2023-12-19T08:02:23.450799+0000","last_unstale":"2023-12-19T08:02:23.452042+0000","last_undegraded":"2023-12-19T08:02:23.452042+0000","last_fullsized":"2023-12-19T08:02:23.452042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:31.625501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455639+0000","last_change":"2023-12-19T08:02:23.451442+0000","last_active":"2023-12-19T08:02:23.455639+0000","last_peered":"2023-12-19T08:02:23.455639+0000","last_clean":"2023-12-19T08:02:23.455639+0000","last_became_active":"2023-12-19T08:02:23.450876+0000","last_became_peered":"2023-12-19T08:02:23.450876+0000","last_unstale":"2023-12-19T08:02:23.455639+0000","last_undegraded":"2023-12-19T08:02:23.455639+0000","last_fullsized":"2023-12-19T08:02:23.455639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:28:39.841583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456168+0000","last_change":"2023-12-19T08:02:23.456168+0000","last_active":"2023-12-19T08:02:23.456168+0000","last_peered":"2023-12-19T08:02:23.456168+0000","last_clean":"2023-12-19T08:02:23.456168+0000","last_became_active":"2023-12-19T08:02:23.455881+0000","last_became_peered":"2023-12-19T08:02:23.455881+0000","last_unstale":"2023-12-19T08:02:23.456168+0000","last_undegraded":"2023-12-19T08:02:23.456168+0000","last_fullsized":"2023-12-19T08:02:23.456168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457631+0000","last_change":"2023-12-19T08:02:23.455226+0000","last_active":"2023-12-19T08:02:23.457631+0000","last_peered":"2023-12-19T08:02:23.457631+0000","last_clean":"2023-12-19T08:02:23.457631+0000","last_became_active":"2023-12-19T08:02:23.454878+0000","last_became_peered":"2023-12-19T08:02:23.454878+0000","last_unstale":"2023-12-19T08:02:23.457631+0000","last_undegraded":"2023-12-19T08:02:23.457631+0000","last_fullsized":"2023-12-19T08:02:23.457631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:06.062668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457663+0000","last_change":"2023-12-19T08:02:23.455332+0000","last_active":"2023-12-19T08:02:23.457663+0000","last_peered":"2023-12-19T08:02:23.457663+0000","last_clean":"2023-12-19T08:02:23.457663+0000","last_became_active":"2023-12-19T08:02:23.454961+0000","last_became_peered":"2023-12-19T08:02:23.454961+0000","last_unstale":"2023-12-19T08:02:23.457663+0000","last_undegraded":"2023-12-19T08:02:23.457663+0000","last_fullsized":"2023-12-19T08:02:23.457663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:01.898282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.470428+0000","last_change":"2023-12-19T08:02:23.464930+0000","last_active":"2023-12-19T08:02:23.470428+0000","last_peered":"2023-12-19T08:02:23.470428+0000","last_clean":"2023-12-19T08:02:23.470428+0000","last_became_active":"2023-12-19T08:02:23.464624+0000","last_became_peered":"2023-12-19T08:02:23.464624+0000","last_unstale":"2023-12-19T08:02:23.470428+0000","last_undegraded":"2023-12-19T08:02:23.470428+0000","last_fullsized":"2023-12-19T08:02:23.470428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.465808+0000","last_change":"2023-12-19T08:02:23.465808+0000","last_active":"2023-12-19T08:02:23.465808+0000","last_peered":"2023-12-19T08:02:23.465808+0000","last_clean":"2023-12-19T08:02:23.465808+0000","last_became_active":"2023-12-19T08:02:23.465517+0000","last_became_peered":"2023-12-19T08:02:23.465517+0000","last_unstale":"2023-12-19T08:02:23.465808+0000","last_undegraded":"2023-12-19T08:02:23.465808+0000","last_fullsized":"2023-12-19T08:02:23.465808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:19.019055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.467585+0000","last_change":"2023-12-19T08:02:23.467585+0000","last_active":"2023-12-19T08:02:23.467585+0000","last_peered":"2023-12-19T08:02:23.467585+0000","last_clean":"2023-12-19T08:02:23.467585+0000","last_became_active":"2023-12-19T08:02:23.466929+0000","last_became_peered":"2023-12-19T08:02:23.466929+0000","last_unstale":"2023-12-19T08:02:23.467585+0000","last_undegraded":"2023-12-19T08:02:23.467585+0000","last_fullsized":"2023-12-19T08:02:23.467585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458749+0000","last_change":"2023-12-19T08:02:23.458749+0000","last_active":"2023-12-19T08:02:23.458749+0000","last_peered":"2023-12-19T08:02:23.458749+0000","last_clean":"2023-12-19T08:02:23.458749+0000","last_became_active":"2023-12-19T08:02:23.457376+0000","last_became_peered":"2023-12-19T08:02:23.457376+0000","last_unstale":"2023-12-19T08:02:23.458749+0000","last_undegraded":"2023-12-19T08:02:23.458749+0000","last_fullsized":"2023-12-19T08:02:23.458749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458983+0000","last_change":"2023-12-19T08:02:23.457115+0000","last_active":"2023-12-19T08:02:23.458983+0000","last_peered":"2023-12-19T08:02:23.458983+0000","last_clean":"2023-12-19T08:02:23.458983+0000","last_became_active":"2023-12-19T08:02:23.456788+0000","last_became_peered":"2023-12-19T08:02:23.456788+0000","last_unstale":"2023-12-19T08:02:23.458983+0000","last_undegraded":"2023-12-19T08:02:23.458983+0000","last_fullsized":"2023-12-19T08:02:23.458983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.468245+0000","last_change":"2023-12-19T08:02:23.463312+0000","last_active":"2023-12-19T08:02:23.468245+0000","last_peered":"2023-12-19T08:02:23.468245+0000","last_clean":"2023-12-19T08:02:23.468245+0000","last_became_active":"2023-12-19T08:02:23.461048+0000","last_became_peered":"2023-12-19T08:02:23.461048+0000","last_unstale":"2023-12-19T08:02:23.468245+0000","last_undegraded":"2023-12-19T08:02:23.468245+0000","last_fullsized":"2023-12-19T08:02:23.468245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.465256+0000","last_change":"2023-12-19T08:02:23.465256+0000","last_active":"2023-12-19T08:02:23.465256+0000","last_peered":"2023-12-19T08:02:23.465256+0000","last_clean":"2023-12-19T08:02:23.465256+0000","last_became_active":"2023-12-19T08:02:23.464984+0000","last_became_peered":"2023-12-19T08:02:23.464984+0000","last_unstale":"2023-12-19T08:02:23.465256+0000","last_undegraded":"2023-12-19T08:02:23.465256+0000","last_fullsized":"2023-12-19T08:02:23.465256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.450979+0000","last_change":"2023-12-19T08:02:23.450979+0000","last_active":"2023-12-19T08:02:23.450979+0000","last_peered":"2023-12-19T08:02:23.450979+0000","last_clean":"2023-12-19T08:02:23.450979+0000","last_became_active":"2023-12-19T08:02:23.450515+0000","last_became_peered":"2023-12-19T08:02:23.450515+0000","last_unstale":"2023-12-19T08:02:23.450979+0000","last_undegraded":"2023-12-19T08:02:23.450979+0000","last_fullsized":"2023-12-19T08:02:23.450979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.449461+0000","last_change":"2023-12-19T08:02:23.449461+0000","last_active":"2023-12-19T08:02:23.449461+0000","last_peered":"2023-12-19T08:02:23.449461+0000","last_clean":"2023-12-19T08:02:23.449461+0000","last_became_active":"2023-12-19T08:02:23.448800+0000","last_became_peered":"2023-12-19T08:02:23.448800+0000","last_unstale":"2023-12-19T08:02:23.449461+0000","last_undegraded":"2023-12-19T08:02:23.449461+0000","last_fullsized":"2023-12-19T08:02:23.449461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:03.323862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.453241+0000","last_change":"2023-12-19T08:02:23.453241+0000","last_active":"2023-12-19T08:02:23.453241+0000","last_peered":"2023-12-19T08:02:23.453241+0000","last_clean":"2023-12-19T08:02:23.453241+0000","last_became_active":"2023-12-19T08:02:23.452954+0000","last_became_peered":"2023-12-19T08:02:23.452954+0000","last_unstale":"2023-12-19T08:02:23.453241+0000","last_undegraded":"2023-12-19T08:02:23.453241+0000","last_fullsized":"2023-12-19T08:02:23.453241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:55.837409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.623024+0000","last_change":"2023-12-19T08:02:22.610102+0000","last_active":"2023-12-19T08:02:23.623024+0000","last_peered":"2023-12-19T08:02:23.623024+0000","last_clean":"2023-12-19T08:02:23.623024+0000","last_became_active":"2023-12-19T08:02:22.609712+0000","last_became_peered":"2023-12-19T08:02:22.609712+0000","last_unstale":"2023-12-19T08:02:23.623024+0000","last_undegraded":"2023-12-19T08:02:23.623024+0000","last_fullsized":"2023-12-19T08:02:23.623024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:53.974542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.461324+0000","last_change":"2023-12-19T08:02:23.461324+0000","last_active":"2023-12-19T08:02:23.461324+0000","last_peered":"2023-12-19T08:02:23.461324+0000","last_clean":"2023-12-19T08:02:23.461324+0000","last_became_active":"2023-12-19T08:02:23.461045+0000","last_became_peered":"2023-12-19T08:02:23.461045+0000","last_unstale":"2023-12-19T08:02:23.461324+0000","last_undegraded":"2023-12-19T08:02:23.461324+0000","last_fullsized":"2023-12-19T08:02:23.461324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.446326+0000","last_change":"2023-12-19T08:02:22.618628+0000","last_active":"2023-12-19T08:02:23.446326+0000","last_peered":"2023-12-19T08:02:23.446326+0000","last_clean":"2023-12-19T08:02:23.446326+0000","last_became_active":"2023-12-19T08:02:22.618329+0000","last_became_peered":"2023-12-19T08:02:22.618329+0000","last_unstale":"2023-12-19T08:02:23.446326+0000","last_undegraded":"2023-12-19T08:02:23.446326+0000","last_fullsized":"2023-12-19T08:02:23.446326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:24.214042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.453986+0000","last_change":"2023-12-19T08:02:23.453986+0000","last_active":"2023-12-19T08:02:23.453986+0000","last_peered":"2023-12-19T08:02:23.453986+0000","last_clean":"2023-12-19T08:02:23.453986+0000","last_became_active":"2023-12-19T08:02:23.453648+0000","last_became_peered":"2023-12-19T08:02:23.453648+0000","last_unstale":"2023-12-19T08:02:23.453986+0000","last_undegraded":"2023-12-19T08:02:23.453986+0000","last_fullsized":"2023-12-19T08:02:23.453986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.620248+0000","last_change":"2023-12-19T08:02:22.617820+0000","last_active":"2023-12-19T08:02:23.620248+0000","last_peered":"2023-12-19T08:02:23.620248+0000","last_clean":"2023-12-19T08:02:23.620248+0000","last_became_active":"2023-12-19T08:02:22.617539+0000","last_became_peered":"2023-12-19T08:02:22.617539+0000","last_unstale":"2023-12-19T08:02:23.620248+0000","last_undegraded":"2023-12-19T08:02:23.620248+0000","last_fullsized":"2023-12-19T08:02:23.620248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:45.405609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.462092+0000","last_change":"2023-12-19T08:02:23.454793+0000","last_active":"2023-12-19T08:02:23.462092+0000","last_peered":"2023-12-19T08:02:23.462092+0000","last_clean":"2023-12-19T08:02:23.462092+0000","last_became_active":"2023-12-19T08:02:23.454469+0000","last_became_peered":"2023-12-19T08:02:23.454469+0000","last_unstale":"2023-12-19T08:02:23.462092+0000","last_undegraded":"2023-12-19T08:02:23.462092+0000","last_fullsized":"2023-12-19T08:02:23.462092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:03.956014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.618755+0000","last_change":"2023-12-19T08:02:22.613357+0000","last_active":"2023-12-19T08:02:23.618755+0000","last_peered":"2023-12-19T08:02:23.618755+0000","last_clean":"2023-12-19T08:02:23.618755+0000","last_became_active":"2023-12-19T08:02:22.613084+0000","last_became_peered":"2023-12-19T08:02:22.613084+0000","last_unstale":"2023-12-19T08:02:23.618755+0000","last_undegraded":"2023-12-19T08:02:23.618755+0000","last_fullsized":"2023-12-19T08:02:23.618755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:40:18.821196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.442025+0000","last_change":"2023-12-19T08:02:23.442025+0000","last_active":"2023-12-19T08:02:23.442025+0000","last_peered":"2023-12-19T08:02:23.442025+0000","last_clean":"2023-12-19T08:02:23.442025+0000","last_became_active":"2023-12-19T08:02:23.441461+0000","last_became_peered":"2023-12-19T08:02:23.441461+0000","last_unstale":"2023-12-19T08:02:23.442025+0000","last_undegraded":"2023-12-19T08:02:23.442025+0000","last_fullsized":"2023-12-19T08:02:23.442025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:57:44.207797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.617535+0000","last_change":"2023-12-19T08:02:22.610175+0000","last_active":"2023-12-19T08:02:23.617535+0000","last_peered":"2023-12-19T08:02:23.617535+0000","last_clean":"2023-12-19T08:02:23.617535+0000","last_became_active":"2023-12-19T08:02:22.609848+0000","last_became_peered":"2023-12-19T08:02:22.609848+0000","last_unstale":"2023-12-19T08:02:23.617535+0000","last_undegraded":"2023-12-19T08:02:23.617535+0000","last_fullsized":"2023-12-19T08:02:23.617535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:46.149837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458288+0000","last_change":"2023-12-19T08:02:23.455486+0000","last_active":"2023-12-19T08:02:23.458288+0000","last_peered":"2023-12-19T08:02:23.458288+0000","last_clean":"2023-12-19T08:02:23.458288+0000","last_became_active":"2023-12-19T08:02:23.455180+0000","last_became_peered":"2023-12-19T08:02:23.455180+0000","last_unstale":"2023-12-19T08:02:23.458288+0000","last_undegraded":"2023-12-19T08:02:23.458288+0000","last_fullsized":"2023-12-19T08:02:23.458288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:33:38.020278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.616438+0000","last_change":"2023-12-19T08:02:22.611027+0000","last_active":"2023-12-19T08:02:23.616438+0000","last_peered":"2023-12-19T08:02:23.616438+0000","last_clean":"2023-12-19T08:02:23.616438+0000","last_became_active":"2023-12-19T08:02:22.610745+0000","last_became_peered":"2023-12-19T08:02:22.610745+0000","last_unstale":"2023-12-19T08:02:23.616438+0000","last_undegraded":"2023-12-19T08:02:23.616438+0000","last_fullsized":"2023-12-19T08:02:23.616438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:32:05.206647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457263+0000","last_change":"2023-12-19T08:02:23.457263+0000","last_active":"2023-12-19T08:02:23.457263+0000","last_peered":"2023-12-19T08:02:23.457263+0000","last_clean":"2023-12-19T08:02:23.457263+0000","last_became_active":"2023-12-19T08:02:23.456955+0000","last_became_peered":"2023-12-19T08:02:23.456955+0000","last_unstale":"2023-12-19T08:02:23.457263+0000","last_undegraded":"2023-12-19T08:02:23.457263+0000","last_fullsized":"2023-12-19T08:02:23.457263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:40.250504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436531+0000","last_change":"2023-12-19T08:02:22.615364+0000","last_active":"2023-12-19T08:02:23.436531+0000","last_peered":"2023-12-19T08:02:23.436531+0000","last_clean":"2023-12-19T08:02:23.436531+0000","last_became_active":"2023-12-19T08:02:22.615005+0000","last_became_peered":"2023-12-19T08:02:22.615005+0000","last_unstale":"2023-12-19T08:02:23.436531+0000","last_undegraded":"2023-12-19T08:02:23.436531+0000","last_fullsized":"2023-12-19T08:02:23.436531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:42.507228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454313+0000","last_change":"2023-12-19T08:02:23.452573+0000","last_active":"2023-12-19T08:02:23.454313+0000","last_peered":"2023-12-19T08:02:23.454313+0000","last_clean":"2023-12-19T08:02:23.454313+0000","last_became_active":"2023-12-19T08:02:23.452295+0000","last_became_peered":"2023-12-19T08:02:23.452295+0000","last_unstale":"2023-12-19T08:02:23.454313+0000","last_undegraded":"2023-12-19T08:02:23.454313+0000","last_fullsized":"2023-12-19T08:02:23.454313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:31:44.372844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.628894+0000","last_change":"2023-12-19T08:02:22.621199+0000","last_active":"2023-12-19T08:02:22.628894+0000","last_peered":"2023-12-19T08:02:22.628894+0000","last_clean":"2023-12-19T08:02:22.628894+0000","last_became_active":"2023-12-19T08:02:22.620894+0000","last_became_peered":"2023-12-19T08:02:22.620894+0000","last_unstale":"2023-12-19T08:02:22.628894+0000","last_undegraded":"2023-12-19T08:02:22.628894+0000","last_fullsized":"2023-12-19T08:02:22.628894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:44.389604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.447084+0000","last_change":"2023-12-19T08:02:23.443563+0000","last_active":"2023-12-19T08:02:23.447084+0000","last_peered":"2023-12-19T08:02:23.447084+0000","last_clean":"2023-12-19T08:02:23.447084+0000","last_became_active":"2023-12-19T08:02:23.443262+0000","last_became_peered":"2023-12-19T08:02:23.443262+0000","last_unstale":"2023-12-19T08:02:23.447084+0000","last_undegraded":"2023-12-19T08:02:23.447084+0000","last_fullsized":"2023-12-19T08:02:23.447084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:19:26.758815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.445672+0000","last_change":"2023-12-19T08:02:22.618977+0000","last_active":"2023-12-19T08:02:23.445672+0000","last_peered":"2023-12-19T08:02:23.445672+0000","last_clean":"2023-12-19T08:02:23.445672+0000","last_became_active":"2023-12-19T08:02:22.618674+0000","last_became_peered":"2023-12-19T08:02:22.618674+0000","last_unstale":"2023-12-19T08:02:23.445672+0000","last_undegraded":"2023-12-19T08:02:23.445672+0000","last_fullsized":"2023-12-19T08:02:23.445672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:49.973599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456468+0000","last_change":"2023-12-19T08:02:23.453623+0000","last_active":"2023-12-19T08:02:23.456468+0000","last_peered":"2023-12-19T08:02:23.456468+0000","last_clean":"2023-12-19T08:02:23.456468+0000","last_became_active":"2023-12-19T08:02:23.453338+0000","last_became_peered":"2023-12-19T08:02:23.453338+0000","last_unstale":"2023-12-19T08:02:23.456468+0000","last_undegraded":"2023-12-19T08:02:23.456468+0000","last_fullsized":"2023-12-19T08:02:23.456468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:22:01.249592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618487+0000","last_change":"2023-12-19T08:02:22.615446+0000","last_active":"2023-12-19T08:02:22.618487+0000","last_peered":"2023-12-19T08:02:22.618487+0000","last_clean":"2023-12-19T08:02:22.618487+0000","last_became_active":"2023-12-19T08:02:22.615148+0000","last_became_peered":"2023-12-19T08:02:22.615148+0000","last_unstale":"2023-12-19T08:02:22.618487+0000","last_undegraded":"2023-12-19T08:02:22.618487+0000","last_fullsized":"2023-12-19T08:02:22.618487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:49.973599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458053+0000","last_change":"2023-12-19T08:02:23.458053+0000","last_active":"2023-12-19T08:02:23.458053+0000","last_peered":"2023-12-19T08:02:23.458053+0000","last_clean":"2023-12-19T08:02:23.458053+0000","last_became_active":"2023-12-19T08:02:23.457772+0000","last_became_peered":"2023-12-19T08:02:23.457772+0000","last_unstale":"2023-12-19T08:02:23.458053+0000","last_undegraded":"2023-12-19T08:02:23.458053+0000","last_fullsized":"2023-12-19T08:02:23.458053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:33:55.460893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627425+0000","last_change":"2023-12-19T08:02:22.616100+0000","last_active":"2023-12-19T08:02:23.627425+0000","last_peered":"2023-12-19T08:02:23.627425+0000","last_clean":"2023-12-19T08:02:23.627425+0000","last_became_active":"2023-12-19T08:02:22.615783+0000","last_became_peered":"2023-12-19T08:02:22.615783+0000","last_unstale":"2023-12-19T08:02:23.627425+0000","last_undegraded":"2023-12-19T08:02:23.627425+0000","last_fullsized":"2023-12-19T08:02:23.627425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:28:46.325113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454261+0000","last_change":"2023-12-19T08:02:23.454261+0000","last_active":"2023-12-19T08:02:23.454261+0000","last_peered":"2023-12-19T08:02:23.454261+0000","last_clean":"2023-12-19T08:02:23.454261+0000","last_became_active":"2023-12-19T08:02:23.453833+0000","last_became_peered":"2023-12-19T08:02:23.453833+0000","last_unstale":"2023-12-19T08:02:23.454261+0000","last_undegraded":"2023-12-19T08:02:23.454261+0000","last_fullsized":"2023-12-19T08:02:23.454261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:47.858020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.624842+0000","last_change":"2023-12-19T08:02:22.617875+0000","last_active":"2023-12-19T08:02:23.624842+0000","last_peered":"2023-12-19T08:02:23.624842+0000","last_clean":"2023-12-19T08:02:23.624842+0000","last_became_active":"2023-12-19T08:02:22.617579+0000","last_became_peered":"2023-12-19T08:02:22.617579+0000","last_unstale":"2023-12-19T08:02:23.624842+0000","last_undegraded":"2023-12-19T08:02:23.624842+0000","last_fullsized":"2023-12-19T08:02:23.624842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:33:54.444879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.463631+0000","last_change":"2023-12-19T08:02:23.463631+0000","last_active":"2023-12-19T08:02:23.463631+0000","last_peered":"2023-12-19T08:02:23.463631+0000","last_clean":"2023-12-19T08:02:23.463631+0000","last_became_active":"2023-12-19T08:02:23.463355+0000","last_became_peered":"2023-12-19T08:02:23.463355+0000","last_unstale":"2023-12-19T08:02:23.463631+0000","last_undegraded":"2023-12-19T08:02:23.463631+0000","last_fullsized":"2023-12-19T08:02:23.463631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:46.421623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627310+0000","last_change":"2023-12-19T08:02:22.620685+0000","last_active":"2023-12-19T08:02:23.627310+0000","last_peered":"2023-12-19T08:02:23.627310+0000","last_clean":"2023-12-19T08:02:23.627310+0000","last_became_active":"2023-12-19T08:02:22.620353+0000","last_became_peered":"2023-12-19T08:02:22.620353+0000","last_unstale":"2023-12-19T08:02:23.627310+0000","last_undegraded":"2023-12-19T08:02:23.627310+0000","last_fullsized":"2023-12-19T08:02:23.627310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:59.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.466084+0000","last_change":"2023-12-19T08:02:23.466084+0000","last_active":"2023-12-19T08:02:23.466084+0000","last_peered":"2023-12-19T08:02:23.466084+0000","last_clean":"2023-12-19T08:02:23.466084+0000","last_became_active":"2023-12-19T08:02:23.465699+0000","last_became_peered":"2023-12-19T08:02:23.465699+0000","last_unstale":"2023-12-19T08:02:23.466084+0000","last_undegraded":"2023-12-19T08:02:23.466084+0000","last_fullsized":"2023-12-19T08:02:23.466084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618546+0000","last_change":"2023-12-19T08:02:22.615110+0000","last_active":"2023-12-19T08:02:22.618546+0000","last_peered":"2023-12-19T08:02:22.618546+0000","last_clean":"2023-12-19T08:02:22.618546+0000","last_became_active":"2023-12-19T08:02:22.614598+0000","last_became_peered":"2023-12-19T08:02:22.614598+0000","last_unstale":"2023-12-19T08:02:22.618546+0000","last_undegraded":"2023-12-19T08:02:22.618546+0000","last_fullsized":"2023-12-19T08:02:22.618546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:45:11.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456440+0000","last_change":"2023-12-19T08:02:23.456440+0000","last_active":"2023-12-19T08:02:23.456440+0000","last_peered":"2023-12-19T08:02:23.456440+0000","last_clean":"2023-12-19T08:02:23.456440+0000","last_became_active":"2023-12-19T08:02:23.453483+0000","last_became_peered":"2023-12-19T08:02:23.453483+0000","last_unstale":"2023-12-19T08:02:23.456440+0000","last_undegraded":"2023-12-19T08:02:23.456440+0000","last_fullsized":"2023-12-19T08:02:23.456440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:49:43.200638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.445655+0000","last_change":"2023-12-19T08:02:22.611908+0000","last_active":"2023-12-19T08:02:23.445655+0000","last_peered":"2023-12-19T08:02:23.445655+0000","last_clean":"2023-12-19T08:02:23.445655+0000","last_became_active":"2023-12-19T08:02:22.611329+0000","last_became_peered":"2023-12-19T08:02:22.611329+0000","last_unstale":"2023-12-19T08:02:23.445655+0000","last_undegraded":"2023-12-19T08:02:23.445655+0000","last_fullsized":"2023-12-19T08:02:23.445655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:35.394367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.455703+0000","last_change":"2023-12-19T08:02:23.454566+0000","last_active":"2023-12-19T08:02:23.455703+0000","last_peered":"2023-12-19T08:02:23.455703+0000","last_clean":"2023-12-19T08:02:23.455703+0000","last_became_active":"2023-12-19T08:02:23.454274+0000","last_became_peered":"2023-12-19T08:02:23.454274+0000","last_unstale":"2023-12-19T08:02:23.455703+0000","last_undegraded":"2023-12-19T08:02:23.455703+0000","last_fullsized":"2023-12-19T08:02:23.455703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:27.758201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.628041+0000","last_change":"2023-12-19T08:02:22.615290+0000","last_active":"2023-12-19T08:02:23.628041+0000","last_peered":"2023-12-19T08:02:23.628041+0000","last_clean":"2023-12-19T08:02:23.628041+0000","last_became_active":"2023-12-19T08:02:22.615007+0000","last_became_peered":"2023-12-19T08:02:22.615007+0000","last_unstale":"2023-12-19T08:02:23.628041+0000","last_undegraded":"2023-12-19T08:02:23.628041+0000","last_fullsized":"2023-12-19T08:02:23.628041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.472599+0000","last_change":"2023-12-19T08:02:23.467532+0000","last_active":"2023-12-19T08:02:23.472599+0000","last_peered":"2023-12-19T08:02:23.472599+0000","last_clean":"2023-12-19T08:02:23.472599+0000","last_became_active":"2023-12-19T08:02:23.462282+0000","last_became_peered":"2023-12-19T08:02:23.462282+0000","last_unstale":"2023-12-19T08:02:23.472599+0000","last_undegraded":"2023-12-19T08:02:23.472599+0000","last_fullsized":"2023-12-19T08:02:23.472599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:25.230055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451691+0000","last_change":"2023-12-19T08:02:22.614966+0000","last_active":"2023-12-19T08:02:23.451691+0000","last_peered":"2023-12-19T08:02:23.451691+0000","last_clean":"2023-12-19T08:02:23.451691+0000","last_became_active":"2023-12-19T08:02:22.614510+0000","last_became_peered":"2023-12-19T08:02:22.614510+0000","last_unstale":"2023-12-19T08:02:23.451691+0000","last_undegraded":"2023-12-19T08:02:23.451691+0000","last_fullsized":"2023-12-19T08:02:23.451691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:59.863209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.469161+0000","last_change":"2023-12-19T08:02:23.469161+0000","last_active":"2023-12-19T08:02:23.469161+0000","last_peered":"2023-12-19T08:02:23.469161+0000","last_clean":"2023-12-19T08:02:23.469161+0000","last_became_active":"2023-12-19T08:02:23.466870+0000","last_became_peered":"2023-12-19T08:02:23.466870+0000","last_unstale":"2023-12-19T08:02:23.469161+0000","last_undegraded":"2023-12-19T08:02:23.469161+0000","last_fullsized":"2023-12-19T08:02:23.469161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:06.813090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.626793+0000","last_change":"2023-12-19T08:02:22.619397+0000","last_active":"2023-12-19T08:02:23.626793+0000","last_peered":"2023-12-19T08:02:23.626793+0000","last_clean":"2023-12-19T08:02:23.626793+0000","last_became_active":"2023-12-19T08:02:22.618886+0000","last_became_peered":"2023-12-19T08:02:22.618886+0000","last_unstale":"2023-12-19T08:02:23.626793+0000","last_undegraded":"2023-12-19T08:02:23.626793+0000","last_fullsized":"2023-12-19T08:02:23.626793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.460303+0000","last_change":"2023-12-19T08:02:23.460303+0000","last_active":"2023-12-19T08:02:23.460303+0000","last_peered":"2023-12-19T08:02:23.460303+0000","last_clean":"2023-12-19T08:02:23.460303+0000","last_became_active":"2023-12-19T08:02:23.459947+0000","last_became_peered":"2023-12-19T08:02:23.459947+0000","last_unstale":"2023-12-19T08:02:23.460303+0000","last_undegraded":"2023-12-19T08:02:23.460303+0000","last_fullsized":"2023-12-19T08:02:23.460303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:17:08.448655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.627304+0000","last_change":"2023-12-19T08:02:22.618977+0000","last_active":"2023-12-19T08:02:23.627304+0000","last_peered":"2023-12-19T08:02:23.627304+0000","last_clean":"2023-12-19T08:02:23.627304+0000","last_became_active":"2023-12-19T08:02:22.618683+0000","last_became_peered":"2023-12-19T08:02:22.618683+0000","last_unstale":"2023-12-19T08:02:23.627304+0000","last_undegraded":"2023-12-19T08:02:23.627304+0000","last_fullsized":"2023-12-19T08:02:23.627304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:02.059302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451854+0000","last_change":"2023-12-19T08:02:23.451854+0000","last_active":"2023-12-19T08:02:23.451854+0000","last_peered":"2023-12-19T08:02:23.451854+0000","last_clean":"2023-12-19T08:02:23.451854+0000","last_became_active":"2023-12-19T08:02:23.451229+0000","last_became_peered":"2023-12-19T08:02:23.451229+0000","last_unstale":"2023-12-19T08:02:23.451854+0000","last_undegraded":"2023-12-19T08:02:23.451854+0000","last_fullsized":"2023-12-19T08:02:23.451854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:55:15.260396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436319+0000","last_change":"2023-12-19T08:02:22.608641+0000","last_active":"2023-12-19T08:02:23.436319+0000","last_peered":"2023-12-19T08:02:23.436319+0000","last_clean":"2023-12-19T08:02:23.436319+0000","last_became_active":"2023-12-19T08:02:22.608369+0000","last_became_peered":"2023-12-19T08:02:22.608369+0000","last_unstale":"2023-12-19T08:02:23.436319+0000","last_undegraded":"2023-12-19T08:02:23.436319+0000","last_fullsized":"2023-12-19T08:02:23.436319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:53.317303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.449846+0000","last_change":"2023-12-19T08:02:23.449846+0000","last_active":"2023-12-19T08:02:23.449846+0000","last_peered":"2023-12-19T08:02:23.449846+0000","last_clean":"2023-12-19T08:02:23.449846+0000","last_became_active":"2023-12-19T08:02:23.448518+0000","last_became_peered":"2023-12-19T08:02:23.448518+0000","last_unstale":"2023-12-19T08:02:23.449846+0000","last_undegraded":"2023-12-19T08:02:23.449846+0000","last_fullsized":"2023-12-19T08:02:23.449846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.620034+0000","last_change":"2023-12-19T08:02:22.616425+0000","last_active":"2023-12-19T08:02:23.620034+0000","last_peered":"2023-12-19T08:02:23.620034+0000","last_clean":"2023-12-19T08:02:23.620034+0000","last_became_active":"2023-12-19T08:02:22.616111+0000","last_became_peered":"2023-12-19T08:02:22.616111+0000","last_unstale":"2023-12-19T08:02:23.620034+0000","last_undegraded":"2023-12-19T08:02:23.620034+0000","last_fullsized":"2023-12-19T08:02:23.620034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:51:46.842007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.456902+0000","last_change":"2023-12-19T08:02:23.456902+0000","last_active":"2023-12-19T08:02:23.456902+0000","last_peered":"2023-12-19T08:02:23.456902+0000","last_clean":"2023-12-19T08:02:23.456902+0000","last_became_active":"2023-12-19T08:02:23.456540+0000","last_became_peered":"2023-12-19T08:02:23.456540+0000","last_unstale":"2023-12-19T08:02:23.456902+0000","last_undegraded":"2023-12-19T08:02:23.456902+0000","last_fullsized":"2023-12-19T08:02:23.456902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:02:50.588814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618530+0000","last_change":"2023-12-19T08:02:22.614364+0000","last_active":"2023-12-19T08:02:22.618530+0000","last_peered":"2023-12-19T08:02:22.618530+0000","last_clean":"2023-12-19T08:02:22.618530+0000","last_became_active":"2023-12-19T08:02:22.613813+0000","last_became_peered":"2023-12-19T08:02:22.613813+0000","last_unstale":"2023-12-19T08:02:22.618530+0000","last_undegraded":"2023-12-19T08:02:22.618530+0000","last_fullsized":"2023-12-19T08:02:22.618530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:36.898878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.457049+0000","last_change":"2023-12-19T08:02:23.457049+0000","last_active":"2023-12-19T08:02:23.457049+0000","last_peered":"2023-12-19T08:02:23.457049+0000","last_clean":"2023-12-19T08:02:23.457049+0000","last_became_active":"2023-12-19T08:02:23.456690+0000","last_became_peered":"2023-12-19T08:02:23.456690+0000","last_unstale":"2023-12-19T08:02:23.457049+0000","last_undegraded":"2023-12-19T08:02:23.457049+0000","last_fullsized":"2023-12-19T08:02:23.457049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:27.513419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618977+0000","last_change":"2023-12-19T08:02:22.617291+0000","last_active":"2023-12-19T08:02:22.618977+0000","last_peered":"2023-12-19T08:02:22.618977+0000","last_clean":"2023-12-19T08:02:22.618977+0000","last_became_active":"2023-12-19T08:02:22.616986+0000","last_became_peered":"2023-12-19T08:02:22.616986+0000","last_unstale":"2023-12-19T08:02:22.618977+0000","last_undegraded":"2023-12-19T08:02:22.618977+0000","last_fullsized":"2023-12-19T08:02:22.618977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:54.821396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.458437+0000","last_change":"2023-12-19T08:02:23.458437+0000","last_active":"2023-12-19T08:02:23.458437+0000","last_peered":"2023-12-19T08:02:23.458437+0000","last_clean":"2023-12-19T08:02:23.458437+0000","last_became_active":"2023-12-19T08:02:23.457776+0000","last_became_peered":"2023-12-19T08:02:23.457776+0000","last_unstale":"2023-12-19T08:02:23.458437+0000","last_undegraded":"2023-12-19T08:02:23.458437+0000","last_fullsized":"2023-12-19T08:02:23.458437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:36.772610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.625030+0000","last_change":"2023-12-19T08:02:22.614778+0000","last_active":"2023-12-19T08:02:23.625030+0000","last_peered":"2023-12-19T08:02:23.625030+0000","last_clean":"2023-12-19T08:02:23.625030+0000","last_became_active":"2023-12-19T08:02:22.614194+0000","last_became_peered":"2023-12-19T08:02:22.614194+0000","last_unstale":"2023-12-19T08:02:23.625030+0000","last_undegraded":"2023-12-19T08:02:23.625030+0000","last_fullsized":"2023-12-19T08:02:23.625030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:26.497406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452176+0000","last_change":"2023-12-19T08:02:23.452176+0000","last_active":"2023-12-19T08:02:23.452176+0000","last_peered":"2023-12-19T08:02:23.452176+0000","last_clean":"2023-12-19T08:02:23.452176+0000","last_became_active":"2023-12-19T08:02:23.451877+0000","last_became_peered":"2023-12-19T08:02:23.451877+0000","last_unstale":"2023-12-19T08:02:23.452176+0000","last_undegraded":"2023-12-19T08:02:23.452176+0000","last_fullsized":"2023-12-19T08:02:23.452176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:50.240450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.618573+0000","last_change":"2023-12-19T08:02:22.617222+0000","last_active":"2023-12-19T08:02:22.618573+0000","last_peered":"2023-12-19T08:02:22.618573+0000","last_clean":"2023-12-19T08:02:22.618573+0000","last_became_active":"2023-12-19T08:02:22.616938+0000","last_became_peered":"2023-12-19T08:02:22.616938+0000","last_unstale":"2023-12-19T08:02:22.618573+0000","last_undegraded":"2023-12-19T08:02:22.618573+0000","last_fullsized":"2023-12-19T08:02:22.618573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:35.394367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.467785+0000","last_change":"2023-12-19T08:02:23.467785+0000","last_active":"2023-12-19T08:02:23.467785+0000","last_peered":"2023-12-19T08:02:23.467785+0000","last_clean":"2023-12-19T08:02:23.467785+0000","last_became_active":"2023-12-19T08:02:23.467315+0000","last_became_peered":"2023-12-19T08:02:23.467315+0000","last_unstale":"2023-12-19T08:02:23.467785+0000","last_undegraded":"2023-12-19T08:02:23.467785+0000","last_fullsized":"2023-12-19T08:02:23.467785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:22.823532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-12-19T08:02:22.619088+0000","last_change":"2023-12-19T08:02:22.617508+0000","last_active":"2023-12-19T08:02:22.619088+0000","last_peered":"2023-12-19T08:02:22.619088+0000","last_clean":"2023-12-19T08:02:22.619088+0000","last_became_active":"2023-12-19T08:02:22.617162+0000","last_became_peered":"2023-12-19T08:02:22.617162+0000","last_unstale":"2023-12-19T08:02:22.619088+0000","last_undegraded":"2023-12-19T08:02:22.619088+0000","last_fullsized":"2023-12-19T08:02:22.619088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:05.797077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451906+0000","last_change":"2023-12-19T08:02:23.451906+0000","last_active":"2023-12-19T08:02:23.451906+0000","last_peered":"2023-12-19T08:02:23.451906+0000","last_clean":"2023-12-19T08:02:23.451906+0000","last_became_active":"2023-12-19T08:02:23.451602+0000","last_became_peered":"2023-12-19T08:02:23.451602+0000","last_unstale":"2023-12-19T08:02:23.451906+0000","last_undegraded":"2023-12-19T08:02:23.451906+0000","last_fullsized":"2023-12-19T08:02:23.451906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:54.990555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.440840+0000","last_change":"2023-12-19T08:02:22.615931+0000","last_active":"2023-12-19T08:02:23.440840+0000","last_peered":"2023-12-19T08:02:23.440840+0000","last_clean":"2023-12-19T08:02:23.440840+0000","last_became_active":"2023-12-19T08:02:22.615334+0000","last_became_peered":"2023-12-19T08:02:22.615334+0000","last_unstale":"2023-12-19T08:02:23.440840+0000","last_undegraded":"2023-12-19T08:02:23.440840+0000","last_fullsized":"2023-12-19T08:02:23.440840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:08:39.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452312+0000","last_change":"2023-12-19T08:02:23.452312+0000","last_active":"2023-12-19T08:02:23.452312+0000","last_peered":"2023-12-19T08:02:23.452312+0000","last_clean":"2023-12-19T08:02:23.452312+0000","last_became_active":"2023-12-19T08:02:23.451950+0000","last_became_peered":"2023-12-19T08:02:23.451950+0000","last_unstale":"2023-12-19T08:02:23.452312+0000","last_undegraded":"2023-12-19T08:02:23.452312+0000","last_fullsized":"2023-12-19T08:02:23.452312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:21:46.421623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.446142+0000","last_change":"2023-12-19T08:02:22.614483+0000","last_active":"2023-12-19T08:02:23.446142+0000","last_peered":"2023-12-19T08:02:23.446142+0000","last_clean":"2023-12-19T08:02:23.446142+0000","last_became_active":"2023-12-19T08:02:22.614210+0000","last_became_peered":"2023-12-19T08:02:22.614210+0000","last_unstale":"2023-12-19T08:02:23.446142+0000","last_undegraded":"2023-12-19T08:02:23.446142+0000","last_fullsized":"2023-12-19T08:02:23.446142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:45:11.904439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.463998+0000","last_change":"2023-12-19T08:02:23.463998+0000","last_active":"2023-12-19T08:02:23.463998+0000","last_peered":"2023-12-19T08:02:23.463998+0000","last_clean":"2023-12-19T08:02:23.463998+0000","last_became_active":"2023-12-19T08:02:23.463721+0000","last_became_peered":"2023-12-19T08:02:23.463721+0000","last_unstale":"2023-12-19T08:02:23.463998+0000","last_undegraded":"2023-12-19T08:02:23.463998+0000","last_fullsized":"2023-12-19T08:02:23.463998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:06.813090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436205+0000","last_change":"2023-12-19T08:02:22.607009+0000","last_active":"2023-12-19T08:02:23.436205+0000","last_peered":"2023-12-19T08:02:23.436205+0000","last_clean":"2023-12-19T08:02:23.436205+0000","last_became_active":"2023-12-19T08:02:22.605969+0000","last_became_peered":"2023-12-19T08:02:22.605969+0000","last_unstale":"2023-12-19T08:02:23.436205+0000","last_undegraded":"2023-12-19T08:02:23.436205+0000","last_fullsized":"2023-12-19T08:02:23.436205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:29:08.762596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.454802+0000","last_change":"2023-12-19T08:02:23.451105+0000","last_active":"2023-12-19T08:02:23.454802+0000","last_peered":"2023-12-19T08:02:23.454802+0000","last_clean":"2023-12-19T08:02:23.454802+0000","last_became_active":"2023-12-19T08:02:23.450756+0000","last_became_peered":"2023-12-19T08:02:23.450756+0000","last_unstale":"2023-12-19T08:02:23.454802+0000","last_undegraded":"2023-12-19T08:02:23.454802+0000","last_fullsized":"2023-12-19T08:02:23.454802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:12.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.436648+0000","last_change":"2023-12-19T08:02:22.609723+0000","last_active":"2023-12-19T08:02:23.436648+0000","last_peered":"2023-12-19T08:02:23.436648+0000","last_clean":"2023-12-19T08:02:23.436648+0000","last_became_active":"2023-12-19T08:02:22.607744+0000","last_became_peered":"2023-12-19T08:02:22.607744+0000","last_unstale":"2023-12-19T08:02:23.436648+0000","last_undegraded":"2023-12-19T08:02:23.436648+0000","last_fullsized":"2023-12-19T08:02:23.436648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:27:53.075744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.452635+0000","last_change":"2023-12-19T08:02:23.452635+0000","last_active":"2023-12-19T08:02:23.452635+0000","last_peered":"2023-12-19T08:02:23.452635+0000","last_clean":"2023-12-19T08:02:23.452635+0000","last_became_active":"2023-12-19T08:02:23.452298+0000","last_became_peered":"2023-12-19T08:02:23.452298+0000","last_unstale":"2023-12-19T08:02:23.452635+0000","last_undegraded":"2023-12-19T08:02:23.452635+0000","last_fullsized":"2023-12-19T08:02:23.452635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:09:25.923471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.440208+0000","last_change":"2023-12-19T08:02:22.617096+0000","last_active":"2023-12-19T08:02:23.440208+0000","last_peered":"2023-12-19T08:02:23.440208+0000","last_clean":"2023-12-19T08:02:23.440208+0000","last_became_active":"2023-12-19T08:02:22.616720+0000","last_became_peered":"2023-12-19T08:02:22.616720+0000","last_unstale":"2023-12-19T08:02:23.440208+0000","last_undegraded":"2023-12-19T08:02:23.440208+0000","last_fullsized":"2023-12-19T08:02:23.440208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:26.497406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":8,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.451143+0000","last_change":"2023-12-19T08:02:23.451143+0000","last_active":"2023-12-19T08:02:23.451143+0000","last_peered":"2023-12-19T08:02:23.451143+0000","last_clean":"2023-12-19T08:02:23.451143+0000","last_became_active":"2023-12-19T08:02:23.450850+0000","last_became_peered":"2023-12-19T08:02:23.450850+0000","last_unstale":"2023-12-19T08:02:23.451143+0000","last_undegraded":"2023-12-19T08:02:23.451143+0000","last_fullsized":"2023-12-19T08:02:23.451143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:22.574984+0000","last_clean_scrub_stamp":"2023-12-19T08:02:22.574984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:40:19.837209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.435261+0000","last_change":"2023-12-19T08:02:22.612387+0000","last_active":"2023-12-19T08:02:23.435261+0000","last_peered":"2023-12-19T08:02:23.435261+0000","last_clean":"2023-12-19T08:02:23.435261+0000","last_became_active":"2023-12-19T08:02:22.612088+0000","last_became_peered":"2023-12-19T08:02:22.612088+0000","last_unstale":"2023-12-19T08:02:23.435261+0000","last_undegraded":"2023-12-19T08:02:23.435261+0000","last_fullsized":"2023-12-19T08:02:23.435261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:21.558971+0000","last_clean_scrub_stamp":"2023-12-19T08:02:21.558971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:30:27.779394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'161","reported_seq":373,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-12-19T08:02:23.444450+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:02:23.444450+0000","last_peered":"2023-12-19T08:02:23.444450+0000","last_clean":"2023-12-19T08:02:23.444450+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:02:23.444450+0000","last_undegraded":"2023-12-19T08:02:23.444450+0000","last_fullsized":"2023-12-19T08:02:23.444450+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575032,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060660,"statfs":{"total":96636764160,"available":96318115840,"internally_reserved":0,"allocated":1642496,"data_stored":1115543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2430000000000001,"15min":1.2410000000000001},"min":{"1min":0.69699999999999995,"5min":0.67600000000000005,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.2649999999999999,"15min":1.256},"min":{"1min":1.0029999999999999,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":1.522,"5min":1.591,"15min":1.845},"last":4.6210000000000004}]},{"osd":1,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.147,"15min":1.147},"min":{"1min":0.81699999999999995,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":1.486,"5min":1.6259999999999999,"15min":1.944},"last":4.9160000000000004},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1499999999999999,"15min":1.153},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.577,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":3.8969999999999998}]},{"osd":2,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.1479999999999999,"15min":1.1639999999999999},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.532,"5min":1.532,"15min":2.375},"last":4.9409999999999998},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.2549999999999999,"15min":1.242},"min":{"1min":0.91000000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.663,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.502}]},{"osd":3,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.27,"15min":1.2849999999999999},"min":{"1min":0.95299999999999996,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":1.627,"5min":1.627,"15min":2.0579999999999998},"last":4.8220000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3320000000000001,"15min":1.349},"min":{"1min":0.96899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.593,"5min":1.631,"15min":2.6720000000000002},"last":4.4980000000000002}]},{"osd":4,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.92000000000000004,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":1.736,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":6.2439999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.85899999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.554,"5min":1.671,"15min":1.9490000000000001},"last":6.8360000000000003}]},{"osd":5,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2310000000000001,"15min":1.2410000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.651,"5min":1.653,"15min":2.8279999999999998},"last":4.5679999999999996},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.365,"15min":1.359},"min":{"1min":0.85099999999999998,"5min":0.751,"15min":0.751},"max":{"1min":1.7969999999999999,"5min":2.2120000000000002,"15min":2.3010000000000002},"last":2.004}]},{"osd":6,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.383,"15min":1.403},"min":{"1min":1.0900000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.7470000000000001,"5min":2.0070000000000001,"15min":2.9220000000000002},"last":8.4760000000000009},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.363,"15min":1.3879999999999999},"min":{"1min":0.97499999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6879999999999999,"5min":1.867,"15min":3.097},"last":4.9039999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302384,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069456,"statfs":{"total":96636764160,"available":96327122944,"internally_reserved":0,"allocated":1024000,"data_stored":517333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0669999999999999,"15min":1.1040000000000001},"min":{"1min":0.82399999999999995,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.665},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.135,"15min":1.129},"min":{"1min":0.872,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.92800000000000005}]},{"osd":1,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1299999999999999,"15min":1.171},"min":{"1min":0.81999999999999995,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5389999999999999,"5min":1.675,"15min":1.881},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.165,"15min":1.171},"min":{"1min":0.82099999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5349999999999999,"5min":1.542,"15min":1.7989999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.226,"15min":1.23},"min":{"1min":0.83999999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.746,"5min":1.847,"15min":1.847},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.21,"15min":1.2230000000000001},"min":{"1min":0.92100000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.694,"5min":1.694,"15min":1.7929999999999999},"last":1.0629999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.25,"15min":1.2709999999999999},"min":{"1min":0.81999999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.5860000000000001,"5min":1.73,"15min":2.0659999999999998},"last":1.2},{"interface":"front","average":{"1min":1.284,"5min":1.3149999999999999,"15min":1.3169999999999999},"min":{"1min":0.93899999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.6120000000000001,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.2809999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1899999999999999,"15min":1.1930000000000001},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6339999999999999,"5min":1.7130000000000001,"15min":2.137},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.403,"15min":1.417},"min":{"1min":0.78300000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.7070000000000001,"5min":1.974,"15min":2.218},"last":0.81000000000000005}]},{"osd":5,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.2969999999999999},"min":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.6899999999999999,"5min":1.9039999999999999,"15min":2.1989999999999998},"last":1.165},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3140000000000001,"15min":1.3169999999999999},"min":{"1min":0.75700000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7250000000000001,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.151}]},{"osd":7,"last update":"Tue Dec 19 08:01:43 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.375,"15min":1.391},"min":{"1min":0.73599999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.7490000000000001,"5min":1.835,"15min":2.2029999999999998},"last":1.141},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4259999999999999,"15min":1.431},"min":{"1min":0.78300000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.738,"5min":2.137,"15min":2.2639999999999998},"last":1.2390000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575031,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298288,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073552,"statfs":{"total":96636764160,"available":96331317248,"internally_reserved":0,"allocated":1024000,"data_stored":517321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.252,"15min":1.2430000000000001},"min":{"1min":0.98199999999999998,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.4750000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.284,"15min":1.278},"min":{"1min":1.111,"5min":0.71499999999999997,"15min":0.50700000000000001},"max":{"1min":1.8129999999999999,"5min":1.954,"15min":2.3140000000000001},"last":1.2909999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.80100000000000005,"5min":0.78200000000000003,"15min":0.61699999999999999},"max":{"1min":1.585,"5min":1.629,"15min":3.5409999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1739999999999999,"15min":1.196},"min":{"1min":0.73899999999999999,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.615,"5min":2.7679999999999998,"15min":3.113},"last":1.361}]},{"osd":2,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.1919999999999999,"15min":1.216},"min":{"1min":0.80700000000000005,"5min":0.74199999999999999,"15min":0.66000000000000003},"max":{"1min":1.546,"5min":1.821,"15min":3.339},"last":1.089},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2070000000000001,"15min":1.224},"min":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.2509999999999999,"15min":1.278},"min":{"1min":0.90100000000000002,"5min":0.80800000000000005,"15min":0.748},"max":{"1min":1.5429999999999999,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.139},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.304,"15min":1.329},"min":{"1min":0.97899999999999998,"5min":0.76600000000000001,"15min":0.58199999999999996},"max":{"1min":1.744,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.256}]},{"osd":4,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.194,"15min":1.2370000000000001},"min":{"1min":0.69199999999999995,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.177},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.25,"15min":1.3100000000000001},"min":{"1min":0.84599999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.637,"5min":2.843,"15min":4.6310000000000002},"last":1.304}]},{"osd":6,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.29,"15min":1.3400000000000001},"min":{"1min":0.90200000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.869,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.389,"5min":1.363,"15min":1.3859999999999999},"min":{"1min":1.0740000000000001,"5min":0.81899999999999995,"15min":0.70599999999999996},"max":{"1min":1.845,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.0429999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.391,"15min":1.444},"min":{"1min":0.84899999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.96,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4570000000000001,"15min":1.4810000000000001},"min":{"1min":1.0920000000000001,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.929,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.2110000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302392,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069448,"statfs":{"total":96636764160,"available":96327114752,"internally_reserved":0,"allocated":1032192,"data_stored":518835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2190000000000001,"15min":1.218},"min":{"1min":0.92100000000000004,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.6000000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.333,"5min":1.3109999999999999,"15min":1.3049999999999999},"min":{"1min":0.94699999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6910000000000001,"5min":1.7909999999999999,"15min":2.738},"last":1.4670000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.77100000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5640000000000001,"5min":1.7589999999999999,"15min":2.589},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.115,"15min":1.1419999999999999},"min":{"1min":0.84599999999999997,"5min":0.60099999999999998,"15min":0.54300000000000004},"max":{"1min":1.528,"5min":1.528,"15min":2.2589999999999999},"last":1.163}]},{"osd":2,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.105,"15min":1.147},"min":{"1min":0.83999999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5409999999999999,"15min":2.4569999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.214,"5min":1.2170000000000001,"15min":1.238},"min":{"1min":0.80100000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.589,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.2170000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.252,"15min":1.294},"min":{"1min":0.85199999999999998,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.4770000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3140000000000001,"15min":1.351},"min":{"1min":0.87,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.0620000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1870000000000001,"15min":1.226},"min":{"1min":0.58299999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.526,"5min":1.8149999999999999,"15min":3.556},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.1659999999999999,"15min":1.2250000000000001},"min":{"1min":0.64700000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.538,"5min":1.764,"15min":3.2599999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.238,"15min":1.27},"min":{"1min":0.76700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.748,"5min":1.748,"15min":2.645},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.3799999999999999,"15min":1.3939999999999999},"min":{"1min":0.71999999999999997,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.8069999999999999,"5min":2.0049999999999999,"15min":2.923},"last":1.546}]},{"osd":7,"last update":"Tue Dec 19 08:01:31 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.323,"15min":1.3779999999999999},"min":{"1min":0.58799999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6930000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.268},{"interface":"front","average":{"1min":1.417,"5min":1.3979999999999999,"15min":1.431},"min":{"1min":1.0700000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.966,"5min":1.966,"15min":2.8460000000000001},"last":1.405}]}]},{"osd":3,"up_from":13,"seq":55834575031,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302380,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069460,"statfs":{"total":96636764160,"available":96327127040,"internally_reserved":0,"allocated":1019904,"data_stored":517299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.84599999999999997,"15min":0.89500000000000002},"min":{"1min":0.58799999999999997,"5min":0.497,"15min":0.45500000000000002},"max":{"1min":1.2070000000000001,"5min":1.617,"15min":5.1079999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.95099999999999996,"15min":0.996},"min":{"1min":0.70199999999999996,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":6.1900000000000004},"last":1.123}]},{"osd":1,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.90300000000000002,"15min":0.95599999999999996},"min":{"1min":0.66900000000000004,"5min":0.54500000000000004,"15min":0.45600000000000002},"max":{"1min":1.1539999999999999,"5min":1.431,"15min":4.8879999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.022,"15min":1.1000000000000001},"min":{"1min":0.79100000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.3660000000000001,"5min":1.5129999999999999,"15min":8.0830000000000002},"last":0.49399999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.069,"15min":1.103},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.52800000000000002},"max":{"1min":1.49,"5min":1.49,"15min":4.7519999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1359999999999999,"15min":1.202},"min":{"1min":0.95599999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3140000000000001,"5min":1.8300000000000001,"15min":5.3330000000000002},"last":0.57299999999999995}]},{"osd":4,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2589999999999999,"15min":1.2989999999999999},"min":{"1min":0.82399999999999995,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.5349999999999999,"5min":1.78,"15min":4.7130000000000001},"last":1.23},{"interface":"front","average":{"1min":1.224,"5min":1.222,"15min":1.294},"min":{"1min":0.94299999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.524,"5min":1.6240000000000001,"15min":5.0220000000000002},"last":1.145}]},{"osd":5,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2470000000000001,"15min":1.325},"min":{"1min":1.028,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.627,"5min":1.7090000000000001,"15min":4.9889999999999999},"last":1.409},{"interface":"front","average":{"1min":1.371,"5min":1.333,"15min":1.417},"min":{"1min":1.125,"5min":0.90500000000000003,"15min":0.65800000000000003},"max":{"1min":1.5860000000000001,"5min":1.7350000000000001,"15min":8.5299999999999994},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.379,"15min":1.458},"min":{"1min":1.2130000000000001,"5min":0.88800000000000001,"15min":0.88700000000000001},"max":{"1min":1.7250000000000001,"5min":1.746,"15min":7.2750000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.399,"5min":1.3720000000000001,"15min":1.409},"min":{"1min":1.1100000000000001,"5min":0.82499999999999996,"15min":0.79100000000000004},"max":{"1min":1.7350000000000001,"5min":1.7789999999999999,"15min":4.6550000000000002},"last":1.26}]},{"osd":7,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.462,"15min":1.5669999999999999},"min":{"1min":1.1000000000000001,"5min":1.0629999999999999,"15min":0.84799999999999998},"max":{"1min":1.865,"5min":1.8720000000000001,"15min":16.081},"last":1.621},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4390000000000001,"15min":1.536},"min":{"1min":1.135,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.7190000000000001,"5min":1.833,"15min":15.285},"last":1.5489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302380,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069460,"statfs":{"total":96636764160,"available":96327127040,"internally_reserved":0,"allocated":1019904,"data_stored":517299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.89600000000000002,"15min":0.92900000000000005},"min":{"1min":0.56399999999999995,"5min":0.56100000000000005,"15min":0.437},"max":{"1min":1.2749999999999999,"5min":1.776,"15min":4.1479999999999997},"last":1.137},{"interface":"front","average":{"1min":1.052,"5min":1.046,"15min":1.0740000000000001},"min":{"1min":0.82199999999999995,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.3220000000000001,"5min":1.8460000000000001,"15min":4.093},"last":0.98199999999999998}]},{"osd":1,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.94599999999999995,"15min":0.97199999999999998},"min":{"1min":0.63700000000000001,"5min":0.59299999999999997,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.5620000000000001,"15min":3.8700000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1,"15min":1.0269999999999999},"min":{"1min":0.65000000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":3.9089999999999998},"last":1.2150000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.97899999999999998,"15min":0.96499999999999997},"min":{"1min":0.68200000000000005,"5min":0.59299999999999997,"15min":0.46300000000000002},"max":{"1min":1.343,"5min":1.619,"15min":1.667},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.78800000000000003,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.5449999999999999,"5min":1.629,"15min":1.7250000000000001},"last":0.999}]},{"osd":4,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3320000000000001,"15min":1.345},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.79300000000000004},"max":{"1min":1.639,"5min":1.7669999999999999,"15min":3.9239999999999999},"last":1.26},{"interface":"front","average":{"1min":1.224,"5min":1.2789999999999999,"15min":1.3240000000000001},"min":{"1min":0.97299999999999998,"5min":0.89400000000000002,"15min":0.82099999999999995},"max":{"1min":1.6160000000000001,"5min":1.736,"15min":4.0170000000000003},"last":1.3779999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3420000000000001,"15min":1.371},"min":{"1min":1.0920000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.6040000000000001,"5min":1.901,"15min":3.948},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.2709999999999999,"15min":1.2609999999999999},"min":{"1min":1.0589999999999999,"5min":0.81699999999999995,"15min":0.80400000000000005},"max":{"1min":1.5720000000000001,"5min":1.6930000000000001,"15min":1.879},"last":0.95399999999999996}]},{"osd":6,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.302,"15min":1.306},"min":{"1min":1.0660000000000001,"5min":0.92700000000000005,"15min":0.752},"max":{"1min":1.583,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.097},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3620000000000001,"15min":1.403},"min":{"1min":1.0369999999999999,"5min":0.97899999999999998,"15min":0.85399999999999998},"max":{"1min":1.7010000000000001,"5min":1.867,"15min":3.9620000000000002},"last":1.3400000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:01:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.369,"15min":1.423},"min":{"1min":1.107,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.607,"5min":1.8109999999999999,"15min":3.9809999999999999},"last":1.298},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.355,"15min":1.3819999999999999},"min":{"1min":1.0029999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.609,"5min":1.8620000000000001,"15min":2.206},"last":1.242}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302408,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069432,"statfs":{"total":96636764160,"available":96327098368,"internally_reserved":0,"allocated":1048576,"data_stored":519571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88300000000000001,"15min":0.85299999999999998},"min":{"1min":0.69799999999999995,"5min":0.60899999999999999,"15min":0.40100000000000002},"max":{"1min":1.534,"5min":1.661,"15min":1.661},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95299999999999996,"15min":0.95099999999999996},"min":{"1min":0.67700000000000005,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":1.4359999999999999,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.97899999999999998}]},{"osd":2,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93600000000000005,"15min":0.92600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4610000000000001,"5min":2.96,"15min":2.96},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0880000000000001,"15min":1.0580000000000001},"min":{"1min":0.91200000000000003,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":1.29,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1879999999999999,"15min":1.157},"min":{"1min":0.82299999999999995,"5min":0.72799999999999998,"15min":0.66100000000000003},"max":{"1min":1.5900000000000001,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.079},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0820000000000001,"15min":1.0780000000000001},"min":{"1min":0.81599999999999995,"5min":0.51200000000000001,"15min":0.46800000000000003},"max":{"1min":1.6020000000000001,"5min":2.407,"15min":2.407},"last":1.212}]},{"osd":4,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.1799999999999999,"15min":1.2},"min":{"1min":0.92800000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.544,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.139},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.341,"15min":1.3140000000000001},"min":{"1min":1.147,"5min":0.95999999999999996,"15min":0.64400000000000002},"max":{"1min":1.6339999999999999,"5min":3.04,"15min":3.04},"last":1.24}]},{"osd":5,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2869999999999999,"15min":1.284},"min":{"1min":0.96299999999999997,"5min":0.87,"15min":0.65100000000000002},"max":{"1min":1.724,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3029999999999999,"15min":1.302},"min":{"1min":0.96499999999999997,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.6919999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.4530000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4019999999999999,"15min":1.3939999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.72799999999999998},"max":{"1min":1.72,"5min":3.089,"15min":3.089},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3700000000000001,"15min":1.375},"min":{"1min":1.079,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.7410000000000001,"5min":2.46,"15min":2.6230000000000002},"last":1.589}]},{"osd":7,"last update":"Tue Dec 19 08:01:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.423,"15min":1.4390000000000001},"min":{"1min":1.1100000000000001,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":1.806,"5min":2.504,"15min":3.6080000000000001},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3779999999999999,"15min":1.413},"min":{"1min":1.0660000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7929999999999999,"5min":2.4590000000000001,"15min":4.101},"last":1.5329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575031,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302956,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068884,"statfs":{"total":96636764160,"available":96326537216,"internally_reserved":0,"allocated":1609728,"data_stored":1106656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.61799999999999999,"5min":0.48699999999999999,"15min":0.45800000000000002},"max":{"1min":1.1659999999999999,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91900000000000004,"15min":0.95799999999999996},"min":{"1min":0.65700000000000003,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.232,"5min":1.3260000000000001,"15min":6.0309999999999997},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.64200000000000002,"15min":0.629},"max":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.5800000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0489999999999999,"15min":1.03},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.504,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.0449999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.96099999999999997,"15min":0.95899999999999996},"min":{"1min":0.56200000000000006,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.671},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1279999999999999,"15min":1.1319999999999999},"min":{"1min":0.84199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.663,"5min":1.663,"15min":1.6719999999999999},"last":1.181}]},{"osd":4,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3140000000000001,"15min":1.3109999999999999},"min":{"1min":1.0149999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6479999999999999,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.3220000000000001,"15min":1.3140000000000001},"min":{"1min":1.004,"5min":0.85499999999999998,"15min":0.76100000000000001},"max":{"1min":1.7969999999999999,"5min":1.855,"15min":2.141},"last":1.3009999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3640000000000001,"15min":1.369},"min":{"1min":0.98599999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.9099999999999999,"5min":2.085,"15min":2.117},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.365,"15min":1.369},"min":{"1min":1.079,"5min":0.92300000000000004,"15min":0.80300000000000005},"max":{"1min":1.8,"5min":2.125,"15min":2.125},"last":1.2410000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.2929999999999999,"15min":1.274},"min":{"1min":0.93600000000000005,"5min":0.92000000000000004,"15min":0.77400000000000002},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1.006},{"interface":"front","average":{"1min":1.403,"5min":1.3100000000000001,"15min":1.329},"min":{"1min":1.0820000000000001,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.871,"5min":1.871,"15min":1.9530000000000001},"last":1.1200000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:01:41 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.345,"15min":1.331},"min":{"1min":0.99299999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.788,"5min":1.887,"15min":2.0539999999999998},"last":1.212},{"interface":"front","average":{"1min":1.419,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":1.0349999999999999,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.903,"5min":2.0059999999999998,"15min":2.016},"last":1.2729999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:02:28.337 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:02:28.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:02:28.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.553+0000 7fb4504f0700 1 Processor -- start 2023-12-19T08:02:28.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.553+0000 7fb4504f0700 1 -- start start 2023-12-19T08:02:28.560 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T08:02:28.557+0000 7f38717f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:02:28.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb4504f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb44806b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb4504f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480cbdf0 con 0x7fb4480cb970 2023-12-19T08:02:28.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb4504f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480cc0d0 con 0x7fb448158e20 2023-12-19T08:02:28.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb4504f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480cc3b0 con 0x7fb4480cecd0 2023-12-19T08:02:28.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb44da8b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb4480cb970 0x7fb44806acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41992/0 (socket says 172.21.15.111:41992) 2023-12-19T08:02:28.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.557+0000 7fb44da8b700 1 -- 172.21.15.111:0/4197835754 learned_addr learned my addr 172.21.15.111:0/4197835754 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44ea8d700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb44806b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339498370 0 0) 0x7fb4480cbdf0 con 0x7fb4480cb970 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434003570 con 0x7fb4480cb970 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079892610 0 0) 0x7fb4480cc0d0 con 0x7fb448158e20 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4480cbdf0 con 0x7fb448158e20 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292003408 0 0) 0x7fb434003570 con 0x7fb4480cb970 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 msgr2=0x7fb44806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb44806b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 >> v1:172.21.15.143:6789/0 conn(0x7fb448158e20 legacy=0x7fb4480ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480cc690 con 0x7fb4480cb970 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fb438001470 con 0x7fb4480cb970 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 >> v1:172.21.15.111:6789/0 conn(0x7fb4480cb970 legacy=0x7fb44806acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434003c20 0x7fb434004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434004810 0x7fb434004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb434005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480cc3b0 con 0x7fb434004810 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43400bbf0 con 0x7fb434003c20 2023-12-19T08:02:28.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43400bdb0 con 0x7fb4480cecd0 2023-12-19T08:02:28.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434004810 0x7fb434004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44e28c700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434003c20 0x7fb434004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44ea8d700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb434005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 -- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 msgr2=0x7fb434005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480cecd0 0x7fb434005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 -- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434003c20 msgr2=0x7fb434004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434003c20 0x7fb434004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 -- 172.21.15.111:0/4197835754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434003570 con 0x7fb434004810 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44da8b700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434004810 0x7fb434004c80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c1e0 tx=0x7fb438001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719d113eb73bb459 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43800cbe0 con 0x7fb434004810 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43800cd40 con 0x7fb434004810 2023-12-19T08:02:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.561+0000 7fb44d28a700 1 -- 172.21.15.111:0/4197835754 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438005b70 con 0x7fb434004810 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 -- 172.21.15.111:0/4197835754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434004810 msgr2=0x7fb434004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 --2- 172.21.15.111:0/4197835754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb434004810 0x7fb434004c80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c1e0 tx=0x7fb438001d40 comp rx=0 tx=0).stop 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb44ea8d700 1 -- 172.21.15.111:0/4197835754 reap_dead start 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 -- 172.21.15.111:0/4197835754 shutdown_connections 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 -- 172.21.15.111:0/4197835754 >> 172.21.15.111:0/4197835754 conn(0x7fb448067bb0 msgr2=0x7fb4480d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 -- 172.21.15.111:0/4197835754 shutdown_connections 2023-12-19T08:02:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 -- 172.21.15.111:0/4197835754 wait complete. 2023-12-19T08:02:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.565+0000 7fb4504f0700 1 Processor -- start 2023-12-19T08:02:28.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- start start 2023-12-19T08:02:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434004810 0x7fb44814cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 0x7fb44815db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b5af0 con 0x7fb4480b4820 2023-12-19T08:02:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b4ee0 con 0x7fb434004810 2023-12-19T08:02:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b51c0 con 0x7fb4480b43b0 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58104/0 (socket says 172.21.15.111:58104) 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 -- 172.21.15.111:0/335403193 learned_addr learned my addr 172.21.15.111:0/335403193 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44da8b700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 0x7fb44815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44e28c700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434004810 0x7fb44814cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 -- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 msgr2=0x7fb44815db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 0x7fb44815db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 -- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434004810 msgr2=0x7fb44814cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434004810 0x7fb44814cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 -- 172.21.15.111:0/335403193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480b54a0 con 0x7fb4480b4820 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44da8b700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 0x7fb44815db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb44ea8d700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb4400094d0 tx=0x7fb44000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e040 con 0x7fb4480b4820 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44000f040 con 0x7fb4480b4820 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480b5780 con 0x7fb4480b4820 2023-12-19T08:02:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000fe20 con 0x7fb4480b4820 2023-12-19T08:02:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.569+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4480c8920 con 0x7fb4480b4820 2023-12-19T08:02:28.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.573+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480b5910 con 0x7fb4480b4820 2023-12-19T08:02:28.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.573+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb440015640 con 0x7fb4480b4820 2023-12-19T08:02:28.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.577+0000 7fb43e7fc700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 0x7fb42c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:28.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.577+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fb44007d650 con 0x7fb4480b4820 2023-12-19T08:02:28.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.577+0000 7fb44e28c700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 0x7fb42c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:28.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.577+0000 7fb44e28c700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 0x7fb42c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb444003d70 tx=0x7fb444009040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:28.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.585+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb44004d630 con 0x7fb4480b4820 2023-12-19T08:02:28.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.809+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb448071120 con 0x7fb4480b4820 2023-12-19T08:02:28.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.821+0000 7fb43e7fc700 1 -- 172.21.15.111:0/335403193 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb440050c50 con 0x7fb4480b4820 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 msgr2=0x7fb42c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 0x7fb42c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb444003d70 tx=0x7fb444009040 comp rx=0 tx=0).stop 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 msgr2=0x7fb44815e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb4400094d0 tx=0x7fb44000b040 comp rx=0 tx=0).stop 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 shutdown_connections 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb42c060670 0x7fb42c062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4480b4820 0x7fb44815e070 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480b43b0 0x7fb44815db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 --2- 172.21.15.111:0/335403193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb434004810 0x7fb44814cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:28.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 >> 172.21.15.111:0/335403193 conn(0x7fb448067bb0 msgr2=0x7fb44806aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:28.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 shutdown_connections 2023-12-19T08:02:28.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:28.825+0000 7fb4504f0700 1 -- 172.21.15.111:0/335403193 wait complete. 2023-12-19T08:02:28.828 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:02:28.847 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:29.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.005+0000 7fdb44e6e700 1 Processor -- start 2023-12-19T08:02:29.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.009+0000 7fdb44e6e700 1 -- start start 2023-12-19T08:02:29.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.009+0000 7fdb44e6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb401556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.009+0000 7fdb44e6e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400c1b90 con 0x7fdb4006c7c0 2023-12-19T08:02:29.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.009+0000 7fdb44e6e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4006a9d0 con 0x7fdb401552f0 2023-12-19T08:02:29.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.009+0000 7fdb44e6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400c8710 con 0x7fdb400ae520 2023-12-19T08:02:29.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdb4006c7c0 0x7fdb4006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42008/0 (socket says 172.21.15.111:42008) 2023-12-19T08:02:29.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3ed9d700 1 -- 172.21.15.111:0/433807362 learned_addr learned my addr 172.21.15.111:0/433807362 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:29.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb35d9b700 1 --2- 172.21.15.111:0/433807362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb401556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463265288 0 0) 0x7fdb400c1b90 con 0x7fdb4006c7c0 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c003570 con 0x7fdb4006c7c0 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905194018 0 0) 0x7fdb4006a9d0 con 0x7fdb401552f0 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb400c1b90 con 0x7fdb401552f0 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb35d9b700 1 -- 172.21.15.111:0/433807362 >> v1:172.21.15.143:6789/0 conn(0x7fdb401552f0 legacy=0x7fdb400c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb35d9b700 1 -- 172.21.15.111:0/433807362 >> v1:172.21.15.111:6789/0 conn(0x7fdb4006c7c0 legacy=0x7fdb4006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb35d9b700 1 -- 172.21.15.111:0/433807362 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4006acb0 con 0x7fdb400ae520 2023-12-19T08:02:29.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb35d9b700 1 --2- 172.21.15.111:0/433807362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb401556c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800aac0 tx=0x7fdb2800a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=167ea0714cb4ce09 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800c970 con 0x7fdb400ae520 2023-12-19T08:02:29.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2800cad0 con 0x7fdb400ae520 2023-12-19T08:02:29.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb3dd9b700 1 -- 172.21.15.111:0/433807362 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800ce70 con 0x7fdb400ae520 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 -- 172.21.15.111:0/433807362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 msgr2=0x7fdb401556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/433807362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb401556c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800aac0 tx=0x7fdb2800a330 comp rx=0 tx=0).stop 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 -- 172.21.15.111:0/433807362 shutdown_connections 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/433807362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb401556c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 -- 172.21.15.111:0/433807362 >> 172.21.15.111:0/433807362 conn(0x7fdb400c8850 msgr2=0x7fdb400c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 -- 172.21.15.111:0/433807362 shutdown_connections 2023-12-19T08:02:29.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.013+0000 7fdb44e6e700 1 -- 172.21.15.111:0/433807362 wait complete. 2023-12-19T08:02:29.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.017+0000 7fdb44e6e700 1 Processor -- start 2023-12-19T08:02:29.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.017+0000 7fdb44e6e700 1 -- start start 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb400b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 0x7fdb400b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40169150 con 0x7fdb4006c7c0 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40169f40 con 0x7fdb401552f0 2023-12-19T08:02:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40169330 con 0x7fdb400ae520 2023-12-19T08:02:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58120/0 (socket says 172.21.15.111:58120) 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 -- 172.21.15.111:0/4200720474 learned_addr learned my addr 172.21.15.111:0/4200720474 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb35d9b700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb400b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3ed9d700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 0x7fdb400b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 -- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 msgr2=0x7fdb400b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb400b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 -- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 msgr2=0x7fdb400b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 0x7fdb400b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 -- 172.21.15.111:0/4200720474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2800a5d0 con 0x7fdb4006c7c0 2023-12-19T08:02:29.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3ed9d700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 0x7fdb400b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb3e59c700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000c5b0 tx=0x7fdb30007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30010510 con 0x7fdb4006c7c0 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb30010670 con 0x7fdb4006c7c0 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb40169610 con 0x7fdb4006c7c0 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb30010a30 con 0x7fdb4006c7c0 2023-12-19T08:02:29.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.021+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb40169bd0 con 0x7fdb4006c7c0 2023-12-19T08:02:29.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.025+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb40071120 con 0x7fdb4006c7c0 2023-12-19T08:02:29.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.025+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdb3001b400 con 0x7fdb4006c7c0 2023-12-19T08:02:29.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.029+0000 7fdb36ffd700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 0x7fdb20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.029+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fdb3007c910 con 0x7fdb4006c7c0 2023-12-19T08:02:29.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.029+0000 7fdb35d9b700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 0x7fdb20062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.029+0000 7fdb35d9b700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 0x7fdb20062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800aa10 tx=0x7fdb2801a000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.033+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdb3004ca00 con 0x7fdb4006c7c0 2023-12-19T08:02:29.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.241+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb400642a0 con 0x7fdb4006c7c0 2023-12-19T08:02:29.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.241+0000 7fdb36ffd700 1 -- 172.21.15.111:0/4200720474 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7fdb3007ae60 con 0x7fdb4006c7c0 2023-12-19T08:02:29.243 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:29.244 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 msgr2=0x7fdb20062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 0x7fdb20062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800aa10 tx=0x7fdb2801a000 comp rx=0 tx=0).stop 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 msgr2=0x7fdb400aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000c5b0 tx=0x7fdb30007e10 comp rx=0 tx=0).stop 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 shutdown_connections 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdb20060670 0x7fdb20062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb401552f0 0x7fdb400b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb400ae520 0x7fdb400b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 --2- 172.21.15.111:0/4200720474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4006c7c0 0x7fdb400aceb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 >> 172.21.15.111:0/4200720474 conn(0x7fdb400c8850 msgr2=0x7fdb400cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 shutdown_connections 2023-12-19T08:02:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.245+0000 7fdb44e6e700 1 -- 172.21.15.111:0/4200720474 wait complete. 2023-12-19T08:02:29.250 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:29.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:02:23.022271+0000', 'modified': '2023-12-19T08:02:26.633728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6429}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6429': {'gid': 6429, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/82065769', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6833', 'nonce': 82065769}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 82065769}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:02:29.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:02:29.670 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T08:02:29.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.833+0000 7fdef546e700 1 Processor -- start 2023-12-19T08:02:29.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.833+0000 7fdef546e700 1 -- start start 2023-12-19T08:02:29.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdef546e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdef0156c50 0x7fdef00ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdef546e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef006dd10 con 0x7fdef0156880 2023-12-19T08:02:29.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdef546e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef00ae050 con 0x7fdef00adc20 2023-12-19T08:02:29.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdef546e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef00c7300 con 0x7fdef0156c50 2023-12-19T08:02:29.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeee7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdef0156c50 0x7fdef00ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeeeffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdef0156880 0x7fdef00c6880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42012/0 (socket says 172.21.15.111:42012) 2023-12-19T08:02:29.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeeeffd700 1 -- 172.21.15.111:0/2883661414 learned_addr learned my addr 172.21.15.111:0/2883661414 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:29.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeee7fc700 1 -- 172.21.15.111:0/2883661414 >> v1:172.21.15.143:6789/0 conn(0x7fdef00adc20 legacy=0x7fdef006a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeee7fc700 1 -- 172.21.15.111:0/2883661414 >> v1:172.21.15.111:6789/0 conn(0x7fdef0156880 legacy=0x7fdef00c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeee7fc700 1 -- 172.21.15.111:0/2883661414 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef006d3e0 con 0x7fdef0156c50 2023-12-19T08:02:29.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeedffb700 1 -- 172.21.15.111:0/2883661414 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588328795 0 0) 0x7fdef006dd10 con 0x7fdef0156880 2023-12-19T08:02:29.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.837+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/2883661414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef00ad6e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5195f0cd5219a1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdeedffb700 1 -- 172.21.15.111:0/2883661414 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4009e80 con 0x7fdef0156c50 2023-12-19T08:02:29.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdeedffb700 1 -- 172.21.15.111:0/2883661414 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee400b860 con 0x7fdef0156c50 2023-12-19T08:02:29.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdeedffb700 1 -- 172.21.15.111:0/2883661414 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee400bbe0 con 0x7fdef0156c50 2023-12-19T08:02:29.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 -- 172.21.15.111:0/2883661414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 msgr2=0x7fdef00ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 --2- 172.21.15.111:0/2883661414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef00ad6e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee40091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 -- 172.21.15.111:0/2883661414 shutdown_connections 2023-12-19T08:02:29.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 --2- 172.21.15.111:0/2883661414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef00ad6e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 -- 172.21.15.111:0/2883661414 >> 172.21.15.111:0/2883661414 conn(0x7fdef0067bb0 msgr2=0x7fdef00cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 -- 172.21.15.111:0/2883661414 shutdown_connections 2023-12-19T08:02:29.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 -- 172.21.15.111:0/2883661414 wait complete. 2023-12-19T08:02:29.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.841+0000 7fdef546e700 1 Processor -- start 2023-12-19T08:02:29.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 -- start start 2023-12-19T08:02:29.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef00adc20 0x7fdef00b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0156880 0x7fdef01532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00b1650 con 0x7fdef0156c50 2023-12-19T08:02:29.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00c9af0 con 0x7fdef00adc20 2023-12-19T08:02:29.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00c9dd0 con 0x7fdef0156880 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58126/0 (socket says 172.21.15.111:58126) 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeeeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef00adc20 0x7fdef00b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 -- 172.21.15.111:0/1058226841 learned_addr learned my addr 172.21.15.111:0/1058226841 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0156880 0x7fdef01532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 -- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0156880 msgr2=0x7fdef01532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0156880 0x7fdef01532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 -- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef00adc20 msgr2=0x7fdef00b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef00adc20 0x7fdef00b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 -- 172.21.15.111:0/1058226841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4009420 con 0x7fdef0156c50 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c8e0 tx=0x7fdee8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fded6ffd700 1 -- 172.21.15.111:0/1058226841 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8012630 con 0x7fdef0156c50 2023-12-19T08:02:29.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fded6ffd700 1 -- 172.21.15.111:0/1058226841 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8012790 con 0x7fdef0156c50 2023-12-19T08:02:29.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef00ca0b0 con 0x7fdef0156c50 2023-12-19T08:02:29.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.845+0000 7fded6ffd700 1 -- 172.21.15.111:0/1058226841 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8012b50 con 0x7fdef0156c50 2023-12-19T08:02:29.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.849+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef0153800 con 0x7fdef0156c50 2023-12-19T08:02:29.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.849+0000 7fded6ffd700 1 -- 172.21.15.111:0/1058226841 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdee801a3a0 con 0x7fdef0156c50 2023-12-19T08:02:29.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fded6ffd700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 0x7fdedc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fded6ffd700 1 -- 172.21.15.111:0/1058226841 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fdee807cea0 con 0x7fdef0156c50 2023-12-19T08:02:29.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 0x7fdedc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 0x7fdedc062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdee0007fd0 tx=0x7fdee0007d10 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fdef7724740 1 Processor -- start 2023-12-19T08:02:29.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.853+0000 7fdef7724740 1 -- start start 2023-12-19T08:02:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2516f30 con 0x23b6790 2023-12-19T08:02:29.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2512ed0 con 0x236bae0 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x24d90b0 con 0x236b710 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58142/0 (socket says 172.21.15.111:58142) 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 -- 172.21.15.111:0/2671210386 learned_addr learned my addr 172.21.15.111:0/2671210386 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 -- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 msgr2=0x25e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 -- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 msgr2=0x25e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 -- 172.21.15.111:0/2671210386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23b79e0 con 0x23b6790 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:29.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdee807ae80 tx=0x7fdee8050ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ad3f22f764d79b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdecffff700 1 -- 172.21.15.111:0/2671210386 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8066100 con 0x23b6790 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdecffff700 1 -- 172.21.15.111:0/2671210386 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee804ed90 con 0x23b6790 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdecffff700 1 -- 172.21.15.111:0/2671210386 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee804d360 con 0x23b6790 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- 172.21.15.111:0/2671210386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 msgr2=0x25e9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fdee807ae80 tx=0x7fdee8050ab0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- 172.21.15.111:0/2671210386 shutdown_connections 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25e9980 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 --2- 172.21.15.111:0/2671210386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- 172.21.15.111:0/2671210386 >> 172.21.15.111:0/2671210386 conn(0x236beb0 msgr2=0x25a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- 172.21.15.111:0/2671210386 shutdown_connections 2023-12-19T08:02:29.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.857+0000 7fdef7724740 1 -- 172.21.15.111:0/2671210386 wait complete. 2023-12-19T08:02:29.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 -- start start 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 Processor -- start 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25ed840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23b6790 0x25cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x24d8d50 con 0x236b710 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x24b1970 con 0x236bae0 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x25126e0 con 0x23b6790 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdef7724740 10 client.0 _ll_register_callbacks cb 0x255c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58146/0 (socket says 172.21.15.111:58146) 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 -- 172.21.15.111:0/76181071 learned_addr learned my addr 172.21.15.111:0/76181071 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23b6790 0x25cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25ed840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 -- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23b6790 msgr2=0x25cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23b6790 0x25cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 -- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 msgr2=0x25ed840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25ed840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 -- 172.21.15.111:0/76181071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef0153800 con 0x236b710 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdee0014500 tx=0x7fdee0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c20d246ad2368b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.861+0000 7fdecd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0014b50 con 0x236b710 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fdee0014cb0 con 0x236b710 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x24b1cc0 con 0x236b710 2023-12-19T08:02:29.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee0014e70 con 0x236b710 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 10 client.6570 fetch_fsmap learned FSMap version 32 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x24e1370 con 0x236b710 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x24b1cc0 con 0x236b710 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdee0006450 con 0x236b710 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 10 client.6570 fetch_fsmap finished waiting for FSMap version 32 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 10 client.6570 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 20 client.6570 populate_metadata read hostname 'smithi111' 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25d0370 0x25f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 4 client.6570 mds_command: new command op to 6429 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 --> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x24b53e0 con 0x25d0370 2023-12-19T08:02:29.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25d0370 0x25f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 10 client.6570 ms_handle_connect on v2:172.21.15.111:6833/82065769 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25d0370 0x25f84a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185f1b413670c016 server_cookie=afaf2a5aeb5190b4 in_seq=0 out_seq=0 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 1 -- 172.21.15.111:0/76181071 <== mds.0 v2:172.21.15.111:6833/82065769 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x24b53e0 con 0x25d0370 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.865+0000 7fdecd7fa700 10 client.6570 handle_command_reply: tid=0 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 client.6570 shutdown 2023-12-19T08:02:29.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 msgr2=0x25ed300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdee0014500 tx=0x7fdee0015000 comp rx=0 tx=0).stop 2023-12-19T08:02:29.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 shutdown_connections 2023-12-19T08:02:29.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25d0370 0x25f84a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25ed840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23b6790 0x25cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 --2- 172.21.15.111:0/76181071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x236b710 0x25ed300 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 >> 172.21.15.111:0/76181071 conn(0x236beb0 msgr2=0x259df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdecd7fa700 0 client.6570 ms_handle_reset on v2:172.21.15.111:6833/82065769 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 shutdown_connections 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 1 -- 172.21.15.111:0/76181071 wait complete. 2023-12-19T08:02:29.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.869+0000 7fdef7724740 20 client.6570 trim_cache size 0 max 16384 2023-12-19T08:02:29.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 Processor -- start 2023-12-19T08:02:29.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 -- start start 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236b710 0x25cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x259fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x23e8650 con 0x23b6790 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x24b6340 con 0x236b710 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x2510890 con 0x236bae0 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x259fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.889+0000 7fdeee7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeee7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:40440/0 (socket says 172.21.15.111:40440) 2023-12-19T08:02:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeee7fc700 1 -- 172.21.15.111:0/163067317 learned_addr learned my addr 172.21.15.111:0/163067317 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236b710 0x25cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 -- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 msgr2=0x25cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 -- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236b710 msgr2=0x25cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236b710 0x25cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 -- 172.21.15.111:0/163067317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24c7d80 con 0x23b6790 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x259fad0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdee8050950 tx=0x7fdee8069000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4ead051b4af587 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fded4ff9700 1 -- 172.21.15.111:0/163067317 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8066100 con 0x23b6790 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fded4ff9700 1 -- 172.21.15.111:0/163067317 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee804cf90 con 0x23b6790 2023-12-19T08:02:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fded4ff9700 1 -- 172.21.15.111:0/163067317 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8002440 con 0x23b6790 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- 172.21.15.111:0/163067317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 msgr2=0x259fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x259fad0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdee8050950 tx=0x7fdee8069000 comp rx=0 tx=0).stop 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- 172.21.15.111:0/163067317 shutdown_connections 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x259fad0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236bae0 0x25cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 --2- 172.21.15.111:0/163067317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236b710 0x25cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- 172.21.15.111:0/163067317 >> 172.21.15.111:0/163067317 conn(0x236beb0 msgr2=0x25e8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- 172.21.15.111:0/163067317 shutdown_connections 2023-12-19T08:02:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- 172.21.15.111:0/163067317 wait complete. 2023-12-19T08:02:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.893+0000 7fdef7724740 1 -- start start 2023-12-19T08:02:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 Processor -- start 2023-12-19T08:02:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x24c1c90 con 0x23b6790 2023-12-19T08:02:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x24e7610 con 0x236bae0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x23e47c0 con 0x236b710 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 10 client.0 _ll_register_callbacks cb 0x23379d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58156/0 (socket says 172.21.15.111:58156) 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 -- 172.21.15.111:0/3867224459 learned_addr learned my addr 172.21.15.111:0/3867224459 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeee7fc700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 -- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 msgr2=0x25ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 -- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 msgr2=0x25a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 -- 172.21.15.111:0/3867224459 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef006d3e0 con 0x23b6790 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdeef7fe700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdee8057df0 tx=0x7fdee8079040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297744f704c5023b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdecd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8066100 con 0x23b6790 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fdee806e020 con 0x23b6790 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdee804ed90 con 0x23b6790 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee8002440 con 0x23b6790 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 10 client.6576 fetch_fsmap learned FSMap version 32 2023-12-19T08:02:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x23b79e0 con 0x23b6790 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.897+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee804ed90 con 0x23b6790 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fdee804d360 con 0x23b6790 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 10 client.6576 fetch_fsmap finished waiting for FSMap version 32 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 10 client.6576 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 20 client.6576 populate_metadata read hostname 'smithi111' 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25a2d30 0x2642320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 4 client.6576 mds_command: new command op to 6429 tid=0[{"prefix": "status"}] 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 --> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] -- command(tid 0: {"prefix": "status"}) v1 -- 0x24e7410 con 0x25a2d30 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25a2d30 0x2642320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdeeeffd700 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25a2d30 0x2642320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e2a38e193743df server_cookie=a9b37fa7e83aa71 in_seq=0 out_seq=0 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdecd7fa700 10 client.6576 ms_handle_connect on v2:172.21.15.111:6833/82065769 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdecd7fa700 1 -- 172.21.15.111:0/3867224459 <== mds.0 v2:172.21.15.111:6833/82065769 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x24e7410 con 0x25a2d30 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdecd7fa700 10 client.6576 handle_command_reply: tid=0 2023-12-19T08:02:29.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 client.6576 shutdown 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 msgr2=0x25a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdee8057df0 tx=0x7fdee8079040 comp rx=0 tx=0).stop 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 shutdown_connections 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:6833/82065769,v1:172.21.15.111:6835/82065769] conn(0x25a2d30 0x2642320 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x236bae0 0x25a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdecd7fa700 0 client.6576 ms_handle_reset on v2:172.21.15.111:6833/82065769 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23b6790 0x25a27f0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 --2- 172.21.15.111:0/3867224459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x236b710 0x25ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 >> 172.21.15.111:0/3867224459 conn(0x236beb0 msgr2=0x25e8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 shutdown_connections 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 1 -- 172.21.15.111:0/3867224459 wait complete. 2023-12-19T08:02:29.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef7724740 20 client.6576 trim_cache size 0 max 16384 2023-12-19T08:02:29.912 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:02:29.912 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:02:29.912 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 6429, 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.3021365950000003, 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 32, 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 85, 2023-12-19T08:02:29.913 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 85, 2023-12-19T08:02:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.224704137 2023-12-19T08:02:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 msgr2=0x7fdedc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 0x7fdedc062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdee0007fd0 tx=0x7fdee0007d10 comp rx=0 tx=0).stop 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 msgr2=0x7fdef016ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.901+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c8e0 tx=0x7fdee8001710 comp rx=0 tx=0).stop 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 shutdown_connections 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdedc0606a0 0x7fdedc062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef0156c50 0x7fdef016ba20 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0156880 0x7fdef01532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 --2- 172.21.15.111:0/1058226841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef00adc20 0x7fdef00b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 >> 172.21.15.111:0/1058226841 conn(0x7fdef0067bb0 msgr2=0x7fdef006b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 shutdown_connections 2023-12-19T08:02:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:29.905+0000 7fdef546e700 1 -- 172.21.15.111:0/1058226841 wait complete. 2023-12-19T08:02:29.926 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:02:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 Processor -- start 2023-12-19T08:02:30.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 -- start start 2023-12-19T08:02:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4960156bb0 0x7f49600ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 0x7f49600afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496015b940 con 0x7f49600b01b0 2023-12-19T08:02:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496015aa50 con 0x7f4960156bb0 2023-12-19T08:02:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f4964f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496015ad30 con 0x7f49600ad3d0 2023-12-19T08:02:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f495dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 0x7f49600afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f495e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4960156bb0 0x7f49600ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f495e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4960156bb0 0x7f49600ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:54834/0 (socket says 172.21.15.111:54834) 2023-12-19T08:02:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f495e59c700 1 -- 172.21.15.111:0/2840237236 learned_addr learned my addr 172.21.15.111:0/2840237236 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.097+0000 7f495dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 0x7f49600afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:40460/0 (socket says 172.21.15.111:40460) 2023-12-19T08:02:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495e59c700 1 -- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 msgr2=0x7f49600afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495e59c700 1 --2- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 0x7f49600afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495e59c700 1 -- 172.21.15.111:0/2840237236 >> v1:172.21.15.143:6789/0 conn(0x7f49600b01b0 legacy=0x7f496006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495e59c700 1 -- 172.21.15.111:0/2840237236 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496015b010 con 0x7f4960156bb0 2023-12-19T08:02:30.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495e59c700 1 --2- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600ace90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4950009f20 tx=0x7f4950009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d24997a874242e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:30.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495d59a700 1 -- 172.21.15.111:0/2840237236 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495000e070 con 0x7f4960156bb0 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495d59a700 1 -- 172.21.15.111:0/2840237236 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495000b9d0 con 0x7f4960156bb0 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f495d59a700 1 -- 172.21.15.111:0/2840237236 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500092e0 con 0x7f4960156bb0 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 -- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 msgr2=0x7f49600ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 --2- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600ace90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4950009f20 tx=0x7f4950009840 comp rx=0 tx=0).stop 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 -- 172.21.15.111:0/2840237236 shutdown_connections 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 --2- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600ad3d0 0x7f49600afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 --2- 172.21.15.111:0/2840237236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600ace90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.101+0000 7f4964f85700 1 -- 172.21.15.111:0/2840237236 >> 172.21.15.111:0/2840237236 conn(0x7f49600ca9d0 msgr2=0x7f49600cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.105+0000 7f4964f85700 1 -- 172.21.15.111:0/2840237236 shutdown_connections 2023-12-19T08:02:30.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.105+0000 7f4964f85700 1 -- 172.21.15.111:0/2840237236 wait complete. 2023-12-19T08:02:30.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.105+0000 7f4964f85700 1 Processor -- start 2023-12-19T08:02:30.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.105+0000 7f4964f85700 1 -- start start 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600b01b0 0x7f496014bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49600b2270 con 0x7f49600ad3d0 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49600b1660 con 0x7f4960156bb0 2023-12-19T08:02:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49600b1940 con 0x7f49600b01b0 2023-12-19T08:02:30.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58164/0 (socket says 172.21.15.111:58164) 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 -- 172.21.15.111:0/1135746924 learned_addr learned my addr 172.21.15.111:0/1135746924 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495dd9b700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600b01b0 0x7f496014bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495ed9d700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 -- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600b01b0 msgr2=0x7f496014bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600b01b0 0x7f496014bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 -- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 msgr2=0x7f49600b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 -- 172.21.15.111:0/1135746924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4950009ae0 con 0x7f49600ad3d0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f495e59c700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f495000b610 tx=0x7f4950018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950008a20 con 0x7f49600ad3d0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4950008b80 con 0x7f49600ad3d0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49600b1c20 con 0x7f49600ad3d0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950026460 con 0x7f49600ad3d0 2023-12-19T08:02:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.109+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496014c1e0 con 0x7f49600ad3d0 2023-12-19T08:02:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.113+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4960071120 con 0x7f49600ad3d0 2023-12-19T08:02:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.113+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4950003040 con 0x7f49600ad3d0 2023-12-19T08:02:30.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.113+0000 7f494e7fc700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 0x7f4940062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:30.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.113+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f495007ef30 con 0x7f49600ad3d0 2023-12-19T08:02:30.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.117+0000 7f495dd9b700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 0x7f4940062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:30.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.117+0000 7f495dd9b700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 0x7f4940062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49600c8a90 tx=0x7f49540091a0 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:30.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.121+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f495004ef10 con 0x7f49600ad3d0 2023-12-19T08:02:30.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.325+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f49600c8500 con 0x7f49600ad3d0 2023-12-19T08:02:30.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f494e7fc700 1 -- 172.21.15.111:0/1135746924 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4950052530 con 0x7f49600ad3d0 2023-12-19T08:02:30.330 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:02:30.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 msgr2=0x7f4940062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 0x7f4940062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49600c8a90 tx=0x7f49540091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 msgr2=0x7f49600c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:30.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f495000b610 tx=0x7f4950018000 comp rx=0 tx=0).stop 2023-12-19T08:02:30.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 shutdown_connections 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4940060670 0x7f4940062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4960156bb0 0x7f49600b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49600b01b0 0x7f496014bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 --2- 172.21.15.111:0/1135746924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49600ad3d0 0x7f49600c4f70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 >> 172.21.15.111:0/1135746924 conn(0x7f49600ca9d0 msgr2=0x7f496014fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 shutdown_connections 2023-12-19T08:02:30.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:30.329+0000 7f4964f85700 1 -- 172.21.15.111:0/1135746924 wait complete. 2023-12-19T08:02:30.351 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:02:30.351 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:02:30.351 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:02:30.352 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:02:30.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.352 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.372 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41507sec preferred_lft 41507sec 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:02:30.373 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:02:30.374 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:02:30.375 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:02:30.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.376 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:02:30.448 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:02:30.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.450 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:02:30.511 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:02:30.511 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:02:30.511 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:02:30.512 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:02:30.512 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:02:30.512 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:02:30.512 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:02:30.512 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:02:30.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:02:30.513 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:02:30.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.646 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:02:30.663 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:02:30.664 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:02:30.665 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:02:30.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:30.665 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:02:30.748 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:02:30.748 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:02:30.748 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:30.806 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:02:30.806 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:02:30.806 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:30.864 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:02:30.942 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:02:30.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:02:31.018 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:02:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:02:31.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:31.022 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:02:31.087 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:02:31.088 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:02:31.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:02:31.174 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:02:31.175 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:02:31.251 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:02:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:02:31.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:31.254 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:02:31.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:02:31.281+0000 7ff2ce904380 -1 init, newargv = 0x5615dced5fe0 newargc=15 2023-12-19T08:02:31.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[49029]: starting ceph client 2023-12-19T08:02:31.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[49029]: starting fuse 2023-12-19T08:02:31.298 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:02:31.299 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:02:31.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:02:31.299 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:02:31.299 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:02:31.300 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:02:31.301 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:02:31.467 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.49029.asok 2023-12-19T08:02:31.646 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49029.asok 2023-12-19T08:02:31.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:31.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49029.asok status 2023-12-19T08:02:31.833 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "49029", 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 6585, 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:02:31.834 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 6585 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2572993973 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2572993973 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:31.835 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.6585 192.168.0.1:0/2572993973", 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2572993973", 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 32, 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 85, 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:02:31.836 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:02:31.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:31.847 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:02:31.847 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:31.866 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:02:31.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:31.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:31.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:31.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:02:32.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.157+0000 7ff945e16700 1 Processor -- start 2023-12-19T08:02:32.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.157+0000 7ff945e16700 1 -- start start 2023-12-19T08:02:32.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.157+0000 7ff945e16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9400c6900 0x7ff9400c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.157+0000 7ff945e16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff94006b900 0x7ff94006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94006b470 con 0x7ff94015cc20 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94015d7b0 con 0x7ff94006b900 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff940161d10 con 0x7ff9400c6900 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945615700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff94015cc20 0x7ff94015d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38174/0 (socket says 172.21.15.111:38174) 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945615700 1 -- 172.21.15.111:0/1374408097 learned_addr learned my addr 172.21.15.111:0/1374408097 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 --2- 172.21.15.111:0/1374408097 >> v2:172.21.15.111:3300/0 conn(0x7ff94006b900 0x7ff94006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff944e14700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9400c6900 0x7ff9400c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102212212 0 0) 0x7ff94006b470 con 0x7ff94015cc20 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003570 con 0x7ff94015cc20 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391650301 0 0) 0x7ff930003570 con 0x7ff94015cc20 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 -- 172.21.15.111:0/1374408097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9400c6900 msgr2=0x7ff9400c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9400c6900 0x7ff9400c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 -- 172.21.15.111:0/1374408097 >> v1:172.21.15.111:6789/0 conn(0x7ff94015cc20 legacy=0x7ff94015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 -- 172.21.15.111:0/1374408097 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94006bd70 con 0x7ff94006b900 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93ffff700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94006da00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009910 tx=0x7ff92c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df3795f5f7da68a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9380017f0 con 0x7ff94006b900 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c011420 con 0x7ff94006b900 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff93f7fe700 1 -- 172.21.15.111:0/1374408097 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c0117c0 con 0x7ff94006b900 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- 172.21.15.111:0/1374408097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 msgr2=0x7ff94006da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94006da00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009910 tx=0x7ff92c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- 172.21.15.111:0/1374408097 shutdown_connections 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94006da00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 --2- 172.21.15.111:0/1374408097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9400c6900 0x7ff9400c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- 172.21.15.111:0/1374408097 >> 172.21.15.111:0/1374408097 conn(0x7ff940067bb0 msgr2=0x7ff9400cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:32.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.161+0000 7ff945e16700 1 -- 172.21.15.111:0/1374408097 shutdown_connections 2023-12-19T08:02:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.165+0000 7ff945e16700 1 -- 172.21.15.111:0/1374408097 wait complete. 2023-12-19T08:02:32.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.165+0000 7ff945e16700 1 Processor -- start 2023-12-19T08:02:32.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.165+0000 7ff945e16700 1 -- start start 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 0x7ff94015e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94015cc20 0x7ff9400afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94015f400 con 0x7ff94006b900 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401591f0 con 0x7ff9400c6900 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94015e7f0 con 0x7ff94015cc20 2023-12-19T08:02:32.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35400/0 (socket says 172.21.15.111:35400) 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 -- 172.21.15.111:0/3410063012 learned_addr learned my addr 172.21.15.111:0/3410063012 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff93ffff700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 0x7ff94015e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945615700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94015cc20 0x7ff9400afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 -- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94015cc20 msgr2=0x7ff9400afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94015cc20 0x7ff9400afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 -- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 msgr2=0x7ff94015e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 0x7ff94015e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 -- 172.21.15.111:0/3410063012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c009420 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff93ffff700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 0x7ff94015e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff944e14700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff93400ec80 tx=0x7ff9340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93400c9d0 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93400cb30 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934013470 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94015ead0 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94015f090 con 0x7ff94006b900 2023-12-19T08:02:32.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.169+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94004be40 con 0x7ff94006b900 2023-12-19T08:02:32.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.173+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff934013820 con 0x7ff94006b900 2023-12-19T08:02:32.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.173+0000 7ff93cff9700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 0x7ff924062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.173+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7ff93407d0b0 con 0x7ff94006b900 2023-12-19T08:02:32.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.173+0000 7ff93ffff700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 0x7ff924062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.177+0000 7ff93ffff700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 0x7ff924062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0093f0 tx=0x7ff92c016000 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.181+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff93404d1a0 con 0x7ff94006b900 2023-12-19T08:02:32.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.377+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff94015f540 con 0x7ff9240606a0 2023-12-19T08:02:32.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.381+0000 7ff93cff9700 1 -- 172.21.15.111:0/3410063012 <== mgr.6151 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff92c011b30 con 0x7ff9240606a0 2023-12-19T08:02:32.384 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:02:32.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:32.384 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:02:32.384 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:32.384 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 msgr2=0x7ff924062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 0x7ff924062b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0093f0 tx=0x7ff92c016000 comp rx=0 tx=0).stop 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 msgr2=0x7ff94015dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff93400ec80 tx=0x7ff9340091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 shutdown_connections 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff9240606a0 0x7ff924062b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9400c6900 0x7ff94015e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff94015cc20 0x7ff9400afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 --2- 172.21.15.111:0/3410063012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff94006b900 0x7ff94015dc50 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 >> 172.21.15.111:0/3410063012 conn(0x7ff940067bb0 msgr2=0x7ff940159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:32.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 shutdown_connections 2023-12-19T08:02:32.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.385+0000 7ff945e16700 1 -- 172.21.15.111:0/3410063012 wait complete. 2023-12-19T08:02:32.408 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:02:32.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.585+0000 7fbf3062e700 1 Processor -- start 2023-12-19T08:02:32.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.585+0000 7fbf3062e700 1 -- start start 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 0x7fbf2806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbf28158ad0 0x7fbf28158f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf280ab6c0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf280ac240 con 0x7fbf28158ad0 2023-12-19T08:02:32.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf3062e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280ab910 con 0x7fbf28069a20 2023-12-19T08:02:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2e3ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 0x7fbf2806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2ebcb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbf28158ad0 0x7fbf28158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:50466/0 (socket says 172.21.15.111:50466) 2023-12-19T08:02:32.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2755685318 learned_addr learned my addr 172.21.15.111:0/2755685318 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2755685318 >> v2:172.21.15.143:3300/0 conn(0x7fbf28158ad0 msgr2=0x7fbf28158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2755685318 >> v2:172.21.15.143:3300/0 conn(0x7fbf28158ad0 0x7fbf28158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2755685318 >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 msgr2=0x7fbf2806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2755685318 >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 0x7fbf2806d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2755685318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280abbf0 con 0x7fbf28069a20 2023-12-19T08:02:32.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2e3ca700 1 --2- 172.21.15.111:0/2755685318 >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 0x7fbf2806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:32.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2755685318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806a9a0 tx=0x7fbf18009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38009321fb75aa92 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2d3c8700 1 -- 172.21.15.111:0/2755685318 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1800b640 con 0x7fbf28069a20 2023-12-19T08:02:32.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2d3c8700 1 -- 172.21.15.111:0/2755685318 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1800b7a0 con 0x7fbf28069a20 2023-12-19T08:02:32.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.589+0000 7fbf2d3c8700 1 -- 172.21.15.111:0/2755685318 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1800bb60 con 0x7fbf28069a20 2023-12-19T08:02:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- 172.21.15.111:0/2755685318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 msgr2=0x7fbf280b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2755685318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806a9a0 tx=0x7fbf18009250 comp rx=0 tx=0).stop 2023-12-19T08:02:32.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- 172.21.15.111:0/2755685318 shutdown_connections 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2755685318 >> v2:172.21.15.143:3300/0 conn(0x7fbf28158ad0 0x7fbf28158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2755685318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28069a20 0x7fbf280b28d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2755685318 >> v2:172.21.15.111:3300/0 conn(0x7fbf2806cfa0 0x7fbf2806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- 172.21.15.111:0/2755685318 >> 172.21.15.111:0/2755685318 conn(0x7fbf280cd870 msgr2=0x7fbf28069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- 172.21.15.111:0/2755685318 shutdown_connections 2023-12-19T08:02:32.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- 172.21.15.111:0/2755685318 wait complete. 2023-12-19T08:02:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 Processor -- start 2023-12-19T08:02:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.593+0000 7fbf3062e700 1 -- start start 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf28069a20 0x7fbf28153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 0x7fbf28156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28169eb0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28159a10 con 0x7fbf28069a20 2023-12-19T08:02:32.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28169580 con 0x7fbf28158ad0 2023-12-19T08:02:32.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35426/0 (socket says 172.21.15.111:35426) 2023-12-19T08:02:32.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2762255277 learned_addr learned my addr 172.21.15.111:0/2762255277 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:32.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2e3ca700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf28069a20 0x7fbf28153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2ebcb700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 0x7fbf28156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 msgr2=0x7fbf28156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 0x7fbf28156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf28069a20 msgr2=0x7fbf28153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf28069a20 0x7fbf28153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 -- 172.21.15.111:0/2762255277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24009420 con 0x7fbf2806cfa0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2ebcb700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 0x7fbf28156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf2dbc9700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbf280aba50 tx=0x7fbf18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18000d10 con 0x7fbf2806cfa0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1800bb20 con 0x7fbf2806cfa0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf180094f0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18011670 con 0x7fbf2806cfa0 2023-12-19T08:02:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.597+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf28169b40 con 0x7fbf2806cfa0 2023-12-19T08:02:32.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.601+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf280ae1d0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.601+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbf180119e0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.601+0000 7fbf1e7fc700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 0x7fbf0c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:32.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.605+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fbf1807e430 con 0x7fbf2806cfa0 2023-12-19T08:02:32.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.605+0000 7fbf2e3ca700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 0x7fbf0c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:32.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.605+0000 7fbf2e3ca700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 0x7fbf0c062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf240093f0 tx=0x7fbf24008040 comp rx=0 tx=0).ready entity=mgr.6151 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:32.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.609+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf280ae1d0 con 0x7fbf2806cfa0 2023-12-19T08:02:32.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:32.821+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbf280ae1d0 con 0x7fbf2806cfa0 2023-12-19T08:02:33.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.457+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fbf1804e990 con 0x7fbf2806cfa0 2023-12-19T08:02:33.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.461+0000 7fbf1e7fc700 1 -- 172.21.15.111:0/2762255277 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v27) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbf18051be0 con 0x7fbf2806cfa0 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 msgr2=0x7fbf0c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 0x7fbf0c062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf240093f0 tx=0x7fbf24008040 comp rx=0 tx=0).stop 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 msgr2=0x7fbf281594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbf280aba50 tx=0x7fbf18017000 comp rx=0 tx=0).stop 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 shutdown_connections 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf0c060670 0x7fbf0c062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf28158ad0 0x7fbf28156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf2806cfa0 0x7fbf281594d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 --2- 172.21.15.111:0/2762255277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf28069a20 0x7fbf28153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 >> 172.21.15.111:0/2762255277 conn(0x7fbf280cd870 msgr2=0x7fbf280adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:33.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 shutdown_connections 2023-12-19T08:02:33.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.465+0000 7fbf3062e700 1 -- 172.21.15.111:0/2762255277 wait complete. 2023-12-19T08:02:33.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:02:33.640 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:33.637+0000 7f741b147000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:33.646 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:33.646+0000 7f7949564000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.653+0000 7f37c95cc700 1 Processor -- start 2023-12-19T08:02:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.653+0000 7f37c95cc700 1 -- start start 2023-12-19T08:02:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c95cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c4158c90 0x7f37c40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c95cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f37c4159770 0x7f37c406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c95cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c406c850 con 0x7f37c406aeb0 2023-12-19T08:02:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c95cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c406b470 con 0x7f37c4159770 2023-12-19T08:02:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c95cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c406b750 con 0x7f37c4158c90 2023-12-19T08:02:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c37fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f37c406aeb0 0x7f37c4159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56162/0 (socket says 172.21.15.111:56162) 2023-12-19T08:02:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c37fe700 1 -- 172.21.15.111:0/3879366897 learned_addr learned my addr 172.21.15.111:0/3879366897 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 --2- 172.21.15.111:0/3879366897 >> v2:172.21.15.143:3300/0 conn(0x7f37c4159770 0x7f37c406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c3fff700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c4158c90 0x7f37c40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c2ffd700 1 -- 172.21.15.111:0/3879366897 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939943583 0 0) 0x7f37c406c850 con 0x7f37c406aeb0 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c2ffd700 1 -- 172.21.15.111:0/3879366897 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac003570 con 0x7f37c406aeb0 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 -- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c4158c90 msgr2=0x7f37c40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c4158c90 0x7f37c40caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 -- 172.21.15.111:0/3879366897 >> v1:172.21.15.143:6789/0 conn(0x7f37c406aeb0 legacy=0x7f37c4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 -- 172.21.15.111:0/3879366897 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c406ba30 con 0x7f37c4159770 2023-12-19T08:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.657+0000 7f37c8dcb700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c406cce0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009e60 tx=0x7f37bc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534298aa25607b24 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c2ffd700 1 -- 172.21.15.111:0/3879366897 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc00e070 con 0x7f37c4159770 2023-12-19T08:02:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c2ffd700 1 -- 172.21.15.111:0/3879366897 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc00bab0 con 0x7f37c4159770 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c2ffd700 1 -- 172.21.15.111:0/3879366897 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc00be70 con 0x7f37c4159770 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 msgr2=0x7f37c406cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c406cce0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009e60 tx=0x7f37bc009780 comp rx=0 tx=0).stop 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- 172.21.15.111:0/3879366897 shutdown_connections 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c406cce0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3879366897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c4158c90 0x7f37c40caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- 172.21.15.111:0/3879366897 >> 172.21.15.111:0/3879366897 conn(0x7f37c4067bb0 msgr2=0x7f37c40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- 172.21.15.111:0/3879366897 shutdown_connections 2023-12-19T08:02:33.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- 172.21.15.111:0/3879366897 wait complete. 2023-12-19T08:02:33.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 Processor -- start 2023-12-19T08:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.661+0000 7f37c95cc700 1 -- start start 2023-12-19T08:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c406aeb0 0x7f37c4146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c40b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40b3b00 con 0x7f37c4158c90 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4157c40 con 0x7f37c4159770 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4157f20 con 0x7f37c406aeb0 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35460/0 (socket says 172.21.15.111:35460) 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 -- 172.21.15.111:0/3407625650 learned_addr learned my addr 172.21.15.111:0/3407625650 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c3fff700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c406aeb0 0x7f37c4146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c8dcb700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c40b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 -- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c406aeb0 msgr2=0x7f37c4146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c406aeb0 0x7f37c4146710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 -- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 msgr2=0x7f37c40b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c40b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 -- 172.21.15.111:0/3407625650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc009a20 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c37fe700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c7c0 tx=0x7f37b8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8005b30 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b8005c90 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c4158200 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8007320 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.665+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c4146c50 con 0x7f37c4158c90 2023-12-19T08:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.669+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c40b3cb0 con 0x7f37c4158c90 2023-12-19T08:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.669+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f37b801b3b0 con 0x7f37c4158c90 2023-12-19T08:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.673+0000 7f37abfff700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 0x7f37b0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.673+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f37b807c2b0 con 0x7f37c4158c90 2023-12-19T08:02:33.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.673+0000 7f37c3fff700 1 -- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 msgr2=0x7f37b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:33.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.673+0000 7f37c3fff700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 0x7f37b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:33.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.677+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f37c40b3cb0 con 0x7f37c4158c90 2023-12-19T08:02:33.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.849+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f37c404be40 con 0x7f37c4158c90 2023-12-19T08:02:33.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.857+0000 7f37abfff700 1 -- 172.21.15.111:0/3407625650 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f37b804c3a0 con 0x7f37c4158c90 2023-12-19T08:02:33.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 msgr2=0x7f37b0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:33.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 0x7f37b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 msgr2=0x7f37c4155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c7c0 tx=0x7f37b8007e00 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 shutdown_connections 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f37b00606c0 0x7f37b0062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37c4159770 0x7f37c40b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37c4158c90 0x7f37c4155190 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 --2- 172.21.15.111:0/3407625650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37c406aeb0 0x7f37c4146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 >> 172.21.15.111:0/3407625650 conn(0x7f37c4067bb0 msgr2=0x7f37c40caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 shutdown_connections 2023-12-19T08:02:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:33.861+0000 7f37c95cc700 1 -- 172.21.15.111:0/3407625650 wait complete. 2023-12-19T08:02:33.883 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:02:34.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.053+0000 7f3650cae700 1 Processor -- start 2023-12-19T08:02:34.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.053+0000 7f3650cae700 1 -- start start 2023-12-19T08:02:34.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f364c1514a0 0x7f364c157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c0cae60 0x7f364c06b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0cbee0 con 0x7f364c06b5d0 2023-12-19T08:02:34.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364c06ba50 con 0x7f364c1514a0 2023-12-19T08:02:34.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c158530 con 0x7f364c0cae60 2023-12-19T08:02:34.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f364c1514a0 0x7f364c157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f364c1514a0 0x7f364c157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35464/0 (socket says 172.21.15.111:35464) 2023-12-19T08:02:34.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 -- 172.21.15.111:0/2470540944 learned_addr learned my addr 172.21.15.111:0/2470540944 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:34.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3649d9b700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c0cae60 0x7f364c06b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 -- 172.21.15.111:0/2470540944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c0cae60 msgr2=0x7f364c06b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c0cae60 0x7f364c06b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 -- 172.21.15.111:0/2470540944 >> v1:172.21.15.143:6789/0 conn(0x7f364c06b5d0 legacy=0x7f364c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 -- 172.21.15.111:0/2470540944 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0cc370 con 0x7f364c1514a0 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364a59c700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c1514a0 0x7f364c157ff0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f363c0099b0 tx=0x7f363c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9203060cc0fac813 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364959a700 1 -- 172.21.15.111:0/2470540944 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c009ea0 con 0x7f364c1514a0 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364959a700 1 -- 172.21.15.111:0/2470540944 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c00b860 con 0x7f364c1514a0 2023-12-19T08:02:34.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f364959a700 1 -- 172.21.15.111:0/2470540944 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00bc00 con 0x7f364c1514a0 2023-12-19T08:02:34.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- 172.21.15.111:0/2470540944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c1514a0 msgr2=0x7f364c157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c1514a0 0x7f364c157ff0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f363c0099b0 tx=0x7f363c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- 172.21.15.111:0/2470540944 shutdown_connections 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c0cae60 0x7f364c06b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 --2- 172.21.15.111:0/2470540944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c1514a0 0x7f364c157ff0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- 172.21.15.111:0/2470540944 >> 172.21.15.111:0/2470540944 conn(0x7f364c0cc580 msgr2=0x7f364c0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- 172.21.15.111:0/2470540944 shutdown_connections 2023-12-19T08:02:34.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.057+0000 7f3650cae700 1 -- 172.21.15.111:0/2470540944 wait complete. 2023-12-19T08:02:34.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 Processor -- start 2023-12-19T08:02:34.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 -- start start 2023-12-19T08:02:34.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 0x7f364c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c1514a0 0x7f364c0a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c15d410 con 0x7f364c06b5d0 2023-12-19T08:02:34.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c15c800 con 0x7f364c0cae60 2023-12-19T08:02:34.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3650cae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c15cae0 con 0x7f364c1514a0 2023-12-19T08:02:34.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f364a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f364a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35474/0 (socket says 172.21.15.111:35474) 2023-12-19T08:02:34.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f364a59c700 1 -- 172.21.15.111:0/984760903 learned_addr learned my addr 172.21.15.111:0/984760903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f3649d9b700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 0x7f364c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.061+0000 7f364ad9d700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c1514a0 0x7f364c0a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 -- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c1514a0 msgr2=0x7f364c0a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c1514a0 0x7f364c0a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 -- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 msgr2=0x7f364c15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 0x7f364c15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 -- 172.21.15.111:0/984760903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0094f0 con 0x7f364c06b5d0 2023-12-19T08:02:34.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f3649d9b700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 0x7f364c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f364a59c700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f364c064900 tx=0x7f363c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c00bbe0 con 0x7f364c06b5d0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c011420 con 0x7f364c06b5d0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c15cdc0 con 0x7f364c06b5d0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c0117c0 con 0x7f364c06b5d0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c15c420 con 0x7f364c06b5d0 2023-12-19T08:02:34.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c071120 con 0x7f364c06b5d0 2023-12-19T08:02:34.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.065+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f363c011bf0 con 0x7f364c06b5d0 2023-12-19T08:02:34.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.069+0000 7f363a7fc700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 0x7f362c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.069+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f363c07e5f0 con 0x7f364c06b5d0 2023-12-19T08:02:34.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.069+0000 7f3649d9b700 1 -- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 msgr2=0x7f362c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:34.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.069+0000 7f3649d9b700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 0x7f362c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.077+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f363c04e780 con 0x7f364c06b5d0 2023-12-19T08:02:34.192 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:34.189+0000 7f741b147000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:34.193 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:34.194+0000 7f7949564000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:34.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.241+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f364c0cb2d0 con 0x7f364c06b5d0 2023-12-19T08:02:34.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.249+0000 7f363a7fc700 1 -- 172.21.15.111:0/984760903 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f363c051da0 con 0x7f364c06b5d0 2023-12-19T08:02:34.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 msgr2=0x7f362c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:34.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 0x7f362c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 msgr2=0x7f364c150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f364c064900 tx=0x7f363c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:34.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 shutdown_connections 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f362c0606c0 0x7f362c062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f364c0cae60 0x7f364c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f364c1514a0 0x7f364c0a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 --2- 172.21.15.111:0/984760903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f364c06b5d0 0x7f364c150610 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 >> 172.21.15.111:0/984760903 conn(0x7f364c0cc580 msgr2=0x7f364c0c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 shutdown_connections 2023-12-19T08:02:34.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.253+0000 7f3650cae700 1 -- 172.21.15.111:0/984760903 wait complete. 2023-12-19T08:02:34.268 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:34.266+0000 7f7949564000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:34.270 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:34.269+0000 7f741b147000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:34.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:34.276 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T08:02:34.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T08:02:34.420 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:34.422+0000 7f7949564000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:34.421 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:34.421+0000 7f741b147000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:34.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 Processor -- start 2023-12-19T08:02:34.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 -- start start 2023-12-19T08:02:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc0c6ec0 0x7fa8bc06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8bc1566a0 0x7fa8bc156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc06d7b0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8bc06cba0 con 0x7fa8bc1566a0 2023-12-19T08:02:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8c12ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc06ce80 con 0x7fa8bc0c6ec0 2023-12-19T08:02:34.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8ba7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc0c6ec0 0x7fa8bc06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8bc1566a0 0x7fa8bc156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8bc1566a0 0x7fa8bc156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:53748/0 (socket says 172.21.15.111:53748) 2023-12-19T08:02:34.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 -- 172.21.15.111:0/3135222954 learned_addr learned my addr 172.21.15.111:0/3135222954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:34.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 -- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc0c6ec0 msgr2=0x7fa8bc06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:34.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 --2- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc0c6ec0 0x7fa8bc06dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 -- 172.21.15.111:0/3135222954 >> v1:172.21.15.143:6789/0 conn(0x7fa8bc0c6af0 legacy=0x7fa8bc06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.521+0000 7fa8bb7fe700 1 -- 172.21.15.111:0/3135222954 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc06d160 con 0x7fa8bc1566a0 2023-12-19T08:02:34.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8bb7fe700 1 --2- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc1566a0 0x7fa8bc156b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009f20 tx=0x7fa8b4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ee22b0fa9a5c41 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:34.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8b9ffb700 1 -- 172.21.15.111:0/3135222954 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400e070 con 0x7fa8bc1566a0 2023-12-19T08:02:34.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8b9ffb700 1 -- 172.21.15.111:0/3135222954 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b400b9d0 con 0x7fa8bc1566a0 2023-12-19T08:02:34.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8b9ffb700 1 -- 172.21.15.111:0/3135222954 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40092e0 con 0x7fa8bc1566a0 2023-12-19T08:02:34.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc1566a0 msgr2=0x7fa8bc156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 --2- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc1566a0 0x7fa8bc156b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009f20 tx=0x7fa8b4009840 comp rx=0 tx=0).stop 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- 172.21.15.111:0/3135222954 shutdown_connections 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 --2- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc1566a0 0x7fa8bc156b20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 --2- 172.21.15.111:0/3135222954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc0c6ec0 0x7fa8bc06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- 172.21.15.111:0/3135222954 >> 172.21.15.111:0/3135222954 conn(0x7fa8bc067b10 msgr2=0x7fa8bc0cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- 172.21.15.111:0/3135222954 shutdown_connections 2023-12-19T08:02:34.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- 172.21.15.111:0/3135222954 wait complete. 2023-12-19T08:02:34.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 Processor -- start 2023-12-19T08:02:34.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.525+0000 7fa8c12ae700 1 -- start start 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc0c6af0 0x7fa8bc0aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc0c6ec0 0x7fa8bc168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc1566a0 0x7fa8bc0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0a8850 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0aa8f0 con 0x7fa8bc0c6ec0 2023-12-19T08:02:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc169430 con 0x7fa8bc1566a0 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc0c6af0 0x7fa8bc0aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc0c6af0 0x7fa8bc0aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35490/0 (socket says 172.21.15.111:35490) 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 -- 172.21.15.111:0/2241152320 learned_addr learned my addr 172.21.15.111:0/2241152320 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8bb7fe700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc1566a0 0x7fa8bc0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc0c6ec0 0x7fa8bc168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc1566a0 msgr2=0x7fa8bc0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc1566a0 0x7fa8bc0c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc0c6ec0 msgr2=0x7fa8bc168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc0c6ec0 0x7fa8bc168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4009ae0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8bb7fe700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc1566a0 0x7fa8bc0c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8baffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc0c6af0 0x7fa8bc0aa3b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c590 tx=0x7fa8ac00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac011590 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac0116f0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac011ab0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8bc0a8ac0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.529+0000 7fa8c12ae700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc169940 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8c12ae700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8bc04be40 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 27) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fa8ac01b3b0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8a2ffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fa8ac07c2a0 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8ba7fc700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.533+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:34.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.541+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa8ac04c430 con 0x7fa8bc0c6af0 2023-12-19T08:02:34.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:34.698+0000 7f7949564000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:34.703 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:34.701+0000 7f741b147000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:34.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.733+0000 7fa8c12ae700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8bc06d440 con 0x7fa8a80606c0 2023-12-19T08:02:34.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.737+0000 7fa8ba7fc700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:34.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:34.737+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:34.773 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:34.774+0000 7f7949564000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:34.776 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:34.773+0000 7f741b147000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:35.003 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:35.002+0000 7f7949564000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:35.003 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:35.001+0000 7f741b147000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:35.085 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:35.086+0000 7f7949564000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:35.087 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:35.085+0000 7f741b147000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:35.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:35.141+0000 7fa8ba7fc700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:35.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:35.141+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:02:35.173 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:35.174+0000 7f7949564000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:35.174 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:35.173+0000 7f741b147000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:35.581 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:35.577+0000 7f741b147000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:35.586 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:35.590+0000 7f7949564000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:35.753 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:35.749+0000 7f741b147000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:35.763 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:35.766+0000 7f7949564000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:35.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:35.945+0000 7fa8ba7fc700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:35.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:35.945+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:02:36.093 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.089+0000 7f741b147000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:36.133 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.134+0000 7f7949564000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:36.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:02:36.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:02:36.312 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.309+0000 7f741b147000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:36.352 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.354+0000 7f7949564000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:36.406 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.405+0000 7f741b147000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:36.444 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.446+0000 7f7949564000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:36.480 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.477+0000 7f741b147000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:36.528 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.530+0000 7f7949564000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:36.557 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.553+0000 7f741b147000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:36.620 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.622+0000 7f7949564000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:36.766 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.765+0000 7f741b147000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:36.845 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.845+0000 7f741b147000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:36.859 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.862+0000 7f7949564000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:36.929 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:36.925+0000 7f741b147000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:36.942 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:36.946+0000 7f7949564000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:37.034 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:37.034+0000 7f7949564000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:37.112 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:37.109+0000 7f741b147000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:37.193 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:37.189+0000 7f741b147000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:37.232 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:37.234+0000 7f7949564000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:37.319 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:37.322+0000 7f7949564000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:37.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:37.549+0000 7fa8ba7fc700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:37.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:37.549+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:02:37.895 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:37.893+0000 7f741b147000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:37.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:37.961+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 28) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fa8ac04c800 con 0x7fa8bc0c6af0 2023-12-19T08:02:38.053 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.054+0000 7f7949564000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:38.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:38.065+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 29) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fa8ac07a430 con 0x7fa8bc0c6af0 2023-12-19T08:02:38.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:38.069+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 msgr2=0x7fa8a8062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:38.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:38.069+0000 7fa8a2ffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f790b056700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f790b056700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f790b056700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f790b056700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f790b056700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.255 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f7908050700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.256 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f7908050700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.256 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f7908050700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.256 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f7908050700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:38.256 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:38.258+0000 7f7908050700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.081+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mon.0 v2:172.21.15.111:3300/0 9 ==== mgrmap(e 30) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fa8ac049920 con 0x7fa8bc0c6af0 2023-12-19T08:02:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.081+0000 7fa8a2ffd700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.081+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8bc06d440 con 0x7fa8a80606c0 2023-12-19T08:02:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.085+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.089+0000 7fa8ba7fc700 1 --2- 172.21.15.111:0/2241152320 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa8a80606c0 0x7fa8a8062af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0ae3e0 tx=0x7fa8b00091a0 comp rx=0 tx=0).ready entity=mgr.5969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:39.104 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:39.106+0000 7f7912064700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T08:02:39.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.101+0000 7fa8a2ffd700 1 -- 172.21.15.111:0/2241152320 <== mgr.5969 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fa8bc06d440 con 0x7fa8a80606c0 2023-12-19T08:02:39.109 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T08:02:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T08:02:39.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:02:39.119 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T08:02:39.196 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T08:02:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.413+0000 7ff552878700 1 Processor -- start 2023-12-19T08:02:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.413+0000 7ff552878700 1 -- start start 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c158e20 0x7ff54c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff54c0cb970 0x7ff54c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 0x7ff54c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0cbdf0 con 0x7ff54c0cb970 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0cc0d0 con 0x7ff54c0ca2c0 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cc3b0 con 0x7ff54c158e20 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff54c0cb970 0x7ff54c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff54c0cb970 0x7ff54c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35500/0 (socket says 172.21.15.111:35500) 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 -- 172.21.15.111:0/2153114542 learned_addr learned my addr 172.21.15.111:0/2153114542 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff550e15700 1 --2- 172.21.15.111:0/2153114542 >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 0x7ff54c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54bfff700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c158e20 0x7ff54c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 -- 172.21.15.111:0/2153114542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c158e20 msgr2=0x7ff54c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c158e20 0x7ff54c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 -- 172.21.15.111:0/2153114542 >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 msgr2=0x7ff54c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 --2- 172.21.15.111:0/2153114542 >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 0x7ff54c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 -- 172.21.15.111:0/2153114542 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0cc690 con 0x7ff54c0cb970 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff550e15700 1 --2- 172.21.15.111:0/2153114542 >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 0x7ff54c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54b7fe700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0cb970 0x7ff54c0ceb00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0cfd10 tx=0x7ff540009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9979c8f0af09d1a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54affd700 1 -- 172.21.15.111:0/2153114542 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000b860 con 0x7ff54c0cb970 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54affd700 1 -- 172.21.15.111:0/2153114542 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000b9c0 con 0x7ff54c0cb970 2023-12-19T08:02:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff54affd700 1 -- 172.21.15.111:0/2153114542 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000bd60 con 0x7ff54c0cb970 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- 172.21.15.111:0/2153114542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0cb970 msgr2=0x7ff54c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0cb970 0x7ff54c0ceb00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0cfd10 tx=0x7ff540009250 comp rx=0 tx=0).stop 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- 172.21.15.111:0/2153114542 shutdown_connections 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- 172.21.15.111:0/2153114542 >> v2:172.21.15.143:3300/0 conn(0x7ff54c0ca2c0 0x7ff54c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0cb970 0x7ff54c0ceb00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 --2- 172.21.15.111:0/2153114542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c158e20 0x7ff54c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- 172.21.15.111:0/2153114542 >> 172.21.15.111:0/2153114542 conn(0x7ff54c067bb0 msgr2=0x7ff54c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- 172.21.15.111:0/2153114542 shutdown_connections 2023-12-19T08:02:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.417+0000 7ff552878700 1 -- 172.21.15.111:0/2153114542 wait complete. 2023-12-19T08:02:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.421+0000 7ff552878700 1 Processor -- start 2023-12-19T08:02:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.421+0000 7ff552878700 1 -- start start 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.421+0000 7ff552878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c0ca2c0 0x7ff54c14e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.421+0000 7ff552878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff54c158e20 0x7ff54c0c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c15ba10 con 0x7ff54c15d460 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c14e770 con 0x7ff54c158e20 2023-12-19T08:02:39.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c15de20 con 0x7ff54c0ca2c0 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35516/0 (socket says 172.21.15.111:35516) 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 -- 172.21.15.111:0/1496988237 learned_addr learned my addr 172.21.15.111:0/1496988237 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff54b7fe700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff54c158e20 0x7ff54c0c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff54bfff700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c0ca2c0 0x7ff54c14e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 -- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c0ca2c0 msgr2=0x7ff54c14e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c0ca2c0 0x7ff54c14e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 -- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff54c158e20 msgr2=0x7ff54c0c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:39.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff54c158e20 0x7ff54c0c86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 -- 172.21.15.111:0/1496988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400094f0 con 0x7ff54c15d460 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff550e15700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d860 tx=0x7ff544008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440125f0 con 0x7ff54c15d460 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544012750 con 0x7ff54c15d460 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c15e100 con 0x7ff54c15d460 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544012ad0 con 0x7ff54c15d460 2023-12-19T08:02:39.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.425+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c15b6a0 con 0x7ff54c15d460 2023-12-19T08:02:39.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.429+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c15bb80 con 0x7ff54c15d460 2023-12-19T08:02:39.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.429+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 30) v1 ==== 79816+0+0 (secure 0 0 0) 0x7ff544014430 con 0x7ff54c15d460 2023-12-19T08:02:39.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.429+0000 7ff533fff700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 0x7ff5380628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:39.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.429+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7ff54407ce20 con 0x7ff54c15d460 2023-12-19T08:02:39.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.429+0000 7ff54bfff700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 0x7ff5380628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:39.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.433+0000 7ff54bfff700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 0x7ff5380628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff53c007540 tx=0x7ff53c00b040 comp rx=0 tx=0).ready entity=mgr.5969 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:39.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.437+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff544050be0 con 0x7ff54c15d460 2023-12-19T08:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:39.649+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff54c071120 con 0x7ff54c15d460 2023-12-19T08:02:40.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.093+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7ff544082020 con 0x7ff54c15d460 2023-12-19T08:02:40.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.093+0000 7ff533fff700 1 -- 172.21.15.111:0/1496988237 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v31) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff544050950 con 0x7ff54c15d460 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 msgr2=0x7ff5380628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 0x7ff5380628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff53c007540 tx=0x7ff53c00b040 comp rx=0 tx=0).stop 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 msgr2=0x7ff54c15a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d860 tx=0x7ff544008040 comp rx=0 tx=0).stop 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 shutdown_connections 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5380603f0 0x7ff5380628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff54c158e20 0x7ff54c0c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c15d460 0x7ff54c15a570 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 --2- 172.21.15.111:0/1496988237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff54c0ca2c0 0x7ff54c14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 >> 172.21.15.111:0/1496988237 conn(0x7ff54c067bb0 msgr2=0x7ff54c14ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:40.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 shutdown_connections 2023-12-19T08:02:40.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.097+0000 7ff552878700 1 -- 172.21.15.111:0/1496988237 wait complete. 2023-12-19T08:02:40.119 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:02:40.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:40.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:40.190 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:02:40.190 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T08:02:40.190 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T08:02:40.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T08:02:40.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[49029]: fuse finished with error 0 and tester_r 0 2023-12-19T08:02:40.232 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:02:40.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:40.232 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T08:02:40.283 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:40.281+0000 7fc302b0a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:40.283 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:40.286+0000 7f706d15f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:02:40.290 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:02:40.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:40.290 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T08:02:40.311 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T08:02:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:40.312 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:02:40.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:40.313 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T08:02:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T08:02:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:40.354 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:02:40.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:02:40.354 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T08:02:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:02:40.377 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T08:02:40.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:40.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 Processor -- start 2023-12-19T08:02:40.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 -- start start 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c00158e20 0x7f1c0006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1c000ce810 0x7f1c000ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0006b730 con 0x7f1c001591f0 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0006ab20 con 0x7f1c000ce810 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1c04ae3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0006ae00 con 0x7f1c00158e20 2023-12-19T08:02:40.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1bff7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c00158e20 0x7f1c0006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1bfeffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1c001591f0 0x7f1c000ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56188/0 (socket says 172.21.15.111:56188) 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1bfffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1c000ce810 0x7f1c000ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.553+0000 7f1bfeffd700 1 -- 172.21.15.111:0/1438152262 learned_addr learned my addr 172.21.15.111:0/1438152262 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bff7fe700 1 -- 172.21.15.111:0/1438152262 >> v2:172.21.15.111:3300/0 conn(0x7f1c000ce810 msgr2=0x7f1c000ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bff7fe700 1 --2- 172.21.15.111:0/1438152262 >> v2:172.21.15.111:3300/0 conn(0x7f1c000ce810 0x7f1c000ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bff7fe700 1 -- 172.21.15.111:0/1438152262 >> v1:172.21.15.143:6789/0 conn(0x7f1c001591f0 legacy=0x7f1c000ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bff7fe700 1 -- 172.21.15.111:0/1438152262 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0006b0e0 con 0x7f1c00158e20 2023-12-19T08:02:40.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bff7fe700 1 --2- 172.21.15.111:0/1438152262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c0006a540 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0006e510 tx=0x7f1bf00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd5217d0bbeda4cd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:40.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bfe7fc700 1 -- 172.21.15.111:0/1438152262 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0011630 con 0x7f1c00158e20 2023-12-19T08:02:40.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bfe7fc700 1 -- 172.21.15.111:0/1438152262 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf0011790 con 0x7f1c00158e20 2023-12-19T08:02:40.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1bfe7fc700 1 -- 172.21.15.111:0/1438152262 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0011b30 con 0x7f1c00158e20 2023-12-19T08:02:40.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 -- 172.21.15.111:0/1438152262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 msgr2=0x7f1c0006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/1438152262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c0006a540 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0006e510 tx=0x7f1bf00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 -- 172.21.15.111:0/1438152262 shutdown_connections 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/1438152262 >> v2:172.21.15.111:3300/0 conn(0x7f1c000ce810 0x7f1c000ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/1438152262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c0006a540 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 -- 172.21.15.111:0/1438152262 >> 172.21.15.111:0/1438152262 conn(0x7f1c00067bb0 msgr2=0x7f1c000cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 -- 172.21.15.111:0/1438152262 shutdown_connections 2023-12-19T08:02:40.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.557+0000 7f1c04ae3700 1 -- 172.21.15.111:0/1438152262 wait complete. 2023-12-19T08:02:40.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 Processor -- start 2023-12-19T08:02:40.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 -- start start 2023-12-19T08:02:40.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c000ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c001591f0 0x7f1c0006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0016a650 con 0x7f1c000ce810 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00152b10 con 0x7f1c00158e20 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1c04ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00169a40 con 0x7f1c001591f0 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1bff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1bff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35550/0 (socket says 172.21.15.111:35550) 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1bff7fe700 1 -- 172.21.15.111:0/3732267789 learned_addr learned my addr 172.21.15.111:0/3732267789 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:40.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1bfeffd700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c000ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.561+0000 7f1bfffff700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c001591f0 0x7f1c0006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c001591f0 msgr2=0x7f1c0006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c001591f0 0x7f1c0006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 msgr2=0x7f1c000ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c000ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 -- 172.21.15.111:0/3732267789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0009420 con 0x7f1c000ce810 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1bff7fe700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1c0006af40 tx=0x7f1bf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0011b10 con 0x7f1c000ce810 2023-12-19T08:02:40.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf00253f0 con 0x7f1c000ce810 2023-12-19T08:02:40.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c00169d20 con 0x7f1c000ce810 2023-12-19T08:02:40.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0025790 con 0x7f1c000ce810 2023-12-19T08:02:40.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0016a2e0 con 0x7f1c000ce810 2023-12-19T08:02:40.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.565+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c000c7e40 con 0x7f1c000ce810 2023-12-19T08:02:40.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.569+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f1bf0025b80 con 0x7f1c000ce810 2023-12-19T08:02:40.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.569+0000 7f1be77fe700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 0x7f1bec062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.569+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f1bf007e900 con 0x7f1c000ce810 2023-12-19T08:02:40.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.569+0000 7f1bfeffd700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 msgr2=0x7f1bec062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:40.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.569+0000 7f1bfeffd700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 0x7f1bec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:40.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.577+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1bf004e730 con 0x7f1c000ce810 2023-12-19T08:02:40.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.773+0000 7f1bfeffd700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 msgr2=0x7f1bec062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:40.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.773+0000 7f1bfeffd700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 0x7f1bec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:40.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.793+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c0004be40 con 0x7f1c000ce810 2023-12-19T08:02:40.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.793+0000 7f1be77fe700 1 -- 172.21.15.111:0/3732267789 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f1bf0051d50 con 0x7f1c000ce810 2023-12-19T08:02:40.795 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:40.795 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 msgr2=0x7f1bec062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 0x7f1bec062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 msgr2=0x7f1c001525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1c0006af40 tx=0x7f1bf0017000 comp rx=0 tx=0).stop 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 shutdown_connections 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bec060700 0x7f1bec062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c00158e20 0x7f1c000ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c001591f0 0x7f1c0006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 --2- 172.21.15.111:0/3732267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c000ce810 0x7f1c001525d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 >> 172.21.15.111:0/3732267789 conn(0x7f1c00067bb0 msgr2=0x7f1c00153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 shutdown_connections 2023-12-19T08:02:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.797+0000 7f1c04ae3700 1 -- 172.21.15.111:0/3732267789 wait complete. 2023-12-19T08:02:40.800 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:40.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:40.832 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:40.834+0000 7f706d15f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:40.850 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:40.849+0000 7fc302b0a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:02:40.912 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:40.914+0000 7f706d15f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:40.928 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:40.925+0000 7fc302b0a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:02:40.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.957+0000 7f97658e3700 1 Processor -- start 2023-12-19T08:02:40.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.957+0000 7f97658e3700 1 -- start start 2023-12-19T08:02:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97658e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 0x7f976006a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97658e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f976006d210 0x7f976006d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97658e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976006c7c0 con 0x7f97600b1410 2023-12-19T08:02:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97658e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976006cd80 con 0x7f976006d210 2023-12-19T08:02:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97658e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976006af10 con 0x7f976006a580 2023-12-19T08:02:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97650e2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f976006d210 0x7f976006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97648e1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f97600b1410 0x7f97600ab440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38204/0 (socket says 172.21.15.111:38204) 2023-12-19T08:02:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f97648e1700 1 -- 172.21.15.111:0/2185508401 learned_addr learned my addr 172.21.15.111:0/2185508401 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 --2- 172.21.15.111:0/2185508401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 0x7f976006a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 -- 172.21.15.111:0/2185508401 >> v2:172.21.15.143:3300/0 conn(0x7f976006d210 msgr2=0x7f976006d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 --2- 172.21.15.111:0/2185508401 >> v2:172.21.15.143:3300/0 conn(0x7f976006d210 0x7f976006d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 -- 172.21.15.111:0/2185508401 >> v1:172.21.15.111:6789/0 conn(0x7f97600b1410 legacy=0x7f97600ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 -- 172.21.15.111:0/2185508401 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976006b8f0 con 0x7f976006a580 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975f7fe700 1 -- 172.21.15.111:0/2185508401 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638059652 0 0) 0x7f976006c7c0 con 0x7f97600b1410 2023-12-19T08:02:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975ffff700 1 --2- 172.21.15.111:0/2185508401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 0x7f976006a9d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f974c0098e0 tx=0x7f974c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ba76bb60a9ae84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975f7fe700 1 -- 172.21.15.111:0/2185508401 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c00b640 con 0x7f976006a580 2023-12-19T08:02:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975f7fe700 1 -- 172.21.15.111:0/2185508401 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c00b7a0 con 0x7f976006a580 2023-12-19T08:02:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.961+0000 7f975f7fe700 1 -- 172.21.15.111:0/2185508401 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c00bb40 con 0x7f976006a580 2023-12-19T08:02:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- 172.21.15.111:0/2185508401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 msgr2=0x7f976006a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 --2- 172.21.15.111:0/2185508401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 0x7f976006a9d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f974c0098e0 tx=0x7f974c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- 172.21.15.111:0/2185508401 shutdown_connections 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 --2- 172.21.15.111:0/2185508401 >> v2:172.21.15.143:3300/0 conn(0x7f976006d210 0x7f976006d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 --2- 172.21.15.111:0/2185508401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006a580 0x7f976006a9d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- 172.21.15.111:0/2185508401 >> 172.21.15.111:0/2185508401 conn(0x7f9760067bb0 msgr2=0x7f97600cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- 172.21.15.111:0/2185508401 shutdown_connections 2023-12-19T08:02:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- 172.21.15.111:0/2185508401 wait complete. 2023-12-19T08:02:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 Processor -- start 2023-12-19T08:02:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.965+0000 7f97658e3700 1 -- start start 2023-12-19T08:02:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006d210 0x7f97600c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600b1410 0x7f97600c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976016ae70 con 0x7f976006a580 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97601442d0 con 0x7f97600b1410 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976016a560 con 0x7f976006d210 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35556/0 (socket says 172.21.15.111:35556) 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 -- 172.21.15.111:0/2498547936 learned_addr learned my addr 172.21.15.111:0/2498547936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f975ffff700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006d210 0x7f97600c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97650e2700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600b1410 0x7f97600c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:40.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 -- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006d210 msgr2=0x7f97600c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006d210 0x7f97600c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 -- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600b1410 msgr2=0x7f97600c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600b1410 0x7f97600c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 -- 172.21.15.111:0/2498547936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c009420 con 0x7f976006a580 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97648e1700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f975400c5b0 tx=0x7f9754001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754005b30 con 0x7f976006a580 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754005c90 con 0x7f976006a580 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976016a820 con 0x7f976006a580 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754007300 con 0x7f976006a580 2023-12-19T08:02:40.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.969+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9760158f50 con 0x7f976006a580 2023-12-19T08:02:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976004be40 con 0x7f976006a580 2023-12-19T08:02:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f975401b3b0 con 0x7f976006a580 2023-12-19T08:02:40.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f975cff9700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 0x7f9744062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:40.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f975407c3a0 con 0x7f976006a580 2023-12-19T08:02:40.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f975ffff700 1 -- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 msgr2=0x7f9744062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:40.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.973+0000 7f975ffff700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 0x7f9744062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:40.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:40.981+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f975404c270 con 0x7f976006a580 2023-12-19T08:02:41.058 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.058+0000 7f706d15f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:41.077 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.073+0000 7fc302b0a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:41.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.161+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97600ab220 con 0x7f976006a580 2023-12-19T08:02:41.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.165+0000 7f975cff9700 1 -- 172.21.15.111:0/2498547936 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16893 (secure 0 0 0) 0x7f975404f890 con 0x7f976006a580 2023-12-19T08:02:41.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:41.168 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:38.058416+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2190915022":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1956986802":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1938739052":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/3973057301":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/2537901821":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6833/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6832/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1521230848":"2023-12-20T08:02:38.058339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:41.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 msgr2=0x7f9744062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:41.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 0x7f9744062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 msgr2=0x7f9760143d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f975400c5b0 tx=0x7f9754001070 comp rx=0 tx=0).stop 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 shutdown_connections 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9744060700 0x7f9744062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600b1410 0x7f97600c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006d210 0x7f97600c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 --2- 172.21.15.111:0/2498547936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f976006a580 0x7f9760143d90 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 >> 172.21.15.111:0/2498547936 conn(0x7f9760067bb0 msgr2=0x7f97600b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 shutdown_connections 2023-12-19T08:02:41.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.169+0000 7f97658e3700 1 -- 172.21.15.111:0/2498547936 wait complete. 2023-12-19T08:02:41.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:02:21.020848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:02:22.062351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T08:02:41.191 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:41.338 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.338+0000 7f706d15f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:41.355 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.353+0000 7fc302b0a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:02:41.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.393+0000 7efd210ae700 1 Processor -- start 2023-12-19T08:02:41.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.393+0000 7efd210ae700 1 -- start start 2023-12-19T08:02:41.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c158e20 0x7efd1c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd1c0cb970 0x7efd1c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1c0cbdf0 con 0x7efd1c0ca2c0 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1c0cc0d0 con 0x7efd1c0cb970 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0cc3b0 con 0x7efd1c158e20 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd1c0cb970 0x7efd1c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd1c0cb970 0x7efd1c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35572/0 (socket says 172.21.15.111:35572) 2023-12-19T08:02:41.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 -- 172.21.15.111:0/2121931853 learned_addr learned my addr 172.21.15.111:0/2121931853 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1ad9d700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c158e20 0x7efd1c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 -- 172.21.15.111:0/2121931853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c158e20 msgr2=0x7efd1c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c158e20 0x7efd1c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 -- 172.21.15.111:0/2121931853 >> v1:172.21.15.143:6789/0 conn(0x7efd1c0ca2c0 legacy=0x7efd1c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 -- 172.21.15.111:0/2121931853 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c0cc690 con 0x7efd1c0cb970 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd1a59c700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c0cb970 0x7efd1c0ceb00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7efd100099b0 tx=0x7efd10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcc310b5a1fd283 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd19d9b700 1 -- 172.21.15.111:0/2121931853 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000b860 con 0x7efd1c0cb970 2023-12-19T08:02:41.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd19d9b700 1 -- 172.21.15.111:0/2121931853 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1000b9c0 con 0x7efd1c0cb970 2023-12-19T08:02:41.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd19d9b700 1 -- 172.21.15.111:0/2121931853 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000bd60 con 0x7efd1c0cb970 2023-12-19T08:02:41.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- 172.21.15.111:0/2121931853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c0cb970 msgr2=0x7efd1c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c0cb970 0x7efd1c0ceb00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7efd100099b0 tx=0x7efd10009250 comp rx=0 tx=0).stop 2023-12-19T08:02:41.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- 172.21.15.111:0/2121931853 shutdown_connections 2023-12-19T08:02:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c0cb970 0x7efd1c0ceb00 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 --2- 172.21.15.111:0/2121931853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c158e20 0x7efd1c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- 172.21.15.111:0/2121931853 >> 172.21.15.111:0/2121931853 conn(0x7efd1c067bb0 msgr2=0x7efd1c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- 172.21.15.111:0/2121931853 shutdown_connections 2023-12-19T08:02:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.397+0000 7efd210ae700 1 -- 172.21.15.111:0/2121931853 wait complete. 2023-12-19T08:02:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 Processor -- start 2023-12-19T08:02:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 -- start start 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c0ca2c0 0x7efd1c16a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1c0b52b0 0x7efd1c0b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c16a630 con 0x7efd1c158e20 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c14cc90 con 0x7efd1c0b52b0 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd210ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0b5c70 con 0x7efd1c0ca2c0 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35582/0 (socket says 172.21.15.111:35582) 2023-12-19T08:02:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 -- 172.21.15.111:0/4141072073 learned_addr learned my addr 172.21.15.111:0/4141072073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:41.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1b59e700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1c0b52b0 0x7efd1c0b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1ad9d700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c0ca2c0 0x7efd1c16a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 -- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c0ca2c0 msgr2=0x7efd1c16a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c0ca2c0 0x7efd1c16a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 -- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1c0b52b0 msgr2=0x7efd1c0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1c0b52b0 0x7efd1c0b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 -- 172.21.15.111:0/4141072073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100094f0 con 0x7efd1c158e20 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.401+0000 7efd1a59c700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efd1c069a80 tx=0x7efd10017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10024440 con 0x7efd1c158e20 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd100245c0 con 0x7efd1c158e20 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10024980 con 0x7efd1c158e20 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1c0b3d60 con 0x7efd1c158e20 2023-12-19T08:02:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1c0b3430 con 0x7efd1c158e20 2023-12-19T08:02:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1c0b3ef0 con 0x7efd1c158e20 2023-12-19T08:02:41.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.405+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7efd1001a420 con 0x7efd1c158e20 2023-12-19T08:02:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.409+0000 7efd02ffd700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 0x7efd08062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.409+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7efd1007ee90 con 0x7efd1c158e20 2023-12-19T08:02:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.409+0000 7efd1ad9d700 1 -- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 msgr2=0x7efd08062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.409+0000 7efd1ad9d700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 0x7efd08062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.413+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efd1004ed60 con 0x7efd1c158e20 2023-12-19T08:02:41.418 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.418+0000 7f706d15f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:41.438 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.437+0000 7fc302b0a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:41.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.581+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd1c06a7a0 con 0x7efd1c158e20 2023-12-19T08:02:41.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.581+0000 7efd02ffd700 1 -- 172.21.15.111:0/4141072073 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd10052380 con 0x7efd1c158e20 2023-12-19T08:02:41.585 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:41.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 msgr2=0x7efd08062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:41.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 0x7efd08062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 msgr2=0x7efd1c14c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efd1c069a80 tx=0x7efd10017000 comp rx=0 tx=0).stop 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 shutdown_connections 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efd08060700 0x7efd08062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1c0b52b0 0x7efd1c0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd1c158e20 0x7efd1c14c750 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 --2- 172.21.15.111:0/4141072073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd1c0ca2c0 0x7efd1c16a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 >> 172.21.15.111:0/4141072073 conn(0x7efd1c067bb0 msgr2=0x7efd1c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 shutdown_connections 2023-12-19T08:02:41.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.585+0000 7efd210ae700 1 -- 172.21.15.111:0/4141072073 wait complete. 2023-12-19T08:02:41.608 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T08:02:41.637 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.637+0000 7f706d15f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:41.668 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.665+0000 7fc302b0a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:02:41.721 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.721+0000 7f706d15f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.753+0000 7f910a5cc700 1 Processor -- start 2023-12-19T08:02:41.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 -- start start 2023-12-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 0x7f91040caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f910406aeb0 0x7f9104159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f910406c850 con 0x7f91041595a0 2023-12-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910406b470 con 0x7f910406aeb0 2023-12-19T08:02:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f910a5cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910406b750 con 0x7f9104158c90 2023-12-19T08:02:41.760 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.757+0000 7fc302b0a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f910406aeb0 0x7f9104159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f91095ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 0x7f91040caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f910406aeb0 0x7f9104159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:56804/0 (socket says 172.21.15.111:56804) 2023-12-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f91095ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 0x7f91040caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52466/0 (socket says 172.21.15.111:52466) 2023-12-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 -- 172.21.15.111:0/1327046038 learned_addr learned my addr 172.21.15.111:0/1327046038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:41.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9109dcb700 1 --1- 172.21.15.111:0/1327046038 >> v1:172.21.15.143:6789/0 conn(0x7f91041595a0 0x7f910415c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:51822/0 (socket says 172.21.15.111:51822) 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 -- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 msgr2=0x7f91040caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 --2- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 0x7f91040caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 -- 172.21.15.111:0/1327046038 >> v1:172.21.15.143:6789/0 conn(0x7f91041595a0 legacy=0x7f910415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.757+0000 7f9108dc9700 1 -- 172.21.15.111:0/1327046038 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910406ba30 con 0x7f910406aeb0 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f9108dc9700 1 --2- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f9104159060 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f90f40099b0 tx=0x7f90f4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad03fcd30d8a500 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f90fbfff700 1 -- 172.21.15.111:0/1327046038 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4009ea0 con 0x7f910406aeb0 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f90fbfff700 1 -- 172.21.15.111:0/1327046038 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f400b860 con 0x7f910406aeb0 2023-12-19T08:02:41.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f90fbfff700 1 -- 172.21.15.111:0/1327046038 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400bc00 con 0x7f910406aeb0 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 -- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 msgr2=0x7f9104159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 --2- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f9104159060 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f90f40099b0 tx=0x7f90f4009250 comp rx=0 tx=0).stop 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 -- 172.21.15.111:0/1327046038 shutdown_connections 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 --2- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f9104159060 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 --2- 172.21.15.111:0/1327046038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9104158c90 0x7f91040caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 -- 172.21.15.111:0/1327046038 >> 172.21.15.111:0/1327046038 conn(0x7f9104067bb0 msgr2=0x7f91040cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 -- 172.21.15.111:0/1327046038 shutdown_connections 2023-12-19T08:02:41.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 -- 172.21.15.111:0/1327046038 wait complete. 2023-12-19T08:02:41.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.761+0000 7f910a5cc700 1 Processor -- start 2023-12-19T08:02:41.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- start start 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f91040c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91041595a0 0x7f91040af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91040c7790 con 0x7f9104158c90 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9104154e60 con 0x7f910406aeb0 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91041559e0 con 0x7f91041595a0 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43266/0 (socket says 172.21.15.111:43266) 2023-12-19T08:02:41.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 -- 172.21.15.111:0/3338433167 learned_addr learned my addr 172.21.15.111:0/3338433167 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:41.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f91095ca700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f91040c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9109dcb700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91041595a0 0x7f91040af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 -- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91041595a0 msgr2=0x7f91040af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91041595a0 0x7f91040af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 -- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 msgr2=0x7f91040c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f91040c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 -- 172.21.15.111:0/3338433167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f40094f0 con 0x7f9104158c90 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f9108dc9700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400bbe0 con 0x7f9104158c90 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4011420 con 0x7f9104158c90 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91041550b0 con 0x7f9104158c90 2023-12-19T08:02:41.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f40117c0 con 0x7f9104158c90 2023-12-19T08:02:41.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.765+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91041556f0 con 0x7f9104158c90 2023-12-19T08:02:41.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9104155b70 con 0x7f9104158c90 2023-12-19T08:02:41.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f90f4011bf0 con 0x7f9104158c90 2023-12-19T08:02:41.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f90f97fa700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 0x7f90e8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:41.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f90f407e500 con 0x7f9104158c90 2023-12-19T08:02:41.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f91095ca700 1 -- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 msgr2=0x7f90e8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:41.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.769+0000 7f91095ca700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 0x7f90e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:41.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.777+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f90f404e450 con 0x7f9104158c90 2023-12-19T08:02:41.809 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:41.809+0000 7f706d15f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:41.850 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:41.849+0000 7fc302b0a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:02:41.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f91041583d0 con 0x7f9104158c90 2023-12-19T08:02:41.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f90f97fa700 1 -- 172.21.15.111:0/3338433167 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90f4051a70 con 0x7f9104158c90 2023-12-19T08:02:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 msgr2=0x7f90e8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 0x7f90e8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 msgr2=0x7f9104157e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4017000 comp rx=0 tx=0).stop 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 shutdown_connections 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f90e8060700 0x7f90e8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91041595a0 0x7f91040af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9104158c90 0x7f9104157e00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 --2- 172.21.15.111:0/3338433167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f910406aeb0 0x7f91040c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.949+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 >> 172.21.15.111:0/3338433167 conn(0x7f9104067bb0 msgr2=0x7f91040afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.953+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 shutdown_connections 2023-12-19T08:02:41.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:41.953+0000 7f910a5cc700 1 -- 172.21.15.111:0/3338433167 wait complete. 2023-12-19T08:02:41.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T08:02:42.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.117+0000 7f1a25dcc700 1 Processor -- start 2023-12-19T08:02:42.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.117+0000 7f1a25dcc700 1 -- start start 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a20142680 0x7f1a200c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a200c8f40 0x7f1a2006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1a20142a50 0x7f1a2006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a200cc7e0 con 0x7f1a200c8f40 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a200cbbf0 con 0x7f1a20142a50 2023-12-19T08:02:42.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200cbeb0 con 0x7f1a20142680 2023-12-19T08:02:42.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a200c8f40 0x7f1a2006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a200c8f40 0x7f1a2006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43268/0 (socket says 172.21.15.111:43268) 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a255cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1a20142a50 0x7f1a2006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 -- 172.21.15.111:0/3259402660 learned_addr learned my addr 172.21.15.111:0/3259402660 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a24dca700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a20142680 0x7f1a200c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 -- 172.21.15.111:0/3259402660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a20142680 msgr2=0x7f1a200c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a20142680 0x7f1a200c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 -- 172.21.15.111:0/3259402660 >> v2:172.21.15.143:3300/0 conn(0x7f1a20142a50 msgr2=0x7f1a2006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 --2- 172.21.15.111:0/3259402660 >> v2:172.21.15.143:3300/0 conn(0x7f1a20142a50 0x7f1a2006d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 -- 172.21.15.111:0/3259402660 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200cc190 con 0x7f1a200c8f40 2023-12-19T08:02:42.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1ffff700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a200c8f40 0x7f1a2006d050 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1a200ad410 tx=0x7f1a0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7caf06699ae1e0c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1f7fe700 1 -- 172.21.15.111:0/3259402660 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00b640 con 0x7f1a200c8f40 2023-12-19T08:02:42.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1f7fe700 1 -- 172.21.15.111:0/3259402660 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c00b7a0 con 0x7f1a200c8f40 2023-12-19T08:02:42.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a1f7fe700 1 -- 172.21.15.111:0/3259402660 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00bb60 con 0x7f1a200c8f40 2023-12-19T08:02:42.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3259402660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a200c8f40 msgr2=0x7f1a2006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a200c8f40 0x7f1a2006d050 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1a200ad410 tx=0x7f1a0c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3259402660 shutdown_connections 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3259402660 >> v2:172.21.15.143:3300/0 conn(0x7f1a20142a50 0x7f1a2006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a200c8f40 0x7f1a2006d050 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3259402660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a20142680 0x7f1a200c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3259402660 >> 172.21.15.111:0/3259402660 conn(0x7f1a200ccb30 msgr2=0x7f1a200ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3259402660 shutdown_connections 2023-12-19T08:02:42.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.121+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3259402660 wait complete. 2023-12-19T08:02:42.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 Processor -- start 2023-12-19T08:02:42.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 -- start start 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20142a50 0x7f1a200b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a200c59a0 0x7f1a20069230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200b2790 con 0x7f1a20142680 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200c3be0 con 0x7f1a20142a50 2023-12-19T08:02:42.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a25dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a201585d0 con 0x7f1a200c59a0 2023-12-19T08:02:42.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a24dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a24dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43274/0 (socket says 172.21.15.111:43274) 2023-12-19T08:02:42.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a24dca700 1 -- 172.21.15.111:0/3733547443 learned_addr learned my addr 172.21.15.111:0/3733547443 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:42.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a1ffff700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20142a50 0x7f1a200b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.125+0000 7f1a255cb700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a200c59a0 0x7f1a20069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 -- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a200c59a0 msgr2=0x7f1a20069230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a200c59a0 0x7f1a20069230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 -- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20142a50 msgr2=0x7f1a200b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20142a50 0x7f1a200b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 -- 172.21.15.111:0/3733547443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0094f0 con 0x7f1a20142680 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a24dca700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400ec80 tx=0x7f1a140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400c850 con 0x7f1a20142680 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1400c9d0 con 0x7f1a20142680 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400cd90 con 0x7f1a20142680 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a201588b0 con 0x7f1a20142680 2023-12-19T08:02:42.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a20158e70 con 0x7f1a20142680 2023-12-19T08:02:42.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a200c3d70 con 0x7f1a20142680 2023-12-19T08:02:42.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.129+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f1a14013680 con 0x7f1a20142680 2023-12-19T08:02:42.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.133+0000 7f1a1cff9700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 0x7f1a04062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.133+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f1a1407d310 con 0x7f1a20142680 2023-12-19T08:02:42.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.133+0000 7f1a1ffff700 1 -- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 msgr2=0x7f1a04062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:42.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.133+0000 7f1a1ffff700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 0x7f1a04062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:42.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.137+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a1404d140 con 0x7f1a20142680 2023-12-19T08:02:42.215 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:42.217+0000 7f706d15f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:42.266 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:42.265+0000 7fc302b0a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:02:42.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.305+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a2004be40 con 0x7f1a20142680 2023-12-19T08:02:42.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.305+0000 7f1a1cff9700 1 -- 172.21.15.111:0/3733547443 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a14050760 con 0x7f1a20142680 2023-12-19T08:02:42.307 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T08:02:42.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 msgr2=0x7f1a04062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:42.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 0x7f1a04062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 msgr2=0x7f1a20157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400ec80 tx=0x7f1a140091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 shutdown_connections 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a04060700 0x7f1a04062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20142a50 0x7f1a200b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a200c59a0 0x7f1a20069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 --2- 172.21.15.111:0/3733547443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a20142680 0x7f1a20157f00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 >> 172.21.15.111:0/3733547443 conn(0x7f1a200ccb30 msgr2=0x7f1a200cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 shutdown_connections 2023-12-19T08:02:42.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.309+0000 7f1a25dcc700 1 -- 172.21.15.111:0/3733547443 wait complete. 2023-12-19T08:02:42.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:42.386 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:42.385+0000 7f706d15f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:42.435 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:42.433+0000 7fc302b0a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:02:42.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 Processor -- start 2023-12-19T08:02:42.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 -- start start 2023-12-19T08:02:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff5c8067c90 0x7ff5c8068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80c6630 con 0x7ff5c80c5900 2023-12-19T08:02:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c80c6440 con 0x7ff5c8067c90 2023-12-19T08:02:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5cdf45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80c6910 con 0x7ff5c814fba0 2023-12-19T08:02:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5c77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff5c8067c90 0x7ff5c8068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5c7fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff5c80c5900 0x7ff5c80c5d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:51824/0 (socket says 172.21.15.111:51824) 2023-12-19T08:02:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5c7fff700 1 -- 172.21.15.111:0/3716647082 learned_addr learned my addr 172.21.15.111:0/3716647082 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.493+0000 7ff5c6ffd700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c77fe700 1 -- 172.21.15.111:0/3716647082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 msgr2=0x7ff5c814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c77fe700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c77fe700 1 -- 172.21.15.111:0/3716647082 >> v1:172.21.15.143:6789/0 conn(0x7ff5c80c5900 legacy=0x7ff5c80c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c77fe700 1 -- 172.21.15.111:0/3716647082 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c80c6bf0 con 0x7ff5c8067c90 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c77fe700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c8068060 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80098e0 tx=0x7ff5b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1766b0fb504034db server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c67fc700 1 -- 172.21.15.111:0/3716647082 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800b640 con 0x7ff5c8067c90 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c67fc700 1 -- 172.21.15.111:0/3716647082 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800b7a0 con 0x7ff5c8067c90 2023-12-19T08:02:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5c67fc700 1 -- 172.21.15.111:0/3716647082 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800bb40 con 0x7ff5c8067c90 2023-12-19T08:02:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 -- 172.21.15.111:0/3716647082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 msgr2=0x7ff5c8068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c8068060 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff5b80098e0 tx=0x7ff5b80091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 -- 172.21.15.111:0/3716647082 shutdown_connections 2023-12-19T08:02:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/3716647082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c8068060 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 -- 172.21.15.111:0/3716647082 >> 172.21.15.111:0/3716647082 conn(0x7ff5c80cd6d0 msgr2=0x7ff5c806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 -- 172.21.15.111:0/3716647082 shutdown_connections 2023-12-19T08:02:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.497+0000 7ff5cdf45700 1 -- 172.21.15.111:0/3716647082 wait complete. 2023-12-19T08:02:42.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 Processor -- start 2023-12-19T08:02:42.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 -- start start 2023-12-19T08:02:42.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c80c5900 0x7ff5c80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c815ca60 con 0x7ff5c8067c90 2023-12-19T08:02:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80b3990 con 0x7ff5c80c5900 2023-12-19T08:02:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5cdf45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80b3c70 con 0x7ff5c814fba0 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.501+0000 7ff5c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43292/0 (socket says 172.21.15.111:43292) 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 -- 172.21.15.111:0/1316459270 learned_addr learned my addr 172.21.15.111:0/1316459270 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c6ffd700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c80c5900 0x7ff5c80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c7fff700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 msgr2=0x7ff5c814c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c80c5900 msgr2=0x7ff5c80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c80c5900 0x7ff5c80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 -- 172.21.15.111:0/1316459270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8009420 con 0x7ff5c8067c90 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c7fff700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5c77fe700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff5c806a0f0 tx=0x7ff5b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b801bc20 con 0x7ff5c8067c90 2023-12-19T08:02:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c80b3f50 con 0x7ff5c8067c90 2023-12-19T08:02:42.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b801bd80 con 0x7ff5c8067c90 2023-12-19T08:02:42.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8011670 con 0x7ff5c8067c90 2023-12-19T08:02:42.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c815c5f0 con 0x7ff5c8067c90 2023-12-19T08:02:42.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.505+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c8071120 con 0x7ff5c8067c90 2023-12-19T08:02:42.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.509+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7ff5b80119e0 con 0x7ff5c8067c90 2023-12-19T08:02:42.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.509+0000 7ff5af7fe700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 0x7ff5b4062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.509+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7ff5b807e380 con 0x7ff5c8067c90 2023-12-19T08:02:42.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.509+0000 7ff5c6ffd700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 msgr2=0x7ff5b4062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:42.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.509+0000 7ff5c6ffd700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 0x7ff5b4062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:42.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.517+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff5b804e250 con 0x7ff5c8067c90 2023-12-19T08:02:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.713+0000 7ff5c6ffd700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 msgr2=0x7ff5b4062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.713+0000 7ff5c6ffd700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 0x7ff5b4062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.717+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c80642a0 con 0x7ff5c8067c90 2023-12-19T08:02:42.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5af7fe700 1 -- 172.21.15.111:0/1316459270 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7ff5b8051870 con 0x7ff5c8067c90 2023-12-19T08:02:42.722 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:42.722 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 msgr2=0x7ff5b4062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 0x7ff5b4062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 msgr2=0x7ff5c814bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff5c806a0f0 tx=0x7ff5b8017000 comp rx=0 tx=0).stop 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 shutdown_connections 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff5b4060730 0x7ff5b4062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c80c5900 0x7ff5c80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c814fba0 0x7ff5c814c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 --2- 172.21.15.111:0/1316459270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c8067c90 0x7ff5c814bd20 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.721+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 >> 172.21.15.111:0/1316459270 conn(0x7ff5c80cd6d0 msgr2=0x7ff5c80b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:42.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.725+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 shutdown_connections 2023-12-19T08:02:42.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.725+0000 7ff5cdf45700 1 -- 172.21.15.111:0/1316459270 wait complete. 2023-12-19T08:02:42.727 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:42.744 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:42.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:42.761+0000 7f706d15f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:42.820 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:42.817+0000 7fc302b0a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:02:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.893+0000 7f31fc678700 1 Processor -- start 2023-12-19T08:02:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.893+0000 7f31fc678700 1 -- start start 2023-12-19T08:02:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fc678700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31f4140ae0 0x7f31f40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fc678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 0x7f31f40ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fc678700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f406e150 con 0x7f31f4140710 2023-12-19T08:02:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fc678700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f4068ab0 con 0x7f31f4140ae0 2023-12-19T08:02:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fc678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f406d540 con 0x7f31f4068630 2023-12-19T08:02:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fa414700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f31f4140710 0x7f31f40c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43598/0 (socket says 172.21.15.111:43598) 2023-12-19T08:02:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fa414700 1 -- 172.21.15.111:0/2106558793 learned_addr learned my addr 172.21.15.111:0/2106558793 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 --2- 172.21.15.111:0/2106558793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 0x7f31f40ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9c13700 1 --2- 172.21.15.111:0/2106558793 >> v2:172.21.15.143:3300/0 conn(0x7f31f4140ae0 0x7f31f40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9412700 1 -- 172.21.15.111:0/2106558793 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776356117 0 0) 0x7f31f406e150 con 0x7f31f4140710 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9412700 1 -- 172.21.15.111:0/2106558793 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e0003570 con 0x7f31f4140710 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 -- 172.21.15.111:0/2106558793 >> v2:172.21.15.143:3300/0 conn(0x7f31f4140ae0 msgr2=0x7f31f40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 --2- 172.21.15.111:0/2106558793 >> v2:172.21.15.143:3300/0 conn(0x7f31f4140ae0 0x7f31f40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 -- 172.21.15.111:0/2106558793 >> v1:172.21.15.111:6789/0 conn(0x7f31f4140710 legacy=0x7f31f40c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 -- 172.21.15.111:0/2106558793 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f406d820 con 0x7f31f4068630 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31fac15700 1 --2- 172.21.15.111:0/2106558793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 0x7f31f40ac250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e60 tx=0x7f31ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d426aad89c9d962f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9412700 1 -- 172.21.15.111:0/2106558793 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec00e070 con 0x7f31f4068630 2023-12-19T08:02:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9412700 1 -- 172.21.15.111:0/2106558793 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec00bab0 con 0x7f31f4068630 2023-12-19T08:02:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.897+0000 7f31f9412700 1 -- 172.21.15.111:0/2106558793 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec00be70 con 0x7f31f4068630 2023-12-19T08:02:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 -- 172.21.15.111:0/2106558793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 msgr2=0x7f31f40ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 --2- 172.21.15.111:0/2106558793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 0x7f31f40ac250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009e60 tx=0x7f31ec009780 comp rx=0 tx=0).stop 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 -- 172.21.15.111:0/2106558793 shutdown_connections 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 --2- 172.21.15.111:0/2106558793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4068630 0x7f31f40ac250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 --2- 172.21.15.111:0/2106558793 >> v2:172.21.15.143:3300/0 conn(0x7f31f4140ae0 0x7f31f40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 -- 172.21.15.111:0/2106558793 >> 172.21.15.111:0/2106558793 conn(0x7f31f40c8850 msgr2=0x7f31f40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 -- 172.21.15.111:0/2106558793 shutdown_connections 2023-12-19T08:02:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.901+0000 7f31fc678700 1 -- 172.21.15.111:0/2106558793 wait complete. 2023-12-19T08:02:42.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 Processor -- start 2023-12-19T08:02:42.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 -- start start 2023-12-19T08:02:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f4068630 0x7f31f4166520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 0x7f31f40c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40c2fc0 con 0x7f31f4140ae0 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40c32a0 con 0x7f31f4068630 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f40c3580 con 0x7f31f4140710 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31f9c13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 0x7f31f40c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31f9c13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52534/0 (socket says 172.21.15.111:52534) 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31f9c13700 1 -- 172.21.15.111:0/1995164346 learned_addr learned my addr 172.21.15.111:0/1995164346 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fa414700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f4068630 0x7f31f4166520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 -- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 msgr2=0x7f31f415a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 -- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f4068630 msgr2=0x7f31f4166520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f4068630 0x7f31f4166520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 -- 172.21.15.111:0/1995164346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009a20 con 0x7f31f4140ae0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31f9c13700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fac15700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 0x7f31f40c5430 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f31ec016000 tx=0x7f31ec019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec012e10 con 0x7f31f4140ae0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec027470 con 0x7f31f4140ae0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f40c3860 con 0x7f31f4140ae0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.905+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec027830 con 0x7f31f4140ae0 2023-12-19T08:02:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f415a8b0 con 0x7f31f4140ae0 2023-12-19T08:02:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f4071120 con 0x7f31f4140ae0 2023-12-19T08:02:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f31ec024070 con 0x7f31f4140ae0 2023-12-19T08:02:42.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31ea7fc700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 0x7f31d8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f31ec07f540 con 0x7f31f4140ae0 2023-12-19T08:02:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31fa414700 1 -- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 msgr2=0x7f31d8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.909+0000 7f31fa414700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 0x7f31d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:42.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:42.917+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f31ec04f370 con 0x7f31f4140ae0 2023-12-19T08:02:42.981 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:42.981+0000 7f706d15f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:43.045 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.041+0000 7fc302b0a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:02:43.071 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.073+0000 7f706d15f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:43.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.085+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31f40642a0 con 0x7f31f4140ae0 2023-12-19T08:02:43.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31ea7fc700 1 -- 172.21.15.111:0/1995164346 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16893 (secure 0 0 0) 0x7f31ec052990 con 0x7f31f4140ae0 2023-12-19T08:02:43.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:43.090 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:38.058416+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2190915022":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1956986802":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1938739052":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/3973057301":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/2537901821":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6833/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6832/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1521230848":"2023-12-20T08:02:38.058339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:43.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 msgr2=0x7f31d8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:43.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 0x7f31d8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 msgr2=0x7f31f40c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 0x7f31f40c5430 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f31ec016000 tx=0x7f31ec019000 comp rx=0 tx=0).stop 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 shutdown_connections 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31d8060700 0x7f31d8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31f4140ae0 0x7f31f40c5430 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f4140710 0x7f31f415a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 --2- 172.21.15.111:0/1995164346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f4068630 0x7f31f4166520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 >> 172.21.15.111:0/1995164346 conn(0x7f31f40c8850 msgr2=0x7f31f406a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:43.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 shutdown_connections 2023-12-19T08:02:43.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.089+0000 7f31fc678700 1 -- 172.21.15.111:0/1995164346 wait complete. 2023-12-19T08:02:43.113 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T08:02:43.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:43.138 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.137+0000 7fc302b0a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:02:43.142 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.145+0000 7f706d15f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:43.220 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.221+0000 7f706d15f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:43.221 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.221+0000 7fc302b0a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:02:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.269+0000 7f3018a25700 1 Processor -- start 2023-12-19T08:02:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 -- start start 2023-12-19T08:02:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3014069070 0x7f3014067960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30140cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3014068be0 con 0x7f3014158d40 2023-12-19T08:02:43.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3014159530 con 0x7f3014069070 2023-12-19T08:02:43.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3018a25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3014067fd0 con 0x7f3014159110 2023-12-19T08:02:43.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f301259c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3014158d40 0x7f30140cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:51840/0 (socket says 172.21.15.111:51840) 2023-12-19T08:02:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f301259c700 1 -- 172.21.15.111:0/4291825445 learned_addr learned my addr 172.21.15.111:0/4291825445 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3011d9b700 1 --2- 172.21.15.111:0/4291825445 >> v2:172.21.15.143:3300/0 conn(0x7f3014069070 0x7f3014067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.273+0000 7f3012d9d700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30140cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3011d9b700 1 -- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 msgr2=0x7f30140cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3011d9b700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30140cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3011d9b700 1 -- 172.21.15.111:0/4291825445 >> v1:172.21.15.143:6789/0 conn(0x7f3014158d40 legacy=0x7f30140cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f301159a700 1 -- 172.21.15.111:0/4291825445 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060434322 0 0) 0x7f3014068be0 con 0x7f3014158d40 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3011d9b700 1 -- 172.21.15.111:0/4291825445 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30140682b0 con 0x7f3014069070 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3012d9d700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30140cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3011d9b700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f3014067960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30080098e0 tx=0x7f30080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebfc616e5cd379a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f301159a700 1 -- 172.21.15.111:0/4291825445 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300800b640 con 0x7f3014069070 2023-12-19T08:02:43.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f301159a700 1 -- 172.21.15.111:0/4291825445 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300800b7a0 con 0x7f3014069070 2023-12-19T08:02:43.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f301159a700 1 -- 172.21.15.111:0/4291825445 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300800bb40 con 0x7f3014069070 2023-12-19T08:02:43.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 -- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 msgr2=0x7f3014067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f3014067960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f30080098e0 tx=0x7f30080091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 -- 172.21.15.111:0/4291825445 shutdown_connections 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30140cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 --2- 172.21.15.111:0/4291825445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f3014067960 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 -- 172.21.15.111:0/4291825445 >> 172.21.15.111:0/4291825445 conn(0x7f30140ccc40 msgr2=0x7f30140cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 -- 172.21.15.111:0/4291825445 shutdown_connections 2023-12-19T08:02:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.277+0000 7f3018a25700 1 -- 172.21.15.111:0/4291825445 wait complete. 2023-12-19T08:02:43.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 Processor -- start 2023-12-19T08:02:43.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- start start 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f30140afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 0x7f3014169d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30141535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30140b0010 con 0x7f3014158d40 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30140b0ae0 con 0x7f3014069070 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30141599e0 con 0x7f3014159110 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3011d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 0x7f3014169d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3011d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 0x7f3014169d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43304/0 (socket says 172.21.15.111:43304) 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3012d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30141535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3011d9b700 1 -- 172.21.15.111:0/1803858400 learned_addr learned my addr 172.21.15.111:0/1803858400 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f30140afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 msgr2=0x7f30141535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30141535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 msgr2=0x7f3014169d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 0x7f3014169d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 -- 172.21.15.111:0/1803858400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c009a20 con 0x7f3014069070 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f301259c700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f30140afad0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f300400c5d0 tx=0x7f3004007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004005cd0 con 0x7f3014069070 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3004005e30 con 0x7f3014069070 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3008009420 con 0x7f3014069070 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004007300 con 0x7f3014069070 2023-12-19T08:02:43.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.281+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30140b0240 con 0x7f3014069070 2023-12-19T08:02:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f300401b3b0 con 0x7f3014069070 2023-12-19T08:02:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3014071120 con 0x7f3014069070 2023-12-19T08:02:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f30027fc700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 0x7f2ff4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f300407c5f0 con 0x7f3014069070 2023-12-19T08:02:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f3011d9b700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 msgr2=0x7f2ff4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.285+0000 7f3011d9b700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 0x7f2ff4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:43.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.297+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f300404c4e0 con 0x7f3014069070 2023-12-19T08:02:43.301 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.301+0000 7fc302b0a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:02:43.442 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.441+0000 7f706d15f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:43.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.489+0000 7f3011d9b700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 msgr2=0x7f2ff4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:43.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.489+0000 7f3011d9b700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 0x7f2ff4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:43.517 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.517+0000 7f706d15f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:43.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.517+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3014068590 con 0x7f3014069070 2023-12-19T08:02:43.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f30027fc700 1 -- 172.21.15.111:0/1803858400 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f3004055070 con 0x7f3014069070 2023-12-19T08:02:43.522 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:43.522 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 msgr2=0x7f2ff4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 0x7f2ff4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 msgr2=0x7f30140afad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f30140afad0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f300400c5d0 tx=0x7f3004007d30 comp rx=0 tx=0).stop 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 shutdown_connections 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ff4060700 0x7f2ff4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3014159110 0x7f30141535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3014158d40 0x7f3014169d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 --2- 172.21.15.111:0/1803858400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3014069070 0x7f30140afad0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.521+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 >> 172.21.15.111:0/1803858400 conn(0x7f30140ccc40 msgr2=0x7f30141576a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:43.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.525+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 shutdown_connections 2023-12-19T08:02:43.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.525+0000 7f3018a25700 1 -- 172.21.15.111:0/1803858400 wait complete. 2023-12-19T08:02:43.527 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:43.537 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.537+0000 7fc302b0a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:02:43.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:43.600 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.601+0000 7f706d15f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:43.622 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.621+0000 7fc302b0a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:02:43.719 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.717+0000 7fc302b0a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:02:43.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 Processor -- start 2023-12-19T08:02:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 -- start start 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23b415d320 0x7f23b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 0x7f23b40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 0x7f23b40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b40c7190 con 0x7f23b415d320 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b40c1b50 con 0x7f23b40c15b0 2023-12-19T08:02:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23bc02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b40c1e30 con 0x7f23b406d690 2023-12-19T08:02:43.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23b415d320 0x7f23b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23b415d320 0x7f23b415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43314/0 (socket says 172.21.15.111:43314) 2023-12-19T08:02:43.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 -- 172.21.15.111:0/2861451560 learned_addr learned my addr 172.21.15.111:0/2861451560 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b95c9700 1 --2- 172.21.15.111:0/2861451560 >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 0x7f23b40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23ba5cb700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 0x7f23b40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 -- 172.21.15.111:0/2861451560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 msgr2=0x7f23b40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 0x7f23b40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.737+0000 7f23b9dca700 1 -- 172.21.15.111:0/2861451560 >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 msgr2=0x7f23b40aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b9dca700 1 --2- 172.21.15.111:0/2861451560 >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 0x7f23b40aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b9dca700 1 -- 172.21.15.111:0/2861451560 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40c6e20 con 0x7f23b415d320 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23ba5cb700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 0x7f23b40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b95c9700 1 --2- 172.21.15.111:0/2861451560 >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 0x7f23b40aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b9dca700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b415d6f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f23b406f530 tx=0x7f23b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be818c985c677e2f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b8dc8700 1 -- 172.21.15.111:0/2861451560 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b000b860 con 0x7f23b415d320 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b8dc8700 1 -- 172.21.15.111:0/2861451560 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b000b9c0 con 0x7f23b415d320 2023-12-19T08:02:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23b8dc8700 1 -- 172.21.15.111:0/2861451560 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b000bd60 con 0x7f23b415d320 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 -- 172.21.15.111:0/2861451560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 msgr2=0x7f23b415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b415d6f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f23b406f530 tx=0x7f23b0009250 comp rx=0 tx=0).stop 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 -- 172.21.15.111:0/2861451560 shutdown_connections 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b406d690 0x7f23b40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 --2- 172.21.15.111:0/2861451560 >> v2:172.21.15.143:3300/0 conn(0x7f23b40c15b0 0x7f23b40aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 --2- 172.21.15.111:0/2861451560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b415d6f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 -- 172.21.15.111:0/2861451560 >> 172.21.15.111:0/2861451560 conn(0x7f23b40c8850 msgr2=0x7f23b40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 -- 172.21.15.111:0/2861451560 shutdown_connections 2023-12-19T08:02:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.741+0000 7f23bc02e700 1 -- 172.21.15.111:0/2861451560 wait complete. 2023-12-19T08:02:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 Processor -- start 2023-12-19T08:02:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 -- start start 2023-12-19T08:02:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b406d690 0x7f23b4171220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b40c15b0 0x7f23b4146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b4160320 con 0x7f23b415d320 2023-12-19T08:02:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b4162cf0 con 0x7f23b406d690 2023-12-19T08:02:43.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23bc02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b40c4630 con 0x7f23b40c15b0 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43330/0 (socket says 172.21.15.111:43330) 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23b9dca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b406d690 0x7f23b4171220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 -- 172.21.15.111:0/1444106499 learned_addr learned my addr 172.21.15.111:0/1444106499 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23b95c9700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b40c15b0 0x7f23b4146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b40c15b0 msgr2=0x7f23b4146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b40c15b0 0x7f23b4146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b406d690 msgr2=0x7f23b4171220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b406d690 0x7f23b4171220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 -- 172.21.15.111:0/1444106499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00094f0 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23ba5cb700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00ce40 tx=0x7f23ac0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.745+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac00ac70 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00adf0 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b4162a10 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac016670 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b4160040 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b41717c0 con 0x7f23b415d320 2023-12-19T08:02:43.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.749+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f23ac016a20 con 0x7f23b415d320 2023-12-19T08:02:43.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.753+0000 7f23a9ffb700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 0x7f2398062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:43.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.753+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f23ac012070 con 0x7f23b415d320 2023-12-19T08:02:43.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.753+0000 7f23b9dca700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 msgr2=0x7f2398062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:43.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.753+0000 7f23b9dca700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 0x7f2398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:43.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.757+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23ac04cbe0 con 0x7f23b415d320 2023-12-19T08:02:43.787 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.789+0000 7f706d15f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:43.875 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:43.877+0000 7f706d15f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:43.929 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:43.925+0000 7fc302b0a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:02:43.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.957+0000 7f23b9dca700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 msgr2=0x7f2398062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:43.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.957+0000 7f23b9dca700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 0x7f2398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:02:43.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.965+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23b4071120 con 0x7f23b415d320 2023-12-19T08:02:43.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.965+0000 7f23a9ffb700 1 -- 172.21.15.111:0/1444106499 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4604 (secure 0 0 0) 0x7f23ac04d5a0 con 0x7f23b415d320 2023-12-19T08:02:43.966 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:43.966 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:23.022271+0000","modified":"2023-12-19T08:02:26.633728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/82065769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":82065769},{"type":"v1","addr":"172.21.15.111:6835","nonce":82065769}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.965+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 msgr2=0x7f2398062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.965+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 0x7f2398062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 msgr2=0x7f23b40ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00ce40 tx=0x7f23ac0077d0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 shutdown_connections 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2398060700 0x7f2398062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23b415d320 0x7f23b40ca2e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23b40c15b0 0x7f23b4146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 --2- 172.21.15.111:0/1444106499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b406d690 0x7f23b4171220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 >> 172.21.15.111:0/1444106499 conn(0x7f23b40c8850 msgr2=0x7f23b4067970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 shutdown_connections 2023-12-19T08:02:43.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:43.969+0000 7f23bc02e700 1 -- 172.21.15.111:0/1444106499 wait complete. 2023-12-19T08:02:43.972 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-12-19T08:02:43.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:44.024 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:44.021+0000 7fc302b0a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:02:44.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 Processor -- start 2023-12-19T08:02:44.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 -- start start 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3dc067c90 0x7fd3dc068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3dc14fba0 0x7fd3dc150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc0c5900 0x7fd3dc0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc0c6630 con 0x7fd3dc14fba0 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc0c6910 con 0x7fd3dc067c90 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.181+0000 7fd3e0b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc0c6bf0 con 0x7fd3dc0c5900 2023-12-19T08:02:44.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3dc14fba0 0x7fd3dc150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3dc14fba0 0x7fd3dc150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43340/0 (socket says 172.21.15.111:43340) 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 -- 172.21.15.111:0/1965848953 learned_addr learned my addr 172.21.15.111:0/1965848953 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3dad9d700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc0c5900 0x7fd3dc0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3da59c700 1 --2- 172.21.15.111:0/1965848953 >> v2:172.21.15.143:3300/0 conn(0x7fd3dc067c90 0x7fd3dc068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 -- 172.21.15.111:0/1965848953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc0c5900 msgr2=0x7fd3dc0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc0c5900 0x7fd3dc0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 -- 172.21.15.111:0/1965848953 >> v2:172.21.15.143:3300/0 conn(0x7fd3dc067c90 msgr2=0x7fd3dc068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/1965848953 >> v2:172.21.15.143:3300/0 conn(0x7fd3dc067c90 0x7fd3dc068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 -- 172.21.15.111:0/1965848953 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0c6ed0 con 0x7fd3dc14fba0 2023-12-19T08:02:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc14fba0 0x7fd3dc150000 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc06a0c0 tx=0x7fd3d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f50e526d2ccbfe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d959a700 1 -- 172.21.15.111:0/1965848953 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000b640 con 0x7fd3dc14fba0 2023-12-19T08:02:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d959a700 1 -- 172.21.15.111:0/1965848953 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d000b7a0 con 0x7fd3dc14fba0 2023-12-19T08:02:44.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3d959a700 1 -- 172.21.15.111:0/1965848953 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000bb60 con 0x7fd3dc14fba0 2023-12-19T08:02:44.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 -- 172.21.15.111:0/1965848953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc14fba0 msgr2=0x7fd3dc150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc14fba0 0x7fd3dc150000 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc06a0c0 tx=0x7fd3d0009250 comp rx=0 tx=0).stop 2023-12-19T08:02:44.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 -- 172.21.15.111:0/1965848953 shutdown_connections 2023-12-19T08:02:44.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc0c5900 0x7fd3dc0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/1965848953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc14fba0 0x7fd3dc150000 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/1965848953 >> v2:172.21.15.143:3300/0 conn(0x7fd3dc067c90 0x7fd3dc068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 -- 172.21.15.111:0/1965848953 >> 172.21.15.111:0/1965848953 conn(0x7fd3dc0cd6d0 msgr2=0x7fd3dc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:44.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 -- 172.21.15.111:0/1965848953 shutdown_connections 2023-12-19T08:02:44.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.185+0000 7fd3e0b85700 1 -- 172.21.15.111:0/1965848953 wait complete. 2023-12-19T08:02:44.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 Processor -- start 2023-12-19T08:02:44.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 -- start start 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 0x7fd3dc0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 0x7fd3dc16a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc150e90 con 0x7fd3dc067c90 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc16a650 con 0x7fd3dc0c5900 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3e0b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc0ca140 con 0x7fd3dc14fba0 2023-12-19T08:02:44.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43356/0 (socket says 172.21.15.111:43356) 2023-12-19T08:02:44.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 -- 172.21.15.111:0/491980945 learned_addr learned my addr 172.21.15.111:0/491980945 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 0x7fd3dc0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3dad9d700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 0x7fd3dc16a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 -- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 msgr2=0x7fd3dc16a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 0x7fd3dc16a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 -- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 msgr2=0x7fd3dc0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 0x7fd3dc0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3da59c700 1 -- 172.21.15.111:0/491980945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00094f0 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 0x7fd3dc0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.189+0000 7fd3dad9d700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 0x7fd3dc16a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3da59c700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00ec80 tx=0x7fd3cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00c9d0 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3cc00cb30 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc013470 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc14ffa0 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc150560 con 0x7fd3dc067c90 2023-12-19T08:02:44.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc071120 con 0x7fd3dc067c90 2023-12-19T08:02:44.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.193+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fd3cc013820 con 0x7fd3dc067c90 2023-12-19T08:02:44.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.197+0000 7fd3ca7fc700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 0x7fd3bc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.197+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fd3cc07d370 con 0x7fd3dc067c90 2023-12-19T08:02:44.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.197+0000 7fd3d9d9b700 1 -- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 msgr2=0x7fd3bc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:44.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.197+0000 7fd3d9d9b700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 0x7fd3bc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:44.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.201+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd3cc04d1a0 con 0x7fd3dc067c90 2023-12-19T08:02:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.377+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3dc0642a0 con 0x7fd3dc067c90 2023-12-19T08:02:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.377+0000 7fd3ca7fc700 1 -- 172.21.15.111:0/491980945 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16893 (secure 0 0 0) 0x7fd3cc0507c0 con 0x7fd3dc067c90 2023-12-19T08:02:44.379 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:44.379 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:38.058416+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:21.020848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:22.062351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2190915022":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1956986802":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1938739052":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/3973057301":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/2537901821":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6833/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6832/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1521230848":"2023-12-20T08:02:38.058339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 msgr2=0x7fd3bc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 0x7fd3bc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 msgr2=0x7fd3dc16a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00ec80 tx=0x7fd3cc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 shutdown_connections 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd3bc060700 0x7fd3bc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc0c5900 0x7fd3dc0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3dc14fba0 0x7fd3dc16a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 --2- 172.21.15.111:0/491980945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3dc067c90 0x7fd3dc16a110 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 >> 172.21.15.111:0/491980945 conn(0x7fd3dc0cd6d0 msgr2=0x7fd3dc14ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 shutdown_connections 2023-12-19T08:02:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.381+0000 7fd3e0b85700 1 -- 172.21.15.111:0/491980945 wait complete. 2023-12-19T08:02:44.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T08:02:44.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.561+0000 7f1639b45700 1 Processor -- start 2023-12-19T08:02:44.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.561+0000 7f1639b45700 1 -- start start 2023-12-19T08:02:44.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16340ad170 0x7f1634159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 0x7f1634156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340b3a10 con 0x7f16340b1b90 2023-12-19T08:02:44.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1634068b90 con 0x7f16340ad170 2023-12-19T08:02:44.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1639b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163415a400 con 0x7f1634067aa0 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1633fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 0x7f1634156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16340ad170 0x7f1634159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1633fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 0x7f1634156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43366/0 (socket says 172.21.15.111:43366) 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1633fff700 1 -- 172.21.15.111:0/222910208 learned_addr learned my addr 172.21.15.111:0/222910208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f1632ffd700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 -- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 msgr2=0x7f16340b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 -- 172.21.15.111:0/222910208 >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 msgr2=0x7f1634156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 --2- 172.21.15.111:0/222910208 >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 0x7f1634156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.565+0000 7f16337fe700 1 -- 172.21.15.111:0/222910208 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340ad540 con 0x7f16340ad170 2023-12-19T08:02:44.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f16337fe700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f1634159ec0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16340ad730 tx=0x7f16240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5297488d4a135be server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:44.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f16327fc700 1 -- 172.21.15.111:0/222910208 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624011630 con 0x7f16340ad170 2023-12-19T08:02:44.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f16327fc700 1 -- 172.21.15.111:0/222910208 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1624011790 con 0x7f16340ad170 2023-12-19T08:02:44.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f16327fc700 1 -- 172.21.15.111:0/222910208 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624011b50 con 0x7f16340ad170 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 -- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 msgr2=0x7f1634159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f1634159ec0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f16340ad730 tx=0x7f16240091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 -- 172.21.15.111:0/222910208 shutdown_connections 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 --2- 172.21.15.111:0/222910208 >> v2:172.21.15.111:3300/0 conn(0x7f16340b1b90 0x7f1634156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 --2- 172.21.15.111:0/222910208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f1634159ec0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 -- 172.21.15.111:0/222910208 >> 172.21.15.111:0/222910208 conn(0x7f16340cdda0 msgr2=0x7f16340ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:44.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 -- 172.21.15.111:0/222910208 shutdown_connections 2023-12-19T08:02:44.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.569+0000 7f1639b45700 1 -- 172.21.15.111:0/222910208 wait complete. 2023-12-19T08:02:44.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 Processor -- start 2023-12-19T08:02:44.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 -- start start 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f16341568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634157320 con 0x7f16340b1b90 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340caab0 con 0x7f16340ad170 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1639b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340c8800 con 0x7f1634067aa0 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43374/0 (socket says 172.21.15.111:43374) 2023-12-19T08:02:44.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 -- 172.21.15.111:0/2726414988 learned_addr learned my addr 172.21.15.111:0/2726414988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1632ffd700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f16341568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f16337fe700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 -- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 msgr2=0x7f16340cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 -- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 msgr2=0x7f16341568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f16341568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.573+0000 7f1633fff700 1 -- 172.21.15.111:0/2726414988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1624009420 con 0x7f16340b1b90 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f1633fff700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f162c009db0 tx=0x7f162c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00c890 con 0x7f16340b1b90 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00f040 con 0x7f16340b1b90 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00cc40 con 0x7f16340b1b90 2023-12-19T08:02:44.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16340ca180 con 0x7f16340b1b90 2023-12-19T08:02:44.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16340ca740 con 0x7f16340b1b90 2023-12-19T08:02:44.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634071120 con 0x7f16340b1b90 2023-12-19T08:02:44.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.577+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f162c014650 con 0x7f16340b1b90 2023-12-19T08:02:44.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.581+0000 7f161b7fe700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 0x7f1620062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:44.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.581+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f162c07d5e0 con 0x7f16340b1b90 2023-12-19T08:02:44.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.581+0000 7f16337fe700 1 -- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 msgr2=0x7f1620062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:02:44.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.581+0000 7f16337fe700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 0x7f1620062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:02:44.587 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.589+0000 7f706d15f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:44.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.585+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f162c04d3a0 con 0x7f16340b1b90 2023-12-19T08:02:44.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.601+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 32) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f162c04d770 con 0x7f16340b1b90 2023-12-19T08:02:44.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.601+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 msgr2=0x7f1620062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:02:44.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.601+0000 7f161b7fe700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 0x7f1620062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f70374e0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f70374e0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f70374e0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f70374e0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.696 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f70374e0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.697 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f7033cd9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.697 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f7033cd9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.697 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f7033cd9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.697 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f7033cd9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.697 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:02:44.697+0000 7f7033cd9700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:02:44.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.809+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1634067460 con 0x7f16340b1b90 2023-12-19T08:02:44.812 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T08:02:44.809+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T08:02:44.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.821+0000 7f161b7fe700 1 -- 172.21.15.111:0/2726414988 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7f162c069b00 con 0x7f16340b1b90 2023-12-19T08:02:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 msgr2=0x7f1634156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f162c009db0 tx=0x7f162c00b040 comp rx=0 tx=0).stop 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 shutdown_connections 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1620060700 0x7f1620062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16340ad170 0x7f16341568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16340b1b90 0x7f1634156de0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 --2- 172.21.15.111:0/2726414988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1634067aa0 0x7f16340cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 >> 172.21.15.111:0/2726414988 conn(0x7f16340cdda0 msgr2=0x7f16340ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 shutdown_connections 2023-12-19T08:02:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.825+0000 7f1639b45700 1 -- 172.21.15.111:0/2726414988 wait complete. 2023-12-19T08:02:44.828 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T08:02:44.831 INFO:tasks.ceph.mds.a.smithi111.stderr: -2525> 2023-12-19T08:02:28.557+0000 7f38717f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:02:44.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T08:02:44.851 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:02:44.849+0000 7fc302b0a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:02:44.879 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:02:45.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.997+0000 7f05cf06b700 1 Processor -- start 2023-12-19T08:02:45.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:44.997+0000 7f05cf06b700 1 -- start start 2023-12-19T08:02:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cf06b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05c815a010 0x7f05c80c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cf06b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c8159b80 con 0x7f05c8141e70 2023-12-19T08:02:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cf06b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c80c8220 con 0x7f05c8142690 2023-12-19T08:02:45.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cf06b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80c7f40 con 0x7f05c815a010 2023-12-19T08:02:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05ce069700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f05c8141e70 0x7f05c80c7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43606/0 (socket says 172.21.15.111:43606) 2023-12-19T08:02:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05ce069700 1 -- 172.21.15.111:0/4105099501 learned_addr learned my addr 172.21.15.111:0/4105099501 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:45.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd868700 1 --2- 172.21.15.111:0/4105099501 >> v2:172.21.15.143:3300/0 conn(0x7f05c815a010 0x7f05c80c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877533524 0 0) 0x7f05c8159b80 con 0x7f05c8141e70 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b4003570 con 0x7f05c8141e70 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd868700 1 -- 172.21.15.111:0/4105099501 >> v1:172.21.15.143:6789/0 conn(0x7f05c8142690 legacy=0x7f05c815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914432577 0 0) 0x7f05c80c8220 con 0x7f05c8142690 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd868700 1 -- 172.21.15.111:0/4105099501 >> v1:172.21.15.111:6789/0 conn(0x7f05c8141e70 legacy=0x7f05c80c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.001+0000 7f05cd868700 1 -- 172.21.15.111:0/4105099501 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c8159530 con 0x7f05c815a010 2023-12-19T08:02:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cd868700 1 --2- 172.21.15.111:0/4105099501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c815a010 0x7f05c80c4ce0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f05b80098e0 tx=0x7f05b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ccc7a0235c500d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:45.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b800b640 con 0x7f05c815a010 2023-12-19T08:02:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b800b7a0 con 0x7f05c815a010 2023-12-19T08:02:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cd067700 1 -- 172.21.15.111:0/4105099501 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b800bb40 con 0x7f05c815a010 2023-12-19T08:02:45.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 -- 172.21.15.111:0/4105099501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c815a010 msgr2=0x7f05c80c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 --2- 172.21.15.111:0/4105099501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c815a010 0x7f05c80c4ce0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f05b80098e0 tx=0x7f05b80091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 -- 172.21.15.111:0/4105099501 shutdown_connections 2023-12-19T08:02:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 --2- 172.21.15.111:0/4105099501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c815a010 0x7f05c80c4ce0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 -- 172.21.15.111:0/4105099501 >> 172.21.15.111:0/4105099501 conn(0x7f05c8067b50 msgr2=0x7f05c80cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 -- 172.21.15.111:0/4105099501 shutdown_connections 2023-12-19T08:02:45.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.005+0000 7f05cf06b700 1 -- 172.21.15.111:0/4105099501 wait complete. 2023-12-19T08:02:45.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 Processor -- start 2023-12-19T08:02:45.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 -- start start 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c8141e70 0x7f05c813bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05c8142690 0x7f05c8158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813adb0 con 0x7f05c815a010 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80b18e0 con 0x7f05c8141e70 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cf06b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813afc0 con 0x7f05c8142690 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05ce86a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:45.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05ce86a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43388/0 (socket says 172.21.15.111:43388) 2023-12-19T08:02:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05ce86a700 1 -- 172.21.15.111:0/2160561765 learned_addr learned my addr 172.21.15.111:0/2160561765 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05cd868700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05c8142690 0x7f05c8158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:45.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.009+0000 7f05ce069700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c8141e70 0x7f05c813bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 -- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05c8142690 msgr2=0x7f05c8158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05c8142690 0x7f05c8158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 -- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c8141e70 msgr2=0x7f05c813bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c8141e70 0x7f05c813bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 -- 172.21.15.111:0/2160561765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8009420 con 0x7f05c815a010 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05ce86a700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f05c0001560 tx=0x7f05c0001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c00040d0 con 0x7f05c815a010 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c000a7e0 con 0x7f05c815a010 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c000aba0 con 0x7f05c815a010 2023-12-19T08:02:45.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80b1290 con 0x7f05c815a010 2023-12-19T08:02:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c813c260 con 0x7f05c815a010 2023-12-19T08:02:45.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.013+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c804be40 con 0x7f05c815a010 2023-12-19T08:02:45.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.017+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 32) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f05c0012420 con 0x7f05c815a010 2023-12-19T08:02:45.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.017+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f05c007c670 con 0x7f05c815a010 2023-12-19T08:02:45.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.021+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f05c004c910 con 0x7f05c815a010 2023-12-19T08:02:45.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.229+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f05c8159810 con 0x7f05c815a010 2023-12-19T08:02:45.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.613+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 33) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f05c004c3c0 con 0x7f05c815a010 2023-12-19T08:02:45.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.613+0000 7f05be7fc700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 0x7f05ac068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:45.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.617+0000 7f05ce069700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 0x7f05ac068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:45.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.617+0000 7f05ce069700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 0x7f05ac068cb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f05c813d880 tx=0x7f05c4009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:45.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.825+0000 7f05be7fc700 1 -- 172.21.15.111:0/2160561765 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f05c0069190 con 0x7f05c815a010 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 msgr2=0x7f05ac068cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 0x7f05ac068cb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f05c813d880 tx=0x7f05c4009040 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 msgr2=0x7f05c8140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f05c0001560 tx=0x7f05c0001770 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 shutdown_connections 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f05ac0668d0 0x7f05ac068cb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c815a010 0x7f05c8140830 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05c8142690 0x7f05c8158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 --2- 172.21.15.111:0/2160561765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c8141e70 0x7f05c813bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 >> 172.21.15.111:0/2160561765 conn(0x7f05c8067b50 msgr2=0x7f05c806d600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 shutdown_connections 2023-12-19T08:02:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:45.829+0000 7f05cf06b700 1 -- 172.21.15.111:0/2160561765 wait complete. 2023-12-19T08:02:45.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T08:02:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.081+0000 7fadae66e700 1 Processor -- start 2023-12-19T08:02:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.081+0000 7fadae66e700 1 -- start start 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadae66e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fada806d760 0x7fada815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadae66e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada815dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadae66e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80adfa0 con 0x7fada815d360 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadae66e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80ad990 con 0x7fada806d760 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadae66e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada815e5a0 con 0x7fada815d730 2023-12-19T08:02:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fada806d760 0x7fada815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fada806d760 0x7fada815eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43396/0 (socket says 172.21.15.111:43396) 2023-12-19T08:02:46.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 -- 172.21.15.111:0/887599769 learned_addr learned my addr 172.21.15.111:0/887599769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:46.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fada815d360 0x7fada806d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43614/0 (socket says 172.21.15.111:43614) 2023-12-19T08:02:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fadacc0b700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada815dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 -- 172.21.15.111:0/887599769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 msgr2=0x7fada815dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada815dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 -- 172.21.15.111:0/887599769 >> v1:172.21.15.111:6789/0 conn(0x7fada815d360 legacy=0x7fada806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 -- 172.21.15.111:0/887599769 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada815e880 con 0x7fada806d760 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.085+0000 7fada77fe700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada815eec0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0099b0 tx=0x7fad9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e2797ebfa846e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fada6ffd700 1 -- 172.21.15.111:0/887599769 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00b860 con 0x7fada806d760 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fada6ffd700 1 -- 172.21.15.111:0/887599769 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c00b9c0 con 0x7fada806d760 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fada6ffd700 1 -- 172.21.15.111:0/887599769 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00bd60 con 0x7fada806d760 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- 172.21.15.111:0/887599769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 msgr2=0x7fada815eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada815eec0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0099b0 tx=0x7fad9c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- 172.21.15.111:0/887599769 shutdown_connections 2023-12-19T08:02:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada815dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 --2- 172.21.15.111:0/887599769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada815eec0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- 172.21.15.111:0/887599769 >> 172.21.15.111:0/887599769 conn(0x7fada80ccc20 msgr2=0x7fada80cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- 172.21.15.111:0/887599769 shutdown_connections 2023-12-19T08:02:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- 172.21.15.111:0/887599769 wait complete. 2023-12-19T08:02:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 Processor -- start 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.089+0000 7fadae66e700 1 -- start start 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada815d360 0x7fada8153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada80c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8069fd0 con 0x7fada806d760 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80b15a0 con 0x7fada815d360 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada8067860 con 0x7fada815d730 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43410/0 (socket says 172.21.15.111:43410) 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 -- 172.21.15.111:0/4084192062 learned_addr learned my addr 172.21.15.111:0/4084192062 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadacc0b700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada80c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada77fe700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada815d360 0x7fada8153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 -- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 msgr2=0x7fada80c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada80c2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 -- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada815d360 msgr2=0x7fada8153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada815d360 0x7fada8153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 -- 172.21.15.111:0/4084192062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c0094f0 con 0x7fada806d760 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fada7fff700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c7c0 tx=0x7fad98007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98005a70 con 0x7fada806d760 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad98005bf0 con 0x7fada806d760 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad980070d0 con 0x7fada806d760 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada80b1020 con 0x7fada806d760 2023-12-19T08:02:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.093+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada80b12c0 con 0x7fada806d760 2023-12-19T08:02:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.097+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8071120 con 0x7fada806d760 2023-12-19T08:02:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.097+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fad98007440 con 0x7fada806d760 2023-12-19T08:02:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.097+0000 7fad8ffff700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 0x7fad94062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.097+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fad9807bee0 con 0x7fada806d760 2023-12-19T08:02:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.101+0000 7fada77fe700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 0x7fad94062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.101+0000 7fada77fe700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 0x7fad94062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0099b0 tx=0x7fad9c017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:46.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.105+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad9804c090 con 0x7fada806d760 2023-12-19T08:02:46.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.269+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fada806a110 con 0x7fada806d760 2023-12-19T08:02:46.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.829+0000 7fad8ffff700 1 -- 172.21.15.111:0/4084192062 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7fad9804f6b0 con 0x7fada806d760 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 msgr2=0x7fad94062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 0x7fad94062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0099b0 tx=0x7fad9c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 msgr2=0x7fada80b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fad9800c7c0 tx=0x7fad98007f70 comp rx=0 tx=0).stop 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 shutdown_connections 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad940603a0 0x7fad94062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada815d360 0x7fada8153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fada815d730 0x7fada80c2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 --2- 172.21.15.111:0/4084192062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada806d760 0x7fada80b0ae0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:46.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 >> 172.21.15.111:0/4084192062 conn(0x7fada80ccc20 msgr2=0x7fada815f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:46.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 shutdown_connections 2023-12-19T08:02:46.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:46.837+0000 7fadae66e700 1 -- 172.21.15.111:0/4084192062 wait complete. 2023-12-19T08:02:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T08:02:46.860 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T08:02:47.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.041+0000 7f81b52cd700 1 Processor -- start 2023-12-19T08:02:47.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.041+0000 7f81b52cd700 1 -- start start 2023-12-19T08:02:47.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81b52cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f81b00c7990 0x7f81b00c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81b52cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81b52cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b00cb4b0 con 0x7f81b00cafd0 2023-12-19T08:02:47.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81b52cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b00cb790 con 0x7f81b00c7990 2023-12-19T08:02:47.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81b52cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00aece0 con 0x7f81b00ae860 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81aeffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f81b00c7990 0x7f81b00c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52676/0 (socket says 172.21.15.111:52676) 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 -- 172.21.15.111:0/2303774971 learned_addr learned my addr 172.21.15.111:0/2303774971 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81ae7fc700 1 --1- 172.21.15.111:0/2303774971 >> v1:172.21.15.143:6789/0 conn(0x7f81b00cafd0 0x7f81b00c9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:51880/0 (socket says 172.21.15.111:51880) 2023-12-19T08:02:47.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 -- 172.21.15.111:0/2303774971 >> v2:172.21.15.143:3300/0 conn(0x7f81b00c7990 msgr2=0x7f81b00c7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2303774971 >> v2:172.21.15.143:3300/0 conn(0x7f81b00c7990 0x7f81b00c7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81adffb700 1 -- 172.21.15.111:0/2303774971 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660334780 0 0) 0x7f81b00cb4b0 con 0x7f81b00cafd0 2023-12-19T08:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 -- 172.21.15.111:0/2303774971 >> v1:172.21.15.143:6789/0 conn(0x7f81b00cafd0 legacy=0x7f81b00c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 -- 172.21.15.111:0/2303774971 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b006c8f0 con 0x7f81b00ae860 2023-12-19T08:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2303774971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009f20 tx=0x7f81a8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b676124b29915fe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81adffb700 1 -- 172.21.15.111:0/2303774971 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800e070 con 0x7f81b00ae860 2023-12-19T08:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81adffb700 1 -- 172.21.15.111:0/2303774971 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800b9d0 con 0x7f81b00ae860 2023-12-19T08:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.045+0000 7f81adffb700 1 -- 172.21.15.111:0/2303774971 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80092e0 con 0x7f81b00ae860 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- 172.21.15.111:0/2303774971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 msgr2=0x7f81b01554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 --2- 172.21.15.111:0/2303774971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009f20 tx=0x7f81a8009840 comp rx=0 tx=0).stop 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- 172.21.15.111:0/2303774971 shutdown_connections 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 --2- 172.21.15.111:0/2303774971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00ae860 0x7f81b01554f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 --2- 172.21.15.111:0/2303774971 >> v2:172.21.15.143:3300/0 conn(0x7f81b00c7990 0x7f81b00c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- 172.21.15.111:0/2303774971 >> 172.21.15.111:0/2303774971 conn(0x7f81b0067b10 msgr2=0x7f81b00cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- 172.21.15.111:0/2303774971 shutdown_connections 2023-12-19T08:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- 172.21.15.111:0/2303774971 wait complete. 2023-12-19T08:02:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 Processor -- start 2023-12-19T08:02:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.049+0000 7f81b52cd700 1 -- start start 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 0x7f81b0158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00c7990 0x7f81b0152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0157c30 con 0x7f81b00cafd0 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0157f10 con 0x7f81b00ae860 2023-12-19T08:02:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b01581f0 con 0x7f81b00c7990 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43426/0 (socket says 172.21.15.111:43426) 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 -- 172.21.15.111:0/1393360022 learned_addr learned my addr 172.21.15.111:0/1393360022 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81aeffd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 0x7f81b0158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81ae7fc700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00c7990 0x7f81b0152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 -- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00c7990 msgr2=0x7f81b0152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00c7990 0x7f81b0152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 -- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 msgr2=0x7f81b0158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 0x7f81b0158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 -- 172.21.15.111:0/1393360022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8009ae0 con 0x7f81b00cafd0 2023-12-19T08:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81aeffd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 0x7f81b0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:47.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81af7fe700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f81b00c6f30 tx=0x7f81a8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8008b10 con 0x7f81b00cafd0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a8008c90 con 0x7f81b00cafd0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b01584d0 con 0x7f81b00cafd0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80266b0 con 0x7f81b00cafd0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.053+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b0159210 con 0x7f81b00cafd0 2023-12-19T08:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.057+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b0159430 con 0x7f81b00cafd0 2023-12-19T08:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.057+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f81a8003040 con 0x7f81b00cafd0 2023-12-19T08:02:47.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.057+0000 7f8196ffd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 0x7f819c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:47.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.061+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6838+0+0 (secure 0 0 0) 0x7f81a807edf0 con 0x7f81b00cafd0 2023-12-19T08:02:47.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.061+0000 7f81aeffd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 0x7f819c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:47.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.061+0000 7f81aeffd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 0x7f819c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81a000bfd0 tx=0x7f81a0008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:47.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.065+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f81a804ecb0 con 0x7f81b00cafd0 2023-12-19T08:02:47.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.241+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f81b004be40 con 0x7f81b00cafd0 2023-12-19T08:02:47.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.845+0000 7f8196ffd700 1 -- 172.21.15.111:0/1393360022 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f81a80522d0 con 0x7f81b00cafd0 2023-12-19T08:02:47.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 msgr2=0x7f819c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 0x7f819c062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81a000bfd0 tx=0x7f81a0008040 comp rx=0 tx=0).stop 2023-12-19T08:02:47.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 msgr2=0x7f81b00c5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f81b00c6f30 tx=0x7f81a8018000 comp rx=0 tx=0).stop 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 shutdown_connections 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f819c060670 0x7f819c062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00cafd0 0x7f81b00c5110 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81b00c7990 0x7f81b0152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 --2- 172.21.15.111:0/1393360022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b00ae860 0x7f81b0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 >> 172.21.15.111:0/1393360022 conn(0x7f81b0067b10 msgr2=0x7f81b00c1840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 shutdown_connections 2023-12-19T08:02:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:47.849+0000 7f81b52cd700 1 -- 172.21.15.111:0/1393360022 wait complete. 2023-12-19T08:02:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T08:02:47.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T08:02:48.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 Processor -- start 2023-12-19T08:02:48.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 -- start start 2023-12-19T08:02:48.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a3c0caa10 0x7f5a3c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 0x7f5a3c158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c15d960 con 0x7f5a3c0cc040 2023-12-19T08:02:48.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c0cbbb0 con 0x7f5a3c0caa10 2023-12-19T08:02:48.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.037+0000 7f5a426e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c06dc90 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a41ee2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 0x7f5a3c158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a41ee2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 0x7f5a3c158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43436/0 (socket says 172.21.15.111:43436) 2023-12-19T08:02:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a41ee2700 1 -- 172.21.15.111:0/831956309 learned_addr learned my addr 172.21.15.111:0/831956309 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a40ee0700 1 --2- 172.21.15.111:0/831956309 >> v2:172.21.15.143:3300/0 conn(0x7f5a3c0caa10 0x7f5a3c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 -- 172.21.15.111:0/831956309 >> v2:172.21.15.143:3300/0 conn(0x7f5a3c0caa10 msgr2=0x7f5a3c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 --2- 172.21.15.111:0/831956309 >> v2:172.21.15.143:3300/0 conn(0x7f5a3c0caa10 0x7f5a3c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 -- 172.21.15.111:0/831956309 >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 msgr2=0x7f5a3c158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 --2- 172.21.15.111:0/831956309 >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 0x7f5a3c158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 -- 172.21.15.111:0/831956309 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0cafa0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a416e1700 1 --2- 172.21.15.111:0/831956309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0ce6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c06ea60 tx=0x7f5a380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c01835a5ea8aca2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a33fff700 1 -- 172.21.15.111:0/831956309 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38011630 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a33fff700 1 -- 172.21.15.111:0/831956309 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a380117b0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a33fff700 1 -- 172.21.15.111:0/831956309 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38011b70 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 -- 172.21.15.111:0/831956309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 msgr2=0x7f5a3c0ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 --2- 172.21.15.111:0/831956309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0ce6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c06ea60 tx=0x7f5a380091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 -- 172.21.15.111:0/831956309 shutdown_connections 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 --2- 172.21.15.111:0/831956309 >> v2:172.21.15.111:3300/0 conn(0x7f5a3c0cc040 0x7f5a3c158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 --2- 172.21.15.111:0/831956309 >> v2:172.21.15.143:3300/0 conn(0x7f5a3c0caa10 0x7f5a3c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 --2- 172.21.15.111:0/831956309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0ce6a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 -- 172.21.15.111:0/831956309 >> 172.21.15.111:0/831956309 conn(0x7f5a3c067bb0 msgr2=0x7f5a3c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.041+0000 7f5a426e3700 1 -- 172.21.15.111:0/831956309 shutdown_connections 2023-12-19T08:02:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 -- 172.21.15.111:0/831956309 wait complete. 2023-12-19T08:02:48.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 Processor -- start 2023-12-19T08:02:48.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 -- start start 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a3c0caa10 0x7f5a3c154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0cc040 0x7f5a3c0c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c0b2cf0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c0c8a50 con 0x7f5a3c0caa10 2023-12-19T08:02:48.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.045+0000 7f5a426e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c0c8d30 con 0x7f5a3c0cc040 2023-12-19T08:02:48.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43444/0 (socket says 172.21.15.111:43444) 2023-12-19T08:02:48.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 -- 172.21.15.111:0/2916459753 learned_addr learned my addr 172.21.15.111:0/2916459753 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a40ee0700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0cc040 0x7f5a3c0c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a416e1700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a3c0caa10 0x7f5a3c154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 -- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0cc040 msgr2=0x7f5a3c0c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0cc040 0x7f5a3c0c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 -- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a3c0caa10 msgr2=0x7f5a3c154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a3c0caa10 0x7f5a3c154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 -- 172.21.15.111:0/2916459753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38009420 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a41ee2700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009e50 tx=0x7f5a3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400ca10 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3400f040 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c0b2f40 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400cdc0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c0c98a0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.049+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3c0b3440 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.053+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5a34014650 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.053+0000 7f5a317fa700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 0x7f5a20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.053+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f5a3407d550 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.053+0000 7f5a416e1700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 0x7f5a20062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.057+0000 7f5a416e1700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 0x7f5a20062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c06ea60 tx=0x7f5a38017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.061+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5a3404d5b0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.233+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f5a3c04be40 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.241+0000 7f5a317fa700 1 -- 172.21.15.111:0/2916459753 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v25) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5a34050bd0 con 0x7f5a3c0ce2d0 2023-12-19T08:02:48.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 msgr2=0x7f5a20062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 0x7f5a20062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c06ea60 tx=0x7f5a38017000 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 msgr2=0x7f5a3c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009e50 tx=0x7f5a3400b040 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 shutdown_connections 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5a20060670 0x7f5a20062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a3c0ce2d0 0x7f5a3c0b3740 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a3c0cc040 0x7f5a3c0c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 --2- 172.21.15.111:0/2916459753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a3c0caa10 0x7f5a3c154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 >> 172.21.15.111:0/2916459753 conn(0x7f5a3c067bb0 msgr2=0x7f5a3c0c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 shutdown_connections 2023-12-19T08:02:48.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.245+0000 7f5a426e3700 1 -- 172.21.15.111:0/2916459753 wait complete. 2023-12-19T08:02:48.268 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T08:02:48.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.429+0000 7fdf4eb85700 1 Processor -- start 2023-12-19T08:02:48.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.429+0000 7fdf4eb85700 1 -- start start 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdf480c6d20 0x7fdf480c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf4806c7c0 0x7fdf4806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf4815aa30 0x7fdf4815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4806a7a0 con 0x7fdf4815aa30 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf480c7630 con 0x7fdf480c6d20 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4eb85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4806d4b0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf4815aa30 0x7fdf4815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf4815aa30 0x7fdf4815ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43448/0 (socket says 172.21.15.111:43448) 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 -- 172.21.15.111:0/4174992877 learned_addr learned my addr 172.21.15.111:0/4174992877 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:48.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf47fff700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf4806c7c0 0x7fdf4806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4c921700 1 --2- 172.21.15.111:0/4174992877 >> v2:172.21.15.143:3300/0 conn(0x7fdf480c6d20 0x7fdf480c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 -- 172.21.15.111:0/4174992877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf4806c7c0 msgr2=0x7fdf4806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf4806c7c0 0x7fdf4806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 -- 172.21.15.111:0/4174992877 >> v2:172.21.15.143:3300/0 conn(0x7fdf480c6d20 msgr2=0x7fdf480c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 --2- 172.21.15.111:0/4174992877 >> v2:172.21.15.143:3300/0 conn(0x7fdf480c6d20 0x7fdf480c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 -- 172.21.15.111:0/4174992877 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4806d790 con 0x7fdf4815aa30 2023-12-19T08:02:48.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.433+0000 7fdf4d122700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4815aa30 0x7fdf4815ae50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdf48068030 tx=0x7fdf40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6723411adb79c5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf477fe700 1 -- 172.21.15.111:0/4174992877 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4000e070 con 0x7fdf4815aa30 2023-12-19T08:02:48.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf477fe700 1 -- 172.21.15.111:0/4174992877 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4000ba50 con 0x7fdf4815aa30 2023-12-19T08:02:48.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf477fe700 1 -- 172.21.15.111:0/4174992877 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf400092e0 con 0x7fdf4815aa30 2023-12-19T08:02:48.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4174992877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4815aa30 msgr2=0x7fdf4815ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4815aa30 0x7fdf4815ae50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdf48068030 tx=0x7fdf40009840 comp rx=0 tx=0).stop 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4174992877 shutdown_connections 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4815aa30 0x7fdf4815ae50 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4174992877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf4806c7c0 0x7fdf4806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4174992877 >> v2:172.21.15.143:3300/0 conn(0x7fdf480c6d20 0x7fdf480c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4174992877 >> 172.21.15.111:0/4174992877 conn(0x7fdf48067bb0 msgr2=0x7fdf480cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4174992877 shutdown_connections 2023-12-19T08:02:48.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.437+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4174992877 wait complete. 2023-12-19T08:02:48.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 Processor -- start 2023-12-19T08:02:48.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 -- start start 2023-12-19T08:02:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf480c6d20 0x7fdf48156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4815aa30 0x7fdf480c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf48153c00 con 0x7fdf4806c7c0 2023-12-19T08:02:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480ca9e0 con 0x7fdf4815aa30 2023-12-19T08:02:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4eb85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480cb720 con 0x7fdf480c6d20 2023-12-19T08:02:48.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4c921700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4c921700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43452/0 (socket says 172.21.15.111:43452) 2023-12-19T08:02:48.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.441+0000 7fdf4c921700 1 -- 172.21.15.111:0/4171940431 learned_addr learned my addr 172.21.15.111:0/4171940431 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:48.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4d122700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4815aa30 0x7fdf480c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf47fff700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf480c6d20 0x7fdf48156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 -- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf480c6d20 msgr2=0x7fdf48156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf480c6d20 0x7fdf48156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 -- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4815aa30 msgr2=0x7fdf480c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4815aa30 0x7fdf480c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 -- 172.21.15.111:0/4171940431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf40009ae0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4c921700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d890 tx=0x7fdf3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0125f0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c012750 con 0x7fdf4806c7c0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c012af0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf480c99c0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf480c9f80 con 0x7fdf4806c7c0 2023-12-19T08:02:48.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.445+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf480ca110 con 0x7fdf4806c7c0 2023-12-19T08:02:48.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.449+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdf3c01e460 con 0x7fdf4806c7c0 2023-12-19T08:02:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.449+0000 7fdf44ff9700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 0x7fdf2c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.449+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fdf3c07d210 con 0x7fdf4806c7c0 2023-12-19T08:02:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.449+0000 7fdf47fff700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 0x7fdf2c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.453+0000 7fdf47fff700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 0x7fdf2c062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf48157e50 tx=0x7fdf34008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.457+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdf3c04d380 con 0x7fdf4806c7c0 2023-12-19T08:02:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.629+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdf48071180 con 0x7fdf4806c7c0 2023-12-19T08:02:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.637+0000 7fdf44ff9700 1 -- 172.21.15.111:0/4171940431 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdf3c0509a0 con 0x7fdf4806c7c0 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 msgr2=0x7fdf2c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 0x7fdf2c062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf48157e50 tx=0x7fdf34008040 comp rx=0 tx=0).stop 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 msgr2=0x7fdf480ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d890 tx=0x7fdf3c008040 comp rx=0 tx=0).stop 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 shutdown_connections 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdf2c0606a0 0x7fdf2c062b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4815aa30 0x7fdf480c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf480c6d20 0x7fdf48156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 --2- 172.21.15.111:0/4171940431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf4806c7c0 0x7fdf480ca4a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 >> 172.21.15.111:0/4171940431 conn(0x7fdf48067bb0 msgr2=0x7fdf480cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 shutdown_connections 2023-12-19T08:02:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.641+0000 7fdf4eb85700 1 -- 172.21.15.111:0/4171940431 wait complete. 2023-12-19T08:02:48.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T08:02:48.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.829+0000 7fb519616700 1 Processor -- start 2023-12-19T08:02:48.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.829+0000 7fb519616700 1 -- start start 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb51413f6a0 0x7fb5140ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 0x7fb5140c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5140adc60 con 0x7fb51413f2d0 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51406d7d0 con 0x7fb51413f6a0 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140c8140 con 0x7fb5140c7780 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb513fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb51413f2d0 0x7fb51415a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43624/0 (socket says 172.21.15.111:43624) 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb513fff700 1 -- 172.21.15.111:0/3677301171 learned_addr learned my addr 172.21.15.111:0/3677301171 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:48.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 --2- 172.21.15.111:0/3677301171 >> v2:172.21.15.111:3300/0 conn(0x7fb51413f6a0 0x7fb5140ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb512ffd700 1 -- 172.21.15.111:0/3677301171 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872398526 0 0) 0x7fb5140adc60 con 0x7fb51413f2d0 2023-12-19T08:02:48.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb518e15700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 0x7fb5140c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb512ffd700 1 -- 172.21.15.111:0/3677301171 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc003570 con 0x7fb51413f2d0 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 -- 172.21.15.111:0/3677301171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 msgr2=0x7fb5140c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 0x7fb5140c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 -- 172.21.15.111:0/3677301171 >> v1:172.21.15.111:6789/0 conn(0x7fb51413f2d0 legacy=0x7fb51415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 -- 172.21.15.111:0/3677301171 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140ad050 con 0x7fb51413f6a0 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb518e15700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 0x7fb5140c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb5137fe700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb51413f6a0 0x7fb5140ac9b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb508009910 tx=0x7fb5080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0095a994c13d0b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb512ffd700 1 -- 172.21.15.111:0/3677301171 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800b640 con 0x7fb51413f6a0 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb512ffd700 1 -- 172.21.15.111:0/3677301171 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50800b7a0 con 0x7fb51413f6a0 2023-12-19T08:02:48.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb512ffd700 1 -- 172.21.15.111:0/3677301171 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800bb60 con 0x7fb51413f6a0 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- 172.21.15.111:0/3677301171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb51413f6a0 msgr2=0x7fb5140ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb51413f6a0 0x7fb5140ac9b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb508009910 tx=0x7fb5080091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- 172.21.15.111:0/3677301171 shutdown_connections 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140c7780 0x7fb5140c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 --2- 172.21.15.111:0/3677301171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb51413f6a0 0x7fb5140ac9b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- 172.21.15.111:0/3677301171 >> 172.21.15.111:0/3677301171 conn(0x7fb514067b50 msgr2=0x7fb5140cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:48.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- 172.21.15.111:0/3677301171 shutdown_connections 2023-12-19T08:02:48.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.833+0000 7fb519616700 1 -- 172.21.15.111:0/3677301171 wait complete. 2023-12-19T08:02:48.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.837+0000 7fb519616700 1 Processor -- start 2023-12-19T08:02:48.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.837+0000 7fb519616700 1 -- start start 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb51413f2d0 0x7fb514157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 0x7fb51406a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb514158d30 con 0x7fb5140c7780 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51413aaa0 con 0x7fb51413f2d0 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb514158120 con 0x7fb51413f6a0 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43470/0 (socket says 172.21.15.111:43470) 2023-12-19T08:02:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 -- 172.21.15.111:0/2216080341 learned_addr learned my addr 172.21.15.111:0/2216080341 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 -- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 msgr2=0x7fb51406a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb518e15700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 0x7fb51406a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 0x7fb51406a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 -- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb51413f2d0 msgr2=0x7fb514157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb51413f2d0 0x7fb514157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 -- 172.21.15.111:0/2216080341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508009420 con 0x7fb5140c7780 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb518e15700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 0x7fb51406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb513fff700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c7c0 tx=0x7fb504007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504005b30 con 0x7fb5140c7780 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504005c90 con 0x7fb5140c7780 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb514158400 con 0x7fb5140c7780 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504007340 con 0x7fb5140c7780 2023-12-19T08:02:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5141589c0 con 0x7fb5140c7780 2023-12-19T08:02:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.841+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51404be40 con 0x7fb5140c7780 2023-12-19T08:02:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.849+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb504025430 con 0x7fb5140c7780 2023-12-19T08:02:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.849+0000 7fb4fbfff700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 0x7fb500062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.849+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fb50407d280 con 0x7fb5140c7780 2023-12-19T08:02:48.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.849+0000 7fb5137fe700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 0x7fb500062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:48.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.849+0000 7fb5137fe700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 0x7fb500062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb51413fb00 tx=0x7fb508017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:48.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:48.853+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb50404d4f0 con 0x7fb5140c7780 2023-12-19T08:02:49.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.017+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fb51406ba60 con 0x7fb5140c7780 2023-12-19T08:02:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.025+0000 7fb4fbfff700 1 -- 172.21.15.111:0/2216080341 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v27) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb504050b10 con 0x7fb5140c7780 2023-12-19T08:02:49.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 msgr2=0x7fb500062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 0x7fb500062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb51413fb00 tx=0x7fb508017000 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 msgr2=0x7fb51413a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c7c0 tx=0x7fb504007e00 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 shutdown_connections 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb500060670 0x7fb500062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb51413f2d0 0x7fb514157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb51413f6a0 0x7fb51406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 --2- 172.21.15.111:0/2216080341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140c7780 0x7fb51413a560 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 >> 172.21.15.111:0/2216080341 conn(0x7fb514067b50 msgr2=0x7fb5140c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 shutdown_connections 2023-12-19T08:02:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.029+0000 7fb519616700 1 -- 172.21.15.111:0/2216080341 wait complete. 2023-12-19T08:02:49.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-12-19T08:02:49.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 Processor -- start 2023-12-19T08:02:49.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 -- start start 2023-12-19T08:02:49.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17600ad150 0x7f176006a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17600ca3e0 0x7f1760159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176006c9f0 con 0x7f17600ca010 2023-12-19T08:02:49.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17600ca840 con 0x7f17600ad150 2023-12-19T08:02:49.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.217+0000 7f1765985700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17600c8d20 con 0x7f17600ca3e0 2023-12-19T08:02:49.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f17600ca010 0x7f17600c7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43632/0 (socket says 172.21.15.111:43632) 2023-12-19T08:02:49.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175effd700 1 -- 172.21.15.111:0/1534029541 learned_addr learned my addr 172.21.15.111:0/1534029541 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:49.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17600ad150 0x7f176006a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175ffff700 1 --2- 172.21.15.111:0/1534029541 >> v2:172.21.15.143:3300/0 conn(0x7f17600ca3e0 0x7f1760159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 -- 172.21.15.111:0/1534029541 >> v2:172.21.15.143:3300/0 conn(0x7f17600ca3e0 msgr2=0x7f1760159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 --2- 172.21.15.111:0/1534029541 >> v2:172.21.15.143:3300/0 conn(0x7f17600ca3e0 0x7f1760159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 -- 172.21.15.111:0/1534029541 >> v1:172.21.15.111:6789/0 conn(0x7f17600ca010 legacy=0x7f17600c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 -- 172.21.15.111:0/1534029541 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600c8480 con 0x7f17600ad150 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175dffb700 1 -- 172.21.15.111:0/1534029541 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262576949 0 0) 0x7f176006c9f0 con 0x7f17600ca010 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175e7fc700 1 --2- 172.21.15.111:0/1534029541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ad150 0x7f176006a3f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f176006a930 tx=0x7f17540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90b3c149d009603 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175dffb700 1 -- 172.21.15.111:0/1534029541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754011630 con 0x7f17600ad150 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175dffb700 1 -- 172.21.15.111:0/1534029541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1754011790 con 0x7f17600ad150 2023-12-19T08:02:49.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f175dffb700 1 -- 172.21.15.111:0/1534029541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754011b30 con 0x7f17600ad150 2023-12-19T08:02:49.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 -- 172.21.15.111:0/1534029541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ad150 msgr2=0x7f176006a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 --2- 172.21.15.111:0/1534029541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ad150 0x7f176006a3f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f176006a930 tx=0x7f17540091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 -- 172.21.15.111:0/1534029541 shutdown_connections 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 --2- 172.21.15.111:0/1534029541 >> v2:172.21.15.143:3300/0 conn(0x7f17600ca3e0 0x7f1760159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 --2- 172.21.15.111:0/1534029541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ad150 0x7f176006a3f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 -- 172.21.15.111:0/1534029541 >> 172.21.15.111:0/1534029541 conn(0x7f1760068500 msgr2=0x7f17600688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 -- 172.21.15.111:0/1534029541 shutdown_connections 2023-12-19T08:02:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.221+0000 7f1765985700 1 -- 172.21.15.111:0/1534029541 wait complete. 2023-12-19T08:02:49.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 Processor -- start 2023-12-19T08:02:49.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 -- start start 2023-12-19T08:02:49.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17600ad150 0x7f17600ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 0x7f17600b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176014e5c0 con 0x7f17600ca3e0 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17600b1b30 con 0x7f17600ad150 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f1765985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17600b1e10 con 0x7f17600ca010 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43478/0 (socket says 172.21.15.111:43478) 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 -- 172.21.15.111:0/2303806052 learned_addr learned my addr 172.21.15.111:0/2303806052 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175effd700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17600ad150 0x7f17600ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175e7fc700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 0x7f17600b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 -- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 msgr2=0x7f17600b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 0x7f17600b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 -- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17600ad150 msgr2=0x7f17600ac010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17600ad150 0x7f17600ac010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175ffff700 1 -- 172.21.15.111:0/2303806052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1754009420 con 0x7f17600ca3e0 2023-12-19T08:02:49.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.225+0000 7f175e7fc700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 0x7f17600b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:02:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f175ffff700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1758009df0 tx=0x7f175800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175800cb50 con 0x7f17600ca3e0 2023-12-19T08:02:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175800f040 con 0x7f17600ca3e0 2023-12-19T08:02:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17600b20f0 con 0x7f17600ca3e0 2023-12-19T08:02:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758014480 con 0x7f17600ca3e0 2023-12-19T08:02:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17600b1720 con 0x7f17600ca3e0 2023-12-19T08:02:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.229+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176014e370 con 0x7f17600ca3e0 2023-12-19T08:02:49.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.233+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1758014830 con 0x7f17600ca3e0 2023-12-19T08:02:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.233+0000 7f1746ffd700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 0x7f174c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.233+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f175807d6c0 con 0x7f17600ca3e0 2023-12-19T08:02:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.233+0000 7f175effd700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 0x7f174c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.237+0000 7f175effd700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 0x7f174c062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17500011e0 tx=0x7f1750009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:49.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.241+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f175804d720 con 0x7f17600ca3e0 2023-12-19T08:02:49.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.469+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f1760071120 con 0x7f17600ca3e0 2023-12-19T08:02:49.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.853+0000 7f1746ffd700 1 -- 172.21.15.111:0/2303806052 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v575) v1 ==== 156+0+0 (secure 0 0 0) 0x7f1758050d40 con 0x7f17600ca3e0 2023-12-19T08:02:49.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 msgr2=0x7f174c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 0x7f174c062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17500011e0 tx=0x7f1750009040 comp rx=0 tx=0).stop 2023-12-19T08:02:49.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 msgr2=0x7f17600c36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:49.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1758009df0 tx=0x7f175800b040 comp rx=0 tx=0).stop 2023-12-19T08:02:49.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 shutdown_connections 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f174c0606a0 0x7f174c062b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17600ca3e0 0x7f17600c36c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17600ca010 0x7f17600b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 --2- 172.21.15.111:0/2303806052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17600ad150 0x7f17600ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 >> 172.21.15.111:0/2303806052 conn(0x7f1760068500 msgr2=0x7f17600c6530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 shutdown_connections 2023-12-19T08:02:49.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:49.857+0000 7f1765985700 1 -- 172.21.15.111:0/2303806052 wait complete. 2023-12-19T08:02:49.882 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-12-19T08:02:49.883 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-12-19T08:02:49.883 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T08:02:49.883 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:50.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.045+0000 7f87c8d45700 1 Processor -- start 2023-12-19T08:02:50.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 -- start start 2023-12-19T08:02:50.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87c40c15b0 0x7f87c40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c40c7190 con 0x7f87c406d690 2023-12-19T08:02:50.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c40c1b50 con 0x7f87c40c15b0 2023-12-19T08:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c8d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40c1e30 con 0x7f87c415d320 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87c40c15b0 0x7f87c40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87c40c15b0 0x7f87c40aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43480/0 (socket says 172.21.15.111:43480) 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 -- 172.21.15.111:0/2643005759 learned_addr learned my addr 172.21.15.111:0/2643005759 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c259c700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 -- 172.21.15.111:0/2643005759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 msgr2=0x7f87c415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 -- 172.21.15.111:0/2643005759 >> v1:172.21.15.143:6789/0 conn(0x7f87c406d690 legacy=0x7f87c40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87bbfff700 1 -- 172.21.15.111:0/2643005759 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40c6e20 con 0x7f87c40c15b0 2023-12-19T08:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.049+0000 7f87c259c700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:50.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87bbfff700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c40aeba0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0099b0 tx=0x7f87ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659214d21ce32b22 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c1d9b700 1 -- 172.21.15.111:0/2643005759 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac00b860 con 0x7f87c40c15b0 2023-12-19T08:02:50.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c1d9b700 1 -- 172.21.15.111:0/2643005759 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac00b9c0 con 0x7f87c40c15b0 2023-12-19T08:02:50.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c1d9b700 1 -- 172.21.15.111:0/2643005759 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac00bd60 con 0x7f87c40c15b0 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 -- 172.21.15.111:0/2643005759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 msgr2=0x7f87c40aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c40aeba0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0099b0 tx=0x7f87ac009250 comp rx=0 tx=0).stop 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 -- 172.21.15.111:0/2643005759 shutdown_connections 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c40aeba0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 --2- 172.21.15.111:0/2643005759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 -- 172.21.15.111:0/2643005759 >> 172.21.15.111:0/2643005759 conn(0x7f87c40c8850 msgr2=0x7f87c40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 -- 172.21.15.111:0/2643005759 shutdown_connections 2023-12-19T08:02:50.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 -- 172.21.15.111:0/2643005759 wait complete. 2023-12-19T08:02:50.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.053+0000 7f87c8d45700 1 Processor -- start 2023-12-19T08:02:50.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- start start 2023-12-19T08:02:50.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 0x7f87c416e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c4149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c416ec90 con 0x7f87c40c15b0 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c416ffb0 con 0x7f87c406d690 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c415e100 con 0x7f87c415d320 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c4149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43482/0 (socket says 172.21.15.111:43482) 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 -- 172.21.15.111:0/3872527393 learned_addr learned my addr 172.21.15.111:0/3872527393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c259c700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 0x7f87c416e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 -- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 msgr2=0x7f87c4149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c4149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 -- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 msgr2=0x7f87c416e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 0x7f87c416e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 -- 172.21.15.111:0/3872527393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac0094f0 con 0x7f87c40c15b0 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c259c700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 0x7f87c416e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87bbfff700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009290 tx=0x7f87ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac026480 con 0x7f87c40c15b0 2023-12-19T08:02:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac01f070 con 0x7f87c40c15b0 2023-12-19T08:02:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c415ecb0 con 0x7f87c40c15b0 2023-12-19T08:02:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac026830 con 0x7f87c40c15b0 2023-12-19T08:02:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.057+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c416f680 con 0x7f87c40c15b0 2023-12-19T08:02:50.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.061+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c4071120 con 0x7f87c40c15b0 2023-12-19T08:02:50.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.061+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f87ac026ba0 con 0x7f87c40c15b0 2023-12-19T08:02:50.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.061+0000 7f87ba7fc700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 0x7f87a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.061+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f87ac021070 con 0x7f87c40c15b0 2023-12-19T08:02:50.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.061+0000 7f87c259c700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 0x7f87a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.065+0000 7f87c259c700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 0x7f87a4062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87c40c61d0 tx=0x7f87b4008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.069+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f87ac04ec30 con 0x7f87c40c15b0 2023-12-19T08:02:50.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.241+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87c41700f0 con 0x7f87c40c15b0 2023-12-19T08:02:50.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.241+0000 7f87ba7fc700 1 -- 172.21.15.111:0/3872527393 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+14241 (secure 0 0 0) 0x7f87ac015090 con 0x7f87c40c15b0 2023-12-19T08:02:50.245 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":90,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:47.838779+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6833/82065769":"2023-12-20T08:02:44.813785+0000","172.21.15.143:0/134871789":"2023-12-20T08:02:44.592147+0000","172.21.15.143:0/479928039":"2023-12-20T08:02:44.592147+0000","172.21.15.143:0/2210431380":"2023-12-20T08:02:44.592147+0000","172.21.15.143:0/982685815":"2023-12-20T08:02:44.592147+0000","172.21.15.111:6835/82065769":"2023-12-20T08:02:44.813785+0000","172.21.15.143:0/465723011":"2023-12-20T08:02:44.592147+0000","172.21.15.143:0/2326278745":"2023-12-20T08:02:44.592147+0000","172.21.15.143:0/2190915022":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1956986802":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1938739052":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/3973057301":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/2537901821":"2023-12-20T08:02:38.058339+0000","172.21.15.143:6833/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/3696222545":"2023-12-20T08:02:44.592147+0000","172.21.15.143:6832/33796":"2023-12-20T08:02:38.058339+0000","172.21.15.143:0/1521230848":"2023-12-20T08:02:38.058339+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 msgr2=0x7f87a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 0x7f87a4062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87c40c61d0 tx=0x7f87b4008040 comp rx=0 tx=0).stop 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 msgr2=0x7f87c415e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009290 tx=0x7f87ac017000 comp rx=0 tx=0).stop 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 shutdown_connections 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f87a4060670 0x7f87a4062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c415d320 0x7f87c4149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87c40c15b0 0x7f87c415e770 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 --2- 172.21.15.111:0/3872527393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c406d690 0x7f87c416e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 >> 172.21.15.111:0/3872527393 conn(0x7f87c40c8850 msgr2=0x7f87c40cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 shutdown_connections 2023-12-19T08:02:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.249+0000 7f87c8d45700 1 -- 172.21.15.111:0/3872527393 wait complete. 2023-12-19T08:02:50.270 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:02:50.271 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:50.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.445+0000 7fe57362e700 1 Processor -- start 2023-12-19T08:02:50.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.445+0000 7fe57362e700 1 -- start start 2023-12-19T08:02:50.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe57362e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe56c15d410 0x7fe56c0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe57362e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c0c7450 con 0x7fe56c06b1b0 2023-12-19T08:02:50.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe57362e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56c06bea0 con 0x7fe56c0ca400 2023-12-19T08:02:50.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe57362e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe56c0c6e00 con 0x7fe56c15d410 2023-12-19T08:02:50.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe571bcb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe56c06b1b0 0x7fe56c06b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43648/0 (socket says 172.21.15.111:43648) 2023-12-19T08:02:50.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe571bcb700 1 -- 172.21.15.111:0/2444322867 learned_addr learned my addr 172.21.15.111:0/2444322867 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2444322867 >> v2:172.21.15.111:3300/0 conn(0x7fe56c15d410 0x7fe56c0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914744466 0 0) 0x7fe56c0c7450 con 0x7fe56c06b1b0 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe558003570 con 0x7fe56c06b1b0 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982482841 0 0) 0x7fe56c06bea0 con 0x7fe56c0ca400 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe5713ca700 1 -- 172.21.15.111:0/2444322867 >> v1:172.21.15.143:6789/0 conn(0x7fe56c0ca400 legacy=0x7fe56c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe5713ca700 1 -- 172.21.15.111:0/2444322867 >> v1:172.21.15.111:6789/0 conn(0x7fe56c06b1b0 legacy=0x7fe56c06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe5713ca700 1 -- 172.21.15.111:0/2444322867 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0c70e0 con 0x7fe56c15d410 2023-12-19T08:02:50.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2444322867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c15d410 0x7fe56c0ab6c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe5680098e0 tx=0x7fe5680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2e2cd77c4b9a53 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800b640 con 0x7fe56c15d410 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56800b7a0 con 0x7fe56c15d410 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.449+0000 7fe563fff700 1 -- 172.21.15.111:0/2444322867 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56800bb40 con 0x7fe56c15d410 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 -- 172.21.15.111:0/2444322867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c15d410 msgr2=0x7fe56c0ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 --2- 172.21.15.111:0/2444322867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c15d410 0x7fe56c0ab6c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe5680098e0 tx=0x7fe5680091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 -- 172.21.15.111:0/2444322867 shutdown_connections 2023-12-19T08:02:50.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 --2- 172.21.15.111:0/2444322867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c15d410 0x7fe56c0ab6c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 -- 172.21.15.111:0/2444322867 >> 172.21.15.111:0/2444322867 conn(0x7fe56c067bb0 msgr2=0x7fe56c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:50.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 -- 172.21.15.111:0/2444322867 shutdown_connections 2023-12-19T08:02:50.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.453+0000 7fe57362e700 1 -- 172.21.15.111:0/2444322867 wait complete. 2023-12-19T08:02:50.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 Processor -- start 2023-12-19T08:02:50.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 -- start start 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 0x7fe56c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe56c15d410 0x7fe56c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0b1aa0 con 0x7fe56c0ca400 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0ac5a0 con 0x7fe56c06b1b0 2023-12-19T08:02:50.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe57362e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c0b1290 con 0x7fe56c15d410 2023-12-19T08:02:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe570bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe570bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43506/0 (socket says 172.21.15.111:43506) 2023-12-19T08:02:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe570bc9700 1 -- 172.21.15.111:0/2483701864 learned_addr learned my addr 172.21.15.111:0/2483701864 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe571bcb700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe56c15d410 0x7fe56c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.457+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 0x7fe56c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 -- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe56c15d410 msgr2=0x7fe56c156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe56c15d410 0x7fe56c156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 -- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 msgr2=0x7fe56c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 0x7fe56c0b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 -- 172.21.15.111:0/2483701864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568009420 con 0x7fe56c0ca400 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 0x7fe56c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe570bc9700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c560 tx=0x7fe55c00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c011590 con 0x7fe56c0ca400 2023-12-19T08:02:50.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c011710 con 0x7fe56c0ca400 2023-12-19T08:02:50.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c011ad0 con 0x7fe56c0ca400 2023-12-19T08:02:50.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c162ed0 con 0x7fe56c0ca400 2023-12-19T08:02:50.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c0b17c0 con 0x7fe56c0ca400 2023-12-19T08:02:50.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.461+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56c0ac710 con 0x7fe56c0ca400 2023-12-19T08:02:50.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.465+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe55c01b3b0 con 0x7fe56c0ca400 2023-12-19T08:02:50.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.465+0000 7fe5617fa700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 0x7fe550062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.465+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fe55c07c1d0 con 0x7fe56c0ca400 2023-12-19T08:02:50.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.465+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 0x7fe550062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.465+0000 7fe5713ca700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 0x7fe550062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5680093f0 tx=0x7fe568017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.469+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe55c04c3e0 con 0x7fe56c0ca400 2023-12-19T08:02:50.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.641+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe56c0ae770 con 0x7fe56c0ca400 2023-12-19T08:02:50.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.641+0000 7fe5617fa700 1 -- 172.21.15.111:0/2483701864 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe55c0170a0 con 0x7fe56c0ca400 2023-12-19T08:02:50.644 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 msgr2=0x7fe550062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 0x7fe550062b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5680093f0 tx=0x7fe568017000 comp rx=0 tx=0).stop 2023-12-19T08:02:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 msgr2=0x7fe56c162990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c560 tx=0x7fe55c00c870 comp rx=0 tx=0).stop 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 shutdown_connections 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5500606a0 0x7fe550062b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe56c15d410 0x7fe56c156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe56c0ca400 0x7fe56c162990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 --2- 172.21.15.111:0/2483701864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe56c06b1b0 0x7fe56c0b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 >> 172.21.15.111:0/2483701864 conn(0x7fe56c067bb0 msgr2=0x7fe56c0c1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 shutdown_connections 2023-12-19T08:02:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.645+0000 7fe57362e700 1 -- 172.21.15.111:0/2483701864 wait complete. 2023-12-19T08:02:50.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-12-19T08:02:50.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 Processor -- start 2023-12-19T08:02:50.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 -- start start 2023-12-19T08:02:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc044159530 0x7fc0440c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440accb0 con 0x7fc044159160 2023-12-19T08:02:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440acf90 con 0x7fc044158200 2023-12-19T08:02:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.837+0000 7fc04a3cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc044158d70 con 0x7fc044159530 2023-12-19T08:02:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc048bc9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc044159530 0x7fc0440c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc0493ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc044159160 0x7fc0440adcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43658/0 (socket says 172.21.15.111:43658) 2023-12-19T08:02:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc0493ca700 1 -- 172.21.15.111:0/916102016 learned_addr learned my addr 172.21.15.111:0/916102016 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:50.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158317463 0 0) 0x7fc0440accb0 con 0x7fc044159160 2023-12-19T08:02:50.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030003570 con 0x7fc044159160 2023-12-19T08:02:50.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538937532 0 0) 0x7fc0440acf90 con 0x7fc044158200 2023-12-19T08:02:50.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0440accb0 con 0x7fc044158200 2023-12-19T08:02:50.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc048bc9700 1 -- 172.21.15.111:0/916102016 >> v1:172.21.15.143:6789/0 conn(0x7fc044158200 legacy=0x7fc044158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc048bc9700 1 -- 172.21.15.111:0/916102016 >> v1:172.21.15.111:6789/0 conn(0x7fc044159160 legacy=0x7fc0440adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc048bc9700 1 -- 172.21.15.111:0/916102016 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440ad270 con 0x7fc044159530 2023-12-19T08:02:50.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc048bc9700 1 --2- 172.21.15.111:0/916102016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044159530 0x7fc0440c5fd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc0340098e0 tx=0x7fc0340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45634172592f8eb9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034011630 con 0x7fc044159530 2023-12-19T08:02:50.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc034011790 con 0x7fc044159530 2023-12-19T08:02:50.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.841+0000 7fc03bfff700 1 -- 172.21.15.111:0/916102016 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034011b30 con 0x7fc044159530 2023-12-19T08:02:50.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- 172.21.15.111:0/916102016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044159530 msgr2=0x7fc0440c5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/916102016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044159530 0x7fc0440c5fd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc0340098e0 tx=0x7fc0340091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- 172.21.15.111:0/916102016 shutdown_connections 2023-12-19T08:02:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/916102016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044159530 0x7fc0440c5fd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- 172.21.15.111:0/916102016 >> 172.21.15.111:0/916102016 conn(0x7fc044067bb0 msgr2=0x7fc0440cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- 172.21.15.111:0/916102016 shutdown_connections 2023-12-19T08:02:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- 172.21.15.111:0/916102016 wait complete. 2023-12-19T08:02:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 Processor -- start 2023-12-19T08:02:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.845+0000 7fc04a3cc700 1 -- start start 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044158200 0x7fc0440af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 0x7fc04414d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc044169500 con 0x7fc044159530 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04414f3a0 con 0x7fc044158200 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc04a3cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440c6000 con 0x7fc044159160 2023-12-19T08:02:50.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc0493ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044158200 0x7fc0440af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc048bc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 0x7fc04414d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc048bc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52802/0 (socket says 172.21.15.111:52802) 2023-12-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc048bc9700 1 -- 172.21.15.111:0/2706968189 learned_addr learned my addr 172.21.15.111:0/2706968189 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:50.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 -- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 msgr2=0x7fc044168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 -- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044158200 msgr2=0x7fc0440af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044158200 0x7fc0440af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 -- 172.21.15.111:0/2706968189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034009420 con 0x7fc044159530 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc048bc9700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:50.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.849+0000 7fc049bcb700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 0x7fc04414d520 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00c560 tx=0x7fc03c007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c011590 con 0x7fc044159530 2023-12-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c011710 con 0x7fc044159530 2023-12-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c011ad0 con 0x7fc044159530 2023-12-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04414ea70 con 0x7fc044159530 2023-12-19T08:02:50.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04414f030 con 0x7fc044159530 2023-12-19T08:02:50.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.853+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04414f530 con 0x7fc044159530 2023-12-19T08:02:50.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.857+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc03c012420 con 0x7fc044159530 2023-12-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.857+0000 7fc0397fa700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 0x7fc028062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.857+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fc03c07c580 con 0x7fc044159530 2023-12-19T08:02:50.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.857+0000 7fc0493ca700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 0x7fc028062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:50.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.861+0000 7fc0493ca700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 0x7fc028062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0440b19c0 tx=0x7fc040009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:50.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:50.869+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc03c04c790 con 0x7fc044159530 2023-12-19T08:02:51.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.093+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fc04404be40 con 0x7fc044159530 2023-12-19T08:02:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.865+0000 7fc0397fa700 1 -- 172.21.15.111:0/2706968189 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v577) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc03c04fdb0 con 0x7fc044159530 2023-12-19T08:02:51.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 msgr2=0x7fc028062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 0x7fc028062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0440b19c0 tx=0x7fc040009040 comp rx=0 tx=0).stop 2023-12-19T08:02:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 msgr2=0x7fc04414d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 0x7fc04414d520 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00c560 tx=0x7fc03c007cf0 comp rx=0 tx=0).stop 2023-12-19T08:02:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 shutdown_connections 2023-12-19T08:02:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc028060670 0x7fc028062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc044159530 0x7fc04414d520 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc044159160 0x7fc044168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 --2- 172.21.15.111:0/2706968189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc044158200 0x7fc0440af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:51.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 >> 172.21.15.111:0/2706968189 conn(0x7fc044067bb0 msgr2=0x7fc04414df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:51.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 shutdown_connections 2023-12-19T08:02:51.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:51.869+0000 7fc04a3cc700 1 -- 172.21.15.111:0/2706968189 wait complete. 2023-12-19T08:02:51.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:02:52.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 Processor -- start 2023-12-19T08:02:52.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 -- start start 2023-12-19T08:02:52.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 0x7fe4f00b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 0x7fe4f00ab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00ae2b0 con 0x7fe4f0069a90 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f01562b0 con 0x7fe4f00ca370 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4f586e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00b0ec0 con 0x7fe4f00b1a50 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4ee7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4f0069a90 0x7fe4f00b0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34108/0 (socket says 172.21.15.111:34108) 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.069+0000 7fe4ee7fc700 1 -- 172.21.15.111:0/3334778267 learned_addr learned my addr 172.21.15.111:0/3334778267 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/3334778267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 0x7fe4f00b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4ef7fe700 1 --2- 172.21.15.111:0/3334778267 >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 0x7fe4f00ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4edffb700 1 -- 172.21.15.111:0/3334778267 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151412724 0 0) 0x7fe4f00ae2b0 con 0x7fe4f0069a90 2023-12-19T08:02:52.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4edffb700 1 -- 172.21.15.111:0/3334778267 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8003570 con 0x7fe4f0069a90 2023-12-19T08:02:52.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 -- 172.21.15.111:0/3334778267 >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 msgr2=0x7fe4f00ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:52.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/3334778267 >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 0x7fe4f00ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 -- 172.21.15.111:0/3334778267 >> v1:172.21.15.111:6789/0 conn(0x7fe4f0069a90 legacy=0x7fe4f00b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 -- 172.21.15.111:0/3334778267 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00ad980 con 0x7fe4f00b1a50 2023-12-19T08:02:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4ef7fe700 1 --2- 172.21.15.111:0/3334778267 >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 0x7fe4f00ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/3334778267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 0x7fe4f00b1e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009910 tx=0x7fe4e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84130fd2ee9d9d5d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4edffb700 1 -- 172.21.15.111:0/3334778267 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011630 con 0x7fe4f00b1a50 2023-12-19T08:02:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4edffb700 1 -- 172.21.15.111:0/3334778267 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e0011790 con 0x7fe4f00b1a50 2023-12-19T08:02:52.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4edffb700 1 -- 172.21.15.111:0/3334778267 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011b30 con 0x7fe4f00b1a50 2023-12-19T08:02:52.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 -- 172.21.15.111:0/3334778267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 msgr2=0x7fe4f00b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 --2- 172.21.15.111:0/3334778267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 0x7fe4f00b1e20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009910 tx=0x7fe4e00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 -- 172.21.15.111:0/3334778267 shutdown_connections 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 --2- 172.21.15.111:0/3334778267 >> v2:172.21.15.143:3300/0 conn(0x7fe4f00ca370 0x7fe4f00ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 --2- 172.21.15.111:0/3334778267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f00b1a50 0x7fe4f00b1e20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 -- 172.21.15.111:0/3334778267 >> 172.21.15.111:0/3334778267 conn(0x7fe4f00cd870 msgr2=0x7fe4f0069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 -- 172.21.15.111:0/3334778267 shutdown_connections 2023-12-19T08:02:52.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.073+0000 7fe4f586e700 1 -- 172.21.15.111:0/3334778267 wait complete. 2023-12-19T08:02:52.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 Processor -- start 2023-12-19T08:02:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 -- start start 2023-12-19T08:02:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 0x7fe4f00c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 0x7fe4f014d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f016cb30 con 0x7fe4f00ca370 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c8810 con 0x7fe4f00b1a50 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4f586e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c8af0 con 0x7fe4f0069a90 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52988/0 (socket says 172.21.15.111:52988) 2023-12-19T08:02:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4ef7fe700 1 -- 172.21.15.111:0/1785185181 learned_addr learned my addr 172.21.15.111:0/1785185181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 0x7fe4f00c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4ee7fc700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 0x7fe4f014d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.077+0000 7fe4ef7fe700 1 -- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 msgr2=0x7fe4f00c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ef7fe700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 0x7fe4f00c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ef7fe700 1 -- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 msgr2=0x7fe4f014d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ef7fe700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 0x7fe4f014d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ef7fe700 1 -- 172.21.15.111:0/1785185181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8009a20 con 0x7fe4f00ca370 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 0x7fe4f00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ee7fc700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 0x7fe4f014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4ef7fe700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009fc0 tx=0x7fe4e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e800cd30 con 0x7fe4f00ca370 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e800f040 con 0x7fe4f00ca370 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e80146d0 con 0x7fe4f00ca370 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e0009420 con 0x7fe4f00ca370 2023-12-19T08:02:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f014d7d0 con 0x7fe4f00ca370 2023-12-19T08:02:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.081+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f016cca0 con 0x7fe4f00ca370 2023-12-19T08:02:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.085+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe4e8003020 con 0x7fe4f00ca370 2023-12-19T08:02:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.085+0000 7fe4d6ffd700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 0x7fe4dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.085+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fe4e807d7f0 con 0x7fe4f00ca370 2023-12-19T08:02:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.085+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 0x7fe4dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.089+0000 7fe4eeffd700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 0x7fe4dc062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00093f0 tx=0x7fe4e0017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.093+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe4e804d850 con 0x7fe4f00ca370 2023-12-19T08:02:52.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.269+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe4f0071120 con 0x7fe4f00ca370 2023-12-19T08:02:52.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.277+0000 7fe4d6ffd700 1 -- 172.21.15.111:0/1785185181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe4e8050e70 con 0x7fe4f00ca370 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 msgr2=0x7fe4dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 0x7fe4dc062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00093f0 tx=0x7fe4e0017000 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 msgr2=0x7fe4f00abcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009fc0 tx=0x7fe4e800b040 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 shutdown_connections 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe4dc060670 0x7fe4dc062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f00b1a50 0x7fe4f014d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00ca370 0x7fe4f00abcb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 --2- 172.21.15.111:0/1785185181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0069a90 0x7fe4f00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 >> 172.21.15.111:0/1785185181 conn(0x7fe4f00cd870 msgr2=0x7fe4f00682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:52.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 shutdown_connections 2023-12-19T08:02:52.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.281+0000 7fe4f586e700 1 -- 172.21.15.111:0/1785185181 wait complete. 2023-12-19T08:02:52.306 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:02:52.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:52.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:02:52.393 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:02:52.393 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:02:52.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:02:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 Processor -- start 2023-12-19T08:02:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 -- start start 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1178158d60 0x7f117815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 0x7f11780c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f117806a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117806c7c0 con 0x7f117806b7a0 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117806caa0 con 0x7f1178158d60 2023-12-19T08:02:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.565+0000 7f117759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11780c55f0 con 0x7f11780c51b0 2023-12-19T08:02:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f116dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 0x7f11780c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1178158d60 0x7f117815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f116dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 0x7f11780c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52994/0 (socket says 172.21.15.111:52994) 2023-12-19T08:02:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f116dd9b700 1 -- 172.21.15.111:0/3316254369 learned_addr learned my addr 172.21.15.111:0/3316254369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f1176d9d700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f117806a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 -- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 msgr2=0x7f117806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f117806a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 -- 172.21.15.111:0/3316254369 >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 msgr2=0x7f11780c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 --2- 172.21.15.111:0/3316254369 >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 0x7f11780c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 -- 172.21.15.111:0/3316254369 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117815ce60 con 0x7f1178158d60 2023-12-19T08:02:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117659c700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1178158d60 0x7f117815c920 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f117806c540 tx=0x7f11680091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd40fc5f960991c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f1175d9b700 1 -- 172.21.15.111:0/3316254369 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168011630 con 0x7f1178158d60 2023-12-19T08:02:52.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f1175d9b700 1 -- 172.21.15.111:0/3316254369 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11680117b0 con 0x7f1178158d60 2023-12-19T08:02:52.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f1175d9b700 1 -- 172.21.15.111:0/3316254369 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168011b70 con 0x7f1178158d60 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117759e700 1 -- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1178158d60 msgr2=0x7f117815c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117759e700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1178158d60 0x7f117815c920 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f117806c540 tx=0x7f11680091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.569+0000 7f117759e700 1 -- 172.21.15.111:0/3316254369 shutdown_connections 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f117806a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 --2- 172.21.15.111:0/3316254369 >> v2:172.21.15.111:3300/0 conn(0x7f117806b7a0 0x7f11780c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 --2- 172.21.15.111:0/3316254369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1178158d60 0x7f117815c920 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 -- 172.21.15.111:0/3316254369 >> 172.21.15.111:0/3316254369 conn(0x7f1178067bb0 msgr2=0x7f11780cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:52.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 -- 172.21.15.111:0/3316254369 shutdown_connections 2023-12-19T08:02:52.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 -- 172.21.15.111:0/3316254369 wait complete. 2023-12-19T08:02:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 Processor -- start 2023-12-19T08:02:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.573+0000 7f117759e700 1 -- start start 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 0x7f117815aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f1178151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178155510 con 0x7f1178158d60 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11781521a0 con 0x7f117806b7a0 2023-12-19T08:02:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1178155070 con 0x7f11780c51b0 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53008/0 (socket says 172.21.15.111:53008) 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 -- 172.21.15.111:0/1888167386 learned_addr learned my addr 172.21.15.111:0/1888167386 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f116dd9b700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f1178151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117659c700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 0x7f117815aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 -- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 msgr2=0x7f1178151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f1178151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 -- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 msgr2=0x7f117815aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 0x7f117815aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 -- 172.21.15.111:0/1888167386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168009420 con 0x7f1178158d60 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f116dd9b700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f1178151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117659c700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 0x7f117815aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f1176d9d700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e90 tx=0x7f117000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000cc20 con 0x7f1178158d60 2023-12-19T08:02:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000f040 con 0x7f1178158d60 2023-12-19T08:02:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170014480 con 0x7f1178158d60 2023-12-19T08:02:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117815b0f0 con 0x7f1178158d60 2023-12-19T08:02:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.577+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117815b6b0 con 0x7f1178158d60 2023-12-19T08:02:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.581+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11781556c0 con 0x7f1178158d60 2023-12-19T08:02:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.581+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1170014830 con 0x7f1178158d60 2023-12-19T08:02:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.581+0000 7f116effd700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 0x7f115c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.581+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f117007dae0 con 0x7f1178158d60 2023-12-19T08:02:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.581+0000 7f117659c700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 0x7f115c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:52.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.585+0000 7f117659c700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 0x7f115c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f117806c540 tx=0x7f1168017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.589+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f11781556c0 con 0x7f1178158d60 2023-12-19T08:02:52.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.793+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11781552d0 con 0x7f1178158d60 2023-12-19T08:02:52.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.797+0000 7f116effd700 1 -- 172.21.15.111:0/1888167386 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f1170018090 con 0x7f1178158d60 2023-12-19T08:02:52.798 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:52.798 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5837,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1890020889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1890020889},{"type":"v1","addr":"172.21.15.143:6836","nonce":1890020889}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":5840,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1560739805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1560739805},{"type":"v1","addr":"172.21.15.143:6837","nonce":1560739805}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6426,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3940329899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3940329899},{"type":"v1","addr":"172.21.15.111:6834","nonce":3940329899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6786,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/496458405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":496458405},{"type":"v1","addr":"172.21.15.111:6835","nonce":496458405}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[]} 2023-12-19T08:02:52.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 msgr2=0x7f115c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 0x7f115c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f117806c540 tx=0x7f1168017000 comp rx=0 tx=0).stop 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 msgr2=0x7f1178154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e90 tx=0x7f117000b040 comp rx=0 tx=0).stop 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 shutdown_connections 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f115c060670 0x7f115c062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1178158d60 0x7f1178154b30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11780c51b0 0x7f1178151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 --2- 172.21.15.111:0/1888167386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117806b7a0 0x7f117815aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:52.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 >> 172.21.15.111:0/1888167386 conn(0x7f1178067bb0 msgr2=0x7f11780c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:52.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 shutdown_connections 2023-12-19T08:02:52.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:52.801+0000 7f117759e700 1 -- 172.21.15.111:0/1888167386 wait complete. 2023-12-19T08:02:52.806 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 34 2023-12-19T08:02:52.825 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:02:52.825 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:02:52.826 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:02:52.826 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:52.826 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:02:52.826 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:02:52.826 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:02:52.826 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:52.826 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:02:52.827 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:02:52.827 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:02:52.827 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:52.827 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:02:52.827 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:02:52.827 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:02:52.827 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:02:52.848 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:02:52.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:02:52.850 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:02:52.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:02:52.850 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:02:52.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:02:52.852 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:02:52.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:02:52.899 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:02:52.899 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:52.900 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:02:52.901 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:52.902 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:02:52.902 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:52.903 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:02:52.904 DEBUG:teuthology.parallel:result is None 2023-12-19T08:02:52.904 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:02:53.007 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:02:53.009 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:02:53.010 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:02:53.021 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:02:53.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.061+0000 7f8c26f23700 1 Processor -- start 2023-12-19T08:02:53.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.061+0000 7f8c26f23700 1 -- start start 2023-12-19T08:02:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 0x7f8c2006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26f23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8c200ce810 0x7f8c200ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26f23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2006b730 con 0x7f8c20158e20 2023-12-19T08:02:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26f23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2006ab20 con 0x7f8c200ce810 2023-12-19T08:02:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2006ae00 con 0x7f8c201591f0 2023-12-19T08:02:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c25f21700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8c20158e20 0x7f8c200ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34122/0 (socket says 172.21.15.111:34122) 2023-12-19T08:02:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c25f21700 1 -- 172.21.15.111:0/3836746109 learned_addr learned my addr 172.21.15.111:0/3836746109 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 --2- 172.21.15.111:0/3836746109 >> v2:172.21.15.111:3300/0 conn(0x7f8c200ce810 0x7f8c200ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c25720700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 0x7f8c2006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c24f1f700 1 -- 172.21.15.111:0/3836746109 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397475232 0 0) 0x7f8c2006b730 con 0x7f8c20158e20 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c24f1f700 1 -- 172.21.15.111:0/3836746109 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c003570 con 0x7f8c20158e20 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 -- 172.21.15.111:0/3836746109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 msgr2=0x7f8c2006a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 0x7f8c2006a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 -- 172.21.15.111:0/3836746109 >> v1:172.21.15.111:6789/0 conn(0x7f8c20158e20 legacy=0x7f8c200ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 -- 172.21.15.111:0/3836746109 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2006b0e0 con 0x7f8c200ce810 2023-12-19T08:02:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c25720700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 0x7f8c2006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:02:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c26722700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c200ce810 0x7f8c200ca950 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009e60 tx=0x7f8c18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c2bdb0255e9e25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c24f1f700 1 -- 172.21.15.111:0/3836746109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800e070 con 0x7f8c200ce810 2023-12-19T08:02:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c24f1f700 1 -- 172.21.15.111:0/3836746109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1800bab0 con 0x7f8c200ce810 2023-12-19T08:02:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.065+0000 7f8c24f1f700 1 -- 172.21.15.111:0/3836746109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800be70 con 0x7f8c200ce810 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 -- 172.21.15.111:0/3836746109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c200ce810 msgr2=0x7f8c200ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c200ce810 0x7f8c200ca950 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009e60 tx=0x7f8c18009780 comp rx=0 tx=0).stop 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 -- 172.21.15.111:0/3836746109 shutdown_connections 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c200ce810 0x7f8c200ca950 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 --2- 172.21.15.111:0/3836746109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c201591f0 0x7f8c2006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 -- 172.21.15.111:0/3836746109 >> 172.21.15.111:0/3836746109 conn(0x7f8c20067bb0 msgr2=0x7f8c200cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 -- 172.21.15.111:0/3836746109 shutdown_connections 2023-12-19T08:02:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.069+0000 7f8c26f23700 1 -- 172.21.15.111:0/3836746109 wait complete. 2023-12-19T08:02:53.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 Processor -- start 2023-12-19T08:02:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 -- start start 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c200ce810 0x7f8c20155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c20158e20 0x7f8c2014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200ca120 con 0x7f8c201591f0 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c201517b0 con 0x7f8c20158e20 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2014bee0 con 0x7f8c200ce810 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53048/0 (socket says 172.21.15.111:53048) 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 -- 172.21.15.111:0/1202218327 learned_addr learned my addr 172.21.15.111:0/1202218327 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c25f21700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c200ce810 0x7f8c20155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c25720700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c20158e20 0x7f8c2014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 -- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c200ce810 msgr2=0x7f8c20155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c200ce810 0x7f8c20155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 -- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c20158e20 msgr2=0x7f8c2014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c20158e20 0x7f8c2014b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.073+0000 7f8c26722700 1 -- 172.21.15.111:0/1202218327 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10009420 con 0x7f8c201591f0 2023-12-19T08:02:53.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c26722700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8c18013000 tx=0x7f8c18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:53.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18012e10 con 0x7f8c201591f0 2023-12-19T08:02:53.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18027470 con 0x7f8c201591f0 2023-12-19T08:02:53.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c18009a20 con 0x7f8c201591f0 2023-12-19T08:02:53.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18027830 con 0x7f8c201591f0 2023-12-19T08:02:53.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c20150f00 con 0x7f8c201591f0 2023-12-19T08:02:53.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.077+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c20151940 con 0x7f8c201591f0 2023-12-19T08:02:53.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.081+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8c18024070 con 0x7f8c201591f0 2023-12-19T08:02:53.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.081+0000 7f8c15ffb700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 0x7f8c04062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:53.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.081+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f8c1807f690 con 0x7f8c201591f0 2023-12-19T08:02:53.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.081+0000 7f8c25f21700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 0x7f8c04062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:53.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.081+0000 7f8c25f21700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 0x7f8c04062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c007fd0 tx=0x7f8c1c009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:53.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.089+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c1804f800 con 0x7f8c201591f0 2023-12-19T08:02:53.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:53.269+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8c2006b3c0 con 0x7f8c201591f0 2023-12-19T08:02:54.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.305+0000 7f8c15ffb700 1 -- 172.21.15.111:0/1202218327 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8c18052e20 con 0x7f8c201591f0 2023-12-19T08:02:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 msgr2=0x7f8c04062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 0x7f8c04062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c007fd0 tx=0x7f8c1c009040 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 msgr2=0x7f8c200c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8c18013000 tx=0x7f8c18019000 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 shutdown_connections 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8c04060670 0x7f8c04062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c20158e20 0x7f8c2014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c201591f0 0x7f8c200c75e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 --2- 172.21.15.111:0/1202218327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c200ce810 0x7f8c20155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 >> 172.21.15.111:0/1202218327 conn(0x7f8c20067bb0 msgr2=0x7f8c2006c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 shutdown_connections 2023-12-19T08:02:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.309+0000 7f8c26f23700 1 -- 172.21.15.111:0/1202218327 wait complete. 2023-12-19T08:02:54.314 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:02:54.336 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:02:54.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.501+0000 7f385f945700 1 Processor -- start 2023-12-19T08:02:54.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 -- start start 2023-12-19T08:02:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3858067c90 0x7f3858068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f385814fba0 0x7f385814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38580c6630 con 0x7f38580c5900 2023-12-19T08:02:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38580c6440 con 0x7f385814fba0 2023-12-19T08:02:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385f945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38580c6910 con 0x7f3858067c90 2023-12-19T08:02:54.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385cee0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f385814fba0 0x7f385814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385cee0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f385814fba0 0x7f385814ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:55866/0 (socket says 172.21.15.111:55866) 2023-12-19T08:02:54.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385cee0700 1 -- 172.21.15.111:0/880799761 learned_addr learned my addr 172.21.15.111:0/880799761 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:54.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385d6e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3858067c90 0x7f3858068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.505+0000 7f385dee2700 1 --1- 172.21.15.111:0/880799761 >> v1:172.21.15.143:6789/0 conn(0x7f38580c5900 0x7f38580c5d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:39024/0 (socket says 172.21.15.111:39024) 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385cee0700 1 -- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3858067c90 msgr2=0x7f3858068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385cee0700 1 --2- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3858067c90 0x7f3858068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385cee0700 1 -- 172.21.15.111:0/880799761 >> v1:172.21.15.143:6789/0 conn(0x7f38580c5900 legacy=0x7f38580c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385cee0700 1 -- 172.21.15.111:0/880799761 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580c6bf0 con 0x7f385814fba0 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f384ffff700 1 -- 172.21.15.111:0/880799761 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633100980 0 0) 0x7f38580c6630 con 0x7f38580c5900 2023-12-19T08:02:54.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385cee0700 1 --2- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f385814fba0 0x7f385814ffd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f38480099b0 tx=0x7f3848009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a128e01d008251e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f384ffff700 1 -- 172.21.15.111:0/880799761 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848009ea0 con 0x7f385814fba0 2023-12-19T08:02:54.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f384ffff700 1 -- 172.21.15.111:0/880799761 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384800b860 con 0x7f385814fba0 2023-12-19T08:02:54.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f384ffff700 1 -- 172.21.15.111:0/880799761 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800bc00 con 0x7f385814fba0 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 -- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f385814fba0 msgr2=0x7f385814ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 --2- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f385814fba0 0x7f385814ffd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f38480099b0 tx=0x7f3848009250 comp rx=0 tx=0).stop 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 -- 172.21.15.111:0/880799761 shutdown_connections 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 --2- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f385814fba0 0x7f385814ffd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 --2- 172.21.15.111:0/880799761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3858067c90 0x7f3858068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 -- 172.21.15.111:0/880799761 >> 172.21.15.111:0/880799761 conn(0x7f38580cd6d0 msgr2=0x7f385806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 -- 172.21.15.111:0/880799761 shutdown_connections 2023-12-19T08:02:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.509+0000 7f385f945700 1 -- 172.21.15.111:0/880799761 wait complete. 2023-12-19T08:02:54.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 Processor -- start 2023-12-19T08:02:54.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 -- start start 2023-12-19T08:02:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3858067c90 0x7f3858152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38580c5900 0x7f3858150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385814f3c0 con 0x7f385814fba0 2023-12-19T08:02:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38580c8530 con 0x7f3858067c90 2023-12-19T08:02:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385f945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38581698a0 con 0x7f38580c5900 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53058/0 (socket says 172.21.15.111:53058) 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 -- 172.21.15.111:0/3577882412 learned_addr learned my addr 172.21.15.111:0/3577882412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385cee0700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38580c5900 0x7f3858150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385d6e1700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3858067c90 0x7f3858152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 -- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38580c5900 msgr2=0x7f3858150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38580c5900 0x7f3858150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 -- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3858067c90 msgr2=0x7f3858152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3858067c90 0x7f3858152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.513+0000 7f385dee2700 1 -- 172.21.15.111:0/3577882412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480094f0 con 0x7f385814fba0 2023-12-19T08:02:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f385dee2700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f385000cd80 tx=0x7f3850001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850005df0 con 0x7f385814fba0 2023-12-19T08:02:54.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385000a870 con 0x7f385814fba0 2023-12-19T08:02:54.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3858169be0 con 0x7f385814fba0 2023-12-19T08:02:54.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385000ac30 con 0x7f385814fba0 2023-12-19T08:02:54.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3858153190 con 0x7f385814fba0 2023-12-19T08:02:54.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.517+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38580c86c0 con 0x7f385814fba0 2023-12-19T08:02:54.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.521+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f38500070f0 con 0x7f385814fba0 2023-12-19T08:02:54.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.521+0000 7f384d7fa700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 0x7f383c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.521+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f385007caa0 con 0x7f385814fba0 2023-12-19T08:02:54.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.521+0000 7f385d6e1700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 0x7f383c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.521+0000 7f385d6e1700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 0x7f383c062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3854009d10 tx=0x7f38540091a0 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.525+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f385004cf70 con 0x7f385814fba0 2023-12-19T08:02:54.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.749+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3858071120 con 0x7f385814fba0 2023-12-19T08:02:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.753+0000 7f384d7fa700 1 -- 172.21.15.111:0/3577882412 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3850050590 con 0x7f385814fba0 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.755 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:02:54.758 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:02:54.761 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:02:54.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.765 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:02:54.766 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:02:54.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:02:54.769 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 msgr2=0x7f383c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 0x7f383c062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3854009d10 tx=0x7f38540091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 msgr2=0x7f38580b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f385000cd80 tx=0x7f3850001b10 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 shutdown_connections 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f383c060670 0x7f383c062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f385814fba0 0x7f38580b5100 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38580c5900 0x7f3858150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 --2- 172.21.15.111:0/3577882412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3858067c90 0x7f3858152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 >> 172.21.15.111:0/3577882412 conn(0x7f38580cd6d0 msgr2=0x7f38580ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:54.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 shutdown_connections 2023-12-19T08:02:54.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.757+0000 7f385f945700 1 -- 172.21.15.111:0/3577882412 wait complete. 2023-12-19T08:02:54.778 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:54.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 Processor -- start 2023-12-19T08:02:54.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 -- start start 2023-12-19T08:02:54.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc13f890 0x7efefc0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efefc0680f0 0x7efefc068570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc06c9f0 con 0x7efefc0c2980 2023-12-19T08:02:54.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc068ab0 con 0x7efefc0680f0 2023-12-19T08:02:54.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7eff016ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0c7f20 con 0x7efefc13f890 2023-12-19T08:02:54.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7efefa7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efefc0c2980 0x7efefc0c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34130/0 (socket says 172.21.15.111:34130) 2023-12-19T08:02:54.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.941+0000 7efefa7fc700 1 -- 172.21.15.111:0/2060676353 learned_addr learned my addr 172.21.15.111:0/2060676353 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:54.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 --2- 172.21.15.111:0/2060676353 >> v2:172.21.15.143:3300/0 conn(0x7efefc0680f0 0x7efefc068570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefaffd700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc13f890 0x7efefc0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efef9ffb700 1 -- 172.21.15.111:0/2060676353 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858424356 0 0) 0x7efefc06c9f0 con 0x7efefc0c2980 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efef9ffb700 1 -- 172.21.15.111:0/2060676353 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee4003570 con 0x7efefc0c2980 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 -- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc13f890 msgr2=0x7efefc0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc13f890 0x7efefc0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 -- 172.21.15.111:0/2060676353 >> v1:172.21.15.111:6789/0 conn(0x7efefc0c2980 legacy=0x7efefc0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 -- 172.21.15.111:0/2060676353 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0c1c50 con 0x7efefc0680f0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efefb7fe700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc068570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e60 tx=0x7efef4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f00e3562a97be6e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efef9ffb700 1 -- 172.21.15.111:0/2060676353 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef400e070 con 0x7efefc0680f0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efef9ffb700 1 -- 172.21.15.111:0/2060676353 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef400bab0 con 0x7efefc0680f0 2023-12-19T08:02:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7efef9ffb700 1 -- 172.21.15.111:0/2060676353 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef400be70 con 0x7efefc0680f0 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 -- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 msgr2=0x7efefc068570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc068570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e60 tx=0x7efef4009780 comp rx=0 tx=0).stop 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 -- 172.21.15.111:0/2060676353 shutdown_connections 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc068570 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 --2- 172.21.15.111:0/2060676353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc13f890 0x7efefc0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 -- 172.21.15.111:0/2060676353 >> 172.21.15.111:0/2060676353 conn(0x7efefc0c8850 msgr2=0x7efefc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 -- 172.21.15.111:0/2060676353 shutdown_connections 2023-12-19T08:02:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.945+0000 7eff016ae700 1 -- 172.21.15.111:0/2060676353 wait complete. 2023-12-19T08:02:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.949+0000 7eff016ae700 1 Processor -- start 2023-12-19T08:02:54.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.949+0000 7eff016ae700 1 -- start start 2023-12-19T08:02:54.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc0c2980 0x7efefc140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc13a140 con 0x7efefc13f890 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc1410c0 con 0x7efefc0680f0 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc1413a0 con 0x7efefc0c2980 2023-12-19T08:02:54.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53062/0 (socket says 172.21.15.111:53062) 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 -- 172.21.15.111:0/819125565 learned_addr learned my addr 172.21.15.111:0/819125565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefa7fc700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc0c2980 0x7efefc140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefaffd700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 -- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc0c2980 msgr2=0x7efefc140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc0c2980 0x7efefc140610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 -- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 msgr2=0x7efefc0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc0c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 -- 172.21.15.111:0/819125565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4009a20 con 0x7efefc13f890 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efefb7fe700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efefc06a2b0 tx=0x7efef4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4012e10 con 0x7efefc13f890 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4027470 con 0x7efefc13f890 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4027830 con 0x7efefc13f890 2023-12-19T08:02:54.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc141680 con 0x7efefc13f890 2023-12-19T08:02:54.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.953+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc141c40 con 0x7efefc13f890 2023-12-19T08:02:54.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.957+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc071120 con 0x7efefc13f890 2023-12-19T08:02:54.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.957+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7efef4024070 con 0x7efefc13f890 2023-12-19T08:02:54.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.957+0000 7efee2ffd700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 0x7efee8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:54.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.957+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5693+0+0 (secure 0 0 0) 0x7efef407f3a0 con 0x7efefc13f890 2023-12-19T08:02:54.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.957+0000 7efefaffd700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 0x7efee8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:54.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.961+0000 7efefaffd700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 0x7efee8062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efeec009d10 tx=0x7efeec0091a0 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:54.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:54.965+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efef404f7f0 con 0x7efefc13f890 2023-12-19T08:02:55.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.133+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efefc0642a0 con 0x7efefc13f890 2023-12-19T08:02:55.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.133+0000 7efee2ffd700 1 -- 172.21.15.111:0/819125565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+13177 (secure 0 0 0) 0x7efef4052e10 con 0x7efefc13f890 2023-12-19T08:02:55.136 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:55.137 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:54.298791+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 msgr2=0x7efee8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 0x7efee8062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efeec009d10 tx=0x7efeec0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 msgr2=0x7efefc0c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efefc06a2b0 tx=0x7efef4019000 comp rx=0 tx=0).stop 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 shutdown_connections 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efee8060670 0x7efee8062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc13f890 0x7efefc0c43e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efefc0c2980 0x7efefc140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 --2- 172.21.15.111:0/819125565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efefc0680f0 0x7efefc0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 >> 172.21.15.111:0/819125565 conn(0x7efefc0c8850 msgr2=0x7efefc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 shutdown_connections 2023-12-19T08:02:55.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.137+0000 7eff016ae700 1 -- 172.21.15.111:0/819125565 wait complete. 2023-12-19T08:02:55.159 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:02:55.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:02:55.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.333+0000 7f1279216700 1 Processor -- start 2023-12-19T08:02:55.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.333+0000 7f1279216700 1 -- start start 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1279216700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f127406aeb0 0x7f1274159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1279216700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12741595a0 0x7f12741599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1279216700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127406c850 con 0x7f1274158c90 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1279216700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12740cb650 con 0x7f12741595a0 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1279216700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127406b470 con 0x7f127406aeb0 2023-12-19T08:02:55.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12741595a0 0x7f12741599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1273fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1274158c90 0x7f12740caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34134/0 (socket says 172.21.15.111:34134) 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1273fff700 1 -- 172.21.15.111:0/2589044782 learned_addr learned my addr 172.21.15.111:0/2589044782 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f12737fe700 1 --2- 172.21.15.111:0/2589044782 >> v2:172.21.15.143:3300/0 conn(0x7f127406aeb0 0x7f1274159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 -- 172.21.15.111:0/2589044782 >> v2:172.21.15.143:3300/0 conn(0x7f127406aeb0 msgr2=0x7f1274159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 --2- 172.21.15.111:0/2589044782 >> v2:172.21.15.143:3300/0 conn(0x7f127406aeb0 0x7f1274159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 -- 172.21.15.111:0/2589044782 >> v1:172.21.15.111:6789/0 conn(0x7f1274158c90 legacy=0x7f12740caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 -- 172.21.15.111:0/2589044782 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127406b750 con 0x7f12741595a0 2023-12-19T08:02:55.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1272ffd700 1 -- 172.21.15.111:0/2589044782 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33897567 0 0) 0x7f127406c850 con 0x7f1274158c90 2023-12-19T08:02:55.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1278a15700 1 --2- 172.21.15.111:0/2589044782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12741595a0 0x7f12741599c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1274068000 tx=0x7f126c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61590887331b381f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1272ffd700 1 -- 172.21.15.111:0/2589044782 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c00e070 con 0x7f12741595a0 2023-12-19T08:02:55.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1272ffd700 1 -- 172.21.15.111:0/2589044782 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c00bc30 con 0x7f12741595a0 2023-12-19T08:02:55.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.337+0000 7f1272ffd700 1 -- 172.21.15.111:0/2589044782 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c012430 con 0x7f12741595a0 2023-12-19T08:02:55.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 -- 172.21.15.111:0/2589044782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12741595a0 msgr2=0x7f12741599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 --2- 172.21.15.111:0/2589044782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12741595a0 0x7f12741599c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1274068000 tx=0x7f126c009780 comp rx=0 tx=0).stop 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 -- 172.21.15.111:0/2589044782 shutdown_connections 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 --2- 172.21.15.111:0/2589044782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12741595a0 0x7f12741599c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 --2- 172.21.15.111:0/2589044782 >> v2:172.21.15.143:3300/0 conn(0x7f127406aeb0 0x7f1274159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 -- 172.21.15.111:0/2589044782 >> 172.21.15.111:0/2589044782 conn(0x7f1274067bb0 msgr2=0x7f12740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 -- 172.21.15.111:0/2589044782 shutdown_connections 2023-12-19T08:02:55.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.341+0000 7f1279216700 1 -- 172.21.15.111:0/2589044782 wait complete. 2023-12-19T08:02:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 Processor -- start 2023-12-19T08:02:55.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 -- start start 2023-12-19T08:02:55.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 0x7f1274143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12741595a0 0x7f12741694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740c62d0 con 0x7f1274158c90 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740b5110 con 0x7f127406aeb0 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1279216700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12740b47e0 con 0x7f12741595a0 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f12737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f12737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53088/0 (socket says 172.21.15.111:53088) 2023-12-19T08:02:55.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f12737fe700 1 -- 172.21.15.111:0/971705043 learned_addr learned my addr 172.21.15.111:0/971705043 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1273fff700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 0x7f1274143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.345+0000 7f1278a15700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12741595a0 0x7f12741694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 -- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12741595a0 msgr2=0x7f12741694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12741595a0 0x7f12741694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 -- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 msgr2=0x7f1274143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 0x7f1274143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 -- 172.21.15.111:0/971705043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c009a20 con 0x7f1274158c90 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f1273fff700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 0x7f1274143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:55.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f12737fe700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f126800d8c0 tx=0x7f1268008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126800de60 con 0x7f1274158c90 2023-12-19T08:02:55.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126800b780 con 0x7f1274158c90 2023-12-19T08:02:55.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12740b4ac0 con 0x7f1274158c90 2023-12-19T08:02:55.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126800bb20 con 0x7f1274158c90 2023-12-19T08:02:55.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12740c5260 con 0x7f1274158c90 2023-12-19T08:02:55.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.349+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127404be40 con 0x7f1274158c90 2023-12-19T08:02:55.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.353+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f126801e460 con 0x7f1274158c90 2023-12-19T08:02:55.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.353+0000 7f125bfff700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 0x7f1260062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.353+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f126807d0c0 con 0x7f1274158c90 2023-12-19T08:02:55.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.353+0000 7f1273fff700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 0x7f1260062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.353+0000 7f1273fff700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 0x7f1260062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12640011e0 tx=0x7f1264009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.361+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f126804d5b0 con 0x7f1274158c90 2023-12-19T08:02:55.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.533+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12740b5250 con 0x7f1274158c90 2023-12-19T08:02:55.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f125bfff700 1 -- 172.21.15.111:0/971705043 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1268050bd0 con 0x7f1274158c90 2023-12-19T08:02:55.538 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 msgr2=0x7f1260062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 0x7f1260062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12640011e0 tx=0x7f1264009040 comp rx=0 tx=0).stop 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 msgr2=0x7f1274168fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f126800d8c0 tx=0x7f1268008040 comp rx=0 tx=0).stop 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 shutdown_connections 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1260060670 0x7f1260062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12741595a0 0x7f12741694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1274158c90 0x7f1274168fa0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 --2- 172.21.15.111:0/971705043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f127406aeb0 0x7f1274143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 >> 172.21.15.111:0/971705043 conn(0x7f1274067bb0 msgr2=0x7f1274157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:55.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 shutdown_connections 2023-12-19T08:02:55.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.537+0000 7f1279216700 1 -- 172.21.15.111:0/971705043 wait complete. 2023-12-19T08:02:55.560 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:02:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.717+0000 7f28e8923700 1 Processor -- start 2023-12-19T08:02:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.717+0000 7f28e8923700 1 -- start start 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e8923700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28e41591f0 0x7f28e406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e8923700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28e40ce810 0x7f28e40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e8923700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e406b730 con 0x7f28e4158e20 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e8923700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e406ab20 con 0x7f28e41591f0 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e8923700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e406ae00 con 0x7f28e40ce810 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28e41591f0 0x7f28e406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28e41591f0 0x7f28e406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53096/0 (socket says 172.21.15.111:53096) 2023-12-19T08:02:55.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 -- 172.21.15.111:0/1819748568 learned_addr learned my addr 172.21.15.111:0/1819748568 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:55.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 -- 172.21.15.111:0/1819748568 >> v2:172.21.15.143:3300/0 conn(0x7f28e40ce810 msgr2=0x7f28e40ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:55.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 --2- 172.21.15.111:0/1819748568 >> v2:172.21.15.143:3300/0 conn(0x7f28e40ce810 0x7f28e40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 -- 172.21.15.111:0/1819748568 >> v1:172.21.15.111:6789/0 conn(0x7f28e4158e20 legacy=0x7f28e40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 -- 172.21.15.111:0/1819748568 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e406b0e0 con 0x7f28e41591f0 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.721+0000 7f28e2ffd700 1 --2- 172.21.15.111:0/1819748568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e406a540 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f28e406dc00 tx=0x7f28d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a837f37f4222db server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e27fc700 1 -- 172.21.15.111:0/1819748568 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d800b860 con 0x7f28e41591f0 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e27fc700 1 -- 172.21.15.111:0/1819748568 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d800b9c0 con 0x7f28e41591f0 2023-12-19T08:02:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e27fc700 1 -- 172.21.15.111:0/1819748568 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d800bd60 con 0x7f28e41591f0 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- 172.21.15.111:0/1819748568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 msgr2=0x7f28e406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- 172.21.15.111:0/1819748568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e406a540 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f28e406dc00 tx=0x7f28d8009250 comp rx=0 tx=0).stop 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- 172.21.15.111:0/1819748568 shutdown_connections 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- 172.21.15.111:0/1819748568 >> v2:172.21.15.143:3300/0 conn(0x7f28e40ce810 0x7f28e40ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- 172.21.15.111:0/1819748568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e406a540 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- 172.21.15.111:0/1819748568 >> 172.21.15.111:0/1819748568 conn(0x7f28e4067bb0 msgr2=0x7f28e40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- 172.21.15.111:0/1819748568 shutdown_connections 2023-12-19T08:02:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- 172.21.15.111:0/1819748568 wait complete. 2023-12-19T08:02:55.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 Processor -- start 2023-12-19T08:02:55.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- start start 2023-12-19T08:02:55.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40ce810 0x7f28e416a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e4158e20 0x7f28e41534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e414b160 con 0x7f28e41591f0 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4169e20 con 0x7f28e4158e20 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.725+0000 7f28e8923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e416aa50 con 0x7f28e40ce810 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53108/0 (socket says 172.21.15.111:53108) 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 -- 172.21.15.111:0/1655225347 learned_addr learned my addr 172.21.15.111:0/1655225347 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e2ffd700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e4158e20 0x7f28e41534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e37fe700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40ce810 0x7f28e416a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 -- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40ce810 msgr2=0x7f28e416a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40ce810 0x7f28e416a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 -- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e4158e20 msgr2=0x7f28e41534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e4158e20 0x7f28e41534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 -- 172.21.15.111:0/1655225347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80094f0 con 0x7f28e41591f0 2023-12-19T08:02:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e3fff700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00bc30 tx=0x7f28dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0092e0 con 0x7f28e41591f0 2023-12-19T08:02:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc00c7a0 con 0x7f28e41591f0 2023-12-19T08:02:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e414d2d0 con 0x7f28e41591f0 2023-12-19T08:02:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00cb80 con 0x7f28e41591f0 2023-12-19T08:02:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e414d890 con 0x7f28e41591f0 2023-12-19T08:02:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.729+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e404be40 con 0x7f28e41591f0 2023-12-19T08:02:55.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.733+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f28dc012070 con 0x7f28e41591f0 2023-12-19T08:02:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.733+0000 7f28cb7fe700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 0x7f28d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.733+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f28dc07c680 con 0x7f28e41591f0 2023-12-19T08:02:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.733+0000 7f28e37fe700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 0x7f28d0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.733+0000 7f28e37fe700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 0x7f28d0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f28d4003860 tx=0x7f28d4009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:55.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.737+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f28dc04cad0 con 0x7f28e41591f0 2023-12-19T08:02:55.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f28e406b3c0 con 0x7f28e41591f0 2023-12-19T08:02:55.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28cb7fe700 1 -- 172.21.15.111:0/1655225347 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+65 (secure 0 0 0) 0x7f28dc0500f0 con 0x7f28e41591f0 2023-12-19T08:02:55.910 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:02:55.911 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:02:55.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 msgr2=0x7f28d0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 0x7f28d0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f28d4003860 tx=0x7f28d4009040 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 msgr2=0x7f28e41539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.909+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00bc30 tx=0x7f28dc009840 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 shutdown_connections 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f28d0060670 0x7f28d0062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28e4158e20 0x7f28e41534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28e41591f0 0x7f28e41539f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 --2- 172.21.15.111:0/1655225347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28e40ce810 0x7f28e416a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 >> 172.21.15.111:0/1655225347 conn(0x7f28e4067bb0 msgr2=0x7f28e40cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 shutdown_connections 2023-12-19T08:02:55.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:55.913+0000 7f28e8923700 1 -- 172.21.15.111:0/1655225347 wait complete. 2023-12-19T08:02:55.935 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:02:55.936 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:02:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.101+0000 7f450042e700 1 Processor -- start 2023-12-19T08:02:56.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.101+0000 7f450042e700 1 -- start start 2023-12-19T08:02:56.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f450042e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f80cb130 0x7f44f80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f450042e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44f815c930 0x7f44f806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f450042e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f80b0ac0 con 0x7f44f806a6a0 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f450042e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f806b030 con 0x7f44f815c930 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f450042e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f806b210 con 0x7f44f80cb130 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44f815c930 0x7f44f806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44f815c930 0x7f44f806aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53114/0 (socket says 172.21.15.111:53114) 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fd9c9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f44f806a6a0 0x7f44f815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34164/0 (socket says 172.21.15.111:34164) 2023-12-19T08:02:56.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 -- 172.21.15.111:0/1615244657 learned_addr learned my addr 172.21.15.111:0/1615244657 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe1ca700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f80cb130 0x7f44f80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 -- 172.21.15.111:0/1615244657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f80cb130 msgr2=0x7f44f80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f80cb130 0x7f44f80cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 -- 172.21.15.111:0/1615244657 >> v1:172.21.15.111:6789/0 conn(0x7f44f806a6a0 legacy=0x7f44f815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 -- 172.21.15.111:0/1615244657 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80ca810 con 0x7f44f815c930 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fd1c8700 1 -- 172.21.15.111:0/1615244657 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896464801 0 0) 0x7f44f80b0ac0 con 0x7f44f806a6a0 2023-12-19T08:02:56.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.105+0000 7f44fe9cb700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f815c930 0x7f44f806aaf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009f20 tx=0x7f44f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4534322c36c4524d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:56.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f44fd1c8700 1 -- 172.21.15.111:0/1615244657 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f000e070 con 0x7f44f815c930 2023-12-19T08:02:56.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f44fd1c8700 1 -- 172.21.15.111:0/1615244657 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f000b9d0 con 0x7f44f815c930 2023-12-19T08:02:56.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f44fd1c8700 1 -- 172.21.15.111:0/1615244657 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f00092e0 con 0x7f44f815c930 2023-12-19T08:02:56.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 -- 172.21.15.111:0/1615244657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f815c930 msgr2=0x7f44f806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f815c930 0x7f44f806aaf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009f20 tx=0x7f44f0009840 comp rx=0 tx=0).stop 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 -- 172.21.15.111:0/1615244657 shutdown_connections 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f815c930 0x7f44f806aaf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 --2- 172.21.15.111:0/1615244657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f80cb130 0x7f44f80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 -- 172.21.15.111:0/1615244657 >> 172.21.15.111:0/1615244657 conn(0x7f44f8067bb0 msgr2=0x7f44f80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 -- 172.21.15.111:0/1615244657 shutdown_connections 2023-12-19T08:02:56.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.109+0000 7f450042e700 1 -- 172.21.15.111:0/1615244657 wait complete. 2023-12-19T08:02:56.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 Processor -- start 2023-12-19T08:02:56.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 -- start start 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f80cb130 0x7f44f816c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f815c930 0x7f44f815d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f815f070 con 0x7f44f806a6a0 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f815f500 con 0x7f44f80cb130 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f450042e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f815e460 con 0x7f44f815c930 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f44fe1ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f44fe1ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53126/0 (socket says 172.21.15.111:53126) 2023-12-19T08:02:56.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.113+0000 7f44fe1ca700 1 -- 172.21.15.111:0/2501029023 learned_addr learned my addr 172.21.15.111:0/2501029023 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:56.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fd9c9700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f80cb130 0x7f44f816c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe9cb700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f815c930 0x7f44f815d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 -- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f815c930 msgr2=0x7f44f815d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f815c930 0x7f44f815d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 -- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f80cb130 msgr2=0x7f44f816c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f80cb130 0x7f44f816c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 -- 172.21.15.111:0/2501029023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f0009ae0 con 0x7f44f806a6a0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44fe1ca700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f44f400ecb0 tx=0x7f44f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f400c9d0 con 0x7f44f806a6a0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f400cb30 con 0x7f44f806a6a0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f815e740 con 0x7f44f806a6a0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f400ced0 con 0x7f44f806a6a0 2023-12-19T08:02:56.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f815ed00 con 0x7f44f806a6a0 2023-12-19T08:02:56.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.117+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f8071120 con 0x7f44f806a6a0 2023-12-19T08:02:56.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.121+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f44f4013680 con 0x7f44f806a6a0 2023-12-19T08:02:56.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.121+0000 7f44ee7fc700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 0x7f44dc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:56.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.121+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f44f407cc70 con 0x7f44f806a6a0 2023-12-19T08:02:56.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.121+0000 7f44fd9c9700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 0x7f44dc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:56.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.121+0000 7f44fd9c9700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 0x7f44dc062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44f80b55d0 tx=0x7f44e8007bf0 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:56.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.129+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44f404d160 con 0x7f44f806a6a0 2023-12-19T08:02:56.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.305+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f44f80cbe90 con 0x7f44f806a6a0 2023-12-19T08:02:56.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.901+0000 7f44ee7fc700 1 -- 172.21.15.111:0/2501029023 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44f4050780 con 0x7f44f806a6a0 2023-12-19T08:02:56.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 msgr2=0x7f44dc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 0x7f44dc062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44f80b55d0 tx=0x7f44e8007bf0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 msgr2=0x7f44f816bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f44f400ecb0 tx=0x7f44f40091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 shutdown_connections 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f44dc060670 0x7f44dc062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f80cb130 0x7f44f816c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f815c930 0x7f44f815d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 --2- 172.21.15.111:0/2501029023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44f806a6a0 0x7f44f816bae0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:56.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 >> 172.21.15.111:0/2501029023 conn(0x7f44f8067bb0 msgr2=0x7f44f80b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:56.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 shutdown_connections 2023-12-19T08:02:56.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:56.905+0000 7f450042e700 1 -- 172.21.15.111:0/2501029023 wait complete. 2023-12-19T08:02:56.911 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:02:56.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:02:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.109+0000 7f02aec6b700 1 Processor -- start 2023-12-19T08:02:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 -- start start 2023-12-19T08:02:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02a80ce2d0 0x7f02a80ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80cc040 0x7f02a8159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80cbbb0 con 0x7f02a80caa10 2023-12-19T08:02:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a806dc90 con 0x7f02a80ce2d0 2023-12-19T08:02:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.113+0000 7f02aec6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a80cafa0 con 0x7f02a80cc040 2023-12-19T08:02:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02a80ce2d0 0x7f02a80ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02a80ce2d0 0x7f02a80ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53134/0 (socket says 172.21.15.111:53134) 2023-12-19T08:02:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 -- 172.21.15.111:0/941317958 learned_addr learned my addr 172.21.15.111:0/941317958 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02ae46a700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80cc040 0x7f02a8159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 -- 172.21.15.111:0/941317958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80cc040 msgr2=0x7f02a8159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80cc040 0x7f02a8159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02acc67700 1 -- 172.21.15.111:0/941317958 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109015608 0 0) 0x7f02a80cbbb0 con 0x7f02a80caa10 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 -- 172.21.15.111:0/941317958 >> v1:172.21.15.111:6789/0 conn(0x7f02a80caa10 legacy=0x7f02a8158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 -- 172.21.15.111:0/941317958 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80cb280 con 0x7f02a80ce2d0 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02adc69700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80ce2d0 0x7f02a80ce6a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f02a40099b0 tx=0x7f02a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89135a3d06c7a4c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02acc67700 1 -- 172.21.15.111:0/941317958 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4009ea0 con 0x7f02a80ce2d0 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02acc67700 1 -- 172.21.15.111:0/941317958 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400b860 con 0x7f02a80ce2d0 2023-12-19T08:02:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02acc67700 1 -- 172.21.15.111:0/941317958 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a400bc00 con 0x7f02a80ce2d0 2023-12-19T08:02:57.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 -- 172.21.15.111:0/941317958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80ce2d0 msgr2=0x7f02a80ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80ce2d0 0x7f02a80ce6a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f02a40099b0 tx=0x7f02a4009250 comp rx=0 tx=0).stop 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 -- 172.21.15.111:0/941317958 shutdown_connections 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80cc040 0x7f02a8159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 --2- 172.21.15.111:0/941317958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80ce2d0 0x7f02a80ce6a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 -- 172.21.15.111:0/941317958 >> 172.21.15.111:0/941317958 conn(0x7f02a8067bb0 msgr2=0x7f02a80cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 -- 172.21.15.111:0/941317958 shutdown_connections 2023-12-19T08:02:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.117+0000 7f02aec6b700 1 -- 172.21.15.111:0/941317958 wait complete. 2023-12-19T08:02:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 Processor -- start 2023-12-19T08:02:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 -- start start 2023-12-19T08:02:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a80cc040 0x7f02a815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80ce2d0 0x7f02a8155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8169270 con 0x7f02a80caa10 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8169530 con 0x7f02a80cc040 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02aec6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8169810 con 0x7f02a80ce2d0 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53138/0 (socket says 172.21.15.111:53138) 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 -- 172.21.15.111:0/2500818039 learned_addr learned my addr 172.21.15.111:0/2500818039 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02ae46a700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80ce2d0 0x7f02a8155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02ad468700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a80cc040 0x7f02a815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 -- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80ce2d0 msgr2=0x7f02a8155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80ce2d0 0x7f02a8155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 -- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a80cc040 msgr2=0x7f02a815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a80cc040 0x7f02a815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.121+0000 7f02adc69700 1 -- 172.21.15.111:0/2500818039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a40094f0 con 0x7f02a80caa10 2023-12-19T08:02:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f02adc69700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f02a80d07c0 tx=0x7f02a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4000d10 con 0x7f02a80caa10 2023-12-19T08:02:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400bbe0 con 0x7f02a80caa10 2023-12-19T08:02:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4011690 con 0x7f02a80caa10 2023-12-19T08:02:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8169af0 con 0x7f02a80caa10 2023-12-19T08:02:57.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a816a130 con 0x7f02a80caa10 2023-12-19T08:02:57.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a804be40 con 0x7f02a80caa10 2023-12-19T08:02:57.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.125+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f02a4011ac0 con 0x7f02a80caa10 2023-12-19T08:02:57.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.129+0000 7f029dffb700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 0x7f028c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:57.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.129+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6063+0+0 (secure 0 0 0) 0x7f02a407e170 con 0x7f02a80caa10 2023-12-19T08:02:57.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.129+0000 7f02ad468700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 0x7f028c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.129+0000 7f02ad468700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 0x7f028c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02a806b350 tx=0x7f0298007d30 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:57.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.137+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02a404e4f0 con 0x7f02a80caa10 2023-12-19T08:02:57.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.305+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f02a80cb560 con 0x7f02a80caa10 2023-12-19T08:02:57.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.913+0000 7f029dffb700 1 -- 172.21.15.111:0/2500818039 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) v1 ==== 170+0+0 (secure 0 0 0) 0x7f02a4051b10 con 0x7f02a80caa10 2023-12-19T08:02:57.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.921+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 msgr2=0x7f028c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.921+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 0x7f028c062b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f02a806b350 tx=0x7f0298007d30 comp rx=0 tx=0).stop 2023-12-19T08:02:57.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.921+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 msgr2=0x7f02a80b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:57.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.921+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f02a80d07c0 tx=0x7f02a4017000 comp rx=0 tx=0).stop 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 shutdown_connections 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f028c0606a0 0x7f028c062b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a80cc040 0x7f02a815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02a80ce2d0 0x7f02a8155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 --2- 172.21.15.111:0/2500818039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80caa10 0x7f02a80b2050 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 >> 172.21.15.111:0/2500818039 conn(0x7f02a8067bb0 msgr2=0x7f02a806a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:57.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 shutdown_connections 2023-12-19T08:02:57.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:57.925+0000 7f02aec6b700 1 -- 172.21.15.111:0/2500818039 wait complete. 2023-12-19T08:02:57.932 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:02:57.954 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:02:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.129+0000 7fd8d63c5700 1 Processor -- start 2023-12-19T08:02:58.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.129+0000 7fd8d63c5700 1 -- start start 2023-12-19T08:02:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d63c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8d00b1a50 0x7fd8d00b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d63c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d00ae2b0 con 0x7fd8d00ca370 2023-12-19T08:02:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d63c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d00ad980 con 0x7fd8d0069a90 2023-12-19T08:02:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d63c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d01562b0 con 0x7fd8d00b1a50 2023-12-19T08:02:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8d00b1a50 0x7fd8d00b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d4962700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd8d00ca370 0x7fd8d00ab910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34188/0 (socket says 172.21.15.111:34188) 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8d4962700 1 -- 172.21.15.111:0/3554318609 learned_addr learned my addr 172.21.15.111:0/3554318609 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cf7fe700 1 -- 172.21.15.111:0/3554318609 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141923209 0 0) 0x7fd8d00ae2b0 con 0x7fd8d00ca370 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cf7fe700 1 -- 172.21.15.111:0/3554318609 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc003570 con 0x7fd8d00ca370 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cffff700 1 -- 172.21.15.111:0/3554318609 >> v1:172.21.15.143:6789/0 conn(0x7fd8d0069a90 legacy=0x7fd8d00b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cffff700 1 -- 172.21.15.111:0/3554318609 >> v1:172.21.15.111:6789/0 conn(0x7fd8d00ca370 legacy=0x7fd8d00ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cffff700 1 -- 172.21.15.111:0/3554318609 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00adc60 con 0x7fd8d00b1a50 2023-12-19T08:02:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cffff700 1 --2- 172.21.15.111:0/3554318609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00b1a50 0x7fd8d00b1e20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098e0 tx=0x7fd8c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e892710c2a31922a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cf7fe700 1 -- 172.21.15.111:0/3554318609 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000b640 con 0x7fd8d00b1a50 2023-12-19T08:02:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cf7fe700 1 -- 172.21.15.111:0/3554318609 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c000b7a0 con 0x7fd8d00b1a50 2023-12-19T08:02:58.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.133+0000 7fd8cf7fe700 1 -- 172.21.15.111:0/3554318609 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000bb40 con 0x7fd8d00b1a50 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- 172.21.15.111:0/3554318609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00b1a50 msgr2=0x7fd8d00b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/3554318609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00b1a50 0x7fd8d00b1e20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098e0 tx=0x7fd8c00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- 172.21.15.111:0/3554318609 shutdown_connections 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/3554318609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00b1a50 0x7fd8d00b1e20 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- 172.21.15.111:0/3554318609 >> 172.21.15.111:0/3554318609 conn(0x7fd8d00cd870 msgr2=0x7fd8d0069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- 172.21.15.111:0/3554318609 shutdown_connections 2023-12-19T08:02:58.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- 172.21.15.111:0/3554318609 wait complete. 2023-12-19T08:02:58.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 Processor -- start 2023-12-19T08:02:58.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.137+0000 7fd8d63c5700 1 -- start start 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d0069a90 0x7fd8d00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 0x7fd8d00c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00c5940 con 0x7fd8d00ca370 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00c4d30 con 0x7fd8d0069a90 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00c5010 con 0x7fd8d00b1a50 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53150/0 (socket says 172.21.15.111:53150) 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 -- 172.21.15.111:0/1390629188 learned_addr learned my addr 172.21.15.111:0/1390629188 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8cffff700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d0069a90 0x7fd8d00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8c77fe700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 0x7fd8d00c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 -- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 msgr2=0x7fd8d00c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 0x7fd8d00c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 -- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d0069a90 msgr2=0x7fd8d00a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d0069a90 0x7fd8d00a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 -- 172.21.15.111:0/1390629188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0009420 con 0x7fd8d00ca370 2023-12-19T08:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8c77fe700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 0x7fd8d00c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d4962700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8001460 tx=0x7fd8c8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800a8a0 con 0x7fd8d00ca370 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800aa00 con 0x7fd8d00ca370 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d00c52f0 con 0x7fd8d00ca370 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800ad80 con 0x7fd8d00ca370 2023-12-19T08:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.141+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d00c7450 con 0x7fd8d00ca370 2023-12-19T08:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.145+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d00c5af0 con 0x7fd8d00ca370 2023-12-19T08:02:58.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.149+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd8c8007300 con 0x7fd8d00ca370 2023-12-19T08:02:58.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.149+0000 7fd8ccff9700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 0x7fd8b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:58.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.149+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fd8c807d8f0 con 0x7fd8d00ca370 2023-12-19T08:02:58.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.149+0000 7fd8cffff700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 0x7fd8b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:58.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.149+0000 7fd8cffff700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 0x7fd8b0062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00093f0 tx=0x7fd8c0017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:58.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.153+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8c804db50 con 0x7fd8d00ca370 2023-12-19T08:02:58.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.357+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd8d0071120 con 0x7fd8d00ca370 2023-12-19T08:02:58.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.933+0000 7fd8ccff9700 1 -- 172.21.15.111:0/1390629188 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v36) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd8c8051170 con 0x7fd8d00ca370 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 msgr2=0x7fd8b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 0x7fd8b0062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00093f0 tx=0x7fd8c0017000 comp rx=0 tx=0).stop 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 msgr2=0x7fd8d0169600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8001460 tx=0x7fd8c8001a40 comp rx=0 tx=0).stop 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 shutdown_connections 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8b0060670 0x7fd8b0062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d00ca370 0x7fd8d0169600 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00b1a50 0x7fd8d00c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 --2- 172.21.15.111:0/1390629188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d0069a90 0x7fd8d00a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:58.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.937+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 >> 172.21.15.111:0/1390629188 conn(0x7fd8d00cd870 msgr2=0x7fd8d00c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:58.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.941+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 shutdown_connections 2023-12-19T08:02:58.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:58.941+0000 7fd8d63c5700 1 -- 172.21.15.111:0/1390629188 wait complete. 2023-12-19T08:02:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 11 and data pool 12 2023-12-19T08:02:58.965 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:59.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 Processor -- start 2023-12-19T08:02:59.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 -- start start 2023-12-19T08:02:59.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5ac067c90 0x7fe5ac068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5ac14fba0 0x7fe5ac150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac0c6630 con 0x7fe5ac0c5900 2023-12-19T08:02:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ac0c6910 con 0x7fe5ac14fba0 2023-12-19T08:02:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.125+0000 7fe5b1278700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ac0c6bf0 con 0x7fe5ac067c90 2023-12-19T08:02:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5ac067c90 0x7fe5ac068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5ac067c90 0x7fe5ac068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:55952/0 (socket says 172.21.15.111:55952) 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 -- 172.21.15.111:0/1237355718 learned_addr learned my addr 172.21.15.111:0/1237355718 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aa7fc700 1 --2- 172.21.15.111:0/1237355718 >> v2:172.21.15.111:3300/0 conn(0x7fe5ac14fba0 0x7fe5ac150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 -- 172.21.15.111:0/1237355718 >> v2:172.21.15.111:3300/0 conn(0x7fe5ac14fba0 msgr2=0x7fe5ac150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1237355718 >> v2:172.21.15.111:3300/0 conn(0x7fe5ac14fba0 0x7fe5ac150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 -- 172.21.15.111:0/1237355718 >> v1:172.21.15.143:6789/0 conn(0x7fe5ac0c5900 legacy=0x7fe5ac0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 -- 172.21.15.111:0/1237355718 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac0c6ed0 con 0x7fe5ac067c90 2023-12-19T08:02:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1237355718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac068060 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06a0c0 tx=0x7fe59c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c5ea90603dd4943 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5a9ffb700 1 -- 172.21.15.111:0/1237355718 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c009e70 con 0x7fe5ac067c90 2023-12-19T08:02:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5a9ffb700 1 -- 172.21.15.111:0/1237355718 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00b860 con 0x7fe5ac067c90 2023-12-19T08:02:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5a9ffb700 1 -- 172.21.15.111:0/1237355718 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00bc00 con 0x7fe5ac067c90 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 -- 172.21.15.111:0/1237355718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 msgr2=0x7fe5ac068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1237355718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac068060 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06a0c0 tx=0x7fe59c009250 comp rx=0 tx=0).stop 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 -- 172.21.15.111:0/1237355718 shutdown_connections 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1237355718 >> v2:172.21.15.111:3300/0 conn(0x7fe5ac14fba0 0x7fe5ac150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1237355718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac068060 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 -- 172.21.15.111:0/1237355718 >> 172.21.15.111:0/1237355718 conn(0x7fe5ac0cd6d0 msgr2=0x7fe5ac06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:59.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 -- 172.21.15.111:0/1237355718 shutdown_connections 2023-12-19T08:02:59.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.129+0000 7fe5b1278700 1 -- 172.21.15.111:0/1237355718 wait complete. 2023-12-19T08:02:59.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 Processor -- start 2023-12-19T08:02:59.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 -- start start 2023-12-19T08:02:59.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5ac0c5900 0x7fe5ac16e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac1522b0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac1500a0 con 0x7fe5ac067c90 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5b1278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac16e120 con 0x7fe5ac0c5900 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.133+0000 7fe5ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53162/0 (socket says 172.21.15.111:53162) 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 -- 172.21.15.111:0/1508479945 learned_addr learned my addr 172.21.15.111:0/1508479945 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5aa7fc700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5ac0c5900 0x7fe5ac16e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 -- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5ac0c5900 msgr2=0x7fe5ac16e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5ac0c5900 0x7fe5ac16e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 -- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 msgr2=0x7fe5ac151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 -- 172.21.15.111:0/1508479945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c0094f0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5ab7fe700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c8e0 tx=0x7fe5a4001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012630 con 0x7fe5ac14fba0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4012790 con 0x7fe5ac14fba0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac16e400 con 0x7fe5ac14fba0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012b50 con 0x7fe5ac14fba0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac16e6e0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.137+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ac150560 con 0x7fe5ac14fba0 2023-12-19T08:02:59.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.141+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe5a400a3b0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.141+0000 7fe592ffd700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 0x7fe598062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.141+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe5a407cfb0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.141+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 0x7fe598062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.145+0000 7fe5aaffd700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 0x7fe598062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06a0c0 tx=0x7fe59c017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.149+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe5a404d0a0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.329+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5ac071120 con 0x7fe5ac14fba0 2023-12-19T08:02:59.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.329+0000 7fe592ffd700 1 -- 172.21.15.111:0/1508479945 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16408 (secure 0 0 0) 0x7fe5a40506c0 con 0x7fe5ac14fba0 2023-12-19T08:02:59.332 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:59.332 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":98,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:58.362612+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 msgr2=0x7fe598062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 0x7fe598062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac06a0c0 tx=0x7fe59c017000 comp rx=0 tx=0).stop 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 msgr2=0x7fe5ac16d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c8e0 tx=0x7fe5a4001710 comp rx=0 tx=0).stop 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 shutdown_connections 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe598060670 0x7fe598062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5ac14fba0 0x7fe5ac16d7b0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5ac0c5900 0x7fe5ac16e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 --2- 172.21.15.111:0/1508479945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5ac067c90 0x7fe5ac151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 >> 172.21.15.111:0/1508479945 conn(0x7fe5ac0cd6d0 msgr2=0x7fe5ac14f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 shutdown_connections 2023-12-19T08:02:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.333+0000 7fe5b1278700 1 -- 172.21.15.111:0/1508479945 wait complete. 2023-12-19T08:02:59.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:02:59.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.513+0000 7f4083216700 1 Processor -- start 2023-12-19T08:02:59.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 -- start start 2023-12-19T08:02:59.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f407c06a580 0x7f407c06aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f407c06d210 0x7f407c06d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c06c7c0 con 0x7f407c0b1410 2023-12-19T08:02:59.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c06af40 con 0x7f407c06d210 2023-12-19T08:02:59.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4083216700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c06cd80 con 0x7f407c06a580 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f407c06d210 0x7f407c06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f407c06d210 0x7f407c06d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53174/0 (socket says 172.21.15.111:53174) 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 -- 172.21.15.111:0/2186071863 learned_addr learned my addr 172.21.15.111:0/2186071863 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4081a13700 1 --2- 172.21.15.111:0/2186071863 >> v2:172.21.15.143:3300/0 conn(0x7f407c06a580 0x7f407c06aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 -- 172.21.15.111:0/2186071863 >> v2:172.21.15.143:3300/0 conn(0x7f407c06a580 msgr2=0x7f407c06aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 --2- 172.21.15.111:0/2186071863 >> v2:172.21.15.143:3300/0 conn(0x7f407c06a580 0x7f407c06aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 -- 172.21.15.111:0/2186071863 >> v1:172.21.15.143:6789/0 conn(0x7f407c0b1410 legacy=0x7f407c0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 -- 172.21.15.111:0/2186071863 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c06b8f0 con 0x7f407c06d210 2023-12-19T08:02:59.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.517+0000 7f4082a15700 1 --2- 172.21.15.111:0/2186071863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c06d620 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f407c068040 tx=0x7f4074009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655834526e47078c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4081212700 1 -- 172.21.15.111:0/2186071863 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407400e070 con 0x7f407c06d210 2023-12-19T08:02:59.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4081212700 1 -- 172.21.15.111:0/2186071863 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407400ba20 con 0x7f407c06d210 2023-12-19T08:02:59.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4081212700 1 -- 172.21.15.111:0/2186071863 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40740092e0 con 0x7f407c06d210 2023-12-19T08:02:59.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 -- 172.21.15.111:0/2186071863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 msgr2=0x7f407c06d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 --2- 172.21.15.111:0/2186071863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c06d620 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f407c068040 tx=0x7f4074009840 comp rx=0 tx=0).stop 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 -- 172.21.15.111:0/2186071863 shutdown_connections 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 --2- 172.21.15.111:0/2186071863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c06d620 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 --2- 172.21.15.111:0/2186071863 >> v2:172.21.15.143:3300/0 conn(0x7f407c06a580 0x7f407c06aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 -- 172.21.15.111:0/2186071863 >> 172.21.15.111:0/2186071863 conn(0x7f407c067bb0 msgr2=0x7f407c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 -- 172.21.15.111:0/2186071863 shutdown_connections 2023-12-19T08:02:59.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.521+0000 7f4083216700 1 -- 172.21.15.111:0/2186071863 wait complete. 2023-12-19T08:02:59.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 Processor -- start 2023-12-19T08:02:59.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 -- start start 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c06a580 0x7f407c159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407c0b1410 0x7f407c0c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c16cf70 con 0x7f407c06d210 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c16a400 con 0x7f407c06a580 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4083216700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c169e40 con 0x7f407c0b1410 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53178/0 (socket says 172.21.15.111:53178) 2023-12-19T08:02:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 -- 172.21.15.111:0/154517876 learned_addr learned my addr 172.21.15.111:0/154517876 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4082a15700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407c0b1410 0x7f407c0c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4082214700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c06a580 0x7f407c159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 -- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407c0b1410 msgr2=0x7f407c0c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407c0b1410 0x7f407c0c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 -- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c06a580 msgr2=0x7f407c159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c06a580 0x7f407c159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 -- 172.21.15.111:0/154517876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4074009ae0 con 0x7f407c06d210 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.525+0000 7f4081a13700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f406c00ec80 tx=0x7f406c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c00c9d0 con 0x7f407c06d210 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c00cb30 con 0x7f407c06d210 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c16a120 con 0x7f407c06d210 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c013470 con 0x7f407c06d210 2023-12-19T08:02:59.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c159c70 con 0x7f407c06d210 2023-12-19T08:02:59.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c04be40 con 0x7f407c06d210 2023-12-19T08:02:59.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.529+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f406c013820 con 0x7f407c06d210 2023-12-19T08:02:59.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.533+0000 7f40727fc700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 0x7f4060062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.533+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f406c07cfd0 con 0x7f407c06d210 2023-12-19T08:02:59.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.533+0000 7f4082214700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 0x7f4060062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.533+0000 7f4082214700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 0x7f4060062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40780037c0 tx=0x7f4078009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.537+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f406c04d0c0 con 0x7f407c06d210 2023-12-19T08:02:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.709+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f407c0ab220 con 0x7f407c06d210 2023-12-19T08:02:59.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.709+0000 7f40727fc700 1 -- 172.21.15.111:0/154517876 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16395 (secure 0 0 0) 0x7f406c0506e0 con 0x7f407c06d210 2023-12-19T08:02:59.713 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:02:59.714 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":99,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:59.441190+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:02:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 msgr2=0x7f4060062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 0x7f4060062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40780037c0 tx=0x7f4078009040 comp rx=0 tx=0).stop 2023-12-19T08:02:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 msgr2=0x7f407c1694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f406c00ec80 tx=0x7f406c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 shutdown_connections 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40600606a0 0x7f4060062b50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407c0b1410 0x7f407c0c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f407c06d210 0x7f407c1694d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 --2- 172.21.15.111:0/154517876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c06a580 0x7f407c159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 >> 172.21.15.111:0/154517876 conn(0x7f407c067bb0 msgr2=0x7f407c0adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 shutdown_connections 2023-12-19T08:02:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.717+0000 7f4083216700 1 -- 172.21.15.111:0/154517876 wait complete. 2023-12-19T08:02:59.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 Processor -- start 2023-12-19T08:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 -- start start 2023-12-19T08:02:59.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ec00c69a0 0x7f2ec00c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec006de00 0x7f2ec006b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec006d970 con 0x7f2ec006c7c0 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec006cd60 con 0x7f2ec006de00 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec6e6b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec006d040 con 0x7f2ec00c69a0 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec5668700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec006de00 0x7f2ec006b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec666a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2ec006c7c0 0x7f2ec015cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34200/0 (socket says 172.21.15.111:34200) 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec5668700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec006de00 0x7f2ec006b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53190/0 (socket says 172.21.15.111:53190) 2023-12-19T08:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec666a700 1 -- 172.21.15.111:0/2909181137 learned_addr learned my addr 172.21.15.111:0/2909181137 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.909+0000 7f2ec5e69700 1 --2- 172.21.15.111:0/2909181137 >> v2:172.21.15.143:3300/0 conn(0x7f2ec00c69a0 0x7f2ec00c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec5668700 1 -- 172.21.15.111:0/2909181137 >> v2:172.21.15.143:3300/0 conn(0x7f2ec00c69a0 msgr2=0x7f2ec00c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec5668700 1 --2- 172.21.15.111:0/2909181137 >> v2:172.21.15.143:3300/0 conn(0x7f2ec00c69a0 0x7f2ec00c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec5668700 1 -- 172.21.15.111:0/2909181137 >> v1:172.21.15.111:6789/0 conn(0x7f2ec006c7c0 legacy=0x7f2ec015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec5668700 1 -- 172.21.15.111:0/2909181137 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec006d320 con 0x7f2ec006de00 2023-12-19T08:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec5668700 1 --2- 172.21.15.111:0/2909181137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec006b880 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00c8390 tx=0x7f2eb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9d73a037b0a807 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec4e67700 1 -- 172.21.15.111:0/2909181137 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000b6b0 con 0x7f2ec006de00 2023-12-19T08:02:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec4e67700 1 -- 172.21.15.111:0/2909181137 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb000b810 con 0x7f2ec006de00 2023-12-19T08:02:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec4e67700 1 -- 172.21.15.111:0/2909181137 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000bbb0 con 0x7f2ec006de00 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/2909181137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 msgr2=0x7f2ec006b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/2909181137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec006b880 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00c8390 tx=0x7f2eb00091a0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/2909181137 shutdown_connections 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/2909181137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec006b880 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/2909181137 >> v2:172.21.15.143:3300/0 conn(0x7f2ec00c69a0 0x7f2ec00c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/2909181137 >> 172.21.15.111:0/2909181137 conn(0x7f2ec0067b50 msgr2=0x7f2ec00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:02:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/2909181137 shutdown_connections 2023-12-19T08:02:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.913+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/2909181137 wait complete. 2023-12-19T08:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 Processor -- start 2023-12-19T08:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 -- start start 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 0x7f2ec00b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec00c69a0 0x7f2ec01507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00c08d0 con 0x7f2ec006de00 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0171340 con 0x7f2ec00c69a0 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec6e6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0171620 con 0x7f2ec006c7c0 2023-12-19T08:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53206/0 (socket says 172.21.15.111:53206) 2023-12-19T08:02:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 -- 172.21.15.111:0/3220312440 learned_addr learned my addr 172.21.15.111:0/3220312440 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec666a700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec00c69a0 0x7f2ec01507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5e69700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 0x7f2ec00b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 -- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 msgr2=0x7f2ec00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 0x7f2ec00b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 -- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec00c69a0 msgr2=0x7f2ec01507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec00c69a0 0x7f2ec01507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.917+0000 7f2ec5668700 1 -- 172.21.15.111:0/3220312440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0009420 con 0x7f2ec006de00 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2ec5e69700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 0x7f2ec00b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2ec5668700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006a0d0 tx=0x7f2eb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000bb90 con 0x7f2ec006de00 2023-12-19T08:02:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0011420 con 0x7f2ec006de00 2023-12-19T08:02:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb00117c0 con 0x7f2ec006de00 2023-12-19T08:02:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec0171900 con 0x7f2ec006de00 2023-12-19T08:02:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec00b15e0 con 0x7f2ec006de00 2023-12-19T08:02:59.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.921+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec00c0a40 con 0x7f2ec006de00 2023-12-19T08:02:59.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.925+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2eb0011bf0 con 0x7f2ec006de00 2023-12-19T08:02:59.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.925+0000 7f2eb5ffb700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:02:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.925+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f2eb007e630 con 0x7f2ec006de00 2023-12-19T08:02:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.925+0000 7f2ec5e69700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:02:59.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.929+0000 7f2ec5e69700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc006fd0 tx=0x7f2ebc008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:02:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:02:59.933+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2eb004e7a0 con 0x7f2ec006de00 2023-12-19T08:03:00.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.145+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2ec004be40 con 0x7f2ec006de00 2023-12-19T08:03:00.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.953+0000 7f2eb5ffb700 1 -- 172.21.15.111:0/3220312440 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2eb0051d40 con 0x7f2ec006de00 2023-12-19T08:03:00.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 msgr2=0x7f2ea4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc006fd0 tx=0x7f2ebc008040 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 msgr2=0x7f2ec00c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006a0d0 tx=0x7f2eb0017000 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 shutdown_connections 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec00c69a0 0x7f2ec01507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec006de00 0x7f2ec00c5190 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 --2- 172.21.15.111:0/3220312440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006c7c0 0x7f2ec00b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 >> 172.21.15.111:0/3220312440 conn(0x7f2ec0067b50 msgr2=0x7f2ec00c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:00.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 shutdown_connections 2023-12-19T08:03:00.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:00.957+0000 7f2ec6e6b700 1 -- 172.21.15.111:0/3220312440 wait complete. 2023-12-19T08:03:00.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:03:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.157+0000 7fd75a8f0700 1 Processor -- start 2023-12-19T08:03:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.157+0000 7fd75a8f0700 1 -- start start 2023-12-19T08:03:01.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd75a8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd7540c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd75a8f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd75406c7c0 0x7fd75406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd75a8f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75406a7a0 con 0x7fd75415aa30 2023-12-19T08:03:01.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd75a8f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7540c7630 con 0x7fd75406c7c0 2023-12-19T08:03:01.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd75a8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75406d4b0 con 0x7fd7540c6d20 2023-12-19T08:03:01.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd758e8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd75415aa30 0x7fd75415ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34210/0 (socket says 172.21.15.111:34210) 2023-12-19T08:03:01.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd75406c7c0 0x7fd75406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd758e8d700 1 -- 172.21.15.111:0/2234815946 learned_addr learned my addr 172.21.15.111:0/2234815946 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:01.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd753fff700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd7540c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd7537fe700 1 -- 172.21.15.111:0/2234815946 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429961759 0 0) 0x7fd75406a7a0 con 0x7fd75415aa30 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd7537fe700 1 -- 172.21.15.111:0/2234815946 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003570 con 0x7fd75415aa30 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 -- 172.21.15.111:0/2234815946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 msgr2=0x7fd7540c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd7540c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 -- 172.21.15.111:0/2234815946 >> v1:172.21.15.111:6789/0 conn(0x7fd75415aa30 legacy=0x7fd75415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 -- 172.21.15.111:0/2234815946 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75406d790 con 0x7fd75406c7c0 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd753fff700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd7540c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:01.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.161+0000 7fd74b7fe700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75406c7c0 0x7fd75406cc30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0098e0 tx=0x7fd73c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2e8b3982b566e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:01.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd7537fe700 1 -- 172.21.15.111:0/2234815946 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c011630 con 0x7fd75406c7c0 2023-12-19T08:03:01.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd7537fe700 1 -- 172.21.15.111:0/2234815946 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c011790 con 0x7fd75406c7c0 2023-12-19T08:03:01.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd7537fe700 1 -- 172.21.15.111:0/2234815946 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c011b30 con 0x7fd75406c7c0 2023-12-19T08:03:01.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- 172.21.15.111:0/2234815946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75406c7c0 msgr2=0x7fd75406cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75406c7c0 0x7fd75406cc30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0098e0 tx=0x7fd73c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- 172.21.15.111:0/2234815946 shutdown_connections 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75406c7c0 0x7fd75406cc30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/2234815946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd7540c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- 172.21.15.111:0/2234815946 >> 172.21.15.111:0/2234815946 conn(0x7fd754067bb0 msgr2=0x7fd7540cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- 172.21.15.111:0/2234815946 shutdown_connections 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- 172.21.15.111:0/2234815946 wait complete. 2023-12-19T08:03:01.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 Processor -- start 2023-12-19T08:03:01.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.165+0000 7fd75a8f0700 1 -- start start 2023-12-19T08:03:01.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd75406c7c0 0x7fd7540aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd75416a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540ace10 con 0x7fd75415aa30 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75416a560 con 0x7fd75406c7c0 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540aecd0 con 0x7fd7540c6d20 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53234/0 (socket says 172.21.15.111:53234) 2023-12-19T08:03:01.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 -- 172.21.15.111:0/445207844 learned_addr learned my addr 172.21.15.111:0/445207844 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:01.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd753fff700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd75406c7c0 0x7fd7540aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd74b7fe700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd75416a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 -- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 msgr2=0x7fd75416a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd75416a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 -- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd75406c7c0 msgr2=0x7fd7540aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd75406c7c0 0x7fd7540aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 -- 172.21.15.111:0/445207844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009420 con 0x7fd75415aa30 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd758e8d700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00cc40 tx=0x7fd74c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c00a940 con 0x7fd75415aa30 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c00aaa0 con 0x7fd75415aa30 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c009420 con 0x7fd75415aa30 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c00ae20 con 0x7fd75415aa30 2023-12-19T08:03:01.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.169+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7540ad060 con 0x7fd75415aa30 2023-12-19T08:03:01.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.173+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd754071120 con 0x7fd75415aa30 2023-12-19T08:03:01.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.173+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd74c007300 con 0x7fd75415aa30 2023-12-19T08:03:01.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.177+0000 7fd750ff9700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 0x7fd734062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:01.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.177+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fd74c07d120 con 0x7fd75415aa30 2023-12-19T08:03:01.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.177+0000 7fd753fff700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 0x7fd734062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:01.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.177+0000 7fd753fff700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 0x7fd734062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7440093f0 tx=0x7fd744008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:01.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.181+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd74c04d290 con 0x7fd75415aa30 2023-12-19T08:03:01.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.389+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd7540ae6f0 con 0x7fd75415aa30 2023-12-19T08:03:01.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.965+0000 7fd750ff9700 1 -- 172.21.15.111:0/445207844 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd74c0508b0 con 0x7fd75415aa30 2023-12-19T08:03:01.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 msgr2=0x7fd734062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 0x7fd734062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7440093f0 tx=0x7fd744008040 comp rx=0 tx=0).stop 2023-12-19T08:03:01.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 msgr2=0x7fd7540ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00cc40 tx=0x7fd74c001a40 comp rx=0 tx=0).stop 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 shutdown_connections 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd734060670 0x7fd734062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd75415aa30 0x7fd7540ad860 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7540c6d20 0x7fd75416a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 --2- 172.21.15.111:0/445207844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd75406c7c0 0x7fd7540aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 >> 172.21.15.111:0/445207844 conn(0x7fd754067bb0 msgr2=0x7fd7540c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 shutdown_connections 2023-12-19T08:03:01.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:01.969+0000 7fd75a8f0700 1 -- 172.21.15.111:0/445207844 wait complete. 2023-12-19T08:03:01.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:02.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 Processor -- start 2023-12-19T08:03:02.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 -- start start 2023-12-19T08:03:02.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e8c06a540 0x7f8e8c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0cc470 0x7f8e8c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c0ca5a0 con 0x7f8e8c158e70 2023-12-19T08:03:02.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c06dac0 con 0x7f8e8c06a540 2023-12-19T08:03:02.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e915c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c06d220 con 0x7f8e8c0cc470 2023-12-19T08:03:02.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e8c06a540 0x7f8e8c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e8c06a540 0x7f8e8c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37316/0 (socket says 172.21.15.111:37316) 2023-12-19T08:03:02.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e837fe700 1 -- 172.21.15.111:0/1210740377 learned_addr learned my addr 172.21.15.111:0/1210740377 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:02.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.165+0000 7f8e90dc2700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0cc470 0x7f8e8c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e837fe700 1 -- 172.21.15.111:0/1210740377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0cc470 msgr2=0x7f8e8c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:02.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e837fe700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0cc470 0x7f8e8c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e837fe700 1 -- 172.21.15.111:0/1210740377 >> v1:172.21.15.143:6789/0 conn(0x7f8e8c158e70 legacy=0x7f8e8c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e837fe700 1 -- 172.21.15.111:0/1210740377 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c06d500 con 0x7f8e8c06a540 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e837fe700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c06a540 0x7f8e8c06c9d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8e740099b0 tx=0x7f8e74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea34d9d0bef8d4d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e8b7fe700 1 -- 172.21.15.111:0/1210740377 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74009ea0 con 0x7f8e8c06a540 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e8b7fe700 1 -- 172.21.15.111:0/1210740377 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7400b860 con 0x7f8e8c06a540 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e8b7fe700 1 -- 172.21.15.111:0/1210740377 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400bc20 con 0x7f8e8c06a540 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 -- 172.21.15.111:0/1210740377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c06a540 msgr2=0x7f8e8c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c06a540 0x7f8e8c06c9d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8e740099b0 tx=0x7f8e74009250 comp rx=0 tx=0).stop 2023-12-19T08:03:02.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 -- 172.21.15.111:0/1210740377 shutdown_connections 2023-12-19T08:03:02.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0cc470 0x7f8e8c0cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 --2- 172.21.15.111:0/1210740377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c06a540 0x7f8e8c06c9d0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 -- 172.21.15.111:0/1210740377 >> 172.21.15.111:0/1210740377 conn(0x7f8e8c0670e0 msgr2=0x7f8e8c0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:02.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 -- 172.21.15.111:0/1210740377 shutdown_connections 2023-12-19T08:03:02.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.169+0000 7f8e915c3700 1 -- 172.21.15.111:0/1210740377 wait complete. 2023-12-19T08:03:02.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 Processor -- start 2023-12-19T08:03:02.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 -- start start 2023-12-19T08:03:02.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 0x7f8e8c171ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c158e70 0x7f8e8c0c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1721e0 con 0x7f8e8c0cc470 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c16e0e0 con 0x7f8e8c06a540 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e915c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c16e3c0 con 0x7f8e8c158e70 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37328/0 (socket says 172.21.15.111:37328) 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 -- 172.21.15.111:0/2986731073 learned_addr learned my addr 172.21.15.111:0/2986731073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e8bfff700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 0x7f8e8c171ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e90dc2700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c158e70 0x7f8e8c0c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 -- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c158e70 msgr2=0x7f8e8c0c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c158e70 0x7f8e8c0c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 -- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 msgr2=0x7f8e8c171ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 0x7f8e8c171ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e837fe700 1 -- 172.21.15.111:0/2986731073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e740094f0 con 0x7f8e8c0cc470 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.173+0000 7f8e8bfff700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 0x7f8e8c171ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e837fe700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8e74006fd0 tx=0x7f8e74017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7401f070 con 0x7f8e8c0cc470 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74000bc0 con 0x7f8e8c0cc470 2023-12-19T08:03:02.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c16e6a0 con 0x7f8e8c0cc470 2023-12-19T08:03:02.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74025420 con 0x7f8e8c0cc470 2023-12-19T08:03:02.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c0b4060 con 0x7f8e8c0cc470 2023-12-19T08:03:02.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c0c3e20 con 0x7f8e8c0cc470 2023-12-19T08:03:02.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.177+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8e74023070 con 0x7f8e8c0cc470 2023-12-19T08:03:02.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.181+0000 7f8e88ff9700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 0x7f8e6c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.181+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f8e7407df80 con 0x7f8e8c0cc470 2023-12-19T08:03:02.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.181+0000 7f8e8bfff700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 0x7f8e6c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.181+0000 7f8e8bfff700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 0x7f8e6c062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c007fd0 tx=0x7f8e7c009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.185+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e7404e0f0 con 0x7f8e8c0cc470 2023-12-19T08:03:02.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.393+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8c04be40 con 0x7f8e8c0cc470 2023-12-19T08:03:02.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e88ff9700 1 -- 172.21.15.111:0/2986731073 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f8e74015090 con 0x7f8e8c0cc470 2023-12-19T08:03:02.398 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:02.399 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:02.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 msgr2=0x7f8e6c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 0x7f8e6c062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c007fd0 tx=0x7f8e7c009040 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 msgr2=0x7f8e8c157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8e74006fd0 tx=0x7f8e74017000 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 shutdown_connections 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e6c0606a0 0x7f8e6c062b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c158e70 0x7f8e8c0c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e8c0cc470 0x7f8e8c157140 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 --2- 172.21.15.111:0/2986731073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c06a540 0x7f8e8c171ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.397+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 >> 172.21.15.111:0/2986731073 conn(0x7f8e8c0670e0 msgr2=0x7f8e8c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.401+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 shutdown_connections 2023-12-19T08:03:02.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.401+0000 7f8e915c3700 1 -- 172.21.15.111:0/2986731073 wait complete. 2023-12-19T08:03:02.404 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:02.421 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.569+0000 7fec5628e700 1 Processor -- start 2023-12-19T08:03:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.569+0000 7fec5628e700 1 -- start start 2023-12-19T08:03:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5628e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec500b1410 0x7fec500ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5628e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5006c7c0 con 0x7fec5006aa00 2023-12-19T08:03:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5628e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5006cd80 con 0x7fec5006a580 2023-12-19T08:03:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5628e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5015b680 con 0x7fec500b1410 2023-12-19T08:03:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec55a8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fec5006aa00 0x7fec5015af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59786/0 (socket says 172.21.15.111:59786) 2023-12-19T08:03:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec55a8d700 1 -- 172.21.15.111:0/3160494137 learned_addr learned my addr 172.21.15.111:0/3160494137 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5528c700 1 --2- 172.21.15.111:0/3160494137 >> v2:172.21.15.111:3300/0 conn(0x7fec500b1410 0x7fec500ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428955902 0 0) 0x7fec5006c7c0 con 0x7fec5006aa00 2023-12-19T08:03:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3c003570 con 0x7fec5006aa00 2023-12-19T08:03:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5528c700 1 -- 172.21.15.111:0/3160494137 >> v1:172.21.15.143:6789/0 conn(0x7fec5006a580 legacy=0x7fec5015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194065104 0 0) 0x7fec5006cd80 con 0x7fec5006a580 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5528c700 1 -- 172.21.15.111:0/3160494137 >> v1:172.21.15.111:6789/0 conn(0x7fec5006aa00 legacy=0x7fec5015af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5528c700 1 -- 172.21.15.111:0/3160494137 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5006b8f0 con 0x7fec500b1410 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec5528c700 1 --2- 172.21.15.111:0/3160494137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec500b1410 0x7fec500ab440 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0098e0 tx=0x7fec4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7015356a4078ab80 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00b640 con 0x7fec500b1410 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c00b7a0 con 0x7fec500b1410 2023-12-19T08:03:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.573+0000 7fec47fff700 1 -- 172.21.15.111:0/3160494137 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00bb40 con 0x7fec500b1410 2023-12-19T08:03:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- 172.21.15.111:0/3160494137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec500b1410 msgr2=0x7fec500ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 --2- 172.21.15.111:0/3160494137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec500b1410 0x7fec500ab440 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0098e0 tx=0x7fec4c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- 172.21.15.111:0/3160494137 shutdown_connections 2023-12-19T08:03:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 --2- 172.21.15.111:0/3160494137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec500b1410 0x7fec500ab440 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- 172.21.15.111:0/3160494137 >> 172.21.15.111:0/3160494137 conn(0x7fec50067bb0 msgr2=0x7fec500cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- 172.21.15.111:0/3160494137 shutdown_connections 2023-12-19T08:03:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- 172.21.15.111:0/3160494137 wait complete. 2023-12-19T08:03:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 Processor -- start 2023-12-19T08:03:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.577+0000 7fec5628e700 1 -- start start 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec5006a580 0x7fec5015c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 0x7fec50158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5015d150 con 0x7fec5006aa00 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5015d430 con 0x7fec500b1410 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5015d710 con 0x7fec5006a580 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37336/0 (socket says 172.21.15.111:37336) 2023-12-19T08:03:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 -- 172.21.15.111:0/846454809 learned_addr learned my addr 172.21.15.111:0/846454809 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5528c700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec5006a580 0x7fec5015c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec55a8d700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 0x7fec50158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 -- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec5006a580 msgr2=0x7fec5015c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec5006a580 0x7fec5015c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 -- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 msgr2=0x7fec50158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 0x7fec50158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 -- 172.21.15.111:0/846454809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c009420 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec55a8d700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 0x7fec50158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec54a8b700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fec4000c7e0 tx=0x7fec40007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40012700 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec40012880 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5015d9f0 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40012c40 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5015ce40 con 0x7fec5006aa00 2023-12-19T08:03:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.581+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5015def0 con 0x7fec5006aa00 2023-12-19T08:03:02.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.585+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fec4001b400 con 0x7fec5006aa00 2023-12-19T08:03:02.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.585+0000 7fec457fa700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 0x7fec34062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.585+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fec4007c900 con 0x7fec5006aa00 2023-12-19T08:03:02.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.585+0000 7fec5528c700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 0x7fec34062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.589+0000 7fec5528c700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 0x7fec34062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00a010 tx=0x7fec4c017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.589+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec4004c9f0 con 0x7fec5006aa00 2023-12-19T08:03:02.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.757+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec5004be40 con 0x7fec5006aa00 2023-12-19T08:03:02.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.761+0000 7fec457fa700 1 -- 172.21.15.111:0/846454809 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16395 (secure 0 0 0) 0x7fec40050010 con 0x7fec5006aa00 2023-12-19T08:03:02.764 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:02.764 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":99,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:02:59.441190+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.761+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 msgr2=0x7fec34062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.761+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 0x7fec34062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00a010 tx=0x7fec4c017000 comp rx=0 tx=0).stop 2023-12-19T08:03:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.761+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 msgr2=0x7fec500ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fec4000c7e0 tx=0x7fec40007df0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 shutdown_connections 2023-12-19T08:03:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec34060670 0x7fec34062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec500b1410 0x7fec50158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5006aa00 0x7fec500ab670 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 --2- 172.21.15.111:0/846454809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec5006a580 0x7fec5015c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 >> 172.21.15.111:0/846454809 conn(0x7fec50067bb0 msgr2=0x7fec500c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 shutdown_connections 2023-12-19T08:03:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.765+0000 7fec5628e700 1 -- 172.21.15.111:0/846454809 wait complete. 2023-12-19T08:03:02.787 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:03:02.787 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:03:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.945+0000 7f2ec646b700 1 Processor -- start 2023-12-19T08:03:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.945+0000 7f2ec646b700 1 -- start start 2023-12-19T08:03:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec646b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec0158df0 0x7f2ec006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec646b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ec006d590 0x7f2ec01591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec646b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec0159f90 con 0x7f2ec0159700 2023-12-19T08:03:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec646b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec006da10 con 0x7f2ec0158df0 2023-12-19T08:03:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec646b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec015a270 con 0x7f2ec006d590 2023-12-19T08:03:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec0158df0 0x7f2ec006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ec0158df0 0x7f2ec006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37352/0 (socket says 172.21.15.111:37352) 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 -- 172.21.15.111:0/1671290581 learned_addr learned my addr 172.21.15.111:0/1671290581 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec4c68700 1 --2- 172.21.15.111:0/1671290581 >> v2:172.21.15.143:3300/0 conn(0x7f2ec006d590 0x7f2ec01591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 -- 172.21.15.111:0/1671290581 >> v2:172.21.15.143:3300/0 conn(0x7f2ec006d590 msgr2=0x7f2ec01591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 --2- 172.21.15.111:0/1671290581 >> v2:172.21.15.143:3300/0 conn(0x7f2ec006d590 0x7f2ec01591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 -- 172.21.15.111:0/1671290581 >> v1:172.21.15.143:6789/0 conn(0x7f2ec0159700 legacy=0x7f2ec015ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 -- 172.21.15.111:0/1671290581 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec00c7930 con 0x7f2ec0158df0 2023-12-19T08:03:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2ec5469700 1 --2- 172.21.15.111:0/1671290581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec006d050 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006b3e0 tx=0x7f2ebc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f067eae2ab3f95b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2eb7fff700 1 -- 172.21.15.111:0/1671290581 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00b860 con 0x7f2ec0158df0 2023-12-19T08:03:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2eb7fff700 1 -- 172.21.15.111:0/1671290581 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc00b9c0 con 0x7f2ec0158df0 2023-12-19T08:03:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.949+0000 7f2eb7fff700 1 -- 172.21.15.111:0/1671290581 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc00bd80 con 0x7f2ec0158df0 2023-12-19T08:03:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- 172.21.15.111:0/1671290581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 msgr2=0x7f2ec006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 --2- 172.21.15.111:0/1671290581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec006d050 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006b3e0 tx=0x7f2ebc009250 comp rx=0 tx=0).stop 2023-12-19T08:03:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- 172.21.15.111:0/1671290581 shutdown_connections 2023-12-19T08:03:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 --2- 172.21.15.111:0/1671290581 >> v2:172.21.15.143:3300/0 conn(0x7f2ec006d590 0x7f2ec01591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 --2- 172.21.15.111:0/1671290581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec006d050 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- 172.21.15.111:0/1671290581 >> 172.21.15.111:0/1671290581 conn(0x7f2ec006c080 msgr2=0x7f2ec00ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- 172.21.15.111:0/1671290581 shutdown_connections 2023-12-19T08:03:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- 172.21.15.111:0/1671290581 wait complete. 2023-12-19T08:03:02.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 Processor -- start 2023-12-19T08:03:02.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.953+0000 7f2ec646b700 1 -- start start 2023-12-19T08:03:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 0x7f2ec014ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 0x7f2ec00b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec01697c0 con 0x7f2ec0158df0 2023-12-19T08:03:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00aa300 con 0x7f2ec0159700 2023-12-19T08:03:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00b18a0 con 0x7f2ec006d590 2023-12-19T08:03:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37368/0 (socket says 172.21.15.111:37368) 2023-12-19T08:03:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 -- 172.21.15.111:0/330814538 learned_addr learned my addr 172.21.15.111:0/330814538 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec5469700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 0x7f2ec014ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec5c6a700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 0x7f2ec00b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 -- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 msgr2=0x7f2ec014ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 0x7f2ec014ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 -- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 msgr2=0x7f2ec00b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 0x7f2ec00b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 -- 172.21.15.111:0/330814538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0094f0 con 0x7f2ec0158df0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec5469700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 0x7f2ec014ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec5c6a700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 0x7f2ec00b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec4c68700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000d860 tx=0x7f2eb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0009610 con 0x7f2ec0158df0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0009770 con 0x7f2ec0158df0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec00b15c0 con 0x7f2ec0158df0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0009b30 con 0x7f2ec0158df0 2023-12-19T08:03:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.957+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec00a9f90 con 0x7f2ec0158df0 2023-12-19T08:03:02.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.961+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec00c7020 con 0x7f2ec0158df0 2023-12-19T08:03:02.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.961+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2eb001e460 con 0x7f2ec0158df0 2023-12-19T08:03:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.965+0000 7f2eb57fa700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.965+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f2eb007d350 con 0x7f2ec0158df0 2023-12-19T08:03:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.965+0000 7f2ec5469700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:02.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.965+0000 7f2ec5469700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006b3e0 tx=0x7f2ebc017000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:02.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:02.969+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ec00c7020 con 0x7f2ec0158df0 2023-12-19T08:03:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.133+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ec00c7020 con 0x7f2ea4060670 2023-12-19T08:03:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.153+0000 7f2eb57fa700 1 -- 172.21.15.111:0/330814538 <== mgr.6044 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287145 (secure 0 0 0) 0x7f2ec00c7020 con 0x7f2ea4060670 2023-12-19T08:03:03.159 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 msgr2=0x7f2ea4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2ec006b3e0 tx=0x7f2ebc017000 comp rx=0 tx=0).stop 2023-12-19T08:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 msgr2=0x7f2ec00b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000d860 tx=0x7f2eb0008040 comp rx=0 tx=0).stop 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 shutdown_connections 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2ea4060670 0x7f2ea4062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ec0159700 0x7f2ec00b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ec0158df0 0x7f2ec00b0a00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 --2- 172.21.15.111:0/330814538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ec006d590 0x7f2ec014ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 >> 172.21.15.111:0/330814538 conn(0x7f2ec006c080 msgr2=0x7f2ec01585e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 shutdown_connections 2023-12-19T08:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.161+0000 7f2ec646b700 1 -- 172.21.15.111:0/330814538 wait complete. 2023-12-19T08:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:03:03.185 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:03:02.628370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2434016,"kb_used_data":10656,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752540704,"statfs":{"total":773094113280,"available":770601680896,"internally_reserved":0,"allocated":10911744,"data_stored":6147849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.722046"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485395+0000","last_change":"2023-12-19T08:02:58.482752+0000","last_active":"2023-12-19T08:02:58.485395+0000","last_peered":"2023-12-19T08:02:58.485395+0000","last_clean":"2023-12-19T08:02:58.485395+0000","last_became_active":"2023-12-19T08:02:58.482462+0000","last_became_peered":"2023-12-19T08:02:58.482462+0000","last_unstale":"2023-12-19T08:02:58.485395+0000","last_undegraded":"2023-12-19T08:02:58.485395+0000","last_fullsized":"2023-12-19T08:02:58.485395+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:56:39.326794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484669+0000","last_change":"2023-12-19T08:02:58.479640+0000","last_active":"2023-12-19T08:02:58.484669+0000","last_peered":"2023-12-19T08:02:58.484669+0000","last_clean":"2023-12-19T08:02:58.484669+0000","last_became_active":"2023-12-19T08:02:58.479292+0000","last_became_peered":"2023-12-19T08:02:58.479292+0000","last_unstale":"2023-12-19T08:02:58.484669+0000","last_undegraded":"2023-12-19T08:02:58.484669+0000","last_fullsized":"2023-12-19T08:02:58.484669+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:35.015348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482162+0000","last_change":"2023-12-19T08:02:58.477499+0000","last_active":"2023-12-19T08:02:58.482162+0000","last_peered":"2023-12-19T08:02:58.482162+0000","last_clean":"2023-12-19T08:02:58.482162+0000","last_became_active":"2023-12-19T08:02:58.477069+0000","last_became_peered":"2023-12-19T08:02:58.477069+0000","last_unstale":"2023-12-19T08:02:58.482162+0000","last_undegraded":"2023-12-19T08:02:58.482162+0000","last_fullsized":"2023-12-19T08:02:58.482162+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.481262+0000","last_change":"2023-12-19T08:02:58.478609+0000","last_active":"2023-12-19T08:02:58.481262+0000","last_peered":"2023-12-19T08:02:58.481262+0000","last_clean":"2023-12-19T08:02:58.481262+0000","last_became_active":"2023-12-19T08:02:58.478259+0000","last_became_peered":"2023-12-19T08:02:58.478259+0000","last_unstale":"2023-12-19T08:02:58.481262+0000","last_undegraded":"2023-12-19T08:02:58.481262+0000","last_fullsized":"2023-12-19T08:02:58.481262+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:28.600644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484522+0000","last_change":"2023-12-19T08:02:58.480655+0000","last_active":"2023-12-19T08:02:58.484522+0000","last_peered":"2023-12-19T08:02:58.484522+0000","last_clean":"2023-12-19T08:02:58.484522+0000","last_became_active":"2023-12-19T08:02:58.478102+0000","last_became_peered":"2023-12-19T08:02:58.478102+0000","last_unstale":"2023-12-19T08:02:58.484522+0000","last_undegraded":"2023-12-19T08:02:58.484522+0000","last_fullsized":"2023-12-19T08:02:58.484522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:22.956703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465121+0000","last_change":"2023-12-19T08:02:58.460614+0000","last_active":"2023-12-19T08:02:58.465121+0000","last_peered":"2023-12-19T08:02:58.465121+0000","last_clean":"2023-12-19T08:02:58.465121+0000","last_became_active":"2023-12-19T08:02:58.459436+0000","last_became_peered":"2023-12-19T08:02:58.459436+0000","last_unstale":"2023-12-19T08:02:58.465121+0000","last_undegraded":"2023-12-19T08:02:58.465121+0000","last_fullsized":"2023-12-19T08:02:58.465121+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:19:29.498138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489076+0000","last_change":"2023-12-19T08:02:58.483542+0000","last_active":"2023-12-19T08:02:58.489076+0000","last_peered":"2023-12-19T08:02:58.489076+0000","last_clean":"2023-12-19T08:02:58.489076+0000","last_became_active":"2023-12-19T08:02:58.483316+0000","last_became_peered":"2023-12-19T08:02:58.483316+0000","last_unstale":"2023-12-19T08:02:58.489076+0000","last_undegraded":"2023-12-19T08:02:58.489076+0000","last_fullsized":"2023-12-19T08:02:58.489076+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482750+0000","last_change":"2023-12-19T08:02:58.471595+0000","last_active":"2023-12-19T08:02:58.482750+0000","last_peered":"2023-12-19T08:02:58.482750+0000","last_clean":"2023-12-19T08:02:58.482750+0000","last_became_active":"2023-12-19T08:02:58.470111+0000","last_became_peered":"2023-12-19T08:02:58.470111+0000","last_unstale":"2023-12-19T08:02:58.482750+0000","last_undegraded":"2023-12-19T08:02:58.482750+0000","last_fullsized":"2023-12-19T08:02:58.482750+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:25.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.487150+0000","last_change":"2023-12-19T08:02:58.477070+0000","last_active":"2023-12-19T08:02:58.487150+0000","last_peered":"2023-12-19T08:02:58.487150+0000","last_clean":"2023-12-19T08:02:58.487150+0000","last_became_active":"2023-12-19T08:02:58.476556+0000","last_became_peered":"2023-12-19T08:02:58.476556+0000","last_unstale":"2023-12-19T08:02:58.487150+0000","last_undegraded":"2023-12-19T08:02:58.487150+0000","last_fullsized":"2023-12-19T08:02:58.487150+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486632+0000","last_change":"2023-12-19T08:02:58.481446+0000","last_active":"2023-12-19T08:02:58.486632+0000","last_peered":"2023-12-19T08:02:58.486632+0000","last_clean":"2023-12-19T08:02:58.486632+0000","last_became_active":"2023-12-19T08:02:58.481121+0000","last_became_peered":"2023-12-19T08:02:58.481121+0000","last_unstale":"2023-12-19T08:02:58.486632+0000","last_undegraded":"2023-12-19T08:02:58.486632+0000","last_fullsized":"2023-12-19T08:02:58.486632+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:10.842548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484740+0000","last_change":"2023-12-19T08:02:58.482213+0000","last_active":"2023-12-19T08:02:58.484740+0000","last_peered":"2023-12-19T08:02:58.484740+0000","last_clean":"2023-12-19T08:02:58.484740+0000","last_became_active":"2023-12-19T08:02:58.481724+0000","last_became_peered":"2023-12-19T08:02:58.481724+0000","last_unstale":"2023-12-19T08:02:58.484740+0000","last_undegraded":"2023-12-19T08:02:58.484740+0000","last_fullsized":"2023-12-19T08:02:58.484740+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:46.500728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486899+0000","last_change":"2023-12-19T08:02:58.482020+0000","last_active":"2023-12-19T08:02:58.486899+0000","last_peered":"2023-12-19T08:02:58.486899+0000","last_clean":"2023-12-19T08:02:58.486899+0000","last_became_active":"2023-12-19T08:02:58.481724+0000","last_became_peered":"2023-12-19T08:02:58.481724+0000","last_unstale":"2023-12-19T08:02:58.486899+0000","last_undegraded":"2023-12-19T08:02:58.486899+0000","last_fullsized":"2023-12-19T08:02:58.486899+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:23:46.634962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482886+0000","last_change":"2023-12-19T08:02:58.470936+0000","last_active":"2023-12-19T08:02:58.482886+0000","last_peered":"2023-12-19T08:02:58.482886+0000","last_clean":"2023-12-19T08:02:58.482886+0000","last_became_active":"2023-12-19T08:02:58.470619+0000","last_became_peered":"2023-12-19T08:02:58.470619+0000","last_unstale":"2023-12-19T08:02:58.482886+0000","last_undegraded":"2023-12-19T08:02:58.482886+0000","last_fullsized":"2023-12-19T08:02:58.482886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:30:47.840369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465024+0000","last_change":"2023-12-19T08:02:58.460303+0000","last_active":"2023-12-19T08:02:58.465024+0000","last_peered":"2023-12-19T08:02:58.465024+0000","last_clean":"2023-12-19T08:02:58.465024+0000","last_became_active":"2023-12-19T08:02:58.459037+0000","last_became_peered":"2023-12-19T08:02:58.459037+0000","last_unstale":"2023-12-19T08:02:58.465024+0000","last_undegraded":"2023-12-19T08:02:58.465024+0000","last_fullsized":"2023-12-19T08:02:58.465024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:13.416628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477088+0000","last_change":"2023-12-19T08:02:58.473780+0000","last_active":"2023-12-19T08:02:58.477088+0000","last_peered":"2023-12-19T08:02:58.477088+0000","last_clean":"2023-12-19T08:02:58.477088+0000","last_became_active":"2023-12-19T08:02:58.472763+0000","last_became_peered":"2023-12-19T08:02:58.472763+0000","last_unstale":"2023-12-19T08:02:58.477088+0000","last_undegraded":"2023-12-19T08:02:58.477088+0000","last_fullsized":"2023-12-19T08:02:58.477088+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:31.945833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476106+0000","last_change":"2023-12-19T08:02:58.474326+0000","last_active":"2023-12-19T08:02:58.476106+0000","last_peered":"2023-12-19T08:02:58.476106+0000","last_clean":"2023-12-19T08:02:58.476106+0000","last_became_active":"2023-12-19T08:02:58.473988+0000","last_became_peered":"2023-12-19T08:02:58.473988+0000","last_unstale":"2023-12-19T08:02:58.476106+0000","last_undegraded":"2023-12-19T08:02:58.476106+0000","last_fullsized":"2023-12-19T08:02:58.476106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:28.600644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485351+0000","last_change":"2023-12-19T08:02:58.482865+0000","last_active":"2023-12-19T08:02:58.485351+0000","last_peered":"2023-12-19T08:02:58.485351+0000","last_clean":"2023-12-19T08:02:58.485351+0000","last_became_active":"2023-12-19T08:02:58.482532+0000","last_became_peered":"2023-12-19T08:02:58.482532+0000","last_unstale":"2023-12-19T08:02:58.485351+0000","last_undegraded":"2023-12-19T08:02:58.485351+0000","last_fullsized":"2023-12-19T08:02:58.485351+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:37:41.128971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474296+0000","last_change":"2023-12-19T08:02:58.471113+0000","last_active":"2023-12-19T08:02:58.474296+0000","last_peered":"2023-12-19T08:02:58.474296+0000","last_clean":"2023-12-19T08:02:58.474296+0000","last_became_active":"2023-12-19T08:02:58.470814+0000","last_became_peered":"2023-12-19T08:02:58.470814+0000","last_unstale":"2023-12-19T08:02:58.474296+0000","last_undegraded":"2023-12-19T08:02:58.474296+0000","last_fullsized":"2023-12-19T08:02:58.474296+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482595+0000","last_change":"2023-12-19T08:02:58.481385+0000","last_active":"2023-12-19T08:02:58.482595+0000","last_peered":"2023-12-19T08:02:58.482595+0000","last_clean":"2023-12-19T08:02:58.482595+0000","last_became_active":"2023-12-19T08:02:58.481112+0000","last_became_peered":"2023-12-19T08:02:58.481112+0000","last_unstale":"2023-12-19T08:02:58.482595+0000","last_undegraded":"2023-12-19T08:02:58.482595+0000","last_fullsized":"2023-12-19T08:02:58.482595+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:57.413537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482615+0000","last_change":"2023-12-19T08:02:58.469795+0000","last_active":"2023-12-19T08:02:58.482615+0000","last_peered":"2023-12-19T08:02:58.482615+0000","last_clean":"2023-12-19T08:02:58.482615+0000","last_became_active":"2023-12-19T08:02:58.469239+0000","last_became_peered":"2023-12-19T08:02:58.469239+0000","last_unstale":"2023-12-19T08:02:58.482615+0000","last_undegraded":"2023-12-19T08:02:58.482615+0000","last_fullsized":"2023-12-19T08:02:58.482615+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:30:56.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482114+0000","last_change":"2023-12-19T08:02:58.469700+0000","last_active":"2023-12-19T08:02:58.482114+0000","last_peered":"2023-12-19T08:02:58.482114+0000","last_clean":"2023-12-19T08:02:58.482114+0000","last_became_active":"2023-12-19T08:02:58.468388+0000","last_became_peered":"2023-12-19T08:02:58.468388+0000","last_unstale":"2023-12-19T08:02:58.482114+0000","last_undegraded":"2023-12-19T08:02:58.482114+0000","last_fullsized":"2023-12-19T08:02:58.482114+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482798+0000","last_change":"2023-12-19T08:02:58.467255+0000","last_active":"2023-12-19T08:02:58.482798+0000","last_peered":"2023-12-19T08:02:58.482798+0000","last_clean":"2023-12-19T08:02:58.482798+0000","last_became_active":"2023-12-19T08:02:58.466981+0000","last_became_peered":"2023-12-19T08:02:58.466981+0000","last_unstale":"2023-12-19T08:02:58.482798+0000","last_undegraded":"2023-12-19T08:02:58.482798+0000","last_fullsized":"2023-12-19T08:02:58.482798+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:44:56.890865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486898+0000","last_change":"2023-12-19T08:02:58.471044+0000","last_active":"2023-12-19T08:02:58.486898+0000","last_peered":"2023-12-19T08:02:58.486898+0000","last_clean":"2023-12-19T08:02:58.486898+0000","last_became_active":"2023-12-19T08:02:58.470725+0000","last_became_peered":"2023-12-19T08:02:58.470725+0000","last_unstale":"2023-12-19T08:02:58.486898+0000","last_undegraded":"2023-12-19T08:02:58.486898+0000","last_fullsized":"2023-12-19T08:02:58.486898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:39.728694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480160+0000","last_change":"2023-12-19T08:02:58.475283+0000","last_active":"2023-12-19T08:02:58.480160+0000","last_peered":"2023-12-19T08:02:58.480160+0000","last_clean":"2023-12-19T08:02:58.480160+0000","last_became_active":"2023-12-19T08:02:58.474995+0000","last_became_peered":"2023-12-19T08:02:58.474995+0000","last_unstale":"2023-12-19T08:02:58.480160+0000","last_undegraded":"2023-12-19T08:02:58.480160+0000","last_fullsized":"2023-12-19T08:02:58.480160+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:06.759558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480196+0000","last_change":"2023-12-19T08:02:58.475220+0000","last_active":"2023-12-19T08:02:58.480196+0000","last_peered":"2023-12-19T08:02:58.480196+0000","last_clean":"2023-12-19T08:02:58.480196+0000","last_became_active":"2023-12-19T08:02:58.474924+0000","last_became_peered":"2023-12-19T08:02:58.474924+0000","last_unstale":"2023-12-19T08:02:58.480196+0000","last_undegraded":"2023-12-19T08:02:58.480196+0000","last_fullsized":"2023-12-19T08:02:58.480196+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:18:29.983942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.487980+0000","last_change":"2023-12-19T08:02:58.486451+0000","last_active":"2023-12-19T08:02:58.487980+0000","last_peered":"2023-12-19T08:02:58.487980+0000","last_clean":"2023-12-19T08:02:58.487980+0000","last_became_active":"2023-12-19T08:02:58.486153+0000","last_became_peered":"2023-12-19T08:02:58.486153+0000","last_unstale":"2023-12-19T08:02:58.487980+0000","last_undegraded":"2023-12-19T08:02:58.487980+0000","last_fullsized":"2023-12-19T08:02:58.487980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:36.374176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476017+0000","last_change":"2023-12-19T08:02:58.474586+0000","last_active":"2023-12-19T08:02:58.476017+0000","last_peered":"2023-12-19T08:02:58.476017+0000","last_clean":"2023-12-19T08:02:58.476017+0000","last_became_active":"2023-12-19T08:02:58.474230+0000","last_became_peered":"2023-12-19T08:02:58.474230+0000","last_unstale":"2023-12-19T08:02:58.476017+0000","last_undegraded":"2023-12-19T08:02:58.476017+0000","last_fullsized":"2023-12-19T08:02:58.476017+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:58.172015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482645+0000","last_change":"2023-12-19T08:02:58.471706+0000","last_active":"2023-12-19T08:02:58.482645+0000","last_peered":"2023-12-19T08:02:58.482645+0000","last_clean":"2023-12-19T08:02:58.482645+0000","last_became_active":"2023-12-19T08:02:58.470015+0000","last_became_peered":"2023-12-19T08:02:58.470015+0000","last_unstale":"2023-12-19T08:02:58.482645+0000","last_undegraded":"2023-12-19T08:02:58.482645+0000","last_fullsized":"2023-12-19T08:02:58.482645+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:51:54.459798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480263+0000","last_change":"2023-12-19T08:02:58.476560+0000","last_active":"2023-12-19T08:02:58.480263+0000","last_peered":"2023-12-19T08:02:58.480263+0000","last_clean":"2023-12-19T08:02:58.480263+0000","last_became_active":"2023-12-19T08:02:58.476295+0000","last_became_peered":"2023-12-19T08:02:58.476295+0000","last_unstale":"2023-12-19T08:02:58.480263+0000","last_undegraded":"2023-12-19T08:02:58.480263+0000","last_fullsized":"2023-12-19T08:02:58.480263+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:07.275271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485205+0000","last_change":"2023-12-19T08:02:58.481489+0000","last_active":"2023-12-19T08:02:58.485205+0000","last_peered":"2023-12-19T08:02:58.485205+0000","last_clean":"2023-12-19T08:02:58.485205+0000","last_became_active":"2023-12-19T08:02:58.480625+0000","last_became_peered":"2023-12-19T08:02:58.480625+0000","last_unstale":"2023-12-19T08:02:58.485205+0000","last_undegraded":"2023-12-19T08:02:58.485205+0000","last_fullsized":"2023-12-19T08:02:58.485205+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:37.495292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482323+0000","last_change":"2023-12-19T08:02:58.478286+0000","last_active":"2023-12-19T08:02:58.482323+0000","last_peered":"2023-12-19T08:02:58.482323+0000","last_clean":"2023-12-19T08:02:58.482323+0000","last_became_active":"2023-12-19T08:02:58.476707+0000","last_became_peered":"2023-12-19T08:02:58.476707+0000","last_unstale":"2023-12-19T08:02:58.482323+0000","last_undegraded":"2023-12-19T08:02:58.482323+0000","last_fullsized":"2023-12-19T08:02:58.482323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485327+0000","last_change":"2023-12-19T08:02:58.482716+0000","last_active":"2023-12-19T08:02:58.485327+0000","last_peered":"2023-12-19T08:02:58.485327+0000","last_clean":"2023-12-19T08:02:58.485327+0000","last_became_active":"2023-12-19T08:02:58.482404+0000","last_became_peered":"2023-12-19T08:02:58.482404+0000","last_unstale":"2023-12-19T08:02:58.485327+0000","last_undegraded":"2023-12-19T08:02:58.485327+0000","last_fullsized":"2023-12-19T08:02:58.485327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:07.275271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474191+0000","last_change":"2023-12-19T08:02:58.471817+0000","last_active":"2023-12-19T08:02:58.474191+0000","last_peered":"2023-12-19T08:02:58.474191+0000","last_clean":"2023-12-19T08:02:58.474191+0000","last_became_active":"2023-12-19T08:02:58.470979+0000","last_became_peered":"2023-12-19T08:02:58.470979+0000","last_unstale":"2023-12-19T08:02:58.474191+0000","last_undegraded":"2023-12-19T08:02:58.474191+0000","last_fullsized":"2023-12-19T08:02:58.474191+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:40.194019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"98'4","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965820+0000","last_change":"2023-12-19T08:02:57.937367+0000","last_active":"2023-12-19T08:02:58.965820+0000","last_peered":"2023-12-19T08:02:58.965820+0000","last_clean":"2023-12-19T08:02:58.965820+0000","last_became_active":"2023-12-19T08:02:57.937097+0000","last_became_peered":"2023-12-19T08:02:57.937097+0000","last_unstale":"2023-12-19T08:02:58.965820+0000","last_undegraded":"2023-12-19T08:02:58.965820+0000","last_fullsized":"2023-12-19T08:02:58.965820+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:46.317912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.483081+0000","last_change":"2023-12-19T08:02:58.478025+0000","last_active":"2023-12-19T08:02:58.483081+0000","last_peered":"2023-12-19T08:02:58.483081+0000","last_clean":"2023-12-19T08:02:58.483081+0000","last_became_active":"2023-12-19T08:02:58.475371+0000","last_became_peered":"2023-12-19T08:02:58.475371+0000","last_unstale":"2023-12-19T08:02:58.483081+0000","last_undegraded":"2023-12-19T08:02:58.483081+0000","last_fullsized":"2023-12-19T08:02:58.483081+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:37.495292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957495+0000","last_change":"2023-12-19T08:02:57.946583+0000","last_active":"2023-12-19T08:02:58.957495+0000","last_peered":"2023-12-19T08:02:58.957495+0000","last_clean":"2023-12-19T08:02:58.957495+0000","last_became_active":"2023-12-19T08:02:57.946266+0000","last_became_peered":"2023-12-19T08:02:57.946266+0000","last_unstale":"2023-12-19T08:02:58.957495+0000","last_undegraded":"2023-12-19T08:02:58.957495+0000","last_fullsized":"2023-12-19T08:02:58.957495+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:32.960684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476236+0000","last_change":"2023-12-19T08:02:58.464646+0000","last_active":"2023-12-19T08:02:58.476236+0000","last_peered":"2023-12-19T08:02:58.476236+0000","last_clean":"2023-12-19T08:02:58.476236+0000","last_became_active":"2023-12-19T08:02:58.464350+0000","last_became_peered":"2023-12-19T08:02:58.464350+0000","last_unstale":"2023-12-19T08:02:58.476236+0000","last_undegraded":"2023-12-19T08:02:58.476236+0000","last_fullsized":"2023-12-19T08:02:58.476236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.958293+0000","last_change":"2023-12-19T08:02:57.943884+0000","last_active":"2023-12-19T08:02:58.958293+0000","last_peered":"2023-12-19T08:02:58.958293+0000","last_clean":"2023-12-19T08:02:58.958293+0000","last_became_active":"2023-12-19T08:02:57.943598+0000","last_became_peered":"2023-12-19T08:02:57.943598+0000","last_unstale":"2023-12-19T08:02:58.958293+0000","last_undegraded":"2023-12-19T08:02:58.958293+0000","last_fullsized":"2023-12-19T08:02:58.958293+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:42.218705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482138+0000","last_change":"2023-12-19T08:02:58.470325+0000","last_active":"2023-12-19T08:02:58.482138+0000","last_peered":"2023-12-19T08:02:58.482138+0000","last_clean":"2023-12-19T08:02:58.482138+0000","last_became_active":"2023-12-19T08:02:58.470011+0000","last_became_peered":"2023-12-19T08:02:58.470011+0000","last_unstale":"2023-12-19T08:02:58.482138+0000","last_undegraded":"2023-12-19T08:02:58.482138+0000","last_fullsized":"2023-12-19T08:02:58.482138+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.961597+0000","last_change":"2023-12-19T08:02:57.945609+0000","last_active":"2023-12-19T08:02:58.961597+0000","last_peered":"2023-12-19T08:02:58.961597+0000","last_clean":"2023-12-19T08:02:58.961597+0000","last_became_active":"2023-12-19T08:02:57.945323+0000","last_became_peered":"2023-12-19T08:02:57.945323+0000","last_unstale":"2023-12-19T08:02:58.961597+0000","last_undegraded":"2023-12-19T08:02:58.961597+0000","last_fullsized":"2023-12-19T08:02:58.961597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:14.153026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480007+0000","last_change":"2023-12-19T08:02:58.474935+0000","last_active":"2023-12-19T08:02:58.480007+0000","last_peered":"2023-12-19T08:02:58.480007+0000","last_clean":"2023-12-19T08:02:58.480007+0000","last_became_active":"2023-12-19T08:02:58.474616+0000","last_became_peered":"2023-12-19T08:02:58.474616+0000","last_unstale":"2023-12-19T08:02:58.480007+0000","last_undegraded":"2023-12-19T08:02:58.480007+0000","last_fullsized":"2023-12-19T08:02:58.480007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:21:48.436227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957229+0000","last_change":"2023-12-19T08:02:57.942738+0000","last_active":"2023-12-19T08:02:58.957229+0000","last_peered":"2023-12-19T08:02:58.957229+0000","last_clean":"2023-12-19T08:02:58.957229+0000","last_became_active":"2023-12-19T08:02:57.942114+0000","last_became_peered":"2023-12-19T08:02:57.942114+0000","last_unstale":"2023-12-19T08:02:58.957229+0000","last_undegraded":"2023-12-19T08:02:58.957229+0000","last_fullsized":"2023-12-19T08:02:58.957229+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:36.209726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484719+0000","last_change":"2023-12-19T08:02:58.482019+0000","last_active":"2023-12-19T08:02:58.484719+0000","last_peered":"2023-12-19T08:02:58.484719+0000","last_clean":"2023-12-19T08:02:58.484719+0000","last_became_active":"2023-12-19T08:02:58.481376+0000","last_became_peered":"2023-12-19T08:02:58.481376+0000","last_unstale":"2023-12-19T08:02:58.484719+0000","last_undegraded":"2023-12-19T08:02:58.484719+0000","last_fullsized":"2023-12-19T08:02:58.484719+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:46:58.190288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.958502+0000","last_change":"2023-12-19T08:02:57.947665+0000","last_active":"2023-12-19T08:02:58.958502+0000","last_peered":"2023-12-19T08:02:58.958502+0000","last_clean":"2023-12-19T08:02:58.958502+0000","last_became_active":"2023-12-19T08:02:57.947271+0000","last_became_peered":"2023-12-19T08:02:57.947271+0000","last_unstale":"2023-12-19T08:02:58.958502+0000","last_undegraded":"2023-12-19T08:02:58.958502+0000","last_fullsized":"2023-12-19T08:02:58.958502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:38.715762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482176+0000","last_change":"2023-12-19T08:02:58.470079+0000","last_active":"2023-12-19T08:02:58.482176+0000","last_peered":"2023-12-19T08:02:58.482176+0000","last_clean":"2023-12-19T08:02:58.482176+0000","last_became_active":"2023-12-19T08:02:58.469063+0000","last_became_peered":"2023-12-19T08:02:58.469063+0000","last_unstale":"2023-12-19T08:02:58.482176+0000","last_undegraded":"2023-12-19T08:02:58.482176+0000","last_fullsized":"2023-12-19T08:02:58.482176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465127+0000","last_change":"2023-12-19T08:02:57.932512+0000","last_active":"2023-12-19T08:02:58.465127+0000","last_peered":"2023-12-19T08:02:58.465127+0000","last_clean":"2023-12-19T08:02:58.465127+0000","last_became_active":"2023-12-19T08:02:57.932245+0000","last_became_peered":"2023-12-19T08:02:57.932245+0000","last_unstale":"2023-12-19T08:02:58.465127+0000","last_undegraded":"2023-12-19T08:02:58.465127+0000","last_fullsized":"2023-12-19T08:02:58.465127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489100+0000","last_change":"2023-12-19T08:02:58.484926+0000","last_active":"2023-12-19T08:02:58.489100+0000","last_peered":"2023-12-19T08:02:58.489100+0000","last_clean":"2023-12-19T08:02:58.489100+0000","last_became_active":"2023-12-19T08:02:58.484622+0000","last_became_peered":"2023-12-19T08:02:58.484622+0000","last_unstale":"2023-12-19T08:02:58.489100+0000","last_undegraded":"2023-12-19T08:02:58.489100+0000","last_fullsized":"2023-12-19T08:02:58.489100+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:35.696666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467545+0000","last_change":"2023-12-19T08:02:57.932729+0000","last_active":"2023-12-19T08:02:58.467545+0000","last_peered":"2023-12-19T08:02:58.467545+0000","last_clean":"2023-12-19T08:02:58.467545+0000","last_became_active":"2023-12-19T08:02:57.932355+0000","last_became_peered":"2023-12-19T08:02:57.932355+0000","last_unstale":"2023-12-19T08:02:58.467545+0000","last_undegraded":"2023-12-19T08:02:58.467545+0000","last_fullsized":"2023-12-19T08:02:58.467545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:24.900257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477601+0000","last_change":"2023-12-19T08:02:58.472746+0000","last_active":"2023-12-19T08:02:58.477601+0000","last_peered":"2023-12-19T08:02:58.477601+0000","last_clean":"2023-12-19T08:02:58.477601+0000","last_became_active":"2023-12-19T08:02:58.471165+0000","last_became_peered":"2023-12-19T08:02:58.471165+0000","last_unstale":"2023-12-19T08:02:58.477601+0000","last_undegraded":"2023-12-19T08:02:58.477601+0000","last_fullsized":"2023-12-19T08:02:58.477601+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:13.085870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464982+0000","last_change":"2023-12-19T08:02:57.947907+0000","last_active":"2023-12-19T08:02:58.464982+0000","last_peered":"2023-12-19T08:02:58.464982+0000","last_clean":"2023-12-19T08:02:58.464982+0000","last_became_active":"2023-12-19T08:02:57.947627+0000","last_became_peered":"2023-12-19T08:02:57.947627+0000","last_unstale":"2023-12-19T08:02:58.464982+0000","last_undegraded":"2023-12-19T08:02:58.464982+0000","last_fullsized":"2023-12-19T08:02:58.464982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:26.014801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486603+0000","last_change":"2023-12-19T08:02:58.483455+0000","last_active":"2023-12-19T08:02:58.486603+0000","last_peered":"2023-12-19T08:02:58.486603+0000","last_clean":"2023-12-19T08:02:58.486603+0000","last_became_active":"2023-12-19T08:02:58.481325+0000","last_became_peered":"2023-12-19T08:02:58.481325+0000","last_unstale":"2023-12-19T08:02:58.486603+0000","last_undegraded":"2023-12-19T08:02:58.486603+0000","last_fullsized":"2023-12-19T08:02:58.486603+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:16:57.879013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464518+0000","last_change":"2023-12-19T08:02:57.942923+0000","last_active":"2023-12-19T08:02:58.464518+0000","last_peered":"2023-12-19T08:02:58.464518+0000","last_clean":"2023-12-19T08:02:58.464518+0000","last_became_active":"2023-12-19T08:02:57.942621+0000","last_became_peered":"2023-12-19T08:02:57.942621+0000","last_unstale":"2023-12-19T08:02:58.464518+0000","last_undegraded":"2023-12-19T08:02:58.464518+0000","last_fullsized":"2023-12-19T08:02:58.464518+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:26:02.397922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467539+0000","last_change":"2023-12-19T08:02:58.462885+0000","last_active":"2023-12-19T08:02:58.467539+0000","last_peered":"2023-12-19T08:02:58.467539+0000","last_clean":"2023-12-19T08:02:58.467539+0000","last_became_active":"2023-12-19T08:02:58.462568+0000","last_became_peered":"2023-12-19T08:02:58.462568+0000","last_unstale":"2023-12-19T08:02:58.467539+0000","last_undegraded":"2023-12-19T08:02:58.467539+0000","last_fullsized":"2023-12-19T08:02:58.467539+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:09:48.640556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"98'3","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965922+0000","last_change":"2023-12-19T08:02:57.944304+0000","last_active":"2023-12-19T08:02:58.965922+0000","last_peered":"2023-12-19T08:02:58.965922+0000","last_clean":"2023-12-19T08:02:58.965922+0000","last_became_active":"2023-12-19T08:02:57.943820+0000","last_became_peered":"2023-12-19T08:02:57.943820+0000","last_unstale":"2023-12-19T08:02:58.965922+0000","last_undegraded":"2023-12-19T08:02:58.965922+0000","last_fullsized":"2023-12-19T08:02:58.965922+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:57.134975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482365+0000","last_change":"2023-12-19T08:02:58.477892+0000","last_active":"2023-12-19T08:02:58.482365+0000","last_peered":"2023-12-19T08:02:58.482365+0000","last_clean":"2023-12-19T08:02:58.482365+0000","last_became_active":"2023-12-19T08:02:58.477626+0000","last_became_peered":"2023-12-19T08:02:58.477626+0000","last_unstale":"2023-12-19T08:02:58.482365+0000","last_undegraded":"2023-12-19T08:02:58.482365+0000","last_fullsized":"2023-12-19T08:02:58.482365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"98'6","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.964176+0000","last_change":"2023-12-19T08:02:57.945475+0000","last_active":"2023-12-19T08:02:58.964176+0000","last_peered":"2023-12-19T08:02:58.964176+0000","last_clean":"2023-12-19T08:02:58.964176+0000","last_became_active":"2023-12-19T08:02:57.945022+0000","last_became_peered":"2023-12-19T08:02:57.945022+0000","last_unstale":"2023-12-19T08:02:58.964176+0000","last_undegraded":"2023-12-19T08:02:58.964176+0000","last_fullsized":"2023-12-19T08:02:58.964176+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:40.374112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476066+0000","last_change":"2023-12-19T08:02:58.471519+0000","last_active":"2023-12-19T08:02:58.476066+0000","last_peered":"2023-12-19T08:02:58.476066+0000","last_clean":"2023-12-19T08:02:58.476066+0000","last_became_active":"2023-12-19T08:02:58.471209+0000","last_became_peered":"2023-12-19T08:02:58.471209+0000","last_unstale":"2023-12-19T08:02:58.476066+0000","last_undegraded":"2023-12-19T08:02:58.476066+0000","last_fullsized":"2023-12-19T08:02:58.476066+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:14:27.163320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.961834+0000","last_change":"2023-12-19T08:02:57.938702+0000","last_active":"2023-12-19T08:02:58.961834+0000","last_peered":"2023-12-19T08:02:58.961834+0000","last_clean":"2023-12-19T08:02:58.961834+0000","last_became_active":"2023-12-19T08:02:57.938414+0000","last_became_peered":"2023-12-19T08:02:57.938414+0000","last_unstale":"2023-12-19T08:02:58.961834+0000","last_undegraded":"2023-12-19T08:02:58.961834+0000","last_fullsized":"2023-12-19T08:02:58.961834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488896+0000","last_change":"2023-12-19T08:02:58.480870+0000","last_active":"2023-12-19T08:02:58.488896+0000","last_peered":"2023-12-19T08:02:58.488896+0000","last_clean":"2023-12-19T08:02:58.488896+0000","last_became_active":"2023-12-19T08:02:58.479791+0000","last_became_peered":"2023-12-19T08:02:58.479791+0000","last_unstale":"2023-12-19T08:02:58.488896+0000","last_undegraded":"2023-12-19T08:02:58.488896+0000","last_fullsized":"2023-12-19T08:02:58.488896+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:40.194019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.463712+0000","last_change":"2023-12-19T08:02:57.941949+0000","last_active":"2023-12-19T08:02:58.463712+0000","last_peered":"2023-12-19T08:02:58.463712+0000","last_clean":"2023-12-19T08:02:58.463712+0000","last_became_active":"2023-12-19T08:02:57.941671+0000","last_became_peered":"2023-12-19T08:02:57.941671+0000","last_unstale":"2023-12-19T08:02:58.463712+0000","last_undegraded":"2023-12-19T08:02:58.463712+0000","last_fullsized":"2023-12-19T08:02:58.463712+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489032+0000","last_change":"2023-12-19T08:02:58.483228+0000","last_active":"2023-12-19T08:02:58.489032+0000","last_peered":"2023-12-19T08:02:58.489032+0000","last_clean":"2023-12-19T08:02:58.489032+0000","last_became_active":"2023-12-19T08:02:58.482736+0000","last_became_peered":"2023-12-19T08:02:58.482736+0000","last_unstale":"2023-12-19T08:02:58.489032+0000","last_undegraded":"2023-12-19T08:02:58.489032+0000","last_fullsized":"2023-12-19T08:02:58.489032+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:16:57.879013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468386+0000","last_change":"2023-12-19T08:02:57.938058+0000","last_active":"2023-12-19T08:02:58.468386+0000","last_peered":"2023-12-19T08:02:58.468386+0000","last_clean":"2023-12-19T08:02:58.468386+0000","last_became_active":"2023-12-19T08:02:57.937723+0000","last_became_peered":"2023-12-19T08:02:57.937723+0000","last_unstale":"2023-12-19T08:02:58.468386+0000","last_undegraded":"2023-12-19T08:02:58.468386+0000","last_fullsized":"2023-12-19T08:02:58.468386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:37.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488880+0000","last_change":"2023-12-19T08:02:58.480363+0000","last_active":"2023-12-19T08:02:58.488880+0000","last_peered":"2023-12-19T08:02:58.488880+0000","last_clean":"2023-12-19T08:02:58.488880+0000","last_became_active":"2023-12-19T08:02:58.479645+0000","last_became_peered":"2023-12-19T08:02:58.479645+0000","last_unstale":"2023-12-19T08:02:58.488880+0000","last_undegraded":"2023-12-19T08:02:58.488880+0000","last_fullsized":"2023-12-19T08:02:58.488880+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"98'8","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957475+0000","last_change":"2023-12-19T08:02:57.930378+0000","last_active":"2023-12-19T08:02:58.957475+0000","last_peered":"2023-12-19T08:02:58.957475+0000","last_clean":"2023-12-19T08:02:58.957475+0000","last_became_active":"2023-12-19T08:02:57.930092+0000","last_became_peered":"2023-12-19T08:02:57.930092+0000","last_unstale":"2023-12-19T08:02:58.957475+0000","last_undegraded":"2023-12-19T08:02:58.957475+0000","last_fullsized":"2023-12-19T08:02:58.957475+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:38.715762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464923+0000","last_change":"2023-12-19T08:02:58.460288+0000","last_active":"2023-12-19T08:02:58.464923+0000","last_peered":"2023-12-19T08:02:58.464923+0000","last_clean":"2023-12-19T08:02:58.464923+0000","last_became_active":"2023-12-19T08:02:58.459962+0000","last_became_peered":"2023-12-19T08:02:58.459962+0000","last_unstale":"2023-12-19T08:02:58.464923+0000","last_undegraded":"2023-12-19T08:02:58.464923+0000","last_fullsized":"2023-12-19T08:02:58.464923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:46.774140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.956555+0000","last_change":"2023-12-19T08:02:57.948366+0000","last_active":"2023-12-19T08:02:58.956555+0000","last_peered":"2023-12-19T08:02:58.956555+0000","last_clean":"2023-12-19T08:02:58.956555+0000","last_became_active":"2023-12-19T08:02:57.947337+0000","last_became_peered":"2023-12-19T08:02:57.947337+0000","last_unstale":"2023-12-19T08:02:58.956555+0000","last_undegraded":"2023-12-19T08:02:58.956555+0000","last_fullsized":"2023-12-19T08:02:58.956555+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482762+0000","last_change":"2023-12-19T08:02:58.477676+0000","last_active":"2023-12-19T08:02:58.482762+0000","last_peered":"2023-12-19T08:02:58.482762+0000","last_clean":"2023-12-19T08:02:58.482762+0000","last_became_active":"2023-12-19T08:02:58.477323+0000","last_became_peered":"2023-12-19T08:02:58.477323+0000","last_unstale":"2023-12-19T08:02:58.482762+0000","last_undegraded":"2023-12-19T08:02:58.482762+0000","last_fullsized":"2023-12-19T08:02:58.482762+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:29.176763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.959213+0000","last_change":"2023-12-19T08:02:57.948575+0000","last_active":"2023-12-19T08:02:58.959213+0000","last_peered":"2023-12-19T08:02:58.959213+0000","last_clean":"2023-12-19T08:02:58.959213+0000","last_became_active":"2023-12-19T08:02:57.948274+0000","last_became_peered":"2023-12-19T08:02:57.948274+0000","last_unstale":"2023-12-19T08:02:58.959213+0000","last_undegraded":"2023-12-19T08:02:58.959213+0000","last_fullsized":"2023-12-19T08:02:58.959213+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:14.153026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482818+0000","last_change":"2023-12-19T08:02:58.476362+0000","last_active":"2023-12-19T08:02:58.482818+0000","last_peered":"2023-12-19T08:02:58.482818+0000","last_clean":"2023-12-19T08:02:58.482818+0000","last_became_active":"2023-12-19T08:02:58.475826+0000","last_became_peered":"2023-12-19T08:02:58.475826+0000","last_unstale":"2023-12-19T08:02:58.482818+0000","last_undegraded":"2023-12-19T08:02:58.482818+0000","last_fullsized":"2023-12-19T08:02:58.482818+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:11:18.306416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965846+0000","last_change":"2023-12-19T08:02:57.932863+0000","last_active":"2023-12-19T08:02:58.965846+0000","last_peered":"2023-12-19T08:02:58.965846+0000","last_clean":"2023-12-19T08:02:58.965846+0000","last_became_active":"2023-12-19T08:02:57.931611+0000","last_became_peered":"2023-12-19T08:02:57.931611+0000","last_unstale":"2023-12-19T08:02:58.965846+0000","last_undegraded":"2023-12-19T08:02:58.965846+0000","last_fullsized":"2023-12-19T08:02:58.965846+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:12.331722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.478432+0000","last_change":"2023-12-19T08:02:58.470384+0000","last_active":"2023-12-19T08:02:58.478432+0000","last_peered":"2023-12-19T08:02:58.478432+0000","last_clean":"2023-12-19T08:02:58.478432+0000","last_became_active":"2023-12-19T08:02:58.470040+0000","last_became_peered":"2023-12-19T08:02:58.470040+0000","last_unstale":"2023-12-19T08:02:58.478432+0000","last_undegraded":"2023-12-19T08:02:58.478432+0000","last_fullsized":"2023-12-19T08:02:58.478432+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:14:39.780359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.460811+0000","last_change":"2023-12-19T08:02:57.946209+0000","last_active":"2023-12-19T08:02:58.460811+0000","last_peered":"2023-12-19T08:02:58.460811+0000","last_clean":"2023-12-19T08:02:58.460811+0000","last_became_active":"2023-12-19T08:02:57.945842+0000","last_became_peered":"2023-12-19T08:02:57.945842+0000","last_unstale":"2023-12-19T08:02:58.460811+0000","last_undegraded":"2023-12-19T08:02:58.460811+0000","last_fullsized":"2023-12-19T08:02:58.460811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:26.150714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477124+0000","last_change":"2023-12-19T08:02:58.471203+0000","last_active":"2023-12-19T08:02:58.477124+0000","last_peered":"2023-12-19T08:02:58.477124+0000","last_clean":"2023-12-19T08:02:58.477124+0000","last_became_active":"2023-12-19T08:02:58.470929+0000","last_became_peered":"2023-12-19T08:02:58.470929+0000","last_unstale":"2023-12-19T08:02:58.477124+0000","last_undegraded":"2023-12-19T08:02:58.477124+0000","last_fullsized":"2023-12-19T08:02:58.477124+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:24.541849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.955322+0000","last_change":"2023-12-19T08:02:57.939030+0000","last_active":"2023-12-19T08:02:58.955322+0000","last_peered":"2023-12-19T08:02:58.955322+0000","last_clean":"2023-12-19T08:02:58.955322+0000","last_became_active":"2023-12-19T08:02:57.938689+0000","last_became_peered":"2023-12-19T08:02:57.938689+0000","last_unstale":"2023-12-19T08:02:58.955322+0000","last_undegraded":"2023-12-19T08:02:58.955322+0000","last_fullsized":"2023-12-19T08:02:58.955322+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:23:28.297819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477910+0000","last_change":"2023-12-19T08:02:58.471400+0000","last_active":"2023-12-19T08:02:58.477910+0000","last_peered":"2023-12-19T08:02:58.477910+0000","last_clean":"2023-12-19T08:02:58.477910+0000","last_became_active":"2023-12-19T08:02:58.471052+0000","last_became_peered":"2023-12-19T08:02:58.471052+0000","last_unstale":"2023-12-19T08:02:58.477910+0000","last_undegraded":"2023-12-19T08:02:58.477910+0000","last_fullsized":"2023-12-19T08:02:58.477910+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:58.172015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.447334+0000","last_change":"2023-12-19T08:02:57.940846+0000","last_active":"2023-12-19T08:02:58.447334+0000","last_peered":"2023-12-19T08:02:58.447334+0000","last_clean":"2023-12-19T08:02:58.447334+0000","last_became_active":"2023-12-19T08:02:57.940539+0000","last_became_peered":"2023-12-19T08:02:57.940539+0000","last_unstale":"2023-12-19T08:02:58.447334+0000","last_undegraded":"2023-12-19T08:02:58.447334+0000","last_fullsized":"2023-12-19T08:02:58.447334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:58:52.740845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486943+0000","last_change":"2023-12-19T08:02:58.472977+0000","last_active":"2023-12-19T08:02:58.486943+0000","last_peered":"2023-12-19T08:02:58.486943+0000","last_clean":"2023-12-19T08:02:58.486943+0000","last_became_active":"2023-12-19T08:02:58.472671+0000","last_became_peered":"2023-12-19T08:02:58.472671+0000","last_unstale":"2023-12-19T08:02:58.486943+0000","last_undegraded":"2023-12-19T08:02:58.486943+0000","last_fullsized":"2023-12-19T08:02:58.486943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.447016+0000","last_change":"2023-12-19T08:02:57.941697+0000","last_active":"2023-12-19T08:02:58.447016+0000","last_peered":"2023-12-19T08:02:58.447016+0000","last_clean":"2023-12-19T08:02:58.447016+0000","last_became_active":"2023-12-19T08:02:57.940734+0000","last_became_peered":"2023-12-19T08:02:57.940734+0000","last_unstale":"2023-12-19T08:02:58.447016+0000","last_undegraded":"2023-12-19T08:02:58.447016+0000","last_fullsized":"2023-12-19T08:02:58.447016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:28.318167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488460+0000","last_change":"2023-12-19T08:02:58.480514+0000","last_active":"2023-12-19T08:02:58.488460+0000","last_peered":"2023-12-19T08:02:58.488460+0000","last_clean":"2023-12-19T08:02:58.488460+0000","last_became_active":"2023-12-19T08:02:58.480038+0000","last_became_peered":"2023-12-19T08:02:58.480038+0000","last_unstale":"2023-12-19T08:02:58.488460+0000","last_undegraded":"2023-12-19T08:02:58.488460+0000","last_fullsized":"2023-12-19T08:02:58.488460+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.960465+0000","last_change":"2023-12-19T08:02:57.942853+0000","last_active":"2023-12-19T08:02:58.960465+0000","last_peered":"2023-12-19T08:02:58.960465+0000","last_clean":"2023-12-19T08:02:58.960465+0000","last_became_active":"2023-12-19T08:02:57.942557+0000","last_became_peered":"2023-12-19T08:02:57.942557+0000","last_unstale":"2023-12-19T08:02:58.960465+0000","last_undegraded":"2023-12-19T08:02:58.960465+0000","last_fullsized":"2023-12-19T08:02:58.960465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486924+0000","last_change":"2023-12-19T08:02:58.477420+0000","last_active":"2023-12-19T08:02:58.486924+0000","last_peered":"2023-12-19T08:02:58.486924+0000","last_clean":"2023-12-19T08:02:58.486924+0000","last_became_active":"2023-12-19T08:02:58.475627+0000","last_became_peered":"2023-12-19T08:02:58.475627+0000","last_unstale":"2023-12-19T08:02:58.486924+0000","last_undegraded":"2023-12-19T08:02:58.486924+0000","last_fullsized":"2023-12-19T08:02:58.486924+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467094+0000","last_change":"2023-12-19T08:02:57.934521+0000","last_active":"2023-12-19T08:02:58.467094+0000","last_peered":"2023-12-19T08:02:58.467094+0000","last_clean":"2023-12-19T08:02:58.467094+0000","last_became_active":"2023-12-19T08:02:57.932148+0000","last_became_peered":"2023-12-19T08:02:57.932148+0000","last_unstale":"2023-12-19T08:02:58.467094+0000","last_undegraded":"2023-12-19T08:02:58.467094+0000","last_fullsized":"2023-12-19T08:02:58.467094+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.481721+0000","last_change":"2023-12-19T08:02:58.468847+0000","last_active":"2023-12-19T08:02:58.481721+0000","last_peered":"2023-12-19T08:02:58.481721+0000","last_clean":"2023-12-19T08:02:58.481721+0000","last_became_active":"2023-12-19T08:02:58.468517+0000","last_became_peered":"2023-12-19T08:02:58.468517+0000","last_unstale":"2023-12-19T08:02:58.481721+0000","last_undegraded":"2023-12-19T08:02:58.481721+0000","last_fullsized":"2023-12-19T08:02:58.481721+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:55.269402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464357+0000","last_change":"2023-12-19T08:02:57.941833+0000","last_active":"2023-12-19T08:02:58.464357+0000","last_peered":"2023-12-19T08:02:58.464357+0000","last_clean":"2023-12-19T08:02:58.464357+0000","last_became_active":"2023-12-19T08:02:57.941493+0000","last_became_peered":"2023-12-19T08:02:57.941493+0000","last_unstale":"2023-12-19T08:02:58.464357+0000","last_undegraded":"2023-12-19T08:02:58.464357+0000","last_fullsized":"2023-12-19T08:02:58.464357+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:13.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477064+0000","last_change":"2023-12-19T08:02:58.473356+0000","last_active":"2023-12-19T08:02:58.477064+0000","last_peered":"2023-12-19T08:02:58.477064+0000","last_clean":"2023-12-19T08:02:58.477064+0000","last_became_active":"2023-12-19T08:02:58.471319+0000","last_became_peered":"2023-12-19T08:02:58.471319+0000","last_unstale":"2023-12-19T08:02:58.477064+0000","last_undegraded":"2023-12-19T08:02:58.477064+0000","last_fullsized":"2023-12-19T08:02:58.477064+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:56:39.326794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464836+0000","last_change":"2023-12-19T08:02:57.946724+0000","last_active":"2023-12-19T08:02:58.464836+0000","last_peered":"2023-12-19T08:02:58.464836+0000","last_clean":"2023-12-19T08:02:58.464836+0000","last_became_active":"2023-12-19T08:02:57.946416+0000","last_became_peered":"2023-12-19T08:02:57.946416+0000","last_unstale":"2023-12-19T08:02:58.464836+0000","last_undegraded":"2023-12-19T08:02:58.464836+0000","last_fullsized":"2023-12-19T08:02:58.464836+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:13.735477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486934+0000","last_change":"2023-12-19T08:02:58.476374+0000","last_active":"2023-12-19T08:02:58.486934+0000","last_peered":"2023-12-19T08:02:58.486934+0000","last_clean":"2023-12-19T08:02:58.486934+0000","last_became_active":"2023-12-19T08:02:58.475541+0000","last_became_peered":"2023-12-19T08:02:58.475541+0000","last_unstale":"2023-12-19T08:02:58.486934+0000","last_undegraded":"2023-12-19T08:02:58.486934+0000","last_fullsized":"2023-12-19T08:02:58.486934+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.469542+0000","last_change":"2023-12-19T08:02:57.942781+0000","last_active":"2023-12-19T08:02:58.469542+0000","last_peered":"2023-12-19T08:02:58.469542+0000","last_clean":"2023-12-19T08:02:58.469542+0000","last_became_active":"2023-12-19T08:02:57.942490+0000","last_became_peered":"2023-12-19T08:02:57.942490+0000","last_unstale":"2023-12-19T08:02:58.469542+0000","last_undegraded":"2023-12-19T08:02:58.469542+0000","last_fullsized":"2023-12-19T08:02:58.469542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:36.209726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489494+0000","last_change":"2023-12-19T08:02:58.488636+0000","last_active":"2023-12-19T08:02:58.489494+0000","last_peered":"2023-12-19T08:02:58.489494+0000","last_clean":"2023-12-19T08:02:58.489494+0000","last_became_active":"2023-12-19T08:02:58.488323+0000","last_became_peered":"2023-12-19T08:02:58.488323+0000","last_unstale":"2023-12-19T08:02:58.489494+0000","last_undegraded":"2023-12-19T08:02:58.489494+0000","last_fullsized":"2023-12-19T08:02:58.489494+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467801+0000","last_change":"2023-12-19T08:02:57.941368+0000","last_active":"2023-12-19T08:02:58.467801+0000","last_peered":"2023-12-19T08:02:58.467801+0000","last_clean":"2023-12-19T08:02:58.467801+0000","last_became_active":"2023-12-19T08:02:57.939982+0000","last_became_peered":"2023-12-19T08:02:57.939982+0000","last_unstale":"2023-12-19T08:02:58.467801+0000","last_undegraded":"2023-12-19T08:02:58.467801+0000","last_fullsized":"2023-12-19T08:02:58.467801+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:53.330499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482956+0000","last_change":"2023-12-19T08:02:58.479112+0000","last_active":"2023-12-19T08:02:58.482956+0000","last_peered":"2023-12-19T08:02:58.482956+0000","last_clean":"2023-12-19T08:02:58.482956+0000","last_became_active":"2023-12-19T08:02:58.477115+0000","last_became_peered":"2023-12-19T08:02:58.477115+0000","last_unstale":"2023-12-19T08:02:58.482956+0000","last_undegraded":"2023-12-19T08:02:58.482956+0000","last_fullsized":"2023-12-19T08:02:58.482956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468931+0000","last_change":"2023-12-19T08:02:57.947446+0000","last_active":"2023-12-19T08:02:58.468931+0000","last_peered":"2023-12-19T08:02:58.468931+0000","last_clean":"2023-12-19T08:02:58.468931+0000","last_became_active":"2023-12-19T08:02:57.946835+0000","last_became_peered":"2023-12-19T08:02:57.946835+0000","last_unstale":"2023-12-19T08:02:58.468931+0000","last_undegraded":"2023-12-19T08:02:58.468931+0000","last_fullsized":"2023-12-19T08:02:58.468931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:47.543596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468541+0000","last_change":"2023-12-19T08:02:58.467190+0000","last_active":"2023-12-19T08:02:58.468541+0000","last_peered":"2023-12-19T08:02:58.468541+0000","last_clean":"2023-12-19T08:02:58.468541+0000","last_became_active":"2023-12-19T08:02:58.466060+0000","last_became_peered":"2023-12-19T08:02:58.466060+0000","last_unstale":"2023-12-19T08:02:58.468541+0000","last_undegraded":"2023-12-19T08:02:58.468541+0000","last_fullsized":"2023-12-19T08:02:58.468541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.459409+0000","last_change":"2023-12-19T08:02:57.940724+0000","last_active":"2023-12-19T08:02:58.459409+0000","last_peered":"2023-12-19T08:02:58.459409+0000","last_clean":"2023-12-19T08:02:58.459409+0000","last_became_active":"2023-12-19T08:02:57.940447+0000","last_became_peered":"2023-12-19T08:02:57.940447+0000","last_unstale":"2023-12-19T08:02:58.459409+0000","last_undegraded":"2023-12-19T08:02:58.459409+0000","last_fullsized":"2023-12-19T08:02:58.459409+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:45.430449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474942+0000","last_change":"2023-12-19T08:02:58.471852+0000","last_active":"2023-12-19T08:02:58.474942+0000","last_peered":"2023-12-19T08:02:58.474942+0000","last_clean":"2023-12-19T08:02:58.474942+0000","last_became_active":"2023-12-19T08:02:58.471576+0000","last_became_peered":"2023-12-19T08:02:58.471576+0000","last_unstale":"2023-12-19T08:02:58.474942+0000","last_undegraded":"2023-12-19T08:02:58.474942+0000","last_fullsized":"2023-12-19T08:02:58.474942+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:28:55.839090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.448116+0000","last_change":"2023-12-19T08:02:57.945400+0000","last_active":"2023-12-19T08:02:58.448116+0000","last_peered":"2023-12-19T08:02:58.448116+0000","last_clean":"2023-12-19T08:02:58.448116+0000","last_became_active":"2023-12-19T08:02:57.945004+0000","last_became_peered":"2023-12-19T08:02:57.945004+0000","last_unstale":"2023-12-19T08:02:58.448116+0000","last_undegraded":"2023-12-19T08:02:58.448116+0000","last_fullsized":"2023-12-19T08:02:58.448116+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:51.160105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"87'183","reported_seq":463,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465010+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:02:58.465010+0000","last_peered":"2023-12-19T08:02:58.465010+0000","last_clean":"2023-12-19T08:02:58.465010+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:02:58.465010+0000","last_undegraded":"2023-12-19T08:02:58.465010+0000","last_fullsized":"2023-12-19T08:02:58.465010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311344,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060496,"statfs":{"total":96636764160,"available":96317947904,"internally_reserved":0,"allocated":1810432,"data_stored":1213502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2450000000000001,"15min":1.2470000000000001},"min":{"1min":0.877,"5min":0.69699999999999995,"15min":0.59599999999999997},"max":{"1min":1.5880000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.458},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.3120000000000001,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":4.6210000000000004,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.397}]},{"osd":1,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.1899999999999999,"15min":1.1639999999999999},"min":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":4.9160000000000004,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.113},{"interface":"front","average":{"1min":1.26,"5min":1.167,"15min":1.1659999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.8969999999999998,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.359}]},{"osd":2,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.1910000000000001,"15min":1.1759999999999999},"min":{"1min":0.92400000000000004,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":4.9409999999999998,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.024},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.268,"15min":1.2450000000000001},"min":{"1min":0.97599999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5149999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3069999999999999,"15min":1.296},"min":{"1min":0.95799999999999996,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":4.8220000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.3700000000000001,"15min":1.3540000000000001},"min":{"1min":0.97299999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":4.4980000000000002,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.5369999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.272,"15min":1.228},"min":{"1min":0.71899999999999997,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.3140000000000001,"15min":1.2490000000000001},"min":{"1min":0.91900000000000004,"5min":0.76500000000000001,"15min":0.64900000000000002},"max":{"1min":6.8360000000000003,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.639}]},{"osd":5,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.3,"15min":1.2509999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.3959999999999999,"15min":1.363},"min":{"1min":0.98199999999999998,"5min":0.751,"15min":0.751},"max":{"1min":2.004,"5min":2.2120000000000002,"15min":2.3010000000000002},"last":1.679}]},{"osd":6,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.4990000000000001,"15min":1.429},"min":{"1min":1.1759999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.605},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.423,"15min":1.395},"min":{"1min":0.95999999999999996,"5min":0.90200000000000002,"15min":0.58299999999999996},"max":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.496}]}]},{"osd":6,"up_from":13,"seq":55834575037,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":617520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.071,"15min":1.103},"min":{"1min":0.75800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.472,"5min":1.5569999999999999,"15min":1.665},"last":1.137},{"interface":"front","average":{"1min":1.109,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.373,"5min":1.756,"15min":1.756},"last":0.70099999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.75800000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4770000000000001,"5min":1.675,"15min":1.881},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.161,"15min":1.161},"min":{"1min":0.745,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4279999999999999,"5min":1.542,"15min":1.7989999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.675,"5min":1.847,"15min":1.847},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1970000000000001,"15min":1.2130000000000001},"min":{"1min":0.70799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4259999999999999,"5min":1.694,"15min":1.7929999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2450000000000001,"15min":1.2669999999999999},"min":{"1min":0.77600000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0659999999999998},"last":1.052},{"interface":"front","average":{"1min":1.252,"5min":1.3,"15min":1.3100000000000001},"min":{"1min":0.80900000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7889999999999999,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.258}]},{"osd":4,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1990000000000001,"15min":1.1839999999999999},"min":{"1min":0.80900000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5900000000000001,"5min":1.7130000000000001,"15min":2.137},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.629,"5min":1.974,"15min":2.218},"last":1.2889999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3120000000000001,"15min":1.2929999999999999},"min":{"1min":0.78300000000000003,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3109999999999999},"min":{"1min":0.83999999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.663,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.93999999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":2.2029999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.448,"15min":1.425},"min":{"1min":1.1379999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.137,"15min":2.2639999999999998},"last":1.4399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575037,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298464,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073376,"statfs":{"total":96636764160,"available":96331137024,"internally_reserved":0,"allocated":1204224,"data_stored":616018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.546,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.093,"5min":1.2410000000000001,"15min":1.264},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.50700000000000001},"max":{"1min":1.5940000000000001,"5min":1.954,"15min":2.3140000000000001},"last":0.89500000000000002}]},{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.341,"5min":1.585,"15min":3.5409999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.77500000000000002,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.647,"5min":2.7679999999999998,"15min":3.113},"last":0.95499999999999996}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1759999999999999,"15min":1.2010000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66000000000000003},"max":{"1min":1.6739999999999999,"5min":1.821,"15min":3.339},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.175,"15min":1.2110000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4279999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":0.79900000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.228,"15min":1.2569999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.4630000000000001,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.93400000000000005,"5min":0.88400000000000001,"15min":0.58199999999999996},"max":{"1min":1.6359999999999999,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.3260000000000001}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.202,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.232},{"interface":"front","average":{"1min":1.286,"5min":1.2629999999999999,"15min":1.2929999999999999},"min":{"1min":0.96899999999999997,"5min":0.70299999999999996,"15min":0.59199999999999997},"max":{"1min":1.621,"5min":2.843,"15min":4.6310000000000002},"last":1.026}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.81499999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.728,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.367,"15min":1.3700000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.70599999999999996},"max":{"1min":1.96,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.379}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3939999999999999,"15min":1.427},"min":{"1min":0.88,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.6739999999999999,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.46,"15min":1.462},"min":{"1min":1.038,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.8200000000000001,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.482}]}]},{"osd":4,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302588,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069252,"statfs":{"total":96636764160,"available":96326914048,"internally_reserved":0,"allocated":1232896,"data_stored":624752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1950000000000001,"15min":1.214},"min":{"1min":0.70499999999999996,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.4490000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.056},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.94999999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.742,"5min":1.742,"15min":2.738},"last":1.8680000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.159,"15min":1.177},"min":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.556,"5min":1.7589999999999999,"15min":2.589},"last":1.403},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1060000000000001,"15min":1.143},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54300000000000004},"max":{"1min":1.407,"5min":1.528,"15min":2.2589999999999999},"last":0.877}]},{"osd":2,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.085,"15min":1.1359999999999999},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5149999999999999,"15min":2.4569999999999999},"last":1.351},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.2350000000000001},"min":{"1min":0.89000000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.524,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.212}]},{"osd":3,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2509999999999999,"15min":1.2829999999999999},"min":{"1min":0.70999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5980000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2969999999999999,"15min":1.3360000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.81899999999999995},"max":{"1min":1.5780000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.754}]},{"osd":5,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1739999999999999,"15min":1.2170000000000001},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5549999999999999,"5min":1.8149999999999999,"15min":3.556},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.2150000000000001},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6850000000000001,"5min":1.764,"15min":3.2599999999999998},"last":1.1679999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7350000000000001,"5min":1.748,"15min":2.645},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.3839999999999999},"min":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6299999999999999,"5min":2.0049999999999999,"15min":2.923},"last":2.0880000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3580000000000001},"min":{"1min":0.82699999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7410000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.476},{"interface":"front","average":{"1min":1.337,"5min":1.3859999999999999,"15min":1.411},"min":{"1min":0.80200000000000005,"5min":0.73099999999999998,"15min":0.70999999999999996},"max":{"1min":1.829,"5min":1.966,"15min":2.8460000000000001},"last":1.919}]}]},{"osd":3,"up_from":13,"seq":55834575037,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306668,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065172,"statfs":{"total":96636764160,"available":96322736128,"internally_reserved":0,"allocated":1216512,"data_stored":623666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.5539999999999998,"5min":2.5539999999999998,"15min":5.1079999999999997},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93200000000000005,"15min":0.97299999999999998},"min":{"1min":0.66100000000000003,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.962,"5min":1.962,"15min":6.1900000000000004},"last":0.95399999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.91700000000000004,"15min":0.93899999999999995},"min":{"1min":0.60999999999999999,"5min":0.54500000000000004,"15min":0.45600000000000002},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":4.8879999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.032,"15min":1.0680000000000001},"min":{"1min":0.49399999999999999,"5min":0.442,"15min":0.442},"max":{"1min":2.423,"5min":2.423,"15min":8.0830000000000002},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.05,"15min":1.081},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":4.7519999999999998},"last":1.292},{"interface":"front","average":{"1min":1.252,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.57299999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.48,"5min":2.48,"15min":5.3330000000000002},"last":1.0569999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2350000000000001,"15min":1.274},"min":{"1min":0.81200000000000006,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":4.7130000000000001},"last":1.601},{"interface":"front","average":{"1min":1.238,"5min":1.222,"15min":1.268},"min":{"1min":0.86499999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.536,"5min":1.6240000000000001,"15min":5.0220000000000002},"last":1.325}]},{"osd":5,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2490000000000001,"15min":1.2969999999999999},"min":{"1min":0.86499999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.996,"5min":1.996,"15min":4.9889999999999999},"last":1.242},{"interface":"front","average":{"1min":1.427,"5min":1.3500000000000001,"15min":1.3779999999999999},"min":{"1min":1.1419999999999999,"5min":0.90500000000000003,"15min":0.65800000000000003},"max":{"1min":2.161,"5min":2.161,"15min":8.5299999999999994},"last":1.163}]},{"osd":6,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3919999999999999,"15min":1.4199999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":7.2750000000000004},"last":2.4830000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3859999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.79100000000000004},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":4.6550000000000002},"last":1.6579999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4550000000000001,"15min":1.51},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.84799999999999998},"max":{"1min":2.504,"5min":2.504,"15min":16.081},"last":2.7530000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4390000000000001,"15min":1.4750000000000001},"min":{"1min":1.0940000000000001,"5min":0.92700000000000005,"15min":0.92000000000000004},"max":{"1min":1.71,"5min":1.833,"15min":15.285},"last":2.3359999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":616698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.91200000000000003,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.56399999999999995,"15min":0.437},"max":{"1min":1.1910000000000001,"5min":1.776,"15min":4.1479999999999997},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.70299999999999996,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.7769999999999999,"5min":1.8460000000000001,"15min":4.093},"last":1.05}]},{"osd":1,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.96899999999999997,"15min":0.96499999999999997},"min":{"1min":0.78600000000000003,"5min":0.60599999999999998,"15min":0.51800000000000002},"max":{"1min":1.327,"5min":1.5620000000000001,"15min":3.8700000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.004,"15min":1.024},"min":{"1min":0.57199999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.835,"5min":1.835,"15min":3.9089999999999998},"last":0.91800000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98799999999999999,"15min":0.96699999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.667},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0980000000000001,"15min":1.111},"min":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.629,"15min":1.7250000000000001},"last":1.1339999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3260000000000001,"15min":1.331},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.79300000000000004},"max":{"1min":1.47,"5min":1.7669999999999999,"15min":3.9239999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.2949999999999999,"15min":1.3120000000000001},"min":{"1min":0.94299999999999995,"5min":0.89400000000000002,"15min":0.82099999999999995},"max":{"1min":1.925,"5min":1.925,"15min":4.0170000000000003},"last":1.486}]},{"osd":5,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3380000000000001,"15min":1.3560000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":3.948},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.2769999999999999,"15min":1.256},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.5760000000000001,"5min":1.6930000000000001,"15min":1.879},"last":1.085}]},{"osd":6,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2969999999999999,"15min":1.3},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.752},"max":{"1min":1.5669999999999999,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.2},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3660000000000001,"15min":1.383},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.85399999999999998},"max":{"1min":1.548,"5min":1.867,"15min":3.9620000000000002},"last":1.359}]},{"osd":7,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.405},"min":{"1min":1.034,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":3.9809999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3520000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.97399999999999998,"15min":0.88200000000000001},"max":{"1min":1.5600000000000001,"5min":1.8620000000000001,"15min":2.206},"last":1.26}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306672,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065168,"statfs":{"total":96636764160,"available":96322732032,"internally_reserved":0,"allocated":1220608,"data_stored":623224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.85699999999999998},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.661,"15min":1.661},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.95399999999999996,"15min":0.94999999999999996},"min":{"1min":0.60399999999999998,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":2.2050000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.76100000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":0.92800000000000005},"min":{"1min":0.61499999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.96,"5min":2.96,"15min":2.96},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1120000000000001,"15min":1.0649999999999999},"min":{"1min":0.67500000000000004,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":2.323,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.90700000000000003}]},{"osd":3,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.2530000000000001,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.117,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.46800000000000003},"max":{"1min":2.1299999999999999,"5min":2.407,"15min":2.407},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1930000000000001,"15min":1.198},"min":{"1min":0.877,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.877,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.359,"15min":1.3160000000000001},"min":{"1min":1.042,"5min":0.95999999999999996,"15min":0.64400000000000002},"max":{"1min":2.2890000000000001,"5min":3.04,"15min":3.04},"last":1.0880000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.2989999999999999,"15min":1.2849999999999999},"min":{"1min":0.90100000000000002,"5min":0.87,"15min":0.65100000000000002},"max":{"1min":1.996,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.194},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3200000000000001,"15min":1.2969999999999999},"min":{"1min":1.0089999999999999,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.9239999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.03}]},{"osd":6,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4299999999999999,"15min":1.393},"min":{"1min":1.1000000000000001,"5min":0.96799999999999997,"15min":0.72799999999999998},"max":{"1min":2.3559999999999999,"5min":3.089,"15min":3.089},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.393,"15min":1.3720000000000001},"min":{"1min":0.96299999999999997,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":2.169,"5min":2.46,"15min":2.6230000000000002},"last":1.129}]},{"osd":7,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4399999999999999,"15min":1.4319999999999999},"min":{"1min":1.1759999999999999,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":2.0430000000000001,"5min":2.504,"15min":3.6080000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.3999999999999999,"15min":1.4019999999999999},"min":{"1min":1.0469999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":4.101},"last":1.079}]}]},{"osd":0,"up_from":13,"seq":55834575037,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303144,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068696,"statfs":{"total":96636764160,"available":96326344704,"internally_reserved":0,"allocated":1802240,"data_stored":1212469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.262,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91600000000000004,"15min":0.93799999999999994},"min":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.385,"5min":1.385,"15min":6.0309999999999997},"last":0.90300000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.95999999999999996,"15min":0.97999999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.054},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.044,"15min":1.03},"min":{"1min":0.75700000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3700000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.1699999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.96099999999999997},"min":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.51400000000000001},"max":{"1min":1.48,"5min":1.5529999999999999,"15min":1.671},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.115,"15min":1.129},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.514,"5min":1.663,"15min":1.6719999999999999},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3129999999999999,"15min":1.3080000000000001},"min":{"1min":0.86499999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6000000000000001,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.5840000000000001,"5min":1.855,"15min":2.141},"last":1.101}]},{"osd":5,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.351,"15min":1.3600000000000001},"min":{"1min":0.83399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.6819999999999999,"5min":2.085,"15min":2.117},"last":1.389},{"interface":"front","average":{"1min":1.339,"5min":1.357,"15min":1.361},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.843,"5min":2.125,"15min":2.125},"last":1.264}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.276,"15min":1.268},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.77400000000000002},"max":{"1min":1.5329999999999999,"5min":1.97,"15min":1.97},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3200000000000001,"15min":1.3220000000000001},"min":{"1min":0.91900000000000004,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.712,"5min":1.871,"15min":1.9530000000000001},"last":1.657}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.341,"15min":1.3240000000000001},"min":{"1min":0.97799999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.8009999999999999,"5min":1.887,"15min":2.0539999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3720000000000001,"15min":1.3759999999999999},"min":{"1min":0.88600000000000001,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.714,"5min":2.0059999999999998,"15min":2.016},"last":1.712}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:03:03.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:03:03.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.389+0000 7fca3186e700 1 Processor -- start 2023-12-19T08:03:03.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.389+0000 7fca3186e700 1 -- start start 2023-12-19T08:03:03.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca3186e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 0x7fca2c069820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca3186e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca2c069d60 0x7fca2c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca3186e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c06a1d0 con 0x7fca2c0cb710 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca3186e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0cbf30 con 0x7fca2c069d60 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca3186e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1563a0 con 0x7fca2c14f880 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fca2c0cb710 0x7fca2c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59788/0 (socket says 172.21.15.111:59788) 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2bfff700 1 -- 172.21.15.111:0/1199627990 learned_addr learned my addr 172.21.15.111:0/1199627990 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 --2- 172.21.15.111:0/1199627990 >> v2:172.21.15.143:3300/0 conn(0x7fca2c069d60 0x7fca2c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2affd700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 0x7fca2c069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708394395 0 0) 0x7fca2c06a1d0 con 0x7fca2c0cb710 2023-12-19T08:03:03.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca18003570 con 0x7fca2c0cb710 2023-12-19T08:03:03.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 832201164 0 0) 0x7fca18003570 con 0x7fca2c0cb710 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 -- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 msgr2=0x7fca2c069820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 0x7fca2c069820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 -- 172.21.15.111:0/1199627990 >> v1:172.21.15.111:6789/0 conn(0x7fca2c0cb710 legacy=0x7fca2c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 -- 172.21.15.111:0/1199627990 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c155a70 con 0x7fca2c069d60 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca2affd700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 0x7fca2c069820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.393+0000 7fca227fc700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c069d60 0x7fca2c0cb1d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fca1400aa90 tx=0x7fca1400a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=642b4262977b46f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24001470 con 0x7fca2c069d60 2023-12-19T08:03:03.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1400ae70 con 0x7fca2c069d60 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca2a7fc700 1 -- 172.21.15.111:0/1199627990 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1400c870 con 0x7fca2c069d60 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c069d60 msgr2=0x7fca2c0cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c069d60 0x7fca2c0cb1d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fca1400aa90 tx=0x7fca1400a330 comp rx=0 tx=0).stop 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- 172.21.15.111:0/1199627990 shutdown_connections 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c069d60 0x7fca2c0cb1d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 --2- 172.21.15.111:0/1199627990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14f880 0x7fca2c069820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- 172.21.15.111:0/1199627990 >> 172.21.15.111:0/1199627990 conn(0x7fca2c068500 msgr2=0x7fca2c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- 172.21.15.111:0/1199627990 shutdown_connections 2023-12-19T08:03:03.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- 172.21.15.111:0/1199627990 wait complete. 2023-12-19T08:03:03.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 Processor -- start 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.397+0000 7fca3186e700 1 -- start start 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c069d60 0x7fca2c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c14f880 0x7fca2c145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c152f10 con 0x7fca2c0cb710 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c14ed80 con 0x7fca2c14f880 2023-12-19T08:03:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c152c30 con 0x7fca2c069d60 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37376/0 (socket says 172.21.15.111:37376) 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 -- 172.21.15.111:0/940274979 learned_addr learned my addr 172.21.15.111:0/940274979 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca2bfff700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c14f880 0x7fca2c145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca2affd700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c069d60 0x7fca2c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 -- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c069d60 msgr2=0x7fca2c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c069d60 0x7fca2c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 -- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c14f880 msgr2=0x7fca2c145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c14f880 0x7fca2c145040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 -- 172.21.15.111:0/940274979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c009a20 con 0x7fca2c0cb710 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca227fc700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fca1400a370 tx=0x7fca14018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14017b80 con 0x7fca2c0cb710 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca14017d00 con 0x7fca2c0cb710 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14020680 con 0x7fca2c0cb710 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1400a5d0 con 0x7fca2c0cb710 2023-12-19T08:03:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.401+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c0c48a0 con 0x7fca2c0cb710 2023-12-19T08:03:03.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.405+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c071120 con 0x7fca2c0cb710 2023-12-19T08:03:03.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.405+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fca140209f0 con 0x7fca2c0cb710 2023-12-19T08:03:03.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.409+0000 7fca237fe700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 0x7fca0c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.409+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fca14013070 con 0x7fca2c0cb710 2023-12-19T08:03:03.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.409+0000 7fca2affd700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 0x7fca0c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.409+0000 7fca2affd700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 0x7fca0c062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0099f0 tx=0x7fca1c00b040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.413+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca1404e4b0 con 0x7fca2c0cb710 2023-12-19T08:03:03.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.577+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca2c068c50 con 0x7fca0c060670 2023-12-19T08:03:03.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.593+0000 7fca237fe700 1 -- 172.21.15.111:0/940274979 <== mgr.6044 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287145 (secure 0 0 0) 0x7fca2c068c50 con 0x7fca0c060670 2023-12-19T08:03:03.601 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:03.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 msgr2=0x7fca0c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 0x7fca0c062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0099f0 tx=0x7fca1c00b040 comp rx=0 tx=0).stop 2023-12-19T08:03:03.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 msgr2=0x7fca2c144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fca1400a370 tx=0x7fca14018000 comp rx=0 tx=0).stop 2023-12-19T08:03:03.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 shutdown_connections 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca0c060670 0x7fca0c062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c14f880 0x7fca2c145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca2c0cb710 0x7fca2c144b00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 --2- 172.21.15.111:0/940274979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c069d60 0x7fca2c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 >> 172.21.15.111:0/940274979 conn(0x7fca2c068500 msgr2=0x7fca2c0ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 shutdown_connections 2023-12-19T08:03:03.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.601+0000 7fca3186e700 1 -- 172.21.15.111:0/940274979 wait complete. 2023-12-19T08:03:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:03:03.628 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:03:02.628370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2434016,"kb_used_data":10656,"kb_used_omap":0,"kb_used_meta":2423296,"kb_avail":752540704,"statfs":{"total":773094113280,"available":770601680896,"internally_reserved":0,"allocated":10911744,"data_stored":6147849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2481455104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.722046"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485395+0000","last_change":"2023-12-19T08:02:58.482752+0000","last_active":"2023-12-19T08:02:58.485395+0000","last_peered":"2023-12-19T08:02:58.485395+0000","last_clean":"2023-12-19T08:02:58.485395+0000","last_became_active":"2023-12-19T08:02:58.482462+0000","last_became_peered":"2023-12-19T08:02:58.482462+0000","last_unstale":"2023-12-19T08:02:58.485395+0000","last_undegraded":"2023-12-19T08:02:58.485395+0000","last_fullsized":"2023-12-19T08:02:58.485395+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:56:39.326794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484669+0000","last_change":"2023-12-19T08:02:58.479640+0000","last_active":"2023-12-19T08:02:58.484669+0000","last_peered":"2023-12-19T08:02:58.484669+0000","last_clean":"2023-12-19T08:02:58.484669+0000","last_became_active":"2023-12-19T08:02:58.479292+0000","last_became_peered":"2023-12-19T08:02:58.479292+0000","last_unstale":"2023-12-19T08:02:58.484669+0000","last_undegraded":"2023-12-19T08:02:58.484669+0000","last_fullsized":"2023-12-19T08:02:58.484669+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:35.015348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482162+0000","last_change":"2023-12-19T08:02:58.477499+0000","last_active":"2023-12-19T08:02:58.482162+0000","last_peered":"2023-12-19T08:02:58.482162+0000","last_clean":"2023-12-19T08:02:58.482162+0000","last_became_active":"2023-12-19T08:02:58.477069+0000","last_became_peered":"2023-12-19T08:02:58.477069+0000","last_unstale":"2023-12-19T08:02:58.482162+0000","last_undegraded":"2023-12-19T08:02:58.482162+0000","last_fullsized":"2023-12-19T08:02:58.482162+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.481262+0000","last_change":"2023-12-19T08:02:58.478609+0000","last_active":"2023-12-19T08:02:58.481262+0000","last_peered":"2023-12-19T08:02:58.481262+0000","last_clean":"2023-12-19T08:02:58.481262+0000","last_became_active":"2023-12-19T08:02:58.478259+0000","last_became_peered":"2023-12-19T08:02:58.478259+0000","last_unstale":"2023-12-19T08:02:58.481262+0000","last_undegraded":"2023-12-19T08:02:58.481262+0000","last_fullsized":"2023-12-19T08:02:58.481262+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:28.600644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484522+0000","last_change":"2023-12-19T08:02:58.480655+0000","last_active":"2023-12-19T08:02:58.484522+0000","last_peered":"2023-12-19T08:02:58.484522+0000","last_clean":"2023-12-19T08:02:58.484522+0000","last_became_active":"2023-12-19T08:02:58.478102+0000","last_became_peered":"2023-12-19T08:02:58.478102+0000","last_unstale":"2023-12-19T08:02:58.484522+0000","last_undegraded":"2023-12-19T08:02:58.484522+0000","last_fullsized":"2023-12-19T08:02:58.484522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:22.956703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465121+0000","last_change":"2023-12-19T08:02:58.460614+0000","last_active":"2023-12-19T08:02:58.465121+0000","last_peered":"2023-12-19T08:02:58.465121+0000","last_clean":"2023-12-19T08:02:58.465121+0000","last_became_active":"2023-12-19T08:02:58.459436+0000","last_became_peered":"2023-12-19T08:02:58.459436+0000","last_unstale":"2023-12-19T08:02:58.465121+0000","last_undegraded":"2023-12-19T08:02:58.465121+0000","last_fullsized":"2023-12-19T08:02:58.465121+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:19:29.498138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489076+0000","last_change":"2023-12-19T08:02:58.483542+0000","last_active":"2023-12-19T08:02:58.489076+0000","last_peered":"2023-12-19T08:02:58.489076+0000","last_clean":"2023-12-19T08:02:58.489076+0000","last_became_active":"2023-12-19T08:02:58.483316+0000","last_became_peered":"2023-12-19T08:02:58.483316+0000","last_unstale":"2023-12-19T08:02:58.489076+0000","last_undegraded":"2023-12-19T08:02:58.489076+0000","last_fullsized":"2023-12-19T08:02:58.489076+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482750+0000","last_change":"2023-12-19T08:02:58.471595+0000","last_active":"2023-12-19T08:02:58.482750+0000","last_peered":"2023-12-19T08:02:58.482750+0000","last_clean":"2023-12-19T08:02:58.482750+0000","last_became_active":"2023-12-19T08:02:58.470111+0000","last_became_peered":"2023-12-19T08:02:58.470111+0000","last_unstale":"2023-12-19T08:02:58.482750+0000","last_undegraded":"2023-12-19T08:02:58.482750+0000","last_fullsized":"2023-12-19T08:02:58.482750+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:25.694478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.487150+0000","last_change":"2023-12-19T08:02:58.477070+0000","last_active":"2023-12-19T08:02:58.487150+0000","last_peered":"2023-12-19T08:02:58.487150+0000","last_clean":"2023-12-19T08:02:58.487150+0000","last_became_active":"2023-12-19T08:02:58.476556+0000","last_became_peered":"2023-12-19T08:02:58.476556+0000","last_unstale":"2023-12-19T08:02:58.487150+0000","last_undegraded":"2023-12-19T08:02:58.487150+0000","last_fullsized":"2023-12-19T08:02:58.487150+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486632+0000","last_change":"2023-12-19T08:02:58.481446+0000","last_active":"2023-12-19T08:02:58.486632+0000","last_peered":"2023-12-19T08:02:58.486632+0000","last_clean":"2023-12-19T08:02:58.486632+0000","last_became_active":"2023-12-19T08:02:58.481121+0000","last_became_peered":"2023-12-19T08:02:58.481121+0000","last_unstale":"2023-12-19T08:02:58.486632+0000","last_undegraded":"2023-12-19T08:02:58.486632+0000","last_fullsized":"2023-12-19T08:02:58.486632+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:01:10.842548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484740+0000","last_change":"2023-12-19T08:02:58.482213+0000","last_active":"2023-12-19T08:02:58.484740+0000","last_peered":"2023-12-19T08:02:58.484740+0000","last_clean":"2023-12-19T08:02:58.484740+0000","last_became_active":"2023-12-19T08:02:58.481724+0000","last_became_peered":"2023-12-19T08:02:58.481724+0000","last_unstale":"2023-12-19T08:02:58.484740+0000","last_undegraded":"2023-12-19T08:02:58.484740+0000","last_fullsized":"2023-12-19T08:02:58.484740+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:46.500728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486899+0000","last_change":"2023-12-19T08:02:58.482020+0000","last_active":"2023-12-19T08:02:58.486899+0000","last_peered":"2023-12-19T08:02:58.486899+0000","last_clean":"2023-12-19T08:02:58.486899+0000","last_became_active":"2023-12-19T08:02:58.481724+0000","last_became_peered":"2023-12-19T08:02:58.481724+0000","last_unstale":"2023-12-19T08:02:58.486899+0000","last_undegraded":"2023-12-19T08:02:58.486899+0000","last_fullsized":"2023-12-19T08:02:58.486899+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:23:46.634962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482886+0000","last_change":"2023-12-19T08:02:58.470936+0000","last_active":"2023-12-19T08:02:58.482886+0000","last_peered":"2023-12-19T08:02:58.482886+0000","last_clean":"2023-12-19T08:02:58.482886+0000","last_became_active":"2023-12-19T08:02:58.470619+0000","last_became_peered":"2023-12-19T08:02:58.470619+0000","last_unstale":"2023-12-19T08:02:58.482886+0000","last_undegraded":"2023-12-19T08:02:58.482886+0000","last_fullsized":"2023-12-19T08:02:58.482886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:30:47.840369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465024+0000","last_change":"2023-12-19T08:02:58.460303+0000","last_active":"2023-12-19T08:02:58.465024+0000","last_peered":"2023-12-19T08:02:58.465024+0000","last_clean":"2023-12-19T08:02:58.465024+0000","last_became_active":"2023-12-19T08:02:58.459037+0000","last_became_peered":"2023-12-19T08:02:58.459037+0000","last_unstale":"2023-12-19T08:02:58.465024+0000","last_undegraded":"2023-12-19T08:02:58.465024+0000","last_fullsized":"2023-12-19T08:02:58.465024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:55:13.416628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477088+0000","last_change":"2023-12-19T08:02:58.473780+0000","last_active":"2023-12-19T08:02:58.477088+0000","last_peered":"2023-12-19T08:02:58.477088+0000","last_clean":"2023-12-19T08:02:58.477088+0000","last_became_active":"2023-12-19T08:02:58.472763+0000","last_became_peered":"2023-12-19T08:02:58.472763+0000","last_unstale":"2023-12-19T08:02:58.477088+0000","last_undegraded":"2023-12-19T08:02:58.477088+0000","last_fullsized":"2023-12-19T08:02:58.477088+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:44:31.945833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476106+0000","last_change":"2023-12-19T08:02:58.474326+0000","last_active":"2023-12-19T08:02:58.476106+0000","last_peered":"2023-12-19T08:02:58.476106+0000","last_clean":"2023-12-19T08:02:58.476106+0000","last_became_active":"2023-12-19T08:02:58.473988+0000","last_became_peered":"2023-12-19T08:02:58.473988+0000","last_unstale":"2023-12-19T08:02:58.476106+0000","last_undegraded":"2023-12-19T08:02:58.476106+0000","last_fullsized":"2023-12-19T08:02:58.476106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:28.600644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485351+0000","last_change":"2023-12-19T08:02:58.482865+0000","last_active":"2023-12-19T08:02:58.485351+0000","last_peered":"2023-12-19T08:02:58.485351+0000","last_clean":"2023-12-19T08:02:58.485351+0000","last_became_active":"2023-12-19T08:02:58.482532+0000","last_became_peered":"2023-12-19T08:02:58.482532+0000","last_unstale":"2023-12-19T08:02:58.485351+0000","last_undegraded":"2023-12-19T08:02:58.485351+0000","last_fullsized":"2023-12-19T08:02:58.485351+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:37:41.128971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474296+0000","last_change":"2023-12-19T08:02:58.471113+0000","last_active":"2023-12-19T08:02:58.474296+0000","last_peered":"2023-12-19T08:02:58.474296+0000","last_clean":"2023-12-19T08:02:58.474296+0000","last_became_active":"2023-12-19T08:02:58.470814+0000","last_became_peered":"2023-12-19T08:02:58.470814+0000","last_unstale":"2023-12-19T08:02:58.474296+0000","last_undegraded":"2023-12-19T08:02:58.474296+0000","last_fullsized":"2023-12-19T08:02:58.474296+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482595+0000","last_change":"2023-12-19T08:02:58.481385+0000","last_active":"2023-12-19T08:02:58.482595+0000","last_peered":"2023-12-19T08:02:58.482595+0000","last_clean":"2023-12-19T08:02:58.482595+0000","last_became_active":"2023-12-19T08:02:58.481112+0000","last_became_peered":"2023-12-19T08:02:58.481112+0000","last_unstale":"2023-12-19T08:02:58.482595+0000","last_undegraded":"2023-12-19T08:02:58.482595+0000","last_fullsized":"2023-12-19T08:02:58.482595+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:57.413537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482615+0000","last_change":"2023-12-19T08:02:58.469795+0000","last_active":"2023-12-19T08:02:58.482615+0000","last_peered":"2023-12-19T08:02:58.482615+0000","last_clean":"2023-12-19T08:02:58.482615+0000","last_became_active":"2023-12-19T08:02:58.469239+0000","last_became_peered":"2023-12-19T08:02:58.469239+0000","last_unstale":"2023-12-19T08:02:58.482615+0000","last_undegraded":"2023-12-19T08:02:58.482615+0000","last_fullsized":"2023-12-19T08:02:58.482615+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:30:56.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482114+0000","last_change":"2023-12-19T08:02:58.469700+0000","last_active":"2023-12-19T08:02:58.482114+0000","last_peered":"2023-12-19T08:02:58.482114+0000","last_clean":"2023-12-19T08:02:58.482114+0000","last_became_active":"2023-12-19T08:02:58.468388+0000","last_became_peered":"2023-12-19T08:02:58.468388+0000","last_unstale":"2023-12-19T08:02:58.482114+0000","last_undegraded":"2023-12-19T08:02:58.482114+0000","last_fullsized":"2023-12-19T08:02:58.482114+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:14.231358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482798+0000","last_change":"2023-12-19T08:02:58.467255+0000","last_active":"2023-12-19T08:02:58.482798+0000","last_peered":"2023-12-19T08:02:58.482798+0000","last_clean":"2023-12-19T08:02:58.482798+0000","last_became_active":"2023-12-19T08:02:58.466981+0000","last_became_peered":"2023-12-19T08:02:58.466981+0000","last_unstale":"2023-12-19T08:02:58.482798+0000","last_undegraded":"2023-12-19T08:02:58.482798+0000","last_fullsized":"2023-12-19T08:02:58.482798+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:44:56.890865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486898+0000","last_change":"2023-12-19T08:02:58.471044+0000","last_active":"2023-12-19T08:02:58.486898+0000","last_peered":"2023-12-19T08:02:58.486898+0000","last_clean":"2023-12-19T08:02:58.486898+0000","last_became_active":"2023-12-19T08:02:58.470725+0000","last_became_peered":"2023-12-19T08:02:58.470725+0000","last_unstale":"2023-12-19T08:02:58.486898+0000","last_undegraded":"2023-12-19T08:02:58.486898+0000","last_fullsized":"2023-12-19T08:02:58.486898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:39.728694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480160+0000","last_change":"2023-12-19T08:02:58.475283+0000","last_active":"2023-12-19T08:02:58.480160+0000","last_peered":"2023-12-19T08:02:58.480160+0000","last_clean":"2023-12-19T08:02:58.480160+0000","last_became_active":"2023-12-19T08:02:58.474995+0000","last_became_peered":"2023-12-19T08:02:58.474995+0000","last_unstale":"2023-12-19T08:02:58.480160+0000","last_undegraded":"2023-12-19T08:02:58.480160+0000","last_fullsized":"2023-12-19T08:02:58.480160+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:06.759558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480196+0000","last_change":"2023-12-19T08:02:58.475220+0000","last_active":"2023-12-19T08:02:58.480196+0000","last_peered":"2023-12-19T08:02:58.480196+0000","last_clean":"2023-12-19T08:02:58.480196+0000","last_became_active":"2023-12-19T08:02:58.474924+0000","last_became_peered":"2023-12-19T08:02:58.474924+0000","last_unstale":"2023-12-19T08:02:58.480196+0000","last_undegraded":"2023-12-19T08:02:58.480196+0000","last_fullsized":"2023-12-19T08:02:58.480196+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:18:29.983942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.487980+0000","last_change":"2023-12-19T08:02:58.486451+0000","last_active":"2023-12-19T08:02:58.487980+0000","last_peered":"2023-12-19T08:02:58.487980+0000","last_clean":"2023-12-19T08:02:58.487980+0000","last_became_active":"2023-12-19T08:02:58.486153+0000","last_became_peered":"2023-12-19T08:02:58.486153+0000","last_unstale":"2023-12-19T08:02:58.487980+0000","last_undegraded":"2023-12-19T08:02:58.487980+0000","last_fullsized":"2023-12-19T08:02:58.487980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:07:36.374176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476017+0000","last_change":"2023-12-19T08:02:58.474586+0000","last_active":"2023-12-19T08:02:58.476017+0000","last_peered":"2023-12-19T08:02:58.476017+0000","last_clean":"2023-12-19T08:02:58.476017+0000","last_became_active":"2023-12-19T08:02:58.474230+0000","last_became_peered":"2023-12-19T08:02:58.474230+0000","last_unstale":"2023-12-19T08:02:58.476017+0000","last_undegraded":"2023-12-19T08:02:58.476017+0000","last_fullsized":"2023-12-19T08:02:58.476017+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:58.172015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482645+0000","last_change":"2023-12-19T08:02:58.471706+0000","last_active":"2023-12-19T08:02:58.482645+0000","last_peered":"2023-12-19T08:02:58.482645+0000","last_clean":"2023-12-19T08:02:58.482645+0000","last_became_active":"2023-12-19T08:02:58.470015+0000","last_became_peered":"2023-12-19T08:02:58.470015+0000","last_unstale":"2023-12-19T08:02:58.482645+0000","last_undegraded":"2023-12-19T08:02:58.482645+0000","last_fullsized":"2023-12-19T08:02:58.482645+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:51:54.459798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480263+0000","last_change":"2023-12-19T08:02:58.476560+0000","last_active":"2023-12-19T08:02:58.480263+0000","last_peered":"2023-12-19T08:02:58.480263+0000","last_clean":"2023-12-19T08:02:58.480263+0000","last_became_active":"2023-12-19T08:02:58.476295+0000","last_became_peered":"2023-12-19T08:02:58.476295+0000","last_unstale":"2023-12-19T08:02:58.480263+0000","last_undegraded":"2023-12-19T08:02:58.480263+0000","last_fullsized":"2023-12-19T08:02:58.480263+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:07.275271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485205+0000","last_change":"2023-12-19T08:02:58.481489+0000","last_active":"2023-12-19T08:02:58.485205+0000","last_peered":"2023-12-19T08:02:58.485205+0000","last_clean":"2023-12-19T08:02:58.485205+0000","last_became_active":"2023-12-19T08:02:58.480625+0000","last_became_peered":"2023-12-19T08:02:58.480625+0000","last_unstale":"2023-12-19T08:02:58.485205+0000","last_undegraded":"2023-12-19T08:02:58.485205+0000","last_fullsized":"2023-12-19T08:02:58.485205+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:37.495292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482323+0000","last_change":"2023-12-19T08:02:58.478286+0000","last_active":"2023-12-19T08:02:58.482323+0000","last_peered":"2023-12-19T08:02:58.482323+0000","last_clean":"2023-12-19T08:02:58.482323+0000","last_became_active":"2023-12-19T08:02:58.476707+0000","last_became_peered":"2023-12-19T08:02:58.476707+0000","last_unstale":"2023-12-19T08:02:58.482323+0000","last_undegraded":"2023-12-19T08:02:58.482323+0000","last_fullsized":"2023-12-19T08:02:58.482323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.485327+0000","last_change":"2023-12-19T08:02:58.482716+0000","last_active":"2023-12-19T08:02:58.485327+0000","last_peered":"2023-12-19T08:02:58.485327+0000","last_clean":"2023-12-19T08:02:58.485327+0000","last_became_active":"2023-12-19T08:02:58.482404+0000","last_became_peered":"2023-12-19T08:02:58.482404+0000","last_unstale":"2023-12-19T08:02:58.485327+0000","last_undegraded":"2023-12-19T08:02:58.485327+0000","last_fullsized":"2023-12-19T08:02:58.485327+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:07.275271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474191+0000","last_change":"2023-12-19T08:02:58.471817+0000","last_active":"2023-12-19T08:02:58.474191+0000","last_peered":"2023-12-19T08:02:58.474191+0000","last_clean":"2023-12-19T08:02:58.474191+0000","last_became_active":"2023-12-19T08:02:58.470979+0000","last_became_peered":"2023-12-19T08:02:58.470979+0000","last_unstale":"2023-12-19T08:02:58.474191+0000","last_undegraded":"2023-12-19T08:02:58.474191+0000","last_fullsized":"2023-12-19T08:02:58.474191+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:40.194019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"98'4","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965820+0000","last_change":"2023-12-19T08:02:57.937367+0000","last_active":"2023-12-19T08:02:58.965820+0000","last_peered":"2023-12-19T08:02:58.965820+0000","last_clean":"2023-12-19T08:02:58.965820+0000","last_became_active":"2023-12-19T08:02:57.937097+0000","last_became_peered":"2023-12-19T08:02:57.937097+0000","last_unstale":"2023-12-19T08:02:58.965820+0000","last_undegraded":"2023-12-19T08:02:58.965820+0000","last_fullsized":"2023-12-19T08:02:58.965820+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:46.317912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.483081+0000","last_change":"2023-12-19T08:02:58.478025+0000","last_active":"2023-12-19T08:02:58.483081+0000","last_peered":"2023-12-19T08:02:58.483081+0000","last_clean":"2023-12-19T08:02:58.483081+0000","last_became_active":"2023-12-19T08:02:58.475371+0000","last_became_peered":"2023-12-19T08:02:58.475371+0000","last_unstale":"2023-12-19T08:02:58.483081+0000","last_undegraded":"2023-12-19T08:02:58.483081+0000","last_fullsized":"2023-12-19T08:02:58.483081+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:37.495292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957495+0000","last_change":"2023-12-19T08:02:57.946583+0000","last_active":"2023-12-19T08:02:58.957495+0000","last_peered":"2023-12-19T08:02:58.957495+0000","last_clean":"2023-12-19T08:02:58.957495+0000","last_became_active":"2023-12-19T08:02:57.946266+0000","last_became_peered":"2023-12-19T08:02:57.946266+0000","last_unstale":"2023-12-19T08:02:58.957495+0000","last_undegraded":"2023-12-19T08:02:58.957495+0000","last_fullsized":"2023-12-19T08:02:58.957495+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:32.960684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476236+0000","last_change":"2023-12-19T08:02:58.464646+0000","last_active":"2023-12-19T08:02:58.476236+0000","last_peered":"2023-12-19T08:02:58.476236+0000","last_clean":"2023-12-19T08:02:58.476236+0000","last_became_active":"2023-12-19T08:02:58.464350+0000","last_became_peered":"2023-12-19T08:02:58.464350+0000","last_unstale":"2023-12-19T08:02:58.476236+0000","last_undegraded":"2023-12-19T08:02:58.476236+0000","last_fullsized":"2023-12-19T08:02:58.476236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.958293+0000","last_change":"2023-12-19T08:02:57.943884+0000","last_active":"2023-12-19T08:02:58.958293+0000","last_peered":"2023-12-19T08:02:58.958293+0000","last_clean":"2023-12-19T08:02:58.958293+0000","last_became_active":"2023-12-19T08:02:57.943598+0000","last_became_peered":"2023-12-19T08:02:57.943598+0000","last_unstale":"2023-12-19T08:02:58.958293+0000","last_undegraded":"2023-12-19T08:02:58.958293+0000","last_fullsized":"2023-12-19T08:02:58.958293+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:42.218705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482138+0000","last_change":"2023-12-19T08:02:58.470325+0000","last_active":"2023-12-19T08:02:58.482138+0000","last_peered":"2023-12-19T08:02:58.482138+0000","last_clean":"2023-12-19T08:02:58.482138+0000","last_became_active":"2023-12-19T08:02:58.470011+0000","last_became_peered":"2023-12-19T08:02:58.470011+0000","last_unstale":"2023-12-19T08:02:58.482138+0000","last_undegraded":"2023-12-19T08:02:58.482138+0000","last_fullsized":"2023-12-19T08:02:58.482138+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.961597+0000","last_change":"2023-12-19T08:02:57.945609+0000","last_active":"2023-12-19T08:02:58.961597+0000","last_peered":"2023-12-19T08:02:58.961597+0000","last_clean":"2023-12-19T08:02:58.961597+0000","last_became_active":"2023-12-19T08:02:57.945323+0000","last_became_peered":"2023-12-19T08:02:57.945323+0000","last_unstale":"2023-12-19T08:02:58.961597+0000","last_undegraded":"2023-12-19T08:02:58.961597+0000","last_fullsized":"2023-12-19T08:02:58.961597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:14.153026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.480007+0000","last_change":"2023-12-19T08:02:58.474935+0000","last_active":"2023-12-19T08:02:58.480007+0000","last_peered":"2023-12-19T08:02:58.480007+0000","last_clean":"2023-12-19T08:02:58.480007+0000","last_became_active":"2023-12-19T08:02:58.474616+0000","last_became_peered":"2023-12-19T08:02:58.474616+0000","last_unstale":"2023-12-19T08:02:58.480007+0000","last_undegraded":"2023-12-19T08:02:58.480007+0000","last_fullsized":"2023-12-19T08:02:58.480007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:21:48.436227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957229+0000","last_change":"2023-12-19T08:02:57.942738+0000","last_active":"2023-12-19T08:02:58.957229+0000","last_peered":"2023-12-19T08:02:58.957229+0000","last_clean":"2023-12-19T08:02:58.957229+0000","last_became_active":"2023-12-19T08:02:57.942114+0000","last_became_peered":"2023-12-19T08:02:57.942114+0000","last_unstale":"2023-12-19T08:02:58.957229+0000","last_undegraded":"2023-12-19T08:02:58.957229+0000","last_fullsized":"2023-12-19T08:02:58.957229+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:36.209726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.484719+0000","last_change":"2023-12-19T08:02:58.482019+0000","last_active":"2023-12-19T08:02:58.484719+0000","last_peered":"2023-12-19T08:02:58.484719+0000","last_clean":"2023-12-19T08:02:58.484719+0000","last_became_active":"2023-12-19T08:02:58.481376+0000","last_became_peered":"2023-12-19T08:02:58.481376+0000","last_unstale":"2023-12-19T08:02:58.484719+0000","last_undegraded":"2023-12-19T08:02:58.484719+0000","last_fullsized":"2023-12-19T08:02:58.484719+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:46:58.190288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.958502+0000","last_change":"2023-12-19T08:02:57.947665+0000","last_active":"2023-12-19T08:02:58.958502+0000","last_peered":"2023-12-19T08:02:58.958502+0000","last_clean":"2023-12-19T08:02:58.958502+0000","last_became_active":"2023-12-19T08:02:57.947271+0000","last_became_peered":"2023-12-19T08:02:57.947271+0000","last_unstale":"2023-12-19T08:02:58.958502+0000","last_undegraded":"2023-12-19T08:02:58.958502+0000","last_fullsized":"2023-12-19T08:02:58.958502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:38.715762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482176+0000","last_change":"2023-12-19T08:02:58.470079+0000","last_active":"2023-12-19T08:02:58.482176+0000","last_peered":"2023-12-19T08:02:58.482176+0000","last_clean":"2023-12-19T08:02:58.482176+0000","last_became_active":"2023-12-19T08:02:58.469063+0000","last_became_peered":"2023-12-19T08:02:58.469063+0000","last_unstale":"2023-12-19T08:02:58.482176+0000","last_undegraded":"2023-12-19T08:02:58.482176+0000","last_fullsized":"2023-12-19T08:02:58.482176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465127+0000","last_change":"2023-12-19T08:02:57.932512+0000","last_active":"2023-12-19T08:02:58.465127+0000","last_peered":"2023-12-19T08:02:58.465127+0000","last_clean":"2023-12-19T08:02:58.465127+0000","last_became_active":"2023-12-19T08:02:57.932245+0000","last_became_peered":"2023-12-19T08:02:57.932245+0000","last_unstale":"2023-12-19T08:02:58.465127+0000","last_undegraded":"2023-12-19T08:02:58.465127+0000","last_fullsized":"2023-12-19T08:02:58.465127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489100+0000","last_change":"2023-12-19T08:02:58.484926+0000","last_active":"2023-12-19T08:02:58.489100+0000","last_peered":"2023-12-19T08:02:58.489100+0000","last_clean":"2023-12-19T08:02:58.489100+0000","last_became_active":"2023-12-19T08:02:58.484622+0000","last_became_peered":"2023-12-19T08:02:58.484622+0000","last_unstale":"2023-12-19T08:02:58.489100+0000","last_undegraded":"2023-12-19T08:02:58.489100+0000","last_fullsized":"2023-12-19T08:02:58.489100+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:35.696666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467545+0000","last_change":"2023-12-19T08:02:57.932729+0000","last_active":"2023-12-19T08:02:58.467545+0000","last_peered":"2023-12-19T08:02:58.467545+0000","last_clean":"2023-12-19T08:02:58.467545+0000","last_became_active":"2023-12-19T08:02:57.932355+0000","last_became_peered":"2023-12-19T08:02:57.932355+0000","last_unstale":"2023-12-19T08:02:58.467545+0000","last_undegraded":"2023-12-19T08:02:58.467545+0000","last_fullsized":"2023-12-19T08:02:58.467545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:03:24.900257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477601+0000","last_change":"2023-12-19T08:02:58.472746+0000","last_active":"2023-12-19T08:02:58.477601+0000","last_peered":"2023-12-19T08:02:58.477601+0000","last_clean":"2023-12-19T08:02:58.477601+0000","last_became_active":"2023-12-19T08:02:58.471165+0000","last_became_peered":"2023-12-19T08:02:58.471165+0000","last_unstale":"2023-12-19T08:02:58.477601+0000","last_undegraded":"2023-12-19T08:02:58.477601+0000","last_fullsized":"2023-12-19T08:02:58.477601+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:13.085870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464982+0000","last_change":"2023-12-19T08:02:57.947907+0000","last_active":"2023-12-19T08:02:58.464982+0000","last_peered":"2023-12-19T08:02:58.464982+0000","last_clean":"2023-12-19T08:02:58.464982+0000","last_became_active":"2023-12-19T08:02:57.947627+0000","last_became_peered":"2023-12-19T08:02:57.947627+0000","last_unstale":"2023-12-19T08:02:58.464982+0000","last_undegraded":"2023-12-19T08:02:58.464982+0000","last_fullsized":"2023-12-19T08:02:58.464982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:26.014801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486603+0000","last_change":"2023-12-19T08:02:58.483455+0000","last_active":"2023-12-19T08:02:58.486603+0000","last_peered":"2023-12-19T08:02:58.486603+0000","last_clean":"2023-12-19T08:02:58.486603+0000","last_became_active":"2023-12-19T08:02:58.481325+0000","last_became_peered":"2023-12-19T08:02:58.481325+0000","last_unstale":"2023-12-19T08:02:58.486603+0000","last_undegraded":"2023-12-19T08:02:58.486603+0000","last_fullsized":"2023-12-19T08:02:58.486603+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:16:57.879013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464518+0000","last_change":"2023-12-19T08:02:57.942923+0000","last_active":"2023-12-19T08:02:58.464518+0000","last_peered":"2023-12-19T08:02:58.464518+0000","last_clean":"2023-12-19T08:02:58.464518+0000","last_became_active":"2023-12-19T08:02:57.942621+0000","last_became_peered":"2023-12-19T08:02:57.942621+0000","last_unstale":"2023-12-19T08:02:58.464518+0000","last_undegraded":"2023-12-19T08:02:58.464518+0000","last_fullsized":"2023-12-19T08:02:58.464518+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:26:02.397922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467539+0000","last_change":"2023-12-19T08:02:58.462885+0000","last_active":"2023-12-19T08:02:58.467539+0000","last_peered":"2023-12-19T08:02:58.467539+0000","last_clean":"2023-12-19T08:02:58.467539+0000","last_became_active":"2023-12-19T08:02:58.462568+0000","last_became_peered":"2023-12-19T08:02:58.462568+0000","last_unstale":"2023-12-19T08:02:58.467539+0000","last_undegraded":"2023-12-19T08:02:58.467539+0000","last_fullsized":"2023-12-19T08:02:58.467539+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:09:48.640556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"98'3","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965922+0000","last_change":"2023-12-19T08:02:57.944304+0000","last_active":"2023-12-19T08:02:58.965922+0000","last_peered":"2023-12-19T08:02:58.965922+0000","last_clean":"2023-12-19T08:02:58.965922+0000","last_became_active":"2023-12-19T08:02:57.943820+0000","last_became_peered":"2023-12-19T08:02:57.943820+0000","last_unstale":"2023-12-19T08:02:58.965922+0000","last_undegraded":"2023-12-19T08:02:58.965922+0000","last_fullsized":"2023-12-19T08:02:58.965922+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:35:57.134975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482365+0000","last_change":"2023-12-19T08:02:58.477892+0000","last_active":"2023-12-19T08:02:58.482365+0000","last_peered":"2023-12-19T08:02:58.482365+0000","last_clean":"2023-12-19T08:02:58.482365+0000","last_became_active":"2023-12-19T08:02:58.477626+0000","last_became_peered":"2023-12-19T08:02:58.477626+0000","last_unstale":"2023-12-19T08:02:58.482365+0000","last_undegraded":"2023-12-19T08:02:58.482365+0000","last_fullsized":"2023-12-19T08:02:58.482365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"98'6","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.964176+0000","last_change":"2023-12-19T08:02:57.945475+0000","last_active":"2023-12-19T08:02:58.964176+0000","last_peered":"2023-12-19T08:02:58.964176+0000","last_clean":"2023-12-19T08:02:58.964176+0000","last_became_active":"2023-12-19T08:02:57.945022+0000","last_became_peered":"2023-12-19T08:02:57.945022+0000","last_unstale":"2023-12-19T08:02:58.964176+0000","last_undegraded":"2023-12-19T08:02:58.964176+0000","last_fullsized":"2023-12-19T08:02:58.964176+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:30:40.374112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.476066+0000","last_change":"2023-12-19T08:02:58.471519+0000","last_active":"2023-12-19T08:02:58.476066+0000","last_peered":"2023-12-19T08:02:58.476066+0000","last_clean":"2023-12-19T08:02:58.476066+0000","last_became_active":"2023-12-19T08:02:58.471209+0000","last_became_peered":"2023-12-19T08:02:58.471209+0000","last_unstale":"2023-12-19T08:02:58.476066+0000","last_undegraded":"2023-12-19T08:02:58.476066+0000","last_fullsized":"2023-12-19T08:02:58.476066+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:14:27.163320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.961834+0000","last_change":"2023-12-19T08:02:57.938702+0000","last_active":"2023-12-19T08:02:58.961834+0000","last_peered":"2023-12-19T08:02:58.961834+0000","last_clean":"2023-12-19T08:02:58.961834+0000","last_became_active":"2023-12-19T08:02:57.938414+0000","last_became_peered":"2023-12-19T08:02:57.938414+0000","last_unstale":"2023-12-19T08:02:58.961834+0000","last_undegraded":"2023-12-19T08:02:58.961834+0000","last_fullsized":"2023-12-19T08:02:58.961834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488896+0000","last_change":"2023-12-19T08:02:58.480870+0000","last_active":"2023-12-19T08:02:58.488896+0000","last_peered":"2023-12-19T08:02:58.488896+0000","last_clean":"2023-12-19T08:02:58.488896+0000","last_became_active":"2023-12-19T08:02:58.479791+0000","last_became_peered":"2023-12-19T08:02:58.479791+0000","last_unstale":"2023-12-19T08:02:58.488896+0000","last_undegraded":"2023-12-19T08:02:58.488896+0000","last_fullsized":"2023-12-19T08:02:58.488896+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:40.194019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.463712+0000","last_change":"2023-12-19T08:02:57.941949+0000","last_active":"2023-12-19T08:02:58.463712+0000","last_peered":"2023-12-19T08:02:58.463712+0000","last_clean":"2023-12-19T08:02:58.463712+0000","last_became_active":"2023-12-19T08:02:57.941671+0000","last_became_peered":"2023-12-19T08:02:57.941671+0000","last_unstale":"2023-12-19T08:02:58.463712+0000","last_undegraded":"2023-12-19T08:02:58.463712+0000","last_fullsized":"2023-12-19T08:02:58.463712+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489032+0000","last_change":"2023-12-19T08:02:58.483228+0000","last_active":"2023-12-19T08:02:58.489032+0000","last_peered":"2023-12-19T08:02:58.489032+0000","last_clean":"2023-12-19T08:02:58.489032+0000","last_became_active":"2023-12-19T08:02:58.482736+0000","last_became_peered":"2023-12-19T08:02:58.482736+0000","last_unstale":"2023-12-19T08:02:58.489032+0000","last_undegraded":"2023-12-19T08:02:58.489032+0000","last_fullsized":"2023-12-19T08:02:58.489032+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:16:57.879013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468386+0000","last_change":"2023-12-19T08:02:57.938058+0000","last_active":"2023-12-19T08:02:58.468386+0000","last_peered":"2023-12-19T08:02:58.468386+0000","last_clean":"2023-12-19T08:02:58.468386+0000","last_became_active":"2023-12-19T08:02:57.937723+0000","last_became_peered":"2023-12-19T08:02:57.937723+0000","last_unstale":"2023-12-19T08:02:58.468386+0000","last_undegraded":"2023-12-19T08:02:58.468386+0000","last_fullsized":"2023-12-19T08:02:58.468386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:37.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488880+0000","last_change":"2023-12-19T08:02:58.480363+0000","last_active":"2023-12-19T08:02:58.488880+0000","last_peered":"2023-12-19T08:02:58.488880+0000","last_clean":"2023-12-19T08:02:58.488880+0000","last_became_active":"2023-12-19T08:02:58.479645+0000","last_became_peered":"2023-12-19T08:02:58.479645+0000","last_unstale":"2023-12-19T08:02:58.488880+0000","last_undegraded":"2023-12-19T08:02:58.488880+0000","last_fullsized":"2023-12-19T08:02:58.488880+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:14.748409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"98'8","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.957475+0000","last_change":"2023-12-19T08:02:57.930378+0000","last_active":"2023-12-19T08:02:58.957475+0000","last_peered":"2023-12-19T08:02:58.957475+0000","last_clean":"2023-12-19T08:02:58.957475+0000","last_became_active":"2023-12-19T08:02:57.930092+0000","last_became_peered":"2023-12-19T08:02:57.930092+0000","last_unstale":"2023-12-19T08:02:58.957475+0000","last_undegraded":"2023-12-19T08:02:58.957475+0000","last_fullsized":"2023-12-19T08:02:58.957475+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:53:38.715762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464923+0000","last_change":"2023-12-19T08:02:58.460288+0000","last_active":"2023-12-19T08:02:58.464923+0000","last_peered":"2023-12-19T08:02:58.464923+0000","last_clean":"2023-12-19T08:02:58.464923+0000","last_became_active":"2023-12-19T08:02:58.459962+0000","last_became_peered":"2023-12-19T08:02:58.459962+0000","last_unstale":"2023-12-19T08:02:58.464923+0000","last_undegraded":"2023-12-19T08:02:58.464923+0000","last_fullsized":"2023-12-19T08:02:58.464923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:46.774140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.956555+0000","last_change":"2023-12-19T08:02:57.948366+0000","last_active":"2023-12-19T08:02:58.956555+0000","last_peered":"2023-12-19T08:02:58.956555+0000","last_clean":"2023-12-19T08:02:58.956555+0000","last_became_active":"2023-12-19T08:02:57.947337+0000","last_became_peered":"2023-12-19T08:02:57.947337+0000","last_unstale":"2023-12-19T08:02:58.956555+0000","last_undegraded":"2023-12-19T08:02:58.956555+0000","last_fullsized":"2023-12-19T08:02:58.956555+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:26.988789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482762+0000","last_change":"2023-12-19T08:02:58.477676+0000","last_active":"2023-12-19T08:02:58.482762+0000","last_peered":"2023-12-19T08:02:58.482762+0000","last_clean":"2023-12-19T08:02:58.482762+0000","last_became_active":"2023-12-19T08:02:58.477323+0000","last_became_peered":"2023-12-19T08:02:58.477323+0000","last_unstale":"2023-12-19T08:02:58.482762+0000","last_undegraded":"2023-12-19T08:02:58.482762+0000","last_fullsized":"2023-12-19T08:02:58.482762+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:02:29.176763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.959213+0000","last_change":"2023-12-19T08:02:57.948575+0000","last_active":"2023-12-19T08:02:58.959213+0000","last_peered":"2023-12-19T08:02:58.959213+0000","last_clean":"2023-12-19T08:02:58.959213+0000","last_became_active":"2023-12-19T08:02:57.948274+0000","last_became_peered":"2023-12-19T08:02:57.948274+0000","last_unstale":"2023-12-19T08:02:58.959213+0000","last_undegraded":"2023-12-19T08:02:58.959213+0000","last_fullsized":"2023-12-19T08:02:58.959213+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:14.153026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482818+0000","last_change":"2023-12-19T08:02:58.476362+0000","last_active":"2023-12-19T08:02:58.482818+0000","last_peered":"2023-12-19T08:02:58.482818+0000","last_clean":"2023-12-19T08:02:58.482818+0000","last_became_active":"2023-12-19T08:02:58.475826+0000","last_became_peered":"2023-12-19T08:02:58.475826+0000","last_unstale":"2023-12-19T08:02:58.482818+0000","last_undegraded":"2023-12-19T08:02:58.482818+0000","last_fullsized":"2023-12-19T08:02:58.482818+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:11:18.306416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"98'1","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.965846+0000","last_change":"2023-12-19T08:02:57.932863+0000","last_active":"2023-12-19T08:02:58.965846+0000","last_peered":"2023-12-19T08:02:58.965846+0000","last_clean":"2023-12-19T08:02:58.965846+0000","last_became_active":"2023-12-19T08:02:57.931611+0000","last_became_peered":"2023-12-19T08:02:57.931611+0000","last_unstale":"2023-12-19T08:02:58.965846+0000","last_undegraded":"2023-12-19T08:02:58.965846+0000","last_fullsized":"2023-12-19T08:02:58.965846+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:34:12.331722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.478432+0000","last_change":"2023-12-19T08:02:58.470384+0000","last_active":"2023-12-19T08:02:58.478432+0000","last_peered":"2023-12-19T08:02:58.478432+0000","last_clean":"2023-12-19T08:02:58.478432+0000","last_became_active":"2023-12-19T08:02:58.470040+0000","last_became_peered":"2023-12-19T08:02:58.470040+0000","last_unstale":"2023-12-19T08:02:58.478432+0000","last_undegraded":"2023-12-19T08:02:58.478432+0000","last_fullsized":"2023-12-19T08:02:58.478432+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:14:39.780359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.460811+0000","last_change":"2023-12-19T08:02:57.946209+0000","last_active":"2023-12-19T08:02:58.460811+0000","last_peered":"2023-12-19T08:02:58.460811+0000","last_clean":"2023-12-19T08:02:58.460811+0000","last_became_active":"2023-12-19T08:02:57.945842+0000","last_became_peered":"2023-12-19T08:02:57.945842+0000","last_unstale":"2023-12-19T08:02:58.460811+0000","last_undegraded":"2023-12-19T08:02:58.460811+0000","last_fullsized":"2023-12-19T08:02:58.460811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:26.150714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477124+0000","last_change":"2023-12-19T08:02:58.471203+0000","last_active":"2023-12-19T08:02:58.477124+0000","last_peered":"2023-12-19T08:02:58.477124+0000","last_clean":"2023-12-19T08:02:58.477124+0000","last_became_active":"2023-12-19T08:02:58.470929+0000","last_became_peered":"2023-12-19T08:02:58.470929+0000","last_unstale":"2023-12-19T08:02:58.477124+0000","last_undegraded":"2023-12-19T08:02:58.477124+0000","last_fullsized":"2023-12-19T08:02:58.477124+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:24.541849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.955322+0000","last_change":"2023-12-19T08:02:57.939030+0000","last_active":"2023-12-19T08:02:58.955322+0000","last_peered":"2023-12-19T08:02:58.955322+0000","last_clean":"2023-12-19T08:02:58.955322+0000","last_became_active":"2023-12-19T08:02:57.938689+0000","last_became_peered":"2023-12-19T08:02:57.938689+0000","last_unstale":"2023-12-19T08:02:58.955322+0000","last_undegraded":"2023-12-19T08:02:58.955322+0000","last_fullsized":"2023-12-19T08:02:58.955322+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:23:28.297819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477910+0000","last_change":"2023-12-19T08:02:58.471400+0000","last_active":"2023-12-19T08:02:58.477910+0000","last_peered":"2023-12-19T08:02:58.477910+0000","last_clean":"2023-12-19T08:02:58.477910+0000","last_became_active":"2023-12-19T08:02:58.471052+0000","last_became_peered":"2023-12-19T08:02:58.471052+0000","last_unstale":"2023-12-19T08:02:58.477910+0000","last_undegraded":"2023-12-19T08:02:58.477910+0000","last_fullsized":"2023-12-19T08:02:58.477910+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:36:58.172015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.447334+0000","last_change":"2023-12-19T08:02:57.940846+0000","last_active":"2023-12-19T08:02:58.447334+0000","last_peered":"2023-12-19T08:02:58.447334+0000","last_clean":"2023-12-19T08:02:58.447334+0000","last_became_active":"2023-12-19T08:02:57.940539+0000","last_became_peered":"2023-12-19T08:02:57.940539+0000","last_unstale":"2023-12-19T08:02:58.447334+0000","last_undegraded":"2023-12-19T08:02:58.447334+0000","last_fullsized":"2023-12-19T08:02:58.447334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:58:52.740845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486943+0000","last_change":"2023-12-19T08:02:58.472977+0000","last_active":"2023-12-19T08:02:58.486943+0000","last_peered":"2023-12-19T08:02:58.486943+0000","last_clean":"2023-12-19T08:02:58.486943+0000","last_became_active":"2023-12-19T08:02:58.472671+0000","last_became_peered":"2023-12-19T08:02:58.472671+0000","last_unstale":"2023-12-19T08:02:58.486943+0000","last_undegraded":"2023-12-19T08:02:58.486943+0000","last_fullsized":"2023-12-19T08:02:58.486943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:43.231637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.447016+0000","last_change":"2023-12-19T08:02:57.941697+0000","last_active":"2023-12-19T08:02:58.447016+0000","last_peered":"2023-12-19T08:02:58.447016+0000","last_clean":"2023-12-19T08:02:58.447016+0000","last_became_active":"2023-12-19T08:02:57.940734+0000","last_became_peered":"2023-12-19T08:02:57.940734+0000","last_unstale":"2023-12-19T08:02:58.447016+0000","last_undegraded":"2023-12-19T08:02:58.447016+0000","last_fullsized":"2023-12-19T08:02:58.447016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:28.318167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.488460+0000","last_change":"2023-12-19T08:02:58.480514+0000","last_active":"2023-12-19T08:02:58.488460+0000","last_peered":"2023-12-19T08:02:58.488460+0000","last_clean":"2023-12-19T08:02:58.488460+0000","last_became_active":"2023-12-19T08:02:58.480038+0000","last_became_peered":"2023-12-19T08:02:58.480038+0000","last_unstale":"2023-12-19T08:02:58.488460+0000","last_undegraded":"2023-12-19T08:02:58.488460+0000","last_fullsized":"2023-12-19T08:02:58.488460+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"98'2","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.960465+0000","last_change":"2023-12-19T08:02:57.942853+0000","last_active":"2023-12-19T08:02:58.960465+0000","last_peered":"2023-12-19T08:02:58.960465+0000","last_clean":"2023-12-19T08:02:58.960465+0000","last_became_active":"2023-12-19T08:02:57.942557+0000","last_became_peered":"2023-12-19T08:02:57.942557+0000","last_unstale":"2023-12-19T08:02:58.960465+0000","last_undegraded":"2023-12-19T08:02:58.960465+0000","last_fullsized":"2023-12-19T08:02:58.960465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486924+0000","last_change":"2023-12-19T08:02:58.477420+0000","last_active":"2023-12-19T08:02:58.486924+0000","last_peered":"2023-12-19T08:02:58.486924+0000","last_clean":"2023-12-19T08:02:58.486924+0000","last_became_active":"2023-12-19T08:02:58.475627+0000","last_became_peered":"2023-12-19T08:02:58.475627+0000","last_unstale":"2023-12-19T08:02:58.486924+0000","last_undegraded":"2023-12-19T08:02:58.486924+0000","last_fullsized":"2023-12-19T08:02:58.486924+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467094+0000","last_change":"2023-12-19T08:02:57.934521+0000","last_active":"2023-12-19T08:02:58.467094+0000","last_peered":"2023-12-19T08:02:58.467094+0000","last_clean":"2023-12-19T08:02:58.467094+0000","last_became_active":"2023-12-19T08:02:57.932148+0000","last_became_peered":"2023-12-19T08:02:57.932148+0000","last_unstale":"2023-12-19T08:02:58.467094+0000","last_undegraded":"2023-12-19T08:02:58.467094+0000","last_fullsized":"2023-12-19T08:02:58.467094+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:54.256469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.481721+0000","last_change":"2023-12-19T08:02:58.468847+0000","last_active":"2023-12-19T08:02:58.481721+0000","last_peered":"2023-12-19T08:02:58.481721+0000","last_clean":"2023-12-19T08:02:58.481721+0000","last_became_active":"2023-12-19T08:02:58.468517+0000","last_became_peered":"2023-12-19T08:02:58.468517+0000","last_unstale":"2023-12-19T08:02:58.481721+0000","last_undegraded":"2023-12-19T08:02:58.481721+0000","last_fullsized":"2023-12-19T08:02:58.481721+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:55.269402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464357+0000","last_change":"2023-12-19T08:02:57.941833+0000","last_active":"2023-12-19T08:02:58.464357+0000","last_peered":"2023-12-19T08:02:58.464357+0000","last_clean":"2023-12-19T08:02:58.464357+0000","last_became_active":"2023-12-19T08:02:57.941493+0000","last_became_peered":"2023-12-19T08:02:57.941493+0000","last_unstale":"2023-12-19T08:02:58.464357+0000","last_undegraded":"2023-12-19T08:02:58.464357+0000","last_fullsized":"2023-12-19T08:02:58.464357+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:47:13.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.477064+0000","last_change":"2023-12-19T08:02:58.473356+0000","last_active":"2023-12-19T08:02:58.477064+0000","last_peered":"2023-12-19T08:02:58.477064+0000","last_clean":"2023-12-19T08:02:58.477064+0000","last_became_active":"2023-12-19T08:02:58.471319+0000","last_became_peered":"2023-12-19T08:02:58.471319+0000","last_unstale":"2023-12-19T08:02:58.477064+0000","last_undegraded":"2023-12-19T08:02:58.477064+0000","last_fullsized":"2023-12-19T08:02:58.477064+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:56:39.326794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.464836+0000","last_change":"2023-12-19T08:02:57.946724+0000","last_active":"2023-12-19T08:02:58.464836+0000","last_peered":"2023-12-19T08:02:58.464836+0000","last_clean":"2023-12-19T08:02:58.464836+0000","last_became_active":"2023-12-19T08:02:57.946416+0000","last_became_peered":"2023-12-19T08:02:57.946416+0000","last_unstale":"2023-12-19T08:02:58.464836+0000","last_undegraded":"2023-12-19T08:02:58.464836+0000","last_fullsized":"2023-12-19T08:02:58.464836+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:13.735477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.486934+0000","last_change":"2023-12-19T08:02:58.476374+0000","last_active":"2023-12-19T08:02:58.486934+0000","last_peered":"2023-12-19T08:02:58.486934+0000","last_clean":"2023-12-19T08:02:58.486934+0000","last_became_active":"2023-12-19T08:02:58.475541+0000","last_became_peered":"2023-12-19T08:02:58.475541+0000","last_unstale":"2023-12-19T08:02:58.486934+0000","last_undegraded":"2023-12-19T08:02:58.486934+0000","last_fullsized":"2023-12-19T08:02:58.486934+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.469542+0000","last_change":"2023-12-19T08:02:57.942781+0000","last_active":"2023-12-19T08:02:58.469542+0000","last_peered":"2023-12-19T08:02:58.469542+0000","last_clean":"2023-12-19T08:02:58.469542+0000","last_became_active":"2023-12-19T08:02:57.942490+0000","last_became_peered":"2023-12-19T08:02:57.942490+0000","last_unstale":"2023-12-19T08:02:58.469542+0000","last_undegraded":"2023-12-19T08:02:58.469542+0000","last_fullsized":"2023-12-19T08:02:58.469542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:59:36.209726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.489494+0000","last_change":"2023-12-19T08:02:58.488636+0000","last_active":"2023-12-19T08:02:58.489494+0000","last_peered":"2023-12-19T08:02:58.489494+0000","last_clean":"2023-12-19T08:02:58.489494+0000","last_became_active":"2023-12-19T08:02:58.488323+0000","last_became_peered":"2023-12-19T08:02:58.488323+0000","last_unstale":"2023-12-19T08:02:58.489494+0000","last_undegraded":"2023-12-19T08:02:58.489494+0000","last_fullsized":"2023-12-19T08:02:58.489494+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:28.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.467801+0000","last_change":"2023-12-19T08:02:57.941368+0000","last_active":"2023-12-19T08:02:58.467801+0000","last_peered":"2023-12-19T08:02:58.467801+0000","last_clean":"2023-12-19T08:02:58.467801+0000","last_became_active":"2023-12-19T08:02:57.939982+0000","last_became_peered":"2023-12-19T08:02:57.939982+0000","last_unstale":"2023-12-19T08:02:58.467801+0000","last_undegraded":"2023-12-19T08:02:58.467801+0000","last_fullsized":"2023-12-19T08:02:58.467801+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:53.330499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.482956+0000","last_change":"2023-12-19T08:02:58.479112+0000","last_active":"2023-12-19T08:02:58.482956+0000","last_peered":"2023-12-19T08:02:58.482956+0000","last_clean":"2023-12-19T08:02:58.482956+0000","last_became_active":"2023-12-19T08:02:58.477115+0000","last_became_peered":"2023-12-19T08:02:58.477115+0000","last_unstale":"2023-12-19T08:02:58.482956+0000","last_undegraded":"2023-12-19T08:02:58.482956+0000","last_fullsized":"2023-12-19T08:02:58.482956+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:20:27.027733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468931+0000","last_change":"2023-12-19T08:02:57.947446+0000","last_active":"2023-12-19T08:02:58.468931+0000","last_peered":"2023-12-19T08:02:58.468931+0000","last_clean":"2023-12-19T08:02:58.468931+0000","last_became_active":"2023-12-19T08:02:57.946835+0000","last_became_peered":"2023-12-19T08:02:57.946835+0000","last_unstale":"2023-12-19T08:02:58.468931+0000","last_undegraded":"2023-12-19T08:02:58.468931+0000","last_fullsized":"2023-12-19T08:02:58.468931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:47.543596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.468541+0000","last_change":"2023-12-19T08:02:58.467190+0000","last_active":"2023-12-19T08:02:58.468541+0000","last_peered":"2023-12-19T08:02:58.468541+0000","last_clean":"2023-12-19T08:02:58.468541+0000","last_became_active":"2023-12-19T08:02:58.466060+0000","last_became_peered":"2023-12-19T08:02:58.466060+0000","last_unstale":"2023-12-19T08:02:58.468541+0000","last_undegraded":"2023-12-19T08:02:58.468541+0000","last_fullsized":"2023-12-19T08:02:58.468541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:46:33.973617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.459409+0000","last_change":"2023-12-19T08:02:57.940724+0000","last_active":"2023-12-19T08:02:58.459409+0000","last_peered":"2023-12-19T08:02:58.459409+0000","last_clean":"2023-12-19T08:02:58.459409+0000","last_became_active":"2023-12-19T08:02:57.940447+0000","last_became_peered":"2023-12-19T08:02:57.940447+0000","last_unstale":"2023-12-19T08:02:58.459409+0000","last_undegraded":"2023-12-19T08:02:58.459409+0000","last_fullsized":"2023-12-19T08:02:58.459409+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:45.430449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.474942+0000","last_change":"2023-12-19T08:02:58.471852+0000","last_active":"2023-12-19T08:02:58.474942+0000","last_peered":"2023-12-19T08:02:58.474942+0000","last_clean":"2023-12-19T08:02:58.474942+0000","last_became_active":"2023-12-19T08:02:58.471576+0000","last_became_peered":"2023-12-19T08:02:58.471576+0000","last_unstale":"2023-12-19T08:02:58.474942+0000","last_undegraded":"2023-12-19T08:02:58.474942+0000","last_fullsized":"2023-12-19T08:02:58.474942+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:57.899360+0000","last_clean_scrub_stamp":"2023-12-19T08:02:57.899360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:28:55.839090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.448116+0000","last_change":"2023-12-19T08:02:57.945400+0000","last_active":"2023-12-19T08:02:58.448116+0000","last_peered":"2023-12-19T08:02:58.448116+0000","last_clean":"2023-12-19T08:02:58.448116+0000","last_became_active":"2023-12-19T08:02:57.945004+0000","last_became_peered":"2023-12-19T08:02:57.945004+0000","last_unstale":"2023-12-19T08:02:58.448116+0000","last_undegraded":"2023-12-19T08:02:58.448116+0000","last_fullsized":"2023-12-19T08:02:58.448116+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:02:56.886428+0000","last_clean_scrub_stamp":"2023-12-19T08:02:56.886428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:53:51.160105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"87'183","reported_seq":463,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-12-19T08:02:58.465010+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:02:58.465010+0000","last_peered":"2023-12-19T08:02:58.465010+0000","last_clean":"2023-12-19T08:02:58.465010+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:02:58.465010+0000","last_undegraded":"2023-12-19T08:02:58.465010+0000","last_fullsized":"2023-12-19T08:02:58.465010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311344,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060496,"statfs":{"total":96636764160,"available":96317947904,"internally_reserved":0,"allocated":1810432,"data_stored":1213502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2450000000000001,"15min":1.2470000000000001},"min":{"1min":0.877,"5min":0.69699999999999995,"15min":0.59599999999999997},"max":{"1min":1.5880000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.458},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.3120000000000001,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.749,"15min":0.67200000000000004},"max":{"1min":4.6210000000000004,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.397}]},{"osd":1,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.1899999999999999,"15min":1.1639999999999999},"min":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.66100000000000003},"max":{"1min":4.9160000000000004,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.113},{"interface":"front","average":{"1min":1.26,"5min":1.167,"15min":1.1659999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.8969999999999998,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.359}]},{"osd":2,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.1910000000000001,"15min":1.1759999999999999},"min":{"1min":0.92400000000000004,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":4.9409999999999998,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.024},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.268,"15min":1.2450000000000001},"min":{"1min":0.97599999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5149999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3069999999999999,"15min":1.296},"min":{"1min":0.95799999999999996,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":4.8220000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.3700000000000001,"15min":1.3540000000000001},"min":{"1min":0.97299999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":4.4980000000000002,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.5369999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.272,"15min":1.228},"min":{"1min":0.71899999999999997,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.3140000000000001,"15min":1.2490000000000001},"min":{"1min":0.91900000000000004,"5min":0.76500000000000001,"15min":0.64900000000000002},"max":{"1min":6.8360000000000003,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.639}]},{"osd":5,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.3,"15min":1.2509999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.3959999999999999,"15min":1.363},"min":{"1min":0.98199999999999998,"5min":0.751,"15min":0.751},"max":{"1min":2.004,"5min":2.2120000000000002,"15min":2.3010000000000002},"last":1.679}]},{"osd":6,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.4990000000000001,"15min":1.429},"min":{"1min":1.1759999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.605},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.423,"15min":1.395},"min":{"1min":0.95999999999999996,"5min":0.90200000000000002,"15min":0.58299999999999996},"max":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.496}]}]},{"osd":6,"up_from":13,"seq":55834575037,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":617520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.071,"15min":1.103},"min":{"1min":0.75800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.472,"5min":1.5569999999999999,"15min":1.665},"last":1.137},{"interface":"front","average":{"1min":1.109,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.373,"5min":1.756,"15min":1.756},"last":0.70099999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.75800000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4770000000000001,"5min":1.675,"15min":1.881},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.161,"15min":1.161},"min":{"1min":0.745,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4279999999999999,"5min":1.542,"15min":1.7989999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.675,"5min":1.847,"15min":1.847},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1970000000000001,"15min":1.2130000000000001},"min":{"1min":0.70799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4259999999999999,"5min":1.694,"15min":1.7929999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2450000000000001,"15min":1.2669999999999999},"min":{"1min":0.77600000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0659999999999998},"last":1.052},{"interface":"front","average":{"1min":1.252,"5min":1.3,"15min":1.3100000000000001},"min":{"1min":0.80900000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7889999999999999,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.258}]},{"osd":4,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1990000000000001,"15min":1.1839999999999999},"min":{"1min":0.80900000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5900000000000001,"5min":1.7130000000000001,"15min":2.137},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.629,"5min":1.974,"15min":2.218},"last":1.2889999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3120000000000001,"15min":1.2929999999999999},"min":{"1min":0.78300000000000003,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3109999999999999},"min":{"1min":0.83999999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.663,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.93999999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":2.2029999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.448,"15min":1.425},"min":{"1min":1.1379999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.137,"15min":2.2639999999999998},"last":1.4399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575037,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298464,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073376,"statfs":{"total":96636764160,"available":96331137024,"internally_reserved":0,"allocated":1204224,"data_stored":616018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.546,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.093,"5min":1.2410000000000001,"15min":1.264},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.50700000000000001},"max":{"1min":1.5940000000000001,"5min":1.954,"15min":2.3140000000000001},"last":0.89500000000000002}]},{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.341,"5min":1.585,"15min":3.5409999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.77500000000000002,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.647,"5min":2.7679999999999998,"15min":3.113},"last":0.95499999999999996}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1759999999999999,"15min":1.2010000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66000000000000003},"max":{"1min":1.6739999999999999,"5min":1.821,"15min":3.339},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.175,"15min":1.2110000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4279999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":0.79900000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.228,"15min":1.2569999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.4630000000000001,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.93400000000000005,"5min":0.88400000000000001,"15min":0.58199999999999996},"max":{"1min":1.6359999999999999,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.3260000000000001}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.202,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.232},{"interface":"front","average":{"1min":1.286,"5min":1.2629999999999999,"15min":1.2929999999999999},"min":{"1min":0.96899999999999997,"5min":0.70299999999999996,"15min":0.59199999999999997},"max":{"1min":1.621,"5min":2.843,"15min":4.6310000000000002},"last":1.026}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.81499999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.728,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.367,"15min":1.3700000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.70599999999999996},"max":{"1min":1.96,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.379}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3939999999999999,"15min":1.427},"min":{"1min":0.88,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.6739999999999999,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.46,"15min":1.462},"min":{"1min":1.038,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.8200000000000001,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.482}]}]},{"osd":4,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302588,"kb_used_data":1204,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069252,"statfs":{"total":96636764160,"available":96326914048,"internally_reserved":0,"allocated":1232896,"data_stored":624752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1950000000000001,"15min":1.214},"min":{"1min":0.70499999999999996,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.4490000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.056},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.94999999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.742,"5min":1.742,"15min":2.738},"last":1.8680000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.159,"15min":1.177},"min":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.556,"5min":1.7589999999999999,"15min":2.589},"last":1.403},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1060000000000001,"15min":1.143},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54300000000000004},"max":{"1min":1.407,"5min":1.528,"15min":2.2589999999999999},"last":0.877}]},{"osd":2,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.085,"15min":1.1359999999999999},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5149999999999999,"15min":2.4569999999999999},"last":1.351},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.2350000000000001},"min":{"1min":0.89000000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.524,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.212}]},{"osd":3,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2509999999999999,"15min":1.2829999999999999},"min":{"1min":0.70999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5980000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2969999999999999,"15min":1.3360000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.81899999999999995},"max":{"1min":1.5780000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.754}]},{"osd":5,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1739999999999999,"15min":1.2170000000000001},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5549999999999999,"5min":1.8149999999999999,"15min":3.556},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.2150000000000001},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6850000000000001,"5min":1.764,"15min":3.2599999999999998},"last":1.1679999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7350000000000001,"5min":1.748,"15min":2.645},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.3839999999999999},"min":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6299999999999999,"5min":2.0049999999999999,"15min":2.923},"last":2.0880000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3580000000000001},"min":{"1min":0.82699999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7410000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.476},{"interface":"front","average":{"1min":1.337,"5min":1.3859999999999999,"15min":1.411},"min":{"1min":0.80200000000000005,"5min":0.73099999999999998,"15min":0.70999999999999996},"max":{"1min":1.829,"5min":1.966,"15min":2.8460000000000001},"last":1.919}]}]},{"osd":3,"up_from":13,"seq":55834575037,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306668,"kb_used_data":1188,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065172,"statfs":{"total":96636764160,"available":96322736128,"internally_reserved":0,"allocated":1216512,"data_stored":623666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.85199999999999998,"15min":0.875},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.5539999999999998,"5min":2.5539999999999998,"15min":5.1079999999999997},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93200000000000005,"15min":0.97299999999999998},"min":{"1min":0.66100000000000003,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.962,"5min":1.962,"15min":6.1900000000000004},"last":0.95399999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.91700000000000004,"15min":0.93899999999999995},"min":{"1min":0.60999999999999999,"5min":0.54500000000000004,"15min":0.45600000000000002},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":4.8879999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.032,"15min":1.0680000000000001},"min":{"1min":0.49399999999999999,"5min":0.442,"15min":0.442},"max":{"1min":2.423,"5min":2.423,"15min":8.0830000000000002},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.05,"15min":1.081},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":4.7519999999999998},"last":1.292},{"interface":"front","average":{"1min":1.252,"5min":1.153,"15min":1.1799999999999999},"min":{"1min":0.57299999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.48,"5min":2.48,"15min":5.3330000000000002},"last":1.0569999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2350000000000001,"15min":1.274},"min":{"1min":0.81200000000000006,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":4.7130000000000001},"last":1.601},{"interface":"front","average":{"1min":1.238,"5min":1.222,"15min":1.268},"min":{"1min":0.86499999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.536,"5min":1.6240000000000001,"15min":5.0220000000000002},"last":1.325}]},{"osd":5,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2490000000000001,"15min":1.2969999999999999},"min":{"1min":0.86499999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.996,"5min":1.996,"15min":4.9889999999999999},"last":1.242},{"interface":"front","average":{"1min":1.427,"5min":1.3500000000000001,"15min":1.3779999999999999},"min":{"1min":1.1419999999999999,"5min":0.90500000000000003,"15min":0.65800000000000003},"max":{"1min":2.161,"5min":2.161,"15min":8.5299999999999994},"last":1.163}]},{"osd":6,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3919999999999999,"15min":1.4199999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":7.2750000000000004},"last":2.4830000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3859999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.79100000000000004},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":4.6550000000000002},"last":1.6579999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4550000000000001,"15min":1.51},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.84799999999999998},"max":{"1min":2.504,"5min":2.504,"15min":16.081},"last":2.7530000000000001},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.4390000000000001,"15min":1.4750000000000001},"min":{"1min":1.0940000000000001,"5min":0.92700000000000005,"15min":0.92000000000000004},"max":{"1min":1.71,"5min":1.833,"15min":15.285},"last":2.3359999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302568,"kb_used_data":1184,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069272,"statfs":{"total":96636764160,"available":96326934528,"internally_reserved":0,"allocated":1212416,"data_stored":616698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.91200000000000003,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.56399999999999995,"15min":0.437},"max":{"1min":1.1910000000000001,"5min":1.776,"15min":4.1479999999999997},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.70299999999999996,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.7769999999999999,"5min":1.8460000000000001,"15min":4.093},"last":1.05}]},{"osd":1,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.96899999999999997,"15min":0.96499999999999997},"min":{"1min":0.78600000000000003,"5min":0.60599999999999998,"15min":0.51800000000000002},"max":{"1min":1.327,"5min":1.5620000000000001,"15min":3.8700000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.004,"15min":1.024},"min":{"1min":0.57199999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.835,"5min":1.835,"15min":3.9089999999999998},"last":0.91800000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98799999999999999,"15min":0.96699999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.667},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0980000000000001,"15min":1.111},"min":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.4670000000000001,"5min":1.629,"15min":1.7250000000000001},"last":1.1339999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.3260000000000001,"15min":1.331},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.79300000000000004},"max":{"1min":1.47,"5min":1.7669999999999999,"15min":3.9239999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.2949999999999999,"15min":1.3120000000000001},"min":{"1min":0.94299999999999995,"5min":0.89400000000000002,"15min":0.82099999999999995},"max":{"1min":1.925,"5min":1.925,"15min":4.0170000000000003},"last":1.486}]},{"osd":5,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3380000000000001,"15min":1.3560000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":3.948},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.2769999999999999,"15min":1.256},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.5760000000000001,"5min":1.6930000000000001,"15min":1.879},"last":1.085}]},{"osd":6,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2969999999999999,"15min":1.3},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.752},"max":{"1min":1.5669999999999999,"5min":1.7849999999999999,"15min":2.0760000000000001},"last":1.2},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3660000000000001,"15min":1.383},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.85399999999999998},"max":{"1min":1.548,"5min":1.867,"15min":3.9620000000000002},"last":1.359}]},{"osd":7,"last update":"Tue Dec 19 08:02:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3660000000000001,"15min":1.405},"min":{"1min":1.034,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":3.9809999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.3520000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.97399999999999998,"15min":0.88200000000000001},"max":{"1min":1.5600000000000001,"5min":1.8620000000000001,"15min":2.206},"last":1.26}]}]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306672,"kb_used_data":1192,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065168,"statfs":{"total":96636764160,"available":96322732032,"internally_reserved":0,"allocated":1220608,"data_stored":623224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":0.85699999999999998},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.40100000000000002},"max":{"1min":1.1539999999999999,"5min":1.661,"15min":1.661},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.95399999999999996,"15min":0.94999999999999996},"min":{"1min":0.60399999999999998,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":2.2050000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.76100000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":0.92800000000000005},"min":{"1min":0.61499999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.96,"5min":2.96,"15min":2.96},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1120000000000001,"15min":1.0649999999999999},"min":{"1min":0.67500000000000004,"5min":0.56100000000000005,"15min":0.46300000000000002},"max":{"1min":2.323,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.90700000000000003}]},{"osd":3,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2070000000000001,"15min":1.165},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.2530000000000001,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.117,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.46800000000000003},"max":{"1min":2.1299999999999999,"5min":2.407,"15min":2.407},"last":0.55500000000000005}]},{"osd":4,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1930000000000001,"15min":1.198},"min":{"1min":0.877,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.877,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.359,"15min":1.3160000000000001},"min":{"1min":1.042,"5min":0.95999999999999996,"15min":0.64400000000000002},"max":{"1min":2.2890000000000001,"5min":3.04,"15min":3.04},"last":1.0880000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.2989999999999999,"15min":1.2849999999999999},"min":{"1min":0.90100000000000002,"5min":0.87,"15min":0.65100000000000002},"max":{"1min":1.996,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.194},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3200000000000001,"15min":1.2969999999999999},"min":{"1min":1.0089999999999999,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.9239999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.03}]},{"osd":6,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4299999999999999,"15min":1.393},"min":{"1min":1.1000000000000001,"5min":0.96799999999999997,"15min":0.72799999999999998},"max":{"1min":2.3559999999999999,"5min":3.089,"15min":3.089},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.393,"15min":1.3720000000000001},"min":{"1min":0.96299999999999997,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":2.169,"5min":2.46,"15min":2.6230000000000002},"last":1.129}]},{"osd":7,"last update":"Tue Dec 19 08:02:32 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4399999999999999,"15min":1.4319999999999999},"min":{"1min":1.1759999999999999,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":2.0430000000000001,"5min":2.504,"15min":3.6080000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.3999999999999999,"15min":1.4019999999999999},"min":{"1min":1.0469999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.7290000000000001,"5min":2.7290000000000001,"15min":4.101},"last":1.079}]}]},{"osd":0,"up_from":13,"seq":55834575037,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303144,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068696,"statfs":{"total":96636764160,"available":96326344704,"internally_reserved":0,"allocated":1802240,"data_stored":1212469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.262,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91600000000000004,"15min":0.93799999999999994},"min":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.385,"5min":1.385,"15min":6.0309999999999997},"last":0.90300000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.95999999999999996,"15min":0.97999999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.054},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.044,"15min":1.03},"min":{"1min":0.75700000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3700000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.1699999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.96099999999999997},"min":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.51400000000000001},"max":{"1min":1.48,"5min":1.5529999999999999,"15min":1.671},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.115,"15min":1.129},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.514,"5min":1.663,"15min":1.6719999999999999},"last":1.1399999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3129999999999999,"15min":1.3080000000000001},"min":{"1min":0.86499999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6000000000000001,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.5840000000000001,"5min":1.855,"15min":2.141},"last":1.101}]},{"osd":5,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.351,"15min":1.3600000000000001},"min":{"1min":0.83399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.6819999999999999,"5min":2.085,"15min":2.117},"last":1.389},{"interface":"front","average":{"1min":1.339,"5min":1.357,"15min":1.361},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.843,"5min":2.125,"15min":2.125},"last":1.264}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.276,"15min":1.268},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.77400000000000002},"max":{"1min":1.5329999999999999,"5min":1.97,"15min":1.97},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3200000000000001,"15min":1.3220000000000001},"min":{"1min":0.91900000000000004,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.712,"5min":1.871,"15min":1.9530000000000001},"last":1.657}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.341,"15min":1.3240000000000001},"min":{"1min":0.97799999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.8009999999999999,"5min":1.887,"15min":2.0539999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3720000000000001,"15min":1.3759999999999999},"min":{"1min":0.88600000000000001,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.714,"5min":2.0059999999999998,"15min":2.016},"last":1.712}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:03:03.632 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:03:03.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:03:03.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 Processor -- start 2023-12-19T08:03:03.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 -- start start 2023-12-19T08:03:03.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe7680caa10 0x7fe768159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7680ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680cbbb0 con 0x7fe7680ce2d0 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76806dc90 con 0x7fe7680caa10 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76de0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680cafa0 con 0x7fe7680cc040 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe7680caa10 0x7fe768159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe7680caa10 0x7fe768159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37384/0 (socket says 172.21.15.111:37384) 2023-12-19T08:03:03.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 -- 172.21.15.111:0/3266555315 learned_addr learned my addr 172.21.15.111:0/3266555315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:03.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe76d60b700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7680ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 -- 172.21.15.111:0/3266555315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 msgr2=0x7fe7680ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:03.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7680ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 -- 172.21.15.111:0/3266555315 >> v1:172.21.15.143:6789/0 conn(0x7fe7680ce2d0 legacy=0x7fe768158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.841+0000 7fe767fff700 1 -- 172.21.15.111:0/3266555315 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680cb280 con 0x7fe7680caa10 2023-12-19T08:03:03.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe767fff700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680caa10 0x7fe768159010 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe7540099b0 tx=0x7fe754009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af5c9beb1497bee server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe7677fe700 1 -- 172.21.15.111:0/3266555315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754009ea0 con 0x7fe7680caa10 2023-12-19T08:03:03.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe7677fe700 1 -- 172.21.15.111:0/3266555315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75400b860 con 0x7fe7680caa10 2023-12-19T08:03:03.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe7677fe700 1 -- 172.21.15.111:0/3266555315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75400bc00 con 0x7fe7680caa10 2023-12-19T08:03:03.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 -- 172.21.15.111:0/3266555315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680caa10 msgr2=0x7fe768159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680caa10 0x7fe768159010 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe7540099b0 tx=0x7fe754009250 comp rx=0 tx=0).stop 2023-12-19T08:03:03.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 -- 172.21.15.111:0/3266555315 shutdown_connections 2023-12-19T08:03:03.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7680ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 --2- 172.21.15.111:0/3266555315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680caa10 0x7fe768159010 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 -- 172.21.15.111:0/3266555315 >> 172.21.15.111:0/3266555315 conn(0x7fe768067bb0 msgr2=0x7fe7680cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:03.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 -- 172.21.15.111:0/3266555315 shutdown_connections 2023-12-19T08:03:03.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 -- 172.21.15.111:0/3266555315 wait complete. 2023-12-19T08:03:03.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.845+0000 7fe76de0c700 1 Processor -- start 2023-12-19T08:03:03.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- start start 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7680caa10 0x7fe7681498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7681586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe768159c50 con 0x7fe7680ce2d0 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe768149de0 con 0x7fe7680caa10 2023-12-19T08:03:03.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe768159040 con 0x7fe7680cc040 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37388/0 (socket says 172.21.15.111:37388) 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 -- 172.21.15.111:0/362469697 learned_addr learned my addr 172.21.15.111:0/362469697 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76ce0a700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7680caa10 0x7fe7681498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe767fff700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7681586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 -- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 msgr2=0x7fe7681586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7681586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 -- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7680caa10 msgr2=0x7fe7681498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7680caa10 0x7fe7681498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 -- 172.21.15.111:0/362469697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540094f0 con 0x7fe7680ce2d0 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76d60b700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe760009e90 tx=0x7fe76000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000cc20 con 0x7fe7680ce2d0 2023-12-19T08:03:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76000f040 con 0x7fe7680ce2d0 2023-12-19T08:03:03.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760014480 con 0x7fe7680ce2d0 2023-12-19T08:03:03.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe768159320 con 0x7fe7680ce2d0 2023-12-19T08:03:03.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.849+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681598e0 con 0x7fe7680ce2d0 2023-12-19T08:03:03.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.853+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768159dc0 con 0x7fe7680ce2d0 2023-12-19T08:03:03.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.853+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe760014830 con 0x7fe7680ce2d0 2023-12-19T08:03:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.853+0000 7fe764ff9700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 0x7fe74c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.853+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe76007d830 con 0x7fe7680ce2d0 2023-12-19T08:03:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.857+0000 7fe76ce0a700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 0x7fe74c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:03.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.857+0000 7fe76ce0a700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 0x7fe74c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0037c0 tx=0x7fe75c009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:03.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:03.861+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe7600512a0 con 0x7fe7680ce2d0 2023-12-19T08:03:03.886 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T08:03:03.885+0000 7f4faa7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:03:04.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.089+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe76804be40 con 0x7fe7680ce2d0 2023-12-19T08:03:04.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.097+0000 7fe764ff9700 1 -- 172.21.15.111:0/362469697 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe760050e30 con 0x7fe7680ce2d0 2023-12-19T08:03:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 msgr2=0x7fe74c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 0x7fe74c062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0037c0 tx=0x7fe75c009040 comp rx=0 tx=0).stop 2023-12-19T08:03:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 msgr2=0x7fe76806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe760009e90 tx=0x7fe76000b040 comp rx=0 tx=0).stop 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 shutdown_connections 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe74c060670 0x7fe74c062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7680ce2d0 0x7fe76806a480 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7680cc040 0x7fe7681586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 --2- 172.21.15.111:0/362469697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7680caa10 0x7fe7681498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 >> 172.21.15.111:0/362469697 conn(0x7fe768067bb0 msgr2=0x7fe768155930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 shutdown_connections 2023-12-19T08:03:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.101+0000 7fe76de0c700 1 -- 172.21.15.111:0/362469697 wait complete. 2023-12-19T08:03:04.107 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:03:04.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:04.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.277+0000 7ff086c4c700 1 Processor -- start 2023-12-19T08:03:04.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.277+0000 7ff086c4c700 1 -- start start 2023-12-19T08:03:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff086c4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff08015d330 0x7ff08006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff086c4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08006b470 con 0x7ff0800c6cd0 2023-12-19T08:03:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff086c4c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08006e110 con 0x7ff0800c6900 2023-12-19T08:03:04.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff086c4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08006be40 con 0x7ff08015d330 2023-12-19T08:03:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff08644b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff08015d330 0x7ff08006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff085449700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff0800c6cd0 0x7ff08015cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59790/0 (socket says 172.21.15.111:59790) 2023-12-19T08:03:04.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff085449700 1 -- 172.21.15.111:0/1179600697 learned_addr learned my addr 172.21.15.111:0/1179600697 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff08644b700 1 -- 172.21.15.111:0/1179600697 >> v1:172.21.15.143:6789/0 conn(0x7ff0800c6900 legacy=0x7ff08006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff08644b700 1 -- 172.21.15.111:0/1179600697 >> v1:172.21.15.111:6789/0 conn(0x7ff0800c6cd0 legacy=0x7ff08015cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff084c48700 1 -- 172.21.15.111:0/1179600697 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675290445 0 0) 0x7ff08006b470 con 0x7ff0800c6cd0 2023-12-19T08:03:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff08644b700 1 -- 172.21.15.111:0/1179600697 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080161d10 con 0x7ff08015d330 2023-12-19T08:03:04.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.281+0000 7ff08644b700 1 --2- 172.21.15.111:0/1179600697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff08015d330 0x7ff08006b900 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff078009e60 tx=0x7ff078009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee470325372a770 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff084c48700 1 -- 172.21.15.111:0/1179600697 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800e070 con 0x7ff08015d330 2023-12-19T08:03:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff084c48700 1 -- 172.21.15.111:0/1179600697 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07800bc30 con 0x7ff08015d330 2023-12-19T08:03:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff084c48700 1 -- 172.21.15.111:0/1179600697 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078012430 con 0x7ff08015d330 2023-12-19T08:03:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 -- 172.21.15.111:0/1179600697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff08015d330 msgr2=0x7ff08006b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 --2- 172.21.15.111:0/1179600697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff08015d330 0x7ff08006b900 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff078009e60 tx=0x7ff078009780 comp rx=0 tx=0).stop 2023-12-19T08:03:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 -- 172.21.15.111:0/1179600697 shutdown_connections 2023-12-19T08:03:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 --2- 172.21.15.111:0/1179600697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff08015d330 0x7ff08006b900 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 -- 172.21.15.111:0/1179600697 >> 172.21.15.111:0/1179600697 conn(0x7ff080067bb0 msgr2=0x7ff0800cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 -- 172.21.15.111:0/1179600697 shutdown_connections 2023-12-19T08:03:04.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.285+0000 7ff086c4c700 1 -- 172.21.15.111:0/1179600697 wait complete. 2023-12-19T08:03:04.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 Processor -- start 2023-12-19T08:03:04.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 -- start start 2023-12-19T08:03:04.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:04.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0800c6cd0 0x7ff0800cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:04.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 0x7ff0801701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:04.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08016e760 con 0x7ff0800c6900 2023-12-19T08:03:04.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800aea80 con 0x7ff08015d330 2023-12-19T08:03:04.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff086c4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800aed20 con 0x7ff0800c6cd0 2023-12-19T08:03:04.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:04.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37414/0 (socket says 172.21.15.111:37414) 2023-12-19T08:03:04.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 -- 172.21.15.111:0/872978353 learned_addr learned my addr 172.21.15.111:0/872978353 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:04.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085449700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0800c6cd0 0x7ff0800cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:04.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff08644b700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 0x7ff0801701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:04.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 -- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0800c6cd0 msgr2=0x7ff0800cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0800c6cd0 0x7ff0800cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 -- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 msgr2=0x7ff0801701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 0x7ff0801701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff085c4a700 1 -- 172.21.15.111:0/872978353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009a20 con 0x7ff0800c6900 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.289+0000 7ff08644b700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 0x7ff0801701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff085c4a700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00c590 tx=0x7ff07c007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c011590 con 0x7ff0800c6900 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c011730 con 0x7ff0800c6900 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0800af000 con 0x7ff0800c6900 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c011af0 con 0x7ff0800c6900 2023-12-19T08:03:04.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0800af5c0 con 0x7ff0800c6900 2023-12-19T08:03:04.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08004be40 con 0x7ff0800c6900 2023-12-19T08:03:04.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.293+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff07c012420 con 0x7ff0800c6900 2023-12-19T08:03:04.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.297+0000 7ff075ffb700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 0x7ff064062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:04.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.297+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7ff07c07c080 con 0x7ff0800c6900 2023-12-19T08:03:04.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.297+0000 7ff085449700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 0x7ff064062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:04.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.297+0000 7ff085449700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 0x7ff064062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0800c5950 tx=0x7ff070009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:04.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.301+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff07c049be0 con 0x7ff0800c6900 2023-12-19T08:03:04.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.505+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08015b600 con 0x7ff0800c6900 2023-12-19T08:03:04.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff075ffb700 1 -- 172.21.15.111:0/872978353 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7ff07c017020 con 0x7ff0800c6900 2023-12-19T08:03:04.510 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:04.510 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:04.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 msgr2=0x7ff064062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 0x7ff064062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0800c5950 tx=0x7ff070009040 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 msgr2=0x7ff08016e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00c590 tx=0x7ff07c007f30 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 shutdown_connections 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff064060670 0x7ff064062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff08015d330 0x7ff0801701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0800c6cd0 0x7ff0800cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 --2- 172.21.15.111:0/872978353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0800c6900 0x7ff08016e220 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 >> 172.21.15.111:0/872978353 conn(0x7ff080067bb0 msgr2=0x7ff080159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 shutdown_connections 2023-12-19T08:03:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:04.509+0000 7ff086c4c700 1 -- 172.21.15.111:0/872978353 wait complete. 2023-12-19T08:03:04.515 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:04.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:02:58.363473+0000', 'modified': '2023-12-19T08:03:01.958376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6894}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6894': {'gid': 6894, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/2842493331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 2842493331}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 2842493331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:03:04.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:03:04.933 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T08:03:05.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 Processor -- start 2023-12-19T08:03:05.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 -- start start 2023-12-19T08:03:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f941406dc00 0x7f941406b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 0x7f94140b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941406b0d0 con 0x7f9414158cc0 2023-12-19T08:03:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941406bec0 con 0x7f941406dc00 2023-12-19T08:03:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.101+0000 7f94190cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94140b4790 con 0x7f941406baa0 2023-12-19T08:03:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f941406dc00 0x7f941406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f941406dc00 0x7f941406b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37416/0 (socket says 172.21.15.111:37416) 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 -- 172.21.15.111:0/2522776390 learned_addr learned my addr 172.21.15.111:0/2522776390 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941359e700 1 --2- 172.21.15.111:0/2522776390 >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 0x7f94140b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 -- 172.21.15.111:0/2522776390 >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 msgr2=0x7f94140b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 --2- 172.21.15.111:0/2522776390 >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 0x7f94140b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 -- 172.21.15.111:0/2522776390 >> v1:172.21.15.143:6789/0 conn(0x7f9414158cc0 legacy=0x7f941406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 -- 172.21.15.111:0/2522776390 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941406a7e0 con 0x7f941406dc00 2023-12-19T08:03:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941359e700 1 --2- 172.21.15.111:0/2522776390 >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 0x7f94140b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:05.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f941259c700 1 --2- 172.21.15.111:0/2522776390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f941406b560 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f941406b040 tx=0x7f9408009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dec7d4c8a982b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f9411d9b700 1 -- 172.21.15.111:0/2522776390 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940800b860 con 0x7f941406dc00 2023-12-19T08:03:05.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f9411d9b700 1 -- 172.21.15.111:0/2522776390 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940800b9c0 con 0x7f941406dc00 2023-12-19T08:03:05.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f9411d9b700 1 -- 172.21.15.111:0/2522776390 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940800bd60 con 0x7f941406dc00 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 -- 172.21.15.111:0/2522776390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 msgr2=0x7f941406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 --2- 172.21.15.111:0/2522776390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f941406b560 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f941406b040 tx=0x7f9408009250 comp rx=0 tx=0).stop 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 -- 172.21.15.111:0/2522776390 shutdown_connections 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 --2- 172.21.15.111:0/2522776390 >> v2:172.21.15.143:3300/0 conn(0x7f941406baa0 0x7f94140b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 --2- 172.21.15.111:0/2522776390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f941406b560 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 -- 172.21.15.111:0/2522776390 >> 172.21.15.111:0/2522776390 conn(0x7f9414067bb0 msgr2=0x7f94140cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 -- 172.21.15.111:0/2522776390 shutdown_connections 2023-12-19T08:03:05.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.105+0000 7f94190cd700 1 -- 172.21.15.111:0/2522776390 wait complete. 2023-12-19T08:03:05.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 Processor -- start 2023-12-19T08:03:05.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 -- start start 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 0x7f94140aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9414158cc0 0x7f94140aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94140aac80 con 0x7f941406dc00 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414155620 con 0x7f9414158cc0 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f94190cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140ad5b0 con 0x7f941406baa0 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37428/0 (socket says 172.21.15.111:37428) 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 -- 172.21.15.111:0/936130178 learned_addr learned my addr 172.21.15.111:0/936130178 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941359e700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9414158cc0 0x7f94140aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f9412d9d700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 0x7f94140aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 -- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 msgr2=0x7f94140aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 0x7f94140aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 -- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9414158cc0 msgr2=0x7f94140aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9414158cc0 0x7f94140aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 -- 172.21.15.111:0/936130178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080094f0 con 0x7f941406dc00 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f9412d9d700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 0x7f94140aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f941259c700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9408009290 tx=0x7f9408017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f93faffd700 1 -- 172.21.15.111:0/936130178 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408026450 con 0x7f941406dc00 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f93faffd700 1 -- 172.21.15.111:0/936130178 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080265b0 con 0x7f941406dc00 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.109+0000 7f93faffd700 1 -- 172.21.15.111:0/936130178 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408026970 con 0x7f941406dc00 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.113+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94140ad890 con 0x7f941406dc00 2023-12-19T08:03:05.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.113+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141552b0 con 0x7f941406dc00 2023-12-19T08:03:05.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.113+0000 7f93faffd700 1 -- 172.21.15.111:0/936130178 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f940801a420 con 0x7f941406dc00 2023-12-19T08:03:05.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.117+0000 7f93faffd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 0x7f9400062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.117+0000 7f93faffd700 1 -- 172.21.15.111:0/936130178 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f940807eaa0 con 0x7f941406dc00 2023-12-19T08:03:05.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.117+0000 7f9412d9d700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 0x7f9400062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.117+0000 7f9412d9d700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 0x7f9400062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9404007fd0 tx=0x7f9404009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.117+0000 7f941b3a0740 1 Processor -- start 2023-12-19T08:03:05.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 -- start start 2023-12-19T08:03:05.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x157bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x157c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x14a9f30 con 0x12fe710 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x14a5ed0 con 0x12feae0 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x146c0b0 con 0x1349790 2023-12-19T08:03:05.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37444/0 (socket says 172.21.15.111:37444) 2023-12-19T08:03:05.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 -- 172.21.15.111:0/3471258997 learned_addr learned my addr 172.21.15.111:0/3471258997 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f9412d9d700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x157bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941359e700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x157c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 -- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 msgr2=0x157c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x157c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 -- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 msgr2=0x157bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x157bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 -- 172.21.15.111:0/3471258997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x134a9e0 con 0x12fe710 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f9412d9d700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x157bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f941259c700 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f940807cb40 tx=0x7f940804ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f11640f81458ee server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f93f3fff700 1 -- 172.21.15.111:0/3471258997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408055b60 con 0x12fe710 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f93f3fff700 1 -- 172.21.15.111:0/3471258997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080598b0 con 0x12fe710 2023-12-19T08:03:05.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.121+0000 7f93f3fff700 1 -- 172.21.15.111:0/3471258997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408026da0 con 0x12fe710 2023-12-19T08:03:05.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- 172.21.15.111:0/3471258997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 msgr2=0x157c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f940807cb40 tx=0x7f940804ec10 comp rx=0 tx=0).stop 2023-12-19T08:03:05.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- 172.21.15.111:0/3471258997 shutdown_connections 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x157c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x157c440 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3471258997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x157bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- 172.21.15.111:0/3471258997 >> 172.21.15.111:0/3471258997 conn(0x12feeb0 msgr2=0x1538930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- 172.21.15.111:0/3471258997 shutdown_connections 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- 172.21.15.111:0/3471258997 wait complete. 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- start start 2023-12-19T08:03:05.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 Processor -- start 2023-12-19T08:03:05.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12fe710 0x1580510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1580a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x146bd50 con 0x1349790 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1444970 con 0x12feae0 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x14f5480 con 0x12fe710 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941b3a0740 10 client.0 _ll_register_callbacks cb 0x14ef6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37452/0 (socket says 172.21.15.111:37452) 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941359e700 1 -- 172.21.15.111:0/3256686973 learned_addr learned my addr 172.21.15.111:0/3256686973 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f9412d9d700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12fe710 0x1580510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.125+0000 7f941259c700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1580a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 -- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12fe710 msgr2=0x1580510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12fe710 0x1580510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 -- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 msgr2=0x1580a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1580a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 -- 172.21.15.111:0/3256686973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141552b0 con 0x1349790 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941359e700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f940c00fe30 tx=0x7f940c0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b267b698a4913af server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c014070 con 0x1349790 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f940c00dd00 con 0x1349790 2023-12-19T08:03:05.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1444cc0 con 0x1349790 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f940c018620 con 0x1349790 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 10 client.7041 fetch_fsmap learned FSMap version 40 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x1474370 con 0x1349790 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1444cc0 con 0x1349790 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 40) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f940c018780 con 0x1349790 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 10 client.7041 fetch_fsmap finished waiting for FSMap version 40 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 10 client.7041 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 20 client.7041 populate_metadata read hostname 'smithi111' 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x15632d0 0x158b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 4 client.7041 mds_command: new command op to 6894 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 --> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x14483e0 con 0x15632d0 2023-12-19T08:03:05.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f9412d9d700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x15632d0 0x158b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f9412d9d700 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x15632d0 0x158b6b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d81fade22f269927 server_cookie=a0d69d87d0a891d6 in_seq=0 out_seq=0 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.129+0000 7f93f17fa700 10 client.7041 ms_handle_connect on v2:172.21.15.111:6834/2842493331 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f93f17fa700 1 -- 172.21.15.111:0/3256686973 <== mds.0 v2:172.21.15.111:6834/2842493331 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x14483e0 con 0x15632d0 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f93f17fa700 10 client.7041 handle_command_reply: tid=0 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 client.7041 shutdown 2023-12-19T08:03:05.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 msgr2=0x1561cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f940c00fe30 tx=0x7f940c0077a0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 shutdown_connections 2023-12-19T08:03:05.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x15632d0 0x158b6b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1580a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f93f17fa700 0 client.7041 ms_handle_reset on v2:172.21.15.111:6834/2842493331 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1349790 0x1561cb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 --2- 172.21.15.111:0/3256686973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12fe710 0x1580510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 >> 172.21.15.111:0/3256686973 conn(0x12feeb0 msgr2=0x1530f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 shutdown_connections 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 1 -- 172.21.15.111:0/3256686973 wait complete. 2023-12-19T08:03:05.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.133+0000 7f941b3a0740 20 client.7041 trim_cache size 0 max 16384 2023-12-19T08:03:05.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 Processor -- start 2023-12-19T08:03:05.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 -- start start 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1533010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x155f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x14aaef0 con 0x12fe710 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1446b80 con 0x12feae0 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x14a8590 con 0x1349790 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37468/0 (socket says 172.21.15.111:37468) 2023-12-19T08:03:05.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 -- 172.21.15.111:0/499923936 learned_addr learned my addr 172.21.15.111:0/499923936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 -- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 msgr2=0x155f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x155f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 -- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 msgr2=0x1533010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1533010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 -- 172.21.15.111:0/499923936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x145ad80 con 0x12fe710 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f9412d9d700 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f9404001210 tx=0x7f9404015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79680c5756902af server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.153+0000 7f93f8ff9700 1 -- 172.21.15.111:0/499923936 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940400ee60 con 0x12fe710 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f93f8ff9700 1 -- 172.21.15.111:0/499923936 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9404014c20 con 0x12fe710 2023-12-19T08:03:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f93f8ff9700 1 -- 172.21.15.111:0/499923936 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404014d90 con 0x12fe710 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- 172.21.15.111:0/499923936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 msgr2=0x1532ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f9404001210 tx=0x7f9404015040 comp rx=0 tx=0).stop 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- 172.21.15.111:0/499923936 shutdown_connections 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12feae0 0x1533010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1349790 0x155f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 --2- 172.21.15.111:0/499923936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x1532ad0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- 172.21.15.111:0/499923936 >> 172.21.15.111:0/499923936 conn(0x12feeb0 msgr2=0x157b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- 172.21.15.111:0/499923936 shutdown_connections 2023-12-19T08:03:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- 172.21.15.111:0/499923936 wait complete. 2023-12-19T08:03:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 -- start start 2023-12-19T08:03:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.157+0000 7f941b3a0740 1 Processor -- start 2023-12-19T08:03:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12feae0 0x15d5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 0x15354c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x14a99d0 con 0x12fe710 2023-12-19T08:03:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x14a3890 con 0x1349790 2023-12-19T08:03:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1454c90 con 0x12feae0 2023-12-19T08:03:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 10 client.0 _ll_register_callbacks cb 0x12ca9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:03:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37472/0 (socket says 172.21.15.111:37472) 2023-12-19T08:03:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 -- 172.21.15.111:0/430908145 learned_addr learned my addr 172.21.15.111:0/430908145 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941359e700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 0x15354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941259c700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12feae0 0x15d5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 -- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12feae0 msgr2=0x15d5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12feae0 0x15d5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 -- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 msgr2=0x15354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 0x15354c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 -- 172.21.15.111:0/430908145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940401d040 con 0x12fe710 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941359e700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 0x15354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f9412d9d700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x1450760 tx=0x7f9404027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8df5c459ccbfd98 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f93f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040259e0 con 0x12fe710 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f9404025b40 con 0x12fe710 2023-12-19T08:03:05.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.161+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94080598b0 con 0x12fe710 2023-12-19T08:03:05.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9404036400 con 0x12fe710 2023-12-19T08:03:05.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 10 client.7047 fetch_fsmap learned FSMap version 40 2023-12-19T08:03:05.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x144f130 con 0x12fe710 2023-12-19T08:03:05.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080598b0 con 0x12fe710 2023-12-19T08:03:05.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 40) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f940402b020 con 0x12fe710 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 10 client.7047 fetch_fsmap finished waiting for FSMap version 40 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 10 client.7047 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 20 client.7047 populate_metadata read hostname 'smithi111' 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x1535e50 0x1536220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 4 client.7047 mds_command: new command op to 6894 tid=0[{"prefix": "status"}] 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 --> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] -- command(tid 0: {"prefix": "status"}) v1 -- 0x147a410 con 0x1535e50 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941259c700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x1535e50 0x1536220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941259c700 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x1535e50 0x1536220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de80336deb639c84 server_cookie=3005876e35e7245a in_seq=0 out_seq=0 2023-12-19T08:03:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 10 client.7047 ms_handle_connect on v2:172.21.15.111:6834/2842493331 2023-12-19T08:03:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 1 -- 172.21.15.111:0/430908145 <== mds.0 v2:172.21.15.111:6834/2842493331 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x147a410 con 0x1535e50 2023-12-19T08:03:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 10 client.7047 handle_command_reply: tid=0 2023-12-19T08:03:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 client.7047 shutdown 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 msgr2=0x15d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x1450760 tx=0x7f9404027000 comp rx=0 tx=0).stop 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 shutdown_connections 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:6834/2842493331,v1:172.21.15.111:6835/2842493331] conn(0x1535e50 0x1536220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1349790 0x15354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12feae0 0x15d5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f93f17fa700 0 client.7047 ms_handle_reset on v2:172.21.15.111:6834/2842493331 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 --2- 172.21.15.111:0/430908145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12fe710 0x15d5960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 >> 172.21.15.111:0/430908145 conn(0x12feeb0 msgr2=0x155f780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 shutdown_connections 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 1 -- 172.21.15.111:0/430908145 wait complete. 2023-12-19T08:03:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.165+0000 7f941b3a0740 20 client.7047 trim_cache size 0 max 16384 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 6894, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.233826326, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 40, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 99, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 99, 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.142537043000001 2023-12-19T08:03:05.171 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:03:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 msgr2=0x7f9400062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 0x7f9400062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9404007fd0 tx=0x7f9404009040 comp rx=0 tx=0).stop 2023-12-19T08:03:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 msgr2=0x7f9414153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9408009290 tx=0x7f9408017000 comp rx=0 tx=0).stop 2023-12-19T08:03:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 shutdown_connections 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9400060670 0x7f9400062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9414158cc0 0x7f94140aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f941406dc00 0x7f9414153580 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 --2- 172.21.15.111:0/936130178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941406baa0 0x7f94140aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 >> 172.21.15.111:0/936130178 conn(0x7f9414067bb0 msgr2=0x7f941415b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 shutdown_connections 2023-12-19T08:03:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.169+0000 7f94190cd700 1 -- 172.21.15.111:0/936130178 wait complete. 2023-12-19T08:03:05.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:03:05.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.353+0000 7f8d43eae700 1 Processor -- start 2023-12-19T08:03:05.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.353+0000 7f8d43eae700 1 -- start start 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 0x7f8d3c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 0x7f8d3c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d3c06d690 0x7f8d3c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c7190 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c1b50 con 0x7f8d3c06d690 2023-12-19T08:03:05.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d43eae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0c1e30 con 0x7f8d3c15d320 2023-12-19T08:03:05.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41449700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 0x7f8d3c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 0x7f8d3c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41449700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 0x7f8d3c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37476/0 (socket says 172.21.15.111:37476) 2023-12-19T08:03:05.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41449700 1 -- 172.21.15.111:0/696859084 learned_addr learned my addr 172.21.15.111:0/696859084 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d4244b700 1 --2- 172.21.15.111:0/696859084 >> v2:172.21.15.143:3300/0 conn(0x7f8d3c06d690 0x7f8d3c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 -- 172.21.15.111:0/696859084 >> v2:172.21.15.143:3300/0 conn(0x7f8d3c06d690 msgr2=0x7f8d3c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/696859084 >> v2:172.21.15.143:3300/0 conn(0x7f8d3c06d690 0x7f8d3c0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 -- 172.21.15.111:0/696859084 >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 msgr2=0x7f8d3c0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/696859084 >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 0x7f8d3c0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 -- 172.21.15.111:0/696859084 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0c6e20 con 0x7f8d3c15d320 2023-12-19T08:03:05.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/696859084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 0x7f8d3c15d6f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c06e2b0 tx=0x7f8d38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35c50f0061a45219 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.357+0000 7f8d40c48700 1 -- 172.21.15.111:0/696859084 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3800e070 con 0x7f8d3c15d320 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d40c48700 1 -- 172.21.15.111:0/696859084 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3800bab0 con 0x7f8d3c15d320 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d40c48700 1 -- 172.21.15.111:0/696859084 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3800be70 con 0x7f8d3c15d320 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 -- 172.21.15.111:0/696859084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 msgr2=0x7f8d3c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 --2- 172.21.15.111:0/696859084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 0x7f8d3c15d6f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c06e2b0 tx=0x7f8d38009780 comp rx=0 tx=0).stop 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 -- 172.21.15.111:0/696859084 shutdown_connections 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 --2- 172.21.15.111:0/696859084 >> v2:172.21.15.143:3300/0 conn(0x7f8d3c06d690 0x7f8d3c0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 --2- 172.21.15.111:0/696859084 >> v2:172.21.15.111:3300/0 conn(0x7f8d3c0c15b0 0x7f8d3c0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 --2- 172.21.15.111:0/696859084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c15d320 0x7f8d3c15d6f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 -- 172.21.15.111:0/696859084 >> 172.21.15.111:0/696859084 conn(0x7f8d3c0c8850 msgr2=0x7f8d3c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 -- 172.21.15.111:0/696859084 shutdown_connections 2023-12-19T08:03:05.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 -- 172.21.15.111:0/696859084 wait complete. 2023-12-19T08:03:05.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.361+0000 7f8d43eae700 1 Processor -- start 2023-12-19T08:03:05.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- start start 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c06d690 0x7f8d3c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d3c15d320 0x7f8d3c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c16db20 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c16e600 con 0x7f8d3c15d320 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c16dd10 con 0x7f8d3c06d690 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37488/0 (socket says 172.21.15.111:37488) 2023-12-19T08:03:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 -- 172.21.15.111:0/2387746734 learned_addr learned my addr 172.21.15.111:0/2387746734 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c06d690 0x7f8d3c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d4244b700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d3c15d320 0x7f8d3c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 -- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c06d690 msgr2=0x7f8d3c0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c06d690 0x7f8d3c0c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 -- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d3c15d320 msgr2=0x7f8d3c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d3c15d320 0x7f8d3c0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 -- 172.21.15.111:0/2387746734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d38009a20 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d41449700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00ea50 tx=0x7f8d2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00c7f0 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c00c950 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c16ea90 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00ccd0 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.365+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c16e290 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.369+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c071120 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.369+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8d2c013680 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.373+0000 7f8d31ffb700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:05.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.373+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f8d2c07cf50 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.373+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:05.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.373+0000 7f8d41c4a700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097c0 tx=0x7f8d38019000 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:05.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.377+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d2c04d040 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.581+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8d3c0642a0 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.585+0000 7f8d31ffb700 1 -- 172.21.15.111:0/2387746734 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8d2c050660 con 0x7f8d3c0c15b0 2023-12-19T08:03:05.587 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:03:05.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.585+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 msgr2=0x7f8d20062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.585+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097c0 tx=0x7f8d38019000 comp rx=0 tx=0).stop 2023-12-19T08:03:05.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.585+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 msgr2=0x7f8d3c0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:05.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.585+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00ea50 tx=0x7f8d2c009250 comp rx=0 tx=0).stop 2023-12-19T08:03:05.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 shutdown_connections 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d20060670 0x7f8d20062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d3c15d320 0x7f8d3c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d3c0c15b0 0x7f8d3c0afc90 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 --2- 172.21.15.111:0/2387746734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d3c06d690 0x7f8d3c0c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 >> 172.21.15.111:0/2387746734 conn(0x7f8d3c0c8850 msgr2=0x7f8d3c0679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 shutdown_connections 2023-12-19T08:03:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:05.589+0000 7f8d43eae700 1 -- 172.21.15.111:0/2387746734 wait complete. 2023-12-19T08:03:05.608 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:03:05.608 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:03:05.609 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:03:05.610 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:05.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.610 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:03:05.629 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:03:05.629 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:03:05.631 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:03:05.631 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41472sec preferred_lft 41472sec 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:03:05.632 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:05.633 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:03:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:03:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.635 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:03:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:03:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:03:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:05.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.638 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:03:05.723 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:03:05.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.727 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:03:05.792 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:03:05.793 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:03:05.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:03:05.794 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:03:05.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.939 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:03:05.955 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:03:05.955 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:03:05.956 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:05.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:05.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:03:06.035 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:03:06.035 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:03:06.035 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:06.098 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:06.098 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:03:06.099 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:06.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:03:06.239 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:03:06.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:03:06.304 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:03:06.318 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:03:06.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:03:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:03:06.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:06.324 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:03:06.377 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:03:06.377 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:03:06.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:03:06.483 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:03:06.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:03:06.563 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:03:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:03:06.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:06.566 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:03:06.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:03:06.581+0000 7ffa0f100380 -1 init, newargv = 0x55e56b06b2c0 newargc=15 2023-12-19T08:03:06.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[50663]: starting ceph client 2023-12-19T08:03:06.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[50663]: starting fuse 2023-12-19T08:03:06.597 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:03:06.598 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:03:06.598 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:03:06.599 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:03:06.600 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:03:06.600 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:06.600 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:03:06.600 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:03:06.755 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.50663.asok 2023-12-19T08:03:06.940 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50663.asok 2023-12-19T08:03:06.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:06.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50663.asok status 2023-12-19T08:03:07.119 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "50663", 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:03:07.120 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 7059, 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 7059 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1422760668 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1422760668 2023-12-19T08:03:07.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.7059 192.168.0.1:0/1422760668", 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/1422760668", 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 40, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 99, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:03:07.122 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:03:07.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:07.133 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:03:07.133 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:03:07.148 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:07.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:07.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:07.223 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:03:07.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.429+0000 7f46bd385700 1 Processor -- start 2023-12-19T08:03:07.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.429+0000 7f46bd385700 1 -- start start 2023-12-19T08:03:07.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46bd385700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 0x7f46b806a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46bd385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 0x7f46b815d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46bd385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b806ba50 con 0x7f46b815d050 2023-12-19T08:03:07.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46bd385700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b806bee0 con 0x7f46b80afef0 2023-12-19T08:03:07.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46bd385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80af600 con 0x7f46b8160e60 2023-12-19T08:03:07.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b67fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 0x7f46b806a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 0x7f46b815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b67fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 0x7f46b806a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:35756/0 (socket says 172.21.15.111:35756) 2023-12-19T08:03:07.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b67fc700 1 -- 172.21.15.111:0/4171278062 learned_addr learned my addr 172.21.15.111:0/4171278062 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:07.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 -- 172.21.15.111:0/4171278062 >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 msgr2=0x7f46b806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:07.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 --2- 172.21.15.111:0/4171278062 >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 0x7f46b806a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 -- 172.21.15.111:0/4171278062 >> v1:172.21.15.143:6789/0 conn(0x7f46b815d050 legacy=0x7f46b80af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 -- 172.21.15.111:0/4171278062 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b806ae40 con 0x7f46b8160e60 2023-12-19T08:03:07.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b77fe700 1 --2- 172.21.15.111:0/4171278062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 0x7f46b815d420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009e60 tx=0x7f46b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba00bdb1254b75cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b5ffb700 1 -- 172.21.15.111:0/4171278062 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000e070 con 0x7f46b8160e60 2023-12-19T08:03:07.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b5ffb700 1 -- 172.21.15.111:0/4171278062 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b000bb60 con 0x7f46b8160e60 2023-12-19T08:03:07.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.433+0000 7f46b5ffb700 1 -- 172.21.15.111:0/4171278062 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0012430 con 0x7f46b8160e60 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- 172.21.15.111:0/4171278062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 msgr2=0x7f46b815d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 --2- 172.21.15.111:0/4171278062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 0x7f46b815d420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009e60 tx=0x7f46b0009780 comp rx=0 tx=0).stop 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- 172.21.15.111:0/4171278062 shutdown_connections 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 --2- 172.21.15.111:0/4171278062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b8160e60 0x7f46b815d420 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 --2- 172.21.15.111:0/4171278062 >> v2:172.21.15.143:3300/0 conn(0x7f46b80afef0 0x7f46b806a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- 172.21.15.111:0/4171278062 >> 172.21.15.111:0/4171278062 conn(0x7f46b8067b60 msgr2=0x7f46b80ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- 172.21.15.111:0/4171278062 shutdown_connections 2023-12-19T08:03:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- 172.21.15.111:0/4171278062 wait complete. 2023-12-19T08:03:07.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 Processor -- start 2023-12-19T08:03:07.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.437+0000 7f46bd385700 1 -- start start 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 0x7f46b814be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b815d050 0x7f46b815be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b816dfc0 con 0x7f46b8160e60 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c1910 con 0x7f46b80afef0 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80c2490 con 0x7f46b815d050 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37498/0 (socket says 172.21.15.111:37498) 2023-12-19T08:03:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 -- 172.21.15.111:0/1960434883 learned_addr learned my addr 172.21.15.111:0/1960434883 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b67fc700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b815d050 0x7f46b815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b6ffd700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 0x7f46b814be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 -- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b815d050 msgr2=0x7f46b815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b815d050 0x7f46b815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 -- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 msgr2=0x7f46b814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 0x7f46b814be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 -- 172.21.15.111:0/1960434883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0009a20 con 0x7f46b8160e60 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b6ffd700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 0x7f46b814be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46b77fe700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b6c0 tx=0x7f46b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0027470 con 0x7f46b8160e60 2023-12-19T08:03:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b00275f0 con 0x7f46b8160e60 2023-12-19T08:03:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b80c2920 con 0x7f46b8160e60 2023-12-19T08:03:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b00279b0 con 0x7f46b8160e60 2023-12-19T08:03:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.441+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b80c1b60 con 0x7f46b8160e60 2023-12-19T08:03:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.445+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b8071120 con 0x7f46b8160e60 2023-12-19T08:03:07.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.445+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f46b0024070 con 0x7f46b8160e60 2023-12-19T08:03:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.449+0000 7f469effd700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 0x7f46a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.449+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f46b007f580 con 0x7f46b8160e60 2023-12-19T08:03:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.449+0000 7f46b6ffd700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 0x7f46a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.449+0000 7f46b6ffd700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 0x7f46a4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46a8007fd0 tx=0x7f46a8009040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.453+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f46b004f710 con 0x7f46b8160e60 2023-12-19T08:03:07.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.653+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b80c25d0 con 0x7f46a4060670 2023-12-19T08:03:07.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.657+0000 7f469effd700 1 -- 172.21.15.111:0/1960434883 <== mgr.6044 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f46b80c25d0 con 0x7f46a4060670 2023-12-19T08:03:07.658 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:03:07.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:07.658 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:03:07.658 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:07.658 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:03:07.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 msgr2=0x7f46a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 0x7f46a4062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f46a8007fd0 tx=0x7f46a8009040 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 msgr2=0x7f46b815c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b6c0 tx=0x7f46b0019000 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 shutdown_connections 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f46a4060670 0x7f46a4062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b8160e60 0x7f46b815c3b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46b815d050 0x7f46b815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 --2- 172.21.15.111:0/1960434883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46b80afef0 0x7f46b814be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 >> 172.21.15.111:0/1960434883 conn(0x7f46b8067b60 msgr2=0x7f46b815ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 shutdown_connections 2023-12-19T08:03:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.661+0000 7f46bd385700 1 -- 172.21.15.111:0/1960434883 wait complete. 2023-12-19T08:03:07.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:03:07.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.845+0000 7f3611a2e700 1 Processor -- start 2023-12-19T08:03:07.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.845+0000 7f3611a2e700 1 -- start start 2023-12-19T08:03:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f360c158e20 0x7f360c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f360c0ca2c0 0x7f360c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c0cbdf0 con 0x7f360c0cb970 2023-12-19T08:03:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360c0cc0d0 con 0x7f360c0ca2c0 2023-12-19T08:03:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360c0cc3b0 con 0x7f360c158e20 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f360c0ca2c0 0x7f360c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f360c0ca2c0 0x7f360c0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37512/0 (socket says 172.21.15.111:37512) 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 -- 172.21.15.111:0/299122479 learned_addr learned my addr 172.21.15.111:0/299122479 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360affd700 1 --2- 172.21.15.111:0/299122479 >> v2:172.21.15.143:3300/0 conn(0x7f360c158e20 0x7f360c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 -- 172.21.15.111:0/299122479 >> v2:172.21.15.143:3300/0 conn(0x7f360c158e20 msgr2=0x7f360c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 --2- 172.21.15.111:0/299122479 >> v2:172.21.15.143:3300/0 conn(0x7f360c158e20 0x7f360c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 -- 172.21.15.111:0/299122479 >> v1:172.21.15.143:6789/0 conn(0x7f360c0cb970 legacy=0x7f360c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 -- 172.21.15.111:0/299122479 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0cc690 con 0x7f360c0ca2c0 2023-12-19T08:03:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f360b7fe700 1 --2- 172.21.15.111:0/299122479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c0ca2c0 0x7f360c0ceb00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f360c0cdc10 tx=0x7f3604009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30624f1bcae6e706 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3609ffb700 1 -- 172.21.15.111:0/299122479 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360400e070 con 0x7f360c0ca2c0 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3609ffb700 1 -- 172.21.15.111:0/299122479 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360400b9d0 con 0x7f360c0ca2c0 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3609ffb700 1 -- 172.21.15.111:0/299122479 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36040092e0 con 0x7f360c0ca2c0 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- 172.21.15.111:0/299122479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c0ca2c0 msgr2=0x7f360c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 --2- 172.21.15.111:0/299122479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c0ca2c0 0x7f360c0ceb00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f360c0cdc10 tx=0x7f3604009840 comp rx=0 tx=0).stop 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- 172.21.15.111:0/299122479 shutdown_connections 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 --2- 172.21.15.111:0/299122479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c0ca2c0 0x7f360c0ceb00 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 --2- 172.21.15.111:0/299122479 >> v2:172.21.15.143:3300/0 conn(0x7f360c158e20 0x7f360c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- 172.21.15.111:0/299122479 >> 172.21.15.111:0/299122479 conn(0x7f360c067bb0 msgr2=0x7f360c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- 172.21.15.111:0/299122479 shutdown_connections 2023-12-19T08:03:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.849+0000 7f3611a2e700 1 -- 172.21.15.111:0/299122479 wait complete. 2023-12-19T08:03:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 Processor -- start 2023-12-19T08:03:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 -- start start 2023-12-19T08:03:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f360c0ca2c0 0x7f360c16e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f360c154d10 0x7f360c14c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c16e700 con 0x7f360c158e20 2023-12-19T08:03:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c16d680 con 0x7f360c0ca2c0 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f3611a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c155db0 con 0x7f360c154d10 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f360a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f360a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37524/0 (socket says 172.21.15.111:37524) 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f360a7fc700 1 -- 172.21.15.111:0/541042114 learned_addr learned my addr 172.21.15.111:0/541042114 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f360affd700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f360c0ca2c0 0x7f360c16e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.853+0000 7f360b7fe700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f360c154d10 0x7f360c14c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 -- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f360c154d10 msgr2=0x7f360c14c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f360c154d10 0x7f360c14c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 -- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f360c0ca2c0 msgr2=0x7f360c16e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f360c0ca2c0 0x7f360c16e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 -- 172.21.15.111:0/541042114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3604009ae0 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f360a7fc700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f360000c5c0 tx=0x7f3600007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600011590 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600011710 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c1551a0 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600011ad0 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c155760 con 0x7f360c158e20 2023-12-19T08:03:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.857+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c06b800 con 0x7f360c158e20 2023-12-19T08:03:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.861+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 34) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f360001c460 con 0x7f360c158e20 2023-12-19T08:03:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.861+0000 7f35f2ffd700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 0x7f35f8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.861+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f360007c910 con 0x7f360c158e20 2023-12-19T08:03:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.861+0000 7f360affd700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 0x7f35f8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:07.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.865+0000 7f360affd700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 0x7f35f8062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006fd0 tx=0x7f35fc008040 comp rx=0 tx=0).ready entity=mgr.6044 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:07.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:07.869+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f360c06b800 con 0x7f360c158e20 2023-12-19T08:03:08.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.073+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f360c06b800 con 0x7f360c158e20 2023-12-19T08:03:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.121+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 35) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f360004ce70 con 0x7f360c158e20 2023-12-19T08:03:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.121+0000 7f35f2ffd700 1 -- 172.21.15.111:0/541042114 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v35) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3600049000 con 0x7f360c158e20 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 msgr2=0x7f35f8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 0x7f35f8062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006fd0 tx=0x7f35fc008040 comp rx=0 tx=0).stop 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 msgr2=0x7f360c16d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f360000c5c0 tx=0x7f3600007f30 comp rx=0 tx=0).stop 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 shutdown_connections 2023-12-19T08:03:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f35f8060670 0x7f35f8062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f360c154d10 0x7f360c14c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f360c158e20 0x7f360c16d140 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 --2- 172.21.15.111:0/541042114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f360c0ca2c0 0x7f360c16e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 >> 172.21.15.111:0/541042114 conn(0x7f360c067bb0 msgr2=0x7f360c14b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 shutdown_connections 2023-12-19T08:03:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.125+0000 7f3611a2e700 1 -- 172.21.15.111:0/541042114 wait complete. 2023-12-19T08:03:08.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:03:08.313 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:08.309+0000 7fd250762000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:08.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 Processor -- start 2023-12-19T08:03:08.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 -- start start 2023-12-19T08:03:08.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 0x7fb20c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c06c850 con 0x7fb20c158c90 2023-12-19T08:03:08.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c0cb650 con 0x7fb20c1595a0 2023-12-19T08:03:08.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21228e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c06b470 con 0x7fb20c06aeb0 2023-12-19T08:03:08.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21128c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb20c158c90 0x7fb20c0caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59812/0 (socket says 172.21.15.111:59812) 2023-12-19T08:03:08.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb21128c700 1 -- 172.21.15.111:0/2556065606 learned_addr learned my addr 172.21.15.111:0/2556065606 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:08.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.313+0000 7fb210a8b700 1 --2- 172.21.15.111:0/2556065606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 0x7fb20c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734351863 0 0) 0x7fb20c06c850 con 0x7fb20c158c90 2023-12-19T08:03:08.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f8003570 con 0x7fb20c158c90 2023-12-19T08:03:08.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064263976 0 0) 0x7fb20c0cb650 con 0x7fb20c1595a0 2023-12-19T08:03:08.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb210a8b700 1 -- 172.21.15.111:0/2556065606 >> v1:172.21.15.143:6789/0 conn(0x7fb20c1595a0 legacy=0x7fb20c06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb210a8b700 1 -- 172.21.15.111:0/2556065606 >> v1:172.21.15.111:6789/0 conn(0x7fb20c158c90 legacy=0x7fb20c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb210a8b700 1 -- 172.21.15.111:0/2556065606 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c06b750 con 0x7fb20c06aeb0 2023-12-19T08:03:08.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb210a8b700 1 --2- 172.21.15.111:0/2556065606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 0x7fb20c159060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009910 tx=0x7fb1fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a196f9def7a76df server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00b680 con 0x7fb20c06aeb0 2023-12-19T08:03:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc00b7e0 con 0x7fb20c06aeb0 2023-12-19T08:03:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb203fff700 1 -- 172.21.15.111:0/2556065606 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00bba0 con 0x7fb20c06aeb0 2023-12-19T08:03:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 -- 172.21.15.111:0/2556065606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 msgr2=0x7fb20c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 --2- 172.21.15.111:0/2556065606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 0x7fb20c159060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009910 tx=0x7fb1fc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 -- 172.21.15.111:0/2556065606 shutdown_connections 2023-12-19T08:03:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 --2- 172.21.15.111:0/2556065606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c06aeb0 0x7fb20c159060 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 -- 172.21.15.111:0/2556065606 >> 172.21.15.111:0/2556065606 conn(0x7fb20c067bb0 msgr2=0x7fb20c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 -- 172.21.15.111:0/2556065606 shutdown_connections 2023-12-19T08:03:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.317+0000 7fb21228e700 1 -- 172.21.15.111:0/2556065606 wait complete. 2023-12-19T08:03:08.322 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:08.321+0000 7f518655b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:08.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.321+0000 7fb21228e700 1 Processor -- start 2023-12-19T08:03:08.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.321+0000 7fb21228e700 1 -- start start 2023-12-19T08:03:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 0x7fb20c14b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c1595a0 0x7fb20c156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c14be60 con 0x7fb20c06aeb0 2023-12-19T08:03:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0c9bd0 con 0x7fb20c1595a0 2023-12-19T08:03:08.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c154b50 con 0x7fb20c158c90 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37542/0 (socket says 172.21.15.111:37542) 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 -- 172.21.15.111:0/3186942199 learned_addr learned my addr 172.21.15.111:0/3186942199 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb211a8d700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c1595a0 0x7fb20c156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb210a8b700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 0x7fb20c14b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 -- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 msgr2=0x7fb20c14b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 0x7fb20c14b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 -- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c1595a0 msgr2=0x7fb20c156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c1595a0 0x7fb20c156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 -- 172.21.15.111:0/3186942199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc009420 con 0x7fb20c06aeb0 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb210a8b700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 0x7fb20c14b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:08.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21128c700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c790 tx=0x7fb208007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:08.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208005b30 con 0x7fb20c06aeb0 2023-12-19T08:03:08.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208005c90 con 0x7fb20c06aeb0 2023-12-19T08:03:08.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208007320 con 0x7fb20c06aeb0 2023-12-19T08:03:08.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c0c9330 con 0x7fb20c06aeb0 2023-12-19T08:03:08.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.325+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c0c98f0 con 0x7fb20c06aeb0 2023-12-19T08:03:08.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.329+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c04be40 con 0x7fb20c06aeb0 2023-12-19T08:03:08.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.329+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 35) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fb208007690 con 0x7fb20c06aeb0 2023-12-19T08:03:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.333+0000 7fb2017fa700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 0x7fb1f0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.333+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fb20807c1f0 con 0x7fb20c06aeb0 2023-12-19T08:03:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.333+0000 7fb210a8b700 1 -- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 msgr2=0x7fb1f0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.333+0000 7fb210a8b700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 0x7fb1f0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.337+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb20804c2e0 con 0x7fb20c06aeb0 2023-12-19T08:03:08.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.509+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb20c06ba30 con 0x7fb20c06aeb0 2023-12-19T08:03:08.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.517+0000 7fb2017fa700 1 -- 172.21.15.111:0/3186942199 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb20804f900 con 0x7fb20c06aeb0 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.517+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 msgr2=0x7fb1f0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.517+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 0x7fb1f0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.517+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 msgr2=0x7fb20c0c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.517+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c790 tx=0x7fb208007df0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 shutdown_connections 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb1f00606c0 0x7fb1f0062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c1595a0 0x7fb20c156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c158c90 0x7fb20c14b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 --2- 172.21.15.111:0/3186942199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb20c06aeb0 0x7fb20c0c8d50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 >> 172.21.15.111:0/3186942199 conn(0x7fb20c067bb0 msgr2=0x7fb20c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 shutdown_connections 2023-12-19T08:03:08.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.521+0000 7fb21228e700 1 -- 172.21.15.111:0/3186942199 wait complete. 2023-12-19T08:03:08.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:03:08.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 Processor -- start 2023-12-19T08:03:08.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 -- start start 2023-12-19T08:03:08.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 0x7fae900684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae900b0cb0 0x7fae900b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae900b1d00 0x7fae90159b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.709+0000 7fae9722e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900b2180 con 0x7fae900b1d00 2023-12-19T08:03:08.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae9722e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae90067790 con 0x7fae90159790 2023-12-19T08:03:08.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae9722e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae90068030 con 0x7fae900b0cb0 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae900b1d00 0x7fae90159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fae900b1d00 0x7fae90159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37546/0 (socket says 172.21.15.111:37546) 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 -- 172.21.15.111:0/1594382915 learned_addr learned my addr 172.21.15.111:0/1594382915 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae94fca700 1 --2- 172.21.15.111:0/1594382915 >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 0x7fae900684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae8ffff700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae900b0cb0 0x7fae900b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 -- 172.21.15.111:0/1594382915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae900b0cb0 msgr2=0x7fae900b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:08.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae900b0cb0 0x7fae900b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 -- 172.21.15.111:0/1594382915 >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 msgr2=0x7fae900684c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:08.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 --2- 172.21.15.111:0/1594382915 >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 0x7fae900684c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 -- 172.21.15.111:0/1594382915 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900b1200 con 0x7fae900b1d00 2023-12-19T08:03:08.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae94fca700 1 --2- 172.21.15.111:0/1594382915 >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 0x7fae900684c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:08.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae957cb700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b1d00 0x7fae90159b60 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fae900b09f0 tx=0x7fae88009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39aca8ca8649287e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:08.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae8f7fe700 1 -- 172.21.15.111:0/1594382915 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8800e070 con 0x7fae900b1d00 2023-12-19T08:03:08.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae8f7fe700 1 -- 172.21.15.111:0/1594382915 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8800ba20 con 0x7fae900b1d00 2023-12-19T08:03:08.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.713+0000 7fae8f7fe700 1 -- 172.21.15.111:0/1594382915 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae880092e0 con 0x7fae900b1d00 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- 172.21.15.111:0/1594382915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b1d00 msgr2=0x7fae90159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b1d00 0x7fae90159b60 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fae900b09f0 tx=0x7fae88009840 comp rx=0 tx=0).stop 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- 172.21.15.111:0/1594382915 shutdown_connections 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b1d00 0x7fae90159b60 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- 172.21.15.111:0/1594382915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae900b0cb0 0x7fae900b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- 172.21.15.111:0/1594382915 >> v2:172.21.15.143:3300/0 conn(0x7fae90159790 0x7fae900684c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- 172.21.15.111:0/1594382915 >> 172.21.15.111:0/1594382915 conn(0x7fae900cdef0 msgr2=0x7fae900ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- 172.21.15.111:0/1594382915 shutdown_connections 2023-12-19T08:03:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- 172.21.15.111:0/1594382915 wait complete. 2023-12-19T08:03:08.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 Processor -- start 2023-12-19T08:03:08.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- start start 2023-12-19T08:03:08.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 0x7fae900cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 0x7fae901549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015db80 con 0x7fae900b0cb0 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c8520 con 0x7fae900b1d00 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae9722e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c7910 con 0x7fae90159790 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.717+0000 7fae94fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37558/0 (socket says 172.21.15.111:37558) 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 -- 172.21.15.111:0/934640620 learned_addr learned my addr 172.21.15.111:0/934640620 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae957cb700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 0x7fae901549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae8ffff700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 0x7fae900cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 -- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 msgr2=0x7fae901549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 0x7fae901549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 -- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 msgr2=0x7fae900cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 0x7fae900cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 -- 172.21.15.111:0/934640620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88009ae0 con 0x7fae900b0cb0 2023-12-19T08:03:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae8ffff700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 0x7fae900cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae957cb700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 0x7fae901549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae94fca700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fae8400d890 tx=0x7fae84008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400b600 con 0x7fae900b0cb0 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8400b760 con 0x7fae900b0cb0 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400bb00 con 0x7fae900b0cb0 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae900c7bf0 con 0x7fae900b0cb0 2023-12-19T08:03:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c81b0 con 0x7fae900b0cb0 2023-12-19T08:03:08.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.721+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae90063510 con 0x7fae900b0cb0 2023-12-19T08:03:08.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.725+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 35) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fae84014430 con 0x7fae900b0cb0 2023-12-19T08:03:08.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.725+0000 7fae8cff9700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 0x7fae74062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:08.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.725+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fae8407d190 con 0x7fae900b0cb0 2023-12-19T08:03:08.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.725+0000 7fae8ffff700 1 -- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 msgr2=0x7fae74062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:08.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.725+0000 7fae8ffff700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 0x7fae74062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:08.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.733+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fae90063510 con 0x7fae900b0cb0 2023-12-19T08:03:08.880 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:08.881+0000 7f518655b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:08.885 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:08.881+0000 7fd250762000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:08.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.905+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fae90063510 con 0x7fae900b0cb0 2023-12-19T08:03:08.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.913+0000 7fae8cff9700 1 -- 172.21.15.111:0/934640620 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fae90063510 con 0x7fae900b0cb0 2023-12-19T08:03:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 msgr2=0x7fae74062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 0x7fae74062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 msgr2=0x7fae90145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fae8400d890 tx=0x7fae84008040 comp rx=0 tx=0).stop 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 shutdown_connections 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fae740606c0 0x7fae74062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae900b1d00 0x7fae900cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae90159790 0x7fae901549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 --2- 172.21.15.111:0/934640620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae900b0cb0 0x7fae90145910 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.917+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 >> 172.21.15.111:0/934640620 conn(0x7fae900cdef0 msgr2=0x7fae9015e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.921+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 shutdown_connections 2023-12-19T08:03:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:08.921+0000 7fae9722e700 1 -- 172.21.15.111:0/934640620 wait complete. 2023-12-19T08:03:08.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T08:03:08.959 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:08.961+0000 7f518655b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:08.967 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:08.965+0000 7fd250762000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:09.105 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.105+0000 7f518655b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:09.113 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.109+0000 7fd250762000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.117+0000 7fb75c02e700 1 Processor -- start 2023-12-19T08:03:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.117+0000 7fb75c02e700 1 -- start start 2023-12-19T08:03:09.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75c02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb75415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75c02e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb75415c760 0x7fb7540b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75c02e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540b0ac0 con 0x7fb7540cb130 2023-12-19T08:03:09.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75c02e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75415cbe0 con 0x7fb75415c760 2023-12-19T08:03:09.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75c02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75406b210 con 0x7fb7540cb500 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb759dca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb7540cb130 0x7fb75406a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59820/0 (socket says 172.21.15.111:59820) 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb759dca700 1 -- 172.21.15.111:0/4185902165 learned_addr learned my addr 172.21.15.111:0/4185902165 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75a5cb700 1 --2- 172.21.15.111:0/4185902165 >> v2:172.21.15.143:3300/0 conn(0x7fb75415c760 0x7fb7540b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb75415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925408814 0 0) 0x7fb7540b0ac0 con 0x7fb7540cb130 2023-12-19T08:03:09.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb740003570 con 0x7fb7540cb130 2023-12-19T08:03:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300460178 0 0) 0x7fb740003570 con 0x7fb7540cb130 2023-12-19T08:03:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 msgr2=0x7fb75415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb75415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 >> v2:172.21.15.143:3300/0 conn(0x7fb75415c760 msgr2=0x7fb7540b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 --2- 172.21.15.111:0/4185902165 >> v2:172.21.15.143:3300/0 conn(0x7fb75415c760 0x7fb7540b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540ca810 con 0x7fb7540cb130 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fb750001470 con 0x7fb7540cb130 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 >> v1:172.21.15.111:6789/0 conn(0x7fb7540cb130 legacy=0x7fb75406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb740003a60 0x7fb740003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb7400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75415cbe0 con 0x7fb740003a60 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75406b210 con 0x7fb740004810 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74000ba60 con 0x7fb7540cb500 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb759dca700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb740003a60 0x7fb740003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb75a5cb700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb7400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 -- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 msgr2=0x7fb7400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7540cb500 0x7fb7400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 -- 172.21.15.111:0/4185902165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb740003a60 msgr2=0x7fb740003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb740003a60 0x7fb740003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb7595c9700 1 -- 172.21.15.111:0/4185902165 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740003570 con 0x7fb740004810 2023-12-19T08:03:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.121+0000 7fb759dca700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb740003a60 0x7fb740003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:09.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb7595c9700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb740004c80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb744009c30 tx=0x7fb7440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8776211d9b1a0185 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:09.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74400e710 con 0x7fb740004810 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74400e870 con 0x7fb740004810 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb758dc8700 1 -- 172.21.15.111:0/4185902165 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74400ec30 con 0x7fb740004810 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 msgr2=0x7fb740004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 --2- 172.21.15.111:0/4185902165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb740004c80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb744009c30 tx=0x7fb7440091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75a5cb700 1 -- 172.21.15.111:0/4185902165 reap_dead start 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- 172.21.15.111:0/4185902165 shutdown_connections 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- 172.21.15.111:0/4185902165 >> 172.21.15.111:0/4185902165 conn(0x7fb754067bb0 msgr2=0x7fb7540cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- 172.21.15.111:0/4185902165 shutdown_connections 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- 172.21.15.111:0/4185902165 wait complete. 2023-12-19T08:03:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 Processor -- start 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.125+0000 7fb75c02e700 1 -- start start 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb754158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7540c9e10 0x7fb7541565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb754156b20 0x7fb75416d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75416ae60 con 0x7fb7540c9e10 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb754159450 con 0x7fb740004810 2023-12-19T08:03:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540c5870 con 0x7fb754156b20 2023-12-19T08:03:09.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7540c9e10 0x7fb7541565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7540c9e10 0x7fb7541565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37564/0 (socket says 172.21.15.111:37564) 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 -- 172.21.15.111:0/2183710767 learned_addr learned my addr 172.21.15.111:0/2183710767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb754158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75a5cb700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb754156b20 0x7fb75416d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb754156b20 msgr2=0x7fb75416d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb754156b20 0x7fb75416d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 msgr2=0x7fb754158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb740004810 0x7fb754158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540c5b50 con 0x7fb7540c9e10 2023-12-19T08:03:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb7595c9700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7540c9e10 0x7fb7541565e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb74401ef80 tx=0x7fb74401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744023070 con 0x7fb7540c9e10 2023-12-19T08:03:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74400e9e0 con 0x7fb7540c9e10 2023-12-19T08:03:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744006e50 con 0x7fb7540c9e10 2023-12-19T08:03:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75416b0b0 con 0x7fb7540c9e10 2023-12-19T08:03:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.129+0000 7fb75c02e700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75416d7d0 con 0x7fb7540c9e10 2023-12-19T08:03:09.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.133+0000 7fb75c02e700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75416b240 con 0x7fb7540c9e10 2023-12-19T08:03:09.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.133+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 35) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fb744013680 con 0x7fb7540c9e10 2023-12-19T08:03:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.133+0000 7fb749ffb700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.133+0000 7fb759dca700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.133+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.137+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fb7440526c0 con 0x7fb7540c9e10 2023-12-19T08:03:09.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.145+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb74404e2c0 con 0x7fb7540c9e10 2023-12-19T08:03:09.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.337+0000 7fb759dca700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:09.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.337+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:09.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.349+0000 7fb75c02e700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75416ae60 con 0x7fb7380606f0 2023-12-19T08:03:09.388 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.389+0000 7f518655b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:09.392 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.389+0000 7fd250762000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:09.466 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.465+0000 7fd250762000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:09.470 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.469+0000 7f518655b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:09.692 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.689+0000 7fd250762000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:09.706 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.705+0000 7f518655b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:09.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.741+0000 7fb759dca700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:09.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:09.741+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:03:09.781 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.777+0000 7fd250762000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:09.791 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.793+0000 7f518655b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:09.866 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:09.865+0000 7fd250762000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:09.882 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:09.881+0000 7f518655b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:10.285 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:10.281+0000 7fd250762000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:10.297 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:10.297+0000 7f518655b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:10.455 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:10.453+0000 7fd250762000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:10.472 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:10.473+0000 7f518655b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:10.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:10.545+0000 7fb759dca700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:10.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:10.545+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:03:10.833 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:10.829+0000 7fd250762000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:10.850 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:10.849+0000 7f518655b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:11.061 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.057+0000 7fd250762000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:11.064 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.065+0000 7f518655b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:11.155 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.157+0000 7f518655b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:11.165 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.161+0000 7fd250762000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:11.234 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.233+0000 7f518655b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:11.254 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.253+0000 7fd250762000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:11.311 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.313+0000 7f518655b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:11.340 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.337+0000 7fd250762000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:11.538 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.537+0000 7f518655b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:11.577 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.573+0000 7fd250762000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:11.617 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.617+0000 7f518655b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:11.664 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.661+0000 7fd250762000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:11.703 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.701+0000 7f518655b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:11.763 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.761+0000 7fd250762000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:11.893 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.893+0000 7f518655b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:11.976 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:11.973+0000 7fd250762000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:11.977 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:11.977+0000 7f518655b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:12.065 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:12.061+0000 7fd250762000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:12.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:12.149+0000 7fb759dca700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:12.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:12.149+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:03:12.689 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.689+0000 7f518655b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:12.705+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 36) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fb74407c2f0 con 0x7fb7540c9e10 2023-12-19T08:03:12.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:12.709+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 msgr2=0x7fb738062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:12.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:12.709+0000 7fb749ffb700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:12.820 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:12.817+0000 7fd250762000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:12.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5143fe4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5143fe4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5143fe4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5143fe4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5143fe4700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.891 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5147fec700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.891 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5147fec700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.891 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5147fec700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.891 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5147fec700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:12.891 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:12.889+0000 7f5147fec700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:13.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.721+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 37) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fb74407bf80 con 0x7fb7540c9e10 2023-12-19T08:03:13.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.721+0000 7fb749ffb700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.721+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75416ae60 con 0x7fb7380606f0 2023-12-19T08:03:13.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.725+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.725+0000 7fb759dca700 1 --2- 172.21.15.111:0/2183710767 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7380606f0 0x7fb738062b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb750001d20 tx=0x7fb750001e90 comp rx=0 tx=0).ready entity=mgr.6326 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:13.741 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:13.741+0000 7f514effa700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T08:03:13.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.737+0000 7fb749ffb700 1 -- 172.21.15.111:0/2183710767 <== mgr.6326 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fb75416ae60 con 0x7fb7380606f0 2023-12-19T08:03:13.744 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T08:03:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T08:03:13.756 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T08:03:13.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.953+0000 7fe2fe80c700 1 Processor -- start 2023-12-19T08:03:13.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.953+0000 7fe2fe80c700 1 -- start start 2023-12-19T08:03:13.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe80c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2f806b050 0x7fe2f806b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe80c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2f815d4e0 0x7fe2f8159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe80c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f806da40 con 0x7fe2f806c7c0 2023-12-19T08:03:13.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe80c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f806cc40 con 0x7fe2f815d4e0 2023-12-19T08:03:13.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe80c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f806cec0 con 0x7fe2f806b050 2023-12-19T08:03:13.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe00b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2f815d4e0 0x7fe2f8159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe00b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2f815d4e0 0x7fe2f8159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50718/0 (socket says 172.21.15.111:50718) 2023-12-19T08:03:13.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fe00b700 1 -- 172.21.15.111:0/74627388 learned_addr learned my addr 172.21.15.111:0/74627388 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:13.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fd009700 1 --1- 172.21.15.111:0/74627388 >> v1:172.21.15.111:6789/0 conn(0x7fe2f806c7c0 0x7fe2f81594b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41520/0 (socket says 172.21.15.111:41520) 2023-12-19T08:03:13.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.957+0000 7fe2fd80a700 1 --2- 172.21.15.111:0/74627388 >> v2:172.21.15.143:3300/0 conn(0x7fe2f806b050 0x7fe2f806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe00b700 1 -- 172.21.15.111:0/74627388 >> v2:172.21.15.143:3300/0 conn(0x7fe2f806b050 msgr2=0x7fe2f806b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe00b700 1 --2- 172.21.15.111:0/74627388 >> v2:172.21.15.143:3300/0 conn(0x7fe2f806b050 0x7fe2f806b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe00b700 1 -- 172.21.15.111:0/74627388 >> v1:172.21.15.111:6789/0 conn(0x7fe2f806c7c0 legacy=0x7fe2f81594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe00b700 1 -- 172.21.15.111:0/74627388 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f806d1a0 con 0x7fe2f815d4e0 2023-12-19T08:03:13.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fc808700 1 -- 172.21.15.111:0/74627388 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615833269 0 0) 0x7fe2f806da40 con 0x7fe2f806c7c0 2023-12-19T08:03:13.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe00b700 1 --2- 172.21.15.111:0/74627388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f815d4e0 0x7fe2f8159bc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8068010 tx=0x7fe2f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a43826883407e7b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fc808700 1 -- 172.21.15.111:0/74627388 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000e070 con 0x7fe2f815d4e0 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fc808700 1 -- 172.21.15.111:0/74627388 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f000ba20 con 0x7fe2f815d4e0 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fc808700 1 -- 172.21.15.111:0/74627388 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00092e0 con 0x7fe2f815d4e0 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 -- 172.21.15.111:0/74627388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f815d4e0 msgr2=0x7fe2f8159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/74627388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f815d4e0 0x7fe2f8159bc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8068010 tx=0x7fe2f0009840 comp rx=0 tx=0).stop 2023-12-19T08:03:13.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 -- 172.21.15.111:0/74627388 shutdown_connections 2023-12-19T08:03:13.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/74627388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f815d4e0 0x7fe2f8159bc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:13.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/74627388 >> v2:172.21.15.143:3300/0 conn(0x7fe2f806b050 0x7fe2f806b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:13.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 -- 172.21.15.111:0/74627388 >> 172.21.15.111:0/74627388 conn(0x7fe2f80c8cd0 msgr2=0x7fe2f80c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:13.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 -- 172.21.15.111:0/74627388 shutdown_connections 2023-12-19T08:03:13.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.961+0000 7fe2fe80c700 1 -- 172.21.15.111:0/74627388 wait complete. 2023-12-19T08:03:13.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 Processor -- start 2023-12-19T08:03:13.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 -- start start 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f806c7c0 0x7fe2f80a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f815d4e0 0x7fe2f80c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f80a97b0 con 0x7fe2f806b050 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f80c5de0 con 0x7fe2f815d4e0 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe80c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f80c9e10 con 0x7fe2f806c7c0 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50722/0 (socket says 172.21.15.111:50722) 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 -- 172.21.15.111:0/92563242 learned_addr learned my addr 172.21.15.111:0/92563242 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd009700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f806c7c0 0x7fe2f80a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fe00b700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f815d4e0 0x7fe2f80c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 -- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f806c7c0 msgr2=0x7fe2f80a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f806c7c0 0x7fe2f80a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 -- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f815d4e0 msgr2=0x7fe2f80c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f815d4e0 0x7fe2f80c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.965+0000 7fe2fd80a700 1 -- 172.21.15.111:0/92563242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0009ae0 con 0x7fe2f806b050 2023-12-19T08:03:13.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2fd80a700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400ec80 tx=0x7fe2f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400c850 con 0x7fe2f806b050 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f400c9b0 con 0x7fe2f806b050 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400cd50 con 0x7fe2f806b050 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f80b0020 con 0x7fe2f806b050 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f80b0990 con 0x7fe2f806b050 2023-12-19T08:03:13.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.969+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f804be40 con 0x7fe2f806b050 2023-12-19T08:03:13.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.973+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 37) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fe2f4013680 con 0x7fe2f806b050 2023-12-19T08:03:13.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.973+0000 7fe2edffb700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 0x7fe2dc0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:13.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.973+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fe2f407cb80 con 0x7fe2f806b050 2023-12-19T08:03:13.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.973+0000 7fe2fd009700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 0x7fe2dc0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:13.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.973+0000 7fe2fd009700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 0x7fe2dc0628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80cb960 tx=0x7fe2e8007d50 comp rx=0 tx=0).ready entity=mgr.6326 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:13.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:13.981+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2f404cf80 con 0x7fe2f806b050 2023-12-19T08:03:14.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.189+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe2f806d480 con 0x7fe2f806b050 2023-12-19T08:03:14.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.773+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fe2f404d350 con 0x7fe2f806b050 2023-12-19T08:03:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.777+0000 7fe2edffb700 1 -- 172.21.15.111:0/92563242 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v38) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe2f4050a10 con 0x7fe2f806b050 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 msgr2=0x7fe2dc0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 0x7fe2dc0628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80cb960 tx=0x7fe2e8007d50 comp rx=0 tx=0).stop 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 msgr2=0x7fe2f8169fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400ec80 tx=0x7fe2f40091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 shutdown_connections 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2dc0603f0 0x7fe2dc0628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f815d4e0 0x7fe2f80c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f806c7c0 0x7fe2f80a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 --2- 172.21.15.111:0/92563242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f806b050 0x7fe2f8169fd0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 >> 172.21.15.111:0/92563242 conn(0x7fe2f80c8cd0 msgr2=0x7fe2f815aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 shutdown_connections 2023-12-19T08:03:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:14.781+0000 7fe2fe80c700 1 -- 172.21.15.111:0/92563242 wait complete. 2023-12-19T08:03:14.801 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:03:14.801 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:03:14.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:03:14.887 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:03:14.887 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T08:03:14.887 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T08:03:14.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:14.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[50663]: fuse finished with error 0 and tester_r 0 2023-12-19T08:03:14.922 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:03:14.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:14.922 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T08:03:14.974 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:14.973+0000 7f9e63161000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:14.985 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:03:14.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:14.986 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T08:03:15.005 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T08:03:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:03:15.006 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:03:15.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:15.007 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T08:03:15.013 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:15.009+0000 7f60ec760000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:15.032 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T08:03:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:03:15.033 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:03:15.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:15.034 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T08:03:15.074 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T08:03:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:03:15.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:15.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 Processor -- start 2023-12-19T08:03:15.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 -- start start 2023-12-19T08:03:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d4159f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb9d406ced0 0x7fb9d40ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40670e0 con 0x7fb9d406a450 2023-12-19T08:03:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d406b560 con 0x7fb9d406ced0 2023-12-19T08:03:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d946e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d406d650 con 0x7fb9d4156990 2023-12-19T08:03:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d27fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb9d406a450 0x7fb9d406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42498/0 (socket says 172.21.15.111:42498) 2023-12-19T08:03:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb9d406ced0 0x7fb9d40ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d27fc700 1 -- 172.21.15.111:0/3952643974 learned_addr learned my addr 172.21.15.111:0/3952643974 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.237+0000 7fb9d2ffd700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d4159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d1ffb700 1 -- 172.21.15.111:0/3952643974 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112454879 0 0) 0x7fb9d40670e0 con 0x7fb9d406a450 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d1ffb700 1 -- 172.21.15.111:0/3952643974 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8003570 con 0x7fb9d406a450 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d3fff700 1 -- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 msgr2=0x7fb9d4159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d3fff700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d4159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d3fff700 1 -- 172.21.15.111:0/3952643974 >> v1:172.21.15.143:6789/0 conn(0x7fb9d406a450 legacy=0x7fb9d406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d3fff700 1 -- 172.21.15.111:0/3952643974 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d415a450 con 0x7fb9d406ced0 2023-12-19T08:03:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d3fff700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40ce3b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009e60 tx=0x7fb9cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b0a645b6274f17 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d1ffb700 1 -- 172.21.15.111:0/3952643974 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00e070 con 0x7fb9d406ced0 2023-12-19T08:03:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d1ffb700 1 -- 172.21.15.111:0/3952643974 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc00bab0 con 0x7fb9d406ced0 2023-12-19T08:03:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d1ffb700 1 -- 172.21.15.111:0/3952643974 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00be70 con 0x7fb9d406ced0 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 -- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 msgr2=0x7fb9d40ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40ce3b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009e60 tx=0x7fb9cc009780 comp rx=0 tx=0).stop 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 -- 172.21.15.111:0/3952643974 shutdown_connections 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40ce3b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3952643974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d4159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 -- 172.21.15.111:0/3952643974 >> 172.21.15.111:0/3952643974 conn(0x7fb9d4069d80 msgr2=0x7fb9d40cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 -- 172.21.15.111:0/3952643974 shutdown_connections 2023-12-19T08:03:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.241+0000 7fb9d946e700 1 -- 172.21.15.111:0/3952643974 wait complete. 2023-12-19T08:03:15.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 Processor -- start 2023-12-19T08:03:15.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 -- start start 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d40ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4152ea0 con 0x7fb9d406a450 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40cf350 con 0x7fb9d406ced0 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d946e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4153440 con 0x7fb9d4156990 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50752/0 (socket says 172.21.15.111:50752) 2023-12-19T08:03:15.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/3069318230 learned_addr learned my addr 172.21.15.111:0/3069318230 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.245+0000 7fb9d3fff700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d40ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d27fc700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 msgr2=0x7fb9d40ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d40ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 msgr2=0x7fb9d40b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/3069318230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc009a20 con 0x7fb9d406a450 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d27fc700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d2ffd700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400ec80 tx=0x7fb9c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c400c9d0 con 0x7fb9d406a450 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c400cb30 con 0x7fb9d406a450 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d40ced90 con 0x7fb9d406a450 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4013470 con 0x7fb9d406a450 2023-12-19T08:03:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d40cf120 con 0x7fb9d406a450 2023-12-19T08:03:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d414e960 con 0x7fb9d406a450 2023-12-19T08:03:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.249+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fb9c4013820 con 0x7fb9d406a450 2023-12-19T08:03:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.253+0000 7fb9caffd700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 0x7fb9bc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.253+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fb9c407d290 con 0x7fb9d406a450 2023-12-19T08:03:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.253+0000 7fb9d27fc700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 msgr2=0x7fb9bc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.253+0000 7fb9d27fc700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 0x7fb9bc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:15.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.257+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9c404d160 con 0x7fb9d406a450 2023-12-19T08:03:15.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.457+0000 7fb9d27fc700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 msgr2=0x7fb9bc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:15.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.457+0000 7fb9d27fc700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 0x7fb9bc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:15.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.465+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9d4071120 con 0x7fb9d406a450 2023-12-19T08:03:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.465+0000 7fb9caffd700 1 -- 172.21.15.111:0/3069318230 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7fb9c4050780 con 0x7fb9d406a450 2023-12-19T08:03:15.469 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 msgr2=0x7fb9bc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 0x7fb9bc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 msgr2=0x7fb9d414e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400ec80 tx=0x7fb9c40091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 shutdown_connections 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9bc060700 0x7fb9bc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9d406ced0 0x7fb9d40b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9d4156990 0x7fb9d40ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 --2- 172.21.15.111:0/3069318230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d406a450 0x7fb9d414e360 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 >> 172.21.15.111:0/3069318230 conn(0x7fb9d4069d80 msgr2=0x7fb9d4158d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:15.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 shutdown_connections 2023-12-19T08:03:15.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.469+0000 7fb9d946e700 1 -- 172.21.15.111:0/3069318230 wait complete. 2023-12-19T08:03:15.475 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:15.493 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:15.519 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:15.521+0000 7f9e63161000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:15.578 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:15.577+0000 7f60ec760000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:15.591 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:15.593+0000 7f9e63161000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:15.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 Processor -- start 2023-12-19T08:03:15.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 -- start start 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febc4140710 0x7febc40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc406e150 con 0x7febc40c75e0 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc406d540 con 0x7febc4140710 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc8a6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc406d820 con 0x7febc4140ae0 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febc4140710 0x7febc40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7febc4140710 0x7febc40c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50754/0 (socket says 172.21.15.111:50754) 2023-12-19T08:03:15.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc259c700 1 -- 172.21.15.111:0/1550365040 learned_addr learned my addr 172.21.15.111:0/1550365040 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:15.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.645+0000 7febc1d9b700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc259c700 1 -- 172.21.15.111:0/1550365040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 msgr2=0x7febc40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:15.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc259c700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc259c700 1 -- 172.21.15.111:0/1550365040 >> v1:172.21.15.111:6789/0 conn(0x7febc40c75e0 legacy=0x7febc406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc259c700 1 -- 172.21.15.111:0/1550365040 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc406db00 con 0x7febc4140710 2023-12-19T08:03:15.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc259c700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc4140710 0x7febc40c70a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7febb40099b0 tx=0x7febb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3126bb0b86e9a98a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:15.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc159a700 1 -- 172.21.15.111:0/1550365040 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4009ea0 con 0x7febc4140710 2023-12-19T08:03:15.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc159a700 1 -- 172.21.15.111:0/1550365040 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb400b860 con 0x7febc4140710 2023-12-19T08:03:15.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc159a700 1 -- 172.21.15.111:0/1550365040 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb400bc00 con 0x7febc4140710 2023-12-19T08:03:15.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 -- 172.21.15.111:0/1550365040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc4140710 msgr2=0x7febc40c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc4140710 0x7febc40c70a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7febb40099b0 tx=0x7febb4009250 comp rx=0 tx=0).stop 2023-12-19T08:03:15.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 -- 172.21.15.111:0/1550365040 shutdown_connections 2023-12-19T08:03:15.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 --2- 172.21.15.111:0/1550365040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc4140710 0x7febc40c70a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 -- 172.21.15.111:0/1550365040 >> 172.21.15.111:0/1550365040 conn(0x7febc40c8850 msgr2=0x7febc40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:15.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 -- 172.21.15.111:0/1550365040 shutdown_connections 2023-12-19T08:03:15.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.649+0000 7febc8a6e700 1 -- 172.21.15.111:0/1550365040 wait complete. 2023-12-19T08:03:15.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 Processor -- start 2023-12-19T08:03:15.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 -- start start 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc4140710 0x7febc413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40a9610 con 0x7febc40c75e0 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40a9e40 con 0x7febc4140710 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc8a6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40aa120 con 0x7febc4140ae0 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50766/0 (socket says 172.21.15.111:50766) 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 -- 172.21.15.111:0/4274869148 learned_addr learned my addr 172.21.15.111:0/4274869148 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:15.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc1d9b700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc4140710 0x7febc413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc2d9d700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 -- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 msgr2=0x7febc40c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40c1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 -- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc4140710 msgr2=0x7febc413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc4140710 0x7febc413f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.653+0000 7febc259c700 1 -- 172.21.15.111:0/4274869148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40094f0 con 0x7febc40c75e0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febc259c700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7febc406a280 tx=0x7febb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb400bbe0 con 0x7febc40c75e0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4011420 con 0x7febc40c75e0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc40aa400 con 0x7febc40c75e0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb40117c0 con 0x7febc40c75e0 2023-12-19T08:03:15.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc413fe50 con 0x7febc40c75e0 2023-12-19T08:03:15.660 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:15.657+0000 7f60ec760000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:15.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.657+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc40a9ca0 con 0x7febc40c75e0 2023-12-19T08:03:15.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.661+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7febb4011bf0 con 0x7febc40c75e0 2023-12-19T08:03:15.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.661+0000 7febb27fc700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 0x7feba4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:15.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.661+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7febb407e6b0 con 0x7febc40c75e0 2023-12-19T08:03:15.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.661+0000 7febc1d9b700 1 -- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 msgr2=0x7feba4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:15.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.661+0000 7febc1d9b700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 0x7feba4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:15.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.665+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7febc40a9ca0 con 0x7febc40c75e0 2023-12-19T08:03:15.733 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:15.733+0000 7f9e63161000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:15.818 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:15.817+0000 7f60ec760000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.837+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc413fe50 con 0x7febc40c75e0 2023-12-19T08:03:15.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.841+0000 7febb27fc700 1 -- 172.21.15.111:0/4274869148 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16895 (secure 0 0 0) 0x7febc413fe50 con 0x7febc40c75e0 2023-12-19T08:03:15.843 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:15.843 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":100,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:12.694352+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/1273338574":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3396798404":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3432369124":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1714582390":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1261051935":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3477898242":"2023-12-20T08:03:12.694202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 msgr2=0x7feba4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 0x7feba4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 msgr2=0x7febc40aa920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7febc406a280 tx=0x7febb4017000 comp rx=0 tx=0).stop 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 shutdown_connections 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7feba4060700 0x7feba4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc4140710 0x7febc413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febc4140ae0 0x7febc40c1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 --2- 172.21.15.111:0/4274869148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febc40c75e0 0x7febc40aa920 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 >> 172.21.15.111:0/4274869148 conn(0x7febc40c8850 msgr2=0x7febc40cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 shutdown_connections 2023-12-19T08:03:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:15.845+0000 7febc8a6e700 1 -- 172.21.15.111:0/4274869148 wait complete. 2023-12-19T08:03:15.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:02:56.370138+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:02:57.369378+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T08:03:15.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:03:15.985 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:15.985+0000 7f9e63161000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:16.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.033+0000 7f32d9745700 1 Processor -- start 2023-12-19T08:03:16.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.033+0000 7f32d9745700 1 -- start start 2023-12-19T08:03:16.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d9745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 0x7f32d406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d9745700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d406d3d0 con 0x7f32d406d860 2023-12-19T08:03:16.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d9745700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40cc880 con 0x7f32d40672f0 2023-12-19T08:03:16.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d9745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d406e1d0 con 0x7f32d40c8b60 2023-12-19T08:03:16.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d27fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32d406d860 0x7f32d40c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41546/0 (socket says 172.21.15.111:41546) 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d27fc700 1 -- 172.21.15.111:0/1389730701 learned_addr learned my addr 172.21.15.111:0/1389730701 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d37fe700 1 --2- 172.21.15.111:0/1389730701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 0x7f32d406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331991272 0 0) 0x7f32d406d3d0 con 0x7f32d406d860 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc003570 con 0x7f32d406d860 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151590385 0 0) 0x7f32d40cc880 con 0x7f32d40672f0 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d406d3d0 con 0x7f32d40672f0 2023-12-19T08:03:16.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134452042 0 0) 0x7f32bc003570 con 0x7f32d406d860 2023-12-19T08:03:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d37fe700 1 -- 172.21.15.111:0/1389730701 >> v1:172.21.15.143:6789/0 conn(0x7f32d40672f0 legacy=0x7f32d40cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d37fe700 1 -- 172.21.15.111:0/1389730701 >> v1:172.21.15.111:6789/0 conn(0x7f32d406d860 legacy=0x7f32d40c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d37fe700 1 -- 172.21.15.111:0/1389730701 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d406c7c0 con 0x7f32d40c8b60 2023-12-19T08:03:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d37fe700 1 --2- 172.21.15.111:0/1389730701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 0x7f32d406dc90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009e90 tx=0x7f32cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56058f89b6107aba server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc00e070 con 0x7f32d40c8b60 2023-12-19T08:03:16.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc00b900 con 0x7f32d40c8b60 2023-12-19T08:03:16.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.037+0000 7f32d1ffb700 1 -- 172.21.15.111:0/1389730701 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c80013f0 con 0x7f32d40c8b60 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- 172.21.15.111:0/1389730701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 msgr2=0x7f32d406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 --2- 172.21.15.111:0/1389730701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 0x7f32d406dc90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009e90 tx=0x7f32cc009780 comp rx=0 tx=0).stop 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- 172.21.15.111:0/1389730701 shutdown_connections 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 --2- 172.21.15.111:0/1389730701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d40c8b60 0x7f32d406dc90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- 172.21.15.111:0/1389730701 >> 172.21.15.111:0/1389730701 conn(0x7f32d40cdc00 msgr2=0x7f32d40ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- 172.21.15.111:0/1389730701 shutdown_connections 2023-12-19T08:03:16.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- 172.21.15.111:0/1389730701 wait complete. 2023-12-19T08:03:16.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 Processor -- start 2023-12-19T08:03:16.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.041+0000 7f32d9745700 1 -- start start 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d406d860 0x7f32d415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 0x7f32d415a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d415af70 con 0x7f32d40672f0 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d415b250 con 0x7f32d40c8b60 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d415b530 con 0x7f32d406d860 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50778/0 (socket says 172.21.15.111:50778) 2023-12-19T08:03:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 -- 172.21.15.111:0/1250315973 learned_addr learned my addr 172.21.15.111:0/1250315973 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d37fe700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 0x7f32d415a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d27fc700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d406d860 0x7f32d415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 -- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d406d860 msgr2=0x7f32d415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d406d860 0x7f32d415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 -- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 msgr2=0x7f32d415a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 0x7f32d415a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 -- 172.21.15.111:0/1250315973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc009a20 con 0x7f32d40672f0 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d37fe700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 0x7f32d415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d2ffd700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f32c400c790 tx=0x7f32c4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4011590 con 0x7f32d40672f0 2023-12-19T08:03:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c40116f0 con 0x7f32d40672f0 2023-12-19T08:03:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d415b810 con 0x7f32d40672f0 2023-12-19T08:03:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4011ab0 con 0x7f32d40672f0 2023-12-19T08:03:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.045+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d415a920 con 0x7f32d40672f0 2023-12-19T08:03:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.049+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d404be40 con 0x7f32d40672f0 2023-12-19T08:03:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.049+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f32c4012420 con 0x7f32d40672f0 2023-12-19T08:03:16.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.053+0000 7f32baffd700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 0x7f32c0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.053+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f32c407cc30 con 0x7f32d40672f0 2023-12-19T08:03:16.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.053+0000 7f32d27fc700 1 -- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 msgr2=0x7f32c0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:16.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.053+0000 7f32d27fc700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 0x7f32c0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:16.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.057+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f32c404cb20 con 0x7f32d40672f0 2023-12-19T08:03:16.061 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.061+0000 7f9e63161000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:16.106 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:16.105+0000 7f60ec760000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:16.184 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:16.181+0000 7f60ec760000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:16.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.241+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32d406caa0 con 0x7f32d40672f0 2023-12-19T08:03:16.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.241+0000 7f32baffd700 1 -- 172.21.15.111:0/1250315973 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32c4050140 con 0x7f32d40672f0 2023-12-19T08:03:16.243 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:03:16.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 msgr2=0x7f32c0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:16.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 0x7f32c0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 msgr2=0x7f32d415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f32c400c790 tx=0x7f32c4007ea0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 shutdown_connections 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32c0060700 0x7f32c0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32d40c8b60 0x7f32d415a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d406d860 0x7f32d415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 --2- 172.21.15.111:0/1250315973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32d40672f0 0x7f32d415c990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 >> 172.21.15.111:0/1250315973 conn(0x7f32d40cdc00 msgr2=0x7f32d40c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 shutdown_connections 2023-12-19T08:03:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.245+0000 7f32d9745700 1 -- 172.21.15.111:0/1250315973 wait complete. 2023-12-19T08:03:16.265 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T08:03:16.282 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.281+0000 7f9e63161000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:16.364 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.365+0000 7f9e63161000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:16.407 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:16.405+0000 7f60ec760000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.433+0000 7f984cd05700 1 Processor -- start 2023-12-19T08:03:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.433+0000 7f984cd05700 1 -- start start 2023-12-19T08:03:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.433+0000 7f984cd05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f98480ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984cd05700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984806d5e0 con 0x7f98480ca2c0 2023-12-19T08:03:16.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984cd05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98480cbdf0 con 0x7f98480cb970 2023-12-19T08:03:16.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984cd05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98480cc0d0 con 0x7f9848158e20 2023-12-19T08:03:16.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9846d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f98480ca2c0 0x7f984806ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41562/0 (socket says 172.21.15.111:41562) 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9846d9d700 1 -- 172.21.15.111:0/1747849983 learned_addr learned my addr 172.21.15.111:0/1747849983 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984659c700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f98480ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377589936 0 0) 0x7f984806d5e0 con 0x7f98480ca2c0 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9830003570 con 0x7f98480ca2c0 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905420955 0 0) 0x7f98480cbdf0 con 0x7f98480cb970 2023-12-19T08:03:16.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984806d5e0 con 0x7f98480cb970 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472719075 0 0) 0x7f9830003570 con 0x7f98480ca2c0 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 msgr2=0x7f98480ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f98480ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 >> v1:172.21.15.143:6789/0 conn(0x7f98480cb970 legacy=0x7f984806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98480cc3b0 con 0x7f98480ca2c0 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f9840001720 con 0x7f98480ca2c0 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 >> v1:172.21.15.111:6789/0 conn(0x7f98480ca2c0 legacy=0x7f984806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f9830004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f98300037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9830003cf0 0x7f9830004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98480cc0d0 con 0x7f9830005230 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983000bbf0 con 0x7f9830003cf0 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983000bdb0 con 0x7f9848158e20 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f98300037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9846d9d700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9830003cf0 0x7f9830004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984659c700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f9830004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 -- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 msgr2=0x7f9830004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f9830004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 -- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9830003cf0 msgr2=0x7f9830004160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9830003cf0 0x7f9830004160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 -- 172.21.15.111:0/1747849983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830003570 con 0x7f9830005230 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984659c700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9848158e20 0x7f9830004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f9845d9b700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f98300037b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f983c001a10 tx=0x7f983c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f7ed908c3e672a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c00c5d0 con 0x7f9830005230 2023-12-19T08:03:16.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c00c730 con 0x7f9830005230 2023-12-19T08:03:16.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.437+0000 7f984559a700 1 -- 172.21.15.111:0/1747849983 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c00cad0 con 0x7f9830005230 2023-12-19T08:03:16.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- 172.21.15.111:0/1747849983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 msgr2=0x7f98300037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 --2- 172.21.15.111:0/1747849983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f98300037b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f983c001a10 tx=0x7f983c007d30 comp rx=0 tx=0).stop 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f9846d9d700 1 -- 172.21.15.111:0/1747849983 reap_dead start 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- 172.21.15.111:0/1747849983 shutdown_connections 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- 172.21.15.111:0/1747849983 >> 172.21.15.111:0/1747849983 conn(0x7f9848067bb0 msgr2=0x7f98480d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- 172.21.15.111:0/1747849983 shutdown_connections 2023-12-19T08:03:16.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- 172.21.15.111:0/1747849983 wait complete. 2023-12-19T08:03:16.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 Processor -- start 2023-12-19T08:03:16.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.441+0000 7f984cd05700 1 -- start start 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98480c4f30 0x7f98480c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9848159420 0x7f9848159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98480c58e0 con 0x7f9830005230 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984815b5d0 con 0x7f9848159420 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984815aca0 con 0x7f98480c4f30 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50806/0 (socket says 172.21.15.111:50806) 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 -- 172.21.15.111:0/2189716967 learned_addr learned my addr 172.21.15.111:0/2189716967 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f9846d9d700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9848159420 0x7f9848159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f9845d9b700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98480c4f30 0x7f98480c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.449 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.449+0000 7f9e63161000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 -- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98480c4f30 msgr2=0x7f98480c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98480c4f30 0x7f98480c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 -- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9848159420 msgr2=0x7f9848159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9848159420 0x7f9848159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 -- 172.21.15.111:0/2189716967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984815af80 con 0x7f9830005230 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984659c700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9838010ca0 tx=0x7f9838008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983800e810 con 0x7f9830005230 2023-12-19T08:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983800e970 con 0x7f9830005230 2023-12-19T08:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983800ed10 con 0x7f9830005230 2023-12-19T08:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984815b260 con 0x7f9830005230 2023-12-19T08:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.445+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9848147100 con 0x7f9830005230 2023-12-19T08:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9848071120 con 0x7f9830005230 2023-12-19T08:03:16.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f983801c610 con 0x7f9830005230 2023-12-19T08:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f98367fc700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 0x7f9828062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f983807d2e0 con 0x7f9830005230 2023-12-19T08:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f9845d9b700 1 -- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 msgr2=0x7f9828062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.449+0000 7f9845d9b700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 0x7f9828062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.453+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f983804d1b0 con 0x7f9830005230 2023-12-19T08:03:16.496 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:16.493+0000 7f60ec760000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:16.586 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:16.585+0000 7f60ec760000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:16.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.625+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f984815ade0 con 0x7f9830005230 2023-12-19T08:03:16.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.625+0000 7f98367fc700 1 -- 172.21.15.111:0/2189716967 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f984815ade0 con 0x7f9830005230 2023-12-19T08:03:16.627 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T08:03:16.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 msgr2=0x7f9828062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:16.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 0x7f9828062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 msgr2=0x7f9848146bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9838010ca0 tx=0x7f9838008040 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 shutdown_connections 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9828060730 0x7f9828062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9848159420 0x7f9848159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98480c4f30 0x7f98480c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 --2- 172.21.15.111:0/2189716967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9830005230 0x7f9848146bc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 >> 172.21.15.111:0/2189716967 conn(0x7f9848067bb0 msgr2=0x7f98480ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 shutdown_connections 2023-12-19T08:03:16.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.629+0000 7f984cd05700 1 -- 172.21.15.111:0/2189716967 wait complete. 2023-12-19T08:03:16.652 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T08:03:16.812 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.813+0000 7f9e63161000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:16.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.813+0000 7fdd2dc6e700 1 Processor -- start 2023-12-19T08:03:16.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.813+0000 7fdd2dc6e700 1 -- start start 2023-12-19T08:03:16.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd2dc6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd281514a0 0x7fdd28157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd2dc6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 0x7fdd28158740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd2dc6e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280cbee0 con 0x7fdd280cae60 2023-12-19T08:03:16.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd2dc6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd28158530 con 0x7fdd281514a0 2023-12-19T08:03:16.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd2dc6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280cc370 con 0x7fdd2806b7a0 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd26ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdd280cae60 0x7fdd2806b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41574/0 (socket says 172.21.15.111:41574) 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd26ffd700 1 -- 172.21.15.111:0/4202070002 learned_addr learned my addr 172.21.15.111:0/4202070002 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 --2- 172.21.15.111:0/4202070002 >> v2:172.21.15.111:3300/0 conn(0x7fdd281514a0 0x7fdd28157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd27fff700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 0x7fdd28158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd267fc700 1 -- 172.21.15.111:0/4202070002 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566734507 0 0) 0x7fdd280cbee0 con 0x7fdd280cae60 2023-12-19T08:03:16.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd267fc700 1 -- 172.21.15.111:0/4202070002 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd10003570 con 0x7fdd280cae60 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 -- 172.21.15.111:0/4202070002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 msgr2=0x7fdd28158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 0x7fdd28158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 -- 172.21.15.111:0/4202070002 >> v1:172.21.15.111:6789/0 conn(0x7fdd280cae60 legacy=0x7fdd2806b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 -- 172.21.15.111:0/4202070002 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280cb2d0 con 0x7fdd281514a0 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd27fff700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 0x7fdd28158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:16.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd277fe700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd281514a0 0x7fdd28157ff0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009e60 tx=0x7fdd18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0951f92fa466de0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd267fc700 1 -- 172.21.15.111:0/4202070002 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800e070 con 0x7fdd281514a0 2023-12-19T08:03:16.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd267fc700 1 -- 172.21.15.111:0/4202070002 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800bb60 con 0x7fdd281514a0 2023-12-19T08:03:16.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.817+0000 7fdd267fc700 1 -- 172.21.15.111:0/4202070002 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18012430 con 0x7fdd281514a0 2023-12-19T08:03:16.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/4202070002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd281514a0 msgr2=0x7fdd28157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd281514a0 0x7fdd28157ff0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009e60 tx=0x7fdd18009780 comp rx=0 tx=0).stop 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/4202070002 shutdown_connections 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd2806b7a0 0x7fdd28158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/4202070002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd281514a0 0x7fdd28157ff0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/4202070002 >> 172.21.15.111:0/4202070002 conn(0x7fdd280cc580 msgr2=0x7fdd280cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/4202070002 shutdown_connections 2023-12-19T08:03:16.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/4202070002 wait complete. 2023-12-19T08:03:16.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 Processor -- start 2023-12-19T08:03:16.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.821+0000 7fdd2dc6e700 1 -- start start 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd280cae60 0x7fdd28153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd281514a0 0x7fdd281400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28155510 con 0x7fdd2806b7a0 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28154470 con 0x7fdd280cae60 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28154750 con 0x7fdd281514a0 2023-12-19T08:03:16.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50828/0 (socket says 172.21.15.111:50828) 2023-12-19T08:03:16.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 -- 172.21.15.111:0/887370504 learned_addr learned my addr 172.21.15.111:0/887370504 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:16.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd27fff700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd281514a0 0x7fdd281400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd26ffd700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd280cae60 0x7fdd28153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 -- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd281514a0 msgr2=0x7fdd281400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd281514a0 0x7fdd281400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 -- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd280cae60 msgr2=0x7fdd28153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd280cae60 0x7fdd28153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 -- 172.21.15.111:0/887370504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18009a20 con 0x7fdd2806b7a0 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd277fe700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fdd28064900 tx=0x7fdd18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18012cf0 con 0x7fdd2806b7a0 2023-12-19T08:03:16.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18012e70 con 0x7fdd2806b7a0 2023-12-19T08:03:16.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28154a30 con 0x7fdd2806b7a0 2023-12-19T08:03:16.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd180276a0 con 0x7fdd2806b7a0 2023-12-19T08:03:16.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28154ff0 con 0x7fdd2806b7a0 2023-12-19T08:03:16.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.825+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28155210 con 0x7fdd2806b7a0 2023-12-19T08:03:16.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.829+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fdd18024070 con 0x7fdd2806b7a0 2023-12-19T08:03:16.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.829+0000 7fdd0f7fe700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 0x7fdd14062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.829+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fdd1807f440 con 0x7fdd2806b7a0 2023-12-19T08:03:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.829+0000 7fdd26ffd700 1 -- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 msgr2=0x7fdd14062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.829+0000 7fdd26ffd700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 0x7fdd14062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:16.837+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd1804f310 con 0x7fdd2806b7a0 2023-12-19T08:03:16.977 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:16.977+0000 7f9e63161000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:17.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd28071120 con 0x7fdd2806b7a0 2023-12-19T08:03:17.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd0f7fe700 1 -- 172.21.15.111:0/887370504 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd18052930 con 0x7fdd2806b7a0 2023-12-19T08:03:17.007 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T08:03:17.007 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.005+0000 7f60ec760000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 msgr2=0x7fdd14062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 0x7fdd14062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 msgr2=0x7fdd28153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.005+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fdd28064900 tx=0x7fdd18019000 comp rx=0 tx=0).stop 2023-12-19T08:03:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 shutdown_connections 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd14060700 0x7fdd14062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd280cae60 0x7fdd28153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd281514a0 0x7fdd281400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 --2- 172.21.15.111:0/887370504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2806b7a0 0x7fdd28153410 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 >> 172.21.15.111:0/887370504 conn(0x7fdd280cc580 msgr2=0x7fdd280b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 shutdown_connections 2023-12-19T08:03:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.009+0000 7fdd2dc6e700 1 -- 172.21.15.111:0/887370504 wait complete. 2023-12-19T08:03:17.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:17.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.180+0000 7fc65ba78700 1 Processor -- start 2023-12-19T08:03:17.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.180+0000 7fc65ba78700 1 -- start start 2023-12-19T08:03:17.184 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.180+0000 7f60ec760000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65ba78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 0x7fc6540a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65ba78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65406dc20 con 0x7fc6540c1d60 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65ba78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65406d940 con 0x7fc6540ae140 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65ba78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540c2200 con 0x7fc6540ae510 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65a015700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc6540c1d60 0x7fc6540680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41580/0 (socket says 172.21.15.111:41580) 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc65a015700 1 -- 172.21.15.111:0/3262070492 learned_addr learned my addr 172.21.15.111:0/3262070492 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:17.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc659013700 1 --2- 172.21.15.111:0/3262070492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 0x7fc6540a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038273827 0 0) 0x7fc65406dc20 con 0x7fc6540c1d60 2023-12-19T08:03:17.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc640003570 con 0x7fc6540c1d60 2023-12-19T08:03:17.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc659013700 1 -- 172.21.15.111:0/3262070492 >> v1:172.21.15.143:6789/0 conn(0x7fc6540ae140 legacy=0x7fc65406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598591722 0 0) 0x7fc65406d940 con 0x7fc6540ae140 2023-12-19T08:03:17.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc659013700 1 -- 172.21.15.111:0/3262070492 >> v1:172.21.15.111:6789/0 conn(0x7fc6540c1d60 legacy=0x7fc6540680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc659013700 1 -- 172.21.15.111:0/3262070492 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540c6e20 con 0x7fc6540ae510 2023-12-19T08:03:17.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc659013700 1 --2- 172.21.15.111:0/3262070492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 0x7fc6540a8850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc644009910 tx=0x7fc6440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c383b02fe73c9544 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:17.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400b680 con 0x7fc6540ae510 2023-12-19T08:03:17.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64400b7e0 con 0x7fc6540ae510 2023-12-19T08:03:17.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.184+0000 7fc658812700 1 -- 172.21.15.111:0/3262070492 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400bba0 con 0x7fc6540ae510 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- 172.21.15.111:0/3262070492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 msgr2=0x7fc6540a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3262070492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 0x7fc6540a8850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc644009910 tx=0x7fc6440091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- 172.21.15.111:0/3262070492 shutdown_connections 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3262070492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540ae510 0x7fc6540a8850 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- 172.21.15.111:0/3262070492 >> 172.21.15.111:0/3262070492 conn(0x7fc6540c8850 msgr2=0x7fc6540c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- 172.21.15.111:0/3262070492 shutdown_connections 2023-12-19T08:03:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- 172.21.15.111:0/3262070492 wait complete. 2023-12-19T08:03:17.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 Processor -- start 2023-12-19T08:03:17.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.188+0000 7fc65ba78700 1 -- start start 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6540ae510 0x7fc65415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540c1d60 0x7fc654156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc654168b20 con 0x7fc6540ae140 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc654169900 con 0x7fc6540ae510 2023-12-19T08:03:17.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc654168d30 con 0x7fc6540c1d60 2023-12-19T08:03:17.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50834/0 (socket says 172.21.15.111:50834) 2023-12-19T08:03:17.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 -- 172.21.15.111:0/3429399589 learned_addr learned my addr 172.21.15.111:0/3429399589 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:17.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659013700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6540ae510 0x7fc65415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65a015700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540c1d60 0x7fc654156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540c1d60 msgr2=0x7fc654156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540c1d60 0x7fc654156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6540ae510 msgr2=0x7fc65415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6540ae510 0x7fc65415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 -- 172.21.15.111:0/3429399589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644009420 con 0x7fc6540ae140 2023-12-19T08:03:17.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc659814700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc65000c7e0 tx=0x7fc650007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500105c0 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65415c260 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc650010740 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc650010b00 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.192+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6541692b0 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc654159700 con 0x7fc6540ae140 2023-12-19T08:03:17.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fc65001b3b0 con 0x7fc6540ae140 2023-12-19T08:03:17.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc649ffb700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 0x7fc638062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fc65007c640 con 0x7fc6540ae140 2023-12-19T08:03:17.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc659013700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 msgr2=0x7fc638062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:17.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.196+0000 7fc659013700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 0x7fc638062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:17.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.204+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc65004c470 con 0x7fc6540ae140 2023-12-19T08:03:17.324 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:17.325+0000 7f9e63161000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:17.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.400+0000 7fc659013700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 msgr2=0x7fc638062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:17.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.400+0000 7fc659013700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 0x7fc638062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.416+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc654071120 con 0x7fc6540ae140 2023-12-19T08:03:17.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.416+0000 7fc649ffb700 1 -- 172.21.15.111:0/3429399589 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7fc650017090 con 0x7fc6540ae140 2023-12-19T08:03:17.420 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 msgr2=0x7fc638062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 0x7fc638062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 msgr2=0x7fc6541514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc65000c7e0 tx=0x7fc650007df0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 shutdown_connections 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc638060700 0x7fc638062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6540ae510 0x7fc65415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6540c1d60 0x7fc654156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 --2- 172.21.15.111:0/3429399589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6540ae140 0x7fc6541514f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 >> 172.21.15.111:0/3429399589 conn(0x7fc6540c8850 msgr2=0x7fc6540676e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 shutdown_connections 2023-12-19T08:03:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.420+0000 7fc65ba78700 1 -- 172.21.15.111:0/3429399589 wait complete. 2023-12-19T08:03:17.426 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:17.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:17.550 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:17.549+0000 7f9e63161000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:17.573 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.568+0000 7f60ec760000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:17.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.604+0000 7f994568e700 1 Processor -- start 2023-12-19T08:03:17.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.604+0000 7f994568e700 1 -- start start 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f994568e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 0x7f99400670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f994568e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f994015ca30 0x7f994015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f994568e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99400cd540 con 0x7f994006d050 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f994568e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99400c7ed0 con 0x7f9940158df0 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f994568e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994015d3c0 con 0x7f994015ca30 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 0x7f99400670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f9944e8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f994006d050 0x7f994006d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41596/0 (socket says 172.21.15.111:41596) 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f9944e8d700 1 -- 172.21.15.111:0/144424398 learned_addr learned my addr 172.21.15.111:0/144424398 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:17.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 --2- 172.21.15.111:0/144424398 >> v2:172.21.15.143:3300/0 conn(0x7f994015ca30 0x7f994015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 -- 172.21.15.111:0/144424398 >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 msgr2=0x7f99400670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:17.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 --2- 172.21.15.111:0/144424398 >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 0x7f99400670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 -- 172.21.15.111:0/144424398 >> v1:172.21.15.111:6789/0 conn(0x7f994006d050 legacy=0x7f994006d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 -- 172.21.15.111:0/144424398 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994015d610 con 0x7f994015ca30 2023-12-19T08:03:17.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993ffff700 1 --2- 172.21.15.111:0/144424398 >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 0x7f99400670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:17.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993f7fe700 1 --2- 172.21.15.111:0/144424398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015ca30 0x7f994015ce80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f99400b2ae0 tx=0x7f99340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d27d35a86aaa249b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:17.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993effd700 1 -- 172.21.15.111:0/144424398 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934011630 con 0x7f994015ca30 2023-12-19T08:03:17.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993effd700 1 -- 172.21.15.111:0/144424398 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934011790 con 0x7f994015ca30 2023-12-19T08:03:17.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.608+0000 7f993effd700 1 -- 172.21.15.111:0/144424398 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934011b30 con 0x7f994015ca30 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- 172.21.15.111:0/144424398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015ca30 msgr2=0x7f994015ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 --2- 172.21.15.111:0/144424398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015ca30 0x7f994015ce80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f99400b2ae0 tx=0x7f99340091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- 172.21.15.111:0/144424398 shutdown_connections 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 --2- 172.21.15.111:0/144424398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015ca30 0x7f994015ce80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 --2- 172.21.15.111:0/144424398 >> v2:172.21.15.111:3300/0 conn(0x7f9940158df0 0x7f99400670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- 172.21.15.111:0/144424398 >> 172.21.15.111:0/144424398 conn(0x7f994006c080 msgr2=0x7f99400ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- 172.21.15.111:0/144424398 shutdown_connections 2023-12-19T08:03:17.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- 172.21.15.111:0/144424398 wait complete. 2023-12-19T08:03:17.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 Processor -- start 2023-12-19T08:03:17.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.612+0000 7f994568e700 1 -- start start 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9940158df0 0x7f99400b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9940148f00 con 0x7f994015ca30 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400c5f60 con 0x7f994006d050 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400c6240 con 0x7f9940158df0 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9944e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:37500/0 (socket says 172.21.15.111:37500) 2023-12-19T08:03:17.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9944e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50850/0 (socket says 172.21.15.111:50850) 2023-12-19T08:03:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 -- 172.21.15.111:0/2732464787 learned_addr learned my addr 172.21.15.111:0/2732464787 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993f7fe700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9940158df0 0x7f99400b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 -- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9940158df0 msgr2=0x7f99400b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9940158df0 0x7f99400b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 -- 172.21.15.111:0/2732464787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 msgr2=0x7f99400b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 -- 172.21.15.111:0/2732464787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930009420 con 0x7f994006d050 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9944e8d700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f993ffff700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f993000ec80 tx=0x7f99300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000c850 con 0x7f994006d050 2023-12-19T08:03:17.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9934009420 con 0x7f994006d050 2023-12-19T08:03:17.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993000c9b0 con 0x7f994006d050 2023-12-19T08:03:17.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000cd30 con 0x7f994006d050 2023-12-19T08:03:17.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.616+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99400c6800 con 0x7f994006d050 2023-12-19T08:03:17.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994004be40 con 0x7f994006d050 2023-12-19T08:03:17.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f9930013680 con 0x7f994006d050 2023-12-19T08:03:17.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f9927fff700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 0x7f992c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f993007d250 con 0x7f994006d050 2023-12-19T08:03:17.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f993f7fe700 1 -- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 msgr2=0x7f992c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:17.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.620+0000 7f993f7fe700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 0x7f992c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:17.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.632+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9930050bb0 con 0x7f994006d050 2023-12-19T08:03:17.644 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:17.645+0000 7f9e63161000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:17.718 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:17.717+0000 7f9e63161000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:17.798 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:17.797+0000 7f9e63161000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:17.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.800+0000 7f60ec760000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:17.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.808+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99400cda00 con 0x7f994006d050 2023-12-19T08:03:17.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.812+0000 7f9927fff700 1 -- 172.21.15.111:0/2732464787 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f9930050740 con 0x7f994006d050 2023-12-19T08:03:17.815 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:17.815 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":100,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:12.694352+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/1273338574":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3396798404":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3432369124":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1714582390":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1261051935":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3477898242":"2023-12-20T08:03:12.694202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:17.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.812+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 msgr2=0x7f992c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.812+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 0x7f992c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.812+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 msgr2=0x7f99401486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.812+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f993000ec80 tx=0x7f99300091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 shutdown_connections 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f992c060700 0x7f992c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f994015ca30 0x7f99400b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9940158df0 0x7f99400b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 --2- 172.21.15.111:0/2732464787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994006d050 0x7f99401486c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 >> 172.21.15.111:0/2732464787 conn(0x7f994006c080 msgr2=0x7f99401581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 shutdown_connections 2023-12-19T08:03:17.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.816+0000 7f994568e700 1 -- 172.21.15.111:0/2732464787 wait complete. 2023-12-19T08:03:17.838 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T08:03:17.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:17.900 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.896+0000 7f60ec760000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:17.984 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:17.980+0000 7f60ec760000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:17.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 Processor -- start 2023-12-19T08:03:17.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 -- start start 2023-12-19T08:03:17.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0dc415bd00 0x7f0dc415c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406b2c0 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406e0f0 con 0x7f0dc40cad10 2023-12-19T08:03:17.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc8ee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc406a6b0 con 0x7f0dc415bd00 2023-12-19T08:03:17.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dbbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0dc40cb0e0 0x7f0dc406d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41612/0 (socket says 172.21.15.111:41612) 2023-12-19T08:03:17.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dbbfff700 1 -- 172.21.15.111:0/2883430497 learned_addr learned my addr 172.21.15.111:0/2883430497 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:17.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc3fff700 1 --2- 172.21.15.111:0/2883430497 >> v2:172.21.15.143:3300/0 conn(0x7f0dc415bd00 0x7f0dc415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799370039 0 0) 0x7f0dc406b2c0 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0003570 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3646152445 0 0) 0x7f0db0003570 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 >> v2:172.21.15.143:3300/0 conn(0x7f0dc415bd00 msgr2=0x7f0dc415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2883430497 >> v2:172.21.15.143:3300/0 conn(0x7f0dc415bd00 0x7f0dc415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 >> v1:172.21.15.143:6789/0 conn(0x7f0dc40cad10 legacy=0x7f0dc406b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.984+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc406a990 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0dac001470 con 0x7f0dc40cb0e0 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 >> v1:172.21.15.111:6789/0 conn(0x7f0dc40cb0e0 legacy=0x7f0dc406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 0x7f0db0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0004810 0x7f0db0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00051c0 0x7f0db000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc406a6b0 con 0x7f0db0003a60 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db000c130 con 0x7f0db00051c0 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db000c2f0 con 0x7f0db0004810 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 0x7f0db0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dbbfff700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0004810 0x7f0db0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc3fff700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00051c0 0x7f0db000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 -- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0004810 msgr2=0x7f0db0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0004810 0x7f0db0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 -- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00051c0 msgr2=0x7f0db000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00051c0 0x7f0db000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 -- 172.21.15.111:0/2883430497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0003570 con 0x7f0db0003a60 2023-12-19T08:03:17.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc3fff700 1 -- 172.21.15.111:0/2883430497 reap_dead start 2023-12-19T08:03:17.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 0x7f0db0003e30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0db4001880 tx=0x7f0db4007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576bf6cb048097f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:17.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4012420 con 0x7f0db0003a60 2023-12-19T08:03:17.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db40125a0 con 0x7f0db0003a60 2023-12-19T08:03:17.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2883430497 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4012960 con 0x7f0db0003a60 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 msgr2=0x7f0db0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 0x7f0db0003e30 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0db4001880 tx=0x7f0db4007d30 comp rx=0 tx=0).stop 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/2883430497 shutdown_connections 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/2883430497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db0003a60 0x7f0db0003e30 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/2883430497 >> 172.21.15.111:0/2883430497 conn(0x7f0dc4067bb0 msgr2=0x7f0dc40cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/2883430497 shutdown_connections 2023-12-19T08:03:17.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.988+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/2883430497 wait complete. 2023-12-19T08:03:17.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 Processor -- start 2023-12-19T08:03:17.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 -- start start 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0003a60 0x7f0dc41497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0dc40c7450 0x7f0dc40c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40c7e10 con 0x7f0dc4169b20 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc415a320 con 0x7f0db0003a60 2023-12-19T08:03:17.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc8ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4159710 con 0x7f0dc40c7450 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dbbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dbbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50864/0 (socket says 172.21.15.111:50864) 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dbbfff700 1 -- 172.21.15.111:0/813015746 learned_addr learned my addr 172.21.15.111:0/813015746 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0003a60 0x7f0dc41497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.992+0000 7f0dc3fff700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0dc40c7450 0x7f0dc40c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:17.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0dc40c7450 msgr2=0x7f0dc40c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0dc40c7450 0x7f0dc40c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0003a60 msgr2=0x7f0dc41497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:17.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0003a60 0x7f0dc41497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:17.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 -- 172.21.15.111:0/813015746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc41599f0 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbbfff700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0dac008390 tx=0x7f0dac001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac008770 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac00e7d0 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc4159cd0 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00eb70 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4149d10 con 0x7f0dc4169b20 2023-12-19T08:03:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:17.996+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc415a150 con 0x7f0dc4169b20 2023-12-19T08:03:18.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.000+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f0dac01d460 con 0x7f0dc4169b20 2023-12-19T08:03:18.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.000+0000 7f0dbb7fe700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 0x7f0da4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.000+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f0dac07d1c0 con 0x7f0dc4169b20 2023-12-19T08:03:18.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.000+0000 7f0dc37fe700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 msgr2=0x7f0da4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:18.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.000+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 0x7f0da4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:18.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.004+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0dac04cee0 con 0x7f0dc4169b20 2023-12-19T08:03:18.014 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:18.013+0000 7f9e63161000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:18.074 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.072+0000 7f60ec760000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:18.095 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:18.093+0000 7f9e63161000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:18.182 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:18.181+0000 7f9e63161000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.204+0000 7f0dc37fe700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 msgr2=0x7f0da4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.204+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 0x7f0da4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:18.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.208+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dc404be40 con 0x7f0dc4169b20 2023-12-19T08:03:18.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.208+0000 7f0dbb7fe700 1 -- 172.21.15.111:0/813015746 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f0dac050500 con 0x7f0dc4169b20 2023-12-19T08:03:18.211 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:18.211 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 msgr2=0x7f0da4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 0x7f0da4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 msgr2=0x7f0dc4158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0dac008390 tx=0x7f0dac001b20 comp rx=0 tx=0).stop 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 shutdown_connections 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0da4060700 0x7f0da4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0dc40c7450 0x7f0dc40c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc4169b20 0x7f0dc4158e30 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 --2- 172.21.15.111:0/813015746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0003a60 0x7f0dc41497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 >> 172.21.15.111:0/813015746 conn(0x7f0dc4067bb0 msgr2=0x7f0dc40afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 shutdown_connections 2023-12-19T08:03:18.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.212+0000 7f0dc8ee3700 1 -- 172.21.15.111:0/813015746 wait complete. 2023-12-19T08:03:18.217 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:18.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:18.323 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.320+0000 7f60ec760000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:18.371 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:18.373+0000 7f9e63161000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 Processor -- start 2023-12-19T08:03:18.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 -- start start 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdec06a540 0x7fbdec06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec0cc470 0x7fbdec0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0ca5a0 con 0x7fbdec158e70 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec06dac0 con 0x7fbdec06a540 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf146b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec06d220 con 0x7fbdec0cc470 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdeb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdec06a540 0x7fbdec06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdebfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbdec158e70 0x7fbdec0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41628/0 (socket says 172.21.15.111:41628) 2023-12-19T08:03:18.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdeb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbdec06a540 0x7fbdec06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50868/0 (socket says 172.21.15.111:50868) 2023-12-19T08:03:18.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdebfff700 1 -- 172.21.15.111:0/1531442141 learned_addr learned my addr 172.21.15.111:0/1531442141 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:18.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdf0c6a700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec0cc470 0x7fbdec0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/1531442141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec0cc470 msgr2=0x7fbdec0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdeb7fe700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec0cc470 0x7fbdec0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.400+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/1531442141 >> v1:172.21.15.111:6789/0 conn(0x7fbdec158e70 legacy=0x7fbdec0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/1531442141 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec06d500 con 0x7fbdec06a540 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdeb7fe700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec06a540 0x7fbdec06c9d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbde00098e0 tx=0x7fbde00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505501183822f1ef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdeaffd700 1 -- 172.21.15.111:0/1531442141 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0009e80 con 0x7fbdec06a540 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdeaffd700 1 -- 172.21.15.111:0/1531442141 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000b860 con 0x7fbdec06a540 2023-12-19T08:03:18.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdeaffd700 1 -- 172.21.15.111:0/1531442141 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000bbe0 con 0x7fbdec06a540 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 -- 172.21.15.111:0/1531442141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec06a540 msgr2=0x7fbdec06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec06a540 0x7fbdec06c9d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbde00098e0 tx=0x7fbde00091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 -- 172.21.15.111:0/1531442141 shutdown_connections 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec0cc470 0x7fbdec0cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 --2- 172.21.15.111:0/1531442141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec06a540 0x7fbdec06c9d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 -- 172.21.15.111:0/1531442141 >> 172.21.15.111:0/1531442141 conn(0x7fbdec0670e0 msgr2=0x7fbdec0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:18.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 -- 172.21.15.111:0/1531442141 shutdown_connections 2023-12-19T08:03:18.409 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.404+0000 7f60ec760000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:18.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.404+0000 7fbdf146b700 1 -- 172.21.15.111:0/1531442141 wait complete. 2023-12-19T08:03:18.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 Processor -- start 2023-12-19T08:03:18.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 -- start start 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec06a540 0x7fbdec0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 0x7fbdec1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0c46b0 con 0x7fbdec0cc470 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec156b10 con 0x7fbdec158e70 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf146b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0c4840 con 0x7fbdec06a540 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50874/0 (socket says 172.21.15.111:50874) 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/228794077 learned_addr learned my addr 172.21.15.111:0/228794077 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdebfff700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec06a540 0x7fbdec0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf0c6a700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 0x7fbdec1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec06a540 msgr2=0x7fbdec0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec06a540 0x7fbdec0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 msgr2=0x7fbdec1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 0x7fbdec1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 -- 172.21.15.111:0/228794077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0009420 con 0x7fbdec0cc470 2023-12-19T08:03:18.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdf0c6a700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 0x7fbdec1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.408+0000 7fbdeb7fe700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009220 tx=0x7fbde0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000bbe0 con 0x7fbdec0cc470 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0006af0 con 0x7fbdec0cc470 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec15a270 con 0x7fbdec0cc470 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0011670 con 0x7fbdec0cc470 2023-12-19T08:03:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec15a830 con 0x7fbdec0cc470 2023-12-19T08:03:18.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec04be40 con 0x7fbdec0cc470 2023-12-19T08:03:18.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fbde0011aa0 con 0x7fbdec0cc470 2023-12-19T08:03:18.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 0x7fbdd8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.412+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fbde007e870 con 0x7fbdec0cc470 2023-12-19T08:03:18.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.416+0000 7fbdebfff700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 msgr2=0x7fbdd8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:18.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.416+0000 7fbdebfff700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 0x7fbdd8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:18.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.420+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbde004e6a0 con 0x7fbdec0cc470 2023-12-19T08:03:18.457 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:18.457+0000 7f9e63161000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:18.505 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.500+0000 7f60ec760000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:18.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.620+0000 7fbdebfff700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 msgr2=0x7fbdd8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:18.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.620+0000 7fbdebfff700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 0x7fbdd8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:18.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.624+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdec0b48a0 con 0x7fbdec0cc470 2023-12-19T08:03:18.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.624+0000 7fbdd3fff700 1 -- 172.21.15.111:0/228794077 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4611 (secure 0 0 0) 0x7fbde0051cc0 con 0x7fbdec0cc470 2023-12-19T08:03:18.627 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:18.627 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:02:58.363473+0000","modified":"2023-12-19T08:03:01.958376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6894":{"gid":6894,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/2842493331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2842493331},{"type":"v1","addr":"172.21.15.111:6835","nonce":2842493331}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2023-12-19T08:03:18.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 msgr2=0x7fbdd8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 0x7fbdd8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 msgr2=0x7fbdec0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009220 tx=0x7fbde0017000 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 shutdown_connections 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbdd8060700 0x7fbdd8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdec158e70 0x7fbdec1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdec0cc470 0x7fbdec0b5000 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 --2- 172.21.15.111:0/228794077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdec06a540 0x7fbdec0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 >> 172.21.15.111:0/228794077 conn(0x7fbdec0670e0 msgr2=0x7fbdec158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 shutdown_connections 2023-12-19T08:03:18.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.628+0000 7fbdf146b700 1 -- 172.21.15.111:0/228794077 wait complete. 2023-12-19T08:03:18.634 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2023-12-19T08:03:18.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:18.719 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.716+0000 7f60ec760000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:18.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.812+0000 7fc0644f0700 1 Processor -- start 2023-12-19T08:03:18.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.812+0000 7fc0644f0700 1 -- start start 2023-12-19T08:03:18.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc0644f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc05c0cc610 0x7fc05c0c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc0644f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c0c6630 con 0x7fc05c067c90 2023-12-19T08:03:18.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc0644f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05c0c6910 con 0x7fc05c1502b0 2023-12-19T08:03:18.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc0644f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05c0c6bf0 con 0x7fc05c0cc610 2023-12-19T08:03:18.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc062a8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc05c0cc610 0x7fc05c0c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06228c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc05c067c90 0x7fc05c14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41630/0 (socket says 172.21.15.111:41630) 2023-12-19T08:03:18.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06228c700 1 -- 172.21.15.111:0/3168077416 learned_addr learned my addr 172.21.15.111:0/3168077416 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:18.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06128a700 1 -- 172.21.15.111:0/3168077416 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062795731 0 0) 0x7fc05c0c6630 con 0x7fc05c067c90 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc062a8d700 1 -- 172.21.15.111:0/3168077416 >> v1:172.21.15.143:6789/0 conn(0x7fc05c1502b0 legacy=0x7fc05c0c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc062a8d700 1 -- 172.21.15.111:0/3168077416 >> v1:172.21.15.111:6789/0 conn(0x7fc05c067c90 legacy=0x7fc05c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc062a8d700 1 -- 172.21.15.111:0/3168077416 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c0c6ed0 con 0x7fc05c0cc610 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc062a8d700 1 --2- 172.21.15.111:0/3168077416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0c6010 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc054009e60 tx=0x7fc054009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3eae04112a8cb1e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06128a700 1 -- 172.21.15.111:0/3168077416 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05400e070 con 0x7fc05c0cc610 2023-12-19T08:03:18.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06128a700 1 -- 172.21.15.111:0/3168077416 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05400bc30 con 0x7fc05c0cc610 2023-12-19T08:03:18.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.816+0000 7fc06128a700 1 -- 172.21.15.111:0/3168077416 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054012430 con 0x7fc05c0cc610 2023-12-19T08:03:18.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- 172.21.15.111:0/3168077416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 msgr2=0x7fc05c0c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 --2- 172.21.15.111:0/3168077416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0c6010 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc054009e60 tx=0x7fc054009780 comp rx=0 tx=0).stop 2023-12-19T08:03:18.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- 172.21.15.111:0/3168077416 shutdown_connections 2023-12-19T08:03:18.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 --2- 172.21.15.111:0/3168077416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0c6010 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- 172.21.15.111:0/3168077416 >> 172.21.15.111:0/3168077416 conn(0x7fc05c0cd6d0 msgr2=0x7fc05c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:18.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- 172.21.15.111:0/3168077416 shutdown_connections 2023-12-19T08:03:18.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- 172.21.15.111:0/3168077416 wait complete. 2023-12-19T08:03:18.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 Processor -- start 2023-12-19T08:03:18.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 -- start start 2023-12-19T08:03:18.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.820+0000 7fc0644f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 0x7fc05c0b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 0x7fc05c0c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0b45a0 con 0x7fc05c0cc610 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0b3990 con 0x7fc05c1502b0 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0b3c70 con 0x7fc05c067c90 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50892/0 (socket says 172.21.15.111:50892) 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 -- 172.21.15.111:0/2804458595 learned_addr learned my addr 172.21.15.111:0/2804458595 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc062a8d700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 0x7fc05c0c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc06228c700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 0x7fc05c0b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 -- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 msgr2=0x7fc05c0b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 0x7fc05c0b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 -- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 msgr2=0x7fc05c0c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:18.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 0x7fc05c0c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:18.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 -- 172.21.15.111:0/2804458595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054009a20 con 0x7fc05c0cc610 2023-12-19T08:03:18.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc062a8d700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 0x7fc05c0c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:18.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc06228c700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 0x7fc05c0b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:18.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc061a8b700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00c4e0 tx=0x7fc04c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:18.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c011590 con 0x7fc05c0cc610 2023-12-19T08:03:18.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c011710 con 0x7fc05c0cc610 2023-12-19T08:03:18.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c011ad0 con 0x7fc05c0cc610 2023-12-19T08:03:18.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05c0b3f50 con 0x7fc05c0cc610 2023-12-19T08:03:18.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05c152bc0 con 0x7fc05c0cc610 2023-12-19T08:03:18.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.824+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05c0b4710 con 0x7fc05c0cc610 2023-12-19T08:03:18.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.828+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fc04c01b3b0 con 0x7fc05c0cc610 2023-12-19T08:03:18.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.828+0000 7fc0527fc700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 0x7fc040062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:18.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.828+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fc04c07c530 con 0x7fc05c0cc610 2023-12-19T08:03:18.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.828+0000 7fc06228c700 1 -- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 msgr2=0x7fc040062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:18.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.828+0000 7fc06228c700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 0x7fc040062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:18.834 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:18.832+0000 7f60ec760000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:18.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:18.832+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc04c04c360 con 0x7fc05c0cc610 2023-12-19T08:03:19.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.016+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc05c071120 con 0x7fc05c0cc610 2023-12-19T08:03:19.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.020+0000 7fc0527fc700 1 -- 172.21.15.111:0/2804458595 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16895 (secure 0 0 0) 0x7fc04c04f980 con 0x7fc05c0cc610 2023-12-19T08:03:19.024 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:19.024 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":100,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:12.694352+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:02:56.370138+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-12-19T08:02:57.369378+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/1273338574":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3396798404":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3432369124":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1714582390":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1261051935":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3477898242":"2023-12-20T08:03:12.694202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:19.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 msgr2=0x7fc040062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:19.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 0x7fc040062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 msgr2=0x7fc05c0b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00c4e0 tx=0x7fc04c00c8a0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 shutdown_connections 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc040060700 0x7fc040062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc05c1502b0 0x7fc05c0c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc05c0cc610 0x7fc05c0b2fb0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 --2- 172.21.15.111:0/2804458595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc05c067c90 0x7fc05c0b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 >> 172.21.15.111:0/2804458595 conn(0x7fc05c0cd6d0 msgr2=0x7fc05c06a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 shutdown_connections 2023-12-19T08:03:19.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.024+0000 7fc0644f0700 1 -- 172.21.15.111:0/2804458595 wait complete. 2023-12-19T08:03:19.047 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T08:03:19.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.208+0000 7f6ebb6f0700 1 Processor -- start 2023-12-19T08:03:19.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 -- start start 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6eb415a8d0 0x7f6eb415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 0x7f6eb40cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4157290 0x7f6eb41576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb415b1e0 con 0x7f6eb415a8d0 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb40cb400 con 0x7f6eb41515a0 2023-12-19T08:03:19.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6ebb6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40cb6e0 con 0x7f6eb4157290 2023-12-19T08:03:19.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb948c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6eb415a8d0 0x7f6eb415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb948c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6eb415a8d0 0x7f6eb415aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50906/0 (socket says 172.21.15.111:50906) 2023-12-19T08:03:19.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb9c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4157290 0x7f6eb41576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb8c8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 0x7f6eb40cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb948c700 1 -- 172.21.15.111:0/1294778039 learned_addr learned my addr 172.21.15.111:0/1294778039 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb948c700 1 -- 172.21.15.111:0/1294778039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4157290 msgr2=0x7f6eb41576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.212+0000 7f6eb948c700 1 --2- 172.21.15.111:0/1294778039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4157290 0x7f6eb41576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eb948c700 1 -- 172.21.15.111:0/1294778039 >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 msgr2=0x7f6eb40cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eb948c700 1 --2- 172.21.15.111:0/1294778039 >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 0x7f6eb40cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eb948c700 1 -- 172.21.15.111:0/1294778039 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40cb9c0 con 0x7f6eb415a8d0 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eb8c8b700 1 --2- 172.21.15.111:0/1294778039 >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 0x7f6eb40cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:19.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eb948c700 1 --2- 172.21.15.111:0/1294778039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb415a8d0 0x7f6eb415aca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6eb406f7a0 tx=0x7f6eb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d358a7c5f57e3a83 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:19.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eabfff700 1 -- 172.21.15.111:0/1294778039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000b640 con 0x7f6eb415a8d0 2023-12-19T08:03:19.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eabfff700 1 -- 172.21.15.111:0/1294778039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb000b7a0 con 0x7f6eb415a8d0 2023-12-19T08:03:19.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6eabfff700 1 -- 172.21.15.111:0/1294778039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000bb60 con 0x7f6eb415a8d0 2023-12-19T08:03:19.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/1294778039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb415a8d0 msgr2=0x7f6eb415aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/1294778039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb415a8d0 0x7f6eb415aca0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6eb406f7a0 tx=0x7f6eb0009250 comp rx=0 tx=0).stop 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/1294778039 shutdown_connections 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/1294778039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4157290 0x7f6eb41576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/1294778039 >> v2:172.21.15.143:3300/0 conn(0x7f6eb41515a0 0x7f6eb40cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/1294778039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb415a8d0 0x7f6eb415aca0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/1294778039 >> 172.21.15.111:0/1294778039 conn(0x7f6eb40686f0 msgr2=0x7f6eb4068af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/1294778039 shutdown_connections 2023-12-19T08:03:19.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.216+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/1294778039 wait complete. 2023-12-19T08:03:19.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 Processor -- start 2023-12-19T08:03:19.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 -- start start 2023-12-19T08:03:19.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6eb4153790 0x7f6eb41674e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4167a20 0x7f6eb40af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40b0b40 con 0x7f6eb4157290 2023-12-19T08:03:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4167ec0 con 0x7f6eb4153790 2023-12-19T08:03:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6ebb6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40afec0 con 0x7f6eb4167a20 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6eb948c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6eb948c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50922/0 (socket says 172.21.15.111:50922) 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6eb948c700 1 -- 172.21.15.111:0/2137967004 learned_addr learned my addr 172.21.15.111:0/2137967004 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6eb8c8b700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6eb4153790 0x7f6eb41674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.220+0000 7f6eb9c8d700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4167a20 0x7f6eb40af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 -- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4167a20 msgr2=0x7f6eb40af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4167a20 0x7f6eb40af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 -- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6eb4153790 msgr2=0x7f6eb41674e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6eb4153790 0x7f6eb41674e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 -- 172.21.15.111:0/2137967004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea400d8d0 con 0x7f6eb4157290 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6eb948c700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000b910 con 0x7f6eb4157290 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb000ba70 con 0x7f6eb4157290 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0011670 con 0x7f6eb4157290 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb00094f0 con 0x7f6eb4157290 2023-12-19T08:03:19.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb40b0480 con 0x7f6eb4157290 2023-12-19T08:03:19.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.224+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb4071120 con 0x7f6eb4157290 2023-12-19T08:03:19.230 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.229+0000 7f9e63161000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:19.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.228+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f6eb00119e0 con 0x7f6eb4157290 2023-12-19T08:03:19.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.228+0000 7f6ea97fa700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 0x7f6e98062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.228+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f6eb007e730 con 0x7f6eb4157290 2023-12-19T08:03:19.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.228+0000 7f6eb8c8b700 1 -- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 msgr2=0x7f6e98062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:19.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.228+0000 7f6eb8c8b700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 0x7f6e98062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.236+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6eb004e560 con 0x7f6eb4157290 2023-12-19T08:03:19.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.248+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 39) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f6eb004e930 con 0x7f6eb4157290 2023-12-19T08:03:19.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.248+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 msgr2=0x7f6e98062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:19.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.248+0000 7f6ea97fa700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 0x7f6e98062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2a51d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2a51d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2a51d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2a51d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2a51d700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2d523700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.347 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2d523700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.347 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2d523700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.347 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2d523700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.347 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:19.345+0000 7f9e2d523700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:19.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.464+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6eb40cbca0 con 0x7f6eb4157290 2023-12-19T08:03:19.467 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T08:03:19.464+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T08:03:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.472+0000 7f6ea97fa700 1 -- 172.21.15.111:0/2137967004 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6eb0027cf0 con 0x7f6eb4157290 2023-12-19T08:03:19.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 msgr2=0x7f6eb4153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0006fd0 tx=0x7f6eb0017000 comp rx=0 tx=0).stop 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 shutdown_connections 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e98060700 0x7f6e98062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6eb4153790 0x7f6eb41674e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6eb4167a20 0x7f6eb40af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 --2- 172.21.15.111:0/2137967004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6eb4157290 0x7f6eb4153250 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 >> 172.21.15.111:0/2137967004 conn(0x7f6eb40686f0 msgr2=0x7f6eb4150940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 shutdown_connections 2023-12-19T08:03:19.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.476+0000 7f6ebb6f0700 1 -- 172.21.15.111:0/2137967004 wait complete. 2023-12-19T08:03:19.482 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T08:03:19.483 INFO:tasks.ceph.mds.a.smithi111.stderr: -1326> 2023-12-19T08:03:03.885+0000 7f4faa7f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:03:19.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T08:03:19.533 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:03:19.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.648+0000 7fc050e8e700 1 Processor -- start 2023-12-19T08:03:19.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.648+0000 7fc050e8e700 1 -- start start 2023-12-19T08:03:19.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc050e8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 0x7fc04c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc050e8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc04c0adab0 0x7fc04c0adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc050e8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c15ae30 con 0x7fc04c154780 2023-12-19T08:03:19.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc050e8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c0abf70 con 0x7fc04c0adab0 2023-12-19T08:03:19.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc050e8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c154b50 con 0x7fc04c06baa0 2023-12-19T08:03:19.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc04c0adab0 0x7fc04c0adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc04c0adab0 0x7fc04c0adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50960/0 (socket says 172.21.15.111:50960) 2023-12-19T08:03:19.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 -- 172.21.15.111:0/3677813756 learned_addr learned my addr 172.21.15.111:0/3677813756 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04affd700 1 --2- 172.21.15.111:0/3677813756 >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 0x7fc04c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 -- 172.21.15.111:0/3677813756 >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 msgr2=0x7fc04c15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 --2- 172.21.15.111:0/3677813756 >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 0x7fc04c15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 -- 172.21.15.111:0/3677813756 >> v1:172.21.15.143:6789/0 conn(0x7fc04c154780 legacy=0x7fc04c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 -- 172.21.15.111:0/3677813756 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0ac210 con 0x7fc04c0adab0 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04affd700 1 --2- 172.21.15.111:0/3677813756 >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 0x7fc04c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.652+0000 7fc04bfff700 1 --2- 172.21.15.111:0/3677813756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0adeb0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0cb990 tx=0x7fc044009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7783ac0ced52c4c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc04a7fc700 1 -- 172.21.15.111:0/3677813756 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400e070 con 0x7fc04c0adab0 2023-12-19T08:03:19.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc04a7fc700 1 -- 172.21.15.111:0/3677813756 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04400ba50 con 0x7fc04c0adab0 2023-12-19T08:03:19.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc04a7fc700 1 -- 172.21.15.111:0/3677813756 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0440092e0 con 0x7fc04c0adab0 2023-12-19T08:03:19.659 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:19.656+0000 7f60ec760000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 -- 172.21.15.111:0/3677813756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 msgr2=0x7fc04c0adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 --2- 172.21.15.111:0/3677813756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0adeb0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0cb990 tx=0x7fc044009840 comp rx=0 tx=0).stop 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 -- 172.21.15.111:0/3677813756 shutdown_connections 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 --2- 172.21.15.111:0/3677813756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0adeb0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 --2- 172.21.15.111:0/3677813756 >> v2:172.21.15.143:3300/0 conn(0x7fc04c06baa0 0x7fc04c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 -- 172.21.15.111:0/3677813756 >> 172.21.15.111:0/3677813756 conn(0x7fc04c067b50 msgr2=0x7fc04c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 -- 172.21.15.111:0/3677813756 shutdown_connections 2023-12-19T08:03:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.656+0000 7fc050e8e700 1 -- 172.21.15.111:0/3677813756 wait complete. 2023-12-19T08:03:19.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 Processor -- start 2023-12-19T08:03:19.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 -- start start 2023-12-19T08:03:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc04c06baa0 0x7fc04c150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04c154780 0x7fc04c0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1690b0 con 0x7fc04c0adab0 2023-12-19T08:03:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0aff40 con 0x7fc04c154780 2023-12-19T08:03:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc050e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0af330 con 0x7fc04c06baa0 2023-12-19T08:03:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50966/0 (socket says 172.21.15.111:50966) 2023-12-19T08:03:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 -- 172.21.15.111:0/18127884 learned_addr learned my addr 172.21.15.111:0/18127884 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04b7fe700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc04c06baa0 0x7fc04c150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04bfff700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04c154780 0x7fc04c0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 -- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc04c06baa0 msgr2=0x7fc04c150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc04c06baa0 0x7fc04c150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 -- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04c154780 msgr2=0x7fc04c0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04c154780 0x7fc04c0c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 -- 172.21.15.111:0/18127884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc044009ae0 con 0x7fc04c0adab0 2023-12-19T08:03:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.660+0000 7fc04affd700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c800 tx=0x7fc040008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040011600 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040011760 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c0af610 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040011ae0 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c0afbd0 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c0b00f0 con 0x7fc04c0adab0 2023-12-19T08:03:19.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fc040010350 con 0x7fc04c0adab0 2023-12-19T08:03:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.664+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc04007c900 con 0x7fc04c0adab0 2023-12-19T08:03:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.668+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc04c0b00f0 con 0x7fc04c0adab0 2023-12-19T08:03:19.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:19.880+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc04c0b00b0 con 0x7fc04c0adab0 2023-12-19T08:03:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.256+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 40) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fc04004c650 con 0x7fc04c0adab0 2023-12-19T08:03:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.260+0000 7fc0337fe700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 0x7fc038068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.264+0000 7fc04b7fe700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 0x7fc038068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.264+0000 7fc04b7fe700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 0x7fc038068cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc03c003860 tx=0x7fc03c009040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.476+0000 7fc0337fe700 1 -- 172.21.15.111:0/18127884 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc040069f60 con 0x7fc04c0adab0 2023-12-19T08:03:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.480+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 msgr2=0x7fc038068cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.480+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 0x7fc038068cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc03c003860 tx=0x7fc03c009040 comp rx=0 tx=0).stop 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.480+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 msgr2=0x7fc04c0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.480+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c800 tx=0x7fc040008040 comp rx=0 tx=0).stop 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 shutdown_connections 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0380668d0 0x7fc038068cb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04c154780 0x7fc04c0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc04c0adab0 0x7fc04c0ae950 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 --2- 172.21.15.111:0/18127884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc04c06baa0 0x7fc04c150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 >> 172.21.15.111:0/18127884 conn(0x7fc04c067b50 msgr2=0x7fc04c1560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 shutdown_connections 2023-12-19T08:03:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.484+0000 7fc050e8e700 1 -- 172.21.15.111:0/18127884 wait complete. 2023-12-19T08:03:20.506 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T08:03:20.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.720+0000 7fd04762e700 1 Processor -- start 2023-12-19T08:03:20.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.720+0000 7fd04762e700 1 -- start start 2023-12-19T08:03:20.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd04762e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 0x7fd0400abf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd04762e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0400ac8e0 con 0x7fd040156e70 2023-12-19T08:03:20.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd04762e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04006d7b0 con 0x7fd0400ac480 2023-12-19T08:03:20.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd04762e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0400ae570 con 0x7fd04006dc40 2023-12-19T08:03:20.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd0453ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd040156e70 0x7fd04006b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41650/0 (socket says 172.21.15.111:41650) 2023-12-19T08:03:20.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd0453ca700 1 -- 172.21.15.111:0/2077057673 learned_addr learned my addr 172.21.15.111:0/2077057673 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:20.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd044bc9700 1 --2- 172.21.15.111:0/2077057673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 0x7fd0400abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd044bc9700 1 -- 172.21.15.111:0/2077057673 >> v1:172.21.15.143:6789/0 conn(0x7fd0400ac480 legacy=0x7fd0400aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd044bc9700 1 -- 172.21.15.111:0/2077057673 >> v1:172.21.15.111:6789/0 conn(0x7fd040156e70 legacy=0x7fd04006b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd037fff700 1 -- 172.21.15.111:0/2077057673 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720369998 0 0) 0x7fd04006d7b0 con 0x7fd0400ac480 2023-12-19T08:03:20.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd044bc9700 1 -- 172.21.15.111:0/2077057673 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04006cba0 con 0x7fd04006dc40 2023-12-19T08:03:20.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd044bc9700 1 --2- 172.21.15.111:0/2077057673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 0x7fd0400abf40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd030009910 tx=0x7fd0300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cdc1e57bb288f2a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:20.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd037fff700 1 -- 172.21.15.111:0/2077057673 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03000b760 con 0x7fd04006dc40 2023-12-19T08:03:20.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd037fff700 1 -- 172.21.15.111:0/2077057673 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03000b8c0 con 0x7fd04006dc40 2023-12-19T08:03:20.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd037fff700 1 -- 172.21.15.111:0/2077057673 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03000bc60 con 0x7fd04006dc40 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.724+0000 7fd04762e700 1 -- 172.21.15.111:0/2077057673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 msgr2=0x7fd0400abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 --2- 172.21.15.111:0/2077057673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 0x7fd0400abf40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd030009910 tx=0x7fd0300091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 -- 172.21.15.111:0/2077057673 shutdown_connections 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 --2- 172.21.15.111:0/2077057673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd04006dc40 0x7fd0400abf40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 -- 172.21.15.111:0/2077057673 >> 172.21.15.111:0/2077057673 conn(0x7fd040067bb0 msgr2=0x7fd0400cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:20.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 -- 172.21.15.111:0/2077057673 shutdown_connections 2023-12-19T08:03:20.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.728+0000 7fd04762e700 1 -- 172.21.15.111:0/2077057673 wait complete. 2023-12-19T08:03:20.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 Processor -- start 2023-12-19T08:03:20.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 -- start start 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 0x7fd040155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd040156e70 0x7fd0400aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd040158630 con 0x7fd04006dc40 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0401560b0 con 0x7fd040156e70 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd04762e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd040151e60 con 0x7fd0400ac480 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50980/0 (socket says 172.21.15.111:50980) 2023-12-19T08:03:20.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 -- 172.21.15.111:0/1436189277 learned_addr learned my addr 172.21.15.111:0/1436189277 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:20.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd045bcb700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd040156e70 0x7fd0400aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd044bc9700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 0x7fd040155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 -- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 msgr2=0x7fd040155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 0x7fd040155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 -- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd040156e70 msgr2=0x7fd0400aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd040156e70 0x7fd0400aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 -- 172.21.15.111:0/1436189277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd030009420 con 0x7fd04006dc40 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd044bc9700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 0x7fd040155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.732+0000 7fd0453ca700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c890 tx=0x7fd03c007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c005b30 con 0x7fd04006dc40 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c005c90 con 0x7fd04006dc40 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd040157ab0 con 0x7fd04006dc40 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c007340 con 0x7fd04006dc40 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd040158070 con 0x7fd04006dc40 2023-12-19T08:03:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd040153df0 con 0x7fd04006dc40 2023-12-19T08:03:20.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.736+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 40) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fd03c01b400 con 0x7fd04006dc40 2023-12-19T08:03:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.740+0000 7fd0357fa700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 0x7fd024062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.740+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd03c07c650 con 0x7fd04006dc40 2023-12-19T08:03:20.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.740+0000 7fd044bc9700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 0x7fd024062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.740+0000 7fd044bc9700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 0x7fd024062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd0300030c0 tx=0x7fd030016000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:20.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.744+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd040153df0 con 0x7fd04006dc40 2023-12-19T08:03:20.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:20.920+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd0400ac8e0 con 0x7fd04006dc40 2023-12-19T08:03:21.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.796+0000 7fd0357fa700 1 -- 172.21.15.111:0/1436189277 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd0400ac8e0 con 0x7fd04006dc40 2023-12-19T08:03:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 msgr2=0x7fd024062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 0x7fd024062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd0300030c0 tx=0x7fd030016000 comp rx=0 tx=0).stop 2023-12-19T08:03:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 msgr2=0x7fd040159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:21.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00c890 tx=0x7fd03c007f30 comp rx=0 tx=0).stop 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 shutdown_connections 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd0240603a0 0x7fd024062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd040156e70 0x7fd0400aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0400ac480 0x7fd040155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 --2- 172.21.15.111:0/1436189277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd04006dc40 0x7fd040159b30 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 >> 172.21.15.111:0/1436189277 conn(0x7fd040067bb0 msgr2=0x7fd0400b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 shutdown_connections 2023-12-19T08:03:21.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:21.804+0000 7fd04762e700 1 -- 172.21.15.111:0/1436189277 wait complete. 2023-12-19T08:03:21.809 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T08:03:21.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T08:03:22.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.000+0000 7f4eb10e3700 1 Processor -- start 2023-12-19T08:03:22.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.000+0000 7f4eb10e3700 1 -- start start 2023-12-19T08:03:22.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb10e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4eac06ed80 0x7f4eac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb10e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06df10 0x7f4eac06ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb10e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac15d720 con 0x7f4eac158da0 2023-12-19T08:03:22.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb10e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac0ca290 con 0x7f4eac06ed80 2023-12-19T08:03:22.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb10e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0ca4a0 con 0x7f4eac06df10 2023-12-19T08:03:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eabfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4eac158da0 0x7f4eac06e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54582/0 (socket says 172.21.15.111:54582) 2023-12-19T08:03:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eabfff700 1 -- 172.21.15.111:0/1061565314 learned_addr learned my addr 172.21.15.111:0/1061565314 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4ea3fff700 1 --2- 172.21.15.111:0/1061565314 >> v2:172.21.15.143:3300/0 conn(0x7f4eac06ed80 0x7f4eac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.004+0000 7f4eb08e2700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06df10 0x7f4eac06ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4ea3fff700 1 -- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06df10 msgr2=0x7f4eac06ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4ea3fff700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06df10 0x7f4eac06ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4ea3fff700 1 -- 172.21.15.111:0/1061565314 >> v1:172.21.15.111:6789/0 conn(0x7f4eac158da0 legacy=0x7f4eac06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4ea3fff700 1 -- 172.21.15.111:0/1061565314 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac0ca780 con 0x7f4eac06ed80 2023-12-19T08:03:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4ea3fff700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06ed80 0x7f4eac06a7d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4e940098e0 tx=0x7f4e940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d13b05e5af2ac7a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eab7fe700 1 -- 172.21.15.111:0/1061565314 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400b640 con 0x7f4eac06ed80 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eab7fe700 1 -- 172.21.15.111:0/1061565314 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400b7a0 con 0x7f4eac06ed80 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eab7fe700 1 -- 172.21.15.111:0/1061565314 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400bb40 con 0x7f4eac06ed80 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 -- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06ed80 msgr2=0x7f4eac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06ed80 0x7f4eac06a7d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4e940098e0 tx=0x7f4e940091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 -- 172.21.15.111:0/1061565314 shutdown_connections 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06df10 0x7f4eac06ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/1061565314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06ed80 0x7f4eac06a7d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 -- 172.21.15.111:0/1061565314 >> 172.21.15.111:0/1061565314 conn(0x7f4eac0cc760 msgr2=0x7f4eac0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 -- 172.21.15.111:0/1061565314 shutdown_connections 2023-12-19T08:03:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.008+0000 7f4eb10e3700 1 -- 172.21.15.111:0/1061565314 wait complete. 2023-12-19T08:03:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.012+0000 7f4eb10e3700 1 Processor -- start 2023-12-19T08:03:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.012+0000 7f4eb10e3700 1 -- start start 2023-12-19T08:03:22.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 0x7f4eac1500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06ed80 0x7f4eac0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac14ea40 con 0x7f4eac158da0 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b3ea0 con 0x7f4eac06df10 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b4d00 con 0x7f4eac06ed80 2023-12-19T08:03:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eabfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 0x7f4eac1500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46528/0 (socket says 172.21.15.111:46528) 2023-12-19T08:03:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 -- 172.21.15.111:0/3450635769 learned_addr learned my addr 172.21.15.111:0/3450635769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4ea3fff700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06ed80 0x7f4eac0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 -- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06ed80 msgr2=0x7f4eac0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06ed80 0x7f4eac0b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 -- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 msgr2=0x7f4eac1500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 0x7f4eac1500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 -- 172.21.15.111:0/3450635769 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94009420 con 0x7f4eac158da0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eabfff700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 0x7f4eac1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb08e2700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009ec0 tx=0x7f4ea400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400cb70 con 0x7f4eac158da0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400f040 con 0x7f4eac158da0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4014480 con 0x7f4eac158da0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.016+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac0b5190 con 0x7f4eac158da0 2023-12-19T08:03:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.020+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac0b43d0 con 0x7f4eac158da0 2023-12-19T08:03:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.020+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac04be40 con 0x7f4eac158da0 2023-12-19T08:03:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.020+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4ea40147f0 con 0x7f4eac158da0 2023-12-19T08:03:22.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.024+0000 7f4ea8ff9700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 0x7f4e8c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:22.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.024+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6838+0+0 (secure 0 0 0) 0x7f4ea407db30 con 0x7f4eac158da0 2023-12-19T08:03:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.024+0000 7f4eabfff700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 0x7f4e8c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.024+0000 7f4eabfff700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 0x7f4e8c062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c007110 tx=0x7f4e9c007f10 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:22.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.032+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4ea404d9f0 con 0x7f4eac158da0 2023-12-19T08:03:22.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.204+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4eac0b4e40 con 0x7f4eac158da0 2023-12-19T08:03:22.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.816+0000 7f4ea8ff9700 1 -- 172.21.15.111:0/3450635769 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4ea40180d0 con 0x7f4eac158da0 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 msgr2=0x7f4e8c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 0x7f4e8c062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c007110 tx=0x7f4e9c007f10 comp rx=0 tx=0).stop 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 msgr2=0x7f4eac1553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009ec0 tx=0x7f4ea400b040 comp rx=0 tx=0).stop 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 shutdown_connections 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4e8c0606a0 0x7f4e8c062b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4eac158da0 0x7f4eac1553c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eac06ed80 0x7f4eac0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 --2- 172.21.15.111:0/3450635769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eac06df10 0x7f4eac1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:22.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 >> 172.21.15.111:0/3450635769 conn(0x7f4eac0cc760 msgr2=0x7f4eac0b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:22.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 shutdown_connections 2023-12-19T08:03:22.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:22.824+0000 7f4eb10e3700 1 -- 172.21.15.111:0/3450635769 wait complete. 2023-12-19T08:03:22.830 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T08:03:22.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T08:03:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.008+0000 7fad5a84c700 1 Processor -- start 2023-12-19T08:03:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.008+0000 7fad5a84c700 1 -- start start 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad540ce2d0 0x7fad540ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 0x7fad5415d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540cc040 0x7fad54158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5415d960 con 0x7fad540ce2d0 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540cbbb0 con 0x7fad540caa10 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5406dc90 con 0x7fad540cc040 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad540ce2d0 0x7fad540ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad540ce2d0 0x7fad540ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46532/0 (socket says 172.21.15.111:46532) 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 -- 172.21.15.111:0/3293841400 learned_addr learned my addr 172.21.15.111:0/3293841400 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad59049700 1 --2- 172.21.15.111:0/3293841400 >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 0x7fad5415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5a04b700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540cc040 0x7fad54158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 -- 172.21.15.111:0/3293841400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540cc040 msgr2=0x7fad54158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540cc040 0x7fad54158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 -- 172.21.15.111:0/3293841400 >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 msgr2=0x7fad5415d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 --2- 172.21.15.111:0/3293841400 >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 0x7fad5415d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 -- 172.21.15.111:0/3293841400 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540cafa0 con 0x7fad540ce2d0 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad59049700 1 --2- 172.21.15.111:0/3293841400 >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 0x7fad5415d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.012+0000 7fad5984a700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540ce2d0 0x7fad540ce6a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fad540696f0 tx=0x7fad50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b214442bd8c0fca server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad58848700 1 -- 172.21.15.111:0/3293841400 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5000b640 con 0x7fad540ce2d0 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad58848700 1 -- 172.21.15.111:0/3293841400 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5000b7a0 con 0x7fad540ce2d0 2023-12-19T08:03:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad58848700 1 -- 172.21.15.111:0/3293841400 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5000bb60 con 0x7fad540ce2d0 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- 172.21.15.111:0/3293841400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540ce2d0 msgr2=0x7fad540ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540ce2d0 0x7fad540ce6a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fad540696f0 tx=0x7fad50009250 comp rx=0 tx=0).stop 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- 172.21.15.111:0/3293841400 shutdown_connections 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540cc040 0x7fad54158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 --2- 172.21.15.111:0/3293841400 >> v2:172.21.15.143:3300/0 conn(0x7fad540caa10 0x7fad5415d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 --2- 172.21.15.111:0/3293841400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540ce2d0 0x7fad540ce6a0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- 172.21.15.111:0/3293841400 >> 172.21.15.111:0/3293841400 conn(0x7fad54067bb0 msgr2=0x7fad540cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- 172.21.15.111:0/3293841400 shutdown_connections 2023-12-19T08:03:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- 172.21.15.111:0/3293841400 wait complete. 2023-12-19T08:03:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 Processor -- start 2023-12-19T08:03:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.016+0000 7fad5a84c700 1 -- start start 2023-12-19T08:03:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 0x7fad5416c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540ce2d0 0x7fad540b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b5be0 con 0x7fad540cc040 2023-12-19T08:03:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b44e0 con 0x7fad540ce2d0 2023-12-19T08:03:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad540b38d0 con 0x7fad540caa10 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46540/0 (socket says 172.21.15.111:46540) 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 -- 172.21.15.111:0/609709338 learned_addr learned my addr 172.21.15.111:0/609709338 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a04b700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540ce2d0 0x7fad540b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5984a700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 0x7fad5416c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 -- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 msgr2=0x7fad5416c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 0x7fad5416c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 -- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540ce2d0 msgr2=0x7fad540b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540ce2d0 0x7fad540b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 -- 172.21.15.111:0/609709338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009420 con 0x7fad540cc040 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5984a700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 0x7fad5416c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad59049700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fad4400ec80 tx=0x7fad440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400c8f0 con 0x7fad540cc040 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4400ca50 con 0x7fad540cc040 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad500094f0 con 0x7fad540cc040 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400cdd0 con 0x7fad540cc040 2023-12-19T08:03:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.020+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad540b3c60 con 0x7fad540cc040 2023-12-19T08:03:23.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.024+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad540b4290 con 0x7fad540cc040 2023-12-19T08:03:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.024+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fad44013680 con 0x7fad540cc040 2023-12-19T08:03:23.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.024+0000 7fad49ffb700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 0x7fad38062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.024+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fad4407cfa0 con 0x7fad540cc040 2023-12-19T08:03:23.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.024+0000 7fad5984a700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 0x7fad38062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.028+0000 7fad5984a700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 0x7fad38062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad540696f0 tx=0x7fad50017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.032+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fad4404d110 con 0x7fad540cc040 2023-12-19T08:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.204+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fad540b46a0 con 0x7fad540cc040 2023-12-19T08:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.212+0000 7fad49ffb700 1 -- 172.21.15.111:0/609709338 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v31) v1 ==== 120+0+0 (secure 0 0 0) 0x7fad44050730 con 0x7fad540cc040 2023-12-19T08:03:23.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 msgr2=0x7fad38062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 0x7fad38062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad540696f0 tx=0x7fad50017000 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 msgr2=0x7fad540b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fad4400ec80 tx=0x7fad440091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 shutdown_connections 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fad38060670 0x7fad38062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540ce2d0 0x7fad540b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad540cc040 0x7fad540b2cf0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 --2- 172.21.15.111:0/609709338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa10 0x7fad5416c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 >> 172.21.15.111:0/609709338 conn(0x7fad54067bb0 msgr2=0x7fad540c5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 shutdown_connections 2023-12-19T08:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.216+0000 7fad5a84c700 1 -- 172.21.15.111:0/609709338 wait complete. 2023-12-19T08:03:23.243 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T08:03:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 Processor -- start 2023-12-19T08:03:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 -- start start 2023-12-19T08:03:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e400c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e400c7a80 0x7f2e400c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e40159b80 con 0x7f2e4015a010 2023-12-19T08:03:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e400c8220 con 0x7f2e400c7a80 2023-12-19T08:03:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.404+0000 7f2e44f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e400c7f40 con 0x7f2e40141e70 2023-12-19T08:03:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e400c7a80 0x7f2e400c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e400c7a80 0x7f2e400c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46550/0 (socket says 172.21.15.111:46550) 2023-12-19T08:03:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 -- 172.21.15.111:0/2583716244 learned_addr learned my addr 172.21.15.111:0/2583716244 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3f7fe700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e400c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 -- 172.21.15.111:0/2583716244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 msgr2=0x7f2e400c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e400c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 -- 172.21.15.111:0/2583716244 >> v1:172.21.15.143:6789/0 conn(0x7f2e4015a010 legacy=0x7f2e4015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 -- 172.21.15.111:0/2583716244 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e40159530 con 0x7f2e400c7a80 2023-12-19T08:03:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3effd700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e400c4ce0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2e340099b0 tx=0x7f2e34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1322bc6c4eaf402 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3e7fc700 1 -- 172.21.15.111:0/2583716244 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34009ea0 con 0x7f2e400c7a80 2023-12-19T08:03:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3e7fc700 1 -- 172.21.15.111:0/2583716244 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3400b860 con 0x7f2e400c7a80 2023-12-19T08:03:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e3e7fc700 1 -- 172.21.15.111:0/2583716244 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400bc00 con 0x7f2e400c7a80 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 -- 172.21.15.111:0/2583716244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 msgr2=0x7f2e400c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e400c4ce0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2e340099b0 tx=0x7f2e34009250 comp rx=0 tx=0).stop 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 -- 172.21.15.111:0/2583716244 shutdown_connections 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e400c4ce0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 --2- 172.21.15.111:0/2583716244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e400c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 -- 172.21.15.111:0/2583716244 >> 172.21.15.111:0/2583716244 conn(0x7f2e40067b50 msgr2=0x7f2e400cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 -- 172.21.15.111:0/2583716244 shutdown_connections 2023-12-19T08:03:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.408+0000 7f2e44f23700 1 -- 172.21.15.111:0/2583716244 wait complete. 2023-12-19T08:03:23.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.412+0000 7f2e44f23700 1 Processor -- start 2023-12-19T08:03:23.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.412+0000 7f2e44f23700 1 -- start start 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e40157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4015a010 0x7f2e40157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e400b0a90 con 0x7f2e400c7a80 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4013b930 con 0x7f2e4015a010 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e40158380 con 0x7f2e40141e70 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46556/0 (socket says 172.21.15.111:46556) 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 -- 172.21.15.111:0/922744898 learned_addr learned my addr 172.21.15.111:0/922744898 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3ffff700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4015a010 0x7f2e40157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3effd700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e40157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 -- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 msgr2=0x7f2e40157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e40157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 -- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4015a010 msgr2=0x7f2e40157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4015a010 0x7f2e40157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 -- 172.21.15.111:0/922744898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340094f0 con 0x7f2e400c7a80 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e3f7fe700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000ecb0 tx=0x7f2e300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000c9d0 con 0x7f2e400c7a80 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3000cb30 con 0x7f2e400c7a80 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e40133200 con 0x7f2e400c7a80 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000ced0 con 0x7f2e400c7a80 2023-12-19T08:03:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.416+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e401337c0 con 0x7f2e400c7a80 2023-12-19T08:03:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.420+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e40133b40 con 0x7f2e400c7a80 2023-12-19T08:03:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.420+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2e30013680 con 0x7f2e400c7a80 2023-12-19T08:03:23.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.424+0000 7f2e277fe700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 0x7f2e2c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.424+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f2e3007cfa0 con 0x7f2e400c7a80 2023-12-19T08:03:23.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.424+0000 7f2e3effd700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 0x7f2e2c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.424+0000 7f2e3effd700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 0x7f2e2c062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e340099b0 tx=0x7f2e34017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.428+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e3004d110 con 0x7f2e400c7a80 2023-12-19T08:03:23.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.596+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2e4004be40 con 0x7f2e400c7a80 2023-12-19T08:03:23.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.604+0000 7f2e277fe700 1 -- 172.21.15.111:0/922744898 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2e30050730 con 0x7f2e400c7a80 2023-12-19T08:03:23.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 msgr2=0x7f2e2c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 0x7f2e2c062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e340099b0 tx=0x7f2e34017000 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 msgr2=0x7f2e4006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000ecb0 tx=0x7f2e300091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 shutdown_connections 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e2c0606a0 0x7f2e2c062b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e4015a010 0x7f2e40157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e40141e70 0x7f2e40157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 --2- 172.21.15.111:0/922744898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e400c7a80 0x7f2e4006ad30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 >> 172.21.15.111:0/922744898 conn(0x7f2e40067b50 msgr2=0x7f2e4015a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 shutdown_connections 2023-12-19T08:03:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.608+0000 7f2e44f23700 1 -- 172.21.15.111:0/922744898 wait complete. 2023-12-19T08:03:23.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T08:03:23.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.796+0000 7f41456ae700 1 Processor -- start 2023-12-19T08:03:23.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.796+0000 7f41456ae700 1 -- start start 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f41456ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41400cb970 0x7f414006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f41456ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f41400cecd0 0x7f414006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f41456ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41400cbdf0 con 0x7f4140158e20 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f41456ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41400cc0d0 con 0x7f41400cb970 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f41456ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41400cc3b0 con 0x7f41400cecd0 2023-12-19T08:03:23.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41400cb970 0x7f414006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41400cb970 0x7f414006acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46564/0 (socket says 172.21.15.111:46564) 2023-12-19T08:03:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 -- 172.21.15.111:0/465109769 learned_addr learned my addr 172.21.15.111:0/465109769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413f7fe700 1 --2- 172.21.15.111:0/465109769 >> v2:172.21.15.143:3300/0 conn(0x7f41400cecd0 0x7f414006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 -- 172.21.15.111:0/465109769 >> v2:172.21.15.143:3300/0 conn(0x7f41400cecd0 msgr2=0x7f414006b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:23.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 --2- 172.21.15.111:0/465109769 >> v2:172.21.15.143:3300/0 conn(0x7f41400cecd0 0x7f414006b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 -- 172.21.15.111:0/465109769 >> v1:172.21.15.111:6789/0 conn(0x7f4140158e20 legacy=0x7f41400ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 -- 172.21.15.111:0/465109769 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41400cc690 con 0x7f41400cb970 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413e7fc700 1 --2- 172.21.15.111:0/465109769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cb970 0x7f414006acc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f41340099b0 tx=0x7f4134009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759e75c41613178c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413dffb700 1 -- 172.21.15.111:0/465109769 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400b820 con 0x7f41400cb970 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413dffb700 1 -- 172.21.15.111:0/465109769 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413400b980 con 0x7f41400cb970 2023-12-19T08:03:23.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.800+0000 7f413dffb700 1 -- 172.21.15.111:0/465109769 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400bd20 con 0x7f41400cb970 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 -- 172.21.15.111:0/465109769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cb970 msgr2=0x7f414006acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 --2- 172.21.15.111:0/465109769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cb970 0x7f414006acc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f41340099b0 tx=0x7f4134009250 comp rx=0 tx=0).stop 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 -- 172.21.15.111:0/465109769 shutdown_connections 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 --2- 172.21.15.111:0/465109769 >> v2:172.21.15.143:3300/0 conn(0x7f41400cecd0 0x7f414006b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 --2- 172.21.15.111:0/465109769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cb970 0x7f414006acc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 -- 172.21.15.111:0/465109769 >> 172.21.15.111:0/465109769 conn(0x7f4140067bb0 msgr2=0x7f41400d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:23.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 -- 172.21.15.111:0/465109769 shutdown_connections 2023-12-19T08:03:23.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.804+0000 7f41456ae700 1 -- 172.21.15.111:0/465109769 wait complete. 2023-12-19T08:03:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 Processor -- start 2023-12-19T08:03:23.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 -- start start 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140158e20 0x7f41400c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41400c9370 0x7f41400c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140156aa0 con 0x7f41400cecd0 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41400b41c0 con 0x7f4140158e20 2023-12-19T08:03:23.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f41456ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41400b35b0 con 0x7f41400c9370 2023-12-19T08:03:23.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413e7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140158e20 0x7f41400c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46580/0 (socket says 172.21.15.111:46580) 2023-12-19T08:03:23.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 -- 172.21.15.111:0/703667150 learned_addr learned my addr 172.21.15.111:0/703667150 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:23.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413f7fe700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41400c9370 0x7f41400c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 -- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41400c9370 msgr2=0x7f41400c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41400c9370 0x7f41400c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 -- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140158e20 msgr2=0x7f41400c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140158e20 0x7f41400c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.808+0000 7f413effd700 1 -- 172.21.15.111:0/703667150 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340094f0 con 0x7f41400cecd0 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f413effd700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f413000c590 tx=0x7f413000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130011590 con 0x7f41400cecd0 2023-12-19T08:03:23.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130011710 con 0x7f41400cecd0 2023-12-19T08:03:23.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41400b3890 con 0x7f41400cecd0 2023-12-19T08:03:23.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130011ad0 con 0x7f41400cecd0 2023-12-19T08:03:23.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41400b3e50 con 0x7f41400cecd0 2023-12-19T08:03:23.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.812+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4140071120 con 0x7f41400cecd0 2023-12-19T08:03:23.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.816+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f413001b3b0 con 0x7f41400cecd0 2023-12-19T08:03:23.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.816+0000 7f4126ffd700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 0x7f412c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:23.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.816+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f413007c110 con 0x7f41400cecd0 2023-12-19T08:03:23.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.816+0000 7f413e7fc700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 0x7f412c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:23.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.816+0000 7f413e7fc700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 0x7f412c062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41340099b0 tx=0x7f4134017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:23.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.824+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f413004c280 con 0x7f41400cecd0 2023-12-19T08:03:23.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:23.996+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f414006a7a0 con 0x7f41400cecd0 2023-12-19T08:03:24.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.004+0000 7f4126ffd700 1 -- 172.21.15.111:0/703667150 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v33) v1 ==== 115+0+0 (secure 0 0 0) 0x7f413004f8a0 con 0x7f41400cecd0 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 msgr2=0x7f412c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 0x7f412c062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41340099b0 tx=0x7f4134017000 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 msgr2=0x7f414014aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f413000c590 tx=0x7f413000c8a0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 shutdown_connections 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f412c060670 0x7f412c062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140158e20 0x7f41400c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41400c9370 0x7f41400c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 --2- 172.21.15.111:0/703667150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41400cecd0 0x7f414014aca0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 >> 172.21.15.111:0/703667150 conn(0x7f4140067bb0 msgr2=0x7f414014caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 shutdown_connections 2023-12-19T08:03:24.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.008+0000 7f41456ae700 1 -- 172.21.15.111:0/703667150 wait complete. 2023-12-19T08:03:24.033 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-12-19T08:03:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.184+0000 7f8bec345700 1 Processor -- start 2023-12-19T08:03:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.184+0000 7f8bec345700 1 -- start start 2023-12-19T08:03:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bec345700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8be415d320 0x7f8be415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bec345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bec345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be40c7190 con 0x7f8be406d690 2023-12-19T08:03:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bec345700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be40c1b50 con 0x7f8be415d320 2023-12-19T08:03:24.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bec345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40c1e30 con 0x7f8be40c15b0 2023-12-19T08:03:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8be415d320 0x7f8be415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8be415d320 0x7f8be415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46584/0 (socket says 172.21.15.111:46584) 2023-12-19T08:03:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 -- 172.21.15.111:0/3873112035 learned_addr learned my addr 172.21.15.111:0/3873112035 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8be98e0700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 -- 172.21.15.111:0/3873112035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 msgr2=0x7f8be40aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be40aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 -- 172.21.15.111:0/3873112035 >> v1:172.21.15.111:6789/0 conn(0x7f8be406d690 legacy=0x7f8be40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.188+0000 7f8bea0e1700 1 -- 172.21.15.111:0/3873112035 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be40c6e20 con 0x7f8be415d320 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bea0e1700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be415d6f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009f20 tx=0x7f8be0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d45ad0a5dcb0d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8be90df700 1 -- 172.21.15.111:0/3873112035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be000e070 con 0x7f8be415d320 2023-12-19T08:03:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8be90df700 1 -- 172.21.15.111:0/3873112035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be000b9d0 con 0x7f8be415d320 2023-12-19T08:03:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8be90df700 1 -- 172.21.15.111:0/3873112035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be00092e0 con 0x7f8be415d320 2023-12-19T08:03:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 -- 172.21.15.111:0/3873112035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 msgr2=0x7f8be415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be415d6f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009f20 tx=0x7f8be0009840 comp rx=0 tx=0).stop 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 -- 172.21.15.111:0/3873112035 shutdown_connections 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be40aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 --2- 172.21.15.111:0/3873112035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be415d6f0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 -- 172.21.15.111:0/3873112035 >> 172.21.15.111:0/3873112035 conn(0x7f8be40c8850 msgr2=0x7f8be40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 -- 172.21.15.111:0/3873112035 shutdown_connections 2023-12-19T08:03:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.192+0000 7f8bec345700 1 -- 172.21.15.111:0/3873112035 wait complete. 2023-12-19T08:03:24.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 Processor -- start 2023-12-19T08:03:24.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 -- start start 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be406d690 0x7f8be416eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be416e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be416e910 con 0x7f8be415d320 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be416fdb0 con 0x7f8be406d690 2023-12-19T08:03:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bec345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be416d770 con 0x7f8be40c15b0 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46600/0 (socket says 172.21.15.111:46600) 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 -- 172.21.15.111:0/174406857 learned_addr learned my addr 172.21.15.111:0/174406857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8be98e0700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be416e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea0e1700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be406d690 0x7f8be416eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 -- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 msgr2=0x7f8be416e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be416e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 -- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be406d690 msgr2=0x7f8be416eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be406d690 0x7f8be416eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 -- 172.21.15.111:0/174406857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be0009ae0 con 0x7f8be415d320 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.196+0000 7f8bea8e2700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc00c4c0 tx=0x7f8bdc007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc011590 con 0x7f8be415d320 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc011710 con 0x7f8be415d320 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be416f1a0 con 0x7f8be415d320 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc011ad0 con 0x7f8be415d320 2023-12-19T08:03:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be416f760 con 0x7f8be415d320 2023-12-19T08:03:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be416ff60 con 0x7f8be415d320 2023-12-19T08:03:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.200+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8bdc01b3b0 con 0x7f8be415d320 2023-12-19T08:03:24.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.204+0000 7f8bda7fc700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 0x7f8bc8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:24.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.204+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f8bdc07c050 con 0x7f8be415d320 2023-12-19T08:03:24.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.204+0000 7f8bea0e1700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 0x7f8bc8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:24.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.204+0000 7f8bea0e1700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 0x7f8bc8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009880 tx=0x7f8be0018000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:24.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.208+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8be416ff60 con 0x7f8be415d320 2023-12-19T08:03:24.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.440+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f8be416ff20 con 0x7f8be415d320 2023-12-19T08:03:24.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bda7fc700 1 -- 172.21.15.111:0/174406857 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v606) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8bdc0170e0 con 0x7f8be415d320 2023-12-19T08:03:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 msgr2=0x7f8bc8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 0x7f8bc8062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009880 tx=0x7f8be0018000 comp rx=0 tx=0).stop 2023-12-19T08:03:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 msgr2=0x7f8be416d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:24.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc00c4c0 tx=0x7f8bdc007f50 comp rx=0 tx=0).stop 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 shutdown_connections 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8bc8060670 0x7f8bc8062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be415d320 0x7f8be416d230 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be40c15b0 0x7f8be416e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 --2- 172.21.15.111:0/174406857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be406d690 0x7f8be416eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.840+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 >> 172.21.15.111:0/174406857 conn(0x7f8be40c8850 msgr2=0x7f8be415c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.844+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 shutdown_connections 2023-12-19T08:03:24.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:24.844+0000 7f8bec345700 1 -- 172.21.15.111:0/174406857 wait complete. 2023-12-19T08:03:24.866 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-12-19T08:03:24.867 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-12-19T08:03:24.867 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T08:03:24.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.028+0000 7f8e05945700 1 Processor -- start 2023-12-19T08:03:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.028+0000 7f8e05945700 1 -- start start 2023-12-19T08:03:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8e05945700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e00158c60 0x7f8e000cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8e05945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e0006e770 0x7f8e000c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8e05945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e00068180 con 0x7f8e00067980 2023-12-19T08:03:25.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8e05945700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e000cb5b0 con 0x7f8e00158c60 2023-12-19T08:03:25.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8e05945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e00068460 con 0x7f8e0006e770 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e00158c60 0x7f8e000cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e00158c60 0x7f8e000cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46606/0 (socket says 172.21.15.111:46606) 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 -- 172.21.15.111:0/695624137 learned_addr learned my addr 172.21.15.111:0/695624137 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dff7fe700 1 --2- 172.21.15.111:0/695624137 >> v2:172.21.15.143:3300/0 conn(0x7f8e0006e770 0x7f8e000c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 -- 172.21.15.111:0/695624137 >> v2:172.21.15.143:3300/0 conn(0x7f8e0006e770 msgr2=0x7f8e000c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 --2- 172.21.15.111:0/695624137 >> v2:172.21.15.143:3300/0 conn(0x7f8e0006e770 0x7f8e000c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 -- 172.21.15.111:0/695624137 >> v1:172.21.15.111:6789/0 conn(0x7f8e00067980 legacy=0x7f8e0006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 -- 172.21.15.111:0/695624137 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00068740 con 0x7f8e00158c60 2023-12-19T08:03:25.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfeffd700 1 --2- 172.21.15.111:0/695624137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e00158c60 0x7f8e000cb070 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8df00099b0 tx=0x7f8df0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9406e333d043ac6a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfe7fc700 1 -- 172.21.15.111:0/695624137 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000b820 con 0x7f8e00158c60 2023-12-19T08:03:25.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfe7fc700 1 -- 172.21.15.111:0/695624137 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df000b980 con 0x7f8e00158c60 2023-12-19T08:03:25.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.032+0000 7f8dfe7fc700 1 -- 172.21.15.111:0/695624137 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df000bd20 con 0x7f8e00158c60 2023-12-19T08:03:25.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 -- 172.21.15.111:0/695624137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e00158c60 msgr2=0x7f8e000cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 --2- 172.21.15.111:0/695624137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e00158c60 0x7f8e000cb070 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8df00099b0 tx=0x7f8df0009250 comp rx=0 tx=0).stop 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 -- 172.21.15.111:0/695624137 shutdown_connections 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 --2- 172.21.15.111:0/695624137 >> v2:172.21.15.143:3300/0 conn(0x7f8e0006e770 0x7f8e000c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 --2- 172.21.15.111:0/695624137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e00158c60 0x7f8e000cb070 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 -- 172.21.15.111:0/695624137 >> 172.21.15.111:0/695624137 conn(0x7f8e0006ade0 msgr2=0x7f8e0006b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 -- 172.21.15.111:0/695624137 shutdown_connections 2023-12-19T08:03:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 -- 172.21.15.111:0/695624137 wait complete. 2023-12-19T08:03:25.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.036+0000 7f8e05945700 1 Processor -- start 2023-12-19T08:03:25.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 -- start start 2023-12-19T08:03:25.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e00067980 0x7f8e000b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e00158c60 0x7f8e000aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000b1730 con 0x7f8e0006e770 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000b0e60 con 0x7f8e00067980 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8e05945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0016d4b0 con 0x7f8e00158c60 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46620/0 (socket says 172.21.15.111:46620) 2023-12-19T08:03:25.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 -- 172.21.15.111:0/4075303278 learned_addr learned my addr 172.21.15.111:0/4075303278 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8dff7fe700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e00158c60 0x7f8e000aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8dfeffd700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e00067980 0x7f8e000b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 -- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e00158c60 msgr2=0x7f8e000aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e00158c60 0x7f8e000aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 -- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e00067980 msgr2=0x7f8e000b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e00067980 0x7f8e000b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 -- 172.21.15.111:0/4075303278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00094f0 con 0x7f8e0006e770 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.040+0000 7f8df7fff700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c590 tx=0x7f8de800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8011590 con 0x7f8e0006e770 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8011710 con 0x7f8e0006e770 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8011ad0 con 0x7f8e0006e770 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e000b10e0 con 0x7f8e0006e770 2023-12-19T08:03:25.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e000ae7d0 con 0x7f8e0006e770 2023-12-19T08:03:25.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e000b18e0 con 0x7f8e0006e770 2023-12-19T08:03:25.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.044+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8de801b3b0 con 0x7f8e0006e770 2023-12-19T08:03:25.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.048+0000 7f8df6ffd700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 0x7f8de0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.048+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f8de807c0b0 con 0x7f8e0006e770 2023-12-19T08:03:25.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.048+0000 7f8dfeffd700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 0x7f8de0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.048+0000 7f8dfeffd700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 0x7f8de0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8df00099b0 tx=0x7f8df0017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.052+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e000b18e0 con 0x7f8e0006e770 2023-12-19T08:03:25.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.228+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e0004be40 con 0x7f8e0006e770 2023-12-19T08:03:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.228+0000 7f8df6ffd700 1 -- 172.21.15.111:0/4075303278 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+14250 (secure 0 0 0) 0x7f8de8017090 con 0x7f8e0006e770 2023-12-19T08:03:25.233 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:25.233 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":104,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:22.807732+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6835/2842493331":"2023-12-20T08:03:19.468378+0000","172.21.15.111:6834/2842493331":"2023-12-20T08:03:19.468378+0000","172.21.15.143:0/3111746713":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/4017351320":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/2914349397":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/3376952395":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/2955652133":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/3301845822":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/2763978015":"2023-12-20T08:03:19.240672+0000","172.21.15.143:0/1273338574":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3396798404":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3432369124":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1714582390":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/1261051935":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:12.694202+0000","172.21.15.143:0/3477898242":"2023-12-20T08:03:12.694202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 msgr2=0x7f8de0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 0x7f8de0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8df00099b0 tx=0x7f8df0017000 comp rx=0 tx=0).stop 2023-12-19T08:03:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 msgr2=0x7f8e000ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c590 tx=0x7f8de800c8a0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 shutdown_connections 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8de00606a0 0x7f8de0062b50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e00158c60 0x7f8e000aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e0006e770 0x7f8e000ae290 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 --2- 172.21.15.111:0/4075303278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e00067980 0x7f8e000b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 >> 172.21.15.111:0/4075303278 conn(0x7f8e0006ade0 msgr2=0x7f8e000c2370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 shutdown_connections 2023-12-19T08:03:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.232+0000 7f8e05945700 1 -- 172.21.15.111:0/4075303278 wait complete. 2023-12-19T08:03:25.258 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:03:25.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:03:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.416+0000 7f71cb59e700 1 Processor -- start 2023-12-19T08:03:25.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.416+0000 7f71cb59e700 1 -- start start 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f71cc154780 0x7f71cc15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f71cc06baa0 0x7f71cc157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 0x7f71cc0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc15ae30 con 0x7f71cc154780 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc0abf70 con 0x7f71cc06baa0 2023-12-19T08:03:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc154b50 con 0x7f71cc0adab0 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f71cc154780 0x7f71cc15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f71cc154780 0x7f71cc15b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46628/0 (socket says 172.21.15.111:46628) 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 -- 172.21.15.111:0/3128068545 learned_addr learned my addr 172.21.15.111:0/3128068545 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cad9d700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 0x7f71cc0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71c9d9b700 1 --2- 172.21.15.111:0/3128068545 >> v2:172.21.15.143:3300/0 conn(0x7f71cc06baa0 0x7f71cc157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 -- 172.21.15.111:0/3128068545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 msgr2=0x7f71cc0adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 0x7f71cc0adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 -- 172.21.15.111:0/3128068545 >> v2:172.21.15.143:3300/0 conn(0x7f71cc06baa0 msgr2=0x7f71cc157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 --2- 172.21.15.111:0/3128068545 >> v2:172.21.15.143:3300/0 conn(0x7f71cc06baa0 0x7f71cc157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 -- 172.21.15.111:0/3128068545 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc1580a0 con 0x7f71cc154780 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71cad9d700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 0x7f71cc0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71ca59c700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc154780 0x7f71cc15b2c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cb990 tx=0x7f71bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5070bc16f11bb4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71c959a700 1 -- 172.21.15.111:0/3128068545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00b860 con 0x7f71cc154780 2023-12-19T08:03:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71c959a700 1 -- 172.21.15.111:0/3128068545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc00b9c0 con 0x7f71cc154780 2023-12-19T08:03:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.420+0000 7f71c959a700 1 -- 172.21.15.111:0/3128068545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00bd60 con 0x7f71cc154780 2023-12-19T08:03:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- 172.21.15.111:0/3128068545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc154780 msgr2=0x7f71cc15b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc154780 0x7f71cc15b2c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cb990 tx=0x7f71bc009250 comp rx=0 tx=0).stop 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- 172.21.15.111:0/3128068545 shutdown_connections 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc0adab0 0x7f71cc0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 --2- 172.21.15.111:0/3128068545 >> v2:172.21.15.143:3300/0 conn(0x7f71cc06baa0 0x7f71cc157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 --2- 172.21.15.111:0/3128068545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc154780 0x7f71cc15b2c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- 172.21.15.111:0/3128068545 >> 172.21.15.111:0/3128068545 conn(0x7f71cc067b50 msgr2=0x7f71cc0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- 172.21.15.111:0/3128068545 shutdown_connections 2023-12-19T08:03:25.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- 172.21.15.111:0/3128068545 wait complete. 2023-12-19T08:03:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 Processor -- start 2023-12-19T08:03:25.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.424+0000 7f71cb59e700 1 -- start start 2023-12-19T08:03:25.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc06baa0 0x7f71cc0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 0x7f71cc159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0af880 con 0x7f71cc0adab0 2023-12-19T08:03:25.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0af2c0 con 0x7f71cc154780 2023-12-19T08:03:25.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0af5a0 con 0x7f71cc06baa0 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46644/0 (socket says 172.21.15.111:46644) 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 -- 172.21.15.111:0/915221192 learned_addr learned my addr 172.21.15.111:0/915221192 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71ca59c700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc06baa0 0x7f71cc0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cad9d700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 0x7f71cc159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 -- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc06baa0 msgr2=0x7f71cc0ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc06baa0 0x7f71cc0ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 -- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 msgr2=0x7f71cc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 0x7f71cc159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 -- 172.21.15.111:0/915221192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc0094f0 con 0x7f71cc0adab0 2023-12-19T08:03:25.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cad9d700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 0x7f71cc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71c9d9b700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f71c0006b00 tx=0x7f71c0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c00125f0 con 0x7f71cc0adab0 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0012770 con 0x7f71cc0adab0 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc16c860 con 0x7f71cc0adab0 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0012b30 con 0x7f71cc0adab0 2023-12-19T08:03:25.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.428+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc0aee90 con 0x7f71cc0adab0 2023-12-19T08:03:25.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.432+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc16c6c0 con 0x7f71cc0adab0 2023-12-19T08:03:25.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.432+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f71c001e460 con 0x7f71cc0adab0 2023-12-19T08:03:25.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.432+0000 7f71ba7fc700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 0x7f71ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.432+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f71c007d180 con 0x7f71cc0adab0 2023-12-19T08:03:25.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.436+0000 7f71ca59c700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 0x7f71ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.436+0000 7f71ca59c700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 0x7f71ac062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cb990 tx=0x7f71bc017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.440+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f71cc16c6c0 con 0x7f71cc0adab0 2023-12-19T08:03:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.616+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71cc16c6c0 con 0x7f71cc0adab0 2023-12-19T08:03:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.620+0000 7f71ba7fc700 1 -- 172.21.15.111:0/915221192 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71cc16c6c0 con 0x7f71cc0adab0 2023-12-19T08:03:25.623 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.620+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 msgr2=0x7f71ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.620+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 0x7f71ac062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cb990 tx=0x7f71bc017000 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.620+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 msgr2=0x7f71cc0c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.620+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f71c0006b00 tx=0x7f71c0008040 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 shutdown_connections 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71ac060670 0x7f71ac062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71cc154780 0x7f71cc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71cc0adab0 0x7f71cc0c7400 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 --2- 172.21.15.111:0/915221192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71cc06baa0 0x7f71cc0ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 >> 172.21.15.111:0/915221192 conn(0x7f71cc067b50 msgr2=0x7f71cc153380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 shutdown_connections 2023-12-19T08:03:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.624+0000 7f71cb59e700 1 -- 172.21.15.111:0/915221192 wait complete. 2023-12-19T08:03:25.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-12-19T08:03:25.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.796+0000 7fb772e78700 1 Processor -- start 2023-12-19T08:03:25.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb772e78700 1 -- start start 2023-12-19T08:03:25.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb772e78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb76c068800 0x7fb76c161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb772e78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c06e150 con 0x7fb76c140710 2023-12-19T08:03:25.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb772e78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c06d540 con 0x7fb76c140ae0 2023-12-19T08:03:25.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb772e78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c06d820 con 0x7fb76c068800 2023-12-19T08:03:25.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb770c14700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb76c140710 0x7fb76c0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54600/0 (socket says 172.21.15.111:54600) 2023-12-19T08:03:25.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb770c14700 1 -- 172.21.15.111:0/1606832114 learned_addr learned my addr 172.21.15.111:0/1606832114 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:25.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb771415700 1 --2- 172.21.15.111:0/1606832114 >> v2:172.21.15.143:3300/0 conn(0x7fb76c068800 0x7fb76c161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350758857 0 0) 0x7fb76c06e150 con 0x7fb76c140710 2023-12-19T08:03:25.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75c003570 con 0x7fb76c140710 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261260559 0 0) 0x7fb75c003570 con 0x7fb76c140710 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.800+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 >> v2:172.21.15.143:3300/0 conn(0x7fb76c068800 msgr2=0x7fb76c161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 --2- 172.21.15.111:0/1606832114 >> v2:172.21.15.143:3300/0 conn(0x7fb76c068800 0x7fb76c161cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 >> v1:172.21.15.143:6789/0 conn(0x7fb76c140ae0 legacy=0x7fb76c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c06db00 con 0x7fb76c140710 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fb760001470 con 0x7fb76c140710 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 >> v1:172.21.15.111:6789/0 conn(0x7fb76c140710 legacy=0x7fb76c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb75c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb75c004810 0x7fb75c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb75c0051c0 0x7fb75c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c06d820 con 0x7fb75c003a60 2023-12-19T08:03:25.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c00c050 con 0x7fb75c0051c0 2023-12-19T08:03:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c00c210 con 0x7fb75c004810 2023-12-19T08:03:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb75c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76bfff700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb75c004810 0x7fb75c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb771415700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb75c0051c0 0x7fb75c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 -- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb75c004810 msgr2=0x7fb75c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb75c004810 0x7fb75c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 -- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb75c0051c0 msgr2=0x7fb75c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb75c0051c0 0x7fb75c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 -- 172.21.15.111:0/1606832114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c003570 con 0x7fb75c003a60 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb771415700 1 -- 172.21.15.111:0/1606832114 reap_dead start 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb770c14700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb75c003e30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c3c0 tx=0x7fb760001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c673071939e53446 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76000cc10 con 0x7fb75c003a60 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76000cd70 con 0x7fb75c003a60 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb76b7fe700 1 -- 172.21.15.111:0/1606832114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760005b70 con 0x7fb75c003a60 2023-12-19T08:03:25.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb772e78700 1 -- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 msgr2=0x7fb75c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb772e78700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb75c003e30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c3c0 tx=0x7fb760001e70 comp rx=0 tx=0).stop 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb772e78700 1 -- 172.21.15.111:0/1606832114 shutdown_connections 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb772e78700 1 --2- 172.21.15.111:0/1606832114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb75c003e30 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.804+0000 7fb772e78700 1 -- 172.21.15.111:0/1606832114 >> 172.21.15.111:0/1606832114 conn(0x7fb76c0c8850 msgr2=0x7fb76c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.808+0000 7fb772e78700 1 -- 172.21.15.111:0/1606832114 shutdown_connections 2023-12-19T08:03:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.808+0000 7fb772e78700 1 -- 172.21.15.111:0/1606832114 wait complete. 2023-12-19T08:03:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.808+0000 7fb772e78700 1 Processor -- start 2023-12-19T08:03:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.808+0000 7fb772e78700 1 -- start start 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 0x7fb76c0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1581e0 0x7fb76c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c158b90 con 0x7fb75c003a60 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13bcc0 con 0x7fb76c1581e0 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1670f0 con 0x7fb76c0b07f0 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46670/0 (socket says 172.21.15.111:46670) 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 -- 172.21.15.111:0/3222290952 learned_addr learned my addr 172.21.15.111:0/3222290952 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb76bfff700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 0x7fb76c0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb771415700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1581e0 0x7fb76c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 -- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 msgr2=0x7fb76c0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 0x7fb76c0b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 -- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1581e0 msgr2=0x7fb76c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1581e0 0x7fb76c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 -- 172.21.15.111:0/3222290952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c15a5b0 con 0x7fb75c003a60 2023-12-19T08:03:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb76bfff700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 0x7fb76c0b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb770c14700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c9b0 tx=0x7fb760015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760022070 con 0x7fb75c003a60 2023-12-19T08:03:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760004c10 con 0x7fb75c003a60 2023-12-19T08:03:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c0b1360 con 0x7fb75c003a60 2023-12-19T08:03:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76000cee0 con 0x7fb75c003a60 2023-12-19T08:03:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.812+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c15bff0 con 0x7fb75c003a60 2023-12-19T08:03:25.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.816+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c071120 con 0x7fb75c003a60 2023-12-19T08:03:25.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.816+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb7600072e0 con 0x7fb75c003a60 2023-12-19T08:03:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.816+0000 7fb768ff9700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 0x7fb750062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.820+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fb76007db70 con 0x7fb75c003a60 2023-12-19T08:03:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.820+0000 7fb76bfff700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 0x7fb750062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:25.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.820+0000 7fb76bfff700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 0x7fb750062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0c7f10 tx=0x7fb758007d10 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:25.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:25.824+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb76004dce0 con 0x7fb75c003a60 2023-12-19T08:03:26.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.052+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fb76c0642a0 con 0x7fb75c003a60 2023-12-19T08:03:26.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.856+0000 7fb768ff9700 1 -- 172.21.15.111:0/3222290952 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v608) v1 ==== 186+0+0 (secure 0 0 0) 0x7fb760017020 con 0x7fb75c003a60 2023-12-19T08:03:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 msgr2=0x7fb750062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 0x7fb750062b50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0c7f10 tx=0x7fb758007d10 comp rx=0 tx=0).stop 2023-12-19T08:03:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 msgr2=0x7fb76c13b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb76000c9b0 tx=0x7fb760015000 comp rx=0 tx=0).stop 2023-12-19T08:03:26.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 shutdown_connections 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb7500606a0 0x7fb750062b50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1581e0 0x7fb76c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0b07f0 0x7fb76c0b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 --2- 172.21.15.111:0/3222290952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb75c003a60 0x7fb76c13b780 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 >> 172.21.15.111:0/3222290952 conn(0x7fb76c0c8850 msgr2=0x7fb76c0ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 shutdown_connections 2023-12-19T08:03:26.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:26.860+0000 7fb772e78700 1 -- 172.21.15.111:0/3222290952 wait complete. 2023-12-19T08:03:26.884 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:03:27.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 Processor -- start 2023-12-19T08:03:27.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 -- start start 2023-12-19T08:03:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 0x7fd3600ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3600ca400 0x7fd3600ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3600c7450 con 0x7fd36006b1b0 2023-12-19T08:03:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3600cad50 con 0x7fd3600ca400 2023-12-19T08:03:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd364f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36006bea0 con 0x7fd36015d410 2023-12-19T08:03:27.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.052+0000 7fd35e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 0x7fd3600ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3600ca400 0x7fd3600ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35ed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd36006b1b0 0x7fd36006b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42618/0 (socket says 172.21.15.111:42618) 2023-12-19T08:03:27.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35ed9d700 1 -- 172.21.15.111:0/1962202768 learned_addr learned my addr 172.21.15.111:0/1962202768 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35dd9b700 1 -- 172.21.15.111:0/1962202768 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156148298 0 0) 0x7fd3600c7450 con 0x7fd36006b1b0 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35dd9b700 1 -- 172.21.15.111:0/1962202768 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c003570 con 0x7fd36006b1b0 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 -- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 msgr2=0x7fd3600ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 0x7fd3600ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 -- 172.21.15.111:0/1962202768 >> v1:172.21.15.143:6789/0 conn(0x7fd36006b1b0 legacy=0x7fd36006b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 -- 172.21.15.111:0/1962202768 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600c6e00 con 0x7fd3600ca400 2023-12-19T08:03:27.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35e59c700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 0x7fd3600ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:27.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd357fff700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3600ca400 0x7fd3600ca810 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd3480098e0 tx=0x7fd3480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8db97655df2c93c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35dd9b700 1 -- 172.21.15.111:0/1962202768 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011630 con 0x7fd3600ca400 2023-12-19T08:03:27.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35dd9b700 1 -- 172.21.15.111:0/1962202768 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348011790 con 0x7fd3600ca400 2023-12-19T08:03:27.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd35dd9b700 1 -- 172.21.15.111:0/1962202768 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011b30 con 0x7fd3600ca400 2023-12-19T08:03:27.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 -- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3600ca400 msgr2=0x7fd3600ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3600ca400 0x7fd3600ca810 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd3480098e0 tx=0x7fd3480091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 -- 172.21.15.111:0/1962202768 shutdown_connections 2023-12-19T08:03:27.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3600ca400 0x7fd3600ca810 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 --2- 172.21.15.111:0/1962202768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36015d410 0x7fd3600ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 -- 172.21.15.111:0/1962202768 >> 172.21.15.111:0/1962202768 conn(0x7fd360067bb0 msgr2=0x7fd3600cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:27.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 -- 172.21.15.111:0/1962202768 shutdown_connections 2023-12-19T08:03:27.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.056+0000 7fd364f45700 1 -- 172.21.15.111:0/1962202768 wait complete. 2023-12-19T08:03:27.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 Processor -- start 2023-12-19T08:03:27.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 -- start start 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36006b1b0 0x7fd36016d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36015d410 0x7fd3600c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600c2eb0 con 0x7fd3600ca400 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36016dc70 con 0x7fd36015d410 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd364f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36016df50 con 0x7fd36006b1b0 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd357fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd357fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46676/0 (socket says 172.21.15.111:46676) 2023-12-19T08:03:27.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd357fff700 1 -- 172.21.15.111:0/462601030 learned_addr learned my addr 172.21.15.111:0/462601030 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd35e59c700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36006b1b0 0x7fd36016d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.060+0000 7fd35ed9d700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36015d410 0x7fd3600c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 -- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36006b1b0 msgr2=0x7fd36016d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36006b1b0 0x7fd36016d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 -- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36015d410 msgr2=0x7fd3600c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36015d410 0x7fd3600c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 -- 172.21.15.111:0/462601030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd350009420 con 0x7fd3600ca400 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd357fff700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd36006a040 tx=0x7fd348017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011900 con 0x7fd3600ca400 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348011a60 con 0x7fd3600ca400 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd348009420 con 0x7fd3600ca400 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348025640 con 0x7fd3600ca400 2023-12-19T08:03:27.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36016e510 con 0x7fd3600ca400 2023-12-19T08:03:27.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.064+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3600c3020 con 0x7fd3600ca400 2023-12-19T08:03:27.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.068+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd348025a30 con 0x7fd3600ca400 2023-12-19T08:03:27.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.068+0000 7fd3567fc700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 0x7fd340062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.068+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fd34807e300 con 0x7fd3600ca400 2023-12-19T08:03:27.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.068+0000 7fd35e59c700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 0x7fd340062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.068+0000 7fd35e59c700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 0x7fd340062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd360159d90 tx=0x7fd3500091a0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.072+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd34804e470 con 0x7fd3600ca400 2023-12-19T08:03:27.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.248+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd360071120 con 0x7fd3600ca400 2023-12-19T08:03:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.256+0000 7fd3567fc700 1 -- 172.21.15.111:0/462601030 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd360071120 con 0x7fd3600ca400 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 msgr2=0x7fd340062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 0x7fd340062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd360159d90 tx=0x7fd3500091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 msgr2=0x7fd3600b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd36006a040 tx=0x7fd348017000 comp rx=0 tx=0).stop 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 shutdown_connections 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd340060670 0x7fd340062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36015d410 0x7fd3600c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3600ca400 0x7fd3600b14e0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 --2- 172.21.15.111:0/462601030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36006b1b0 0x7fd36016d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 >> 172.21.15.111:0/462601030 conn(0x7fd360067bb0 msgr2=0x7fd3600c2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 shutdown_connections 2023-12-19T08:03:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.260+0000 7fd364f45700 1 -- 172.21.15.111:0/462601030 wait complete. 2023-12-19T08:03:27.281 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:03:27.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:03:27.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:03:27.368 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:03:27.368 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:03:27.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.524+0000 7fa8baf85700 1 Processor -- start 2023-12-19T08:03:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.524+0000 7fa8baf85700 1 -- start start 2023-12-19T08:03:27.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8baf85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 0x7fa8b4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8baf85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8b414fba0 0x7fa8b414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8baf85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b40c6630 con 0x7fa8b40c5900 2023-12-19T08:03:27.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8baf85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40c6440 con 0x7fa8b414fba0 2023-12-19T08:03:27.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8baf85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40c6910 con 0x7fa8b4067c90 2023-12-19T08:03:27.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b8d21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 0x7fa8b4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8b414fba0 0x7fa8b414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b8d21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 0x7fa8b4068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:47312/0 (socket says 172.21.15.111:47312) 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b8d21700 1 -- 172.21.15.111:0/3096473658 learned_addr learned my addr 172.21.15.111:0/3096473658 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b9522700 1 --1- 172.21.15.111:0/3096473658 >> v1:172.21.15.143:6789/0 conn(0x7fa8b40c5900 0x7fa8b40c5d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42632/0 (socket says 172.21.15.111:42632) 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b3fff700 1 -- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 msgr2=0x7fa8b4068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b3fff700 1 --2- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 0x7fa8b4068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b3fff700 1 -- 172.21.15.111:0/3096473658 >> v1:172.21.15.143:6789/0 conn(0x7fa8b40c5900 legacy=0x7fa8b40c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.528+0000 7fa8b3fff700 1 -- 172.21.15.111:0/3096473658 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b40c6bf0 con 0x7fa8b414fba0 2023-12-19T08:03:27.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8b3fff700 1 --2- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b414fba0 0x7fa8b414ffd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00098e0 tx=0x7fa8a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=938f62d08705a4ac server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8b37fe700 1 -- 172.21.15.111:0/3096473658 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000b640 con 0x7fa8b414fba0 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8b37fe700 1 -- 172.21.15.111:0/3096473658 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a000b7a0 con 0x7fa8b414fba0 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8b37fe700 1 -- 172.21.15.111:0/3096473658 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000bb40 con 0x7fa8b414fba0 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 -- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b414fba0 msgr2=0x7fa8b414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 --2- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b414fba0 0x7fa8b414ffd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00098e0 tx=0x7fa8a00091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 -- 172.21.15.111:0/3096473658 shutdown_connections 2023-12-19T08:03:27.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 --2- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b414fba0 0x7fa8b414ffd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 --2- 172.21.15.111:0/3096473658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b4067c90 0x7fa8b4068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 -- 172.21.15.111:0/3096473658 >> 172.21.15.111:0/3096473658 conn(0x7fa8b40cd6d0 msgr2=0x7fa8b406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:27.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 -- 172.21.15.111:0/3096473658 shutdown_connections 2023-12-19T08:03:27.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.532+0000 7fa8baf85700 1 -- 172.21.15.111:0/3096473658 wait complete. 2023-12-19T08:03:27.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 Processor -- start 2023-12-19T08:03:27.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 -- start start 2023-12-19T08:03:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b4067c90 0x7fa8b40b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b40c5900 0x7fa8b40c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40b4d30 con 0x7fa8b414fba0 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40b4360 con 0x7fa8b4067c90 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8baf85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b40b3750 con 0x7fa8b40c5900 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46686/0 (socket says 172.21.15.111:46686) 2023-12-19T08:03:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 -- 172.21.15.111:0/1919662207 learned_addr learned my addr 172.21.15.111:0/1919662207 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b8d21700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b4067c90 0x7fa8b40b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b3fff700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b40c5900 0x7fa8b40c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 -- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b40c5900 msgr2=0x7fa8b40c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b40c5900 0x7fa8b40c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 -- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b4067c90 msgr2=0x7fa8b40b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b4067c90 0x7fa8b40b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.536+0000 7fa8b9522700 1 -- 172.21.15.111:0/1919662207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0009420 con 0x7fa8b414fba0 2023-12-19T08:03:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8b9522700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00cd00 tx=0x7fa8ac001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00ce60 con 0x7fa8b414fba0 2023-12-19T08:03:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac005df0 con 0x7fa8b414fba0 2023-12-19T08:03:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00ab00 con 0x7fa8b414fba0 2023-12-19T08:03:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b40b3a30 con 0x7fa8b414fba0 2023-12-19T08:03:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b40b3ff0 con 0x7fa8b414fba0 2023-12-19T08:03:27.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.540+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40b44f0 con 0x7fa8b414fba0 2023-12-19T08:03:27.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.544+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa8ac0070f0 con 0x7fa8b414fba0 2023-12-19T08:03:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.544+0000 7fa8b0ff9700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 0x7fa898062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.544+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7fa8ac07ccf0 con 0x7fa8b414fba0 2023-12-19T08:03:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.544+0000 7fa8b8d21700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 0x7fa898062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.544+0000 7fa8b8d21700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 0x7fa898062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8006fd0 tx=0x7fa8a8009250 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.548+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa8ac04cee0 con 0x7fa8b414fba0 2023-12-19T08:03:27.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.764+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b4071120 con 0x7fa8b414fba0 2023-12-19T08:03:27.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8b0ff9700 1 -- 172.21.15.111:0/1919662207 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fa8ac050500 con 0x7fa8b414fba0 2023-12-19T08:03:27.770 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2146629082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2146629082},{"type":"v1","addr":"172.21.15.111:6833","nonce":2146629082}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3633853063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":3633853063},{"type":"v1","addr":"172.21.15.143:6837","nonce":3633853063}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6882,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2806683193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2806683193},{"type":"v1","addr":"172.21.15.143:6836","nonce":2806683193}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2201389409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2201389409},{"type":"v1","addr":"172.21.15.111:6835","nonce":2201389409}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[]} 2023-12-19T08:03:27.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 msgr2=0x7fa898062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 0x7fa898062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8006fd0 tx=0x7fa8a8009250 comp rx=0 tx=0).stop 2023-12-19T08:03:27.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 msgr2=0x7fa8b4151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:27.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00cd00 tx=0x7fa8ac001b10 comp rx=0 tx=0).stop 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 shutdown_connections 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa898060670 0x7fa898062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b414fba0 0x7fa8b4151320 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8b40c5900 0x7fa8b40c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 --2- 172.21.15.111:0/1919662207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8b4067c90 0x7fa8b40b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 >> 172.21.15.111:0/1919662207 conn(0x7fa8b40cd6d0 msgr2=0x7fa8b40ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 shutdown_connections 2023-12-19T08:03:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:27.768+0000 7fa8baf85700 1 -- 172.21.15.111:0/1919662207 wait complete. 2023-12-19T08:03:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 42 2023-12-19T08:03:27.791 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:03:27.792 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:03:27.792 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:03:27.792 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:03:27.792 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:03:27.792 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:03:27.792 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:03:27.792 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:03:27.792 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:03:27.793 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:03:27.793 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:03:27.793 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:03:27.793 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:03:27.793 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:03:27.793 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:03:27.793 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:03:27.807 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:03:27.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:03:27.813 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:03:27.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:03:27.815 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:03:27.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:03:27.816 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:03:27.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:03:27.863 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:03:27.864 DEBUG:teuthology.parallel:result is None 2023-12-19T08:03:27.864 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:03:27.865 DEBUG:teuthology.parallel:result is None 2023-12-19T08:03:27.868 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:03:27.868 DEBUG:teuthology.parallel:result is None 2023-12-19T08:03:27.870 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:03:27.870 DEBUG:teuthology.parallel:result is None 2023-12-19T08:03:27.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:03:27.969 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:03:27.973 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:03:27.975 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:03:27.982 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:03:28.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.036+0000 7f3c69c16700 1 Processor -- start 2023-12-19T08:03:28.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.036+0000 7f3c69c16700 1 -- start start 2023-12-19T08:03:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.036+0000 7f3c69c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.036+0000 7f3c69c16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640ca5a0 con 0x7f3c640cc470 2023-12-19T08:03:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c69c16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6406dac0 con 0x7f3c6406a540 2023-12-19T08:03:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c69c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c640b0260 con 0x7f3c64158e70 2023-12-19T08:03:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c69415700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c640cc470 0x7f3c640cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54624/0 (socket says 172.21.15.111:54624) 2023-12-19T08:03:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c69415700 1 -- 172.21.15.111:0/2026876773 learned_addr learned my addr 172.21.15.111:0/2026876773 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c68c14700 1 --2- 172.21.15.111:0/2026876773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:28.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617562432 0 0) 0x7f3c640ca5a0 con 0x7f3c640cc470 2023-12-19T08:03:28.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003570 con 0x7f3c640cc470 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922145705 0 0) 0x7f3c6406dac0 con 0x7f3c6406a540 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c640ca5a0 con 0x7f3c6406a540 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c68c14700 1 -- 172.21.15.111:0/2026876773 >> v1:172.21.15.143:6789/0 conn(0x7f3c6406a540 legacy=0x7f3c6406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c68c14700 1 -- 172.21.15.111:0/2026876773 >> v1:172.21.15.111:6789/0 conn(0x7f3c640cc470 legacy=0x7f3c640cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c68c14700 1 -- 172.21.15.111:0/2026876773 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6406d220 con 0x7f3c64158e70 2023-12-19T08:03:28.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c68c14700 1 --2- 172.21.15.111:0/2026876773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640afd20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009910 tx=0x7f3c580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e58a53e223aec3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:28.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58011630 con 0x7f3c64158e70 2023-12-19T08:03:28.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58011790 con 0x7f3c64158e70 2023-12-19T08:03:28.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.040+0000 7f3c637fe700 1 -- 172.21.15.111:0/2026876773 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58011b10 con 0x7f3c64158e70 2023-12-19T08:03:28.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 -- 172.21.15.111:0/2026876773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 msgr2=0x7f3c640afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:28.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 --2- 172.21.15.111:0/2026876773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640afd20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009910 tx=0x7f3c580091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:28.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 -- 172.21.15.111:0/2026876773 shutdown_connections 2023-12-19T08:03:28.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 --2- 172.21.15.111:0/2026876773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640afd20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:28.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 -- 172.21.15.111:0/2026876773 >> 172.21.15.111:0/2026876773 conn(0x7f3c640670e0 msgr2=0x7f3c640674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:28.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 -- 172.21.15.111:0/2026876773 shutdown_connections 2023-12-19T08:03:28.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.044+0000 7f3c69c16700 1 -- 172.21.15.111:0/2026876773 wait complete. 2023-12-19T08:03:28.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 Processor -- start 2023-12-19T08:03:28.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 -- start start 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c640cc470 0x7f3c640ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64155840 con 0x7f3c6406a540 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6415b620 con 0x7f3c640cc470 2023-12-19T08:03:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69c16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6415b900 con 0x7f3c64158e70 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46728/0 (socket says 172.21.15.111:46728) 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 -- 172.21.15.111:0/1194518463 learned_addr learned my addr 172.21.15.111:0/1194518463 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69415700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c63fff700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c640cc470 0x7f3c640ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 -- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 msgr2=0x7f3c640ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 -- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c640cc470 msgr2=0x7f3c640ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c640cc470 0x7f3c640ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 -- 172.21.15.111:0/1194518463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58009420 con 0x7f3c6406a540 2023-12-19T08:03:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c69415700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.048+0000 7f3c68c14700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3c58013000 tx=0x7f3c58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58011b10 con 0x7f3c6406a540 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58025400 con 0x7f3c6406a540 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6415bbe0 con 0x7f3c6406a540 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580257a0 con 0x7f3c6406a540 2023-12-19T08:03:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6415a930 con 0x7f3c6406a540 2023-12-19T08:03:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6404be40 con 0x7f3c6406a540 2023-12-19T08:03:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.052+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3c5802e440 con 0x7f3c6406a540 2023-12-19T08:03:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.056+0000 7f3c60ff9700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 0x7f3c48062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.056+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6424+0+0 (secure 0 0 0) 0x7f3c5807e480 con 0x7f3c6406a540 2023-12-19T08:03:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.056+0000 7f3c63fff700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 0x7f3c48062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:28.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.056+0000 7f3c63fff700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 0x7f3c48062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c640d0780 tx=0x7f3c50009040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:28.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.064+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c5804e690 con 0x7f3c6406a540 2023-12-19T08:03:28.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:28.248+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3c6406bb10 con 0x7f3c6406a540 2023-12-19T08:03:29.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.300+0000 7f3c60ff9700 1 -- 172.21.15.111:0/1194518463 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v109) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c58051cb0 con 0x7f3c6406a540 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.300+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 msgr2=0x7f3c48062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.300+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 0x7f3c48062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c640d0780 tx=0x7f3c50009040 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.300+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 msgr2=0x7f3c64155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.300+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3c58013000 tx=0x7f3c58017000 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 shutdown_connections 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c48060670 0x7f3c48062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c640cc470 0x7f3c640ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c64158e70 0x7f3c640ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 --2- 172.21.15.111:0/1194518463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c6406a540 0x7f3c64155300 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 >> 172.21.15.111:0/1194518463 conn(0x7f3c640670e0 msgr2=0x7f3c640cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:29.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 shutdown_connections 2023-12-19T08:03:29.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.304+0000 7f3c69c16700 1 -- 172.21.15.111:0/1194518463 wait complete. 2023-12-19T08:03:29.308 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:03:29.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:03:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.492+0000 7f4ef68f0700 1 Processor -- start 2023-12-19T08:03:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.492+0000 7f4ef68f0700 1 -- start start 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 0x7f4ef00cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ef00cda30 0x7f4ef0157060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00c90e0 0x7f4ef015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00cd5a0 con 0x7f4ef00cda30 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00cc990 con 0x7f4ef0156c90 2023-12-19T08:03:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00ccc70 con 0x7f4ef00c90e0 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ef00cda30 0x7f4ef0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ef00cda30 0x7f4ef0157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46730/0 (socket says 172.21.15.111:46730) 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 -- 172.21.15.111:0/2375781541 learned_addr learned my addr 172.21.15.111:0/2375781541 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef4e8d700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00c90e0 0x7f4ef015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4eeffff700 1 --2- 172.21.15.111:0/2375781541 >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 0x7f4ef00cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 -- 172.21.15.111:0/2375781541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00c90e0 msgr2=0x7f4ef015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00c90e0 0x7f4ef015ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 -- 172.21.15.111:0/2375781541 >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 msgr2=0x7f4ef00cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 --2- 172.21.15.111:0/2375781541 >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 0x7f4ef00cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 -- 172.21.15.111:0/2375781541 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00ccf50 con 0x7f4ef00cda30 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4eeffff700 1 --2- 172.21.15.111:0/2375781541 >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 0x7f4ef00cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ee77fe700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef00cda30 0x7f4ef0157060 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0064950 tx=0x7f4ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddad5777c7f3180 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4eef7fe700 1 -- 172.21.15.111:0/2375781541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800b860 con 0x7f4ef00cda30 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4eef7fe700 1 -- 172.21.15.111:0/2375781541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed800b9c0 con 0x7f4ef00cda30 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4eef7fe700 1 -- 172.21.15.111:0/2375781541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800bd60 con 0x7f4ef00cda30 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 -- 172.21.15.111:0/2375781541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef00cda30 msgr2=0x7f4ef0157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.496+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef00cda30 0x7f4ef0157060 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0064950 tx=0x7f4ed8009250 comp rx=0 tx=0).stop 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 -- 172.21.15.111:0/2375781541 shutdown_connections 2023-12-19T08:03:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00c90e0 0x7f4ef015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/2375781541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef00cda30 0x7f4ef0157060 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/2375781541 >> v2:172.21.15.143:3300/0 conn(0x7f4ef0156c90 0x7f4ef00cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 -- 172.21.15.111:0/2375781541 >> 172.21.15.111:0/2375781541 conn(0x7f4ef006c7c0 msgr2=0x7f4ef006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 -- 172.21.15.111:0/2375781541 shutdown_connections 2023-12-19T08:03:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 -- 172.21.15.111:0/2375781541 wait complete. 2023-12-19T08:03:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.500+0000 7f4ef68f0700 1 Processor -- start 2023-12-19T08:03:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 -- start start 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef00c90e0 0x7f4ef014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00cda30 0x7f4ef00b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef0158770 con 0x7f4ef0156c90 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b19f0 con 0x7f4ef00c90e0 2023-12-19T08:03:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef68f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00b1cd0 con 0x7f4ef00cda30 2023-12-19T08:03:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46742/0 (socket says 172.21.15.111:46742) 2023-12-19T08:03:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 -- 172.21.15.111:0/3190090350 learned_addr learned my addr 172.21.15.111:0/3190090350 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4eeffff700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef00c90e0 0x7f4ef014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ee77fe700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00cda30 0x7f4ef00b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 -- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00cda30 msgr2=0x7f4ef00b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00cda30 0x7f4ef00b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 -- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef00c90e0 msgr2=0x7f4ef014cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef00c90e0 0x7f4ef014cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.504+0000 7f4ef4e8d700 1 -- 172.21.15.111:0/3190090350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80094f0 con 0x7f4ef0156c90 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4ef4e8d700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009fa0 tx=0x7f4ee8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee800ebc0 con 0x7f4ef0156c90 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee800ed20 con 0x7f4ef0156c90 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8014670 con 0x7f4ef0156c90 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef00b1fb0 con 0x7f4ef0156c90 2023-12-19T08:03:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef00b3b90 con 0x7f4ef0156c90 2023-12-19T08:03:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.508+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef0071120 con 0x7f4ef0156c90 2023-12-19T08:03:29.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.512+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4ee8014a20 con 0x7f4ef0156c90 2023-12-19T08:03:29.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.512+0000 7f4eecff9700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 0x7f4ed0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.512+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f4ee807cf10 con 0x7f4ef0156c90 2023-12-19T08:03:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.516+0000 7f4eeffff700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 0x7f4ed0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.516+0000 7f4eeffff700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 0x7f4ed0062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0008040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.520+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4ee804d400 con 0x7f4ef0156c90 2023-12-19T08:03:29.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.752+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4ef00b0670 con 0x7f4ef0156c90 2023-12-19T08:03:29.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.752+0000 7f4eecff9700 1 -- 172.21.15.111:0/3190090350 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4ef00b0670 con 0x7f4ef0156c90 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:03:29.757 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:03:29.758 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.760 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:03:29.761 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.762 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.763 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.764 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:03:29.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:03:29.766 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:03:29.768 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:03:29.769 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:03:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 msgr2=0x7f4ed0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 0x7f4ed0062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0008040 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 msgr2=0x7f4ef00af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009fa0 tx=0x7f4ee8009900 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 shutdown_connections 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ed0060670 0x7f4ed0062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ef0156c90 0x7f4ef00af1e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef00cda30 0x7f4ef00b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 --2- 172.21.15.111:0/3190090350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef00c90e0 0x7f4ef014cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.756+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 >> 172.21.15.111:0/3190090350 conn(0x7f4ef006c7c0 msgr2=0x7f4ef014db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.760+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 shutdown_connections 2023-12-19T08:03:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.760+0000 7f4ef68f0700 1 -- 172.21.15.111:0/3190090350 wait complete. 2023-12-19T08:03:29.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:29.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.940+0000 7fce9def0700 1 Processor -- start 2023-12-19T08:03:29.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.940+0000 7fce9def0700 1 -- start start 2023-12-19T08:03:29.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce9def0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce981502b0 0x7fce98068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce9def0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980c6630 con 0x7fce98067c90 2023-12-19T08:03:29.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce9def0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980c6910 con 0x7fce980c5900 2023-12-19T08:03:29.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce9def0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce980c6bf0 con 0x7fce981502b0 2023-12-19T08:03:29.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce977fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fce98067c90 0x7fce9814fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54632/0 (socket says 172.21.15.111:54632) 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce977fe700 1 -- 172.21.15.111:0/2270899673 learned_addr learned my addr 172.21.15.111:0/2270899673 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce96ffd700 1 --2- 172.21.15.111:0/2270899673 >> v2:172.21.15.111:3300/0 conn(0x7fce981502b0 0x7fce98068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce96ffd700 1 -- 172.21.15.111:0/2270899673 >> v1:172.21.15.143:6789/0 conn(0x7fce980c5900 legacy=0x7fce980c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce96ffd700 1 -- 172.21.15.111:0/2270899673 >> v1:172.21.15.111:6789/0 conn(0x7fce98067c90 legacy=0x7fce9814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce96ffd700 1 -- 172.21.15.111:0/2270899673 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980c6ed0 con 0x7fce981502b0 2023-12-19T08:03:29.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.944+0000 7fce96ffd700 1 --2- 172.21.15.111:0/2270899673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce981502b0 0x7fce98068060 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0098e0 tx=0x7fce8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae5c3f879b6926a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce967fc700 1 -- 172.21.15.111:0/2270899673 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c009e80 con 0x7fce981502b0 2023-12-19T08:03:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce967fc700 1 -- 172.21.15.111:0/2270899673 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c00b860 con 0x7fce981502b0 2023-12-19T08:03:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce967fc700 1 -- 172.21.15.111:0/2270899673 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c00bbe0 con 0x7fce981502b0 2023-12-19T08:03:29.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 -- 172.21.15.111:0/2270899673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce981502b0 msgr2=0x7fce98068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 --2- 172.21.15.111:0/2270899673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce981502b0 0x7fce98068060 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0098e0 tx=0x7fce8c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 -- 172.21.15.111:0/2270899673 shutdown_connections 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 --2- 172.21.15.111:0/2270899673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce981502b0 0x7fce98068060 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 -- 172.21.15.111:0/2270899673 >> 172.21.15.111:0/2270899673 conn(0x7fce980cd6d0 msgr2=0x7fce9806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 -- 172.21.15.111:0/2270899673 shutdown_connections 2023-12-19T08:03:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.948+0000 7fce9def0700 1 -- 172.21.15.111:0/2270899673 wait complete. 2023-12-19T08:03:29.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 Processor -- start 2023-12-19T08:03:29.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 -- start start 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce98067c90 0x7fce980b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 0x7fce9814e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b4970 con 0x7fce980c5900 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b4c50 con 0x7fce98067c90 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce9def0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b4f30 con 0x7fce981502b0 2023-12-19T08:03:29.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce96ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce96ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46758/0 (socket says 172.21.15.111:46758) 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce96ffd700 1 -- 172.21.15.111:0/1767260141 learned_addr learned my addr 172.21.15.111:0/1767260141 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce977fe700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce98067c90 0x7fce980b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.952+0000 7fce97fff700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 0x7fce9814e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 -- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 msgr2=0x7fce9814e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 0x7fce9814e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 -- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce98067c90 msgr2=0x7fce980b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce98067c90 0x7fce980b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 -- 172.21.15.111:0/1767260141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c009420 con 0x7fce980c5900 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce97fff700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 0x7fce9814e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:29.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce96ffd700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fce9806a0f0 tx=0x7fce8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c011420 con 0x7fce980c5900 2023-12-19T08:03:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c011580 con 0x7fce980c5900 2023-12-19T08:03:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c011940 con 0x7fce980c5900 2023-12-19T08:03:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce980b5210 con 0x7fce980c5900 2023-12-19T08:03:29.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce980b4590 con 0x7fce980c5900 2023-12-19T08:03:29.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.956+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce980b5710 con 0x7fce980c5900 2023-12-19T08:03:29.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.960+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fce8c011cf0 con 0x7fce980c5900 2023-12-19T08:03:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.960+0000 7fce7f7fe700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 0x7fce84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:29.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.960+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fce8c07e2d0 con 0x7fce980c5900 2023-12-19T08:03:29.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.960+0000 7fce977fe700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 0x7fce84062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:29.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.964+0000 7fce977fe700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 0x7fce84062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce88007130 tx=0x7fce88007eb0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:29.968+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce8c04e720 con 0x7fce980c5900 2023-12-19T08:03:30.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.136+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce98071120 con 0x7fce980c5900 2023-12-19T08:03:30.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.136+0000 7fce7f7fe700 1 -- 172.21.15.111:0/1767260141 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+13178 (secure 0 0 0) 0x7fce8c051d40 con 0x7fce980c5900 2023-12-19T08:03:30.141 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:30.141 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":109,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:29.293820+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 msgr2=0x7fce84062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 0x7fce84062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce88007130 tx=0x7fce88007eb0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 msgr2=0x7fce981522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fce9806a0f0 tx=0x7fce8c017000 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 shutdown_connections 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce84060670 0x7fce84062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce981502b0 0x7fce9814e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce980c5900 0x7fce981522b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 --2- 172.21.15.111:0/1767260141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce98067c90 0x7fce980b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 >> 172.21.15.111:0/1767260141 conn(0x7fce980cd6d0 msgr2=0x7fce9806d610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 shutdown_connections 2023-12-19T08:03:30.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.140+0000 7fce9def0700 1 -- 172.21.15.111:0/1767260141 wait complete. 2023-12-19T08:03:30.165 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:03:30.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:03:30.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.336+0000 7fd48924c700 1 Processor -- start 2023-12-19T08:03:30.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.336+0000 7fd48924c700 1 -- start start 2023-12-19T08:03:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd48924c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd48413f2d0 0x7fd4840ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd48924c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840adc60 con 0x7fd48413f6a0 2023-12-19T08:03:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd48924c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840ad050 con 0x7fd4840c7780 2023-12-19T08:03:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd48924c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4840ad330 con 0x7fd48413f2d0 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd47b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd48413f6a0 0x7fd48415a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54634/0 (socket says 172.21.15.111:54634) 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd47b7fe700 1 -- 172.21.15.111:0/2104308006 learned_addr learned my addr 172.21.15.111:0/2104308006 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd483fff700 1 --2- 172.21.15.111:0/2104308006 >> v2:172.21.15.143:3300/0 conn(0x7fd48413f2d0 0x7fd4840ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181658720 0 0) 0x7fd4840adc60 con 0x7fd48413f6a0 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd470003570 con 0x7fd48413f6a0 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392759092 0 0) 0x7fd4840ad050 con 0x7fd4840c7780 2023-12-19T08:03:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4840adc60 con 0x7fd4840c7780 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1484785609 0 0) 0x7fd470003570 con 0x7fd48413f6a0 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 >> v2:172.21.15.143:3300/0 conn(0x7fd48413f2d0 msgr2=0x7fd4840ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 --2- 172.21.15.111:0/2104308006 >> v2:172.21.15.143:3300/0 conn(0x7fd48413f2d0 0x7fd4840ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 >> v1:172.21.15.143:6789/0 conn(0x7fd4840c7780 legacy=0x7fd48406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840ad610 con 0x7fd48413f6a0 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fd46c001470 con 0x7fd48413f6a0 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd483fff700 1 --2- 172.21.15.111:0/2104308006 >> v2:172.21.15.143:3300/0 conn(0x7fd48413f2d0 0x7fd4840ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 >> v1:172.21.15.111:6789/0 conn(0x7fd48413f6a0 legacy=0x7fd48415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd470003bf0 0x7fd470003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd470004810 0x7fd470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 0x7fd47000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840ad330 con 0x7fd4700051c0 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47000c0b0 con 0x7fd470003bf0 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47000e480 con 0x7fd470004810 2023-12-19T08:03:30.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.340+0000 7fd488a4b700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 0x7fd47000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd483fff700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd470003bf0 0x7fd470003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd47b7fe700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd470004810 0x7fd470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 -- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd470004810 msgr2=0x7fd470004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd470004810 0x7fd470004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 -- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd470003bf0 msgr2=0x7fd470003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd470003bf0 0x7fd470003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 -- 172.21.15.111:0/2104308006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470003570 con 0x7fd4700051c0 2023-12-19T08:03:30.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 -- 172.21.15.111:0/2104308006 reap_dead start 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd488a4b700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 0x7fd47000ba60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd47c002120 tx=0x7fd47c0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18924833f53bd5e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c00c980 con 0x7fd4700051c0 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c00cae0 con 0x7fd4700051c0 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd4837fe700 1 -- 172.21.15.111:0/2104308006 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c00ce80 con 0x7fd4700051c0 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 -- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 msgr2=0x7fd47000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 0x7fd47000ba60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd47c002120 tx=0x7fd47c0017a0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 -- 172.21.15.111:0/2104308006 shutdown_connections 2023-12-19T08:03:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 --2- 172.21.15.111:0/2104308006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4700051c0 0x7fd47000ba60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 -- 172.21.15.111:0/2104308006 >> 172.21.15.111:0/2104308006 conn(0x7fd484067b50 msgr2=0x7fd4840cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 -- 172.21.15.111:0/2104308006 shutdown_connections 2023-12-19T08:03:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.344+0000 7fd48924c700 1 -- 172.21.15.111:0/2104308006 wait complete. 2023-12-19T08:03:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 Processor -- start 2023-12-19T08:03:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 -- start start 2023-12-19T08:03:30.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4700051c0 0x7fd48416bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd484158720 0x7fd4840c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4840a4650 con 0x7fd4841582b0 2023-12-19T08:03:30.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48416bad0 con 0x7fd4700051c0 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd48924c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48416aec0 con 0x7fd484158720 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd47b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd47b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46776/0 (socket says 172.21.15.111:46776) 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.348+0000 7fd47b7fe700 1 -- 172.21.15.111:0/1201644604 learned_addr learned my addr 172.21.15.111:0/1201644604 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd483fff700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4700051c0 0x7fd48416bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd488a4b700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd484158720 0x7fd4840c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 -- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd484158720 msgr2=0x7fd4840c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd484158720 0x7fd4840c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 -- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4700051c0 msgr2=0x7fd48416bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4700051c0 0x7fd48416bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 -- 172.21.15.111:0/1201644604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48416b1a0 con 0x7fd4841582b0 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd47b7fe700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd46c007390 tx=0x7fd46c001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c018030 con 0x7fd4841582b0 2023-12-19T08:03:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c00f6c0 con 0x7fd4841582b0 2023-12-19T08:03:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48416b480 con 0x7fd4841582b0 2023-12-19T08:03:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00fa80 con 0x7fd4841582b0 2023-12-19T08:03:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4840b1390 con 0x7fd4841582b0 2023-12-19T08:03:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.352+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48404be40 con 0x7fd4841582b0 2023-12-19T08:03:30.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.356+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd46c01b070 con 0x7fd4841582b0 2023-12-19T08:03:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.356+0000 7fd480ff9700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 0x7fd464062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.356+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fd46c019070 con 0x7fd4841582b0 2023-12-19T08:03:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.356+0000 7fd483fff700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 0x7fd464062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.356+0000 7fd483fff700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 0x7fd464062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd4840c5d80 tx=0x7fd4740091a0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.364+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd46c04cb00 con 0x7fd4841582b0 2023-12-19T08:03:30.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.532+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd48406ba60 con 0x7fd4841582b0 2023-12-19T08:03:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.532+0000 7fd480ff9700 1 -- 172.21.15.111:0/1201644604 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd46c050120 con 0x7fd4841582b0 2023-12-19T08:03:30.536 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:03:30.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 msgr2=0x7fd464062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 0x7fd464062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd4840c5d80 tx=0x7fd4740091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 msgr2=0x7fd48416a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd46c007390 tx=0x7fd46c001df0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 shutdown_connections 2023-12-19T08:03:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd464060670 0x7fd464062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd484158720 0x7fd4840c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4841582b0 0x7fd48416a440 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 --2- 172.21.15.111:0/1201644604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4700051c0 0x7fd48416bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 >> 172.21.15.111:0/1201644604 conn(0x7fd484067b50 msgr2=0x7fd48413b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:30.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 shutdown_connections 2023-12-19T08:03:30.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.536+0000 7fd48924c700 1 -- 172.21.15.111:0/1201644604 wait complete. 2023-12-19T08:03:30.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:03:30.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.724+0000 7f21188ae700 1 Processor -- start 2023-12-19T08:03:30.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 -- start start 2023-12-19T08:03:30.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2114140710 0x7f21140c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2114068800 0x7f21140ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211406e150 con 0x7f2114140ae0 2023-12-19T08:03:30.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211406d540 con 0x7f2114068800 2023-12-19T08:03:30.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f21188ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211406d820 con 0x7f2114140710 2023-12-19T08:03:30.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f210bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2114140ae0 0x7f21140680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54646/0 (socket says 172.21.15.111:54646) 2023-12-19T08:03:30.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f210bfff700 1 -- 172.21.15.111:0/175842596 learned_addr learned my addr 172.21.15.111:0/175842596 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:30.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f2112d9d700 1 --2- 172.21.15.111:0/175842596 >> v2:172.21.15.111:3300/0 conn(0x7f2114068800 0x7f21140ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f211259c700 1 --2- 172.21.15.111:0/175842596 >> v2:172.21.15.143:3300/0 conn(0x7f2114140710 0x7f21140c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.728+0000 7f2111d9b700 1 -- 172.21.15.111:0/175842596 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238870981 0 0) 0x7f211406e150 con 0x7f2114140ae0 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2111d9b700 1 -- 172.21.15.111:0/175842596 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2100003570 con 0x7f2114140ae0 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2112d9d700 1 -- 172.21.15.111:0/175842596 >> v2:172.21.15.143:3300/0 conn(0x7f2114140710 msgr2=0x7f21140c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2112d9d700 1 --2- 172.21.15.111:0/175842596 >> v2:172.21.15.143:3300/0 conn(0x7f2114140710 0x7f21140c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2112d9d700 1 -- 172.21.15.111:0/175842596 >> v1:172.21.15.111:6789/0 conn(0x7f2114140ae0 legacy=0x7f21140680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2112d9d700 1 -- 172.21.15.111:0/175842596 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211406db00 con 0x7f2114068800 2023-12-19T08:03:30.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2112d9d700 1 --2- 172.21.15.111:0/175842596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114068800 0x7f21140ac250 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2114068c30 tx=0x7f210c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac271e01c7deb38 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2111d9b700 1 -- 172.21.15.111:0/175842596 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c00e070 con 0x7f2114068800 2023-12-19T08:03:30.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2111d9b700 1 -- 172.21.15.111:0/175842596 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c00bab0 con 0x7f2114068800 2023-12-19T08:03:30.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f2111d9b700 1 -- 172.21.15.111:0/175842596 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c00be70 con 0x7f2114068800 2023-12-19T08:03:30.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 -- 172.21.15.111:0/175842596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114068800 msgr2=0x7f21140ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 --2- 172.21.15.111:0/175842596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114068800 0x7f21140ac250 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2114068c30 tx=0x7f210c009780 comp rx=0 tx=0).stop 2023-12-19T08:03:30.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 -- 172.21.15.111:0/175842596 shutdown_connections 2023-12-19T08:03:30.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 --2- 172.21.15.111:0/175842596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114068800 0x7f21140ac250 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 --2- 172.21.15.111:0/175842596 >> v2:172.21.15.143:3300/0 conn(0x7f2114140710 0x7f21140c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 -- 172.21.15.111:0/175842596 >> 172.21.15.111:0/175842596 conn(0x7f21140c8850 msgr2=0x7f21140c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:30.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 -- 172.21.15.111:0/175842596 shutdown_connections 2023-12-19T08:03:30.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.732+0000 7f21188ae700 1 -- 172.21.15.111:0/175842596 wait complete. 2023-12-19T08:03:30.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 Processor -- start 2023-12-19T08:03:30.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 -- start start 2023-12-19T08:03:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2114068800 0x7f211416da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2114140710 0x7f2114156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211413f330 con 0x7f2114140ae0 2023-12-19T08:03:30.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21140a9820 con 0x7f2114140710 2023-12-19T08:03:30.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f21188ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211416df80 con 0x7f2114068800 2023-12-19T08:03:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f2112d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f210bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2114140710 0x7f2114156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f2112d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46792/0 (socket says 172.21.15.111:46792) 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f2112d9d700 1 -- 172.21.15.111:0/1921955811 learned_addr learned my addr 172.21.15.111:0/1921955811 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.736+0000 7f211259c700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2114068800 0x7f211416da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 -- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2114068800 msgr2=0x7f211416da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2114068800 0x7f211416da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 -- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2114140710 msgr2=0x7f2114156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2114140710 0x7f2114156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 -- 172.21.15.111:0/1921955811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c009a20 con 0x7f2114140ae0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f2112d9d700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f210c009fd0 tx=0x7f210c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c012dd0 con 0x7f2114140ae0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c027470 con 0x7f2114140ae0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c027830 con 0x7f2114140ae0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21141570d0 con 0x7f2114140ae0 2023-12-19T08:03:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21140a9dd0 con 0x7f2114140ae0 2023-12-19T08:03:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.740+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2114071120 con 0x7f2114140ae0 2023-12-19T08:03:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.744+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f210c024070 con 0x7f2114140ae0 2023-12-19T08:03:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.744+0000 7f210a7fc700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 0x7f20f4062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.744+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f210c07ee10 con 0x7f2114140ae0 2023-12-19T08:03:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.744+0000 7f211259c700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 0x7f20f4062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.744+0000 7f211259c700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 0x7f20f4062bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2104009db0 tx=0x7f21040091a0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:30.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.752+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f210c04f380 con 0x7f2114140ae0 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.920+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f21140642a0 con 0x7f2114140ae0 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.920+0000 7f210a7fc700 1 -- 172.21.15.111:0/1921955811 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+65 (secure 0 0 0) 0x7f210c0529a0 con 0x7f2114140ae0 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:03:30.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:30.925 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 msgr2=0x7f20f4062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 0x7f20f4062bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2104009db0 tx=0x7f21040091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 msgr2=0x7f21140aab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f210c009fd0 tx=0x7f210c019000 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 shutdown_connections 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f20f4060740 0x7f20f4062bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2114140710 0x7f2114156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2114140ae0 0x7f21140aab10 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 --2- 172.21.15.111:0/1921955811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2114068800 0x7f211416da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 >> 172.21.15.111:0/1921955811 conn(0x7f21140c8850 msgr2=0x7f21140c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:30.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 shutdown_connections 2023-12-19T08:03:30.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:30.924+0000 7f21188ae700 1 -- 172.21.15.111:0/1921955811 wait complete. 2023-12-19T08:03:30.943 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:03:30.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:03:31.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.104+0000 7f5ce6e0c700 1 Processor -- start 2023-12-19T08:03:31.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.104+0000 7f5ce6e0c700 1 -- start start 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 0x7f5ce006af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce00cb3f0 con 0x7f5ce00cac50 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce006d220 con 0x7f5ce00aedc0 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00cc140 con 0x7f5ce00af190 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce660b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5ce00cac50 0x7f5ce00af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54662/0 (socket says 172.21.15.111:54662) 2023-12-19T08:03:31.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce660b700 1 -- 172.21.15.111:0/48234373 learned_addr learned my addr 172.21.15.111:0/48234373 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:31.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce5609700 1 --2- 172.21.15.111:0/48234373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 0x7f5ce006af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:31.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce4e08700 1 -- 172.21.15.111:0/48234373 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717670170 0 0) 0x7f5ce00cb3f0 con 0x7f5ce00cac50 2023-12-19T08:03:31.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce4e08700 1 -- 172.21.15.111:0/48234373 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc003570 con 0x7f5ce00cac50 2023-12-19T08:03:31.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce5609700 1 -- 172.21.15.111:0/48234373 >> v1:172.21.15.143:6789/0 conn(0x7f5ce00aedc0 legacy=0x7f5ce006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce5609700 1 -- 172.21.15.111:0/48234373 >> v1:172.21.15.111:6789/0 conn(0x7f5ce00cac50 legacy=0x7f5ce00af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce5609700 1 -- 172.21.15.111:0/48234373 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce006c8f0 con 0x7f5ce00af190 2023-12-19T08:03:31.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce5609700 1 --2- 172.21.15.111:0/48234373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 0x7f5ce006af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009910 tx=0x7f5cd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d5035b044b3576 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:31.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce4e08700 1 -- 172.21.15.111:0/48234373 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000b680 con 0x7f5ce00af190 2023-12-19T08:03:31.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce4e08700 1 -- 172.21.15.111:0/48234373 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd000b7e0 con 0x7f5ce00af190 2023-12-19T08:03:31.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce4e08700 1 -- 172.21.15.111:0/48234373 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000bba0 con 0x7f5ce00af190 2023-12-19T08:03:31.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/48234373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 msgr2=0x7f5ce006af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/48234373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 0x7f5ce006af50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009910 tx=0x7f5cd00091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/48234373 shutdown_connections 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/48234373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00af190 0x7f5ce006af50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.108+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/48234373 >> 172.21.15.111:0/48234373 conn(0x7f5ce00cc760 msgr2=0x7f5ce00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/48234373 shutdown_connections 2023-12-19T08:03:31.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/48234373 wait complete. 2023-12-19T08:03:31.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 Processor -- start 2023-12-19T08:03:31.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- start start 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 0x7f5ce00c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00cac50 0x7f5ce00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce015a060 con 0x7f5ce00af190 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00c5bf0 con 0x7f5ce00aedc0 2023-12-19T08:03:31.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.112+0000 7f5ce6e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00c4fe0 con 0x7f5ce00cac50 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46800/0 (socket says 172.21.15.111:46800) 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 -- 172.21.15.111:0/2948636570 learned_addr learned my addr 172.21.15.111:0/2948636570 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce660b700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00cac50 0x7f5ce00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5e0a700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 0x7f5ce00c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:31.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 -- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00cac50 msgr2=0x7f5ce00b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00cac50 0x7f5ce00b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 -- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 msgr2=0x7f5ce00c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 0x7f5ce00c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 -- 172.21.15.111:0/2948636570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd0009420 con 0x7f5ce00af190 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5e0a700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 0x7f5ce00c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:31.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce5609700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0068ed0 tx=0x7f5cd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:31.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd000bb60 con 0x7f5ce00af190 2023-12-19T08:03:31.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd0000c90 con 0x7f5ce00af190 2023-12-19T08:03:31.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce00c52c0 con 0x7f5ce00af190 2023-12-19T08:03:31.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd001a660 con 0x7f5ce00af190 2023-12-19T08:03:31.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.116+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce00c5880 con 0x7f5ce00af190 2023-12-19T08:03:31.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.120+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce00c5d80 con 0x7f5ce00af190 2023-12-19T08:03:31.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.120+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5cd001aa10 con 0x7f5ce00af190 2023-12-19T08:03:31.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.120+0000 7f5cd5ffb700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 0x7f5cc4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:31.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.120+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f5cd007e630 con 0x7f5ce00af190 2023-12-19T08:03:31.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.120+0000 7f5ce5e0a700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 0x7f5cc4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:31.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.124+0000 7f5ce5e0a700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 0x7f5cc4062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc007fd0 tx=0x7f5cdc009040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:31.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.128+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5cd004ea80 con 0x7f5ce00af190 2023-12-19T08:03:31.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.296+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5ce004be40 con 0x7f5ce00af190 2023-12-19T08:03:31.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.900+0000 7f5cd5ffb700 1 -- 172.21.15.111:0/2948636570 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5cd00520a0 con 0x7f5ce00af190 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 msgr2=0x7f5cc4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 0x7f5cc4062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc007fd0 tx=0x7f5cdc009040 comp rx=0 tx=0).stop 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 msgr2=0x7f5ce0147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0068ed0 tx=0x7f5cd0017000 comp rx=0 tx=0).stop 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 shutdown_connections 2023-12-19T08:03:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5cc4060670 0x7f5cc4062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce00cac50 0x7f5ce00b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ce00af190 0x7f5ce0147200 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 --2- 172.21.15.111:0/2948636570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce00aedc0 0x7f5ce00c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 >> 172.21.15.111:0/2948636570 conn(0x7f5ce00cc760 msgr2=0x7f5ce00ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 shutdown_connections 2023-12-19T08:03:31.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:31.904+0000 7f5ce6e0c700 1 -- 172.21.15.111:0/2948636570 wait complete. 2023-12-19T08:03:31.912 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:03:31.930 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:03:32.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.092+0000 7f63b3ef0700 1 Processor -- start 2023-12-19T08:03:32.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.092+0000 7f63b3ef0700 1 -- start start 2023-12-19T08:03:32.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b3ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 0x7f63ac06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:32.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b3ef0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac0cbdf0 con 0x7f63ac158e20 2023-12-19T08:03:32.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b3ef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac0cc0d0 con 0x7f63ac0cecd0 2023-12-19T08:03:32.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b3ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0cc3b0 con 0x7f63ac0cb970 2023-12-19T08:03:32.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b1c8c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f63ac158e20 0x7f63ac0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34312/0 (socket says 172.21.15.111:34312) 2023-12-19T08:03:32.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b1c8c700 1 -- 172.21.15.111:0/1613188800 learned_addr learned my addr 172.21.15.111:0/1613188800 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b148b700 1 --2- 172.21.15.111:0/1613188800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 0x7f63ac06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759730703 0 0) 0x7f63ac0cbdf0 con 0x7f63ac158e20 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.096+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6398003570 con 0x7f63ac158e20 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484989426 0 0) 0x7f63ac0cc0d0 con 0x7f63ac0cecd0 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac0cbdf0 con 0x7f63ac0cecd0 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b148b700 1 -- 172.21.15.111:0/1613188800 >> v1:172.21.15.143:6789/0 conn(0x7f63ac0cecd0 legacy=0x7f63ac06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b148b700 1 -- 172.21.15.111:0/1613188800 >> v1:172.21.15.111:6789/0 conn(0x7f63ac158e20 legacy=0x7f63ac0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b148b700 1 -- 172.21.15.111:0/1613188800 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0cc690 con 0x7f63ac0cb970 2023-12-19T08:03:32.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b148b700 1 --2- 172.21.15.111:0/1613188800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 0x7f63ac06acc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f639c009910 tx=0x7f639c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60f5a20238e98647 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:32.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011630 con 0x7f63ac0cb970 2023-12-19T08:03:32.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c011790 con 0x7f63ac0cb970 2023-12-19T08:03:32.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b0c8a700 1 -- 172.21.15.111:0/1613188800 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011b30 con 0x7f63ac0cb970 2023-12-19T08:03:32.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 -- 172.21.15.111:0/1613188800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 msgr2=0x7f63ac06acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/1613188800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 0x7f63ac06acc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f639c009910 tx=0x7f639c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 -- 172.21.15.111:0/1613188800 shutdown_connections 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/1613188800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac0cb970 0x7f63ac06acc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 -- 172.21.15.111:0/1613188800 >> 172.21.15.111:0/1613188800 conn(0x7f63ac067bb0 msgr2=0x7f63ac0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 -- 172.21.15.111:0/1613188800 shutdown_connections 2023-12-19T08:03:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.100+0000 7f63b3ef0700 1 -- 172.21.15.111:0/1613188800 wait complete. 2023-12-19T08:03:32.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 Processor -- start 2023-12-19T08:03:32.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 -- start start 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ac0cecd0 0x7f63ac1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 0x7f63ac159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac154c60 con 0x7f63ac158e20 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac1536e0 con 0x7f63ac0cecd0 2023-12-19T08:03:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.104+0000 7f63b3ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0b27a0 con 0x7f63ac16e400 2023-12-19T08:03:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45268/0 (socket says 172.21.15.111:45268) 2023-12-19T08:03:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 -- 172.21.15.111:0/3546223907 learned_addr learned my addr 172.21.15.111:0/3546223907 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b1c8c700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ac0cecd0 0x7f63ac1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b248d700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 0x7f63ac159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 -- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 msgr2=0x7f63ac159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 0x7f63ac159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 -- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ac0cecd0 msgr2=0x7f63ac1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ac0cecd0 0x7f63ac1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 -- 172.21.15.111:0/3546223907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009420 con 0x7f63ac158e20 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b248d700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 0x7f63ac159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b148b700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f639c009f00 tx=0x7f639c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c01f070 con 0x7f63ac158e20 2023-12-19T08:03:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c000bc0 con 0x7f63ac158e20 2023-12-19T08:03:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac154610 con 0x7f63ac158e20 2023-12-19T08:03:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c025420 con 0x7f63ac158e20 2023-12-19T08:03:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.108+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac0b1a40 con 0x7f63ac158e20 2023-12-19T08:03:32.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.112+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac154470 con 0x7f63ac158e20 2023-12-19T08:03:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.112+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f639c023070 con 0x7f63ac158e20 2023-12-19T08:03:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.116+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 0x7f6390062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.116+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6063+0+0 (secure 0 0 0) 0x7f639c07dff0 con 0x7f63ac158e20 2023-12-19T08:03:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.116+0000 7f63b1c8c700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 0x7f6390062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:32.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.116+0000 7f63b1c8c700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 0x7f6390062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f63a8007fd0 tx=0x7f63a8009040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:32.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.116+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f639c04e3f0 con 0x7f63ac158e20 2023-12-19T08:03:32.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.288+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f63ac154470 con 0x7f63ac158e20 2023-12-19T08:03:32.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.920+0000 7f63a1ffb700 1 -- 172.21.15.111:0/3546223907 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7f63ac154470 con 0x7f63ac158e20 2023-12-19T08:03:32.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 msgr2=0x7f6390062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 0x7f6390062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f63a8007fd0 tx=0x7f63a8009040 comp rx=0 tx=0).stop 2023-12-19T08:03:32.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 msgr2=0x7f63ac1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:32.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f639c009f00 tx=0x7f639c017000 comp rx=0 tx=0).stop 2023-12-19T08:03:32.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 shutdown_connections 2023-12-19T08:03:32.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6390060670 0x7f6390062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63ac16e400 0x7f63ac159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac158e20 0x7f63ac1598d0 secure :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f639c009f00 tx=0x7f639c017000 comp rx=0 tx=0).stop 2023-12-19T08:03:32.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 --2- 172.21.15.111:0/3546223907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ac0cecd0 0x7f63ac1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:32.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.932+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 >> 172.21.15.111:0/3546223907 conn(0x7f63ac067bb0 msgr2=0x7f63ac06afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:32.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.936+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 shutdown_connections 2023-12-19T08:03:32.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:32.936+0000 7f63b3ef0700 1 -- 172.21.15.111:0/3546223907 wait complete. 2023-12-19T08:03:32.948 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:03:32.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:03:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.128+0000 7fba8346e700 1 Processor -- start 2023-12-19T08:03:33.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.128+0000 7fba8346e700 1 -- start start 2023-12-19T08:03:33.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba8346e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0cb970 0x7fba7c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba8346e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c0cbdf0 con 0x7fba7c0cecd0 2023-12-19T08:03:33.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba8346e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c0cc0d0 con 0x7fba7c158e20 2023-12-19T08:03:33.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba8346e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0cc3b0 con 0x7fba7c0cb970 2023-12-19T08:03:33.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba81a0b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fba7c0cecd0 0x7fba7c06b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34328/0 (socket says 172.21.15.111:34328) 2023-12-19T08:03:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba81a0b700 1 -- 172.21.15.111:0/4122531928 learned_addr learned my addr 172.21.15.111:0/4122531928 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba80a09700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0cb970 0x7fba7c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315312116 0 0) 0x7fba7c0cbdf0 con 0x7fba7c0cecd0 2023-12-19T08:03:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba68003570 con 0x7fba7c0cecd0 2023-12-19T08:03:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891638198 0 0) 0x7fba7c0cc0d0 con 0x7fba7c158e20 2023-12-19T08:03:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c0cbdf0 con 0x7fba7c158e20 2023-12-19T08:03:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402968162 0 0) 0x7fba68003570 con 0x7fba7c0cecd0 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0cb970 msgr2=0x7fba7c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0cb970 0x7fba7c06acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 >> v1:172.21.15.143:6789/0 conn(0x7fba7c158e20 legacy=0x7fba7c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c0cc690 con 0x7fba7c0cecd0 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fba740017f0 con 0x7fba7c0cecd0 2023-12-19T08:03:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 >> v1:172.21.15.111:6789/0 conn(0x7fba7c0cecd0 legacy=0x7fba7c06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba68003c20 0x7fba68004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba680051c0 0x7fba6800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0cc3b0 con 0x7fba68004810 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6800bfa0 con 0x7fba68003c20 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.132+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6800c0e0 con 0x7fba680051c0 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8120a700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba68003c20 0x7fba68004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba81a0b700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba680051c0 0x7fba6800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 -- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba680051c0 msgr2=0x7fba6800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba680051c0 0x7fba6800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 -- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba68003c20 msgr2=0x7fba68004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba68003c20 0x7fba68004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 -- 172.21.15.111:0/4122531928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68003570 con 0x7fba68004810 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba81a0b700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba680051c0 0x7fba6800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba80a09700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba68004c80 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009c30 tx=0x7fba6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c0b04fefed540b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c00e710 con 0x7fba68004810 2023-12-19T08:03:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c00e870 con 0x7fba68004810 2023-12-19T08:03:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba73fff700 1 -- 172.21.15.111:0/4122531928 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c00ec30 con 0x7fba68004810 2023-12-19T08:03:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 -- 172.21.15.111:0/4122531928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 msgr2=0x7fba68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 --2- 172.21.15.111:0/4122531928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba68004c80 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009c30 tx=0x7fba6c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba81a0b700 1 -- 172.21.15.111:0/4122531928 reap_dead start 2023-12-19T08:03:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 -- 172.21.15.111:0/4122531928 shutdown_connections 2023-12-19T08:03:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 -- 172.21.15.111:0/4122531928 >> 172.21.15.111:0/4122531928 conn(0x7fba7c067bb0 msgr2=0x7fba7c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 -- 172.21.15.111:0/4122531928 shutdown_connections 2023-12-19T08:03:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.136+0000 7fba8346e700 1 -- 172.21.15.111:0/4122531928 wait complete. 2023-12-19T08:03:33.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.140+0000 7fba8346e700 1 Processor -- start 2023-12-19T08:03:33.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.140+0000 7fba8346e700 1 -- start start 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0b59e0 0x7fba7c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba7c15e300 0x7fba7c0c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0c7790 con 0x7fba68004810 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c150340 con 0x7fba7c15e300 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c15e830 con 0x7fba7c0b59e0 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45286/0 (socket says 172.21.15.111:45286) 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 -- 172.21.15.111:0/3540100295 learned_addr learned my addr 172.21.15.111:0/3540100295 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:33.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba81a0b700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba7c15e300 0x7fba7c0c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba80a09700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0b59e0 0x7fba7c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 -- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0b59e0 msgr2=0x7fba7c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0b59e0 0x7fba7c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 -- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba7c15e300 msgr2=0x7fba7c0c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba7c15e300 0x7fba7c0c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 -- 172.21.15.111:0/3540100295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c15eb10 con 0x7fba68004810 2023-12-19T08:03:33.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8120a700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fba78002c00 tx=0x7fba78007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:33.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800f8d0 con 0x7fba68004810 2023-12-19T08:03:33.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800fa30 con 0x7fba68004810 2023-12-19T08:03:33.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c15edf0 con 0x7fba68004810 2023-12-19T08:03:33.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800fdf0 con 0x7fba68004810 2023-12-19T08:03:33.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.144+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c0c9940 con 0x7fba68004810 2023-12-19T08:03:33.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.148+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c071120 con 0x7fba68004810 2023-12-19T08:03:33.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.148+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fba78006a30 con 0x7fba68004810 2023-12-19T08:03:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.152+0000 7fba717fa700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 0x7fba60062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.152+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fba7807d6a0 con 0x7fba68004810 2023-12-19T08:03:33.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.152+0000 7fba80a09700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 0x7fba60062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:33.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.152+0000 7fba80a09700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 0x7fba60062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0091e0 tx=0x7fba6c01a000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:33.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.152+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fba7804d920 con 0x7fba68004810 2023-12-19T08:03:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.356+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fba7c06a7a0 con 0x7fba68004810 2023-12-19T08:03:33.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.928+0000 7fba717fa700 1 -- 172.21.15.111:0/3540100295 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v44) v1 ==== 174+0+0 (secure 0 0 0) 0x7fba78050f40 con 0x7fba68004810 2023-12-19T08:03:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 msgr2=0x7fba60062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 0x7fba60062b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0091e0 tx=0x7fba6c01a000 comp rx=0 tx=0).stop 2023-12-19T08:03:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 msgr2=0x7fba7c14fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fba78002c00 tx=0x7fba78007d30 comp rx=0 tx=0).stop 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 shutdown_connections 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba600606a0 0x7fba60062b50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba7c15e300 0x7fba7c0c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba7c0b59e0 0x7fba7c15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 --2- 172.21.15.111:0/3540100295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba68004810 0x7fba7c14fe00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.932+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 >> 172.21.15.111:0/3540100295 conn(0x7fba7c067bb0 msgr2=0x7fba7c0c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.936+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 shutdown_connections 2023-12-19T08:03:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:33.936+0000 7fba8346e700 1 -- 172.21.15.111:0/3540100295 wait complete. 2023-12-19T08:03:33.941 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 13 and data pool 14 2023-12-19T08:03:33.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:34.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.116+0000 7f817966e700 1 Processor -- start 2023-12-19T08:03:34.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.116+0000 7f817966e700 1 -- start start 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.116+0000 7f817966e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8174067c90 0x7f8174068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.116+0000 7f817966e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f817414fba0 0x7f8174150000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740c6630 con 0x7f81740c5900 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740c6910 con 0x7f8174067c90 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740c6bf0 con 0x7f817414fba0 2023-12-19T08:03:34.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8174067c90 0x7f8174068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8174067c90 0x7f8174068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45292/0 (socket says 172.21.15.111:45292) 2023-12-19T08:03:34.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 -- 172.21.15.111:0/1038971987 learned_addr learned my addr 172.21.15.111:0/1038971987 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 -- 172.21.15.111:0/1038971987 >> v2:172.21.15.143:3300/0 conn(0x7f817414fba0 msgr2=0x7f8174150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 --2- 172.21.15.111:0/1038971987 >> v2:172.21.15.143:3300/0 conn(0x7f817414fba0 0x7f8174150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 -- 172.21.15.111:0/1038971987 >> v1:172.21.15.143:6789/0 conn(0x7f81740c5900 legacy=0x7f81740c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 -- 172.21.15.111:0/1038971987 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81740c6ed0 con 0x7f8174067c90 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f8172ffd700 1 --2- 172.21.15.111:0/1038971987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8174067c90 0x7f8174068060 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f81740c56b0 tx=0x7f8164009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2113c89caf60b7f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f81727fc700 1 -- 172.21.15.111:0/1038971987 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816400b820 con 0x7f8174067c90 2023-12-19T08:03:34.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f81727fc700 1 -- 172.21.15.111:0/1038971987 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816400b980 con 0x7f8174067c90 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f81727fc700 1 -- 172.21.15.111:0/1038971987 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816400bd20 con 0x7f8174067c90 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- 172.21.15.111:0/1038971987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8174067c90 msgr2=0x7f8174068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 --2- 172.21.15.111:0/1038971987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8174067c90 0x7f8174068060 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f81740c56b0 tx=0x7f8164009250 comp rx=0 tx=0).stop 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- 172.21.15.111:0/1038971987 shutdown_connections 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 --2- 172.21.15.111:0/1038971987 >> v2:172.21.15.143:3300/0 conn(0x7f817414fba0 0x7f8174150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 --2- 172.21.15.111:0/1038971987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8174067c90 0x7f8174068060 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- 172.21.15.111:0/1038971987 >> 172.21.15.111:0/1038971987 conn(0x7f81740cd6d0 msgr2=0x7f817406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:34.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- 172.21.15.111:0/1038971987 shutdown_connections 2023-12-19T08:03:34.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.120+0000 7f817966e700 1 -- 172.21.15.111:0/1038971987 wait complete. 2023-12-19T08:03:34.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.124+0000 7f817966e700 1 Processor -- start 2023-12-19T08:03:34.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.124+0000 7f817966e700 1 -- start start 2023-12-19T08:03:34.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8174067c90 0x7f8174169df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81740c5900 0x7f817414e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81741523a0 con 0x7f817414fba0 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8174152830 con 0x7f81740c5900 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817416a330 con 0x7f8174067c90 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45306/0 (socket says 172.21.15.111:45306) 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 -- 172.21.15.111:0/3873804675 learned_addr learned my addr 172.21.15.111:0/3873804675 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f816bfff700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81740c5900 0x7f817414e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f8172ffd700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8174067c90 0x7f8174169df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 -- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8174067c90 msgr2=0x7f8174169df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8174067c90 0x7f8174169df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 -- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81740c5900 msgr2=0x7f817414e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81740c5900 0x7f817414e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 -- 172.21.15.111:0/3873804675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81640094f0 con 0x7f817414fba0 2023-12-19T08:03:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f81737fe700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f816c00bbb0 tx=0x7f816c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c0092e0 con 0x7f817414fba0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c00c6f0 con 0x7f817414fba0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c00ca90 con 0x7f817414fba0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8174150cb0 con 0x7f817414fba0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.128+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8174151a70 con 0x7f817414fba0 2023-12-19T08:03:34.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.132+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8174152550 con 0x7f817414fba0 2023-12-19T08:03:34.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.132+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f816c012070 con 0x7f817414fba0 2023-12-19T08:03:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.132+0000 7f816affd700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 0x7f8154062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.132+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f816c07c990 con 0x7f817414fba0 2023-12-19T08:03:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.136+0000 7f8172ffd700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 0x7f8154062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.136+0000 7f8172ffd700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 0x7f8154062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81740c56b0 tx=0x7f8164016000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.140+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8174152550 con 0x7f817414fba0 2023-12-19T08:03:34.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.312+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8174071120 con 0x7f817414fba0 2023-12-19T08:03:34.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.312+0000 7f816affd700 1 -- 172.21.15.111:0/3873804675 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16419 (secure 0 0 0) 0x7f816c04ca80 con 0x7f817414fba0 2023-12-19T08:03:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:34.318 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":112,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:33.363166+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 msgr2=0x7f8154062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 0x7f8154062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81740c56b0 tx=0x7f8164016000 comp rx=0 tx=0).stop 2023-12-19T08:03:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 msgr2=0x7f81740c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f816c00bbb0 tx=0x7f816c009840 comp rx=0 tx=0).stop 2023-12-19T08:03:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 shutdown_connections 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8154060670 0x7f8154062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81740c5900 0x7f817414e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f817414fba0 0x7f81740c72a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 --2- 172.21.15.111:0/3873804675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8174067c90 0x7f8174169df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 >> 172.21.15.111:0/3873804675 conn(0x7f81740cd6d0 msgr2=0x7f817406a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 shutdown_connections 2023-12-19T08:03:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.316+0000 7f817966e700 1 -- 172.21.15.111:0/3873804675 wait complete. 2023-12-19T08:03:34.341 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:34.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.496+0000 7fbe6b545700 1 Processor -- start 2023-12-19T08:03:34.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.496+0000 7fbe6b545700 1 -- start start 2023-12-19T08:03:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbe641502b0 0x7fbe64068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe640c5900 0x7fbe640c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe640c6630 con 0x7fbe64067c90 2023-12-19T08:03:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640c6910 con 0x7fbe640c5900 2023-12-19T08:03:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640c6bf0 con 0x7fbe641502b0 2023-12-19T08:03:34.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe640c5900 0x7fbe640c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe640c5900 0x7fbe640c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45314/0 (socket says 172.21.15.111:45314) 2023-12-19T08:03:34.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 -- 172.21.15.111:0/594589661 learned_addr learned my addr 172.21.15.111:0/594589661 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe68ae0700 1 --2- 172.21.15.111:0/594589661 >> v2:172.21.15.143:3300/0 conn(0x7fbe641502b0 0x7fbe64068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 -- 172.21.15.111:0/594589661 >> v2:172.21.15.143:3300/0 conn(0x7fbe641502b0 msgr2=0x7fbe64068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 --2- 172.21.15.111:0/594589661 >> v2:172.21.15.143:3300/0 conn(0x7fbe641502b0 0x7fbe64068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 -- 172.21.15.111:0/594589661 >> v1:172.21.15.143:6789/0 conn(0x7fbe64067c90 legacy=0x7fbe6414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 -- 172.21.15.111:0/594589661 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640c6ed0 con 0x7fbe640c5900 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe69ae2700 1 --2- 172.21.15.111:0/594589661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe640c5900 0x7fbe640c5d60 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fbe640c56b0 tx=0x7fbe5c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4141b2f67b93284a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe5bfff700 1 -- 172.21.15.111:0/594589661 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00e070 con 0x7fbe640c5900 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe5bfff700 1 -- 172.21.15.111:0/594589661 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c00ba50 con 0x7fbe640c5900 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe5bfff700 1 -- 172.21.15.111:0/594589661 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0092e0 con 0x7fbe640c5900 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- 172.21.15.111:0/594589661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe640c5900 msgr2=0x7fbe640c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 --2- 172.21.15.111:0/594589661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe640c5900 0x7fbe640c5d60 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fbe640c56b0 tx=0x7fbe5c009840 comp rx=0 tx=0).stop 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- 172.21.15.111:0/594589661 shutdown_connections 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 --2- 172.21.15.111:0/594589661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe640c5900 0x7fbe640c5d60 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 --2- 172.21.15.111:0/594589661 >> v2:172.21.15.143:3300/0 conn(0x7fbe641502b0 0x7fbe64068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- 172.21.15.111:0/594589661 >> 172.21.15.111:0/594589661 conn(0x7fbe640cd6d0 msgr2=0x7fbe6406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:34.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- 172.21.15.111:0/594589661 shutdown_connections 2023-12-19T08:03:34.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.500+0000 7fbe6b545700 1 -- 172.21.15.111:0/594589661 wait complete. 2023-12-19T08:03:34.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 Processor -- start 2023-12-19T08:03:34.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 -- start start 2023-12-19T08:03:34.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe64067c90 0x7fbe64169d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe640c5900 0x7fbe640b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c72a0 con 0x7fbe641502b0 2023-12-19T08:03:34.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c8100 con 0x7fbe640c5900 2023-12-19T08:03:34.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe6b545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6416a2c0 con 0x7fbe64067c90 2023-12-19T08:03:34.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe69ae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.504+0000 7fbe69ae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45328/0 (socket says 172.21.15.111:45328) 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 -- 172.21.15.111:0/3426769667 learned_addr learned my addr 172.21.15.111:0/3426769667 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe68ae0700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe640c5900 0x7fbe640b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe692e1700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe64067c90 0x7fbe64169d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 -- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe64067c90 msgr2=0x7fbe64169d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe64067c90 0x7fbe64169d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 -- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe640c5900 msgr2=0x7fbe640b5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe640c5900 0x7fbe640b5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 -- 172.21.15.111:0/3426769667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c009ae0 con 0x7fbe641502b0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe69ae2700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fbe6406a0f0 tx=0x7fbe5c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00be60 con 0x7fbe641502b0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c008d20 con 0x7fbe641502b0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640c74f0 con 0x7fbe641502b0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0266b0 con 0x7fbe641502b0 2023-12-19T08:03:34.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe640c7ab0 con 0x7fbe641502b0 2023-12-19T08:03:34.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.508+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe640c8270 con 0x7fbe641502b0 2023-12-19T08:03:34.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.512+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbe5c003040 con 0x7fbe641502b0 2023-12-19T08:03:34.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.512+0000 7fbe597fa700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 0x7fbe48062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.512+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fbe5c07ecf0 con 0x7fbe641502b0 2023-12-19T08:03:34.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.512+0000 7fbe692e1700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 0x7fbe48062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.512+0000 7fbe692e1700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 0x7fbe48062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe60003a80 tx=0x7fbe60007ec0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.520+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbe5c04ecd0 con 0x7fbe641502b0 2023-12-19T08:03:34.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.692+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe64071120 con 0x7fbe641502b0 2023-12-19T08:03:34.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe597fa700 1 -- 172.21.15.111:0/3426769667 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16406 (secure 0 0 0) 0x7fbe5c021090 con 0x7fbe641502b0 2023-12-19T08:03:34.699 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:34.699 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":113,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:34.458790+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 msgr2=0x7fbe48062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 0x7fbe48062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe60003a80 tx=0x7fbe60007ec0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 msgr2=0x7fbe640b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fbe6406a0f0 tx=0x7fbe5c018000 comp rx=0 tx=0).stop 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 shutdown_connections 2023-12-19T08:03:34.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe48060670 0x7fbe48062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe640c5900 0x7fbe640b5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe641502b0 0x7fbe640b5640 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 --2- 172.21.15.111:0/3426769667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe64067c90 0x7fbe64169d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 >> 172.21.15.111:0/3426769667 conn(0x7fbe640cd6d0 msgr2=0x7fbe640c5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 shutdown_connections 2023-12-19T08:03:34.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.696+0000 7fbe6b545700 1 -- 172.21.15.111:0/3426769667 wait complete. 2023-12-19T08:03:34.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:03:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.888+0000 7f68bdb85700 1 Processor -- start 2023-12-19T08:03:34.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.888+0000 7f68bdb85700 1 -- start start 2023-12-19T08:03:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68bdb85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 0x7f68b80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68bdb85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68b80ca2c0 0x7f68b80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68bdb85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b80cbdf0 con 0x7f68b80cb970 2023-12-19T08:03:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68bdb85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80cc0d0 con 0x7f68b80ca2c0 2023-12-19T08:03:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68bdb85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80cc3b0 con 0x7f68b8158e20 2023-12-19T08:03:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 0x7f68b80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68b80ca2c0 0x7f68b80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 0x7f68b80ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:51636/0 (socket says 172.21.15.111:51636) 2023-12-19T08:03:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b77fe700 1 -- 172.21.15.111:0/891294819 learned_addr learned my addr 172.21.15.111:0/891294819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b6ffd700 1 --1- 172.21.15.111:0/891294819 >> v1:172.21.15.143:6789/0 conn(0x7f68b80cb970 0x7f68b806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36842/0 (socket says 172.21.15.111:36842) 2023-12-19T08:03:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b7fff700 1 -- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 msgr2=0x7f68b80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b7fff700 1 --2- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 0x7f68b80ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.892+0000 7f68b7fff700 1 -- 172.21.15.111:0/891294819 >> v1:172.21.15.143:6789/0 conn(0x7f68b80cb970 legacy=0x7f68b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68b7fff700 1 -- 172.21.15.111:0/891294819 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80cc690 con 0x7f68b80ca2c0 2023-12-19T08:03:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68b7fff700 1 --2- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b80ca2c0 0x7f68b80ceb00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009e60 tx=0x7f68b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d08c836b2fe12456 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68b67fc700 1 -- 172.21.15.111:0/891294819 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b000e070 con 0x7f68b80ca2c0 2023-12-19T08:03:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68b67fc700 1 -- 172.21.15.111:0/891294819 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b000bb60 con 0x7f68b80ca2c0 2023-12-19T08:03:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68b67fc700 1 -- 172.21.15.111:0/891294819 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0012430 con 0x7f68b80ca2c0 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 -- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b80ca2c0 msgr2=0x7f68b80ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 --2- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b80ca2c0 0x7f68b80ceb00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009e60 tx=0x7f68b0009780 comp rx=0 tx=0).stop 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 -- 172.21.15.111:0/891294819 shutdown_connections 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 --2- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b80ca2c0 0x7f68b80ceb00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 --2- 172.21.15.111:0/891294819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b8158e20 0x7f68b80ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 -- 172.21.15.111:0/891294819 >> 172.21.15.111:0/891294819 conn(0x7f68b8067bb0 msgr2=0x7f68b80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 -- 172.21.15.111:0/891294819 shutdown_connections 2023-12-19T08:03:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.896+0000 7f68bdb85700 1 -- 172.21.15.111:0/891294819 wait complete. 2023-12-19T08:03:34.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 Processor -- start 2023-12-19T08:03:34.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 -- start start 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b80ca2c0 0x7f68b816e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 0x7f68b815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b816e430 con 0x7f68b80b4f60 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80b5a90 con 0x7f68b8158e20 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68bdb85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80b5d70 con 0x7f68b80ca2c0 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45338/0 (socket says 172.21.15.111:45338) 2023-12-19T08:03:34.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 -- 172.21.15.111:0/2417322492 learned_addr learned my addr 172.21.15.111:0/2417322492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:34.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b6ffd700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 0x7f68b815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b77fe700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b80ca2c0 0x7f68b816e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 -- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b80ca2c0 msgr2=0x7f68b816e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b80ca2c0 0x7f68b816e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 -- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 msgr2=0x7f68b815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 0x7f68b815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.900+0000 7f68b7fff700 1 -- 172.21.15.111:0/2417322492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0009a20 con 0x7f68b80b4f60 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f68b6ffd700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 0x7f68b815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f68b7fff700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f68b8069a80 tx=0x7f68b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0027470 con 0x7f68b80b4f60 2023-12-19T08:03:34.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b00275f0 con 0x7f68b80b4f60 2023-12-19T08:03:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b816dde0 con 0x7f68b80b4f60 2023-12-19T08:03:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b00279b0 con 0x7f68b80b4f60 2023-12-19T08:03:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b815f4e0 con 0x7f68b80b4f60 2023-12-19T08:03:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.904+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8071120 con 0x7f68b80b4f60 2023-12-19T08:03:34.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.908+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f68b0024070 con 0x7f68b80b4f60 2023-12-19T08:03:34.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.908+0000 7f689f7fe700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 0x7f68a4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:34.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.908+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f68b007f520 con 0x7f68b80b4f60 2023-12-19T08:03:34.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.908+0000 7f68b77fe700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 0x7f68a4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:34.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.908+0000 7f68b77fe700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 0x7f68a4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68b8156590 tx=0x7f68a8009250 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:34.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:34.916+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f68b004f6b0 con 0x7f68b80b4f60 2023-12-19T08:03:35.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.124+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f68b806a7a0 con 0x7f68b80b4f60 2023-12-19T08:03:35.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.956+0000 7f689f7fe700 1 -- 172.21.15.111:0/2417322492 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v47) v1 ==== 118+0+0 (secure 0 0 0) 0x7f68b0052cd0 con 0x7f68b80b4f60 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 msgr2=0x7f68a4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 0x7f68a4062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68b8156590 tx=0x7f68a8009250 comp rx=0 tx=0).stop 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 msgr2=0x7f68b80b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f68b8069a80 tx=0x7f68b0019000 comp rx=0 tx=0).stop 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 shutdown_connections 2023-12-19T08:03:35.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f68a4060670 0x7f68a4062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68b8158e20 0x7f68b815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68b80b4f60 0x7f68b80b5400 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 --2- 172.21.15.111:0/2417322492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68b80ca2c0 0x7f68b816e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 >> 172.21.15.111:0/2417322492 conn(0x7f68b8067bb0 msgr2=0x7f68b806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 shutdown_connections 2023-12-19T08:03:35.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:35.960+0000 7f68bdb85700 1 -- 172.21.15.111:0/2417322492 wait complete. 2023-12-19T08:03:35.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:03:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.156+0000 7fc48528e700 1 Processor -- start 2023-12-19T08:03:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.156+0000 7fc48528e700 1 -- start start 2023-12-19T08:03:36.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc48528e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc480158c90 0x7fc4800caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc48528e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc48006aeb0 0x7fc480159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc48528e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48006c850 con 0x7fc4801595a0 2023-12-19T08:03:36.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc48528e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48006b470 con 0x7fc480158c90 2023-12-19T08:03:36.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc48528e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48006b750 con 0x7fc48006aeb0 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc480158c90 0x7fc4800caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc484a8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc4801595a0 0x7fc48015c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34338/0 (socket says 172.21.15.111:34338) 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc484a8d700 1 -- 172.21.15.111:0/1544243330 learned_addr learned my addr 172.21.15.111:0/1544243330 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47f7fe700 1 --2- 172.21.15.111:0/1544243330 >> v2:172.21.15.143:3300/0 conn(0x7fc48006aeb0 0x7fc480159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 -- 172.21.15.111:0/1544243330 >> v2:172.21.15.143:3300/0 conn(0x7fc48006aeb0 msgr2=0x7fc480159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 --2- 172.21.15.111:0/1544243330 >> v2:172.21.15.143:3300/0 conn(0x7fc48006aeb0 0x7fc480159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 -- 172.21.15.111:0/1544243330 >> v1:172.21.15.111:6789/0 conn(0x7fc4801595a0 legacy=0x7fc48015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 -- 172.21.15.111:0/1544243330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48006ba30 con 0x7fc480158c90 2023-12-19T08:03:36.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47effd700 1 -- 172.21.15.111:0/1544243330 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315790131 0 0) 0x7fc48006c850 con 0x7fc4801595a0 2023-12-19T08:03:36.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47ffff700 1 --2- 172.21.15.111:0/1544243330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc480158c90 0x7fc4800caf40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc480068000 tx=0x7fc4700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57108835207bde39 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:36.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47effd700 1 -- 172.21.15.111:0/1544243330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470011630 con 0x7fc480158c90 2023-12-19T08:03:36.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47effd700 1 -- 172.21.15.111:0/1544243330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470011790 con 0x7fc480158c90 2023-12-19T08:03:36.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.160+0000 7fc47effd700 1 -- 172.21.15.111:0/1544243330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470011b30 con 0x7fc480158c90 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 -- 172.21.15.111:0/1544243330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc480158c90 msgr2=0x7fc4800caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 --2- 172.21.15.111:0/1544243330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc480158c90 0x7fc4800caf40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc480068000 tx=0x7fc4700091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 -- 172.21.15.111:0/1544243330 shutdown_connections 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 --2- 172.21.15.111:0/1544243330 >> v2:172.21.15.143:3300/0 conn(0x7fc48006aeb0 0x7fc480159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 --2- 172.21.15.111:0/1544243330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc480158c90 0x7fc4800caf40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 -- 172.21.15.111:0/1544243330 >> 172.21.15.111:0/1544243330 conn(0x7fc480067bb0 msgr2=0x7fc4800cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 -- 172.21.15.111:0/1544243330 shutdown_connections 2023-12-19T08:03:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.164+0000 7fc48528e700 1 -- 172.21.15.111:0/1544243330 wait complete. 2023-12-19T08:03:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 Processor -- start 2023-12-19T08:03:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 -- start start 2023-12-19T08:03:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc480158c90 0x7fc480157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 0x7fc4800c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800c5a20 con 0x7fc48006aeb0 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800c5eb0 con 0x7fc4801595a0 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc48528e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800c4e10 con 0x7fc480158c90 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc47ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc47ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45354/0 (socket says 172.21.15.111:45354) 2023-12-19T08:03:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc47ffff700 1 -- 172.21.15.111:0/3759673302 learned_addr learned my addr 172.21.15.111:0/3759673302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc47f7fe700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc480158c90 0x7fc480157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.168+0000 7fc484a8d700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 0x7fc4800c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 -- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc480158c90 msgr2=0x7fc480157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc480158c90 0x7fc480157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 -- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 msgr2=0x7fc4800c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 0x7fc4800c6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 -- 172.21.15.111:0/3759673302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470009420 con 0x7fc48006aeb0 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc484a8d700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 0x7fc4800c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc47ffff700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc480069b70 tx=0x7fc470017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470011b10 con 0x7fc48006aeb0 2023-12-19T08:03:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4700233f0 con 0x7fc48006aeb0 2023-12-19T08:03:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470023790 con 0x7fc48006aeb0 2023-12-19T08:03:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4800c50f0 con 0x7fc48006aeb0 2023-12-19T08:03:36.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4800c56b0 con 0x7fc48006aeb0 2023-12-19T08:03:36.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.172+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48004be40 con 0x7fc48006aeb0 2023-12-19T08:03:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.176+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc470023b80 con 0x7fc48006aeb0 2023-12-19T08:03:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.176+0000 7fc467fff700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 0x7fc46c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.176+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fc47007dfd0 con 0x7fc48006aeb0 2023-12-19T08:03:36.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.176+0000 7fc47f7fe700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 0x7fc46c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:36.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.180+0000 7fc47f7fe700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 0x7fc46c062bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc4800c9ee0 tx=0x7fc4740091a0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:36.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.180+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc47004e1c0 con 0x7fc48006aeb0 2023-12-19T08:03:36.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.384+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc48006bd10 con 0x7fc48006aeb0 2023-12-19T08:03:36.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:03:36.401 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:03:36.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.956+0000 7fc467fff700 1 -- 172.21.15.111:0/3759673302 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc4700150c0 con 0x7fc48006aeb0 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 msgr2=0x7fc46c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 0x7fc46c062bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc4800c9ee0 tx=0x7fc4740091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 msgr2=0x7fc48014b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc480069b70 tx=0x7fc470017000 comp rx=0 tx=0).stop 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 shutdown_connections 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc46c060740 0x7fc46c062bf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4801595a0 0x7fc4800c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc480158c90 0x7fc480157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 --2- 172.21.15.111:0/3759673302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc48006aeb0 0x7fc48014b890 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:36.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 >> 172.21.15.111:0/3759673302 conn(0x7fc480067bb0 msgr2=0x7fc48006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:36.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 shutdown_connections 2023-12-19T08:03:36.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:36.960+0000 7fc48528e700 1 -- 172.21.15.111:0/3759673302 wait complete. 2023-12-19T08:03:36.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:37.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.144+0000 7fbad2c0c700 1 Processor -- start 2023-12-19T08:03:37.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.144+0000 7fbad2c0c700 1 -- start start 2023-12-19T08:03:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad2c0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 0x7fbacc0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad2c0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 0x7fbacc15af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad2c0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc06c7c0 con 0x7fbacc06a580 2023-12-19T08:03:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad2c0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbacc06cd80 con 0x7fbacc06a9d0 2023-12-19T08:03:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad2c0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc06b8f0 con 0x7fbacc0b1410 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 0x7fbacc0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad240b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 0x7fbacc15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1409700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbacc06a580 0x7fbacc15a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36846/0 (socket says 172.21.15.111:36846) 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1409700 1 -- 172.21.15.111:0/325311065 learned_addr learned my addr 172.21.15.111:0/325311065 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad0c08700 1 -- 172.21.15.111:0/325311065 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489419047 0 0) 0x7fbacc06c7c0 con 0x7fbacc06a580 2023-12-19T08:03:37.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad0c08700 1 -- 172.21.15.111:0/325311065 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003570 con 0x7fbacc06a580 2023-12-19T08:03:37.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 -- 172.21.15.111:0/325311065 >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 msgr2=0x7fbacc15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:37.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 --2- 172.21.15.111:0/325311065 >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 0x7fbacc15af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 -- 172.21.15.111:0/325311065 >> v1:172.21.15.143:6789/0 conn(0x7fbacc06a580 legacy=0x7fbacc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 -- 172.21.15.111:0/325311065 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0ac1e0 con 0x7fbacc0b1410 2023-12-19T08:03:37.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad240b700 1 --2- 172.21.15.111:0/325311065 >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 0x7fbacc15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.148+0000 7fbad1c0a700 1 --2- 172.21.15.111:0/325311065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 0x7fbacc0ab440 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbac80098e0 tx=0x7fbac80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e03369e91e129076 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad0c08700 1 -- 172.21.15.111:0/325311065 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8011630 con 0x7fbacc0b1410 2023-12-19T08:03:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad0c08700 1 -- 172.21.15.111:0/325311065 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8011790 con 0x7fbacc0b1410 2023-12-19T08:03:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad0c08700 1 -- 172.21.15.111:0/325311065 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8011b30 con 0x7fbacc0b1410 2023-12-19T08:03:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 -- 172.21.15.111:0/325311065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 msgr2=0x7fbacc0ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/325311065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 0x7fbacc0ab440 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbac80098e0 tx=0x7fbac80091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 -- 172.21.15.111:0/325311065 shutdown_connections 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/325311065 >> v2:172.21.15.143:3300/0 conn(0x7fbacc06a9d0 0x7fbacc15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/325311065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc0b1410 0x7fbacc0ab440 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 -- 172.21.15.111:0/325311065 >> 172.21.15.111:0/325311065 conn(0x7fbacc067bb0 msgr2=0x7fbacc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 -- 172.21.15.111:0/325311065 shutdown_connections 2023-12-19T08:03:37.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.152+0000 7fbad2c0c700 1 -- 172.21.15.111:0/325311065 wait complete. 2023-12-19T08:03:37.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 Processor -- start 2023-12-19T08:03:37.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 -- start start 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbacc06a580 0x7fbacc14a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc06a9d0 0x7fbacc0c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc159a70 con 0x7fbacc0b1410 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc158e80 con 0x7fbacc06a580 2023-12-19T08:03:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad2c0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc159140 con 0x7fbacc06a9d0 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad240b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad240b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45362/0 (socket says 172.21.15.111:45362) 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad240b700 1 -- 172.21.15.111:0/654784498 learned_addr learned my addr 172.21.15.111:0/654784498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad1409700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc06a9d0 0x7fbacc0c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.156+0000 7fbad1c0a700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbacc06a580 0x7fbacc14a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 -- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc06a9d0 msgr2=0x7fbacc0c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc06a9d0 0x7fbacc0c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 -- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbacc06a580 msgr2=0x7fbacc14a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbacc06a580 0x7fbacc14a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 -- 172.21.15.111:0/654784498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac4009a20 con 0x7fbacc0b1410 2023-12-19T08:03:37.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad240b700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009fc0 tx=0x7fbac400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4009d20 con 0x7fbacc0b1410 2023-12-19T08:03:37.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac400f040 con 0x7fbacc0b1410 2023-12-19T08:03:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac8009420 con 0x7fbacc0b1410 2023-12-19T08:03:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac400ce20 con 0x7fbacc0b1410 2023-12-19T08:03:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc159700 con 0x7fbacc0b1410 2023-12-19T08:03:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.160+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc04be40 con 0x7fbacc0b1410 2023-12-19T08:03:37.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.164+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbac4003020 con 0x7fbacc0b1410 2023-12-19T08:03:37.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.164+0000 7fbac1ffb700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 0x7fbab0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.164+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fbac407d6e0 con 0x7fbacc0b1410 2023-12-19T08:03:37.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.164+0000 7fbad1c0a700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 0x7fbab0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.168+0000 7fbad1c0a700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 0x7fbab0062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbac80091e0 tx=0x7fbac8017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.172+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbac404d6c0 con 0x7fbacc0b1410 2023-12-19T08:03:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.372+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbacc0ab220 con 0x7fbacc0b1410 2023-12-19T08:03:37.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.376+0000 7fbac1ffb700 1 -- 172.21.15.111:0/654784498 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7fbac4050ce0 con 0x7fbacc0b1410 2023-12-19T08:03:37.379 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:37.379 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 msgr2=0x7fbab0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 0x7fbab0062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbac80091e0 tx=0x7fbac8017000 comp rx=0 tx=0).stop 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 msgr2=0x7fbacc0c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009fc0 tx=0x7fbac400b040 comp rx=0 tx=0).stop 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 shutdown_connections 2023-12-19T08:03:37.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbab0060670 0x7fbab0062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbacc0b1410 0x7fbacc0c8560 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbacc06a9d0 0x7fbacc0c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 --2- 172.21.15.111:0/654784498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbacc06a580 0x7fbacc14a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 >> 172.21.15.111:0/654784498 conn(0x7fbacc067bb0 msgr2=0x7fbacc153f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 shutdown_connections 2023-12-19T08:03:37.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.380+0000 7fbad2c0c700 1 -- 172.21.15.111:0/654784498 wait complete. 2023-12-19T08:03:37.385 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:03:37.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:03:37.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.564+0000 7f39fa8e3700 1 Processor -- start 2023-12-19T08:03:37.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.568+0000 7f39fa8e3700 1 -- start start 2023-12-19T08:03:37.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39f41591f0 0x7f39f406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39f40ce810 0x7f39f40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f406b730 con 0x7f39f4158e20 2023-12-19T08:03:37.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f406ab20 con 0x7f39f40ce810 2023-12-19T08:03:37.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f406ae00 con 0x7f39f41591f0 2023-12-19T08:03:37.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39f40ce810 0x7f39f40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39f40ce810 0x7f39f40ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45372/0 (socket says 172.21.15.111:45372) 2023-12-19T08:03:37.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 -- 172.21.15.111:0/154030808 learned_addr learned my addr 172.21.15.111:0/154030808 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:37.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f98e1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39f4158e20 0x7f39f40ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34346/0 (socket says 172.21.15.111:34346) 2023-12-19T08:03:37.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f90e0700 1 --2- 172.21.15.111:0/154030808 >> v2:172.21.15.143:3300/0 conn(0x7f39f41591f0 0x7f39f406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 -- 172.21.15.111:0/154030808 >> v2:172.21.15.143:3300/0 conn(0x7f39f41591f0 msgr2=0x7f39f406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 --2- 172.21.15.111:0/154030808 >> v2:172.21.15.143:3300/0 conn(0x7f39f41591f0 0x7f39f406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 -- 172.21.15.111:0/154030808 >> v1:172.21.15.111:6789/0 conn(0x7f39f4158e20 legacy=0x7f39f40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 -- 172.21.15.111:0/154030808 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f406b0e0 con 0x7f39f40ce810 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f88df700 1 -- 172.21.15.111:0/154030808 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570443549 0 0) 0x7f39f406b730 con 0x7f39f4158e20 2023-12-19T08:03:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa0e2700 1 --2- 172.21.15.111:0/154030808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f40ce810 0x7f39f40ca950 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f39f40642c0 tx=0x7f39ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d59d67eb4ecfc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f88df700 1 -- 172.21.15.111:0/154030808 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec00e070 con 0x7f39f40ce810 2023-12-19T08:03:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f88df700 1 -- 172.21.15.111:0/154030808 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec00ba20 con 0x7f39f40ce810 2023-12-19T08:03:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39f88df700 1 -- 172.21.15.111:0/154030808 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0092e0 con 0x7f39f40ce810 2023-12-19T08:03:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 -- 172.21.15.111:0/154030808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f40ce810 msgr2=0x7f39f40ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.572+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/154030808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f40ce810 0x7f39f40ca950 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f39f40642c0 tx=0x7f39ec009840 comp rx=0 tx=0).stop 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 -- 172.21.15.111:0/154030808 shutdown_connections 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/154030808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f40ce810 0x7f39f40ca950 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/154030808 >> v2:172.21.15.143:3300/0 conn(0x7f39f41591f0 0x7f39f406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 -- 172.21.15.111:0/154030808 >> 172.21.15.111:0/154030808 conn(0x7f39f4067bb0 msgr2=0x7f39f40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 -- 172.21.15.111:0/154030808 shutdown_connections 2023-12-19T08:03:37.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 -- 172.21.15.111:0/154030808 wait complete. 2023-12-19T08:03:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 Processor -- start 2023-12-19T08:03:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.576+0000 7f39fa8e3700 1 -- start start 2023-12-19T08:03:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39f40ce810 0x7f39f415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39f4158e20 0x7f39f40c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40c8280 con 0x7f39f41591f0 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f415c680 con 0x7f39f4158e20 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f40b5030 con 0x7f39f40ce810 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45388/0 (socket says 172.21.15.111:45388) 2023-12-19T08:03:37.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 -- 172.21.15.111:0/3807852225 learned_addr learned my addr 172.21.15.111:0/3807852225 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39f98e1700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39f40ce810 0x7f39f415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39f90e0700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39f4158e20 0x7f39f40c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 -- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39f40ce810 msgr2=0x7f39f415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39f40ce810 0x7f39f415c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 -- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39f4158e20 msgr2=0x7f39f40c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39f4158e20 0x7f39f40c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 -- 172.21.15.111:0/3807852225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec009ae0 con 0x7f39f41591f0 2023-12-19T08:03:37.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa0e2700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f39ec009ab0 tx=0x7f39ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec008be0 con 0x7f39f41591f0 2023-12-19T08:03:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec008d60 con 0x7f39f41591f0 2023-12-19T08:03:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f4159c60 con 0x7f39f41591f0 2023-12-19T08:03:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0266b0 con 0x7f39f41591f0 2023-12-19T08:03:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.580+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f40c7cc0 con 0x7f39f41591f0 2023-12-19T08:03:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.584+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f404be40 con 0x7f39f41591f0 2023-12-19T08:03:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.584+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f39ec003040 con 0x7f39f41591f0 2023-12-19T08:03:37.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.584+0000 7f39e9ffb700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 0x7f39d8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.588+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f39ec07e810 con 0x7f39f41591f0 2023-12-19T08:03:37.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.588+0000 7f39f98e1700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 0x7f39d8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.588+0000 7f39f98e1700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 0x7f39d8062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39f00011e0 tx=0x7f39f0007bf0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.592+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39ec04e7f0 con 0x7f39f41591f0 2023-12-19T08:03:37.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.776+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39f40c8060 con 0x7f39f41591f0 2023-12-19T08:03:37.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.776+0000 7f39e9ffb700 1 -- 172.21.15.111:0/3807852225 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16406 (secure 0 0 0) 0x7f39ec076e60 con 0x7f39f41591f0 2023-12-19T08:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":113,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:03:34.458790+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:03:37.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 msgr2=0x7f39d8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 0x7f39d8062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39f00011e0 tx=0x7f39f0007bf0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 msgr2=0x7f39f40c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f39ec009ab0 tx=0x7f39ec018000 comp rx=0 tx=0).stop 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 shutdown_connections 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39d8060670 0x7f39d8062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39f4158e20 0x7f39f40c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39f41591f0 0x7f39f40c73b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 --2- 172.21.15.111:0/3807852225 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39f40ce810 0x7f39f415c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 >> 172.21.15.111:0/3807852225 conn(0x7f39f4067bb0 msgr2=0x7f39f40cbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 shutdown_connections 2023-12-19T08:03:37.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.784+0000 7f39fa8e3700 1 -- 172.21.15.111:0/3807852225 wait complete. 2023-12-19T08:03:37.807 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:03:37.808 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:03:37.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.968+0000 7f1cad4a5700 1 Processor -- start 2023-12-19T08:03:37.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.968+0000 7f1cad4a5700 1 -- start start 2023-12-19T08:03:37.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1cad4a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1ca8158c10 0x7f1ca8158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1cad4a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1ca806b7d0 0x7f1ca80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1cad4a5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca806ac30 con 0x7f1ca806de60 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1cad4a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca80b4590 con 0x7f1ca8158c10 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1cad4a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca80ccc80 con 0x7f1ca806b7d0 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca67fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1ca806de60 0x7f1ca806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34356/0 (socket says 172.21.15.111:34356) 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca67fc700 1 -- 172.21.15.111:0/2112048559 learned_addr learned my addr 172.21.15.111:0/2112048559 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/2112048559 >> v2:172.21.15.111:3300/0 conn(0x7f1ca8158c10 0x7f1ca8158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca77fe700 1 --2- 172.21.15.111:0/2112048559 >> v2:172.21.15.143:3300/0 conn(0x7f1ca806b7d0 0x7f1ca80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2112048559 >> v2:172.21.15.143:3300/0 conn(0x7f1ca806b7d0 msgr2=0x7f1ca80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:37.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/2112048559 >> v2:172.21.15.143:3300/0 conn(0x7f1ca806b7d0 0x7f1ca80cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2112048559 >> v1:172.21.15.111:6789/0 conn(0x7f1ca806de60 legacy=0x7f1ca806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2112048559 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca806d180 con 0x7f1ca8158c10 2023-12-19T08:03:37.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/2112048559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca8158c10 0x7f1ca8158fe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80b5e60 tx=0x7f1c98009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e700a3de9a0df8e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca5ffb700 1 -- 172.21.15.111:0/2112048559 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98005c10 con 0x7f1ca8158c10 2023-12-19T08:03:37.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca5ffb700 1 -- 172.21.15.111:0/2112048559 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98005d70 con 0x7f1ca8158c10 2023-12-19T08:03:37.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.972+0000 7f1ca5ffb700 1 -- 172.21.15.111:0/2112048559 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98006260 con 0x7f1ca8158c10 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 -- 172.21.15.111:0/2112048559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca8158c10 msgr2=0x7f1ca8158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/2112048559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca8158c10 0x7f1ca8158fe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80b5e60 tx=0x7f1c98009120 comp rx=0 tx=0).stop 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 -- 172.21.15.111:0/2112048559 shutdown_connections 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/2112048559 >> v2:172.21.15.143:3300/0 conn(0x7f1ca806b7d0 0x7f1ca80cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/2112048559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca8158c10 0x7f1ca8158fe0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 -- 172.21.15.111:0/2112048559 >> 172.21.15.111:0/2112048559 conn(0x7f1ca80d0920 msgr2=0x7f1ca80b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 -- 172.21.15.111:0/2112048559 shutdown_connections 2023-12-19T08:03:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.976+0000 7f1cad4a5700 1 -- 172.21.15.111:0/2112048559 wait complete. 2023-12-19T08:03:37.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 Processor -- start 2023-12-19T08:03:37.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 -- start start 2023-12-19T08:03:37.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca806b7d0 0x7f1ca816c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca8158c10 0x7f1ca81693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca8147e80 con 0x7f1ca806de60 2023-12-19T08:03:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca816be80 con 0x7f1ca806b7d0 2023-12-19T08:03:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca816b2b0 con 0x7f1ca8158c10 2023-12-19T08:03:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45400/0 (socket says 172.21.15.111:45400) 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 -- 172.21.15.111:0/3538912778 learned_addr learned my addr 172.21.15.111:0/3538912778 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca806b7d0 0x7f1ca816c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca77fe700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca8158c10 0x7f1ca81693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 -- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca8158c10 msgr2=0x7f1ca81693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca8158c10 0x7f1ca81693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 -- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca806b7d0 msgr2=0x7f1ca816c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca806b7d0 0x7f1ca816c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 -- 172.21.15.111:0/3538912778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980093c0 con 0x7f1ca806de60 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1ca67fc700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c7c0 tx=0x7f1c9c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c005b30 con 0x7f1ca806de60 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c005cb0 con 0x7f1ca806de60 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca816b550 con 0x7f1ca806de60 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.980+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c004a80 con 0x7f1ca806de60 2023-12-19T08:03:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.984+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca816bb10 con 0x7f1ca806de60 2023-12-19T08:03:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.984+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca804be40 con 0x7f1ca806de60 2023-12-19T08:03:37.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.984+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1c9c0070d0 con 0x7f1ca806de60 2023-12-19T08:03:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.988+0000 7f1c8effd700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 0x7f1c94062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.988+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f1c9c07bf40 con 0x7f1ca806de60 2023-12-19T08:03:37.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.988+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 0x7f1c94062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:37.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.988+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 0x7f1c94062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800a010 tx=0x7f1c98016000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:37.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:37.992+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1c9c04c0d0 con 0x7f1ca806de60 2023-12-19T08:03:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.160+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ca806bc80 con 0x7f1c94060670 2023-12-19T08:03:38.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.180+0000 7f1c8effd700 1 -- 172.21.15.111:0/3538912778 <== mgr.6389 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287221 (secure 0 0 0) 0x7f1ca806bc80 con 0x7f1c94060670 2023-12-19T08:03:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:38.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 msgr2=0x7f1c94062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 0x7f1c94062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800a010 tx=0x7f1c98016000 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 msgr2=0x7f1ca8147940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c7c0 tx=0x7f1c9c007f70 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 shutdown_connections 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1c94060670 0x7f1c94062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ca8158c10 0x7f1ca81693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ca806de60 0x7f1ca8147940 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 --2- 172.21.15.111:0/3538912778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ca806b7d0 0x7f1ca816c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 >> 172.21.15.111:0/3538912778 conn(0x7f1ca80d0920 msgr2=0x7f1ca80ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 shutdown_connections 2023-12-19T08:03:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.188+0000 7f1cad4a5700 1 -- 172.21.15.111:0/3538912778 wait complete. 2023-12-19T08:03:38.194 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:03:38.210 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:03:37.281558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443616,"kb_used_data":12064,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752531104,"statfs":{"total":773094113280,"available":770591850496,"internally_reserved":0,"allocated":12353536,"data_stored":6937329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.372655"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486516+0000","last_change":"2023-12-19T08:03:33.483526+0000","last_active":"2023-12-19T08:03:33.486516+0000","last_peered":"2023-12-19T08:03:33.486516+0000","last_clean":"2023-12-19T08:03:33.486516+0000","last_became_active":"2023-12-19T08:03:33.483260+0000","last_became_peered":"2023-12-19T08:03:33.483260+0000","last_unstale":"2023-12-19T08:03:33.486516+0000","last_undegraded":"2023-12-19T08:03:33.486516+0000","last_fullsized":"2023-12-19T08:03:33.486516+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:31:29.470742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497432+0000","last_change":"2023-12-19T08:03:33.494403+0000","last_active":"2023-12-19T08:03:33.497432+0000","last_peered":"2023-12-19T08:03:33.497432+0000","last_clean":"2023-12-19T08:03:33.497432+0000","last_became_active":"2023-12-19T08:03:33.493595+0000","last_became_peered":"2023-12-19T08:03:33.493595+0000","last_unstale":"2023-12-19T08:03:33.497432+0000","last_undegraded":"2023-12-19T08:03:33.497432+0000","last_fullsized":"2023-12-19T08:03:33.497432+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.477738+0000","last_change":"2023-12-19T08:03:33.476010+0000","last_active":"2023-12-19T08:03:33.477738+0000","last_peered":"2023-12-19T08:03:33.477738+0000","last_clean":"2023-12-19T08:03:33.477738+0000","last_became_active":"2023-12-19T08:03:33.475333+0000","last_became_peered":"2023-12-19T08:03:33.475333+0000","last_unstale":"2023-12-19T08:03:33.477738+0000","last_undegraded":"2023-12-19T08:03:33.477738+0000","last_fullsized":"2023-12-19T08:03:33.477738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:42:12.891064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.495594+0000","last_change":"2023-12-19T08:03:33.490631+0000","last_active":"2023-12-19T08:03:33.495594+0000","last_peered":"2023-12-19T08:03:33.495594+0000","last_clean":"2023-12-19T08:03:33.495594+0000","last_became_active":"2023-12-19T08:03:33.490351+0000","last_became_peered":"2023-12-19T08:03:33.490351+0000","last_unstale":"2023-12-19T08:03:33.495594+0000","last_undegraded":"2023-12-19T08:03:33.495594+0000","last_fullsized":"2023-12-19T08:03:33.495594+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497950+0000","last_change":"2023-12-19T08:03:33.493902+0000","last_active":"2023-12-19T08:03:33.497950+0000","last_peered":"2023-12-19T08:03:33.497950+0000","last_clean":"2023-12-19T08:03:33.497950+0000","last_became_active":"2023-12-19T08:03:33.491759+0000","last_became_peered":"2023-12-19T08:03:33.491759+0000","last_unstale":"2023-12-19T08:03:33.497950+0000","last_undegraded":"2023-12-19T08:03:33.497950+0000","last_fullsized":"2023-12-19T08:03:33.497950+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:52.769135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500253+0000","last_change":"2023-12-19T08:03:33.496646+0000","last_active":"2023-12-19T08:03:33.500253+0000","last_peered":"2023-12-19T08:03:33.500253+0000","last_clean":"2023-12-19T08:03:33.500253+0000","last_became_active":"2023-12-19T08:03:33.496380+0000","last_became_peered":"2023-12-19T08:03:33.496380+0000","last_unstale":"2023-12-19T08:03:33.500253+0000","last_undegraded":"2023-12-19T08:03:33.500253+0000","last_fullsized":"2023-12-19T08:03:33.500253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:27:36.213826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.488405+0000","last_change":"2023-12-19T08:03:33.486510+0000","last_active":"2023-12-19T08:03:33.488405+0000","last_peered":"2023-12-19T08:03:33.488405+0000","last_clean":"2023-12-19T08:03:33.488405+0000","last_became_active":"2023-12-19T08:03:33.486191+0000","last_became_peered":"2023-12-19T08:03:33.486191+0000","last_unstale":"2023-12-19T08:03:33.488405+0000","last_undegraded":"2023-12-19T08:03:33.488405+0000","last_fullsized":"2023-12-19T08:03:33.488405+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:23.472892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489090+0000","last_change":"2023-12-19T08:03:33.483741+0000","last_active":"2023-12-19T08:03:33.489090+0000","last_peered":"2023-12-19T08:03:33.489090+0000","last_clean":"2023-12-19T08:03:33.489090+0000","last_became_active":"2023-12-19T08:03:33.483444+0000","last_became_peered":"2023-12-19T08:03:33.483444+0000","last_unstale":"2023-12-19T08:03:33.489090+0000","last_undegraded":"2023-12-19T08:03:33.489090+0000","last_fullsized":"2023-12-19T08:03:33.489090+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:08.878219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494472+0000","last_change":"2023-12-19T08:03:33.491060+0000","last_active":"2023-12-19T08:03:33.494472+0000","last_peered":"2023-12-19T08:03:33.494472+0000","last_clean":"2023-12-19T08:03:33.494472+0000","last_became_active":"2023-12-19T08:03:33.490647+0000","last_became_peered":"2023-12-19T08:03:33.490647+0000","last_unstale":"2023-12-19T08:03:33.494472+0000","last_undegraded":"2023-12-19T08:03:33.494472+0000","last_fullsized":"2023-12-19T08:03:33.494472+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501231+0000","last_change":"2023-12-19T08:03:33.497094+0000","last_active":"2023-12-19T08:03:33.501231+0000","last_peered":"2023-12-19T08:03:33.501231+0000","last_clean":"2023-12-19T08:03:33.501231+0000","last_became_active":"2023-12-19T08:03:33.496734+0000","last_became_peered":"2023-12-19T08:03:33.496734+0000","last_unstale":"2023-12-19T08:03:33.501231+0000","last_undegraded":"2023-12-19T08:03:33.501231+0000","last_fullsized":"2023-12-19T08:03:33.501231+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:05:10.019107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492393+0000","last_change":"2023-12-19T08:03:33.488004+0000","last_active":"2023-12-19T08:03:33.492393+0000","last_peered":"2023-12-19T08:03:33.492393+0000","last_clean":"2023-12-19T08:03:33.492393+0000","last_became_active":"2023-12-19T08:03:33.487599+0000","last_became_peered":"2023-12-19T08:03:33.487599+0000","last_unstale":"2023-12-19T08:03:33.492393+0000","last_undegraded":"2023-12-19T08:03:33.492393+0000","last_fullsized":"2023-12-19T08:03:33.492393+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.493887+0000","last_change":"2023-12-19T08:03:33.485971+0000","last_active":"2023-12-19T08:03:33.493887+0000","last_peered":"2023-12-19T08:03:33.493887+0000","last_clean":"2023-12-19T08:03:33.493887+0000","last_became_active":"2023-12-19T08:03:33.485690+0000","last_became_peered":"2023-12-19T08:03:33.485690+0000","last_unstale":"2023-12-19T08:03:33.493887+0000","last_undegraded":"2023-12-19T08:03:33.493887+0000","last_fullsized":"2023-12-19T08:03:33.493887+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:41.763317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490153+0000","last_change":"2023-12-19T08:03:33.479400+0000","last_active":"2023-12-19T08:03:33.490153+0000","last_peered":"2023-12-19T08:03:33.490153+0000","last_clean":"2023-12-19T08:03:33.490153+0000","last_became_active":"2023-12-19T08:03:33.478895+0000","last_became_peered":"2023-12-19T08:03:33.478895+0000","last_unstale":"2023-12-19T08:03:33.490153+0000","last_undegraded":"2023-12-19T08:03:33.490153+0000","last_fullsized":"2023-12-19T08:03:33.490153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498294+0000","last_change":"2023-12-19T08:03:33.496111+0000","last_active":"2023-12-19T08:03:33.498294+0000","last_peered":"2023-12-19T08:03:33.498294+0000","last_clean":"2023-12-19T08:03:33.498294+0000","last_became_active":"2023-12-19T08:03:33.495834+0000","last_became_peered":"2023-12-19T08:03:33.495834+0000","last_unstale":"2023-12-19T08:03:33.498294+0000","last_undegraded":"2023-12-19T08:03:33.498294+0000","last_fullsized":"2023-12-19T08:03:33.498294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:02.622620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.496692+0000","last_change":"2023-12-19T08:03:33.495064+0000","last_active":"2023-12-19T08:03:33.496692+0000","last_peered":"2023-12-19T08:03:33.496692+0000","last_clean":"2023-12-19T08:03:33.496692+0000","last_became_active":"2023-12-19T08:03:33.494667+0000","last_became_peered":"2023-12-19T08:03:33.494667+0000","last_unstale":"2023-12-19T08:03:33.496692+0000","last_undegraded":"2023-12-19T08:03:33.496692+0000","last_fullsized":"2023-12-19T08:03:33.496692+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492322+0000","last_change":"2023-12-19T08:03:33.480488+0000","last_active":"2023-12-19T08:03:33.492322+0000","last_peered":"2023-12-19T08:03:33.492322+0000","last_clean":"2023-12-19T08:03:33.492322+0000","last_became_active":"2023-12-19T08:03:33.480184+0000","last_became_peered":"2023-12-19T08:03:33.480184+0000","last_unstale":"2023-12-19T08:03:33.492322+0000","last_undegraded":"2023-12-19T08:03:33.492322+0000","last_fullsized":"2023-12-19T08:03:33.492322+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:46.444257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499112+0000","last_change":"2023-12-19T08:03:33.488645+0000","last_active":"2023-12-19T08:03:33.499112+0000","last_peered":"2023-12-19T08:03:33.499112+0000","last_clean":"2023-12-19T08:03:33.499112+0000","last_became_active":"2023-12-19T08:03:33.487066+0000","last_became_peered":"2023-12-19T08:03:33.487066+0000","last_unstale":"2023-12-19T08:03:33.499112+0000","last_undegraded":"2023-12-19T08:03:33.499112+0000","last_fullsized":"2023-12-19T08:03:33.499112+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:08.647435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490125+0000","last_change":"2023-12-19T08:03:33.485107+0000","last_active":"2023-12-19T08:03:33.490125+0000","last_peered":"2023-12-19T08:03:33.490125+0000","last_clean":"2023-12-19T08:03:33.490125+0000","last_became_active":"2023-12-19T08:03:33.484406+0000","last_became_peered":"2023-12-19T08:03:33.484406+0000","last_unstale":"2023-12-19T08:03:33.490125+0000","last_undegraded":"2023-12-19T08:03:33.490125+0000","last_fullsized":"2023-12-19T08:03:33.490125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:16.739695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497961+0000","last_change":"2023-12-19T08:03:33.495729+0000","last_active":"2023-12-19T08:03:33.497961+0000","last_peered":"2023-12-19T08:03:33.497961+0000","last_clean":"2023-12-19T08:03:33.497961+0000","last_became_active":"2023-12-19T08:03:33.495399+0000","last_became_peered":"2023-12-19T08:03:33.495399+0000","last_unstale":"2023-12-19T08:03:33.497961+0000","last_undegraded":"2023-12-19T08:03:33.497961+0000","last_fullsized":"2023-12-19T08:03:33.497961+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490800+0000","last_change":"2023-12-19T08:03:33.480669+0000","last_active":"2023-12-19T08:03:33.490800+0000","last_peered":"2023-12-19T08:03:33.490800+0000","last_clean":"2023-12-19T08:03:33.490800+0000","last_became_active":"2023-12-19T08:03:33.478912+0000","last_became_peered":"2023-12-19T08:03:33.478912+0000","last_unstale":"2023-12-19T08:03:33.490800+0000","last_undegraded":"2023-12-19T08:03:33.490800+0000","last_fullsized":"2023-12-19T08:03:33.490800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.493657+0000","last_change":"2023-12-19T08:03:33.483556+0000","last_active":"2023-12-19T08:03:33.493657+0000","last_peered":"2023-12-19T08:03:33.493657+0000","last_clean":"2023-12-19T08:03:33.493657+0000","last_became_active":"2023-12-19T08:03:33.483280+0000","last_became_peered":"2023-12-19T08:03:33.483280+0000","last_unstale":"2023-12-19T08:03:33.493657+0000","last_undegraded":"2023-12-19T08:03:33.493657+0000","last_fullsized":"2023-12-19T08:03:33.493657+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498772+0000","last_change":"2023-12-19T08:03:33.497052+0000","last_active":"2023-12-19T08:03:33.498772+0000","last_peered":"2023-12-19T08:03:33.498772+0000","last_clean":"2023-12-19T08:03:33.498772+0000","last_became_active":"2023-12-19T08:03:33.496466+0000","last_became_peered":"2023-12-19T08:03:33.496466+0000","last_unstale":"2023-12-19T08:03:33.498772+0000","last_undegraded":"2023-12-19T08:03:33.498772+0000","last_fullsized":"2023-12-19T08:03:33.498772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:01.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502234+0000","last_change":"2023-12-19T08:03:33.501256+0000","last_active":"2023-12-19T08:03:33.502234+0000","last_peered":"2023-12-19T08:03:33.502234+0000","last_clean":"2023-12-19T08:03:33.502234+0000","last_became_active":"2023-12-19T08:03:33.500903+0000","last_became_peered":"2023-12-19T08:03:33.500903+0000","last_unstale":"2023-12-19T08:03:33.502234+0000","last_undegraded":"2023-12-19T08:03:33.502234+0000","last_fullsized":"2023-12-19T08:03:33.502234+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.485944+0000","last_change":"2023-12-19T08:03:33.482986+0000","last_active":"2023-12-19T08:03:33.485944+0000","last_peered":"2023-12-19T08:03:33.485944+0000","last_clean":"2023-12-19T08:03:33.485944+0000","last_became_active":"2023-12-19T08:03:33.482686+0000","last_became_peered":"2023-12-19T08:03:33.482686+0000","last_unstale":"2023-12-19T08:03:33.485944+0000","last_undegraded":"2023-12-19T08:03:33.485944+0000","last_fullsized":"2023-12-19T08:03:33.485944+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:00.428547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490864+0000","last_change":"2023-12-19T08:03:33.487930+0000","last_active":"2023-12-19T08:03:33.490864+0000","last_peered":"2023-12-19T08:03:33.490864+0000","last_clean":"2023-12-19T08:03:33.490864+0000","last_became_active":"2023-12-19T08:03:33.487635+0000","last_became_peered":"2023-12-19T08:03:33.487635+0000","last_unstale":"2023-12-19T08:03:33.490864+0000","last_undegraded":"2023-12-19T08:03:33.490864+0000","last_fullsized":"2023-12-19T08:03:33.490864+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497979+0000","last_change":"2023-12-19T08:03:33.495114+0000","last_active":"2023-12-19T08:03:33.497979+0000","last_peered":"2023-12-19T08:03:33.497979+0000","last_clean":"2023-12-19T08:03:33.497979+0000","last_became_active":"2023-12-19T08:03:33.494838+0000","last_became_peered":"2023-12-19T08:03:33.494838+0000","last_unstale":"2023-12-19T08:03:33.497979+0000","last_undegraded":"2023-12-19T08:03:33.497979+0000","last_fullsized":"2023-12-19T08:03:33.497979+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.496253+0000","last_change":"2023-12-19T08:03:33.491303+0000","last_active":"2023-12-19T08:03:33.496253+0000","last_peered":"2023-12-19T08:03:33.496253+0000","last_clean":"2023-12-19T08:03:33.496253+0000","last_became_active":"2023-12-19T08:03:33.490331+0000","last_became_peered":"2023-12-19T08:03:33.490331+0000","last_unstale":"2023-12-19T08:03:33.496253+0000","last_undegraded":"2023-12-19T08:03:33.496253+0000","last_fullsized":"2023-12-19T08:03:33.496253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:48.089629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490783+0000","last_change":"2023-12-19T08:03:33.479506+0000","last_active":"2023-12-19T08:03:33.490783+0000","last_peered":"2023-12-19T08:03:33.490783+0000","last_clean":"2023-12-19T08:03:33.490783+0000","last_became_active":"2023-12-19T08:03:33.479143+0000","last_became_peered":"2023-12-19T08:03:33.479143+0000","last_unstale":"2023-12-19T08:03:33.490783+0000","last_undegraded":"2023-12-19T08:03:33.490783+0000","last_fullsized":"2023-12-19T08:03:33.490783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501637+0000","last_change":"2023-12-19T08:03:33.501025+0000","last_active":"2023-12-19T08:03:33.501637+0000","last_peered":"2023-12-19T08:03:33.501637+0000","last_clean":"2023-12-19T08:03:33.501637+0000","last_became_active":"2023-12-19T08:03:33.500409+0000","last_became_peered":"2023-12-19T08:03:33.500409+0000","last_unstale":"2023-12-19T08:03:33.501637+0000","last_undegraded":"2023-12-19T08:03:33.501637+0000","last_fullsized":"2023-12-19T08:03:33.501637+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:59:20.172093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490359+0000","last_change":"2023-12-19T08:03:33.487426+0000","last_active":"2023-12-19T08:03:33.490359+0000","last_peered":"2023-12-19T08:03:33.490359+0000","last_clean":"2023-12-19T08:03:33.490359+0000","last_became_active":"2023-12-19T08:03:33.487097+0000","last_became_peered":"2023-12-19T08:03:33.487097+0000","last_unstale":"2023-12-19T08:03:33.490359+0000","last_undegraded":"2023-12-19T08:03:33.490359+0000","last_fullsized":"2023-12-19T08:03:33.490359+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497397+0000","last_change":"2023-12-19T08:03:33.495076+0000","last_active":"2023-12-19T08:03:33.497397+0000","last_peered":"2023-12-19T08:03:33.497397+0000","last_clean":"2023-12-19T08:03:33.497397+0000","last_became_active":"2023-12-19T08:03:33.494471+0000","last_became_peered":"2023-12-19T08:03:33.494471+0000","last_unstale":"2023-12-19T08:03:33.497397+0000","last_undegraded":"2023-12-19T08:03:33.497397+0000","last_fullsized":"2023-12-19T08:03:33.497397+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492288+0000","last_change":"2023-12-19T08:03:33.479076+0000","last_active":"2023-12-19T08:03:33.492288+0000","last_peered":"2023-12-19T08:03:33.492288+0000","last_clean":"2023-12-19T08:03:33.492288+0000","last_became_active":"2023-12-19T08:03:33.478792+0000","last_became_peered":"2023-12-19T08:03:33.478792+0000","last_unstale":"2023-12-19T08:03:33.492288+0000","last_undegraded":"2023-12-19T08:03:33.492288+0000","last_fullsized":"2023-12-19T08:03:33.492288+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:13:46.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498260+0000","last_change":"2023-12-19T08:03:33.487582+0000","last_active":"2023-12-19T08:03:33.498260+0000","last_peered":"2023-12-19T08:03:33.498260+0000","last_clean":"2023-12-19T08:03:33.498260+0000","last_became_active":"2023-12-19T08:03:33.487325+0000","last_became_peered":"2023-12-19T08:03:33.487325+0000","last_unstale":"2023-12-19T08:03:33.498260+0000","last_undegraded":"2023-12-19T08:03:33.498260+0000","last_fullsized":"2023-12-19T08:03:33.498260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:19:20.739839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"112'4","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956886+0000","last_change":"2023-12-19T08:03:32.957444+0000","last_active":"2023-12-19T08:03:33.956886+0000","last_peered":"2023-12-19T08:03:33.956886+0000","last_clean":"2023-12-19T08:03:33.956886+0000","last_became_active":"2023-12-19T08:03:32.956954+0000","last_became_peered":"2023-12-19T08:03:32.956954+0000","last_unstale":"2023-12-19T08:03:33.956886+0000","last_undegraded":"2023-12-19T08:03:33.956886+0000","last_fullsized":"2023-12-19T08:03:33.956886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:49.416066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500161+0000","last_change":"2023-12-19T08:03:33.492120+0000","last_active":"2023-12-19T08:03:33.500161+0000","last_peered":"2023-12-19T08:03:33.500161+0000","last_clean":"2023-12-19T08:03:33.500161+0000","last_became_active":"2023-12-19T08:03:33.491790+0000","last_became_peered":"2023-12-19T08:03:33.491790+0000","last_unstale":"2023-12-19T08:03:33.500161+0000","last_undegraded":"2023-12-19T08:03:33.500161+0000","last_fullsized":"2023-12-19T08:03:33.500161+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:06:27.601956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.952119+0000","last_change":"2023-12-19T08:03:32.952797+0000","last_active":"2023-12-19T08:03:33.952119+0000","last_peered":"2023-12-19T08:03:33.952119+0000","last_clean":"2023-12-19T08:03:33.952119+0000","last_became_active":"2023-12-19T08:03:32.951746+0000","last_became_peered":"2023-12-19T08:03:32.951746+0000","last_unstale":"2023-12-19T08:03:33.952119+0000","last_undegraded":"2023-12-19T08:03:33.952119+0000","last_fullsized":"2023-12-19T08:03:33.952119+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:54:48.344896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498557+0000","last_change":"2023-12-19T08:03:33.497519+0000","last_active":"2023-12-19T08:03:33.498557+0000","last_peered":"2023-12-19T08:03:33.498557+0000","last_clean":"2023-12-19T08:03:33.498557+0000","last_became_active":"2023-12-19T08:03:33.497078+0000","last_became_peered":"2023-12-19T08:03:33.497078+0000","last_unstale":"2023-12-19T08:03:33.498557+0000","last_undegraded":"2023-12-19T08:03:33.498557+0000","last_fullsized":"2023-12-19T08:03:33.498557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:02.622620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.954038+0000","last_change":"2023-12-19T08:03:32.955148+0000","last_active":"2023-12-19T08:03:33.954038+0000","last_peered":"2023-12-19T08:03:33.954038+0000","last_clean":"2023-12-19T08:03:33.954038+0000","last_became_active":"2023-12-19T08:03:32.954848+0000","last_became_peered":"2023-12-19T08:03:32.954848+0000","last_unstale":"2023-12-19T08:03:33.954038+0000","last_undegraded":"2023-12-19T08:03:33.954038+0000","last_fullsized":"2023-12-19T08:03:33.954038+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:07.861485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499094+0000","last_change":"2023-12-19T08:03:33.481278+0000","last_active":"2023-12-19T08:03:33.499094+0000","last_peered":"2023-12-19T08:03:33.499094+0000","last_clean":"2023-12-19T08:03:33.499094+0000","last_became_active":"2023-12-19T08:03:33.480989+0000","last_became_peered":"2023-12-19T08:03:33.480989+0000","last_unstale":"2023-12-19T08:03:33.499094+0000","last_undegraded":"2023-12-19T08:03:33.499094+0000","last_fullsized":"2023-12-19T08:03:33.499094+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:15:05.924454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.955220+0000","last_change":"2023-12-19T08:03:32.956243+0000","last_active":"2023-12-19T08:03:33.955220+0000","last_peered":"2023-12-19T08:03:33.955220+0000","last_clean":"2023-12-19T08:03:33.955220+0000","last_became_active":"2023-12-19T08:03:32.955865+0000","last_became_peered":"2023-12-19T08:03:32.955865+0000","last_unstale":"2023-12-19T08:03:33.955220+0000","last_undegraded":"2023-12-19T08:03:33.955220+0000","last_fullsized":"2023-12-19T08:03:33.955220+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.152351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489002+0000","last_change":"2023-12-19T08:03:33.479280+0000","last_active":"2023-12-19T08:03:33.489002+0000","last_peered":"2023-12-19T08:03:33.489002+0000","last_clean":"2023-12-19T08:03:33.489002+0000","last_became_active":"2023-12-19T08:03:33.478884+0000","last_became_peered":"2023-12-19T08:03:33.478884+0000","last_unstale":"2023-12-19T08:03:33.489002+0000","last_undegraded":"2023-12-19T08:03:33.489002+0000","last_fullsized":"2023-12-19T08:03:33.489002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.953764+0000","last_change":"2023-12-19T08:03:32.952536+0000","last_active":"2023-12-19T08:03:33.953764+0000","last_peered":"2023-12-19T08:03:33.953764+0000","last_clean":"2023-12-19T08:03:33.953764+0000","last_became_active":"2023-12-19T08:03:32.951133+0000","last_became_peered":"2023-12-19T08:03:32.951133+0000","last_unstale":"2023-12-19T08:03:33.953764+0000","last_undegraded":"2023-12-19T08:03:33.953764+0000","last_fullsized":"2023-12-19T08:03:33.953764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:02.590163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501966+0000","last_change":"2023-12-19T08:03:33.498372+0000","last_active":"2023-12-19T08:03:33.501966+0000","last_peered":"2023-12-19T08:03:33.501966+0000","last_clean":"2023-12-19T08:03:33.501966+0000","last_became_active":"2023-12-19T08:03:33.497657+0000","last_became_peered":"2023-12-19T08:03:33.497657+0000","last_unstale":"2023-12-19T08:03:33.501966+0000","last_undegraded":"2023-12-19T08:03:33.501966+0000","last_fullsized":"2023-12-19T08:03:33.501966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.947692+0000","last_change":"2023-12-19T08:03:32.947425+0000","last_active":"2023-12-19T08:03:33.947692+0000","last_peered":"2023-12-19T08:03:33.947692+0000","last_clean":"2023-12-19T08:03:33.947692+0000","last_became_active":"2023-12-19T08:03:32.947072+0000","last_became_peered":"2023-12-19T08:03:32.947072+0000","last_unstale":"2023-12-19T08:03:33.947692+0000","last_undegraded":"2023-12-19T08:03:33.947692+0000","last_fullsized":"2023-12-19T08:03:33.947692+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:54:48.344896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500270+0000","last_change":"2023-12-19T08:03:33.497118+0000","last_active":"2023-12-19T08:03:33.500270+0000","last_peered":"2023-12-19T08:03:33.500270+0000","last_clean":"2023-12-19T08:03:33.500270+0000","last_became_active":"2023-12-19T08:03:33.496816+0000","last_became_peered":"2023-12-19T08:03:33.496816+0000","last_unstale":"2023-12-19T08:03:33.500270+0000","last_undegraded":"2023-12-19T08:03:33.500270+0000","last_fullsized":"2023-12-19T08:03:33.500270+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:54.598683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.464292+0000","last_change":"2023-12-19T08:03:32.943454+0000","last_active":"2023-12-19T08:03:33.464292+0000","last_peered":"2023-12-19T08:03:33.464292+0000","last_clean":"2023-12-19T08:03:33.464292+0000","last_became_active":"2023-12-19T08:03:32.943171+0000","last_became_peered":"2023-12-19T08:03:32.943171+0000","last_unstale":"2023-12-19T08:03:33.464292+0000","last_undegraded":"2023-12-19T08:03:33.464292+0000","last_fullsized":"2023-12-19T08:03:33.464292+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:15:04.907719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490170+0000","last_change":"2023-12-19T08:03:33.487373+0000","last_active":"2023-12-19T08:03:33.490170+0000","last_peered":"2023-12-19T08:03:33.490170+0000","last_clean":"2023-12-19T08:03:33.490170+0000","last_became_active":"2023-12-19T08:03:33.486362+0000","last_became_peered":"2023-12-19T08:03:33.486362+0000","last_unstale":"2023-12-19T08:03:33.490170+0000","last_undegraded":"2023-12-19T08:03:33.490170+0000","last_fullsized":"2023-12-19T08:03:33.490170+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476403+0000","last_change":"2023-12-19T08:03:32.958981+0000","last_active":"2023-12-19T08:03:33.476403+0000","last_peered":"2023-12-19T08:03:33.476403+0000","last_clean":"2023-12-19T08:03:33.476403+0000","last_became_active":"2023-12-19T08:03:32.958411+0000","last_became_peered":"2023-12-19T08:03:32.958411+0000","last_unstale":"2023-12-19T08:03:33.476403+0000","last_undegraded":"2023-12-19T08:03:33.476403+0000","last_fullsized":"2023-12-19T08:03:33.476403+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:16.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.485111+0000","last_change":"2023-12-19T08:03:33.481237+0000","last_active":"2023-12-19T08:03:33.485111+0000","last_peered":"2023-12-19T08:03:33.485111+0000","last_clean":"2023-12-19T08:03:33.485111+0000","last_became_active":"2023-12-19T08:03:33.478910+0000","last_became_peered":"2023-12-19T08:03:33.478910+0000","last_unstale":"2023-12-19T08:03:33.485111+0000","last_undegraded":"2023-12-19T08:03:33.485111+0000","last_fullsized":"2023-12-19T08:03:33.485111+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:41:51.857662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.480934+0000","last_change":"2023-12-19T08:03:32.952096+0000","last_active":"2023-12-19T08:03:33.480934+0000","last_peered":"2023-12-19T08:03:33.480934+0000","last_clean":"2023-12-19T08:03:33.480934+0000","last_became_active":"2023-12-19T08:03:32.951815+0000","last_became_peered":"2023-12-19T08:03:32.951815+0000","last_unstale":"2023-12-19T08:03:33.480934+0000","last_undegraded":"2023-12-19T08:03:33.480934+0000","last_fullsized":"2023-12-19T08:03:33.480934+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.152351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.488423+0000","last_change":"2023-12-19T08:03:33.486532+0000","last_active":"2023-12-19T08:03:33.488423+0000","last_peered":"2023-12-19T08:03:33.488423+0000","last_clean":"2023-12-19T08:03:33.488423+0000","last_became_active":"2023-12-19T08:03:33.486254+0000","last_became_peered":"2023-12-19T08:03:33.486254+0000","last_unstale":"2023-12-19T08:03:33.488423+0000","last_undegraded":"2023-12-19T08:03:33.488423+0000","last_fullsized":"2023-12-19T08:03:33.488423+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:30:24.758243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.465131+0000","last_change":"2023-12-19T08:03:32.948461+0000","last_active":"2023-12-19T08:03:33.465131+0000","last_peered":"2023-12-19T08:03:33.465131+0000","last_clean":"2023-12-19T08:03:33.465131+0000","last_became_active":"2023-12-19T08:03:32.948131+0000","last_became_peered":"2023-12-19T08:03:32.948131+0000","last_unstale":"2023-12-19T08:03:33.465131+0000","last_undegraded":"2023-12-19T08:03:33.465131+0000","last_fullsized":"2023-12-19T08:03:33.465131+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:09.077379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.484623+0000","last_change":"2023-12-19T08:03:33.482184+0000","last_active":"2023-12-19T08:03:33.484623+0000","last_peered":"2023-12-19T08:03:33.484623+0000","last_clean":"2023-12-19T08:03:33.484623+0000","last_became_active":"2023-12-19T08:03:33.481907+0000","last_became_peered":"2023-12-19T08:03:33.481907+0000","last_unstale":"2023-12-19T08:03:33.484623+0000","last_undegraded":"2023-12-19T08:03:33.484623+0000","last_fullsized":"2023-12-19T08:03:33.484623+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:04.746187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"112'3","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.954987+0000","last_change":"2023-12-19T08:03:32.954273+0000","last_active":"2023-12-19T08:03:33.954987+0000","last_peered":"2023-12-19T08:03:33.954987+0000","last_clean":"2023-12-19T08:03:33.954987+0000","last_became_active":"2023-12-19T08:03:32.953922+0000","last_became_peered":"2023-12-19T08:03:32.953922+0000","last_unstale":"2023-12-19T08:03:33.954987+0000","last_undegraded":"2023-12-19T08:03:33.954987+0000","last_fullsized":"2023-12-19T08:03:33.954987+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:28.068462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489125+0000","last_change":"2023-12-19T08:03:33.486370+0000","last_active":"2023-12-19T08:03:33.489125+0000","last_peered":"2023-12-19T08:03:33.489125+0000","last_clean":"2023-12-19T08:03:33.489125+0000","last_became_active":"2023-12-19T08:03:33.486057+0000","last_became_peered":"2023-12-19T08:03:33.486057+0000","last_unstale":"2023-12-19T08:03:33.489125+0000","last_undegraded":"2023-12-19T08:03:33.489125+0000","last_fullsized":"2023-12-19T08:03:33.489125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:01.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.953940+0000","last_change":"2023-12-19T08:03:32.949662+0000","last_active":"2023-12-19T08:03:33.953940+0000","last_peered":"2023-12-19T08:03:33.953940+0000","last_clean":"2023-12-19T08:03:33.953940+0000","last_became_active":"2023-12-19T08:03:32.949247+0000","last_became_peered":"2023-12-19T08:03:32.949247+0000","last_unstale":"2023-12-19T08:03:33.953940+0000","last_undegraded":"2023-12-19T08:03:33.953940+0000","last_fullsized":"2023-12-19T08:03:33.953940+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:30.492951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489115+0000","last_change":"2023-12-19T08:03:33.484062+0000","last_active":"2023-12-19T08:03:33.489115+0000","last_peered":"2023-12-19T08:03:33.489115+0000","last_clean":"2023-12-19T08:03:33.489115+0000","last_became_active":"2023-12-19T08:03:33.483742+0000","last_became_peered":"2023-12-19T08:03:33.483742+0000","last_unstale":"2023-12-19T08:03:33.489115+0000","last_undegraded":"2023-12-19T08:03:33.489115+0000","last_fullsized":"2023-12-19T08:03:33.489115+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:44:15.395278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956617+0000","last_change":"2023-12-19T08:03:32.960033+0000","last_active":"2023-12-19T08:03:33.956617+0000","last_peered":"2023-12-19T08:03:33.956617+0000","last_clean":"2023-12-19T08:03:33.956617+0000","last_became_active":"2023-12-19T08:03:32.959695+0000","last_became_peered":"2023-12-19T08:03:32.959695+0000","last_unstale":"2023-12-19T08:03:33.956617+0000","last_undegraded":"2023-12-19T08:03:33.956617+0000","last_fullsized":"2023-12-19T08:03:33.956617+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:47.360353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494277+0000","last_change":"2023-12-19T08:03:33.490631+0000","last_active":"2023-12-19T08:03:33.494277+0000","last_peered":"2023-12-19T08:03:33.494277+0000","last_clean":"2023-12-19T08:03:33.494277+0000","last_became_active":"2023-12-19T08:03:33.490357+0000","last_became_peered":"2023-12-19T08:03:33.490357+0000","last_unstale":"2023-12-19T08:03:33.494277+0000","last_undegraded":"2023-12-19T08:03:33.494277+0000","last_fullsized":"2023-12-19T08:03:33.494277+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:52.314950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475238+0000","last_change":"2023-12-19T08:03:32.940732+0000","last_active":"2023-12-19T08:03:33.475238+0000","last_peered":"2023-12-19T08:03:33.475238+0000","last_clean":"2023-12-19T08:03:33.475238+0000","last_became_active":"2023-12-19T08:03:32.940308+0000","last_became_peered":"2023-12-19T08:03:32.940308+0000","last_unstale":"2023-12-19T08:03:33.475238+0000","last_undegraded":"2023-12-19T08:03:33.475238+0000","last_fullsized":"2023-12-19T08:03:33.475238+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:16.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501227+0000","last_change":"2023-12-19T08:03:33.497321+0000","last_active":"2023-12-19T08:03:33.501227+0000","last_peered":"2023-12-19T08:03:33.501227+0000","last_clean":"2023-12-19T08:03:33.501227+0000","last_became_active":"2023-12-19T08:03:33.496926+0000","last_became_peered":"2023-12-19T08:03:33.496926+0000","last_unstale":"2023-12-19T08:03:33.501227+0000","last_undegraded":"2023-12-19T08:03:33.501227+0000","last_fullsized":"2023-12-19T08:03:33.501227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:47:33.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.473052+0000","last_change":"2023-12-19T08:03:32.946645+0000","last_active":"2023-12-19T08:03:33.473052+0000","last_peered":"2023-12-19T08:03:33.473052+0000","last_clean":"2023-12-19T08:03:33.473052+0000","last_became_active":"2023-12-19T08:03:32.946315+0000","last_became_peered":"2023-12-19T08:03:32.946315+0000","last_unstale":"2023-12-19T08:03:33.473052+0000","last_undegraded":"2023-12-19T08:03:33.473052+0000","last_fullsized":"2023-12-19T08:03:33.473052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:16:08.683512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494721+0000","last_change":"2023-12-19T08:03:33.491992+0000","last_active":"2023-12-19T08:03:33.494721+0000","last_peered":"2023-12-19T08:03:33.494721+0000","last_clean":"2023-12-19T08:03:33.494721+0000","last_became_active":"2023-12-19T08:03:33.491708+0000","last_became_peered":"2023-12-19T08:03:33.491708+0000","last_unstale":"2023-12-19T08:03:33.494721+0000","last_undegraded":"2023-12-19T08:03:33.494721+0000","last_fullsized":"2023-12-19T08:03:33.494721+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:59.545609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.951374+0000","last_change":"2023-12-19T08:03:32.947262+0000","last_active":"2023-12-19T08:03:33.951374+0000","last_peered":"2023-12-19T08:03:33.951374+0000","last_clean":"2023-12-19T08:03:33.951374+0000","last_became_active":"2023-12-19T08:03:32.946936+0000","last_became_peered":"2023-12-19T08:03:32.946936+0000","last_unstale":"2023-12-19T08:03:33.951374+0000","last_undegraded":"2023-12-19T08:03:33.951374+0000","last_fullsized":"2023-12-19T08:03:33.951374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:28.068462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498240+0000","last_change":"2023-12-19T08:03:33.486454+0000","last_active":"2023-12-19T08:03:33.498240+0000","last_peered":"2023-12-19T08:03:33.498240+0000","last_clean":"2023-12-19T08:03:33.498240+0000","last_became_active":"2023-12-19T08:03:33.485613+0000","last_became_peered":"2023-12-19T08:03:33.485613+0000","last_unstale":"2023-12-19T08:03:33.498240+0000","last_undegraded":"2023-12-19T08:03:33.498240+0000","last_fullsized":"2023-12-19T08:03:33.498240+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:10.094114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956284+0000","last_change":"2023-12-19T08:03:32.960500+0000","last_active":"2023-12-19T08:03:33.956284+0000","last_peered":"2023-12-19T08:03:33.956284+0000","last_clean":"2023-12-19T08:03:33.956284+0000","last_became_active":"2023-12-19T08:03:32.959932+0000","last_became_peered":"2023-12-19T08:03:32.959932+0000","last_unstale":"2023-12-19T08:03:33.956284+0000","last_undegraded":"2023-12-19T08:03:33.956284+0000","last_fullsized":"2023-12-19T08:03:33.956284+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:56.567271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490799+0000","last_change":"2023-12-19T08:03:33.488521+0000","last_active":"2023-12-19T08:03:33.490799+0000","last_peered":"2023-12-19T08:03:33.490799+0000","last_clean":"2023-12-19T08:03:33.490799+0000","last_became_active":"2023-12-19T08:03:33.487699+0000","last_became_peered":"2023-12-19T08:03:33.487699+0000","last_unstale":"2023-12-19T08:03:33.490799+0000","last_undegraded":"2023-12-19T08:03:33.490799+0000","last_fullsized":"2023-12-19T08:03:33.490799+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:08.878219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956421+0000","last_change":"2023-12-19T08:03:32.956306+0000","last_active":"2023-12-19T08:03:33.956421+0000","last_peered":"2023-12-19T08:03:33.956421+0000","last_clean":"2023-12-19T08:03:33.956421+0000","last_became_active":"2023-12-19T08:03:32.955688+0000","last_became_peered":"2023-12-19T08:03:32.955688+0000","last_unstale":"2023-12-19T08:03:33.956421+0000","last_undegraded":"2023-12-19T08:03:33.956421+0000","last_fullsized":"2023-12-19T08:03:33.956421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:12.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498754+0000","last_change":"2023-12-19T08:03:33.497007+0000","last_active":"2023-12-19T08:03:33.498754+0000","last_peered":"2023-12-19T08:03:33.498754+0000","last_clean":"2023-12-19T08:03:33.498754+0000","last_became_active":"2023-12-19T08:03:33.496664+0000","last_became_peered":"2023-12-19T08:03:33.496664+0000","last_unstale":"2023-12-19T08:03:33.498754+0000","last_undegraded":"2023-12-19T08:03:33.498754+0000","last_fullsized":"2023-12-19T08:03:33.498754+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.955182+0000","last_change":"2023-12-19T08:03:32.956036+0000","last_active":"2023-12-19T08:03:33.955182+0000","last_peered":"2023-12-19T08:03:33.955182+0000","last_clean":"2023-12-19T08:03:33.955182+0000","last_became_active":"2023-12-19T08:03:32.953950+0000","last_became_peered":"2023-12-19T08:03:32.953950+0000","last_unstale":"2023-12-19T08:03:33.955182+0000","last_undegraded":"2023-12-19T08:03:33.955182+0000","last_fullsized":"2023-12-19T08:03:33.955182+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:38:54.022009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499051+0000","last_change":"2023-12-19T08:03:33.485647+0000","last_active":"2023-12-19T08:03:33.499051+0000","last_peered":"2023-12-19T08:03:33.499051+0000","last_clean":"2023-12-19T08:03:33.499051+0000","last_became_active":"2023-12-19T08:03:33.484717+0000","last_became_peered":"2023-12-19T08:03:33.484717+0000","last_unstale":"2023-12-19T08:03:33.499051+0000","last_undegraded":"2023-12-19T08:03:33.499051+0000","last_fullsized":"2023-12-19T08:03:33.499051+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:04:56.590204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956145+0000","last_change":"2023-12-19T08:03:32.947216+0000","last_active":"2023-12-19T08:03:33.956145+0000","last_peered":"2023-12-19T08:03:33.956145+0000","last_clean":"2023-12-19T08:03:33.956145+0000","last_became_active":"2023-12-19T08:03:32.946904+0000","last_became_peered":"2023-12-19T08:03:32.946904+0000","last_unstale":"2023-12-19T08:03:33.956145+0000","last_undegraded":"2023-12-19T08:03:33.956145+0000","last_fullsized":"2023-12-19T08:03:33.956145+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:03.209463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490820+0000","last_change":"2023-12-19T08:03:33.488004+0000","last_active":"2023-12-19T08:03:33.490820+0000","last_peered":"2023-12-19T08:03:33.490820+0000","last_clean":"2023-12-19T08:03:33.490820+0000","last_became_active":"2023-12-19T08:03:33.487110+0000","last_became_peered":"2023-12-19T08:03:33.487110+0000","last_unstale":"2023-12-19T08:03:33.490820+0000","last_undegraded":"2023-12-19T08:03:33.490820+0000","last_fullsized":"2023-12-19T08:03:33.490820+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486272+0000","last_change":"2023-12-19T08:03:32.951521+0000","last_active":"2023-12-19T08:03:33.486272+0000","last_peered":"2023-12-19T08:03:33.486272+0000","last_clean":"2023-12-19T08:03:33.486272+0000","last_became_active":"2023-12-19T08:03:32.951121+0000","last_became_peered":"2023-12-19T08:03:32.951121+0000","last_unstale":"2023-12-19T08:03:33.486272+0000","last_undegraded":"2023-12-19T08:03:33.486272+0000","last_fullsized":"2023-12-19T08:03:33.486272+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:10.361201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502086+0000","last_change":"2023-12-19T08:03:33.501554+0000","last_active":"2023-12-19T08:03:33.502086+0000","last_peered":"2023-12-19T08:03:33.502086+0000","last_clean":"2023-12-19T08:03:33.502086+0000","last_became_active":"2023-12-19T08:03:33.500854+0000","last_became_peered":"2023-12-19T08:03:33.500854+0000","last_unstale":"2023-12-19T08:03:33.502086+0000","last_undegraded":"2023-12-19T08:03:33.502086+0000","last_fullsized":"2023-12-19T08:03:33.502086+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.477634+0000","last_change":"2023-12-19T08:03:32.954269+0000","last_active":"2023-12-19T08:03:33.477634+0000","last_peered":"2023-12-19T08:03:33.477634+0000","last_clean":"2023-12-19T08:03:33.477634+0000","last_became_active":"2023-12-19T08:03:32.953991+0000","last_became_peered":"2023-12-19T08:03:32.953991+0000","last_unstale":"2023-12-19T08:03:33.477634+0000","last_undegraded":"2023-12-19T08:03:33.477634+0000","last_fullsized":"2023-12-19T08:03:33.477634+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:49.416066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500235+0000","last_change":"2023-12-19T08:03:33.495729+0000","last_active":"2023-12-19T08:03:33.500235+0000","last_peered":"2023-12-19T08:03:33.500235+0000","last_clean":"2023-12-19T08:03:33.500235+0000","last_became_active":"2023-12-19T08:03:33.495455+0000","last_became_peered":"2023-12-19T08:03:33.495455+0000","last_unstale":"2023-12-19T08:03:33.500235+0000","last_undegraded":"2023-12-19T08:03:33.500235+0000","last_fullsized":"2023-12-19T08:03:33.500235+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:45.551623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486312+0000","last_change":"2023-12-19T08:03:32.943826+0000","last_active":"2023-12-19T08:03:33.486312+0000","last_peered":"2023-12-19T08:03:33.486312+0000","last_clean":"2023-12-19T08:03:33.486312+0000","last_became_active":"2023-12-19T08:03:32.943509+0000","last_became_peered":"2023-12-19T08:03:32.943509+0000","last_unstale":"2023-12-19T08:03:33.486312+0000","last_undegraded":"2023-12-19T08:03:33.486312+0000","last_fullsized":"2023-12-19T08:03:33.486312+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:20.621987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490767+0000","last_change":"2023-12-19T08:03:33.488323+0000","last_active":"2023-12-19T08:03:33.490767+0000","last_peered":"2023-12-19T08:03:33.490767+0000","last_clean":"2023-12-19T08:03:33.490767+0000","last_became_active":"2023-12-19T08:03:33.488058+0000","last_became_peered":"2023-12-19T08:03:33.488058+0000","last_unstale":"2023-12-19T08:03:33.490767+0000","last_undegraded":"2023-12-19T08:03:33.490767+0000","last_fullsized":"2023-12-19T08:03:33.490767+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956833+0000","last_change":"2023-12-19T08:03:32.959707+0000","last_active":"2023-12-19T08:03:33.956833+0000","last_peered":"2023-12-19T08:03:33.956833+0000","last_clean":"2023-12-19T08:03:33.956833+0000","last_became_active":"2023-12-19T08:03:32.959376+0000","last_became_peered":"2023-12-19T08:03:32.959376+0000","last_unstale":"2023-12-19T08:03:33.956833+0000","last_undegraded":"2023-12-19T08:03:33.956833+0000","last_fullsized":"2023-12-19T08:03:33.956833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:44:14.378543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498150+0000","last_change":"2023-12-19T08:03:33.482782+0000","last_active":"2023-12-19T08:03:33.498150+0000","last_peered":"2023-12-19T08:03:33.498150+0000","last_clean":"2023-12-19T08:03:33.498150+0000","last_became_active":"2023-12-19T08:03:33.480963+0000","last_became_peered":"2023-12-19T08:03:33.480963+0000","last_unstale":"2023-12-19T08:03:33.498150+0000","last_undegraded":"2023-12-19T08:03:33.498150+0000","last_fullsized":"2023-12-19T08:03:33.498150+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:22:44.764760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475117+0000","last_change":"2023-12-19T08:03:32.947496+0000","last_active":"2023-12-19T08:03:33.475117+0000","last_peered":"2023-12-19T08:03:33.475117+0000","last_clean":"2023-12-19T08:03:33.475117+0000","last_became_active":"2023-12-19T08:03:32.947206+0000","last_became_peered":"2023-12-19T08:03:32.947206+0000","last_unstale":"2023-12-19T08:03:33.475117+0000","last_undegraded":"2023-12-19T08:03:33.475117+0000","last_fullsized":"2023-12-19T08:03:33.475117+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:59.743549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500192+0000","last_change":"2023-12-19T08:03:33.492167+0000","last_active":"2023-12-19T08:03:33.500192+0000","last_peered":"2023-12-19T08:03:33.500192+0000","last_clean":"2023-12-19T08:03:33.500192+0000","last_became_active":"2023-12-19T08:03:33.491882+0000","last_became_peered":"2023-12-19T08:03:33.491882+0000","last_unstale":"2023-12-19T08:03:33.500192+0000","last_undegraded":"2023-12-19T08:03:33.500192+0000","last_fullsized":"2023-12-19T08:03:33.500192+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:43.966569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475078+0000","last_change":"2023-12-19T08:03:32.951747+0000","last_active":"2023-12-19T08:03:33.475078+0000","last_peered":"2023-12-19T08:03:33.475078+0000","last_clean":"2023-12-19T08:03:33.475078+0000","last_became_active":"2023-12-19T08:03:32.950473+0000","last_became_peered":"2023-12-19T08:03:32.950473+0000","last_unstale":"2023-12-19T08:03:33.475078+0000","last_undegraded":"2023-12-19T08:03:33.475078+0000","last_fullsized":"2023-12-19T08:03:33.475078+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:02.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490849+0000","last_change":"2023-12-19T08:03:33.487630+0000","last_active":"2023-12-19T08:03:33.490849+0000","last_peered":"2023-12-19T08:03:33.490849+0000","last_clean":"2023-12-19T08:03:33.490849+0000","last_became_active":"2023-12-19T08:03:33.487236+0000","last_became_peered":"2023-12-19T08:03:33.487236+0000","last_unstale":"2023-12-19T08:03:33.490849+0000","last_undegraded":"2023-12-19T08:03:33.490849+0000","last_fullsized":"2023-12-19T08:03:33.490849+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.482614+0000","last_change":"2023-12-19T08:03:32.955468+0000","last_active":"2023-12-19T08:03:33.482614+0000","last_peered":"2023-12-19T08:03:33.482614+0000","last_clean":"2023-12-19T08:03:33.482614+0000","last_became_active":"2023-12-19T08:03:32.955003+0000","last_became_peered":"2023-12-19T08:03:32.955003+0000","last_unstale":"2023-12-19T08:03:33.482614+0000","last_undegraded":"2023-12-19T08:03:33.482614+0000","last_fullsized":"2023-12-19T08:03:33.482614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:59:34.316305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499572+0000","last_change":"2023-12-19T08:03:33.494746+0000","last_active":"2023-12-19T08:03:33.499572+0000","last_peered":"2023-12-19T08:03:33.499572+0000","last_clean":"2023-12-19T08:03:33.499572+0000","last_became_active":"2023-12-19T08:03:33.494277+0000","last_became_peered":"2023-12-19T08:03:33.494277+0000","last_unstale":"2023-12-19T08:03:33.499572+0000","last_undegraded":"2023-12-19T08:03:33.499572+0000","last_fullsized":"2023-12-19T08:03:33.499572+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:22:23.439986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475710+0000","last_change":"2023-12-19T08:03:32.944458+0000","last_active":"2023-12-19T08:03:33.475710+0000","last_peered":"2023-12-19T08:03:33.475710+0000","last_clean":"2023-12-19T08:03:33.475710+0000","last_became_active":"2023-12-19T08:03:32.942038+0000","last_became_peered":"2023-12-19T08:03:32.942038+0000","last_unstale":"2023-12-19T08:03:33.475710+0000","last_undegraded":"2023-12-19T08:03:33.475710+0000","last_fullsized":"2023-12-19T08:03:33.475710+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:47.360353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490584+0000","last_change":"2023-12-19T08:03:33.488638+0000","last_active":"2023-12-19T08:03:33.490584+0000","last_peered":"2023-12-19T08:03:33.490584+0000","last_clean":"2023-12-19T08:03:33.490584+0000","last_became_active":"2023-12-19T08:03:33.488060+0000","last_became_peered":"2023-12-19T08:03:33.488060+0000","last_unstale":"2023-12-19T08:03:33.490584+0000","last_undegraded":"2023-12-19T08:03:33.490584+0000","last_fullsized":"2023-12-19T08:03:33.490584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:50.998299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.478802+0000","last_change":"2023-12-19T08:03:32.952935+0000","last_active":"2023-12-19T08:03:33.478802+0000","last_peered":"2023-12-19T08:03:33.478802+0000","last_clean":"2023-12-19T08:03:33.478802+0000","last_became_active":"2023-12-19T08:03:32.952655+0000","last_became_peered":"2023-12-19T08:03:32.952655+0000","last_unstale":"2023-12-19T08:03:33.478802+0000","last_undegraded":"2023-12-19T08:03:33.478802+0000","last_fullsized":"2023-12-19T08:03:33.478802+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:56.567271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502254+0000","last_change":"2023-12-19T08:03:33.501262+0000","last_active":"2023-12-19T08:03:33.502254+0000","last_peered":"2023-12-19T08:03:33.502254+0000","last_clean":"2023-12-19T08:03:33.502254+0000","last_became_active":"2023-12-19T08:03:33.500905+0000","last_became_peered":"2023-12-19T08:03:33.500905+0000","last_unstale":"2023-12-19T08:03:33.502254+0000","last_undegraded":"2023-12-19T08:03:33.502254+0000","last_fullsized":"2023-12-19T08:03:33.502254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.484229+0000","last_change":"2023-12-19T08:03:32.955341+0000","last_active":"2023-12-19T08:03:33.484229+0000","last_peered":"2023-12-19T08:03:33.484229+0000","last_clean":"2023-12-19T08:03:33.484229+0000","last_became_active":"2023-12-19T08:03:32.954966+0000","last_became_peered":"2023-12-19T08:03:32.954966+0000","last_unstale":"2023-12-19T08:03:33.484229+0000","last_undegraded":"2023-12-19T08:03:33.484229+0000","last_fullsized":"2023-12-19T08:03:33.484229+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:11:44.588253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498287+0000","last_change":"2023-12-19T08:03:33.489139+0000","last_active":"2023-12-19T08:03:33.498287+0000","last_peered":"2023-12-19T08:03:33.498287+0000","last_clean":"2023-12-19T08:03:33.498287+0000","last_became_active":"2023-12-19T08:03:33.488848+0000","last_became_peered":"2023-12-19T08:03:33.488848+0000","last_unstale":"2023-12-19T08:03:33.498287+0000","last_undegraded":"2023-12-19T08:03:33.498287+0000","last_fullsized":"2023-12-19T08:03:33.498287+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:20:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476286+0000","last_change":"2023-12-19T08:03:32.957401+0000","last_active":"2023-12-19T08:03:33.476286+0000","last_peered":"2023-12-19T08:03:33.476286+0000","last_clean":"2023-12-19T08:03:33.476286+0000","last_became_active":"2023-12-19T08:03:32.957114+0000","last_became_peered":"2023-12-19T08:03:32.957114+0000","last_unstale":"2023-12-19T08:03:33.476286+0000","last_undegraded":"2023-12-19T08:03:33.476286+0000","last_fullsized":"2023-12-19T08:03:33.476286+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:48:52.259196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497181+0000","last_change":"2023-12-19T08:03:33.493629+0000","last_active":"2023-12-19T08:03:33.497181+0000","last_peered":"2023-12-19T08:03:33.497181+0000","last_clean":"2023-12-19T08:03:33.497181+0000","last_became_active":"2023-12-19T08:03:33.493301+0000","last_became_peered":"2023-12-19T08:03:33.493301+0000","last_unstale":"2023-12-19T08:03:33.497181+0000","last_undegraded":"2023-12-19T08:03:33.497181+0000","last_fullsized":"2023-12-19T08:03:33.497181+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:52.314950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.473015+0000","last_change":"2023-12-19T08:03:32.951471+0000","last_active":"2023-12-19T08:03:33.473015+0000","last_peered":"2023-12-19T08:03:33.473015+0000","last_clean":"2023-12-19T08:03:33.473015+0000","last_became_active":"2023-12-19T08:03:32.951146+0000","last_became_peered":"2023-12-19T08:03:32.951146+0000","last_unstale":"2023-12-19T08:03:33.473015+0000","last_undegraded":"2023-12-19T08:03:33.473015+0000","last_fullsized":"2023-12-19T08:03:33.473015+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:02.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'205","reported_seq":551,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476503+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:03:33.476503+0000","last_peered":"2023-12-19T08:03:33.476503+0000","last_clean":"2023-12-19T08:03:33.476503+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:03:33.476503+0000","last_undegraded":"2023-12-19T08:03:33.476503+0000","last_fullsized":"2023-12-19T08:03:33.476503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1305093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.218,"15min":1.246},"min":{"1min":0.751,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.615,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.135},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.298,"15min":1.2729999999999999},"min":{"1min":0.83199999999999996,"5min":0.749,"15min":0.70799999999999996},"max":{"1min":1.625,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.625}]},{"osd":1,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.169},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.526,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.169,"15min":1.1739999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.843,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.0289999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.1699999999999999},"min":{"1min":0.81499999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.512,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.288},{"interface":"front","average":{"1min":1.226,"5min":1.2609999999999999,"15min":1.2430000000000001},"min":{"1min":0.82999999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8799999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.2450000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2889999999999999,"15min":1.2909999999999999},"min":{"1min":0.92400000000000004,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":1.6499999999999999,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.381,"15min":1.3500000000000001},"min":{"1min":0.88400000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8049999999999999,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.5149999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3069999999999999,"15min":1.2310000000000001},"min":{"1min":0.86699999999999999,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":1.9570000000000001,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.458},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3169999999999999,"15min":1.2470000000000001},"min":{"1min":0.76800000000000002,"5min":0.76500000000000001,"15min":0.64900000000000002},"max":{"1min":1.6699999999999999,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.415}]},{"osd":5,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3129999999999999,"15min":1.242},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.9139999999999999,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.417,"15min":1.361},"min":{"1min":0.94799999999999995,"5min":0.751,"15min":0.751},"max":{"1min":2.036,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.585}]},{"osd":6,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.5009999999999999,"15min":1.4219999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.996,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.748},{"interface":"front","average":{"1min":1.393,"5min":1.4319999999999999,"15min":1.3859999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.58299999999999996},"max":{"1min":2.1400000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.5589999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302764,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069076,"statfs":{"total":96636764160,"available":96326733824,"internally_reserved":0,"allocated":1413120,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.071,"15min":1.103},"min":{"1min":0.75800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.472,"5min":1.5569999999999999,"15min":1.665},"last":1.198},{"interface":"front","average":{"1min":1.109,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.373,"5min":1.756,"15min":1.756},"last":1.429}]},{"osd":1,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.75800000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4770000000000001,"5min":1.675,"15min":1.881},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.073,"5min":1.161,"15min":1.161},"min":{"1min":0.745,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4279999999999999,"5min":1.542,"15min":1.7989999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.675,"5min":1.847,"15min":1.847},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1970000000000001,"15min":1.2130000000000001},"min":{"1min":0.70799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4259999999999999,"5min":1.694,"15min":1.7929999999999999},"last":1.2270000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2450000000000001,"15min":1.2669999999999999},"min":{"1min":0.77600000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0659999999999998},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.3,"15min":1.3100000000000001},"min":{"1min":0.80900000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7889999999999999,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.2909999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1990000000000001,"15min":1.1839999999999999},"min":{"1min":0.80900000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5900000000000001,"5min":1.7130000000000001,"15min":2.137},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.629,"5min":1.974,"15min":2.218},"last":1.546}]},{"osd":5,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3120000000000001,"15min":1.2929999999999999},"min":{"1min":0.78300000000000003,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3109999999999999},"min":{"1min":0.83999999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.663,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.93999999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":2.2029999999999998},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.448,"15min":1.425},"min":{"1min":1.1379999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.137,"15min":2.2639999999999998},"last":1.4039999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298648,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073192,"statfs":{"total":96636764160,"available":96330948608,"internally_reserved":0,"allocated":1392640,"data_stored":721797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.546,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.2410000000000001,"15min":1.264},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.50700000000000001},"max":{"1min":1.5940000000000001,"5min":1.954,"15min":2.3140000000000001},"last":1.27}]},{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.341,"5min":1.585,"15min":3.5409999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.77500000000000002,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.647,"5min":2.7679999999999998,"15min":3.113},"last":1.3460000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1759999999999999,"15min":1.2010000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66000000000000003},"max":{"1min":1.6739999999999999,"5min":1.821,"15min":3.339},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.175,"15min":1.2110000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4279999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.228,"15min":1.2569999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.4630000000000001,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.93400000000000005,"5min":0.88400000000000001,"15min":0.58199999999999996},"max":{"1min":1.6359999999999999,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.619}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.202,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.2629999999999999,"15min":1.2929999999999999},"min":{"1min":0.96899999999999997,"5min":0.70299999999999996,"15min":0.59199999999999997},"max":{"1min":1.621,"5min":2.843,"15min":4.6310000000000002},"last":1.5840000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.81499999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.728,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.367,"15min":1.3700000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.70599999999999996},"max":{"1min":1.96,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.6359999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3939999999999999,"15min":1.427},"min":{"1min":0.88,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.6739999999999999,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.46,"15min":1.462},"min":{"1min":1.038,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.8200000000000001,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.819}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":721957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1950000000000001,"15min":1.214},"min":{"1min":0.70499999999999996,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.4490000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.419},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.94999999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.742,"5min":1.742,"15min":2.738},"last":1.383}]},{"osd":1,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.159,"15min":1.177},"min":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.556,"5min":1.7589999999999999,"15min":2.589},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1060000000000001,"15min":1.143},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54300000000000004},"max":{"1min":1.407,"5min":1.528,"15min":2.2589999999999999},"last":0.86599999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.085,"15min":1.1359999999999999},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5149999999999999,"15min":2.4569999999999999},"last":1.196},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.2350000000000001},"min":{"1min":0.89000000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.524,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.121}]},{"osd":3,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2509999999999999,"15min":1.2829999999999999},"min":{"1min":0.70999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5980000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.151},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2969999999999999,"15min":1.3360000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.81899999999999995},"max":{"1min":1.5780000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.472}]},{"osd":5,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1739999999999999,"15min":1.2170000000000001},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5549999999999999,"5min":1.8149999999999999,"15min":3.556},"last":1.26},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.2150000000000001},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6850000000000001,"5min":1.764,"15min":3.2599999999999998},"last":1.242}]},{"osd":6,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7350000000000001,"5min":1.748,"15min":2.645},"last":1.516},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.3839999999999999},"min":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6299999999999999,"5min":2.0049999999999999,"15min":2.923},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3580000000000001},"min":{"1min":0.82699999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7410000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.333},{"interface":"front","average":{"1min":1.337,"5min":1.3859999999999999,"15min":1.411},"min":{"1min":0.80200000000000005,"5min":0.73099999999999998,"15min":0.70999999999999996},"max":{"1min":1.829,"5min":1.966,"15min":2.8460000000000001},"last":1.637}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":714669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84599999999999997,"15min":0.85099999999999998},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.1359999999999999,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94199999999999995,"15min":0.95199999999999996},"min":{"1min":0.61699999999999999,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.518,"5min":1.962,"15min":1.962},"last":1.0589999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89900000000000002,"15min":0.91300000000000003},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.45600000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.0209999999999999,"15min":1.0289999999999999},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.694,"5min":2.423,"15min":2.423},"last":0.84299999999999997}]},{"osd":2,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0580000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6910000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.879},{"interface":"front","average":{"1min":1.157,"5min":1.1539999999999999,"15min":1.1519999999999999},"min":{"1min":0.68799999999999994,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.599,"5min":2.48,"15min":2.48},"last":0.92800000000000005}]},{"osd":4,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2370000000000001,"15min":1.254},"min":{"1min":0.84299999999999997,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.6120000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.407},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.22,"15min":1.242},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.649,"5min":1.649,"15min":1.8120000000000001},"last":1.379}]},{"osd":5,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2609999999999999,"15min":1.268},"min":{"1min":0.90300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.7569999999999999,"5min":1.996,"15min":1.996},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.331,"15min":1.3280000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.65800000000000003},"max":{"1min":1.821,"5min":2.161,"15min":2.161},"last":1.022}]},{"osd":6,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.3959999999999999,"15min":1.381},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4830000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.339,"15min":1.3600000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.79100000000000004},"max":{"1min":1.6890000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.282}]},{"osd":7,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4490000000000001,"15min":1.448},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.84799999999999998},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.421,"15min":1.411},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.4359999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306840,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065000,"statfs":{"total":96636764160,"available":96322560000,"internally_reserved":0,"allocated":1392640,"data_stored":721797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89100000000000001,"15min":0.90600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.437},"max":{"1min":1.353,"5min":1.776,"15min":1.776},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.012,"15min":1.0529999999999999},"min":{"1min":0.70399999999999996,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.2669999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96599999999999997,"15min":0.95299999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.51800000000000002},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.98599999999999999,"15min":1.0109999999999999},"min":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.528,"5min":1.835,"15min":1.835},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.96499999999999997},"min":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.3220000000000001,"5min":1.657,"15min":1.667},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.181,"5min":1.103,"15min":1.113},"min":{"1min":0.745,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.335,"15min":1.323},"min":{"1min":0.98099999999999998,"5min":0.83699999999999997,"15min":0.79300000000000004},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.2929999999999999,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.82099999999999995},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.984},"last":1.2929999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.341},"min":{"1min":0.95799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9610000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.417},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2629999999999999,"15min":1.2509999999999999},"min":{"1min":0.91400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.5369999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2929999999999999,"15min":1.294},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.345},{"interface":"front","average":{"1min":1.375,"5min":1.3640000000000001,"15min":1.365},"min":{"1min":1.0129999999999999,"5min":0.95999999999999996,"15min":0.85399999999999998},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.5780000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3600000000000001,"15min":1.385},"min":{"1min":1.0509999999999999,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.8620000000000001,"5min":1.9730000000000001,"15min":2.214},"last":1.145},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3440000000000001,"15min":1.365},"min":{"1min":0.98699999999999999,"5min":0.97399999999999998,"15min":0.88200000000000001},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.242}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306832,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065008,"statfs":{"total":96636764160,"available":96322568192,"internally_reserved":0,"allocated":1384448,"data_stored":714739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89100000000000001,"15min":0.86099999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.647,"5min":1.661,"15min":1.661},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":1.284,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.999}]},{"osd":2,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94199999999999995,"15min":0.92300000000000004},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.4670000000000001,"5min":2.96,"15min":2.96},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1020000000000001,"15min":1.0609999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.389,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.357}]},{"osd":3,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.21,"15min":1.161},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.571,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.032},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.097,"15min":1.0600000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.46800000000000003},"max":{"1min":1.524,"5min":2.407,"15min":2.407},"last":0.86099999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.373,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.3480000000000001,"15min":1.3089999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.623,"5min":3.04,"15min":3.04},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.302,"15min":1.278},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.65100000000000002},"max":{"1min":1.837,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.837},{"interface":"front","average":{"1min":1.24,"5min":1.3140000000000001,"15min":1.2849999999999999},"min":{"1min":0.88200000000000001,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.8049999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.8049999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.417,"15min":1.381},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.696,"5min":3.089,"15min":3.089},"last":1.661},{"interface":"front","average":{"1min":1.333,"5min":1.3879999999999999,"15min":1.359},"min":{"1min":0.873,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.881,"5min":2.46,"15min":2.46},"last":1.881}]},{"osd":7,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.431,"15min":1.4139999999999999},"min":{"1min":1.018,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":1.7729999999999999,"5min":2.504,"15min":2.504},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.391,"15min":1.377},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.6899999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.6899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307440,"kb_used_data":1960,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064400,"statfs":{"total":96636764160,"available":96321945600,"internally_reserved":0,"allocated":2007040,"data_stored":1313368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.262,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91600000000000004,"15min":0.93799999999999994},"min":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.385,"5min":1.385,"15min":6.0309999999999997},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.95999999999999996,"15min":0.97999999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.044,"15min":1.03},"min":{"1min":0.75700000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3700000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.96299999999999997}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.96099999999999997},"min":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.51400000000000001},"max":{"1min":1.48,"5min":1.5529999999999999,"15min":1.671},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.115,"15min":1.129},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.514,"5min":1.663,"15min":1.6719999999999999},"last":1.252}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3129999999999999,"15min":1.3080000000000001},"min":{"1min":0.86499999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6000000000000001,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.5840000000000001,"5min":1.855,"15min":2.141},"last":1.268}]},{"osd":5,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.351,"15min":1.3600000000000001},"min":{"1min":0.83399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.6819999999999999,"5min":2.085,"15min":2.117},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.357,"15min":1.361},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.843,"5min":2.125,"15min":2.125},"last":1.45}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.276,"15min":1.268},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.77400000000000002},"max":{"1min":1.5329999999999999,"5min":1.97,"15min":1.97},"last":1.331},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3200000000000001,"15min":1.3220000000000001},"min":{"1min":0.91900000000000004,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.712,"5min":1.871,"15min":1.9530000000000001},"last":1.544}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.341,"15min":1.3240000000000001},"min":{"1min":0.97799999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.8009999999999999,"5min":1.887,"15min":2.0539999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3720000000000001,"15min":1.3759999999999999},"min":{"1min":0.88600000000000001,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.714,"5min":2.0059999999999998,"15min":2.016},"last":1.397}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:03:38.216 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:03:38.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.416+0000 7f6e0b59e700 1 Processor -- start 2023-12-19T08:03:38.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.416+0000 7f6e0b59e700 1 -- start start 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e0c0c6900 0x7f6e0c0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c06b470 con 0x7f6e0c06b900 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0c06e110 con 0x7f6e0c15cc20 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c15d790 con 0x7f6e0c0c6900 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e09d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6e0c06b900 0x7f6e0c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34362/0 (socket says 172.21.15.111:34362) 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e09d9b700 1 -- 172.21.15.111:0/3479384883 learned_addr learned my addr 172.21.15.111:0/3479384883 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:38.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0a59c700 1 --2- 172.21.15.111:0/3479384883 >> v2:172.21.15.111:3300/0 conn(0x7f6e0c0c6900 0x7f6e0c0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275201099 0 0) 0x7f6e0c06e110 con 0x7f6e0c15cc20 2023-12-19T08:03:38.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df4003570 con 0x7f6e0c15cc20 2023-12-19T08:03:38.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821636128 0 0) 0x7f6e0c06b470 con 0x7f6e0c06b900 2023-12-19T08:03:38.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c06e110 con 0x7f6e0c06b900 2023-12-19T08:03:38.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0a59c700 1 -- 172.21.15.111:0/3479384883 >> v1:172.21.15.143:6789/0 conn(0x7f6e0c15cc20 legacy=0x7f6e0c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0a59c700 1 -- 172.21.15.111:0/3479384883 >> v1:172.21.15.111:6789/0 conn(0x7f6e0c06b900 legacy=0x7f6e0c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0a59c700 1 -- 172.21.15.111:0/3479384883 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c161d10 con 0x7f6e0c0c6900 2023-12-19T08:03:38.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0a59c700 1 --2- 172.21.15.111:0/3479384883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c0c6900 0x7f6e0c0c6cd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0098e0 tx=0x7f6dfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482186d081aefa5b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.420+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc011630 con 0x7f6e0c0c6900 2023-12-19T08:03:38.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc011790 con 0x7f6e0c0c6900 2023-12-19T08:03:38.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0959a700 1 -- 172.21.15.111:0/3479384883 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc011b30 con 0x7f6e0c0c6900 2023-12-19T08:03:38.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 -- 172.21.15.111:0/3479384883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c0c6900 msgr2=0x7f6e0c0c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/3479384883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c0c6900 0x7f6e0c0c6cd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0098e0 tx=0x7f6dfc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 -- 172.21.15.111:0/3479384883 shutdown_connections 2023-12-19T08:03:38.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/3479384883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c0c6900 0x7f6e0c0c6cd0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 -- 172.21.15.111:0/3479384883 >> 172.21.15.111:0/3479384883 conn(0x7f6e0c067bb0 msgr2=0x7f6e0c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:38.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 -- 172.21.15.111:0/3479384883 shutdown_connections 2023-12-19T08:03:38.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 -- 172.21.15.111:0/3479384883 wait complete. 2023-12-19T08:03:38.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.424+0000 7f6e0b59e700 1 Processor -- start 2023-12-19T08:03:38.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 -- start start 2023-12-19T08:03:38.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e0c06b900 0x7f6e0c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e0c0c6900 0x7f6e0c0c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c0b05b0 con 0x7f6e0c15cc20 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c0af9e0 con 0x7f6e0c06b900 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c0afc80 con 0x7f6e0c0c6900 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45430/0 (socket says 172.21.15.111:45430) 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 -- 172.21.15.111:0/2467746516 learned_addr learned my addr 172.21.15.111:0/2467746516 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0a59c700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e0c06b900 0x7f6e0c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e09d9b700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e0c0c6900 0x7f6e0c0c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 -- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e0c0c6900 msgr2=0x7f6e0c0c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e0c0c6900 0x7f6e0c0c2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 -- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e0c06b900 msgr2=0x7f6e0c158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e0c06b900 0x7f6e0c158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 -- 172.21.15.111:0/2467746516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc009420 con 0x7f6e0c15cc20 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0ad9d700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400c860 tx=0x7f6e04002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04012630 con 0x7f6e0c15cc20 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e040127b0 con 0x7f6e0c15cc20 2023-12-19T08:03:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04012b70 con 0x7f6e0c15cc20 2023-12-19T08:03:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.428+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c0aff60 con 0x7f6e0c15cc20 2023-12-19T08:03:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.432+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c1592a0 con 0x7f6e0c15cc20 2023-12-19T08:03:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.432+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c0b0760 con 0x7f6e0c15cc20 2023-12-19T08:03:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.432+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6e0401b400 con 0x7f6e0c15cc20 2023-12-19T08:03:38.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.436+0000 7f6dfa7fc700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 0x7f6dec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.436+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f6e0407cfa0 con 0x7f6e0c15cc20 2023-12-19T08:03:38.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.436+0000 7f6e0a59c700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 0x7f6dec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.436+0000 7f6e0a59c700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 0x7f6dec062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0093f0 tx=0x7f6dfc017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.440+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6e040acbe0 con 0x7f6e0c15cc20 2023-12-19T08:03:38.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.612+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e0c04be40 con 0x7f6dec060670 2023-12-19T08:03:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.628+0000 7f6dfa7fc700 1 -- 172.21.15.111:0/2467746516 <== mgr.6389 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287221 (secure 0 0 0) 0x7f6e0c04be40 con 0x7f6dec060670 2023-12-19T08:03:38.639 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 msgr2=0x7f6dec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 0x7f6dec062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0093f0 tx=0x7f6dfc017000 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 msgr2=0x7f6e0c160010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400c860 tx=0x7f6e04002700 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 shutdown_connections 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6dec060670 0x7f6dec062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e0c15cc20 0x7f6e0c160010 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e0c0c6900 0x7f6e0c0c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 --2- 172.21.15.111:0/2467746516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e0c06b900 0x7f6e0c158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 >> 172.21.15.111:0/2467746516 conn(0x7f6e0c067bb0 msgr2=0x7f6e0c159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 shutdown_connections 2023-12-19T08:03:38.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.636+0000 7f6e0b59e700 1 -- 172.21.15.111:0/2467746516 wait complete. 2023-12-19T08:03:38.643 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:03:38.668 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:03:37.281558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":139,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2443616,"kb_used_data":12064,"kb_used_omap":0,"kb_used_meta":2431488,"kb_avail":752531104,"statfs":{"total":773094113280,"available":770591850496,"internally_reserved":0,"allocated":12353536,"data_stored":6937329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489843712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.372655"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486516+0000","last_change":"2023-12-19T08:03:33.483526+0000","last_active":"2023-12-19T08:03:33.486516+0000","last_peered":"2023-12-19T08:03:33.486516+0000","last_clean":"2023-12-19T08:03:33.486516+0000","last_became_active":"2023-12-19T08:03:33.483260+0000","last_became_peered":"2023-12-19T08:03:33.483260+0000","last_unstale":"2023-12-19T08:03:33.486516+0000","last_undegraded":"2023-12-19T08:03:33.486516+0000","last_fullsized":"2023-12-19T08:03:33.486516+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:31:29.470742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497432+0000","last_change":"2023-12-19T08:03:33.494403+0000","last_active":"2023-12-19T08:03:33.497432+0000","last_peered":"2023-12-19T08:03:33.497432+0000","last_clean":"2023-12-19T08:03:33.497432+0000","last_became_active":"2023-12-19T08:03:33.493595+0000","last_became_peered":"2023-12-19T08:03:33.493595+0000","last_unstale":"2023-12-19T08:03:33.497432+0000","last_undegraded":"2023-12-19T08:03:33.497432+0000","last_fullsized":"2023-12-19T08:03:33.497432+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.477738+0000","last_change":"2023-12-19T08:03:33.476010+0000","last_active":"2023-12-19T08:03:33.477738+0000","last_peered":"2023-12-19T08:03:33.477738+0000","last_clean":"2023-12-19T08:03:33.477738+0000","last_became_active":"2023-12-19T08:03:33.475333+0000","last_became_peered":"2023-12-19T08:03:33.475333+0000","last_unstale":"2023-12-19T08:03:33.477738+0000","last_undegraded":"2023-12-19T08:03:33.477738+0000","last_fullsized":"2023-12-19T08:03:33.477738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:42:12.891064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.495594+0000","last_change":"2023-12-19T08:03:33.490631+0000","last_active":"2023-12-19T08:03:33.495594+0000","last_peered":"2023-12-19T08:03:33.495594+0000","last_clean":"2023-12-19T08:03:33.495594+0000","last_became_active":"2023-12-19T08:03:33.490351+0000","last_became_peered":"2023-12-19T08:03:33.490351+0000","last_unstale":"2023-12-19T08:03:33.495594+0000","last_undegraded":"2023-12-19T08:03:33.495594+0000","last_fullsized":"2023-12-19T08:03:33.495594+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497950+0000","last_change":"2023-12-19T08:03:33.493902+0000","last_active":"2023-12-19T08:03:33.497950+0000","last_peered":"2023-12-19T08:03:33.497950+0000","last_clean":"2023-12-19T08:03:33.497950+0000","last_became_active":"2023-12-19T08:03:33.491759+0000","last_became_peered":"2023-12-19T08:03:33.491759+0000","last_unstale":"2023-12-19T08:03:33.497950+0000","last_undegraded":"2023-12-19T08:03:33.497950+0000","last_fullsized":"2023-12-19T08:03:33.497950+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:52.769135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500253+0000","last_change":"2023-12-19T08:03:33.496646+0000","last_active":"2023-12-19T08:03:33.500253+0000","last_peered":"2023-12-19T08:03:33.500253+0000","last_clean":"2023-12-19T08:03:33.500253+0000","last_became_active":"2023-12-19T08:03:33.496380+0000","last_became_peered":"2023-12-19T08:03:33.496380+0000","last_unstale":"2023-12-19T08:03:33.500253+0000","last_undegraded":"2023-12-19T08:03:33.500253+0000","last_fullsized":"2023-12-19T08:03:33.500253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:27:36.213826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.488405+0000","last_change":"2023-12-19T08:03:33.486510+0000","last_active":"2023-12-19T08:03:33.488405+0000","last_peered":"2023-12-19T08:03:33.488405+0000","last_clean":"2023-12-19T08:03:33.488405+0000","last_became_active":"2023-12-19T08:03:33.486191+0000","last_became_peered":"2023-12-19T08:03:33.486191+0000","last_unstale":"2023-12-19T08:03:33.488405+0000","last_undegraded":"2023-12-19T08:03:33.488405+0000","last_fullsized":"2023-12-19T08:03:33.488405+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:14:23.472892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489090+0000","last_change":"2023-12-19T08:03:33.483741+0000","last_active":"2023-12-19T08:03:33.489090+0000","last_peered":"2023-12-19T08:03:33.489090+0000","last_clean":"2023-12-19T08:03:33.489090+0000","last_became_active":"2023-12-19T08:03:33.483444+0000","last_became_peered":"2023-12-19T08:03:33.483444+0000","last_unstale":"2023-12-19T08:03:33.489090+0000","last_undegraded":"2023-12-19T08:03:33.489090+0000","last_fullsized":"2023-12-19T08:03:33.489090+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:08.878219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494472+0000","last_change":"2023-12-19T08:03:33.491060+0000","last_active":"2023-12-19T08:03:33.494472+0000","last_peered":"2023-12-19T08:03:33.494472+0000","last_clean":"2023-12-19T08:03:33.494472+0000","last_became_active":"2023-12-19T08:03:33.490647+0000","last_became_peered":"2023-12-19T08:03:33.490647+0000","last_unstale":"2023-12-19T08:03:33.494472+0000","last_undegraded":"2023-12-19T08:03:33.494472+0000","last_fullsized":"2023-12-19T08:03:33.494472+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501231+0000","last_change":"2023-12-19T08:03:33.497094+0000","last_active":"2023-12-19T08:03:33.501231+0000","last_peered":"2023-12-19T08:03:33.501231+0000","last_clean":"2023-12-19T08:03:33.501231+0000","last_became_active":"2023-12-19T08:03:33.496734+0000","last_became_peered":"2023-12-19T08:03:33.496734+0000","last_unstale":"2023-12-19T08:03:33.501231+0000","last_undegraded":"2023-12-19T08:03:33.501231+0000","last_fullsized":"2023-12-19T08:03:33.501231+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:05:10.019107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492393+0000","last_change":"2023-12-19T08:03:33.488004+0000","last_active":"2023-12-19T08:03:33.492393+0000","last_peered":"2023-12-19T08:03:33.492393+0000","last_clean":"2023-12-19T08:03:33.492393+0000","last_became_active":"2023-12-19T08:03:33.487599+0000","last_became_peered":"2023-12-19T08:03:33.487599+0000","last_unstale":"2023-12-19T08:03:33.492393+0000","last_undegraded":"2023-12-19T08:03:33.492393+0000","last_fullsized":"2023-12-19T08:03:33.492393+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.493887+0000","last_change":"2023-12-19T08:03:33.485971+0000","last_active":"2023-12-19T08:03:33.493887+0000","last_peered":"2023-12-19T08:03:33.493887+0000","last_clean":"2023-12-19T08:03:33.493887+0000","last_became_active":"2023-12-19T08:03:33.485690+0000","last_became_peered":"2023-12-19T08:03:33.485690+0000","last_unstale":"2023-12-19T08:03:33.493887+0000","last_undegraded":"2023-12-19T08:03:33.493887+0000","last_fullsized":"2023-12-19T08:03:33.493887+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:41.763317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490153+0000","last_change":"2023-12-19T08:03:33.479400+0000","last_active":"2023-12-19T08:03:33.490153+0000","last_peered":"2023-12-19T08:03:33.490153+0000","last_clean":"2023-12-19T08:03:33.490153+0000","last_became_active":"2023-12-19T08:03:33.478895+0000","last_became_peered":"2023-12-19T08:03:33.478895+0000","last_unstale":"2023-12-19T08:03:33.490153+0000","last_undegraded":"2023-12-19T08:03:33.490153+0000","last_fullsized":"2023-12-19T08:03:33.490153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498294+0000","last_change":"2023-12-19T08:03:33.496111+0000","last_active":"2023-12-19T08:03:33.498294+0000","last_peered":"2023-12-19T08:03:33.498294+0000","last_clean":"2023-12-19T08:03:33.498294+0000","last_became_active":"2023-12-19T08:03:33.495834+0000","last_became_peered":"2023-12-19T08:03:33.495834+0000","last_unstale":"2023-12-19T08:03:33.498294+0000","last_undegraded":"2023-12-19T08:03:33.498294+0000","last_fullsized":"2023-12-19T08:03:33.498294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:02.622620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.496692+0000","last_change":"2023-12-19T08:03:33.495064+0000","last_active":"2023-12-19T08:03:33.496692+0000","last_peered":"2023-12-19T08:03:33.496692+0000","last_clean":"2023-12-19T08:03:33.496692+0000","last_became_active":"2023-12-19T08:03:33.494667+0000","last_became_peered":"2023-12-19T08:03:33.494667+0000","last_unstale":"2023-12-19T08:03:33.496692+0000","last_undegraded":"2023-12-19T08:03:33.496692+0000","last_fullsized":"2023-12-19T08:03:33.496692+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492322+0000","last_change":"2023-12-19T08:03:33.480488+0000","last_active":"2023-12-19T08:03:33.492322+0000","last_peered":"2023-12-19T08:03:33.492322+0000","last_clean":"2023-12-19T08:03:33.492322+0000","last_became_active":"2023-12-19T08:03:33.480184+0000","last_became_peered":"2023-12-19T08:03:33.480184+0000","last_unstale":"2023-12-19T08:03:33.492322+0000","last_undegraded":"2023-12-19T08:03:33.492322+0000","last_fullsized":"2023-12-19T08:03:33.492322+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:46.444257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499112+0000","last_change":"2023-12-19T08:03:33.488645+0000","last_active":"2023-12-19T08:03:33.499112+0000","last_peered":"2023-12-19T08:03:33.499112+0000","last_clean":"2023-12-19T08:03:33.499112+0000","last_became_active":"2023-12-19T08:03:33.487066+0000","last_became_peered":"2023-12-19T08:03:33.487066+0000","last_unstale":"2023-12-19T08:03:33.499112+0000","last_undegraded":"2023-12-19T08:03:33.499112+0000","last_fullsized":"2023-12-19T08:03:33.499112+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:08.647435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490125+0000","last_change":"2023-12-19T08:03:33.485107+0000","last_active":"2023-12-19T08:03:33.490125+0000","last_peered":"2023-12-19T08:03:33.490125+0000","last_clean":"2023-12-19T08:03:33.490125+0000","last_became_active":"2023-12-19T08:03:33.484406+0000","last_became_peered":"2023-12-19T08:03:33.484406+0000","last_unstale":"2023-12-19T08:03:33.490125+0000","last_undegraded":"2023-12-19T08:03:33.490125+0000","last_fullsized":"2023-12-19T08:03:33.490125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:27:16.739695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497961+0000","last_change":"2023-12-19T08:03:33.495729+0000","last_active":"2023-12-19T08:03:33.497961+0000","last_peered":"2023-12-19T08:03:33.497961+0000","last_clean":"2023-12-19T08:03:33.497961+0000","last_became_active":"2023-12-19T08:03:33.495399+0000","last_became_peered":"2023-12-19T08:03:33.495399+0000","last_unstale":"2023-12-19T08:03:33.497961+0000","last_undegraded":"2023-12-19T08:03:33.497961+0000","last_fullsized":"2023-12-19T08:03:33.497961+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490800+0000","last_change":"2023-12-19T08:03:33.480669+0000","last_active":"2023-12-19T08:03:33.490800+0000","last_peered":"2023-12-19T08:03:33.490800+0000","last_clean":"2023-12-19T08:03:33.490800+0000","last_became_active":"2023-12-19T08:03:33.478912+0000","last_became_peered":"2023-12-19T08:03:33.478912+0000","last_unstale":"2023-12-19T08:03:33.490800+0000","last_undegraded":"2023-12-19T08:03:33.490800+0000","last_fullsized":"2023-12-19T08:03:33.490800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.493657+0000","last_change":"2023-12-19T08:03:33.483556+0000","last_active":"2023-12-19T08:03:33.493657+0000","last_peered":"2023-12-19T08:03:33.493657+0000","last_clean":"2023-12-19T08:03:33.493657+0000","last_became_active":"2023-12-19T08:03:33.483280+0000","last_became_peered":"2023-12-19T08:03:33.483280+0000","last_unstale":"2023-12-19T08:03:33.493657+0000","last_undegraded":"2023-12-19T08:03:33.493657+0000","last_fullsized":"2023-12-19T08:03:33.493657+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:32:19.692755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498772+0000","last_change":"2023-12-19T08:03:33.497052+0000","last_active":"2023-12-19T08:03:33.498772+0000","last_peered":"2023-12-19T08:03:33.498772+0000","last_clean":"2023-12-19T08:03:33.498772+0000","last_became_active":"2023-12-19T08:03:33.496466+0000","last_became_peered":"2023-12-19T08:03:33.496466+0000","last_unstale":"2023-12-19T08:03:33.498772+0000","last_undegraded":"2023-12-19T08:03:33.498772+0000","last_fullsized":"2023-12-19T08:03:33.498772+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:01.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502234+0000","last_change":"2023-12-19T08:03:33.501256+0000","last_active":"2023-12-19T08:03:33.502234+0000","last_peered":"2023-12-19T08:03:33.502234+0000","last_clean":"2023-12-19T08:03:33.502234+0000","last_became_active":"2023-12-19T08:03:33.500903+0000","last_became_peered":"2023-12-19T08:03:33.500903+0000","last_unstale":"2023-12-19T08:03:33.502234+0000","last_undegraded":"2023-12-19T08:03:33.502234+0000","last_fullsized":"2023-12-19T08:03:33.502234+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.485944+0000","last_change":"2023-12-19T08:03:33.482986+0000","last_active":"2023-12-19T08:03:33.485944+0000","last_peered":"2023-12-19T08:03:33.485944+0000","last_clean":"2023-12-19T08:03:33.485944+0000","last_became_active":"2023-12-19T08:03:33.482686+0000","last_became_peered":"2023-12-19T08:03:33.482686+0000","last_unstale":"2023-12-19T08:03:33.485944+0000","last_undegraded":"2023-12-19T08:03:33.485944+0000","last_fullsized":"2023-12-19T08:03:33.485944+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:14:00.428547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490864+0000","last_change":"2023-12-19T08:03:33.487930+0000","last_active":"2023-12-19T08:03:33.490864+0000","last_peered":"2023-12-19T08:03:33.490864+0000","last_clean":"2023-12-19T08:03:33.490864+0000","last_became_active":"2023-12-19T08:03:33.487635+0000","last_became_peered":"2023-12-19T08:03:33.487635+0000","last_unstale":"2023-12-19T08:03:33.490864+0000","last_undegraded":"2023-12-19T08:03:33.490864+0000","last_fullsized":"2023-12-19T08:03:33.490864+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497979+0000","last_change":"2023-12-19T08:03:33.495114+0000","last_active":"2023-12-19T08:03:33.497979+0000","last_peered":"2023-12-19T08:03:33.497979+0000","last_clean":"2023-12-19T08:03:33.497979+0000","last_became_active":"2023-12-19T08:03:33.494838+0000","last_became_peered":"2023-12-19T08:03:33.494838+0000","last_unstale":"2023-12-19T08:03:33.497979+0000","last_undegraded":"2023-12-19T08:03:33.497979+0000","last_fullsized":"2023-12-19T08:03:33.497979+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.496253+0000","last_change":"2023-12-19T08:03:33.491303+0000","last_active":"2023-12-19T08:03:33.496253+0000","last_peered":"2023-12-19T08:03:33.496253+0000","last_clean":"2023-12-19T08:03:33.496253+0000","last_became_active":"2023-12-19T08:03:33.490331+0000","last_became_peered":"2023-12-19T08:03:33.490331+0000","last_unstale":"2023-12-19T08:03:33.496253+0000","last_undegraded":"2023-12-19T08:03:33.496253+0000","last_fullsized":"2023-12-19T08:03:33.496253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:32:48.089629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490783+0000","last_change":"2023-12-19T08:03:33.479506+0000","last_active":"2023-12-19T08:03:33.490783+0000","last_peered":"2023-12-19T08:03:33.490783+0000","last_clean":"2023-12-19T08:03:33.490783+0000","last_became_active":"2023-12-19T08:03:33.479143+0000","last_became_peered":"2023-12-19T08:03:33.479143+0000","last_unstale":"2023-12-19T08:03:33.490783+0000","last_undegraded":"2023-12-19T08:03:33.490783+0000","last_fullsized":"2023-12-19T08:03:33.490783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501637+0000","last_change":"2023-12-19T08:03:33.501025+0000","last_active":"2023-12-19T08:03:33.501637+0000","last_peered":"2023-12-19T08:03:33.501637+0000","last_clean":"2023-12-19T08:03:33.501637+0000","last_became_active":"2023-12-19T08:03:33.500409+0000","last_became_peered":"2023-12-19T08:03:33.500409+0000","last_unstale":"2023-12-19T08:03:33.501637+0000","last_undegraded":"2023-12-19T08:03:33.501637+0000","last_fullsized":"2023-12-19T08:03:33.501637+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:59:20.172093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490359+0000","last_change":"2023-12-19T08:03:33.487426+0000","last_active":"2023-12-19T08:03:33.490359+0000","last_peered":"2023-12-19T08:03:33.490359+0000","last_clean":"2023-12-19T08:03:33.490359+0000","last_became_active":"2023-12-19T08:03:33.487097+0000","last_became_peered":"2023-12-19T08:03:33.487097+0000","last_unstale":"2023-12-19T08:03:33.490359+0000","last_undegraded":"2023-12-19T08:03:33.490359+0000","last_fullsized":"2023-12-19T08:03:33.490359+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497397+0000","last_change":"2023-12-19T08:03:33.495076+0000","last_active":"2023-12-19T08:03:33.497397+0000","last_peered":"2023-12-19T08:03:33.497397+0000","last_clean":"2023-12-19T08:03:33.497397+0000","last_became_active":"2023-12-19T08:03:33.494471+0000","last_became_peered":"2023-12-19T08:03:33.494471+0000","last_unstale":"2023-12-19T08:03:33.497397+0000","last_undegraded":"2023-12-19T08:03:33.497397+0000","last_fullsized":"2023-12-19T08:03:33.497397+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.492288+0000","last_change":"2023-12-19T08:03:33.479076+0000","last_active":"2023-12-19T08:03:33.492288+0000","last_peered":"2023-12-19T08:03:33.492288+0000","last_clean":"2023-12-19T08:03:33.492288+0000","last_became_active":"2023-12-19T08:03:33.478792+0000","last_became_peered":"2023-12-19T08:03:33.478792+0000","last_unstale":"2023-12-19T08:03:33.492288+0000","last_undegraded":"2023-12-19T08:03:33.492288+0000","last_fullsized":"2023-12-19T08:03:33.492288+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:13:46.264168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498260+0000","last_change":"2023-12-19T08:03:33.487582+0000","last_active":"2023-12-19T08:03:33.498260+0000","last_peered":"2023-12-19T08:03:33.498260+0000","last_clean":"2023-12-19T08:03:33.498260+0000","last_became_active":"2023-12-19T08:03:33.487325+0000","last_became_peered":"2023-12-19T08:03:33.487325+0000","last_unstale":"2023-12-19T08:03:33.498260+0000","last_undegraded":"2023-12-19T08:03:33.498260+0000","last_fullsized":"2023-12-19T08:03:33.498260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:19:20.739839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"112'4","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956886+0000","last_change":"2023-12-19T08:03:32.957444+0000","last_active":"2023-12-19T08:03:33.956886+0000","last_peered":"2023-12-19T08:03:33.956886+0000","last_clean":"2023-12-19T08:03:33.956886+0000","last_became_active":"2023-12-19T08:03:32.956954+0000","last_became_peered":"2023-12-19T08:03:32.956954+0000","last_unstale":"2023-12-19T08:03:33.956886+0000","last_undegraded":"2023-12-19T08:03:33.956886+0000","last_fullsized":"2023-12-19T08:03:33.956886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:49.416066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500161+0000","last_change":"2023-12-19T08:03:33.492120+0000","last_active":"2023-12-19T08:03:33.500161+0000","last_peered":"2023-12-19T08:03:33.500161+0000","last_clean":"2023-12-19T08:03:33.500161+0000","last_became_active":"2023-12-19T08:03:33.491790+0000","last_became_peered":"2023-12-19T08:03:33.491790+0000","last_unstale":"2023-12-19T08:03:33.500161+0000","last_undegraded":"2023-12-19T08:03:33.500161+0000","last_fullsized":"2023-12-19T08:03:33.500161+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:06:27.601956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.952119+0000","last_change":"2023-12-19T08:03:32.952797+0000","last_active":"2023-12-19T08:03:33.952119+0000","last_peered":"2023-12-19T08:03:33.952119+0000","last_clean":"2023-12-19T08:03:33.952119+0000","last_became_active":"2023-12-19T08:03:32.951746+0000","last_became_peered":"2023-12-19T08:03:32.951746+0000","last_unstale":"2023-12-19T08:03:33.952119+0000","last_undegraded":"2023-12-19T08:03:33.952119+0000","last_fullsized":"2023-12-19T08:03:33.952119+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:54:48.344896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498557+0000","last_change":"2023-12-19T08:03:33.497519+0000","last_active":"2023-12-19T08:03:33.498557+0000","last_peered":"2023-12-19T08:03:33.498557+0000","last_clean":"2023-12-19T08:03:33.498557+0000","last_became_active":"2023-12-19T08:03:33.497078+0000","last_became_peered":"2023-12-19T08:03:33.497078+0000","last_unstale":"2023-12-19T08:03:33.498557+0000","last_undegraded":"2023-12-19T08:03:33.498557+0000","last_fullsized":"2023-12-19T08:03:33.498557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:02.622620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.954038+0000","last_change":"2023-12-19T08:03:32.955148+0000","last_active":"2023-12-19T08:03:33.954038+0000","last_peered":"2023-12-19T08:03:33.954038+0000","last_clean":"2023-12-19T08:03:33.954038+0000","last_became_active":"2023-12-19T08:03:32.954848+0000","last_became_peered":"2023-12-19T08:03:32.954848+0000","last_unstale":"2023-12-19T08:03:33.954038+0000","last_undegraded":"2023-12-19T08:03:33.954038+0000","last_fullsized":"2023-12-19T08:03:33.954038+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:07.861485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499094+0000","last_change":"2023-12-19T08:03:33.481278+0000","last_active":"2023-12-19T08:03:33.499094+0000","last_peered":"2023-12-19T08:03:33.499094+0000","last_clean":"2023-12-19T08:03:33.499094+0000","last_became_active":"2023-12-19T08:03:33.480989+0000","last_became_peered":"2023-12-19T08:03:33.480989+0000","last_unstale":"2023-12-19T08:03:33.499094+0000","last_undegraded":"2023-12-19T08:03:33.499094+0000","last_fullsized":"2023-12-19T08:03:33.499094+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:15:05.924454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.955220+0000","last_change":"2023-12-19T08:03:32.956243+0000","last_active":"2023-12-19T08:03:33.955220+0000","last_peered":"2023-12-19T08:03:33.955220+0000","last_clean":"2023-12-19T08:03:33.955220+0000","last_became_active":"2023-12-19T08:03:32.955865+0000","last_became_peered":"2023-12-19T08:03:32.955865+0000","last_unstale":"2023-12-19T08:03:33.955220+0000","last_undegraded":"2023-12-19T08:03:33.955220+0000","last_fullsized":"2023-12-19T08:03:33.955220+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.152351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489002+0000","last_change":"2023-12-19T08:03:33.479280+0000","last_active":"2023-12-19T08:03:33.489002+0000","last_peered":"2023-12-19T08:03:33.489002+0000","last_clean":"2023-12-19T08:03:33.489002+0000","last_became_active":"2023-12-19T08:03:33.478884+0000","last_became_peered":"2023-12-19T08:03:33.478884+0000","last_unstale":"2023-12-19T08:03:33.489002+0000","last_undegraded":"2023-12-19T08:03:33.489002+0000","last_fullsized":"2023-12-19T08:03:33.489002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.953764+0000","last_change":"2023-12-19T08:03:32.952536+0000","last_active":"2023-12-19T08:03:33.953764+0000","last_peered":"2023-12-19T08:03:33.953764+0000","last_clean":"2023-12-19T08:03:33.953764+0000","last_became_active":"2023-12-19T08:03:32.951133+0000","last_became_peered":"2023-12-19T08:03:32.951133+0000","last_unstale":"2023-12-19T08:03:33.953764+0000","last_undegraded":"2023-12-19T08:03:33.953764+0000","last_fullsized":"2023-12-19T08:03:33.953764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:25:02.590163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501966+0000","last_change":"2023-12-19T08:03:33.498372+0000","last_active":"2023-12-19T08:03:33.501966+0000","last_peered":"2023-12-19T08:03:33.501966+0000","last_clean":"2023-12-19T08:03:33.501966+0000","last_became_active":"2023-12-19T08:03:33.497657+0000","last_became_peered":"2023-12-19T08:03:33.497657+0000","last_unstale":"2023-12-19T08:03:33.501966+0000","last_undegraded":"2023-12-19T08:03:33.501966+0000","last_fullsized":"2023-12-19T08:03:33.501966+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.947692+0000","last_change":"2023-12-19T08:03:32.947425+0000","last_active":"2023-12-19T08:03:33.947692+0000","last_peered":"2023-12-19T08:03:33.947692+0000","last_clean":"2023-12-19T08:03:33.947692+0000","last_became_active":"2023-12-19T08:03:32.947072+0000","last_became_peered":"2023-12-19T08:03:32.947072+0000","last_unstale":"2023-12-19T08:03:33.947692+0000","last_undegraded":"2023-12-19T08:03:33.947692+0000","last_fullsized":"2023-12-19T08:03:33.947692+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:54:48.344896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500270+0000","last_change":"2023-12-19T08:03:33.497118+0000","last_active":"2023-12-19T08:03:33.500270+0000","last_peered":"2023-12-19T08:03:33.500270+0000","last_clean":"2023-12-19T08:03:33.500270+0000","last_became_active":"2023-12-19T08:03:33.496816+0000","last_became_peered":"2023-12-19T08:03:33.496816+0000","last_unstale":"2023-12-19T08:03:33.500270+0000","last_undegraded":"2023-12-19T08:03:33.500270+0000","last_fullsized":"2023-12-19T08:03:33.500270+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:54.598683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.464292+0000","last_change":"2023-12-19T08:03:32.943454+0000","last_active":"2023-12-19T08:03:33.464292+0000","last_peered":"2023-12-19T08:03:33.464292+0000","last_clean":"2023-12-19T08:03:33.464292+0000","last_became_active":"2023-12-19T08:03:32.943171+0000","last_became_peered":"2023-12-19T08:03:32.943171+0000","last_unstale":"2023-12-19T08:03:33.464292+0000","last_undegraded":"2023-12-19T08:03:33.464292+0000","last_fullsized":"2023-12-19T08:03:33.464292+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:15:04.907719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490170+0000","last_change":"2023-12-19T08:03:33.487373+0000","last_active":"2023-12-19T08:03:33.490170+0000","last_peered":"2023-12-19T08:03:33.490170+0000","last_clean":"2023-12-19T08:03:33.490170+0000","last_became_active":"2023-12-19T08:03:33.486362+0000","last_became_peered":"2023-12-19T08:03:33.486362+0000","last_unstale":"2023-12-19T08:03:33.490170+0000","last_undegraded":"2023-12-19T08:03:33.490170+0000","last_fullsized":"2023-12-19T08:03:33.490170+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:00:37.251846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476403+0000","last_change":"2023-12-19T08:03:32.958981+0000","last_active":"2023-12-19T08:03:33.476403+0000","last_peered":"2023-12-19T08:03:33.476403+0000","last_clean":"2023-12-19T08:03:33.476403+0000","last_became_active":"2023-12-19T08:03:32.958411+0000","last_became_peered":"2023-12-19T08:03:32.958411+0000","last_unstale":"2023-12-19T08:03:33.476403+0000","last_undegraded":"2023-12-19T08:03:33.476403+0000","last_fullsized":"2023-12-19T08:03:33.476403+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:16.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.485111+0000","last_change":"2023-12-19T08:03:33.481237+0000","last_active":"2023-12-19T08:03:33.485111+0000","last_peered":"2023-12-19T08:03:33.485111+0000","last_clean":"2023-12-19T08:03:33.485111+0000","last_became_active":"2023-12-19T08:03:33.478910+0000","last_became_peered":"2023-12-19T08:03:33.478910+0000","last_unstale":"2023-12-19T08:03:33.485111+0000","last_undegraded":"2023-12-19T08:03:33.485111+0000","last_fullsized":"2023-12-19T08:03:33.485111+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:41:51.857662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.480934+0000","last_change":"2023-12-19T08:03:32.952096+0000","last_active":"2023-12-19T08:03:33.480934+0000","last_peered":"2023-12-19T08:03:33.480934+0000","last_clean":"2023-12-19T08:03:33.480934+0000","last_became_active":"2023-12-19T08:03:32.951815+0000","last_became_peered":"2023-12-19T08:03:32.951815+0000","last_unstale":"2023-12-19T08:03:33.480934+0000","last_undegraded":"2023-12-19T08:03:33.480934+0000","last_fullsized":"2023-12-19T08:03:33.480934+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.152351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.488423+0000","last_change":"2023-12-19T08:03:33.486532+0000","last_active":"2023-12-19T08:03:33.488423+0000","last_peered":"2023-12-19T08:03:33.488423+0000","last_clean":"2023-12-19T08:03:33.488423+0000","last_became_active":"2023-12-19T08:03:33.486254+0000","last_became_peered":"2023-12-19T08:03:33.486254+0000","last_unstale":"2023-12-19T08:03:33.488423+0000","last_undegraded":"2023-12-19T08:03:33.488423+0000","last_fullsized":"2023-12-19T08:03:33.488423+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:30:24.758243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.465131+0000","last_change":"2023-12-19T08:03:32.948461+0000","last_active":"2023-12-19T08:03:33.465131+0000","last_peered":"2023-12-19T08:03:33.465131+0000","last_clean":"2023-12-19T08:03:33.465131+0000","last_became_active":"2023-12-19T08:03:32.948131+0000","last_became_peered":"2023-12-19T08:03:32.948131+0000","last_unstale":"2023-12-19T08:03:33.465131+0000","last_undegraded":"2023-12-19T08:03:33.465131+0000","last_fullsized":"2023-12-19T08:03:33.465131+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:09.077379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.484623+0000","last_change":"2023-12-19T08:03:33.482184+0000","last_active":"2023-12-19T08:03:33.484623+0000","last_peered":"2023-12-19T08:03:33.484623+0000","last_clean":"2023-12-19T08:03:33.484623+0000","last_became_active":"2023-12-19T08:03:33.481907+0000","last_became_peered":"2023-12-19T08:03:33.481907+0000","last_unstale":"2023-12-19T08:03:33.484623+0000","last_undegraded":"2023-12-19T08:03:33.484623+0000","last_fullsized":"2023-12-19T08:03:33.484623+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:04.746187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"112'3","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.954987+0000","last_change":"2023-12-19T08:03:32.954273+0000","last_active":"2023-12-19T08:03:33.954987+0000","last_peered":"2023-12-19T08:03:33.954987+0000","last_clean":"2023-12-19T08:03:33.954987+0000","last_became_active":"2023-12-19T08:03:32.953922+0000","last_became_peered":"2023-12-19T08:03:32.953922+0000","last_unstale":"2023-12-19T08:03:33.954987+0000","last_undegraded":"2023-12-19T08:03:33.954987+0000","last_fullsized":"2023-12-19T08:03:33.954987+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:28.068462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489125+0000","last_change":"2023-12-19T08:03:33.486370+0000","last_active":"2023-12-19T08:03:33.489125+0000","last_peered":"2023-12-19T08:03:33.489125+0000","last_clean":"2023-12-19T08:03:33.489125+0000","last_became_active":"2023-12-19T08:03:33.486057+0000","last_became_peered":"2023-12-19T08:03:33.486057+0000","last_unstale":"2023-12-19T08:03:33.489125+0000","last_undegraded":"2023-12-19T08:03:33.489125+0000","last_fullsized":"2023-12-19T08:03:33.489125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:44:01.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":16,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.953940+0000","last_change":"2023-12-19T08:03:32.949662+0000","last_active":"2023-12-19T08:03:33.953940+0000","last_peered":"2023-12-19T08:03:33.953940+0000","last_clean":"2023-12-19T08:03:33.953940+0000","last_became_active":"2023-12-19T08:03:32.949247+0000","last_became_peered":"2023-12-19T08:03:32.949247+0000","last_unstale":"2023-12-19T08:03:33.953940+0000","last_undegraded":"2023-12-19T08:03:33.953940+0000","last_fullsized":"2023-12-19T08:03:33.953940+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:30.492951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.489115+0000","last_change":"2023-12-19T08:03:33.484062+0000","last_active":"2023-12-19T08:03:33.489115+0000","last_peered":"2023-12-19T08:03:33.489115+0000","last_clean":"2023-12-19T08:03:33.489115+0000","last_became_active":"2023-12-19T08:03:33.483742+0000","last_became_peered":"2023-12-19T08:03:33.483742+0000","last_unstale":"2023-12-19T08:03:33.489115+0000","last_undegraded":"2023-12-19T08:03:33.489115+0000","last_fullsized":"2023-12-19T08:03:33.489115+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:44:15.395278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956617+0000","last_change":"2023-12-19T08:03:32.960033+0000","last_active":"2023-12-19T08:03:33.956617+0000","last_peered":"2023-12-19T08:03:33.956617+0000","last_clean":"2023-12-19T08:03:33.956617+0000","last_became_active":"2023-12-19T08:03:32.959695+0000","last_became_peered":"2023-12-19T08:03:32.959695+0000","last_unstale":"2023-12-19T08:03:33.956617+0000","last_undegraded":"2023-12-19T08:03:33.956617+0000","last_fullsized":"2023-12-19T08:03:33.956617+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:47.360353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494277+0000","last_change":"2023-12-19T08:03:33.490631+0000","last_active":"2023-12-19T08:03:33.494277+0000","last_peered":"2023-12-19T08:03:33.494277+0000","last_clean":"2023-12-19T08:03:33.494277+0000","last_became_active":"2023-12-19T08:03:33.490357+0000","last_became_peered":"2023-12-19T08:03:33.490357+0000","last_unstale":"2023-12-19T08:03:33.494277+0000","last_undegraded":"2023-12-19T08:03:33.494277+0000","last_fullsized":"2023-12-19T08:03:33.494277+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:52.314950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475238+0000","last_change":"2023-12-19T08:03:32.940732+0000","last_active":"2023-12-19T08:03:33.475238+0000","last_peered":"2023-12-19T08:03:33.475238+0000","last_clean":"2023-12-19T08:03:33.475238+0000","last_became_active":"2023-12-19T08:03:32.940308+0000","last_became_peered":"2023-12-19T08:03:32.940308+0000","last_unstale":"2023-12-19T08:03:33.475238+0000","last_undegraded":"2023-12-19T08:03:33.475238+0000","last_fullsized":"2023-12-19T08:03:33.475238+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:16.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.501227+0000","last_change":"2023-12-19T08:03:33.497321+0000","last_active":"2023-12-19T08:03:33.501227+0000","last_peered":"2023-12-19T08:03:33.501227+0000","last_clean":"2023-12-19T08:03:33.501227+0000","last_became_active":"2023-12-19T08:03:33.496926+0000","last_became_peered":"2023-12-19T08:03:33.496926+0000","last_unstale":"2023-12-19T08:03:33.501227+0000","last_undegraded":"2023-12-19T08:03:33.501227+0000","last_fullsized":"2023-12-19T08:03:33.501227+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:47:33.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.473052+0000","last_change":"2023-12-19T08:03:32.946645+0000","last_active":"2023-12-19T08:03:33.473052+0000","last_peered":"2023-12-19T08:03:33.473052+0000","last_clean":"2023-12-19T08:03:33.473052+0000","last_became_active":"2023-12-19T08:03:32.946315+0000","last_became_peered":"2023-12-19T08:03:32.946315+0000","last_unstale":"2023-12-19T08:03:33.473052+0000","last_undegraded":"2023-12-19T08:03:33.473052+0000","last_fullsized":"2023-12-19T08:03:33.473052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:16:08.683512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.494721+0000","last_change":"2023-12-19T08:03:33.491992+0000","last_active":"2023-12-19T08:03:33.494721+0000","last_peered":"2023-12-19T08:03:33.494721+0000","last_clean":"2023-12-19T08:03:33.494721+0000","last_became_active":"2023-12-19T08:03:33.491708+0000","last_became_peered":"2023-12-19T08:03:33.491708+0000","last_unstale":"2023-12-19T08:03:33.494721+0000","last_undegraded":"2023-12-19T08:03:33.494721+0000","last_fullsized":"2023-12-19T08:03:33.494721+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:59.545609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.951374+0000","last_change":"2023-12-19T08:03:32.947262+0000","last_active":"2023-12-19T08:03:33.951374+0000","last_peered":"2023-12-19T08:03:33.951374+0000","last_clean":"2023-12-19T08:03:33.951374+0000","last_became_active":"2023-12-19T08:03:32.946936+0000","last_became_peered":"2023-12-19T08:03:32.946936+0000","last_unstale":"2023-12-19T08:03:33.951374+0000","last_undegraded":"2023-12-19T08:03:33.951374+0000","last_fullsized":"2023-12-19T08:03:33.951374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:28.068462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498240+0000","last_change":"2023-12-19T08:03:33.486454+0000","last_active":"2023-12-19T08:03:33.498240+0000","last_peered":"2023-12-19T08:03:33.498240+0000","last_clean":"2023-12-19T08:03:33.498240+0000","last_became_active":"2023-12-19T08:03:33.485613+0000","last_became_peered":"2023-12-19T08:03:33.485613+0000","last_unstale":"2023-12-19T08:03:33.498240+0000","last_undegraded":"2023-12-19T08:03:33.498240+0000","last_fullsized":"2023-12-19T08:03:33.498240+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:10.094114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956284+0000","last_change":"2023-12-19T08:03:32.960500+0000","last_active":"2023-12-19T08:03:33.956284+0000","last_peered":"2023-12-19T08:03:33.956284+0000","last_clean":"2023-12-19T08:03:33.956284+0000","last_became_active":"2023-12-19T08:03:32.959932+0000","last_became_peered":"2023-12-19T08:03:32.959932+0000","last_unstale":"2023-12-19T08:03:33.956284+0000","last_undegraded":"2023-12-19T08:03:33.956284+0000","last_fullsized":"2023-12-19T08:03:33.956284+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:56.567271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490799+0000","last_change":"2023-12-19T08:03:33.488521+0000","last_active":"2023-12-19T08:03:33.490799+0000","last_peered":"2023-12-19T08:03:33.490799+0000","last_clean":"2023-12-19T08:03:33.490799+0000","last_became_active":"2023-12-19T08:03:33.487699+0000","last_became_peered":"2023-12-19T08:03:33.487699+0000","last_unstale":"2023-12-19T08:03:33.490799+0000","last_undegraded":"2023-12-19T08:03:33.490799+0000","last_fullsized":"2023-12-19T08:03:33.490799+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:08.878219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956421+0000","last_change":"2023-12-19T08:03:32.956306+0000","last_active":"2023-12-19T08:03:33.956421+0000","last_peered":"2023-12-19T08:03:33.956421+0000","last_clean":"2023-12-19T08:03:33.956421+0000","last_became_active":"2023-12-19T08:03:32.955688+0000","last_became_peered":"2023-12-19T08:03:32.955688+0000","last_unstale":"2023-12-19T08:03:33.956421+0000","last_undegraded":"2023-12-19T08:03:33.956421+0000","last_fullsized":"2023-12-19T08:03:33.956421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:12.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498754+0000","last_change":"2023-12-19T08:03:33.497007+0000","last_active":"2023-12-19T08:03:33.498754+0000","last_peered":"2023-12-19T08:03:33.498754+0000","last_clean":"2023-12-19T08:03:33.498754+0000","last_became_active":"2023-12-19T08:03:33.496664+0000","last_became_peered":"2023-12-19T08:03:33.496664+0000","last_unstale":"2023-12-19T08:03:33.498754+0000","last_undegraded":"2023-12-19T08:03:33.498754+0000","last_fullsized":"2023-12-19T08:03:33.498754+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"112'1","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.955182+0000","last_change":"2023-12-19T08:03:32.956036+0000","last_active":"2023-12-19T08:03:33.955182+0000","last_peered":"2023-12-19T08:03:33.955182+0000","last_clean":"2023-12-19T08:03:33.955182+0000","last_became_active":"2023-12-19T08:03:32.953950+0000","last_became_peered":"2023-12-19T08:03:32.953950+0000","last_unstale":"2023-12-19T08:03:33.955182+0000","last_undegraded":"2023-12-19T08:03:33.955182+0000","last_fullsized":"2023-12-19T08:03:33.955182+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:38:54.022009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499051+0000","last_change":"2023-12-19T08:03:33.485647+0000","last_active":"2023-12-19T08:03:33.499051+0000","last_peered":"2023-12-19T08:03:33.499051+0000","last_clean":"2023-12-19T08:03:33.499051+0000","last_became_active":"2023-12-19T08:03:33.484717+0000","last_became_peered":"2023-12-19T08:03:33.484717+0000","last_unstale":"2023-12-19T08:03:33.499051+0000","last_undegraded":"2023-12-19T08:03:33.499051+0000","last_fullsized":"2023-12-19T08:03:33.499051+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:04:56.590204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956145+0000","last_change":"2023-12-19T08:03:32.947216+0000","last_active":"2023-12-19T08:03:33.956145+0000","last_peered":"2023-12-19T08:03:33.956145+0000","last_clean":"2023-12-19T08:03:33.956145+0000","last_became_active":"2023-12-19T08:03:32.946904+0000","last_became_peered":"2023-12-19T08:03:32.946904+0000","last_unstale":"2023-12-19T08:03:33.956145+0000","last_undegraded":"2023-12-19T08:03:33.956145+0000","last_fullsized":"2023-12-19T08:03:33.956145+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:03.209463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490820+0000","last_change":"2023-12-19T08:03:33.488004+0000","last_active":"2023-12-19T08:03:33.490820+0000","last_peered":"2023-12-19T08:03:33.490820+0000","last_clean":"2023-12-19T08:03:33.490820+0000","last_became_active":"2023-12-19T08:03:33.487110+0000","last_became_peered":"2023-12-19T08:03:33.487110+0000","last_unstale":"2023-12-19T08:03:33.490820+0000","last_undegraded":"2023-12-19T08:03:33.490820+0000","last_fullsized":"2023-12-19T08:03:33.490820+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:51:17.407689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486272+0000","last_change":"2023-12-19T08:03:32.951521+0000","last_active":"2023-12-19T08:03:33.486272+0000","last_peered":"2023-12-19T08:03:33.486272+0000","last_clean":"2023-12-19T08:03:33.486272+0000","last_became_active":"2023-12-19T08:03:32.951121+0000","last_became_peered":"2023-12-19T08:03:32.951121+0000","last_unstale":"2023-12-19T08:03:33.486272+0000","last_undegraded":"2023-12-19T08:03:33.486272+0000","last_fullsized":"2023-12-19T08:03:33.486272+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:08:10.361201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502086+0000","last_change":"2023-12-19T08:03:33.501554+0000","last_active":"2023-12-19T08:03:33.502086+0000","last_peered":"2023-12-19T08:03:33.502086+0000","last_clean":"2023-12-19T08:03:33.502086+0000","last_became_active":"2023-12-19T08:03:33.500854+0000","last_became_peered":"2023-12-19T08:03:33.500854+0000","last_unstale":"2023-12-19T08:03:33.502086+0000","last_undegraded":"2023-12-19T08:03:33.502086+0000","last_fullsized":"2023-12-19T08:03:33.502086+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:38:31.023862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.477634+0000","last_change":"2023-12-19T08:03:32.954269+0000","last_active":"2023-12-19T08:03:33.477634+0000","last_peered":"2023-12-19T08:03:33.477634+0000","last_clean":"2023-12-19T08:03:33.477634+0000","last_became_active":"2023-12-19T08:03:32.953991+0000","last_became_peered":"2023-12-19T08:03:32.953991+0000","last_unstale":"2023-12-19T08:03:33.477634+0000","last_undegraded":"2023-12-19T08:03:33.477634+0000","last_fullsized":"2023-12-19T08:03:33.477634+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:21:49.416066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500235+0000","last_change":"2023-12-19T08:03:33.495729+0000","last_active":"2023-12-19T08:03:33.500235+0000","last_peered":"2023-12-19T08:03:33.500235+0000","last_clean":"2023-12-19T08:03:33.500235+0000","last_became_active":"2023-12-19T08:03:33.495455+0000","last_became_peered":"2023-12-19T08:03:33.495455+0000","last_unstale":"2023-12-19T08:03:33.500235+0000","last_undegraded":"2023-12-19T08:03:33.500235+0000","last_fullsized":"2023-12-19T08:03:33.500235+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:45.551623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.486312+0000","last_change":"2023-12-19T08:03:32.943826+0000","last_active":"2023-12-19T08:03:33.486312+0000","last_peered":"2023-12-19T08:03:33.486312+0000","last_clean":"2023-12-19T08:03:33.486312+0000","last_became_active":"2023-12-19T08:03:32.943509+0000","last_became_peered":"2023-12-19T08:03:32.943509+0000","last_unstale":"2023-12-19T08:03:33.486312+0000","last_undegraded":"2023-12-19T08:03:33.486312+0000","last_fullsized":"2023-12-19T08:03:33.486312+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:20.621987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490767+0000","last_change":"2023-12-19T08:03:33.488323+0000","last_active":"2023-12-19T08:03:33.490767+0000","last_peered":"2023-12-19T08:03:33.490767+0000","last_clean":"2023-12-19T08:03:33.490767+0000","last_became_active":"2023-12-19T08:03:33.488058+0000","last_became_peered":"2023-12-19T08:03:33.488058+0000","last_unstale":"2023-12-19T08:03:33.490767+0000","last_undegraded":"2023-12-19T08:03:33.490767+0000","last_fullsized":"2023-12-19T08:03:33.490767+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:48.377087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"112'2","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.956833+0000","last_change":"2023-12-19T08:03:32.959707+0000","last_active":"2023-12-19T08:03:33.956833+0000","last_peered":"2023-12-19T08:03:33.956833+0000","last_clean":"2023-12-19T08:03:33.956833+0000","last_became_active":"2023-12-19T08:03:32.959376+0000","last_became_peered":"2023-12-19T08:03:32.959376+0000","last_unstale":"2023-12-19T08:03:33.956833+0000","last_undegraded":"2023-12-19T08:03:33.956833+0000","last_fullsized":"2023-12-19T08:03:33.956833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:44:14.378543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498150+0000","last_change":"2023-12-19T08:03:33.482782+0000","last_active":"2023-12-19T08:03:33.498150+0000","last_peered":"2023-12-19T08:03:33.498150+0000","last_clean":"2023-12-19T08:03:33.498150+0000","last_became_active":"2023-12-19T08:03:33.480963+0000","last_became_peered":"2023-12-19T08:03:33.480963+0000","last_unstale":"2023-12-19T08:03:33.498150+0000","last_undegraded":"2023-12-19T08:03:33.498150+0000","last_fullsized":"2023-12-19T08:03:33.498150+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:22:44.764760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475117+0000","last_change":"2023-12-19T08:03:32.947496+0000","last_active":"2023-12-19T08:03:33.475117+0000","last_peered":"2023-12-19T08:03:33.475117+0000","last_clean":"2023-12-19T08:03:33.475117+0000","last_became_active":"2023-12-19T08:03:32.947206+0000","last_became_peered":"2023-12-19T08:03:32.947206+0000","last_unstale":"2023-12-19T08:03:33.475117+0000","last_undegraded":"2023-12-19T08:03:33.475117+0000","last_fullsized":"2023-12-19T08:03:33.475117+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:59.743549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.500192+0000","last_change":"2023-12-19T08:03:33.492167+0000","last_active":"2023-12-19T08:03:33.500192+0000","last_peered":"2023-12-19T08:03:33.500192+0000","last_clean":"2023-12-19T08:03:33.500192+0000","last_became_active":"2023-12-19T08:03:33.491882+0000","last_became_peered":"2023-12-19T08:03:33.491882+0000","last_unstale":"2023-12-19T08:03:33.500192+0000","last_undegraded":"2023-12-19T08:03:33.500192+0000","last_fullsized":"2023-12-19T08:03:33.500192+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:43.966569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475078+0000","last_change":"2023-12-19T08:03:32.951747+0000","last_active":"2023-12-19T08:03:33.475078+0000","last_peered":"2023-12-19T08:03:33.475078+0000","last_clean":"2023-12-19T08:03:33.475078+0000","last_became_active":"2023-12-19T08:03:32.950473+0000","last_became_peered":"2023-12-19T08:03:32.950473+0000","last_unstale":"2023-12-19T08:03:33.475078+0000","last_undegraded":"2023-12-19T08:03:33.475078+0000","last_fullsized":"2023-12-19T08:03:33.475078+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:02.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490849+0000","last_change":"2023-12-19T08:03:33.487630+0000","last_active":"2023-12-19T08:03:33.490849+0000","last_peered":"2023-12-19T08:03:33.490849+0000","last_clean":"2023-12-19T08:03:33.490849+0000","last_became_active":"2023-12-19T08:03:33.487236+0000","last_became_peered":"2023-12-19T08:03:33.487236+0000","last_unstale":"2023-12-19T08:03:33.490849+0000","last_undegraded":"2023-12-19T08:03:33.490849+0000","last_fullsized":"2023-12-19T08:03:33.490849+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:37:31.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.482614+0000","last_change":"2023-12-19T08:03:32.955468+0000","last_active":"2023-12-19T08:03:33.482614+0000","last_peered":"2023-12-19T08:03:33.482614+0000","last_clean":"2023-12-19T08:03:33.482614+0000","last_became_active":"2023-12-19T08:03:32.955003+0000","last_became_peered":"2023-12-19T08:03:32.955003+0000","last_unstale":"2023-12-19T08:03:33.482614+0000","last_undegraded":"2023-12-19T08:03:33.482614+0000","last_fullsized":"2023-12-19T08:03:33.482614+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:59:34.316305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.499572+0000","last_change":"2023-12-19T08:03:33.494746+0000","last_active":"2023-12-19T08:03:33.499572+0000","last_peered":"2023-12-19T08:03:33.499572+0000","last_clean":"2023-12-19T08:03:33.499572+0000","last_became_active":"2023-12-19T08:03:33.494277+0000","last_became_peered":"2023-12-19T08:03:33.494277+0000","last_unstale":"2023-12-19T08:03:33.499572+0000","last_undegraded":"2023-12-19T08:03:33.499572+0000","last_fullsized":"2023-12-19T08:03:33.499572+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:22:23.439986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.475710+0000","last_change":"2023-12-19T08:03:32.944458+0000","last_active":"2023-12-19T08:03:33.475710+0000","last_peered":"2023-12-19T08:03:33.475710+0000","last_clean":"2023-12-19T08:03:33.475710+0000","last_became_active":"2023-12-19T08:03:32.942038+0000","last_became_peered":"2023-12-19T08:03:32.942038+0000","last_unstale":"2023-12-19T08:03:33.475710+0000","last_undegraded":"2023-12-19T08:03:33.475710+0000","last_fullsized":"2023-12-19T08:03:33.475710+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:47.360353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.490584+0000","last_change":"2023-12-19T08:03:33.488638+0000","last_active":"2023-12-19T08:03:33.490584+0000","last_peered":"2023-12-19T08:03:33.490584+0000","last_clean":"2023-12-19T08:03:33.490584+0000","last_became_active":"2023-12-19T08:03:33.488060+0000","last_became_peered":"2023-12-19T08:03:33.488060+0000","last_unstale":"2023-12-19T08:03:33.490584+0000","last_undegraded":"2023-12-19T08:03:33.490584+0000","last_fullsized":"2023-12-19T08:03:33.490584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:24:50.998299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.478802+0000","last_change":"2023-12-19T08:03:32.952935+0000","last_active":"2023-12-19T08:03:33.478802+0000","last_peered":"2023-12-19T08:03:33.478802+0000","last_clean":"2023-12-19T08:03:33.478802+0000","last_became_active":"2023-12-19T08:03:32.952655+0000","last_became_peered":"2023-12-19T08:03:32.952655+0000","last_unstale":"2023-12-19T08:03:33.478802+0000","last_undegraded":"2023-12-19T08:03:33.478802+0000","last_fullsized":"2023-12-19T08:03:33.478802+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:56.567271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.502254+0000","last_change":"2023-12-19T08:03:33.501262+0000","last_active":"2023-12-19T08:03:33.502254+0000","last_peered":"2023-12-19T08:03:33.502254+0000","last_clean":"2023-12-19T08:03:33.502254+0000","last_became_active":"2023-12-19T08:03:33.500905+0000","last_became_peered":"2023-12-19T08:03:33.500905+0000","last_unstale":"2023-12-19T08:03:33.502254+0000","last_undegraded":"2023-12-19T08:03:33.502254+0000","last_fullsized":"2023-12-19T08:03:33.502254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:30:13.422943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.484229+0000","last_change":"2023-12-19T08:03:32.955341+0000","last_active":"2023-12-19T08:03:33.484229+0000","last_peered":"2023-12-19T08:03:33.484229+0000","last_clean":"2023-12-19T08:03:33.484229+0000","last_became_active":"2023-12-19T08:03:32.954966+0000","last_became_peered":"2023-12-19T08:03:32.954966+0000","last_unstale":"2023-12-19T08:03:33.484229+0000","last_undegraded":"2023-12-19T08:03:33.484229+0000","last_fullsized":"2023-12-19T08:03:33.484229+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:11:44.588253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.498287+0000","last_change":"2023-12-19T08:03:33.489139+0000","last_active":"2023-12-19T08:03:33.498287+0000","last_peered":"2023-12-19T08:03:33.498287+0000","last_clean":"2023-12-19T08:03:33.498287+0000","last_became_active":"2023-12-19T08:03:33.488848+0000","last_became_peered":"2023-12-19T08:03:33.488848+0000","last_unstale":"2023-12-19T08:03:33.498287+0000","last_undegraded":"2023-12-19T08:03:33.498287+0000","last_fullsized":"2023-12-19T08:03:33.498287+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:20:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476286+0000","last_change":"2023-12-19T08:03:32.957401+0000","last_active":"2023-12-19T08:03:33.476286+0000","last_peered":"2023-12-19T08:03:33.476286+0000","last_clean":"2023-12-19T08:03:33.476286+0000","last_became_active":"2023-12-19T08:03:32.957114+0000","last_became_peered":"2023-12-19T08:03:32.957114+0000","last_unstale":"2023-12-19T08:03:33.476286+0000","last_undegraded":"2023-12-19T08:03:33.476286+0000","last_fullsized":"2023-12-19T08:03:33.476286+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:48:52.259196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.497181+0000","last_change":"2023-12-19T08:03:33.493629+0000","last_active":"2023-12-19T08:03:33.497181+0000","last_peered":"2023-12-19T08:03:33.497181+0000","last_clean":"2023-12-19T08:03:33.497181+0000","last_became_active":"2023-12-19T08:03:33.493301+0000","last_became_peered":"2023-12-19T08:03:33.493301+0000","last_unstale":"2023-12-19T08:03:33.497181+0000","last_undegraded":"2023-12-19T08:03:33.497181+0000","last_fullsized":"2023-12-19T08:03:33.497181+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:32.903120+0000","last_clean_scrub_stamp":"2023-12-19T08:03:32.903120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:52.314950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.473015+0000","last_change":"2023-12-19T08:03:32.951471+0000","last_active":"2023-12-19T08:03:33.473015+0000","last_peered":"2023-12-19T08:03:33.473015+0000","last_clean":"2023-12-19T08:03:33.473015+0000","last_became_active":"2023-12-19T08:03:32.951146+0000","last_became_peered":"2023-12-19T08:03:32.951146+0000","last_unstale":"2023-12-19T08:03:33.473015+0000","last_undegraded":"2023-12-19T08:03:33.473015+0000","last_fullsized":"2023-12-19T08:03:33.473015+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:03:31.886386+0000","last_clean_scrub_stamp":"2023-12-19T08:03:31.886386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:09:02.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"101'205","reported_seq":551,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-12-19T08:03:33.476503+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:03:33.476503+0000","last_peered":"2023-12-19T08:03:33.476503+0000","last_clean":"2023-12-19T08:03:33.476503+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:03:33.476503+0000","last_undegraded":"2023-12-19T08:03:33.476503+0000","last_fullsized":"2023-12-19T08:03:33.476503+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311512,"kb_used_data":1936,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060328,"statfs":{"total":96636764160,"available":96317775872,"internally_reserved":0,"allocated":1982464,"data_stored":1305093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.218,"15min":1.246},"min":{"1min":0.751,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.615,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.135},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.298,"15min":1.2729999999999999},"min":{"1min":0.83199999999999996,"5min":0.749,"15min":0.70799999999999996},"max":{"1min":1.625,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.625}]},{"osd":1,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1859999999999999,"15min":1.169},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.526,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.169,"15min":1.1739999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.843,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.0289999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1919999999999999,"15min":1.1699999999999999},"min":{"1min":0.81499999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.512,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.288},{"interface":"front","average":{"1min":1.226,"5min":1.2609999999999999,"15min":1.2430000000000001},"min":{"1min":0.82999999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8799999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.2450000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2889999999999999,"15min":1.2909999999999999},"min":{"1min":0.92400000000000004,"5min":0.83299999999999996,"15min":0.73199999999999998},"max":{"1min":1.6499999999999999,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.381,"15min":1.3500000000000001},"min":{"1min":0.88400000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8049999999999999,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.5149999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3069999999999999,"15min":1.2310000000000001},"min":{"1min":0.86699999999999999,"5min":0.67900000000000005,"15min":0.67300000000000004},"max":{"1min":1.9570000000000001,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.458},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3169999999999999,"15min":1.2470000000000001},"min":{"1min":0.76800000000000002,"5min":0.76500000000000001,"15min":0.64900000000000002},"max":{"1min":1.6699999999999999,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.415}]},{"osd":5,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3129999999999999,"15min":1.242},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.9139999999999999,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.417,"15min":1.361},"min":{"1min":0.94799999999999995,"5min":0.751,"15min":0.751},"max":{"1min":2.036,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.585}]},{"osd":6,"last update":"Tue Dec 19 08:03:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.5009999999999999,"15min":1.4219999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.996,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.748},{"interface":"front","average":{"1min":1.393,"5min":1.4319999999999999,"15min":1.3859999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.58299999999999996},"max":{"1min":2.1400000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.5589999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302764,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069076,"statfs":{"total":96636764160,"available":96326733824,"internally_reserved":0,"allocated":1413120,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.071,"15min":1.103},"min":{"1min":0.75800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.472,"5min":1.5569999999999999,"15min":1.665},"last":1.198},{"interface":"front","average":{"1min":1.109,"5min":1.113,"15min":1.1279999999999999},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.373,"5min":1.756,"15min":1.756},"last":1.429}]},{"osd":1,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1279999999999999,"15min":1.161},"min":{"1min":0.75800000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4770000000000001,"5min":1.675,"15min":1.881},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.073,"5min":1.161,"15min":1.161},"min":{"1min":0.745,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4279999999999999,"5min":1.542,"15min":1.7989999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2150000000000001,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.675,"5min":1.847,"15min":1.847},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.1970000000000001,"15min":1.2130000000000001},"min":{"1min":0.70799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4259999999999999,"5min":1.694,"15min":1.7929999999999999},"last":1.2270000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2450000000000001,"15min":1.2669999999999999},"min":{"1min":0.77600000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0659999999999998},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.3,"15min":1.3100000000000001},"min":{"1min":0.80900000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7889999999999999,"5min":1.9570000000000001,"15min":2.1760000000000002},"last":1.2909999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1990000000000001,"15min":1.1839999999999999},"min":{"1min":0.80900000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5900000000000001,"5min":1.7130000000000001,"15min":2.137},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.387,"15min":1.405},"min":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.629,"5min":1.974,"15min":2.218},"last":1.546}]},{"osd":5,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3120000000000001,"15min":1.2929999999999999},"min":{"1min":0.78300000000000003,"5min":0.67600000000000005,"15min":0.67500000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3109999999999999},"min":{"1min":0.83999999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.663,"5min":1.8100000000000001,"15min":2.3050000000000002},"last":1.337}]},{"osd":7,"last update":"Tue Dec 19 08:02:44 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.93999999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":2.2029999999999998},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.448,"15min":1.425},"min":{"1min":1.1379999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8919999999999999,"5min":2.137,"15min":2.2639999999999998},"last":1.4039999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298648,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073192,"statfs":{"total":96636764160,"available":96330948608,"internally_reserved":0,"allocated":1392640,"data_stored":721797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.67900000000000005,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.546,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.2410000000000001,"15min":1.264},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.50700000000000001},"max":{"1min":1.5940000000000001,"5min":1.954,"15min":2.3140000000000001},"last":1.27}]},{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.103,"15min":1.1419999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.341,"5min":1.585,"15min":3.5409999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.77500000000000002,"5min":0.72799999999999998,"15min":0.68899999999999995},"max":{"1min":1.647,"5min":2.7679999999999998,"15min":3.113},"last":1.3460000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1759999999999999,"15min":1.2010000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66000000000000003},"max":{"1min":1.6739999999999999,"5min":1.821,"15min":3.339},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.175,"15min":1.2110000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4279999999999999,"5min":1.6679999999999999,"15min":2.9039999999999999},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.228,"15min":1.2569999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.4630000000000001,"5min":2.9489999999999998,"15min":3.5870000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.93400000000000005,"5min":0.88400000000000001,"15min":0.58199999999999996},"max":{"1min":1.6359999999999999,"5min":2.1560000000000001,"15min":4.3879999999999999},"last":1.619}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.202,"15min":1.222},"min":{"1min":0.80500000000000005,"5min":0.629,"15min":0.52600000000000002},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":5.0819999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.2629999999999999,"15min":1.2929999999999999},"min":{"1min":0.96899999999999997,"5min":0.70299999999999996,"15min":0.59199999999999997},"max":{"1min":1.621,"5min":2.843,"15min":4.6310000000000002},"last":1.5840000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.81499999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.728,"5min":2.0190000000000001,"15min":3.9649999999999999},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.367,"15min":1.3700000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.70599999999999996},"max":{"1min":1.96,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.6359999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3939999999999999,"15min":1.427},"min":{"1min":0.88,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.6739999999999999,"5min":2.9929999999999999,"15min":4.4539999999999997},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.46,"15min":1.462},"min":{"1min":1.038,"5min":0.96499999999999997,"15min":0.67800000000000005},"max":{"1min":1.8200000000000001,"5min":2.0779999999999998,"15min":4.1980000000000004},"last":1.819}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302752,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069088,"statfs":{"total":96636764160,"available":96326746112,"internally_reserved":0,"allocated":1400832,"data_stored":721957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1950000000000001,"15min":1.214},"min":{"1min":0.70499999999999996,"5min":0.63400000000000001,"15min":0.59599999999999997},"max":{"1min":1.4490000000000001,"5min":1.6279999999999999,"15min":2.4900000000000002},"last":1.419},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.94999999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.742,"5min":1.742,"15min":2.738},"last":1.383}]},{"osd":1,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.159,"15min":1.177},"min":{"1min":0.78600000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.556,"5min":1.7589999999999999,"15min":2.589},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1060000000000001,"15min":1.143},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.54300000000000004},"max":{"1min":1.407,"5min":1.528,"15min":2.2589999999999999},"last":0.86599999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.085,"15min":1.1359999999999999},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.411,"5min":1.5149999999999999,"15min":2.4569999999999999},"last":1.196},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.2350000000000001},"min":{"1min":0.89000000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.524,"5min":1.8959999999999999,"15min":2.8050000000000002},"last":1.121}]},{"osd":3,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2509999999999999,"15min":1.2829999999999999},"min":{"1min":0.70999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.5980000000000001,"5min":1.7050000000000001,"15min":3.048},"last":1.151},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2969999999999999,"15min":1.3360000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.81899999999999995},"max":{"1min":1.5780000000000001,"5min":1.8580000000000001,"15min":2.661},"last":1.472}]},{"osd":5,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1739999999999999,"15min":1.2170000000000001},"min":{"1min":0.625,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5549999999999999,"5min":1.8149999999999999,"15min":3.556},"last":1.26},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.2150000000000001},"min":{"1min":0.65600000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.6850000000000001,"5min":1.764,"15min":3.2599999999999998},"last":1.242}]},{"osd":6,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.2350000000000001,"15min":1.2509999999999999},"min":{"1min":0.73899999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.7350000000000001,"5min":1.748,"15min":2.645},"last":1.516},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.379,"15min":1.3839999999999999},"min":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.67900000000000005},"max":{"1min":1.6299999999999999,"5min":2.0049999999999999,"15min":2.923},"last":1.4410000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:02:35 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3580000000000001},"min":{"1min":0.82699999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.7410000000000001,"5min":1.8200000000000001,"15min":3.1850000000000001},"last":1.333},{"interface":"front","average":{"1min":1.337,"5min":1.3859999999999999,"15min":1.411},"min":{"1min":0.80200000000000005,"5min":0.73099999999999998,"15min":0.70999999999999996},"max":{"1min":1.829,"5min":1.966,"15min":2.8460000000000001},"last":1.637}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306828,"kb_used_data":1348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065012,"statfs":{"total":96636764160,"available":96322572288,"internally_reserved":0,"allocated":1380352,"data_stored":714669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.84599999999999997,"15min":0.85099999999999998},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.1359999999999999,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94199999999999995,"15min":0.95199999999999996},"min":{"1min":0.61699999999999999,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.518,"5min":1.962,"15min":1.962},"last":1.0589999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.89900000000000002,"15min":0.91300000000000003},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.45600000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.0209999999999999,"15min":1.0289999999999999},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.694,"5min":2.423,"15min":2.423},"last":0.84299999999999997}]},{"osd":2,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0580000000000001,"15min":1.0720000000000001},"min":{"1min":0.59999999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.6910000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.879},{"interface":"front","average":{"1min":1.157,"5min":1.1539999999999999,"15min":1.1519999999999999},"min":{"1min":0.68799999999999994,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.599,"5min":2.48,"15min":2.48},"last":0.92800000000000005}]},{"osd":4,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2370000000000001,"15min":1.254},"min":{"1min":0.84299999999999997,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.6120000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.407},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.22,"15min":1.242},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.649,"5min":1.649,"15min":1.8120000000000001},"last":1.379}]},{"osd":5,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.2609999999999999,"15min":1.268},"min":{"1min":0.90300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.7569999999999999,"5min":1.996,"15min":1.996},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.331,"15min":1.3280000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.65800000000000003},"max":{"1min":1.821,"5min":2.161,"15min":2.161},"last":1.022}]},{"osd":6,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.3959999999999999,"15min":1.381},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4830000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.339,"15min":1.3600000000000001},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.79100000000000004},"max":{"1min":1.6890000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.282}]},{"osd":7,"last update":"Tue Dec 19 08:03:31 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4490000000000001,"15min":1.448},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.84799999999999998},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.421,"15min":1.411},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.4359999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306840,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065000,"statfs":{"total":96636764160,"available":96322560000,"internally_reserved":0,"allocated":1392640,"data_stored":721797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89100000000000001,"15min":0.90600000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.437},"max":{"1min":1.353,"5min":1.776,"15min":1.776},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.012,"15min":1.0529999999999999},"min":{"1min":0.70399999999999996,"5min":0.60399999999999998,"15min":0.52200000000000002},"max":{"1min":1.2669999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96599999999999997,"15min":0.95299999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.51800000000000002},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.98599999999999999,"15min":1.0109999999999999},"min":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.528,"5min":1.835,"15min":1.835},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.96499999999999997},"min":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.3220000000000001,"5min":1.657,"15min":1.667},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.181,"5min":1.103,"15min":1.113},"min":{"1min":0.745,"5min":0.65900000000000003,"15min":0.57699999999999996},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.335,"15min":1.323},"min":{"1min":0.98099999999999998,"5min":0.83699999999999997,"15min":0.79300000000000004},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.2929999999999999,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.82099999999999995},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.984},"last":1.2929999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.341},"min":{"1min":0.95799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9610000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.417},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2629999999999999,"15min":1.2509999999999999},"min":{"1min":0.91400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.8,"5min":1.8,"15min":1.8},"last":1.5369999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2929999999999999,"15min":1.294},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.345},{"interface":"front","average":{"1min":1.375,"5min":1.3640000000000001,"15min":1.365},"min":{"1min":1.0129999999999999,"5min":0.95999999999999996,"15min":0.85399999999999998},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.5780000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3600000000000001,"15min":1.385},"min":{"1min":1.0509999999999999,"5min":0.95899999999999996,"15min":0.83699999999999997},"max":{"1min":1.8620000000000001,"5min":1.9730000000000001,"15min":2.214},"last":1.145},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3440000000000001,"15min":1.365},"min":{"1min":0.98699999999999999,"5min":0.97399999999999998,"15min":0.88200000000000001},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.242}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306832,"kb_used_data":1352,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94065008,"statfs":{"total":96636764160,"available":96322568192,"internally_reserved":0,"allocated":1384448,"data_stored":714739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89100000000000001,"15min":0.86099999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.647,"5min":1.661,"15min":1.661},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.41199999999999998},"max":{"1min":1.284,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.999}]},{"osd":2,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94199999999999995,"15min":0.92300000000000004},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.4670000000000001,"5min":2.96,"15min":2.96},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1020000000000001,"15min":1.0609999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.389,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.357}]},{"osd":3,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.21,"15min":1.161},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.571,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.032},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.097,"15min":1.0600000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.46800000000000003},"max":{"1min":1.524,"5min":2.407,"15min":2.407},"last":0.86099999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.373,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.3480000000000001,"15min":1.3089999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.623,"5min":3.04,"15min":3.04},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.302,"15min":1.278},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.65100000000000002},"max":{"1min":1.837,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.837},{"interface":"front","average":{"1min":1.24,"5min":1.3140000000000001,"15min":1.2849999999999999},"min":{"1min":0.88200000000000001,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.8049999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.8049999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.417,"15min":1.381},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.696,"5min":3.089,"15min":3.089},"last":1.661},{"interface":"front","average":{"1min":1.333,"5min":1.3879999999999999,"15min":1.359},"min":{"1min":0.873,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.881,"5min":2.46,"15min":2.46},"last":1.881}]},{"osd":7,"last update":"Tue Dec 19 08:03:34 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.431,"15min":1.4139999999999999},"min":{"1min":1.018,"5min":0.95699999999999996,"15min":0.83199999999999996},"max":{"1min":1.7729999999999999,"5min":2.504,"15min":2.504},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.391,"15min":1.377},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.6899999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.6899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307440,"kb_used_data":1960,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064400,"statfs":{"total":96636764160,"available":96321945600,"internally_reserved":0,"allocated":2007040,"data_stored":1313368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83699999999999997,"15min":0.86299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.262,"5min":1.5409999999999999,"15min":5.8129999999999997},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.91600000000000004,"15min":0.93799999999999994},"min":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.379},"max":{"1min":1.385,"5min":1.385,"15min":6.0309999999999997},"last":0.76700000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.95999999999999996,"15min":0.97999999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.044,"15min":1.03},"min":{"1min":0.75700000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3700000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.96299999999999997}]},{"osd":3,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.96099999999999997},"min":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.51400000000000001},"max":{"1min":1.48,"5min":1.5529999999999999,"15min":1.671},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.115,"15min":1.129},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.514,"5min":1.663,"15min":1.6719999999999999},"last":1.252}]},{"osd":4,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3129999999999999,"15min":1.3080000000000001},"min":{"1min":0.86499999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6000000000000001,"5min":1.8620000000000001,"15min":2.1669999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.76100000000000001},"max":{"1min":1.5840000000000001,"5min":1.855,"15min":2.141},"last":1.268}]},{"osd":5,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.351,"15min":1.3600000000000001},"min":{"1min":0.83399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.6819999999999999,"5min":2.085,"15min":2.117},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.357,"15min":1.361},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.843,"5min":2.125,"15min":2.125},"last":1.45}]},{"osd":6,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.276,"15min":1.268},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.77400000000000002},"max":{"1min":1.5329999999999999,"5min":1.97,"15min":1.97},"last":1.331},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3200000000000001,"15min":1.3220000000000001},"min":{"1min":0.91900000000000004,"5min":0.75600000000000001,"15min":0.69299999999999995},"max":{"1min":1.712,"5min":1.871,"15min":1.9530000000000001},"last":1.544}]},{"osd":7,"last update":"Tue Dec 19 08:02:43 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.341,"15min":1.3240000000000001},"min":{"1min":0.97799999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.8009999999999999,"5min":1.887,"15min":2.0539999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3720000000000001,"15min":1.3759999999999999},"min":{"1min":0.88600000000000001,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.714,"5min":2.0059999999999998,"15min":2.016},"last":1.397}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:03:38.673 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:03:38.673 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:03:38.880 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T08:03:38.876+0000 7fd187df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:03:38.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.880+0000 7f6a13af0700 1 Processor -- start 2023-12-19T08:03:38.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 -- start start 2023-12-19T08:03:38.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 0x7f6a0c06b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6a0c06b5d0 0x7f6a0c0b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0aff50 con 0x7f6a0c155420 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0c0c95b0 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.884+0000 7f6a13af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c06ba50 con 0x7f6a0c1557f0 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1208d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6a0c06b5d0 0x7f6a0c0b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 0x7f6a0c06b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1188c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6a0c155420 0x7f6a0c06da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34372/0 (socket says 172.21.15.111:34372) 2023-12-19T08:03:38.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1188c700 1 -- 172.21.15.111:0/3294813720 learned_addr learned my addr 172.21.15.111:0/3294813720 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:38.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 -- 172.21.15.111:0/3294813720 >> v2:172.21.15.111:3300/0 conn(0x7f6a0c06b5d0 msgr2=0x7f6a0c0b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:38.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 --2- 172.21.15.111:0/3294813720 >> v2:172.21.15.111:3300/0 conn(0x7f6a0c06b5d0 0x7f6a0c0b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 -- 172.21.15.111:0/3294813720 >> v1:172.21.15.111:6789/0 conn(0x7f6a0c155420 legacy=0x7f6a0c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 -- 172.21.15.111:0/3294813720 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c0af380 con 0x7f6a0c1557f0 2023-12-19T08:03:38.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1108b700 1 --2- 172.21.15.111:0/3294813720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 0x7f6a0c06b090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0098e0 tx=0x7f69fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5ea4912b94d5ff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1088a700 1 -- 172.21.15.111:0/3294813720 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00b640 con 0x7f6a0c1557f0 2023-12-19T08:03:38.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1088a700 1 -- 172.21.15.111:0/3294813720 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc00b7a0 con 0x7f6a0c1557f0 2023-12-19T08:03:38.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a1088a700 1 -- 172.21.15.111:0/3294813720 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00bb40 con 0x7f6a0c1557f0 2023-12-19T08:03:38.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 -- 172.21.15.111:0/3294813720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 msgr2=0x7f6a0c06b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 --2- 172.21.15.111:0/3294813720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 0x7f6a0c06b090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0098e0 tx=0x7f69fc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 -- 172.21.15.111:0/3294813720 shutdown_connections 2023-12-19T08:03:38.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 --2- 172.21.15.111:0/3294813720 >> v2:172.21.15.111:3300/0 conn(0x7f6a0c06b5d0 0x7f6a0c0b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 --2- 172.21.15.111:0/3294813720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c1557f0 0x7f6a0c06b090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 -- 172.21.15.111:0/3294813720 >> 172.21.15.111:0/3294813720 conn(0x7f6a0c0cc580 msgr2=0x7f6a0c0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:38.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 -- 172.21.15.111:0/3294813720 shutdown_connections 2023-12-19T08:03:38.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.888+0000 7f6a13af0700 1 -- 172.21.15.111:0/3294813720 wait complete. 2023-12-19T08:03:38.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 Processor -- start 2023-12-19T08:03:38.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 -- start start 2023-12-19T08:03:38.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c155420 0x7f6a0c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c1557f0 0x7f6a0c151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c0cb720 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c0c9d00 con 0x7f6a0c1557f0 2023-12-19T08:03:38.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a13af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c0c9fe0 con 0x7f6a0c155420 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45448/0 (socket says 172.21.15.111:45448) 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 -- 172.21.15.111:0/298792641 learned_addr learned my addr 172.21.15.111:0/298792641 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1208d700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c1557f0 0x7f6a0c151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1108b700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c155420 0x7f6a0c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 -- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c155420 msgr2=0x7f6a0c0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c155420 0x7f6a0c0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 -- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c1557f0 msgr2=0x7f6a0c151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.892+0000 7f6a1188c700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c1557f0 0x7f6a0c151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:38.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a1188c700 1 -- 172.21.15.111:0/298792641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc009420 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a1188c700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c7c0 tx=0x7f6a08007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08005b30 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a08005c90 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08004a80 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0c0ca2c0 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0c0a8850 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0c071120 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.896+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6a080070d0 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.900+0000 7f6a01ffb700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 0x7f69f0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:38.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.900+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f6a0807bf70 con 0x7f6a0c06b5d0 2023-12-19T08:03:38.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.900+0000 7f6a1108b700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 0x7f69f0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:38.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.900+0000 7f6a1108b700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 0x7f69f0062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0093f0 tx=0x7f69fc017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:38.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:38.904+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6a08049be0 con 0x7f6a0c06b5d0 2023-12-19T08:03:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.132+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6a0c0af620 con 0x7f6a0c06b5d0 2023-12-19T08:03:39.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.144+0000 7f6a01ffb700 1 -- 172.21.15.111:0/298792641 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6a0804fb90 con 0x7f6a0c06b5d0 2023-12-19T08:03:39.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 msgr2=0x7f69f0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 0x7f69f0062b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0093f0 tx=0x7f69fc017000 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 msgr2=0x7f6a0c0c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c7c0 tx=0x7f6a08007f70 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 shutdown_connections 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f69f00606a0 0x7f69f0062b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a0c1557f0 0x7f6a0c151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a0c155420 0x7f6a0c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 --2- 172.21.15.111:0/298792641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a0c06b5d0 0x7f6a0c0c9720 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 >> 172.21.15.111:0/298792641 conn(0x7f6a0c0cc580 msgr2=0x7f6a0c0ab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 shutdown_connections 2023-12-19T08:03:39.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.148+0000 7f6a13af0700 1 -- 172.21.15.111:0/298792641 wait complete. 2023-12-19T08:03:39.155 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:03:39.174 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:03:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.328+0000 7f865dccd700 1 Processor -- start 2023-12-19T08:03:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.328+0000 7f865dccd700 1 -- start start 2023-12-19T08:03:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8658140710 0x7f86580c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8658140ae0 0x7f86580680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86580c75e0 0x7f8658068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865806e150 con 0x7f8658140ae0 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865806d540 con 0x7f86580c75e0 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865806d820 con 0x7f8658140710 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8658140ae0 0x7f86580680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8658140ae0 0x7f86580680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45452/0 (socket says 172.21.15.111:45452) 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 -- 172.21.15.111:0/2462116301 learned_addr learned my addr 172.21.15.111:0/2462116301 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f86577fe700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8658140710 0x7f86580c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f8657fff700 1 --2- 172.21.15.111:0/2462116301 >> v2:172.21.15.143:3300/0 conn(0x7f86580c75e0 0x7f8658068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 -- 172.21.15.111:0/2462116301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8658140710 msgr2=0x7f86580c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8658140710 0x7f86580c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 -- 172.21.15.111:0/2462116301 >> v2:172.21.15.143:3300/0 conn(0x7f86580c75e0 msgr2=0x7f8658068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 --2- 172.21.15.111:0/2462116301 >> v2:172.21.15.143:3300/0 conn(0x7f86580c75e0 0x7f8658068630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 -- 172.21.15.111:0/2462116301 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865806db00 con 0x7f8658140ae0 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f864effd700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580680f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f86580ae930 tx=0x7f8640009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69aa148d13afcc63 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f8656ffd700 1 -- 172.21.15.111:0/2462116301 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000b640 con 0x7f8658140ae0 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f8656ffd700 1 -- 172.21.15.111:0/2462116301 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864000b7a0 con 0x7f8658140ae0 2023-12-19T08:03:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f8656ffd700 1 -- 172.21.15.111:0/2462116301 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000bb60 con 0x7f8658140ae0 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- 172.21.15.111:0/2462116301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 msgr2=0x7f86580680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580680f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f86580ae930 tx=0x7f8640009250 comp rx=0 tx=0).stop 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- 172.21.15.111:0/2462116301 shutdown_connections 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- 172.21.15.111:0/2462116301 >> v2:172.21.15.143:3300/0 conn(0x7f86580c75e0 0x7f8658068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580680f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 --2- 172.21.15.111:0/2462116301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8658140710 0x7f86580c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- 172.21.15.111:0/2462116301 >> 172.21.15.111:0/2462116301 conn(0x7f86580c8850 msgr2=0x7f86580c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- 172.21.15.111:0/2462116301 shutdown_connections 2023-12-19T08:03:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.332+0000 7f865dccd700 1 -- 172.21.15.111:0/2462116301 wait complete. 2023-12-19T08:03:39.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 Processor -- start 2023-12-19T08:03:39.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 -- start start 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 0x7f8658156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8658140710 0x7f865813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865813c270 con 0x7f8658140ae0 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865813b660 con 0x7f8658140710 2023-12-19T08:03:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.336+0000 7f865dccd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865813b940 con 0x7f86580c75e0 2023-12-19T08:03:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45460/0 (socket says 172.21.15.111:45460) 2023-12-19T08:03:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 -- 172.21.15.111:0/2452143365 learned_addr learned my addr 172.21.15.111:0/2452143365 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f864effd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8658140710 0x7f865813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f86577fe700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 0x7f8658156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 -- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 msgr2=0x7f8658156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 0x7f8658156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 -- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8658140710 msgr2=0x7f865813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8658140710 0x7f865813acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 -- 172.21.15.111:0/2452143365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400094f0 con 0x7f8658140ae0 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f86577fe700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 0x7f8658156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f8657fff700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f8650009df0 tx=0x7f865000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865000cbe0 con 0x7f8658140ae0 2023-12-19T08:03:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865813bc20 con 0x7f8658140ae0 2023-12-19T08:03:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000f040 con 0x7f8658140ae0 2023-12-19T08:03:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650014480 con 0x7f8658140ae0 2023-12-19T08:03:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8658156ec0 con 0x7f8658140ae0 2023-12-19T08:03:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.340+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8658071120 con 0x7f8658140ae0 2023-12-19T08:03:39.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.344+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8650014830 con 0x7f8658140ae0 2023-12-19T08:03:39.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.344+0000 7f864ffff700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 0x7f8638062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:39.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.344+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f865007d8b0 con 0x7f8658140ae0 2023-12-19T08:03:39.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.344+0000 7f86577fe700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 0x7f8638062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:39.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.348+0000 7f86577fe700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 0x7f8638062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8648009cf0 tx=0x7f8648008040 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:39.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.352+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f865004d890 con 0x7f8658140ae0 2023-12-19T08:03:39.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.556+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86580c2e90 con 0x7f8658140ae0 2023-12-19T08:03:39.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.556+0000 7f864ffff700 1 -- 172.21.15.111:0/2452143365 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f8650050eb0 con 0x7f8658140ae0 2023-12-19T08:03:39.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:39.560 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:03:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 msgr2=0x7f8638062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 0x7f8638062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8648009cf0 tx=0x7f8648008040 comp rx=0 tx=0).stop 2023-12-19T08:03:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 msgr2=0x7f86580b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f8650009df0 tx=0x7f865000b040 comp rx=0 tx=0).stop 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 shutdown_connections 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8638060670 0x7f8638062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8658140710 0x7f865813acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8658140ae0 0x7f86580b0c30 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 --2- 172.21.15.111:0/2452143365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86580c75e0 0x7f8658156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 >> 172.21.15.111:0/2452143365 conn(0x7f86580c8850 msgr2=0x7f86580cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 shutdown_connections 2023-12-19T08:03:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:39.560+0000 7f865dccd700 1 -- 172.21.15.111:0/2452143365 wait complete. 2023-12-19T08:03:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:03:39.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:03:33.363924+0000', 'modified': '2023-12-19T08:03:36.955831+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7389}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7389': {'gid': 7389, 'name': 'a', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/1122478953', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1122478953}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1122478953}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:03:39.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:03:39.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T08:03:40.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.156+0000 7fe6eaa78700 1 Processor -- start 2023-12-19T08:03:40.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.156+0000 7fe6eaa78700 1 -- start start 2023-12-19T08:03:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6eaa78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6e4158e50 0x7fe6e406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6eaa78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6e406ae50 0x7fe6e40aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6eaa78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e406e160 con 0x7fe6e4159220 2023-12-19T08:03:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6eaa78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e406d0c0 con 0x7fe6e4158e50 2023-12-19T08:03:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6eaa78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e406d3a0 con 0x7fe6e406ae50 2023-12-19T08:03:40.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6e4158e50 0x7fe6e406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6e4158e50 0x7fe6e406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45474/0 (socket says 172.21.15.111:45474) 2023-12-19T08:03:40.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 -- 172.21.15.111:0/3416252924 learned_addr learned my addr 172.21.15.111:0/3416252924 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e9015700 1 --2- 172.21.15.111:0/3416252924 >> v2:172.21.15.143:3300/0 conn(0x7fe6e406ae50 0x7fe6e40aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 -- 172.21.15.111:0/3416252924 >> v2:172.21.15.143:3300/0 conn(0x7fe6e406ae50 msgr2=0x7fe6e40aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 --2- 172.21.15.111:0/3416252924 >> v2:172.21.15.143:3300/0 conn(0x7fe6e406ae50 0x7fe6e40aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 -- 172.21.15.111:0/3416252924 >> v1:172.21.15.143:6789/0 conn(0x7fe6e4159220 legacy=0x7fe6e406a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e37fe700 1 -- 172.21.15.111:0/3416252924 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235159762 0 0) 0x7fe6e406e160 con 0x7fe6e4159220 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 -- 172.21.15.111:0/3416252924 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e406d680 con 0x7fe6e4158e50 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e8814700 1 --2- 172.21.15.111:0/3416252924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e406c7c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099b0 tx=0x7fe6d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701955d55e40cb8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e37fe700 1 -- 172.21.15.111:0/3416252924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8009ea0 con 0x7fe6e4158e50 2023-12-19T08:03:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e37fe700 1 -- 172.21.15.111:0/3416252924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d800b820 con 0x7fe6e4158e50 2023-12-19T08:03:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.160+0000 7fe6e37fe700 1 -- 172.21.15.111:0/3416252924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d800bbc0 con 0x7fe6e4158e50 2023-12-19T08:03:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- 172.21.15.111:0/3416252924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 msgr2=0x7fe6e406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/3416252924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e406c7c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099b0 tx=0x7fe6d8009250 comp rx=0 tx=0).stop 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- 172.21.15.111:0/3416252924 shutdown_connections 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/3416252924 >> v2:172.21.15.143:3300/0 conn(0x7fe6e406ae50 0x7fe6e40aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/3416252924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e406c7c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- 172.21.15.111:0/3416252924 >> 172.21.15.111:0/3416252924 conn(0x7fe6e4067b60 msgr2=0x7fe6e40ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- 172.21.15.111:0/3416252924 shutdown_connections 2023-12-19T08:03:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- 172.21.15.111:0/3416252924 wait complete. 2023-12-19T08:03:40.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 Processor -- start 2023-12-19T08:03:40.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.164+0000 7fe6eaa78700 1 -- start start 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6e406ae50 0x7fe6e40afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6e4159220 0x7fe6e40c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40c8f50 con 0x7fe6e4158e50 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40c9e60 con 0x7fe6e4159220 2023-12-19T08:03:40.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e415be40 con 0x7fe6e406ae50 2023-12-19T08:03:40.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45484/0 (socket says 172.21.15.111:45484) 2023-12-19T08:03:40.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 -- 172.21.15.111:0/2253799349 learned_addr learned my addr 172.21.15.111:0/2253799349 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6e406ae50 0x7fe6e40afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e9015700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6e4159220 0x7fe6e40c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 -- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6e406ae50 msgr2=0x7fe6e40afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6e406ae50 0x7fe6e40afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 -- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6e4159220 msgr2=0x7fe6e40c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6e4159220 0x7fe6e40c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 -- 172.21.15.111:0/2253799349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80094f0 con 0x7fe6e4158e50 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c7e0 tx=0x7fe6d0007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e0ff9700 1 -- 172.21.15.111:0/2253799349 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0011590 con 0x7fe6e4158e50 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e0ff9700 1 -- 172.21.15.111:0/2253799349 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0011710 con 0x7fe6e4158e50 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e4159910 con 0x7fe6e4158e50 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6e0ff9700 1 -- 172.21.15.111:0/2253799349 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0011ad0 con 0x7fe6e4158e50 2023-12-19T08:03:40.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.168+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e40c8be0 con 0x7fe6e4158e50 2023-12-19T08:03:40.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.172+0000 7fe6e0ff9700 1 -- 172.21.15.111:0/2253799349 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe6d001b3b0 con 0x7fe6e4158e50 2023-12-19T08:03:40.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6e0ff9700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 0x7fe6c8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6e0ff9700 1 -- 172.21.15.111:0/2253799349 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe6d007c140 con 0x7fe6e4158e50 2023-12-19T08:03:40.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 0x7fe6c8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 0x7fe6c8062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099b0 tx=0x7fe6d8017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6ecd82740 1 Processor -- start 2023-12-19T08:03:40.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.176+0000 7fe6ecd82740 1 -- start start 2023-12-19T08:03:40.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x141ef30 con 0x1273710 2023-12-19T08:03:40.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x141aed0 con 0x1273ae0 2023-12-19T08:03:40.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x13e10b0 con 0x12be790 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45486/0 (socket says 172.21.15.111:45486) 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 -- 172.21.15.111:0/1456145235 learned_addr learned my addr 172.21.15.111:0/1456145235 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e8814700 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e9015700 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 -- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 msgr2=0x14f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 -- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 msgr2=0x14f0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 -- 172.21.15.111:0/1456145235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x12bf9e0 con 0x1273710 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0001ed0 tx=0x7fe6d004c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06e5284d420598d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6cdffb700 1 -- 172.21.15.111:0/1456145235 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0011e80 con 0x1273710 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6cdffb700 1 -- 172.21.15.111:0/1456145235 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d004e0d0 con 0x1273710 2023-12-19T08:03:40.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6cdffb700 1 -- 172.21.15.111:0/1456145235 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00550c0 con 0x1273710 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1456145235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 msgr2=0x14f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0001ed0 tx=0x7fe6d004c400 comp rx=0 tx=0).stop 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1456145235 shutdown_connections 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f1440 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1456145235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1456145235 >> 172.21.15.111:0/1456145235 conn(0x1273eb0 msgr2=0x14ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1456145235 shutdown_connections 2023-12-19T08:03:40.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.180+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1456145235 wait complete. 2023-12-19T08:03:40.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 -- start start 2023-12-19T08:03:40.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 Processor -- start 2023-12-19T08:03:40.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14d6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x13e0d50 con 0x1273710 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x13b9970 con 0x1273ae0 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1421400 con 0x12be790 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6ecd82740 10 client.0 _ll_register_callbacks cb 0x14646a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6e8814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6e8814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45490/0 (socket says 172.21.15.111:45490) 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6e8814700 1 -- 172.21.15.111:0/2503321954 learned_addr learned my addr 172.21.15.111:0/2503321954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.184+0000 7fe6e9015700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14d6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 -- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 msgr2=0x14d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14d6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 -- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 msgr2=0x14f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 -- 172.21.15.111:0/2503321954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e40c8be0 con 0x1273710 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e8814700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802a4e0 tx=0x7fe6d8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155921ea481f24d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d802f070 con 0x1273710 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x13b9cc0 con 0x1273710 2023-12-19T08:03:40.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe6d8011420 con 0x1273710 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d8021070 con 0x1273710 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 10 client.7524 fetch_fsmap learned FSMap version 48 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x13e9370 con 0x1273710 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x13b9cc0 con 0x1273710 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 48) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe6d80117f0 con 0x1273710 2023-12-19T08:03:40.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 10 client.7524 fetch_fsmap finished waiting for FSMap version 48 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 10 client.7524 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 20 client.7524 populate_metadata read hostname 'smithi111' 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x14d82a0 0x15006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 4 client.7524 mds_command: new command op to 7389 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 --> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x13bd3e0 con 0x14d82a0 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x14d82a0 0x15006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6c2ffd700 10 client.7524 ms_handle_connect on v2:172.21.15.111:6834/1122478953 2023-12-19T08:03:40.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.188+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x14d82a0 0x15006b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa60614411a4a59c server_cookie=6119d17dc7ab729f in_seq=0 out_seq=0 2023-12-19T08:03:40.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/2503321954 <== mds.0 v2:172.21.15.111:6834/1122478953 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x13bd3e0 con 0x14d82a0 2023-12-19T08:03:40.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6c2ffd700 10 client.7524 handle_command_reply: tid=0 2023-12-19T08:03:40.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 client.7524 shutdown 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 msgr2=0x14f5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802a4e0 tx=0x7fe6d8021000 comp rx=0 tx=0).stop 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 shutdown_connections 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x14d82a0 0x15006b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273ae0 0x14f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x12be790 0x14d6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/2503321954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x14f5590 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6c2ffd700 0 client.7524 ms_handle_reset on v2:172.21.15.111:6834/1122478953 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 >> 172.21.15.111:0/2503321954 conn(0x1273eb0 msgr2=0x14a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 shutdown_connections 2023-12-19T08:03:40.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 1 -- 172.21.15.111:0/2503321954 wait complete. 2023-12-19T08:03:40.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.192+0000 7fe6ecd82740 20 client.7524 trim_cache size 0 max 16384 2023-12-19T08:03:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 Processor -- start 2023-12-19T08:03:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 -- start start 2023-12-19T08:03:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 0x14d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x14d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x141d590 con 0x12be790 2023-12-19T08:03:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x13f0550 con 0x1273710 2023-12-19T08:03:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.212+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1416b50 con 0x1273ae0 2023-12-19T08:03:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e8814700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 0x14d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45506/0 (socket says 172.21.15.111:45506) 2023-12-19T08:03:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 -- 172.21.15.111:0/640834040 learned_addr learned my addr 172.21.15.111:0/640834040 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x14d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 -- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 msgr2=0x14d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x14d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 -- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 msgr2=0x14d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 0x14d4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 -- 172.21.15.111:0/640834040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x13cfd80 con 0x12be790 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e8814700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 0x14d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:03:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x14d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6e9015700 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc007780 tx=0x7fe6dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea541c1f46b7c075 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ce7fc700 1 -- 172.21.15.111:0/640834040 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00e040 con 0x12be790 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ce7fc700 1 -- 172.21.15.111:0/640834040 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00f040 con 0x12be790 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ce7fc700 1 -- 172.21.15.111:0/640834040 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc017620 con 0x12be790 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 -- 172.21.15.111:0/640834040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 msgr2=0x14a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc007780 tx=0x7fe6dc00b040 comp rx=0 tx=0).stop 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 -- 172.21.15.111:0/640834040 shutdown_connections 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x12be790 0x14a7ad0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x14d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/640834040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1273710 0x14d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 -- 172.21.15.111:0/640834040 >> 172.21.15.111:0/640834040 conn(0x1273eb0 msgr2=0x14ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 -- 172.21.15.111:0/640834040 shutdown_connections 2023-12-19T08:03:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.216+0000 7fe6ecd82740 1 -- 172.21.15.111:0/640834040 wait complete. 2023-12-19T08:03:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 -- start start 2023-12-19T08:03:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 Processor -- start 2023-12-19T08:03:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x154afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 0x14aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x13ef610 con 0x1273710 2023-12-19T08:03:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x13bbb80 con 0x12be790 2023-12-19T08:03:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x13ee040 con 0x1273ae0 2023-12-19T08:03:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6ecd82740 10 client.0 _ll_register_callbacks cb 0x123f9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:03:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e8814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e8814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45516/0 (socket says 172.21.15.111:45516) 2023-12-19T08:03:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e8814700 1 -- 172.21.15.111:0/1624644753 learned_addr learned my addr 172.21.15.111:0/1624644753 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x154afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e9015700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 0x14aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.220+0000 7fe6e8814700 1 -- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 msgr2=0x154afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e8814700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x154afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e8814700 1 -- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 msgr2=0x14aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e8814700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 0x14aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e8814700 1 -- 172.21.15.111:0/1624644753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x13e9370 con 0x1273710 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x154afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e9015700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 0x14aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e8814700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800bce0 tx=0x7fe6d802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61f775c1f3052c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8021070 con 0x1273710 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe6d8034e10 con 0x1273710 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6d004e0d0 con 0x1273710 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d801a840 con 0x1273710 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 10 client.7530 fetch_fsmap learned FSMap version 48 2023-12-19T08:03:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x7fe6dc0074b0 con 0x1273710 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d004e0d0 con 0x1273710 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 48) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe6d801a9a0 con 0x1273710 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 10 client.7530 fetch_fsmap finished waiting for FSMap version 48 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 10 client.7530 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 20 client.7530 populate_metadata read hostname 'smithi111' 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x1551b30 0x1553f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 4 client.7530 mds_command: new command op to 7389 tid=0[{"prefix": "status"}] 2023-12-19T08:03:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 --> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] -- command(tid 0: {"prefix": "status"}) v1 -- 0x13ef410 con 0x1551b30 2023-12-19T08:03:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x1551b30 0x1553f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6e3fff700 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x1551b30 0x1553f10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8439ad6dbc99159 server_cookie=f38348a21221b0bf in_seq=0 out_seq=0 2023-12-19T08:03:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 10 client.7530 ms_handle_connect on v2:172.21.15.111:6834/1122478953 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1624644753 <== mds.0 v2:172.21.15.111:6834/1122478953 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x13ef410 con 0x1551b30 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6c2ffd700 10 client.7530 handle_command_reply: tid=0 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.224+0000 7fe6ecd82740 1 client.7530 shutdown 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 msgr2=0x154aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800bce0 tx=0x7fe6d802f000 comp rx=0 tx=0).stop 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 shutdown_connections 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:6834/1122478953,v1:172.21.15.111:6835/1122478953] conn(0x1551b30 0x1553f10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x12be790 0x14aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1273ae0 0x154afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6c2ffd700 0 client.7530 ms_handle_reset on v2:172.21.15.111:6834/1122478953 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 --2- 172.21.15.111:0/1624644753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1273710 0x154aa90 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 >> 172.21.15.111:0/1624644753 conn(0x1273eb0 msgr2=0x14a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 shutdown_connections 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 1 -- 172.21.15.111:0/1624644753 wait complete. 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6ecd82740 20 client.7530 trim_cache size 0 max 16384 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:03:40.232 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 7389, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2985927549999996, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 48, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 113, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 113, 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.249302076999999 2023-12-19T08:03:40.233 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 msgr2=0x7fe6c8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 0x7fe6c8062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80099b0 tx=0x7fe6d8017000 comp rx=0 tx=0).stop 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 msgr2=0x7fe6e40c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000c7e0 tx=0x7fe6d0007dc0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 shutdown_connections 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe6c8060670 0x7fe6c8062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6e4159220 0x7fe6e40c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6e4158e50 0x7fe6e40c93e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 --2- 172.21.15.111:0/2253799349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6e406ae50 0x7fe6e40afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 >> 172.21.15.111:0/2253799349 conn(0x7fe6e4067b60 msgr2=0x7fe6e41544f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 shutdown_connections 2023-12-19T08:03:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.228+0000 7fe6eaa78700 1 -- 172.21.15.111:0/2253799349 wait complete. 2023-12-19T08:03:40.253 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:03:40.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 Processor -- start 2023-12-19T08:03:40.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 -- start start 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff0cc0ae140 0x7ff0cc0a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0cc0c1d60 0x7ff0cc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0cc06dc20 con 0x7ff0cc0ae510 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0cc0c2200 con 0x7ff0cc0ae140 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.416+0000 7ff0d3ef0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0cc06d940 con 0x7ff0cc0c1d60 2023-12-19T08:03:40.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff0cc0ae140 0x7ff0cc0a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d248d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0cc0c1d60 0x7ff0cc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff0cc0ae140 0x7ff0cc0a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45526/0 (socket says 172.21.15.111:45526) 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 -- 172.21.15.111:0/2897550582 learned_addr learned my addr 172.21.15.111:0/2897550582 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d148b700 1 --1- 172.21.15.111:0/2897550582 >> v1:172.21.15.143:6789/0 conn(0x7ff0cc0ae510 0x7ff0cc06d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36876/0 (socket says 172.21.15.111:36876) 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 -- 172.21.15.111:0/2897550582 >> v2:172.21.15.143:3300/0 conn(0x7ff0cc0c1d60 msgr2=0x7ff0cc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 --2- 172.21.15.111:0/2897550582 >> v2:172.21.15.143:3300/0 conn(0x7ff0cc0c1d60 0x7ff0cc0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 -- 172.21.15.111:0/2897550582 >> v1:172.21.15.143:6789/0 conn(0x7ff0cc0ae510 legacy=0x7ff0cc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 -- 172.21.15.111:0/2897550582 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc0c6e20 con 0x7ff0cc0ae140 2023-12-19T08:03:40.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d1c8c700 1 --2- 172.21.15.111:0/2897550582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0a8850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c23b0 tx=0x7ff0c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7f04cc7c2dd08f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d0c8a700 1 -- 172.21.15.111:0/2897550582 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8009e70 con 0x7ff0cc0ae140 2023-12-19T08:03:40.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d0c8a700 1 -- 172.21.15.111:0/2897550582 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c800b860 con 0x7ff0cc0ae140 2023-12-19T08:03:40.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d0c8a700 1 -- 172.21.15.111:0/2897550582 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800bc00 con 0x7ff0cc0ae140 2023-12-19T08:03:40.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/2897550582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae140 msgr2=0x7ff0cc0a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/2897550582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0a8850 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c23b0 tx=0x7ff0c8009250 comp rx=0 tx=0).stop 2023-12-19T08:03:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/2897550582 shutdown_connections 2023-12-19T08:03:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/2897550582 >> v2:172.21.15.143:3300/0 conn(0x7ff0cc0c1d60 0x7ff0cc0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/2897550582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0a8850 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.420+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/2897550582 >> 172.21.15.111:0/2897550582 conn(0x7ff0cc0c8850 msgr2=0x7ff0cc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/2897550582 shutdown_connections 2023-12-19T08:03:40.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/2897550582 wait complete. 2023-12-19T08:03:40.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 Processor -- start 2023-12-19T08:03:40.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- start start 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0cc0c1d60 0x7ff0cc15b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc0acf30 con 0x7ff0cc0ae510 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc15c6b0 con 0x7ff0cc0ae140 2023-12-19T08:03:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.424+0000 7ff0d3ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc169620 con 0x7ff0cc0c1d60 2023-12-19T08:03:40.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45528/0 (socket says 172.21.15.111:45528) 2023-12-19T08:03:40.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 -- 172.21.15.111:0/1239753916 learned_addr learned my addr 172.21.15.111:0/1239753916 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:40.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d248d700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0cc0c1d60 0x7ff0cc15b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d1c8c700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 -- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0cc0c1d60 msgr2=0x7ff0cc15b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0cc0c1d60 0x7ff0cc15b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 -- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0cc0ae140 msgr2=0x7ff0cc0ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 -- 172.21.15.111:0/1239753916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c80094f0 con 0x7ff0cc0ae510 2023-12-19T08:03:40.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d148b700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c790 tx=0x7ff0bc007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc005a70 con 0x7ff0cc0ae510 2023-12-19T08:03:40.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc005bf0 con 0x7ff0cc0ae510 2023-12-19T08:03:40.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0070d0 con 0x7ff0cc0ae510 2023-12-19T08:03:40.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0cc0ad700 con 0x7ff0cc0ae510 2023-12-19T08:03:40.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.428+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc0ad140 con 0x7ff0cc0ae510 2023-12-19T08:03:40.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.432+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc071120 con 0x7ff0cc0ae510 2023-12-19T08:03:40.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.432+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff0bc007480 con 0x7ff0cc0ae510 2023-12-19T08:03:40.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.432+0000 7ff0c1ffb700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 0x7ff0b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:40.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.432+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7ff0bc07c110 con 0x7ff0cc0ae510 2023-12-19T08:03:40.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.432+0000 7ff0d1c8c700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 0x7ff0b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:40.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.436+0000 7ff0d1c8c700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 0x7ff0b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c23b0 tx=0x7ff0c8017000 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:40.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.440+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff0bc04c200 con 0x7ff0cc0ae510 2023-12-19T08:03:40.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.644+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff0cc0642a0 con 0x7ff0cc0ae510 2023-12-19T08:03:40.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.644+0000 7ff0c1ffb700 1 -- 172.21.15.111:0/1239753916 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff0bc04f820 con 0x7ff0cc0ae510 2023-12-19T08:03:40.648 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:03:40.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 msgr2=0x7ff0b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 0x7ff0b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c23b0 tx=0x7ff0c8017000 comp rx=0 tx=0).stop 2023-12-19T08:03:40.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 msgr2=0x7ff0cc15c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:40.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c790 tx=0x7ff0bc007f40 comp rx=0 tx=0).stop 2023-12-19T08:03:40.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 shutdown_connections 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff0b0060670 0x7ff0b0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0cc0c1d60 0x7ff0cc15b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0cc0ae510 0x7ff0cc15c170 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 --2- 172.21.15.111:0/1239753916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0cc0ae140 0x7ff0cc0ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 >> 172.21.15.111:0/1239753916 conn(0x7ff0cc0c8850 msgr2=0x7ff0cc0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 shutdown_connections 2023-12-19T08:03:40.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:40.648+0000 7ff0d3ef0700 1 -- 172.21.15.111:0/1239753916 wait complete. 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:03:40.674 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:40.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.674 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.692 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41437sec preferred_lft 41437sec 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.693 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:03:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:03:40.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.696 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:03:40.776 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:03:40.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.779 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:03:40.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:03:40.846 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:03:40.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.978 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:03:40.994 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:03:40.994 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:03:40.995 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:40.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:03:40.995 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:03:41.073 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:03:41.073 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:03:41.073 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:41.131 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:03:41.132 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:03:41.132 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:41.194 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:03:41.277 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:03:41.277 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:03:41.352 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:03:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:03:41.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:41.356 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:03:41.418 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:03:41.419 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:03:41.469 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:03:41.499 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:03:41.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:03:41.576 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:03:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:03:41.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:41.579 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:03:41.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:03:41.588+0000 7f13db90d380 -1 init, newargv = 0x56508b782af0 newargc=15 2023-12-19T08:03:41.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[52291]: starting ceph client 2023-12-19T08:03:41.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[52291]: starting fuse 2023-12-19T08:03:41.606 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:03:41.606 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:41.607 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:03:41.608 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:03:41.768 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.52291.asok 2023-12-19T08:03:41.953 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52291.asok 2023-12-19T08:03:41.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:41.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52291.asok status 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:03:42.136 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "52291", 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 7542, 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 7542 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:03:42.137 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3987531724 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3987531724 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.7542 192.168.0.1:0/3987531724", 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/3987531724", 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 48, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 113, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:03:42.138 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:03:42.139 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:03:42.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:42.150 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:03:42.150 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:42.170 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:03:42.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:42.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:03:42.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:03:42.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:03:42.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.472+0000 7f6ea4078700 1 Processor -- start 2023-12-19T08:03:42.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 -- start start 2023-12-19T08:03:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c0b37a0 0x7f6e9c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0b3ca0 con 0x7f6e9c156ea0 2023-12-19T08:03:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9c15b870 con 0x7f6e9c156ad0 2023-12-19T08:03:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.476+0000 7f6ea4078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c06d300 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea2615700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c0b37a0 0x7f6e9c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33056/0 (socket says 172.21.15.111:33056) 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 -- 172.21.15.111:0/1040522008 learned_addr learned my addr 172.21.15.111:0/1040522008 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1613700 1 --1- 172.21.15.111:0/1040522008 >> v1:172.21.15.111:6789/0 conn(0x7f6e9c156ea0 0x7f6e9c0cc720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46938/0 (socket says 172.21.15.111:46938) 2023-12-19T08:03:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 -- 172.21.15.111:0/1040522008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c0b37a0 msgr2=0x7f6e9c0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 --2- 172.21.15.111:0/1040522008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c0b37a0 0x7f6e9c0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 -- 172.21.15.111:0/1040522008 >> v1:172.21.15.111:6789/0 conn(0x7f6e9c156ea0 legacy=0x7f6e9c0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 -- 172.21.15.111:0/1040522008 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c06d5e0 con 0x7f6e9c156ad0 2023-12-19T08:03:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea0e12700 1 -- 172.21.15.111:0/1040522008 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923057543 0 0) 0x7f6e9c0b3ca0 con 0x7f6e9c156ea0 2023-12-19T08:03:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.480+0000 7f6ea1e14700 1 --2- 172.21.15.111:0/1040522008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f6e980099b0 tx=0x7f6e98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe2e07ad0753d82 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.484+0000 7f6ea0e12700 1 -- 172.21.15.111:0/1040522008 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98009ea0 con 0x7f6e9c156ad0 2023-12-19T08:03:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.484+0000 7f6ea0e12700 1 -- 172.21.15.111:0/1040522008 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9800b860 con 0x7f6e9c156ad0 2023-12-19T08:03:42.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.484+0000 7f6ea0e12700 1 -- 172.21.15.111:0/1040522008 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800bc00 con 0x7f6e9c156ad0 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 -- 172.21.15.111:0/1040522008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c156ad0 msgr2=0x7f6e9c0cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 --2- 172.21.15.111:0/1040522008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f6e980099b0 tx=0x7f6e98009250 comp rx=0 tx=0).stop 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 -- 172.21.15.111:0/1040522008 shutdown_connections 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 --2- 172.21.15.111:0/1040522008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c0b37a0 0x7f6e9c0b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 --2- 172.21.15.111:0/1040522008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c156ad0 0x7f6e9c0cc1e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 -- 172.21.15.111:0/1040522008 >> 172.21.15.111:0/1040522008 conn(0x7f6e9c0cdda0 msgr2=0x7f6e9c0ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 -- 172.21.15.111:0/1040522008 shutdown_connections 2023-12-19T08:03:42.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.488+0000 7f6ea4078700 1 -- 172.21.15.111:0/1040522008 wait complete. 2023-12-19T08:03:42.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 Processor -- start 2023-12-19T08:03:42.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 -- start start 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 0x7f6e9c159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 0x7f6e9c16d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 0x7f6e9c1537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c16e1c0 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c069080 con 0x7f6e9c156ea0 2023-12-19T08:03:42.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea4078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c16d920 con 0x7f6e9c156ad0 2023-12-19T08:03:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1613700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 0x7f6e9c16d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 0x7f6e9c159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea2615700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 0x7f6e9c1537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1613700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 0x7f6e9c16d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:33600/0 (socket says 172.21.15.111:33600) 2023-12-19T08:03:42.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea2615700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 0x7f6e9c1537d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:33738/0 (socket says 172.21.15.111:33738) 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1613700 1 -- 172.21.15.111:0/607724457 learned_addr learned my addr 172.21.15.111:0/607724457 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 -- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 msgr2=0x7f6e9c16d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 0x7f6e9c16d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 -- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 msgr2=0x7f6e9c1537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 0x7f6e9c1537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 -- 172.21.15.111:0/607724457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980094f0 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.492+0000 7f6ea1e14700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 0x7f6e9c159ef0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c06c080 tx=0x7f6e98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98000d10 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9800bbe0 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98011630 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c16dc00 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c158340 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c158530 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.496+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6e98011a60 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.500+0000 7f6e91ffb700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 0x7f6e80062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.500+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f6e9807ddb0 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.500+0000 7f6ea1613700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 0x7f6e80062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.504+0000 7f6ea1613700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 0x7f6e80062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c007d60 tx=0x7f6e8c007cb0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.508+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6e98082070 con 0x7f6e9c0b37a0 2023-12-19T08:03:42.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.700+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e9c071120 con 0x7f6e80060670 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:03:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.704+0000 7f6e91ffb700 1 -- 172.21.15.111:0/607724457 <== mgr.6389 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6e9c071120 con 0x7f6e80060670 2023-12-19T08:03:42.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 msgr2=0x7f6e80062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 0x7f6e80062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c007d60 tx=0x7f6e8c007cb0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 msgr2=0x7f6e9c159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 0x7f6e9c159ef0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c06c080 tx=0x7f6e98017000 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 shutdown_connections 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6e80060670 0x7f6e80062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e9c156ea0 0x7f6e9c1537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e9c156ad0 0x7f6e9c16d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 --2- 172.21.15.111:0/607724457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e9c0b37a0 0x7f6e9c159ef0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 >> 172.21.15.111:0/607724457 conn(0x7f6e9c0cdda0 msgr2=0x7f6e9c0ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 shutdown_connections 2023-12-19T08:03:42.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.708+0000 7f6ea4078700 1 -- 172.21.15.111:0/607724457 wait complete. 2023-12-19T08:03:42.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:03:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.888+0000 7f6b4ae8e700 1 Processor -- start 2023-12-19T08:03:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 -- start start 2023-12-19T08:03:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b440adab0 0x7f6b440adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4415ae30 con 0x7f6b44154780 2023-12-19T08:03:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b440abf70 con 0x7f6b440adab0 2023-12-19T08:03:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4ae8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44154b50 con 0x7f6b4406baa0 2023-12-19T08:03:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:33616/0 (socket says 172.21.15.111:33616) 2023-12-19T08:03:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 -- 172.21.15.111:0/903895578 learned_addr learned my addr 172.21.15.111:0/903895578 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4a68d700 1 --2- 172.21.15.111:0/903895578 >> v2:172.21.15.143:3300/0 conn(0x7f6b440adab0 0x7f6b440adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 -- 172.21.15.111:0/903895578 >> v2:172.21.15.143:3300/0 conn(0x7f6b440adab0 msgr2=0x7f6b440adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 --2- 172.21.15.111:0/903895578 >> v2:172.21.15.143:3300/0 conn(0x7f6b440adab0 0x7f6b440adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 -- 172.21.15.111:0/903895578 >> v1:172.21.15.143:6789/0 conn(0x7f6b44154780 legacy=0x7f6b44157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b48e8a700 1 -- 172.21.15.111:0/903895578 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142350359 0 0) 0x7f6b4415ae30 con 0x7f6b44154780 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.892+0000 7f6b4968b700 1 -- 172.21.15.111:0/903895578 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440ac210 con 0x7f6b4406baa0 2023-12-19T08:03:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4968b700 1 --2- 172.21.15.111:0/903895578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b340099b0 tx=0x7f6b34009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5f56a45ae59ae2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b48e8a700 1 -- 172.21.15.111:0/903895578 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34009ea0 con 0x7f6b4406baa0 2023-12-19T08:03:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b48e8a700 1 -- 172.21.15.111:0/903895578 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3400b860 con 0x7f6b4406baa0 2023-12-19T08:03:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b48e8a700 1 -- 172.21.15.111:0/903895578 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3400bc00 con 0x7f6b4406baa0 2023-12-19T08:03:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/903895578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 msgr2=0x7f6b4415b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/903895578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b340099b0 tx=0x7f6b34009250 comp rx=0 tx=0).stop 2023-12-19T08:03:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/903895578 shutdown_connections 2023-12-19T08:03:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/903895578 >> v2:172.21.15.143:3300/0 conn(0x7f6b440adab0 0x7f6b440adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/903895578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b4415b2c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/903895578 >> 172.21.15.111:0/903895578 conn(0x7f6b44067b50 msgr2=0x7f6b440cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/903895578 shutdown_connections 2023-12-19T08:03:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.896+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/903895578 wait complete. 2023-12-19T08:03:42.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 Processor -- start 2023-12-19T08:03:42.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 -- start start 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b44169b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b440adab0 0x7f6b441524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441529f0 con 0x7f6b44154780 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4414d290 con 0x7f6b440adab0 2023-12-19T08:03:42.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.900+0000 7f6b4ae8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441690d0 con 0x7f6b4406baa0 2023-12-19T08:03:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33064/0 (socket says 172.21.15.111:33064) 2023-12-19T08:03:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 -- 172.21.15.111:0/1433078508 learned_addr learned my addr 172.21.15.111:0/1433078508 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:42.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4968b700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b440adab0 0x7f6b441524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b49e8c700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b44169b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 -- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 msgr2=0x7f6b44169b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b44169b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 -- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b440adab0 msgr2=0x7f6b441524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b440adab0 0x7f6b441524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 -- 172.21.15.111:0/1433078508 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340094f0 con 0x7f6b44154780 2023-12-19T08:03:42.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4a68d700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00e8d0 tx=0x7f6b3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c013680 con 0x7f6b44154780 2023-12-19T08:03:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c0137e0 con 0x7f6b44154780 2023-12-19T08:03:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c013ba0 con 0x7f6b44154780 2023-12-19T08:03:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4414ccd0 con 0x7f6b44154780 2023-12-19T08:03:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.904+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b44169410 con 0x7f6b44154780 2023-12-19T08:03:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.908+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4404be40 con 0x7f6b44154780 2023-12-19T08:03:42.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.908+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 41) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6b3c0123d0 con 0x7f6b44154780 2023-12-19T08:03:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.908+0000 7f6b39ffb700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 0x7f6b28062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.908+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f6b3c07cd30 con 0x7f6b44154780 2023-12-19T08:03:42.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.912+0000 7f6b49e8c700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 0x7f6b28062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:42.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.912+0000 7f6b49e8c700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 0x7f6b28062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b40001560 tx=0x7f6b400016f0 comp rx=0 tx=0).ready entity=mgr.6389 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:42.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:42.916+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6b3c04cee0 con 0x7f6b44154780 2023-12-19T08:03:43.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.128+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6b440cc1a0 con 0x7f6b44154780 2023-12-19T08:03:43.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.504+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 42) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f6b3c07c0a0 con 0x7f6b44154780 2023-12-19T08:03:43.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.504+0000 7f6b39ffb700 1 -- 172.21.15.111:0/1433078508 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v42) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6b3c04cee0 con 0x7f6b44154780 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 msgr2=0x7f6b28062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 0x7f6b28062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b40001560 tx=0x7f6b400016f0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 msgr2=0x7f6b4414c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00e8d0 tx=0x7f6b3c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 shutdown_connections 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6b280606a0 0x7f6b28062b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b440adab0 0x7f6b441524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b44154780 0x7f6b4414c2a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 --2- 172.21.15.111:0/1433078508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b4406baa0 0x7f6b44169b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 >> 172.21.15.111:0/1433078508 conn(0x7f6b44067b50 msgr2=0x7f6b4406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 shutdown_connections 2023-12-19T08:03:43.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.508+0000 7f6b4ae8e700 1 -- 172.21.15.111:0/1433078508 wait complete. 2023-12-19T08:03:43.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:03:43.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.680+0000 7fd69b59e700 1 Processor -- start 2023-12-19T08:03:43.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.680+0000 7fd69b59e700 1 -- start start 2023-12-19T08:03:43.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd69c154780 0x7fd69c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:43.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c15ae30 con 0x7fd69c06baa0 2023-12-19T08:03:43.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0abf70 con 0x7fd69c0adab0 2023-12-19T08:03:43.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c154b50 con 0x7fd69c154780 2023-12-19T08:03:43.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd699d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd69c06baa0 0x7fd69c157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46946/0 (socket says 172.21.15.111:46946) 2023-12-19T08:03:43.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd699d9b700 1 -- 172.21.15.111:0/1792691071 learned_addr learned my addr 172.21.15.111:0/1792691071 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:43.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69a59c700 1 --2- 172.21.15.111:0/1792691071 >> v2:172.21.15.143:3300/0 conn(0x7fd69c154780 0x7fd69c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:43.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812420347 0 0) 0x7fd69c15ae30 con 0x7fd69c06baa0 2023-12-19T08:03:43.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd684003570 con 0x7fd69c06baa0 2023-12-19T08:03:43.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000536922 0 0) 0x7fd69c0abf70 con 0x7fd69c0adab0 2023-12-19T08:03:43.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.684+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c15ae30 con 0x7fd69c0adab0 2023-12-19T08:03:43.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69a59c700 1 -- 172.21.15.111:0/1792691071 >> v1:172.21.15.143:6789/0 conn(0x7fd69c0adab0 legacy=0x7fd69c0adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69a59c700 1 -- 172.21.15.111:0/1792691071 >> v1:172.21.15.111:6789/0 conn(0x7fd69c06baa0 legacy=0x7fd69c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69a59c700 1 -- 172.21.15.111:0/1792691071 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0ac210 con 0x7fd69c154780 2023-12-19T08:03:43.691 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:43.688+0000 7f0a82eea000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69a59c700 1 --2- 172.21.15.111:0/1792691071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c15b2c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd68c0098e0 tx=0x7fd68c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70943c5ee5bfcc2f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c011630 con 0x7fd69c154780 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c011790 con 0x7fd69c154780 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69959a700 1 -- 172.21.15.111:0/1792691071 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c011b30 con 0x7fd69c154780 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 -- 172.21.15.111:0/1792691071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 msgr2=0x7fd69c15b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 --2- 172.21.15.111:0/1792691071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c15b2c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd68c0098e0 tx=0x7fd68c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 -- 172.21.15.111:0/1792691071 shutdown_connections 2023-12-19T08:03:43.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 --2- 172.21.15.111:0/1792691071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c15b2c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 -- 172.21.15.111:0/1792691071 >> 172.21.15.111:0/1792691071 conn(0x7fd69c067b50 msgr2=0x7fd69c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:43.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 -- 172.21.15.111:0/1792691071 shutdown_connections 2023-12-19T08:03:43.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.688+0000 7fd69b59e700 1 -- 172.21.15.111:0/1792691071 wait complete. 2023-12-19T08:03:43.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 Processor -- start 2023-12-19T08:03:43.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 -- start start 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c06baa0 0x7fd69c0b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c0c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c14fb60 con 0x7fd69c0adab0 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c15a240 con 0x7fd69c154780 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c159630 con 0x7fd69c06baa0 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd699d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd699d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33092/0 (socket says 172.21.15.111:33092) 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69a59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c06baa0 0x7fd69c0b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd699d9b700 1 -- 172.21.15.111:0/2120544856 learned_addr learned my addr 172.21.15.111:0/2120544856 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.692+0000 7fd69ad9d700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c0c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:43.699 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:43.704+0000 7f33133c1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 -- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c06baa0 msgr2=0x7fd69c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c06baa0 0x7fd69c0b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 -- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 msgr2=0x7fd69c0c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c0c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 -- 172.21.15.111:0/2120544856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c009420 con 0x7fd69c0adab0 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd699d9b700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c7c0 tx=0x7fd690007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690005c00 con 0x7fd69c0adab0 2023-12-19T08:03:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd690005d60 con 0x7fd69c0adab0 2023-12-19T08:03:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c159910 con 0x7fd69c0adab0 2023-12-19T08:03:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690007320 con 0x7fd69c0adab0 2023-12-19T08:03:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c159ed0 con 0x7fd69c0adab0 2023-12-19T08:03:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.696+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c04be40 con 0x7fd69c0adab0 2023-12-19T08:03:43.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.700+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 42) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fd69001b3b0 con 0x7fd69c0adab0 2023-12-19T08:03:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.700+0000 7fd68a7fc700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 0x7fd67c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.700+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fd69007c230 con 0x7fd69c0adab0 2023-12-19T08:03:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.700+0000 7fd69a59c700 1 -- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 msgr2=0x7fd67c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.700+0000 7fd69a59c700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 0x7fd67c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:43.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.708+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd69004c320 con 0x7fd69c0adab0 2023-12-19T08:03:43.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.888+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd69c0ca340 con 0x7fd69c0adab0 2023-12-19T08:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.896+0000 7fd68a7fc700 1 -- 172.21.15.111:0/2120544856 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd69004f940 con 0x7fd69c0adab0 2023-12-19T08:03:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 msgr2=0x7fd67c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 0x7fd67c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 msgr2=0x7fd69c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c7c0 tx=0x7fd690007dd0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 shutdown_connections 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd67c0606c0 0x7fd67c062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd69c154780 0x7fd69c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd69c0adab0 0x7fd69c0c8170 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 --2- 172.21.15.111:0/2120544856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd69c06baa0 0x7fd69c0b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.900+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 >> 172.21.15.111:0/2120544856 conn(0x7fd69c067b50 msgr2=0x7fd69c152470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.904+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 shutdown_connections 2023-12-19T08:03:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:43.904+0000 7fd69b59e700 1 -- 172.21.15.111:0/2120544856 wait complete. 2023-12-19T08:03:43.931 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:03:44.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.084+0000 7f9bfd26e700 1 Processor -- start 2023-12-19T08:03:44.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.084+0000 7f9bfd26e700 1 -- start start 2023-12-19T08:03:44.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bfd26e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bfd26e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 0x7f9bf80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bfd26e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80cbdf0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bfd26e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf80cc0d0 con 0x7f9bf80cb970 2023-12-19T08:03:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bfd26e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80cc3b0 con 0x7f9bf8158e20 2023-12-19T08:03:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bf77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9bf80ca2c0 0x7f9bf806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46954/0 (socket says 172.21.15.111:46954) 2023-12-19T08:03:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bf77fe700 1 -- 172.21.15.111:0/3294637131 learned_addr learned my addr 172.21.15.111:0/3294637131 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.088+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/3294637131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf67fc700 1 --2- 172.21.15.111:0/3294637131 >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 0x7f9bf80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/3294637131 >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 msgr2=0x7f9bf80ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/3294637131 >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 0x7f9bf80ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/3294637131 >> v1:172.21.15.111:6789/0 conn(0x7f9bf80ca2c0 legacy=0x7f9bf806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf5ffb700 1 -- 172.21.15.111:0/3294637131 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433984618 0 0) 0x7f9bf80cbdf0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/3294637131 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80cc690 con 0x7f9bf8158e20 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf67fc700 1 --2- 172.21.15.111:0/3294637131 >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 0x7f9bf80ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/3294637131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80ce5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098e0 tx=0x7f9be80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e38907e2e4cede8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf5ffb700 1 -- 172.21.15.111:0/3294637131 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8009e80 con 0x7f9bf8158e20 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf5ffb700 1 -- 172.21.15.111:0/3294637131 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be800b860 con 0x7f9bf8158e20 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bf5ffb700 1 -- 172.21.15.111:0/3294637131 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be800bbe0 con 0x7f9bf8158e20 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 -- 172.21.15.111:0/3294637131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 msgr2=0x7f9bf80ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/3294637131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80ce5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098e0 tx=0x7f9be80091a0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 -- 172.21.15.111:0/3294637131 shutdown_connections 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/3294637131 >> v2:172.21.15.143:3300/0 conn(0x7f9bf80cb970 0x7f9bf80ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/3294637131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80ce5c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 -- 172.21.15.111:0/3294637131 >> 172.21.15.111:0/3294637131 conn(0x7f9bf8067bb0 msgr2=0x7f9bf80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 -- 172.21.15.111:0/3294637131 shutdown_connections 2023-12-19T08:03:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.092+0000 7f9bfd26e700 1 -- 172.21.15.111:0/3294637131 wait complete. 2023-12-19T08:03:44.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.096+0000 7f9bfd26e700 1 Processor -- start 2023-12-19T08:03:44.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.096+0000 7f9bfd26e700 1 -- start start 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf80b4de0 0x7f9bf80c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81491c0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80b43a0 con 0x7f9bf80b4de0 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80b34f0 con 0x7f9bf8158e20 2023-12-19T08:03:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33098/0 (socket says 172.21.15.111:33098) 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/706353917 learned_addr learned my addr 172.21.15.111:0/706353917 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf67fc700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf77fe700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf80b4de0 0x7f9bf80c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 msgr2=0x7f9bf80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf80b4de0 msgr2=0x7f9bf80c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf80b4de0 0x7f9bf80c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 -- 172.21.15.111:0/706353917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8009420 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bf6ffd700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80cc4f0 tx=0x7f9be8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be800b9d0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be800bb30 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8011630 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf80b3790 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.100+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf80b3d50 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf8071120 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 42) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f9be80119a0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bdeffd700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 0x7f9be4062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f9be807e530 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bf67fc700 1 -- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 msgr2=0x7f9be4062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:44.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.104+0000 7f9bf67fc700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 0x7f9be4062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:44.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.112+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9be804e620 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.253 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:44.248+0000 7f0a82eea000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:44.257 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:44.260+0000 7f33133c1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:03:44.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.284+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9bf806a7a0 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.292+0000 7f9bdeffd700 1 -- 172.21.15.111:0/706353917 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9be8051c40 con 0x7f9bf80ca2c0 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 msgr2=0x7f9be4062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 0x7f9be4062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 msgr2=0x7f9bf8148c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80cc4f0 tx=0x7f9be8017000 comp rx=0 tx=0).stop 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 shutdown_connections 2023-12-19T08:03:44.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9be40606f0 0x7f9be4062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bf80b4de0 0x7f9bf80c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bf8158e20 0x7f9bf80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 --2- 172.21.15.111:0/706353917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bf80ca2c0 0x7f9bf8148c80 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 >> 172.21.15.111:0/706353917 conn(0x7f9bf8067bb0 msgr2=0x7f9bf8149360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 shutdown_connections 2023-12-19T08:03:44.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.296+0000 7f9bfd26e700 1 -- 172.21.15.111:0/706353917 wait complete. 2023-12-19T08:03:44.323 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /bad-path 1M 2023-12-19T08:03:44.329 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:44.320+0000 7f0a82eea000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:44.333 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:44.336+0000 7f33133c1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:03:44.473 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:44.468+0000 7f0a82eea000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:44.483 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:44.484+0000 7f33133c1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:44.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.500+0000 7f437982e700 1 Processor -- start 2023-12-19T08:03:44.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.500+0000 7f437982e700 1 -- start start 2023-12-19T08:03:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4374159b30 0x7f4374068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4374069900 con 0x7f43741551c0 2023-12-19T08:03:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740c7340 con 0x7f4374155590 2023-12-19T08:03:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43741559f0 con 0x7f4374159b30 2023-12-19T08:03:44.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4372ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f43741551c0 0x7f43740690d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46964/0 (socket says 172.21.15.111:46964) 2023-12-19T08:03:44.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4372ffd700 1 -- 172.21.15.111:0/2286311194 learned_addr learned my addr 172.21.15.111:0/2286311194 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:44.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f43737fe700 1 --2- 172.21.15.111:0/2286311194 >> v2:172.21.15.143:3300/0 conn(0x7f4374159b30 0x7f4374068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247974730 0 0) 0x7f43740c7340 con 0x7f4374155590 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c003570 con 0x7f4374155590 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400489636 0 0) 0x7f4374069900 con 0x7f43741551c0 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f43737fe700 1 -- 172.21.15.111:0/2286311194 >> v1:172.21.15.143:6789/0 conn(0x7f4374155590 legacy=0x7f43740ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f43737fe700 1 -- 172.21.15.111:0/2286311194 >> v1:172.21.15.111:6789/0 conn(0x7f43741551c0 legacy=0x7f43740690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f43737fe700 1 -- 172.21.15.111:0/2286311194 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374159f20 con 0x7f4374159b30 2023-12-19T08:03:44.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f43737fe700 1 --2- 172.21.15.111:0/2286311194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374159b30 0x7f4374068380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f436c009e60 tx=0x7f436c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c24b7b2ccc8928 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:44.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c00e070 con 0x7f4374159b30 2023-12-19T08:03:44.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c00bb60 con 0x7f4374159b30 2023-12-19T08:03:44.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f4371ffb700 1 -- 172.21.15.111:0/2286311194 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c012430 con 0x7f4374159b30 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- 172.21.15.111:0/2286311194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374159b30 msgr2=0x7f4374068380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 --2- 172.21.15.111:0/2286311194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374159b30 0x7f4374068380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f436c009e60 tx=0x7f436c009780 comp rx=0 tx=0).stop 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- 172.21.15.111:0/2286311194 shutdown_connections 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 --2- 172.21.15.111:0/2286311194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374159b30 0x7f4374068380 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- 172.21.15.111:0/2286311194 >> 172.21.15.111:0/2286311194 conn(0x7f43740c8db0 msgr2=0x7f43740c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- 172.21.15.111:0/2286311194 shutdown_connections 2023-12-19T08:03:44.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.504+0000 7f437982e700 1 -- 172.21.15.111:0/2286311194 wait complete. 2023-12-19T08:03:44.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 Processor -- start 2023-12-19T08:03:44.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 -- start start 2023-12-19T08:03:44.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 0x7f43741501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 0x7f4374157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4374159b30 0x7f43740af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374150710 con 0x7f4374159b30 2023-12-19T08:03:44.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374168bc0 con 0x7f4374155590 2023-12-19T08:03:44.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f437982e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740b1200 con 0x7f43741551c0 2023-12-19T08:03:44.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 0x7f43741501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f43727fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 0x7f4374157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f43727fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 0x7f4374157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:33812/0 (socket says 172.21.15.111:33812) 2023-12-19T08:03:44.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 learned_addr learned my addr 172.21.15.111:0/2247129664 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:44.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f43737fe700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4374159b30 0x7f43740af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 msgr2=0x7f4374157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 0x7f4374157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4374159b30 msgr2=0x7f43740af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4374159b30 0x7f43740af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c009a20 con 0x7f43741551c0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.508+0000 7f4372ffd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 0x7f43741501d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f436400c5d0 tx=0x7f4364007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364005b30 con 0x7f43741551c0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364005c90 con 0x7f43741551c0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43740b14e0 con 0x7f43741551c0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364007300 con 0x7f43741551c0 2023-12-19T08:03:44.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4374156190 con 0x7f43741551c0 2023-12-19T08:03:44.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43741563b0 con 0x7f43741551c0 2023-12-19T08:03:44.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.512+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 42) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f436401b3b0 con 0x7f43741551c0 2023-12-19T08:03:44.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.516+0000 7f435affd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:44.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.516+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f436407c150 con 0x7f43741551c0 2023-12-19T08:03:44.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.516+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:44.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.516+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:03:44.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.524+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f436404c280 con 0x7f43741551c0 2023-12-19T08:03:44.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.712+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4374071120 con 0x7f43600606c0 2023-12-19T08:03:44.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.720+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:44.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:44.720+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:03:44.747 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:44.744+0000 7f0a82eea000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:44.764 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:44.768+0000 7f33133c1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:03:44.820 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:44.816+0000 7f0a82eea000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:44.842 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:44.844+0000 7f33133c1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:45.043 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:45.040+0000 7f0a82eea000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:45.068 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:45.072+0000 7f33133c1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:03:45.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:45.124+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:45.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:45.124+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:03:45.129 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:45.124+0000 7f0a82eea000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:45.156 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:45.160+0000 7f33133c1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:03:45.230 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:45.224+0000 7f0a82eea000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:45.247 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:45.248+0000 7f33133c1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:03:45.632 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:45.628+0000 7f0a82eea000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:45.663 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:45.664+0000 7f33133c1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:03:45.810 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:45.808+0000 7f0a82eea000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:45.825 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:45.828+0000 7f33133c1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:03:45.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:45.928+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:45.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:45.928+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:03:46.186 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.180+0000 7f0a82eea000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:46.200 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.204+0000 7f33133c1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:03:46.402 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.396+0000 7f0a82eea000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:46.416 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.420+0000 7f33133c1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:03:46.491 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.488+0000 7f0a82eea000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:46.501 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.504+0000 7f33133c1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:03:46.574 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.568+0000 7f0a82eea000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:46.575 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.576+0000 7f33133c1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:03:46.643 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.648+0000 7f33133c1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:46.658 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.652+0000 7f0a82eea000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:03:46.871 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.876+0000 7f33133c1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:46.894 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.888+0000 7f0a82eea000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:03:46.950 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:46.952+0000 7f33133c1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:46.974 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:46.972+0000 7f0a82eea000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:03:47.037 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:47.040+0000 7f33133c1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:47.078 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:47.072+0000 7f0a82eea000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:03:47.227 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:47.232+0000 7f33133c1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:47.287 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:47.284+0000 7f0a82eea000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:03:47.313 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:47.316+0000 7f33133c1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:47.380 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:47.376+0000 7f0a82eea000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:03:47.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:47.532+0000 7f43727fc700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:03:47.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:47.532+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:03:48.034 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.036+0000 7f33133c1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:48.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:48.048+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mgrmap(e 43) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f436404c650 con 0x7f43741551c0 2023-12-19T08:03:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:48.048+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:03:48.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:48.048+0000 7f435affd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:48.123 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:03:48.120+0000 7f0a82eea000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:03:48.237 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d55b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.237 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d55b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.237 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d55b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.237 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d55b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.237 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d55b8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d25b2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d25b2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d25b2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d25b2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:48.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:03:48.240+0000 7f32d25b2700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:03:49.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.064+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mon.2 v2:172.21.15.143:3301/0 8 ==== mgrmap(e 44) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f436407a2b0 con 0x7f43741551c0 2023-12-19T08:03:49.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.064+0000 7f435affd700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.064+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/bad-path", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4374071120 con 0x7f43600606c0 2023-12-19T08:03:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.068+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.072+0000 7f43727fc700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f437406e020 tx=0x7f4368007dc0 comp rx=0 tx=0).ready entity=mgr.7390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:49.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.152+0000 7f435affd700 1 -- 172.21.15.111:0/2247129664 <== mgr.7390 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4374071120 con 0x7f43600606c0 2023-12-19T08:03:49.155 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /bad-path 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 msgr2=0x7f4360062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f437406e020 tx=0x7f4368007dc0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 msgr2=0x7f43741501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 0x7f43741501d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f436400c5d0 tx=0x7f4364007d30 comp rx=0 tx=0).stop 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 shutdown_connections 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43600606c0 0x7f4360062b70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4374155590 0x7f4374157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4374159b30 0x7f43740af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 --2- 172.21.15.111:0/2247129664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43741551c0 0x7f43741501d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 >> 172.21.15.111:0/2247129664 conn(0x7f43740c8db0 msgr2=0x7f437406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:49.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 shutdown_connections 2023-12-19T08:03:49.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.156+0000 7f437982e700 1 -- 172.21.15.111:0/2247129664 wait complete. 2023-12-19T08:03:49.188 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-12-19T08:03:49.188 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-12-19T08:03:49.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 Processor -- start 2023-12-19T08:03:49.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 -- start start 2023-12-19T08:03:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24880c0d30 0x7f2488068050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f24880aeb50 0x7f24880ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880ad6b0 con 0x7f24880c1100 2023-12-19T08:03:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2488068730 con 0x7f24880aeb50 2023-12-19T08:03:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2488068a10 con 0x7f24880c0d30 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f2485d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f24880c1100 0x7f24880c8ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46966/0 (socket says 172.21.15.111:46966) 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f2485d9b700 1 -- 172.21.15.111:0/957232776 learned_addr learned my addr 172.21.15.111:0/957232776 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248659c700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24880c0d30 0x7f2488068050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f2486d9d700 1 --2- 172.21.15.111:0/957232776 >> v2:172.21.15.143:3300/0 conn(0x7f24880aeb50 0x7f24880ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297719221 0 0) 0x7f24880ad6b0 con 0x7f24880c1100 2023-12-19T08:03:49.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.356+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2470003570 con 0x7f24880c1100 2023-12-19T08:03:49.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598300323 0 0) 0x7f2470003570 con 0x7f24880c1100 2023-12-19T08:03:49.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24880c0d30 msgr2=0x7f2488068050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:49.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24880c0d30 0x7f2488068050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 >> v2:172.21.15.143:3300/0 conn(0x7f24880aeb50 msgr2=0x7f24880ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 --2- 172.21.15.111:0/957232776 >> v2:172.21.15.143:3300/0 conn(0x7f24880aeb50 0x7f24880ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2488068cf0 con 0x7f24880c1100 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f247c001470 con 0x7f24880c1100 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 >> v1:172.21.15.111:6789/0 conn(0x7f24880c1100 legacy=0x7f24880c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470003a60 0x7f2470003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2470004810 0x7f2470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24700051c0 0x7f247000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2488068730 con 0x7f2470004810 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2488068a10 con 0x7f24700051c0 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247000c020 con 0x7f2470003a60 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2470004810 0x7f2470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2486d9d700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24700051c0 0x7f247000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248659c700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470003a60 0x7f2470003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 -- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470003a60 msgr2=0x7f2470003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470003a60 0x7f2470003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 -- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24700051c0 msgr2=0x7f247000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24700051c0 0x7f247000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 -- 172.21.15.111:0/957232776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470003570 con 0x7f2470004810 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2486d9d700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24700051c0 0x7f247000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248659c700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470003a60 0x7f2470003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2485d9b700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2470004810 0x7f2470004c80 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c3c0 tx=0x7f247c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f31d42e0423422 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c00cc90 con 0x7f2470004810 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c00ce30 con 0x7f2470004810 2023-12-19T08:03:49.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248559a700 1 -- 172.21.15.111:0/957232776 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c005b70 con 0x7f2470004810 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 -- 172.21.15.111:0/957232776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2470004810 msgr2=0x7f2470004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 --2- 172.21.15.111:0/957232776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2470004810 0x7f2470004c80 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c3c0 tx=0x7f247c001e90 comp rx=0 tx=0).stop 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f2486d9d700 1 -- 172.21.15.111:0/957232776 reap_dead start 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 -- 172.21.15.111:0/957232776 shutdown_connections 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 -- 172.21.15.111:0/957232776 >> 172.21.15.111:0/957232776 conn(0x7f24880ce5e0 msgr2=0x7f24880ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 -- 172.21.15.111:0/957232776 shutdown_connections 2023-12-19T08:03:49.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.360+0000 7f248759e700 1 -- 172.21.15.111:0/957232776 wait complete. 2023-12-19T08:03:49.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 Processor -- start 2023-12-19T08:03:49.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 -- start start 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470004810 0x7f248814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 0x7f24880c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880b26e0 con 0x7f248814da00 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248814de50 con 0x7f24880c2710 2023-12-19T08:03:49.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.364+0000 7f248759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24880c2c20 con 0x7f2470004810 2023-12-19T08:03:49.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2485d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 0x7f24880c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f248659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470004810 0x7f248814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2485d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 0x7f24880c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33130/0 (socket says 172.21.15.111:33130) 2023-12-19T08:03:49.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:33844/0 (socket says 172.21.15.111:33844) 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2485d9b700 1 -- 172.21.15.111:0/2892874069 learned_addr learned my addr 172.21.15.111:0/2892874069 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 -- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470004810 msgr2=0x7f248814d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470004810 0x7f248814d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 -- 172.21.15.111:0/2892874069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 msgr2=0x7f24880c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 0x7f24880c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 -- 172.21.15.111:0/2892874069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24880b1db0 con 0x7f24880c2710 2023-12-19T08:03:49.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f2486d9d700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2488160d20 tx=0x7f248000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000fbd0 con 0x7f24880c2710 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248000fd30 con 0x7f24880c2710 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24880b2090 con 0x7f24880c2710 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480016650 con 0x7f24880c2710 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248815cd50 con 0x7f24880c2710 2023-12-19T08:03:49.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.368+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248804be40 con 0x7f24880c2710 2023-12-19T08:03:49.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.372+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 44) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f2480016a00 con 0x7f24880c2710 2023-12-19T08:03:49.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.372+0000 7f24767fc700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 0x7f24680628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:03:49.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.372+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f248007d3a0 con 0x7f24880c2710 2023-12-19T08:03:49.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.372+0000 7f248659c700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 0x7f24680628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:03:49.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.372+0000 7f248659c700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 0x7f24680628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f24880700d0 tx=0x7f2478008040 comp rx=0 tx=0).ready entity=mgr.7390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:03:49.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.380+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f248004d6d0 con 0x7f24880c2710 2023-12-19T08:03:49.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.572+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2488068fd0 con 0x7f24680603f0 2023-12-19T08:03:49.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.584+0000 7f24767fc700 1 -- 172.21.15.111:0/2892874069 <== mgr.7390 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7f2488068fd0 con 0x7f24680603f0 2023-12-19T08:03:49.590 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:03:49.590 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:03:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 msgr2=0x7f24680628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 0x7f24680628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f24880700d0 tx=0x7f2478008040 comp rx=0 tx=0).stop 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 msgr2=0x7f248815f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2488160d20 tx=0x7f248000b040 comp rx=0 tx=0).stop 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 shutdown_connections 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f24680603f0 0x7f24680628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c2710 0x7f248815f510 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f248814da00 0x7f24880c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 --2- 172.21.15.111:0/2892874069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2470004810 0x7f248814d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:03:49.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.588+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 >> 172.21.15.111:0/2892874069 conn(0x7f24880ce5e0 msgr2=0x7f248815ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:03:49.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.592+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 shutdown_connections 2023-12-19T08:03:49.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:03:49.592+0000 7f248759e700 1 -- 172.21.15.111:0/2892874069 wait complete. 2023-12-19T08:03:49.616 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:03:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-12-19T08:04:00.174 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:00.176+0000 7f32c4d97700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:00.175 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:00.176+0000 7f32c4d97700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:00.175 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:00.176+0000 7f32c4d97700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:00.175 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:00.176+0000 7f32c4d97700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:00.175 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:00.176+0000 7f32c4d97700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:06.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:04:06.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:04:36.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:04:36.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:04:49.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /bad-path 2023-12-19T08:04:49.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.787+0000 7fc171123700 1 Processor -- start 2023-12-19T08:04:49.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.787+0000 7fc171123700 1 -- start start 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc16c15cf90 0x7fc16c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc16c15d360 0x7fc16c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c06b550 con 0x7fc16c15d360 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c06af40 con 0x7fc16c15cf90 2023-12-19T08:04:49.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc171123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0ae0c0 con 0x7fc16c06dde0 2023-12-19T08:04:49.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc16c15d360 0x7fc16c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc16c15d360 0x7fc16c0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58692/0 (socket says 172.21.15.111:58692) 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 -- 172.21.15.111:0/693920795 learned_addr learned my addr 172.21.15.111:0/693920795 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc16bfff700 1 --2- 172.21.15.111:0/693920795 >> v2:172.21.15.143:3300/0 conn(0x7fc16c15cf90 0x7fc16c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc170922700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c0aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 -- 172.21.15.111:0/693920795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 msgr2=0x7fc16c0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c0aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 -- 172.21.15.111:0/693920795 >> v2:172.21.15.143:3300/0 conn(0x7fc16c15cf90 msgr2=0x7fc16c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 --2- 172.21.15.111:0/693920795 >> v2:172.21.15.143:3300/0 conn(0x7fc16c15cf90 0x7fc16c06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.791+0000 7fc1637fe700 1 -- 172.21.15.111:0/693920795 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0ae3a0 con 0x7fc16c15d360 2023-12-19T08:04:49.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc1637fe700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c0ae950 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc16c069830 tx=0x7fc15c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7532e8f0ee2722a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:49.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc16b7fe700 1 -- 172.21.15.111:0/693920795 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00b640 con 0x7fc16c15d360 2023-12-19T08:04:49.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc16b7fe700 1 -- 172.21.15.111:0/693920795 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c00b7a0 con 0x7fc16c15d360 2023-12-19T08:04:49.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc16b7fe700 1 -- 172.21.15.111:0/693920795 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00bb60 con 0x7fc16c15d360 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 -- 172.21.15.111:0/693920795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 msgr2=0x7fc16c0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c0ae950 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc16c069830 tx=0x7fc15c009250 comp rx=0 tx=0).stop 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 -- 172.21.15.111:0/693920795 shutdown_connections 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c0aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 --2- 172.21.15.111:0/693920795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c0ae950 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 --2- 172.21.15.111:0/693920795 >> v2:172.21.15.143:3300/0 conn(0x7fc16c15cf90 0x7fc16c06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 -- 172.21.15.111:0/693920795 >> 172.21.15.111:0/693920795 conn(0x7fc16c067bb0 msgr2=0x7fc16c0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 -- 172.21.15.111:0/693920795 shutdown_connections 2023-12-19T08:04:49.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.795+0000 7fc171123700 1 -- 172.21.15.111:0/693920795 wait complete. 2023-12-19T08:04:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 Processor -- start 2023-12-19T08:04:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 -- start start 2023-12-19T08:04:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc16c15cf90 0x7fc16c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c15f3a0 con 0x7fc16c15d360 2023-12-19T08:04:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c15e790 con 0x7fc16c15cf90 2023-12-19T08:04:49.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc171123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c15ea70 con 0x7fc16c06dde0 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.799+0000 7fc170922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58700/0 (socket says 172.21.15.111:58700) 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 -- 172.21.15.111:0/703849321 learned_addr learned my addr 172.21.15.111:0/703849321 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc1637fe700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc16c15cf90 0x7fc16c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc16bfff700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 -- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 msgr2=0x7fc16c153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 -- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc16c15cf90 msgr2=0x7fc16c1518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc16c15cf90 0x7fc16c1518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 -- 172.21.15.111:0/703849321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0094f0 con 0x7fc16c15d360 2023-12-19T08:04:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc170922700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc164009df0 tx=0x7fc16400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16400caa0 con 0x7fc16c15d360 2023-12-19T08:04:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16400f040 con 0x7fc16c15d360 2023-12-19T08:04:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c15ed50 con 0x7fc16c15d360 2023-12-19T08:04:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16400ce70 con 0x7fc16c15d360 2023-12-19T08:04:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c153980 con 0x7fc16c15d360 2023-12-19T08:04:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.803+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c04be40 con 0x7fc16c15d360 2023-12-19T08:04:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.807+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 46) v1 ==== 80194+0+0 (secure 0 0 0) 0x7fc164014690 con 0x7fc16c15d360 2023-12-19T08:04:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.807+0000 7fc168ff9700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 0x7fc14c062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.807+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fc16407d9d0 con 0x7fc16c15d360 2023-12-19T08:04:49.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.807+0000 7fc16bfff700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 0x7fc14c062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.811+0000 7fc16bfff700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 0x7fc14c062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc154009d20 tx=0x7fc1540091a0 comp rx=0 tx=0).ready entity=mgr.7390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:49.815+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc16404d670 con 0x7fc16c15d360 2023-12-19T08:04:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.003+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc16c0c8f30 con 0x7fc14c0607e0 2023-12-19T08:04:50.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.023+0000 7fc168ff9700 1 -- 172.21.15.111:0/703849321 <== mgr.7390 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7fc16c0c8f30 con 0x7fc14c0607e0 2023-12-19T08:04:50.030 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:50.031 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:03:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-12-19T08:04:50.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 msgr2=0x7fc14c062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 0x7fc14c062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc154009d20 tx=0x7fc1540091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 msgr2=0x7fc16c151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc164009df0 tx=0x7fc16400b040 comp rx=0 tx=0).stop 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 shutdown_connections 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc14c0607e0 0x7fc14c062c90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc16c15cf90 0x7fc16c1518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc16c15d360 0x7fc16c151e20 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 --2- 172.21.15.111:0/703849321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc16c06dde0 0x7fc16c153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 >> 172.21.15.111:0/703849321 conn(0x7fc16c067bb0 msgr2=0x7fc16c0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 shutdown_connections 2023-12-19T08:04:50.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.027+0000 7fc171123700 1 -- 172.21.15.111:0/703849321 wait complete. 2023-12-19T08:04:50.051 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:03:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-12-19T08:04:50.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /bad-path 2023-12-19T08:04:50.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.211+0000 7f5e64944700 1 Processor -- start 2023-12-19T08:04:50.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.211+0000 7f5e64944700 1 -- start start 2023-12-19T08:04:50.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e64944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e6006af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e64944700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600cb3f0 con 0x7f5e600cac50 2023-12-19T08:04:50.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e64944700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6006d220 con 0x7f5e600aedc0 2023-12-19T08:04:50.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e64944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600cc140 con 0x7f5e600af190 2023-12-19T08:04:50.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5f59e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5e600cac50 0x7f5e600af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58578/0 (socket says 172.21.15.111:58578) 2023-12-19T08:04:50.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5f59e700 1 -- 172.21.15.111:0/3029283889 learned_addr learned my addr 172.21.15.111:0/3029283889 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:50.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5e59c700 1 --2- 172.21.15.111:0/3029283889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e6006af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943434481 0 0) 0x7f5e600cb3f0 con 0x7f5e600cac50 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e48003570 con 0x7f5e600cac50 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374708844 0 0) 0x7f5e6006d220 con 0x7f5e600aedc0 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e600cb3f0 con 0x7f5e600aedc0 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5e59c700 1 -- 172.21.15.111:0/3029283889 >> v1:172.21.15.143:6789/0 conn(0x7f5e600aedc0 legacy=0x7f5e6006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5e59c700 1 -- 172.21.15.111:0/3029283889 >> v1:172.21.15.111:6789/0 conn(0x7f5e600cac50 legacy=0x7f5e600af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.215+0000 7f5e5e59c700 1 -- 172.21.15.111:0/3029283889 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6006c8f0 con 0x7f5e600af190 2023-12-19T08:04:50.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e5e59c700 1 --2- 172.21.15.111:0/3029283889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e6006af50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009910 tx=0x7f5e540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb9d284e4724f28 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54011630 con 0x7f5e600af190 2023-12-19T08:04:50.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54011790 con 0x7f5e600af190 2023-12-19T08:04:50.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e5dd9b700 1 -- 172.21.15.111:0/3029283889 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54011b30 con 0x7f5e600af190 2023-12-19T08:04:50.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 -- 172.21.15.111:0/3029283889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 msgr2=0x7f5e6006af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 --2- 172.21.15.111:0/3029283889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e6006af50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009910 tx=0x7f5e540091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 -- 172.21.15.111:0/3029283889 shutdown_connections 2023-12-19T08:04:50.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 --2- 172.21.15.111:0/3029283889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e6006af50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 -- 172.21.15.111:0/3029283889 >> 172.21.15.111:0/3029283889 conn(0x7f5e600cc760 msgr2=0x7f5e600ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:50.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 -- 172.21.15.111:0/3029283889 shutdown_connections 2023-12-19T08:04:50.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.219+0000 7f5e64944700 1 -- 172.21.15.111:0/3029283889 wait complete. 2023-12-19T08:04:50.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 Processor -- start 2023-12-19T08:04:50.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 -- start start 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e600b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e600cac50 0x7f5e600c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c3e90 con 0x7f5e600aedc0 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600b4660 con 0x7f5e600cac50 2023-12-19T08:04:50.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c38d0 con 0x7f5e600af190 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58706/0 (socket says 172.21.15.111:58706) 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 -- 172.21.15.111:0/884795586 learned_addr learned my addr 172.21.15.111:0/884795586 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5f59e700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e600cac50 0x7f5e600c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5e59c700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e600b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 -- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 msgr2=0x7f5e600b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e600b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 -- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e600cac50 msgr2=0x7f5e600c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e600cac50 0x7f5e600c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 -- 172.21.15.111:0/884795586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e54009420 con 0x7f5e600aedc0 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e5ed9d700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c5b0 tx=0x7f5e50007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50010510 con 0x7f5e600aedc0 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50010670 con 0x7f5e600aedc0 2023-12-19T08:04:50.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e600c3bb0 con 0x7f5e600aedc0 2023-12-19T08:04:50.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.223+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50010a30 con 0x7f5e600aedc0 2023-12-19T08:04:50.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.227+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e600b5030 con 0x7f5e600aedc0 2023-12-19T08:04:50.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.227+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6004be40 con 0x7f5e600aedc0 2023-12-19T08:04:50.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.227+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 46) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f5e5001b400 con 0x7f5e600aedc0 2023-12-19T08:04:50.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.227+0000 7f5e46ffd700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 0x7f5e4c062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.231+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f5e5007cdc0 con 0x7f5e600aedc0 2023-12-19T08:04:50.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.231+0000 7f5e5e59c700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 0x7f5e4c062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.231+0000 7f5e5e59c700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 0x7f5e4c062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009860 tx=0x7f5e54017000 comp rx=0 tx=0).ready entity=mgr.7390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.235+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5e5004cb70 con 0x7f5e600aedc0 2023-12-19T08:04:50.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.419+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e600b4490 con 0x7f5e4c0607e0 2023-12-19T08:04:50.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.523+0000 7f5e46ffd700 1 -- 172.21.15.111:0/884795586 <== mgr.7390 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f5e54000bc0 con 0x7f5e4c0607e0 2023-12-19T08:04:50.530 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /bad-path 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 msgr2=0x7f5e4c062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 0x7f5e4c062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009860 tx=0x7f5e54017000 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 msgr2=0x7f5e600b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c5b0 tx=0x7f5e50007e10 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 shutdown_connections 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5e4c0607e0 0x7f5e4c062c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e600cac50 0x7f5e600c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e600af190 0x7f5e600b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 --2- 172.21.15.111:0/884795586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e600aedc0 0x7f5e600b4af0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 >> 172.21.15.111:0/884795586 conn(0x7f5e600cc760 msgr2=0x7f5e600afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 shutdown_connections 2023-12-19T08:04:50.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.527+0000 7f5e64944700 1 -- 172.21.15.111:0/884795586 wait complete. 2023-12-19T08:04:50.558 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-12-19T08:04:50.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T08:04:50.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.723+0000 7f51c9616700 1 Processor -- start 2023-12-19T08:04:50.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.723+0000 7f51c9616700 1 -- start start 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c9616700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51c4154780 0x7f51c415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c9616700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 0x7f51c4157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c9616700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c415ae30 con 0x7f51c40adab0 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c9616700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c40abf70 con 0x7f51c4154780 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c9616700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4154b50 con 0x7f51c406baa0 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51c4154780 0x7f51c415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c8e15700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f51c40adab0 0x7f51c40adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58582/0 (socket says 172.21.15.111:58582) 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c8e15700 1 -- 172.21.15.111:0/3096921929 learned_addr learned my addr 172.21.15.111:0/3096921929 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.727+0000 7f51c37fe700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 0x7f51c4157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c3fff700 1 -- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 msgr2=0x7f51c4157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c3fff700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 0x7f51c4157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c3fff700 1 -- 172.21.15.111:0/3096921929 >> v1:172.21.15.111:6789/0 conn(0x7f51c40adab0 legacy=0x7f51c40adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c3fff700 1 -- 172.21.15.111:0/3096921929 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c41580a0 con 0x7f51c4154780 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c37fe700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 0x7f51c4157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c3fff700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c4154780 0x7f51c415b2c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0a9b9b1eb4ac90 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c2ffd700 1 -- 172.21.15.111:0/3096921929 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4009e80 con 0x7f51c4154780 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c2ffd700 1 -- 172.21.15.111:0/3096921929 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b400b860 con 0x7f51c4154780 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c2ffd700 1 -- 172.21.15.111:0/3096921929 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400bbe0 con 0x7f51c4154780 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 -- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c4154780 msgr2=0x7f51c415b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c4154780 0x7f51c415b2c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f51b40098e0 tx=0x7f51b40091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 -- 172.21.15.111:0/3096921929 shutdown_connections 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c406baa0 0x7f51c4157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 --2- 172.21.15.111:0/3096921929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c4154780 0x7f51c415b2c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 -- 172.21.15.111:0/3096921929 >> 172.21.15.111:0/3096921929 conn(0x7f51c4067b50 msgr2=0x7f51c40cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 -- 172.21.15.111:0/3096921929 shutdown_connections 2023-12-19T08:04:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.731+0000 7f51c9616700 1 -- 172.21.15.111:0/3096921929 wait complete. 2023-12-19T08:04:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 Processor -- start 2023-12-19T08:04:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 -- start start 2023-12-19T08:04:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c40adab0 0x7f51c41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 0x7f51c40c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4159890 con 0x7f51c406baa0 2023-12-19T08:04:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4158c80 con 0x7f51c40adab0 2023-12-19T08:04:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c9616700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4158f60 con 0x7f51c4154780 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c40adab0 0x7f51c41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58714/0 (socket says 172.21.15.111:58714) 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 -- 172.21.15.111:0/2139311097 learned_addr learned my addr 172.21.15.111:0/2139311097 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c8e15700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 0x7f51c40c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 -- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 msgr2=0x7f51c40c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 0x7f51c40c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 -- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c40adab0 msgr2=0x7f51c41582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:50.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c40adab0 0x7f51c41582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:50.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c3fff700 1 -- 172.21.15.111:0/2139311097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4009420 con 0x7f51c406baa0 2023-12-19T08:04:50.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.735+0000 7f51c8e15700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 0x7f51c40c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:04:50.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51c3fff700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f51c406a0d0 tx=0x7f51b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400bbe0 con 0x7f51c406baa0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4000cc0 con 0x7f51c406baa0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4159240 con 0x7f51c406baa0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4011650 con 0x7f51c406baa0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c41428e0 con 0x7f51c406baa0 2023-12-19T08:04:50.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c40ace50 con 0x7f51c406baa0 2023-12-19T08:04:50.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.739+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 46) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f51b4011a40 con 0x7f51c406baa0 2023-12-19T08:04:50.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.743+0000 7f51abfff700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 0x7f51b0062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:50.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.743+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f51b407e980 con 0x7f51c406baa0 2023-12-19T08:04:50.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.743+0000 7f51c37fe700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 0x7f51b0062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:50.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.743+0000 7f51c37fe700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 0x7f51b0062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f51c406dda0 tx=0x7f51b8008040 comp rx=0 tx=0).ready entity=mgr.7390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:50.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.747+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f51c40ace50 con 0x7f51c406baa0 2023-12-19T08:04:50.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:50.955+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f51c40ace50 con 0x7f51c406baa0 2023-12-19T08:04:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.587+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f51b404eb00 con 0x7f51c406baa0 2023-12-19T08:04:51.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.591+0000 7f51abfff700 1 -- 172.21.15.111:0/2139311097 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v47) v1 ==== 93+0+0 (secure 0 0 0) 0x7f51b40253f0 con 0x7f51c406baa0 2023-12-19T08:04:51.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 msgr2=0x7f51b0062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 0x7f51b0062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f51c406dda0 tx=0x7f51b8008040 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 msgr2=0x7f51c41423a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f51c406a0d0 tx=0x7f51b4017000 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 shutdown_connections 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00607e0 0x7f51b0062c90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c40adab0 0x7f51c41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c4154780 0x7f51c40c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 --2- 172.21.15.111:0/2139311097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406baa0 0x7f51c41423a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 >> 172.21.15.111:0/2139311097 conn(0x7f51c4067b50 msgr2=0x7f51c4154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 shutdown_connections 2023-12-19T08:04:51.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:51.595+0000 7f51c9616700 1 -- 172.21.15.111:0/2139311097 wait complete. 2023-12-19T08:04:51.621 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:04:51.621 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:04:51.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:04:51.702 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:04:51.703 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T08:04:51.703 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T08:04:51.703 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T08:04:51.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[52291]: fuse finished with error 0 and tester_r 0 2023-12-19T08:04:51.743 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:04:51.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:04:51.743 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T08:04:51.780 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:51.775+0000 7f295fdab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:04:51.784 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:51.786+0000 7f4e0a767000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:04:51.890 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:04:51.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:04:51.890 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T08:04:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:04:51.912 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T08:04:51.912 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:04:51.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:04:51.913 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T08:04:51.929 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T08:04:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:04:51.931 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:04:51.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:04:51.931 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T08:04:51.974 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T08:04:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:04:51.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:04:52.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.131+0000 7f561426e700 1 Processor -- start 2023-12-19T08:04:52.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.131+0000 7f561426e700 1 -- start start 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561426e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f560c140710 0x7f560c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561426e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 0x7f560c0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561426e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c06e150 con 0x7f560c140ae0 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561426e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f560c06d540 con 0x7f560c140710 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561426e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c06d820 con 0x7f560c068800 2023-12-19T08:04:52.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f5611809700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f560c140ae0 0x7f560c0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43690/0 (socket says 172.21.15.111:43690) 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f5611809700 1 -- 172.21.15.111:0/4093119052 learned_addr learned my addr 172.21.15.111:0/4093119052 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 --2- 172.21.15.111:0/4093119052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 0x7f560c0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561200a700 1 --2- 172.21.15.111:0/4093119052 >> v2:172.21.15.143:3300/0 conn(0x7f560c140710 0x7f560c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 -- 172.21.15.111:0/4093119052 >> v2:172.21.15.143:3300/0 conn(0x7f560c140710 msgr2=0x7f560c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 --2- 172.21.15.111:0/4093119052 >> v2:172.21.15.143:3300/0 conn(0x7f560c140710 0x7f560c0c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f5611008700 1 -- 172.21.15.111:0/4093119052 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570009028 0 0) 0x7f560c06e150 con 0x7f560c140ae0 2023-12-19T08:04:52.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 -- 172.21.15.111:0/4093119052 >> v1:172.21.15.111:6789/0 conn(0x7f560c140ae0 legacy=0x7f560c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 -- 172.21.15.111:0/4093119052 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c06db00 con 0x7f560c068800 2023-12-19T08:04:52.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.135+0000 7f561280b700 1 --2- 172.21.15.111:0/4093119052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 0x7f560c0ac250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5604009e60 tx=0x7f5604009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891cca4fdddbc350 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:52.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f5611008700 1 -- 172.21.15.111:0/4093119052 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560400e070 con 0x7f560c068800 2023-12-19T08:04:52.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f5611008700 1 -- 172.21.15.111:0/4093119052 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560400bb60 con 0x7f560c068800 2023-12-19T08:04:52.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f5611008700 1 -- 172.21.15.111:0/4093119052 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604012430 con 0x7f560c068800 2023-12-19T08:04:52.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 -- 172.21.15.111:0/4093119052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 msgr2=0x7f560c0ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 --2- 172.21.15.111:0/4093119052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 0x7f560c0ac250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5604009e60 tx=0x7f5604009780 comp rx=0 tx=0).stop 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 -- 172.21.15.111:0/4093119052 shutdown_connections 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 --2- 172.21.15.111:0/4093119052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c068800 0x7f560c0ac250 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 --2- 172.21.15.111:0/4093119052 >> v2:172.21.15.143:3300/0 conn(0x7f560c140710 0x7f560c0c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 -- 172.21.15.111:0/4093119052 >> 172.21.15.111:0/4093119052 conn(0x7f560c0c8850 msgr2=0x7f560c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 -- 172.21.15.111:0/4093119052 shutdown_connections 2023-12-19T08:04:52.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.139+0000 7f561426e700 1 -- 172.21.15.111:0/4093119052 wait complete. 2023-12-19T08:04:52.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 Processor -- start 2023-12-19T08:04:52.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 -- start start 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c068800 0x7f560c167d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c140710 0x7f560c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c159fa0 con 0x7f560c140ae0 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c159cc0 con 0x7f560c068800 2023-12-19T08:04:52.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561426e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560c0aed40 con 0x7f560c140710 2023-12-19T08:04:52.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33860/0 (socket says 172.21.15.111:33860) 2023-12-19T08:04:52.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 -- 172.21.15.111:0/100218067 learned_addr learned my addr 172.21.15.111:0/100218067 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:52.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561200a700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c068800 0x7f560c167d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f5611809700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c140710 0x7f560c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c140710 msgr2=0x7f560c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c140710 0x7f560c156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c068800 msgr2=0x7f560c167d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c068800 0x7f560c167d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 -- 172.21.15.111:0/100218067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604009a20 con 0x7f560c140ae0 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f561280b700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f560c06a280 tx=0x7f5604019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:52.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604027470 con 0x7f560c140ae0 2023-12-19T08:04:52.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.143+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56040275d0 con 0x7f560c140ae0 2023-12-19T08:04:52.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.147+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c0af020 con 0x7f560c140ae0 2023-12-19T08:04:52.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.147+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604027990 con 0x7f560c140ae0 2023-12-19T08:04:52.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.147+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c156e40 con 0x7f560c140ae0 2023-12-19T08:04:52.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.147+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c0aeba0 con 0x7f560c140ae0 2023-12-19T08:04:52.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.147+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f5604024070 con 0x7f560c140ae0 2023-12-19T08:04:52.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.151+0000 7f56027fc700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 0x7f55f0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.151+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f560407f830 con 0x7f560c140ae0 2023-12-19T08:04:52.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.151+0000 7f561200a700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 msgr2=0x7f55f0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:52.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.151+0000 7f561200a700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 0x7f55f0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:52.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.155+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f560c0aeba0 con 0x7f560c140ae0 2023-12-19T08:04:52.330 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:52.330+0000 7f4e0a767000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:04:52.345 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:52.339+0000 7f295fdab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:04:52.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.355+0000 7f561200a700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 msgr2=0x7f55f0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:52.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.355+0000 7f561200a700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 0x7f55f0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:04:52.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.363+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f560c0aeba0 con 0x7f560c140ae0 2023-12-19T08:04:52.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.363+0000 7f56027fc700 1 -- 172.21.15.111:0/100218067 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f560c0aeba0 con 0x7f560c140ae0 2023-12-19T08:04:52.371 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:52.371 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:04:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 msgr2=0x7f55f0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 0x7f55f0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 msgr2=0x7f560c0b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f560c06a280 tx=0x7f5604019000 comp rx=0 tx=0).stop 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 shutdown_connections 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f55f0060790 0x7f55f0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f560c140ae0 0x7f560c0b0f30 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f560c140710 0x7f560c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 --2- 172.21.15.111:0/100218067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f560c068800 0x7f560c167d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 >> 172.21.15.111:0/100218067 conn(0x7f560c0c8850 msgr2=0x7f560c0ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 shutdown_connections 2023-12-19T08:04:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.367+0000 7f561426e700 1 -- 172.21.15.111:0/100218067 wait complete. 2023-12-19T08:04:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:04:52.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:04:52.411 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:52.410+0000 7f4e0a767000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:04:52.419 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:52.415+0000 7f295fdab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:04:52.555 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:52.554+0000 7f4e0a767000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:04:52.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.559+0000 7ffa294f0700 1 Processor -- start 2023-12-19T08:04:52.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.559+0000 7ffa294f0700 1 -- start start 2023-12-19T08:04:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa2413f890 0x7ffa240aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 0x7ffa24068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2406c9f0 con 0x7ffa240c2980 2023-12-19T08:04:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa240c7f20 con 0x7ffa2413f890 2023-12-19T08:04:52.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa24068a50 con 0x7ffa240680f0 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa2413f890 0x7ffa240aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa2413f890 0x7ffa240aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33876/0 (socket says 172.21.15.111:33876) 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 -- 172.21.15.111:0/466349054 learned_addr learned my addr 172.21.15.111:0/466349054 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa237fe700 1 --2- 172.21.15.111:0/466349054 >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 0x7ffa24068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 -- 172.21.15.111:0/466349054 >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 msgr2=0x7ffa24068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 --2- 172.21.15.111:0/466349054 >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 0x7ffa24068510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 -- 172.21.15.111:0/466349054 >> v1:172.21.15.143:6789/0 conn(0x7ffa240c2980 legacy=0x7ffa240c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 -- 172.21.15.111:0/466349054 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240c1c50 con 0x7ffa2413f890 2023-12-19T08:04:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa237fe700 1 --2- 172.21.15.111:0/466349054 >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 0x7ffa24068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:04:52.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa22ffd700 1 --2- 172.21.15.111:0/466349054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240aeba0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ffa2406a6a0 tx=0x7ffa14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eafc93ab2ca71b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa227fc700 1 -- 172.21.15.111:0/466349054 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400e070 con 0x7ffa2413f890 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa227fc700 1 -- 172.21.15.111:0/466349054 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1400ba10 con 0x7ffa2413f890 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa227fc700 1 -- 172.21.15.111:0/466349054 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa140092e0 con 0x7ffa2413f890 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- 172.21.15.111:0/466349054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 msgr2=0x7ffa240aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 --2- 172.21.15.111:0/466349054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240aeba0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ffa2406a6a0 tx=0x7ffa14009840 comp rx=0 tx=0).stop 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- 172.21.15.111:0/466349054 shutdown_connections 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 --2- 172.21.15.111:0/466349054 >> v2:172.21.15.143:3300/0 conn(0x7ffa240680f0 0x7ffa24068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 --2- 172.21.15.111:0/466349054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240aeba0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- 172.21.15.111:0/466349054 >> 172.21.15.111:0/466349054 conn(0x7ffa240c8850 msgr2=0x7ffa240c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- 172.21.15.111:0/466349054 shutdown_connections 2023-12-19T08:04:52.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.563+0000 7ffa294f0700 1 -- 172.21.15.111:0/466349054 wait complete. 2023-12-19T08:04:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 Processor -- start 2023-12-19T08:04:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 -- start start 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa240680f0 0x7ffa2413dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa240c2980 0x7ffa240c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2415e560 con 0x7ffa2413f890 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2415dfa0 con 0x7ffa240680f0 2023-12-19T08:04:52.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.567+0000 7ffa294f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2415e280 con 0x7ffa240c2980 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33886/0 (socket says 172.21.15.111:33886) 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 -- 172.21.15.111:0/3060147759 learned_addr learned my addr 172.21.15.111:0/3060147759 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa1a7fc700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa240c2980 0x7ffa240c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa22ffd700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa240680f0 0x7ffa2413dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 -- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa240c2980 msgr2=0x7ffa240c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa240c2980 0x7ffa240c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 -- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa240680f0 msgr2=0x7ffa2413dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa240680f0 0x7ffa2413dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 -- 172.21.15.111:0/3060147759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14009ae0 con 0x7ffa2413f890 2023-12-19T08:04:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa237fe700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c880 tx=0x7ffa1c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011600 con 0x7ffa2413f890 2023-12-19T08:04:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c011780 con 0x7ffa2413f890 2023-12-19T08:04:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa24168d80 con 0x7ffa2413f890 2023-12-19T08:04:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011b40 con 0x7ffa2413f890 2023-12-19T08:04:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2415d200 con 0x7ffa2413f890 2023-12-19T08:04:52.578 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:52.571+0000 7f295fdab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:04:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.571+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa24169280 con 0x7ffa2413f890 2023-12-19T08:04:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.574+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7ffa1c022350 con 0x7ffa2413f890 2023-12-19T08:04:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.574+0000 7ffa1b7fe700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 0x7ffa04062cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.574+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7ffa1c07d170 con 0x7ffa2413f890 2023-12-19T08:04:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.574+0000 7ffa22ffd700 1 -- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 msgr2=0x7ffa04062cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.574+0000 7ffa22ffd700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 0x7ffa04062cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.578+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffa1c04ce10 con 0x7ffa2413f890 2023-12-19T08:04:52.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.754+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa24071120 con 0x7ffa2413f890 2023-12-19T08:04:52.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.754+0000 7ffa1b7fe700 1 -- 172.21.15.111:0/3060147759 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17097 (secure 0 0 0) 0x7ffa1c050430 con 0x7ffa2413f890 2023-12-19T08:04:52.762 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:52.762 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":116,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:04:50.566091+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2526647480":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/144894101":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1370433618":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/596992054":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/4178727723":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1808666646":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:48.038487+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 msgr2=0x7ffa04062cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 0x7ffa04062cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 msgr2=0x7ffa240c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c880 tx=0x7ffa1c008040 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 shutdown_connections 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffa04060840 0x7ffa04062cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa2413f890 0x7ffa240c5020 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa240c2980 0x7ffa240c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 --2- 172.21.15.111:0/3060147759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa240680f0 0x7ffa2413dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 >> 172.21.15.111:0/3060147759 conn(0x7ffa240c8850 msgr2=0x7ffa240b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 shutdown_connections 2023-12-19T08:04:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.758+0000 7ffa294f0700 1 -- 172.21.15.111:0/3060147759 wait complete. 2023-12-19T08:04:52.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:03:31.360391+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:03:32.348699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T08:04:52.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:04:52.831 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:52.830+0000 7f4e0a767000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:04:52.869 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:52.862+0000 7f295fdab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:04:52.909 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:52.910+0000 7f4e0a767000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:04:52.948 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:52.942+0000 7f295fdab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:04:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 Processor -- start 2023-12-19T08:04:53.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 -- start start 2023-12-19T08:04:53.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f782c154780 0x7f782c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c15ae30 con 0x7f782c06baa0 2023-12-19T08:04:53.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c0abf70 con 0x7f782c154780 2023-12-19T08:04:53.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7831ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c154b50 con 0x7f782c0adab0 2023-12-19T08:04:53.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f78316e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f78316e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:60016/0 (socket says 172.21.15.111:60016) 2023-12-19T08:04:53.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f78316e2700 1 -- 172.21.15.111:0/2036695246 learned_addr learned my addr 172.21.15.111:0/2036695246 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:52.998+0000 7f7830ee1700 1 --2- 172.21.15.111:0/2036695246 >> v2:172.21.15.143:3300/0 conn(0x7f782c154780 0x7f782c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f78316e2700 1 -- 172.21.15.111:0/2036695246 >> v2:172.21.15.143:3300/0 conn(0x7f782c154780 msgr2=0x7f782c15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f78316e2700 1 --2- 172.21.15.111:0/2036695246 >> v2:172.21.15.143:3300/0 conn(0x7f782c154780 0x7f782c15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f78316e2700 1 -- 172.21.15.111:0/2036695246 >> v1:172.21.15.143:6789/0 conn(0x7f782c06baa0 legacy=0x7f782c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f78316e2700 1 -- 172.21.15.111:0/2036695246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c0ac210 con 0x7f782c0adab0 2023-12-19T08:04:53.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f78316e2700 1 --2- 172.21.15.111:0/2036695246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7824009f20 tx=0x7f7824009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e442990c134a33 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f782b7fe700 1 -- 172.21.15.111:0/2036695246 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782400e070 con 0x7f782c0adab0 2023-12-19T08:04:53.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f782b7fe700 1 -- 172.21.15.111:0/2036695246 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782400ba80 con 0x7f782c0adab0 2023-12-19T08:04:53.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f782b7fe700 1 -- 172.21.15.111:0/2036695246 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78240092e0 con 0x7f782c0adab0 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 -- 172.21.15.111:0/2036695246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 msgr2=0x7f782c0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2036695246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7824009f20 tx=0x7f7824009840 comp rx=0 tx=0).stop 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 -- 172.21.15.111:0/2036695246 shutdown_connections 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2036695246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c0adab0 0x7f782c0adef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2036695246 >> v2:172.21.15.143:3300/0 conn(0x7f782c154780 0x7f782c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 -- 172.21.15.111:0/2036695246 >> 172.21.15.111:0/2036695246 conn(0x7f782c067b50 msgr2=0x7f782c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 -- 172.21.15.111:0/2036695246 shutdown_connections 2023-12-19T08:04:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.002+0000 7f7831ee3700 1 -- 172.21.15.111:0/2036695246 wait complete. 2023-12-19T08:04:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 Processor -- start 2023-12-19T08:04:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 -- start start 2023-12-19T08:04:53.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 0x7f782c167010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c154780 0x7f782c0afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0b1910 con 0x7f782c0adab0 2023-12-19T08:04:53.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0af990 con 0x7f782c06baa0 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.006+0000 7f7831ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0af060 con 0x7f782c154780 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33890/0 (socket says 172.21.15.111:33890) 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 -- 172.21.15.111:0/2218974516 learned_addr learned my addr 172.21.15.111:0/2218974516 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f78316e2700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c154780 0x7f782c0afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7830ee1700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 0x7f782c167010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 -- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c154780 msgr2=0x7f782c0afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c154780 0x7f782c0afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 -- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 msgr2=0x7f782c167010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 0x7f782c167010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 -- 172.21.15.111:0/2218974516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824009ae0 con 0x7f782c0adab0 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7830ee1700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 0x7f782c167010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:04:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f782bfff700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f781800c590 tx=0x7f7818007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818011590 con 0x7f782c0adab0 2023-12-19T08:04:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7818011710 con 0x7f782c0adab0 2023-12-19T08:04:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c0af340 con 0x7f782c0adab0 2023-12-19T08:04:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818011ad0 con 0x7f782c0adab0 2023-12-19T08:04:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c151a40 con 0x7f782c0adab0 2023-12-19T08:04:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.010+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c0afb40 con 0x7f782c0adab0 2023-12-19T08:04:53.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.014+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f781801b3b0 con 0x7f782c0adab0 2023-12-19T08:04:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.014+0000 7f7828ff9700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 0x7f7810062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.014+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f781807c440 con 0x7f782c0adab0 2023-12-19T08:04:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.014+0000 7f7830ee1700 1 -- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 msgr2=0x7f7810062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.014+0000 7f7830ee1700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 0x7f7810062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:53.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.022+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f782c0afb40 con 0x7f782c0adab0 2023-12-19T08:04:53.134 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:53.134+0000 7f4e0a767000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:04:53.178 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:53.170+0000 7f295fdab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:04:53.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.190+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f782c0afb00 con 0x7f782c0adab0 2023-12-19T08:04:53.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.190+0000 7f7828ff9700 1 -- 172.21.15.111:0/2218974516 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f781804c1f0 con 0x7f782c0adab0 2023-12-19T08:04:53.198 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:04:53.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 msgr2=0x7f7810062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:53.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 0x7f7810062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 msgr2=0x7f782c151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f781800c590 tx=0x7f7818007f50 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 shutdown_connections 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.194+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7810060790 0x7f7810062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f782c154780 0x7f782c0afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f782c0adab0 0x7f782c151500 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 --2- 172.21.15.111:0/2218974516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f782c06baa0 0x7f782c167010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 >> 172.21.15.111:0/2218974516 conn(0x7f782c067b50 msgr2=0x7f782c0c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 shutdown_connections 2023-12-19T08:04:53.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.198+0000 7f7831ee3700 1 -- 172.21.15.111:0/2218974516 wait complete. 2023-12-19T08:04:53.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:53.222+0000 7f4e0a767000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:04:53.224 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T08:04:53.262 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:53.254+0000 7f295fdab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:04:53.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:53.306+0000 7f4e0a767000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:04:53.356 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:53.350+0000 7f295fdab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:04:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 Processor -- start 2023-12-19T08:04:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 -- start start 2023-12-19T08:04:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59580ca240 0x7f59580ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59580cabc0 con 0x7f595815c520 2023-12-19T08:04:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595806b520 con 0x7f59580ca240 2023-12-19T08:04:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595e6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59580cb910 con 0x7f595815c8f0 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f594ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f594ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:60036/0 (socket says 172.21.15.111:60036) 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f594ffff700 1 -- 172.21.15.111:0/651288921 learned_addr learned my addr 172.21.15.111:0/651288921 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.414+0000 7f595cc8d700 1 --2- 172.21.15.111:0/651288921 >> v2:172.21.15.143:3300/0 conn(0x7f59580ca240 0x7f59580ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f594ffff700 1 -- 172.21.15.111:0/651288921 >> v2:172.21.15.143:3300/0 conn(0x7f59580ca240 msgr2=0x7f59580ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:53.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f594ffff700 1 --2- 172.21.15.111:0/651288921 >> v2:172.21.15.143:3300/0 conn(0x7f59580ca240 0x7f59580ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f594ffff700 1 -- 172.21.15.111:0/651288921 >> v1:172.21.15.143:6789/0 conn(0x7f595815c520 legacy=0x7f595806a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f594ffff700 1 -- 172.21.15.111:0/651288921 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59580c6ab0 con 0x7f595815c8f0 2023-12-19T08:04:53.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f594ffff700 1 --2- 172.21.15.111:0/651288921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f59400099b0 tx=0x7f5940009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f85f3bbc889b27 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f59577fe700 1 -- 172.21.15.111:0/651288921 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940009ea0 con 0x7f595815c8f0 2023-12-19T08:04:53.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f59577fe700 1 -- 172.21.15.111:0/651288921 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594000b8f0 con 0x7f595815c8f0 2023-12-19T08:04:53.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f59577fe700 1 -- 172.21.15.111:0/651288921 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594000bc90 con 0x7f595815c8f0 2023-12-19T08:04:53.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 -- 172.21.15.111:0/651288921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 msgr2=0x7f59580a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 --2- 172.21.15.111:0/651288921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f59400099b0 tx=0x7f5940009250 comp rx=0 tx=0).stop 2023-12-19T08:04:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 -- 172.21.15.111:0/651288921 shutdown_connections 2023-12-19T08:04:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 --2- 172.21.15.111:0/651288921 >> v2:172.21.15.143:3300/0 conn(0x7f59580ca240 0x7f59580ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 --2- 172.21.15.111:0/651288921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c8f0 0x7f59580a8850 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 -- 172.21.15.111:0/651288921 >> 172.21.15.111:0/651288921 conn(0x7f5958067bb0 msgr2=0x7f59580cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 -- 172.21.15.111:0/651288921 shutdown_connections 2023-12-19T08:04:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.418+0000 7f595e6f0700 1 -- 172.21.15.111:0/651288921 wait complete. 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 Processor -- start 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 -- start start 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59580ca240 0x7f59581488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595815c8f0 0x7f59580c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580affa0 con 0x7f595815c8f0 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580c5910 con 0x7f59580ca240 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595e6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59580af3d0 con 0x7f595815c520 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:60040/0 (socket says 172.21.15.111:60040) 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 -- 172.21.15.111:0/3184825410 learned_addr learned my addr 172.21.15.111:0/3184825410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f595cc8d700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595815c8f0 0x7f59580c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f5957fff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59580ca240 0x7f59581488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59580ca240 msgr2=0x7f59581488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59580ca240 0x7f59581488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595815c8f0 msgr2=0x7f59580c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595815c8f0 0x7f59580c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 -- 172.21.15.111:0/3184825410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400094f0 con 0x7f595815c520 2023-12-19T08:04:53.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.422+0000 7f594ffff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f595806a040 tx=0x7f5940017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594000bc70 con 0x7f595815c520 2023-12-19T08:04:53.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594001a450 con 0x7f595815c520 2023-12-19T08:04:53.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59580af670 con 0x7f595815c520 2023-12-19T08:04:53.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594001a810 con 0x7f595815c520 2023-12-19T08:04:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59580afc30 con 0x7f595815c520 2023-12-19T08:04:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59580b0110 con 0x7f595815c520 2023-12-19T08:04:53.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.426+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f594001abc0 con 0x7f595815c520 2023-12-19T08:04:53.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.430+0000 7f5954ff9700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 0x7f5938062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.430+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f594007eaf0 con 0x7f595815c520 2023-12-19T08:04:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.430+0000 7f5957fff700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 msgr2=0x7f5938062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:53.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.430+0000 7f5957fff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 0x7f5938062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:53.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.438+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f594004e920 con 0x7f595815c520 2023-12-19T08:04:53.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.630+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f595806b240 con 0x7f595815c520 2023-12-19T08:04:53.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.630+0000 7f5954ff9700 1 -- 172.21.15.111:0/3184825410 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5940051f40 con 0x7f595815c520 2023-12-19T08:04:53.639 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T08:04:53.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f5957fff700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 msgr2=0x7f5938062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:53.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f5957fff700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 0x7f5938062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 msgr2=0x7f5938062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 0x7f5938062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 msgr2=0x7f59580c4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f595806a040 tx=0x7f5940017000 comp rx=0 tx=0).stop 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 shutdown_connections 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5938060790 0x7f5938062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595815c8f0 0x7f59580c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f595815c520 0x7f59580c4e90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 --2- 172.21.15.111:0/3184825410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59580ca240 0x7f59581488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 >> 172.21.15.111:0/3184825410 conn(0x7f5958067bb0 msgr2=0x7f595815a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 shutdown_connections 2023-12-19T08:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.634+0000 7f595e6f0700 1 -- 172.21.15.111:0/3184825410 wait complete. 2023-12-19T08:04:53.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T08:04:53.762 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:53.762+0000 7f4e0a767000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:04:53.798 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:53.790+0000 7f295fdab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:04:53.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.822+0000 7f3c3c62e700 1 Processor -- start 2023-12-19T08:04:53.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.822+0000 7f3c3c62e700 1 -- start start 2023-12-19T08:04:53.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3c62e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c34069a20 0x7f3c3406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3c62e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 0x7f3c340b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3c62e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340ab6c0 con 0x7f3c3406cfa0 2023-12-19T08:04:53.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3c62e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c340ac240 con 0x7f3c34069a20 2023-12-19T08:04:53.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3c62e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340ab910 con 0x7f3c341591e0 2023-12-19T08:04:53.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c39bc9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c34069a20 0x7f3c3406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3a3ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c3406cfa0 0x7f3c34158ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43696/0 (socket says 172.21.15.111:43696) 2023-12-19T08:04:53.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3a3ca700 1 -- 172.21.15.111:0/3406655000 learned_addr learned my addr 172.21.15.111:0/3406655000 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:53.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3abcb700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 0x7f3c340b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c39bc9700 1 -- 172.21.15.111:0/3406655000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 msgr2=0x7f3c340b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:53.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c39bc9700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 0x7f3c340b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c39bc9700 1 -- 172.21.15.111:0/3406655000 >> v1:172.21.15.111:6789/0 conn(0x7f3c3406cfa0 legacy=0x7f3c34158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c39bc9700 1 -- 172.21.15.111:0/3406655000 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340abbf0 con 0x7f3c34069a20 2023-12-19T08:04:53.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.826+0000 7f3c3abcb700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 0x7f3c340b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:04:53.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c39bc9700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34069a20 0x7f3c3406d370 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3c240098e0 tx=0x7f3c240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f567256cb9601c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c393c8700 1 -- 172.21.15.111:0/3406655000 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400b640 con 0x7f3c34069a20 2023-12-19T08:04:53.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c393c8700 1 -- 172.21.15.111:0/3406655000 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2400b7a0 con 0x7f3c34069a20 2023-12-19T08:04:53.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c393c8700 1 -- 172.21.15.111:0/3406655000 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2400bb40 con 0x7f3c34069a20 2023-12-19T08:04:53.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3406655000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34069a20 msgr2=0x7f3c3406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34069a20 0x7f3c3406d370 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3c240098e0 tx=0x7f3c240091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3406655000 shutdown_connections 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c341591e0 0x7f3c340b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3406655000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c34069a20 0x7f3c3406d370 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3406655000 >> 172.21.15.111:0/3406655000 conn(0x7f3c340cd870 msgr2=0x7f3c34069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3406655000 shutdown_connections 2023-12-19T08:04:53.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.830+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3406655000 wait complete. 2023-12-19T08:04:53.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 Processor -- start 2023-12-19T08:04:53.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- start start 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069a20 0x7f3c34157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c3406cfa0 0x7f3c34151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340c9ab0 con 0x7f3c341591e0 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3415b1e0 con 0x7f3c34069a20 2023-12-19T08:04:53.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c34151fa0 con 0x7f3c3406cfa0 2023-12-19T08:04:53.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33922/0 (socket says 172.21.15.111:33922) 2023-12-19T08:04:53.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 -- 172.21.15.111:0/3469270481 learned_addr learned my addr 172.21.15.111:0/3469270481 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:53.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c39bc9700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c3406cfa0 0x7f3c34151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3a3ca700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069a20 0x7f3c34157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 -- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c3406cfa0 msgr2=0x7f3c34151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c3406cfa0 0x7f3c34151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 -- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069a20 msgr2=0x7f3c34157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069a20 0x7f3c34157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 -- 172.21.15.111:0/3469270481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c009a20 con 0x7f3c341591e0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3abcb700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009f50 tx=0x7f3c2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00cb50 con 0x7f3c341591e0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c00f040 con 0x7f3c341591e0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c014480 con 0x7f3c341591e0 2023-12-19T08:04:53.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c24009420 con 0x7f3c341591e0 2023-12-19T08:04:53.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.834+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3415a940 con 0x7f3c341591e0 2023-12-19T08:04:53.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c34071120 con 0x7f3c341591e0 2023-12-19T08:04:53.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f3c2c003020 con 0x7f3c341591e0 2023-12-19T08:04:53.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c2a7fc700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 0x7f3c18062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:53.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f3c2c07da50 con 0x7f3c341591e0 2023-12-19T08:04:53.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c3a3ca700 1 -- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 msgr2=0x7f3c18062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:53.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.838+0000 7f3c3a3ca700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 0x7f3c18062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:53.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:53.846+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c2c04d6c0 con 0x7f3c341591e0 2023-12-19T08:04:53.935 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:53.934+0000 7f4e0a767000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:04:53.962 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:53.954+0000 7f295fdab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:04:54.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.014+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c3406f6d0 con 0x7f3c341591e0 2023-12-19T08:04:54.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.014+0000 7f3c2a7fc700 1 -- 172.21.15.111:0/3469270481 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c2c018020 con 0x7f3c341591e0 2023-12-19T08:04:54.021 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 msgr2=0x7f3c18062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 0x7f3c18062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 msgr2=0x7f3c3415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009f50 tx=0x7f3c2c00b040 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 shutdown_connections 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c18060790 0x7f3c18062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c341591e0 0x7f3c3415a130 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c3406cfa0 0x7f3c34151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 --2- 172.21.15.111:0/3469270481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069a20 0x7f3c34157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 >> 172.21.15.111:0/3469270481 conn(0x7f3c340cd870 msgr2=0x7f3c34154860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 shutdown_connections 2023-12-19T08:04:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.018+0000 7f3c3c62e700 1 -- 172.21.15.111:0/3469270481 wait complete. 2023-12-19T08:04:54.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:04:54.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.206+0000 7f78aefcc700 1 Processor -- start 2023-12-19T08:04:54.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.206+0000 7f78aefcc700 1 -- start start 2023-12-19T08:04:54.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78a80c6cd0 0x7f78a806b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f78a815cc20 0x7f78a815d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a806b470 con 0x7f78a80c6900 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a806be40 con 0x7f78a815cc20 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a806e110 con 0x7f78a80c6cd0 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78adfca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f78a80c6900 0x7f78a806da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43700/0 (socket says 172.21.15.111:43700) 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78adfca700 1 -- 172.21.15.111:0/3305092330 learned_addr learned my addr 172.21.15.111:0/3305092330 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:54.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 --2- 172.21.15.111:0/3305092330 >> v2:172.21.15.111:3300/0 conn(0x7f78a815cc20 0x7f78a815d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ad7c9700 1 --2- 172.21.15.111:0/3305092330 >> v2:172.21.15.143:3300/0 conn(0x7f78a80c6cd0 0x7f78a806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78acfc8700 1 -- 172.21.15.111:0/3305092330 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515752804 0 0) 0x7f78a806b470 con 0x7f78a80c6900 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78acfc8700 1 -- 172.21.15.111:0/3305092330 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7894003570 con 0x7f78a80c6900 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 -- 172.21.15.111:0/3305092330 >> v2:172.21.15.143:3300/0 conn(0x7f78a80c6cd0 msgr2=0x7f78a806b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 --2- 172.21.15.111:0/3305092330 >> v2:172.21.15.143:3300/0 conn(0x7f78a80c6cd0 0x7f78a806b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 -- 172.21.15.111:0/3305092330 >> v1:172.21.15.111:6789/0 conn(0x7f78a80c6900 legacy=0x7f78a806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 -- 172.21.15.111:0/3305092330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8161d10 con 0x7f78a815cc20 2023-12-19T08:04:54.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78ae7cb700 1 --2- 172.21.15.111:0/3305092330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a815cc20 0x7f78a815d080 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f78a806a030 tx=0x7f78a0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cb075707724d30 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:54.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78acfc8700 1 -- 172.21.15.111:0/3305092330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a000e070 con 0x7f78a815cc20 2023-12-19T08:04:54.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78acfc8700 1 -- 172.21.15.111:0/3305092330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a000bab0 con 0x7f78a815cc20 2023-12-19T08:04:54.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78acfc8700 1 -- 172.21.15.111:0/3305092330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a000be70 con 0x7f78a815cc20 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- 172.21.15.111:0/3305092330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a815cc20 msgr2=0x7f78a815d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 --2- 172.21.15.111:0/3305092330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a815cc20 0x7f78a815d080 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f78a806a030 tx=0x7f78a0009780 comp rx=0 tx=0).stop 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- 172.21.15.111:0/3305092330 shutdown_connections 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 --2- 172.21.15.111:0/3305092330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a815cc20 0x7f78a815d080 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 --2- 172.21.15.111:0/3305092330 >> v2:172.21.15.143:3300/0 conn(0x7f78a80c6cd0 0x7f78a806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.210+0000 7f78aefcc700 1 -- 172.21.15.111:0/3305092330 >> 172.21.15.111:0/3305092330 conn(0x7f78a8067bb0 msgr2=0x7f78a80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.214+0000 7f78aefcc700 1 -- 172.21.15.111:0/3305092330 shutdown_connections 2023-12-19T08:04:54.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.214+0000 7f78aefcc700 1 -- 172.21.15.111:0/3305092330 wait complete. 2023-12-19T08:04:54.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.214+0000 7f78aefcc700 1 Processor -- start 2023-12-19T08:04:54.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.214+0000 7f78aefcc700 1 -- start start 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a80c6900 0x7f78a81536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a815cc20 0x7f78a80ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8156fd0 con 0x7f78a80c6cd0 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a816e0b0 con 0x7f78a80c6900 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80aec50 con 0x7f78a815cc20 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33940/0 (socket says 172.21.15.111:33940) 2023-12-19T08:04:54.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 -- 172.21.15.111:0/1250532440 learned_addr learned my addr 172.21.15.111:0/1250532440 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:54.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ae7cb700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a815cc20 0x7f78a80ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78adfca700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a80c6900 0x7f78a81536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a815cc20 msgr2=0x7f78a80ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a815cc20 0x7f78a80ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a80c6900 msgr2=0x7f78a81536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a80c6900 0x7f78a81536e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 -- 172.21.15.111:0/1250532440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0009a20 con 0x7f78a80c6cd0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78ad7c9700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f789800ec80 tx=0x7f78980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800c850 con 0x7f78a80c6cd0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789800c9b0 con 0x7f78a80c6cd0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80aef30 con 0x7f78a80c6cd0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800cd50 con 0x7f78a80c6cd0 2023-12-19T08:04:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.218+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80af570 con 0x7f78a80c6cd0 2023-12-19T08:04:54.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a80af890 con 0x7f78a80c6cd0 2023-12-19T08:04:54.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f7898013680 con 0x7f78a80c6cd0 2023-12-19T08:04:54.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f789dffb700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 0x7f788c062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f789807d2f0 con 0x7f78a80c6cd0 2023-12-19T08:04:54.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f78adfca700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 msgr2=0x7f788c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:54.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.222+0000 7f78adfca700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 0x7f788c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:54.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.230+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f78a80af890 con 0x7f78a80c6cd0 2023-12-19T08:04:54.316 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:54.318+0000 7f4e0a767000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:04:54.363 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:54.354+0000 7f295fdab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:04:54.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.426+0000 7f78adfca700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 msgr2=0x7f788c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.426+0000 7f78adfca700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 0x7f788c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:04:54.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.434+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a80af890 con 0x7f78a80c6cd0 2023-12-19T08:04:54.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.434+0000 7f789dffb700 1 -- 172.21.15.111:0/1250532440 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f78a80af890 con 0x7f78a80c6cd0 2023-12-19T08:04:54.441 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:54.441 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:04:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 msgr2=0x7f788c062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 0x7f788c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 msgr2=0x7f78a816db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f789800ec80 tx=0x7f78980091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 shutdown_connections 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f788c060790 0x7f788c062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a815cc20 0x7f78a80ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78a80c6cd0 0x7f78a816db70 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 --2- 172.21.15.111:0/1250532440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a80c6900 0x7f78a81536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 >> 172.21.15.111:0/1250532440 conn(0x7f78a8067bb0 msgr2=0x7f78a815b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 shutdown_connections 2023-12-19T08:04:54.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.438+0000 7f78aefcc700 1 -- 172.21.15.111:0/1250532440 wait complete. 2023-12-19T08:04:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:04:54.464 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:04:54.547 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:54.546+0000 7f4e0a767000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:04:54.593 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:54.586+0000 7f295fdab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:04:54.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.614+0000 7fbebb24c700 1 Processor -- start 2023-12-19T08:04:54.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.614+0000 7fbebb24c700 1 -- start start 2023-12-19T08:04:54.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebb24c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 0x7fbeb406d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebb24c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbeb4068160 0x7fbeb4068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebb24c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb40ad070 con 0x7fbeb40670e0 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebb24c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb40ac780 con 0x7fbeb4068160 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebb24c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb4068e90 con 0x7fbeb406d280 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebaa4b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbeb40670e0 0x7fbeb4067540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43704/0 (socket says 172.21.15.111:43704) 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbebaa4b700 1 -- 172.21.15.111:0/3535051079 learned_addr learned my addr 172.21.15.111:0/3535051079 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeba24a700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 0x7fbeb406d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 --2- 172.21.15.111:0/3535051079 >> v2:172.21.15.143:3300/0 conn(0x7fbeb4068160 0x7fbeb4068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9248700 1 -- 172.21.15.111:0/3535051079 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091029630 0 0) 0x7fbeb40ad070 con 0x7fbeb40670e0 2023-12-19T08:04:54.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9248700 1 -- 172.21.15.111:0/3535051079 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0003570 con 0x7fbeb40670e0 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 -- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 msgr2=0x7fbeb406d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 0x7fbeb406d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 -- 172.21.15.111:0/3535051079 >> v1:172.21.15.111:6789/0 conn(0x7fbeb40670e0 legacy=0x7fbeb4067540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 -- 172.21.15.111:0/3535051079 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40aca20 con 0x7fbeb4068160 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeba24a700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 0x7fbeb406d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9a49700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb4068590 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098e0 tx=0x7fbea40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f45ead7896269b21 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9248700 1 -- 172.21.15.111:0/3535051079 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4011630 con 0x7fbeb4068160 2023-12-19T08:04:54.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9248700 1 -- 172.21.15.111:0/3535051079 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea4011790 con 0x7fbeb4068160 2023-12-19T08:04:54.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.618+0000 7fbeb9248700 1 -- 172.21.15.111:0/3535051079 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4011b30 con 0x7fbeb4068160 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 msgr2=0x7fbeb4068590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb4068590 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbea40098e0 tx=0x7fbea40091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- 172.21.15.111:0/3535051079 shutdown_connections 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb4068590 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 --2- 172.21.15.111:0/3535051079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb406d280 0x7fbeb406d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- 172.21.15.111:0/3535051079 >> 172.21.15.111:0/3535051079 conn(0x7fbeb406c080 msgr2=0x7fbeb40ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- 172.21.15.111:0/3535051079 shutdown_connections 2023-12-19T08:04:54.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- 172.21.15.111:0/3535051079 wait complete. 2023-12-19T08:04:54.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 Processor -- start 2023-12-19T08:04:54.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.622+0000 7fbebb24c700 1 -- start start 2023-12-19T08:04:54.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 0x7fbeb414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb40af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 0x7fbeb40afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb4159570 con 0x7fbeb406d280 2023-12-19T08:04:54.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb40b0b20 con 0x7fbeb4068160 2023-12-19T08:04:54.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb41597e0 con 0x7fbeb40670e0 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebaa4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 0x7fbeb40afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 0x7fbeb414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebaa4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 0x7fbeb40afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33944/0 (socket says 172.21.15.111:33944) 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebaa4b700 1 -- 172.21.15.111:0/2174293303 learned_addr learned my addr 172.21.15.111:0/2174293303 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeb9a49700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb40af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 -- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 msgr2=0x7fbeb40af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb40af880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 -- 172.21.15.111:0/2174293303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 msgr2=0x7fbeb40afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 0x7fbeb40afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 -- 172.21.15.111:0/2174293303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0009420 con 0x7fbeb40670e0 2023-12-19T08:04:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeba24a700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 0x7fbeb414aff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000ea00 tx=0x7fbeb00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000c6a0 con 0x7fbeb40670e0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb000c800 con 0x7fbeb40670e0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea4009420 con 0x7fbeb40670e0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000cba0 con 0x7fbeb40670e0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.626+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb4159da0 con 0x7fbeb40670e0 2023-12-19T08:04:54.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb4153d90 con 0x7fbeb40670e0 2023-12-19T08:04:54.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fbeb0013470 con 0x7fbeb40670e0 2023-12-19T08:04:54.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbeaa7fc700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 0x7fbe98062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:54.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fbeb007d2e0 con 0x7fbeb40670e0 2023-12-19T08:04:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbeb9a49700 1 -- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 msgr2=0x7fbe98062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.630+0000 7fbeb9a49700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 0x7fbe98062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:54.644 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:54.646+0000 7f4e0a767000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:04:54.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.638+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbeb004d060 con 0x7fbeb40670e0 2023-12-19T08:04:54.695 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:54.690+0000 7f295fdab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:04:54.730 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:54.730+0000 7f4e0a767000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:04:54.776 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:54.770+0000 7f295fdab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:04:54.812 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:54.810+0000 7f4e0a767000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:04:54.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.814+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbeb404be40 con 0x7fbeb40670e0 2023-12-19T08:04:54.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.814+0000 7fbeaa7fc700 1 -- 172.21.15.111:0/2174293303 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17097 (secure 0 0 0) 0x7fbeb0050680 con 0x7fbeb40670e0 2023-12-19T08:04:54.822 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:54.822 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":116,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:04:50.566091+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2526647480":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/144894101":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1370433618":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/596992054":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/4178727723":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1808666646":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:48.038487+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:04:54.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 msgr2=0x7fbe98062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:54.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 0x7fbe98062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 msgr2=0x7fbeb414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 0x7fbeb414aff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000ea00 tx=0x7fbeb00091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 shutdown_connections 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbe98060790 0x7fbe98062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbeb4068160 0x7fbeb40af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb406d280 0x7fbeb40afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 --2- 172.21.15.111:0/2174293303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbeb40670e0 0x7fbeb414aff0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 >> 172.21.15.111:0/2174293303 conn(0x7fbeb406c080 msgr2=0x7fbeb4152390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 shutdown_connections 2023-12-19T08:04:54.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.818+0000 7fbebb24c700 1 -- 172.21.15.111:0/2174293303 wait complete. 2023-12-19T08:04:54.845 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T08:04:54.845 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:04:54.865 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:54.858+0000 7f295fdab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:04:55.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:54.998+0000 7f6caf585700 1 Processor -- start 2023-12-19T08:04:55.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 -- start start 2023-12-19T08:04:55.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ca80b0110 0x7f6ca80b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 0x7f6ca8153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca80ccb40 con 0x7f6ca80ccfd0 2023-12-19T08:04:55.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80cd470 con 0x7f6ca80b0110 2023-12-19T08:04:55.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6caf585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca8153bd0 con 0x7f6ca8153290 2023-12-19T08:04:55.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6cad321700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ca80b0110 0x7f6ca80b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6cadb22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 0x7f6ca8153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6cad321700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ca80b0110 0x7f6ca80b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33956/0 (socket says 172.21.15.111:33956) 2023-12-19T08:04:55.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6cadb22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 0x7f6ca8153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:46092/0 (socket says 172.21.15.111:46092) 2023-12-19T08:04:55.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.002+0000 7f6cad321700 1 -- 172.21.15.111:0/1380787441 learned_addr learned my addr 172.21.15.111:0/1380787441 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6cad321700 1 -- 172.21.15.111:0/1380787441 >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 msgr2=0x7f6ca8153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:55.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6cad321700 1 --2- 172.21.15.111:0/1380787441 >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 0x7f6ca8153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6cad321700 1 -- 172.21.15.111:0/1380787441 >> v1:172.21.15.143:6789/0 conn(0x7f6ca80ccfd0 legacy=0x7f6ca815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6cad321700 1 -- 172.21.15.111:0/1380787441 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80cbf30 con 0x7f6ca80b0110 2023-12-19T08:04:55.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6cad321700 1 --2- 172.21.15.111:0/1380787441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80b04e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80cdd50 tx=0x7f6ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e669d81ac45298b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6c9ffff700 1 -- 172.21.15.111:0/1380787441 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4009e70 con 0x7f6ca80b0110 2023-12-19T08:04:55.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6c9ffff700 1 -- 172.21.15.111:0/1380787441 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca400b860 con 0x7f6ca80b0110 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6c9ffff700 1 -- 172.21.15.111:0/1380787441 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400bc00 con 0x7f6ca80b0110 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 -- 172.21.15.111:0/1380787441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 msgr2=0x7f6ca80b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 --2- 172.21.15.111:0/1380787441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80b04e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80cdd50 tx=0x7f6ca4009250 comp rx=0 tx=0).stop 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 -- 172.21.15.111:0/1380787441 shutdown_connections 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 --2- 172.21.15.111:0/1380787441 >> v2:172.21.15.143:3300/0 conn(0x7f6ca8153290 0x7f6ca8153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 --2- 172.21.15.111:0/1380787441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80b04e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 -- 172.21.15.111:0/1380787441 >> 172.21.15.111:0/1380787441 conn(0x7f6ca806c7c0 msgr2=0x7f6ca806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 -- 172.21.15.111:0/1380787441 shutdown_connections 2023-12-19T08:04:55.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.006+0000 7f6caf585700 1 -- 172.21.15.111:0/1380787441 wait complete. 2023-12-19T08:04:55.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 Processor -- start 2023-12-19T08:04:55.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 -- start start 2023-12-19T08:04:55.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca80ccfd0 0x7f6ca8150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8153290 0x7f6ca80c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80ab320 con 0x7f6ca80b0110 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca8151400 con 0x7f6ca8153290 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6caf585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80a9330 con 0x7f6ca80ccfd0 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33970/0 (socket says 172.21.15.111:33970) 2023-12-19T08:04:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 -- 172.21.15.111:0/866466787 learned_addr learned my addr 172.21.15.111:0/866466787 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cadb22700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8153290 0x7f6ca80c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cacb20700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca80ccfd0 0x7f6ca8150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca80ccfd0 msgr2=0x7f6ca8150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca80ccfd0 0x7f6ca8150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8153290 msgr2=0x7f6ca80c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8153290 0x7f6ca80c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 -- 172.21.15.111:0/866466787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40094f0 con 0x7f6ca80b0110 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.010+0000 7f6cad321700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80649b0 tx=0x7f6ca4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4011420 con 0x7f6ca80b0110 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4011580 con 0x7f6ca80b0110 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4011940 con 0x7f6ca80b0110 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca80a9610 con 0x7f6ca80b0110 2023-12-19T08:04:55.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca80a9bd0 con 0x7f6ca80b0110 2023-12-19T08:04:55.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca8071120 con 0x7f6ca80b0110 2023-12-19T08:04:55.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.014+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f6ca402c3b0 con 0x7f6ca80b0110 2023-12-19T08:04:55.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.018+0000 7f6c9d7fa700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 0x7f6c8c062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.018+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f6ca407e5a0 con 0x7f6ca80b0110 2023-12-19T08:04:55.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.018+0000 7f6cacb20700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 msgr2=0x7f6c8c062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:55.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.018+0000 7f6cacb20700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 0x7f6c8c062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:55.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.022+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ca404e3f0 con 0x7f6ca80b0110 2023-12-19T08:04:55.051 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:55.050+0000 7f4e0a767000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:04:55.102 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:55.094+0000 7f295fdab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:04:55.132 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:55.134+0000 7f4e0a767000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:04:55.181 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:55.174+0000 7f295fdab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:04:55.226 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:55.226+0000 7f4e0a767000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:04:55.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.222+0000 7f6cacb20700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 msgr2=0x7f6c8c062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:55.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.222+0000 7f6cacb20700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 0x7f6c8c062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:04:55.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.234+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ca80ab460 con 0x7f6ca80b0110 2023-12-19T08:04:55.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.234+0000 7f6c9d7fa700 1 -- 172.21.15.111:0/866466787 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f6ca4051a10 con 0x7f6ca80b0110 2023-12-19T08:04:55.241 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:55.241 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:04:55.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 msgr2=0x7f6c8c062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:55.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 0x7f6c8c062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 msgr2=0x7f6ca80a8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80649b0 tx=0x7f6ca4017000 comp rx=0 tx=0).stop 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 shutdown_connections 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c8c0607c0 0x7f6c8c062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8153290 0x7f6ca80c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca80ccfd0 0x7f6ca8150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 --2- 172.21.15.111:0/866466787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ca80b0110 0x7f6ca80a8ca0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 >> 172.21.15.111:0/866466787 conn(0x7f6ca806c7c0 msgr2=0x7f6ca80c3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 shutdown_connections 2023-12-19T08:04:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.238+0000 7f6caf585700 1 -- 172.21.15.111:0/866466787 wait complete. 2023-12-19T08:04:55.246 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:04:55.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:04:55.274 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:55.266+0000 7f295fdab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:04:55.429 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:55.430+0000 7f4e0a767000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:04:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.422+0000 7f40f3305700 1 Processor -- start 2023-12-19T08:04:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 -- start start 2023-12-19T08:04:55.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f40ec140710 0x7f40ec0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 0x7f40ec0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec06e150 con 0x7f40ec0c75e0 2023-12-19T08:04:55.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec06d540 con 0x7f40ec140710 2023-12-19T08:04:55.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f3305700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec06d820 con 0x7f40ec140ae0 2023-12-19T08:04:55.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f08a0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 0x7f40ec0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f10a1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f40ec140710 0x7f40ec0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f08a0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 0x7f40ec0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:46110/0 (socket says 172.21.15.111:46110) 2023-12-19T08:04:55.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f08a0700 1 -- 172.21.15.111:0/4207047330 learned_addr learned my addr 172.21.15.111:0/4207047330 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f18a2700 1 --1- 172.21.15.111:0/4207047330 >> v1:172.21.15.143:6789/0 conn(0x7f40ec0c75e0 0x7f40ec06a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:37904/0 (socket says 172.21.15.111:37904) 2023-12-19T08:04:55.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f10a1700 1 -- 172.21.15.111:0/4207047330 >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 msgr2=0x7f40ec0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:55.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f10a1700 1 --2- 172.21.15.111:0/4207047330 >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 0x7f40ec0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f10a1700 1 -- 172.21.15.111:0/4207047330 >> v1:172.21.15.143:6789/0 conn(0x7f40ec0c75e0 legacy=0x7f40ec06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.426+0000 7f40f10a1700 1 -- 172.21.15.111:0/4207047330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec06db00 con 0x7f40ec140710 2023-12-19T08:04:55.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f10a1700 1 --2- 172.21.15.111:0/4207047330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140710 0x7f40ec0c70a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0c9960 tx=0x7f40e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686dbb8456ae7384 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40e3fff700 1 -- 172.21.15.111:0/4207047330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8011630 con 0x7f40ec140710 2023-12-19T08:04:55.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40e3fff700 1 -- 172.21.15.111:0/4207047330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e8011790 con 0x7f40ec140710 2023-12-19T08:04:55.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40e3fff700 1 -- 172.21.15.111:0/4207047330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8011b30 con 0x7f40ec140710 2023-12-19T08:04:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 -- 172.21.15.111:0/4207047330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140710 msgr2=0x7f40ec0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 --2- 172.21.15.111:0/4207047330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140710 0x7f40ec0c70a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0c9960 tx=0x7f40e80091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 -- 172.21.15.111:0/4207047330 shutdown_connections 2023-12-19T08:04:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 --2- 172.21.15.111:0/4207047330 >> v2:172.21.15.143:3300/0 conn(0x7f40ec140ae0 0x7f40ec0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 --2- 172.21.15.111:0/4207047330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140710 0x7f40ec0c70a0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 -- 172.21.15.111:0/4207047330 >> 172.21.15.111:0/4207047330 conn(0x7f40ec0c8850 msgr2=0x7f40ec0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:55.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 -- 172.21.15.111:0/4207047330 shutdown_connections 2023-12-19T08:04:55.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.430+0000 7f40f3305700 1 -- 172.21.15.111:0/4207047330 wait complete. 2023-12-19T08:04:55.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 Processor -- start 2023-12-19T08:04:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 -- start start 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40ec0c75e0 0x7f40ec0c3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40ec140710 0x7f40ec13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec0c3100 con 0x7f40ec140ae0 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec140000 con 0x7f40ec0c75e0 2023-12-19T08:04:55.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f3305700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec0c39c0 con 0x7f40ec140710 2023-12-19T08:04:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33976/0 (socket says 172.21.15.111:33976) 2023-12-19T08:04:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 -- 172.21.15.111:0/1730663553 learned_addr learned my addr 172.21.15.111:0/1730663553 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f10a1700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40ec0c75e0 0x7f40ec0c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f08a0700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40ec140710 0x7f40ec13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40ec140710 msgr2=0x7f40ec13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40ec140710 0x7f40ec13fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40ec0c75e0 msgr2=0x7f40ec0c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40ec0c75e0 0x7f40ec0c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.434+0000 7f40f18a2700 1 -- 172.21.15.111:0/1730663553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e8009420 con 0x7f40ec140ae0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40f18a2700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f40e400cc70 tx=0x7f40e4001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4005df0 con 0x7f40ec140ae0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e400a5b0 con 0x7f40ec140ae0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e400a930 con 0x7f40ec140ae0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec0c3ca0 con 0x7f40ec140ae0 2023-12-19T08:04:55.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec0c4260 con 0x7f40ec140ae0 2023-12-19T08:04:55.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec1589b0 con 0x7f40ec140ae0 2023-12-19T08:04:55.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.438+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f40e400ace0 con 0x7f40ec140ae0 2023-12-19T08:04:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.442+0000 7f40e17fa700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 0x7f40d0062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.442+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f40e407d0c0 con 0x7f40ec140ae0 2023-12-19T08:04:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.442+0000 7f40f10a1700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 msgr2=0x7f40d0062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:55.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.442+0000 7f40f10a1700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 0x7f40d0062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:55.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.446+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f40ec1589b0 con 0x7f40ec140ae0 2023-12-19T08:04:55.490 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:55.482+0000 7f295fdab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:04:55.521 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:55.522+0000 7f4e0a767000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:04:55.581 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:55.574+0000 7f295fdab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:04:55.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.646+0000 7f40f10a1700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 msgr2=0x7f40d0062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:55.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.646+0000 7f40f10a1700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 0x7f40d0062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:04:55.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.662+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40ec1589b0 con 0x7f40ec140ae0 2023-12-19T08:04:55.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40e17fa700 1 -- 172.21.15.111:0/1730663553 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4611 (secure 0 0 0) 0x7f40ec1589b0 con 0x7f40ec140ae0 2023-12-19T08:04:55.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:55.672 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:03:33.363924+0000","modified":"2023-12-19T08:03:36.955831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7389":{"gid":7389,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1122478953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1122478953},{"type":"v1","addr":"172.21.15.111:6835","nonce":1122478953}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 msgr2=0x7f40d0062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 0x7f40d0062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 msgr2=0x7f40ec159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f40e400cc70 tx=0x7f40e4001a70 comp rx=0 tx=0).stop 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 shutdown_connections 2023-12-19T08:04:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40d00607c0 0x7f40d0062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40ec140ae0 0x7f40ec159ee0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40ec140710 0x7f40ec13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 --2- 172.21.15.111:0/1730663553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40ec0c75e0 0x7f40ec0c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 >> 172.21.15.111:0/1730663553 conn(0x7f40ec0c8850 msgr2=0x7f40ec0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 shutdown_connections 2023-12-19T08:04:55.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.666+0000 7f40f3305700 1 -- 172.21.15.111:0/1730663553 wait complete. 2023-12-19T08:04:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 48 2023-12-19T08:04:55.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:04:55.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.850+0000 7f2dd9985700 1 Processor -- start 2023-12-19T08:04:55.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.850+0000 7f2dd9985700 1 -- start start 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd9985700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2dd4067c90 0x7f2dd4068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd9985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd40c5900 0x7f2dd40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd9985700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd40c6630 con 0x7f2dd414fba0 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd9985700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd40c6910 con 0x7f2dd4067c90 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd9985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd40c6bf0 con 0x7f2dd40c5900 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2dd4067c90 0x7f2dd4068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2dd4067c90 0x7f2dd4068060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:46118/0 (socket says 172.21.15.111:46118) 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 -- 172.21.15.111:0/14842429 learned_addr learned my addr 172.21.15.111:0/14842429 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd37fe700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd40c5900 0x7f2dd40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 -- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd40c5900 msgr2=0x7f2dd40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd40c5900 0x7f2dd40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 -- 172.21.15.111:0/14842429 >> v1:172.21.15.143:6789/0 conn(0x7f2dd414fba0 legacy=0x7f2dd4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.854+0000 7f2dd2ffd700 1 -- 172.21.15.111:0/14842429 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd40c6ed0 con 0x7f2dd4067c90 2023-12-19T08:04:55.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd2ffd700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd4068060 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40099b0 tx=0x7f2dc4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c48975d12726142 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd1ffb700 1 -- 172.21.15.111:0/14842429 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4009ea0 con 0x7f2dd4067c90 2023-12-19T08:04:55.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd1ffb700 1 -- 172.21.15.111:0/14842429 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc400b860 con 0x7f2dd4067c90 2023-12-19T08:04:55.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd1ffb700 1 -- 172.21.15.111:0/14842429 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc400bc00 con 0x7f2dd4067c90 2023-12-19T08:04:55.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 -- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 msgr2=0x7f2dd4068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd4068060 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40099b0 tx=0x7f2dc4009250 comp rx=0 tx=0).stop 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 -- 172.21.15.111:0/14842429 shutdown_connections 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd40c5900 0x7f2dd40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 --2- 172.21.15.111:0/14842429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd4068060 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 -- 172.21.15.111:0/14842429 >> 172.21.15.111:0/14842429 conn(0x7f2dd40cd6d0 msgr2=0x7f2dd406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 -- 172.21.15.111:0/14842429 shutdown_connections 2023-12-19T08:04:55.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.858+0000 7f2dd9985700 1 -- 172.21.15.111:0/14842429 wait complete. 2023-12-19T08:04:55.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 Processor -- start 2023-12-19T08:04:55.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 -- start start 2023-12-19T08:04:55.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd41509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 0x7f2dd4152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4153560 con 0x7f2dd40c5900 2023-12-19T08:04:55.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd40c8310 con 0x7f2dd4067c90 2023-12-19T08:04:55.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd9985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41537f0 con 0x7f2dd414fba0 2023-12-19T08:04:55.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 0x7f2dd4152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd41509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:55.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:60136/0 (socket says 172.21.15.111:60136) 2023-12-19T08:04:55.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd37fe700 1 -- 172.21.15.111:0/1746054470 learned_addr learned my addr 172.21.15.111:0/1746054470 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.862+0000 7f2dd27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 0x7f2dd4152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33980/0 (socket says 172.21.15.111:33980) 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 -- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 msgr2=0x7f2dd41509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd41509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 -- 172.21.15.111:0/1746054470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 msgr2=0x7f2dd4152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 0x7f2dd4152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 -- 172.21.15.111:0/1746054470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40094f0 con 0x7f2dd414fba0 2023-12-19T08:04:55.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd37fe700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009e90 tx=0x7f2dcc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:55.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00cb70 con 0x7f2dd414fba0 2023-12-19T08:04:55.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc00f040 con 0x7f2dd414fba0 2023-12-19T08:04:55.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc014480 con 0x7f2dd414fba0 2023-12-19T08:04:55.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd40c85b0 con 0x7f2dd414fba0 2023-12-19T08:04:55.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd4153ad0 con 0x7f2dd414fba0 2023-12-19T08:04:55.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.866+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd40c8ab0 con 0x7f2dd414fba0 2023-12-19T08:04:55.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.870+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f2dcc0147f0 con 0x7f2dd414fba0 2023-12-19T08:04:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.870+0000 7f2dbaffd700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 0x7f2dc0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.870+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f2dcc07df60 con 0x7f2dd414fba0 2023-12-19T08:04:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.870+0000 7f2dd2ffd700 1 -- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 msgr2=0x7f2dc0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.870+0000 7f2dd2ffd700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 0x7f2dc0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:55.878+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2dcc04dc00 con 0x7f2dd414fba0 2023-12-19T08:04:56.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.058+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd4071120 con 0x7f2dd414fba0 2023-12-19T08:04:56.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.058+0000 7f2dbaffd700 1 -- 172.21.15.111:0/1746054470 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17097 (secure 0 0 0) 0x7f2dcc018090 con 0x7f2dd414fba0 2023-12-19T08:04:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:04:56.068 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":116,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:04:50.566091+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:03:31.360391+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-12-19T08:03:32.348699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.1b","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2526647480":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/144894101":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1370433618":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/596992054":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/4178727723":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1808666646":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:48.038487+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:04:56.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 msgr2=0x7f2dc0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:56.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 0x7f2dc0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 msgr2=0x7f2dd4153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009e90 tx=0x7f2dcc00b040 comp rx=0 tx=0).stop 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 shutdown_connections 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2dc0060790 0x7f2dc0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd414fba0 0x7f2dd4153020 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd40c5900 0x7f2dd4152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 --2- 172.21.15.111:0/1746054470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd4067c90 0x7f2dd41509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 >> 172.21.15.111:0/1746054470 conn(0x7f2dd40cd6d0 msgr2=0x7f2dd40ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:56.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 shutdown_connections 2023-12-19T08:04:56.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.062+0000 7f2dd9985700 1 -- 172.21.15.111:0/1746054470 wait complete. 2023-12-19T08:04:56.096 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T08:04:56.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 Processor -- start 2023-12-19T08:04:56.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 -- start start 2023-12-19T08:04:56.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa93c158cc0 0x7fa93c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa93c06dc00 0x7fa93c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c06b0d0 con 0x7fa93c06baa0 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93c06bec0 con 0x7fa93c158cc0 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa943b85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93c0b4790 con 0x7fa93c06dc00 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa93c158cc0 0x7fa93c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941120700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa93c06dc00 0x7fa93c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa93c158cc0 0x7fa93c06d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33990/0 (socket says 172.21.15.111:33990) 2023-12-19T08:04:56.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 -- 172.21.15.111:0/1833871913 learned_addr learned my addr 172.21.15.111:0/1833871913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 -- 172.21.15.111:0/1833871913 >> v2:172.21.15.143:3300/0 conn(0x7fa93c06dc00 msgr2=0x7fa93c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 --2- 172.21.15.111:0/1833871913 >> v2:172.21.15.143:3300/0 conn(0x7fa93c06dc00 0x7fa93c06b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 -- 172.21.15.111:0/1833871913 >> v1:172.21.15.143:6789/0 conn(0x7fa93c06baa0 legacy=0x7fa93c0b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.258+0000 7fa941921700 1 -- 172.21.15.111:0/1833871913 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c06d710 con 0x7fa93c158cc0 2023-12-19T08:04:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa941921700 1 --2- 172.21.15.111:0/1833871913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c158cc0 0x7fa93c06d1d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa93c068090 tx=0x7fa938009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54347dd0338f6cbe server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:56.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa94091f700 1 -- 172.21.15.111:0/1833871913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93800e070 con 0x7fa93c158cc0 2023-12-19T08:04:56.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa94091f700 1 -- 172.21.15.111:0/1833871913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800ba20 con 0x7fa93c158cc0 2023-12-19T08:04:56.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa94091f700 1 -- 172.21.15.111:0/1833871913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9380092e0 con 0x7fa93c158cc0 2023-12-19T08:04:56.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 -- 172.21.15.111:0/1833871913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c158cc0 msgr2=0x7fa93c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 --2- 172.21.15.111:0/1833871913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c158cc0 0x7fa93c06d1d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa93c068090 tx=0x7fa938009840 comp rx=0 tx=0).stop 2023-12-19T08:04:56.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 -- 172.21.15.111:0/1833871913 shutdown_connections 2023-12-19T08:04:56.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 --2- 172.21.15.111:0/1833871913 >> v2:172.21.15.143:3300/0 conn(0x7fa93c06dc00 0x7fa93c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 --2- 172.21.15.111:0/1833871913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c158cc0 0x7fa93c06d1d0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 -- 172.21.15.111:0/1833871913 >> 172.21.15.111:0/1833871913 conn(0x7fa93c067bb0 msgr2=0x7fa93c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:56.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 -- 172.21.15.111:0/1833871913 shutdown_connections 2023-12-19T08:04:56.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.262+0000 7fa943b85700 1 -- 172.21.15.111:0/1833871913 wait complete. 2023-12-19T08:04:56.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 Processor -- start 2023-12-19T08:04:56.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 -- start start 2023-12-19T08:04:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa93c06baa0 0x7fa93c0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa93c158cc0 0x7fa93c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0ae800 con 0x7fa93c06dc00 2023-12-19T08:04:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0adbf0 con 0x7fa93c158cc0 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa943b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0aded0 con 0x7fa93c06baa0 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33992/0 (socket says 172.21.15.111:33992) 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 -- 172.21.15.111:0/328897887 learned_addr learned my addr 172.21.15.111:0/328897887 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941921700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa93c06baa0 0x7fa93c0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa942122700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa93c158cc0 0x7fa93c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 -- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa93c06baa0 msgr2=0x7fa93c0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa93c06baa0 0x7fa93c0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 -- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa93c158cc0 msgr2=0x7fa93c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa93c158cc0 0x7fa93c0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.266+0000 7fa941120700 1 -- 172.21.15.111:0/328897887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009ae0 con 0x7fa93c06dc00 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa941120700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00ec80 tx=0x7fa92c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c00c850 con 0x7fa93c06dc00 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92c00c9b0 con 0x7fa93c06dc00 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c0ae1b0 con 0x7fa93c06dc00 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c00cd50 con 0x7fa93c06dc00 2023-12-19T08:04:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c0ad900 con 0x7fa93c06dc00 2023-12-19T08:04:56.278 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.278+0000 7f4e0a767000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:04:56.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c0ae9b0 con 0x7fa93c06dc00 2023-12-19T08:04:56.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.270+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 47) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fa92c013680 con 0x7fa93c06dc00 2023-12-19T08:04:56.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.274+0000 7fa931ffb700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 0x7fa920062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.274+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fa92c07d310 con 0x7fa93c06dc00 2023-12-19T08:04:56.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.274+0000 7fa941921700 1 -- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 msgr2=0x7fa920062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:04:56.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.274+0000 7fa941921700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 0x7fa920062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:04:56.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.278+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa93c0ae9b0 con 0x7fa93c06dc00 2023-12-19T08:04:56.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.290+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 48) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fa92c04d490 con 0x7fa93c06dc00 2023-12-19T08:04:56.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.290+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 msgr2=0x7fa920062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:04:56.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.290+0000 7fa931ffb700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 0x7fa920062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.383 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.382+0000 7f4dd1ae1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.383 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.382+0000 7f4dd1ae1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.383 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.382+0000 7f4dd1ae1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.383 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.382+0000 7f4dd1ae1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.383 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.382+0000 7f4dd1ae1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.386+0000 7f4dd52e8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.386+0000 7f4dd52e8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.386+0000 7f4dd52e8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.386+0000 7f4dd52e8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.384 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:04:56.386+0000 7f4dd52e8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:04:56.392 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:04:56.386+0000 7f295fdab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:04:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.490+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa93c071120 con 0x7fa93c06dc00 2023-12-19T08:04:56.498 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T08:04:56.490+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T08:04:56.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.502+0000 7fa931ffb700 1 -- 172.21.15.111:0/328897887 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa92c068400 con 0x7fa93c06dc00 2023-12-19T08:04:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 msgr2=0x7fa93c0ad3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00ec80 tx=0x7fa92c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 shutdown_connections 2023-12-19T08:04:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa920060790 0x7fa920062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa93c158cc0 0x7fa93c0af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa93c06dc00 0x7fa93c0ad3c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 --2- 172.21.15.111:0/328897887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa93c06baa0 0x7fa93c0b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 >> 172.21.15.111:0/328897887 conn(0x7fa93c067bb0 msgr2=0x7fa93c0abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 shutdown_connections 2023-12-19T08:04:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.506+0000 7fa943b85700 1 -- 172.21.15.111:0/328897887 wait complete. 2023-12-19T08:04:56.514 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T08:04:56.522 INFO:tasks.ceph.mds.a.smithi111.stderr: -6113> 2023-12-19T08:03:38.876+0000 7fd187df1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:04:56.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T08:04:56.568 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:04:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.678+0000 7fb11d123700 1 Processor -- start 2023-12-19T08:04:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.678+0000 7fb11d123700 1 -- start start 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11d123700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1180c53f0 0x7fb11815c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11d123700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118159b80 con 0x7fb118141e70 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11d123700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1180c8220 con 0x7fb11815a010 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11d123700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1180c7f40 con 0x7fb1180c53f0 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb117fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb118141e70 0x7fb1180c7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43724/0 (socket says 172.21.15.111:43724) 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb117fff700 1 -- 172.21.15.111:0/3860181669 learned_addr learned my addr 172.21.15.111:0/3860181669 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11c922700 1 --2- 172.21.15.111:0/3860181669 >> v2:172.21.15.143:3300/0 conn(0x7fb1180c53f0 0x7fb11815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11c922700 1 -- 172.21.15.111:0/3860181669 >> v1:172.21.15.143:6789/0 conn(0x7fb11815a010 legacy=0x7fb1180c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11c922700 1 -- 172.21.15.111:0/3860181669 >> v1:172.21.15.111:6789/0 conn(0x7fb118141e70 legacy=0x7fb1180c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11c922700 1 -- 172.21.15.111:0/3860181669 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118159530 con 0x7fb1180c53f0 2023-12-19T08:04:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb116ffd700 1 -- 172.21.15.111:0/3860181669 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547634802 0 0) 0x7fb118159b80 con 0x7fb118141e70 2023-12-19T08:04:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.682+0000 7fb11c922700 1 --2- 172.21.15.111:0/3860181669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1180c53f0 0x7fb11815c970 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb110009e60 tx=0x7fb110009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=288fa224397df74f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb116ffd700 1 -- 172.21.15.111:0/3860181669 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11000e070 con 0x7fb1180c53f0 2023-12-19T08:04:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb116ffd700 1 -- 172.21.15.111:0/3860181669 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11000bc30 con 0x7fb1180c53f0 2023-12-19T08:04:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb116ffd700 1 -- 172.21.15.111:0/3860181669 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110012430 con 0x7fb1180c53f0 2023-12-19T08:04:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 -- 172.21.15.111:0/3860181669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1180c53f0 msgr2=0x7fb11815c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 --2- 172.21.15.111:0/3860181669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1180c53f0 0x7fb11815c970 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb110009e60 tx=0x7fb110009780 comp rx=0 tx=0).stop 2023-12-19T08:04:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 -- 172.21.15.111:0/3860181669 shutdown_connections 2023-12-19T08:04:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 --2- 172.21.15.111:0/3860181669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1180c53f0 0x7fb11815c970 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 -- 172.21.15.111:0/3860181669 >> 172.21.15.111:0/3860181669 conn(0x7fb118067b50 msgr2=0x7fb1180cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:56.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 -- 172.21.15.111:0/3860181669 shutdown_connections 2023-12-19T08:04:56.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.686+0000 7fb11d123700 1 -- 172.21.15.111:0/3860181669 wait complete. 2023-12-19T08:04:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 Processor -- start 2023-12-19T08:04:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- start start 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb118141e70 0x7fb1180af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 0x7fb118158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11816d060 con 0x7fb1180c53f0 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1180aeaf0 con 0x7fb11815a010 2023-12-19T08:04:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1180aed90 con 0x7fb118141e70 2023-12-19T08:04:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34028/0 (socket says 172.21.15.111:34028) 2023-12-19T08:04:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 -- 172.21.15.111:0/959643563 learned_addr learned my addr 172.21.15.111:0/959643563 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11c922700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 0x7fb118158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb1177fe700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb118141e70 0x7fb1180af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 -- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb118141e70 msgr2=0x7fb1180af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb118141e70 0x7fb1180af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 -- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 msgr2=0x7fb118158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 0x7fb118158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 -- 172.21.15.111:0/959643563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110009a20 con 0x7fb1180c53f0 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11c922700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 0x7fb118158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb117fff700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb10800c7e0 tx=0x7fb108007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb108005b30 con 0x7fb1180c53f0 2023-12-19T08:04:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb108005c90 con 0x7fb1180c53f0 2023-12-19T08:04:56.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11815dc90 con 0x7fb1180c53f0 2023-12-19T08:04:56.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb108007300 con 0x7fb1180c53f0 2023-12-19T08:04:56.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.690+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11813c3c0 con 0x7fb1180c53f0 2023-12-19T08:04:56.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.694+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11815dad0 con 0x7fb1180c53f0 2023-12-19T08:04:56.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.694+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 48) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fb10801b3b0 con 0x7fb1180c53f0 2023-12-19T08:04:56.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.694+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 7435+0+0 (secure 0 0 0) 0x7fb10807bf10 con 0x7fb1180c53f0 2023-12-19T08:04:56.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.702+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb10804c1a0 con 0x7fb1180c53f0 2023-12-19T08:04:56.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:56.918+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb11804be40 con 0x7fb1180c53f0 2023-12-19T08:04:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.298+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 49) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fb1080550c0 con 0x7fb1180c53f0 2023-12-19T08:04:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.298+0000 7fb0fffff700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 0x7fb104068f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.306+0000 7fb1177fe700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 0x7fb104068f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.310+0000 7fb1177fe700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 0x7fb104068f00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb11806bf90 tx=0x7fb10c007ec0 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:57.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.506+0000 7fb0fffff700 1 -- 172.21.15.111:0/959643563 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb108068fc0 con 0x7fb1180c53f0 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 msgr2=0x7fb104068f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 0x7fb104068f00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb11806bf90 tx=0x7fb10c007ec0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 msgr2=0x7fb11816cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb10800c7e0 tx=0x7fb108007df0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 shutdown_connections 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb104066b20 0x7fb104068f00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb11815a010 0x7fb118158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb118141e70 0x7fb1180af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 --2- 172.21.15.111:0/959643563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1180c53f0 0x7fb11816cb20 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 >> 172.21.15.111:0/959643563 conn(0x7fb118067b50 msgr2=0x7fb1180c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:57.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 shutdown_connections 2023-12-19T08:04:57.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.510+0000 7fb11d123700 1 -- 172.21.15.111:0/959643563 wait complete. 2023-12-19T08:04:57.538 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T08:04:57.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.750+0000 7f715c878700 1 Processor -- start 2023-12-19T08:04:57.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.750+0000 7f715c878700 1 -- start start 2023-12-19T08:04:57.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715c878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 0x7f7158068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715c878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7158069310 0x7f71580688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715c878700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715806d0d0 con 0x7f7158158e50 2023-12-19T08:04:57.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715c878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715806c7c0 con 0x7f7158069310 2023-12-19T08:04:57.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715c878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71580c7b40 con 0x7f7158159220 2023-12-19T08:04:57.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7158158e50 0x7f71580c72c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43728/0 (socket says 172.21.15.111:43728) 2023-12-19T08:04:57.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.754+0000 7f715659c700 1 -- 172.21.15.111:0/1543233016 learned_addr learned my addr 172.21.15.111:0/1543233016 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:57.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 --2- 172.21.15.111:0/1543233016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 0x7f7158068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7156d9d700 1 --2- 172.21.15.111:0/1543233016 >> v2:172.21.15.143:3300/0 conn(0x7f7158069310 0x7f71580688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715559a700 1 -- 172.21.15.111:0/1543233016 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965750048 0 0) 0x7f715806d0d0 con 0x7f7158158e50 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715559a700 1 -- 172.21.15.111:0/1543233016 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7140003570 con 0x7f7158158e50 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 -- 172.21.15.111:0/1543233016 >> v2:172.21.15.143:3300/0 conn(0x7f7158069310 msgr2=0x7f71580688c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 --2- 172.21.15.111:0/1543233016 >> v2:172.21.15.143:3300/0 conn(0x7f7158069310 0x7f71580688c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 -- 172.21.15.111:0/1543233016 >> v1:172.21.15.111:6789/0 conn(0x7f7158158e50 legacy=0x7f71580c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 -- 172.21.15.111:0/1543233016 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715806ca80 con 0x7f7158159220 2023-12-19T08:04:57.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f7155d9b700 1 --2- 172.21.15.111:0/1543233016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 0x7f7158068380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f714c0098e0 tx=0x7f714c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a78cdf47852102b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:57.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715559a700 1 -- 172.21.15.111:0/1543233016 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c011630 con 0x7f7158159220 2023-12-19T08:04:57.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715559a700 1 -- 172.21.15.111:0/1543233016 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c011790 con 0x7f7158159220 2023-12-19T08:04:57.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715559a700 1 -- 172.21.15.111:0/1543233016 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c011b30 con 0x7f7158159220 2023-12-19T08:04:57.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715c878700 1 -- 172.21.15.111:0/1543233016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 msgr2=0x7f7158068380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715c878700 1 --2- 172.21.15.111:0/1543233016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 0x7f7158068380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f714c0098e0 tx=0x7f714c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715c878700 1 -- 172.21.15.111:0/1543233016 shutdown_connections 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.758+0000 7f715c878700 1 --2- 172.21.15.111:0/1543233016 >> v2:172.21.15.143:3300/0 conn(0x7f7158069310 0x7f71580688c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 --2- 172.21.15.111:0/1543233016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158159220 0x7f7158068380 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- 172.21.15.111:0/1543233016 >> 172.21.15.111:0/1543233016 conn(0x7f71580c8db0 msgr2=0x7f71580c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- 172.21.15.111:0/1543233016 shutdown_connections 2023-12-19T08:04:57.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- 172.21.15.111:0/1543233016 wait complete. 2023-12-19T08:04:57.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 Processor -- start 2023-12-19T08:04:57.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- start start 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158069310 0x7f71580af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7158158e50 0x7f7158152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580c42d0 con 0x7f7158159220 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580c36c0 con 0x7f7158158e50 2023-12-19T08:04:57.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715c878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71580c39a0 con 0x7f7158069310 2023-12-19T08:04:57.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f7156d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f7156d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34036/0 (socket says 172.21.15.111:34036) 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f7156d9d700 1 -- 172.21.15.111:0/4268028442 learned_addr learned my addr 172.21.15.111:0/4268028442 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f715659c700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158069310 0x7f71580af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f7155d9b700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7158158e50 0x7f7158152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.762+0000 7f7156d9d700 1 -- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158069310 msgr2=0x7f71580af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f7156d9d700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158069310 0x7f71580af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f7156d9d700 1 -- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7158158e50 msgr2=0x7f7158152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f7156d9d700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7158158e50 0x7f7158152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f7156d9d700 1 -- 172.21.15.111:0/4268028442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c009420 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f7156d9d700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e90 tx=0x7f715000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715000cc20 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715000f040 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71580c3c80 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150014480 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71580c3350 con 0x7f7158159220 2023-12-19T08:04:57.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.766+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158071120 con 0x7f7158159220 2023-12-19T08:04:57.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.770+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 49) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f7150014830 con 0x7f7158159220 2023-12-19T08:04:57.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.770+0000 7f71467fc700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 0x7f7138062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:57.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.770+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 7435+0+0 (secure 0 0 0) 0x7f715007d600 con 0x7f7158159220 2023-12-19T08:04:57.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.770+0000 7f715659c700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 0x7f7138062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:57.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.770+0000 7f715659c700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 0x7f7138062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71480058d0 tx=0x7f7148009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:57.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.778+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f71500511a0 con 0x7f7158159220 2023-12-19T08:04:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:57.950+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f715806cd60 con 0x7f7158159220 2023-12-19T08:04:58.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.626+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f715004dae0 con 0x7f7158159220 2023-12-19T08:04:58.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.638+0000 7f71467fc700 1 -- 172.21.15.111:0/4268028442 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7150050f10 con 0x7f7158159220 2023-12-19T08:04:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 msgr2=0x7f7138062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 0x7f7138062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71480058d0 tx=0x7f7148009040 comp rx=0 tx=0).stop 2023-12-19T08:04:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 msgr2=0x7f71581402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e90 tx=0x7f715000b040 comp rx=0 tx=0).stop 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 shutdown_connections 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f71380603a0 0x7f7138062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7158158e50 0x7f7158152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7158159220 0x7f71581402a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 --2- 172.21.15.111:0/4268028442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7158069310 0x7f71580af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 >> 172.21.15.111:0/4268028442 conn(0x7f71580c8db0 msgr2=0x7f71580adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 shutdown_connections 2023-12-19T08:04:58.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.642+0000 7f715c878700 1 -- 172.21.15.111:0/4268028442 wait complete. 2023-12-19T08:04:58.651 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T08:04:58.668 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T08:04:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.838+0000 7fd30302e700 1 Processor -- start 2023-12-19T08:04:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.838+0000 7fd30302e700 1 -- start start 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd30302e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc0cb500 0x7fd2fc15c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd30302e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2fc15c760 0x7fd2fc0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd30302e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0b0ac0 con 0x7fd2fc0cb130 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd30302e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2fc15cbe0 con 0x7fd2fc15c760 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd30302e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc06b210 con 0x7fd2fc0cb500 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2fc15c760 0x7fd2fc0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2fc15c760 0x7fd2fc0b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34050/0 (socket says 172.21.15.111:34050) 2023-12-19T08:04:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 -- 172.21.15.111:0/2633398990 learned_addr learned my addr 172.21.15.111:0/2633398990 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd2fbfff700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc0cb500 0x7fd2fc15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 -- 172.21.15.111:0/2633398990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc0cb500 msgr2=0x7fd2fc15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc0cb500 0x7fd2fc15c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 -- 172.21.15.111:0/2633398990 >> v1:172.21.15.143:6789/0 conn(0x7fd2fc0cb130 legacy=0x7fd2fc06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 -- 172.21.15.111:0/2633398990 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0ca810 con 0x7fd2fc15c760 2023-12-19T08:04:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.842+0000 7fd3015cb700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc15c760 0x7fd2fc0b0f50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009f20 tx=0x7fd2f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8019a52ca6f1d34e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd2fb7fe700 1 -- 172.21.15.111:0/2633398990 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400e070 con 0x7fd2fc15c760 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd2fb7fe700 1 -- 172.21.15.111:0/2633398990 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400ba80 con 0x7fd2fc15c760 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd2fb7fe700 1 -- 172.21.15.111:0/2633398990 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40092e0 con 0x7fd2fc15c760 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 -- 172.21.15.111:0/2633398990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc15c760 msgr2=0x7fd2fc0b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc15c760 0x7fd2fc0b0f50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009f20 tx=0x7fd2f4009840 comp rx=0 tx=0).stop 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 -- 172.21.15.111:0/2633398990 shutdown_connections 2023-12-19T08:04:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc15c760 0x7fd2fc0b0f50 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 --2- 172.21.15.111:0/2633398990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc0cb500 0x7fd2fc15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 -- 172.21.15.111:0/2633398990 >> 172.21.15.111:0/2633398990 conn(0x7fd2fc067bb0 msgr2=0x7fd2fc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 -- 172.21.15.111:0/2633398990 shutdown_connections 2023-12-19T08:04:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 -- 172.21.15.111:0/2633398990 wait complete. 2023-12-19T08:04:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.846+0000 7fd30302e700 1 Processor -- start 2023-12-19T08:04:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 -- start start 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 0x7fd2fc14b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc15c760 0x7fd2fc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc08a300 con 0x7fd2fc0cb130 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0896f0 con 0x7fd2fc0cb500 2023-12-19T08:04:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd30302e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc0899d0 con 0x7fd2fc15c760 2023-12-19T08:04:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34054/0 (socket says 172.21.15.111:34054) 2023-12-19T08:04:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 -- 172.21.15.111:0/466733944 learned_addr learned my addr 172.21.15.111:0/466733944 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd3015cb700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc15c760 0x7fd2fc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd2fbfff700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 0x7fd2fc14b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 -- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc15c760 msgr2=0x7fd2fc0b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc15c760 0x7fd2fc0b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 -- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 msgr2=0x7fd2fc14b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 0x7fd2fc14b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 -- 172.21.15.111:0/466733944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4009ae0 con 0x7fd2fc0cb130 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd2fbfff700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 0x7fd2fc14b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:04:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd300dca700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c4e0 tx=0x7fd2f000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0011590 con 0x7fd2fc0cb130 2023-12-19T08:04:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0011710 con 0x7fd2fc0cb130 2023-12-19T08:04:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.850+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0011ad0 con 0x7fd2fc0cb130 2023-12-19T08:04:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.854+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc089cb0 con 0x7fd2fc0cb130 2023-12-19T08:04:58.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.854+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc0b5280 con 0x7fd2fc0cb130 2023-12-19T08:04:58.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.854+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc08a130 con 0x7fd2fc0cb130 2023-12-19T08:04:58.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.854+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd2f001b3b0 con 0x7fd2fc0cb130 2023-12-19T08:04:58.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.858+0000 7fd2f8ff9700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 0x7fd2e0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:58.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.858+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6997+0+0 (secure 0 0 0) 0x7fd2f007c350 con 0x7fd2fc0cb130 2023-12-19T08:04:58.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.858+0000 7fd2fbfff700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 0x7fd2e0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:58.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.858+0000 7fd2fbfff700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 0x7fd2e0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc088ee0 tx=0x7fd2e8008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:58.862+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd2f004c320 con 0x7fd2fc0cb130 2023-12-19T08:04:59.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.038+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2fc071120 con 0x7fd2fc0cb130 2023-12-19T08:04:59.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.654+0000 7fd2f8ff9700 1 -- 172.21.15.111:0/466733944 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd2f004f940 con 0x7fd2fc0cb130 2023-12-19T08:04:59.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 msgr2=0x7fd2e0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 0x7fd2e0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc088ee0 tx=0x7fd2e8008040 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 msgr2=0x7fd2fc0890f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000c4e0 tx=0x7fd2f000c8a0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 shutdown_connections 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd2e0060670 0x7fd2e0062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2fc0cb500 0x7fd2fc14b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2fc15c760 0x7fd2fc0b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 --2- 172.21.15.111:0/466733944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2fc0cb130 0x7fd2fc0890f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 >> 172.21.15.111:0/466733944 conn(0x7fd2fc067bb0 msgr2=0x7fd2fc0b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 shutdown_connections 2023-12-19T08:04:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.662+0000 7fd30302e700 1 -- 172.21.15.111:0/466733944 wait complete. 2023-12-19T08:04:59.673 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T08:04:59.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T08:04:59.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.858+0000 7fe5eef63700 1 Processor -- start 2023-12-19T08:04:59.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.862+0000 7fe5eef63700 1 -- start start 2023-12-19T08:04:59.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.862+0000 7fe5eef63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 0x7fe5e80caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:59.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5eef63700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e806c850 con 0x7fe5e806aeb0 2023-12-19T08:04:59.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5eef63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e806b470 con 0x7fe5e8159770 2023-12-19T08:04:59.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5eef63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e806b750 con 0x7fe5e8158c90 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ed760700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe5e806aeb0 0x7fe5e8159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43732/0 (socket says 172.21.15.111:43732) 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ed760700 1 -- 172.21.15.111:0/415614169 learned_addr learned my addr 172.21.15.111:0/415614169 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5edf61700 1 --2- 172.21.15.111:0/415614169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 0x7fe5e80caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004683557 0 0) 0x7fe5e806c850 con 0x7fe5e806aeb0 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d4003570 con 0x7fe5e806aeb0 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673699641 0 0) 0x7fe5e806b470 con 0x7fe5e8159770 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e806c850 con 0x7fe5e8159770 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5edf61700 1 -- 172.21.15.111:0/415614169 >> v1:172.21.15.143:6789/0 conn(0x7fe5e8159770 legacy=0x7fe5e806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5edf61700 1 -- 172.21.15.111:0/415614169 >> v1:172.21.15.111:6789/0 conn(0x7fe5e806aeb0 legacy=0x7fe5e8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5edf61700 1 -- 172.21.15.111:0/415614169 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e806ba30 con 0x7fe5e8158c90 2023-12-19T08:04:59.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5edf61700 1 --2- 172.21.15.111:0/415614169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 0x7fe5e80caf40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009910 tx=0x7fe5e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8715428eed3992a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:59.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4011630 con 0x7fe5e8158c90 2023-12-19T08:04:59.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e4011790 con 0x7fe5e8158c90 2023-12-19T08:04:59.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.866+0000 7fe5ecf5f700 1 -- 172.21.15.111:0/415614169 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4011b30 con 0x7fe5e8158c90 2023-12-19T08:04:59.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- 172.21.15.111:0/415614169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 msgr2=0x7fe5e80caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 --2- 172.21.15.111:0/415614169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 0x7fe5e80caf40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009910 tx=0x7fe5e40091a0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- 172.21.15.111:0/415614169 shutdown_connections 2023-12-19T08:04:59.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 --2- 172.21.15.111:0/415614169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e8158c90 0x7fe5e80caf40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- 172.21.15.111:0/415614169 >> 172.21.15.111:0/415614169 conn(0x7fe5e8067bb0 msgr2=0x7fe5e80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:04:59.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- 172.21.15.111:0/415614169 shutdown_connections 2023-12-19T08:04:59.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- 172.21.15.111:0/415614169 wait complete. 2023-12-19T08:04:59.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 Processor -- start 2023-12-19T08:04:59.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.870+0000 7fe5eef63700 1 -- start start 2023-12-19T08:04:59.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 0x7fe5e8156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:59.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:59.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5e8159770 0x7fe5e814b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:59.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8154bd0 con 0x7fe5e8158c90 2023-12-19T08:04:59.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80c9210 con 0x7fe5e8159770 2023-12-19T08:04:59.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80c94f0 con 0x7fe5e806aeb0 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ed760700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ed760700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34070/0 (socket says 172.21.15.111:34070) 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ed760700 1 -- 172.21.15.111:0/640034707 learned_addr learned my addr 172.21.15.111:0/640034707 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 0x7fe5e8156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ee762700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5e8159770 0x7fe5e814b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 -- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5e8159770 msgr2=0x7fe5e814b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5e8159770 0x7fe5e814b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 -- 172.21.15.111:0/640034707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 msgr2=0x7fe5e814b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 -- 172.21.15.111:0/640034707 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009420 con 0x7fe5e806aeb0 2023-12-19T08:04:59.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ed760700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:04:59.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5edf61700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 0x7fe5e8156df0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4016000 tx=0x7fe5e4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:59.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e401f070 con 0x7fe5e806aeb0 2023-12-19T08:04:59.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e4002f70 con 0x7fe5e806aeb0 2023-12-19T08:04:59.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e80c97d0 con 0x7fe5e806aeb0 2023-12-19T08:04:59.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4011900 con 0x7fe5e806aeb0 2023-12-19T08:04:59.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.874+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e8157330 con 0x7fe5e806aeb0 2023-12-19T08:04:59.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.878+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e804be40 con 0x7fe5e806aeb0 2023-12-19T08:04:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.878+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe5e4023070 con 0x7fe5e806aeb0 2023-12-19T08:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.882+0000 7fe5ddffb700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 0x7fe5cc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.882+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fe5e407da50 con 0x7fe5e806aeb0 2023-12-19T08:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.882+0000 7fe5ed760700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 0x7fe5cc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:04:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.882+0000 7fe5ed760700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 0x7fe5cc062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80cf490 tx=0x7fe5d8009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:04:59.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:04:59.890+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe5e404dbd0 con 0x7fe5e806aeb0 2023-12-19T08:05:00.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.070+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fe5e806bd10 con 0x7fe5e806aeb0 2023-12-19T08:05:00.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.082+0000 7fe5ddffb700 1 -- 172.21.15.111:0/640034707 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v37) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe5e40511f0 con 0x7fe5e806aeb0 2023-12-19T08:05:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 msgr2=0x7fe5cc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 0x7fe5cc062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80cf490 tx=0x7fe5d8009040 comp rx=0 tx=0).stop 2023-12-19T08:05:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 msgr2=0x7fe5e8156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 0x7fe5e8156df0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4016000 tx=0x7fe5e4017000 comp rx=0 tx=0).stop 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 shutdown_connections 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe5cc0606a0 0x7fe5cc062b50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5e8159770 0x7fe5e814b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e8158c90 0x7fe5e814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 --2- 172.21.15.111:0/640034707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5e806aeb0 0x7fe5e8156df0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 >> 172.21.15.111:0/640034707 conn(0x7fe5e8067bb0 msgr2=0x7fe5e80cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 shutdown_connections 2023-12-19T08:05:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.086+0000 7fe5eef63700 1 -- 172.21.15.111:0/640034707 wait complete. 2023-12-19T08:05:00.116 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T08:05:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.278+0000 7f166a345700 1 Processor -- start 2023-12-19T08:05:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 -- start start 2023-12-19T08:05:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 0x7f166406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f166406a7a0 0x7f166406ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664159340 con 0x7f16641554b0 2023-12-19T08:05:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166406d790 con 0x7f166406a7a0 2023-12-19T08:05:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f166a345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166406cd00 con 0x7f16640ca2d0 2023-12-19T08:05:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f1663fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f16641554b0 0x7f16641597d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43746/0 (socket says 172.21.15.111:43746) 2023-12-19T08:05:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f1663fff700 1 -- 172.21.15.111:0/824471512 learned_addr learned my addr 172.21.15.111:0/824471512 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.282+0000 7f16688e2700 1 --2- 172.21.15.111:0/824471512 >> v2:172.21.15.111:3300/0 conn(0x7f166406a7a0 0x7f166406ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f165b7fe700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 0x7f166406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16637fe700 1 -- 172.21.15.111:0/824471512 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497632791 0 0) 0x7f1664159340 con 0x7f16641554b0 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16637fe700 1 -- 172.21.15.111:0/824471512 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1650003570 con 0x7f16641554b0 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16688e2700 1 -- 172.21.15.111:0/824471512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 msgr2=0x7f166406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16688e2700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 0x7f166406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16688e2700 1 -- 172.21.15.111:0/824471512 >> v1:172.21.15.111:6789/0 conn(0x7f16641554b0 legacy=0x7f16641597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16688e2700 1 -- 172.21.15.111:0/824471512 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640ca700 con 0x7f166406a7a0 2023-12-19T08:05:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f165b7fe700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 0x7f166406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16688e2700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166406a7a0 0x7f166406ac20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e60 tx=0x7f165c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74db6fa811c813f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16637fe700 1 -- 172.21.15.111:0/824471512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c00e070 con 0x7f166406a7a0 2023-12-19T08:05:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16637fe700 1 -- 172.21.15.111:0/824471512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c00bab0 con 0x7f166406a7a0 2023-12-19T08:05:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f16637fe700 1 -- 172.21.15.111:0/824471512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c00be70 con 0x7f166406a7a0 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 -- 172.21.15.111:0/824471512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166406a7a0 msgr2=0x7f166406ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166406a7a0 0x7f166406ac20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f165c009e60 tx=0x7f165c009780 comp rx=0 tx=0).stop 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 -- 172.21.15.111:0/824471512 shutdown_connections 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166406a7a0 0x7f166406ac20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 --2- 172.21.15.111:0/824471512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640ca2d0 0x7f166406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 -- 172.21.15.111:0/824471512 >> 172.21.15.111:0/824471512 conn(0x7f1664067bb0 msgr2=0x7f16640cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 -- 172.21.15.111:0/824471512 shutdown_connections 2023-12-19T08:05:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.286+0000 7f166a345700 1 -- 172.21.15.111:0/824471512 wait complete. 2023-12-19T08:05:00.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 Processor -- start 2023-12-19T08:05:00.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 -- start start 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a7a0 0x7f166414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 0x7f16640af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640b14d0 con 0x7f16640ca2d0 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166414dc10 con 0x7f16641554b0 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f166a345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c3ae0 con 0x7f166406a7a0 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34078/0 (socket says 172.21.15.111:34078) 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 -- 172.21.15.111:0/3575427788 learned_addr learned my addr 172.21.15.111:0/3575427788 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f16688e2700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 0x7f16640af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f1663fff700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a7a0 0x7f166414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 -- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a7a0 msgr2=0x7f166414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a7a0 0x7f166414d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 -- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 msgr2=0x7f16640af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 0x7f16640af0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 -- 172.21.15.111:0/3575427788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c009420 con 0x7f16640ca2d0 2023-12-19T08:05:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f16688e2700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 0x7f16640af0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.290+0000 7f165b7fe700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f164c00ec80 tx=0x7f164c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c00c9d0 con 0x7f16640ca2d0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c00cb30 con 0x7f16640ca2d0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c00ceb0 con 0x7f16640ca2d0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c009a20 con 0x7f16640ca2d0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16640c40a0 con 0x7f16640ca2d0 2023-12-19T08:05:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166406b0a0 con 0x7f16640ca2d0 2023-12-19T08:05:00.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.294+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f164c013680 con 0x7f16640ca2d0 2023-12-19T08:05:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.298+0000 7f1660ff9700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.298+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f164c07cff0 con 0x7f16640ca2d0 2023-12-19T08:05:00.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.298+0000 7f1663fff700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.298+0000 7f1663fff700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 0x7f1644062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16540058d0 tx=0x7f1654009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.306+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f164c04d1d0 con 0x7f16640ca2d0 2023-12-19T08:05:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.478+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1664158ef0 con 0x7f16640ca2d0 2023-12-19T08:05:00.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.482+0000 7f1660ff9700 1 -- 172.21.15.111:0/3575427788 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f164c054e60 con 0x7f16640ca2d0 2023-12-19T08:05:00.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 msgr2=0x7f1644062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 0x7f1644062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16540058d0 tx=0x7f1654009040 comp rx=0 tx=0).stop 2023-12-19T08:05:00.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 msgr2=0x7f16640aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f164c00ec80 tx=0x7f164c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 shutdown_connections 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16440606a0 0x7f1644062b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16641554b0 0x7f16640af0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16640ca2d0 0x7f16640aeba0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 --2- 172.21.15.111:0/3575427788 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a7a0 0x7f166414d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 >> 172.21.15.111:0/3575427788 conn(0x7f1664067bb0 msgr2=0x7f166406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 shutdown_connections 2023-12-19T08:05:00.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.486+0000 7f166a345700 1 -- 172.21.15.111:0/3575427788 wait complete. 2023-12-19T08:05:00.517 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T08:05:00.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.670+0000 7f77b9a8e700 1 Processor -- start 2023-12-19T08:05:00.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.670+0000 7f77b9a8e700 1 -- start start 2023-12-19T08:05:00.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b9a8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f77b4158fd0 0x7f77b406dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b9a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b40cc1e0 0x7f77b40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b9a8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b406d750 con 0x7f77b40cc610 2023-12-19T08:05:00.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b9a8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40cbaf0 con 0x7f77b4158fd0 2023-12-19T08:05:00.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b9a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b406e120 con 0x7f77b40cc1e0 2023-12-19T08:05:00.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f77b4158fd0 0x7f77b406dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f77b4158fd0 0x7f77b406dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34084/0 (socket says 172.21.15.111:34084) 2023-12-19T08:05:00.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 -- 172.21.15.111:0/2410587140 learned_addr learned my addr 172.21.15.111:0/2410587140 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:00.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b3fff700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b40cc1e0 0x7f77b40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 -- 172.21.15.111:0/2410587140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b40cc1e0 msgr2=0x7f77b40c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b40cc1e0 0x7f77b40c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 -- 172.21.15.111:0/2410587140 >> v1:172.21.15.143:6789/0 conn(0x7f77b40cc610 legacy=0x7f77b40c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 -- 172.21.15.111:0/2410587140 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b406cb40 con 0x7f77b4158fd0 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b8a8c700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b4158fd0 0x7f77b406dbe0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f77a80099b0 tx=0x7f77a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e7440376a8d340 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b37fe700 1 -- 172.21.15.111:0/2410587140 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800b860 con 0x7f77b4158fd0 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b37fe700 1 -- 172.21.15.111:0/2410587140 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a800b9c0 con 0x7f77b4158fd0 2023-12-19T08:05:00.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.674+0000 7f77b37fe700 1 -- 172.21.15.111:0/2410587140 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800bd60 con 0x7f77b4158fd0 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2410587140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b4158fd0 msgr2=0x7f77b406dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b4158fd0 0x7f77b406dbe0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f77a80099b0 tx=0x7f77a8009250 comp rx=0 tx=0).stop 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2410587140 shutdown_connections 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b40cc1e0 0x7f77b40c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2410587140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b4158fd0 0x7f77b406dbe0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2410587140 >> 172.21.15.111:0/2410587140 conn(0x7f77b40cdc00 msgr2=0x7f77b40ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:00.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2410587140 shutdown_connections 2023-12-19T08:05:00.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.678+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2410587140 wait complete. 2023-12-19T08:05:00.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 Processor -- start 2023-12-19T08:05:00.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 -- start start 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77b40cc1e0 0x7f77b4153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b4158fd0 0x7f77b415bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40c5ed0 con 0x7f77b40cc610 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b415a220 con 0x7f77b40cc1e0 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b9a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b415c220 con 0x7f77b4158fd0 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34098/0 (socket says 172.21.15.111:34098) 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b3fff700 1 -- 172.21.15.111:0/2287653636 learned_addr learned my addr 172.21.15.111:0/2287653636 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b8a8c700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77b40cc1e0 0x7f77b4153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.682+0000 7f77b928d700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b4158fd0 0x7f77b415bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 -- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b4158fd0 msgr2=0x7f77b415bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b4158fd0 0x7f77b415bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 -- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77b40cc1e0 msgr2=0x7f77b4153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77b40cc1e0 0x7f77b4153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 -- 172.21.15.111:0/2287653636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80094f0 con 0x7f77b40cc610 2023-12-19T08:05:00.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b3fff700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f77a000d890 tx=0x7f77a0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a00125f0 con 0x7f77b40cc610 2023-12-19T08:05:00.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a0012750 con 0x7f77b40cc610 2023-12-19T08:05:00.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0012af0 con 0x7f77b40cc610 2023-12-19T08:05:00.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b416cad0 con 0x7f77b40cc610 2023-12-19T08:05:00.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b416d090 con 0x7f77b40cc610 2023-12-19T08:05:00.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.686+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b404be40 con 0x7f77b40cc610 2023-12-19T08:05:00.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.690+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f77a001e460 con 0x7f77b40cc610 2023-12-19T08:05:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.690+0000 7f77b0ff9700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 0x7f7798062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.690+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f77a007d270 con 0x7f77b40cc610 2023-12-19T08:05:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.690+0000 7f77b8a8c700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 0x7f7798062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.694+0000 7f77b8a8c700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 0x7f7798062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f77a80099b0 tx=0x7f77a8017000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:00.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.694+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f77a004d3b0 con 0x7f77b40cc610 2023-12-19T08:05:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.866+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f77b406ce20 con 0x7f77b40cc610 2023-12-19T08:05:00.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.878+0000 7f77b0ff9700 1 -- 172.21.15.111:0/2287653636 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7f77a00509d0 con 0x7f77b40cc610 2023-12-19T08:05:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 msgr2=0x7f7798062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 0x7f7798062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f77a80099b0 tx=0x7f77a8017000 comp rx=0 tx=0).stop 2023-12-19T08:05:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 msgr2=0x7f77b415b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f77a000d890 tx=0x7f77a0008040 comp rx=0 tx=0).stop 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 shutdown_connections 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7798060670 0x7f7798062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77b4158fd0 0x7f77b415bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77b40cc610 0x7f77b415b680 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 --2- 172.21.15.111:0/2287653636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77b40cc1e0 0x7f77b4153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 >> 172.21.15.111:0/2287653636 conn(0x7f77b40cdc00 msgr2=0x7f77b40ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 shutdown_connections 2023-12-19T08:05:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:00.882+0000 7f77b9a8e700 1 -- 172.21.15.111:0/2287653636 wait complete. 2023-12-19T08:05:00.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-12-19T08:05:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 Processor -- start 2023-12-19T08:05:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 -- start start 2023-12-19T08:05:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fecf80ce810 0x7fecf806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf806b730 con 0x7fecf81591f0 2023-12-19T08:05:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf806ab20 con 0x7fecf8158e20 2023-12-19T08:05:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf7fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf806ae00 con 0x7fecf80ce810 2023-12-19T08:05:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf67fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fecf81591f0 0x7fecf80ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:43758/0 (socket says 172.21.15.111:43758) 2023-12-19T08:05:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.070+0000 7fecf67fc700 1 -- 172.21.15.111:0/2257493701 learned_addr learned my addr 172.21.15.111:0/2257493701 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf77fe700 1 --2- 172.21.15.111:0/2257493701 >> v2:172.21.15.143:3300/0 conn(0x7fecf80ce810 0x7fecf806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf5ffb700 1 -- 172.21.15.111:0/2257493701 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59179375 0 0) 0x7fecf806b730 con 0x7fecf81591f0 2023-12-19T08:05:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf5ffb700 1 -- 172.21.15.111:0/2257493701 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0003570 con 0x7fecf81591f0 2023-12-19T08:05:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf77fe700 1 -- 172.21.15.111:0/2257493701 >> v1:172.21.15.143:6789/0 conn(0x7fecf8158e20 legacy=0x7fecf80ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf77fe700 1 -- 172.21.15.111:0/2257493701 >> v1:172.21.15.111:6789/0 conn(0x7fecf81591f0 legacy=0x7fecf80ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf77fe700 1 -- 172.21.15.111:0/2257493701 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf806b0e0 con 0x7fecf80ce810 2023-12-19T08:05:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf77fe700 1 --2- 172.21.15.111:0/2257493701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf806a540 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009e60 tx=0x7fecf0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d4a5b96e45e1487 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf5ffb700 1 -- 172.21.15.111:0/2257493701 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf000e070 con 0x7fecf80ce810 2023-12-19T08:05:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf5ffb700 1 -- 172.21.15.111:0/2257493701 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf000bb60 con 0x7fecf80ce810 2023-12-19T08:05:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf5ffb700 1 -- 172.21.15.111:0/2257493701 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0012430 con 0x7fecf80ce810 2023-12-19T08:05:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 -- 172.21.15.111:0/2257493701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 msgr2=0x7fecf806a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 --2- 172.21.15.111:0/2257493701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf806a540 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009e60 tx=0x7fecf0009780 comp rx=0 tx=0).stop 2023-12-19T08:05:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 -- 172.21.15.111:0/2257493701 shutdown_connections 2023-12-19T08:05:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 --2- 172.21.15.111:0/2257493701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf806a540 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 -- 172.21.15.111:0/2257493701 >> 172.21.15.111:0/2257493701 conn(0x7fecf8067bb0 msgr2=0x7fecf80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 -- 172.21.15.111:0/2257493701 shutdown_connections 2023-12-19T08:05:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.074+0000 7fecf7fff700 1 -- 172.21.15.111:0/2257493701 wait complete. 2023-12-19T08:05:01.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 Processor -- start 2023-12-19T08:05:01.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 -- start start 2023-12-19T08:05:01.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf8154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fecf8158e20 0x7fecf815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81691e0 con 0x7fecf81591f0 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf816a000 con 0x7fecf80ce810 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf7fff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81693f0 con 0x7fecf8158e20 2023-12-19T08:05:01.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34108/0 (socket says 172.21.15.111:34108) 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.078+0000 7fecf77fe700 1 -- 172.21.15.111:0/3195234913 learned_addr learned my addr 172.21.15.111:0/3195234913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf67fc700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fecf8158e20 0x7fecf815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf6ffd700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf8154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 -- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fecf8158e20 msgr2=0x7fecf815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fecf8158e20 0x7fecf815bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 -- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 msgr2=0x7fecf8154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf8154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 -- 172.21.15.111:0/3195234913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0009a20 con 0x7fecf81591f0 2023-12-19T08:05:01.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf77fe700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fecf000b6c0 tx=0x7fecf0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0027470 con 0x7fecf81591f0 2023-12-19T08:05:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf00275f0 con 0x7fecf81591f0 2023-12-19T08:05:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf81696d0 con 0x7fecf81591f0 2023-12-19T08:05:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.082+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf00279b0 con 0x7fecf81591f0 2023-12-19T08:05:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.086+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf8169c90 con 0x7fecf81591f0 2023-12-19T08:05:01.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.086+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf804be40 con 0x7fecf81591f0 2023-12-19T08:05:01.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.086+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fecf0024070 con 0x7fecf81591f0 2023-12-19T08:05:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.090+0000 7fecdeffd700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 0x7fece4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.090+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fecf007f610 con 0x7fecf81591f0 2023-12-19T08:05:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.090+0000 7fecf6ffd700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 0x7fece4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.090+0000 7fecf6ffd700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 0x7fece4062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fece8003860 tx=0x7fece8009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.094+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fecf004f7f0 con 0x7fecf81591f0 2023-12-19T08:05:01.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.326+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fecf806b3c0 con 0x7fecf81591f0 2023-12-19T08:05:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.662+0000 7fecdeffd700 1 -- 172.21.15.111:0/3195234913 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v670) v1 ==== 183+0+0 (secure 0 0 0) 0x7fecf0052e10 con 0x7fecf81591f0 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 msgr2=0x7fece4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 0x7fece4062b50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fece8003860 tx=0x7fece8009040 comp rx=0 tx=0).stop 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 msgr2=0x7fecf80b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fecf000b6c0 tx=0x7fecf0019000 comp rx=0 tx=0).stop 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 shutdown_connections 2023-12-19T08:05:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fece40606a0 0x7fece4062b50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf81591f0 0x7fecf80b4d00 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fecf8158e20 0x7fecf815bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 --2- 172.21.15.111:0/3195234913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecf80ce810 0x7fecf8154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 >> 172.21.15.111:0/3195234913 conn(0x7fecf8067bb0 msgr2=0x7fecf80c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 shutdown_connections 2023-12-19T08:05:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.666+0000 7fecf7fff700 1 -- 172.21.15.111:0/3195234913 wait complete. 2023-12-19T08:05:01.697 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-12-19T08:05:01.698 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-12-19T08:05:01.698 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T08:05:01.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:05:01.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 Processor -- start 2023-12-19T08:05:01.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 -- start start 2023-12-19T08:05:01.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc06015d320 0x7fc06015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0600c7190 con 0x7fc0600aeba0 2023-12-19T08:05:01.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0600c1b50 con 0x7fc0600c15b0 2023-12-19T08:05:01.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.858+0000 7fc067d85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0600aefd0 con 0x7fc06015d320 2023-12-19T08:05:01.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc066322700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc0600aeba0 0x7fc0600680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49870/0 (socket says 172.21.15.111:49870) 2023-12-19T08:05:01.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc066322700 1 -- 172.21.15.111:0/1303648140 learned_addr learned my addr 172.21.15.111:0/1303648140 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:01.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc065b21700 1 --2- 172.21.15.111:0/1303648140 >> v2:172.21.15.111:3300/0 conn(0x7fc06015d320 0x7fc06015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141912333 0 0) 0x7fc0600c7190 con 0x7fc0600aeba0 2023-12-19T08:05:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c003570 con 0x7fc0600aeba0 2023-12-19T08:05:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049384759 0 0) 0x7fc0600c1b50 con 0x7fc0600c15b0 2023-12-19T08:05:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0600c7190 con 0x7fc0600c15b0 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc065b21700 1 -- 172.21.15.111:0/1303648140 >> v1:172.21.15.143:6789/0 conn(0x7fc0600c15b0 legacy=0x7fc06006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc065b21700 1 -- 172.21.15.111:0/1303648140 >> v1:172.21.15.111:6789/0 conn(0x7fc0600aeba0 legacy=0x7fc0600680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc065b21700 1 -- 172.21.15.111:0/1303648140 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0600c1e30 con 0x7fc06015d320 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc065b21700 1 --2- 172.21.15.111:0/1303648140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc06015d320 0x7fc06015d6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0098e0 tx=0x7fc05c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd312ee61b44728 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011630 con 0x7fc06015d320 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c011790 con 0x7fc06015d320 2023-12-19T08:05:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc064b1f700 1 -- 172.21.15.111:0/1303648140 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011b30 con 0x7fc06015d320 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 -- 172.21.15.111:0/1303648140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc06015d320 msgr2=0x7fc06015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 --2- 172.21.15.111:0/1303648140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc06015d320 0x7fc06015d6f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0098e0 tx=0x7fc05c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 -- 172.21.15.111:0/1303648140 shutdown_connections 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 --2- 172.21.15.111:0/1303648140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc06015d320 0x7fc06015d6f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 -- 172.21.15.111:0/1303648140 >> 172.21.15.111:0/1303648140 conn(0x7fc0600c8850 msgr2=0x7fc0600c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 -- 172.21.15.111:0/1303648140 shutdown_connections 2023-12-19T08:05:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.862+0000 7fc067d85700 1 -- 172.21.15.111:0/1303648140 wait complete. 2023-12-19T08:05:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 Processor -- start 2023-12-19T08:05:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 -- start start 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0600aeba0 0x7fc0601706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 0x7fc060067620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600c4960 con 0x7fc0600c15b0 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600c4df0 con 0x7fc0600aeba0 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc067d85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0600c3d50 con 0x7fc06015d320 2023-12-19T08:05:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc065320700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc065320700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49488/0 (socket says 172.21.15.111:49488) 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc065320700 1 -- 172.21.15.111:0/687245992 learned_addr learned my addr 172.21.15.111:0/687245992 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc065b21700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0600aeba0 0x7fc0601706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.866+0000 7fc066322700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 0x7fc060067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 -- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 msgr2=0x7fc060067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 0x7fc060067620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 -- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0600aeba0 msgr2=0x7fc0601706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0600aeba0 0x7fc0601706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 -- 172.21.15.111:0/687245992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c009420 con 0x7fc0600c15b0 2023-12-19T08:05:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc066322700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 0x7fc060067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc065320700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7c0 tx=0x7fc050007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050010510 con 0x7fc0600c15b0 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc050010670 con 0x7fc0600c15b0 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0600c4030 con 0x7fc0600c15b0 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050010a10 con 0x7fc0600c15b0 2023-12-19T08:05:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0600c45f0 con 0x7fc0600c15b0 2023-12-19T08:05:01.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.870+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0600c4b10 con 0x7fc0600c15b0 2023-12-19T08:05:01.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.874+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc05001b3b0 con 0x7fc0600c15b0 2023-12-19T08:05:01.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.874+0000 7fc055ffb700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 0x7fc044062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:01.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.874+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7fc05007c580 con 0x7fc0600c15b0 2023-12-19T08:05:01.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.874+0000 7fc065b21700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 0x7fc044062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:01.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.874+0000 7fc065b21700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 0x7fc044062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0093f0 tx=0x7fc05c017000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:01.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:01.878+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc0600c4b10 con 0x7fc0600c15b0 2023-12-19T08:05:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.054+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc060071120 con 0x7fc0600c15b0 2023-12-19T08:05:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc055ffb700 1 -- 172.21.15.111:0/687245992 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14319 (secure 0 0 0) 0x7fc050017090 con 0x7fc0600c15b0 2023-12-19T08:05:02.065 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:02.065 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":120,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:04:59.639813+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6835/1122478953":"2023-12-20T08:04:56.497550+0000","172.21.15.111:6834/1122478953":"2023-12-20T08:04:56.497550+0000","172.21.15.143:0/3275110363":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/3431282327":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/3244738998":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/1659494740":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/4253013021":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/2586102325":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/2526647480":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/144894101":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1370433618":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/596992054":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1492674104":"2023-12-20T08:04:56.283457+0000","172.21.15.143:0/4178727723":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/1808666646":"2023-12-20T08:03:48.038487+0000","172.21.15.143:0/2838896212":"2023-12-20T08:04:56.283457+0000","172.21.15.143:6833/33796":"2023-12-20T08:03:48.038487+0000","172.21.15.143:6832/33796":"2023-12-20T08:03:48.038487+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 msgr2=0x7fc044062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 0x7fc044062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0093f0 tx=0x7fc05c017000 comp rx=0 tx=0).stop 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 msgr2=0x7fc0600670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7c0 tx=0x7fc050007e10 comp rx=0 tx=0).stop 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 shutdown_connections 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc044060670 0x7fc044062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc06015d320 0x7fc060067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0600c15b0 0x7fc0600670e0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 --2- 172.21.15.111:0/687245992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0600aeba0 0x7fc0601706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.058+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 >> 172.21.15.111:0/687245992 conn(0x7fc0600c8850 msgr2=0x7fc06015b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:02.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.062+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 shutdown_connections 2023-12-19T08:05:02.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.062+0000 7fc067d85700 1 -- 172.21.15.111:0/687245992 wait complete. 2023-12-19T08:05:02.085 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:05:02.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:05:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.246+0000 7f00de785700 1 Processor -- start 2023-12-19T08:05:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.246+0000 7f00de785700 1 -- start start 2023-12-19T08:05:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00de785700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00d806aeb0 0x7f00d806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00de785700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d8159340 con 0x7f00d80ca2d0 2023-12-19T08:05:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00de785700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d80ca700 con 0x7f00d81554b0 2023-12-19T08:05:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00de785700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d806d790 con 0x7f00d806aeb0 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00dcd22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00d806aeb0 0x7f00d806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00cf7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f00d80ca2d0 0x7f00d806a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49880/0 (socket says 172.21.15.111:49880) 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00cf7fe700 1 -- 172.21.15.111:0/159097052 learned_addr learned my addr 172.21.15.111:0/159097052 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00dcd22700 1 -- 172.21.15.111:0/159097052 >> v1:172.21.15.143:6789/0 conn(0x7f00d81554b0 legacy=0x7f00d81597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00dcd22700 1 -- 172.21.15.111:0/159097052 >> v1:172.21.15.111:6789/0 conn(0x7f00d80ca2d0 legacy=0x7f00d806a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00dcd22700 1 -- 172.21.15.111:0/159097052 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d806cef0 con 0x7f00d806aeb0 2023-12-19T08:05:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00dcd22700 1 --2- 172.21.15.111:0/159097052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d806aeb0 0x7f00d806c7c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f00d000af70 tx=0x7f00d000a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1049965edd3866d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.250+0000 7f00d77fe700 1 -- 172.21.15.111:0/159097052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d000f070 con 0x7f00d806aeb0 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00d77fe700 1 -- 172.21.15.111:0/159097052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d000a2e0 con 0x7f00d806aeb0 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00d77fe700 1 -- 172.21.15.111:0/159097052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d000cd20 con 0x7f00d806aeb0 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 -- 172.21.15.111:0/159097052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d806aeb0 msgr2=0x7f00d806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 --2- 172.21.15.111:0/159097052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d806aeb0 0x7f00d806c7c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f00d000af70 tx=0x7f00d000a890 comp rx=0 tx=0).stop 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 -- 172.21.15.111:0/159097052 shutdown_connections 2023-12-19T08:05:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 --2- 172.21.15.111:0/159097052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d806aeb0 0x7f00d806c7c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 -- 172.21.15.111:0/159097052 >> 172.21.15.111:0/159097052 conn(0x7f00d8067bb0 msgr2=0x7f00d80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:02.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 -- 172.21.15.111:0/159097052 shutdown_connections 2023-12-19T08:05:02.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 -- 172.21.15.111:0/159097052 wait complete. 2023-12-19T08:05:02.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.254+0000 7f00de785700 1 Processor -- start 2023-12-19T08:05:02.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- start start 2023-12-19T08:05:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00d806aeb0 0x7f00d81531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00d80ca2d0 0x7f00d80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80aeba0 con 0x7f00d81554b0 2023-12-19T08:05:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80af440 con 0x7f00d80ca2d0 2023-12-19T08:05:02.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80c4d80 con 0x7f00d806aeb0 2023-12-19T08:05:02.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49512/0 (socket says 172.21.15.111:49512) 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 -- 172.21.15.111:0/3036264354 learned_addr learned my addr 172.21.15.111:0/3036264354 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00d7fff700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00d806aeb0 0x7f00d81531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00cf7fe700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00d80ca2d0 0x7f00d80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 -- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00d806aeb0 msgr2=0x7f00d81531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00d806aeb0 0x7f00d81531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 -- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00d80ca2d0 msgr2=0x7f00d80af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00d80ca2d0 0x7f00d80af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 -- 172.21.15.111:0/3036264354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d000ab30 con 0x7f00d81554b0 2023-12-19T08:05:02.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00dcd22700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f00d8069e00 tx=0x7f00d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0009e90 con 0x7f00d81554b0 2023-12-19T08:05:02.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d001d440 con 0x7f00d81554b0 2023-12-19T08:05:02.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d80aedf0 con 0x7f00d81554b0 2023-12-19T08:05:02.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d001d820 con 0x7f00d81554b0 2023-12-19T08:05:02.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.258+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d8153700 con 0x7f00d81554b0 2023-12-19T08:05:02.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.262+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d80af5b0 con 0x7f00d81554b0 2023-12-19T08:05:02.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.262+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f00d0003050 con 0x7f00d81554b0 2023-12-19T08:05:02.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.266+0000 7f00d4ff9700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 0x7f00b8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.266+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f00d007e920 con 0x7f00d81554b0 2023-12-19T08:05:02.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.266+0000 7f00d7fff700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 0x7f00b8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.266+0000 7f00d7fff700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 0x7f00b8062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00c8003860 tx=0x7f00c8009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.274+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00d004e950 con 0x7f00d81554b0 2023-12-19T08:05:02.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.442+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00d8071120 con 0x7f00d81554b0 2023-12-19T08:05:02.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.442+0000 7f00d4ff9700 1 -- 172.21.15.111:0/3036264354 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f00d0051f70 con 0x7f00d81554b0 2023-12-19T08:05:02.451 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:05:02.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 msgr2=0x7f00b8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 0x7f00b8062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00c8003860 tx=0x7f00c8009040 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 msgr2=0x7f00d80afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f00d8069e00 tx=0x7f00d0019000 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 shutdown_connections 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f00b8060670 0x7f00b8062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00d80ca2d0 0x7f00d80af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00d81554b0 0x7f00d80afe10 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 --2- 172.21.15.111:0/3036264354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00d806aeb0 0x7f00d81531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 >> 172.21.15.111:0/3036264354 conn(0x7f00d8067bb0 msgr2=0x7f00d806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 shutdown_connections 2023-12-19T08:05:02.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.446+0000 7f00de785700 1 -- 172.21.15.111:0/3036264354 wait complete. 2023-12-19T08:05:02.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-12-19T08:05:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.630+0000 7f64b4fc5700 1 Processor -- start 2023-12-19T08:05:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.630+0000 7f64b4fc5700 1 -- start start 2023-12-19T08:05:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64b4fc5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64b00ab960 0x7f64b0159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64b4fc5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64b0155350 0x7f64b006b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64b4fc5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00b0080 con 0x7f64b01557f0 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64b4fc5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b015a010 con 0x7f64b00ab960 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64b4fc5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b006dee0 con 0x7f64b0155350 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64aed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f64b01557f0 0x7f64b006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49884/0 (socket says 172.21.15.111:49884) 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64aed9d700 1 -- 172.21.15.111:0/1179368978 learned_addr learned my addr 172.21.15.111:0/1179368978 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 --2- 172.21.15.111:0/1179368978 >> v2:172.21.15.111:3300/0 conn(0x7f64b00ab960 0x7f64b0159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64add9b700 1 --2- 172.21.15.111:0/1179368978 >> v2:172.21.15.143:3300/0 conn(0x7f64b0155350 0x7f64b006b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 -- 172.21.15.111:0/1179368978 >> v2:172.21.15.143:3300/0 conn(0x7f64b0155350 msgr2=0x7f64b006b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 --2- 172.21.15.111:0/1179368978 >> v2:172.21.15.143:3300/0 conn(0x7f64b0155350 0x7f64b006b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 -- 172.21.15.111:0/1179368978 >> v1:172.21.15.111:6789/0 conn(0x7f64b01557f0 legacy=0x7f64b006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 -- 172.21.15.111:0/1179368978 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b006e740 con 0x7f64b00ab960 2023-12-19T08:05:02.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ae59c700 1 --2- 172.21.15.111:0/1179368978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b00ab960 0x7f64b0159ad0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f64b0070490 tx=0x7f64a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e941c3c617bc9e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ad59a700 1 -- 172.21.15.111:0/1179368978 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0011630 con 0x7f64b00ab960 2023-12-19T08:05:02.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ad59a700 1 -- 172.21.15.111:0/1179368978 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0011790 con 0x7f64b00ab960 2023-12-19T08:05:02.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.634+0000 7f64ad59a700 1 -- 172.21.15.111:0/1179368978 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0011b30 con 0x7f64b00ab960 2023-12-19T08:05:02.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 -- 172.21.15.111:0/1179368978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b00ab960 msgr2=0x7f64b0159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/1179368978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b00ab960 0x7f64b0159ad0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f64b0070490 tx=0x7f64a00091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 -- 172.21.15.111:0/1179368978 shutdown_connections 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/1179368978 >> v2:172.21.15.143:3300/0 conn(0x7f64b0155350 0x7f64b006b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/1179368978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b00ab960 0x7f64b0159ad0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 -- 172.21.15.111:0/1179368978 >> 172.21.15.111:0/1179368978 conn(0x7f64b0067410 msgr2=0x7f64b00677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 -- 172.21.15.111:0/1179368978 shutdown_connections 2023-12-19T08:05:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.638+0000 7f64b4fc5700 1 -- 172.21.15.111:0/1179368978 wait complete. 2023-12-19T08:05:02.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 Processor -- start 2023-12-19T08:05:02.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 -- start start 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64b00ab960 0x7f64b00b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64b01557f0 0x7f64b00a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00a8c60 con 0x7f64b0155350 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0157b60 con 0x7f64b01557f0 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64b4fc5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01575a0 con 0x7f64b00ab960 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49536/0 (socket says 172.21.15.111:49536) 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 -- 172.21.15.111:0/637498804 learned_addr learned my addr 172.21.15.111:0/637498804 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64aed9d700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64b01557f0 0x7f64b00a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64ae59c700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64b00ab960 0x7f64b00b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 -- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64b00ab960 msgr2=0x7f64b00b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64b00ab960 0x7f64b00b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 -- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64b01557f0 msgr2=0x7f64b00a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64b01557f0 0x7f64b00a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 -- 172.21.15.111:0/637498804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0009420 con 0x7f64b0155350 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.642+0000 7f64add9b700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ec80 tx=0x7f64a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a400c850 con 0x7f64b0155350 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a400c9b0 con 0x7f64b0155350 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a400cd50 con 0x7f64b0155350 2023-12-19T08:05:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0157880 con 0x7f64b0155350 2023-12-19T08:05:02.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b00b0c80 con 0x7f64b0155350 2023-12-19T08:05:02.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.646+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b00a8e10 con 0x7f64b0155350 2023-12-19T08:05:02.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.650+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f64a4013680 con 0x7f64b0155350 2023-12-19T08:05:02.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.650+0000 7f649e7fc700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 0x7f6490062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:02.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.650+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f64a407ced0 con 0x7f64b0155350 2023-12-19T08:05:02.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.650+0000 7f64ae59c700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 0x7f6490062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:02.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.650+0000 7f64ae59c700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 0x7f6490062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64a00091e0 tx=0x7f64a0017000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:02.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.654+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f64b00a8e10 con 0x7f64b0155350 2023-12-19T08:05:02.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:02.890+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f64b00b0f00 con 0x7f64b0155350 2023-12-19T08:05:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.678+0000 7f649e7fc700 1 -- 172.21.15.111:0/637498804 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v672) v1 ==== 151+0+0 (secure 0 0 0) 0x7f64a404d010 con 0x7f64b0155350 2023-12-19T08:05:03.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 msgr2=0x7f6490062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 0x7f6490062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f64a00091e0 tx=0x7f64a0017000 comp rx=0 tx=0).stop 2023-12-19T08:05:03.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 msgr2=0x7f64b0156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ec80 tx=0x7f64a40091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 shutdown_connections 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6490060670 0x7f6490062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64b01557f0 0x7f64b00a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b0155350 0x7f64b0156be0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 --2- 172.21.15.111:0/637498804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64b00ab960 0x7f64b00b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 >> 172.21.15.111:0/637498804 conn(0x7f64b0067410 msgr2=0x7f64b0154660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 shutdown_connections 2023-12-19T08:05:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.682+0000 7f64b4fc5700 1 -- 172.21.15.111:0/637498804 wait complete. 2023-12-19T08:05:03.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:05:03.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.874+0000 7f43466cd700 1 Processor -- start 2023-12-19T08:05:03.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.874+0000 7f43466cd700 1 -- start start 2023-12-19T08:05:03.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f434015c520 0x7f43400a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f434015c8f0 0x7f434006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400c6ab0 con 0x7f434006ace0 2023-12-19T08:05:03.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434006b520 con 0x7f434015c520 2023-12-19T08:05:03.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400cb910 con 0x7f434015c8f0 2023-12-19T08:05:03.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f434015c520 0x7f43400a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f4344c6a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f434006ace0 0x7f43400ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49890/0 (socket says 172.21.15.111:49890) 2023-12-19T08:05:03.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f4344c6a700 1 -- 172.21.15.111:0/1649808804 learned_addr learned my addr 172.21.15.111:0/1649808804 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:03.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43377fe700 1 --2- 172.21.15.111:0/1649808804 >> v2:172.21.15.143:3300/0 conn(0x7f434015c8f0 0x7f434006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 -- 172.21.15.111:0/1649808804 >> v2:172.21.15.143:3300/0 conn(0x7f434015c8f0 msgr2=0x7f434006a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:03.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 --2- 172.21.15.111:0/1649808804 >> v2:172.21.15.143:3300/0 conn(0x7f434015c8f0 0x7f434006a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 -- 172.21.15.111:0/1649808804 >> v1:172.21.15.111:6789/0 conn(0x7f434006ace0 legacy=0x7f43400ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 -- 172.21.15.111:0/1649808804 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434006b240 con 0x7f434015c520 2023-12-19T08:05:03.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433ffff700 1 --2- 172.21.15.111:0/1649808804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f43400a8850 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f43400c9a50 tx=0x7f4330009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112260950b5c7d4c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:03.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433f7fe700 1 -- 172.21.15.111:0/1649808804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000e070 con 0x7f434015c520 2023-12-19T08:05:03.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433f7fe700 1 -- 172.21.15.111:0/1649808804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000bcc0 con 0x7f434015c520 2023-12-19T08:05:03.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f433f7fe700 1 -- 172.21.15.111:0/1649808804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330012680 con 0x7f434015c520 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- 172.21.15.111:0/1649808804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 msgr2=0x7f43400a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 --2- 172.21.15.111:0/1649808804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f43400a8850 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f43400c9a50 tx=0x7f4330009780 comp rx=0 tx=0).stop 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- 172.21.15.111:0/1649808804 shutdown_connections 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 --2- 172.21.15.111:0/1649808804 >> v2:172.21.15.143:3300/0 conn(0x7f434015c8f0 0x7f434006a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 --2- 172.21.15.111:0/1649808804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f43400a8850 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- 172.21.15.111:0/1649808804 >> 172.21.15.111:0/1649808804 conn(0x7f4340067bb0 msgr2=0x7f43400cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- 172.21.15.111:0/1649808804 shutdown_connections 2023-12-19T08:05:03.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.878+0000 7f43466cd700 1 -- 172.21.15.111:0/1649808804 wait complete. 2023-12-19T08:05:03.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.882+0000 7f43466cd700 1 Processor -- start 2023-12-19T08:05:03.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.882+0000 7f43466cd700 1 -- start start 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f434006ace0 0x7f43400a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f434015c8f0 0x7f43400c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43400a9aa0 con 0x7f434015c520 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43400a8e90 con 0x7f434015c8f0 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43466cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43400a9170 con 0x7f434006ace0 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49554/0 (socket says 172.21.15.111:49554) 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 -- 172.21.15.111:0/1277675216 learned_addr learned my addr 172.21.15.111:0/1277675216 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:03.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f433ffff700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f434006ace0 0x7f43400a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f4344c6a700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f434015c8f0 0x7f43400c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 -- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f434006ace0 msgr2=0x7f43400a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f434006ace0 0x7f43400a9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 -- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f434015c8f0 msgr2=0x7f43400c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f434015c8f0 0x7f43400c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.886+0000 7f43377fe700 1 -- 172.21.15.111:0/1277675216 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330009a20 con 0x7f434015c520 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f43377fe700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f432800d860 tx=0x7f4328008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43280125f0 con 0x7f434015c520 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4328012750 con 0x7f434015c520 2023-12-19T08:05:03.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43400a9450 con 0x7f434015c520 2023-12-19T08:05:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328012b10 con 0x7f434015c520 2023-12-19T08:05:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434016a930 con 0x7f434015c520 2023-12-19T08:05:03.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.890+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43400a9c50 con 0x7f434015c520 2023-12-19T08:05:03.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.894+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f432801e460 con 0x7f434015c520 2023-12-19T08:05:03.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.894+0000 7f433cff9700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 0x7f4320062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:03.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.894+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f432807d120 con 0x7f434015c520 2023-12-19T08:05:03.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.894+0000 7f433ffff700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 0x7f4320062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:03.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.894+0000 7f433ffff700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 0x7f4320062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f4330019000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:03.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:03.898+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43400a9c50 con 0x7f434015c520 2023-12-19T08:05:04.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.066+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f43400a9c50 con 0x7f434015c520 2023-12-19T08:05:04.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.078+0000 7f433cff9700 1 -- 172.21.15.111:0/1277675216 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f43400a9c50 con 0x7f434015c520 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 msgr2=0x7f4320062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 0x7f4320062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f4330019000 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 msgr2=0x7f434016a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f432800d860 tx=0x7f4328008040 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 shutdown_connections 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43200606a0 0x7f4320062b50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f434015c8f0 0x7f43400c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f434015c520 0x7f434016a3f0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 --2- 172.21.15.111:0/1277675216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f434006ace0 0x7f43400a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 >> 172.21.15.111:0/1277675216 conn(0x7f4340067bb0 msgr2=0x7f43400c0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:04.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 shutdown_connections 2023-12-19T08:05:04.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.082+0000 7f43466cd700 1 -- 172.21.15.111:0/1277675216 wait complete. 2023-12-19T08:05:04.113 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:05:04.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:05:04.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:05:04.204 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:05:04.204 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:05:04.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:05:04.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.370+0000 7f43d726e700 1 Processor -- start 2023-12-19T08:05:04.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.370+0000 7f43d726e700 1 -- start start 2023-12-19T08:05:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d726e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43d00ae520 0x7f43d01556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d726e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43d006c7c0 0x7f43d006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d726e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d006d0f0 con 0x7f43d01552f0 2023-12-19T08:05:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d726e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d00c8710 con 0x7f43d00ae520 2023-12-19T08:05:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d726e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d00c1b90 con 0x7f43d006c7c0 2023-12-19T08:05:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43d00ae520 0x7f43d01556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43d00ae520 0x7f43d01556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49566/0 (socket says 172.21.15.111:49566) 2023-12-19T08:05:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 -- 172.21.15.111:0/1345516224 learned_addr learned my addr 172.21.15.111:0/1345516224 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d580b700 1 --2- 172.21.15.111:0/1345516224 >> v2:172.21.15.143:3300/0 conn(0x7f43d006c7c0 0x7f43d006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 -- 172.21.15.111:0/1345516224 >> v2:172.21.15.143:3300/0 conn(0x7f43d006c7c0 msgr2=0x7f43d006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 --2- 172.21.15.111:0/1345516224 >> v2:172.21.15.143:3300/0 conn(0x7f43d006c7c0 0x7f43d006cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 -- 172.21.15.111:0/1345516224 >> v1:172.21.15.143:6789/0 conn(0x7f43d01552f0 legacy=0x7f43d00c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 -- 172.21.15.111:0/1345516224 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d006a9d0 con 0x7f43d00ae520 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43c7fff700 1 -- 172.21.15.111:0/1345516224 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110151547 0 0) 0x7f43d006d0f0 con 0x7f43d01552f0 2023-12-19T08:05:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43d4809700 1 --2- 172.21.15.111:0/1345516224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d00ae520 0x7f43d01556c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f43d006e2b0 tx=0x7f43c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5afa013e3c81c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43c7fff700 1 -- 172.21.15.111:0/1345516224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0009e70 con 0x7f43d00ae520 2023-12-19T08:05:04.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43c7fff700 1 -- 172.21.15.111:0/1345516224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c000b860 con 0x7f43d00ae520 2023-12-19T08:05:04.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.374+0000 7f43c7fff700 1 -- 172.21.15.111:0/1345516224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c000bc00 con 0x7f43d00ae520 2023-12-19T08:05:04.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- 172.21.15.111:0/1345516224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d00ae520 msgr2=0x7f43d01556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 --2- 172.21.15.111:0/1345516224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d00ae520 0x7f43d01556c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f43d006e2b0 tx=0x7f43c0009250 comp rx=0 tx=0).stop 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- 172.21.15.111:0/1345516224 shutdown_connections 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 --2- 172.21.15.111:0/1345516224 >> v2:172.21.15.143:3300/0 conn(0x7f43d006c7c0 0x7f43d006cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 --2- 172.21.15.111:0/1345516224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d00ae520 0x7f43d01556c0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- 172.21.15.111:0/1345516224 >> 172.21.15.111:0/1345516224 conn(0x7f43d00c8850 msgr2=0x7f43d00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- 172.21.15.111:0/1345516224 shutdown_connections 2023-12-19T08:05:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- 172.21.15.111:0/1345516224 wait complete. 2023-12-19T08:05:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 Processor -- start 2023-12-19T08:05:04.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.378+0000 7f43d726e700 1 -- start start 2023-12-19T08:05:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d006c7c0 0x7f43d00c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d00ae520 0x7f43d00b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d00c3770 con 0x7f43d01552f0 2023-12-19T08:05:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d00c3a50 con 0x7f43d006c7c0 2023-12-19T08:05:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d00c3d30 con 0x7f43d00ae520 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49582/0 (socket says 172.21.15.111:49582) 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 -- 172.21.15.111:0/2051774072 learned_addr learned my addr 172.21.15.111:0/2051774072 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d500a700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d006c7c0 0x7f43d00c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d4809700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d00ae520 0x7f43d00b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 -- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d00ae520 msgr2=0x7f43d00b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d00ae520 0x7f43d00b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 -- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d006c7c0 msgr2=0x7f43d00c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d006c7c0 0x7f43d00c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 -- 172.21.15.111:0/2051774072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c00094f0 con 0x7f43d01552f0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d580b700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009df0 tx=0x7f43c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800caa0 con 0x7f43d01552f0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800f040 con 0x7f43d01552f0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d00c4010 con 0x7f43d01552f0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800ce70 con 0x7f43d01552f0 2023-12-19T08:05:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.382+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d00c33d0 con 0x7f43d01552f0 2023-12-19T08:05:04.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.386+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d00c4510 con 0x7f43d01552f0 2023-12-19T08:05:04.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.386+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f43c8014690 con 0x7f43d01552f0 2023-12-19T08:05:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.390+0000 7f43c57fa700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 0x7f43b4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.390+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f43c807d4e0 con 0x7f43d01552f0 2023-12-19T08:05:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.390+0000 7f43d500a700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 0x7f43b4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.390+0000 7f43d500a700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 0x7f43b4062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f43cc007fd0 tx=0x7f43cc007d30 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.394+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43c8051040 con 0x7f43d01552f0 2023-12-19T08:05:04.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.606+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43d0071120 con 0x7f43d01552f0 2023-12-19T08:05:04.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.606+0000 7f43c57fa700 1 -- 172.21.15.111:0/2051774072 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f43c8018df0 con 0x7f43d01552f0 2023-12-19T08:05:04.613 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:04.614 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3420165988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3420165988},{"type":"v1","addr":"172.21.15.143:6837","nonce":3420165988}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7377,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1993897071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1993897071},{"type":"v1","addr":"172.21.15.111:6833","nonce":1993897071}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3217398520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3217398520},{"type":"v1","addr":"172.21.15.143:6835","nonce":3217398520}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":7776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3436255106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3436255106},{"type":"v1","addr":"172.21.15.111:6835","nonce":3436255106}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[]} 2023-12-19T08:05:04.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 msgr2=0x7f43b4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 0x7f43b4062b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f43cc007fd0 tx=0x7f43cc007d30 comp rx=0 tx=0).stop 2023-12-19T08:05:04.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 msgr2=0x7f43d014b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009df0 tx=0x7f43c800b040 comp rx=0 tx=0).stop 2023-12-19T08:05:04.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 shutdown_connections 2023-12-19T08:05:04.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f43b40606a0 0x7f43b4062b50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43d01552f0 0x7f43d014b460 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d00ae520 0x7f43d00b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 --2- 172.21.15.111:0/2051774072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d006c7c0 0x7f43d00c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 >> 172.21.15.111:0/2051774072 conn(0x7f43d00c8850 msgr2=0x7f43d00cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:04.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 shutdown_connections 2023-12-19T08:05:04.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.610+0000 7f43d726e700 1 -- 172.21.15.111:0/2051774072 wait complete. 2023-12-19T08:05:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 50 2023-12-19T08:05:04.643 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:05:04.644 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:05:04.644 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:05:04.644 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:05:04.644 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:05:04.644 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:05:04.645 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:05:04.645 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:05:04.645 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:05:04.645 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:05:04.645 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:05:04.645 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:05:04.645 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:05:04.646 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:05:04.646 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:05:04.646 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:05:04.664 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:05:04.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:05:04.665 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:05:04.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:05:04.666 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:05:04.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:05:04.668 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:05:04.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:05:04.721 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:05:04.722 DEBUG:teuthology.parallel:result is None 2023-12-19T08:05:04.723 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:05:04.724 DEBUG:teuthology.parallel:result is None 2023-12-19T08:05:04.725 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:05:04.725 DEBUG:teuthology.parallel:result is None 2023-12-19T08:05:04.727 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:05:04.727 DEBUG:teuthology.parallel:result is None 2023-12-19T08:05:04.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:05:04.826 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:05:04.827 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:05:04.838 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:05:04.838 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:05:04.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 Processor -- start 2023-12-19T08:05:04.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 -- start start 2023-12-19T08:05:04.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff20806c7c0 0x7ff2080b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20806d7d0 con 0x7ff2081407f0 2023-12-19T08:05:04.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20806cc50 con 0x7ff208140420 2023-12-19T08:05:04.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.882+0000 7ff20daa5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2080ae280 con 0x7ff20806c7c0 2023-12-19T08:05:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff207fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff2081407f0 0x7ff2080ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49906/0 (socket says 172.21.15.111:49906) 2023-12-19T08:05:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff207fff700 1 -- 172.21.15.111:0/371705397 learned_addr learned my addr 172.21.15.111:0/371705397 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff206ffd700 1 --2- 172.21.15.111:0/371705397 >> v2:172.21.15.143:3300/0 conn(0x7ff20806c7c0 0x7ff2080b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff2067fc700 1 -- 172.21.15.111:0/371705397 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834859788 0 0) 0x7ff20806d7d0 con 0x7ff2081407f0 2023-12-19T08:05:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff2067fc700 1 -- 172.21.15.111:0/371705397 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f0003570 con 0x7ff2081407f0 2023-12-19T08:05:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff206ffd700 1 -- 172.21.15.111:0/371705397 >> v1:172.21.15.143:6789/0 conn(0x7ff208140420 legacy=0x7ff20815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff206ffd700 1 -- 172.21.15.111:0/371705397 >> v1:172.21.15.111:6789/0 conn(0x7ff2081407f0 legacy=0x7ff2080ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff206ffd700 1 -- 172.21.15.111:0/371705397 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20806cf30 con 0x7ff20806c7c0 2023-12-19T08:05:04.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff206ffd700 1 --2- 172.21.15.111:0/371705397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff20806c7c0 0x7ff2080b0cd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0098e0 tx=0x7ff1fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b229bb18dad0a42b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff2067fc700 1 -- 172.21.15.111:0/371705397 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc00b640 con 0x7ff20806c7c0 2023-12-19T08:05:04.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff2067fc700 1 -- 172.21.15.111:0/371705397 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc00b7a0 con 0x7ff20806c7c0 2023-12-19T08:05:04.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff2067fc700 1 -- 172.21.15.111:0/371705397 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc00bb40 con 0x7ff20806c7c0 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 -- 172.21.15.111:0/371705397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff20806c7c0 msgr2=0x7ff2080b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 --2- 172.21.15.111:0/371705397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff20806c7c0 0x7ff2080b0cd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0098e0 tx=0x7ff1fc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 -- 172.21.15.111:0/371705397 shutdown_connections 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 --2- 172.21.15.111:0/371705397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff20806c7c0 0x7ff2080b0cd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 -- 172.21.15.111:0/371705397 >> 172.21.15.111:0/371705397 conn(0x7ff208067b60 msgr2=0x7ff2080ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 -- 172.21.15.111:0/371705397 shutdown_connections 2023-12-19T08:05:04.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.886+0000 7ff20daa5700 1 -- 172.21.15.111:0/371705397 wait complete. 2023-12-19T08:05:04.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 Processor -- start 2023-12-19T08:05:04.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 -- start start 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff208140420 0x7ff2081364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 0x7ff208157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c8890 con 0x7ff20806c7c0 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20813de10 con 0x7ff208140420 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff20daa5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20813e0f0 con 0x7ff2081407f0 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff2077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff2077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49648/0 (socket says 172.21.15.111:49648) 2023-12-19T08:05:04.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.890+0000 7ff2077fe700 1 -- 172.21.15.111:0/4236084334 learned_addr learned my addr 172.21.15.111:0/4236084334 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:04.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff206ffd700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff208140420 0x7ff2081364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff207fff700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 0x7ff208157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 -- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 msgr2=0x7ff208157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 0x7ff208157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 -- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff208140420 msgr2=0x7ff2081364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff208140420 0x7ff2081364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 -- 172.21.15.111:0/4236084334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc009420 con 0x7ff20806c7c0 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff207fff700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 0x7ff208157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff2077fe700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c590 tx=0x7ff1f800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011590 con 0x7ff20806c7c0 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8011710 con 0x7ff20806c7c0 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011ad0 con 0x7ff20806c7c0 2023-12-19T08:05:04.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20813e3d0 con 0x7ff20806c7c0 2023-12-19T08:05:04.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.894+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080c84d0 con 0x7ff20806c7c0 2023-12-19T08:05:04.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.898+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208071120 con 0x7ff20806c7c0 2023-12-19T08:05:04.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.898+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff1f801b3b0 con 0x7ff20806c7c0 2023-12-19T08:05:04.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.898+0000 7ff1ef7fe700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 0x7ff1f4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:04.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.898+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6467+0+0 (secure 0 0 0) 0x7ff1f807c140 con 0x7ff20806c7c0 2023-12-19T08:05:04.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.898+0000 7ff206ffd700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 0x7ff1f4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:04.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.902+0000 7ff206ffd700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 0x7ff1f4062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0093f0 tx=0x7ff1fc017000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:04.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:04.906+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff1f804c320 con 0x7ff20806c7c0 2023-12-19T08:05:05.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:05.090+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff20806d210 con 0x7ff20806c7c0 2023-12-19T08:05:06.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.106+0000 7ff1ef7fe700 1 -- 172.21.15.111:0/4236084334 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v125) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff1f804f940 con 0x7ff20806c7c0 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 msgr2=0x7ff1f4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 0x7ff1f4062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0093f0 tx=0x7ff1fc017000 comp rx=0 tx=0).stop 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 msgr2=0x7ff2080c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c590 tx=0x7ff1f800c8a0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 shutdown_connections 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1f4060670 0x7ff1f4062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff208140420 0x7ff2081364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2081407f0 0x7ff208157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 --2- 172.21.15.111:0/4236084334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff20806c7c0 0x7ff2080c7f90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 >> 172.21.15.111:0/4236084334 conn(0x7ff208067b60 msgr2=0x7ff20806db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:06.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 shutdown_connections 2023-12-19T08:05:06.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.110+0000 7ff20daa5700 1 -- 172.21.15.111:0/4236084334 wait complete. 2023-12-19T08:05:06.120 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:05:06.140 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:05:06.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.290+0000 7fd17e5cc700 1 Processor -- start 2023-12-19T08:05:06.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.290+0000 7fd17e5cc700 1 -- start start 2023-12-19T08:05:06.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17e5cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd178158e20 0x7fd17806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17e5cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd1780ce810 0x7fd1780ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17e5cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17806b730 con 0x7fd1781591f0 2023-12-19T08:05:06.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17e5cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17806ab20 con 0x7fd1780ce810 2023-12-19T08:05:06.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17e5cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17806ae00 con 0x7fd178158e20 2023-12-19T08:05:06.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17cdc9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd1781591f0 0x7fd1780ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49910/0 (socket says 172.21.15.111:49910) 2023-12-19T08:05:06.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17cdc9700 1 -- 172.21.15.111:0/1500356282 learned_addr learned my addr 172.21.15.111:0/1500356282 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:06.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17ddcb700 1 --2- 172.21.15.111:0/1500356282 >> v2:172.21.15.111:3300/0 conn(0x7fd1780ce810 0x7fd1780ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17d5ca700 1 --2- 172.21.15.111:0/1500356282 >> v2:172.21.15.143:3300/0 conn(0x7fd178158e20 0x7fd17806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17ddcb700 1 -- 172.21.15.111:0/1500356282 >> v2:172.21.15.143:3300/0 conn(0x7fd178158e20 msgr2=0x7fd17806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:06.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17ddcb700 1 --2- 172.21.15.111:0/1500356282 >> v2:172.21.15.143:3300/0 conn(0x7fd178158e20 0x7fd17806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17ddcb700 1 -- 172.21.15.111:0/1500356282 >> v1:172.21.15.111:6789/0 conn(0x7fd1781591f0 legacy=0x7fd1780ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.294+0000 7fd17ddcb700 1 -- 172.21.15.111:0/1500356282 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17806b0e0 con 0x7fd1780ce810 2023-12-19T08:05:06.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17ddcb700 1 --2- 172.21.15.111:0/1500356282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd1780ca950 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd17806e510 tx=0x7fd170009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae91fbfd2a32d3a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd16ffff700 1 -- 172.21.15.111:0/1500356282 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000e070 con 0x7fd1780ce810 2023-12-19T08:05:06.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd16ffff700 1 -- 172.21.15.111:0/1500356282 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000bb30 con 0x7fd1780ce810 2023-12-19T08:05:06.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd16ffff700 1 -- 172.21.15.111:0/1500356282 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170012430 con 0x7fd1780ce810 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 -- 172.21.15.111:0/1500356282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 msgr2=0x7fd1780ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/1500356282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd1780ca950 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fd17806e510 tx=0x7fd170009780 comp rx=0 tx=0).stop 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 -- 172.21.15.111:0/1500356282 shutdown_connections 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/1500356282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd1780ca950 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/1500356282 >> v2:172.21.15.143:3300/0 conn(0x7fd178158e20 0x7fd17806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 -- 172.21.15.111:0/1500356282 >> 172.21.15.111:0/1500356282 conn(0x7fd178067bb0 msgr2=0x7fd1780cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:06.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 -- 172.21.15.111:0/1500356282 shutdown_connections 2023-12-19T08:05:06.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.298+0000 7fd17e5cc700 1 -- 172.21.15.111:0/1500356282 wait complete. 2023-12-19T08:05:06.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 Processor -- start 2023-12-19T08:05:06.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 -- start start 2023-12-19T08:05:06.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 0x7fd1780c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1781591f0 0x7fd178154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd178156310 con 0x7fd1780ce810 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17815ac70 con 0x7fd178158e20 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17e5cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17815af50 con 0x7fd1781591f0 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17d5ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17d5ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49660/0 (socket says 172.21.15.111:49660) 2023-12-19T08:05:06.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17d5ca700 1 -- 172.21.15.111:0/2066665855 learned_addr learned my addr 172.21.15.111:0/2066665855 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:06.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.302+0000 7fd17cdc9700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 0x7fd1780c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17ddcb700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1781591f0 0x7fd178154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 -- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1781591f0 msgr2=0x7fd178154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1781591f0 0x7fd178154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 -- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 msgr2=0x7fd1780c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 0x7fd1780c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 -- 172.21.15.111:0/2066665855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170009a20 con 0x7fd1780ce810 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17cdc9700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 0x7fd1780c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17d5ca700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd17400d890 tx=0x7fd174008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400b600 con 0x7fd1780ce810 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17400b760 con 0x7fd1780ce810 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17815b230 con 0x7fd1780ce810 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400bae0 con 0x7fd1780ce810 2023-12-19T08:05:06.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1780c9e40 con 0x7fd1780ce810 2023-12-19T08:05:06.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17804be40 con 0x7fd1780ce810 2023-12-19T08:05:06.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.306+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd17401e460 con 0x7fd1780ce810 2023-12-19T08:05:06.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.310+0000 7fd16d7fa700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 0x7fd15c062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.310+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fd17407cfe0 con 0x7fd1780ce810 2023-12-19T08:05:06.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.310+0000 7fd17cdc9700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 0x7fd15c062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.310+0000 7fd17cdc9700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 0x7fd15c062bf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd168007fd0 tx=0x7fd168009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.314+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd17404d510 con 0x7fd1780ce810 2023-12-19T08:05:06.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.558+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd17815b6b0 con 0x7fd1780ce810 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.558+0000 7fd16d7fa700 1 -- 172.21.15.111:0/2066665855 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd174050b30 con 0x7fd1780ce810 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:05:06.561 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:05:06.562 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:05:06.563 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.565 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.566 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.567 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.568 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.569 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.571 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:06.572 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:05:06.573 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 msgr2=0x7fd15c062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 0x7fd15c062bf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd168007fd0 tx=0x7fd168009040 comp rx=0 tx=0).stop 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 msgr2=0x7fd17815a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd17400d890 tx=0x7fd174008040 comp rx=0 tx=0).stop 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 shutdown_connections 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd15c060740 0x7fd15c062bf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd178158e20 0x7fd1780c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1781591f0 0x7fd178154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 --2- 172.21.15.111:0/2066665855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1780ce810 0x7fd17815a680 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 >> 172.21.15.111:0/2066665855 conn(0x7fd178067bb0 msgr2=0x7fd1780cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:06.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 shutdown_connections 2023-12-19T08:05:06.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.562+0000 7fd17e5cc700 1 -- 172.21.15.111:0/2066665855 wait complete. 2023-12-19T08:05:06.583 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:05:06.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:05:06.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:05:06.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.734+0000 7fde0f59e700 1 Processor -- start 2023-12-19T08:05:06.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.734+0000 7fde0f59e700 1 -- start start 2023-12-19T08:05:06.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1015c120 0x7fde100b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde100b0340 0x7fde1006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde100af5a0 con 0x7fde1015c4f0 2023-12-19T08:05:06.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde100b2280 con 0x7fde100b0340 2023-12-19T08:05:06.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1006cf40 con 0x7fde1015c120 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde100b0340 0x7fde1006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde100b0340 0x7fde1006d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49670/0 (socket says 172.21.15.111:49670) 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 -- 172.21.15.111:0/3723130658 learned_addr learned my addr 172.21.15.111:0/3723130658 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0e59c700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1015c120 0x7fde100b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 -- 172.21.15.111:0/3723130658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1015c120 msgr2=0x7fde100b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1015c120 0x7fde100b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 -- 172.21.15.111:0/3723130658 >> v1:172.21.15.143:6789/0 conn(0x7fde1015c4f0 legacy=0x7fde100afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.738+0000 7fde0ed9d700 1 -- 172.21.15.111:0/3723130658 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100af2c0 con 0x7fde100b0340 2023-12-19T08:05:06.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0ed9d700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde100b0340 0x7fde1006d3d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fde08009f20 tx=0x7fde08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bc9e22e5756937 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0d59a700 1 -- 172.21.15.111:0/3723130658 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800e070 con 0x7fde100b0340 2023-12-19T08:05:06.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0d59a700 1 -- 172.21.15.111:0/3723130658 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0800ba80 con 0x7fde100b0340 2023-12-19T08:05:06.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0d59a700 1 -- 172.21.15.111:0/3723130658 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080092e0 con 0x7fde100b0340 2023-12-19T08:05:06.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- 172.21.15.111:0/3723130658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde100b0340 msgr2=0x7fde1006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde100b0340 0x7fde1006d3d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fde08009f20 tx=0x7fde08009840 comp rx=0 tx=0).stop 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- 172.21.15.111:0/3723130658 shutdown_connections 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde100b0340 0x7fde1006d3d0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 --2- 172.21.15.111:0/3723130658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1015c120 0x7fde100b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- 172.21.15.111:0/3723130658 >> 172.21.15.111:0/3723130658 conn(0x7fde10067bb0 msgr2=0x7fde100add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- 172.21.15.111:0/3723130658 shutdown_connections 2023-12-19T08:05:06.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- 172.21.15.111:0/3723130658 wait complete. 2023-12-19T08:05:06.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 Processor -- start 2023-12-19T08:05:06.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.742+0000 7fde0f59e700 1 -- start start 2023-12-19T08:05:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde100b0340 0x7fde1014c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1015c120 0x7fde10150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100aa0c0 con 0x7fde1015c4f0 2023-12-19T08:05:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde101538e0 con 0x7fde1015c120 2023-12-19T08:05:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde10153600 con 0x7fde100b0340 2023-12-19T08:05:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49684/0 (socket says 172.21.15.111:49684) 2023-12-19T08:05:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 -- 172.21.15.111:0/2315451734 learned_addr learned my addr 172.21.15.111:0/2315451734 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0e59c700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde100b0340 0x7fde1014c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0dd9b700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1015c120 0x7fde10150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 -- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde100b0340 msgr2=0x7fde1014c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde100b0340 0x7fde1014c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 -- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1015c120 msgr2=0x7fde10150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1015c120 0x7fde10150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 -- 172.21.15.111:0/2315451734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08009ae0 con 0x7fde1015c4f0 2023-12-19T08:05:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0ed9d700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b6c0 tx=0x7fde08018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08008e00 con 0x7fde1015c4f0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde100a9a70 con 0x7fde1015c4f0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde08025460 con 0x7fde1015c4f0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08025820 con 0x7fde1015c4f0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.746+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1014ccc0 con 0x7fde1015c4f0 2023-12-19T08:05:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.750+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde100aa270 con 0x7fde1015c4f0 2023-12-19T08:05:06.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.750+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fde08003040 con 0x7fde1015c4f0 2023-12-19T08:05:06.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.750+0000 7fddfe7fc700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 0x7fddf0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:06.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.750+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fde0807e780 con 0x7fde1015c4f0 2023-12-19T08:05:06.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.750+0000 7fde0e59c700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 0x7fddf0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:06.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.754+0000 7fde0e59c700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 0x7fddf0062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde100ab600 tx=0x7fde00008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:06.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.754+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde100aa270 con 0x7fde1015c4f0 2023-12-19T08:05:06.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.922+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde1004be40 con 0x7fde1015c4f0 2023-12-19T08:05:06.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.926+0000 7fddfe7fc700 1 -- 172.21.15.111:0/2315451734 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+13186 (secure 0 0 0) 0x7fde0804eb60 con 0x7fde1015c4f0 2023-12-19T08:05:06.929 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:06.929 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":125,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:05:06.102697+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 msgr2=0x7fddf0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 0x7fddf0062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde100ab600 tx=0x7fde00008040 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 msgr2=0x7fde10150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fde0800b6c0 tx=0x7fde08018000 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 shutdown_connections 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fddf0060670 0x7fddf0062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1015c120 0x7fde10150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde1015c4f0 0x7fde10150ad0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 --2- 172.21.15.111:0/2315451734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde100b0340 0x7fde1014c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 >> 172.21.15.111:0/2315451734 conn(0x7fde10067bb0 msgr2=0x7fde1006ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 shutdown_connections 2023-12-19T08:05:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:06.930+0000 7fde0f59e700 1 -- 172.21.15.111:0/2315451734 wait complete. 2023-12-19T08:05:06.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:05:06.952 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:05:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 Processor -- start 2023-12-19T08:05:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 -- start start 2023-12-19T08:05:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd89406aeb0 0x7fd894159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8941595a0 0x7fd8941599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89406c850 con 0x7fd894158c90 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8940cb650 con 0x7fd8941595a0 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89b40c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89406b470 con 0x7fd89406aeb0 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89a40a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd894158c90 0x7fd8940caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49916/0 (socket says 172.21.15.111:49916) 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89ac0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8941595a0 0x7fd8941599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd89a40a700 1 -- 172.21.15.111:0/1756528516 learned_addr learned my addr 172.21.15.111:0/1756528516 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.118+0000 7fd899c09700 1 --2- 172.21.15.111:0/1756528516 >> v2:172.21.15.143:3300/0 conn(0x7fd89406aeb0 0x7fd894159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89ac0b700 1 -- 172.21.15.111:0/1756528516 >> v2:172.21.15.143:3300/0 conn(0x7fd89406aeb0 msgr2=0x7fd894159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89ac0b700 1 --2- 172.21.15.111:0/1756528516 >> v2:172.21.15.143:3300/0 conn(0x7fd89406aeb0 0x7fd894159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89ac0b700 1 -- 172.21.15.111:0/1756528516 >> v1:172.21.15.111:6789/0 conn(0x7fd894158c90 legacy=0x7fd8940caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89ac0b700 1 -- 172.21.15.111:0/1756528516 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89406b750 con 0x7fd8941595a0 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89ac0b700 1 --2- 172.21.15.111:0/1756528516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8941595a0 0x7fd8941599c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd894068000 tx=0x7fd88c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daf92c68cdcb30a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd899408700 1 -- 172.21.15.111:0/1756528516 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00e070 con 0x7fd8941595a0 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd899408700 1 -- 172.21.15.111:0/1756528516 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00bc30 con 0x7fd8941595a0 2023-12-19T08:05:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd899408700 1 -- 172.21.15.111:0/1756528516 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c012430 con 0x7fd8941595a0 2023-12-19T08:05:07.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 -- 172.21.15.111:0/1756528516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8941595a0 msgr2=0x7fd8941599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 --2- 172.21.15.111:0/1756528516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8941595a0 0x7fd8941599c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd894068000 tx=0x7fd88c009780 comp rx=0 tx=0).stop 2023-12-19T08:05:07.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 -- 172.21.15.111:0/1756528516 shutdown_connections 2023-12-19T08:05:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 --2- 172.21.15.111:0/1756528516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8941595a0 0x7fd8941599c0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 --2- 172.21.15.111:0/1756528516 >> v2:172.21.15.143:3300/0 conn(0x7fd89406aeb0 0x7fd894159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 -- 172.21.15.111:0/1756528516 >> 172.21.15.111:0/1756528516 conn(0x7fd894067bb0 msgr2=0x7fd8940cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 -- 172.21.15.111:0/1756528516 shutdown_connections 2023-12-19T08:05:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.122+0000 7fd89b40c700 1 -- 172.21.15.111:0/1756528516 wait complete. 2023-12-19T08:05:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 Processor -- start 2023-12-19T08:05:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 -- start start 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 0x7fd8941487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8941595a0 0x7fd8940c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894154f40 con 0x7fd894158c90 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c64a0 con 0x7fd89406aeb0 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89b40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940c6780 con 0x7fd8941595a0 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49706/0 (socket says 172.21.15.111:49706) 2023-12-19T08:05:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 -- 172.21.15.111:0/3122600974 learned_addr learned my addr 172.21.15.111:0/3122600974 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89a40a700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 0x7fd8941487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89ac0b700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8941595a0 0x7fd8940c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 -- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8941595a0 msgr2=0x7fd8940c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8941595a0 0x7fd8940c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 -- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 msgr2=0x7fd8941487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 0x7fd8941487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd899c09700 1 -- 172.21.15.111:0/3122600974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c009a20 con 0x7fd894158c90 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.126+0000 7fd89a40a700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 0x7fd8941487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd899c09700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd88400d890 tx=0x7fd884008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8840125f0 con 0x7fd894158c90 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884012750 con 0x7fd894158c90 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8941548f0 con 0x7fd894158c90 2023-12-19T08:05:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884012af0 con 0x7fd894158c90 2023-12-19T08:05:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd894148d10 con 0x7fd894158c90 2023-12-19T08:05:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.130+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8941550f0 con 0x7fd894158c90 2023-12-19T08:05:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.134+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd88401e460 con 0x7fd894158c90 2023-12-19T08:05:07.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.134+0000 7fd88a7fc700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 0x7fd878062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.134+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fd88407cec0 con 0x7fd894158c90 2023-12-19T08:05:07.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.134+0000 7fd89a40a700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 0x7fd878062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.134+0000 7fd89a40a700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 0x7fd878062b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8940ce300 tx=0x7fd890009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.142+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8941550f0 con 0x7fd894158c90 2023-12-19T08:05:07.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.306+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd89404be40 con 0x7fd894158c90 2023-12-19T08:05:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.310+0000 7fd88a7fc700 1 -- 172.21.15.111:0/3122600974 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v125) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd88404d350 con 0x7fd894158c90 2023-12-19T08:05:07.311 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 msgr2=0x7fd878062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 0x7fd878062b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8940ce300 tx=0x7fd890009040 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 msgr2=0x7fd8940c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd88400d890 tx=0x7fd884008040 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 shutdown_connections 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd8780606a0 0x7fd878062b50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8941595a0 0x7fd8940c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd894158c90 0x7fd8940c5ba0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 --2- 172.21.15.111:0/3122600974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd89406aeb0 0x7fd8941487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 >> 172.21.15.111:0/3122600974 conn(0x7fd894067bb0 msgr2=0x7fd8940b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 shutdown_connections 2023-12-19T08:05:07.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.314+0000 7fd89b40c700 1 -- 172.21.15.111:0/3122600974 wait complete. 2023-12-19T08:05:07.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:05:07.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.486+0000 7fe1348cc700 1 Processor -- start 2023-12-19T08:05:07.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.486+0000 7fe1348cc700 1 -- start start 2023-12-19T08:05:07.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe1348cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe13015bd00 0x7fe13015c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe1348cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13006b2c0 con 0x7fe1300cad10 2023-12-19T08:05:07.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe1348cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13006e0f0 con 0x7fe1300cb0e0 2023-12-19T08:05:07.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe1348cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe13006a6b0 con 0x7fe13015bd00 2023-12-19T08:05:07.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe1300cad10 0x7fe13006b750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49922/0 (socket says 172.21.15.111:49922) 2023-12-19T08:05:07.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12ed9d700 1 -- 172.21.15.111:0/2924274113 learned_addr learned my addr 172.21.15.111:0/2924274113 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:07.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12f59e700 1 --2- 172.21.15.111:0/2924274113 >> v2:172.21.15.111:3300/0 conn(0x7fe13015bd00 0x7fe13015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837686302 0 0) 0x7fe13006b2c0 con 0x7fe1300cad10 2023-12-19T08:05:07.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c003570 con 0x7fe1300cad10 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933556634 0 0) 0x7fe13006e0f0 con 0x7fe1300cb0e0 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13006b2c0 con 0x7fe1300cb0e0 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12f59e700 1 -- 172.21.15.111:0/2924274113 >> v1:172.21.15.143:6789/0 conn(0x7fe1300cb0e0 legacy=0x7fe13006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12f59e700 1 -- 172.21.15.111:0/2924274113 >> v1:172.21.15.111:6789/0 conn(0x7fe1300cad10 legacy=0x7fe13006b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12f59e700 1 -- 172.21.15.111:0/2924274113 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13006a990 con 0x7fe13015bd00 2023-12-19T08:05:07.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12f59e700 1 --2- 172.21.15.111:0/2924274113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe13015c150 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe128009e60 tx=0x7fe128009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd63ffa78af20ec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800e070 con 0x7fe13015bd00 2023-12-19T08:05:07.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.490+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12800bab0 con 0x7fe13015bd00 2023-12-19T08:05:07.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe12e59c700 1 -- 172.21.15.111:0/2924274113 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800be70 con 0x7fe13015bd00 2023-12-19T08:05:07.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 -- 172.21.15.111:0/2924274113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 msgr2=0x7fe13015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2924274113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe13015c150 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fe128009e60 tx=0x7fe128009780 comp rx=0 tx=0).stop 2023-12-19T08:05:07.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 -- 172.21.15.111:0/2924274113 shutdown_connections 2023-12-19T08:05:07.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2924274113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe13015c150 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 -- 172.21.15.111:0/2924274113 >> 172.21.15.111:0/2924274113 conn(0x7fe130067bb0 msgr2=0x7fe1300cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:07.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 -- 172.21.15.111:0/2924274113 shutdown_connections 2023-12-19T08:05:07.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.494+0000 7fe1348cc700 1 -- 172.21.15.111:0/2924274113 wait complete. 2023-12-19T08:05:07.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 Processor -- start 2023-12-19T08:05:07.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 -- start start 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300cad10 0x7fe13015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1300cb0e0 0x7fe130158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130169e80 con 0x7fe13015bd00 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301595c0 con 0x7fe1300cb0e0 2023-12-19T08:05:07.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe1348cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301598a0 con 0x7fe1300cad10 2023-12-19T08:05:07.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49726/0 (socket says 172.21.15.111:49726) 2023-12-19T08:05:07.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 -- 172.21.15.111:0/2160111231 learned_addr learned my addr 172.21.15.111:0/2160111231 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:07.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12ed9d700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300cad10 0x7fe13015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe127fff700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1300cb0e0 0x7fe130158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 -- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300cad10 msgr2=0x7fe13015a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300cad10 0x7fe13015a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 -- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1300cb0e0 msgr2=0x7fe130158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1300cb0e0 0x7fe130158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 -- 172.21.15.111:0/2160111231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128009a20 con 0x7fe13015bd00 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.498+0000 7fe12f59e700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe128009460 tx=0x7fe128019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128012dd0 con 0x7fe13015bd00 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe128026480 con 0x7fe13015bd00 2023-12-19T08:05:07.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128026840 con 0x7fe13015bd00 2023-12-19T08:05:07.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe130159b80 con 0x7fe13015bd00 2023-12-19T08:05:07.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13015a140 con 0x7fe13015bd00 2023-12-19T08:05:07.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe130070470 con 0x7fe13015bd00 2023-12-19T08:05:07.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.502+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe12801c400 con 0x7fe13015bd00 2023-12-19T08:05:07.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.506+0000 7fe126ffd700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 0x7fe110062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.506+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5693+0+0 (secure 0 0 0) 0x7fe12807ed90 con 0x7fe13015bd00 2023-12-19T08:05:07.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.506+0000 7fe12ed9d700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 0x7fe110062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.506+0000 7fe12ed9d700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 0x7fe110062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe120007fd0 tx=0x7fe120009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.510+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe12804f0d0 con 0x7fe13015bd00 2023-12-19T08:05:07.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe13015a360 con 0x7fe13015bd00 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe126ffd700 1 -- 172.21.15.111:0/2160111231 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v125) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe1280526f0 con 0x7fe13015bd00 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:07.681 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 msgr2=0x7fe110062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 0x7fe110062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe120007fd0 tx=0x7fe120009040 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 msgr2=0x7fe130169940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.678+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe128009460 tx=0x7fe128019000 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 shutdown_connections 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe110060670 0x7fe110062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1300cb0e0 0x7fe130158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe13015bd00 0x7fe130169940 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 --2- 172.21.15.111:0/2160111231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300cad10 0x7fe13015a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 >> 172.21.15.111:0/2160111231 conn(0x7fe130067bb0 msgr2=0x7fe1300b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 shutdown_connections 2023-12-19T08:05:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.682+0000 7fe1348cc700 1 -- 172.21.15.111:0/2160111231 wait complete. 2023-12-19T08:05:07.705 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:05:07.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:05:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.870+0000 7f483228e700 1 Processor -- start 2023-12-19T08:05:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.870+0000 7f483228e700 1 -- start start 2023-12-19T08:05:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f483228e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 0x7f482c06aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f483228e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c06b420 con 0x7f482c06ed80 2023-12-19T08:05:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f483228e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c15d720 con 0x7f482c158da0 2023-12-19T08:05:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f483228e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0ca290 con 0x7f482c06df10 2023-12-19T08:05:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4830a8b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f482c06ed80 0x7f482c06a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49934/0 (socket says 172.21.15.111:49934) 2023-12-19T08:05:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4830a8b700 1 -- 172.21.15.111:0/3074017465 learned_addr learned my addr 172.21.15.111:0/3074017465 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4831a8d700 1 --2- 172.21.15.111:0/3074017465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 0x7f482c06aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723368804 0 0) 0x7f482c15d720 con 0x7f482c158da0 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4818003570 con 0x7f482c158da0 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278811735 0 0) 0x7f482c06b420 con 0x7f482c06ed80 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4831a8d700 1 -- 172.21.15.111:0/3074017465 >> v1:172.21.15.143:6789/0 conn(0x7f482c158da0 legacy=0x7f482c06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4831a8d700 1 -- 172.21.15.111:0/3074017465 >> v1:172.21.15.111:6789/0 conn(0x7f482c06ed80 legacy=0x7f482c06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4831a8d700 1 -- 172.21.15.111:0/3074017465 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c0ca4a0 con 0x7f482c06df10 2023-12-19T08:05:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4831a8d700 1 --2- 172.21.15.111:0/3074017465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 0x7f482c06aee0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e90 tx=0x7f4824009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4681bd2fc5514d26 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482400e070 con 0x7f482c06df10 2023-12-19T08:05:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482400bc30 con 0x7f482c06df10 2023-12-19T08:05:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.874+0000 7f4823fff700 1 -- 172.21.15.111:0/3074017465 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824012430 con 0x7f482c06df10 2023-12-19T08:05:07.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- 172.21.15.111:0/3074017465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 msgr2=0x7f482c06aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 --2- 172.21.15.111:0/3074017465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 0x7f482c06aee0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e90 tx=0x7f4824009780 comp rx=0 tx=0).stop 2023-12-19T08:05:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- 172.21.15.111:0/3074017465 shutdown_connections 2023-12-19T08:05:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 --2- 172.21.15.111:0/3074017465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c06df10 0x7f482c06aee0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- 172.21.15.111:0/3074017465 >> 172.21.15.111:0/3074017465 conn(0x7f482c0cc760 msgr2=0x7f482c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- 172.21.15.111:0/3074017465 shutdown_connections 2023-12-19T08:05:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- 172.21.15.111:0/3074017465 wait complete. 2023-12-19T08:05:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 Processor -- start 2023-12-19T08:05:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.878+0000 7f483228e700 1 -- start start 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c158da0 0x7f482c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0c7c00 con 0x7f482c06ed80 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0b2aa0 con 0x7f482c06df10 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483228e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0c7e40 con 0x7f482c158da0 2023-12-19T08:05:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483128c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49734/0 (socket says 172.21.15.111:49734) 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483128c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:37146/0 (socket says 172.21.15.111:37146) 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 -- 172.21.15.111:0/1555572689 learned_addr learned my addr 172.21.15.111:0/1555572689 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4831a8d700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c158da0 0x7f482c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 -- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c158da0 msgr2=0x7f482c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c158da0 0x7f482c15b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 -- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 msgr2=0x7f482c0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 -- 172.21.15.111:0/1555572689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824009a20 con 0x7f482c06ed80 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f483128c700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f4830a8b700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c810 tx=0x7f481c007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.882+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c005b30 con 0x7f482c06ed80 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c005c90 con 0x7f482c06ed80 2023-12-19T08:05:07.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c007320 con 0x7f482c06ed80 2023-12-19T08:05:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c0c8120 con 0x7f482c06ed80 2023-12-19T08:05:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c0c86e0 con 0x7f482c06ed80 2023-12-19T08:05:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c04be40 con 0x7f482c06ed80 2023-12-19T08:05:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.886+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f481c01b3b0 con 0x7f482c06ed80 2023-12-19T08:05:07.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.890+0000 7f48217fa700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 0x7f4810062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:07.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.890+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f481c07be70 con 0x7f482c06ed80 2023-12-19T08:05:07.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.890+0000 7f483128c700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 0x7f4810062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:07.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.890+0000 7f483128c700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 0x7f4810062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f482c0b1a50 tx=0x7f4828007d50 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:07.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:07.894+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f481c04c360 con 0x7f482c06ed80 2023-12-19T08:05:08.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.066+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f482c0b2be0 con 0x7f482c06ed80 2023-12-19T08:05:08.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.714+0000 7f48217fa700 1 -- 172.21.15.111:0/1555572689 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v126) v1 ==== 138+0+0 (secure 0 0 0) 0x7f481c0170b0 con 0x7f482c06ed80 2023-12-19T08:05:08.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.718+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 msgr2=0x7f4810062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.718+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 0x7f4810062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f482c0b1a50 tx=0x7f4828007d50 comp rx=0 tx=0).stop 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.718+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 msgr2=0x7f482c0c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.718+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c810 tx=0x7f481c007e60 comp rx=0 tx=0).stop 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 shutdown_connections 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4810060670 0x7f4810062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f482c158da0 0x7f482c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482c06ed80 0x7f482c0c76c0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 --2- 172.21.15.111:0/1555572689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f482c06df10 0x7f482c0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 >> 172.21.15.111:0/1555572689 conn(0x7f482c0cc760 msgr2=0x7f482c0ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 shutdown_connections 2023-12-19T08:05:08.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.722+0000 7f483228e700 1 -- 172.21.15.111:0/1555572689 wait complete. 2023-12-19T08:05:08.728 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:05:08.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:05:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 Processor -- start 2023-12-19T08:05:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 -- start start 2023-12-19T08:05:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fe00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00c7450 con 0x7f1fe00cab10 2023-12-19T08:05:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006bea0 con 0x7f1fe015d410 2023-12-19T08:05:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.914+0000 7f1fe6878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00c6e00 con 0x7f1fe006b8c0 2023-12-19T08:05:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fd77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1fe00cab10 0x7f1fe006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49936/0 (socket says 172.21.15.111:49936) 2023-12-19T08:05:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fd77fe700 1 -- 172.21.15.111:0/1209182695 learned_addr learned my addr 172.21.15.111:0/1209182695 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fe4e15700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fe00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680250624 0 0) 0x7f1fe00c7450 con 0x7f1fe00cab10 2023-12-19T08:05:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc003570 con 0x7f1fe00cab10 2023-12-19T08:05:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709643547 0 0) 0x7f1fe006bea0 con 0x7f1fe015d410 2023-12-19T08:05:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe00c7450 con 0x7f1fe015d410 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697007545 0 0) 0x7f1fcc003570 con 0x7f1fe00cab10 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 msgr2=0x7f1fe00ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fe00ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 >> v1:172.21.15.143:6789/0 conn(0x7f1fe015d410 legacy=0x7f1fe00ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00c70e0 con 0x7f1fe00cab10 2023-12-19T08:05:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f1fc8001470 con 0x7f1fe00cab10 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 >> v1:172.21.15.111:6789/0 conn(0x7f1fe00cab10 legacy=0x7f1fe006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fcc004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fcc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fcc004d50 0x7f1fcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00c6e00 con 0x7f1fcc003c20 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc00bb70 con 0x7f1fcc004d50 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc00bd30 con 0x7f1fe006b8c0 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fcc004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fd77fe700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fcc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fe4e15700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fcc004d50 0x7f1fcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 -- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 msgr2=0x7f1fcc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe006b8c0 0x7f1fcc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 -- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fcc004d50 msgr2=0x7f1fcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fcc004d50 0x7f1fcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 -- 172.21.15.111:0/1209182695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc003570 con 0x7f1fcc003c20 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fe4e15700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fcc004d50 0x7f1fcc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.918+0000 7f1fdffff700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fcc004010 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0002120 tx=0x7f1fd00017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1a327793697f4f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0001ab0 con 0x7f1fcc003c20 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd000c980 con 0x7f1fcc003c20 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fdf7fe700 1 -- 172.21.15.111:0/1209182695 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd000cd20 con 0x7f1fcc003c20 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- 172.21.15.111:0/1209182695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 msgr2=0x7f1fcc004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 --2- 172.21.15.111:0/1209182695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fcc004010 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0002120 tx=0x7f1fd00017a0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe4e15700 1 -- 172.21.15.111:0/1209182695 reap_dead start 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- 172.21.15.111:0/1209182695 shutdown_connections 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- 172.21.15.111:0/1209182695 >> 172.21.15.111:0/1209182695 conn(0x7f1fe0067bb0 msgr2=0x7f1fe00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- 172.21.15.111:0/1209182695 shutdown_connections 2023-12-19T08:05:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- 172.21.15.111:0/1209182695 wait complete. 2023-12-19T08:05:08.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 Processor -- start 2023-12-19T08:05:08.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.922+0000 7f1fe6878700 1 -- start start 2023-12-19T08:05:08.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c3660 0x7f1fe00b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 0x7f1fe014eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00af250 con 0x7f1fcc003c20 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b1220 con 0x7f1fe00c3660 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00aec40 con 0x7f1fe015da70 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49764/0 (socket says 172.21.15.111:49764) 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 -- 172.21.15.111:0/675387186 learned_addr learned my addr 172.21.15.111:0/675387186 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fd77fe700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c3660 0x7f1fe00b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe4e15700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 0x7f1fe014eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 -- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 msgr2=0x7f1fe014eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 0x7f1fe014eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 -- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c3660 msgr2=0x7f1fe00b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c3660 0x7f1fe00b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 -- 172.21.15.111:0/675387186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe015f9f0 con 0x7f1fcc003c20 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe4e15700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 0x7f1fe014eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdffff700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c6a0 tx=0x7f1fd0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0001ab0 con 0x7f1fcc003c20 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0006db0 con 0x7f1fcc003c20 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe00aeee0 con 0x7f1fcc003c20 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd001e6b0 con 0x7f1fcc003c20 2023-12-19T08:05:08.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.926+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe015dee0 con 0x7f1fcc003c20 2023-12-19T08:05:08.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.930+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe015e160 con 0x7f1fcc003c20 2023-12-19T08:05:08.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.930+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1fd0003030 con 0x7f1fcc003c20 2023-12-19T08:05:08.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.930+0000 7f1fdcff9700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 0x7f1fc0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:08.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.930+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6063+0+0 (secure 0 0 0) 0x7f1fd007f2e0 con 0x7f1fcc003c20 2023-12-19T08:05:08.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.930+0000 7f1fd77fe700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 0x7f1fc0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:08.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.934+0000 7f1fd77fe700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 0x7f1fc0062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8001200 tx=0x7f1fc8007040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:08.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:08.938+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1fd004f5c0 con 0x7f1fcc003c20 2023-12-19T08:05:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.110+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1fe00af070 con 0x7f1fcc003c20 2023-12-19T08:05:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.726+0000 7f1fdcff9700 1 -- 172.21.15.111:0/675387186 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v127) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1fd0052be0 con 0x7f1fcc003c20 2023-12-19T08:05:09.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 msgr2=0x7f1fc0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 0x7f1fc0062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8001200 tx=0x7f1fc8007040 comp rx=0 tx=0).stop 2023-12-19T08:05:09.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 msgr2=0x7f1fe015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c6a0 tx=0x7f1fd0016000 comp rx=0 tx=0).stop 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 shutdown_connections 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1fc0060670 0x7f1fc0062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c3660 0x7f1fe00b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe015da70 0x7f1fe014eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 --2- 172.21.15.111:0/675387186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc003c20 0x7f1fe015f3c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 >> 172.21.15.111:0/675387186 conn(0x7f1fe0067bb0 msgr2=0x7f1fe0159530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:09.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.738+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 shutdown_connections 2023-12-19T08:05:09.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.742+0000 7f1fe6878700 1 -- 172.21.15.111:0/675387186 wait complete. 2023-12-19T08:05:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:05:09.772 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:05:09.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 Processor -- start 2023-12-19T08:05:09.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 -- start start 2023-12-19T08:05:09.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d5806baa0 0x7f8d5815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d580adab0 0x7f8d580adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5815ae30 con 0x7f8d58154780 2023-12-19T08:05:09.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d580abf70 con 0x7f8d5806baa0 2023-12-19T08:05:09.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.926+0000 7f8d5e40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58154b50 con 0x7f8d580adab0 2023-12-19T08:05:09.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d5806baa0 0x7f8d5815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d5806baa0 0x7f8d5815b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49772/0 (socket says 172.21.15.111:49772) 2023-12-19T08:05:09.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 -- 172.21.15.111:0/3857135232 learned_addr learned my addr 172.21.15.111:0/3857135232 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:09.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5d40a700 1 --1- 172.21.15.111:0/3857135232 >> v1:172.21.15.111:6789/0 conn(0x7f8d58154780 0x7f8d58157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49938/0 (socket says 172.21.15.111:49938) 2023-12-19T08:05:09.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5dc0b700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d580adab0 0x7f8d580adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 -- 172.21.15.111:0/3857135232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d580adab0 msgr2=0x7f8d580adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d580adab0 0x7f8d580adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 -- 172.21.15.111:0/3857135232 >> v1:172.21.15.111:6789/0 conn(0x7f8d58154780 legacy=0x7f8d58157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d4ffff700 1 -- 172.21.15.111:0/3857135232 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730803086 0 0) 0x7f8d5815ae30 con 0x7f8d58154780 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 -- 172.21.15.111:0/3857135232 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580ac210 con 0x7f8d5806baa0 2023-12-19T08:05:09.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5cc09700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d5806baa0 0x7f8d5815b2c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f8d480099b0 tx=0x7f8d48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2084daf9632aeb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:09.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d4ffff700 1 -- 172.21.15.111:0/3857135232 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48009ea0 con 0x7f8d5806baa0 2023-12-19T08:05:09.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d4ffff700 1 -- 172.21.15.111:0/3857135232 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4800b860 con 0x7f8d5806baa0 2023-12-19T08:05:09.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d4ffff700 1 -- 172.21.15.111:0/3857135232 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4800bc00 con 0x7f8d5806baa0 2023-12-19T08:05:09.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5e40c700 1 -- 172.21.15.111:0/3857135232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d5806baa0 msgr2=0x7f8d5815b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.930+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d5806baa0 0x7f8d5815b2c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f8d480099b0 tx=0x7f8d48009250 comp rx=0 tx=0).stop 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 -- 172.21.15.111:0/3857135232 shutdown_connections 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d580adab0 0x7f8d580adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/3857135232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d5806baa0 0x7f8d5815b2c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 -- 172.21.15.111:0/3857135232 >> 172.21.15.111:0/3857135232 conn(0x7f8d58067b50 msgr2=0x7f8d580cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 -- 172.21.15.111:0/3857135232 shutdown_connections 2023-12-19T08:05:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 -- 172.21.15.111:0/3857135232 wait complete. 2023-12-19T08:05:09.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 Processor -- start 2023-12-19T08:05:09.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.934+0000 7f8d5e40c700 1 -- start start 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d5806baa0 0x7f8d580c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 0x7f8d581584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580c9ed0 con 0x7f8d58154780 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580ca1b0 con 0x7f8d580adab0 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580ca490 con 0x7f8d5806baa0 2023-12-19T08:05:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49784/0 (socket says 172.21.15.111:49784) 2023-12-19T08:05:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 -- 172.21.15.111:0/888018188 learned_addr learned my addr 172.21.15.111:0/888018188 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5d40a700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d5806baa0 0x7f8d580c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5cc09700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 0x7f8d581584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 -- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d5806baa0 msgr2=0x7f8d580c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d5806baa0 0x7f8d580c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 -- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 msgr2=0x7f8d581584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 0x7f8d581584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 -- 172.21.15.111:0/888018188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480094f0 con 0x7f8d58154780 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5cc09700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 0x7f8d581584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5dc0b700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009df0 tx=0x7f8d5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000cb50 con 0x7f8d58154780 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000f040 con 0x7f8d58154780 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50014480 con 0x7f8d58154780 2023-12-19T08:05:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d580c6100 con 0x7f8d58154780 2023-12-19T08:05:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.938+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5815a4b0 con 0x7f8d58154780 2023-12-19T08:05:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.942+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d580c5f60 con 0x7f8d58154780 2023-12-19T08:05:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.942+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8d50014830 con 0x7f8d58154780 2023-12-19T08:05:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.946+0000 7f8d4d7fa700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 0x7f8d3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.946+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f8d5007dc50 con 0x7f8d58154780 2023-12-19T08:05:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.946+0000 7f8d5d40a700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 0x7f8d3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.946+0000 7f8d5d40a700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 0x7f8d3c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d54007060 tx=0x7f8d54007ef0 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:09.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:09.950+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d5004dd20 con 0x7f8d58154780 2023-12-19T08:05:10.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.162+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8d580c5f60 con 0x7f8d58154780 2023-12-19T08:05:10.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.762+0000 7f8d4d7fa700 1 -- 172.21.15.111:0/888018188 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8d580c5f60 con 0x7f8d58154780 2023-12-19T08:05:10.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.774+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 msgr2=0x7f8d3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.774+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 0x7f8d3c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d54007060 tx=0x7f8d54007ef0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.774+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 msgr2=0x7f8d580c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.774+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009df0 tx=0x7f8d5000b040 comp rx=0 tx=0).stop 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 shutdown_connections 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d3c060670 0x7f8d3c062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d580adab0 0x7f8d581584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d58154780 0x7f8d580c4ce0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 --2- 172.21.15.111:0/888018188 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d5806baa0 0x7f8d580c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.778+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 >> 172.21.15.111:0/888018188 conn(0x7f8d58067b50 msgr2=0x7f8d580cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:10.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.782+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 shutdown_connections 2023-12-19T08:05:10.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.782+0000 7f8d5e40c700 1 -- 172.21.15.111:0/888018188 wait complete. 2023-12-19T08:05:10.788 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 15 and data pool 16 2023-12-19T08:05:10.804 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:05:10.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.962+0000 7f929e4a3700 1 Processor -- start 2023-12-19T08:05:10.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.962+0000 7f929e4a3700 1 -- start start 2023-12-19T08:05:10.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929e4a3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92980c6900 0x7f92980c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929e4a3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929806b470 con 0x7f929806b900 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929e4a3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929806e110 con 0x7f929815cc20 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929e4a3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929815d790 con 0x7f92980c6900 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929cca0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f929806b900 0x7f929806da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49942/0 (socket says 172.21.15.111:49942) 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929cca0700 1 -- 172.21.15.111:0/1573633067 learned_addr learned my addr 172.21.15.111:0/1573633067 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:10.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f929d4a1700 1 --2- 172.21.15.111:0/1573633067 >> v2:172.21.15.143:3300/0 conn(0x7f92980c6900 0x7f92980c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:10.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f928ffff700 1 -- 172.21.15.111:0/1573633067 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193465858 0 0) 0x7f929806b470 con 0x7f929806b900 2023-12-19T08:05:10.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.966+0000 7f928ffff700 1 -- 172.21.15.111:0/1573633067 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9284003570 con 0x7f929806b900 2023-12-19T08:05:10.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929d4a1700 1 -- 172.21.15.111:0/1573633067 >> v1:172.21.15.143:6789/0 conn(0x7f929815cc20 legacy=0x7f929815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929d4a1700 1 -- 172.21.15.111:0/1573633067 >> v1:172.21.15.111:6789/0 conn(0x7f929806b900 legacy=0x7f929806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929d4a1700 1 -- 172.21.15.111:0/1573633067 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9298161d10 con 0x7f92980c6900 2023-12-19T08:05:10.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929d4a1700 1 --2- 172.21.15.111:0/1573633067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92980c6900 0x7f92980c6cd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92940098e0 tx=0x7f92940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3d5bf543ceba54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:10.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f928ffff700 1 -- 172.21.15.111:0/1573633067 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929400b640 con 0x7f92980c6900 2023-12-19T08:05:10.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f928ffff700 1 -- 172.21.15.111:0/1573633067 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929400b7a0 con 0x7f92980c6900 2023-12-19T08:05:10.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f928ffff700 1 -- 172.21.15.111:0/1573633067 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929400bb40 con 0x7f92980c6900 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 -- 172.21.15.111:0/1573633067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92980c6900 msgr2=0x7f92980c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 --2- 172.21.15.111:0/1573633067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92980c6900 0x7f92980c6cd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92940098e0 tx=0x7f92940091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 -- 172.21.15.111:0/1573633067 shutdown_connections 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 --2- 172.21.15.111:0/1573633067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92980c6900 0x7f92980c6cd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 -- 172.21.15.111:0/1573633067 >> 172.21.15.111:0/1573633067 conn(0x7f9298067bb0 msgr2=0x7f92980cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 -- 172.21.15.111:0/1573633067 shutdown_connections 2023-12-19T08:05:10.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.970+0000 7f929e4a3700 1 -- 172.21.15.111:0/1573633067 wait complete. 2023-12-19T08:05:10.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 Processor -- start 2023-12-19T08:05:10.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 -- start start 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92980c6900 0x7f929814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 0x7f92980c0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92980b0e60 con 0x7f929806b900 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298159570 con 0x7f929815cc20 2023-12-19T08:05:10.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929e4a3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92980b0250 con 0x7f92980c6900 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49792/0 (socket says 172.21.15.111:49792) 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 -- 172.21.15.111:0/683020181 learned_addr learned my addr 172.21.15.111:0/683020181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929dca2700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 0x7f92980c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929cca0700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92980c6900 0x7f929814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 -- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92980c6900 msgr2=0x7f929814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92980c6900 0x7f929814a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 -- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 msgr2=0x7f92980c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:10.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 0x7f92980c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:10.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929d4a1700 1 -- 172.21.15.111:0/683020181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294009420 con 0x7f929806b900 2023-12-19T08:05:10.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.974+0000 7f929dca2700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 0x7f92980c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:10.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f929d4a1700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9294009f80 tx=0x7f9294017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:10.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294000d10 con 0x7f929806b900 2023-12-19T08:05:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929400bb20 con 0x7f929806b900 2023-12-19T08:05:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9294011650 con 0x7f929806b900 2023-12-19T08:05:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92980b0530 con 0x7f929806b900 2023-12-19T08:05:10.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92980b0af0 con 0x7f929806b900 2023-12-19T08:05:10.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.978+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929804be40 con 0x7f929806b900 2023-12-19T08:05:10.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.982+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9294011a00 con 0x7f929806b900 2023-12-19T08:05:10.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.982+0000 7f928d7fa700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 0x7f927c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:10.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.982+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f929407e3e0 con 0x7f929806b900 2023-12-19T08:05:10.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.982+0000 7f929cca0700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 0x7f927c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:10.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.982+0000 7f929cca0700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 0x7f927c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92980cb430 tx=0x7f9288009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:10.990+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f929404e4d0 con 0x7f929806b900 2023-12-19T08:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.162+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f929806bd70 con 0x7f929806b900 2023-12-19T08:05:11.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.162+0000 7f928d7fa700 1 -- 172.21.15.111:0/683020181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16419 (secure 0 0 0) 0x7f9294011cb0 con 0x7f929806b900 2023-12-19T08:05:11.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:11.168 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":128,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:05:10.165218+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:05:11.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 msgr2=0x7f927c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 0x7f927c062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92980cb430 tx=0x7f9288009040 comp rx=0 tx=0).stop 2023-12-19T08:05:11.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 msgr2=0x7f92980b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9294009f80 tx=0x7f9294017000 comp rx=0 tx=0).stop 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 shutdown_connections 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f927c060670 0x7f927c062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f929815cc20 0x7f92980c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92980c6900 0x7f929814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 --2- 172.21.15.111:0/683020181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f929806b900 0x7f92980b12f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 >> 172.21.15.111:0/683020181 conn(0x7f9298067bb0 msgr2=0x7f929815a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 shutdown_connections 2023-12-19T08:05:11.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.166+0000 7f929e4a3700 1 -- 172.21.15.111:0/683020181 wait complete. 2023-12-19T08:05:11.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:05:11.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.346+0000 7f174b0f0700 1 Processor -- start 2023-12-19T08:05:11.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.346+0000 7f174b0f0700 1 -- start start 2023-12-19T08:05:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174b0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f17440a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174b0f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17440c1d60 0x7f17440680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174b0f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174406dc20 con 0x7f17440ae510 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174b0f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17440c2200 con 0x7f17440c1d60 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174b0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174406d940 con 0x7f17440ae140 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17440c1d60 0x7f17440680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17440c1d60 0x7f17440680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49804/0 (socket says 172.21.15.111:49804) 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 -- 172.21.15.111:0/3168440002 learned_addr learned my addr 172.21.15.111:0/3168440002 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f1748e8c700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f17440a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 -- 172.21.15.111:0/3168440002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 msgr2=0x7f17440a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f17440a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 -- 172.21.15.111:0/3168440002 >> v1:172.21.15.143:6789/0 conn(0x7f17440ae510 legacy=0x7f174406d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 -- 172.21.15.111:0/3168440002 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440c6e20 con 0x7f17440c1d60 2023-12-19T08:05:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f174968d700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440680f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f173c0099b0 tx=0x7f173c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2048635edfc6b93a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.350+0000 7f17437fe700 1 -- 172.21.15.111:0/3168440002 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c009ea0 con 0x7f17440c1d60 2023-12-19T08:05:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f17437fe700 1 -- 172.21.15.111:0/3168440002 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c00b860 con 0x7f17440c1d60 2023-12-19T08:05:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f17437fe700 1 -- 172.21.15.111:0/3168440002 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c00bc00 con 0x7f17440c1d60 2023-12-19T08:05:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- 172.21.15.111:0/3168440002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 msgr2=0x7f17440680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440680f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f173c0099b0 tx=0x7f173c009250 comp rx=0 tx=0).stop 2023-12-19T08:05:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- 172.21.15.111:0/3168440002 shutdown_connections 2023-12-19T08:05:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440680f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3168440002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f17440a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- 172.21.15.111:0/3168440002 >> 172.21.15.111:0/3168440002 conn(0x7f17440c8850 msgr2=0x7f17440c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- 172.21.15.111:0/3168440002 shutdown_connections 2023-12-19T08:05:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- 172.21.15.111:0/3168440002 wait complete. 2023-12-19T08:05:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 Processor -- start 2023-12-19T08:05:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.354+0000 7f174b0f0700 1 -- start start 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f174415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17440ae510 0x7f17440af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17440afaf0 con 0x7f17440c1d60 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174415b900 con 0x7f17440ae510 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174415bbe0 con 0x7f17440ae140 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49820/0 (socket says 172.21.15.111:49820) 2023-12-19T08:05:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 -- 172.21.15.111:0/3382597057 learned_addr learned my addr 172.21.15.111:0/3382597057 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f1743fff700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17440ae510 0x7f17440af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f1748e8c700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f174415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 -- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 msgr2=0x7f174415b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f174415b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 -- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17440ae510 msgr2=0x7f17440af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17440ae510 0x7f17440af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 -- 172.21.15.111:0/3382597057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c0094f0 con 0x7f17440c1d60 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174968d700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f17440c63d0 tx=0x7f173c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c00bba0 con 0x7f17440c1d60 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c01a640 con 0x7f17440c1d60 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174415bec0 con 0x7f17440c1d60 2023-12-19T08:05:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c01a9e0 con 0x7f17440c1d60 2023-12-19T08:05:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.358+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17440c4040 con 0x7f17440c1d60 2023-12-19T08:05:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.362+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17440c4260 con 0x7f17440c1d60 2023-12-19T08:05:11.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.362+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f173c011420 con 0x7f17440c1d60 2023-12-19T08:05:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.362+0000 7f1740ff9700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 0x7f1728062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.362+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f173c07ef10 con 0x7f17440c1d60 2023-12-19T08:05:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.366+0000 7f1748e8c700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 0x7f1728062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.366+0000 7f1748e8c700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 0x7f1728062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1738007600 tx=0x7f173800b040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.370+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f173c04f1a0 con 0x7f17440c1d60 2023-12-19T08:05:11.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.542+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1744071120 con 0x7f17440c1d60 2023-12-19T08:05:11.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.542+0000 7f1740ff9700 1 -- 172.21.15.111:0/3382597057 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16419 (secure 0 0 0) 0x7f173c0527c0 con 0x7f17440c1d60 2023-12-19T08:05:11.545 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:11.545 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":128,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:05:10.165218+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:05:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 msgr2=0x7f1728062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 0x7f1728062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1738007600 tx=0x7f173800b040 comp rx=0 tx=0).stop 2023-12-19T08:05:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 msgr2=0x7f17440acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f17440c63d0 tx=0x7f173c017000 comp rx=0 tx=0).stop 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 shutdown_connections 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1728060670 0x7f1728062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17440ae510 0x7f17440af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17440c1d60 0x7f17440acd80 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 --2- 172.21.15.111:0/3382597057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17440ae140 0x7f174415b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 >> 172.21.15.111:0/3382597057 conn(0x7f17440c8850 msgr2=0x7f1744157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 shutdown_connections 2023-12-19T08:05:11.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.546+0000 7f174b0f0700 1 -- 172.21.15.111:0/3382597057 wait complete. 2023-12-19T08:05:11.578 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:05:11.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.738+0000 7fba6be6e700 1 Processor -- start 2023-12-19T08:05:11.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.738+0000 7fba6be6e700 1 -- start start 2023-12-19T08:05:11.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba640c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba6406c7c0 0x7fba6406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fba6415aa30 0x7fba6415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6406a7a0 con 0x7fba6415aa30 2023-12-19T08:05:11.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640c7630 con 0x7fba6406c7c0 2023-12-19T08:05:11.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6be6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406d4b0 con 0x7fba640c6d20 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fba6415aa30 0x7fba6415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fba6415aa30 0x7fba6415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56150/0 (socket says 172.21.15.111:56150) 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 -- 172.21.15.111:0/3988461392 learned_addr learned my addr 172.21.15.111:0/3988461392 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba69c0a700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba640c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba69409700 1 --2- 172.21.15.111:0/3988461392 >> v2:172.21.15.143:3300/0 conn(0x7fba6406c7c0 0x7fba6406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 -- 172.21.15.111:0/3988461392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 msgr2=0x7fba640c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba640c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 -- 172.21.15.111:0/3988461392 >> v2:172.21.15.143:3300/0 conn(0x7fba6406c7c0 msgr2=0x7fba6406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 --2- 172.21.15.111:0/3988461392 >> v2:172.21.15.143:3300/0 conn(0x7fba6406c7c0 0x7fba6406cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 -- 172.21.15.111:0/3988461392 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6406d790 con 0x7fba6415aa30 2023-12-19T08:05:11.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.742+0000 7fba6a40b700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6415aa30 0x7fba6415ae30 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fba64068060 tx=0x7fba5c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0b46cf6c8268b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba68c08700 1 -- 172.21.15.111:0/3988461392 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00e070 con 0x7fba6415aa30 2023-12-19T08:05:11.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba68c08700 1 -- 172.21.15.111:0/3988461392 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c00ba50 con 0x7fba6415aa30 2023-12-19T08:05:11.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba68c08700 1 -- 172.21.15.111:0/3988461392 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c0092e0 con 0x7fba6415aa30 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 -- 172.21.15.111:0/3988461392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6415aa30 msgr2=0x7fba6415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6415aa30 0x7fba6415ae30 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fba64068060 tx=0x7fba5c009840 comp rx=0 tx=0).stop 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 -- 172.21.15.111:0/3988461392 shutdown_connections 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6415aa30 0x7fba6415ae30 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 --2- 172.21.15.111:0/3988461392 >> v2:172.21.15.143:3300/0 conn(0x7fba6406c7c0 0x7fba6406cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 --2- 172.21.15.111:0/3988461392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba640c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 -- 172.21.15.111:0/3988461392 >> 172.21.15.111:0/3988461392 conn(0x7fba64067bb0 msgr2=0x7fba640cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 -- 172.21.15.111:0/3988461392 shutdown_connections 2023-12-19T08:05:11.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.746+0000 7fba6be6e700 1 -- 172.21.15.111:0/3988461392 wait complete. 2023-12-19T08:05:11.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.750+0000 7fba6be6e700 1 Processor -- start 2023-12-19T08:05:11.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.750+0000 7fba6be6e700 1 -- start start 2023-12-19T08:05:11.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.750+0000 7fba6be6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.750+0000 7fba6be6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba641537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 0x7fba640c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6416d2b0 con 0x7fba6406c7c0 2023-12-19T08:05:11.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba640c4300 con 0x7fba6415aa30 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6416c3c0 con 0x7fba640c6d20 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56166/0 (socket says 172.21.15.111:56166) 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 -- 172.21.15.111:0/2308385081 learned_addr learned my addr 172.21.15.111:0/2308385081 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6a40b700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 0x7fba640c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69409700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba641537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 -- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 msgr2=0x7fba641537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba641537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 -- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 msgr2=0x7fba640c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 0x7fba640c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 -- 172.21.15.111:0/2308385081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c009ae0 con 0x7fba6406c7c0 2023-12-19T08:05:11.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6a40b700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 0x7fba640c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba69c0a700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fba6000d8c0 tx=0x7fba60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba600125f0 con 0x7fba6406c7c0 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60012750 con 0x7fba6406c7c0 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60012af0 con 0x7fba6406c7c0 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6416c6a0 con 0x7fba6406c7c0 2023-12-19T08:05:11.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.754+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6416cc60 con 0x7fba6406c7c0 2023-12-19T08:05:11.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64071120 con 0x7fba6406c7c0 2023-12-19T08:05:11.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fba6001e460 con 0x7fba6406c7c0 2023-12-19T08:05:11.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba59ffb700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 0x7fba48062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:11.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fba6007d350 con 0x7fba6406c7c0 2023-12-19T08:05:11.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba69409700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 0x7fba48062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:11.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.758+0000 7fba69409700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 0x7fba48062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba641599d0 tx=0x7fba54008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:11.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.762+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fba6004d440 con 0x7fba6406c7c0 2023-12-19T08:05:11.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:11.966+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fba6406da70 con 0x7fba6406c7c0 2023-12-19T08:05:12.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.786+0000 7fba59ffb700 1 -- 172.21.15.111:0/2308385081 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7fba60050a60 con 0x7fba6406c7c0 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 msgr2=0x7fba48062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 0x7fba48062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba641599d0 tx=0x7fba54008040 comp rx=0 tx=0).stop 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 msgr2=0x7fba640c3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fba6000d8c0 tx=0x7fba60008040 comp rx=0 tx=0).stop 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 shutdown_connections 2023-12-19T08:05:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fba48060670 0x7fba48062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba6415aa30 0x7fba640c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba640c6d20 0x7fba641537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 --2- 172.21.15.111:0/2308385081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6406c7c0 0x7fba640c3dc0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 >> 172.21.15.111:0/2308385081 conn(0x7fba64067bb0 msgr2=0x7fba640addd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 shutdown_connections 2023-12-19T08:05:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.794+0000 7fba6be6e700 1 -- 172.21.15.111:0/2308385081 wait complete. 2023-12-19T08:05:12.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:05:12.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 Processor -- start 2023-12-19T08:05:12.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 -- start start 2023-12-19T08:05:12.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f93d8158e20 0x7f93d80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:12.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d806d5e0 con 0x7f93d80ca2c0 2023-12-19T08:05:12.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80cbdf0 con 0x7f93d80cb970 2023-12-19T08:05:12.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93dc8ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d80cc0d0 con 0x7f93d8158e20 2023-12-19T08:05:12.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93d6d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f93d80ca2c0 0x7f93d806ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42172/0 (socket says 172.21.15.111:42172) 2023-12-19T08:05:12.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93d6d9d700 1 -- 172.21.15.111:0/760354499 learned_addr learned my addr 172.21.15.111:0/760354499 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:12.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.986+0000 7f93d659c700 1 --2- 172.21.15.111:0/760354499 >> v2:172.21.15.143:3300/0 conn(0x7f93d8158e20 0x7f93d80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:12.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420953944 0 0) 0x7f93d806d5e0 con 0x7f93d80ca2c0 2023-12-19T08:05:12.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0003570 con 0x7f93d80ca2c0 2023-12-19T08:05:12.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787747854 0 0) 0x7f93d80cbdf0 con 0x7f93d80cb970 2023-12-19T08:05:12.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d806d5e0 con 0x7f93d80cb970 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d659c700 1 -- 172.21.15.111:0/760354499 >> v1:172.21.15.143:6789/0 conn(0x7f93d80cb970 legacy=0x7f93d806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d659c700 1 -- 172.21.15.111:0/760354499 >> v1:172.21.15.111:6789/0 conn(0x7f93d80ca2c0 legacy=0x7f93d806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d659c700 1 -- 172.21.15.111:0/760354499 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d80cc3b0 con 0x7f93d8158e20 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d659c700 1 --2- 172.21.15.111:0/760354499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80ce5c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009910 tx=0x7f93c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcce55f569a7bea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011630 con 0x7f93d8158e20 2023-12-19T08:05:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8011790 con 0x7f93d8158e20 2023-12-19T08:05:12.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93d559a700 1 -- 172.21.15.111:0/760354499 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011b30 con 0x7f93d8158e20 2023-12-19T08:05:12.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 -- 172.21.15.111:0/760354499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 msgr2=0x7f93d80ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:12.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/760354499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80ce5c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009910 tx=0x7f93c80091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 -- 172.21.15.111:0/760354499 shutdown_connections 2023-12-19T08:05:12.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/760354499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80ce5c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:12.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 -- 172.21.15.111:0/760354499 >> 172.21.15.111:0/760354499 conn(0x7f93d8067bb0 msgr2=0x7f93d80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:12.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 -- 172.21.15.111:0/760354499 shutdown_connections 2023-12-19T08:05:12.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.990+0000 7f93dc8ae700 1 -- 172.21.15.111:0/760354499 wait complete. 2023-12-19T08:05:12.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 Processor -- start 2023-12-19T08:05:12.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 -- start start 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93d815d570 0x7f93d8168990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c8400 con 0x7f93d80ca2c0 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c7190 con 0x7f93d8158e20 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93dc8ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c7a30 con 0x7f93d815d570 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56180/0 (socket says 172.21.15.111:56180) 2023-12-19T08:05:12.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.994+0000 7f93d659c700 1 -- 172.21.15.111:0/1333294342 learned_addr learned my addr 172.21.15.111:0/1333294342 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:13.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d6d9d700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93d815d570 0x7f93d8168990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d5d9b700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 -- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93d815d570 msgr2=0x7f93d8168990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93d815d570 0x7f93d8168990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 -- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 msgr2=0x7f93d80c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 -- 172.21.15.111:0/1333294342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c8009420 con 0x7f93d80ca2c0 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d5d9b700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:13.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93d659c700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009f00 tx=0x7f93c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c801f070 con 0x7f93d80ca2c0 2023-12-19T08:05:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8011b10 con 0x7f93d80ca2c0 2023-12-19T08:05:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d815dae0 con 0x7f93d80ca2c0 2023-12-19T08:05:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8023660 con 0x7f93d80ca2c0 2023-12-19T08:05:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d80c76c0 con 0x7f93d80ca2c0 2023-12-19T08:05:13.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:12.998+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d8071120 con 0x7f93d80ca2c0 2023-12-19T08:05:13.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.002+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f93c802c3b0 con 0x7f93d80ca2c0 2023-12-19T08:05:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.002+0000 7f93c67fc700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 0x7f93b8062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.002+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f93c807e0e0 con 0x7f93d80ca2c0 2023-12-19T08:05:13.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.002+0000 7f93d5d9b700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 0x7f93b8062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.006+0000 7f93d5d9b700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 0x7f93b8062bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f93d80cf050 tx=0x7f93cc009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:13.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.010+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f93c804e2f0 con 0x7f93d80ca2c0 2023-12-19T08:05:13.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.214+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f93d80cc690 con 0x7f93d80ca2c0 2023-12-19T08:05:13.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93c67fc700 1 -- 172.21.15.111:0/1333294342 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) v1 ==== 122+0+0 (secure 0 0 0) 0x7f93c8051910 con 0x7f93d80ca2c0 2023-12-19T08:05:13.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 msgr2=0x7f93b8062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 0x7f93b8062bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f93d80cf050 tx=0x7f93cc009040 comp rx=0 tx=0).stop 2023-12-19T08:05:13.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 msgr2=0x7f93d80c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009f00 tx=0x7f93c8017000 comp rx=0 tx=0).stop 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 shutdown_connections 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f93b8060740 0x7f93b8062bf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d8158e20 0x7f93d80c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93d815d570 0x7f93d8168990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.794+0000 7f93dc8ae700 1 --2- 172.21.15.111:0/1333294342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93d80ca2c0 0x7f93d80c9d80 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.798+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 >> 172.21.15.111:0/1333294342 conn(0x7f93d8067bb0 msgr2=0x7f93d80cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.798+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 shutdown_connections 2023-12-19T08:05:13.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.798+0000 7f93dc8ae700 1 -- 172.21.15.111:0/1333294342 wait complete. 2023-12-19T08:05:13.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:05:13.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.978+0000 7f8103545700 1 Processor -- start 2023-12-19T08:05:13.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.978+0000 7f8103545700 1 -- start start 2023-12-19T08:05:13.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8103545700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80fc068800 0x7f80fc161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8103545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc06e150 con 0x7f80fc140ae0 2023-12-19T08:05:13.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8103545700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc06d540 con 0x7f80fc140710 2023-12-19T08:05:13.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8103545700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80fc06d820 con 0x7f80fc068800 2023-12-19T08:05:13.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8101ae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80fc068800 0x7f80fc161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8101ae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80fc068800 0x7f80fc161cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56190/0 (socket says 172.21.15.111:56190) 2023-12-19T08:05:13.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8101ae2700 1 -- 172.21.15.111:0/1769376164 learned_addr learned my addr 172.21.15.111:0/1769376164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:13.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f8100ae0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f80fc140ae0 0x7f80fc0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42180/0 (socket says 172.21.15.111:42180) 2023-12-19T08:05:13.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.982+0000 7f81012e1700 1 --1- 172.21.15.111:0/1769376164 >> v1:172.21.15.143:6789/0 conn(0x7f80fc140710 0x7f80fc0c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:52296/0 (socket says 172.21.15.111:52296) 2023-12-19T08:05:13.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8101ae2700 1 -- 172.21.15.111:0/1769376164 >> v1:172.21.15.143:6789/0 conn(0x7f80fc140710 legacy=0x7f80fc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8101ae2700 1 -- 172.21.15.111:0/1769376164 >> v1:172.21.15.111:6789/0 conn(0x7f80fc140ae0 legacy=0x7f80fc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8101ae2700 1 -- 172.21.15.111:0/1769376164 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc06db00 con 0x7f80fc068800 2023-12-19T08:05:13.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8101ae2700 1 --2- 172.21.15.111:0/1769376164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc161cd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009f20 tx=0x7f80f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef574f13dcb7df6c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:13.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f80f3fff700 1 -- 172.21.15.111:0/1769376164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400e070 con 0x7f80fc068800 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f80f3fff700 1 -- 172.21.15.111:0/1769376164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f400ba80 con 0x7f80fc068800 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f80f3fff700 1 -- 172.21.15.111:0/1769376164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f40092e0 con 0x7f80fc068800 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 -- 172.21.15.111:0/1769376164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 msgr2=0x7f80fc161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 --2- 172.21.15.111:0/1769376164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc161cd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009f20 tx=0x7f80f4009840 comp rx=0 tx=0).stop 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 -- 172.21.15.111:0/1769376164 shutdown_connections 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 --2- 172.21.15.111:0/1769376164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc161cd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 -- 172.21.15.111:0/1769376164 >> 172.21.15.111:0/1769376164 conn(0x7f80fc0c8850 msgr2=0x7f80fc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:13.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 -- 172.21.15.111:0/1769376164 shutdown_connections 2023-12-19T08:05:13.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.986+0000 7f8103545700 1 -- 172.21.15.111:0/1769376164 wait complete. 2023-12-19T08:05:13.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 Processor -- start 2023-12-19T08:05:13.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- start start 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 0x7f80fc1593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 0x7f80fc168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc159c50 con 0x7f80fc068800 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc159f30 con 0x7f80fc140710 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc0a89f0 con 0x7f80fc140ae0 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56202/0 (socket says 172.21.15.111:56202) 2023-12-19T08:05:13.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 -- 172.21.15.111:0/1943381315 learned_addr learned my addr 172.21.15.111:0/1943381315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8100ae0700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 0x7f80fc1593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8101ae2700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 0x7f80fc168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 -- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 msgr2=0x7f80fc168ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 0x7f80fc168ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 -- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 msgr2=0x7f80fc1593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 0x7f80fc1593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 -- 172.21.15.111:0/1943381315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4009ae0 con 0x7f80fc068800 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8101ae2700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 0x7f80fc168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8100ae0700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 0x7f80fc1593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f81012e1700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c7c0 tx=0x7f80f8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8005a70 con 0x7f80fc068800 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8005bd0 con 0x7f80fc068800 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc0a8cd0 con 0x7f80fc068800 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80070f0 con 0x7f80fc068800 2023-12-19T08:05:13.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.990+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc169db0 con 0x7f80fc068800 2023-12-19T08:05:13.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.994+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc0a8850 con 0x7f80fc068800 2023-12-19T08:05:13.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.994+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f80f8007460 con 0x7f80fc068800 2023-12-19T08:05:13.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.998+0000 7f80f17fa700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 0x7f80e0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:13.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.998+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f80f807c710 con 0x7f80fc068800 2023-12-19T08:05:14.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.998+0000 7f8100ae0700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 0x7f80e0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:13.998+0000 7f8100ae0700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 0x7f80e0062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f80ec007fd0 tx=0x7f80ec007db0 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.002+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f80fc0a8850 con 0x7f80fc068800 2023-12-19T08:05:14.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.210+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80fc0a8850 con 0x7f80fc068800 2023-12-19T08:05:14.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.210+0000 7f80f17fa700 1 -- 172.21.15.111:0/1943381315 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7f80fc0a8850 con 0x7f80fc068800 2023-12-19T08:05:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:14.215 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 msgr2=0x7f80e0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 0x7f80e0062b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f80ec007fd0 tx=0x7f80ec007db0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 msgr2=0x7f80fc0aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c7c0 tx=0x7f80f8007f70 comp rx=0 tx=0).stop 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 shutdown_connections 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f80e00606a0 0x7f80e0062b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc140710 0x7f80fc1593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc140ae0 0x7f80fc168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 --2- 172.21.15.111:0/1943381315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc068800 0x7f80fc0aad10 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 >> 172.21.15.111:0/1943381315 conn(0x7f80fc0c8850 msgr2=0x7f80fc06b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:14.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 shutdown_connections 2023-12-19T08:05:14.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.214+0000 7f8103545700 1 -- 172.21.15.111:0/1943381315 wait complete. 2023-12-19T08:05:14.220 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:05:14.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:05:14.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.398+0000 7fb84e4f0700 1 Processor -- start 2023-12-19T08:05:14.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.398+0000 7fb84e4f0700 1 -- start start 2023-12-19T08:05:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84e4f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb848158ee0 0x7fb848067af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84e4f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8481592b0 0x7fb84806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84e4f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480688d0 con 0x7fb848068030 2023-12-19T08:05:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84e4f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8480b1510 con 0x7fb848158ee0 2023-12-19T08:05:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84e4f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84806a180 con 0x7fb8481592b0 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb848158ee0 0x7fb848067af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84ca8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb848068030 0x7fb84815cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42190/0 (socket says 172.21.15.111:42190) 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb848158ee0 0x7fb848067af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56208/0 (socket says 172.21.15.111:56208) 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb84ca8d700 1 -- 172.21.15.111:0/713708308 learned_addr learned my addr 172.21.15.111:0/713708308 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb83ffff700 1 --2- 172.21.15.111:0/713708308 >> v2:172.21.15.143:3300/0 conn(0x7fb8481592b0 0x7fb84806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 -- 172.21.15.111:0/713708308 >> v2:172.21.15.143:3300/0 conn(0x7fb8481592b0 msgr2=0x7fb84806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 --2- 172.21.15.111:0/713708308 >> v2:172.21.15.143:3300/0 conn(0x7fb8481592b0 0x7fb84806d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 -- 172.21.15.111:0/713708308 >> v1:172.21.15.111:6789/0 conn(0x7fb848068030 legacy=0x7fb84815cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 -- 172.21.15.111:0/713708308 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480685f0 con 0x7fb848158ee0 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb847fff700 1 --2- 172.21.15.111:0/713708308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848158ee0 0x7fb848067af0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb8480ac380 tx=0x7fb8380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08f87d35c01f478 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb8477fe700 1 -- 172.21.15.111:0/713708308 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83800b6b0 con 0x7fb848158ee0 2023-12-19T08:05:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb8477fe700 1 -- 172.21.15.111:0/713708308 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83800b810 con 0x7fb848158ee0 2023-12-19T08:05:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.402+0000 7fb8477fe700 1 -- 172.21.15.111:0/713708308 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83800bbb0 con 0x7fb848158ee0 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 -- 172.21.15.111:0/713708308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848158ee0 msgr2=0x7fb848067af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/713708308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848158ee0 0x7fb848067af0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb8480ac380 tx=0x7fb8380091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 -- 172.21.15.111:0/713708308 shutdown_connections 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/713708308 >> v2:172.21.15.143:3300/0 conn(0x7fb8481592b0 0x7fb84806d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/713708308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848158ee0 0x7fb848067af0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 -- 172.21.15.111:0/713708308 >> 172.21.15.111:0/713708308 conn(0x7fb8480cd990 msgr2=0x7fb8480ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 -- 172.21.15.111:0/713708308 shutdown_connections 2023-12-19T08:05:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.406+0000 7fb84e4f0700 1 -- 172.21.15.111:0/713708308 wait complete. 2023-12-19T08:05:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 Processor -- start 2023-12-19T08:05:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 -- start start 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 0x7fb848144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 0x7fb848159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480aca20 con 0x7fb848068030 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb848153930 con 0x7fb848158ee0 2023-12-19T08:05:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84e4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84816a4a0 con 0x7fb8481592b0 2023-12-19T08:05:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56214/0 (socket says 172.21.15.111:56214) 2023-12-19T08:05:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 -- 172.21.15.111:0/995484932 learned_addr learned my addr 172.21.15.111:0/995484932 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 -- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 msgr2=0x7fb848159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84ca8d700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 0x7fb848159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb83ffff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 0x7fb848144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 0x7fb848159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 -- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 msgr2=0x7fb848144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 0x7fb848144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb847fff700 1 -- 172.21.15.111:0/995484932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838009420 con 0x7fb848068030 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb84ca8d700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 0x7fb848159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.410+0000 7fb83ffff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 0x7fb848144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb847fff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb84806a6e0 tx=0x7fb838017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838011420 con 0x7fb848068030 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838011580 con 0x7fb848068030 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84816a1c0 con 0x7fb848068030 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838011940 con 0x7fb848068030 2023-12-19T08:05:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8481535c0 con 0x7fb848068030 2023-12-19T08:05:14.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.414+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb848071120 con 0x7fb848068030 2023-12-19T08:05:14.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.418+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb838011cf0 con 0x7fb848068030 2023-12-19T08:05:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.418+0000 7fb844ff9700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 0x7fb828062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.418+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fb83807e100 con 0x7fb848068030 2023-12-19T08:05:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.418+0000 7fb83ffff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 0x7fb828062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.418+0000 7fb83ffff700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 0x7fb828062bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb84816cb00 tx=0x7fb830008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.426+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb83804e270 con 0x7fb848068030 2023-12-19T08:05:14.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.598+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8480b0ec0 con 0x7fb848068030 2023-12-19T08:05:14.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb844ff9700 1 -- 172.21.15.111:0/995484932 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16406 (secure 0 0 0) 0x7fb838051890 con 0x7fb848068030 2023-12-19T08:05:14.604 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:14.604 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":129,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:05:11.740685+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:05:14.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 msgr2=0x7fb828062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 0x7fb828062bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb84816cb00 tx=0x7fb830008040 comp rx=0 tx=0).stop 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 msgr2=0x7fb848151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb84806a6e0 tx=0x7fb838017000 comp rx=0 tx=0).stop 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 shutdown_connections 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb828060740 0x7fb828062bf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb848158ee0 0x7fb848144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8481592b0 0x7fb848159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 --2- 172.21.15.111:0/995484932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb848068030 0x7fb848151d90 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 >> 172.21.15.111:0/995484932 conn(0x7fb8480cd990 msgr2=0x7fb8481558a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:14.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 shutdown_connections 2023-12-19T08:05:14.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.602+0000 7fb84e4f0700 1 -- 172.21.15.111:0/995484932 wait complete. 2023-12-19T08:05:14.628 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:05:14.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:05:14.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.786+0000 7f19a9ae3700 1 Processor -- start 2023-12-19T08:05:14.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.786+0000 7f19a9ae3700 1 -- start start 2023-12-19T08:05:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f19a4158e20 0x7f19a406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19a415ca80 0x7f19a40ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a406b730 con 0x7f19a415ca80 2023-12-19T08:05:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a406ab20 con 0x7f19a4158e20 2023-12-19T08:05:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a9ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a406ae00 con 0x7f19a41591f0 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19a415ca80 0x7f19a40ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:58726/0 (socket says 172.21.15.111:58726) 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a8ae1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f19a4158e20 0x7f19a406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19a415ca80 0x7f19a40ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56220/0 (socket says 172.21.15.111:56220) 2023-12-19T08:05:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a3fff700 1 -- 172.21.15.111:0/2723588614 learned_addr learned my addr 172.21.15.111:0/2723588614 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 -- 172.21.15.111:0/2723588614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 msgr2=0x7f19a40ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 -- 172.21.15.111:0/2723588614 >> v2:172.21.15.143:3300/0 conn(0x7f19a4158e20 msgr2=0x7f19a406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 --2- 172.21.15.111:0/2723588614 >> v2:172.21.15.143:3300/0 conn(0x7f19a4158e20 0x7f19a406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 -- 172.21.15.111:0/2723588614 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a406b0e0 con 0x7f19a415ca80 2023-12-19T08:05:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a3fff700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:14.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.790+0000 7f19a92e2700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a415ca80 0x7f19a40ce640 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f19a406e510 tx=0x7f199c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86657ca4482048ce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.794+0000 7f19a37fe700 1 -- 172.21.15.111:0/2723588614 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00e070 con 0x7f19a415ca80 2023-12-19T08:05:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.794+0000 7f19a37fe700 1 -- 172.21.15.111:0/2723588614 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c00bab0 con 0x7f19a415ca80 2023-12-19T08:05:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.794+0000 7f19a37fe700 1 -- 172.21.15.111:0/2723588614 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00be70 con 0x7f19a415ca80 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- 172.21.15.111:0/2723588614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a415ca80 msgr2=0x7f19a40ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a415ca80 0x7f19a40ce640 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f19a406e510 tx=0x7f199c009780 comp rx=0 tx=0).stop 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- 172.21.15.111:0/2723588614 shutdown_connections 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a415ca80 0x7f19a40ce640 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/2723588614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a41591f0 0x7f19a40ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/2723588614 >> v2:172.21.15.143:3300/0 conn(0x7f19a4158e20 0x7f19a406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- 172.21.15.111:0/2723588614 >> 172.21.15.111:0/2723588614 conn(0x7f19a4067bb0 msgr2=0x7f19a40cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- 172.21.15.111:0/2723588614 shutdown_connections 2023-12-19T08:05:14.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- 172.21.15.111:0/2723588614 wait complete. 2023-12-19T08:05:14.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 Processor -- start 2023-12-19T08:05:14.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.798+0000 7f19a9ae3700 1 -- start start 2023-12-19T08:05:14.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 0x7f19a4169c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 0x7f19a41512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a415ca80 0x7f19a414afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4152860 con 0x7f19a41591f0 2023-12-19T08:05:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a414b500 con 0x7f19a415ca80 2023-12-19T08:05:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a41522a0 con 0x7f19a4158e20 2023-12-19T08:05:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a8ae1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 0x7f19a4169c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 0x7f19a41512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a8ae1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 0x7f19a4169c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:58732/0 (socket says 172.21.15.111:58732) 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a8ae1700 1 -- 172.21.15.111:0/4072778860 learned_addr learned my addr 172.21.15.111:0/4072778860 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a92e2700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a415ca80 0x7f19a414afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 -- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 msgr2=0x7f19a4169c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 0x7f19a4169c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 -- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a415ca80 msgr2=0x7f19a414afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a415ca80 0x7f19a414afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 -- 172.21.15.111:0/4072778860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19900094f0 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a3fff700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 0x7f19a41512b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f19a41595c0 tx=0x7f1990009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000c820 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199000c980 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c009a20 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000cd00 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.802+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a41691b0 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.806+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a404be40 con 0x7f19a41591f0 2023-12-19T08:05:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.806+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1990013680 con 0x7f19a41591f0 2023-12-19T08:05:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.810+0000 7f19a0ff9700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 0x7f1988062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.810+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f199007ce20 con 0x7f19a41591f0 2023-12-19T08:05:14.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.810+0000 7f19a8ae1700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 0x7f1988062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.810+0000 7f19a8ae1700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 0x7f1988062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1998009e70 tx=0x7f19980091a0 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.814+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f199004cf10 con 0x7f19a41591f0 2023-12-19T08:05:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:14.978+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19a406a860 con 0x7f1988060670 2023-12-19T08:05:15.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.002+0000 7f19a0ff9700 1 -- 172.21.15.111:0/4072778860 <== mgr.7501 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287072 (secure 0 0 0) 0x7f19a406a860 con 0x7f1988060670 2023-12-19T08:05:15.009 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 msgr2=0x7f1988062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 0x7f1988062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1998009e70 tx=0x7f19980091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 msgr2=0x7f19a41512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 0x7f19a41512b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f19a41595c0 tx=0x7f1990009250 comp rx=0 tx=0).stop 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 shutdown_connections 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1988060670 0x7f1988062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a415ca80 0x7f19a414afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a41591f0 0x7f19a41512b0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 --2- 172.21.15.111:0/4072778860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a4158e20 0x7f19a4169c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.006+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 >> 172.21.15.111:0/4072778860 conn(0x7f19a4067bb0 msgr2=0x7f19a40cb360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:15.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.010+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 shutdown_connections 2023-12-19T08:05:15.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.010+0000 7f19a9ae3700 1 -- 172.21.15.111:0/4072778860 wait complete. 2023-12-19T08:05:15.014 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:05:15.033 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:05:14.319455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449276,"kb_used_data":13628,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525444,"statfs":{"total":773094113280,"available":770586054656,"internally_reserved":0,"allocated":13955072,"data_stored":7828401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.599203"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774442+0000","last_change":"2023-12-19T08:05:10.774442+0000","last_active":"2023-12-19T08:05:10.774442+0000","last_peered":"2023-12-19T08:05:10.774442+0000","last_clean":"2023-12-19T08:05:10.774442+0000","last_became_active":"2023-12-19T08:05:10.773088+0000","last_became_peered":"2023-12-19T08:05:10.773088+0000","last_unstale":"2023-12-19T08:05:10.774442+0000","last_undegraded":"2023-12-19T08:05:10.774442+0000","last_fullsized":"2023-12-19T08:05:10.774442+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:22.217801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.781626+0000","last_change":"2023-12-19T08:05:10.781626+0000","last_active":"2023-12-19T08:05:10.781626+0000","last_peered":"2023-12-19T08:05:10.781626+0000","last_clean":"2023-12-19T08:05:10.781626+0000","last_became_active":"2023-12-19T08:05:10.781355+0000","last_became_peered":"2023-12-19T08:05:10.781355+0000","last_unstale":"2023-12-19T08:05:10.781626+0000","last_undegraded":"2023-12-19T08:05:10.781626+0000","last_fullsized":"2023-12-19T08:05:10.781626+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.784678+0000","last_change":"2023-12-19T08:05:10.784678+0000","last_active":"2023-12-19T08:05:10.784678+0000","last_peered":"2023-12-19T08:05:10.784678+0000","last_clean":"2023-12-19T08:05:10.784678+0000","last_became_active":"2023-12-19T08:05:10.783781+0000","last_became_peered":"2023-12-19T08:05:10.783781+0000","last_unstale":"2023-12-19T08:05:10.784678+0000","last_undegraded":"2023-12-19T08:05:10.784678+0000","last_fullsized":"2023-12-19T08:05:10.784678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.914967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773003+0000","last_change":"2023-12-19T08:05:10.773003+0000","last_active":"2023-12-19T08:05:10.773003+0000","last_peered":"2023-12-19T08:05:10.773003+0000","last_clean":"2023-12-19T08:05:10.773003+0000","last_became_active":"2023-12-19T08:05:10.772644+0000","last_became_peered":"2023-12-19T08:05:10.772644+0000","last_unstale":"2023-12-19T08:05:10.773003+0000","last_undegraded":"2023-12-19T08:05:10.773003+0000","last_fullsized":"2023-12-19T08:05:10.773003+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:07:29.016139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.761247+0000","last_change":"2023-12-19T08:05:10.752718+0000","last_active":"2023-12-19T08:05:10.761247+0000","last_peered":"2023-12-19T08:05:10.761247+0000","last_clean":"2023-12-19T08:05:10.761247+0000","last_became_active":"2023-12-19T08:05:10.752389+0000","last_became_peered":"2023-12-19T08:05:10.752389+0000","last_unstale":"2023-12-19T08:05:10.761247+0000","last_undegraded":"2023-12-19T08:05:10.761247+0000","last_fullsized":"2023-12-19T08:05:10.761247+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777106+0000","last_change":"2023-12-19T08:05:10.777106+0000","last_active":"2023-12-19T08:05:10.777106+0000","last_peered":"2023-12-19T08:05:10.777106+0000","last_clean":"2023-12-19T08:05:10.777106+0000","last_became_active":"2023-12-19T08:05:10.776757+0000","last_became_peered":"2023-12-19T08:05:10.776757+0000","last_unstale":"2023-12-19T08:05:10.777106+0000","last_undegraded":"2023-12-19T08:05:10.777106+0000","last_fullsized":"2023-12-19T08:05:10.777106+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:35.788963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769819+0000","last_change":"2023-12-19T08:05:10.769819+0000","last_active":"2023-12-19T08:05:10.769819+0000","last_peered":"2023-12-19T08:05:10.769819+0000","last_clean":"2023-12-19T08:05:10.769819+0000","last_became_active":"2023-12-19T08:05:10.769466+0000","last_became_peered":"2023-12-19T08:05:10.769466+0000","last_unstale":"2023-12-19T08:05:10.769819+0000","last_undegraded":"2023-12-19T08:05:10.769819+0000","last_fullsized":"2023-12-19T08:05:10.769819+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:00.297592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768141+0000","last_change":"2023-12-19T08:05:10.768141+0000","last_active":"2023-12-19T08:05:10.768141+0000","last_peered":"2023-12-19T08:05:10.768141+0000","last_clean":"2023-12-19T08:05:10.768141+0000","last_became_active":"2023-12-19T08:05:10.767843+0000","last_became_peered":"2023-12-19T08:05:10.767843+0000","last_unstale":"2023-12-19T08:05:10.768141+0000","last_undegraded":"2023-12-19T08:05:10.768141+0000","last_fullsized":"2023-12-19T08:05:10.768141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:56.736611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.789632+0000","last_change":"2023-12-19T08:05:10.789632+0000","last_active":"2023-12-19T08:05:10.789632+0000","last_peered":"2023-12-19T08:05:10.789632+0000","last_clean":"2023-12-19T08:05:10.789632+0000","last_became_active":"2023-12-19T08:05:10.787575+0000","last_became_peered":"2023-12-19T08:05:10.787575+0000","last_unstale":"2023-12-19T08:05:10.789632+0000","last_undegraded":"2023-12-19T08:05:10.789632+0000","last_fullsized":"2023-12-19T08:05:10.789632+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.786690+0000","last_change":"2023-12-19T08:05:10.775590+0000","last_active":"2023-12-19T08:05:10.786690+0000","last_peered":"2023-12-19T08:05:10.786690+0000","last_clean":"2023-12-19T08:05:10.786690+0000","last_became_active":"2023-12-19T08:05:10.775156+0000","last_became_peered":"2023-12-19T08:05:10.775156+0000","last_unstale":"2023-12-19T08:05:10.786690+0000","last_undegraded":"2023-12-19T08:05:10.786690+0000","last_fullsized":"2023-12-19T08:05:10.786690+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.791092+0000","last_change":"2023-12-19T08:05:10.787247+0000","last_active":"2023-12-19T08:05:10.791092+0000","last_peered":"2023-12-19T08:05:10.791092+0000","last_clean":"2023-12-19T08:05:10.791092+0000","last_became_active":"2023-12-19T08:05:10.786952+0000","last_became_peered":"2023-12-19T08:05:10.786952+0000","last_unstale":"2023-12-19T08:05:10.791092+0000","last_undegraded":"2023-12-19T08:05:10.791092+0000","last_fullsized":"2023-12-19T08:05:10.791092+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775726+0000","last_change":"2023-12-19T08:05:10.771089+0000","last_active":"2023-12-19T08:05:10.775726+0000","last_peered":"2023-12-19T08:05:10.775726+0000","last_clean":"2023-12-19T08:05:10.775726+0000","last_became_active":"2023-12-19T08:05:10.770810+0000","last_became_peered":"2023-12-19T08:05:10.770810+0000","last_unstale":"2023-12-19T08:05:10.775726+0000","last_undegraded":"2023-12-19T08:05:10.775726+0000","last_fullsized":"2023-12-19T08:05:10.775726+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:24:25.145068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768508+0000","last_change":"2023-12-19T08:05:10.768508+0000","last_active":"2023-12-19T08:05:10.768508+0000","last_peered":"2023-12-19T08:05:10.768508+0000","last_clean":"2023-12-19T08:05:10.768508+0000","last_became_active":"2023-12-19T08:05:10.768072+0000","last_became_peered":"2023-12-19T08:05:10.768072+0000","last_unstale":"2023-12-19T08:05:10.768508+0000","last_undegraded":"2023-12-19T08:05:10.768508+0000","last_fullsized":"2023-12-19T08:05:10.768508+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.914967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773075+0000","last_change":"2023-12-19T08:05:10.773075+0000","last_active":"2023-12-19T08:05:10.773075+0000","last_peered":"2023-12-19T08:05:10.773075+0000","last_clean":"2023-12-19T08:05:10.773075+0000","last_became_active":"2023-12-19T08:05:10.772743+0000","last_became_peered":"2023-12-19T08:05:10.772743+0000","last_unstale":"2023-12-19T08:05:10.773075+0000","last_undegraded":"2023-12-19T08:05:10.773075+0000","last_fullsized":"2023-12-19T08:05:10.773075+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776349+0000","last_change":"2023-12-19T08:05:10.776349+0000","last_active":"2023-12-19T08:05:10.776349+0000","last_peered":"2023-12-19T08:05:10.776349+0000","last_clean":"2023-12-19T08:05:10.776349+0000","last_became_active":"2023-12-19T08:05:10.776083+0000","last_became_peered":"2023-12-19T08:05:10.776083+0000","last_unstale":"2023-12-19T08:05:10.776349+0000","last_undegraded":"2023-12-19T08:05:10.776349+0000","last_fullsized":"2023-12-19T08:05:10.776349+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.783657+0000","last_change":"2023-12-19T08:05:10.783657+0000","last_active":"2023-12-19T08:05:10.783657+0000","last_peered":"2023-12-19T08:05:10.783657+0000","last_clean":"2023-12-19T08:05:10.783657+0000","last_became_active":"2023-12-19T08:05:10.783368+0000","last_became_peered":"2023-12-19T08:05:10.783368+0000","last_unstale":"2023-12-19T08:05:10.783657+0000","last_undegraded":"2023-12-19T08:05:10.783657+0000","last_fullsized":"2023-12-19T08:05:10.783657+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782649+0000","last_change":"2023-12-19T08:05:10.782649+0000","last_active":"2023-12-19T08:05:10.782649+0000","last_peered":"2023-12-19T08:05:10.782649+0000","last_clean":"2023-12-19T08:05:10.782649+0000","last_became_active":"2023-12-19T08:05:10.782191+0000","last_became_peered":"2023-12-19T08:05:10.782191+0000","last_unstale":"2023-12-19T08:05:10.782649+0000","last_undegraded":"2023-12-19T08:05:10.782649+0000","last_fullsized":"2023-12-19T08:05:10.782649+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:59.230868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774741+0000","last_change":"2023-12-19T08:05:10.774741+0000","last_active":"2023-12-19T08:05:10.774741+0000","last_peered":"2023-12-19T08:05:10.774741+0000","last_clean":"2023-12-19T08:05:10.774741+0000","last_became_active":"2023-12-19T08:05:10.774330+0000","last_became_peered":"2023-12-19T08:05:10.774330+0000","last_unstale":"2023-12-19T08:05:10.774741+0000","last_undegraded":"2023-12-19T08:05:10.774741+0000","last_fullsized":"2023-12-19T08:05:10.774741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:53.094931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778808+0000","last_change":"2023-12-19T08:05:10.778808+0000","last_active":"2023-12-19T08:05:10.778808+0000","last_peered":"2023-12-19T08:05:10.778808+0000","last_clean":"2023-12-19T08:05:10.778808+0000","last_became_active":"2023-12-19T08:05:10.778308+0000","last_became_peered":"2023-12-19T08:05:10.778308+0000","last_unstale":"2023-12-19T08:05:10.778808+0000","last_undegraded":"2023-12-19T08:05:10.778808+0000","last_fullsized":"2023-12-19T08:05:10.778808+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:51:50.011314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775707+0000","last_change":"2023-12-19T08:05:10.767923+0000","last_active":"2023-12-19T08:05:10.775707+0000","last_peered":"2023-12-19T08:05:10.775707+0000","last_clean":"2023-12-19T08:05:10.775707+0000","last_became_active":"2023-12-19T08:05:10.766548+0000","last_became_peered":"2023-12-19T08:05:10.766548+0000","last_unstale":"2023-12-19T08:05:10.775707+0000","last_undegraded":"2023-12-19T08:05:10.775707+0000","last_fullsized":"2023-12-19T08:05:10.775707+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:53:30.585462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777398+0000","last_change":"2023-12-19T08:05:10.777398+0000","last_active":"2023-12-19T08:05:10.777398+0000","last_peered":"2023-12-19T08:05:10.777398+0000","last_clean":"2023-12-19T08:05:10.777398+0000","last_became_active":"2023-12-19T08:05:10.777066+0000","last_became_peered":"2023-12-19T08:05:10.777066+0000","last_unstale":"2023-12-19T08:05:10.777398+0000","last_undegraded":"2023-12-19T08:05:10.777398+0000","last_fullsized":"2023-12-19T08:05:10.777398+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:31.320978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775491+0000","last_change":"2023-12-19T08:05:10.767733+0000","last_active":"2023-12-19T08:05:10.775491+0000","last_peered":"2023-12-19T08:05:10.775491+0000","last_clean":"2023-12-19T08:05:10.775491+0000","last_became_active":"2023-12-19T08:05:10.767191+0000","last_became_peered":"2023-12-19T08:05:10.767191+0000","last_unstale":"2023-12-19T08:05:10.775491+0000","last_undegraded":"2023-12-19T08:05:10.775491+0000","last_fullsized":"2023-12-19T08:05:10.775491+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769177+0000","last_change":"2023-12-19T08:05:10.769177+0000","last_active":"2023-12-19T08:05:10.769177+0000","last_peered":"2023-12-19T08:05:10.769177+0000","last_clean":"2023-12-19T08:05:10.769177+0000","last_became_active":"2023-12-19T08:05:10.768843+0000","last_became_peered":"2023-12-19T08:05:10.768843+0000","last_unstale":"2023-12-19T08:05:10.769177+0000","last_undegraded":"2023-12-19T08:05:10.769177+0000","last_fullsized":"2023-12-19T08:05:10.769177+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:47.089554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.765424+0000","last_change":"2023-12-19T08:05:10.760793+0000","last_active":"2023-12-19T08:05:10.765424+0000","last_peered":"2023-12-19T08:05:10.765424+0000","last_clean":"2023-12-19T08:05:10.765424+0000","last_became_active":"2023-12-19T08:05:10.759127+0000","last_became_peered":"2023-12-19T08:05:10.759127+0000","last_unstale":"2023-12-19T08:05:10.765424+0000","last_undegraded":"2023-12-19T08:05:10.765424+0000","last_fullsized":"2023-12-19T08:05:10.765424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777571+0000","last_change":"2023-12-19T08:05:10.774250+0000","last_active":"2023-12-19T08:05:10.777571+0000","last_peered":"2023-12-19T08:05:10.777571+0000","last_clean":"2023-12-19T08:05:10.777571+0000","last_became_active":"2023-12-19T08:05:10.773915+0000","last_became_peered":"2023-12-19T08:05:10.773915+0000","last_unstale":"2023-12-19T08:05:10.777571+0000","last_undegraded":"2023-12-19T08:05:10.777571+0000","last_fullsized":"2023-12-19T08:05:10.777571+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:43.010981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780488+0000","last_change":"2023-12-19T08:05:10.776781+0000","last_active":"2023-12-19T08:05:10.780488+0000","last_peered":"2023-12-19T08:05:10.780488+0000","last_clean":"2023-12-19T08:05:10.780488+0000","last_became_active":"2023-12-19T08:05:10.776453+0000","last_became_peered":"2023-12-19T08:05:10.776453+0000","last_unstale":"2023-12-19T08:05:10.780488+0000","last_undegraded":"2023-12-19T08:05:10.780488+0000","last_fullsized":"2023-12-19T08:05:10.780488+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:01.734825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.791011+0000","last_change":"2023-12-19T08:05:10.785260+0000","last_active":"2023-12-19T08:05:10.791011+0000","last_peered":"2023-12-19T08:05:10.791011+0000","last_clean":"2023-12-19T08:05:10.791011+0000","last_became_active":"2023-12-19T08:05:10.784983+0000","last_became_peered":"2023-12-19T08:05:10.784983+0000","last_unstale":"2023-12-19T08:05:10.791011+0000","last_undegraded":"2023-12-19T08:05:10.791011+0000","last_fullsized":"2023-12-19T08:05:10.791011+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.783431+0000","last_change":"2023-12-19T08:05:10.783431+0000","last_active":"2023-12-19T08:05:10.783431+0000","last_peered":"2023-12-19T08:05:10.783431+0000","last_clean":"2023-12-19T08:05:10.783431+0000","last_became_active":"2023-12-19T08:05:10.783155+0000","last_became_peered":"2023-12-19T08:05:10.783155+0000","last_unstale":"2023-12-19T08:05:10.783431+0000","last_undegraded":"2023-12-19T08:05:10.783431+0000","last_fullsized":"2023-12-19T08:05:10.783431+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778468+0000","last_change":"2023-12-19T08:05:10.778468+0000","last_active":"2023-12-19T08:05:10.778468+0000","last_peered":"2023-12-19T08:05:10.778468+0000","last_clean":"2023-12-19T08:05:10.778468+0000","last_became_active":"2023-12-19T08:05:10.778156+0000","last_became_peered":"2023-12-19T08:05:10.778156+0000","last_unstale":"2023-12-19T08:05:10.778468+0000","last_undegraded":"2023-12-19T08:05:10.778468+0000","last_fullsized":"2023-12-19T08:05:10.778468+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:29.645164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780457+0000","last_change":"2023-12-19T08:05:10.776011+0000","last_active":"2023-12-19T08:05:10.780457+0000","last_peered":"2023-12-19T08:05:10.780457+0000","last_clean":"2023-12-19T08:05:10.780457+0000","last_became_active":"2023-12-19T08:05:10.775407+0000","last_became_peered":"2023-12-19T08:05:10.775407+0000","last_unstale":"2023-12-19T08:05:10.780457+0000","last_undegraded":"2023-12-19T08:05:10.780457+0000","last_fullsized":"2023-12-19T08:05:10.780457+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:26.239674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772983+0000","last_change":"2023-12-19T08:05:10.772983+0000","last_active":"2023-12-19T08:05:10.772983+0000","last_peered":"2023-12-19T08:05:10.772983+0000","last_clean":"2023-12-19T08:05:10.772983+0000","last_became_active":"2023-12-19T08:05:10.772689+0000","last_became_peered":"2023-12-19T08:05:10.772689+0000","last_unstale":"2023-12-19T08:05:10.772983+0000","last_undegraded":"2023-12-19T08:05:10.772983+0000","last_fullsized":"2023-12-19T08:05:10.772983+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:56:46.726157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780205+0000","last_change":"2023-12-19T08:05:10.780205+0000","last_active":"2023-12-19T08:05:10.780205+0000","last_peered":"2023-12-19T08:05:10.780205+0000","last_clean":"2023-12-19T08:05:10.780205+0000","last_became_active":"2023-12-19T08:05:10.779905+0000","last_became_peered":"2023-12-19T08:05:10.779905+0000","last_unstale":"2023-12-19T08:05:10.780205+0000","last_undegraded":"2023-12-19T08:05:10.780205+0000","last_fullsized":"2023-12-19T08:05:10.780205+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:07.921933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778411+0000","last_change":"2023-12-19T08:05:10.778411+0000","last_active":"2023-12-19T08:05:10.778411+0000","last_peered":"2023-12-19T08:05:10.778411+0000","last_clean":"2023-12-19T08:05:10.778411+0000","last_became_active":"2023-12-19T08:05:10.776251+0000","last_became_peered":"2023-12-19T08:05:10.776251+0000","last_unstale":"2023-12-19T08:05:10.778411+0000","last_undegraded":"2023-12-19T08:05:10.778411+0000","last_fullsized":"2023-12-19T08:05:10.778411+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:07:29.016139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790101+0000","last_change":"2023-12-19T08:05:09.755630+0000","last_active":"2023-12-19T08:05:10.790101+0000","last_peered":"2023-12-19T08:05:10.790101+0000","last_clean":"2023-12-19T08:05:10.790101+0000","last_became_active":"2023-12-19T08:05:09.755308+0000","last_became_peered":"2023-12-19T08:05:09.755308+0000","last_unstale":"2023-12-19T08:05:10.790101+0000","last_undegraded":"2023-12-19T08:05:10.790101+0000","last_fullsized":"2023-12-19T08:05:10.790101+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:25:55.379811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.766993+0000","last_change":"2023-12-19T08:05:10.766993+0000","last_active":"2023-12-19T08:05:10.766993+0000","last_peered":"2023-12-19T08:05:10.766993+0000","last_clean":"2023-12-19T08:05:10.766993+0000","last_became_active":"2023-12-19T08:05:10.766691+0000","last_became_peered":"2023-12-19T08:05:10.766691+0000","last_unstale":"2023-12-19T08:05:10.766993+0000","last_undegraded":"2023-12-19T08:05:10.766993+0000","last_fullsized":"2023-12-19T08:05:10.766993+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:10:00.156664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.767191+0000","last_change":"2023-12-19T08:05:09.759559+0000","last_active":"2023-12-19T08:05:10.767191+0000","last_peered":"2023-12-19T08:05:10.767191+0000","last_clean":"2023-12-19T08:05:10.767191+0000","last_became_active":"2023-12-19T08:05:09.759254+0000","last_became_peered":"2023-12-19T08:05:09.759254+0000","last_unstale":"2023-12-19T08:05:10.767191+0000","last_undegraded":"2023-12-19T08:05:10.767191+0000","last_fullsized":"2023-12-19T08:05:10.767191+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:04.314389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780438+0000","last_change":"2023-12-19T08:05:10.775818+0000","last_active":"2023-12-19T08:05:10.780438+0000","last_peered":"2023-12-19T08:05:10.780438+0000","last_clean":"2023-12-19T08:05:10.780438+0000","last_became_active":"2023-12-19T08:05:10.775043+0000","last_became_peered":"2023-12-19T08:05:10.775043+0000","last_unstale":"2023-12-19T08:05:10.780438+0000","last_undegraded":"2023-12-19T08:05:10.780438+0000","last_fullsized":"2023-12-19T08:05:10.780438+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:00.667330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.792071+0000","last_change":"2023-12-19T08:05:09.755987+0000","last_active":"2023-12-19T08:05:10.792071+0000","last_peered":"2023-12-19T08:05:10.792071+0000","last_clean":"2023-12-19T08:05:10.792071+0000","last_became_active":"2023-12-19T08:05:09.755658+0000","last_became_peered":"2023-12-19T08:05:09.755658+0000","last_unstale":"2023-12-19T08:05:10.792071+0000","last_undegraded":"2023-12-19T08:05:10.792071+0000","last_fullsized":"2023-12-19T08:05:10.792071+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:27.956229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782019+0000","last_change":"2023-12-19T08:05:10.773159+0000","last_active":"2023-12-19T08:05:10.782019+0000","last_peered":"2023-12-19T08:05:10.782019+0000","last_clean":"2023-12-19T08:05:10.782019+0000","last_became_active":"2023-12-19T08:05:10.772852+0000","last_became_peered":"2023-12-19T08:05:10.772852+0000","last_unstale":"2023-12-19T08:05:10.782019+0000","last_undegraded":"2023-12-19T08:05:10.782019+0000","last_fullsized":"2023-12-19T08:05:10.782019+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787068+0000","last_change":"2023-12-19T08:05:09.746463+0000","last_active":"2023-12-19T08:05:10.787068+0000","last_peered":"2023-12-19T08:05:10.787068+0000","last_clean":"2023-12-19T08:05:10.787068+0000","last_became_active":"2023-12-19T08:05:09.746144+0000","last_became_peered":"2023-12-19T08:05:09.746144+0000","last_unstale":"2023-12-19T08:05:10.787068+0000","last_undegraded":"2023-12-19T08:05:10.787068+0000","last_fullsized":"2023-12-19T08:05:10.787068+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:20.928370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771039+0000","last_change":"2023-12-19T08:05:10.771039+0000","last_active":"2023-12-19T08:05:10.771039+0000","last_peered":"2023-12-19T08:05:10.771039+0000","last_clean":"2023-12-19T08:05:10.771039+0000","last_became_active":"2023-12-19T08:05:10.770692+0000","last_became_peered":"2023-12-19T08:05:10.770692+0000","last_unstale":"2023-12-19T08:05:10.771039+0000","last_undegraded":"2023-12-19T08:05:10.771039+0000","last_fullsized":"2023-12-19T08:05:10.771039+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:47.089554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.760560+0000","last_change":"2023-12-19T08:05:09.763291+0000","last_active":"2023-12-19T08:05:10.760560+0000","last_peered":"2023-12-19T08:05:10.760560+0000","last_clean":"2023-12-19T08:05:10.760560+0000","last_became_active":"2023-12-19T08:05:09.763011+0000","last_became_peered":"2023-12-19T08:05:09.763011+0000","last_unstale":"2023-12-19T08:05:10.760560+0000","last_undegraded":"2023-12-19T08:05:10.760560+0000","last_fullsized":"2023-12-19T08:05:10.760560+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:58.690695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778597+0000","last_change":"2023-12-19T08:05:10.778597+0000","last_active":"2023-12-19T08:05:10.778597+0000","last_peered":"2023-12-19T08:05:10.778597+0000","last_clean":"2023-12-19T08:05:10.778597+0000","last_became_active":"2023-12-19T08:05:10.777912+0000","last_became_peered":"2023-12-19T08:05:10.777912+0000","last_unstale":"2023-12-19T08:05:10.778597+0000","last_undegraded":"2023-12-19T08:05:10.778597+0000","last_fullsized":"2023-12-19T08:05:10.778597+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:05.748640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790026+0000","last_change":"2023-12-19T08:05:09.758390+0000","last_active":"2023-12-19T08:05:10.790026+0000","last_peered":"2023-12-19T08:05:10.790026+0000","last_clean":"2023-12-19T08:05:10.790026+0000","last_became_active":"2023-12-19T08:05:09.758085+0000","last_became_peered":"2023-12-19T08:05:09.758085+0000","last_unstale":"2023-12-19T08:05:10.790026+0000","last_undegraded":"2023-12-19T08:05:10.790026+0000","last_fullsized":"2023-12-19T08:05:10.790026+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:10.432397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793680+0000","last_change":"2023-12-19T08:05:10.793680+0000","last_active":"2023-12-19T08:05:10.793680+0000","last_peered":"2023-12-19T08:05:10.793680+0000","last_clean":"2023-12-19T08:05:10.793680+0000","last_became_active":"2023-12-19T08:05:10.793273+0000","last_became_peered":"2023-12-19T08:05:10.793273+0000","last_unstale":"2023-12-19T08:05:10.793680+0000","last_undegraded":"2023-12-19T08:05:10.793680+0000","last_fullsized":"2023-12-19T08:05:10.793680+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:10.127412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.754943+0000","last_change":"2023-12-19T08:05:09.755954+0000","last_active":"2023-12-19T08:05:10.754943+0000","last_peered":"2023-12-19T08:05:10.754943+0000","last_clean":"2023-12-19T08:05:10.754943+0000","last_became_active":"2023-12-19T08:05:09.755559+0000","last_became_peered":"2023-12-19T08:05:09.755559+0000","last_unstale":"2023-12-19T08:05:10.754943+0000","last_undegraded":"2023-12-19T08:05:10.754943+0000","last_fullsized":"2023-12-19T08:05:10.754943+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:09.115449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771141+0000","last_change":"2023-12-19T08:05:10.771141+0000","last_active":"2023-12-19T08:05:10.771141+0000","last_peered":"2023-12-19T08:05:10.771141+0000","last_clean":"2023-12-19T08:05:10.771141+0000","last_became_active":"2023-12-19T08:05:10.770596+0000","last_became_peered":"2023-12-19T08:05:10.770596+0000","last_unstale":"2023-12-19T08:05:10.771141+0000","last_undegraded":"2023-12-19T08:05:10.771141+0000","last_fullsized":"2023-12-19T08:05:10.771141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:59.230868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.747950+0000","last_change":"2023-12-19T08:05:09.743271+0000","last_active":"2023-12-19T08:05:09.747950+0000","last_peered":"2023-12-19T08:05:09.747950+0000","last_clean":"2023-12-19T08:05:09.747950+0000","last_became_active":"2023-12-19T08:05:09.742927+0000","last_became_peered":"2023-12-19T08:05:09.742927+0000","last_unstale":"2023-12-19T08:05:09.747950+0000","last_undegraded":"2023-12-19T08:05:09.747950+0000","last_fullsized":"2023-12-19T08:05:09.747950+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:11.961192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778867+0000","last_change":"2023-12-19T08:05:10.778867+0000","last_active":"2023-12-19T08:05:10.778867+0000","last_peered":"2023-12-19T08:05:10.778867+0000","last_clean":"2023-12-19T08:05:10.778867+0000","last_became_active":"2023-12-19T08:05:10.778562+0000","last_became_peered":"2023-12-19T08:05:10.778562+0000","last_unstale":"2023-12-19T08:05:10.778867+0000","last_undegraded":"2023-12-19T08:05:10.778867+0000","last_fullsized":"2023-12-19T08:05:10.778867+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:56.176007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.755551+0000","last_change":"2023-12-19T08:05:09.755907+0000","last_active":"2023-12-19T08:05:10.755551+0000","last_peered":"2023-12-19T08:05:10.755551+0000","last_clean":"2023-12-19T08:05:10.755551+0000","last_became_active":"2023-12-19T08:05:09.755575+0000","last_became_peered":"2023-12-19T08:05:09.755575+0000","last_unstale":"2023-12-19T08:05:10.755551+0000","last_undegraded":"2023-12-19T08:05:10.755551+0000","last_fullsized":"2023-12-19T08:05:10.755551+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:21:03.789828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787587+0000","last_change":"2023-12-19T08:05:10.779996+0000","last_active":"2023-12-19T08:05:10.787587+0000","last_peered":"2023-12-19T08:05:10.787587+0000","last_clean":"2023-12-19T08:05:10.787587+0000","last_became_active":"2023-12-19T08:05:10.779678+0000","last_became_peered":"2023-12-19T08:05:10.779678+0000","last_unstale":"2023-12-19T08:05:10.787587+0000","last_undegraded":"2023-12-19T08:05:10.787587+0000","last_fullsized":"2023-12-19T08:05:10.787587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.757324+0000","last_change":"2023-12-19T08:05:09.753931+0000","last_active":"2023-12-19T08:05:09.757324+0000","last_peered":"2023-12-19T08:05:09.757324+0000","last_clean":"2023-12-19T08:05:09.757324+0000","last_became_active":"2023-12-19T08:05:09.753586+0000","last_became_peered":"2023-12-19T08:05:09.753586+0000","last_unstale":"2023-12-19T08:05:09.757324+0000","last_undegraded":"2023-12-19T08:05:09.757324+0000","last_fullsized":"2023-12-19T08:05:09.757324+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:25.227711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771169+0000","last_change":"2023-12-19T08:05:10.771169+0000","last_active":"2023-12-19T08:05:10.771169+0000","last_peered":"2023-12-19T08:05:10.771169+0000","last_clean":"2023-12-19T08:05:10.771169+0000","last_became_active":"2023-12-19T08:05:10.770894+0000","last_became_peered":"2023-12-19T08:05:10.770894+0000","last_unstale":"2023-12-19T08:05:10.771169+0000","last_undegraded":"2023-12-19T08:05:10.771169+0000","last_fullsized":"2023-12-19T08:05:10.771169+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:52.574569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793945+0000","last_change":"2023-12-19T08:05:09.760406+0000","last_active":"2023-12-19T08:05:10.793945+0000","last_peered":"2023-12-19T08:05:10.793945+0000","last_clean":"2023-12-19T08:05:10.793945+0000","last_became_active":"2023-12-19T08:05:09.760133+0000","last_became_peered":"2023-12-19T08:05:09.760133+0000","last_unstale":"2023-12-19T08:05:10.793945+0000","last_undegraded":"2023-12-19T08:05:10.793945+0000","last_fullsized":"2023-12-19T08:05:10.793945+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:52.082968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787958+0000","last_change":"2023-12-19T08:05:10.782245+0000","last_active":"2023-12-19T08:05:10.787958+0000","last_peered":"2023-12-19T08:05:10.787958+0000","last_clean":"2023-12-19T08:05:10.787958+0000","last_became_active":"2023-12-19T08:05:10.781962+0000","last_became_peered":"2023-12-19T08:05:10.781962+0000","last_unstale":"2023-12-19T08:05:10.787958+0000","last_undegraded":"2023-12-19T08:05:10.787958+0000","last_fullsized":"2023-12-19T08:05:10.787958+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.796687+0000","last_change":"2023-12-19T08:05:09.763870+0000","last_active":"2023-12-19T08:05:10.796687+0000","last_peered":"2023-12-19T08:05:10.796687+0000","last_clean":"2023-12-19T08:05:10.796687+0000","last_became_active":"2023-12-19T08:05:09.763573+0000","last_became_peered":"2023-12-19T08:05:09.763573+0000","last_unstale":"2023-12-19T08:05:10.796687+0000","last_undegraded":"2023-12-19T08:05:10.796687+0000","last_fullsized":"2023-12-19T08:05:10.796687+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:11.020493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793501+0000","last_change":"2023-12-19T08:05:10.790316+0000","last_active":"2023-12-19T08:05:10.793501+0000","last_peered":"2023-12-19T08:05:10.793501+0000","last_clean":"2023-12-19T08:05:10.793501+0000","last_became_active":"2023-12-19T08:05:10.790006+0000","last_became_peered":"2023-12-19T08:05:10.790006+0000","last_unstale":"2023-12-19T08:05:10.793501+0000","last_undegraded":"2023-12-19T08:05:10.793501+0000","last_fullsized":"2023-12-19T08:05:10.793501+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790183+0000","last_change":"2023-12-19T08:05:09.755432+0000","last_active":"2023-12-19T08:05:10.790183+0000","last_peered":"2023-12-19T08:05:10.790183+0000","last_clean":"2023-12-19T08:05:10.790183+0000","last_became_active":"2023-12-19T08:05:09.755099+0000","last_became_peered":"2023-12-19T08:05:09.755099+0000","last_unstale":"2023-12-19T08:05:10.790183+0000","last_undegraded":"2023-12-19T08:05:10.790183+0000","last_fullsized":"2023-12-19T08:05:10.790183+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:15:23.970857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776252+0000","last_change":"2023-12-19T08:05:10.776252+0000","last_active":"2023-12-19T08:05:10.776252+0000","last_peered":"2023-12-19T08:05:10.776252+0000","last_clean":"2023-12-19T08:05:10.776252+0000","last_became_active":"2023-12-19T08:05:10.775947+0000","last_became_peered":"2023-12-19T08:05:10.775947+0000","last_unstale":"2023-12-19T08:05:10.776252+0000","last_undegraded":"2023-12-19T08:05:10.776252+0000","last_fullsized":"2023-12-19T08:05:10.776252+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:59.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.754046+0000","last_change":"2023-12-19T08:05:09.749720+0000","last_active":"2023-12-19T08:05:09.754046+0000","last_peered":"2023-12-19T08:05:09.754046+0000","last_clean":"2023-12-19T08:05:09.754046+0000","last_became_active":"2023-12-19T08:05:09.749444+0000","last_became_peered":"2023-12-19T08:05:09.749444+0000","last_unstale":"2023-12-19T08:05:09.754046+0000","last_undegraded":"2023-12-19T08:05:09.754046+0000","last_fullsized":"2023-12-19T08:05:09.754046+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:44.364952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773965+0000","last_change":"2023-12-19T08:05:10.773965+0000","last_active":"2023-12-19T08:05:10.773965+0000","last_peered":"2023-12-19T08:05:10.773965+0000","last_clean":"2023-12-19T08:05:10.773965+0000","last_became_active":"2023-12-19T08:05:10.773678+0000","last_became_peered":"2023-12-19T08:05:10.773678+0000","last_unstale":"2023-12-19T08:05:10.773965+0000","last_undegraded":"2023-12-19T08:05:10.773965+0000","last_fullsized":"2023-12-19T08:05:10.773965+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:08.185509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.758384+0000","last_change":"2023-12-19T08:05:09.751013+0000","last_active":"2023-12-19T08:05:09.758384+0000","last_peered":"2023-12-19T08:05:09.758384+0000","last_clean":"2023-12-19T08:05:09.758384+0000","last_became_active":"2023-12-19T08:05:09.748978+0000","last_became_peered":"2023-12-19T08:05:09.748978+0000","last_unstale":"2023-12-19T08:05:09.758384+0000","last_undegraded":"2023-12-19T08:05:09.758384+0000","last_fullsized":"2023-12-19T08:05:09.758384+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:04:08.099960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769807+0000","last_change":"2023-12-19T08:05:10.769807+0000","last_active":"2023-12-19T08:05:10.769807+0000","last_peered":"2023-12-19T08:05:10.769807+0000","last_clean":"2023-12-19T08:05:10.769807+0000","last_became_active":"2023-12-19T08:05:10.769523+0000","last_became_peered":"2023-12-19T08:05:10.769523+0000","last_unstale":"2023-12-19T08:05:10.769807+0000","last_undegraded":"2023-12-19T08:05:10.769807+0000","last_fullsized":"2023-12-19T08:05:10.769807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:38.581216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.792055+0000","last_change":"2023-12-19T08:05:09.753610+0000","last_active":"2023-12-19T08:05:10.792055+0000","last_peered":"2023-12-19T08:05:10.792055+0000","last_clean":"2023-12-19T08:05:10.792055+0000","last_became_active":"2023-12-19T08:05:09.753326+0000","last_became_peered":"2023-12-19T08:05:09.753326+0000","last_unstale":"2023-12-19T08:05:10.792055+0000","last_undegraded":"2023-12-19T08:05:10.792055+0000","last_fullsized":"2023-12-19T08:05:10.792055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:58.690695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.767947+0000","last_change":"2023-12-19T08:05:10.767947+0000","last_active":"2023-12-19T08:05:10.767947+0000","last_peered":"2023-12-19T08:05:10.767947+0000","last_clean":"2023-12-19T08:05:10.767947+0000","last_became_active":"2023-12-19T08:05:10.767638+0000","last_became_peered":"2023-12-19T08:05:10.767638+0000","last_unstale":"2023-12-19T08:05:10.767947+0000","last_undegraded":"2023-12-19T08:05:10.767947+0000","last_fullsized":"2023-12-19T08:05:10.767947+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:58.006252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.766770+0000","last_change":"2023-12-19T08:05:09.758625+0000","last_active":"2023-12-19T08:05:10.766770+0000","last_peered":"2023-12-19T08:05:10.766770+0000","last_clean":"2023-12-19T08:05:10.766770+0000","last_became_active":"2023-12-19T08:05:09.758259+0000","last_became_peered":"2023-12-19T08:05:09.758259+0000","last_unstale":"2023-12-19T08:05:10.766770+0000","last_undegraded":"2023-12-19T08:05:10.766770+0000","last_fullsized":"2023-12-19T08:05:10.766770+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:52:38.660192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772611+0000","last_change":"2023-12-19T08:05:10.772611+0000","last_active":"2023-12-19T08:05:10.772611+0000","last_peered":"2023-12-19T08:05:10.772611+0000","last_clean":"2023-12-19T08:05:10.772611+0000","last_became_active":"2023-12-19T08:05:10.772332+0000","last_became_peered":"2023-12-19T08:05:10.772332+0000","last_unstale":"2023-12-19T08:05:10.772611+0000","last_undegraded":"2023-12-19T08:05:10.772611+0000","last_fullsized":"2023-12-19T08:05:10.772611+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:28.968192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.763626+0000","last_change":"2023-12-19T08:05:09.761029+0000","last_active":"2023-12-19T08:05:10.763626+0000","last_peered":"2023-12-19T08:05:10.763626+0000","last_clean":"2023-12-19T08:05:10.763626+0000","last_became_active":"2023-12-19T08:05:09.760710+0000","last_became_peered":"2023-12-19T08:05:09.760710+0000","last_unstale":"2023-12-19T08:05:10.763626+0000","last_undegraded":"2023-12-19T08:05:10.763626+0000","last_fullsized":"2023-12-19T08:05:10.763626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:09.115449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.765517+0000","last_change":"2023-12-19T08:05:10.761050+0000","last_active":"2023-12-19T08:05:10.765517+0000","last_peered":"2023-12-19T08:05:10.765517+0000","last_clean":"2023-12-19T08:05:10.765517+0000","last_became_active":"2023-12-19T08:05:10.760201+0000","last_became_peered":"2023-12-19T08:05:10.760201+0000","last_unstale":"2023-12-19T08:05:10.765517+0000","last_undegraded":"2023-12-19T08:05:10.765517+0000","last_fullsized":"2023-12-19T08:05:10.765517+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768258+0000","last_change":"2023-12-19T08:05:09.765165+0000","last_active":"2023-12-19T08:05:10.768258+0000","last_peered":"2023-12-19T08:05:10.768258+0000","last_clean":"2023-12-19T08:05:10.768258+0000","last_became_active":"2023-12-19T08:05:09.764156+0000","last_became_peered":"2023-12-19T08:05:09.764156+0000","last_unstale":"2023-12-19T08:05:10.768258+0000","last_undegraded":"2023-12-19T08:05:10.768258+0000","last_fullsized":"2023-12-19T08:05:10.768258+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:20.928370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.764990+0000","last_change":"2023-12-19T08:05:10.764990+0000","last_active":"2023-12-19T08:05:10.764990+0000","last_peered":"2023-12-19T08:05:10.764990+0000","last_clean":"2023-12-19T08:05:10.764990+0000","last_became_active":"2023-12-19T08:05:10.764634+0000","last_became_peered":"2023-12-19T08:05:10.764634+0000","last_unstale":"2023-12-19T08:05:10.764990+0000","last_undegraded":"2023-12-19T08:05:10.764990+0000","last_fullsized":"2023-12-19T08:05:10.764990+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:46:03.707714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.788438+0000","last_change":"2023-12-19T08:05:09.754825+0000","last_active":"2023-12-19T08:05:10.788438+0000","last_peered":"2023-12-19T08:05:10.788438+0000","last_clean":"2023-12-19T08:05:10.788438+0000","last_became_active":"2023-12-19T08:05:09.754484+0000","last_became_peered":"2023-12-19T08:05:09.754484+0000","last_unstale":"2023-12-19T08:05:10.788438+0000","last_undegraded":"2023-12-19T08:05:10.788438+0000","last_fullsized":"2023-12-19T08:05:10.788438+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:29:00.220530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768967+0000","last_change":"2023-12-19T08:05:10.768967+0000","last_active":"2023-12-19T08:05:10.768967+0000","last_peered":"2023-12-19T08:05:10.768967+0000","last_clean":"2023-12-19T08:05:10.768967+0000","last_became_active":"2023-12-19T08:05:10.768651+0000","last_became_peered":"2023-12-19T08:05:10.768651+0000","last_unstale":"2023-12-19T08:05:10.768967+0000","last_undegraded":"2023-12-19T08:05:10.768967+0000","last_fullsized":"2023-12-19T08:05:10.768967+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:25:32.228235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.761259+0000","last_change":"2023-12-19T08:05:09.757787+0000","last_active":"2023-12-19T08:05:09.761259+0000","last_peered":"2023-12-19T08:05:09.761259+0000","last_clean":"2023-12-19T08:05:09.761259+0000","last_became_active":"2023-12-19T08:05:09.757517+0000","last_became_peered":"2023-12-19T08:05:09.757517+0000","last_unstale":"2023-12-19T08:05:09.761259+0000","last_undegraded":"2023-12-19T08:05:09.761259+0000","last_fullsized":"2023-12-19T08:05:09.761259+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:00.843396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776792+0000","last_change":"2023-12-19T08:05:10.776792+0000","last_active":"2023-12-19T08:05:10.776792+0000","last_peered":"2023-12-19T08:05:10.776792+0000","last_clean":"2023-12-19T08:05:10.776792+0000","last_became_active":"2023-12-19T08:05:10.776509+0000","last_became_peered":"2023-12-19T08:05:10.776509+0000","last_unstale":"2023-12-19T08:05:10.776792+0000","last_undegraded":"2023-12-19T08:05:10.776792+0000","last_fullsized":"2023-12-19T08:05:10.776792+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:56.176007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.762272+0000","last_change":"2023-12-19T08:05:09.759657+0000","last_active":"2023-12-19T08:05:09.762272+0000","last_peered":"2023-12-19T08:05:09.762272+0000","last_clean":"2023-12-19T08:05:09.762272+0000","last_became_active":"2023-12-19T08:05:09.757285+0000","last_became_peered":"2023-12-19T08:05:09.757285+0000","last_unstale":"2023-12-19T08:05:09.762272+0000","last_undegraded":"2023-12-19T08:05:09.762272+0000","last_fullsized":"2023-12-19T08:05:09.762272+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:07:10.755579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772239+0000","last_change":"2023-12-19T08:05:10.772239+0000","last_active":"2023-12-19T08:05:10.772239+0000","last_peered":"2023-12-19T08:05:10.772239+0000","last_clean":"2023-12-19T08:05:10.772239+0000","last_became_active":"2023-12-19T08:05:10.771953+0000","last_became_peered":"2023-12-19T08:05:10.771953+0000","last_unstale":"2023-12-19T08:05:10.772239+0000","last_undegraded":"2023-12-19T08:05:10.772239+0000","last_fullsized":"2023-12-19T08:05:10.772239+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:02:59.883271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.761314+0000","last_change":"2023-12-19T08:05:09.758304+0000","last_active":"2023-12-19T08:05:09.761314+0000","last_peered":"2023-12-19T08:05:09.761314+0000","last_clean":"2023-12-19T08:05:09.761314+0000","last_became_active":"2023-12-19T08:05:09.758016+0000","last_became_peered":"2023-12-19T08:05:09.758016+0000","last_unstale":"2023-12-19T08:05:09.761314+0000","last_undegraded":"2023-12-19T08:05:09.761314+0000","last_fullsized":"2023-12-19T08:05:09.761314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:18.623976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768293+0000","last_change":"2023-12-19T08:05:10.768293+0000","last_active":"2023-12-19T08:05:10.768293+0000","last_peered":"2023-12-19T08:05:10.768293+0000","last_clean":"2023-12-19T08:05:10.768293+0000","last_became_active":"2023-12-19T08:05:10.768011+0000","last_became_peered":"2023-12-19T08:05:10.768011+0000","last_unstale":"2023-12-19T08:05:10.768293+0000","last_undegraded":"2023-12-19T08:05:10.768293+0000","last_fullsized":"2023-12-19T08:05:10.768293+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:47.996715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.788658+0000","last_change":"2023-12-19T08:05:09.750171+0000","last_active":"2023-12-19T08:05:10.788658+0000","last_peered":"2023-12-19T08:05:10.788658+0000","last_clean":"2023-12-19T08:05:10.788658+0000","last_became_active":"2023-12-19T08:05:09.748948+0000","last_became_peered":"2023-12-19T08:05:09.748948+0000","last_unstale":"2023-12-19T08:05:10.788658+0000","last_undegraded":"2023-12-19T08:05:10.788658+0000","last_fullsized":"2023-12-19T08:05:10.788658+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:34.777001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777094+0000","last_change":"2023-12-19T08:05:10.771680+0000","last_active":"2023-12-19T08:05:10.777094+0000","last_peered":"2023-12-19T08:05:10.777094+0000","last_clean":"2023-12-19T08:05:10.777094+0000","last_became_active":"2023-12-19T08:05:10.771387+0000","last_became_peered":"2023-12-19T08:05:10.771387+0000","last_unstale":"2023-12-19T08:05:10.777094+0000","last_undegraded":"2023-12-19T08:05:10.777094+0000","last_fullsized":"2023-12-19T08:05:10.777094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:32.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.757304+0000","last_change":"2023-12-19T08:05:09.750759+0000","last_active":"2023-12-19T08:05:09.757304+0000","last_peered":"2023-12-19T08:05:09.757304+0000","last_clean":"2023-12-19T08:05:09.757304+0000","last_became_active":"2023-12-19T08:05:09.749668+0000","last_became_peered":"2023-12-19T08:05:09.749668+0000","last_unstale":"2023-12-19T08:05:09.757304+0000","last_undegraded":"2023-12-19T08:05:09.757304+0000","last_fullsized":"2023-12-19T08:05:09.757304+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:41.999018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778113+0000","last_change":"2023-12-19T08:05:10.778113+0000","last_active":"2023-12-19T08:05:10.778113+0000","last_peered":"2023-12-19T08:05:10.778113+0000","last_clean":"2023-12-19T08:05:10.778113+0000","last_became_active":"2023-12-19T08:05:10.776391+0000","last_became_peered":"2023-12-19T08:05:10.776391+0000","last_unstale":"2023-12-19T08:05:10.778113+0000","last_undegraded":"2023-12-19T08:05:10.778113+0000","last_fullsized":"2023-12-19T08:05:10.778113+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:05.748640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.764281+0000","last_change":"2023-12-19T08:05:09.761947+0000","last_active":"2023-12-19T08:05:09.764281+0000","last_peered":"2023-12-19T08:05:09.764281+0000","last_clean":"2023-12-19T08:05:09.764281+0000","last_became_active":"2023-12-19T08:05:09.761624+0000","last_became_peered":"2023-12-19T08:05:09.761624+0000","last_unstale":"2023-12-19T08:05:09.764281+0000","last_undegraded":"2023-12-19T08:05:09.764281+0000","last_fullsized":"2023-12-19T08:05:09.764281+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:34.273452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778150+0000","last_change":"2023-12-19T08:05:10.771421+0000","last_active":"2023-12-19T08:05:10.778150+0000","last_peered":"2023-12-19T08:05:10.778150+0000","last_clean":"2023-12-19T08:05:10.778150+0000","last_became_active":"2023-12-19T08:05:10.771051+0000","last_became_peered":"2023-12-19T08:05:10.771051+0000","last_unstale":"2023-12-19T08:05:10.778150+0000","last_undegraded":"2023-12-19T08:05:10.778150+0000","last_fullsized":"2023-12-19T08:05:10.778150+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.746373+0000","last_change":"2023-12-19T08:05:09.755735+0000","last_active":"2023-12-19T08:05:10.746373+0000","last_peered":"2023-12-19T08:05:10.746373+0000","last_clean":"2023-12-19T08:05:10.746373+0000","last_became_active":"2023-12-19T08:05:09.755429+0000","last_became_peered":"2023-12-19T08:05:09.755429+0000","last_unstale":"2023-12-19T08:05:10.746373+0000","last_undegraded":"2023-12-19T08:05:10.746373+0000","last_fullsized":"2023-12-19T08:05:10.746373+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:10.432397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777206+0000","last_change":"2023-12-19T08:05:10.777206+0000","last_active":"2023-12-19T08:05:10.777206+0000","last_peered":"2023-12-19T08:05:10.777206+0000","last_clean":"2023-12-19T08:05:10.777206+0000","last_became_active":"2023-12-19T08:05:10.776907+0000","last_became_peered":"2023-12-19T08:05:10.776907+0000","last_unstale":"2023-12-19T08:05:10.777206+0000","last_undegraded":"2023-12-19T08:05:10.777206+0000","last_fullsized":"2023-12-19T08:05:10.777206+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:20.721351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.755183+0000","last_change":"2023-12-19T08:05:09.758631+0000","last_active":"2023-12-19T08:05:10.755183+0000","last_peered":"2023-12-19T08:05:10.755183+0000","last_clean":"2023-12-19T08:05:10.755183+0000","last_became_active":"2023-12-19T08:05:09.758326+0000","last_became_peered":"2023-12-19T08:05:09.758326+0000","last_unstale":"2023-12-19T08:05:10.755183+0000","last_undegraded":"2023-12-19T08:05:10.755183+0000","last_fullsized":"2023-12-19T08:05:10.755183+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:52.082968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774231+0000","last_change":"2023-12-19T08:05:10.774231+0000","last_active":"2023-12-19T08:05:10.774231+0000","last_peered":"2023-12-19T08:05:10.774231+0000","last_clean":"2023-12-19T08:05:10.774231+0000","last_became_active":"2023-12-19T08:05:10.772602+0000","last_became_peered":"2023-12-19T08:05:10.772602+0000","last_unstale":"2023-12-19T08:05:10.774231+0000","last_undegraded":"2023-12-19T08:05:10.774231+0000","last_fullsized":"2023-12-19T08:05:10.774231+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:53:40.273343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.758344+0000","last_change":"2023-12-19T08:05:09.752552+0000","last_active":"2023-12-19T08:05:09.758344+0000","last_peered":"2023-12-19T08:05:09.758344+0000","last_clean":"2023-12-19T08:05:09.758344+0000","last_became_active":"2023-12-19T08:05:09.752177+0000","last_became_peered":"2023-12-19T08:05:09.752177+0000","last_unstale":"2023-12-19T08:05:09.758344+0000","last_undegraded":"2023-12-19T08:05:09.758344+0000","last_fullsized":"2023-12-19T08:05:09.758344+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:56.165286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782145+0000","last_change":"2023-12-19T08:05:10.782145+0000","last_active":"2023-12-19T08:05:10.782145+0000","last_peered":"2023-12-19T08:05:10.782145+0000","last_clean":"2023-12-19T08:05:10.782145+0000","last_became_active":"2023-12-19T08:05:10.779891+0000","last_became_peered":"2023-12-19T08:05:10.779891+0000","last_unstale":"2023-12-19T08:05:10.782145+0000","last_undegraded":"2023-12-19T08:05:10.782145+0000","last_fullsized":"2023-12-19T08:05:10.782145+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.763540+0000","last_change":"2023-12-19T08:05:09.761593+0000","last_active":"2023-12-19T08:05:09.763540+0000","last_peered":"2023-12-19T08:05:09.763540+0000","last_clean":"2023-12-19T08:05:09.763540+0000","last_became_active":"2023-12-19T08:05:09.761305+0000","last_became_peered":"2023-12-19T08:05:09.761305+0000","last_unstale":"2023-12-19T08:05:09.763540+0000","last_undegraded":"2023-12-19T08:05:09.763540+0000","last_fullsized":"2023-12-19T08:05:09.763540+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:58.692928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.786190+0000","last_change":"2023-12-19T08:05:10.779392+0000","last_active":"2023-12-19T08:05:10.786190+0000","last_peered":"2023-12-19T08:05:10.786190+0000","last_clean":"2023-12-19T08:05:10.786190+0000","last_became_active":"2023-12-19T08:05:10.779120+0000","last_became_peered":"2023-12-19T08:05:10.779120+0000","last_unstale":"2023-12-19T08:05:10.786190+0000","last_undegraded":"2023-12-19T08:05:10.786190+0000","last_fullsized":"2023-12-19T08:05:10.786190+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.760590+0000","last_change":"2023-12-19T08:05:09.755414+0000","last_active":"2023-12-19T08:05:09.760590+0000","last_peered":"2023-12-19T08:05:09.760590+0000","last_clean":"2023-12-19T08:05:09.760590+0000","last_became_active":"2023-12-19T08:05:09.755149+0000","last_became_peered":"2023-12-19T08:05:09.755149+0000","last_unstale":"2023-12-19T08:05:09.760590+0000","last_undegraded":"2023-12-19T08:05:09.760590+0000","last_fullsized":"2023-12-19T08:05:09.760590+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:49.412606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793525+0000","last_change":"2023-12-19T08:05:10.790478+0000","last_active":"2023-12-19T08:05:10.793525+0000","last_peered":"2023-12-19T08:05:10.793525+0000","last_clean":"2023-12-19T08:05:10.793525+0000","last_became_active":"2023-12-19T08:05:10.790177+0000","last_became_peered":"2023-12-19T08:05:10.790177+0000","last_unstale":"2023-12-19T08:05:10.793525+0000","last_undegraded":"2023-12-19T08:05:10.793525+0000","last_fullsized":"2023-12-19T08:05:10.793525+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790594+0000","last_change":"2023-12-19T08:05:09.759704+0000","last_active":"2023-12-19T08:05:10.790594+0000","last_peered":"2023-12-19T08:05:10.790594+0000","last_clean":"2023-12-19T08:05:10.790594+0000","last_became_active":"2023-12-19T08:05:09.759290+0000","last_became_peered":"2023-12-19T08:05:09.759290+0000","last_unstale":"2023-12-19T08:05:10.790594+0000","last_undegraded":"2023-12-19T08:05:10.790594+0000","last_fullsized":"2023-12-19T08:05:10.790594+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:33:08.436215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'227","reported_seq":641,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.750072+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:05:09.750072+0000","last_peered":"2023-12-19T08:05:09.750072+0000","last_clean":"2023-12-19T08:05:09.750072+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:05:09.750072+0000","last_undegraded":"2023-12-19T08:05:09.750072+0000","last_fullsized":"2023-12-19T08:05:09.750072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1419254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2170000000000001,"15min":1.2509999999999999},"min":{"1min":0.72399999999999998,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.524,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.258,"5min":1.2769999999999999,"15min":1.274},"min":{"1min":0.95399999999999996,"5min":0.749,"15min":0.70799999999999996},"max":{"1min":1.5800000000000001,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":0.82299999999999995}]},{"osd":1,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.161,"15min":1.1639999999999999},"min":{"1min":0.71399999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.464,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.204,"15min":1.1819999999999999},"min":{"1min":0.85599999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.833,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.345}]},{"osd":2,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.163,"15min":1.165},"min":{"1min":0.82699999999999996,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.3540000000000001,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.2889999999999999,"15min":1.256},"min":{"1min":0.85499999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.946,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.236}]},{"osd":3,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2849999999999999,"15min":1.292},"min":{"1min":0.90100000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.5509999999999999,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.319},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.353},"min":{"1min":1.099,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5509999999999999,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.3879999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.3260000000000001,"15min":1.2350000000000001},"min":{"1min":0.81499999999999995,"5min":0.71899999999999997,"15min":0.67300000000000004},"max":{"1min":2.0419999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.3069999999999999,"15min":1.242},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.546,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.4319999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.335,"15min":1.2509999999999999},"min":{"1min":0.79600000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.657,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.536},{"interface":"front","average":{"1min":1.383,"5min":1.4339999999999999,"15min":1.369},"min":{"1min":1.016,"5min":0.751,"15min":0.751},"max":{"1min":2.1219999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.524}]},{"osd":6,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.51,"15min":1.4259999999999999},"min":{"1min":1.0629999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7649999999999999,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.617},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.4370000000000001,"15min":1.385},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.58299999999999996},"max":{"1min":1.7210000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.5069999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575061,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302952,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068888,"statfs":{"total":96636764160,"available":96326541312,"internally_reserved":0,"allocated":1605632,"data_stored":830838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.085,"15min":1.099},"min":{"1min":0.80400000000000005,"5min":0.70699999999999996,"15min":0.36799999999999999},"max":{"1min":1.48,"5min":1.5569999999999999,"15min":1.665},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1419999999999999,"15min":1.137},"min":{"1min":0.92400000000000004,"5min":0.624,"15min":0.49399999999999999},"max":{"1min":1.734,"5min":1.756,"15min":1.756},"last":1.173}]},{"osd":1,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.149,"15min":1.1559999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53400000000000003},"max":{"1min":1.552,"5min":1.675,"15min":1.675},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.119,"15min":1.1519999999999999},"min":{"1min":0.84199999999999997,"5min":0.73799999999999999,"15min":0.51900000000000002},"max":{"1min":1.4299999999999999,"5min":1.5349999999999999,"15min":1.7470000000000001},"last":0.84599999999999997}]},{"osd":2,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2150000000000001,"15min":1.2330000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.59099999999999997},"max":{"1min":1.5589999999999999,"5min":1.746,"15min":1.847},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.2150000000000001,"15min":1.2130000000000001},"min":{"1min":0.94999999999999996,"5min":0.70799999999999996,"15min":0.58999999999999997},"max":{"1min":1.6200000000000001,"5min":1.694,"15min":1.7929999999999999},"last":0.89300000000000002}]},{"osd":3,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.234,"15min":1.2629999999999999},"min":{"1min":0.83699999999999997,"5min":0.77200000000000002,"15min":0.65400000000000003},"max":{"1min":1.6519999999999999,"5min":1.8200000000000001,"15min":1.958},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.302,"15min":1.304},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.63200000000000001},"max":{"1min":1.6060000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.361}]},{"osd":4,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.47199999999999998},"max":{"1min":1.546,"5min":1.7130000000000001,"15min":1.8480000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.389,"15min":1.395},"min":{"1min":0.96799999999999997,"5min":0.72299999999999998,"15min":0.70099999999999996},"max":{"1min":1.843,"5min":1.974,"15min":2.1440000000000001},"last":1.119}]},{"osd":5,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.288},"min":{"1min":0.81799999999999995,"5min":0.70799999999999996,"15min":0.67500000000000004},"max":{"1min":1.6950000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.413},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3109999999999999,"15min":1.296},"min":{"1min":0.77600000000000002,"5min":0.75700000000000001,"15min":0.56499999999999995},"max":{"1min":1.6200000000000001,"5min":1.8100000000000001,"15min":2.2450000000000001},"last":1.095}]},{"osd":7,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.399,"15min":1.3700000000000001},"min":{"1min":0.90700000000000003,"5min":0.73599999999999999,"15min":0.55100000000000005},"max":{"1min":1.8049999999999999,"5min":1.9239999999999999,"15min":2.04},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4550000000000001,"15min":1.4219999999999999},"min":{"1min":1.048,"5min":0.76600000000000001,"15min":0.58099999999999996},"max":{"1min":1.835,"5min":2.137,"15min":2.137},"last":1.4610000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575061,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298848,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072992,"statfs":{"total":96636764160,"available":96330743808,"internally_reserved":0,"allocated":1597440,"data_stored":835854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2050000000000001,"15min":1.2390000000000001},"min":{"1min":1.0649999999999999,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.6140000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":1.0700000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.538,"5min":1.8700000000000001,"15min":1.954},"last":0.88400000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1140000000000001,"15min":1.1399999999999999},"min":{"1min":0.89900000000000002,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.5620000000000001,"5min":1.585,"15min":1.958},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.173,"15min":1.1819999999999999},"min":{"1min":0.91400000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.627,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.097}]},{"osd":2,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.204,"15min":1.204},"min":{"1min":0.85499999999999998,"5min":0.72899999999999998,"15min":0.68500000000000005},"max":{"1min":1.571,"5min":1.821,"15min":2.0259999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.88700000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6499999999999999,"5min":1.6679999999999999,"15min":1.7490000000000001},"last":1.137}]},{"osd":3,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2430000000000001,"15min":1.254},"min":{"1min":0.92700000000000005,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.593,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.327,"15min":1.3129999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.76600000000000001},"max":{"1min":1.6379999999999999,"5min":2.1560000000000001,"15min":2.2130000000000001},"last":1.0569999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.1970000000000001,"15min":1.1990000000000001},"min":{"1min":0.86899999999999999,"5min":0.629,"15min":0.629},"max":{"1min":1.6970000000000001,"5min":1.7749999999999999,"15min":2.1419999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.369,"5min":1.3,"15min":1.2849999999999999},"min":{"1min":0.85599999999999998,"5min":0.76700000000000002,"15min":0.59199999999999997},"max":{"1min":1.673,"5min":2.843,"15min":2.843},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.337,"15min":1.3140000000000001},"min":{"1min":0.84699999999999998,"5min":0.754,"15min":0.754},"max":{"1min":1.6879999999999999,"5min":2.0190000000000001,"15min":2.121},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.425,"5min":1.381,"15min":1.359},"min":{"1min":0.98399999999999999,"5min":0.90800000000000003,"15min":0.77300000000000002},"max":{"1min":1.6140000000000001,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.2549999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4119999999999999,"15min":1.419},"min":{"1min":1.1639999999999999,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.675,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":1.403},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4850000000000001,"15min":1.4570000000000001},"min":{"1min":1.206,"5min":0.96499999999999997,"15min":0.91500000000000004},"max":{"1min":1.7410000000000001,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.96299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575061,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307036,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064804,"statfs":{"total":96636764160,"available":96322359296,"internally_reserved":0,"allocated":1593344,"data_stored":828818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.1859999999999999,"15min":1.21},"min":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.7210000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.92400000000000004,"5min":0.80000000000000004,"15min":0.70299999999999996},"max":{"1min":1.5649999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.2}]},{"osd":1,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.149,"15min":1.157},"min":{"1min":0.69799999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5640000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.151},{"interface":"front","average":{"1min":1.127,"5min":1.1100000000000001,"15min":1.1379999999999999},"min":{"1min":0.63200000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4990000000000001,"5min":1.569,"15min":1.579},"last":0.754}]},{"osd":2,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0860000000000001,"15min":1.1299999999999999},"min":{"1min":0.81599999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.52,"5min":1.5369999999999999,"15min":1.746},"last":0.878},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2290000000000001},"min":{"1min":0.79200000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5980000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.92300000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.264,"15min":1.276},"min":{"1min":0.76300000000000001,"5min":0.70999999999999996,"15min":0.65900000000000003},"max":{"1min":1.6379999999999999,"5min":1.831,"15min":1.831},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.337,"5min":1.2869999999999999,"15min":1.3220000000000001},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.1100000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.76100000000000001,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.6850000000000001,"5min":1.905,"15min":1.905},"last":1.01},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2070000000000001,"15min":1.198},"min":{"1min":0.72999999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.704,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.96399999999999997}]},{"osd":6,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.248,"15min":1.2270000000000001},"min":{"1min":0.88400000000000001,"5min":0.73899999999999999,"15min":0.67400000000000004},"max":{"1min":1.5409999999999999,"5min":1.748,"15min":1.748},"last":0.996},{"interface":"front","average":{"1min":1.351,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":0.97099999999999997,"5min":0.70899999999999996,"15min":0.67900000000000005},"max":{"1min":1.6799999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.3240000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.359,"15min":1.341},"min":{"1min":1.0009999999999999,"5min":0.58799999999999997,"15min":0.46300000000000002},"max":{"1min":1.756,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.036},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.407,"15min":1.387},"min":{"1min":0.96799999999999997,"5min":0.80200000000000005,"15min":0.70999999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.175}]}]},{"osd":3,"up_from":13,"seq":55834575061,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":829280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.85299999999999998,"15min":0.85099999999999998},"min":{"1min":0.51500000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.145,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96099999999999997,"15min":0.96299999999999997},"min":{"1min":0.52700000000000002,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.3620000000000001,"5min":1.962,"15min":1.962},"last":1.1559999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.89100000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.45600000000000002},"max":{"1min":1.228,"5min":1.609,"15min":1.609},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0389999999999999,"15min":1.038},"min":{"1min":0.77300000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.399,"5min":2.423,"15min":2.423},"last":0.90300000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0489999999999999,"15min":1.077},"min":{"1min":0.73999999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5580000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1759999999999999,"15min":1.163},"min":{"1min":0.70599999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.52,"5min":2.48,"15min":2.48},"last":1.0209999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2290000000000001,"15min":1.256},"min":{"1min":0.93700000000000006,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.55,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.82899999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.464,"5min":1.649,"15min":1.8120000000000001},"last":1.2430000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.91600000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.4790000000000001,"5min":1.996,"15min":1.996},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.339,"15min":1.335},"min":{"1min":0.85299999999999998,"5min":0.753,"15min":0.753},"max":{"1min":1.577,"5min":2.161,"15min":2.161},"last":1.552}]},{"osd":6,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4079999999999999,"15min":1.3879999999999999},"min":{"1min":0.89800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6439999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3300000000000001,"15min":1.363},"min":{"1min":1.0429999999999999,"5min":0.89700000000000002,"15min":0.82499999999999996},"max":{"1min":1.774,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.639}]},{"osd":7,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.4490000000000001,"15min":1.4550000000000001},"min":{"1min":0.999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8640000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.571},{"interface":"front","average":{"1min":1.403,"5min":1.4079999999999999,"15min":1.411},"min":{"1min":0.97499999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.752,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.3879999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575061,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":828760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89100000000000001,"15min":0.90400000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.437},"max":{"1min":1.3700000000000001,"5min":1.776,"15min":1.776},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0489999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.452,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78900000000000003}]},{"osd":1,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.351,"5min":1.669,"15min":1.669},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.98099999999999998,"15min":1.008},"min":{"1min":0.74399999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.5429999999999999,"5min":1.835,"15min":1.835},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94999999999999996,"15min":0.96699999999999997},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.361,"5min":1.657,"15min":1.667},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.0980000000000001,"15min":1.121},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5289999999999999,"5min":1.827,"15min":1.827},"last":0.871}]},{"osd":4,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.631,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.306,"15min":1.3009999999999999},"min":{"1min":0.98099999999999998,"5min":0.85899999999999999,"15min":0.82099999999999995},"max":{"1min":1.6859999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.069}]},{"osd":5,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.329,"15min":1.341},"min":{"1min":1.089,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.621,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.2509999999999999,"15min":1.2569999999999999},"min":{"1min":0.86199999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.5820000000000001,"5min":1.8,"15min":1.8},"last":1.234}]},{"osd":6,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.282,"15min":1.296},"min":{"1min":0.97599999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7390000000000001,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.306},{"interface":"front","average":{"1min":1.391,"5min":1.367,"15min":1.369},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.92100000000000004},"max":{"1min":1.7130000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.1080000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.375,"15min":1.3899999999999999},"min":{"1min":1.1339999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.9119999999999999,"5min":1.9730000000000001,"15min":2.214},"last":1.177},{"interface":"front","average":{"1min":1.363,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":1.0649999999999999,"5min":0.98699999999999999,"15min":0.88200000000000001},"max":{"1min":1.6779999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.355}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":835888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.876,"15min":0.85999999999999999},"min":{"1min":0.51700000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2470000000000001,"5min":1.661,"15min":1.661},"last":1.141},{"interface":"front","average":{"1min":0.871,"5min":0.93799999999999994,"15min":0.94099999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.375,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.79400000000000004}]},{"osd":2,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.94599999999999995,"15min":0.92400000000000004},"min":{"1min":0.57299999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.29,"5min":2.96,"15min":2.96},"last":1.054},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0780000000000001,"15min":1.0529999999999999},"min":{"1min":0.78600000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.3460000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1990000000000001,"15min":1.1559999999999999},"min":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.359,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.008},{"interface":"front","average":{"1min":0.999,"5min":1.079,"15min":1.0660000000000001},"min":{"1min":0.60299999999999998,"5min":0.55500000000000005,"15min":0.51200000000000001},"max":{"1min":1.385,"5min":2.407,"15min":2.407},"last":0.61599999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.181,"15min":1.1899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.879,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3500000000000001,"15min":1.3089999999999999},"min":{"1min":0.89300000000000002,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.649,"5min":3.04,"15min":3.04},"last":1.206}]},{"osd":5,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.284,"15min":1.278},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.5,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.103},{"interface":"front","average":{"1min":1.272,"5min":1.3129999999999999,"15min":1.288},"min":{"1min":0.85899999999999999,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.976,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.3380000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.4019999999999999,"15min":1.3819999999999999},"min":{"1min":0.93899999999999995,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.726,"5min":3.089,"15min":3.089},"last":1.284},{"interface":"front","average":{"1min":1.304,"5min":1.373,"15min":1.367},"min":{"1min":0.76300000000000001,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.665,"5min":2.46,"15min":2.46},"last":0.98899999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.407,"15min":1.417},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.88500000000000001},"max":{"1min":1.7609999999999999,"5min":2.504,"15min":2.504},"last":1.381},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.401,"15min":1.3839999999999999},"min":{"1min":0.94299999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7749999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.256}]}]},{"osd":0,"up_from":13,"seq":55834575061,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1419709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.58999999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.349,"5min":1.4590000000000001,"15min":1.5409999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.88,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.496},"max":{"1min":1.466,"5min":1.466,"15min":1.5289999999999999},"last":1.0660000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":0.97499999999999998,"15min":0.98099999999999998},"min":{"1min":0.79500000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.258},{"interface":"front","average":{"1min":1.012,"5min":1.028,"15min":1.034},"min":{"1min":0.73499999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3779999999999999,"5min":1.585,"15min":1.6000000000000001},"last":0.95399999999999996}]},{"osd":3,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94699999999999995,"15min":0.96399999999999997},"min":{"1min":0.68700000000000006,"5min":0.54500000000000004,"15min":0.51400000000000001},"max":{"1min":1.4790000000000001,"5min":1.5529999999999999,"15min":1.671},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.1240000000000001,"15min":1.133},"min":{"1min":0.93899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.528,"5min":1.663,"15min":1.6659999999999999},"last":1.1759999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":1.073,"5min":0.753,"15min":0.71399999999999997},"max":{"1min":1.6319999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":1.0880000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.6519999999999999,"5min":1.855,"15min":1.8999999999999999},"last":1.5660000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":1.0760000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.7250000000000001,"5min":2.085,"15min":2.085},"last":1.427},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3580000000000001},"min":{"1min":1.135,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5940000000000001,"5min":2.125,"15min":2.125},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2669999999999999,"15min":1.264},"min":{"1min":1.0429999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5629999999999999,"5min":1.97,"15min":1.97},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.3220000000000001},"min":{"1min":0.95399999999999996,"5min":0.80700000000000005,"15min":0.75600000000000001},"max":{"1min":1.671,"5min":1.871,"15min":1.871},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3169999999999999,"15min":1.3120000000000001},"min":{"1min":0.88200000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.5780000000000001,"5min":1.887,"15min":1.887},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.377,"15min":1.3700000000000001},"min":{"1min":1.1619999999999999,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.6259999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:05:15.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:05:15.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.234+0000 7f9f1b59e700 1 Processor -- start 2023-12-19T08:05:15.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.234+0000 7f9f1b59e700 1 -- start start 2023-12-19T08:05:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c0b1410 0x7f9f1c0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9f1c06a9d0 0x7f9f1c15af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c06c7c0 con 0x7f9f1c06a580 2023-12-19T08:05:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c06cd80 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c06b8f0 con 0x7f9f1c0b1410 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9f1c06a9d0 0x7f9f1c15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9f1c06a9d0 0x7f9f1c15af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56230/0 (socket says 172.21.15.111:56230) 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 -- 172.21.15.111:0/3925943252 learned_addr learned my addr 172.21.15.111:0/3925943252 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1a59c700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c0b1410 0x7f9f1c0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 -- 172.21.15.111:0/3925943252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c0b1410 msgr2=0x7f9f1c0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c0b1410 0x7f9f1c0ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 -- 172.21.15.111:0/3925943252 >> v1:172.21.15.143:6789/0 conn(0x7f9f1c06a580 legacy=0x7f9f1c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 -- 172.21.15.111:0/3925943252 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c0ac1e0 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1ad9d700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a9d0 0x7f9f1c15af70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009f20 tx=0x7f9f14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4c8e4e0ef9220 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1959a700 1 -- 172.21.15.111:0/3925943252 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1400e070 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1959a700 1 -- 172.21.15.111:0/3925943252 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1400b9d0 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.238+0000 7f9f1959a700 1 -- 172.21.15.111:0/3925943252 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140092e0 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3925943252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a9d0 msgr2=0x7f9f1c15af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a9d0 0x7f9f1c15af70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009f20 tx=0x7f9f14009840 comp rx=0 tx=0).stop 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3925943252 shutdown_connections 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a9d0 0x7f9f1c15af70 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3925943252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c0b1410 0x7f9f1c0ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3925943252 >> 172.21.15.111:0/3925943252 conn(0x7f9f1c067bb0 msgr2=0x7f9f1c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3925943252 shutdown_connections 2023-12-19T08:05:15.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3925943252 wait complete. 2023-12-19T08:05:15.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 Processor -- start 2023-12-19T08:05:15.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.242+0000 7f9f1b59e700 1 -- start start 2023-12-19T08:05:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c06a9d0 0x7f9f1c153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 0x7f9f1c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c159200 con 0x7f9f1c06a580 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c4f00 con 0x7f9f1c0b1410 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c42f0 con 0x7f9f1c06a9d0 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56232/0 (socket says 172.21.15.111:56232) 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 -- 172.21.15.111:0/3773539750 learned_addr learned my addr 172.21.15.111:0/3773539750 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f19d9b700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c06a9d0 0x7f9f1c153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1ad9d700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 0x7f9f1c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 -- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c06a9d0 msgr2=0x7f9f1c153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c06a9d0 0x7f9f1c153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 -- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 msgr2=0x7f9f1c0c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 0x7f9f1c0c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 -- 172.21.15.111:0/3773539750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14009ae0 con 0x7f9f1c06a580 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1ad9d700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 0x7f9f1c0c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1a59c700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00e930 tx=0x7f9f0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00eed0 con 0x7f9f1c06a580 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c00c850 con 0x7f9f1c06a580 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c0c45d0 con 0x7f9f1c06a580 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00cbf0 con 0x7f9f1c06a580 2023-12-19T08:05:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.246+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c0c4b90 con 0x7f9f1c06a580 2023-12-19T08:05:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.250+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c0ad170 con 0x7f9f1c06a580 2023-12-19T08:05:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.250+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9f0c013470 con 0x7f9f1c06a580 2023-12-19T08:05:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.250+0000 7f9f0a7fc700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 0x7f9efc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.250+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f9f0c07cf20 con 0x7f9f1c06a580 2023-12-19T08:05:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.254+0000 7f9f19d9b700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 0x7f9efc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.254+0000 7f9f19d9b700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 0x7f9efc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c15b7a0 tx=0x7f9f10009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.258+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9f1c0ad170 con 0x7f9f1c06a580 2023-12-19T08:05:15.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.438+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f1c0ad170 con 0x7f9efc060670 2023-12-19T08:05:15.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.458+0000 7f9f0a7fc700 1 -- 172.21.15.111:0/3773539750 <== mgr.7501 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287072 (secure 0 0 0) 0x7f9f1c0ad170 con 0x7f9efc060670 2023-12-19T08:05:15.466 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:15.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 msgr2=0x7f9efc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 0x7f9efc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c15b7a0 tx=0x7f9f10009040 comp rx=0 tx=0).stop 2023-12-19T08:05:15.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 msgr2=0x7f9f1c157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00e930 tx=0x7f9f0c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 shutdown_connections 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9efc060670 0x7f9efc062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f1c0b1410 0x7f9f1c0c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f1c06a9d0 0x7f9f1c153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 --2- 172.21.15.111:0/3773539750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c06a580 0x7f9f1c157d30 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.462+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 >> 172.21.15.111:0/3773539750 conn(0x7f9f1c067bb0 msgr2=0x7f9f1c0c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.466+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 shutdown_connections 2023-12-19T08:05:15.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.466+0000 7f9f1b59e700 1 -- 172.21.15.111:0/3773539750 wait complete. 2023-12-19T08:05:15.470 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:05:15.489 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:05:14.319455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449276,"kb_used_data":13628,"kb_used_omap":0,"kb_used_meta":2435584,"kb_avail":752525444,"statfs":{"total":773094113280,"available":770586054656,"internally_reserved":0,"allocated":13955072,"data_stored":7828401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2494038016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.599203"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774442+0000","last_change":"2023-12-19T08:05:10.774442+0000","last_active":"2023-12-19T08:05:10.774442+0000","last_peered":"2023-12-19T08:05:10.774442+0000","last_clean":"2023-12-19T08:05:10.774442+0000","last_became_active":"2023-12-19T08:05:10.773088+0000","last_became_peered":"2023-12-19T08:05:10.773088+0000","last_unstale":"2023-12-19T08:05:10.774442+0000","last_undegraded":"2023-12-19T08:05:10.774442+0000","last_fullsized":"2023-12-19T08:05:10.774442+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:54:22.217801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.781626+0000","last_change":"2023-12-19T08:05:10.781626+0000","last_active":"2023-12-19T08:05:10.781626+0000","last_peered":"2023-12-19T08:05:10.781626+0000","last_clean":"2023-12-19T08:05:10.781626+0000","last_became_active":"2023-12-19T08:05:10.781355+0000","last_became_peered":"2023-12-19T08:05:10.781355+0000","last_unstale":"2023-12-19T08:05:10.781626+0000","last_undegraded":"2023-12-19T08:05:10.781626+0000","last_fullsized":"2023-12-19T08:05:10.781626+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.784678+0000","last_change":"2023-12-19T08:05:10.784678+0000","last_active":"2023-12-19T08:05:10.784678+0000","last_peered":"2023-12-19T08:05:10.784678+0000","last_clean":"2023-12-19T08:05:10.784678+0000","last_became_active":"2023-12-19T08:05:10.783781+0000","last_became_peered":"2023-12-19T08:05:10.783781+0000","last_unstale":"2023-12-19T08:05:10.784678+0000","last_undegraded":"2023-12-19T08:05:10.784678+0000","last_fullsized":"2023-12-19T08:05:10.784678+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.914967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773003+0000","last_change":"2023-12-19T08:05:10.773003+0000","last_active":"2023-12-19T08:05:10.773003+0000","last_peered":"2023-12-19T08:05:10.773003+0000","last_clean":"2023-12-19T08:05:10.773003+0000","last_became_active":"2023-12-19T08:05:10.772644+0000","last_became_peered":"2023-12-19T08:05:10.772644+0000","last_unstale":"2023-12-19T08:05:10.773003+0000","last_undegraded":"2023-12-19T08:05:10.773003+0000","last_fullsized":"2023-12-19T08:05:10.773003+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:07:29.016139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.761247+0000","last_change":"2023-12-19T08:05:10.752718+0000","last_active":"2023-12-19T08:05:10.761247+0000","last_peered":"2023-12-19T08:05:10.761247+0000","last_clean":"2023-12-19T08:05:10.761247+0000","last_became_active":"2023-12-19T08:05:10.752389+0000","last_became_peered":"2023-12-19T08:05:10.752389+0000","last_unstale":"2023-12-19T08:05:10.761247+0000","last_undegraded":"2023-12-19T08:05:10.761247+0000","last_fullsized":"2023-12-19T08:05:10.761247+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777106+0000","last_change":"2023-12-19T08:05:10.777106+0000","last_active":"2023-12-19T08:05:10.777106+0000","last_peered":"2023-12-19T08:05:10.777106+0000","last_clean":"2023-12-19T08:05:10.777106+0000","last_became_active":"2023-12-19T08:05:10.776757+0000","last_became_peered":"2023-12-19T08:05:10.776757+0000","last_unstale":"2023-12-19T08:05:10.777106+0000","last_undegraded":"2023-12-19T08:05:10.777106+0000","last_fullsized":"2023-12-19T08:05:10.777106+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:35.788963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769819+0000","last_change":"2023-12-19T08:05:10.769819+0000","last_active":"2023-12-19T08:05:10.769819+0000","last_peered":"2023-12-19T08:05:10.769819+0000","last_clean":"2023-12-19T08:05:10.769819+0000","last_became_active":"2023-12-19T08:05:10.769466+0000","last_became_peered":"2023-12-19T08:05:10.769466+0000","last_unstale":"2023-12-19T08:05:10.769819+0000","last_undegraded":"2023-12-19T08:05:10.769819+0000","last_fullsized":"2023-12-19T08:05:10.769819+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:00.297592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768141+0000","last_change":"2023-12-19T08:05:10.768141+0000","last_active":"2023-12-19T08:05:10.768141+0000","last_peered":"2023-12-19T08:05:10.768141+0000","last_clean":"2023-12-19T08:05:10.768141+0000","last_became_active":"2023-12-19T08:05:10.767843+0000","last_became_peered":"2023-12-19T08:05:10.767843+0000","last_unstale":"2023-12-19T08:05:10.768141+0000","last_undegraded":"2023-12-19T08:05:10.768141+0000","last_fullsized":"2023-12-19T08:05:10.768141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:56.736611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.789632+0000","last_change":"2023-12-19T08:05:10.789632+0000","last_active":"2023-12-19T08:05:10.789632+0000","last_peered":"2023-12-19T08:05:10.789632+0000","last_clean":"2023-12-19T08:05:10.789632+0000","last_became_active":"2023-12-19T08:05:10.787575+0000","last_became_peered":"2023-12-19T08:05:10.787575+0000","last_unstale":"2023-12-19T08:05:10.789632+0000","last_undegraded":"2023-12-19T08:05:10.789632+0000","last_fullsized":"2023-12-19T08:05:10.789632+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.786690+0000","last_change":"2023-12-19T08:05:10.775590+0000","last_active":"2023-12-19T08:05:10.786690+0000","last_peered":"2023-12-19T08:05:10.786690+0000","last_clean":"2023-12-19T08:05:10.786690+0000","last_became_active":"2023-12-19T08:05:10.775156+0000","last_became_peered":"2023-12-19T08:05:10.775156+0000","last_unstale":"2023-12-19T08:05:10.786690+0000","last_undegraded":"2023-12-19T08:05:10.786690+0000","last_fullsized":"2023-12-19T08:05:10.786690+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.791092+0000","last_change":"2023-12-19T08:05:10.787247+0000","last_active":"2023-12-19T08:05:10.791092+0000","last_peered":"2023-12-19T08:05:10.791092+0000","last_clean":"2023-12-19T08:05:10.791092+0000","last_became_active":"2023-12-19T08:05:10.786952+0000","last_became_peered":"2023-12-19T08:05:10.786952+0000","last_unstale":"2023-12-19T08:05:10.791092+0000","last_undegraded":"2023-12-19T08:05:10.791092+0000","last_fullsized":"2023-12-19T08:05:10.791092+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775726+0000","last_change":"2023-12-19T08:05:10.771089+0000","last_active":"2023-12-19T08:05:10.775726+0000","last_peered":"2023-12-19T08:05:10.775726+0000","last_clean":"2023-12-19T08:05:10.775726+0000","last_became_active":"2023-12-19T08:05:10.770810+0000","last_became_peered":"2023-12-19T08:05:10.770810+0000","last_unstale":"2023-12-19T08:05:10.775726+0000","last_undegraded":"2023-12-19T08:05:10.775726+0000","last_fullsized":"2023-12-19T08:05:10.775726+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:24:25.145068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768508+0000","last_change":"2023-12-19T08:05:10.768508+0000","last_active":"2023-12-19T08:05:10.768508+0000","last_peered":"2023-12-19T08:05:10.768508+0000","last_clean":"2023-12-19T08:05:10.768508+0000","last_became_active":"2023-12-19T08:05:10.768072+0000","last_became_peered":"2023-12-19T08:05:10.768072+0000","last_unstale":"2023-12-19T08:05:10.768508+0000","last_undegraded":"2023-12-19T08:05:10.768508+0000","last_fullsized":"2023-12-19T08:05:10.768508+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.914967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773075+0000","last_change":"2023-12-19T08:05:10.773075+0000","last_active":"2023-12-19T08:05:10.773075+0000","last_peered":"2023-12-19T08:05:10.773075+0000","last_clean":"2023-12-19T08:05:10.773075+0000","last_became_active":"2023-12-19T08:05:10.772743+0000","last_became_peered":"2023-12-19T08:05:10.772743+0000","last_unstale":"2023-12-19T08:05:10.773075+0000","last_undegraded":"2023-12-19T08:05:10.773075+0000","last_fullsized":"2023-12-19T08:05:10.773075+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776349+0000","last_change":"2023-12-19T08:05:10.776349+0000","last_active":"2023-12-19T08:05:10.776349+0000","last_peered":"2023-12-19T08:05:10.776349+0000","last_clean":"2023-12-19T08:05:10.776349+0000","last_became_active":"2023-12-19T08:05:10.776083+0000","last_became_peered":"2023-12-19T08:05:10.776083+0000","last_unstale":"2023-12-19T08:05:10.776349+0000","last_undegraded":"2023-12-19T08:05:10.776349+0000","last_fullsized":"2023-12-19T08:05:10.776349+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.783657+0000","last_change":"2023-12-19T08:05:10.783657+0000","last_active":"2023-12-19T08:05:10.783657+0000","last_peered":"2023-12-19T08:05:10.783657+0000","last_clean":"2023-12-19T08:05:10.783657+0000","last_became_active":"2023-12-19T08:05:10.783368+0000","last_became_peered":"2023-12-19T08:05:10.783368+0000","last_unstale":"2023-12-19T08:05:10.783657+0000","last_undegraded":"2023-12-19T08:05:10.783657+0000","last_fullsized":"2023-12-19T08:05:10.783657+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:13:53.019181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782649+0000","last_change":"2023-12-19T08:05:10.782649+0000","last_active":"2023-12-19T08:05:10.782649+0000","last_peered":"2023-12-19T08:05:10.782649+0000","last_clean":"2023-12-19T08:05:10.782649+0000","last_became_active":"2023-12-19T08:05:10.782191+0000","last_became_peered":"2023-12-19T08:05:10.782191+0000","last_unstale":"2023-12-19T08:05:10.782649+0000","last_undegraded":"2023-12-19T08:05:10.782649+0000","last_fullsized":"2023-12-19T08:05:10.782649+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:59.230868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774741+0000","last_change":"2023-12-19T08:05:10.774741+0000","last_active":"2023-12-19T08:05:10.774741+0000","last_peered":"2023-12-19T08:05:10.774741+0000","last_clean":"2023-12-19T08:05:10.774741+0000","last_became_active":"2023-12-19T08:05:10.774330+0000","last_became_peered":"2023-12-19T08:05:10.774330+0000","last_unstale":"2023-12-19T08:05:10.774741+0000","last_undegraded":"2023-12-19T08:05:10.774741+0000","last_fullsized":"2023-12-19T08:05:10.774741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:53.094931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778808+0000","last_change":"2023-12-19T08:05:10.778808+0000","last_active":"2023-12-19T08:05:10.778808+0000","last_peered":"2023-12-19T08:05:10.778808+0000","last_clean":"2023-12-19T08:05:10.778808+0000","last_became_active":"2023-12-19T08:05:10.778308+0000","last_became_peered":"2023-12-19T08:05:10.778308+0000","last_unstale":"2023-12-19T08:05:10.778808+0000","last_undegraded":"2023-12-19T08:05:10.778808+0000","last_fullsized":"2023-12-19T08:05:10.778808+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:51:50.011314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775707+0000","last_change":"2023-12-19T08:05:10.767923+0000","last_active":"2023-12-19T08:05:10.775707+0000","last_peered":"2023-12-19T08:05:10.775707+0000","last_clean":"2023-12-19T08:05:10.775707+0000","last_became_active":"2023-12-19T08:05:10.766548+0000","last_became_peered":"2023-12-19T08:05:10.766548+0000","last_unstale":"2023-12-19T08:05:10.775707+0000","last_undegraded":"2023-12-19T08:05:10.775707+0000","last_fullsized":"2023-12-19T08:05:10.775707+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:53:30.585462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777398+0000","last_change":"2023-12-19T08:05:10.777398+0000","last_active":"2023-12-19T08:05:10.777398+0000","last_peered":"2023-12-19T08:05:10.777398+0000","last_clean":"2023-12-19T08:05:10.777398+0000","last_became_active":"2023-12-19T08:05:10.777066+0000","last_became_peered":"2023-12-19T08:05:10.777066+0000","last_unstale":"2023-12-19T08:05:10.777398+0000","last_undegraded":"2023-12-19T08:05:10.777398+0000","last_fullsized":"2023-12-19T08:05:10.777398+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:00:31.320978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.775491+0000","last_change":"2023-12-19T08:05:10.767733+0000","last_active":"2023-12-19T08:05:10.775491+0000","last_peered":"2023-12-19T08:05:10.775491+0000","last_clean":"2023-12-19T08:05:10.775491+0000","last_became_active":"2023-12-19T08:05:10.767191+0000","last_became_peered":"2023-12-19T08:05:10.767191+0000","last_unstale":"2023-12-19T08:05:10.775491+0000","last_undegraded":"2023-12-19T08:05:10.775491+0000","last_fullsized":"2023-12-19T08:05:10.775491+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769177+0000","last_change":"2023-12-19T08:05:10.769177+0000","last_active":"2023-12-19T08:05:10.769177+0000","last_peered":"2023-12-19T08:05:10.769177+0000","last_clean":"2023-12-19T08:05:10.769177+0000","last_became_active":"2023-12-19T08:05:10.768843+0000","last_became_peered":"2023-12-19T08:05:10.768843+0000","last_unstale":"2023-12-19T08:05:10.769177+0000","last_undegraded":"2023-12-19T08:05:10.769177+0000","last_fullsized":"2023-12-19T08:05:10.769177+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:47.089554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.765424+0000","last_change":"2023-12-19T08:05:10.760793+0000","last_active":"2023-12-19T08:05:10.765424+0000","last_peered":"2023-12-19T08:05:10.765424+0000","last_clean":"2023-12-19T08:05:10.765424+0000","last_became_active":"2023-12-19T08:05:10.759127+0000","last_became_peered":"2023-12-19T08:05:10.759127+0000","last_unstale":"2023-12-19T08:05:10.765424+0000","last_undegraded":"2023-12-19T08:05:10.765424+0000","last_fullsized":"2023-12-19T08:05:10.765424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777571+0000","last_change":"2023-12-19T08:05:10.774250+0000","last_active":"2023-12-19T08:05:10.777571+0000","last_peered":"2023-12-19T08:05:10.777571+0000","last_clean":"2023-12-19T08:05:10.777571+0000","last_became_active":"2023-12-19T08:05:10.773915+0000","last_became_peered":"2023-12-19T08:05:10.773915+0000","last_unstale":"2023-12-19T08:05:10.777571+0000","last_undegraded":"2023-12-19T08:05:10.777571+0000","last_fullsized":"2023-12-19T08:05:10.777571+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:43.010981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780488+0000","last_change":"2023-12-19T08:05:10.776781+0000","last_active":"2023-12-19T08:05:10.780488+0000","last_peered":"2023-12-19T08:05:10.780488+0000","last_clean":"2023-12-19T08:05:10.780488+0000","last_became_active":"2023-12-19T08:05:10.776453+0000","last_became_peered":"2023-12-19T08:05:10.776453+0000","last_unstale":"2023-12-19T08:05:10.780488+0000","last_undegraded":"2023-12-19T08:05:10.780488+0000","last_fullsized":"2023-12-19T08:05:10.780488+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:50:01.734825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.791011+0000","last_change":"2023-12-19T08:05:10.785260+0000","last_active":"2023-12-19T08:05:10.791011+0000","last_peered":"2023-12-19T08:05:10.791011+0000","last_clean":"2023-12-19T08:05:10.791011+0000","last_became_active":"2023-12-19T08:05:10.784983+0000","last_became_peered":"2023-12-19T08:05:10.784983+0000","last_unstale":"2023-12-19T08:05:10.791011+0000","last_undegraded":"2023-12-19T08:05:10.791011+0000","last_fullsized":"2023-12-19T08:05:10.791011+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:01:00.148034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.783431+0000","last_change":"2023-12-19T08:05:10.783431+0000","last_active":"2023-12-19T08:05:10.783431+0000","last_peered":"2023-12-19T08:05:10.783431+0000","last_clean":"2023-12-19T08:05:10.783431+0000","last_became_active":"2023-12-19T08:05:10.783155+0000","last_became_peered":"2023-12-19T08:05:10.783155+0000","last_unstale":"2023-12-19T08:05:10.783431+0000","last_undegraded":"2023-12-19T08:05:10.783431+0000","last_fullsized":"2023-12-19T08:05:10.783431+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778468+0000","last_change":"2023-12-19T08:05:10.778468+0000","last_active":"2023-12-19T08:05:10.778468+0000","last_peered":"2023-12-19T08:05:10.778468+0000","last_clean":"2023-12-19T08:05:10.778468+0000","last_became_active":"2023-12-19T08:05:10.778156+0000","last_became_peered":"2023-12-19T08:05:10.778156+0000","last_unstale":"2023-12-19T08:05:10.778468+0000","last_undegraded":"2023-12-19T08:05:10.778468+0000","last_fullsized":"2023-12-19T08:05:10.778468+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:29.645164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780457+0000","last_change":"2023-12-19T08:05:10.776011+0000","last_active":"2023-12-19T08:05:10.780457+0000","last_peered":"2023-12-19T08:05:10.780457+0000","last_clean":"2023-12-19T08:05:10.780457+0000","last_became_active":"2023-12-19T08:05:10.775407+0000","last_became_peered":"2023-12-19T08:05:10.775407+0000","last_unstale":"2023-12-19T08:05:10.780457+0000","last_undegraded":"2023-12-19T08:05:10.780457+0000","last_fullsized":"2023-12-19T08:05:10.780457+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:26.239674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772983+0000","last_change":"2023-12-19T08:05:10.772983+0000","last_active":"2023-12-19T08:05:10.772983+0000","last_peered":"2023-12-19T08:05:10.772983+0000","last_clean":"2023-12-19T08:05:10.772983+0000","last_became_active":"2023-12-19T08:05:10.772689+0000","last_became_peered":"2023-12-19T08:05:10.772689+0000","last_unstale":"2023-12-19T08:05:10.772983+0000","last_undegraded":"2023-12-19T08:05:10.772983+0000","last_fullsized":"2023-12-19T08:05:10.772983+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:56:46.726157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780205+0000","last_change":"2023-12-19T08:05:10.780205+0000","last_active":"2023-12-19T08:05:10.780205+0000","last_peered":"2023-12-19T08:05:10.780205+0000","last_clean":"2023-12-19T08:05:10.780205+0000","last_became_active":"2023-12-19T08:05:10.779905+0000","last_became_peered":"2023-12-19T08:05:10.779905+0000","last_unstale":"2023-12-19T08:05:10.780205+0000","last_undegraded":"2023-12-19T08:05:10.780205+0000","last_fullsized":"2023-12-19T08:05:10.780205+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:07.921933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778411+0000","last_change":"2023-12-19T08:05:10.778411+0000","last_active":"2023-12-19T08:05:10.778411+0000","last_peered":"2023-12-19T08:05:10.778411+0000","last_clean":"2023-12-19T08:05:10.778411+0000","last_became_active":"2023-12-19T08:05:10.776251+0000","last_became_peered":"2023-12-19T08:05:10.776251+0000","last_unstale":"2023-12-19T08:05:10.778411+0000","last_undegraded":"2023-12-19T08:05:10.778411+0000","last_fullsized":"2023-12-19T08:05:10.778411+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:07:29.016139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":13,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790101+0000","last_change":"2023-12-19T08:05:09.755630+0000","last_active":"2023-12-19T08:05:10.790101+0000","last_peered":"2023-12-19T08:05:10.790101+0000","last_clean":"2023-12-19T08:05:10.790101+0000","last_became_active":"2023-12-19T08:05:09.755308+0000","last_became_peered":"2023-12-19T08:05:09.755308+0000","last_unstale":"2023-12-19T08:05:10.790101+0000","last_undegraded":"2023-12-19T08:05:10.790101+0000","last_fullsized":"2023-12-19T08:05:10.790101+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:25:55.379811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.766993+0000","last_change":"2023-12-19T08:05:10.766993+0000","last_active":"2023-12-19T08:05:10.766993+0000","last_peered":"2023-12-19T08:05:10.766993+0000","last_clean":"2023-12-19T08:05:10.766993+0000","last_became_active":"2023-12-19T08:05:10.766691+0000","last_became_peered":"2023-12-19T08:05:10.766691+0000","last_unstale":"2023-12-19T08:05:10.766993+0000","last_undegraded":"2023-12-19T08:05:10.766993+0000","last_fullsized":"2023-12-19T08:05:10.766993+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:10:00.156664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.767191+0000","last_change":"2023-12-19T08:05:09.759559+0000","last_active":"2023-12-19T08:05:10.767191+0000","last_peered":"2023-12-19T08:05:10.767191+0000","last_clean":"2023-12-19T08:05:10.767191+0000","last_became_active":"2023-12-19T08:05:09.759254+0000","last_became_peered":"2023-12-19T08:05:09.759254+0000","last_unstale":"2023-12-19T08:05:10.767191+0000","last_undegraded":"2023-12-19T08:05:10.767191+0000","last_fullsized":"2023-12-19T08:05:10.767191+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:04.314389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.780438+0000","last_change":"2023-12-19T08:05:10.775818+0000","last_active":"2023-12-19T08:05:10.780438+0000","last_peered":"2023-12-19T08:05:10.780438+0000","last_clean":"2023-12-19T08:05:10.780438+0000","last_became_active":"2023-12-19T08:05:10.775043+0000","last_became_peered":"2023-12-19T08:05:10.775043+0000","last_unstale":"2023-12-19T08:05:10.780438+0000","last_undegraded":"2023-12-19T08:05:10.780438+0000","last_fullsized":"2023-12-19T08:05:10.780438+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:37:00.667330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.792071+0000","last_change":"2023-12-19T08:05:09.755987+0000","last_active":"2023-12-19T08:05:10.792071+0000","last_peered":"2023-12-19T08:05:10.792071+0000","last_clean":"2023-12-19T08:05:10.792071+0000","last_became_active":"2023-12-19T08:05:09.755658+0000","last_became_peered":"2023-12-19T08:05:09.755658+0000","last_unstale":"2023-12-19T08:05:10.792071+0000","last_undegraded":"2023-12-19T08:05:10.792071+0000","last_fullsized":"2023-12-19T08:05:10.792071+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:27.956229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782019+0000","last_change":"2023-12-19T08:05:10.773159+0000","last_active":"2023-12-19T08:05:10.782019+0000","last_peered":"2023-12-19T08:05:10.782019+0000","last_clean":"2023-12-19T08:05:10.782019+0000","last_became_active":"2023-12-19T08:05:10.772852+0000","last_became_peered":"2023-12-19T08:05:10.772852+0000","last_unstale":"2023-12-19T08:05:10.782019+0000","last_undegraded":"2023-12-19T08:05:10.782019+0000","last_fullsized":"2023-12-19T08:05:10.782019+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787068+0000","last_change":"2023-12-19T08:05:09.746463+0000","last_active":"2023-12-19T08:05:10.787068+0000","last_peered":"2023-12-19T08:05:10.787068+0000","last_clean":"2023-12-19T08:05:10.787068+0000","last_became_active":"2023-12-19T08:05:09.746144+0000","last_became_peered":"2023-12-19T08:05:09.746144+0000","last_unstale":"2023-12-19T08:05:10.787068+0000","last_undegraded":"2023-12-19T08:05:10.787068+0000","last_fullsized":"2023-12-19T08:05:10.787068+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:20.928370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771039+0000","last_change":"2023-12-19T08:05:10.771039+0000","last_active":"2023-12-19T08:05:10.771039+0000","last_peered":"2023-12-19T08:05:10.771039+0000","last_clean":"2023-12-19T08:05:10.771039+0000","last_became_active":"2023-12-19T08:05:10.770692+0000","last_became_peered":"2023-12-19T08:05:10.770692+0000","last_unstale":"2023-12-19T08:05:10.771039+0000","last_undegraded":"2023-12-19T08:05:10.771039+0000","last_fullsized":"2023-12-19T08:05:10.771039+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:47.089554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.760560+0000","last_change":"2023-12-19T08:05:09.763291+0000","last_active":"2023-12-19T08:05:10.760560+0000","last_peered":"2023-12-19T08:05:10.760560+0000","last_clean":"2023-12-19T08:05:10.760560+0000","last_became_active":"2023-12-19T08:05:09.763011+0000","last_became_peered":"2023-12-19T08:05:09.763011+0000","last_unstale":"2023-12-19T08:05:10.760560+0000","last_undegraded":"2023-12-19T08:05:10.760560+0000","last_fullsized":"2023-12-19T08:05:10.760560+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:58.690695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778597+0000","last_change":"2023-12-19T08:05:10.778597+0000","last_active":"2023-12-19T08:05:10.778597+0000","last_peered":"2023-12-19T08:05:10.778597+0000","last_clean":"2023-12-19T08:05:10.778597+0000","last_became_active":"2023-12-19T08:05:10.777912+0000","last_became_peered":"2023-12-19T08:05:10.777912+0000","last_unstale":"2023-12-19T08:05:10.778597+0000","last_undegraded":"2023-12-19T08:05:10.778597+0000","last_fullsized":"2023-12-19T08:05:10.778597+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:05.748640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790026+0000","last_change":"2023-12-19T08:05:09.758390+0000","last_active":"2023-12-19T08:05:10.790026+0000","last_peered":"2023-12-19T08:05:10.790026+0000","last_clean":"2023-12-19T08:05:10.790026+0000","last_became_active":"2023-12-19T08:05:09.758085+0000","last_became_peered":"2023-12-19T08:05:09.758085+0000","last_unstale":"2023-12-19T08:05:10.790026+0000","last_undegraded":"2023-12-19T08:05:10.790026+0000","last_fullsized":"2023-12-19T08:05:10.790026+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:10.432397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793680+0000","last_change":"2023-12-19T08:05:10.793680+0000","last_active":"2023-12-19T08:05:10.793680+0000","last_peered":"2023-12-19T08:05:10.793680+0000","last_clean":"2023-12-19T08:05:10.793680+0000","last_became_active":"2023-12-19T08:05:10.793273+0000","last_became_peered":"2023-12-19T08:05:10.793273+0000","last_unstale":"2023-12-19T08:05:10.793680+0000","last_undegraded":"2023-12-19T08:05:10.793680+0000","last_fullsized":"2023-12-19T08:05:10.793680+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:10.127412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.754943+0000","last_change":"2023-12-19T08:05:09.755954+0000","last_active":"2023-12-19T08:05:10.754943+0000","last_peered":"2023-12-19T08:05:10.754943+0000","last_clean":"2023-12-19T08:05:10.754943+0000","last_became_active":"2023-12-19T08:05:09.755559+0000","last_became_peered":"2023-12-19T08:05:09.755559+0000","last_unstale":"2023-12-19T08:05:10.754943+0000","last_undegraded":"2023-12-19T08:05:10.754943+0000","last_fullsized":"2023-12-19T08:05:10.754943+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:09.115449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771141+0000","last_change":"2023-12-19T08:05:10.771141+0000","last_active":"2023-12-19T08:05:10.771141+0000","last_peered":"2023-12-19T08:05:10.771141+0000","last_clean":"2023-12-19T08:05:10.771141+0000","last_became_active":"2023-12-19T08:05:10.770596+0000","last_became_peered":"2023-12-19T08:05:10.770596+0000","last_unstale":"2023-12-19T08:05:10.771141+0000","last_undegraded":"2023-12-19T08:05:10.771141+0000","last_fullsized":"2023-12-19T08:05:10.771141+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:59.230868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.747950+0000","last_change":"2023-12-19T08:05:09.743271+0000","last_active":"2023-12-19T08:05:09.747950+0000","last_peered":"2023-12-19T08:05:09.747950+0000","last_clean":"2023-12-19T08:05:09.747950+0000","last_became_active":"2023-12-19T08:05:09.742927+0000","last_became_peered":"2023-12-19T08:05:09.742927+0000","last_unstale":"2023-12-19T08:05:09.747950+0000","last_undegraded":"2023-12-19T08:05:09.747950+0000","last_fullsized":"2023-12-19T08:05:09.747950+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:26:11.961192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778867+0000","last_change":"2023-12-19T08:05:10.778867+0000","last_active":"2023-12-19T08:05:10.778867+0000","last_peered":"2023-12-19T08:05:10.778867+0000","last_clean":"2023-12-19T08:05:10.778867+0000","last_became_active":"2023-12-19T08:05:10.778562+0000","last_became_peered":"2023-12-19T08:05:10.778562+0000","last_unstale":"2023-12-19T08:05:10.778867+0000","last_undegraded":"2023-12-19T08:05:10.778867+0000","last_fullsized":"2023-12-19T08:05:10.778867+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:56.176007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.755551+0000","last_change":"2023-12-19T08:05:09.755907+0000","last_active":"2023-12-19T08:05:10.755551+0000","last_peered":"2023-12-19T08:05:10.755551+0000","last_clean":"2023-12-19T08:05:10.755551+0000","last_became_active":"2023-12-19T08:05:09.755575+0000","last_became_peered":"2023-12-19T08:05:09.755575+0000","last_unstale":"2023-12-19T08:05:10.755551+0000","last_undegraded":"2023-12-19T08:05:10.755551+0000","last_fullsized":"2023-12-19T08:05:10.755551+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:21:03.789828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787587+0000","last_change":"2023-12-19T08:05:10.779996+0000","last_active":"2023-12-19T08:05:10.787587+0000","last_peered":"2023-12-19T08:05:10.787587+0000","last_clean":"2023-12-19T08:05:10.787587+0000","last_became_active":"2023-12-19T08:05:10.779678+0000","last_became_peered":"2023-12-19T08:05:10.779678+0000","last_unstale":"2023-12-19T08:05:10.787587+0000","last_undegraded":"2023-12-19T08:05:10.787587+0000","last_fullsized":"2023-12-19T08:05:10.787587+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.757324+0000","last_change":"2023-12-19T08:05:09.753931+0000","last_active":"2023-12-19T08:05:09.757324+0000","last_peered":"2023-12-19T08:05:09.757324+0000","last_clean":"2023-12-19T08:05:09.757324+0000","last_became_active":"2023-12-19T08:05:09.753586+0000","last_became_peered":"2023-12-19T08:05:09.753586+0000","last_unstale":"2023-12-19T08:05:09.757324+0000","last_undegraded":"2023-12-19T08:05:09.757324+0000","last_fullsized":"2023-12-19T08:05:09.757324+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:25.227711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.771169+0000","last_change":"2023-12-19T08:05:10.771169+0000","last_active":"2023-12-19T08:05:10.771169+0000","last_peered":"2023-12-19T08:05:10.771169+0000","last_clean":"2023-12-19T08:05:10.771169+0000","last_became_active":"2023-12-19T08:05:10.770894+0000","last_became_peered":"2023-12-19T08:05:10.770894+0000","last_unstale":"2023-12-19T08:05:10.771169+0000","last_undegraded":"2023-12-19T08:05:10.771169+0000","last_fullsized":"2023-12-19T08:05:10.771169+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:16:52.574569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793945+0000","last_change":"2023-12-19T08:05:09.760406+0000","last_active":"2023-12-19T08:05:10.793945+0000","last_peered":"2023-12-19T08:05:10.793945+0000","last_clean":"2023-12-19T08:05:10.793945+0000","last_became_active":"2023-12-19T08:05:09.760133+0000","last_became_peered":"2023-12-19T08:05:09.760133+0000","last_unstale":"2023-12-19T08:05:10.793945+0000","last_undegraded":"2023-12-19T08:05:10.793945+0000","last_fullsized":"2023-12-19T08:05:10.793945+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:52.082968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.787958+0000","last_change":"2023-12-19T08:05:10.782245+0000","last_active":"2023-12-19T08:05:10.787958+0000","last_peered":"2023-12-19T08:05:10.787958+0000","last_clean":"2023-12-19T08:05:10.787958+0000","last_became_active":"2023-12-19T08:05:10.781962+0000","last_became_peered":"2023-12-19T08:05:10.781962+0000","last_unstale":"2023-12-19T08:05:10.787958+0000","last_undegraded":"2023-12-19T08:05:10.787958+0000","last_fullsized":"2023-12-19T08:05:10.787958+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":15,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.796687+0000","last_change":"2023-12-19T08:05:09.763870+0000","last_active":"2023-12-19T08:05:10.796687+0000","last_peered":"2023-12-19T08:05:10.796687+0000","last_clean":"2023-12-19T08:05:10.796687+0000","last_became_active":"2023-12-19T08:05:09.763573+0000","last_became_peered":"2023-12-19T08:05:09.763573+0000","last_unstale":"2023-12-19T08:05:10.796687+0000","last_undegraded":"2023-12-19T08:05:10.796687+0000","last_fullsized":"2023-12-19T08:05:10.796687+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:39:11.020493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793501+0000","last_change":"2023-12-19T08:05:10.790316+0000","last_active":"2023-12-19T08:05:10.793501+0000","last_peered":"2023-12-19T08:05:10.793501+0000","last_clean":"2023-12-19T08:05:10.793501+0000","last_became_active":"2023-12-19T08:05:10.790006+0000","last_became_peered":"2023-12-19T08:05:10.790006+0000","last_unstale":"2023-12-19T08:05:10.793501+0000","last_undegraded":"2023-12-19T08:05:10.793501+0000","last_fullsized":"2023-12-19T08:05:10.793501+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:49:30.708286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790183+0000","last_change":"2023-12-19T08:05:09.755432+0000","last_active":"2023-12-19T08:05:10.790183+0000","last_peered":"2023-12-19T08:05:10.790183+0000","last_clean":"2023-12-19T08:05:10.790183+0000","last_became_active":"2023-12-19T08:05:09.755099+0000","last_became_peered":"2023-12-19T08:05:09.755099+0000","last_unstale":"2023-12-19T08:05:10.790183+0000","last_undegraded":"2023-12-19T08:05:10.790183+0000","last_fullsized":"2023-12-19T08:05:10.790183+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:15:23.970857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776252+0000","last_change":"2023-12-19T08:05:10.776252+0000","last_active":"2023-12-19T08:05:10.776252+0000","last_peered":"2023-12-19T08:05:10.776252+0000","last_clean":"2023-12-19T08:05:10.776252+0000","last_became_active":"2023-12-19T08:05:10.775947+0000","last_became_peered":"2023-12-19T08:05:10.775947+0000","last_unstale":"2023-12-19T08:05:10.776252+0000","last_undegraded":"2023-12-19T08:05:10.776252+0000","last_fullsized":"2023-12-19T08:05:10.776252+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:59.702657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.754046+0000","last_change":"2023-12-19T08:05:09.749720+0000","last_active":"2023-12-19T08:05:09.754046+0000","last_peered":"2023-12-19T08:05:09.754046+0000","last_clean":"2023-12-19T08:05:09.754046+0000","last_became_active":"2023-12-19T08:05:09.749444+0000","last_became_peered":"2023-12-19T08:05:09.749444+0000","last_unstale":"2023-12-19T08:05:09.754046+0000","last_undegraded":"2023-12-19T08:05:09.754046+0000","last_fullsized":"2023-12-19T08:05:09.754046+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:44.364952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.773965+0000","last_change":"2023-12-19T08:05:10.773965+0000","last_active":"2023-12-19T08:05:10.773965+0000","last_peered":"2023-12-19T08:05:10.773965+0000","last_clean":"2023-12-19T08:05:10.773965+0000","last_became_active":"2023-12-19T08:05:10.773678+0000","last_became_peered":"2023-12-19T08:05:10.773678+0000","last_unstale":"2023-12-19T08:05:10.773965+0000","last_undegraded":"2023-12-19T08:05:10.773965+0000","last_fullsized":"2023-12-19T08:05:10.773965+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:17:08.185509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.758384+0000","last_change":"2023-12-19T08:05:09.751013+0000","last_active":"2023-12-19T08:05:09.758384+0000","last_peered":"2023-12-19T08:05:09.758384+0000","last_clean":"2023-12-19T08:05:09.758384+0000","last_became_active":"2023-12-19T08:05:09.748978+0000","last_became_peered":"2023-12-19T08:05:09.748978+0000","last_unstale":"2023-12-19T08:05:09.758384+0000","last_undegraded":"2023-12-19T08:05:09.758384+0000","last_fullsized":"2023-12-19T08:05:09.758384+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:04:08.099960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.769807+0000","last_change":"2023-12-19T08:05:10.769807+0000","last_active":"2023-12-19T08:05:10.769807+0000","last_peered":"2023-12-19T08:05:10.769807+0000","last_clean":"2023-12-19T08:05:10.769807+0000","last_became_active":"2023-12-19T08:05:10.769523+0000","last_became_peered":"2023-12-19T08:05:10.769523+0000","last_unstale":"2023-12-19T08:05:10.769807+0000","last_undegraded":"2023-12-19T08:05:10.769807+0000","last_fullsized":"2023-12-19T08:05:10.769807+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:11:38.581216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":18,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.792055+0000","last_change":"2023-12-19T08:05:09.753610+0000","last_active":"2023-12-19T08:05:10.792055+0000","last_peered":"2023-12-19T08:05:10.792055+0000","last_clean":"2023-12-19T08:05:10.792055+0000","last_became_active":"2023-12-19T08:05:09.753326+0000","last_became_peered":"2023-12-19T08:05:09.753326+0000","last_unstale":"2023-12-19T08:05:10.792055+0000","last_undegraded":"2023-12-19T08:05:10.792055+0000","last_fullsized":"2023-12-19T08:05:10.792055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:33:58.690695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.767947+0000","last_change":"2023-12-19T08:05:10.767947+0000","last_active":"2023-12-19T08:05:10.767947+0000","last_peered":"2023-12-19T08:05:10.767947+0000","last_clean":"2023-12-19T08:05:10.767947+0000","last_became_active":"2023-12-19T08:05:10.767638+0000","last_became_peered":"2023-12-19T08:05:10.767638+0000","last_unstale":"2023-12-19T08:05:10.767947+0000","last_undegraded":"2023-12-19T08:05:10.767947+0000","last_fullsized":"2023-12-19T08:05:10.767947+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:45:58.006252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.766770+0000","last_change":"2023-12-19T08:05:09.758625+0000","last_active":"2023-12-19T08:05:10.766770+0000","last_peered":"2023-12-19T08:05:10.766770+0000","last_clean":"2023-12-19T08:05:10.766770+0000","last_became_active":"2023-12-19T08:05:09.758259+0000","last_became_peered":"2023-12-19T08:05:09.758259+0000","last_unstale":"2023-12-19T08:05:10.766770+0000","last_undegraded":"2023-12-19T08:05:10.766770+0000","last_fullsized":"2023-12-19T08:05:10.766770+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:52:38.660192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772611+0000","last_change":"2023-12-19T08:05:10.772611+0000","last_active":"2023-12-19T08:05:10.772611+0000","last_peered":"2023-12-19T08:05:10.772611+0000","last_clean":"2023-12-19T08:05:10.772611+0000","last_became_active":"2023-12-19T08:05:10.772332+0000","last_became_peered":"2023-12-19T08:05:10.772332+0000","last_unstale":"2023-12-19T08:05:10.772611+0000","last_undegraded":"2023-12-19T08:05:10.772611+0000","last_fullsized":"2023-12-19T08:05:10.772611+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:04:28.968192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.763626+0000","last_change":"2023-12-19T08:05:09.761029+0000","last_active":"2023-12-19T08:05:10.763626+0000","last_peered":"2023-12-19T08:05:10.763626+0000","last_clean":"2023-12-19T08:05:10.763626+0000","last_became_active":"2023-12-19T08:05:09.760710+0000","last_became_peered":"2023-12-19T08:05:09.760710+0000","last_unstale":"2023-12-19T08:05:10.763626+0000","last_undegraded":"2023-12-19T08:05:10.763626+0000","last_fullsized":"2023-12-19T08:05:10.763626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:09.115449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.765517+0000","last_change":"2023-12-19T08:05:10.761050+0000","last_active":"2023-12-19T08:05:10.765517+0000","last_peered":"2023-12-19T08:05:10.765517+0000","last_clean":"2023-12-19T08:05:10.765517+0000","last_became_active":"2023-12-19T08:05:10.760201+0000","last_became_peered":"2023-12-19T08:05:10.760201+0000","last_unstale":"2023-12-19T08:05:10.765517+0000","last_undegraded":"2023-12-19T08:05:10.765517+0000","last_fullsized":"2023-12-19T08:05:10.765517+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:24:06.658231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768258+0000","last_change":"2023-12-19T08:05:09.765165+0000","last_active":"2023-12-19T08:05:10.768258+0000","last_peered":"2023-12-19T08:05:10.768258+0000","last_clean":"2023-12-19T08:05:10.768258+0000","last_became_active":"2023-12-19T08:05:09.764156+0000","last_became_peered":"2023-12-19T08:05:09.764156+0000","last_unstale":"2023-12-19T08:05:10.768258+0000","last_undegraded":"2023-12-19T08:05:10.768258+0000","last_fullsized":"2023-12-19T08:05:10.768258+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:09:20.928370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.764990+0000","last_change":"2023-12-19T08:05:10.764990+0000","last_active":"2023-12-19T08:05:10.764990+0000","last_peered":"2023-12-19T08:05:10.764990+0000","last_clean":"2023-12-19T08:05:10.764990+0000","last_became_active":"2023-12-19T08:05:10.764634+0000","last_became_peered":"2023-12-19T08:05:10.764634+0000","last_unstale":"2023-12-19T08:05:10.764990+0000","last_undegraded":"2023-12-19T08:05:10.764990+0000","last_fullsized":"2023-12-19T08:05:10.764990+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:46:03.707714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.788438+0000","last_change":"2023-12-19T08:05:09.754825+0000","last_active":"2023-12-19T08:05:10.788438+0000","last_peered":"2023-12-19T08:05:10.788438+0000","last_clean":"2023-12-19T08:05:10.788438+0000","last_became_active":"2023-12-19T08:05:09.754484+0000","last_became_peered":"2023-12-19T08:05:09.754484+0000","last_unstale":"2023-12-19T08:05:10.788438+0000","last_undegraded":"2023-12-19T08:05:10.788438+0000","last_fullsized":"2023-12-19T08:05:10.788438+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:29:00.220530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768967+0000","last_change":"2023-12-19T08:05:10.768967+0000","last_active":"2023-12-19T08:05:10.768967+0000","last_peered":"2023-12-19T08:05:10.768967+0000","last_clean":"2023-12-19T08:05:10.768967+0000","last_became_active":"2023-12-19T08:05:10.768651+0000","last_became_peered":"2023-12-19T08:05:10.768651+0000","last_unstale":"2023-12-19T08:05:10.768967+0000","last_undegraded":"2023-12-19T08:05:10.768967+0000","last_fullsized":"2023-12-19T08:05:10.768967+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:25:32.228235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.761259+0000","last_change":"2023-12-19T08:05:09.757787+0000","last_active":"2023-12-19T08:05:09.761259+0000","last_peered":"2023-12-19T08:05:09.761259+0000","last_clean":"2023-12-19T08:05:09.761259+0000","last_became_active":"2023-12-19T08:05:09.757517+0000","last_became_peered":"2023-12-19T08:05:09.757517+0000","last_unstale":"2023-12-19T08:05:09.761259+0000","last_undegraded":"2023-12-19T08:05:09.761259+0000","last_fullsized":"2023-12-19T08:05:09.761259+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:04:00.843396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.776792+0000","last_change":"2023-12-19T08:05:10.776792+0000","last_active":"2023-12-19T08:05:10.776792+0000","last_peered":"2023-12-19T08:05:10.776792+0000","last_clean":"2023-12-19T08:05:10.776792+0000","last_became_active":"2023-12-19T08:05:10.776509+0000","last_became_peered":"2023-12-19T08:05:10.776509+0000","last_unstale":"2023-12-19T08:05:10.776792+0000","last_undegraded":"2023-12-19T08:05:10.776792+0000","last_fullsized":"2023-12-19T08:05:10.776792+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:56.176007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.762272+0000","last_change":"2023-12-19T08:05:09.759657+0000","last_active":"2023-12-19T08:05:09.762272+0000","last_peered":"2023-12-19T08:05:09.762272+0000","last_clean":"2023-12-19T08:05:09.762272+0000","last_became_active":"2023-12-19T08:05:09.757285+0000","last_became_peered":"2023-12-19T08:05:09.757285+0000","last_unstale":"2023-12-19T08:05:09.762272+0000","last_undegraded":"2023-12-19T08:05:09.762272+0000","last_fullsized":"2023-12-19T08:05:09.762272+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:07:10.755579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.772239+0000","last_change":"2023-12-19T08:05:10.772239+0000","last_active":"2023-12-19T08:05:10.772239+0000","last_peered":"2023-12-19T08:05:10.772239+0000","last_clean":"2023-12-19T08:05:10.772239+0000","last_became_active":"2023-12-19T08:05:10.771953+0000","last_became_peered":"2023-12-19T08:05:10.771953+0000","last_unstale":"2023-12-19T08:05:10.772239+0000","last_undegraded":"2023-12-19T08:05:10.772239+0000","last_fullsized":"2023-12-19T08:05:10.772239+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:02:59.883271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.761314+0000","last_change":"2023-12-19T08:05:09.758304+0000","last_active":"2023-12-19T08:05:09.761314+0000","last_peered":"2023-12-19T08:05:09.761314+0000","last_clean":"2023-12-19T08:05:09.761314+0000","last_became_active":"2023-12-19T08:05:09.758016+0000","last_became_peered":"2023-12-19T08:05:09.758016+0000","last_unstale":"2023-12-19T08:05:09.761314+0000","last_undegraded":"2023-12-19T08:05:09.761314+0000","last_fullsized":"2023-12-19T08:05:09.761314+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:18.623976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.768293+0000","last_change":"2023-12-19T08:05:10.768293+0000","last_active":"2023-12-19T08:05:10.768293+0000","last_peered":"2023-12-19T08:05:10.768293+0000","last_clean":"2023-12-19T08:05:10.768293+0000","last_became_active":"2023-12-19T08:05:10.768011+0000","last_became_peered":"2023-12-19T08:05:10.768011+0000","last_unstale":"2023-12-19T08:05:10.768293+0000","last_undegraded":"2023-12-19T08:05:10.768293+0000","last_fullsized":"2023-12-19T08:05:10.768293+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:14:47.996715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":12,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.788658+0000","last_change":"2023-12-19T08:05:09.750171+0000","last_active":"2023-12-19T08:05:10.788658+0000","last_peered":"2023-12-19T08:05:10.788658+0000","last_clean":"2023-12-19T08:05:10.788658+0000","last_became_active":"2023-12-19T08:05:09.748948+0000","last_became_peered":"2023-12-19T08:05:09.748948+0000","last_unstale":"2023-12-19T08:05:10.788658+0000","last_undegraded":"2023-12-19T08:05:10.788658+0000","last_fullsized":"2023-12-19T08:05:10.788658+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:34.777001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777094+0000","last_change":"2023-12-19T08:05:10.771680+0000","last_active":"2023-12-19T08:05:10.777094+0000","last_peered":"2023-12-19T08:05:10.777094+0000","last_clean":"2023-12-19T08:05:10.777094+0000","last_became_active":"2023-12-19T08:05:10.771387+0000","last_became_peered":"2023-12-19T08:05:10.771387+0000","last_unstale":"2023-12-19T08:05:10.777094+0000","last_undegraded":"2023-12-19T08:05:10.777094+0000","last_fullsized":"2023-12-19T08:05:10.777094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:32.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.757304+0000","last_change":"2023-12-19T08:05:09.750759+0000","last_active":"2023-12-19T08:05:09.757304+0000","last_peered":"2023-12-19T08:05:09.757304+0000","last_clean":"2023-12-19T08:05:09.757304+0000","last_became_active":"2023-12-19T08:05:09.749668+0000","last_became_peered":"2023-12-19T08:05:09.749668+0000","last_unstale":"2023-12-19T08:05:09.757304+0000","last_undegraded":"2023-12-19T08:05:09.757304+0000","last_fullsized":"2023-12-19T08:05:09.757304+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:41.999018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778113+0000","last_change":"2023-12-19T08:05:10.778113+0000","last_active":"2023-12-19T08:05:10.778113+0000","last_peered":"2023-12-19T08:05:10.778113+0000","last_clean":"2023-12-19T08:05:10.778113+0000","last_became_active":"2023-12-19T08:05:10.776391+0000","last_became_peered":"2023-12-19T08:05:10.776391+0000","last_unstale":"2023-12-19T08:05:10.778113+0000","last_undegraded":"2023-12-19T08:05:10.778113+0000","last_fullsized":"2023-12-19T08:05:10.778113+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:05.748640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.764281+0000","last_change":"2023-12-19T08:05:09.761947+0000","last_active":"2023-12-19T08:05:09.764281+0000","last_peered":"2023-12-19T08:05:09.764281+0000","last_clean":"2023-12-19T08:05:09.764281+0000","last_became_active":"2023-12-19T08:05:09.761624+0000","last_became_peered":"2023-12-19T08:05:09.761624+0000","last_unstale":"2023-12-19T08:05:09.764281+0000","last_undegraded":"2023-12-19T08:05:09.764281+0000","last_fullsized":"2023-12-19T08:05:09.764281+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:34.273452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.778150+0000","last_change":"2023-12-19T08:05:10.771421+0000","last_active":"2023-12-19T08:05:10.778150+0000","last_peered":"2023-12-19T08:05:10.778150+0000","last_clean":"2023-12-19T08:05:10.778150+0000","last_became_active":"2023-12-19T08:05:10.771051+0000","last_became_peered":"2023-12-19T08:05:10.771051+0000","last_unstale":"2023-12-19T08:05:10.778150+0000","last_undegraded":"2023-12-19T08:05:10.778150+0000","last_fullsized":"2023-12-19T08:05:10.778150+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.746373+0000","last_change":"2023-12-19T08:05:09.755735+0000","last_active":"2023-12-19T08:05:10.746373+0000","last_peered":"2023-12-19T08:05:10.746373+0000","last_clean":"2023-12-19T08:05:10.746373+0000","last_became_active":"2023-12-19T08:05:09.755429+0000","last_became_peered":"2023-12-19T08:05:09.755429+0000","last_unstale":"2023-12-19T08:05:10.746373+0000","last_undegraded":"2023-12-19T08:05:10.746373+0000","last_fullsized":"2023-12-19T08:05:10.746373+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:10.432397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.777206+0000","last_change":"2023-12-19T08:05:10.777206+0000","last_active":"2023-12-19T08:05:10.777206+0000","last_peered":"2023-12-19T08:05:10.777206+0000","last_clean":"2023-12-19T08:05:10.777206+0000","last_became_active":"2023-12-19T08:05:10.776907+0000","last_became_peered":"2023-12-19T08:05:10.776907+0000","last_unstale":"2023-12-19T08:05:10.777206+0000","last_undegraded":"2023-12-19T08:05:10.777206+0000","last_fullsized":"2023-12-19T08:05:10.777206+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:00:20.721351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.755183+0000","last_change":"2023-12-19T08:05:09.758631+0000","last_active":"2023-12-19T08:05:10.755183+0000","last_peered":"2023-12-19T08:05:10.755183+0000","last_clean":"2023-12-19T08:05:10.755183+0000","last_became_active":"2023-12-19T08:05:09.758326+0000","last_became_peered":"2023-12-19T08:05:09.758326+0000","last_unstale":"2023-12-19T08:05:10.755183+0000","last_undegraded":"2023-12-19T08:05:10.755183+0000","last_fullsized":"2023-12-19T08:05:10.755183+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:52.082968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.774231+0000","last_change":"2023-12-19T08:05:10.774231+0000","last_active":"2023-12-19T08:05:10.774231+0000","last_peered":"2023-12-19T08:05:10.774231+0000","last_clean":"2023-12-19T08:05:10.774231+0000","last_became_active":"2023-12-19T08:05:10.772602+0000","last_became_peered":"2023-12-19T08:05:10.772602+0000","last_unstale":"2023-12-19T08:05:10.774231+0000","last_undegraded":"2023-12-19T08:05:10.774231+0000","last_fullsized":"2023-12-19T08:05:10.774231+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:53:40.273343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.758344+0000","last_change":"2023-12-19T08:05:09.752552+0000","last_active":"2023-12-19T08:05:09.758344+0000","last_peered":"2023-12-19T08:05:09.758344+0000","last_clean":"2023-12-19T08:05:09.758344+0000","last_became_active":"2023-12-19T08:05:09.752177+0000","last_became_peered":"2023-12-19T08:05:09.752177+0000","last_unstale":"2023-12-19T08:05:09.758344+0000","last_undegraded":"2023-12-19T08:05:09.758344+0000","last_fullsized":"2023-12-19T08:05:09.758344+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:56.165286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":8,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.782145+0000","last_change":"2023-12-19T08:05:10.782145+0000","last_active":"2023-12-19T08:05:10.782145+0000","last_peered":"2023-12-19T08:05:10.782145+0000","last_clean":"2023-12-19T08:05:10.782145+0000","last_became_active":"2023-12-19T08:05:10.779891+0000","last_became_peered":"2023-12-19T08:05:10.779891+0000","last_unstale":"2023-12-19T08:05:10.782145+0000","last_undegraded":"2023-12-19T08:05:10.782145+0000","last_fullsized":"2023-12-19T08:05:10.782145+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.763540+0000","last_change":"2023-12-19T08:05:09.761593+0000","last_active":"2023-12-19T08:05:09.763540+0000","last_peered":"2023-12-19T08:05:09.763540+0000","last_clean":"2023-12-19T08:05:09.763540+0000","last_became_active":"2023-12-19T08:05:09.761305+0000","last_became_peered":"2023-12-19T08:05:09.761305+0000","last_unstale":"2023-12-19T08:05:09.763540+0000","last_undegraded":"2023-12-19T08:05:09.763540+0000","last_fullsized":"2023-12-19T08:05:09.763540+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:40:58.692928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.786190+0000","last_change":"2023-12-19T08:05:10.779392+0000","last_active":"2023-12-19T08:05:10.786190+0000","last_peered":"2023-12-19T08:05:10.786190+0000","last_clean":"2023-12-19T08:05:10.786190+0000","last_became_active":"2023-12-19T08:05:10.779120+0000","last_became_peered":"2023-12-19T08:05:10.779120+0000","last_unstale":"2023-12-19T08:05:10.786190+0000","last_undegraded":"2023-12-19T08:05:10.786190+0000","last_fullsized":"2023-12-19T08:05:10.786190+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:43.538524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.760590+0000","last_change":"2023-12-19T08:05:09.755414+0000","last_active":"2023-12-19T08:05:09.760590+0000","last_peered":"2023-12-19T08:05:09.760590+0000","last_clean":"2023-12-19T08:05:09.760590+0000","last_became_active":"2023-12-19T08:05:09.755149+0000","last_became_peered":"2023-12-19T08:05:09.755149+0000","last_unstale":"2023-12-19T08:05:09.760590+0000","last_undegraded":"2023-12-19T08:05:09.760590+0000","last_fullsized":"2023-12-19T08:05:09.760590+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:49.412606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.793525+0000","last_change":"2023-12-19T08:05:10.790478+0000","last_active":"2023-12-19T08:05:10.793525+0000","last_peered":"2023-12-19T08:05:10.793525+0000","last_clean":"2023-12-19T08:05:10.793525+0000","last_became_active":"2023-12-19T08:05:10.790177+0000","last_became_peered":"2023-12-19T08:05:10.790177+0000","last_unstale":"2023-12-19T08:05:10.793525+0000","last_undegraded":"2023-12-19T08:05:10.793525+0000","last_fullsized":"2023-12-19T08:05:10.793525+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:09.713091+0000","last_clean_scrub_stamp":"2023-12-19T08:05:09.713091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:19:28.266097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":11,"reported_epoch":128,"state":"active+clean","last_fresh":"2023-12-19T08:05:10.790594+0000","last_change":"2023-12-19T08:05:09.759704+0000","last_active":"2023-12-19T08:05:10.790594+0000","last_peered":"2023-12-19T08:05:10.790594+0000","last_clean":"2023-12-19T08:05:10.790594+0000","last_became_active":"2023-12-19T08:05:09.759290+0000","last_became_peered":"2023-12-19T08:05:09.759290+0000","last_unstale":"2023-12-19T08:05:10.790594+0000","last_undegraded":"2023-12-19T08:05:10.790594+0000","last_fullsized":"2023-12-19T08:05:10.790594+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:05:08.701128+0000","last_clean_scrub_stamp":"2023-12-19T08:05:08.701128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:33:08.436215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'227","reported_seq":641,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-12-19T08:05:09.750072+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:05:09.750072+0000","last_peered":"2023-12-19T08:05:09.750072+0000","last_clean":"2023-12-19T08:05:09.750072+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:05:09.750072+0000","last_undegraded":"2023-12-19T08:05:09.750072+0000","last_fullsized":"2023-12-19T08:05:09.750072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575062,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1419254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2170000000000001,"15min":1.2509999999999999},"min":{"1min":0.72399999999999998,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.524,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.258,"5min":1.2769999999999999,"15min":1.274},"min":{"1min":0.95399999999999996,"5min":0.749,"15min":0.70799999999999996},"max":{"1min":1.5800000000000001,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":0.82299999999999995}]},{"osd":1,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.161,"15min":1.1639999999999999},"min":{"1min":0.71399999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.464,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.204,"15min":1.1819999999999999},"min":{"1min":0.85599999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.833,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.345}]},{"osd":2,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.163,"15min":1.165},"min":{"1min":0.82699999999999996,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.3540000000000001,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.2889999999999999,"15min":1.256},"min":{"1min":0.85499999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.946,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.236}]},{"osd":3,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2849999999999999,"15min":1.292},"min":{"1min":0.90100000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.5509999999999999,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.319},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3799999999999999,"15min":1.353},"min":{"1min":1.099,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5509999999999999,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.3879999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.3260000000000001,"15min":1.2350000000000001},"min":{"1min":0.81499999999999995,"5min":0.71899999999999997,"15min":0.67300000000000004},"max":{"1min":2.0419999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.3069999999999999,"15min":1.242},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.546,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.4319999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.335,"15min":1.2509999999999999},"min":{"1min":0.79600000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.657,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.536},{"interface":"front","average":{"1min":1.383,"5min":1.4339999999999999,"15min":1.369},"min":{"1min":1.016,"5min":0.751,"15min":0.751},"max":{"1min":2.1219999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.524}]},{"osd":6,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.51,"15min":1.4259999999999999},"min":{"1min":1.0629999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7649999999999999,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.617},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.4370000000000001,"15min":1.385},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.58299999999999996},"max":{"1min":1.7210000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.5069999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575061,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302952,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068888,"statfs":{"total":96636764160,"available":96326541312,"internally_reserved":0,"allocated":1605632,"data_stored":830838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.085,"15min":1.099},"min":{"1min":0.80400000000000005,"5min":0.70699999999999996,"15min":0.36799999999999999},"max":{"1min":1.48,"5min":1.5569999999999999,"15min":1.665},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1419999999999999,"15min":1.137},"min":{"1min":0.92400000000000004,"5min":0.624,"15min":0.49399999999999999},"max":{"1min":1.734,"5min":1.756,"15min":1.756},"last":1.173}]},{"osd":1,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.149,"15min":1.1559999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.53400000000000003},"max":{"1min":1.552,"5min":1.675,"15min":1.675},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.119,"15min":1.1519999999999999},"min":{"1min":0.84199999999999997,"5min":0.73799999999999999,"15min":0.51900000000000002},"max":{"1min":1.4299999999999999,"5min":1.5349999999999999,"15min":1.7470000000000001},"last":0.84599999999999997}]},{"osd":2,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2150000000000001,"15min":1.2330000000000001},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.59099999999999997},"max":{"1min":1.5589999999999999,"5min":1.746,"15min":1.847},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.2150000000000001,"15min":1.2130000000000001},"min":{"1min":0.94999999999999996,"5min":0.70799999999999996,"15min":0.58999999999999997},"max":{"1min":1.6200000000000001,"5min":1.694,"15min":1.7929999999999999},"last":0.89300000000000002}]},{"osd":3,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.234,"15min":1.2629999999999999},"min":{"1min":0.83699999999999997,"5min":0.77200000000000002,"15min":0.65400000000000003},"max":{"1min":1.6519999999999999,"5min":1.8200000000000001,"15min":1.958},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.302,"15min":1.304},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.63200000000000001},"max":{"1min":1.6060000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.361}]},{"osd":4,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.66100000000000003,"5min":0.55500000000000005,"15min":0.47199999999999998},"max":{"1min":1.546,"5min":1.7130000000000001,"15min":1.8480000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.389,"15min":1.395},"min":{"1min":0.96799999999999997,"5min":0.72299999999999998,"15min":0.70099999999999996},"max":{"1min":1.843,"5min":1.974,"15min":2.1440000000000001},"last":1.119}]},{"osd":5,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3029999999999999,"15min":1.288},"min":{"1min":0.81799999999999995,"5min":0.70799999999999996,"15min":0.67500000000000004},"max":{"1min":1.6950000000000001,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.413},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3109999999999999,"15min":1.296},"min":{"1min":0.77600000000000002,"5min":0.75700000000000001,"15min":0.56499999999999995},"max":{"1min":1.6200000000000001,"5min":1.8100000000000001,"15min":2.2450000000000001},"last":1.095}]},{"osd":7,"last update":"Tue Dec 19 08:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.399,"15min":1.3700000000000001},"min":{"1min":0.90700000000000003,"5min":0.73599999999999999,"15min":0.55100000000000005},"max":{"1min":1.8049999999999999,"5min":1.9239999999999999,"15min":2.04},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4550000000000001,"15min":1.4219999999999999},"min":{"1min":1.048,"5min":0.76600000000000001,"15min":0.58099999999999996},"max":{"1min":1.835,"5min":2.137,"15min":2.137},"last":1.4610000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575061,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298848,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072992,"statfs":{"total":96636764160,"available":96330743808,"internally_reserved":0,"allocated":1597440,"data_stored":835854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2050000000000001,"15min":1.2390000000000001},"min":{"1min":1.0649999999999999,"5min":0.66000000000000003,"15min":0.53000000000000003},"max":{"1min":1.6140000000000001,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.2250000000000001,"15min":1.268},"min":{"1min":1.0700000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.538,"5min":1.8700000000000001,"15min":1.954},"last":0.88400000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1140000000000001,"15min":1.1399999999999999},"min":{"1min":0.89900000000000002,"5min":0.61899999999999999,"15min":0.61699999999999999},"max":{"1min":1.5620000000000001,"5min":1.585,"15min":1.958},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.173,"15min":1.1819999999999999},"min":{"1min":0.91400000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.627,"5min":2.7679999999999998,"15min":2.7679999999999998},"last":1.097}]},{"osd":2,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.204,"15min":1.204},"min":{"1min":0.85499999999999998,"5min":0.72899999999999998,"15min":0.68500000000000005},"max":{"1min":1.571,"5min":1.821,"15min":2.0259999999999998},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.88700000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6499999999999999,"5min":1.6679999999999999,"15min":1.7490000000000001},"last":1.137}]},{"osd":3,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2430000000000001,"15min":1.254},"min":{"1min":0.92700000000000005,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.593,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.327,"15min":1.3129999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.76600000000000001},"max":{"1min":1.6379999999999999,"5min":2.1560000000000001,"15min":2.2130000000000001},"last":1.0569999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.1970000000000001,"15min":1.1990000000000001},"min":{"1min":0.86899999999999999,"5min":0.629,"15min":0.629},"max":{"1min":1.6970000000000001,"5min":1.7749999999999999,"15min":2.1419999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.369,"5min":1.3,"15min":1.2849999999999999},"min":{"1min":0.85599999999999998,"5min":0.76700000000000002,"15min":0.59199999999999997},"max":{"1min":1.673,"5min":2.843,"15min":2.843},"last":1.3149999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.337,"15min":1.3140000000000001},"min":{"1min":0.84699999999999998,"5min":0.754,"15min":0.754},"max":{"1min":1.6879999999999999,"5min":2.0190000000000001,"15min":2.121},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.425,"5min":1.381,"15min":1.359},"min":{"1min":0.98399999999999999,"5min":0.90800000000000003,"15min":0.77300000000000002},"max":{"1min":1.6140000000000001,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":1.2549999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:04:47 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4119999999999999,"15min":1.419},"min":{"1min":1.1639999999999999,"5min":0.84899999999999998,"15min":0.74399999999999999},"max":{"1min":1.675,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":1.403},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4850000000000001,"15min":1.4570000000000001},"min":{"1min":1.206,"5min":0.96499999999999997,"15min":0.91500000000000004},"max":{"1min":1.7410000000000001,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.96299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575061,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307036,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064804,"statfs":{"total":96636764160,"available":96322359296,"internally_reserved":0,"allocated":1593344,"data_stored":828818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.1859999999999999,"15min":1.21},"min":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.7210000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.92400000000000004,"5min":0.80000000000000004,"15min":0.70299999999999996},"max":{"1min":1.5649999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.2}]},{"osd":1,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.149,"15min":1.157},"min":{"1min":0.69799999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5640000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.151},{"interface":"front","average":{"1min":1.127,"5min":1.1100000000000001,"15min":1.1379999999999999},"min":{"1min":0.63200000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4990000000000001,"5min":1.569,"15min":1.579},"last":0.754}]},{"osd":2,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0860000000000001,"15min":1.1299999999999999},"min":{"1min":0.81599999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.52,"5min":1.5369999999999999,"15min":1.746},"last":0.878},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2090000000000001,"15min":1.2290000000000001},"min":{"1min":0.79200000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.5980000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.92300000000000004}]},{"osd":3,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.264,"15min":1.276},"min":{"1min":0.76300000000000001,"5min":0.70999999999999996,"15min":0.65900000000000003},"max":{"1min":1.6379999999999999,"5min":1.831,"15min":1.831},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.337,"5min":1.2869999999999999,"15min":1.3220000000000001},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":1.8580000000000001,"15min":1.8580000000000001},"last":1.1100000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.23,"15min":1.1990000000000001},"min":{"1min":0.76100000000000001,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.6850000000000001,"5min":1.905,"15min":1.905},"last":1.01},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2070000000000001,"15min":1.198},"min":{"1min":0.72999999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.704,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.96399999999999997}]},{"osd":6,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.248,"15min":1.2270000000000001},"min":{"1min":0.88400000000000001,"5min":0.73899999999999999,"15min":0.67400000000000004},"max":{"1min":1.5409999999999999,"5min":1.748,"15min":1.748},"last":0.996},{"interface":"front","average":{"1min":1.351,"5min":1.409,"15min":1.3759999999999999},"min":{"1min":0.97099999999999997,"5min":0.70899999999999996,"15min":0.67900000000000005},"max":{"1min":1.6799999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.3240000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:04:41 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.359,"15min":1.341},"min":{"1min":1.0009999999999999,"5min":0.58799999999999997,"15min":0.46300000000000002},"max":{"1min":1.756,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.036},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.407,"15min":1.387},"min":{"1min":0.96799999999999997,"5min":0.80200000000000005,"15min":0.70999999999999996},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.175}]}]},{"osd":3,"up_from":13,"seq":55834575061,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":829280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.85299999999999998,"15min":0.85099999999999998},"min":{"1min":0.51500000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.145,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96099999999999997,"15min":0.96299999999999997},"min":{"1min":0.52700000000000002,"5min":0.49199999999999999,"15min":0.39200000000000002},"max":{"1min":1.3620000000000001,"5min":1.962,"15min":1.962},"last":1.1559999999999999}]},{"osd":1,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.89100000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.45600000000000002},"max":{"1min":1.228,"5min":1.609,"15min":1.609},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0389999999999999,"15min":1.038},"min":{"1min":0.77300000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.399,"5min":2.423,"15min":2.423},"last":0.90300000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0489999999999999,"15min":1.077},"min":{"1min":0.73999999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5580000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1759999999999999,"15min":1.163},"min":{"1min":0.70599999999999996,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.52,"5min":2.48,"15min":2.48},"last":1.0209999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2290000000000001,"15min":1.256},"min":{"1min":0.93700000000000006,"5min":0.69199999999999995,"15min":0.66800000000000004},"max":{"1min":1.55,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.222,"15min":1.2430000000000001},"min":{"1min":0.82899999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.464,"5min":1.649,"15min":1.8120000000000001},"last":1.2430000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.91600000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.4790000000000001,"5min":1.996,"15min":1.996},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.339,"15min":1.335},"min":{"1min":0.85299999999999998,"5min":0.753,"15min":0.753},"max":{"1min":1.577,"5min":2.161,"15min":2.161},"last":1.552}]},{"osd":6,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.4079999999999999,"15min":1.3879999999999999},"min":{"1min":0.89800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6439999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3300000000000001,"15min":1.363},"min":{"1min":1.0429999999999999,"5min":0.89700000000000002,"15min":0.82499999999999996},"max":{"1min":1.774,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.639}]},{"osd":7,"last update":"Tue Dec 19 08:04:36 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.4490000000000001,"15min":1.4550000000000001},"min":{"1min":0.999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8640000000000001,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.571},{"interface":"front","average":{"1min":1.403,"5min":1.4079999999999999,"15min":1.411},"min":{"1min":0.97499999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.752,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.3879999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575061,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307032,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064808,"statfs":{"total":96636764160,"available":96322363392,"internally_reserved":0,"allocated":1589248,"data_stored":828760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89100000000000001,"15min":0.90400000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.437},"max":{"1min":1.3700000000000001,"5min":1.776,"15min":1.776},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.0129999999999999,"15min":1.0489999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.452,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78900000000000003}]},{"osd":1,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.351,"5min":1.669,"15min":1.669},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.98099999999999998,"15min":1.008},"min":{"1min":0.74399999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.5429999999999999,"5min":1.835,"15min":1.835},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94999999999999996,"15min":0.96699999999999997},"min":{"1min":0.60999999999999999,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.361,"5min":1.657,"15min":1.667},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.0980000000000001,"15min":1.121},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.5289999999999999,"5min":1.827,"15min":1.827},"last":0.871}]},{"osd":4,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.325,"15min":1.3240000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.631,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.306,"15min":1.3009999999999999},"min":{"1min":0.98099999999999998,"5min":0.85899999999999999,"15min":0.82099999999999995},"max":{"1min":1.6859999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.069}]},{"osd":5,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.329,"15min":1.341},"min":{"1min":1.089,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.621,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.2509999999999999,"15min":1.2569999999999999},"min":{"1min":0.86199999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.5820000000000001,"5min":1.8,"15min":1.8},"last":1.234}]},{"osd":6,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.282,"15min":1.296},"min":{"1min":0.97599999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7390000000000001,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.306},{"interface":"front","average":{"1min":1.391,"5min":1.367,"15min":1.369},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.92100000000000004},"max":{"1min":1.7130000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.1080000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:04:39 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.375,"15min":1.3899999999999999},"min":{"1min":1.1339999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.9119999999999999,"5min":1.9730000000000001,"15min":2.214},"last":1.177},{"interface":"front","average":{"1min":1.363,"5min":1.3460000000000001,"15min":1.3660000000000001},"min":{"1min":1.0649999999999999,"5min":0.98699999999999999,"15min":0.88200000000000001},"max":{"1min":1.6779999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.355}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307044,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064796,"statfs":{"total":96636764160,"available":96322351104,"internally_reserved":0,"allocated":1601536,"data_stored":835888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.876,"15min":0.85999999999999999},"min":{"1min":0.51700000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2470000000000001,"5min":1.661,"15min":1.661},"last":1.141},{"interface":"front","average":{"1min":0.871,"5min":0.93799999999999994,"15min":0.94099999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.375,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.79400000000000004}]},{"osd":2,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.94599999999999995,"15min":0.92400000000000004},"min":{"1min":0.57299999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.29,"5min":2.96,"15min":2.96},"last":1.054},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0780000000000001,"15min":1.0529999999999999},"min":{"1min":0.78600000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.3460000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1990000000000001,"15min":1.1559999999999999},"min":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.359,"5min":3.1800000000000002,"15min":3.1800000000000002},"last":1.008},{"interface":"front","average":{"1min":0.999,"5min":1.079,"15min":1.0660000000000001},"min":{"1min":0.60299999999999998,"5min":0.55500000000000005,"15min":0.51200000000000001},"max":{"1min":1.385,"5min":2.407,"15min":2.407},"last":0.61599999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.181,"15min":1.1899999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.879,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3500000000000001,"15min":1.3089999999999999},"min":{"1min":0.89300000000000002,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.649,"5min":3.04,"15min":3.04},"last":1.206}]},{"osd":5,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.284,"15min":1.278},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.5,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.103},{"interface":"front","average":{"1min":1.272,"5min":1.3129999999999999,"15min":1.288},"min":{"1min":0.85899999999999999,"5min":0.83199999999999996,"15min":0.63800000000000001},"max":{"1min":1.976,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.3380000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.4019999999999999,"15min":1.3819999999999999},"min":{"1min":0.93899999999999995,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.726,"5min":3.089,"15min":3.089},"last":1.284},{"interface":"front","average":{"1min":1.304,"5min":1.373,"15min":1.367},"min":{"1min":0.76300000000000001,"5min":0.75900000000000001,"15min":0.747},"max":{"1min":1.665,"5min":2.46,"15min":2.46},"last":0.98899999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:04:34 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.407,"15min":1.417},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.88500000000000001},"max":{"1min":1.7609999999999999,"5min":2.504,"15min":2.504},"last":1.381},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.401,"15min":1.3839999999999999},"min":{"1min":0.94299999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7749999999999999,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.256}]}]},{"osd":0,"up_from":13,"seq":55834575061,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307620,"kb_used_data":2140,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064220,"statfs":{"total":96636764160,"available":96321761280,"internally_reserved":0,"allocated":2191360,"data_stored":1419709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.83799999999999997,"15min":0.84499999999999997},"min":{"1min":0.58999999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.349,"5min":1.4590000000000001,"15min":1.5409999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.88,"5min":0.89100000000000001,"15min":0.91500000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.496},"max":{"1min":1.466,"5min":1.466,"15min":1.5289999999999999},"last":1.0660000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":0.97499999999999998,"15min":0.98099999999999998},"min":{"1min":0.79500000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.258},{"interface":"front","average":{"1min":1.012,"5min":1.028,"15min":1.034},"min":{"1min":0.73499999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.3779999999999999,"5min":1.585,"15min":1.6000000000000001},"last":0.95399999999999996}]},{"osd":3,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.94699999999999995,"15min":0.96399999999999997},"min":{"1min":0.68700000000000006,"5min":0.54500000000000004,"15min":0.51400000000000001},"max":{"1min":1.4790000000000001,"5min":1.5529999999999999,"15min":1.671},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.1240000000000001,"15min":1.133},"min":{"1min":0.93899999999999995,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.528,"5min":1.663,"15min":1.6659999999999999},"last":1.1759999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3240000000000001,"15min":1.3149999999999999},"min":{"1min":1.073,"5min":0.753,"15min":0.71399999999999997},"max":{"1min":1.6319999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":1.0880000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.6519999999999999,"5min":1.855,"15min":1.8999999999999999},"last":1.5660000000000001}]},{"osd":5,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":1.0760000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.7250000000000001,"5min":2.085,"15min":2.085},"last":1.427},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3580000000000001},"min":{"1min":1.135,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5940000000000001,"5min":2.125,"15min":2.125},"last":1.4690000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2669999999999999,"15min":1.264},"min":{"1min":1.0429999999999999,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.5629999999999999,"5min":1.97,"15min":1.97},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":1.3220000000000001},"min":{"1min":0.95399999999999996,"5min":0.80700000000000005,"15min":0.75600000000000001},"max":{"1min":1.671,"5min":1.871,"15min":1.871},"last":1.403}]},{"osd":7,"last update":"Tue Dec 19 08:04:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3169999999999999,"15min":1.3120000000000001},"min":{"1min":0.88200000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.5780000000000001,"5min":1.887,"15min":1.887},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.377,"15min":1.3700000000000001},"min":{"1min":1.1619999999999999,"5min":0.79100000000000004,"15min":0.67700000000000005},"max":{"1min":1.6259999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.5409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:05:15.494 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:05:15.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:05:15.702 INFO:tasks.ceph.mds.d.smithi143.stderr:2023-12-19T08:05:15.702+0000 7fa08a08b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:05:15.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.706+0000 7f6ad4d85700 1 Processor -- start 2023-12-19T08:05:15.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.706+0000 7f6ad4d85700 1 -- start start 2023-12-19T08:05:15.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ad4d85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 0x7f6ad00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ad4d85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ad00cb970 0x7f6ad00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ad4d85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00cbdf0 con 0x7f6ad00ca2c0 2023-12-19T08:05:15.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ad4d85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00cc0d0 con 0x7f6ad00cb970 2023-12-19T08:05:15.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ad4d85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00cc3b0 con 0x7f6ad0158e20 2023-12-19T08:05:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ad00cb970 0x7f6ad00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ad00cb970 0x7f6ad00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56248/0 (socket says 172.21.15.111:56248) 2023-12-19T08:05:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 -- 172.21.15.111:0/239692983 learned_addr learned my addr 172.21.15.111:0/239692983 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ace59c700 1 --2- 172.21.15.111:0/239692983 >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 0x7f6ad00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 -- 172.21.15.111:0/239692983 >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 msgr2=0x7f6ad00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 --2- 172.21.15.111:0/239692983 >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 0x7f6ad00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 -- 172.21.15.111:0/239692983 >> v1:172.21.15.143:6789/0 conn(0x7f6ad00ca2c0 legacy=0x7f6ad006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 -- 172.21.15.111:0/239692983 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00cc690 con 0x7f6ad00cb970 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ace59c700 1 --2- 172.21.15.111:0/239692983 >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 0x7f6ad00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:15.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6ac7fff700 1 --2- 172.21.15.111:0/239692983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad00cb970 0x7f6ad00ceb00 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80099b0 tx=0x7f6ab8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae2d9c85b06f7e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6acdd9b700 1 -- 172.21.15.111:0/239692983 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800b820 con 0x7f6ad00cb970 2023-12-19T08:05:15.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6acdd9b700 1 -- 172.21.15.111:0/239692983 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab800b980 con 0x7f6ad00cb970 2023-12-19T08:05:15.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.710+0000 7f6acdd9b700 1 -- 172.21.15.111:0/239692983 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab800bd20 con 0x7f6ad00cb970 2023-12-19T08:05:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- 172.21.15.111:0/239692983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad00cb970 msgr2=0x7f6ad00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/239692983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad00cb970 0x7f6ad00ceb00 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80099b0 tx=0x7f6ab8009250 comp rx=0 tx=0).stop 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- 172.21.15.111:0/239692983 shutdown_connections 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/239692983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad00cb970 0x7f6ad00ceb00 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/239692983 >> v2:172.21.15.143:3300/0 conn(0x7f6ad0158e20 0x7f6ad00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- 172.21.15.111:0/239692983 >> 172.21.15.111:0/239692983 conn(0x7f6ad0067bb0 msgr2=0x7f6ad00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- 172.21.15.111:0/239692983 shutdown_connections 2023-12-19T08:05:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- 172.21.15.111:0/239692983 wait complete. 2023-12-19T08:05:15.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 Processor -- start 2023-12-19T08:05:15.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.714+0000 7f6ad4d85700 1 -- start start 2023-12-19T08:05:15.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ad00ca2c0 0x7f6ad016c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ad0158e20 0x7f6ad00b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad016c9a0 con 0x7f6ad006abc0 2023-12-19T08:05:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad006b5a0 con 0x7f6ad0158e20 2023-12-19T08:05:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad014c520 con 0x7f6ad00ca2c0 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56250/0 (socket says 172.21.15.111:56250) 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 -- 172.21.15.111:0/2392235999 learned_addr learned my addr 172.21.15.111:0/2392235999 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ac7fff700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ad0158e20 0x7f6ad00b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ace59c700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ad00ca2c0 0x7f6ad016c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 -- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ad00ca2c0 msgr2=0x7f6ad016c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ad00ca2c0 0x7f6ad016c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 -- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ad0158e20 msgr2=0x7f6ad00b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ad0158e20 0x7f6ad00b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 -- 172.21.15.111:0/2392235999 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80094f0 con 0x7f6ad006abc0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6aced9d700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c940 tx=0x7f6ac8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8012630 con 0x7f6ad006abc0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac80127b0 con 0x7f6ad006abc0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8012b70 con 0x7f6ad006abc0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad00c9320 con 0x7f6ad006abc0 2023-12-19T08:05:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.718+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad00c89f0 con 0x7f6ad006abc0 2023-12-19T08:05:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.722+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad00c94b0 con 0x7f6ad006abc0 2023-12-19T08:05:15.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.722+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6ac801a3a0 con 0x7f6ad006abc0 2023-12-19T08:05:15.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.726+0000 7f6ac67fc700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:15.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.726+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f6ac807ce30 con 0x7f6ad006abc0 2023-12-19T08:05:15.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.726+0000 7f6ace59c700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:15.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.726+0000 7f6ace59c700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 0x7f6ab0062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0006f00 tx=0x7f6ac0008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:15.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.730+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6ac804cfc0 con 0x7f6ad006abc0 2023-12-19T08:05:15.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.958+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6ad006a7a0 con 0x7f6ad006abc0 2023-12-19T08:05:15.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.966+0000 7f6ac67fc700 1 -- 172.21.15.111:0/2392235999 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6ac80505e0 con 0x7f6ad006abc0 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 msgr2=0x7f6ab0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 0x7f6ab0062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0006f00 tx=0x7f6ac0008040 comp rx=0 tx=0).stop 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 msgr2=0x7f6ad006b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c940 tx=0x7f6ac8001710 comp rx=0 tx=0).stop 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 shutdown_connections 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6ab0060670 0x7f6ab0062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ad0158e20 0x7f6ad00b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad006abc0 0x7f6ad006b060 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 --2- 172.21.15.111:0/2392235999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ad00ca2c0 0x7f6ad016c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:15.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 >> 172.21.15.111:0/2392235999 conn(0x7f6ad0067bb0 msgr2=0x7f6ad00d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:15.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 shutdown_connections 2023-12-19T08:05:15.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:15.970+0000 7f6ad4d85700 1 -- 172.21.15.111:0/2392235999 wait complete. 2023-12-19T08:05:15.976 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:05:15.995 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:05:16.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.150+0000 7fc1a1a2e700 1 Processor -- start 2023-12-19T08:05:16.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 -- start start 2023-12-19T08:05:16.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc19c0c15b0 0x7fc19c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc19c0aeba0 0x7fc19c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0c7190 con 0x7fc19c15d320 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c0aefd0 con 0x7fc19c0aeba0 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc1a1a2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c0c1b50 con 0x7fc19c0c15b0 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc19c0aeba0 0x7fc19c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc19c0aeba0 0x7fc19c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56260/0 (socket says 172.21.15.111:56260) 2023-12-19T08:05:16.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 -- 172.21.15.111:0/4024682886 learned_addr learned my addr 172.21.15.111:0/4024682886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:16.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19a7fc700 1 --2- 172.21.15.111:0/4024682886 >> v2:172.21.15.143:3300/0 conn(0x7fc19c0c15b0 0x7fc19c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 -- 172.21.15.111:0/4024682886 >> v2:172.21.15.143:3300/0 conn(0x7fc19c0c15b0 msgr2=0x7fc19c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:16.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 --2- 172.21.15.111:0/4024682886 >> v2:172.21.15.143:3300/0 conn(0x7fc19c0c15b0 0x7fc19c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 -- 172.21.15.111:0/4024682886 >> v1:172.21.15.143:6789/0 conn(0x7fc19c15d320 legacy=0x7fc19c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.154+0000 7fc19b7fe700 1 -- 172.21.15.111:0/4024682886 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0c1e30 con 0x7fc19c0aeba0 2023-12-19T08:05:16.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc19b7fe700 1 --2- 172.21.15.111:0/4024682886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0aeba0 0x7fc19c0680f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc19c06f530 tx=0x7fc194009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec998e44f86f709 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:16.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc199ffb700 1 -- 172.21.15.111:0/4024682886 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19400e070 con 0x7fc19c0aeba0 2023-12-19T08:05:16.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc199ffb700 1 -- 172.21.15.111:0/4024682886 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19400ba50 con 0x7fc19c0aeba0 2023-12-19T08:05:16.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc199ffb700 1 -- 172.21.15.111:0/4024682886 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940092e0 con 0x7fc19c0aeba0 2023-12-19T08:05:16.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/4024682886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0aeba0 msgr2=0x7fc19c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/4024682886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0aeba0 0x7fc19c0680f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc19c06f530 tx=0x7fc194009840 comp rx=0 tx=0).stop 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/4024682886 shutdown_connections 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/4024682886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0aeba0 0x7fc19c0680f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/4024682886 >> v2:172.21.15.143:3300/0 conn(0x7fc19c0c15b0 0x7fc19c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/4024682886 >> 172.21.15.111:0/4024682886 conn(0x7fc19c0c8850 msgr2=0x7fc19c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/4024682886 shutdown_connections 2023-12-19T08:05:16.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.158+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/4024682886 wait complete. 2023-12-19T08:05:16.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 Processor -- start 2023-12-19T08:05:16.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 -- start start 2023-12-19T08:05:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c0aeba0 0x7fc19c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c15d320 0x7fc19c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c15ef80 con 0x7fc19c0c15b0 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c157be0 con 0x7fc19c15d320 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c157ec0 con 0x7fc19c0aeba0 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56268/0 (socket says 172.21.15.111:56268) 2023-12-19T08:05:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 -- 172.21.15.111:0/542911715 learned_addr learned my addr 172.21.15.111:0/542911715 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19affd700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c0aeba0 0x7fc19c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19b7fe700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c15d320 0x7fc19c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 -- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c0aeba0 msgr2=0x7fc19c15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c0aeba0 0x7fc19c15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 -- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c15d320 msgr2=0x7fc19c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c15d320 0x7fc19c158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 -- 172.21.15.111:0/542911715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194009ae0 con 0x7fc19c0c15b0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc19a7fc700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc19000d890 tx=0x7fc190008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19000b600 con 0x7fc19c0c15b0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19000b760 con 0x7fc19c0c15b0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c1581a0 con 0x7fc19c0c15b0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.162+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19000bae0 con 0x7fc19c0c15b0 2023-12-19T08:05:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.166+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c15eb50 con 0x7fc19c0c15b0 2023-12-19T08:05:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.166+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c158620 con 0x7fc19c0c15b0 2023-12-19T08:05:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.166+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc19001e460 con 0x7fc19c0c15b0 2023-12-19T08:05:16.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.170+0000 7fc182ffd700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 0x7fc188062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.170+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fc19007d3a0 con 0x7fc19c0c15b0 2023-12-19T08:05:16.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.170+0000 7fc19affd700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 0x7fc188062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.170+0000 7fc19affd700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 0x7fc188062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0037c0 tx=0x7fc18c009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:16.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.174+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc19004d530 con 0x7fc19c0c15b0 2023-12-19T08:05:16.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.382+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc19c071120 con 0x7fc19c0c15b0 2023-12-19T08:05:16.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.382+0000 7fc182ffd700 1 -- 172.21.15.111:0/542911715 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7fc190050b50 con 0x7fc19c0c15b0 2023-12-19T08:05:16.387 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:16.387 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:05:16.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 msgr2=0x7fc188062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 0x7fc188062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0037c0 tx=0x7fc18c009040 comp rx=0 tx=0).stop 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 msgr2=0x7fc19c1589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc19000d890 tx=0x7fc190008040 comp rx=0 tx=0).stop 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 shutdown_connections 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc188060670 0x7fc188062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc19c15d320 0x7fc19c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19c0c15b0 0x7fc19c1589a0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 --2- 172.21.15.111:0/542911715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc19c0aeba0 0x7fc19c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 >> 172.21.15.111:0/542911715 conn(0x7fc19c0c8850 msgr2=0x7fc19c0c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 shutdown_connections 2023-12-19T08:05:16.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.386+0000 7fc1a1a2e700 1 -- 172.21.15.111:0/542911715 wait complete. 2023-12-19T08:05:16.393 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:05:16.812 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:05:10.166367+0000', 'modified': '2023-12-19T08:05:13.786643+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7890}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7890': {'gid': 7890, 'name': 'd', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6837/1501582072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6835', 'nonce': 1501582072}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 1501582072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:05:16.812 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:05:16.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-19T08:05:16.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.978+0000 7fe45d345700 1 Processor -- start 2023-12-19T08:05:16.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe45d345700 1 -- start start 2023-12-19T08:05:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe45d345700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe45806b090 0x7fe45806b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe45d345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4580c8200 con 0x7fe4580c7140 2023-12-19T08:05:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe45d345700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45806b9a0 con 0x7fe4580c6ca0 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe45d345700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4580c7680 con 0x7fe45806b090 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe45806b090 0x7fe45806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe45806b090 0x7fe45806b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56276/0 (socket says 172.21.15.111:56276) 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 -- 172.21.15.111:0/1219990223 learned_addr learned my addr 172.21.15.111:0/1219990223 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 -- 172.21.15.111:0/1219990223 >> v1:172.21.15.143:6789/0 conn(0x7fe4580c6ca0 legacy=0x7fe4580ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:16.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 -- 172.21.15.111:0/1219990223 >> v1:172.21.15.111:6789/0 conn(0x7fe4580c7140 legacy=0x7fe458155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 -- 172.21.15.111:0/1219990223 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4580c7960 con 0x7fe45806b090 2023-12-19T08:05:16.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe456ffd700 1 --2- 172.21.15.111:0/1219990223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe45806b460 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe44800bc50 tx=0x7fe44800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2abc71d6c99205 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:16.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe4567fc700 1 -- 172.21.15.111:0/1219990223 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44800be60 con 0x7fe45806b090 2023-12-19T08:05:16.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe4567fc700 1 -- 172.21.15.111:0/1219990223 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44800de00 con 0x7fe45806b090 2023-12-19T08:05:16.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.982+0000 7fe4567fc700 1 -- 172.21.15.111:0/1219990223 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4480196b0 con 0x7fe45806b090 2023-12-19T08:05:16.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- 172.21.15.111:0/1219990223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 msgr2=0x7fe45806b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 --2- 172.21.15.111:0/1219990223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe45806b460 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe44800bc50 tx=0x7fe44800b4f0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- 172.21.15.111:0/1219990223 shutdown_connections 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 --2- 172.21.15.111:0/1219990223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe45806b460 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- 172.21.15.111:0/1219990223 >> 172.21.15.111:0/1219990223 conn(0x7fe458067bb0 msgr2=0x7fe4580cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- 172.21.15.111:0/1219990223 shutdown_connections 2023-12-19T08:05:16.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- 172.21.15.111:0/1219990223 wait complete. 2023-12-19T08:05:16.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 Processor -- start 2023-12-19T08:05:16.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.986+0000 7fe45d345700 1 -- start start 2023-12-19T08:05:16.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4580c6ca0 0x7fe4580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4580c7140 0x7fe458157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:16.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe458169bd0 con 0x7fe45806b090 2023-12-19T08:05:16.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4580a7ab0 con 0x7fe4580c6ca0 2023-12-19T08:05:16.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4580a58f0 con 0x7fe4580c7140 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56278/0 (socket says 172.21.15.111:56278) 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 -- 172.21.15.111:0/3750964218 learned_addr learned my addr 172.21.15.111:0/3750964218 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe44ffff700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4580c6ca0 0x7fe4580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe4577fe700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4580c7140 0x7fe458157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:16.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 -- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4580c7140 msgr2=0x7fe458157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4580c7140 0x7fe458157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 -- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4580c6ca0 msgr2=0x7fe4580a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4580c6ca0 0x7fe4580a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 -- 172.21.15.111:0/3750964218 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44800b790 con 0x7fe45806b090 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe448006fd0 tx=0x7fe44801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe44effd700 1 -- 172.21.15.111:0/3750964218 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4480107d0 con 0x7fe45806b090 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe44effd700 1 -- 172.21.15.111:0/3750964218 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe448010950 con 0x7fe45806b090 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe44effd700 1 -- 172.21.15.111:0/3750964218 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448010d10 con 0x7fe45806b090 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.990+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4580a5bd0 con 0x7fe45806b090 2023-12-19T08:05:16.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.994+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4580a6190 con 0x7fe45806b090 2023-12-19T08:05:16.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.994+0000 7fe44effd700 1 -- 172.21.15.111:0/3750964218 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe44802c430 con 0x7fe45806b090 2023-12-19T08:05:17.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.998+0000 7fe44effd700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 0x7fe438062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.998+0000 7fe44effd700 1 -- 172.21.15.111:0/3750964218 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe448014070 con 0x7fe45806b090 2023-12-19T08:05:17.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.998+0000 7fe44ffff700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 0x7fe438062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.998+0000 7fe45f60e740 1 Processor -- start 2023-12-19T08:05:17.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:16.998+0000 7fe44ffff700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 0x7fe438062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe4580c56d0 tx=0x7fe440007ec0 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 -- start start 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x14be790 0x16f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x161ef30 con 0x1473ae0 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x161aed0 con 0x14be790 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x15e10b0 con 0x1473710 2023-12-19T08:05:17.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56288/0 (socket says 172.21.15.111:56288) 2023-12-19T08:05:17.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 -- 172.21.15.111:0/1310702480 learned_addr learned my addr 172.21.15.111:0/1310702480 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe44ffff700 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe4577fe700 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x14be790 0x16f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 -- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 msgr2=0x16f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 -- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x14be790 msgr2=0x16f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x14be790 0x16f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 -- 172.21.15.111:0/1310702480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x14bf9e0 con 0x1473ae0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe456ffd700 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe448000e50 tx=0x7fe448006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fd81e094bdc49a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe437fff700 1 -- 172.21.15.111:0/1310702480 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448055cd0 con 0x1473ae0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe437fff700 1 -- 172.21.15.111:0/1310702480 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe448059a20 con 0x1473ae0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe437fff700 1 -- 172.21.15.111:0/1310702480 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448050de0 con 0x1473ae0 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 -- 172.21.15.111:0/1310702480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 msgr2=0x16f0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.002+0000 7fe45f60e740 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe448000e50 tx=0x7fe448006f60 comp rx=0 tx=0).stop 2023-12-19T08:05:17.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- 172.21.15.111:0/1310702480 shutdown_connections 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x14be790 0x16f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- 172.21.15.111:0/1310702480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f0f00 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- 172.21.15.111:0/1310702480 >> 172.21.15.111:0/1310702480 conn(0x1473eb0 msgr2=0x16ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- 172.21.15.111:0/1310702480 shutdown_connections 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- 172.21.15.111:0/1310702480 wait complete. 2023-12-19T08:05:17.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- start start 2023-12-19T08:05:17.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 Processor -- start 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16d6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x15e0d50 con 0x1473ae0 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x15b9970 con 0x1473710 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x166a480 con 0x14be790 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe45f60e740 10 client.0 _ll_register_callbacks cb 0x16646a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:05:17.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe44ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:42910/0 (socket says 172.21.15.111:42910) 2023-12-19T08:05:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe456ffd700 1 -- 172.21.15.111:0/2413244602 learned_addr learned my addr 172.21.15.111:0/2413244602 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.006+0000 7fe4577fe700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16d6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 -- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 msgr2=0x16d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16d6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 -- 172.21.15.111:0/2413244602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 msgr2=0x16f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 -- 172.21.15.111:0/2413244602 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4580a6190 con 0x1473710 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe456ffd700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe448052740 tx=0x7fe448069000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7abdb3a60dc5a621 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448051f30 con 0x1473710 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe448067e30 con 0x1473710 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x15b9cc0 con 0x1473710 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe448059590 con 0x1473710 2023-12-19T08:05:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 10 client.7840 fetch_fsmap learned FSMap version 56 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x15e9370 con 0x1473710 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x15b9cc0 con 0x1473710 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 56) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe44804e830 con 0x1473710 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 10 client.7840 fetch_fsmap finished waiting for FSMap version 56 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 10 client.7840 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 20 client.7840 populate_metadata read hostname 'smithi111' 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x16d83d0 0x17006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 4 client.7840 mds_command: new command op to 7890 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:05:17.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 --> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x15bd3e0 con 0x16d83d0 2023-12-19T08:05:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.010+0000 7fe44ffff700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x16d83d0 0x17006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe4357fa700 10 client.7840 ms_handle_connect on v2:172.21.15.143:6835/1501582072 2023-12-19T08:05:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe44ffff700 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x16d83d0 0x17006b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=105496160140f6c3 server_cookie=52828d70a81ecf40 in_seq=0 out_seq=0 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe4357fa700 1 -- 172.21.15.111:0/2413244602 <== mds.0 v2:172.21.15.143:6835/1501582072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x15bd3e0 con 0x16d83d0 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe4357fa700 10 client.7840 handle_command_reply: tid=0 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 client.7840 shutdown 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 msgr2=0x16f5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe448052740 tx=0x7fe448069000 comp rx=0 tx=0).stop 2023-12-19T08:05:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 shutdown_connections 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x16d83d0 0x17006b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16d6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe4357fa700 0 client.7840 ms_handle_reset on v2:172.21.15.143:6835/1501582072 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x16f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2413244602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x16f5510 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 >> 172.21.15.111:0/2413244602 conn(0x1473eb0 msgr2=0x16a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 shutdown_connections 2023-12-19T08:05:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 1 -- 172.21.15.111:0/2413244602 wait complete. 2023-12-19T08:05:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.014+0000 7fe45f60e740 20 client.7840 trim_cache size 0 max 16384 2023-12-19T08:05:17.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 Processor -- start 2023-12-19T08:05:17.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 -- start start 2023-12-19T08:05:17.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473ae0 0x16f4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x15f0550 con 0x14be790 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1616b50 con 0x1473ae0 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x15c9e00 con 0x1473710 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56298/0 (socket says 172.21.15.111:56298) 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe4577fe700 1 -- 172.21.15.111:0/3569813948 learned_addr learned my addr 172.21.15.111:0/3569813948 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.038+0000 7fe44ffff700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473ae0 0x16f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 -- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473ae0 msgr2=0x16f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473ae0 0x16f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 -- 172.21.15.111:0/3569813948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 msgr2=0x16a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 -- 172.21.15.111:0/3569813948 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x15cfd80 con 0x1473710 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe4577fe700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe456ffd700 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f3ed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe44800e010 tx=0x7fe448079040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e04ea9cd17be17b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe44cff9700 1 -- 172.21.15.111:0/3569813948 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448051f30 con 0x1473710 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe44cff9700 1 -- 172.21.15.111:0/3569813948 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44806ce80 con 0x1473710 2023-12-19T08:05:17.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe44cff9700 1 -- 172.21.15.111:0/3569813948 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4480523a0 con 0x1473710 2023-12-19T08:05:17.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 -- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 msgr2=0x16f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f3ed0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe44800e010 tx=0x7fe448079040 comp rx=0 tx=0).stop 2023-12-19T08:05:17.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 -- 172.21.15.111:0/3569813948 shutdown_connections 2023-12-19T08:05:17.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473ae0 0x16f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x14be790 0x16a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 --2- 172.21.15.111:0/3569813948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1473710 0x16f3ed0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 -- 172.21.15.111:0/3569813948 >> 172.21.15.111:0/3569813948 conn(0x1473eb0 msgr2=0x16f10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 -- 172.21.15.111:0/3569813948 shutdown_connections 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.042+0000 7fe45f60e740 1 -- 172.21.15.111:0/3569813948 wait complete. 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 -- start start 2023-12-19T08:05:17.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 Processor -- start 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x17532a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x17537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16aa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x14ec7c0 con 0x1473ae0 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x14f0650 con 0x1473710 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x15f3910 con 0x14be790 2023-12-19T08:05:17.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.046+0000 7fe45f60e740 10 client.0 _ll_register_callbacks cb 0x143f9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x17537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x17532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe456ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x17532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:42928/0 (socket says 172.21.15.111:42928) 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe456ffd700 1 -- 172.21.15.111:0/2813248705 learned_addr learned my addr 172.21.15.111:0/2813248705 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4577fe700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 -- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 msgr2=0x16aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16aa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 -- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 msgr2=0x17532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x17532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 -- 172.21.15.111:0/2813248705 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45000f040 con 0x1473ae0 2023-12-19T08:05:17.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe44ffff700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x17537e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x161a730 tx=0x7fe440014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e608bccaf72631c6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440024b20 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fe44001c040 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x15ef410 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe440024e90 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 10 client.8037 fetch_fsmap learned FSMap version 56 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x14bf9e0 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x15ef410 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 56) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fe4400343a0 con 0x1473ae0 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 10 client.8037 fetch_fsmap finished waiting for FSMap version 56 2023-12-19T08:05:17.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 10 client.8037 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 20 client.8037 populate_metadata read hostname 'smithi111' 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x175a130 0x175c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 4 client.8037 mds_command: new command op to 7890 tid=0[{"prefix": "status"}] 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.050+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 --> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] -- command(tid 0: {"prefix": "status"}) v1 -- 0x14cb4e0 con 0x175a130 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe456ffd700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x175a130 0x175c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe456ffd700 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x175a130 0x175c510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad771731154d8649 server_cookie=f2e4a42eb61f4ddf in_seq=0 out_seq=0 2023-12-19T08:05:17.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe4357fa700 10 client.8037 ms_handle_connect on v2:172.21.15.143:6835/1501582072 2023-12-19T08:05:17.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe4357fa700 1 -- 172.21.15.111:0/2813248705 <== mds.0 v2:172.21.15.143:6835/1501582072 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x14cb4e0 con 0x175a130 2023-12-19T08:05:17.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe4357fa700 10 client.8037 handle_command_reply: tid=0 2023-12-19T08:05:17.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 client.8037 shutdown 2023-12-19T08:05:17.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 msgr2=0x17537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x17537e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x161a730 tx=0x7fe440014000 comp rx=0 tx=0).stop 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 shutdown_connections 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:6835/1501582072,v1:172.21.15.143:6837/1501582072] conn(0x175a130 0x175c510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x14be790 0x16aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1473ae0 0x17537e0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 --2- 172.21.15.111:0/2813248705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1473710 0x17532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 >> 172.21.15.111:0/2813248705 conn(0x1473eb0 msgr2=0x16f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe4357fa700 0 client.8037 ms_handle_reset on v2:172.21.15.143:6835/1501582072 2023-12-19T08:05:17.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 shutdown_connections 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 1 -- 172.21.15.111:0/2813248705 wait complete. 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45f60e740 20 client.8037 trim_cache size 0 max 16384 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 7890, 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:05:17.059 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2976558310000001, 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 56, 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 129, 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 129, 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.213876444 2023-12-19T08:05:17.060 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 msgr2=0x7fe438062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 0x7fe438062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe4580c56d0 tx=0x7fe440007ec0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 msgr2=0x7fe4580a7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe448006fd0 tx=0x7fe44801b000 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 shutdown_connections 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe438060670 0x7fe438062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4580c6ca0 0x7fe4580a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4580c7140 0x7fe458157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 --2- 172.21.15.111:0/3750964218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe45806b090 0x7fe4580a7570 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.054+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 >> 172.21.15.111:0/3750964218 conn(0x7fe458067bb0 msgr2=0x7fe458158550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.058+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 shutdown_connections 2023-12-19T08:05:17.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.058+0000 7fe45d345700 1 -- 172.21.15.111:0/3750964218 wait complete. 2023-12-19T08:05:17.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:05:17.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.238+0000 7f39d2ccd700 1 Processor -- start 2023-12-19T08:05:17.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 -- start start 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140710 0x7f39cc0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39cc140ae0 0x7f39cc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39cc0c75e0 0x7f39cc068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc06e150 con 0x7f39cc140ae0 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc06d540 con 0x7f39cc0c75e0 2023-12-19T08:05:17.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d2ccd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc06d820 con 0x7f39cc140710 2023-12-19T08:05:17.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39cc140ae0 0x7f39cc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39cc140ae0 0x7f39cc0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56310/0 (socket says 172.21.15.111:56310) 2023-12-19T08:05:17.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 -- 172.21.15.111:0/704288155 learned_addr learned my addr 172.21.15.111:0/704288155 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d126a700 1 --2- 172.21.15.111:0/704288155 >> v2:172.21.15.143:3300/0 conn(0x7f39cc0c75e0 0x7f39cc068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39d0a69700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140710 0x7f39cc0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 -- 172.21.15.111:0/704288155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140710 msgr2=0x7f39cc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140710 0x7f39cc0c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 -- 172.21.15.111:0/704288155 >> v2:172.21.15.143:3300/0 conn(0x7f39cc0c75e0 msgr2=0x7f39cc068630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 --2- 172.21.15.111:0/704288155 >> v2:172.21.15.143:3300/0 conn(0x7f39cc0c75e0 0x7f39cc068630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.242+0000 7f39cbfff700 1 -- 172.21.15.111:0/704288155 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc06db00 con 0x7f39cc140ae0 2023-12-19T08:05:17.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39cbfff700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140ae0 0x7f39cc0680f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0ae930 tx=0x7f39b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74434c4a0bb3b99 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39cb7fe700 1 -- 172.21.15.111:0/704288155 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800b640 con 0x7f39cc140ae0 2023-12-19T08:05:17.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39cb7fe700 1 -- 172.21.15.111:0/704288155 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b800b7a0 con 0x7f39cc140ae0 2023-12-19T08:05:17.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39cb7fe700 1 -- 172.21.15.111:0/704288155 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800bb60 con 0x7f39cc140ae0 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 -- 172.21.15.111:0/704288155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140ae0 msgr2=0x7f39cc0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140ae0 0x7f39cc0680f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0ae930 tx=0x7f39b8009250 comp rx=0 tx=0).stop 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 -- 172.21.15.111:0/704288155 shutdown_connections 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/704288155 >> v2:172.21.15.143:3300/0 conn(0x7f39cc0c75e0 0x7f39cc068630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140ae0 0x7f39cc0680f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/704288155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140710 0x7f39cc0c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 -- 172.21.15.111:0/704288155 >> 172.21.15.111:0/704288155 conn(0x7f39cc0c8850 msgr2=0x7f39cc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 -- 172.21.15.111:0/704288155 shutdown_connections 2023-12-19T08:05:17.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 -- 172.21.15.111:0/704288155 wait complete. 2023-12-19T08:05:17.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.246+0000 7f39d2ccd700 1 Processor -- start 2023-12-19T08:05:17.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- start start 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c75e0 0x7f39cc156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 0x7f39cc15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc13cfb0 con 0x7f39cc140710 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc13c3a0 con 0x7f39cc0c75e0 2023-12-19T08:05:17.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc13c680 con 0x7f39cc140ae0 2023-12-19T08:05:17.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56322/0 (socket says 172.21.15.111:56322) 2023-12-19T08:05:17.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 -- 172.21.15.111:0/3266999482 learned_addr learned my addr 172.21.15.111:0/3266999482 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:17.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d126a700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 0x7f39cc15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d0a69700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c75e0 0x7f39cc156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 -- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 msgr2=0x7f39cc15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 0x7f39cc15a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 -- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c75e0 msgr2=0x7f39cc156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c75e0 0x7f39cc156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 -- 172.21.15.111:0/3266999482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80094f0 con 0x7f39cc140710 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d126a700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 0x7f39cc15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39cbfff700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f39b80094c0 tx=0x7f39b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800b910 con 0x7f39cc140710 2023-12-19T08:05:17.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b800ba70 con 0x7f39cc140710 2023-12-19T08:05:17.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b8026680 con 0x7f39cc140710 2023-12-19T08:05:17.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc13c960 con 0x7f39cc140710 2023-12-19T08:05:17.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.250+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc13c070 con 0x7f39cc140710 2023-12-19T08:05:17.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.254+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc13d160 con 0x7f39cc140710 2023-12-19T08:05:17.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.258+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f39b8023030 con 0x7f39cc140710 2023-12-19T08:05:17.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.258+0000 7f39c8ff9700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 0x7f39b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:17.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.258+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f39b807e420 con 0x7f39cc140710 2023-12-19T08:05:17.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.258+0000 7f39d0a69700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 0x7f39b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:17.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.258+0000 7f39d0a69700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 0x7f39b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009d40 tx=0x7f39c0008040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:17.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.266+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f39cc13d160 con 0x7f39cc140710 2023-12-19T08:05:17.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.474+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39cc071120 con 0x7f39cc140710 2023-12-19T08:05:17.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.474+0000 7f39c8ff9700 1 -- 172.21.15.111:0/3266999482 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f39b804e6b0 con 0x7f39cc140710 2023-12-19T08:05:17.479 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 msgr2=0x7f39b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 0x7f39b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009d40 tx=0x7f39c0008040 comp rx=0 tx=0).stop 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 msgr2=0x7f39cc15a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f39b80094c0 tx=0x7f39b8017000 comp rx=0 tx=0).stop 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 shutdown_connections 2023-12-19T08:05:17.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f39b0060670 0x7f39b0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140ae0 0x7f39cc15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc140710 0x7f39cc15a050 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 --2- 172.21.15.111:0/3266999482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c75e0 0x7f39cc156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 >> 172.21.15.111:0/3266999482 conn(0x7f39cc0c8850 msgr2=0x7f39cc0cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 shutdown_connections 2023-12-19T08:05:17.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:17.478+0000 7f39d2ccd700 1 -- 172.21.15.111:0/3266999482 wait complete. 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:05:17.504 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:05:17.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.505 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.525 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41340sec preferred_lft 41340sec 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:05:17.526 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:05:17.527 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:05:17.528 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:05:17.528 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:05:17.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.529 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:05:17.608 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:05:17.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.611 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:05:17.677 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:05:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:05:17.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.679 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:05:17.679 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:05:17.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:05:17.680 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:05:17.680 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:05:17.680 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:05:17.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.814 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:05:17.831 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:05:17.831 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:05:17.832 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:05:17.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:05:17.833 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:05:17.911 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:05:17.911 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:05:17.912 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:17.972 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:05:17.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:05:17.973 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:18.033 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:05:18.115 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:05:18.116 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:05:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:05:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:05:18.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:18.203 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:05:18.264 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:05:18.264 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:05:18.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:05:18.349 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:05:18.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:05:18.425 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:05:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:05:18.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:18.428 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:05:18.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:05:18.434+0000 7f16933cc380 -1 init, newargv = 0x5572ea54ba90 newargc=15 2023-12-19T08:05:18.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[53999]: starting ceph client 2023-12-19T08:05:18.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[53999]: starting fuse 2023-12-19T08:05:18.455 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:05:18.455 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:05:18.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:05:18.455 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:05:18.455 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:05:18.455 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:05:18.455 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:05:18.456 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:05:18.457 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:05:18.618 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.53999.asok 2023-12-19T08:05:18.802 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53999.asok 2023-12-19T08:05:18.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:18.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53999.asok status 2023-12-19T08:05:18.980 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "53999", 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:05:18.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 8046, 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 8046 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:05:18.982 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 400827063 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 400827063 2023-12-19T08:05:18.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.8046 192.168.0.1:0/400827063", 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/400827063", 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 56, 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 129, 2023-12-19T08:05:18.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:05:18.985 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:05:18.985 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:05:18.985 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:05:18.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:18.994 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:05:18.994 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:19.013 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:05:19.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:19.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:19.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:05:19.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:05:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.290+0000 7ff1cde16700 1 Processor -- start 2023-12-19T08:05:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.290+0000 7ff1cde16700 1 -- start start 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1cde16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1c80c6cd0 0x7ff1c806b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1cde16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c806b470 con 0x7ff1c815cc20 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1cde16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c806e110 con 0x7ff1c80c6900 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1cde16700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c806be40 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1c80c6cd0 0x7ff1c806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff1c80c6cd0 0x7ff1c806b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56330/0 (socket says 172.21.15.111:56330) 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 -- 172.21.15.111:0/3593029423 learned_addr learned my addr 172.21.15.111:0/3593029423 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1cd615700 1 --1- 172.21.15.111:0/3593029423 >> v1:172.21.15.111:6789/0 conn(0x7ff1c815cc20 0x7ff1c815d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42206/0 (socket says 172.21.15.111:42206) 2023-12-19T08:05:19.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 -- 172.21.15.111:0/3593029423 >> v1:172.21.15.143:6789/0 conn(0x7ff1c80c6900 legacy=0x7ff1c806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 -- 172.21.15.111:0/3593029423 >> v1:172.21.15.111:6789/0 conn(0x7ff1c815cc20 legacy=0x7ff1c815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 -- 172.21.15.111:0/3593029423 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8161d10 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c7fff700 1 --2- 172.21.15.111:0/3593029423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6cd0 0x7ff1c806b900 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40099b0 tx=0x7ff1b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be4c351e0142c70 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c77fe700 1 -- 172.21.15.111:0/3593029423 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400b860 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c77fe700 1 -- 172.21.15.111:0/3593029423 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b400b9c0 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.294+0000 7ff1c77fe700 1 -- 172.21.15.111:0/3593029423 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400bd60 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- 172.21.15.111:0/3593029423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6cd0 msgr2=0x7ff1c806b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3593029423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6cd0 0x7ff1c806b900 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40099b0 tx=0x7ff1b4009250 comp rx=0 tx=0).stop 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- 172.21.15.111:0/3593029423 shutdown_connections 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3593029423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6cd0 0x7ff1c806b900 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- 172.21.15.111:0/3593029423 >> 172.21.15.111:0/3593029423 conn(0x7ff1c8067bb0 msgr2=0x7ff1c80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- 172.21.15.111:0/3593029423 shutdown_connections 2023-12-19T08:05:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- 172.21.15.111:0/3593029423 wait complete. 2023-12-19T08:05:19.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 Processor -- start 2023-12-19T08:05:19.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.298+0000 7ff1cde16700 1 -- start start 2023-12-19T08:05:19.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 0x7ff1c8159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 0x7ff1c80c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c80c5af0 con 0x7ff1c80c6900 2023-12-19T08:05:19.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8159bb0 con 0x7ff1c815cc20 2023-12-19T08:05:19.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c816f1d0 con 0x7ff1c80c6cd0 2023-12-19T08:05:19.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56346/0 (socket says 172.21.15.111:56346) 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 -- 172.21.15.111:0/3864886553 learned_addr learned my addr 172.21.15.111:0/3864886553 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cd615700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 0x7ff1c80c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1c7fff700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 0x7ff1c8159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 -- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 msgr2=0x7ff1c8159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 0x7ff1c8159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 -- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 msgr2=0x7ff1c80c0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 0x7ff1c80c0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 -- 172.21.15.111:0/3864886553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b40094f0 con 0x7ff1c80c6900 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cd615700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 0x7ff1c80c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1c7fff700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 0x7ff1c8159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:19.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cce14700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c590 tx=0x7ff1bc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc011590 con 0x7ff1c80c6900 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0116f0 con 0x7ff1c80c6900 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc011ab0 con 0x7ff1c80c6900 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c80c10f0 con 0x7ff1c80c6900 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.302+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c80c13f0 con 0x7ff1c80c6900 2023-12-19T08:05:19.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.306+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c804be40 con 0x7ff1c80c6900 2023-12-19T08:05:19.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.306+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff1bc01b3b0 con 0x7ff1c80c6900 2023-12-19T08:05:19.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.306+0000 7ff1c4ff9700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 0x7ff1ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.306+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7ff1bc07c310 con 0x7ff1c80c6900 2023-12-19T08:05:19.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.306+0000 7ff1c7fff700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 0x7ff1ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.310+0000 7ff1c7fff700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 0x7ff1ac062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40099b0 tx=0x7ff1b4017000 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.314+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff1bc04c400 con 0x7ff1c80c6900 2023-12-19T08:05:19.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.506+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1c80cc490 con 0x7ff1ac060670 2023-12-19T08:05:19.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.510+0000 7ff1c4ff9700 1 -- 172.21.15.111:0/3864886553 <== mgr.7501 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff1c80cc490 con 0x7ff1ac060670 2023-12-19T08:05:19.512 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:05:19.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:05:19.512 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:05:19.512 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:05:19.513 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 msgr2=0x7ff1ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 0x7ff1ac062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40099b0 tx=0x7ff1b4017000 comp rx=0 tx=0).stop 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 msgr2=0x7ff1c816e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c590 tx=0x7ff1bc00c950 comp rx=0 tx=0).stop 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 shutdown_connections 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff1ac060670 0x7ff1ac062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c815cc20 0x7ff1c80c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1c80c6cd0 0x7ff1c8159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 --2- 172.21.15.111:0/3864886553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1c80c6900 0x7ff1c816e510 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 >> 172.21.15.111:0/3864886553 conn(0x7ff1c8067bb0 msgr2=0x7ff1c8160d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:19.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 shutdown_connections 2023-12-19T08:05:19.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.514+0000 7ff1cde16700 1 -- 172.21.15.111:0/3864886553 wait complete. 2023-12-19T08:05:19.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:05:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.698+0000 7f231f59e700 1 Processor -- start 2023-12-19T08:05:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.698+0000 7f231f59e700 1 -- start start 2023-12-19T08:05:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f232006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232006b870 con 0x7f2320158fe0 2023-12-19T08:05:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232006ac60 con 0x7f232006c9f0 2023-12-19T08:05:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232006af40 con 0x7f232006d9f0 2023-12-19T08:05:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2320158fe0 0x7f23200afd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42216/0 (socket says 172.21.15.111:42216) 2023-12-19T08:05:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231ed9d700 1 -- 172.21.15.111:0/1832087357 learned_addr learned my addr 172.21.15.111:0/1832087357 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f232006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511423793 0 0) 0x7f232006b870 con 0x7f2320158fe0 2023-12-19T08:05:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003570 con 0x7f2320158fe0 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869045390 0 0) 0x7f2308003570 con 0x7f2320158fe0 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 msgr2=0x7f232006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f232006cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 >> v1:172.21.15.143:6789/0 conn(0x7f232006c9f0 legacy=0x7f23201588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232006b220 con 0x7f2320158fe0 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f23180017f0 con 0x7f2320158fe0 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 >> v1:172.21.15.111:6789/0 conn(0x7f2320158fe0 legacy=0x7f23200afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2308003a90 0x7f2308003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f2308004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2308004d50 0x7f23080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232006af40 con 0x7f2308003a90 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230800bba0 con 0x7f2308004d50 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230800bd60 con 0x7f232006d9f0 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f2308004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231ed9d700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2308004d50 0x7f23080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231e59c700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2308003a90 0x7f2308003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 -- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2308004d50 msgr2=0x7f23080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2308004d50 0x7f23080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 -- 172.21.15.111:0/1832087357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2308003a90 msgr2=0x7f2308003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2308003a90 0x7f2308003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.702+0000 7f231dd9b700 1 -- 172.21.15.111:0/1832087357 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308003570 con 0x7f232006d9f0 2023-12-19T08:05:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231dd9b700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f2308004810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2314009c00 tx=0x7f23140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e667def1c11fc4c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400e710 con 0x7f232006d9f0 2023-12-19T08:05:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231400e870 con 0x7f232006d9f0 2023-12-19T08:05:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231d59a700 1 -- 172.21.15.111:0/1832087357 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400ec30 con 0x7f232006d9f0 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 msgr2=0x7f2308004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 --2- 172.21.15.111:0/1832087357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232006d9f0 0x7f2308004810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2314009c00 tx=0x7f23140091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231ed9d700 1 -- 172.21.15.111:0/1832087357 reap_dead start 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- 172.21.15.111:0/1832087357 shutdown_connections 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- 172.21.15.111:0/1832087357 >> 172.21.15.111:0/1832087357 conn(0x7f2320067bb0 msgr2=0x7f23200cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- 172.21.15.111:0/1832087357 shutdown_connections 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- 172.21.15.111:0/1832087357 wait complete. 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 Processor -- start 2023-12-19T08:05:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.706+0000 7f231f59e700 1 -- start start 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 0x7f2320150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232014b070 0x7f232014b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232014c430 con 0x7f232014ba80 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200b4d40 con 0x7f232006d9f0 2023-12-19T08:05:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200b4130 con 0x7f232014b070 2023-12-19T08:05:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56370/0 (socket says 172.21.15.111:56370) 2023-12-19T08:05:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 -- 172.21.15.111:0/2928680912 learned_addr learned my addr 172.21.15.111:0/2928680912 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231dd9b700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232014b070 0x7f232014b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231e59c700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 0x7f2320150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 -- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232014b070 msgr2=0x7f232014b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232014b070 0x7f232014b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 -- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 msgr2=0x7f2320150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 0x7f2320150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 -- 172.21.15.111:0/2928680912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200b4410 con 0x7f232014ba80 2023-12-19T08:05:19.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231e59c700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 0x7f2320150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231ed9d700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f23201593b0 tx=0x7f23180021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231800ace0 con 0x7f232014ba80 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231800ae80 con 0x7f232014ba80 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23200b46f0 con 0x7f232014ba80 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318013640 con 0x7f232014ba80 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.710+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23200b3e00 con 0x7f232014ba80 2023-12-19T08:05:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.714+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232006d300 con 0x7f232014ba80 2023-12-19T08:05:19.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.714+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 50) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2318013a70 con 0x7f232014ba80 2023-12-19T08:05:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.714+0000 7f230e7fc700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 0x7f2300062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:19.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.714+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f231807d720 con 0x7f232014ba80 2023-12-19T08:05:19.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.718+0000 7f231e59c700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 0x7f2300062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:19.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.718+0000 7f231e59c700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 0x7f2300062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2310000f60 tx=0x7f2310009040 comp rx=0 tx=0).ready entity=mgr.7501 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:19.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.722+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f232006d300 con 0x7f232014ba80 2023-12-19T08:05:19.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:19.926+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f232006d300 con 0x7f232014ba80 2023-12-19T08:05:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.542+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 51) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f231804dbe0 con 0x7f232014ba80 2023-12-19T08:05:20.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.546+0000 7f230e7fc700 1 -- 172.21.15.111:0/2928680912 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v51) v1 ==== 92+0+0 (secure 0 0 0) 0x7f231804d810 con 0x7f232014ba80 2023-12-19T08:05:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 msgr2=0x7f2300062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 0x7f2300062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2310000f60 tx=0x7f2310009040 comp rx=0 tx=0).stop 2023-12-19T08:05:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 msgr2=0x7f232014bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f23201593b0 tx=0x7f23180021d0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 shutdown_connections 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2300060670 0x7f2300062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f232014ba80 0x7f232014bef0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232014b070 0x7f232014b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 --2- 172.21.15.111:0/2928680912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232006d9f0 0x7f2320150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 >> 172.21.15.111:0/2928680912 conn(0x7f2320067bb0 msgr2=0x7f23200cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 shutdown_connections 2023-12-19T08:05:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.550+0000 7f231f59e700 1 -- 172.21.15.111:0/2928680912 wait complete. 2023-12-19T08:05:20.573 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:05:20.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.734+0000 7f3c920e3700 1 Processor -- start 2023-12-19T08:05:20.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.734+0000 7f3c920e3700 1 -- start start 2023-12-19T08:05:20.740 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f28f5fce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:05:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c920e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c8c150420 0x7f3c8c0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c920e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0af4c0 con 0x7f3c8c0ac720 2023-12-19T08:05:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c920e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0cae80 con 0x7f3c8c156e80 2023-12-19T08:05:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c920e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0af1e0 con 0x7f3c8c150420 2023-12-19T08:05:20.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c918e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c8c0ac720 0x7f3c8c0cb310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42232/0 (socket says 172.21.15.111:42232) 2023-12-19T08:05:20.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c918e2700 1 -- 172.21.15.111:0/3367665372 learned_addr learned my addr 172.21.15.111:0/3367665372 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:20.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c910e1700 1 --2- 172.21.15.111:0/3367665372 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c150420 0x7f3c8c0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:20.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.738+0000 7f3c83fff700 1 -- 172.21.15.111:0/3367665372 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492545788 0 0) 0x7f3c8c0af4c0 con 0x7f3c8c0ac720 2023-12-19T08:05:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c83fff700 1 -- 172.21.15.111:0/3367665372 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c78003570 con 0x7f3c8c0ac720 2023-12-19T08:05:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c910e1700 1 -- 172.21.15.111:0/3367665372 >> v1:172.21.15.143:6789/0 conn(0x7f3c8c156e80 legacy=0x7f3c8c0ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c910e1700 1 -- 172.21.15.111:0/3367665372 >> v1:172.21.15.111:6789/0 conn(0x7f3c8c0ac720 legacy=0x7f3c8c0cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c910e1700 1 -- 172.21.15.111:0/3367665372 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c0ca830 con 0x7f3c8c150420 2023-12-19T08:05:20.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c910e1700 1 --2- 172.21.15.111:0/3367665372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c0c7170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3c880098e0 tx=0x7f3c880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db5921b6840793bd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:20.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c83fff700 1 -- 172.21.15.111:0/3367665372 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8800b640 con 0x7f3c8c150420 2023-12-19T08:05:20.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c83fff700 1 -- 172.21.15.111:0/3367665372 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8800b7a0 con 0x7f3c8c150420 2023-12-19T08:05:20.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c83fff700 1 -- 172.21.15.111:0/3367665372 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8800bb40 con 0x7f3c8c150420 2023-12-19T08:05:20.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 -- 172.21.15.111:0/3367665372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 msgr2=0x7f3c8c0c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 --2- 172.21.15.111:0/3367665372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c0c7170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3c880098e0 tx=0x7f3c880091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 -- 172.21.15.111:0/3367665372 shutdown_connections 2023-12-19T08:05:20.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 --2- 172.21.15.111:0/3367665372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c0c7170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 -- 172.21.15.111:0/3367665372 >> 172.21.15.111:0/3367665372 conn(0x7f3c8c0ccb30 msgr2=0x7f3c8c0ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:20.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 -- 172.21.15.111:0/3367665372 shutdown_connections 2023-12-19T08:05:20.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.742+0000 7f3c920e3700 1 -- 172.21.15.111:0/3367665372 wait complete. 2023-12-19T08:05:20.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:20.746+0000 7f17861ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:05:20.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 Processor -- start 2023-12-19T08:05:20.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 -- start start 2023-12-19T08:05:20.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8c0ac720 0x7f3c8c144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:20.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c06d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c16aae0 con 0x7f3c8c156e80 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c145010 con 0x7f3c8c150420 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.746+0000 7f3c920e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c169970 con 0x7f3c8c0ac720 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c908e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c06d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56392/0 (socket says 172.21.15.111:56392) 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 -- 172.21.15.111:0/4078207190 learned_addr learned my addr 172.21.15.111:0/4078207190 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c910e1700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8c0ac720 0x7f3c8c144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 -- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8c0ac720 msgr2=0x7f3c8c144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8c0ac720 0x7f3c8c144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 -- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 msgr2=0x7f3c8c06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c06d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 -- 172.21.15.111:0/4078207190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88009420 con 0x7f3c8c156e80 2023-12-19T08:05:20.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c908e0700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c06d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c918e2700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400cca0 tx=0x7f3c84001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400a8d0 con 0x7f3c8c156e80 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8400aa30 con 0x7f3c8c156e80 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400adb0 con 0x7f3c8c156e80 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c169be0 con 0x7f3c8c156e80 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c16a420 con 0x7f3c8c156e80 2023-12-19T08:05:20.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.750+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c04be40 con 0x7f3c8c156e80 2023-12-19T08:05:20.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.754+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 51) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f3c84007300 con 0x7f3c8c156e80 2023-12-19T08:05:20.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.754+0000 7f3c817fa700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 0x7f3c70062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:20.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.754+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f3c8407d0b0 con 0x7f3c8c156e80 2023-12-19T08:05:20.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.754+0000 7f3c910e1700 1 -- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 msgr2=0x7f3c70062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:20.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.754+0000 7f3c910e1700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 0x7f3c70062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:05:20.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.762+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c8404d220 con 0x7f3c8c156e80 2023-12-19T08:05:20.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.934+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3c8c0cab10 con 0x7f3c8c156e80 2023-12-19T08:05:20.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.946+0000 7f3c817fa700 1 -- 172.21.15.111:0/4078207190 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3c84050840 con 0x7f3c8c156e80 2023-12-19T08:05:20.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 msgr2=0x7f3c70062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:05:20.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 0x7f3c70062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 msgr2=0x7f3c8c1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:20.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400cca0 tx=0x7f3c84001ae0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 shutdown_connections 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c700606c0 0x7f3c70062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8c150420 0x7f3c8c06d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c156e80 0x7f3c8c1434f0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 --2- 172.21.15.111:0/4078207190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8c0ac720 0x7f3c8c144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 >> 172.21.15.111:0/4078207190 conn(0x7f3c8c0ccb30 msgr2=0x7f3c8c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 shutdown_connections 2023-12-19T08:05:20.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:20.950+0000 7f3c920e3700 1 -- 172.21.15.111:0/4078207190 wait complete. 2023-12-19T08:05:20.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:05:21.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.146+0000 7f7c15305700 1 Processor -- start 2023-12-19T08:05:21.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.146+0000 7f7c15305700 1 -- start start 2023-12-19T08:05:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c15305700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c100b2470 0x7f7c100cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c15305700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1006d270 con 0x7f7c1015b6b0 2023-12-19T08:05:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c15305700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1006d550 con 0x7f7c1015abd0 2023-12-19T08:05:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c15305700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1006d830 con 0x7f7c100b2470 2023-12-19T08:05:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0e7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7c1015b6b0 0x7f7c101567f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42248/0 (socket says 172.21.15.111:42248) 2023-12-19T08:05:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0e7fc700 1 -- 172.21.15.111:0/905606725 learned_addr learned my addr 172.21.15.111:0/905606725 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0f7fe700 1 --2- 172.21.15.111:0/905606725 >> v2:172.21.15.143:3300/0 conn(0x7f7c100b2470 0x7f7c100cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0dffb700 1 -- 172.21.15.111:0/905606725 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184253341 0 0) 0x7f7c1006d270 con 0x7f7c1015b6b0 2023-12-19T08:05:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0dffb700 1 -- 172.21.15.111:0/905606725 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8003570 con 0x7f7c1015b6b0 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0f7fe700 1 -- 172.21.15.111:0/905606725 >> v1:172.21.15.143:6789/0 conn(0x7f7c1015abd0 legacy=0x7f7c1015afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0f7fe700 1 -- 172.21.15.111:0/905606725 >> v1:172.21.15.111:6789/0 conn(0x7f7c1015b6b0 legacy=0x7f7c101567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0f7fe700 1 -- 172.21.15.111:0/905606725 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1006db10 con 0x7f7c100b2470 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0f7fe700 1 --2- 172.21.15.111:0/905606725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c100b2470 0x7f7c100cd3f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e60 tx=0x7f7c08009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b392a9bff567f72 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0dffb700 1 -- 172.21.15.111:0/905606725 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0800e070 con 0x7f7c100b2470 2023-12-19T08:05:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0dffb700 1 -- 172.21.15.111:0/905606725 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0800bb60 con 0x7f7c100b2470 2023-12-19T08:05:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.150+0000 7f7c0dffb700 1 -- 172.21.15.111:0/905606725 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08012430 con 0x7f7c100b2470 2023-12-19T08:05:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- 172.21.15.111:0/905606725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c100b2470 msgr2=0x7f7c100cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 --2- 172.21.15.111:0/905606725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c100b2470 0x7f7c100cd3f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e60 tx=0x7f7c08009780 comp rx=0 tx=0).stop 2023-12-19T08:05:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- 172.21.15.111:0/905606725 shutdown_connections 2023-12-19T08:05:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 --2- 172.21.15.111:0/905606725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c100b2470 0x7f7c100cd3f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- 172.21.15.111:0/905606725 >> 172.21.15.111:0/905606725 conn(0x7f7c100cdef0 msgr2=0x7f7c100ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- 172.21.15.111:0/905606725 shutdown_connections 2023-12-19T08:05:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- 172.21.15.111:0/905606725 wait complete. 2023-12-19T08:05:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 Processor -- start 2023-12-19T08:05:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.154+0000 7f7c15305700 1 -- start start 2023-12-19T08:05:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c1015abd0 0x7f7c1014f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 0x7f7c1014fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100ad950 con 0x7f7c100b2470 2023-12-19T08:05:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100adc30 con 0x7f7c1015abd0 2023-12-19T08:05:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100adf10 con 0x7f7c1015b6b0 2023-12-19T08:05:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0e7fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c1015abd0 0x7f7c1014f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56394/0 (socket says 172.21.15.111:56394) 2023-12-19T08:05:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 -- 172.21.15.111:0/3681652857 learned_addr learned my addr 172.21.15.111:0/3681652857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0f7fe700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 0x7f7c1014fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 -- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 msgr2=0x7f7c1014fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 0x7f7c1014fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 -- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c1015abd0 msgr2=0x7f7c1014f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c1015abd0 0x7f7c1014f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 -- 172.21.15.111:0/3681652857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009a20 con 0x7f7c100b2470 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0f7fe700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 0x7f7c1014fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c0effd700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c590 tx=0x7f7c0000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00011590 con 0x7f7c100b2470 2023-12-19T08:05:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00011710 con 0x7f7c100b2470 2023-12-19T08:05:21.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c100ae1f0 con 0x7f7c100b2470 2023-12-19T08:05:21.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00011ad0 con 0x7f7c100b2470 2023-12-19T08:05:21.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.158+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c10153560 con 0x7f7c100b2470 2023-12-19T08:05:21.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c10071120 con 0x7f7c100b2470 2023-12-19T08:05:21.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 51) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f7c0001b3b0 con 0x7f7c100b2470 2023-12-19T08:05:21.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7bf6ffd700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 0x7f7bfc062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7c0e7fc700 1 -- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 msgr2=0x7f7bfc062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:21.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7f7c0007c280 con 0x7f7c100b2470 2023-12-19T08:05:21.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.162+0000 7f7c0e7fc700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 0x7f7bfc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:05:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.170+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c0004c370 con 0x7f7c100b2470 2023-12-19T08:05:21.291 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:21.289+0000 7f17861ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:05:21.304 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:21.302+0000 7f28f5fce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.342+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7c1014dca0 con 0x7f7c100b2470 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.346+0000 7f7bf6ffd700 1 -- 172.21.15.111:0/3681652857 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7c0004f990 con 0x7f7c100b2470 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 msgr2=0x7f7bfc062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 0x7f7bfc062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 msgr2=0x7f7c1014a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c590 tx=0x7f7c0000c8a0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 shutdown_connections 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7bfc0606c0 0x7f7bfc062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c1015abd0 0x7f7c1014f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c1015b6b0 0x7f7c1014fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 --2- 172.21.15.111:0/3681652857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c100b2470 0x7f7c1014a9a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 >> 172.21.15.111:0/3681652857 conn(0x7f7c100cdef0 msgr2=0x7f7c1014c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 shutdown_connections 2023-12-19T08:05:21.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.350+0000 7f7c15305700 1 -- 172.21.15.111:0/3681652857 wait complete. 2023-12-19T08:05:21.367 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:21.365+0000 7f17861ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:05:21.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:21.372 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T08:05:21.378 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:21.374+0000 7f28f5fce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:05:21.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T08:05:21.503 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:21.501+0000 7f17861ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:05:21.533 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:21.530+0000 7f28f5fce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:05:21.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.602+0000 7fccae0cd700 1 Processor -- start 2023-12-19T08:05:21.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 -- start start 2023-12-19T08:05:21.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 0x7fcca80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca80b0ac0 con 0x7fcca806a6a0 2023-12-19T08:05:21.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca806b030 con 0x7fcca80cb130 2023-12-19T08:05:21.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.606+0000 7fccae0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca806b210 con 0x7fcca815c930 2023-12-19T08:05:21.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca6ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcca806a6a0 0x7fcca815c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:42264/0 (socket says 172.21.15.111:42264) 2023-12-19T08:05:21.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2902225806 learned_addr learned my addr 172.21.15.111:0/2902225806 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:05:21.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca77fe700 1 --2- 172.21.15.111:0/2902225806 >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 0x7fcca80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca67fc700 1 -- 172.21.15.111:0/2902225806 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445010826 0 0) 0x7fcca80b0ac0 con 0x7fcca806a6a0 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca67fc700 1 -- 172.21.15.111:0/2902225806 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc90003570 con 0x7fcca806a6a0 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 -- 172.21.15.111:0/2902225806 >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 msgr2=0x7fcca80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 --2- 172.21.15.111:0/2902225806 >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 0x7fcca80cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 -- 172.21.15.111:0/2902225806 >> v1:172.21.15.111:6789/0 conn(0x7fcca806a6a0 legacy=0x7fcca815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 -- 172.21.15.111:0/2902225806 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca80ca810 con 0x7fcca815c930 2023-12-19T08:05:21.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca77fe700 1 --2- 172.21.15.111:0/2902225806 >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 0x7fcca80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:05:21.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca7fff700 1 --2- 172.21.15.111:0/2902225806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca806aaf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009e60 tx=0x7fcca0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a419aa0f11738ada server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:21.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca67fc700 1 -- 172.21.15.111:0/2902225806 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca000e070 con 0x7fcca815c930 2023-12-19T08:05:21.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca67fc700 1 -- 172.21.15.111:0/2902225806 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca000bab0 con 0x7fcca815c930 2023-12-19T08:05:21.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fcca67fc700 1 -- 172.21.15.111:0/2902225806 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca000be70 con 0x7fcca815c930 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 -- 172.21.15.111:0/2902225806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 msgr2=0x7fcca806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2902225806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca806aaf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009e60 tx=0x7fcca0009780 comp rx=0 tx=0).stop 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 -- 172.21.15.111:0/2902225806 shutdown_connections 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2902225806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca806aaf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2902225806 >> v2:172.21.15.143:3300/0 conn(0x7fcca80cb130 0x7fcca80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 -- 172.21.15.111:0/2902225806 >> 172.21.15.111:0/2902225806 conn(0x7fcca8067bb0 msgr2=0x7fcca80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 -- 172.21.15.111:0/2902225806 shutdown_connections 2023-12-19T08:05:21.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.610+0000 7fccae0cd700 1 -- 172.21.15.111:0/2902225806 wait complete. 2023-12-19T08:05:21.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.614+0000 7fccae0cd700 1 Processor -- start 2023-12-19T08:05:21.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.614+0000 7fccae0cd700 1 -- start start 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.614+0000 7fccae0cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcca80cb130 0x7fcca814d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca80b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81480e0 con 0x7fcca806a6a0 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81483c0 con 0x7fcca80cb130 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81486a0 con 0x7fcca815c930 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56400/0 (socket says 172.21.15.111:56400) 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 -- 172.21.15.111:0/2010329761 learned_addr learned my addr 172.21.15.111:0/2010329761 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca7fff700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca80b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcca80cb130 0x7fcca814d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 msgr2=0x7fcca80b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca80b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcca80cb130 msgr2=0x7fcca814d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcca80cb130 0x7fcca814d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:21.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc98009420 con 0x7fcca806a6a0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcca77fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800ec80 tx=0x7fcc980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9800c9d0 con 0x7fcca806a6a0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9800cb30 con 0x7fcca806a6a0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca0009a20 con 0x7fcca806a6a0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9800ceb0 con 0x7fcca806a6a0 2023-12-19T08:05:21.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca8148c60 con 0x7fcca806a6a0 2023-12-19T08:05:21.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.618+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca8071120 con 0x7fcca806a6a0 2023-12-19T08:05:21.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.622+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 51) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fcc98013680 con 0x7fcca806a6a0 2023-12-19T08:05:21.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.622+0000 7fcc8f7fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:21.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.622+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fcc9807d0c0 con 0x7fcca806a6a0 2023-12-19T08:05:21.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.622+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:21.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.622+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:05:21.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.630+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcc9804d250 con 0x7fcca806a6a0 2023-12-19T08:05:21.756 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:21.753+0000 7f17861ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:05:21.786 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:21.782+0000 7f28f5fce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:05:21.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.818+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fcca80cbe90 con 0x7fcc940606c0 2023-12-19T08:05:21.826 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:21.825+0000 7f17861ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:05:21.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.826+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:21.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:21.826+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:05:21.857 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:21.854+0000 7f28f5fce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:05:22.034 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:22.033+0000 7f17861ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:05:22.063 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:22.058+0000 7f28f5fce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:05:22.105 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:22.105+0000 7f17861ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:05:22.142 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:22.138+0000 7f28f5fce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:05:22.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:22.185+0000 7f17861ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:05:22.224 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:22.222+0000 7f28f5fce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:05:22.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:22.230+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:22.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:22.230+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:05:22.557 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:22.557+0000 7f17861ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:05:22.596 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:22.594+0000 7f28f5fce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:05:22.720 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:22.721+0000 7f17861ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:05:22.754 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:22.750+0000 7f28f5fce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:05:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:23.034+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:23.034+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:05:23.064 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.061+0000 7f17861ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:05:23.095 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.090+0000 7f28f5fce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:05:23.287 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.285+0000 7f17861ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:05:23.326 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.322+0000 7f28f5fce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:05:23.380 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.381+0000 7f17861ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:05:23.417 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.414+0000 7f28f5fce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:05:23.460 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.461+0000 7f17861ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:05:23.501 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.498+0000 7f28f5fce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:05:23.545 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.545+0000 7f17861ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:05:23.584 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.582+0000 7f28f5fce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:05:23.787 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.785+0000 7f17861ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:05:23.819 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.814+0000 7f28f5fce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:05:23.869 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.869+0000 7f17861ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:05:23.907 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:23.902+0000 7f28f5fce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:05:23.965 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:23.965+0000 7f17861ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:05:24.004 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:24.002+0000 7f28f5fce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:05:24.175 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:24.173+0000 7f17861ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:05:24.218 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:24.214+0000 7f28f5fce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:05:24.263 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:24.261+0000 7f17861ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:05:24.310 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:24.306+0000 7f28f5fce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:05:24.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:24.638+0000 7fcca6ffd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:05:24.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:24.638+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:05:25.016 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.013+0000 7f17861ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:05:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:25.034+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 52) v1 ==== 79689+0+0 (secure 0 0 0) 0x7fcc9804d620 con 0x7fcca806a6a0 2023-12-19T08:05:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:25.034+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:05:25.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:25.034+0000 7fcc8f7fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:25.070 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:05:25.066+0000 7f28f5fce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1748456700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1748456700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1748456700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1748456700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1748456700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1745450700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1745450700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.223 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1745450700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.224 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1745450700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:25.224 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:05:25.221+0000 7f1745450700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:05:26.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.038+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 53) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fcc9807b250 con 0x7fcca806a6a0 2023-12-19T08:05:26.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.042+0000 7fcc8f7fe700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.042+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fcca80cbe90 con 0x7fcc940606c0 2023-12-19T08:05:26.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.046+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.050+0000 7fcca6ffd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c007fd0 tx=0x7fcc9c009040 comp rx=0 tx=0).ready entity=mgr.8073 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.162+0000 7fcc8f7fe700 1 -- 172.21.15.111:0/2010329761 <== mgr.8073 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcca80cbe90 con 0x7fcc940606c0 2023-12-19T08:05:26.165 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T08:05:26.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 msgr2=0x7fcc94062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c007fd0 tx=0x7fcc9c009040 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 msgr2=0x7fcca8147ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800ec80 tx=0x7fcc980091a0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 shutdown_connections 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcc940606c0 0x7fcc94062b70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcca80cb130 0x7fcca814d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcca815c930 0x7fcca80b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 --2- 172.21.15.111:0/2010329761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcca806a6a0 0x7fcca8147ab0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 >> 172.21.15.111:0/2010329761 conn(0x7fcca8067bb0 msgr2=0x7fcca815b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 shutdown_connections 2023-12-19T08:05:26.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.166+0000 7fccae0cd700 1 -- 172.21.15.111:0/2010329761 wait complete. 2023-12-19T08:05:26.189 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T08:05:26.190 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-08_06_00 in ~34s... 2023-12-19T08:05:26.190 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-12-19T08:05:26.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T08:05:26.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.354+0000 7f16bd145700 1 Processor -- start 2023-12-19T08:05:26.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.354+0000 7f16bd145700 1 -- start start 2023-12-19T08:05:26.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16bd145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 0x7f16b80c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16bd145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16b8159220 0x7f16b8068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16bd145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806d0d0 con 0x7f16b8069310 2023-12-19T08:05:26.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16bd145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b806c7c0 con 0x7f16b8159220 2023-12-19T08:05:26.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16bd145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c7b40 con 0x7f16b8158e50 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16b8159220 0x7f16b8068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16b8159220 0x7f16b8068380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37268/0 (socket says 172.21.15.111:37268) 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 -- 172.21.15.111:0/4277169801 learned_addr learned my addr 172.21.15.111:0/4277169801 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16b6d9d700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 0x7f16b80c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 -- 172.21.15.111:0/4277169801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 msgr2=0x7f16b80c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 0x7f16b80c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 -- 172.21.15.111:0/4277169801 >> v1:172.21.15.111:6789/0 conn(0x7f16b8069310 legacy=0x7f16b815cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 -- 172.21.15.111:0/4277169801 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b806ca80 con 0x7f16b8159220 2023-12-19T08:05:26.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16b6d9d700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 0x7f16b80c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:05:26.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16affff700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b8068380 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f16a00099b0 tx=0x7f16a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad64dfbcbdc9727 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:26.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.358+0000 7f16b659c700 1 -- 172.21.15.111:0/4277169801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a000b860 con 0x7f16b8159220 2023-12-19T08:05:26.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16b659c700 1 -- 172.21.15.111:0/4277169801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a000b9c0 con 0x7f16b8159220 2023-12-19T08:05:26.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16b659c700 1 -- 172.21.15.111:0/4277169801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a000bd80 con 0x7f16b8159220 2023-12-19T08:05:26.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- 172.21.15.111:0/4277169801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 msgr2=0x7f16b8068380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b8068380 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f16a00099b0 tx=0x7f16a0009250 comp rx=0 tx=0).stop 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- 172.21.15.111:0/4277169801 shutdown_connections 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b8068380 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 --2- 172.21.15.111:0/4277169801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8158e50 0x7f16b80c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- 172.21.15.111:0/4277169801 >> 172.21.15.111:0/4277169801 conn(0x7f16b80c8db0 msgr2=0x7f16b80c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- 172.21.15.111:0/4277169801 shutdown_connections 2023-12-19T08:05:26.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- 172.21.15.111:0/4277169801 wait complete. 2023-12-19T08:05:26.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 Processor -- start 2023-12-19T08:05:26.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.362+0000 7f16bd145700 1 -- start start 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8069310 0x7f16b8154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16b8158e50 0x7f16b80b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c6230 con 0x7f16b8159220 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b1030 con 0x7f16b8158e50 2023-12-19T08:05:26.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b1310 con 0x7f16b8069310 2023-12-19T08:05:26.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16b8158e50 0x7f16b80b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8069310 0x7f16b8154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37284/0 (socket says 172.21.15.111:37284) 2023-12-19T08:05:26.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 -- 172.21.15.111:0/2744986404 learned_addr learned my addr 172.21.15.111:0/2744986404 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 -- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8069310 msgr2=0x7f16b8154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8069310 0x7f16b8154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 -- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16b8158e50 msgr2=0x7f16b80b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16b8158e50 0x7f16b80b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 -- 172.21.15.111:0/2744986404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00094f0 con 0x7f16b8159220 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16b759e700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c8e0 tx=0x7f16b0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0012630 con 0x7f16b8159220 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b0012790 con 0x7f16b8159220 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0012b50 con 0x7f16b8159220 2023-12-19T08:05:26.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80b15f0 con 0x7f16b8159220 2023-12-19T08:05:26.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.366+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80b1bb0 con 0x7f16b8159220 2023-12-19T08:05:26.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.370+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80b1da0 con 0x7f16b8159220 2023-12-19T08:05:26.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.370+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 53) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f16b001a3a0 con 0x7f16b8159220 2023-12-19T08:05:26.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.370+0000 7f16aeffd700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 0x7f16980628d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:05:26.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.370+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f16b007cbc0 con 0x7f16b8159220 2023-12-19T08:05:26.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.370+0000 7f16b6d9d700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 0x7f16980628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:05:26.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.374+0000 7f16b6d9d700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 0x7f16980628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16b806e890 tx=0x7f16a8006f90 comp rx=0 tx=0).ready entity=mgr.8073 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:05:26.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.378+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f16b004cef0 con 0x7f16b8159220 2023-12-19T08:05:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.570+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16b8071120 con 0x7f1698060420 2023-12-19T08:05:26.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.586+0000 7f16aeffd700 1 -- 172.21.15.111:0/2744986404 <== mgr.8073 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f16b8071120 con 0x7f1698060420 2023-12-19T08:05:26.589 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:05:26.589 INFO:teuthology.orchestra.run.smithi111.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 msgr2=0x7f16980628d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 0x7f16980628d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16b806e890 tx=0x7f16a8006f90 comp rx=0 tx=0).stop 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 msgr2=0x7f16b80b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f16b000c8e0 tx=0x7f16b0001710 comp rx=0 tx=0).stop 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 shutdown_connections 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1698060420 0x7f16980628d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16b8158e50 0x7f16b80b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b8159220 0x7f16b80b09e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 --2- 172.21.15.111:0/2744986404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16b8069310 0x7f16b8154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:05:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 >> 172.21.15.111:0/2744986404 conn(0x7f16b80c8db0 msgr2=0x7f16b80ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:05:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 shutdown_connections 2023-12-19T08:05:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:05:26.590+0000 7f16bd145700 1 -- 172.21.15.111:0/2744986404 wait complete. 2023-12-19T08:05:26.614 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{}]} 2023-12-19T08:05:26.614 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{}]} 2023-12-19T08:05:26.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:26.614 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:26.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:26.647 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:26.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:27.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:27.648 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:27.687 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:27.687 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:27.687 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:28.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:28.688 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:28.725 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:28.726 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:28.726 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:29.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:29.727 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:29.767 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:29.767 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:29.768 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:30.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:30.769 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:30.808 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:30.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:30.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:31.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:31.810 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:31.849 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:31.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:31.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:32.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:32.851 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:32.887 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:32.887 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:32.887 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:33.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:33.889 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:33.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:33.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:33.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:34.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:34.928 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:34.968 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:34.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:34.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:35.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:35.969 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:36.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:36.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:36.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:36.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:05:36.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:05:37.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:37.010 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:37.048 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:37.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:37.048 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:38.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:38.049 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:38.090 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:38.090 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:38.090 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:39.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:39.091 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:39.130 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:39.130 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:39.130 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:40.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:40.131 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:40.170 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:40.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:40.170 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:41.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:41.171 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:41.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:41.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:41.207 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:42.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:42.208 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:42.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:42.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:42.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:43.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:43.247 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:43.280 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:43.280 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:43.280 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:44.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:44.281 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:44.318 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:44.318 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:44.318 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:45.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:45.319 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:45.356 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:45.356 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:45.356 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:46.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:46.357 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:46.394 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:46.394 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:46.394 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:47.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:47.395 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:47.433 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:47.433 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:47.433 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:48.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:48.434 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:48.469 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:48.469 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:48.469 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:49.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:49.471 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:49.508 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:49.509 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:49.509 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:50.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:50.510 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:50.549 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:50.549 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:50.549 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:51.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:51.550 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:51.588 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:51.588 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:51.588 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:52.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:52.589 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:52.627 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:52.627 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:52.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:53.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:53.629 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:53.663 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:53.663 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:53.663 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:54.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:54.664 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:54.701 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:54.701 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:54.701 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:55.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:55.702 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:55.739 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:55.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:55.740 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:56.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:56.741 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:56.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:56.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:56.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:57.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:57.780 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:57.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:57.817 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:57.817 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:58.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:58.819 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:58.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:58.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:58.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:05:59.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:05:59.857 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:05:59.894 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T08:05:59.894 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T08:05:59.895 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:06:00.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:00.185+0000 7f1737c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:00.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:00.185+0000 7f1737c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:00.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:00.185+0000 7f1737c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:00.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:00.185+0000 7f1737c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:00.185 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:00.185+0000 7f1737c35700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:00.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:00.896 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:06:00.930 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_06_00_UTC 2023-12-19T08:06:00.932 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-08_06_00_UTC'} 2023-12-19T08:06:00.932 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-08_06_00_UTC'} 2023-12-19T08:06:00.932 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T08:06:00.932 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-08_06_00_UTC'] 2023-12-19T08:06:00.932 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702973126.1900723, now = 1702973160.9329326, timo = 34 2023-12-19T08:06:01.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T08:06:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 Processor -- start 2023-12-19T08:06:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 -- start start 2023-12-19T08:06:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f38bc06b730 0x7f38bc06bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc0a9b70 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc06aaf0 con 0x7f38bc06a580 2023-12-19T08:06:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bbfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc06b0b0 con 0x7f38bc06b730 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38baffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f38bc06b730 0x7f38bc06bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38b3fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f38bc06a580 0x7f38bc0a9460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:53064/0 (socket says 172.21.15.111:53064) 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38b3fff700 1 -- 172.21.15.111:0/4264496396 learned_addr learned my addr 172.21.15.111:0/4264496396 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38bb7fe700 1 --1- 172.21.15.111:0/4264496396 >> v1:172.21.15.111:6789/0 conn(0x7f38bc0ab4b0 0x7f38bc06d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55698/0 (socket says 172.21.15.111:55698) 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38baffd700 1 -- 172.21.15.111:0/4264496396 >> v1:172.21.15.143:6789/0 conn(0x7f38bc06a580 legacy=0x7f38bc0a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38baffd700 1 -- 172.21.15.111:0/4264496396 >> v1:172.21.15.111:6789/0 conn(0x7f38bc0ab4b0 legacy=0x7f38bc06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.097+0000 7f38baffd700 1 -- 172.21.15.111:0/4264496396 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc06add0 con 0x7f38bc06b730 2023-12-19T08:06:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38baffd700 1 --2- 172.21.15.111:0/4264496396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc06b730 0x7f38bc06bb00 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0098e0 tx=0x7f38ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66561dffa66a29f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38ba7fc700 1 -- 172.21.15.111:0/4264496396 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac009e80 con 0x7f38bc06b730 2023-12-19T08:06:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38ba7fc700 1 -- 172.21.15.111:0/4264496396 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac00b860 con 0x7f38bc06b730 2023-12-19T08:06:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38ba7fc700 1 -- 172.21.15.111:0/4264496396 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00bbe0 con 0x7f38bc06b730 2023-12-19T08:06:02.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- 172.21.15.111:0/4264496396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc06b730 msgr2=0x7f38bc06bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 --2- 172.21.15.111:0/4264496396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc06b730 0x7f38bc06bb00 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0098e0 tx=0x7f38ac0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- 172.21.15.111:0/4264496396 shutdown_connections 2023-12-19T08:06:02.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 --2- 172.21.15.111:0/4264496396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc06b730 0x7f38bc06bb00 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- 172.21.15.111:0/4264496396 >> 172.21.15.111:0/4264496396 conn(0x7f38bc067bb0 msgr2=0x7f38bc0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:02.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- 172.21.15.111:0/4264496396 shutdown_connections 2023-12-19T08:06:02.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- 172.21.15.111:0/4264496396 wait complete. 2023-12-19T08:06:02.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 Processor -- start 2023-12-19T08:06:02.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.101+0000 7f38bbfff700 1 -- start start 2023-12-19T08:06:02.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38bc06b730 0x7f38bc1564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 0x7f38bc0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc157590 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc15aa80 con 0x7f38bc06b730 2023-12-19T08:06:02.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc152210 con 0x7f38bc06a580 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 0x7f38bc0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38baffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38baffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52212/0 (socket says 172.21.15.111:52212) 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38baffd700 1 -- 172.21.15.111:0/546095280 learned_addr learned my addr 172.21.15.111:0/546095280 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38b3fff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38bc06b730 0x7f38bc1564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 -- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 msgr2=0x7f38bc151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 -- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38bc06b730 msgr2=0x7f38bc1564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38bc06b730 0x7f38bc1564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 -- 172.21.15.111:0/546095280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac009420 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38baffd700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:02.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bb7fe700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 0x7f38bc0aff90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f38b400cc70 tx=0x7f38b4001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4005df0 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b400a640 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc15a750 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b400aa00 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.105+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc156da0 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.109+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc1573d0 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.109+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 55) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f38b40070f0 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.109+0000 7f38b2ffd700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 0x7f389c062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.109+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f38b407cf20 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.109+0000 7f38baffd700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 0x7f389c062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.113+0000 7f38baffd700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 0x7f389c062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0030c0 tx=0x7f38ac017000 comp rx=0 tx=0).ready entity=mgr.8073 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.117+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38b404ce60 con 0x7f38bc0ab4b0 2023-12-19T08:06:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.309+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f38bc04be40 con 0x7f389c0607e0 2023-12-19T08:06:02.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.433+0000 7f38b2ffd700 1 -- 172.21.15.111:0/546095280 <== mgr.8073 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f38bc04be40 con 0x7f389c0607e0 2023-12-19T08:06:02.437 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path snap_test_dir1 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.433+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 msgr2=0x7f389c062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.433+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 0x7f389c062c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38ac0030c0 tx=0x7f38ac017000 comp rx=0 tx=0).stop 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.433+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 msgr2=0x7f38bc0aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.433+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 0x7f38bc0aff90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f38b400cc70 tx=0x7f38b4001a50 comp rx=0 tx=0).stop 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 shutdown_connections 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f389c0607e0 0x7f389c062c90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38bc06b730 0x7f38bc1564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38bc0ab4b0 0x7f38bc0aff90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 --2- 172.21.15.111:0/546095280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38bc06a580 0x7f38bc151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 >> 172.21.15.111:0/546095280 conn(0x7f38bc067bb0 msgr2=0x7f38bc0ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 shutdown_connections 2023-12-19T08:06:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.437+0000 7f38bbfff700 1 -- 172.21.15.111:0/546095280 wait complete. 2023-12-19T08:06:02.459 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T08:06:02.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:02.460 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T08:06:02.492 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_06_00_UTC 2023-12-19T08:06:02.494 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_06_00_UTC 2023-12-19T08:06:02.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:02.494 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_06_00_UTC) 2023-12-19T08:06:02.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:02.577 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T08:06:02.652 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T08:06:02.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 Processor -- start 2023-12-19T08:06:02.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 -- start start 2023-12-19T08:06:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8cc06dcf0 0x7fb8cc159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc06c850 con 0x7fb8cc06aeb0 2023-12-19T08:06:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc0cb650 con 0x7fb8cc158c90 2023-12-19T08:06:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1b23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8cc06b470 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8cbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb8cc06aeb0 0x7fb8cc159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55702/0 (socket says 172.21.15.111:55702) 2023-12-19T08:06:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8cbfff700 1 -- 172.21.15.111:0/1247318473 learned_addr learned my addr 172.21.15.111:0/1247318473 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8d1322700 1 --2- 172.21.15.111:0/1247318473 >> v2:172.21.15.143:3300/0 conn(0x7fb8cc06dcf0 0x7fb8cc159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229936047 0 0) 0x7fb8cc06c850 con 0x7fb8cc06aeb0 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc003570 con 0x7fb8cc06aeb0 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.873+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249256054 0 0) 0x7fb8cc0cb650 con 0x7fb8cc158c90 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8cc06c850 con 0x7fb8cc158c90 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1322700 1 -- 172.21.15.111:0/1247318473 >> v1:172.21.15.143:6789/0 conn(0x7fb8cc158c90 legacy=0x7fb8cc0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1322700 1 -- 172.21.15.111:0/1247318473 >> v1:172.21.15.111:6789/0 conn(0x7fb8cc06aeb0 legacy=0x7fb8cc159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1322700 1 -- 172.21.15.111:0/1247318473 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc06b750 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1322700 1 --2- 172.21.15.111:0/1247318473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc159770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009e60 tx=0x7fb8c4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a0922d533dec6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c400e070 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c400bab0 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8cb7fe700 1 -- 172.21.15.111:0/1247318473 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c400be70 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 -- 172.21.15.111:0/1247318473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 msgr2=0x7fb8cc159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/1247318473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc159770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009e60 tx=0x7fb8c4009780 comp rx=0 tx=0).stop 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 -- 172.21.15.111:0/1247318473 shutdown_connections 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/1247318473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc159770 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 -- 172.21.15.111:0/1247318473 >> 172.21.15.111:0/1247318473 conn(0x7fb8cc067bb0 msgr2=0x7fb8cc0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 -- 172.21.15.111:0/1247318473 shutdown_connections 2023-12-19T08:06:02.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.877+0000 7fb8d1b23700 1 -- 172.21.15.111:0/1247318473 wait complete. 2023-12-19T08:06:02.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 Processor -- start 2023-12-19T08:06:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 -- start start 2023-12-19T08:06:02.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 0x7fb8cc155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc0cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1569d0 con 0x7fb8cc158c90 2023-12-19T08:06:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1560a0 con 0x7fb8cc06dcf0 2023-12-19T08:06:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1b23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1693d0 con 0x7fb8cc06aeb0 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1322700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1322700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60474/0 (socket says 172.21.15.111:60474) 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d1322700 1 -- 172.21.15.111:0/198658011 learned_addr learned my addr 172.21.15.111:0/198658011 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8d0b21700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 0x7fb8cc155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.881+0000 7fb8cbfff700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc0cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 -- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 msgr2=0x7fb8cc155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 0x7fb8cc155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 -- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 msgr2=0x7fb8cc0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc0cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 -- 172.21.15.111:0/198658011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4009a20 con 0x7fb8cc158c90 2023-12-19T08:06:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d0b21700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 0x7fb8cc155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1322700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb8c400b610 tx=0x7fb8c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4012de0 con 0x7fb8cc158c90 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c4026480 con 0x7fb8cc158c90 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc1690d0 con 0x7fb8cc158c90 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4026840 con 0x7fb8cc158c90 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc0b4630 con 0x7fb8cc158c90 2023-12-19T08:06:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.885+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc04be40 con 0x7fb8cc158c90 2023-12-19T08:06:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.889+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 55) v1 ==== 80194+0+0 (secure 0 0 0) 0x7fb8c401c400 con 0x7fb8cc158c90 2023-12-19T08:06:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.889+0000 7fb8c8ff9700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 0x7fb8b0062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.889+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fb8c407f0a0 con 0x7fb8cc158c90 2023-12-19T08:06:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.889+0000 7fb8d0b21700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 0x7fb8b0062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:02.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.889+0000 7fb8d0b21700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 0x7fb8b0062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0007fd0 tx=0x7fb8c0007d50 comp rx=0 tx=0).ready entity=mgr.8073 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:02.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:02.897+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb8c404ee50 con 0x7fb8cc158c90 2023-12-19T08:06:03.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:03.117+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb8cc169510 con 0x7fb8cc158c90 2023-12-19T08:06:04.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.129+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fb8c404f220 con 0x7fb8cc158c90 2023-12-19T08:06:04.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8c8ff9700 1 -- 172.21.15.111:0/198658011 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v56) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb8c4026bb0 con 0x7fb8cc158c90 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 msgr2=0x7fb8b0062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 0x7fb8b0062c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0007fd0 tx=0x7fb8c0007d50 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 msgr2=0x7fb8cc0cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb8c400b610 tx=0x7fb8c4019000 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 shutdown_connections 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b00607e0 0x7fb8b0062c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8cc06dcf0 0x7fb8cc0cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.133+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8cc158c90 0x7fb8cc0cf1d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.137+0000 7fb8d1b23700 1 --2- 172.21.15.111:0/198658011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8cc06aeb0 0x7fb8cc155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.137+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 >> 172.21.15.111:0/198658011 conn(0x7fb8cc067bb0 msgr2=0x7fb8cc0cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:04.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.137+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 shutdown_connections 2023-12-19T08:06:04.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.137+0000 7fb8d1b23700 1 -- 172.21.15.111:0/198658011 wait complete. 2023-12-19T08:06:04.159 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:06:04.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:06:04.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:06:04.238 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:06:04.238 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T08:06:04.238 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T08:06:04.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:04.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[53999]: fuse finished with error 0 and tester_r 0 2023-12-19T08:06:04.278 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:06:04.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:04.279 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T08:06:04.326 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:04.324+0000 7f26bb5cc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:06:04.329 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:04.325+0000 7fab325c1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:06:04.463 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:06:04.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:04.463 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T08:06:04.482 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T08:06:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:06:04.483 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T08:06:04.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:04.483 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T08:06:04.524 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T08:06:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:06:04.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T08:06:04.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:04.526 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T08:06:04.545 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T08:06:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T08:06:04.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:04.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 Processor -- start 2023-12-19T08:06:04.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 -- start start 2023-12-19T08:06:04.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8600c6900 0x7fa8600c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86006b900 0x7fa86006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86006b470 con 0x7fa86015cc20 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86015d7b0 con 0x7fa8600c6900 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa866016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa860161d10 con 0x7fa86006b900 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa865014700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8600c6900 0x7fa8600c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa865014700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8600c6900 0x7fa8600c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60498/0 (socket says 172.21.15.111:60498) 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa865014700 1 -- 172.21.15.111:0/4066638391 learned_addr learned my addr 172.21.15.111:0/4066638391 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.725+0000 7fa864813700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86006b900 0x7fa86006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa865014700 1 -- 172.21.15.111:0/4066638391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86006b900 msgr2=0x7fa86006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa865014700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86006b900 0x7fa86006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa865014700 1 -- 172.21.15.111:0/4066638391 >> v1:172.21.15.111:6789/0 conn(0x7fa86015cc20 legacy=0x7fa86015d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa865014700 1 -- 172.21.15.111:0/4066638391 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86006bd70 con 0x7fa8600c6900 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa865014700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8600c6900 0x7fa8600c6cd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0099e0 tx=0x7fa85c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b27405a6f47472a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa857fff700 1 -- 172.21.15.111:0/4066638391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c00b820 con 0x7fa8600c6900 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa857fff700 1 -- 172.21.15.111:0/4066638391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c00b980 con 0x7fa8600c6900 2023-12-19T08:06:04.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa857fff700 1 -- 172.21.15.111:0/4066638391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c00bd20 con 0x7fa8600c6900 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 -- 172.21.15.111:0/4066638391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8600c6900 msgr2=0x7fa8600c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8600c6900 0x7fa8600c6cd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0099e0 tx=0x7fa85c009250 comp rx=0 tx=0).stop 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 -- 172.21.15.111:0/4066638391 shutdown_connections 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86006b900 0x7fa86006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 --2- 172.21.15.111:0/4066638391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8600c6900 0x7fa8600c6cd0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 -- 172.21.15.111:0/4066638391 >> 172.21.15.111:0/4066638391 conn(0x7fa860067bb0 msgr2=0x7fa8600cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 -- 172.21.15.111:0/4066638391 shutdown_connections 2023-12-19T08:06:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.729+0000 7fa866016700 1 -- 172.21.15.111:0/4066638391 wait complete. 2023-12-19T08:06:04.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 Processor -- start 2023-12-19T08:06:04.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 -- start start 2023-12-19T08:06:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8600c6900 0x7fa860170fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86015cc20 0x7fa86014a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8601608c0 con 0x7fa86006b900 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa860160300 con 0x7fa8600c6900 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa866016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8601605e0 con 0x7fa86015cc20 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60500/0 (socket says 172.21.15.111:60500) 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 -- 172.21.15.111:0/1576536466 learned_addr learned my addr 172.21.15.111:0/1576536466 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865815700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86015cc20 0x7fa86014a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa864813700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8600c6900 0x7fa860170fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86015cc20 msgr2=0x7fa86014a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86015cc20 0x7fa86014a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8600c6900 msgr2=0x7fa860170fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8600c6900 0x7fa860170fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 -- 172.21.15.111:0/1576536466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0094f0 con 0x7fa86006b900 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.733+0000 7fa865014700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fa860064040 tx=0x7fa85c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c011dc0 con 0x7fa86006b900 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c026450 con 0x7fa86006b900 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86015df70 con 0x7fa86006b900 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c026810 con 0x7fa86006b900 2023-12-19T08:06:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86015e530 con 0x7fa86006b900 2023-12-19T08:06:04.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86004be40 con 0x7fa86006b900 2023-12-19T08:06:04.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.737+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fa85c01a420 con 0x7fa86006b900 2023-12-19T08:06:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.741+0000 7fa8557fa700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 0x7fa844062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.741+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fa85c07ed00 con 0x7fa86006b900 2023-12-19T08:06:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.741+0000 7fa864813700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 msgr2=0x7fa844062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.741+0000 7fa864813700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 0x7fa844062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:04.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.749+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa85c04eb50 con 0x7fa86006b900 2023-12-19T08:06:04.863 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:04.860+0000 7f26bb5cc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:06:04.897 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:04.893+0000 7fab325c1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:06:04.933 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:04.932+0000 7f26bb5cc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:06:04.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.945+0000 7fa864813700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 msgr2=0x7fa844062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:04.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.945+0000 7fa864813700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 0x7fa844062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:06:04.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.965+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa86006a580 con 0x7fa86006b900 2023-12-19T08:06:04.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.965+0000 7fa8557fa700 1 -- 172.21.15.111:0/1576536466 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7fa85c015090 con 0x7fa86006b900 2023-12-19T08:06:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:04.971 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:06:04.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 msgr2=0x7fa844062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:04.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 0x7fa844062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 msgr2=0x7fa8600c0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:04.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fa860064040 tx=0x7fa85c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:04.974 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fab325c1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:06:04.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 shutdown_connections 2023-12-19T08:06:04.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa844060790 0x7fa844062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8600c6900 0x7fa860170fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa86015cc20 0x7fa86014a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 --2- 172.21.15.111:0/1576536466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86006b900 0x7fa8600c0860 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:04.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 >> 172.21.15.111:0/1576536466 conn(0x7fa860067bb0 msgr2=0x7fa860159e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:04.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 shutdown_connections 2023-12-19T08:06:04.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:04.969+0000 7fa866016700 1 -- 172.21.15.111:0/1576536466 wait complete. 2023-12-19T08:06:04.976 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:06:04.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:05.120 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.113+0000 7fab325c1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:05.122 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.120+0000 7f26bb5cc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.149+0000 7fdc563cc700 1 Processor -- start 2023-12-19T08:06:05.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc563cc700 1 -- start start 2023-12-19T08:06:05.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc563cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc50158e20 0x7fdc5006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc563cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5006b730 con 0x7fdc501591f0 2023-12-19T08:06:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc563cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5006ab20 con 0x7fdc500ce810 2023-12-19T08:06:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc563cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5006ae00 con 0x7fdc50158e20 2023-12-19T08:06:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc54bc9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc501591f0 0x7fdc500ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55720/0 (socket says 172.21.15.111:55720) 2023-12-19T08:06:05.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc54bc9700 1 -- 172.21.15.111:0/2713966256 learned_addr learned my addr 172.21.15.111:0/2713966256 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc553ca700 1 --2- 172.21.15.111:0/2713966256 >> v2:172.21.15.143:3300/0 conn(0x7fdc50158e20 0x7fdc5006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374109761 0 0) 0x7fdc5006b730 con 0x7fdc501591f0 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc3c003570 con 0x7fdc501591f0 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc553ca700 1 -- 172.21.15.111:0/2713966256 >> v1:172.21.15.143:6789/0 conn(0x7fdc500ce810 legacy=0x7fdc500ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380925515 0 0) 0x7fdc5006ab20 con 0x7fdc500ce810 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc553ca700 1 -- 172.21.15.111:0/2713966256 >> v1:172.21.15.111:6789/0 conn(0x7fdc501591f0 legacy=0x7fdc500ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc553ca700 1 -- 172.21.15.111:0/2713966256 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5006b0e0 con 0x7fdc50158e20 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.153+0000 7fdc553ca700 1 --2- 172.21.15.111:0/2713966256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc50158e20 0x7fdc5006a540 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796ff6f865c05224 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00b640 con 0x7fdc50158e20 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c00b7a0 con 0x7fdc50158e20 2023-12-19T08:06:05.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc47fff700 1 -- 172.21.15.111:0/2713966256 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00bb40 con 0x7fdc50158e20 2023-12-19T08:06:05.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- 172.21.15.111:0/2713966256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc50158e20 msgr2=0x7fdc5006a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2713966256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc50158e20 0x7fdc5006a540 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0098e0 tx=0x7fdc4c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- 172.21.15.111:0/2713966256 shutdown_connections 2023-12-19T08:06:05.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2713966256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc50158e20 0x7fdc5006a540 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- 172.21.15.111:0/2713966256 >> 172.21.15.111:0/2713966256 conn(0x7fdc50067bb0 msgr2=0x7fdc500cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:05.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- 172.21.15.111:0/2713966256 shutdown_connections 2023-12-19T08:06:05.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- 172.21.15.111:0/2713966256 wait complete. 2023-12-19T08:06:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 Processor -- start 2023-12-19T08:06:05.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.157+0000 7fdc563cc700 1 -- start start 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 0x7fdc50153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 0x7fdc5013fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 0x7fdc50140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501407a0 con 0x7fdc50158e20 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50157630 con 0x7fdc500ce810 2023-12-19T08:06:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5015bdd0 con 0x7fdc501591f0 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc54bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 0x7fdc5013fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc55bcb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 0x7fdc50140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 0x7fdc50153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc54bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 0x7fdc5013fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60514/0 (socket says 172.21.15.111:60514) 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc54bc9700 1 -- 172.21.15.111:0/2151973565 learned_addr learned my addr 172.21.15.111:0/2151973565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 -- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 msgr2=0x7fdc50140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 0x7fdc50140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 -- 172.21.15.111:0/2151973565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 msgr2=0x7fdc5013fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 0x7fdc5013fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 -- 172.21.15.111:0/2151973565 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c009420 con 0x7fdc500ce810 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc55bcb700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 0x7fdc50140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc553ca700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 0x7fdc50153730 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c016000 tx=0x7fdc4c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c01f070 con 0x7fdc500ce810 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c002f70 con 0x7fdc500ce810 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5015c340 con 0x7fdc500ce810 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00b910 con 0x7fdc500ce810 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.161+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc50154830 con 0x7fdc500ce810 2023-12-19T08:06:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.165+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50154a20 con 0x7fdc500ce810 2023-12-19T08:06:05.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.165+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fdc4c023070 con 0x7fdc500ce810 2023-12-19T08:06:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.169+0000 7fdc457fa700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 0x7fdc34062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.169+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fdc4c07e380 con 0x7fdc500ce810 2023-12-19T08:06:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.169+0000 7fdc54bc9700 1 -- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 msgr2=0x7fdc34062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:05.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.169+0000 7fdc54bc9700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 0x7fdc34062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:05.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.173+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc4c04e090 con 0x7fdc500ce810 2023-12-19T08:06:05.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.361+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc5004be40 con 0x7fdc500ce810 2023-12-19T08:06:05.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc457fa700 1 -- 172.21.15.111:0/2151973565 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17016 (secure 0 0 0) 0x7fdc4c0516b0 con 0x7fdc500ce810 2023-12-19T08:06:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:05.369 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":131,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:05.127681+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2962591998":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/396308477":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3598389140":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/1868422455":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3413002548":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/4220277516":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6833/33796":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6832/33796":"2023-12-20T08:05:25.018350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 msgr2=0x7fdc34062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:05.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 0x7fdc34062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 msgr2=0x7fdc50153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 0x7fdc50153730 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c016000 tx=0x7fdc4c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:05.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 shutdown_connections 2023-12-19T08:06:05.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc34060790 0x7fdc34062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc501591f0 0x7fdc50140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc50158e20 0x7fdc5013fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 --2- 172.21.15.111:0/2151973565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc500ce810 0x7fdc50153730 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 >> 172.21.15.111:0/2151973565 conn(0x7fdc50067bb0 msgr2=0x7fdc50152630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 shutdown_connections 2023-12-19T08:06:05.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.365+0000 7fdc563cc700 1 -- 172.21.15.111:0/2151973565 wait complete. 2023-12-19T08:06:05.388 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.388+0000 7f26bb5cc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:06:05.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:05:08.129042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:05:09.170914+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 131, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T08:06:05.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:06:05.407 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.401+0000 7fab325c1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:06:05.458 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.456+0000 7f26bb5cc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:05.492 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.485+0000 7fab325c1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:05.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 Processor -- start 2023-12-19T08:06:05.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 -- start start 2023-12-19T08:06:05.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11b0154780 0x7f11b015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f11b00adab0 0x7f11b00adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b015ae30 con 0x7f11b006baa0 2023-12-19T08:06:05.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00abf70 con 0x7f11b00adab0 2023-12-19T08:06:05.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b54e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b0154b50 con 0x7f11b0154780 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f11b00adab0 0x7f11b00adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f11b00adab0 0x7f11b00adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60516/0 (socket says 172.21.15.111:60516) 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 -- 172.21.15.111:0/363895916 learned_addr learned my addr 172.21.15.111:0/363895916 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11affff700 1 --2- 172.21.15.111:0/363895916 >> v2:172.21.15.143:3300/0 conn(0x7f11b0154780 0x7f11b015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 -- 172.21.15.111:0/363895916 >> v2:172.21.15.143:3300/0 conn(0x7f11b0154780 msgr2=0x7f11b015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 --2- 172.21.15.111:0/363895916 >> v2:172.21.15.143:3300/0 conn(0x7f11b0154780 0x7f11b015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 -- 172.21.15.111:0/363895916 >> v1:172.21.15.143:6789/0 conn(0x7f11b006baa0 legacy=0x7f11b0157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 -- 172.21.15.111:0/363895916 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00ac210 con 0x7f11b00adab0 2023-12-19T08:06:05.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11b4ce2700 1 --2- 172.21.15.111:0/363895916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b00adab0 0x7f11b00adeb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f11b006a3f0 tx=0x7f11a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e0d22ad4897af8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.545+0000 7f11aeffd700 1 -- 172.21.15.111:0/363895916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800e070 con 0x7f11b00adab0 2023-12-19T08:06:05.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11aeffd700 1 -- 172.21.15.111:0/363895916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a800ba20 con 0x7f11b00adab0 2023-12-19T08:06:05.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11aeffd700 1 -- 172.21.15.111:0/363895916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80092e0 con 0x7f11b00adab0 2023-12-19T08:06:05.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 -- 172.21.15.111:0/363895916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b00adab0 msgr2=0x7f11b00adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 --2- 172.21.15.111:0/363895916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b00adab0 0x7f11b00adeb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f11b006a3f0 tx=0x7f11a8009840 comp rx=0 tx=0).stop 2023-12-19T08:06:05.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 -- 172.21.15.111:0/363895916 shutdown_connections 2023-12-19T08:06:05.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 --2- 172.21.15.111:0/363895916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b00adab0 0x7f11b00adeb0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 --2- 172.21.15.111:0/363895916 >> v2:172.21.15.143:3300/0 conn(0x7f11b0154780 0x7f11b015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 -- 172.21.15.111:0/363895916 >> 172.21.15.111:0/363895916 conn(0x7f11b0067b50 msgr2=0x7f11b00cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:05.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 -- 172.21.15.111:0/363895916 shutdown_connections 2023-12-19T08:06:05.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 -- 172.21.15.111:0/363895916 wait complete. 2023-12-19T08:06:05.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.549+0000 7f11b54e3700 1 Processor -- start 2023-12-19T08:06:05.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 -- start start 2023-12-19T08:06:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 0x7f11b0150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b0158f20 con 0x7f11b0154780 2023-12-19T08:06:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b01583a0 con 0x7f11b006baa0 2023-12-19T08:06:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b54e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b0158680 con 0x7f11b00adab0 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49492/0 (socket says 172.21.15.111:49492) 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11affff700 1 -- 172.21.15.111:0/2596424987 learned_addr learned my addr 172.21.15.111:0/2596424987 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11af7fe700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 0x7f11b0150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60520/0 (socket says 172.21.15.111:60520) 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 -- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 msgr2=0x7f11b0150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 0x7f11b0150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 -- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 msgr2=0x7f11b0157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 -- 172.21.15.111:0/2596424987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a8009ae0 con 0x7f11b0154780 2023-12-19T08:06:05.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11affff700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:05.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11af7fe700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 0x7f11b0150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:05.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.553+0000 7f11b4ce2700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f11a8015000 tx=0x7f11a8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8021070 con 0x7f11b0154780 2023-12-19T08:06:05.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a801caa0 con 0x7f11b0154780 2023-12-19T08:06:05.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a801ce40 con 0x7f11b0154780 2023-12-19T08:06:05.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b0158960 con 0x7f11b0154780 2023-12-19T08:06:05.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b0158090 con 0x7f11b0154780 2023-12-19T08:06:05.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b01587c0 con 0x7f11b0154780 2023-12-19T08:06:05.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.557+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f11a8003040 con 0x7f11b0154780 2023-12-19T08:06:05.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.561+0000 7f1197fff700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 0x7f119c062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.561+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f11a8081de0 con 0x7f11b0154780 2023-12-19T08:06:05.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.561+0000 7f11affff700 1 -- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 msgr2=0x7f119c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:05.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.561+0000 7f11affff700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 0x7f119c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:05.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.565+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f11b01587c0 con 0x7f11b0154780 2023-12-19T08:06:05.668 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.664+0000 7f26bb5cc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:06:05.730 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.725+0000 7fab325c1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:06:05.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.741+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11b0166ed0 con 0x7f11b0154780 2023-12-19T08:06:05.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.741+0000 7f1197fff700 1 -- 172.21.15.111:0/2596424987 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11b0166ed0 con 0x7f11b0154780 2023-12-19T08:06:05.745 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:06:05.747 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.744+0000 7f26bb5cc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:06:05.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.741+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 msgr2=0x7f119c062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:05.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.741+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 0x7f119c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 msgr2=0x7f11b01598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f11a8015000 tx=0x7f11a8018000 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 shutdown_connections 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f119c060790 0x7f119c062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11b0154780 0x7f11b01598d0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11b00adab0 0x7f11b0150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 --2- 172.21.15.111:0/2596424987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11b006baa0 0x7f11b0157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 >> 172.21.15.111:0/2596424987 conn(0x7f11b0067b50 msgr2=0x7f11b0151400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 shutdown_connections 2023-12-19T08:06:05.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.745+0000 7f11b54e3700 1 -- 172.21.15.111:0/2596424987 wait complete. 2023-12-19T08:06:05.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T08:06:05.817 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.813+0000 7fab325c1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:06:05.832 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:05.832+0000 7f26bb5cc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:06:05.909 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:05.905+0000 7fab325c1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:06:05.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.965+0000 7f1bfb4ae700 1 Processor -- start 2023-12-19T08:06:05.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.965+0000 7f1bfb4ae700 1 -- start start 2023-12-19T08:06:05.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1bf406db40 0x7f1bf40cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1bf40c63f0 0x7f1bf40ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406df10 0x7f1bf4159900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf40c9db0 con 0x7f1bf406db40 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf40cb2e0 con 0x7f1bf40c63f0 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bfb4ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40cb5c0 con 0x7f1bf406df10 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1bf406db40 0x7f1bf40cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1bf406db40 0x7f1bf40cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60532/0 (socket says 172.21.15.111:60532) 2023-12-19T08:06:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 -- 172.21.15.111:0/807600704 learned_addr learned my addr 172.21.15.111:0/807600704 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf8a49700 1 --2- 172.21.15.111:0/807600704 >> v2:172.21.15.143:3300/0 conn(0x7f1bf40c63f0 0x7f1bf40ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf9a4b700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406df10 0x7f1bf4159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 -- 172.21.15.111:0/807600704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406df10 msgr2=0x7f1bf4159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406df10 0x7f1bf4159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 -- 172.21.15.111:0/807600704 >> v2:172.21.15.143:3300/0 conn(0x7f1bf40c63f0 msgr2=0x7f1bf40ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 --2- 172.21.15.111:0/807600704 >> v2:172.21.15.143:3300/0 conn(0x7f1bf40c63f0 0x7f1bf40ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 -- 172.21.15.111:0/807600704 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40cb8a0 con 0x7f1bf406db40 2023-12-19T08:06:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bf924a700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406db40 0x7f1bf40cc3e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406a250 tx=0x7f1bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d886980e76bace48 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bebfff700 1 -- 172.21.15.111:0/807600704 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf000b640 con 0x7f1bf406db40 2023-12-19T08:06:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bebfff700 1 -- 172.21.15.111:0/807600704 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf000b7a0 con 0x7f1bf406db40 2023-12-19T08:06:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.969+0000 7f1bebfff700 1 -- 172.21.15.111:0/807600704 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf000bb60 con 0x7f1bf406db40 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/807600704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406db40 msgr2=0x7f1bf40cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406db40 0x7f1bf40cc3e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1bf406a250 tx=0x7f1bf0009250 comp rx=0 tx=0).stop 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/807600704 shutdown_connections 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406df10 0x7f1bf4159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/807600704 >> v2:172.21.15.143:3300/0 conn(0x7f1bf40c63f0 0x7f1bf40ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/807600704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406db40 0x7f1bf40cc3e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/807600704 >> 172.21.15.111:0/807600704 conn(0x7f1bf4067bb0 msgr2=0x7f1bf40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/807600704 shutdown_connections 2023-12-19T08:06:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.973+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/807600704 wait complete. 2023-12-19T08:06:05.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 Processor -- start 2023-12-19T08:06:05.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 -- start start 2023-12-19T08:06:05.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406db40 0x7f1bf40b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bf40c63f0 0x7f1bf414a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4144300 con 0x7f1bf406df10 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4153630 con 0x7f1bf40c63f0 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bfb4ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf4152a20 con 0x7f1bf406db40 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60546/0 (socket says 172.21.15.111:60546) 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 -- 172.21.15.111:0/704345593 learned_addr learned my addr 172.21.15.111:0/704345593 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf924a700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406db40 0x7f1bf40b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf9a4b700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bf40c63f0 0x7f1bf414a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 -- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406db40 msgr2=0x7f1bf40b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406db40 0x7f1bf40b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 -- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bf40c63f0 msgr2=0x7f1bf414a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bf40c63f0 0x7f1bf414a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 -- 172.21.15.111:0/704345593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf00094f0 con 0x7f1bf406df10 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.977+0000 7f1bf8a49700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009e30 tx=0x7f1be40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be400c8c0 con 0x7f1bf406df10 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be400ca20 con 0x7f1bf406df10 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf4152d00 con 0x7f1bf406df10 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be400cde0 con 0x7f1bf406df10 2023-12-19T08:06:05.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf41532c0 con 0x7f1bf406df10 2023-12-19T08:06:05.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.981+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf4071120 con 0x7f1bf406df10 2023-12-19T08:06:05.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.985+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f1be4011660 con 0x7f1bf406df10 2023-12-19T08:06:05.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.985+0000 7f1be97fa700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 0x7f1bd8062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:05.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.985+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f1be407efd0 con 0x7f1bf406df10 2023-12-19T08:06:05.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.985+0000 7f1bf924a700 1 -- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 msgr2=0x7f1bd8062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:05.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.985+0000 7f1bf924a700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 0x7f1bd8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:05.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:05.989+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1be404ee50 con 0x7f1bf406df10 2023-12-19T08:06:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.161+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1bf40c9a40 con 0x7f1bf406df10 2023-12-19T08:06:06.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.161+0000 7f1be97fa700 1 -- 172.21.15.111:0/704345593 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1be4052470 con 0x7f1bf406df10 2023-12-19T08:06:06.167 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T08:06:06.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 msgr2=0x7f1bd8062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:06.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 0x7f1bd8062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 msgr2=0x7f1bf4149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009e30 tx=0x7f1be40091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 shutdown_connections 2023-12-19T08:06:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1bd8060790 0x7f1bd8062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bf40c63f0 0x7f1bf414a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bf406df10 0x7f1bf4149db0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 --2- 172.21.15.111:0/704345593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bf406db40 0x7f1bf40b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 >> 172.21.15.111:0/704345593 conn(0x7f1bf4067bb0 msgr2=0x7f1bf414c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:06.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 shutdown_connections 2023-12-19T08:06:06.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.165+0000 7f1bfb4ae700 1 -- 172.21.15.111:0/704345593 wait complete. 2023-12-19T08:06:06.191 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T08:06:06.213 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:06.212+0000 7f26bb5cc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:06:06.326 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:06.321+0000 7fab325c1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:06:06.367 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:06.364+0000 7f26bb5cc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:06:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 Processor -- start 2023-12-19T08:06:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 -- start start 2023-12-19T08:06:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f394006c7c0 0x7f394006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f394006ac30 0x7f394006b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394006a7a0 con 0x7f39400c6d20 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394006d4b0 con 0x7f394006c7c0 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3947145700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394006d790 con 0x7f394006ac30 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3944ee1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39400c6d20 0x7f39400c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55734/0 (socket says 172.21.15.111:55734) 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f3944ee1700 1 -- 172.21.15.111:0/3418684953 learned_addr learned my addr 172.21.15.111:0/3418684953 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f393ffff700 1 --2- 172.21.15.111:0/3418684953 >> v2:172.21.15.111:3300/0 conn(0x7f394006c7c0 0x7f394006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f39456e2700 1 --2- 172.21.15.111:0/3418684953 >> v2:172.21.15.143:3300/0 conn(0x7f394006ac30 0x7f394006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065439416 0 0) 0x7f394006a7a0 con 0x7f39400c6d20 2023-12-19T08:06:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.397+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003570 con 0x7f39400c6d20 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492536210 0 0) 0x7f3930003570 con 0x7f39400c6d20 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393ffff700 1 -- 172.21.15.111:0/3418684953 >> v2:172.21.15.143:3300/0 conn(0x7f394006ac30 msgr2=0x7f394006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393ffff700 1 --2- 172.21.15.111:0/3418684953 >> v2:172.21.15.143:3300/0 conn(0x7f394006ac30 0x7f394006b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393ffff700 1 -- 172.21.15.111:0/3418684953 >> v1:172.21.15.111:6789/0 conn(0x7f39400c6d20 legacy=0x7f39400c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393ffff700 1 -- 172.21.15.111:0/3418684953 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394006da70 con 0x7f394006c7c0 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393ffff700 1 --2- 172.21.15.111:0/3418684953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006c7c0 0x7f394006cc60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f39400695c0 tx=0x7f392c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f427a01edc040c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934001470 con 0x7f394006c7c0 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c011420 con 0x7f394006c7c0 2023-12-19T08:06:06.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f393f7fe700 1 -- 172.21.15.111:0/3418684953 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c0117a0 con 0x7f394006c7c0 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 -- 172.21.15.111:0/3418684953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006c7c0 msgr2=0x7f394006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 --2- 172.21.15.111:0/3418684953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006c7c0 0x7f394006cc60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f39400695c0 tx=0x7f392c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 -- 172.21.15.111:0/3418684953 shutdown_connections 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 --2- 172.21.15.111:0/3418684953 >> v2:172.21.15.143:3300/0 conn(0x7f394006ac30 0x7f394006b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 --2- 172.21.15.111:0/3418684953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006c7c0 0x7f394006cc60 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 -- 172.21.15.111:0/3418684953 >> 172.21.15.111:0/3418684953 conn(0x7f3940067bb0 msgr2=0x7f39400cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 -- 172.21.15.111:0/3418684953 shutdown_connections 2023-12-19T08:06:06.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 -- 172.21.15.111:0/3418684953 wait complete. 2023-12-19T08:06:06.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.401+0000 7f3947145700 1 Processor -- start 2023-12-19T08:06:06.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- start start 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 0x7f39400acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39400c6d20 0x7f39400ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400ae320 con 0x7f394006ac30 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39400af700 con 0x7f39400c6d20 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39400ad6a0 con 0x7f394006c7c0 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60556/0 (socket says 172.21.15.111:60556) 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 -- 172.21.15.111:0/3900694704 learned_addr learned my addr 172.21.15.111:0/3900694704 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f39456e2700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39400c6d20 0x7f39400ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f393ffff700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 0x7f39400acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 -- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 msgr2=0x7f39400acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 0x7f39400acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 -- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39400c6d20 msgr2=0x7f39400ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39400c6d20 0x7f39400ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 -- 172.21.15.111:0/3900694704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c009420 con 0x7f394006ac30 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f393ffff700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 0x7f39400acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3944ee1700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f393400cb90 tx=0x7f3934001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934007150 con 0x7f394006ac30 2023-12-19T08:06:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39400ad980 con 0x7f394006ac30 2023-12-19T08:06:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39340072d0 con 0x7f394006ac30 2023-12-19T08:06:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934007690 con 0x7f394006ac30 2023-12-19T08:06:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.405+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39400adf40 con 0x7f394006ac30 2023-12-19T08:06:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.409+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3940071120 con 0x7f394006ac30 2023-12-19T08:06:06.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.409+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f3934004a40 con 0x7f394006ac30 2023-12-19T08:06:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.413+0000 7f393cff9700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 0x7f3924062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.413+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f393407c980 con 0x7f394006ac30 2023-12-19T08:06:06.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.413+0000 7f393ffff700 1 -- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 msgr2=0x7f3924062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:06.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.413+0000 7f393ffff700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 0x7f3924062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.417+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f393404c760 con 0x7f394006ac30 2023-12-19T08:06:06.505 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:06.501+0000 7fab325c1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:06:06.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.585+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39400ae6f0 con 0x7f394006ac30 2023-12-19T08:06:06.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.585+0000 7f393cff9700 1 -- 172.21.15.111:0/3900694704 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3934018020 con 0x7f394006ac30 2023-12-19T08:06:06.591 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 msgr2=0x7f3924062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 0x7f3924062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 msgr2=0x7f39400afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f393400cb90 tx=0x7f3934001a30 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 shutdown_connections 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3924060790 0x7f3924062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39400c6d20 0x7f39400ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f394006c7c0 0x7f39400acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.589+0000 7f3947145700 1 --2- 172.21.15.111:0/3900694704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f394006ac30 0x7f39400afd80 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.593+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 >> 172.21.15.111:0/3900694704 conn(0x7f3940067bb0 msgr2=0x7f39400c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:06.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.593+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 shutdown_connections 2023-12-19T08:06:06.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.593+0000 7f3947145700 1 -- 172.21.15.111:0/3900694704 wait complete. 2023-12-19T08:06:06.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:06.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:06.708+0000 7f26bb5cc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:06:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.777+0000 7fec40803700 1 Processor -- start 2023-12-19T08:06:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.777+0000 7fec40803700 1 -- start start 2023-12-19T08:06:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec40803700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec40803700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec3c0cc470 0x7fec3c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec40803700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3c0ca5a0 con 0x7fec3c158e70 2023-12-19T08:06:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec40803700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c06dac0 con 0x7fec3c0cc470 2023-12-19T08:06:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec40803700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c06d220 con 0x7fec3c06a540 2023-12-19T08:06:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fec3c158e70 0x7fec3c0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55748/0 (socket says 172.21.15.111:55748) 2023-12-19T08:06:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3b7fe700 1 -- 172.21.15.111:0/1740166891 learned_addr learned my addr 172.21.15.111:0/1740166891 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 --2- 172.21.15.111:0/1740166891 >> v2:172.21.15.111:3300/0 conn(0x7fec3c0cc470 0x7fec3c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3affd700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3a7fc700 1 -- 172.21.15.111:0/1740166891 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101677168 0 0) 0x7fec3c0ca5a0 con 0x7fec3c158e70 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3a7fc700 1 -- 172.21.15.111:0/1740166891 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec24003570 con 0x7fec3c158e70 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 -- 172.21.15.111:0/1740166891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 msgr2=0x7fec3c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 -- 172.21.15.111:0/1740166891 >> v1:172.21.15.111:6789/0 conn(0x7fec3c158e70 legacy=0x7fec3c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 -- 172.21.15.111:0/1740166891 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c06d500 con 0x7fec3c0cc470 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3bfff700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c0cc8d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fec34009e60 tx=0x7fec34009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4e47ce5b613805 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3a7fc700 1 -- 172.21.15.111:0/1740166891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3400e070 con 0x7fec3c0cc470 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3a7fc700 1 -- 172.21.15.111:0/1740166891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3400bab0 con 0x7fec3c0cc470 2023-12-19T08:06:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.781+0000 7fec3a7fc700 1 -- 172.21.15.111:0/1740166891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3400be50 con 0x7fec3c0cc470 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- 172.21.15.111:0/1740166891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 msgr2=0x7fec3c0cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c0cc8d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fec34009e60 tx=0x7fec34009780 comp rx=0 tx=0).stop 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- 172.21.15.111:0/1740166891 shutdown_connections 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c0cc8d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- 172.21.15.111:0/1740166891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- 172.21.15.111:0/1740166891 >> 172.21.15.111:0/1740166891 conn(0x7fec3c0670e0 msgr2=0x7fec3c0674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- 172.21.15.111:0/1740166891 shutdown_connections 2023-12-19T08:06:06.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- 172.21.15.111:0/1740166891 wait complete. 2023-12-19T08:06:06.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 Processor -- start 2023-12-19T08:06:06.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- start start 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec3c158e70 0x7fec3c16c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c15e880 con 0x7fec3c0cc470 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c15eb30 con 0x7fec3c158e70 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec40803700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c15ee10 con 0x7fec3c06a540 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec3affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec3affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60586/0 (socket says 172.21.15.111:60586) 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec3affd700 1 -- 172.21.15.111:0/124440260 learned_addr learned my addr 172.21.15.111:0/124440260 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec3bfff700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec3c158e70 0x7fec3c16c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.785+0000 7fec3b7fe700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:06.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 msgr2=0x7fec3c0c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c0c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec3c158e70 msgr2=0x7fec3c16c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec3c158e70 0x7fec3c16c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 -- 172.21.15.111:0/124440260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34009a20 con 0x7fec3c0cc470 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec3affd700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fec3000ec80 tx=0x7fec300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3000c850 con 0x7fec3c0cc470 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3000c9b0 con 0x7fec3c0cc470 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c15f0f0 con 0x7fec3c0cc470 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3000cd50 con 0x7fec3c0cc470 2023-12-19T08:06:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c16c920 con 0x7fec3c0cc470 2023-12-19T08:06:06.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.789+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c15f5f0 con 0x7fec3c0cc470 2023-12-19T08:06:06.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.793+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fec30013680 con 0x7fec3c0cc470 2023-12-19T08:06:06.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.793+0000 7fec237fe700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 0x7fec28062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.793+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fec3007d220 con 0x7fec3c0cc470 2023-12-19T08:06:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.793+0000 7fec3b7fe700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 msgr2=0x7fec28062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.793+0000 7fec3b7fe700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 0x7fec28062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:06.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.801+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec3004d0a0 con 0x7fec3c0cc470 2023-12-19T08:06:06.857 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:06:06.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:06:06.899 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:06.893+0000 7fab325c1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:06:06.938 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:06.936+0000 7f26bb5cc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:07.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.997+0000 7fec3b7fe700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 msgr2=0x7fec28062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:07.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:06.997+0000 7fec3b7fe700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 0x7fec28062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:06:07.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.009+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec3c04be40 con 0x7fec3c0cc470 2023-12-19T08:06:07.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.009+0000 7fec237fe700 1 -- 172.21.15.111:0/124440260 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7fec300506c0 con 0x7fec3c0cc470 2023-12-19T08:06:07.015 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:07.015 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 msgr2=0x7fec28062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 0x7fec28062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 msgr2=0x7fec3c15e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fec3000ec80 tx=0x7fec300091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 shutdown_connections 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec280607c0 0x7fec28062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec3c158e70 0x7fec3c16c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec3c0cc470 0x7fec3c15e340 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 --2- 172.21.15.111:0/124440260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec3c06a540 0x7fec3c0c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 >> 172.21.15.111:0/124440260 conn(0x7fec3c0670e0 msgr2=0x7fec3c0674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 shutdown_connections 2023-12-19T08:06:07.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.013+0000 7fec40803700 1 -- 172.21.15.111:0/124440260 wait complete. 2023-12-19T08:06:07.021 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:06:07.035 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.032+0000 7f26bb5cc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:06:07.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:07.118 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.116+0000 7f26bb5cc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:06:07.140 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.133+0000 7fab325c1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:07.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.193+0000 7f03637c5700 1 Processor -- start 2023-12-19T08:06:07.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.193+0000 7f03637c5700 1 -- start start 2023-12-19T08:06:07.199 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.196+0000 7f26bb5cc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:06:07.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f035c140ae0 0x7f035c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c06e150 con 0x7f035c068630 2023-12-19T08:06:07.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c068a70 con 0x7f035c140710 2023-12-19T08:06:07.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f035c06d540 con 0x7f035c140ae0 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0361d62700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f035c068630 0x7f035c0ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55764/0 (socket says 172.21.15.111:55764) 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0361d62700 1 -- 172.21.15.111:0/2559166012 learned_addr learned my addr 172.21.15.111:0/2559166012 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0360d60700 1 --2- 172.21.15.111:0/2559166012 >> v2:172.21.15.143:3300/0 conn(0x7f035c140ae0 0x7f035c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346180694 0 0) 0x7f035c06e150 con 0x7f035c068630 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0348003570 con 0x7f035c068630 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186734676 0 0) 0x7f035c068a70 con 0x7f035c140710 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c06e150 con 0x7f035c140710 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0360d60700 1 -- 172.21.15.111:0/2559166012 >> v1:172.21.15.143:6789/0 conn(0x7f035c140710 legacy=0x7f035c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0360d60700 1 -- 172.21.15.111:0/2559166012 >> v1:172.21.15.111:6789/0 conn(0x7f035c068630 legacy=0x7f035c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0360d60700 1 -- 172.21.15.111:0/2559166012 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c06d820 con 0x7f035c140ae0 2023-12-19T08:06:07.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0360d60700 1 --2- 172.21.15.111:0/2559166012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c140ae0 0x7f035c0680f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f034c0098e0 tx=0x7f034c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b952199694d63449 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:07.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c011630 con 0x7f035c140ae0 2023-12-19T08:06:07.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c011790 con 0x7f035c140ae0 2023-12-19T08:06:07.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f0353fff700 1 -- 172.21.15.111:0/2559166012 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c011b30 con 0x7f035c140ae0 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 -- 172.21.15.111:0/2559166012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c140ae0 msgr2=0x7f035c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.197+0000 7f03637c5700 1 --2- 172.21.15.111:0/2559166012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c140ae0 0x7f035c0680f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f034c0098e0 tx=0x7f034c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- 172.21.15.111:0/2559166012 shutdown_connections 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 --2- 172.21.15.111:0/2559166012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c140ae0 0x7f035c0680f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- 172.21.15.111:0/2559166012 >> 172.21.15.111:0/2559166012 conn(0x7f035c0c8850 msgr2=0x7f035c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:07.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- 172.21.15.111:0/2559166012 shutdown_connections 2023-12-19T08:06:07.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- 172.21.15.111:0/2559166012 wait complete. 2023-12-19T08:06:07.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 Processor -- start 2023-12-19T08:06:07.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- start start 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 0x7f035c0a9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f035c140ae0 0x7f035c0afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c157760 con 0x7f035c140710 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0a9800 con 0x7f035c068630 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f03637c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035c0a9ae0 con 0x7f035c140ae0 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f0360d60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f0360d60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60588/0 (socket says 172.21.15.111:60588) 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f0360d60700 1 -- 172.21.15.111:0/2105767504 learned_addr learned my addr 172.21.15.111:0/2105767504 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:07.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.201+0000 7f0361561700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 0x7f035c0a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0361d62700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f035c140ae0 0x7f035c0afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 -- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f035c140ae0 msgr2=0x7f035c0afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f035c140ae0 0x7f035c0afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 -- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 msgr2=0x7f035c0a9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 0x7f035c0a9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 -- 172.21.15.111:0/2105767504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c009420 con 0x7f035c140710 2023-12-19T08:06:07.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0361561700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 0x7f035c0a9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f0360d60700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f035c06e100 tx=0x7f034c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c01f070 con 0x7f035c140710 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c000c90 con 0x7f035c140710 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c0a9dc0 con 0x7f035c140710 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c025420 con 0x7f035c140710 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c13e450 con 0x7f035c140710 2023-12-19T08:06:07.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c157910 con 0x7f035c140710 2023-12-19T08:06:07.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f034c023070 con 0x7f035c140710 2023-12-19T08:06:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.205+0000 7f03517fa700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 0x7f0340062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.209+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f034c07e3f0 con 0x7f035c140710 2023-12-19T08:06:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.209+0000 7f0361561700 1 -- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 msgr2=0x7f0340062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.209+0000 7f0361561700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 0x7f0340062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:07.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.213+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f035c157910 con 0x7f035c140710 2023-12-19T08:06:07.234 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.229+0000 7fab325c1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:06:07.314 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.309+0000 7fab325c1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:06:07.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.389+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f035c071120 con 0x7f035c140710 2023-12-19T08:06:07.511 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.436+0000 7f26bb5cc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:07.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.389+0000 7f03517fa700 1 -- 172.21.15.111:0/2105767504 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17016 (secure 0 0 0) 0x7f034c055e60 con 0x7f035c140710 2023-12-19T08:06:07.512 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:07.512 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.389+0000 7fab325c1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:06:07.512 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":131,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:05.127681+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2962591998":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/396308477":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3598389140":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/1868422455":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3413002548":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/4220277516":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6833/33796":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6832/33796":"2023-12-20T08:05:25.018350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:07.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 msgr2=0x7f0340062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:07.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 0x7f0340062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 msgr2=0x7f035c13df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f035c06e100 tx=0x7f034c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 shutdown_connections 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0340060790 0x7f0340062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f035c140ae0 0x7f035c0afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f035c140710 0x7f035c13df10 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 --2- 172.21.15.111:0/2105767504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f035c068630 0x7f035c0a9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 >> 172.21.15.111:0/2105767504 conn(0x7f035c0c8850 msgr2=0x7f035c0c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 shutdown_connections 2023-12-19T08:06:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.393+0000 7f03637c5700 1 -- 172.21.15.111:0/2105767504 wait complete. 2023-12-19T08:06:07.516 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T08:06:07.517 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:07.518 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.516+0000 7f26bb5cc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:06:07.613 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.612+0000 7f26bb5cc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:06:07.637 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.633+0000 7fab325c1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:07.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.673+0000 7ff770a2e700 1 Processor -- start 2023-12-19T08:06:07.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.673+0000 7ff770a2e700 1 -- start start 2023-12-19T08:06:07.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff770a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 0x7ff76c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff770a2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff76c140ae0 0x7ff76c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff770a2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c06e150 con 0x7ff76c0c75e0 2023-12-19T08:06:07.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff770a2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c06d540 con 0x7ff76c140ae0 2023-12-19T08:06:07.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff770a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c06d820 con 0x7ff76c140710 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff76c0c75e0 0x7ff76c06a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55780/0 (socket says 172.21.15.111:55780) 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76ad9d700 1 -- 172.21.15.111:0/4128810954 learned_addr learned my addr 172.21.15.111:0/4128810954 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76a59c700 1 --2- 172.21.15.111:0/4128810954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 0x7ff76c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff769d9b700 1 --2- 172.21.15.111:0/4128810954 >> v2:172.21.15.143:3300/0 conn(0x7ff76c140ae0 0x7ff76c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806465390 0 0) 0x7ff76c06e150 con 0x7ff76c0c75e0 2023-12-19T08:06:07.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff754003570 con 0x7ff76c0c75e0 2023-12-19T08:06:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76a59c700 1 -- 172.21.15.111:0/4128810954 >> v2:172.21.15.143:3300/0 conn(0x7ff76c140ae0 msgr2=0x7ff76c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76a59c700 1 --2- 172.21.15.111:0/4128810954 >> v2:172.21.15.143:3300/0 conn(0x7ff76c140ae0 0x7ff76c0680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270353359 0 0) 0x7ff754003570 con 0x7ff76c0c75e0 2023-12-19T08:06:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76a59c700 1 -- 172.21.15.111:0/4128810954 >> v1:172.21.15.111:6789/0 conn(0x7ff76c0c75e0 legacy=0x7ff76c06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.677+0000 7ff76a59c700 1 -- 172.21.15.111:0/4128810954 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c06db00 con 0x7ff76c140710 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff76a59c700 1 --2- 172.21.15.111:0/4128810954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 0x7ff76c0c70a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098e0 tx=0x7ff75c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4e623a06ec52ad3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7640017f0 con 0x7ff76c140710 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c011420 con 0x7ff76c140710 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff76959a700 1 -- 172.21.15.111:0/4128810954 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c0117c0 con 0x7ff76c140710 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 -- 172.21.15.111:0/4128810954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 msgr2=0x7ff76c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 --2- 172.21.15.111:0/4128810954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 0x7ff76c0c70a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098e0 tx=0x7ff75c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 -- 172.21.15.111:0/4128810954 shutdown_connections 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 --2- 172.21.15.111:0/4128810954 >> v2:172.21.15.143:3300/0 conn(0x7ff76c140ae0 0x7ff76c0680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 --2- 172.21.15.111:0/4128810954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c140710 0x7ff76c0c70a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 -- 172.21.15.111:0/4128810954 >> 172.21.15.111:0/4128810954 conn(0x7ff76c0c8850 msgr2=0x7ff76c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 -- 172.21.15.111:0/4128810954 shutdown_connections 2023-12-19T08:06:07.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.681+0000 7ff770a2e700 1 -- 172.21.15.111:0/4128810954 wait complete. 2023-12-19T08:06:07.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 Processor -- start 2023-12-19T08:06:07.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 -- start start 2023-12-19T08:06:07.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c0c75e0 0x7ff76c13f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff76c140ae0 0x7ff76c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0a9500 con 0x7ff76c140710 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c13fc30 con 0x7ff76c140ae0 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff770a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0a88f0 con 0x7ff76c0c75e0 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60592/0 (socket says 172.21.15.111:60592) 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 -- 172.21.15.111:0/1349938170 learned_addr learned my addr 172.21.15.111:0/1349938170 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff76a59c700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c0c75e0 0x7ff76c13f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff76ad9d700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff76c140ae0 0x7ff76c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c0c75e0 msgr2=0x7ff76c13f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c0c75e0 0x7ff76c13f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff76c140ae0 msgr2=0x7ff76c159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff76c140ae0 0x7ff76c159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 -- 172.21.15.111:0/1349938170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c009420 con 0x7ff76c140710 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.685+0000 7ff769d9b700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff76000ec80 tx=0x7ff7600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76000c850 con 0x7ff76c140710 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76000c9b0 con 0x7ff76c140710 2023-12-19T08:06:07.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76000cd50 con 0x7ff76c140710 2023-12-19T08:06:07.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c0a8bd0 con 0x7ff76c140710 2023-12-19T08:06:07.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c0a9190 con 0x7ff76c140710 2023-12-19T08:06:07.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c0a96b0 con 0x7ff76c140710 2023-12-19T08:06:07.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.689+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7ff760013680 con 0x7ff76c140710 2023-12-19T08:06:07.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.693+0000 7ff75a7fc700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 0x7ff74c062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:07.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.693+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7ff76007d210 con 0x7ff76c140710 2023-12-19T08:06:07.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.693+0000 7ff76a59c700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 msgr2=0x7ff74c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:07.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.693+0000 7ff76a59c700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 0x7ff74c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:07.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.701+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff76c0a96b0 con 0x7ff76c140710 2023-12-19T08:06:07.720 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.717+0000 7fab325c1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:06:07.812 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:07.805+0000 7fab325c1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:06:07.816 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.812+0000 7f26bb5cc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:06:07.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.897+0000 7ff76a59c700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 msgr2=0x7ff74c062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:07.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.897+0000 7ff76a59c700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 0x7ff74c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:06:07.903 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:07.900+0000 7f26bb5cc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:06:07.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.905+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff76c071120 con 0x7ff76c140710 2023-12-19T08:06:07.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.905+0000 7ff75a7fc700 1 -- 172.21.15.111:0/1349938170 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7ff76004d090 con 0x7ff76c140710 2023-12-19T08:06:07.910 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:07.910 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:06:07.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 msgr2=0x7ff74c062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:07.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 0x7ff74c062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 msgr2=0x7ff76c1598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:07.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff76000ec80 tx=0x7ff7600091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 shutdown_connections 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff74c060790 0x7ff74c062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff76c140ae0 0x7ff76c159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff76c140710 0x7ff76c1598b0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 --2- 172.21.15.111:0/1349938170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff76c0c75e0 0x7ff76c13f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 >> 172.21.15.111:0/1349938170 conn(0x7ff76c0c8850 msgr2=0x7ff76c0ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 shutdown_connections 2023-12-19T08:06:07.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:07.909+0000 7ff770a2e700 1 -- 172.21.15.111:0/1349938170 wait complete. 2023-12-19T08:06:07.918 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:06:07.937 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:08.025 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:08.021+0000 7fab325c1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:06:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.089+0000 7f0e3406e700 1 Processor -- start 2023-12-19T08:06:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.089+0000 7f0e3406e700 1 -- start start 2023-12-19T08:06:08.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0e2c068800 0x7f0e2c161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c06e150 con 0x7f0e2c140ae0 2023-12-19T08:06:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c06d540 con 0x7f0e2c140710 2023-12-19T08:06:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2c06d820 con 0x7f0e2c068800 2023-12-19T08:06:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e31609700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0e2c140ae0 0x7f0e2c0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55792/0 (socket says 172.21.15.111:55792) 2023-12-19T08:06:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e31609700 1 -- 172.21.15.111:0/3039303808 learned_addr learned my addr 172.21.15.111:0/3039303808 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3260b700 1 --2- 172.21.15.111:0/3039303808 >> v2:172.21.15.111:3300/0 conn(0x7f0e2c068800 0x7f0e2c161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3260b700 1 -- 172.21.15.111:0/3039303808 >> v1:172.21.15.143:6789/0 conn(0x7f0e2c140710 legacy=0x7f0e2c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e30e08700 1 -- 172.21.15.111:0/3039303808 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615852006 0 0) 0x7f0e2c06e150 con 0x7f0e2c140ae0 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3260b700 1 -- 172.21.15.111:0/3039303808 >> v1:172.21.15.111:6789/0 conn(0x7f0e2c140ae0 legacy=0x7f0e2c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3260b700 1 -- 172.21.15.111:0/3039303808 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c06db00 con 0x7f0e2c068800 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3260b700 1 --2- 172.21.15.111:0/3039303808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c161cd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009e60 tx=0x7f0e24009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6189cf5c36f3b0e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e30e08700 1 -- 172.21.15.111:0/3039303808 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2400e070 con 0x7f0e2c068800 2023-12-19T08:06:08.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e30e08700 1 -- 172.21.15.111:0/3039303808 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2400bc30 con 0x7f0e2c068800 2023-12-19T08:06:08.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e30e08700 1 -- 172.21.15.111:0/3039303808 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24012430 con 0x7f0e2c068800 2023-12-19T08:06:08.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- 172.21.15.111:0/3039303808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 msgr2=0x7f0e2c161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3039303808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c161cd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009e60 tx=0x7f0e24009780 comp rx=0 tx=0).stop 2023-12-19T08:06:08.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- 172.21.15.111:0/3039303808 shutdown_connections 2023-12-19T08:06:08.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3039303808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c161cd0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- 172.21.15.111:0/3039303808 >> 172.21.15.111:0/3039303808 conn(0x7f0e2c0c8850 msgr2=0x7f0e2c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:08.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- 172.21.15.111:0/3039303808 shutdown_connections 2023-12-19T08:06:08.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.093+0000 7f0e3406e700 1 -- 172.21.15.111:0/3039303808 wait complete. 2023-12-19T08:06:08.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 Processor -- start 2023-12-19T08:06:08.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 -- start start 2023-12-19T08:06:08.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c140710 0x7f0e2c158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c140ae0 0x7f0e2c166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0c5d20 con 0x7f0e2c068800 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c158d50 con 0x7f0e2c140ae0 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c159030 con 0x7f0e2c140710 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60600/0 (socket says 172.21.15.111:60600) 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 -- 172.21.15.111:0/3824180615 learned_addr learned my addr 172.21.15.111:0/3824180615 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3260b700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c140ae0 0x7f0e2c166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c140710 msgr2=0x7f0e2c158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c140710 0x7f0e2c158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c140ae0 msgr2=0x7f0e2c166bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c140ae0 0x7f0e2c166bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 -- 172.21.15.111:0/3824180615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24009a20 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e31e0a700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800c590 tx=0x7f0e2800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011590 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280116f0 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28011ab0 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.097+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c0c2f30 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.101+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c0ab000 con 0x7f0e2c068800 2023-12-19T08:06:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.101+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c071120 con 0x7f0e2c068800 2023-12-19T08:06:08.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.101+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f0e2801b3b0 con 0x7f0e2c068800 2023-12-19T08:06:08.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.101+0000 7f0e21ffb700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 0x7f0e10062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.101+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f0e2807c6a0 con 0x7f0e2c068800 2023-12-19T08:06:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.105+0000 7f0e31609700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 msgr2=0x7f0e10062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.105+0000 7f0e31609700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 0x7f0e10062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:08.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.109+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e2804c480 con 0x7f0e2c068800 2023-12-19T08:06:08.118 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:08.113+0000 7fab325c1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:06:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.309+0000 7f0e31609700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 msgr2=0x7f0e10062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.309+0000 7f0e31609700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 0x7f0e10062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:06:08.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.317+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e2c0642a0 con 0x7f0e2c068800 2023-12-19T08:06:08.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.317+0000 7f0e21ffb700 1 -- 172.21.15.111:0/3824180615 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4608 (secure 0 0 0) 0x7f0e28017090 con 0x7f0e2c068800 2023-12-19T08:06:08.322 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:08.322 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:05:10.166367+0000","modified":"2023-12-19T08:05:13.786643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"d","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/1501582072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1501582072},{"type":"v1","addr":"172.21.15.143:6837","nonce":1501582072}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2023-12-19T08:06:08.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 msgr2=0x7f0e10062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:08.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 0x7f0e10062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 msgr2=0x7f0e2c158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800c590 tx=0x7f0e2800c950 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 shutdown_connections 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0e10060790 0x7f0e10062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e2c140ae0 0x7f0e2c166bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e2c140710 0x7f0e2c158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 --2- 172.21.15.111:0/3824180615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e2c068800 0x7f0e2c158120 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 >> 172.21.15.111:0/3824180615 conn(0x7f0e2c0c8850 msgr2=0x7f0e2c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 shutdown_connections 2023-12-19T08:06:08.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.321+0000 7f0e3406e700 1 -- 172.21.15.111:0/3824180615 wait complete. 2023-12-19T08:06:08.327 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 56 2023-12-19T08:06:08.343 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.497+0000 7fd0832cd700 1 Processor -- start 2023-12-19T08:06:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.497+0000 7fd0832cd700 1 -- start start 2023-12-19T08:06:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd0832cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd07c067c90 0x7fd07c068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd0832cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0c6630 con 0x7fd07c0c5900 2023-12-19T08:06:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd0832cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0c6910 con 0x7fd07c14fba0 2023-12-19T08:06:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd0832cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c0c6bf0 con 0x7fd07c067c90 2023-12-19T08:06:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd08186a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd07c0c5900 0x7fd07c0c5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55800/0 (socket says 172.21.15.111:55800) 2023-12-19T08:06:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd08186a700 1 -- 172.21.15.111:0/499447699 learned_addr learned my addr 172.21.15.111:0/499447699 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd081069700 1 --2- 172.21.15.111:0/499447699 >> v2:172.21.15.143:3300/0 conn(0x7fd07c067c90 0x7fd07c068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685801306 0 0) 0x7fd07c0c6630 con 0x7fd07c0c5900 2023-12-19T08:06:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068003570 con 0x7fd07c0c5900 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd081069700 1 -- 172.21.15.111:0/499447699 >> v1:172.21.15.143:6789/0 conn(0x7fd07c14fba0 legacy=0x7fd07c150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574647269 0 0) 0x7fd068003570 con 0x7fd07c0c5900 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd081069700 1 -- 172.21.15.111:0/499447699 >> v1:172.21.15.111:6789/0 conn(0x7fd07c0c5900 legacy=0x7fd07c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.501+0000 7fd081069700 1 -- 172.21.15.111:0/499447699 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0c6ed0 con 0x7fd07c067c90 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd081069700 1 --2- 172.21.15.111:0/499447699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c067c90 0x7fd07c068060 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd0780098e0 tx=0x7fd0780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aba5f3ee89ce4bb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740017f0 con 0x7fd07c067c90 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd078011420 con 0x7fd07c067c90 2023-12-19T08:06:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd073fff700 1 -- 172.21.15.111:0/499447699 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780117a0 con 0x7fd07c067c90 2023-12-19T08:06:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- 172.21.15.111:0/499447699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c067c90 msgr2=0x7fd07c068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 --2- 172.21.15.111:0/499447699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c067c90 0x7fd07c068060 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd0780098e0 tx=0x7fd0780091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- 172.21.15.111:0/499447699 shutdown_connections 2023-12-19T08:06:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 --2- 172.21.15.111:0/499447699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c067c90 0x7fd07c068060 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- 172.21.15.111:0/499447699 >> 172.21.15.111:0/499447699 conn(0x7fd07c0cd6d0 msgr2=0x7fd07c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- 172.21.15.111:0/499447699 shutdown_connections 2023-12-19T08:06:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- 172.21.15.111:0/499447699 wait complete. 2023-12-19T08:06:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 Processor -- start 2023-12-19T08:06:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.505+0000 7fd0832cd700 1 -- start start 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 0x7fd07c151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c14fba0 0x7fd07c151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c152ef0 con 0x7fd07c067c90 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0c7ba0 con 0x7fd07c14fba0 2023-12-19T08:06:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0832cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1522e0 con 0x7fd07c0c5900 2023-12-19T08:06:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60616/0 (socket says 172.21.15.111:60616) 2023-12-19T08:06:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 -- 172.21.15.111:0/6899541 learned_addr learned my addr 172.21.15.111:0/6899541 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd080868700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 0x7fd07c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd08186a700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c14fba0 0x7fd07c151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 -- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 msgr2=0x7fd07c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 0x7fd07c151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 -- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c14fba0 msgr2=0x7fd07c151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c14fba0 0x7fd07c151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 -- 172.21.15.111:0/6899541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009420 con 0x7fd07c067c90 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd080868700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 0x7fd07c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:08.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd081069700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd078009fd0 tx=0x7fd078016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07801cb80 con 0x7fd07c067c90 2023-12-19T08:06:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.509+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07801cd00 con 0x7fd07c067c90 2023-12-19T08:06:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.513+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c1525c0 con 0x7fd07c067c90 2023-12-19T08:06:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.513+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07801a6a0 con 0x7fd07c067c90 2023-12-19T08:06:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.513+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c152b80 con 0x7fd07c067c90 2023-12-19T08:06:08.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.513+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c071120 con 0x7fd07c067c90 2023-12-19T08:06:08.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.513+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 56) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fd07801aad0 con 0x7fd07c067c90 2023-12-19T08:06:08.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.517+0000 7fd0717fa700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 0x7fd060062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.517+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fd07807e990 con 0x7fd07c067c90 2023-12-19T08:06:08.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.517+0000 7fd080868700 1 -- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 msgr2=0x7fd060062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:08.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.517+0000 7fd080868700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 0x7fd060062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:08.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.521+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd07804e770 con 0x7fd07c067c90 2023-12-19T08:06:08.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.689+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd07c0642a0 con 0x7fd07c067c90 2023-12-19T08:06:08.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0717fa700 1 -- 172.21.15.111:0/6899541 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17016 (secure 0 0 0) 0x7fd078051d90 con 0x7fd07c067c90 2023-12-19T08:06:08.697 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:08.698 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":131,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:05.127681+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:05:08.129042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-12-19T08:05:09.170914+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":131,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2962591998":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/396308477":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3598389140":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/1868422455":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3413002548":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/4220277516":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6833/33796":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6832/33796":"2023-12-20T08:05:25.018350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:08.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 msgr2=0x7fd060062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:08.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 0x7fd060062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 msgr2=0x7fd07c0c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd078009fd0 tx=0x7fd078016000 comp rx=0 tx=0).stop 2023-12-19T08:06:08.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.693+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 shutdown_connections 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd060060790 0x7fd060062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd07c14fba0 0x7fd07c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd07c0c5900 0x7fd07c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 --2- 172.21.15.111:0/6899541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd07c067c90 0x7fd07c0c9bf0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 >> 172.21.15.111:0/6899541 conn(0x7fd07c0cd6d0 msgr2=0x7fd07c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 shutdown_connections 2023-12-19T08:06:08.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.697+0000 7fd0832cd700 1 -- 172.21.15.111:0/6899541 wait complete. 2023-12-19T08:06:08.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.708+0000 7f26bb5cc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:06:08.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T08:06:08.822 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26829df700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.822 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26829df700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.822 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26829df700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.822 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26829df700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26829df700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26869e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26869e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26869e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26869e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.823 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:08.820+0000 7f26869e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:08.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 Processor -- start 2023-12-19T08:06:08.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 -- start start 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72b4158cc0 0x7f72b406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f72b40b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b406b0d0 con 0x7f72b406dc00 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40b4790 con 0x7f72b4158cc0 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b9cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b406d710 con 0x7f72b40b3870 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b2ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f72b406dc00 0x7f72b406b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55806/0 (socket says 172.21.15.111:55806) 2023-12-19T08:06:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b2ffd700 1 -- 172.21.15.111:0/1808147250 learned_addr learned my addr 172.21.15.111:0/1808147250 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1808147250 >> v2:172.21.15.111:3300/0 conn(0x7f72b4158cc0 0x7f72b406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f72b40b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671964503 0 0) 0x7f72b406b0d0 con 0x7f72b406dc00 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f729c003570 con 0x7f72b406dc00 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639190819 0 0) 0x7f729c003570 con 0x7f72b406dc00 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 msgr2=0x7f72b40b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f72b40b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 >> v2:172.21.15.111:3300/0 conn(0x7f72b4158cc0 msgr2=0x7f72b406d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:08.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 --2- 172.21.15.111:0/1808147250 >> v2:172.21.15.111:3300/0 conn(0x7f72b4158cc0 0x7f72b406d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b406a7e0 con 0x7f72b406dc00 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f72a8001470 con 0x7f72b406dc00 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.873+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 >> v1:172.21.15.111:6789/0 conn(0x7f72b406dc00 legacy=0x7f72b406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c003a60 0x7f729c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f729c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f729c004d50 0x7f729c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40b4790 con 0x7f729c004d50 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b406d710 con 0x7f729c003a60 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c00ba60 con 0x7f72b40b3870 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f729c004d50 0x7f729c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b2ffd700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f729c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 -- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 msgr2=0x7f729c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b40b3870 0x7f729c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c003a60 0x7f729c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 -- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c003a60 msgr2=0x7f729c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:08.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c003a60 0x7f729c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 -- 172.21.15.111:0/1808147250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c003570 con 0x7f729c004d50 2023-12-19T08:06:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c003a60 0x7f729c003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f729c004d50 0x7f729c0051c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00e210 tx=0x7f72ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc87a50368aa52ae server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00ea60 con 0x7f729c004d50 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac00f040 con 0x7f729c004d50 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b27fc700 1 -- 172.21.15.111:0/1808147250 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac00ee10 con 0x7f729c004d50 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1808147250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f729c004d50 msgr2=0x7f729c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 --2- 172.21.15.111:0/1808147250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f729c004d50 0x7f729c0051c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00e210 tx=0x7f72ac00b040 comp rx=0 tx=0).stop 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b3fff700 1 -- 172.21.15.111:0/1808147250 reap_dead start 2023-12-19T08:06:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1808147250 shutdown_connections 2023-12-19T08:06:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1808147250 >> 172.21.15.111:0/1808147250 conn(0x7f72b4067bb0 msgr2=0x7f72b40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1808147250 shutdown_connections 2023-12-19T08:06:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.877+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1808147250 wait complete. 2023-12-19T08:06:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 Processor -- start 2023-12-19T08:06:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- start start 2023-12-19T08:06:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b4155bb0 0x7f72b4156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 0x7f72b416ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4156650 con 0x7f72b414e040 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b416cf80 con 0x7f72b4155bb0 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41529d0 con 0x7f72b416c5d0 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60640/0 (socket says 172.21.15.111:60640) 2023-12-19T08:06:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 -- 172.21.15.111:0/1886953866 learned_addr learned my addr 172.21.15.111:0/1886953866 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b2ffd700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b4155bb0 0x7f72b4156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 0x7f72b416ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 -- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 msgr2=0x7f72b416ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 0x7f72b416ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 -- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b4155bb0 msgr2=0x7f72b4156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b4155bb0 0x7f72b4156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 -- 172.21.15.111:0/1886953866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b41520a0 con 0x7f72b414e040 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b3fff700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 0x7f72b416ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b37fe700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009270 tx=0x7f72a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a400e870 con 0x7f72b414e040 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a400e9d0 con 0x7f72b414e040 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b4152380 con 0x7f72b414e040 2023-12-19T08:06:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a400ed70 con 0x7f72b414e040 2023-12-19T08:06:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.881+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b416d2f0 con 0x7f72b414e040 2023-12-19T08:06:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.885+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4152800 con 0x7f72b414e040 2023-12-19T08:06:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.889+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 57) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f72a4025420 con 0x7f72b414e040 2023-12-19T08:06:08.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.889+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 7261+0+0 (secure 0 0 0) 0x7f72a407ccf0 con 0x7f72b414e040 2023-12-19T08:06:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:08.893+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72a404d020 con 0x7f72b414e040 2023-12-19T08:06:08.924 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:08.917+0000 7fab325c1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:06:09.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.097+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f72b4071120 con 0x7f72b414e040 2023-12-19T08:06:09.102 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T08:06:09.097+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T08:06:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f729b7fe700 1 -- 172.21.15.111:0/1886953866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7f72a4050640 con 0x7f72b414e040 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 msgr2=0x7f72b40a9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009270 tx=0x7f72a4008040 comp rx=0 tx=0).stop 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 shutdown_connections 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72b4155bb0 0x7f72b4156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72b416c5d0 0x7f72b416ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 --2- 172.21.15.111:0/1886953866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72b414e040 0x7f72b40a9e10 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 >> 172.21.15.111:0/1886953866 conn(0x7f72b4067bb0 msgr2=0x7f72b415c170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 shutdown_connections 2023-12-19T08:06:09.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.109+0000 7f72b9cf0700 1 -- 172.21.15.111:0/1886953866 wait complete. 2023-12-19T08:06:09.117 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T08:06:09.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T08:06:09.180 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:06:09.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.289+0000 7f6bc5e0c700 1 Processor -- start 2023-12-19T08:06:09.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.289+0000 7f6bc5e0c700 1 -- start start 2023-12-19T08:06:09.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.289+0000 7f6bc5e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bc0159770 0x7f6bc006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc006c850 con 0x7f6bc006aeb0 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc006b470 con 0x7f6bc0159770 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc006b750 con 0x7f6bc0158c90 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bc0159770 0x7f6bc006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bc0159770 0x7f6bc006cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60654/0 (socket says 172.21.15.111:60654) 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 -- 172.21.15.111:0/695083412 learned_addr learned my addr 172.21.15.111:0/695083412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc4e0a700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 -- 172.21.15.111:0/695083412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 msgr2=0x7f6bc00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:09.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc00caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 -- 172.21.15.111:0/695083412 >> v1:172.21.15.111:6789/0 conn(0x7f6bc006aeb0 legacy=0x7f6bc0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:09.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 -- 172.21.15.111:0/695083412 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc006ba30 con 0x7f6bc0159770 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc560b700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc0159770 0x7f6bc006cce0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009f20 tx=0x7f6bb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1286c40454b0d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bb77fe700 1 -- 172.21.15.111:0/695083412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb800e070 con 0x7f6bc0159770 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bb77fe700 1 -- 172.21.15.111:0/695083412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb800ba80 con 0x7f6bc0159770 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bb77fe700 1 -- 172.21.15.111:0/695083412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb80092e0 con 0x7f6bc0159770 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/695083412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc0159770 msgr2=0x7f6bc006cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc0159770 0x7f6bc006cce0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8009f20 tx=0x7f6bb8009840 comp rx=0 tx=0).stop 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/695083412 shutdown_connections 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc0159770 0x7f6bc006cce0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/695083412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc00caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/695083412 >> 172.21.15.111:0/695083412 conn(0x7f6bc0067bb0 msgr2=0x7f6bc00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:09.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/695083412 shutdown_connections 2023-12-19T08:06:09.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.293+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/695083412 wait complete. 2023-12-19T08:06:09.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.297+0000 7f6bc5e0c700 1 Processor -- start 2023-12-19T08:06:09.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.297+0000 7f6bc5e0c700 1 -- start start 2023-12-19T08:06:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc01535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 0x7f6bc00c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0153b20 con 0x7f6bc006aeb0 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00c87b0 con 0x7f6bc0159770 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00c8a90 con 0x7f6bc0158c90 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60666/0 (socket says 172.21.15.111:60666) 2023-12-19T08:06:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 -- 172.21.15.111:0/1977037453 learned_addr learned my addr 172.21.15.111:0/1977037453 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bb7fff700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc01535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc560b700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 0x7f6bc00c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 -- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 msgr2=0x7f6bc01535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc01535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 -- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 msgr2=0x7f6bc00c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 0x7f6bc00c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 -- 172.21.15.111:0/1977037453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8009ae0 con 0x7f6bc006aeb0 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc560b700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 0x7f6bc00c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:09.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc4e0a700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00d890 tx=0x7f6bbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc00b600 con 0x7f6bc006aeb0 2023-12-19T08:06:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc00b760 con 0x7f6bc006aeb0 2023-12-19T08:06:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc00c8d70 con 0x7f6bc006aeb0 2023-12-19T08:06:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc00bae0 con 0x7f6bc006aeb0 2023-12-19T08:06:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.301+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc00c9330 con 0x7f6bc006aeb0 2023-12-19T08:06:09.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.305+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc004be40 con 0x7f6bc006aeb0 2023-12-19T08:06:09.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.305+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 57) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f6bbc01e460 con 0x7f6bc006aeb0 2023-12-19T08:06:09.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.309+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 7347+0+0 (secure 0 0 0) 0x7f6bbc07d140 con 0x7f6bc006aeb0 2023-12-19T08:06:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.313+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6bbc04d390 con 0x7f6bc006aeb0 2023-12-19T08:06:09.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.521+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6bc006bd10 con 0x7f6bc006aeb0 2023-12-19T08:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.733+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 58) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f6bbc04ce40 con 0x7f6bc006aeb0 2023-12-19T08:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.733+0000 7f6bb4ff9700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 0x7f6bac068d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:09.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.737+0000 7f6bb7fff700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 0x7f6bac068d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:09.741+0000 7f6bb7fff700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 0x7f6bac068d80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0062c50 tx=0x7f6bb0009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:10.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.161+0000 7f6bb4ff9700 1 -- 172.21.15.111:0/1977037453 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6bbc068160 con 0x7f6bc006aeb0 2023-12-19T08:06:10.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 msgr2=0x7f6bac068d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 0x7f6bac068d80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0062c50 tx=0x7f6bb0009040 comp rx=0 tx=0).stop 2023-12-19T08:06:10.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 msgr2=0x7f6bc00c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00d890 tx=0x7f6bbc008040 comp rx=0 tx=0).stop 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 shutdown_connections 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bac0669a0 0x7f6bac068d80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bc0159770 0x7f6bc00c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bc0158c90 0x7f6bc01535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 --2- 172.21.15.111:0/1977037453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bc006aeb0 0x7f6bc00c8140 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 >> 172.21.15.111:0/1977037453 conn(0x7f6bc0067bb0 msgr2=0x7f6bc00cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 shutdown_connections 2023-12-19T08:06:10.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.165+0000 7f6bc5e0c700 1 -- 172.21.15.111:0/1977037453 wait complete. 2023-12-19T08:06:10.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T08:06:10.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.365+0000 7f9c58e0c700 1 Processor -- start 2023-12-19T08:06:10.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.365+0000 7f9c58e0c700 1 -- start start 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c58e0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c540af190 0x7f9c5406af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c58e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c540af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c58e0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c540cb3f0 con 0x7f9c540aedc0 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c58e0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c540cc140 con 0x7f9c540af190 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c58e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540af610 con 0x7f9c540cac50 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c537fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9c540aedc0 0x7f9c5406d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55830/0 (socket says 172.21.15.111:55830) 2023-12-19T08:06:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c537fe700 1 -- 172.21.15.111:0/3347303670 learned_addr learned my addr 172.21.15.111:0/3347303670 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:10.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153541038 0 0) 0x7f9c540cb3f0 con 0x7f9c540aedc0 2023-12-19T08:06:10.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c53fff700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c540af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c3c003570 con 0x7f9c540aedc0 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> v2:172.21.15.143:3300/0 conn(0x7f9c540af190 0x7f9c5406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354352885 0 0) 0x7f9c3c003570 con 0x7f9c540aedc0 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 msgr2=0x7f9c540af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c540af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 >> v2:172.21.15.143:3300/0 conn(0x7f9c540af190 msgr2=0x7f9c5406af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 --2- 172.21.15.111:0/3347303670 >> v2:172.21.15.143:3300/0 conn(0x7f9c540af190 0x7f9c5406af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5406d220 con 0x7f9c540aedc0 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f9c44001470 con 0x7f9c540aedc0 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> v2:172.21.15.143:3300/0 conn(0x7f9c540af190 0x7f9c5406af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 >> v1:172.21.15.111:6789/0 conn(0x7f9c540aedc0 legacy=0x7f9c5406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c3c003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3c004d50 0x7f9c3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540cc140 con 0x7f9c3c004020 2023-12-19T08:06:10.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540af610 con 0x7f9c3c004d50 2023-12-19T08:06:10.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c00bb90 con 0x7f9c540cac50 2023-12-19T08:06:10.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c537fe700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c3c003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.369+0000 7f9c53fff700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3c004d50 0x7f9c3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 -- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 msgr2=0x7f9c3c003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c3c003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 -- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3c004d50 msgr2=0x7f9c3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3c004d50 0x7f9c3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 -- 172.21.15.111:0/3347303670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c003570 con 0x7f9c3c004020 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c537fe700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c540cac50 0x7f9c3c003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c3c004810 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9c48006fd0 tx=0x7f9c48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b849e58015cfa139 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800c070 con 0x7f9c3c004020 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4800e5e0 con 0x7f9c3c004020 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c527fc700 1 -- 172.21.15.111:0/3347303670 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48009510 con 0x7f9c3c004020 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 -- 172.21.15.111:0/3347303670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 msgr2=0x7f9c3c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/3347303670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c3c004810 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9c48006fd0 tx=0x7f9c48008040 comp rx=0 tx=0).stop 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c53fff700 1 -- 172.21.15.111:0/3347303670 reap_dead start 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 -- 172.21.15.111:0/3347303670 shutdown_connections 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 -- 172.21.15.111:0/3347303670 >> 172.21.15.111:0/3347303670 conn(0x7f9c540cc760 msgr2=0x7f9c540ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 -- 172.21.15.111:0/3347303670 shutdown_connections 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.373+0000 7f9c58e0c700 1 -- 172.21.15.111:0/3347303670 wait complete. 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 Processor -- start 2023-12-19T08:06:10.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 -- start start 2023-12-19T08:06:10.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c54158710 0x7f9c540c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540c5d90 0x7f9c540c3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540c6200 con 0x7f9c3c004020 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540c4580 con 0x7f9c540c5d90 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c58e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540c4860 con 0x7f9c54158710 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60680/0 (socket says 172.21.15.111:60680) 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.377+0000 7f9c537fe700 1 -- 172.21.15.111:0/4220411011 learned_addr learned my addr 172.21.15.111:0/4220411011 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c54158710 0x7f9c540c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c53fff700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540c5d90 0x7f9c540c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 -- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c54158710 msgr2=0x7f9c540c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c54158710 0x7f9c540c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 -- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540c5d90 msgr2=0x7f9c540c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540c5d90 0x7f9c540c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 -- 172.21.15.111:0/4220411011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c540c4b40 con 0x7f9c3c004020 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c537fe700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9c440014e0 tx=0x7f9c44008bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400fbf0 con 0x7f9c3c004020 2023-12-19T08:06:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4400fd50 con 0x7f9c3c004020 2023-12-19T08:06:10.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c540c4e20 con 0x7f9c3c004020 2023-12-19T08:06:10.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44016690 con 0x7f9c3c004020 2023-12-19T08:06:10.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c540c3f90 con 0x7f9c3c004020 2023-12-19T08:06:10.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.381+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5404be40 con 0x7f9c3c004020 2023-12-19T08:06:10.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.385+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 58) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f9c44016a40 con 0x7f9c3c004020 2023-12-19T08:06:10.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.385+0000 7f9c3b7fe700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 0x7f9c40062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:10.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.385+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 7347+0+0 (secure 0 0 0) 0x7f9c4407d2a0 con 0x7f9c3c004020 2023-12-19T08:06:10.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.385+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 0x7f9c40062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:10.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.389+0000 7f9c52ffd700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 0x7f9c40062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c5408b350 tx=0x7f9c4801a000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:10.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.393+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c4404d4d0 con 0x7f9c3c004020 2023-12-19T08:06:10.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:10.561+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c540c6d00 con 0x7f9c3c004020 2023-12-19T08:06:11.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.169+0000 7f9c3b7fe700 1 -- 172.21.15.111:0/4220411011 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9c44050af0 con 0x7f9c3c004020 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 msgr2=0x7f9c40062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 0x7f9c40062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c5408b350 tx=0x7f9c4801a000 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 msgr2=0x7f9c541581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9c440014e0 tx=0x7f9c44008bf0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 shutdown_connections 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c400603a0 0x7f9c40062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540c5d90 0x7f9c540c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c54158710 0x7f9c540c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 --2- 172.21.15.111:0/4220411011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c3c004020 0x7f9c541581d0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 >> 172.21.15.111:0/4220411011 conn(0x7f9c540cc760 msgr2=0x7f9c540b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:11.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 shutdown_connections 2023-12-19T08:06:11.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.173+0000 7f9c58e0c700 1 -- 172.21.15.111:0/4220411011 wait complete. 2023-12-19T08:06:11.182 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T08:06:11.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T08:06:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 Processor -- start 2023-12-19T08:06:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 -- start start 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdcf015cf90 0x7fdcf006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcf006dde0 0x7fdcf00af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf006b550 con 0x7fdcf015d360 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf006af40 con 0x7fdcf015cf90 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.353+0000 7fdceffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf00af5a0 con 0x7fdcf006dde0 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdce7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdcf015d360 0x7fdcf00ae950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55840/0 (socket says 172.21.15.111:55840) 2023-12-19T08:06:11.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdce7fff700 1 -- 172.21.15.111:0/2846858432 learned_addr learned my addr 172.21.15.111:0/2846858432 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 --2- 172.21.15.111:0/2846858432 >> v2:172.21.15.111:3300/0 conn(0x7fdcf015cf90 0x7fdcf006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcef7fe700 1 --2- 172.21.15.111:0/2846858432 >> v2:172.21.15.143:3300/0 conn(0x7fdcf006dde0 0x7fdcf00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcee7fc700 1 -- 172.21.15.111:0/2846858432 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13576990 0 0) 0x7fdcf006b550 con 0x7fdcf015d360 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcee7fc700 1 -- 172.21.15.111:0/2846858432 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcdc003570 con 0x7fdcf015d360 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 -- 172.21.15.111:0/2846858432 >> v2:172.21.15.143:3300/0 conn(0x7fdcf006dde0 msgr2=0x7fdcf00af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 --2- 172.21.15.111:0/2846858432 >> v2:172.21.15.143:3300/0 conn(0x7fdcf006dde0 0x7fdcf00af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 -- 172.21.15.111:0/2846858432 >> v1:172.21.15.111:6789/0 conn(0x7fdcf015d360 legacy=0x7fdcf00ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 -- 172.21.15.111:0/2846858432 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00ae0c0 con 0x7fdcf015cf90 2023-12-19T08:06:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceeffd700 1 --2- 172.21.15.111:0/2846858432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf015cf90 0x7fdcf006b9e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fdcf006a000 tx=0x7fdce00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b555cd7953b465 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcee7fc700 1 -- 172.21.15.111:0/2846858432 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0011630 con 0x7fdcf015cf90 2023-12-19T08:06:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcee7fc700 1 -- 172.21.15.111:0/2846858432 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce00117b0 con 0x7fdcf015cf90 2023-12-19T08:06:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdcee7fc700 1 -- 172.21.15.111:0/2846858432 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0011b70 con 0x7fdcf015cf90 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 -- 172.21.15.111:0/2846858432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf015cf90 msgr2=0x7fdcf006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 --2- 172.21.15.111:0/2846858432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf015cf90 0x7fdcf006b9e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fdcf006a000 tx=0x7fdce00091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 -- 172.21.15.111:0/2846858432 shutdown_connections 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 --2- 172.21.15.111:0/2846858432 >> v2:172.21.15.143:3300/0 conn(0x7fdcf006dde0 0x7fdcf00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 --2- 172.21.15.111:0/2846858432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf015cf90 0x7fdcf006b9e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 -- 172.21.15.111:0/2846858432 >> 172.21.15.111:0/2846858432 conn(0x7fdcf0067bb0 msgr2=0x7fdcf00c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 -- 172.21.15.111:0/2846858432 shutdown_connections 2023-12-19T08:06:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.357+0000 7fdceffff700 1 -- 172.21.15.111:0/2846858432 wait complete. 2023-12-19T08:06:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.361+0000 7fdceffff700 1 Processor -- start 2023-12-19T08:06:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.361+0000 7fdceffff700 1 -- start start 2023-12-19T08:06:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcf015cf90 0x7fdcf0159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 0x7fdcf015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf00b0ec0 con 0x7fdcf006dde0 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf0159b10 con 0x7fdcf015d360 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf00b0620 con 0x7fdcf015cf90 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60698/0 (socket says 172.21.15.111:60698) 2023-12-19T08:06:11.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 -- 172.21.15.111:0/276319397 learned_addr learned my addr 172.21.15.111:0/276319397 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:11.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdce7fff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcf015cf90 0x7fdcf0159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdcef7fe700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 0x7fdcf015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 -- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcf015cf90 msgr2=0x7fdcf0159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcf015cf90 0x7fdcf0159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 -- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 msgr2=0x7fdcf015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 0x7fdcf015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 -- 172.21.15.111:0/276319397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce0009420 con 0x7fdcf006dde0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdcef7fe700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 0x7fdcf015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceeffd700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00699a0 tx=0x7fdce0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce001f070 con 0x7fdcf006dde0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0011920 con 0x7fdcf006dde0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0023410 con 0x7fdcf006dde0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf00b0900 con 0x7fdcf006dde0 2023-12-19T08:06:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.365+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf0160600 con 0x7fdcf006dde0 2023-12-19T08:06:11.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.369+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf004be40 con 0x7fdcf006dde0 2023-12-19T08:06:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.369+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdce002c3b0 con 0x7fdcf006dde0 2023-12-19T08:06:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.369+0000 7fdce6ffd700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 0x7fdcd0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.369+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6909+0+0 (secure 0 0 0) 0x7fdce007e0b0 con 0x7fdcf006dde0 2023-12-19T08:06:11.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.369+0000 7fdce7fff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 0x7fdcd0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.373+0000 7fdce7fff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 0x7fdcd0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0157580 tx=0x7fdcd8009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:11.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.377+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdce004e160 con 0x7fdcf006dde0 2023-12-19T08:06:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:11.545+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdcf00c8f30 con 0x7fdcf006dde0 2023-12-19T08:06:12.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.189+0000 7fdce6ffd700 1 -- 172.21.15.111:0/276319397 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdce0051700 con 0x7fdcf006dde0 2023-12-19T08:06:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 msgr2=0x7fdcd0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 0x7fdcd0062b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0157580 tx=0x7fdcd8009040 comp rx=0 tx=0).stop 2023-12-19T08:06:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 msgr2=0x7fdcf016d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00699a0 tx=0x7fdce0017000 comp rx=0 tx=0).stop 2023-12-19T08:06:12.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 shutdown_connections 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdcd0060670 0x7fdcd0062b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf015d360 0x7fdcf015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcf015cf90 0x7fdcf0159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 --2- 172.21.15.111:0/276319397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcf006dde0 0x7fdcf016d2a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 >> 172.21.15.111:0/276319397 conn(0x7fdcf0067bb0 msgr2=0x7fdcf00c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 shutdown_connections 2023-12-19T08:06:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.193+0000 7fdceffff700 1 -- 172.21.15.111:0/276319397 wait complete. 2023-12-19T08:06:12.202 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T08:06:12.222 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T08:06:12.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.393+0000 7fa4c46f0700 1 Processor -- start 2023-12-19T08:06:12.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.393+0000 7fa4c46f0700 1 -- start start 2023-12-19T08:06:12.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c46f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4bc158e20 0x7fa4bc0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c46f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c46f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc0cbdf0 con 0x7fa4bc0cb970 2023-12-19T08:06:12.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c46f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4bc0cc0d0 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c46f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4bc0cc3b0 con 0x7fa4bc158e20 2023-12-19T08:06:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52308/0 (socket says 172.21.15.111:52308) 2023-12-19T08:06:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c1c8b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa4bc0cb970 0x7fa4bc06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48524/0 (socket says 172.21.15.111:48524) 2023-12-19T08:06:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 -- 172.21.15.111:0/2720355159 learned_addr learned my addr 172.21.15.111:0/2720355159 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c248c700 1 --2- 172.21.15.111:0/2720355159 >> v2:172.21.15.143:3300/0 conn(0x7fa4bc158e20 0x7fa4bc0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 -- 172.21.15.111:0/2720355159 >> v2:172.21.15.143:3300/0 conn(0x7fa4bc158e20 msgr2=0x7fa4bc0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 --2- 172.21.15.111:0/2720355159 >> v2:172.21.15.143:3300/0 conn(0x7fa4bc158e20 0x7fa4bc0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 -- 172.21.15.111:0/2720355159 >> v1:172.21.15.111:6789/0 conn(0x7fa4bc0cb970 legacy=0x7fa4bc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 -- 172.21.15.111:0/2720355159 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0cc690 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c148a700 1 -- 172.21.15.111:0/2720355159 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103621822 0 0) 0x7fa4bc0cbdf0 con 0x7fa4bc0cb970 2023-12-19T08:06:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c2c8d700 1 --2- 172.21.15.111:0/2720355159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009f20 tx=0x7fa4b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170fdfadca0d0b9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c148a700 1 -- 172.21.15.111:0/2720355159 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b400e070 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c148a700 1 -- 172.21.15.111:0/2720355159 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b400b990 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.397+0000 7fa4c148a700 1 -- 172.21.15.111:0/2720355159 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40092e0 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- 172.21.15.111:0/2720355159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc0ca2c0 msgr2=0x7fa4bc0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/2720355159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009f20 tx=0x7fa4b4009840 comp rx=0 tx=0).stop 2023-12-19T08:06:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- 172.21.15.111:0/2720355159 shutdown_connections 2023-12-19T08:06:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/2720355159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0ceb00 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/2720355159 >> v2:172.21.15.143:3300/0 conn(0x7fa4bc158e20 0x7fa4bc0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- 172.21.15.111:0/2720355159 >> 172.21.15.111:0/2720355159 conn(0x7fa4bc067bb0 msgr2=0x7fa4bc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:12.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- 172.21.15.111:0/2720355159 shutdown_connections 2023-12-19T08:06:12.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- 172.21.15.111:0/2720355159 wait complete. 2023-12-19T08:06:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 Processor -- start 2023-12-19T08:06:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.401+0000 7fa4c46f0700 1 -- start start 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4bc14b970 0x7fa4bc0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0b41a0 con 0x7fa4bc158e20 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0b4b70 con 0x7fa4bc14b970 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0b3590 con 0x7fa4bc0ca2c0 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52314/0 (socket says 172.21.15.111:52314) 2023-12-19T08:06:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 -- 172.21.15.111:0/1766608816 learned_addr learned my addr 172.21.15.111:0/1766608816 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c248c700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c2c8d700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4bc14b970 0x7fa4bc0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 -- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4bc0ca2c0 msgr2=0x7fa4bc0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 -- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4bc14b970 msgr2=0x7fa4bc0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4bc14b970 0x7fa4bc0c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 -- 172.21.15.111:0/1766608816 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4009ae0 con 0x7fa4bc158e20 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c1c8b700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00c810 tx=0x7fa4ac007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac005b40 con 0x7fa4bc158e20 2023-12-19T08:06:12.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac005ce0 con 0x7fa4bc158e20 2023-12-19T08:06:12.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc0b3870 con 0x7fa4bc158e20 2023-12-19T08:06:12.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac007340 con 0x7fa4bc158e20 2023-12-19T08:06:12.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.405+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc0b3c00 con 0x7fa4bc158e20 2023-12-19T08:06:12.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.409+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc0b4310 con 0x7fa4bc158e20 2023-12-19T08:06:12.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.409+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa4ac01b400 con 0x7fa4bc158e20 2023-12-19T08:06:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.409+0000 7fa4b27fc700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 0x7fa4a0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.409+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7fa4ac07c780 con 0x7fa4bc158e20 2023-12-19T08:06:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.413+0000 7fa4c248c700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 0x7fa4a0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.413+0000 7fa4c248c700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 0x7fa4a0062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009ca0 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.417+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa4ac04c930 con 0x7fa4bc158e20 2023-12-19T08:06:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.585+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa4bc071120 con 0x7fa4bc158e20 2023-12-19T08:06:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.597+0000 7fa4b27fc700 1 -- 172.21.15.111:0/1766608816 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v43) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa4ac04ff50 con 0x7fa4bc158e20 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 msgr2=0x7fa4a0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 0x7fa4a0062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009ca0 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 msgr2=0x7fa4bc0b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00c810 tx=0x7fa4ac007db0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 shutdown_connections 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4a0060670 0x7fa4a0062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4bc14b970 0x7fa4bc0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4bc158e20 0x7fa4bc0b4630 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 --2- 172.21.15.111:0/1766608816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4bc0ca2c0 0x7fa4bc0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 >> 172.21.15.111:0/1766608816 conn(0x7fa4bc067bb0 msgr2=0x7fa4bc06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 shutdown_connections 2023-12-19T08:06:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.601+0000 7fa4c46f0700 1 -- 172.21.15.111:0/1766608816 wait complete. 2023-12-19T08:06:12.624 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T08:06:12.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.785+0000 7fe8c528e700 1 Processor -- start 2023-12-19T08:06:12.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.785+0000 7fe8c528e700 1 -- start start 2023-12-19T08:06:12.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8c0158c90 0x7fe8c00caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 0x7fe8c0159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c01595a0 0x7fe8c0159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c006c850 con 0x7fe8c0158c90 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c006b470 con 0x7fe8c006aeb0 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c528e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c006b750 con 0x7fe8c01595a0 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8c0158c90 0x7fe8c00caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8c0158c90 0x7fe8c00caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52330/0 (socket says 172.21.15.111:52330) 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 -- 172.21.15.111:0/996890315 learned_addr learned my addr 172.21.15.111:0/996890315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8b77fe700 1 --2- 172.21.15.111:0/996890315 >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 0x7fe8c0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8c4a8d700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c01595a0 0x7fe8c0159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 -- 172.21.15.111:0/996890315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c01595a0 msgr2=0x7fe8c0159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:12.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c01595a0 0x7fe8c0159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 -- 172.21.15.111:0/996890315 >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 msgr2=0x7fe8c0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:12.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 --2- 172.21.15.111:0/996890315 >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 0x7fe8c0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 -- 172.21.15.111:0/996890315 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c006ba30 con 0x7fe8c0158c90 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8b77fe700 1 --2- 172.21.15.111:0/996890315 >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 0x7fe8c0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bffff700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c0158c90 0x7fe8c00caf40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0068000 tx=0x7fe8b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b2ef760e43e419 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bf7fe700 1 -- 172.21.15.111:0/996890315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000b640 con 0x7fe8c0158c90 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bf7fe700 1 -- 172.21.15.111:0/996890315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b000b7a0 con 0x7fe8c0158c90 2023-12-19T08:06:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.789+0000 7fe8bf7fe700 1 -- 172.21.15.111:0/996890315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000bb60 con 0x7fe8c0158c90 2023-12-19T08:06:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 -- 172.21.15.111:0/996890315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c0158c90 msgr2=0x7fe8c00caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c0158c90 0x7fe8c00caf40 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0068000 tx=0x7fe8b0009250 comp rx=0 tx=0).stop 2023-12-19T08:06:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 -- 172.21.15.111:0/996890315 shutdown_connections 2023-12-19T08:06:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c01595a0 0x7fe8c0159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 --2- 172.21.15.111:0/996890315 >> v2:172.21.15.143:3300/0 conn(0x7fe8c006aeb0 0x7fe8c0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 --2- 172.21.15.111:0/996890315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c0158c90 0x7fe8c00caf40 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 -- 172.21.15.111:0/996890315 >> 172.21.15.111:0/996890315 conn(0x7fe8c0067bb0 msgr2=0x7fe8c00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 -- 172.21.15.111:0/996890315 shutdown_connections 2023-12-19T08:06:12.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 -- 172.21.15.111:0/996890315 wait complete. 2023-12-19T08:06:12.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.793+0000 7fe8c528e700 1 Processor -- start 2023-12-19T08:06:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 -- start start 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8c006aeb0 0x7fe8c00c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 0x7fe8c0155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00c8890 con 0x7fe8c01595a0 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00c8b70 con 0x7fe8c006aeb0 2023-12-19T08:06:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c528e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c00c8e50 con 0x7fe8c0158c90 2023-12-19T08:06:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52346/0 (socket says 172.21.15.111:52346) 2023-12-19T08:06:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 -- 172.21.15.111:0/2989012052 learned_addr learned my addr 172.21.15.111:0/2989012052 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8b77fe700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 0x7fe8c0155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8bffff700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8c006aeb0 0x7fe8c00c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 -- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 msgr2=0x7fe8c0155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 0x7fe8c0155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 -- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8c006aeb0 msgr2=0x7fe8c00c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8c006aeb0 0x7fe8c00c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 -- 172.21.15.111:0/2989012052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a8009420 con 0x7fe8c01595a0 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8b77fe700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 0x7fe8c0155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.797+0000 7fe8c4a8d700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009e90 tx=0x7fe8b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b800cc20 con 0x7fe8c01595a0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b800f040 con 0x7fe8c01595a0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8014480 con 0x7fe8c01595a0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b00094f0 con 0x7fe8c01595a0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c00c9410 con 0x7fe8c01595a0 2023-12-19T08:06:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c00c9630 con 0x7fe8c01595a0 2023-12-19T08:06:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.801+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fe8b8014830 con 0x7fe8c01595a0 2023-12-19T08:06:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.805+0000 7fe8bcff9700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 0x7fe8a0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.805+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7fe8b807d7b0 con 0x7fe8c01595a0 2023-12-19T08:06:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.805+0000 7fe8bffff700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 0x7fe8a0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:12.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.805+0000 7fe8bffff700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 0x7fe8a0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0068000 tx=0x7fe8b0017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.809+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe8b8051260 con 0x7fe8c01595a0 2023-12-19T08:06:12.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.981+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe8c004be40 con 0x7fe8c01595a0 2023-12-19T08:06:12.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.985+0000 7fe8bcff9700 1 -- 172.21.15.111:0/2989012052 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe8b8050df0 con 0x7fe8c01595a0 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 msgr2=0x7fe8a0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 0x7fe8a0062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0068000 tx=0x7fe8b0017000 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 msgr2=0x7fe8c00b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009e90 tx=0x7fe8b800b040 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 shutdown_connections 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe8a0060670 0x7fe8a0062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c01595a0 0x7fe8c00b4eb0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8c0158c90 0x7fe8c0155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 --2- 172.21.15.111:0/2989012052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8c006aeb0 0x7fe8c00c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.989+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 >> 172.21.15.111:0/2989012052 conn(0x7fe8c0067bb0 msgr2=0x7fe8c0157e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.993+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 shutdown_connections 2023-12-19T08:06:12.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:12.993+0000 7fe8c528e700 1 -- 172.21.15.111:0/2989012052 wait complete. 2023-12-19T08:06:13.020 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T08:06:13.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.177+0000 7fde4bb45700 1 Processor -- start 2023-12-19T08:06:13.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.177+0000 7fde4bb45700 1 -- start start 2023-12-19T08:06:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde4bb45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde44158e20 0x7fde440ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde4bb45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde440cb970 0x7fde440ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde4bb45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde440cbdf0 con 0x7fde440ca2c0 2023-12-19T08:06:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde4bb45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde440cc0d0 con 0x7fde440cb970 2023-12-19T08:06:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde4bb45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde440cc3b0 con 0x7fde44158e20 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde440cb970 0x7fde440ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde440cb970 0x7fde440ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52352/0 (socket says 172.21.15.111:52352) 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 -- 172.21.15.111:0/4178521093 learned_addr learned my addr 172.21.15.111:0/4178521093 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde498e1700 1 --2- 172.21.15.111:0/4178521093 >> v2:172.21.15.143:3300/0 conn(0x7fde44158e20 0x7fde440ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 -- 172.21.15.111:0/4178521093 >> v2:172.21.15.143:3300/0 conn(0x7fde44158e20 msgr2=0x7fde440ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 --2- 172.21.15.111:0/4178521093 >> v2:172.21.15.143:3300/0 conn(0x7fde44158e20 0x7fde440ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 -- 172.21.15.111:0/4178521093 >> v1:172.21.15.143:6789/0 conn(0x7fde440ca2c0 legacy=0x7fde4406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 -- 172.21.15.111:0/4178521093 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde440cc690 con 0x7fde440cb970 2023-12-19T08:06:13.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.181+0000 7fde490e0700 1 --2- 172.21.15.111:0/4178521093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440cb970 0x7fde440ceb00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fde340099b0 tx=0x7fde34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe19291f5354d862 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde488df700 1 -- 172.21.15.111:0/4178521093 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3400b820 con 0x7fde440cb970 2023-12-19T08:06:13.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde488df700 1 -- 172.21.15.111:0/4178521093 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3400b980 con 0x7fde440cb970 2023-12-19T08:06:13.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde488df700 1 -- 172.21.15.111:0/4178521093 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3400bd20 con 0x7fde440cb970 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 -- 172.21.15.111:0/4178521093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440cb970 msgr2=0x7fde440ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 --2- 172.21.15.111:0/4178521093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440cb970 0x7fde440ceb00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fde340099b0 tx=0x7fde34009250 comp rx=0 tx=0).stop 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 -- 172.21.15.111:0/4178521093 shutdown_connections 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 --2- 172.21.15.111:0/4178521093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440cb970 0x7fde440ceb00 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 --2- 172.21.15.111:0/4178521093 >> v2:172.21.15.143:3300/0 conn(0x7fde44158e20 0x7fde440ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 -- 172.21.15.111:0/4178521093 >> 172.21.15.111:0/4178521093 conn(0x7fde44067bb0 msgr2=0x7fde440d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 -- 172.21.15.111:0/4178521093 shutdown_connections 2023-12-19T08:06:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.185+0000 7fde4bb45700 1 -- 172.21.15.111:0/4178521093 wait complete. 2023-12-19T08:06:13.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 Processor -- start 2023-12-19T08:06:13.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 -- start start 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde44158e20 0x7fde4414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4414b030 0x7fde4414b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4414f480 con 0x7fde440ca2c0 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4414b9f0 con 0x7fde44158e20 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.189+0000 7fde4bb45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde44151820 con 0x7fde4414b030 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52360/0 (socket says 172.21.15.111:52360) 2023-12-19T08:06:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 -- 172.21.15.111:0/1519867670 learned_addr learned my addr 172.21.15.111:0/1519867670 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde490e0700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde44158e20 0x7fde4414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde4a0e2700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4414b030 0x7fde4414b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 -- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4414b030 msgr2=0x7fde4414b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4414b030 0x7fde4414b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 -- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde44158e20 msgr2=0x7fde4414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde44158e20 0x7fde4414fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 -- 172.21.15.111:0/1519867670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde340094f0 con 0x7fde440ca2c0 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde498e1700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fde4000d890 tx=0x7fde40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400125f0 con 0x7fde440ca2c0 2023-12-19T08:06:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40012770 con 0x7fde440ca2c0 2023-12-19T08:06:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde44146390 con 0x7fde440ca2c0 2023-12-19T08:06:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40012b30 con 0x7fde440ca2c0 2023-12-19T08:06:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde44150c10 con 0x7fde440ca2c0 2023-12-19T08:06:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.193+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde44071120 con 0x7fde440ca2c0 2023-12-19T08:06:13.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.197+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fde4001e460 con 0x7fde440ca2c0 2023-12-19T08:06:13.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.197+0000 7fde39ffb700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 0x7fde28062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.197+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7fde4007d1d0 con 0x7fde440ca2c0 2023-12-19T08:06:13.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.197+0000 7fde490e0700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 0x7fde28062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.201+0000 7fde490e0700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 0x7fde28062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde340099b0 tx=0x7fde34017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.205+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde4004d300 con 0x7fde440ca2c0 2023-12-19T08:06:13.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.377+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fde4406a7a0 con 0x7fde440ca2c0 2023-12-19T08:06:13.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.385+0000 7fde39ffb700 1 -- 172.21.15.111:0/1519867670 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v45) v1 ==== 115+0+0 (secure 0 0 0) 0x7fde40050920 con 0x7fde440ca2c0 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 msgr2=0x7fde28062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 0x7fde28062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde340099b0 tx=0x7fde34017000 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 msgr2=0x7fde44145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fde4000d890 tx=0x7fde40008040 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 shutdown_connections 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fde28060670 0x7fde28062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde44158e20 0x7fde4414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4414b030 0x7fde4414b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 --2- 172.21.15.111:0/1519867670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde440ca2c0 0x7fde44145e50 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 >> 172.21.15.111:0/1519867670 conn(0x7fde44067bb0 msgr2=0x7fde44157c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 shutdown_connections 2023-12-19T08:06:13.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.389+0000 7fde4bb45700 1 -- 172.21.15.111:0/1519867670 wait complete. 2023-12-19T08:06:13.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-12-19T08:06:13.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 Processor -- start 2023-12-19T08:06:13.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 -- start start 2023-12-19T08:06:13.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc9a4158d60 0x7fc9a415c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40c5610 con 0x7fc9a40c51b0 2023-12-19T08:06:13.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a406c7c0 con 0x7fc9a4158d60 2023-12-19T08:06:13.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a940c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a406caa0 con 0x7fc9a406b7a0 2023-12-19T08:06:13.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:58140/0 (socket says 172.21.15.111:58140) 2023-12-19T08:06:13.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 -- 172.21.15.111:0/3895470191 learned_addr learned my addr 172.21.15.111:0/3895470191 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:13.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a3fff700 1 --2- 172.21.15.111:0/3895470191 >> v2:172.21.15.143:3300/0 conn(0x7fc9a4158d60 0x7fc9a415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 -- 172.21.15.111:0/3895470191 >> v2:172.21.15.143:3300/0 conn(0x7fc9a4158d60 msgr2=0x7fc9a415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:13.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 --2- 172.21.15.111:0/3895470191 >> v2:172.21.15.143:3300/0 conn(0x7fc9a4158d60 0x7fc9a415c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 -- 172.21.15.111:0/3895470191 >> v1:172.21.15.143:6789/0 conn(0x7fc9a40c51b0 legacy=0x7fc9a406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.581+0000 7fc9a37fe700 1 -- 172.21.15.111:0/3895470191 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a415ce60 con 0x7fc9a406b7a0 2023-12-19T08:06:13.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a37fe700 1 --2- 172.21.15.111:0/3895470191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc9980099b0 tx=0x7fc998009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2c0af542593941 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a2ffd700 1 -- 172.21.15.111:0/3895470191 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998009ea0 con 0x7fc9a406b7a0 2023-12-19T08:06:13.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a2ffd700 1 -- 172.21.15.111:0/3895470191 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99800b860 con 0x7fc9a406b7a0 2023-12-19T08:06:13.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a2ffd700 1 -- 172.21.15.111:0/3895470191 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800bc20 con 0x7fc9a406b7a0 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 -- 172.21.15.111:0/3895470191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 msgr2=0x7fc9a40c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 --2- 172.21.15.111:0/3895470191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc9980099b0 tx=0x7fc998009250 comp rx=0 tx=0).stop 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 -- 172.21.15.111:0/3895470191 shutdown_connections 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 --2- 172.21.15.111:0/3895470191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a406b7a0 0x7fc9a40c4c70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 --2- 172.21.15.111:0/3895470191 >> v2:172.21.15.143:3300/0 conn(0x7fc9a4158d60 0x7fc9a415c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 -- 172.21.15.111:0/3895470191 >> 172.21.15.111:0/3895470191 conn(0x7fc9a4067bb0 msgr2=0x7fc9a40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 -- 172.21.15.111:0/3895470191 shutdown_connections 2023-12-19T08:06:13.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.585+0000 7fc9a940c700 1 -- 172.21.15.111:0/3895470191 wait complete. 2023-12-19T08:06:13.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 Processor -- start 2023-12-19T08:06:13.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 -- start start 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 0x7fc9a40c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a40c51b0 0x7fc9a41562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4156810 con 0x7fc9a4158d60 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a415a990 con 0x7fc9a406b7a0 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.589+0000 7fc9a940c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a415b360 con 0x7fc9a40c51b0 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52364/0 (socket says 172.21.15.111:52364) 2023-12-19T08:06:13.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 -- 172.21.15.111:0/1512770723 learned_addr learned my addr 172.21.15.111:0/1512770723 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a3fff700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 0x7fc9a40c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a37fe700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a40c51b0 0x7fc9a41562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 -- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a40c51b0 msgr2=0x7fc9a41562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a40c51b0 0x7fc9a41562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 -- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 msgr2=0x7fc9a40c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 0x7fc9a40c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 -- 172.21.15.111:0/1512770723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9980094f0 con 0x7fc9a4158d60 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a3fff700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 0x7fc9a40c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a8c0b700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c8e0 tx=0x7fc99c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00ce80 con 0x7fc9a4158d60 2023-12-19T08:06:13.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c00a7e0 con 0x7fc9a4158d60 2023-12-19T08:06:13.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a4156a40 con 0x7fc9a4158d60 2023-12-19T08:06:13.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00ab80 con 0x7fc9a4158d60 2023-12-19T08:06:13.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a415a620 con 0x7fc9a4158d60 2023-12-19T08:06:13.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.593+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a415a480 con 0x7fc9a4158d60 2023-12-19T08:06:13.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.597+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc99c012420 con 0x7fc9a4158d60 2023-12-19T08:06:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.597+0000 7fc98bfff700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:13.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.597+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7fc99c051350 con 0x7fc9a4158d60 2023-12-19T08:06:13.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.597+0000 7fc9a3fff700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.597+0000 7fc9a3fff700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 0x7fc990062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc994006f00 tx=0x7fc994008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:13.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.605+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc9a415a480 con 0x7fc9a4158d60 2023-12-19T08:06:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:13.833+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fc9a40c5610 con 0x7fc9a4158d60 2023-12-19T08:06:14.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.197+0000 7fc98bfff700 1 -- 172.21.15.111:0/1512770723 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v722) v1 ==== 148+0+0 (secure 0 0 0) 0x7fc9a40c5610 con 0x7fc9a4158d60 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 msgr2=0x7fc990062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 0x7fc990062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc994006f00 tx=0x7fc994008040 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 msgr2=0x7fc9a415ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00c8e0 tx=0x7fc99c001710 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 shutdown_connections 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc990060670 0x7fc990062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9a4158d60 0x7fc9a415ae20 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9a40c51b0 0x7fc9a41562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 --2- 172.21.15.111:0/1512770723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9a406b7a0 0x7fc9a40c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 >> 172.21.15.111:0/1512770723 conn(0x7fc9a4067bb0 msgr2=0x7fc9a40b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:14.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 shutdown_connections 2023-12-19T08:06:14.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.201+0000 7fc9a940c700 1 -- 172.21.15.111:0/1512770723 wait complete. 2023-12-19T08:06:14.228 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-12-19T08:06:14.228 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-12-19T08:06:14.229 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T08:06:14.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:14.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.389+0000 7f6760d45700 1 Processor -- start 2023-12-19T08:06:14.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.389+0000 7f6760d45700 1 -- start start 2023-12-19T08:06:14.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f6760d45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f675c140710 0x7f675c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f6760d45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c06e150 con 0x7f675c068800 2023-12-19T08:06:14.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f6760d45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c06d540 con 0x7f675c140ae0 2023-12-19T08:06:14.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f6760d45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675c06d820 con 0x7f675c140710 2023-12-19T08:06:14.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f675c140710 0x7f675c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f675c068800 0x7f675c0ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48534/0 (socket says 172.21.15.111:48534) 2023-12-19T08:06:14.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675ad9d700 1 -- 172.21.15.111:0/2571798124 learned_addr learned my addr 172.21.15.111:0/2571798124 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:14.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675a59c700 1 -- 172.21.15.111:0/2571798124 >> v1:172.21.15.143:6789/0 conn(0x7f675c140ae0 legacy=0x7f675c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675a59c700 1 -- 172.21.15.111:0/2571798124 >> v1:172.21.15.111:6789/0 conn(0x7f675c068800 legacy=0x7f675c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675a59c700 1 -- 172.21.15.111:0/2571798124 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c06db00 con 0x7f675c140710 2023-12-19T08:06:14.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675a59c700 1 --2- 172.21.15.111:0/2571798124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140710 0x7f675c0c70a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f674c0098e0 tx=0x7f674c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2aeb6238df084 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675959a700 1 -- 172.21.15.111:0/2571798124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c009e80 con 0x7f675c140710 2023-12-19T08:06:14.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675959a700 1 -- 172.21.15.111:0/2571798124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c00b860 con 0x7f675c140710 2023-12-19T08:06:14.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.393+0000 7f675959a700 1 -- 172.21.15.111:0/2571798124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00bbe0 con 0x7f675c140710 2023-12-19T08:06:14.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- 172.21.15.111:0/2571798124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140710 msgr2=0x7f675c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 --2- 172.21.15.111:0/2571798124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140710 0x7f675c0c70a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f674c0098e0 tx=0x7f674c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- 172.21.15.111:0/2571798124 shutdown_connections 2023-12-19T08:06:14.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 --2- 172.21.15.111:0/2571798124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140710 0x7f675c0c70a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- 172.21.15.111:0/2571798124 >> 172.21.15.111:0/2571798124 conn(0x7f675c0c8850 msgr2=0x7f675c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:14.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- 172.21.15.111:0/2571798124 shutdown_connections 2023-12-19T08:06:14.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- 172.21.15.111:0/2571798124 wait complete. 2023-12-19T08:06:14.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 Processor -- start 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- start start 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f675c068800 0x7f675c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 0x7f675c0c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0a92b0 con 0x7f675c140ae0 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0c2b80 con 0x7f675c140710 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.397+0000 7f6760d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0a8980 con 0x7f675c068800 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52388/0 (socket says 172.21.15.111:52388) 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 -- 172.21.15.111:0/3900613634 learned_addr learned my addr 172.21.15.111:0/3900613634 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f6759d9b700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 0x7f675c0c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675a59c700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f675c068800 0x7f675c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 -- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f675c068800 msgr2=0x7f675c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f675c068800 0x7f675c156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 -- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 msgr2=0x7f675c0c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 0x7f675c0c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 -- 172.21.15.111:0/3900613634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009420 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f6759d9b700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 0x7f675c0c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f675ad9d700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f675400cbc0 tx=0x7f6754001960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675400a8e0 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675400aa40 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c0a8c60 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675400adc0 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c0c19c0 con 0x7f675c140ae0 2023-12-19T08:06:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.401+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c0a9460 con 0x7f675c140ae0 2023-12-19T08:06:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.405+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6754007300 con 0x7f675c140ae0 2023-12-19T08:06:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.405+0000 7f674a7fc700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 0x7f673c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.405+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7f675407d020 con 0x7f675c140ae0 2023-12-19T08:06:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.405+0000 7f675a59c700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 0x7f673c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.409+0000 7f675a59c700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 0x7f673c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f674c0093f0 tx=0x7f674c017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.413+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f675c0a9460 con 0x7f675c140ae0 2023-12-19T08:06:14.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.581+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f675c071120 con 0x7f675c140ae0 2023-12-19T08:06:14.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f674a7fc700 1 -- 172.21.15.111:0/3900613634 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+14362 (secure 0 0 0) 0x7f6754054e60 con 0x7f675c140ae0 2023-12-19T08:06:14.588 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:14.589 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":135,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:12.182540+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6837/1501582072":"2023-12-20T08:06:09.101059+0000","172.21.15.143:6835/1501582072":"2023-12-20T08:06:09.101059+0000","172.21.15.143:0/4155123299":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/2270028498":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/1932034904":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/2168214307":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/710694177":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/2962591998":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/396308477":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3598389140":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/1868422455":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/33593772":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/3413002548":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/1914134911":"2023-12-20T08:06:08.711219+0000","172.21.15.143:0/4220277516":"2023-12-20T08:05:25.018350+0000","172.21.15.143:6833/33796":"2023-12-20T08:05:25.018350+0000","172.21.15.143:0/3465467344":"2023-12-20T08:06:08.711219+0000","172.21.15.143:6832/33796":"2023-12-20T08:05:25.018350+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 msgr2=0x7f673c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 0x7f673c062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f674c0093f0 tx=0x7f674c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 msgr2=0x7f675c0c2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f675400cbc0 tx=0x7f6754001960 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 shutdown_connections 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f673c060670 0x7f673c062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f675c140710 0x7f675c0c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c140ae0 0x7f675c0c2570 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 --2- 172.21.15.111:0/3900613634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f675c068800 0x7f675c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 >> 172.21.15.111:0/3900613634 conn(0x7f675c0c8850 msgr2=0x7f675c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:14.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 shutdown_connections 2023-12-19T08:06:14.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.585+0000 7f6760d45700 1 -- 172.21.15.111:0/3900613634 wait complete. 2023-12-19T08:06:14.611 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:06:14.612 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:06:14.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.769+0000 7ff79d8f0700 1 Processor -- start 2023-12-19T08:06:14.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.769+0000 7ff79d8f0700 1 -- start start 2023-12-19T08:06:14.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff79d8f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7980c6d20 0x7ff7980c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff79d8f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79806a7a0 con 0x7ff79806ac30 2023-12-19T08:06:14.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff79d8f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980c7630 con 0x7ff79806c7c0 2023-12-19T08:06:14.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff79d8f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79806d4b0 con 0x7ff7980c6d20 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff7977fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff79806ac30 0x7ff79806b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48548/0 (socket says 172.21.15.111:48548) 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff7977fe700 1 -- 172.21.15.111:0/4109435911 learned_addr learned my addr 172.21.15.111:0/4109435911 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff796ffd700 1 --2- 172.21.15.111:0/4109435911 >> v2:172.21.15.143:3300/0 conn(0x7ff7980c6d20 0x7ff7980c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452961469 0 0) 0x7ff79806a7a0 con 0x7ff79806ac30 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff780003570 con 0x7ff79806ac30 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167285593 0 0) 0x7ff7980c7630 con 0x7ff79806c7c0 2023-12-19T08:06:14.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff79806a7a0 con 0x7ff79806c7c0 2023-12-19T08:06:14.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff796ffd700 1 -- 172.21.15.111:0/4109435911 >> v1:172.21.15.143:6789/0 conn(0x7ff79806c7c0 legacy=0x7ff79806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110993877 0 0) 0x7ff780003570 con 0x7ff79806ac30 2023-12-19T08:06:14.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff796ffd700 1 -- 172.21.15.111:0/4109435911 >> v1:172.21.15.111:6789/0 conn(0x7ff79806ac30 legacy=0x7ff79806b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff796ffd700 1 -- 172.21.15.111:0/4109435911 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79806d790 con 0x7ff7980c6d20 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff796ffd700 1 --2- 172.21.15.111:0/4109435911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7980c6d20 0x7ff7980c70f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff7880098e0 tx=0x7ff7880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b87ef39a4b5d4499 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7900017f0 con 0x7ff7980c6d20 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788011420 con 0x7ff7980c6d20 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.773+0000 7ff795ffb700 1 -- 172.21.15.111:0/4109435911 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7880117c0 con 0x7ff7980c6d20 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- 172.21.15.111:0/4109435911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7980c6d20 msgr2=0x7ff7980c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/4109435911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7980c6d20 0x7ff7980c70f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff7880098e0 tx=0x7ff7880091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- 172.21.15.111:0/4109435911 shutdown_connections 2023-12-19T08:06:14.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/4109435911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7980c6d20 0x7ff7980c70f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- 172.21.15.111:0/4109435911 >> 172.21.15.111:0/4109435911 conn(0x7ff798067bb0 msgr2=0x7ff7980cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:14.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- 172.21.15.111:0/4109435911 shutdown_connections 2023-12-19T08:06:14.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- 172.21.15.111:0/4109435911 wait complete. 2023-12-19T08:06:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 Processor -- start 2023-12-19T08:06:14.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.777+0000 7ff79d8f0700 1 -- start start 2023-12-19T08:06:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79806ac30 0x7ff7980cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff79806c7c0 0x7ff7980c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980c8d50 con 0x7ff7980c6d20 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980af600 con 0x7ff79806ac30 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7980c8ad0 con 0x7ff79806c7c0 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52394/0 (socket says 172.21.15.111:52394) 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff796ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79806ac30 0x7ff7980cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 -- 172.21.15.111:0/2647040003 learned_addr learned my addr 172.21.15.111:0/2647040003 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:14.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7967fc700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff79806c7c0 0x7ff7980c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 -- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff79806c7c0 msgr2=0x7ff7980c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff79806c7c0 0x7ff7980c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 -- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79806ac30 msgr2=0x7ff7980cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79806ac30 0x7ff7980cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 -- 172.21.15.111:0/2647040003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff788009420 con 0x7ff7980c6d20 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff7977fe700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff79000ccf0 tx=0x7ff790007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790004dc0 con 0x7ff7980c6d20 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79000a620 con 0x7ff7980c6d20 2023-12-19T08:06:14.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79000a9e0 con 0x7ff7980c6d20 2023-12-19T08:06:14.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7980aecd0 con 0x7ff7980c6d20 2023-12-19T08:06:14.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.781+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7980af290 con 0x7ff7980c6d20 2023-12-19T08:06:14.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.785+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7980af7b0 con 0x7ff7980c6d20 2023-12-19T08:06:14.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.785+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff790007150 con 0x7ff7980c6d20 2023-12-19T08:06:14.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.785+0000 7ff77effd700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 0x7ff784062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:14.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.785+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7ff79007c320 con 0x7ff7980c6d20 2023-12-19T08:06:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.785+0000 7ff796ffd700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 0x7ff784062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.789+0000 7ff796ffd700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 0x7ff784062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7880093f0 tx=0x7ff788016000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.789+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff7980af7b0 con 0x7ff7980c6d20 2023-12-19T08:06:14.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.957+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7980af7b0 con 0x7ff7980c6d20 2023-12-19T08:06:14.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.961+0000 7ff77effd700 1 -- 172.21.15.111:0/2647040003 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff7980af7b0 con 0x7ff7980c6d20 2023-12-19T08:06:14.964 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:06:14.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 msgr2=0x7ff784062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 0x7ff784062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff7880093f0 tx=0x7ff788016000 comp rx=0 tx=0).stop 2023-12-19T08:06:14.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 msgr2=0x7ff798168e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:14.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff79000ccf0 tx=0x7ff790007d20 comp rx=0 tx=0).stop 2023-12-19T08:06:14.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 shutdown_connections 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff784060670 0x7ff784062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7980c6d20 0x7ff798168e70 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff79806c7c0 0x7ff7980c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 --2- 172.21.15.111:0/2647040003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff79806ac30 0x7ff7980cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 >> 172.21.15.111:0/2647040003 conn(0x7ff798067bb0 msgr2=0x7ff7980ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 shutdown_connections 2023-12-19T08:06:14.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:14.965+0000 7ff79d8f0700 1 -- 172.21.15.111:0/2647040003 wait complete. 2023-12-19T08:06:14.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-12-19T08:06:15.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 Processor -- start 2023-12-19T08:06:15.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 -- start start 2023-12-19T08:06:15.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f44200c6900 0x7f44200c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f442015cc20 0x7f442015d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442006b470 con 0x7f442006b900 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442006e110 con 0x7f442015cc20 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44264e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4420161d10 con 0x7f44200c6900 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f4424ce0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f442006b900 0x7f442006da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48558/0 (socket says 172.21.15.111:48558) 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f4425ce2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f442015cc20 0x7f442015d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f4424ce0700 1 -- 172.21.15.111:0/1971732615 learned_addr learned my addr 172.21.15.111:0/1971732615 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:15.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.141+0000 7f44254e1700 1 --2- 172.21.15.111:0/1971732615 >> v2:172.21.15.143:3300/0 conn(0x7f44200c6900 0x7f44200c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4417fff700 1 -- 172.21.15.111:0/1971732615 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555941793 0 0) 0x7f442006b470 con 0x7f442006b900 2023-12-19T08:06:15.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4417fff700 1 -- 172.21.15.111:0/1971732615 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440c003570 con 0x7f442006b900 2023-12-19T08:06:15.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4425ce2700 1 -- 172.21.15.111:0/1971732615 >> v2:172.21.15.143:3300/0 conn(0x7f44200c6900 msgr2=0x7f44200c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4425ce2700 1 --2- 172.21.15.111:0/1971732615 >> v2:172.21.15.143:3300/0 conn(0x7f44200c6900 0x7f44200c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4425ce2700 1 -- 172.21.15.111:0/1971732615 >> v1:172.21.15.111:6789/0 conn(0x7f442006b900 legacy=0x7f442006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4425ce2700 1 -- 172.21.15.111:0/1971732615 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442006bd70 con 0x7f442015cc20 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4425ce2700 1 --2- 172.21.15.111:0/1971732615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f442015cc20 0x7f442015d080 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f442006c570 tx=0x7f4418009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3745b054a3999e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4417fff700 1 -- 172.21.15.111:0/1971732615 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800e070 con 0x7f442015cc20 2023-12-19T08:06:15.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4417fff700 1 -- 172.21.15.111:0/1971732615 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441800bab0 con 0x7f442015cc20 2023-12-19T08:06:15.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f4417fff700 1 -- 172.21.15.111:0/1971732615 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800be70 con 0x7f442015cc20 2023-12-19T08:06:15.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 -- 172.21.15.111:0/1971732615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f442015cc20 msgr2=0x7f442015d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:15.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 --2- 172.21.15.111:0/1971732615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f442015cc20 0x7f442015d080 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f442006c570 tx=0x7f4418009780 comp rx=0 tx=0).stop 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 -- 172.21.15.111:0/1971732615 shutdown_connections 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 --2- 172.21.15.111:0/1971732615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f442015cc20 0x7f442015d080 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 --2- 172.21.15.111:0/1971732615 >> v2:172.21.15.143:3300/0 conn(0x7f44200c6900 0x7f44200c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 -- 172.21.15.111:0/1971732615 >> 172.21.15.111:0/1971732615 conn(0x7f4420067bb0 msgr2=0x7f44200cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 -- 172.21.15.111:0/1971732615 shutdown_connections 2023-12-19T08:06:15.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.145+0000 7f44264e3700 1 -- 172.21.15.111:0/1971732615 wait complete. 2023-12-19T08:06:15.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 Processor -- start 2023-12-19T08:06:15.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 -- start start 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f442006b900 0x7f44200c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f442015cc20 0x7f4420155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200b18b0 con 0x7f44200c6900 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44200af590 con 0x7f442006b900 2023-12-19T08:06:15.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44264e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44200ae980 con 0x7f442015cc20 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52404/0 (socket says 172.21.15.111:52404) 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 -- 172.21.15.111:0/2183263344 learned_addr learned my addr 172.21.15.111:0/2183263344 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4425ce2700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f442015cc20 0x7f4420155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f44254e1700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f442006b900 0x7f44200c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 -- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f442015cc20 msgr2=0x7f4420155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:15.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f442015cc20 0x7f4420155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:15.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 -- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f442006b900 msgr2=0x7f44200c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:15.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f442006b900 0x7f44200c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:15.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.149+0000 7f4424ce0700 1 -- 172.21.15.111:0/2183263344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009a20 con 0x7f44200c6900 2023-12-19T08:06:15.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f4424ce0700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f441000c7c0 tx=0x7f4410007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:15.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410005cd0 con 0x7f44200c6900 2023-12-19T08:06:15.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410005e30 con 0x7f44200c6900 2023-12-19T08:06:15.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44200aec60 con 0x7f44200c6900 2023-12-19T08:06:15.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410007320 con 0x7f44200c6900 2023-12-19T08:06:15.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44200af220 con 0x7f44200c6900 2023-12-19T08:06:15.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44201617a0 con 0x7f44200c6900 2023-12-19T08:06:15.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.153+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f441001b3b0 con 0x7f44200c6900 2023-12-19T08:06:15.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.157+0000 7f44157fa700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 0x7f4404062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:15.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.157+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7f441007c2e0 con 0x7f44200c6900 2023-12-19T08:06:15.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.157+0000 7f44254e1700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 0x7f4404062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:15.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.157+0000 7f44254e1700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 0x7f4404062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44200ae480 tx=0x7f441c0091a0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:15.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.161+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44201617a0 con 0x7f44200c6900 2023-12-19T08:06:15.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:15.397+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f44201617a0 con 0x7f44200c6900 2023-12-19T08:06:16.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.217+0000 7f44157fa700 1 -- 172.21.15.111:0/2183263344 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v724) v1 ==== 181+0+0 (secure 0 0 0) 0x7f44201617a0 con 0x7f44200c6900 2023-12-19T08:06:16.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.217+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 msgr2=0x7f4404062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.217+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 0x7f4404062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44200ae480 tx=0x7f441c0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.217+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 msgr2=0x7f44200c8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.217+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f441000c7c0 tx=0x7f4410007e00 comp rx=0 tx=0).stop 2023-12-19T08:06:16.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 shutdown_connections 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4404060670 0x7f4404062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f442015cc20 0x7f4420155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44200c6900 0x7f44200c8830 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 --2- 172.21.15.111:0/2183263344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f442006b900 0x7f44200c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 >> 172.21.15.111:0/2183263344 conn(0x7f4420067bb0 msgr2=0x7f442015b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 shutdown_connections 2023-12-19T08:06:16.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.221+0000 7f44264e3700 1 -- 172.21.15.111:0/2183263344 wait complete. 2023-12-19T08:06:16.246 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T08:06:16.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.409+0000 7ff8b7a65700 1 Processor -- start 2023-12-19T08:06:16.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.409+0000 7ff8b7a65700 1 -- start start 2023-12-19T08:06:16.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b7a65700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b7a65700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8b006ac30 0x7ff8b006b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b7a65700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b006a7a0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b7a65700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b006d4b0 con 0x7ff8b006ac30 2023-12-19T08:06:16.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b7a65700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b006d790 con 0x7ff8b006c7c0 2023-12-19T08:06:16.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:58196/0 (socket says 172.21.15.111:58196) 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 -- 172.21.15.111:0/3420111425 learned_addr learned my addr 172.21.15.111:0/3420111425 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5801700 1 --1- 172.21.15.111:0/3420111425 >> v1:172.21.15.111:6789/0 conn(0x7ff8b00c6d20 0x7ff8b00c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48570/0 (socket says 172.21.15.111:48570) 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b6002700 1 --2- 172.21.15.111:0/3420111425 >> v2:172.21.15.143:3300/0 conn(0x7ff8b006ac30 0x7ff8b006b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 -- 172.21.15.111:0/3420111425 >> v2:172.21.15.143:3300/0 conn(0x7ff8b006ac30 msgr2=0x7ff8b006b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 --2- 172.21.15.111:0/3420111425 >> v2:172.21.15.143:3300/0 conn(0x7ff8b006ac30 0x7ff8b006b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 -- 172.21.15.111:0/3420111425 >> v1:172.21.15.111:6789/0 conn(0x7ff8b00c6d20 legacy=0x7ff8b00c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.413+0000 7ff8b5000700 1 -- 172.21.15.111:0/3420111425 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b006da70 con 0x7ff8b006c7c0 2023-12-19T08:06:16.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b5000700 1 --2- 172.21.15.111:0/3420111425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00099b0 tx=0x7ff8a0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41932ad94378b796 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8a7fff700 1 -- 172.21.15.111:0/3420111425 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000b860 con 0x7ff8b006c7c0 2023-12-19T08:06:16.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8a7fff700 1 -- 172.21.15.111:0/3420111425 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a000b9c0 con 0x7ff8b006c7c0 2023-12-19T08:06:16.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8a7fff700 1 -- 172.21.15.111:0/3420111425 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000bd60 con 0x7ff8b006c7c0 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 -- 172.21.15.111:0/3420111425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 msgr2=0x7ff8b006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/3420111425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00099b0 tx=0x7ff8a0009250 comp rx=0 tx=0).stop 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 -- 172.21.15.111:0/3420111425 shutdown_connections 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/3420111425 >> v2:172.21.15.143:3300/0 conn(0x7ff8b006ac30 0x7ff8b006b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/3420111425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006c7c0 0x7ff8b006cc60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 -- 172.21.15.111:0/3420111425 >> 172.21.15.111:0/3420111425 conn(0x7ff8b0067bb0 msgr2=0x7ff8b00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 -- 172.21.15.111:0/3420111425 shutdown_connections 2023-12-19T08:06:16.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.417+0000 7ff8b7a65700 1 -- 172.21.15.111:0/3420111425 wait complete. 2023-12-19T08:06:16.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.421+0000 7ff8b7a65700 1 Processor -- start 2023-12-19T08:06:16.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.421+0000 7ff8b7a65700 1 -- start start 2023-12-19T08:06:16.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.421+0000 7ff8b7a65700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006ac30 0x7ff8b0145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 0x7ff8b00c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b016a8a0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b01463c0 con 0x7ff8b006c7c0 2023-12-19T08:06:16.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b0169c90 con 0x7ff8b006ac30 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52410/0 (socket says 172.21.15.111:52410) 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 -- 172.21.15.111:0/2299078683 learned_addr learned my addr 172.21.15.111:0/2299078683 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b5801700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006ac30 0x7ff8b0145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b5000700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 0x7ff8b00c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 -- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006ac30 msgr2=0x7ff8b0145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006ac30 0x7ff8b0145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 -- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 msgr2=0x7ff8b00c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 0x7ff8b00c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 -- 172.21.15.111:0/2299078683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00094f0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b5000700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 0x7ff8b00c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b6002700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009fa0 tx=0x7ff8a8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800ec70 con 0x7ff8b00c6d20 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800edd0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b0169f70 con 0x7ff8b00c6d20 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8014670 con 0x7ff8b00c6d20 2023-12-19T08:06:16.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b016a530 con 0x7ff8b00c6d20 2023-12-19T08:06:16.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.425+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b0071120 con 0x7ff8b00c6d20 2023-12-19T08:06:16.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.429+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff8a80149e0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.429+0000 7ff8a57fa700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 0x7ff894062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.429+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7ff8a807d320 con 0x7ff8b00c6d20 2023-12-19T08:06:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.429+0000 7ff8b5801700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 0x7ff894062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.433+0000 7ff8b5801700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 0x7ff894062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac004fd0 tx=0x7ff8ac007eb0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.437+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff8a804d4f0 con 0x7ff8b00c6d20 2023-12-19T08:06:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.613+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff8b006dd50 con 0x7ff8b00c6d20 2023-12-19T08:06:16.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.621+0000 7ff8a57fa700 1 -- 172.21.15.111:0/2299078683 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff8a8050b10 con 0x7ff8b00c6d20 2023-12-19T08:06:16.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 msgr2=0x7ff894062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 0x7ff894062b50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac004fd0 tx=0x7ff8ac007eb0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 msgr2=0x7ff8b00a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009fa0 tx=0x7ff8a8009900 comp rx=0 tx=0).stop 2023-12-19T08:06:16.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 shutdown_connections 2023-12-19T08:06:16.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff8940606a0 0x7ff894062b50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8b006c7c0 0x7ff8b00c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b00c6d20 0x7ff8b00a9820 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 --2- 172.21.15.111:0/2299078683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8b006ac30 0x7ff8b0145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 >> 172.21.15.111:0/2299078683 conn(0x7ff8b0067bb0 msgr2=0x7ff8b00c18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:16.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 shutdown_connections 2023-12-19T08:06:16.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.625+0000 7ff8b7a65700 1 -- 172.21.15.111:0/2299078683 wait complete. 2023-12-19T08:06:16.654 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T08:06:16.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:06:16.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T08:06:16.742 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T08:06:16.742 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T08:06:16.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:16.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.909+0000 7f548abcc700 1 Processor -- start 2023-12-19T08:06:16.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.909+0000 7f548abcc700 1 -- start start 2023-12-19T08:06:16.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f548abcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 0x7f54840afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f548abcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f548406a540 0x7f548406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f548abcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840ca5a0 con 0x7f54840cc470 2023-12-19T08:06:16.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f548abcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548406dac0 con 0x7f5484158e70 2023-12-19T08:06:16.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f548abcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54840b0260 con 0x7f548406a540 2023-12-19T08:06:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5489bca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 0x7f54840afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5489bca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 0x7f54840afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52422/0 (socket says 172.21.15.111:52422) 2023-12-19T08:06:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5489bca700 1 -- 172.21.15.111:0/380248363 learned_addr learned my addr 172.21.15.111:0/380248363 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 --2- 172.21.15.111:0/380248363 >> v2:172.21.15.143:3300/0 conn(0x7f548406a540 0x7f548406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 -- 172.21.15.111:0/380248363 >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 msgr2=0x7f54840afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:16.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 --2- 172.21.15.111:0/380248363 >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 0x7f54840afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 -- 172.21.15.111:0/380248363 >> v1:172.21.15.143:6789/0 conn(0x7f54840cc470 legacy=0x7f54840cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 -- 172.21.15.111:0/380248363 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548406d220 con 0x7f548406a540 2023-12-19T08:06:16.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f54893c9700 1 --2- 172.21.15.111:0/380248363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548406a540 0x7f548406c9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f54840cfd20 tx=0x7f54740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14abc4875f387c86 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5488bc8700 1 -- 172.21.15.111:0/380248363 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474011630 con 0x7f548406a540 2023-12-19T08:06:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5488bc8700 1 -- 172.21.15.111:0/380248363 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474011790 con 0x7f548406a540 2023-12-19T08:06:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.913+0000 7f5488bc8700 1 -- 172.21.15.111:0/380248363 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474011b30 con 0x7f548406a540 2023-12-19T08:06:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 -- 172.21.15.111:0/380248363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548406a540 msgr2=0x7f548406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 --2- 172.21.15.111:0/380248363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548406a540 0x7f548406c9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f54840cfd20 tx=0x7f54740091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 -- 172.21.15.111:0/380248363 shutdown_connections 2023-12-19T08:06:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 --2- 172.21.15.111:0/380248363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548406a540 0x7f548406c9d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 --2- 172.21.15.111:0/380248363 >> v2:172.21.15.111:3300/0 conn(0x7f5484158e70 0x7f54840afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 -- 172.21.15.111:0/380248363 >> 172.21.15.111:0/380248363 conn(0x7f54840670e0 msgr2=0x7f54840674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 -- 172.21.15.111:0/380248363 shutdown_connections 2023-12-19T08:06:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 -- 172.21.15.111:0/380248363 wait complete. 2023-12-19T08:06:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.917+0000 7f548abcc700 1 Processor -- start 2023-12-19T08:06:16.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 -- start start 2023-12-19T08:06:16.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 0x7f5484155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5484158e70 0x7f5484156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54840b38c0 con 0x7f548406a540 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54840b3ba0 con 0x7f54840cc470 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548abcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54840b3e80 con 0x7f5484158e70 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52432/0 (socket says 172.21.15.111:52432) 2023-12-19T08:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 -- 172.21.15.111:0/1480488715 learned_addr learned my addr 172.21.15.111:0/1480488715 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:16.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f54893c9700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 0x7f5484155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f548a3cb700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5484158e70 0x7f5484156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 -- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5484158e70 msgr2=0x7f5484156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5484158e70 0x7f5484156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 -- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 msgr2=0x7f5484155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 0x7f5484155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f5489bca700 1 -- 172.21.15.111:0/1480488715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5474009420 con 0x7f548406a540 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.921+0000 7f54893c9700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 0x7f5484155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f5489bca700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f548000ea50 tx=0x7f5480009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548000eee0 con 0x7f548406a540 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548000c850 con 0x7f548406a540 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54840b4160 con 0x7f548406a540 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548000cbd0 con 0x7f548406a540 2023-12-19T08:06:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54840c4aa0 con 0x7f548406a540 2023-12-19T08:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.925+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54840b0b40 con 0x7f548406a540 2023-12-19T08:06:16.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.929+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5480013470 con 0x7f548406a540 2023-12-19T08:06:16.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.929+0000 7f5479ffb700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 0x7f5468062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:16.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.929+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7f548007ce20 con 0x7f548406a540 2023-12-19T08:06:16.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.929+0000 7f54893c9700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 0x7f5468062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:16.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.929+0000 7f54893c9700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 0x7f5468062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f54740093f0 tx=0x7f5474017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:16.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:16.933+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54840b0b40 con 0x7f548406a540 2023-12-19T08:06:17.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.137+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54840b0b40 con 0x7f548406a540 2023-12-19T08:06:17.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.137+0000 7f5479ffb700 1 -- 172.21.15.111:0/1480488715 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f5480017df0 con 0x7f548406a540 2023-12-19T08:06:17.142 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:17.143 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2579318738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":2579318738},{"type":"v1","addr":"172.21.15.143:6837","nonce":2579318738}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7714,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/2872508919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2872508919},{"type":"v1","addr":"172.21.15.143:6836","nonce":2872508919}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/3100919337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3100919337},{"type":"v1","addr":"172.21.15.111:6834","nonce":3100919337}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/413203040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":413203040},{"type":"v1","addr":"172.21.15.111:6835","nonce":413203040}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[]} 2023-12-19T08:06:17.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 msgr2=0x7f5468062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 0x7f5468062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f54740093f0 tx=0x7f5474017000 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 msgr2=0x7f54840c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f548000ea50 tx=0x7f5480009250 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 shutdown_connections 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5468060670 0x7f5468062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54840cc470 0x7f5484155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5484158e70 0x7f5484156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 --2- 172.21.15.111:0/1480488715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548406a540 0x7f54840c4560 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 >> 172.21.15.111:0/1480488715 conn(0x7f54840670e0 msgr2=0x7f5484158260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 shutdown_connections 2023-12-19T08:06:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.141+0000 7f548abcc700 1 -- 172.21.15.111:0/1480488715 wait complete. 2023-12-19T08:06:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 58 2023-12-19T08:06:17.162 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T08:06:17.162 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T08:06:17.162 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T08:06:17.162 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:06:17.162 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T08:06:17.163 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T08:06:17.163 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T08:06:17.163 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:06:17.163 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T08:06:17.163 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T08:06:17.163 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T08:06:17.163 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:06:17.163 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T08:06:17.164 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T08:06:17.164 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T08:06:17.164 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T08:06:17.180 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T08:06:17.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T08:06:17.186 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T08:06:17.186 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T08:06:17.189 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T08:06:17.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T08:06:17.191 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T08:06:17.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T08:06:17.237 INFO:tasks.ceph.mds.c:Started 2023-12-19T08:06:17.238 DEBUG:teuthology.parallel:result is None 2023-12-19T08:06:17.241 INFO:tasks.ceph.mds.a:Started 2023-12-19T08:06:17.242 DEBUG:teuthology.parallel:result is None 2023-12-19T08:06:17.242 INFO:tasks.ceph.mds.d:Started 2023-12-19T08:06:17.243 DEBUG:teuthology.parallel:result is None 2023-12-19T08:06:17.248 INFO:tasks.ceph.mds.b:Started 2023-12-19T08:06:17.249 DEBUG:teuthology.parallel:result is None 2023-12-19T08:06:17.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T08:06:17.341 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T08:06:17.347 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T08:06:17.358 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T08:06:17.358 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T08:06:17.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.397+0000 7f3c4aa03700 1 Processor -- start 2023-12-19T08:06:17.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.401+0000 7f3c4aa03700 1 -- start start 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.401+0000 7f3c4aa03700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c44158e20 0x7f3c4406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.401+0000 7f3c4aa03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c440ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4406b730 con 0x7f3c441591f0 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4406ab20 con 0x7f3c44158e20 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4406ae00 con 0x7f3c440ce810 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49200700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c441591f0 0x7f3c440ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48594/0 (socket says 172.21.15.111:48594) 2023-12-19T08:06:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49200700 1 -- 172.21.15.111:0/710471219 learned_addr learned my addr 172.21.15.111:0/710471219 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:17.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 --2- 172.21.15.111:0/710471219 >> v2:172.21.15.111:3300/0 conn(0x7f3c44158e20 0x7f3c4406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4a202700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c440ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 -- 172.21.15.111:0/710471219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 msgr2=0x7f3c440ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c440ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c489ff700 1 -- 172.21.15.111:0/710471219 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791999741 0 0) 0x7f3c4406b730 con 0x7f3c441591f0 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 -- 172.21.15.111:0/710471219 >> v1:172.21.15.111:6789/0 conn(0x7f3c441591f0 legacy=0x7f3c440ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 -- 172.21.15.111:0/710471219 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4406b0e0 con 0x7f3c44158e20 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c49a01700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c4406a540 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f3c400098e0 tx=0x7f3c400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945791cc30ee2e52 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c489ff700 1 -- 172.21.15.111:0/710471219 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40009e80 con 0x7f3c44158e20 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c489ff700 1 -- 172.21.15.111:0/710471219 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4000b860 con 0x7f3c44158e20 2023-12-19T08:06:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c489ff700 1 -- 172.21.15.111:0/710471219 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4000bbe0 con 0x7f3c44158e20 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- 172.21.15.111:0/710471219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 msgr2=0x7f3c4406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c4406a540 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f3c400098e0 tx=0x7f3c400091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- 172.21.15.111:0/710471219 shutdown_connections 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c440ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/710471219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c4406a540 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- 172.21.15.111:0/710471219 >> 172.21.15.111:0/710471219 conn(0x7f3c44067bb0 msgr2=0x7f3c440cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- 172.21.15.111:0/710471219 shutdown_connections 2023-12-19T08:06:17.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.405+0000 7f3c4aa03700 1 -- 172.21.15.111:0/710471219 wait complete. 2023-12-19T08:06:17.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 Processor -- start 2023-12-19T08:06:17.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 -- start start 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c4414e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c441591f0 0x7f3c440b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440b46c0 con 0x7f3c44158e20 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44152500 con 0x7f3c441591f0 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.409+0000 7f3c4aa03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440b43e0 con 0x7f3c440ce810 2023-12-19T08:06:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52502/0 (socket says 172.21.15.111:52502) 2023-12-19T08:06:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 -- 172.21.15.111:0/947637583 learned_addr learned my addr 172.21.15.111:0/947637583 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49a01700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c4414e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c4a202700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c441591f0 0x7f3c440b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 -- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 msgr2=0x7f3c4414e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c4414e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 -- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c441591f0 msgr2=0x7f3c440b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c441591f0 0x7f3c440b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 -- 172.21.15.111:0/947637583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40009420 con 0x7f3c44158e20 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c49200700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c5b0 tx=0x7f3c34001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34005b30 con 0x7f3c44158e20 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c34005c90 con 0x7f3c44158e20 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34007320 con 0x7f3c44158e20 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c44151eb0 con 0x7f3c44158e20 2023-12-19T08:06:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.413+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440b5040 con 0x7f3c44158e20 2023-12-19T08:06:17.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.417+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44151d10 con 0x7f3c44158e20 2023-12-19T08:06:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.417+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3c3401b3b0 con 0x7f3c44158e20 2023-12-19T08:06:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.417+0000 7f3c39ffb700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 0x7f3c28062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.417+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6495+0+0 (secure 0 0 0) 0x7f3c3407c0a0 con 0x7f3c44158e20 2023-12-19T08:06:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.417+0000 7f3c49a01700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 0x7f3c28062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.421+0000 7f3c49a01700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 0x7f3c28062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c400091e0 tx=0x7f3c40017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:17.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.421+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c44151d10 con 0x7f3c44158e20 2023-12-19T08:06:17.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:17.601+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3c44151d10 con 0x7f3c44158e20 2023-12-19T08:06:18.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.657+0000 7f3c39ffb700 1 -- 172.21.15.111:0/947637583 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c44151d10 con 0x7f3c44158e20 2023-12-19T08:06:18.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.661+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 msgr2=0x7f3c28062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.661+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 0x7f3c28062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3c400091e0 tx=0x7f3c40017000 comp rx=0 tx=0).stop 2023-12-19T08:06:18.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.661+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 msgr2=0x7f3c44148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.661+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c5b0 tx=0x7f3c34001070 comp rx=0 tx=0).stop 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 shutdown_connections 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3c28060670 0x7f3c28062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c441591f0 0x7f3c440b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c44158e20 0x7f3c44148950 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 --2- 172.21.15.111:0/947637583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c440ce810 0x7f3c4414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 >> 172.21.15.111:0/947637583 conn(0x7f3c44067bb0 msgr2=0x7f3c441548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 shutdown_connections 2023-12-19T08:06:18.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.665+0000 7f3c4aa03700 1 -- 172.21.15.111:0/947637583 wait complete. 2023-12-19T08:06:18.672 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T08:06:18.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T08:06:18.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.865+0000 7efdc1f45700 1 Processor -- start 2023-12-19T08:06:18.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 -- start start 2023-12-19T08:06:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efdbc0c2980 0x7efdbc0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc06c9f0 con 0x7efdbc13f890 2023-12-19T08:06:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc0c7f20 con 0x7efdbc0c2980 2023-12-19T08:06:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdc1f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc068a50 con 0x7efdbc0680f0 2023-12-19T08:06:18.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdbb7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7efdbc13f890 0x7efdbc0c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:48182/0 (socket says 172.21.15.111:48182) 2023-12-19T08:06:18.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.869+0000 7efdbb7fe700 1 -- 172.21.15.111:0/1374496741 learned_addr learned my addr 172.21.15.111:0/1374496741 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:18.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 --2- 172.21.15.111:0/1374496741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbaffd700 1 --2- 172.21.15.111:0/1374496741 >> v2:172.21.15.143:3300/0 conn(0x7efdbc0c2980 0x7efdbc0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdba7fc700 1 -- 172.21.15.111:0/1374496741 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707346332 0 0) 0x7efdbc06c9f0 con 0x7efdbc13f890 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdba7fc700 1 -- 172.21.15.111:0/1374496741 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4003570 con 0x7efdbc13f890 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 -- 172.21.15.111:0/1374496741 >> v2:172.21.15.143:3300/0 conn(0x7efdbc0c2980 msgr2=0x7efdbc0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 --2- 172.21.15.111:0/1374496741 >> v2:172.21.15.143:3300/0 conn(0x7efdbc0c2980 0x7efdbc0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 -- 172.21.15.111:0/1374496741 >> v1:172.21.15.143:6789/0 conn(0x7efdbc13f890 legacy=0x7efdbc0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 -- 172.21.15.111:0/1374496741 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc0c1c50 con 0x7efdbc0680f0 2023-12-19T08:06:18.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdbbfff700 1 --2- 172.21.15.111:0/1374496741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc068510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009e60 tx=0x7efdb4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b71dbed927675bb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:18.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdba7fc700 1 -- 172.21.15.111:0/1374496741 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400e070 con 0x7efdbc0680f0 2023-12-19T08:06:18.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdba7fc700 1 -- 172.21.15.111:0/1374496741 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb400bab0 con 0x7efdbc0680f0 2023-12-19T08:06:18.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdba7fc700 1 -- 172.21.15.111:0/1374496741 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb400be70 con 0x7efdbc0680f0 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 -- 172.21.15.111:0/1374496741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 msgr2=0x7efdbc068510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 --2- 172.21.15.111:0/1374496741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc068510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009e60 tx=0x7efdb4009780 comp rx=0 tx=0).stop 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 -- 172.21.15.111:0/1374496741 shutdown_connections 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 --2- 172.21.15.111:0/1374496741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc068510 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 --2- 172.21.15.111:0/1374496741 >> v2:172.21.15.143:3300/0 conn(0x7efdbc0c2980 0x7efdbc0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 -- 172.21.15.111:0/1374496741 >> 172.21.15.111:0/1374496741 conn(0x7efdbc0c8850 msgr2=0x7efdbc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 -- 172.21.15.111:0/1374496741 shutdown_connections 2023-12-19T08:06:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.873+0000 7efdc1f45700 1 -- 172.21.15.111:0/1374496741 wait complete. 2023-12-19T08:06:18.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 Processor -- start 2023-12-19T08:06:18.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 -- start start 2023-12-19T08:06:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdbc13f890 0x7efdbc149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0c4d60 con 0x7efdbc0c2980 2023-12-19T08:06:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0a88f0 con 0x7efdbc13f890 2023-12-19T08:06:18.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.877+0000 7efdc1f45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc0c4fe0 con 0x7efdbc0680f0 2023-12-19T08:06:18.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52510/0 (socket says 172.21.15.111:52510) 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 -- 172.21.15.111:0/2230978323 learned_addr learned my addr 172.21.15.111:0/2230978323 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbbfff700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdbc13f890 0x7efdbc149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbb7fe700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 -- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 msgr2=0x7efdbc0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc0c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 -- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdbc13f890 msgr2=0x7efdbc149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdbc13f890 0x7efdbc149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 -- 172.21.15.111:0/2230978323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb4009a20 con 0x7efdbc0c2980 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdbaffd700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec50 tx=0x7efdb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000c850 con 0x7efdbc0c2980 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb000c9b0 con 0x7efdbc0c2980 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc0c52c0 con 0x7efdbc0c2980 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000cd70 con 0x7efdbc0c2980 2023-12-19T08:06:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc0c5880 con 0x7efdbc0c2980 2023-12-19T08:06:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.881+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdbc071120 con 0x7efdbc0c2980 2023-12-19T08:06:18.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.885+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7efdb0013680 con 0x7efdbc0c2980 2023-12-19T08:06:18.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.885+0000 7efda37fe700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 0x7efda8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:18.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.885+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7efdb007cc40 con 0x7efdbc0c2980 2023-12-19T08:06:18.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.885+0000 7efdbb7fe700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 0x7efda8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:18.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.889+0000 7efdbb7fe700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 0x7efda8062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdac007fd0 tx=0x7efdac009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:18.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:18.893+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efdb004d070 con 0x7efdbc0c2980 2023-12-19T08:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.125+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efdbc0642a0 con 0x7efdbc0c2980 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.125+0000 7efda37fe700 1 -- 172.21.15.111:0/2230978323 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efdb0050690 con 0x7efdbc0c2980 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:06:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T08:06:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T08:06:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T08:06:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T08:06:19.143 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:06:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 msgr2=0x7efda8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 0x7efda8062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdac007fd0 tx=0x7efdac009040 comp rx=0 tx=0).stop 2023-12-19T08:06:19.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 msgr2=0x7efdbc0c4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7efdb000ec50 tx=0x7efdb00091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 shutdown_connections 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7efda8060670 0x7efda8062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdbc13f890 0x7efdbc149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdbc0c2980 0x7efdbc0c4820 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 --2- 172.21.15.111:0/2230978323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efdbc0680f0 0x7efdbc0c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 >> 172.21.15.111:0/2230978323 conn(0x7efdbc0c8850 msgr2=0x7efdbc15dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 shutdown_connections 2023-12-19T08:06:19.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.129+0000 7efdc1f45700 1 -- 172.21.15.111:0/2230978323 wait complete. 2023-12-19T08:06:19.153 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.317+0000 7f51dce2e700 1 Processor -- start 2023-12-19T08:06:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.317+0000 7f51dce2e700 1 -- start start 2023-12-19T08:06:19.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51dce2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51d8068800 0x7f51d815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51dce2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80c7190 con 0x7f51d815d320 2023-12-19T08:06:19.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51dce2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80c1b50 con 0x7f51d80c15b0 2023-12-19T08:06:19.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51dce2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80c1e30 con 0x7f51d8068800 2023-12-19T08:06:19.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f51d815d320 0x7f51d806d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48608/0 (socket says 172.21.15.111:48608) 2023-12-19T08:06:19.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d659c700 1 -- 172.21.15.111:0/2572633098 learned_addr learned my addr 172.21.15.111:0/2572633098 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:19.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d6d9d700 1 --2- 172.21.15.111:0/2572633098 >> v2:172.21.15.143:3300/0 conn(0x7f51d8068800 0x7f51d815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896755374 0 0) 0x7f51d80c7190 con 0x7f51d815d320 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c4003570 con 0x7f51d815d320 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963775798 0 0) 0x7f51d80c1b50 con 0x7f51d80c15b0 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d80c7190 con 0x7f51d80c15b0 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283503281 0 0) 0x7f51c4003570 con 0x7f51d815d320 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2572633098 >> v1:172.21.15.143:6789/0 conn(0x7f51d80c15b0 legacy=0x7f51d80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2572633098 >> v1:172.21.15.111:6789/0 conn(0x7f51d815d320 legacy=0x7f51d806d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.321+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2572633098 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80c6e20 con 0x7f51d8068800 2023-12-19T08:06:19.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51d6d9d700 1 --2- 172.21.15.111:0/2572633098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d8068800 0x7f51d815d6f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f51d00098e0 tx=0x7f51d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8f117ded515600 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80017f0 con 0x7f51d8068800 2023-12-19T08:06:19.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0011420 con 0x7f51d8068800 2023-12-19T08:06:19.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51d5d9b700 1 -- 172.21.15.111:0/2572633098 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00117c0 con 0x7f51d8068800 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 -- 172.21.15.111:0/2572633098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d8068800 msgr2=0x7f51d815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2572633098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d8068800 0x7f51d815d6f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f51d00098e0 tx=0x7f51d00091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 -- 172.21.15.111:0/2572633098 shutdown_connections 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2572633098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d8068800 0x7f51d815d6f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 -- 172.21.15.111:0/2572633098 >> 172.21.15.111:0/2572633098 conn(0x7f51d80c8850 msgr2=0x7f51d80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:19.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 -- 172.21.15.111:0/2572633098 shutdown_connections 2023-12-19T08:06:19.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.325+0000 7f51dce2e700 1 -- 172.21.15.111:0/2572633098 wait complete. 2023-12-19T08:06:19.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 Processor -- start 2023-12-19T08:06:19.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- start start 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51d8068800 0x7f51d815d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 0x7f51d80cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c45a0 con 0x7f51d815d320 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c3990 con 0x7f51d80c15b0 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c3c70 con 0x7f51d8068800 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52524/0 (socket says 172.21.15.111:52524) 2023-12-19T08:06:19.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2356798558 learned_addr learned my addr 172.21.15.111:0/2356798558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:19.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51cdd9b700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 0x7f51d80cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d659c700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51d8068800 0x7f51d815d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51d8068800 msgr2=0x7f51d815d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51d8068800 0x7f51d815d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 msgr2=0x7f51d80cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 0x7f51d80cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 -- 172.21.15.111:0/2356798558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0009420 con 0x7f51d815d320 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51cdd9b700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 0x7f51d80cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51d6d9d700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f51d00093f0 tx=0x7f51d0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d001bd50 con 0x7f51d815d320 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0020070 con 0x7f51d815d320 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80c3f50 con 0x7f51d815d320 2023-12-19T08:06:19.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00276d0 con 0x7f51d815d320 2023-12-19T08:06:19.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.329+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d81585b0 con 0x7f51d815d320 2023-12-19T08:06:19.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.333+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d81587a0 con 0x7f51d815d320 2023-12-19T08:06:19.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.333+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f51d0027a40 con 0x7f51d815d320 2023-12-19T08:06:19.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.333+0000 7f51ceffd700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 0x7f51b8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.333+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f51d0022070 con 0x7f51d815d320 2023-12-19T08:06:19.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.337+0000 7f51d659c700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 0x7f51b8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.337+0000 7f51d659c700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 0x7f51b8062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005fd0 tx=0x7f51c80014e0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.341+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f51d81587a0 con 0x7f51d815d320 2023-12-19T08:06:19.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.517+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51d81587a0 con 0x7f51d815d320 2023-12-19T08:06:19.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.517+0000 7f51ceffd700 1 -- 172.21.15.111:0/2356798558 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+13231 (secure 0 0 0) 0x7f51d81587a0 con 0x7f51d815d320 2023-12-19T08:06:19.522 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:19.522 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":140,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:18.654101+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 msgr2=0x7f51b8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 0x7f51b8062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005fd0 tx=0x7f51c80014e0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 msgr2=0x7f51d8144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f51d00093f0 tx=0x7f51d0016000 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 shutdown_connections 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b80606a0 0x7f51b8062b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51d80c15b0 0x7f51d80cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51d815d320 0x7f51d8144650 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 --2- 172.21.15.111:0/2356798558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51d8068800 0x7f51d815d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 >> 172.21.15.111:0/2356798558 conn(0x7f51d80c8850 msgr2=0x7f51d806b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 shutdown_connections 2023-12-19T08:06:19.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.521+0000 7f51dce2e700 1 -- 172.21.15.111:0/2356798558 wait complete. 2023-12-19T08:06:19.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T08:06:19.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T08:06:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 Processor -- start 2023-12-19T08:06:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 -- start start 2023-12-19T08:06:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f27340ce2d0 0x7f27340ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340cc040 0x7f2734159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340cbbb0 con 0x7f27340caa10 2023-12-19T08:06:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273406dc90 con 0x7f27340ce2d0 2023-12-19T08:06:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f273a843700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340cafa0 con 0x7f27340cc040 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f2739841700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f27340ce2d0 0x7f27340ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f2739040700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f27340caa10 0x7f2734158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48620/0 (socket says 172.21.15.111:48620) 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.697+0000 7f2739040700 1 -- 172.21.15.111:0/961923527 learned_addr learned my addr 172.21.15.111:0/961923527 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a042700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340cc040 0x7f2734159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273883f700 1 -- 172.21.15.111:0/961923527 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778909124 0 0) 0x7f27340cbbb0 con 0x7f27340caa10 2023-12-19T08:06:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273883f700 1 -- 172.21.15.111:0/961923527 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2720003570 con 0x7f27340caa10 2023-12-19T08:06:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f2739841700 1 -- 172.21.15.111:0/961923527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340cc040 msgr2=0x7f2734159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f2739841700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340cc040 0x7f2734159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f2739841700 1 -- 172.21.15.111:0/961923527 >> v1:172.21.15.111:6789/0 conn(0x7f27340caa10 legacy=0x7f2734158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f2739841700 1 -- 172.21.15.111:0/961923527 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340cb280 con 0x7f27340ce2d0 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f2739841700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340ce2d0 0x7f27340ce6a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f27300098e0 tx=0x7f27300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c766c9993cffa6e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273883f700 1 -- 172.21.15.111:0/961923527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730011630 con 0x7f27340ce2d0 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273883f700 1 -- 172.21.15.111:0/961923527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730011790 con 0x7f27340ce2d0 2023-12-19T08:06:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273883f700 1 -- 172.21.15.111:0/961923527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730011b30 con 0x7f27340ce2d0 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 -- 172.21.15.111:0/961923527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340ce2d0 msgr2=0x7f27340ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340ce2d0 0x7f27340ce6a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f27300098e0 tx=0x7f27300091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 -- 172.21.15.111:0/961923527 shutdown_connections 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340cc040 0x7f2734159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 --2- 172.21.15.111:0/961923527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340ce2d0 0x7f27340ce6a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 -- 172.21.15.111:0/961923527 >> 172.21.15.111:0/961923527 conn(0x7f2734067bb0 msgr2=0x7f27340cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 -- 172.21.15.111:0/961923527 shutdown_connections 2023-12-19T08:06:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.701+0000 7f273a843700 1 -- 172.21.15.111:0/961923527 wait complete. 2023-12-19T08:06:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 Processor -- start 2023-12-19T08:06:19.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 -- start start 2023-12-19T08:06:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27340caa10 0x7f273415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340ce2d0 0x7f27341541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2734158c10 con 0x7f27340cc040 2023-12-19T08:06:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273415bf10 con 0x7f27340caa10 2023-12-19T08:06:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2734168dc0 con 0x7f27340ce2d0 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52548/0 (socket says 172.21.15.111:52548) 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 -- 172.21.15.111:0/2299155454 learned_addr learned my addr 172.21.15.111:0/2299155454 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739841700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27340caa10 0x7f273415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a042700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340ce2d0 0x7f27341541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 -- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340ce2d0 msgr2=0x7f27341541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340ce2d0 0x7f27341541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 -- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27340caa10 msgr2=0x7f273415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27340caa10 0x7f273415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 -- 172.21.15.111:0/2299155454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730009420 con 0x7f27340cc040 2023-12-19T08:06:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2739040700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f272400c7c0 tx=0x7f2724007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724005cd0 con 0x7f27340cc040 2023-12-19T08:06:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724005e30 con 0x7f27340cc040 2023-12-19T08:06:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273415b670 con 0x7f27340cc040 2023-12-19T08:06:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.705+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724007320 con 0x7f27340cc040 2023-12-19T08:06:19.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.709+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273415bc30 con 0x7f27340cc040 2023-12-19T08:06:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.709+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273404be40 con 0x7f27340cc040 2023-12-19T08:06:19.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.709+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f272401b3b0 con 0x7f27340cc040 2023-12-19T08:06:19.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.709+0000 7f2729ffb700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 0x7f2718062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:19.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.713+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f272407bfe0 con 0x7f27340cc040 2023-12-19T08:06:19.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.713+0000 7f2739841700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 0x7f2718062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:19.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.713+0000 7f2739841700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 0x7f2718062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27300091e0 tx=0x7f2730017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:19.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.717+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f272404c4b0 con 0x7f27340cc040 2023-12-19T08:06:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.885+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27340cb560 con 0x7f27340cc040 2023-12-19T08:06:19.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.885+0000 7f2729ffb700 1 -- 172.21.15.111:0/2299155454 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27240170a0 con 0x7f27340cc040 2023-12-19T08:06:19.891 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 msgr2=0x7f2718062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 0x7f2718062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f27300091e0 tx=0x7f2730017000 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 msgr2=0x7f27341586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f272400c7c0 tx=0x7f2724007e00 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 shutdown_connections 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2718060670 0x7f2718062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27340ce2d0 0x7f27341541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27340cc040 0x7f27341586d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 --2- 172.21.15.111:0/2299155454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27340caa10 0x7f273415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 >> 172.21.15.111:0/2299155454 conn(0x7f2734067bb0 msgr2=0x7f2734155260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 shutdown_connections 2023-12-19T08:06:19.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:19.889+0000 7f273a843700 1 -- 172.21.15.111:0/2299155454 wait complete. 2023-12-19T08:06:19.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T08:06:20.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.065+0000 7f9906a8e700 1 Processor -- start 2023-12-19T08:06:20.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.065+0000 7f9906a8e700 1 -- start start 2023-12-19T08:06:20.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.069+0000 7f9906a8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f990006e1d0 0x7f9900159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.069+0000 7f9906a8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990015a260 con 0x7f990006ad70 2023-12-19T08:06:20.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.069+0000 7f9906a8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000afad0 con 0x7f990006b210 2023-12-19T08:06:20.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.069+0000 7f9906a8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99000afdb0 con 0x7f990006e1d0 2023-12-19T08:06:20.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.069+0000 7f990528b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f990006ad70 0x7f990014f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48622/0 (socket says 172.21.15.111:48622) 2023-12-19T08:06:20.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f990528b700 1 -- 172.21.15.111:0/2941131358 learned_addr learned my addr 172.21.15.111:0/2941131358 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:20.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9905a8c700 1 --2- 172.21.15.111:0/2941131358 >> v2:172.21.15.111:3300/0 conn(0x7f990006e1d0 0x7f9900159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9904a8a700 1 -- 172.21.15.111:0/2941131358 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5976592 0 0) 0x7f990015a260 con 0x7f990006ad70 2023-12-19T08:06:20.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9904a8a700 1 -- 172.21.15.111:0/2941131358 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003570 con 0x7f990006ad70 2023-12-19T08:06:20.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9905a8c700 1 -- 172.21.15.111:0/2941131358 >> v1:172.21.15.143:6789/0 conn(0x7f990006b210 legacy=0x7f990014fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9905a8c700 1 -- 172.21.15.111:0/2941131358 >> v1:172.21.15.111:6789/0 conn(0x7f990006ad70 legacy=0x7f990014f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9905a8c700 1 -- 172.21.15.111:0/2941131358 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99000b0090 con 0x7f990006e1d0 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9905a8c700 1 --2- 172.21.15.111:0/2941131358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f9900159d20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098e0 tx=0x7f98fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6523a64bfa68a5f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9904a8a700 1 -- 172.21.15.111:0/2941131358 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00b640 con 0x7f990006e1d0 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9904a8a700 1 -- 172.21.15.111:0/2941131358 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc00b7a0 con 0x7f990006e1d0 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9904a8a700 1 -- 172.21.15.111:0/2941131358 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00bb40 con 0x7f990006e1d0 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 -- 172.21.15.111:0/2941131358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 msgr2=0x7f9900159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 --2- 172.21.15.111:0/2941131358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f9900159d20 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0098e0 tx=0x7f98fc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 -- 172.21.15.111:0/2941131358 shutdown_connections 2023-12-19T08:06:20.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 --2- 172.21.15.111:0/2941131358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f9900159d20 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 -- 172.21.15.111:0/2941131358 >> 172.21.15.111:0/2941131358 conn(0x7f9900067bb0 msgr2=0x7f99000add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:20.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 -- 172.21.15.111:0/2941131358 shutdown_connections 2023-12-19T08:06:20.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.073+0000 7f9906a8e700 1 -- 172.21.15.111:0/2941131358 wait complete. 2023-12-19T08:06:20.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 Processor -- start 2023-12-19T08:06:20.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 -- start start 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 0x7f9900156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f990006b210 0x7f990014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99001539c0 con 0x7f990006e1d0 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990014f4f0 con 0x7f990006b210 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9906a8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99000acbd0 con 0x7f990006ad70 2023-12-19T08:06:20.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52564/0 (socket says 172.21.15.111:52564) 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 -- 172.21.15.111:0/3452127338 learned_addr learned my addr 172.21.15.111:0/3452127338 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990528b700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f990006b210 0x7f990014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9905a8c700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 0x7f9900156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 -- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 msgr2=0x7f9900156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 0x7f9900156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 -- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f990006b210 msgr2=0x7f990014efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f990006b210 0x7f990014efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f990628d700 1 -- 172.21.15.111:0/3452127338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc009420 con 0x7f990006e1d0 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.077+0000 7f9905a8c700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 0x7f9900156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:20.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f990628d700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c8e0 tx=0x7f98f8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f800ce80 con 0x7f990006e1d0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f800a7e0 con 0x7f990006e1d0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f800ab80 con 0x7f990006e1d0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99000ace90 con 0x7f990006e1d0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99000ad450 con 0x7f990006e1d0 2023-12-19T08:06:20.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.081+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99000a8850 con 0x7f990006e1d0 2023-12-19T08:06:20.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.085+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f98f8012420 con 0x7f990006e1d0 2023-12-19T08:06:20.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.085+0000 7f98f5ffb700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 0x7f98e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.085+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f98f807cca0 con 0x7f990006e1d0 2023-12-19T08:06:20.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.085+0000 7f9905a8c700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 0x7f98e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.089+0000 7f9905a8c700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 0x7f98e4062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98fc003080 tx=0x7f98fc017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.093+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f99000a8850 con 0x7f990006e1d0 2023-12-19T08:06:20.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.269+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f99000a8850 con 0x7f990006e1d0 2023-12-19T08:06:20.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.269+0000 7f98f5ffb700 1 -- 172.21.15.111:0/3452127338 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v140) v1 ==== 84+0+65 (secure 0 0 0) 0x7f99000a8850 con 0x7f990006e1d0 2023-12-19T08:06:20.274 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:20.275 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:06:20.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 msgr2=0x7f98e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 0x7f98e4062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98fc003080 tx=0x7f98fc017000 comp rx=0 tx=0).stop 2023-12-19T08:06:20.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 msgr2=0x7f99000ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c8e0 tx=0x7f98f8001710 comp rx=0 tx=0).stop 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 shutdown_connections 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98e4060670 0x7f98e4062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f990006b210 0x7f990014efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990006e1d0 0x7f99000ac690 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 --2- 172.21.15.111:0/3452127338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f990006ad70 0x7f9900156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 >> 172.21.15.111:0/3452127338 conn(0x7f9900067bb0 msgr2=0x7f99000cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 shutdown_connections 2023-12-19T08:06:20.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.273+0000 7f9906a8e700 1 -- 172.21.15.111:0/3452127338 wait complete. 2023-12-19T08:06:20.298 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T08:06:20.298 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T08:06:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 Processor -- start 2023-12-19T08:06:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 -- start start 2023-12-19T08:06:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015d360 0x7fc66006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc66006dde0 0x7fc6600af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006b550 con 0x7fc66015cf90 2023-12-19T08:06:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66006af40 con 0x7fc66006dde0 2023-12-19T08:06:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.461+0000 7fc66720c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600af5a0 con 0x7fc66015d360 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc66006dde0 0x7fc6600af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc66006dde0 0x7fc6600af060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52576/0 (socket says 172.21.15.111:52576) 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 -- 172.21.15.111:0/1154986084 learned_addr learned my addr 172.21.15.111:0/1154986084 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc665a09700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015d360 0x7fc66006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 -- 172.21.15.111:0/1154986084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015d360 msgr2=0x7fc66006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015d360 0x7fc66006b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 -- 172.21.15.111:0/1154986084 >> v1:172.21.15.143:6789/0 conn(0x7fc66015cf90 legacy=0x7fc6600ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 -- 172.21.15.111:0/1154986084 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600ae0c0 con 0x7fc66006dde0 2023-12-19T08:06:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc666a0b700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc6600af060 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fc658009f20 tx=0x7fc658009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b39f4a087043cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc665208700 1 -- 172.21.15.111:0/1154986084 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800e070 con 0x7fc66006dde0 2023-12-19T08:06:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc665208700 1 -- 172.21.15.111:0/1154986084 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800ba40 con 0x7fc66006dde0 2023-12-19T08:06:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc665208700 1 -- 172.21.15.111:0/1154986084 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580092e0 con 0x7fc66006dde0 2023-12-19T08:06:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 -- 172.21.15.111:0/1154986084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 msgr2=0x7fc6600af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc6600af060 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fc658009f20 tx=0x7fc658009840 comp rx=0 tx=0).stop 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 -- 172.21.15.111:0/1154986084 shutdown_connections 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc6600af060 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 --2- 172.21.15.111:0/1154986084 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015d360 0x7fc66006b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 -- 172.21.15.111:0/1154986084 >> 172.21.15.111:0/1154986084 conn(0x7fc660067bb0 msgr2=0x7fc6600c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 -- 172.21.15.111:0/1154986084 shutdown_connections 2023-12-19T08:06:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.465+0000 7fc66720c700 1 -- 172.21.15.111:0/1154986084 wait complete. 2023-12-19T08:06:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.469+0000 7fc66720c700 1 Processor -- start 2023-12-19T08:06:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.469+0000 7fc66720c700 1 -- start start 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015cf90 0x7fc66015b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc66015d360 0x7fc66015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015bac0 con 0x7fc66006dde0 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015e800 con 0x7fc66015d360 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015bd00 con 0x7fc66015cf90 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52586/0 (socket says 172.21.15.111:52586) 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 -- 172.21.15.111:0/1920939697 learned_addr learned my addr 172.21.15.111:0/1920939697 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc666a0b700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc66015d360 0x7fc66015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc665a09700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015cf90 0x7fc66015b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 -- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015cf90 msgr2=0x7fc66015b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015cf90 0x7fc66015b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 -- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc66015d360 msgr2=0x7fc66015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc66015d360 0x7fc66015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 -- 172.21.15.111:0/1920939697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009ae0 con 0x7fc66006dde0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66620a700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c5d0 tx=0x7fc65c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c011590 con 0x7fc66006dde0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c011710 con 0x7fc66006dde0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66015ea50 con 0x7fc66006dde0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c011ad0 con 0x7fc66006dde0 2023-12-19T08:06:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.473+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66015f010 con 0x7fc66006dde0 2023-12-19T08:06:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.477+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66016dba0 con 0x7fc66006dde0 2023-12-19T08:06:20.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.477+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc65c01c460 con 0x7fc66006dde0 2023-12-19T08:06:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.477+0000 7fc6567fc700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 0x7fc644062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.477+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fc65c07c5f0 con 0x7fc66006dde0 2023-12-19T08:06:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.481+0000 7fc665a09700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 0x7fc644062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.481+0000 7fc665a09700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 0x7fc644062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6600ca960 tx=0x7fc650008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.485+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc65c04ca20 con 0x7fc66006dde0 2023-12-19T08:06:20.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:20.653+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc66004be40 con 0x7fc66006dde0 2023-12-19T08:06:21.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.253+0000 7fc6567fc700 1 -- 172.21.15.111:0/1920939697 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v141) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc65c050040 con 0x7fc66006dde0 2023-12-19T08:06:21.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 msgr2=0x7fc644062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 0x7fc644062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6600ca960 tx=0x7fc650008040 comp rx=0 tx=0).stop 2023-12-19T08:06:21.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 msgr2=0x7fc66016d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c5d0 tx=0x7fc65c007f60 comp rx=0 tx=0).stop 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 shutdown_connections 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc644060670 0x7fc644062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc66015d360 0x7fc66015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc66015cf90 0x7fc66015b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 --2- 172.21.15.111:0/1920939697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc66006dde0 0x7fc66016d600 secure :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c5d0 tx=0x7fc65c007f60 comp rx=0 tx=0).stop 2023-12-19T08:06:21.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.257+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 >> 172.21.15.111:0/1920939697 conn(0x7fc660067bb0 msgr2=0x7fc66006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:21.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.261+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 shutdown_connections 2023-12-19T08:06:21.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.261+0000 7fc66720c700 1 -- 172.21.15.111:0/1920939697 wait complete. 2023-12-19T08:06:21.268 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T08:06:21.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T08:06:21.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.449+0000 7f43aaae3700 1 Processor -- start 2023-12-19T08:06:21.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.449+0000 7f43aaae3700 1 -- start start 2023-12-19T08:06:21.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aaae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 0x7f43a4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aaae3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43a41595a0 0x7f43a4159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aaae3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a406c850 con 0x7f43a4158c90 2023-12-19T08:06:21.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aaae3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a40cb650 con 0x7f43a41595a0 2023-12-19T08:06:21.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aaae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a406b470 con 0x7f43a406aeb0 2023-12-19T08:06:21.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 0x7f43a4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a9ae1700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f43a4158c90 0x7f43a40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:48236/0 (socket says 172.21.15.111:48236) 2023-12-19T08:06:21.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a9ae1700 1 -- 172.21.15.111:0/3664316434 learned_addr learned my addr 172.21.15.111:0/3664316434 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:21.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43aa2e2700 1 --2- 172.21.15.111:0/3664316434 >> v2:172.21.15.143:3300/0 conn(0x7f43a41595a0 0x7f43a4159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a8adf700 1 -- 172.21.15.111:0/3664316434 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916257718 0 0) 0x7f43a406c850 con 0x7f43a4158c90 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a8adf700 1 -- 172.21.15.111:0/3664316434 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4390003570 con 0x7f43a4158c90 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 -- 172.21.15.111:0/3664316434 >> v2:172.21.15.143:3300/0 conn(0x7f43a41595a0 msgr2=0x7f43a4159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 --2- 172.21.15.111:0/3664316434 >> v2:172.21.15.143:3300/0 conn(0x7f43a41595a0 0x7f43a4159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 -- 172.21.15.111:0/3664316434 >> v1:172.21.15.143:6789/0 conn(0x7f43a4158c90 legacy=0x7f43a40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 -- 172.21.15.111:0/3664316434 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a406b750 con 0x7f43a406aeb0 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.453+0000 7f43a92e0700 1 --2- 172.21.15.111:0/3664316434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 0x7f43a4159060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f43940098e0 tx=0x7f43940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2291f19cb07946 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43a8adf700 1 -- 172.21.15.111:0/3664316434 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394011630 con 0x7f43a406aeb0 2023-12-19T08:06:21.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43a8adf700 1 -- 172.21.15.111:0/3664316434 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394011790 con 0x7f43a406aeb0 2023-12-19T08:06:21.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43a8adf700 1 -- 172.21.15.111:0/3664316434 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394011b30 con 0x7f43a406aeb0 2023-12-19T08:06:21.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 -- 172.21.15.111:0/3664316434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 msgr2=0x7f43a4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 --2- 172.21.15.111:0/3664316434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 0x7f43a4159060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f43940098e0 tx=0x7f43940091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 -- 172.21.15.111:0/3664316434 shutdown_connections 2023-12-19T08:06:21.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 --2- 172.21.15.111:0/3664316434 >> v2:172.21.15.143:3300/0 conn(0x7f43a41595a0 0x7f43a4159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 --2- 172.21.15.111:0/3664316434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a406aeb0 0x7f43a4159060 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 -- 172.21.15.111:0/3664316434 >> 172.21.15.111:0/3664316434 conn(0x7f43a4067bb0 msgr2=0x7f43a40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:21.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 -- 172.21.15.111:0/3664316434 shutdown_connections 2023-12-19T08:06:21.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.457+0000 7f43aaae3700 1 -- 172.21.15.111:0/3664316434 wait complete. 2023-12-19T08:06:21.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 Processor -- start 2023-12-19T08:06:21.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 -- start start 2023-12-19T08:06:21.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43a406aeb0 0x7f43a41579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 0x7f43a4156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40b3d30 con 0x7f43a4158c90 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4155530 con 0x7f43a406aeb0 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aaae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40b3f80 con 0x7f43a41595a0 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52592/0 (socket says 172.21.15.111:52592) 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 -- 172.21.15.111:0/2536332778 learned_addr learned my addr 172.21.15.111:0/2536332778 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a9ae1700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43a406aeb0 0x7f43a41579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aa2e2700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 0x7f43a4156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 -- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 msgr2=0x7f43a4156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 0x7f43a4156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 -- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43a406aeb0 msgr2=0x7f43a41579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:21.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43a406aeb0 0x7f43a41579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:21.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43a92e0700 1 -- 172.21.15.111:0/2536332778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009420 con 0x7f43a4158c90 2023-12-19T08:06:21.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.461+0000 7f43aa2e2700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 0x7f43a4156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:21.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f43a92e0700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f43a406e2e0 tx=0x7f4394017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:21.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394011b10 con 0x7f43a4158c90 2023-12-19T08:06:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43940253f0 con 0x7f43a4158c90 2023-12-19T08:06:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43940257b0 con 0x7f43a4158c90 2023-12-19T08:06:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40b4260 con 0x7f43a4158c90 2023-12-19T08:06:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a40b4820 con 0x7f43a4158c90 2023-12-19T08:06:21.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.465+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a404be40 con 0x7f43a4158c90 2023-12-19T08:06:21.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.469+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4394025ba0 con 0x7f43a4158c90 2023-12-19T08:06:21.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.469+0000 7f4399ffb700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 0x7f4388062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.469+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f439407e3c0 con 0x7f43a4158c90 2023-12-19T08:06:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.469+0000 7f43a9ae1700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 0x7f4388062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:21.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.469+0000 7f43a9ae1700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 0x7f4388062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43a00058d0 tx=0x7f43a0009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:21.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.473+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f439404e720 con 0x7f43a4158c90 2023-12-19T08:06:21.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:21.641+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f43a406ba30 con 0x7f43a4158c90 2023-12-19T08:06:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.265+0000 7f4399ffb700 1 -- 172.21.15.111:0/2536332778 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v142) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4394051d40 con 0x7f43a4158c90 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 msgr2=0x7f4388062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 0x7f4388062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43a00058d0 tx=0x7f43a0009040 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 msgr2=0x7f43a4169560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f43a406e2e0 tx=0x7f4394017000 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 shutdown_connections 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4388060670 0x7f4388062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43a41595a0 0x7f43a4156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43a4158c90 0x7f43a4169560 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 --2- 172.21.15.111:0/2536332778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43a406aeb0 0x7f43a41579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 >> 172.21.15.111:0/2536332778 conn(0x7f43a4067bb0 msgr2=0x7f43a40ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:22.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.281+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 shutdown_connections 2023-12-19T08:06:22.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.285+0000 7f43aaae3700 1 -- 172.21.15.111:0/2536332778 wait complete. 2023-12-19T08:06:22.298 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T08:06:22.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T08:06:22.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.477+0000 7f636e8e3700 1 Processor -- start 2023-12-19T08:06:22.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.477+0000 7f636e8e3700 1 -- start start 2023-12-19T08:06:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e8e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f63680caa10 0x7f6368159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e8e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680cbbb0 con 0x7f63680cc040 2023-12-19T08:06:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e8e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636806dc90 con 0x7f63680ce2d0 2023-12-19T08:06:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e8e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63680cafa0 con 0x7f63680caa10 2023-12-19T08:06:22.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e0e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f63680cc040 0x7f636806a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33020/0 (socket says 172.21.15.111:33020) 2023-12-19T08:06:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636e0e2700 1 -- 172.21.15.111:0/345575197 learned_addr learned my addr 172.21.15.111:0/345575197 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:22.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636d0e0700 1 --2- 172.21.15.111:0/345575197 >> v2:172.21.15.143:3300/0 conn(0x7f63680caa10 0x7f6368159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636c8df700 1 -- 172.21.15.111:0/345575197 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257508555 0 0) 0x7f63680cbbb0 con 0x7f63680cc040 2023-12-19T08:06:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636c8df700 1 -- 172.21.15.111:0/345575197 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6354003570 con 0x7f63680cc040 2023-12-19T08:06:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636d0e0700 1 -- 172.21.15.111:0/345575197 >> v1:172.21.15.143:6789/0 conn(0x7f63680ce2d0 legacy=0x7f6368158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636d0e0700 1 -- 172.21.15.111:0/345575197 >> v1:172.21.15.111:6789/0 conn(0x7f63680cc040 legacy=0x7f636806a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636d0e0700 1 -- 172.21.15.111:0/345575197 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63680cb280 con 0x7f63680caa10 2023-12-19T08:06:22.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636d0e0700 1 --2- 172.21.15.111:0/345575197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680caa10 0x7f6368159010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f63580098e0 tx=0x7f63580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3441f9f2de24608 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636c8df700 1 -- 172.21.15.111:0/345575197 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800b640 con 0x7f63680caa10 2023-12-19T08:06:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636c8df700 1 -- 172.21.15.111:0/345575197 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800b7a0 con 0x7f63680caa10 2023-12-19T08:06:22.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.481+0000 7f636c8df700 1 -- 172.21.15.111:0/345575197 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800bb40 con 0x7f63680caa10 2023-12-19T08:06:22.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 -- 172.21.15.111:0/345575197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680caa10 msgr2=0x7f6368159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 --2- 172.21.15.111:0/345575197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680caa10 0x7f6368159010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f63580098e0 tx=0x7f63580091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 -- 172.21.15.111:0/345575197 shutdown_connections 2023-12-19T08:06:22.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 --2- 172.21.15.111:0/345575197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680caa10 0x7f6368159010 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 -- 172.21.15.111:0/345575197 >> 172.21.15.111:0/345575197 conn(0x7f6368067bb0 msgr2=0x7f63680cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:22.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 -- 172.21.15.111:0/345575197 shutdown_connections 2023-12-19T08:06:22.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.485+0000 7f636e8e3700 1 -- 172.21.15.111:0/345575197 wait complete. 2023-12-19T08:06:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 Processor -- start 2023-12-19T08:06:22.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 -- start start 2023-12-19T08:06:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:22.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680cc040 0x7f63681557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63680ce2d0 0x7f63680b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63680c7f20 con 0x7f63680caa10 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63681590e0 con 0x7f63680cc040 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636815ba40 con 0x7f63680ce2d0 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636d8e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:22.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636d8e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52956/0 (socket says 172.21.15.111:52956) 2023-12-19T08:06:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636d8e1700 1 -- 172.21.15.111:0/582717933 learned_addr learned my addr 172.21.15.111:0/582717933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636e0e2700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63680ce2d0 0x7f63680b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.489+0000 7f636d0e0700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680cc040 0x7f63681557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 -- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63680ce2d0 msgr2=0x7f63680b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63680ce2d0 0x7f63680b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 -- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680cc040 msgr2=0x7f63681557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:22.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680cc040 0x7f63681557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:22.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 -- 172.21.15.111:0/582717933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358009420 con 0x7f63680caa10 2023-12-19T08:06:22.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636d8e1700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f636400c510 tx=0x7f636400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364011590 con 0x7f63680caa10 2023-12-19T08:06:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364011710 con 0x7f63680caa10 2023-12-19T08:06:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6368159330 con 0x7f63680caa10 2023-12-19T08:06:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364011ad0 con 0x7f63680caa10 2023-12-19T08:06:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63681596c0 con 0x7f63680caa10 2023-12-19T08:06:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.493+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636804be40 con 0x7f63680caa10 2023-12-19T08:06:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.497+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f636401b3b0 con 0x7f63680caa10 2023-12-19T08:06:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.497+0000 7f635dffb700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 0x7f634c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.497+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f636407bfc0 con 0x7f63680caa10 2023-12-19T08:06:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.501+0000 7f636d0e0700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 0x7f634c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.501+0000 7f636d0e0700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 0x7f634c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63580093f0 tx=0x7f6358017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.505+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f636404c1a0 con 0x7f63680caa10 2023-12-19T08:06:22.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:22.713+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f63680cb560 con 0x7f63680caa10 2023-12-19T08:06:23.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.285+0000 7f635dffb700 1 -- 172.21.15.111:0/582717933 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v60) v1 ==== 174+0+0 (secure 0 0 0) 0x7f636404f7c0 con 0x7f63680caa10 2023-12-19T08:06:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.301+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 msgr2=0x7f634c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.301+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 0x7f634c062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63580093f0 tx=0x7f6358017000 comp rx=0 tx=0).stop 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.301+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 msgr2=0x7f63680c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.301+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f636400c510 tx=0x7f636400c820 comp rx=0 tx=0).stop 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 shutdown_connections 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f634c060670 0x7f634c062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63680cc040 0x7f63681557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63680ce2d0 0x7f63680b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 --2- 172.21.15.111:0/582717933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63680caa10 0x7f63680c79e0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 >> 172.21.15.111:0/582717933 conn(0x7f6368067bb0 msgr2=0x7f636815bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 shutdown_connections 2023-12-19T08:06:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.305+0000 7f636e8e3700 1 -- 172.21.15.111:0/582717933 wait complete. 2023-12-19T08:06:23.316 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 17 and data pool 18 2023-12-19T08:06:23.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:23.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 Processor -- start 2023-12-19T08:06:23.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 -- start start 2023-12-19T08:06:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 0x7f65140c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f651406b900 0x7f651406da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651406b470 con 0x7f651415cc20 2023-12-19T08:06:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651415d7b0 con 0x7f651406b900 2023-12-19T08:06:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.489+0000 7f651a88e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6514161d10 con 0x7f65140c6900 2023-12-19T08:06:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651988c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 0x7f65140c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f651406b900 0x7f651406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f651406b900 0x7f651406da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:58244/0 (socket says 172.21.15.111:58244) 2023-12-19T08:06:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 -- 172.21.15.111:0/228715953 learned_addr learned my addr 172.21.15.111:0/228715953 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651988c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 0x7f65140c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52884/0 (socket says 172.21.15.111:52884) 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a08d700 1 --1- 172.21.15.111:0/228715953 >> v1:172.21.15.143:6789/0 conn(0x7f651415cc20 0x7f651415d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:50430/0 (socket says 172.21.15.111:50430) 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 -- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 msgr2=0x7f65140c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 --2- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 0x7f65140c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 -- 172.21.15.111:0/228715953 >> v1:172.21.15.143:6789/0 conn(0x7f651415cc20 legacy=0x7f651415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 -- 172.21.15.111:0/228715953 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651406bd70 con 0x7f651406b900 2023-12-19T08:06:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651908b700 1 --2- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f651406b900 0x7f651406da00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f65040099e0 tx=0x7f6504009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afd46ad18612d69b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651888a700 1 -- 172.21.15.111:0/228715953 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650400b820 con 0x7f651406b900 2023-12-19T08:06:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651888a700 1 -- 172.21.15.111:0/228715953 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650400b980 con 0x7f651406b900 2023-12-19T08:06:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651888a700 1 -- 172.21.15.111:0/228715953 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650400bd20 con 0x7f651406b900 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 -- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f651406b900 msgr2=0x7f651406da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 --2- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f651406b900 0x7f651406da00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f65040099e0 tx=0x7f6504009250 comp rx=0 tx=0).stop 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 -- 172.21.15.111:0/228715953 shutdown_connections 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 --2- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f651406b900 0x7f651406da00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 --2- 172.21.15.111:0/228715953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65140c6900 0x7f65140c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 -- 172.21.15.111:0/228715953 >> 172.21.15.111:0/228715953 conn(0x7f6514067bb0 msgr2=0x7f65140cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 -- 172.21.15.111:0/228715953 shutdown_connections 2023-12-19T08:06:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.493+0000 7f651a88e700 1 -- 172.21.15.111:0/228715953 wait complete. 2023-12-19T08:06:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 Processor -- start 2023-12-19T08:06:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 -- start start 2023-12-19T08:06:23.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651406b900 0x7f651414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65140c6900 0x7f65140b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651415ec50 con 0x7f651415cc20 2023-12-19T08:06:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651415ef30 con 0x7f65140c6900 2023-12-19T08:06:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a88e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651415f210 con 0x7f651406b900 2023-12-19T08:06:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52970/0 (socket says 172.21.15.111:52970) 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651a08d700 1 -- 172.21.15.111:0/3965325190 learned_addr learned my addr 172.21.15.111:0/3965325190 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651988c700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651406b900 0x7f651414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.497+0000 7f651908b700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65140c6900 0x7f65140b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 -- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651406b900 msgr2=0x7f651414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651406b900 0x7f651414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 -- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65140c6900 msgr2=0x7f65140b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65140c6900 0x7f65140b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 -- 172.21.15.111:0/3965325190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65040094f0 con 0x7f651415cc20 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a08d700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f650c00cc70 tx=0x7f650c001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c005df0 con 0x7f651415cc20 2023-12-19T08:06:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c00a5b0 con 0x7f651415cc20 2023-12-19T08:06:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651415f4f0 con 0x7f651415cc20 2023-12-19T08:06:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00a930 con 0x7f651415cc20 2023-12-19T08:06:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65141502a0 con 0x7f651415cc20 2023-12-19T08:06:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.501+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651404be40 con 0x7f651415cc20 2023-12-19T08:06:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.505+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f650c00ace0 con 0x7f651415cc20 2023-12-19T08:06:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.505+0000 7f6509ffb700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 0x7f64f8062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.505+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f650c07cc90 con 0x7f651415cc20 2023-12-19T08:06:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.505+0000 7f651988c700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 0x7f64f8062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.505+0000 7f651988c700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 0x7f64f8062ba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6510009db0 tx=0x7f65100091a0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.513+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f650c04cdf0 con 0x7f651415cc20 2023-12-19T08:06:23.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.689+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f651406a580 con 0x7f651415cc20 2023-12-19T08:06:23.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.689+0000 7f6509ffb700 1 -- 172.21.15.111:0/3965325190 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16464 (secure 0 0 0) 0x7f650c050410 con 0x7f651415cc20 2023-12-19T08:06:23.694 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:23.695 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":143,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:22.716436+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:23.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 msgr2=0x7f64f8062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 0x7f64f8062ba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6510009db0 tx=0x7f65100091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 msgr2=0x7f6514153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f650c00cc70 tx=0x7f650c001a70 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 shutdown_connections 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f80606f0 0x7f64f8062ba0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65140c6900 0x7f65140b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f651415cc20 0x7f6514153ae0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 --2- 172.21.15.111:0/3965325190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651406b900 0x7f651414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 >> 172.21.15.111:0/3965325190 conn(0x7f6514067bb0 msgr2=0x7f6514159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 shutdown_connections 2023-12-19T08:06:23.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.693+0000 7f651a88e700 1 -- 172.21.15.111:0/3965325190 wait complete. 2023-12-19T08:06:23.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:23.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.869+0000 7f9d43f85700 1 Processor -- start 2023-12-19T08:06:23.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.869+0000 7f9d43f85700 1 -- start start 2023-12-19T08:06:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d43f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cea90 0x7f9d3c0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d43f85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d3c0cc330 0x7f9d3c068a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d43f85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0cbea0 con 0x7f9d3c0cc780 2023-12-19T08:06:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d43f85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c0cb030 con 0x7f9d3c0cc330 2023-12-19T08:06:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d43f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0cb2d0 con 0x7f9d3c0cea90 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d3c0cc330 0x7f9d3c068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d3c0cc330 0x7f9d3c068a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52978/0 (socket says 172.21.15.111:52978) 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 -- 172.21.15.111:0/3697994470 learned_addr learned my addr 172.21.15.111:0/3697994470 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d42522700 1 --1- 172.21.15.111:0/3697994470 >> v1:172.21.15.111:6789/0 conn(0x7f9d3c0cc780 0x7f9d3c142720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33028/0 (socket says 172.21.15.111:33028) 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41d21700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cea90 0x7f9d3c0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 -- 172.21.15.111:0/3697994470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cea90 msgr2=0x7f9d3c0cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cea90 0x7f9d3c0cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 -- 172.21.15.111:0/3697994470 >> v1:172.21.15.111:6789/0 conn(0x7f9d3c0cc780 legacy=0x7f9d3c142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 -- 172.21.15.111:0/3697994470 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c067e00 con 0x7f9d3c0cc330 2023-12-19T08:06:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.873+0000 7f9d41520700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc330 0x7f9d3c068a70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0099b0 tx=0x7f9d2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df18a34d18b3523c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d40d1f700 1 -- 172.21.15.111:0/3697994470 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c00b860 con 0x7f9d3c0cc330 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d40d1f700 1 -- 172.21.15.111:0/3697994470 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c00b9c0 con 0x7f9d3c0cc330 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d40d1f700 1 -- 172.21.15.111:0/3697994470 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c00bd60 con 0x7f9d3c0cc330 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- 172.21.15.111:0/3697994470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc330 msgr2=0x7f9d3c068a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc330 0x7f9d3c068a70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0099b0 tx=0x7f9d2c009250 comp rx=0 tx=0).stop 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- 172.21.15.111:0/3697994470 shutdown_connections 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc330 0x7f9d3c068a70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 --2- 172.21.15.111:0/3697994470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cea90 0x7f9d3c0cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- 172.21.15.111:0/3697994470 >> 172.21.15.111:0/3697994470 conn(0x7f9d3c0cd650 msgr2=0x7f9d3c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- 172.21.15.111:0/3697994470 shutdown_connections 2023-12-19T08:06:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- 172.21.15.111:0/3697994470 wait complete. 2023-12-19T08:06:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 Processor -- start 2023-12-19T08:06:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.877+0000 7f9d43f85700 1 -- start start 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cc330 0x7f9d3c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 0x7f9d3c0c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c140fd0 con 0x7f9d3c0cc780 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0c78f0 con 0x7f9d3c0cea90 2023-12-19T08:06:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c15f750 con 0x7f9d3c0cc330 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52986/0 (socket says 172.21.15.111:52986) 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 -- 172.21.15.111:0/1434825300 learned_addr learned my addr 172.21.15.111:0/1434825300 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d42522700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 0x7f9d3c0c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41d21700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cc330 0x7f9d3c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 -- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cc330 msgr2=0x7f9d3c15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cc330 0x7f9d3c15e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 -- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 msgr2=0x7f9d3c0c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 0x7f9d3c0c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 -- 172.21.15.111:0/1434825300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c0094f0 con 0x7f9d3c0cc780 2023-12-19T08:06:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d42522700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 0x7f9d3c0c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d41520700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c06a0f0 tx=0x7f9d2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c024440 con 0x7f9d3c0cc780 2023-12-19T08:06:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c0245c0 con 0x7f9d3c0cc780 2023-12-19T08:06:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c15f470 con 0x7f9d3c0cc780 2023-12-19T08:06:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c024980 con 0x7f9d3c0cc780 2023-12-19T08:06:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.881+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c0c7580 con 0x7f9d3c0cc780 2023-12-19T08:06:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.885+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c0c7a80 con 0x7f9d3c0cc780 2023-12-19T08:06:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.885+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9d2c01a420 con 0x7f9d3c0cc780 2023-12-19T08:06:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.889+0000 7f9d31ffb700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 0x7f9d20062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.889+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f9d2c07eaa0 con 0x7f9d3c0cc780 2023-12-19T08:06:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.889+0000 7f9d41d21700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 0x7f9d20062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.889+0000 7f9d41d21700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 0x7f9d20062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d38006fd0 tx=0x7f9d38008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:23.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:23.893+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d2c04eb80 con 0x7f9d3c0cc780 2023-12-19T08:06:24.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.061+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d3c071120 con 0x7f9d3c0cc780 2023-12-19T08:06:24.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.061+0000 7f9d31ffb700 1 -- 172.21.15.111:0/1434825300 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16464 (secure 0 0 0) 0x7f9d2c0521a0 con 0x7f9d3c0cc780 2023-12-19T08:06:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:24.068 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":143,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:22.716436+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 msgr2=0x7f9d20062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 0x7f9d20062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d38006fd0 tx=0x7f9d38008040 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 msgr2=0x7f9d3c15eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c06a0f0 tx=0x7f9d2c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 shutdown_connections 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9d20060670 0x7f9d20062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d3c0cea90 0x7f9d3c0c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d3c0cc780 0x7f9d3c15eda0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 --2- 172.21.15.111:0/1434825300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d3c0cc330 0x7f9d3c15e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 >> 172.21.15.111:0/1434825300 conn(0x7f9d3c0cd650 msgr2=0x7f9d3c0c6650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:24.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 shutdown_connections 2023-12-19T08:06:24.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.065+0000 7f9d43f85700 1 -- 172.21.15.111:0/1434825300 wait complete. 2023-12-19T08:06:24.092 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T08:06:24.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.253+0000 7f432562e700 1 Processor -- start 2023-12-19T08:06:24.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.253+0000 7f432562e700 1 -- start start 2023-12-19T08:06:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f432562e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43200cecd0 0x7f432006b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f432562e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43200cbdf0 con 0x7f43200cb970 2023-12-19T08:06:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f432562e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43200cc0d0 con 0x7f4320158e20 2023-12-19T08:06:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f432562e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43200cc3b0 con 0x7f43200cecd0 2023-12-19T08:06:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431e7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f43200cb970 0x7f432006acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33030/0 (socket says 172.21.15.111:33030) 2023-12-19T08:06:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431e7fc700 1 -- 172.21.15.111:0/2592410520 learned_addr learned my addr 172.21.15.111:0/2592410520 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431f7fe700 1 --2- 172.21.15.111:0/2592410520 >> v2:172.21.15.143:3300/0 conn(0x7f43200cecd0 0x7f432006b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646378470 0 0) 0x7f43200cbdf0 con 0x7f43200cb970 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4308003570 con 0x7f43200cb970 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985205246 0 0) 0x7f43200cc0d0 con 0x7f4320158e20 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43200cbdf0 con 0x7f4320158e20 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431f7fe700 1 -- 172.21.15.111:0/2592410520 >> v1:172.21.15.143:6789/0 conn(0x7f4320158e20 legacy=0x7f43200ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431f7fe700 1 -- 172.21.15.111:0/2592410520 >> v1:172.21.15.111:6789/0 conn(0x7f43200cb970 legacy=0x7f432006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431f7fe700 1 -- 172.21.15.111:0/2592410520 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43200cc690 con 0x7f43200cecd0 2023-12-19T08:06:24.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.257+0000 7f431f7fe700 1 --2- 172.21.15.111:0/2592410520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432006b3d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4318009e90 tx=0x7f4318009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6813b2e0aa3c4965 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431800e070 con 0x7f43200cecd0 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431800bb90 con 0x7f43200cecd0 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f431dffb700 1 -- 172.21.15.111:0/2592410520 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318012430 con 0x7f43200cecd0 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 -- 172.21.15.111:0/2592410520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 msgr2=0x7f432006b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 --2- 172.21.15.111:0/2592410520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432006b3d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4318009e90 tx=0x7f4318009780 comp rx=0 tx=0).stop 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 -- 172.21.15.111:0/2592410520 shutdown_connections 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 --2- 172.21.15.111:0/2592410520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432006b3d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 -- 172.21.15.111:0/2592410520 >> 172.21.15.111:0/2592410520 conn(0x7f4320067bb0 msgr2=0x7f43200d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 -- 172.21.15.111:0/2592410520 shutdown_connections 2023-12-19T08:06:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.261+0000 7f432562e700 1 -- 172.21.15.111:0/2592410520 wait complete. 2023-12-19T08:06:24.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 Processor -- start 2023-12-19T08:06:24.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 -- start start 2023-12-19T08:06:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4320158e20 0x7f4320155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320156890 con 0x7f4320156410 2023-12-19T08:06:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43200c9d30 con 0x7f43200cecd0 2023-12-19T08:06:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f432562e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432014fff0 con 0x7f4320158e20 2023-12-19T08:06:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53002/0 (socket says 172.21.15.111:53002) 2023-12-19T08:06:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 -- 172.21.15.111:0/1391395722 learned_addr learned my addr 172.21.15.111:0/1391395722 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431effd700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431e7fc700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4320158e20 0x7f4320155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 -- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4320158e20 msgr2=0x7f4320155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4320158e20 0x7f4320155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 -- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 msgr2=0x7f432014bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432014bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 -- 172.21.15.111:0/1391395722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318009a20 con 0x7f4320156410 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.265+0000 7f431f7fe700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f4318013000 tx=0x7f4318019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431801b420 con 0x7f4320156410 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431801b5e0 con 0x7f4320156410 2023-12-19T08:06:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431801b9a0 con 0x7f4320156410 2023-12-19T08:06:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43200c9120 con 0x7f4320156410 2023-12-19T08:06:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43200c96e0 con 0x7f4320156410 2023-12-19T08:06:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43200c9ee0 con 0x7f4320156410 2023-12-19T08:06:24.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.269+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f431801bd10 con 0x7f4320156410 2023-12-19T08:06:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.273+0000 7f4306ffd700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 0x7f430c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.273+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f431807e670 con 0x7f4320156410 2023-12-19T08:06:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.273+0000 7f431effd700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 0x7f430c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:24.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.273+0000 7f431effd700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 0x7f430c062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4310007fd0 tx=0x7f4310007d50 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:24.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.281+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43200c9ee0 con 0x7f4320156410 2023-12-19T08:06:24.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:24.489+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f43200c9ea0 con 0x7f4320156410 2023-12-19T08:06:25.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.309+0000 7f4306ffd700 1 -- 172.21.15.111:0/1391395722 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4318052180 con 0x7f4320156410 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 msgr2=0x7f430c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 0x7f430c062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4310007fd0 tx=0x7f4310007d50 comp rx=0 tx=0).stop 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 msgr2=0x7f43200c84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f4318013000 tx=0x7f4318019000 comp rx=0 tx=0).stop 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 shutdown_connections 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f430c0606a0 0x7f430c062b50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4320156410 0x7f43200c84b0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4320158e20 0x7f4320155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 --2- 172.21.15.111:0/1391395722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43200cecd0 0x7f432014bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 >> 172.21.15.111:0/1391395722 conn(0x7f4320067bb0 msgr2=0x7f432014caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 shutdown_connections 2023-12-19T08:06:25.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.313+0000 7f432562e700 1 -- 172.21.15.111:0/1391395722 wait complete. 2023-12-19T08:06:25.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T08:06:25.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.505+0000 7f2c5ca6b700 1 Processor -- start 2023-12-19T08:06:25.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.505+0000 7f2c5ca6b700 1 -- start start 2023-12-19T08:06:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c5ca6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 0x7f2c5815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c5ca6b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c580adab0 0x7f2c580adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c5ca6b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5815ae30 con 0x7f2c5806baa0 2023-12-19T08:06:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c5ca6b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c580abf70 con 0x7f2c580adab0 2023-12-19T08:06:25.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c5ca6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58154b50 con 0x7f2c58154780 2023-12-19T08:06:25.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c56ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2c5806baa0 0x7f2c58157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33044/0 (socket says 172.21.15.111:33044) 2023-12-19T08:06:25.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c56ffd700 1 -- 172.21.15.111:0/4044174642 learned_addr learned my addr 172.21.15.111:0/4044174642 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:25.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c577fe700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 0x7f2c5815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.509+0000 7f2c57fff700 1 --2- 172.21.15.111:0/4044174642 >> v2:172.21.15.143:3300/0 conn(0x7f2c580adab0 0x7f2c580adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c567fc700 1 -- 172.21.15.111:0/4044174642 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486834685 0 0) 0x7f2c5815ae30 con 0x7f2c5806baa0 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c567fc700 1 -- 172.21.15.111:0/4044174642 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c40003570 con 0x7f2c5806baa0 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c57fff700 1 -- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 msgr2=0x7f2c5815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c57fff700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 0x7f2c5815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c57fff700 1 -- 172.21.15.111:0/4044174642 >> v1:172.21.15.111:6789/0 conn(0x7f2c5806baa0 legacy=0x7f2c58157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c57fff700 1 -- 172.21.15.111:0/4044174642 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580ac210 con 0x7f2c580adab0 2023-12-19T08:06:25.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c577fe700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 0x7f2c5815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T08:06:25.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c57fff700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c580adab0 0x7f2c580adf10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e60 tx=0x7f2c50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb714921a73ae81 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:25.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c567fc700 1 -- 172.21.15.111:0/4044174642 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5000e070 con 0x7f2c580adab0 2023-12-19T08:06:25.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c567fc700 1 -- 172.21.15.111:0/4044174642 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5000bab0 con 0x7f2c580adab0 2023-12-19T08:06:25.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c567fc700 1 -- 172.21.15.111:0/4044174642 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5000be70 con 0x7f2c580adab0 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c580adab0 msgr2=0x7f2c580adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c580adab0 0x7f2c580adf10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e60 tx=0x7f2c50009780 comp rx=0 tx=0).stop 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/4044174642 shutdown_connections 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c580adab0 0x7f2c580adf10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/4044174642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c58154780 0x7f2c5815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/4044174642 >> 172.21.15.111:0/4044174642 conn(0x7f2c58067b50 msgr2=0x7f2c580cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/4044174642 shutdown_connections 2023-12-19T08:06:25.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.513+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/4044174642 wait complete. 2023-12-19T08:06:25.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.517+0000 7f2c5ca6b700 1 Processor -- start 2023-12-19T08:06:25.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.517+0000 7f2c5ca6b700 1 -- start start 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 0x7f2c580b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c580adab0 0x7f2c58159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 0x7f2c581667f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58167800 con 0x7f2c58154780 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58166d30 con 0x7f2c5806baa0 2023-12-19T08:06:25.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581584d0 con 0x7f2c580adab0 2023-12-19T08:06:25.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c57fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 0x7f2c581667f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c57fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 0x7f2c581667f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53004/0 (socket says 172.21.15.111:53004) 2023-12-19T08:06:25.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c57fff700 1 -- 172.21.15.111:0/246052650 learned_addr learned my addr 172.21.15.111:0/246052650 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 0x7f2c580b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c56ffd700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c580adab0 0x7f2c58159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 -- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c580adab0 msgr2=0x7f2c58159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c580adab0 0x7f2c58159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 -- 172.21.15.111:0/246052650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 msgr2=0x7f2c581667f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 0x7f2c581667f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:25.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 -- 172.21.15.111:0/246052650 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c48009420 con 0x7f2c5806baa0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c577fe700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 0x7f2c580b1570 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800ecb0 tx=0x7f2c480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800c850 con 0x7f2c5806baa0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4800c9b0 con 0x7f2c5806baa0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c50009a20 con 0x7f2c5806baa0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800cd50 con 0x7f2c5806baa0 2023-12-19T08:06:25.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.521+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c58167240 con 0x7f2c5806baa0 2023-12-19T08:06:25.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.525+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c581582a0 con 0x7f2c5806baa0 2023-12-19T08:06:25.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.525+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2c48013680 con 0x7f2c5806baa0 2023-12-19T08:06:25.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.525+0000 7f2c3f7fe700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 0x7f2c44062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:25.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.525+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f2c4807cf30 con 0x7f2c5806baa0 2023-12-19T08:06:25.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.525+0000 7f2c56ffd700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 0x7f2c44062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:25.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.529+0000 7f2c56ffd700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 0x7f2c44062af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c580c9910 tx=0x7f2c4c009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:25.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.537+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c4804d010 con 0x7f2c5806baa0 2023-12-19T08:06:25.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:25.741+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2c5804be40 con 0x7f2c5806baa0 2023-12-19T08:06:26.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.325+0000 7f2c3f7fe700 1 -- 172.21.15.111:0/246052650 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2c48050630 con 0x7f2c5806baa0 2023-12-19T08:06:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 msgr2=0x7f2c44062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 0x7f2c44062af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c580c9910 tx=0x7f2c4c009040 comp rx=0 tx=0).stop 2023-12-19T08:06:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 msgr2=0x7f2c580b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 0x7f2c580b1570 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800ecb0 tx=0x7f2c480091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 shutdown_connections 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c44060640 0x7f2c44062af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c58154780 0x7f2c581667f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c580adab0 0x7f2c58159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 --2- 172.21.15.111:0/246052650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5806baa0 0x7f2c580b1570 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 >> 172.21.15.111:0/246052650 conn(0x7f2c58067b50 msgr2=0x7f2c580c89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 shutdown_connections 2023-12-19T08:06:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.333+0000 7f2c5ca6b700 1 -- 172.21.15.111:0/246052650 wait complete. 2023-12-19T08:06:26.359 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.529+0000 7f6bfb0f0700 1 Processor -- start 2023-12-19T08:06:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 -- start start 2023-12-19T08:06:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bf4140710 0x7f6bf40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140ae0 0x7f6bf40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf406e150 con 0x7f6bf40c75e0 2023-12-19T08:06:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf406d540 con 0x7f6bf4140710 2023-12-19T08:06:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bfb0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf406d820 con 0x7f6bf4140ae0 2023-12-19T08:06:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bf4140710 0x7f6bf40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bf4140710 0x7f6bf40c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53020/0 (socket says 172.21.15.111:53020) 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/1133834762 learned_addr learned my addr 172.21.15.111:0/1133834762 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf968d700 1 --1- 172.21.15.111:0/1133834762 >> v1:172.21.15.111:6789/0 conn(0x7f6bf40c75e0 0x7f6bf406a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33060/0 (socket says 172.21.15.111:33060) 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf3fff700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140ae0 0x7f6bf40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/1133834762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140ae0 msgr2=0x7f6bf40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140ae0 0x7f6bf40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/1133834762 >> v1:172.21.15.111:6789/0 conn(0x7f6bf40c75e0 legacy=0x7f6bf406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/1133834762 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf406db00 con 0x7f6bf4140710 2023-12-19T08:06:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.533+0000 7f6bf8e8c700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf4140710 0x7f6bf40c70a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6be80099b0 tx=0x7f6be8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1baec068ecf9b766 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bf37fe700 1 -- 172.21.15.111:0/1133834762 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800b860 con 0x7f6bf4140710 2023-12-19T08:06:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bf37fe700 1 -- 172.21.15.111:0/1133834762 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be800b9c0 con 0x7f6bf4140710 2023-12-19T08:06:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bf37fe700 1 -- 172.21.15.111:0/1133834762 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800bd60 con 0x7f6bf4140710 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/1133834762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf4140710 msgr2=0x7f6bf40c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf4140710 0x7f6bf40c70a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6be80099b0 tx=0x7f6be8009250 comp rx=0 tx=0).stop 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/1133834762 shutdown_connections 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140ae0 0x7f6bf40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/1133834762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf4140710 0x7f6bf40c70a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/1133834762 >> 172.21.15.111:0/1133834762 conn(0x7f6bf40c8850 msgr2=0x7f6bf40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/1133834762 shutdown_connections 2023-12-19T08:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/1133834762 wait complete. 2023-12-19T08:06:26.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.537+0000 7f6bfb0f0700 1 Processor -- start 2023-12-19T08:06:26.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- start start 2023-12-19T08:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140710 0x7f6bf4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf4140ae0 0x7f6bf4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf40c5680 con 0x7f6bf40c75e0 2023-12-19T08:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4157c00 con 0x7f6bf4140ae0 2023-12-19T08:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4140000 con 0x7f6bf4140710 2023-12-19T08:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53026/0 (socket says 172.21.15.111:53026) 2023-12-19T08:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/741483748 learned_addr learned my addr 172.21.15.111:0/741483748 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf3fff700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140710 0x7f6bf4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf968d700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf4140ae0 0x7f6bf4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140710 msgr2=0x7f6bf4159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140710 0x7f6bf4159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf4140ae0 msgr2=0x7f6bf4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf4140ae0 0x7f6bf4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 -- 172.21.15.111:0/741483748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80094f0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf8e8c700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006fd0 tx=0x7f6be8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8024440 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be80245a0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf40c50c0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8024940 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.541+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4157890 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.545+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4071120 con 0x7f6bf40c75e0 2023-12-19T08:06:26.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.545+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f6be801a420 con 0x7f6bf40c75e0 2023-12-19T08:06:26.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.545+0000 7f6bf0ff9700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 0x7f6bd8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.549+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f6be807eaa0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.549+0000 7f6bf3fff700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 0x7f6bd8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.549+0000 7f6bf3fff700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 0x7f6bd8062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1370 tx=0x7f6be0008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.553+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6be804ec80 con 0x7f6bf40c75e0 2023-12-19T08:06:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.761+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf40642a0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bf0ff9700 1 -- 172.21.15.111:0/741483748 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7f6be80522a0 con 0x7f6bf40c75e0 2023-12-19T08:06:26.769 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:26.769 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T08:06:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 msgr2=0x7f6bd8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 0x7f6bd8062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40b1370 tx=0x7f6be0008040 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 msgr2=0x7f6bf413fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006fd0 tx=0x7f6be8017000 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 shutdown_connections 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6bd8060670 0x7f6bd8062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf4140ae0 0x7f6bf4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf4140710 0x7f6bf4159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 --2- 172.21.15.111:0/741483748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bf40c75e0 0x7f6bf413fac0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.765+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 >> 172.21.15.111:0/741483748 conn(0x7f6bf40c8850 msgr2=0x7f6bf40c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:26.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.769+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 shutdown_connections 2023-12-19T08:06:26.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.769+0000 7f6bfb0f0700 1 -- 172.21.15.111:0/741483748 wait complete. 2023-12-19T08:06:26.775 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T08:06:26.792 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T08:06:26.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.937+0000 7fdd753c5700 1 Processor -- start 2023-12-19T08:06:26.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.937+0000 7fdd753c5700 1 -- start start 2023-12-19T08:06:26.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd70140710 0x7fdd700c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd70068800 0x7fdd700ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7006e150 con 0x7fdd70140ae0 2023-12-19T08:06:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7006d540 con 0x7fdd70068800 2023-12-19T08:06:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7006d820 con 0x7fdd70140710 2023-12-19T08:06:26.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd70068800 0x7fdd700ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd70068800 0x7fdd700ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53032/0 (socket says 172.21.15.111:53032) 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 -- 172.21.15.111:0/2804000255 learned_addr learned my addr 172.21.15.111:0/2804000255 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 -- 172.21.15.111:0/2804000255 >> v2:172.21.15.143:3300/0 conn(0x7fdd70140710 msgr2=0x7fdd700c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 --2- 172.21.15.111:0/2804000255 >> v2:172.21.15.143:3300/0 conn(0x7fdd70140710 0x7fdd700c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 -- 172.21.15.111:0/2804000255 >> v1:172.21.15.143:6789/0 conn(0x7fdd70140ae0 legacy=0x7fdd700680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 -- 172.21.15.111:0/2804000255 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7006db00 con 0x7fdd70068800 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6f7fe700 1 --2- 172.21.15.111:0/2804000255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd700ac250 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fdd70068c30 tx=0x7fdd68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1993e0c910c630fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6dffb700 1 -- 172.21.15.111:0/2804000255 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800b820 con 0x7fdd70068800 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6dffb700 1 -- 172.21.15.111:0/2804000255 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6800b980 con 0x7fdd70068800 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd6dffb700 1 -- 172.21.15.111:0/2804000255 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800bd20 con 0x7fdd70068800 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- 172.21.15.111:0/2804000255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 msgr2=0x7fdd700ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 --2- 172.21.15.111:0/2804000255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd700ac250 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fdd70068c30 tx=0x7fdd68009250 comp rx=0 tx=0).stop 2023-12-19T08:06:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- 172.21.15.111:0/2804000255 shutdown_connections 2023-12-19T08:06:26.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 --2- 172.21.15.111:0/2804000255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd700ac250 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 --2- 172.21.15.111:0/2804000255 >> v2:172.21.15.143:3300/0 conn(0x7fdd70140710 0x7fdd700c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.941+0000 7fdd753c5700 1 -- 172.21.15.111:0/2804000255 >> 172.21.15.111:0/2804000255 conn(0x7fdd700c8850 msgr2=0x7fdd700c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:26.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.945+0000 7fdd753c5700 1 -- 172.21.15.111:0/2804000255 shutdown_connections 2023-12-19T08:06:26.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.945+0000 7fdd753c5700 1 -- 172.21.15.111:0/2804000255 wait complete. 2023-12-19T08:06:26.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.945+0000 7fdd753c5700 1 Processor -- start 2023-12-19T08:06:26.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.945+0000 7fdd753c5700 1 -- start start 2023-12-19T08:06:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd70140710 0x7fdd70167ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 0x7fdd700acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70167ad0 con 0x7fdd70068800 2023-12-19T08:06:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7015ac20 con 0x7fdd70140ae0 2023-12-19T08:06:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd753c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7015af00 con 0x7fdd70140710 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53046/0 (socket says 172.21.15.111:53046) 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 -- 172.21.15.111:0/3346159859 learned_addr learned my addr 172.21.15.111:0/3346159859 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6f7fe700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 0x7fdd700acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6e7fc700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd70140710 0x7fdd70167ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 -- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd70140710 msgr2=0x7fdd70167ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd70140710 0x7fdd70167ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 -- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 msgr2=0x7fdd700acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 0x7fdd700acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 -- 172.21.15.111:0/3346159859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd680094f0 con 0x7fdd70068800 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6f7fe700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 0x7fdd700acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd6effd700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000b810 tx=0x7fdd60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.949+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000f680 con 0x7fdd70068800 2023-12-19T08:06:26.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6000f800 con 0x7fdd70068800 2023-12-19T08:06:26.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7015b1e0 con 0x7fdd70068800 2023-12-19T08:06:26.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000fbc0 con 0x7fdd70068800 2023-12-19T08:06:26.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7015b7a0 con 0x7fdd70068800 2023-12-19T08:06:26.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd70071120 con 0x7fdd70068800 2023-12-19T08:06:26.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.953+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdd60018400 con 0x7fdd70068800 2023-12-19T08:06:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.957+0000 7fdd56ffd700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 0x7fdd5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.957+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fdd6007cdd0 con 0x7fdd70068800 2023-12-19T08:06:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.957+0000 7fdd6e7fc700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 0x7fdd5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.957+0000 7fdd6e7fc700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 0x7fdd5c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd700ad8e0 tx=0x7fdd64009620 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:26.961+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd6004cf50 con 0x7fdd70068800 2023-12-19T08:06:27.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.133+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd700642a0 con 0x7fdd70068800 2023-12-19T08:06:27.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.137+0000 7fdd56ffd700 1 -- 172.21.15.111:0/3346159859 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16451 (secure 0 0 0) 0x7fdd60050570 con 0x7fdd70068800 2023-12-19T08:06:27.142 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:27.142 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":144,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T08:06:24.283567+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T08:06:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 msgr2=0x7fdd5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 0x7fdd5c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdd700ad8e0 tx=0x7fdd64009620 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 msgr2=0x7fdd70167590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000b810 tx=0x7fdd60009250 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 shutdown_connections 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdd5c060670 0x7fdd5c062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd70140ae0 0x7fdd700acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd70140710 0x7fdd70167ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 --2- 172.21.15.111:0/3346159859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd70068800 0x7fdd70167590 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 >> 172.21.15.111:0/3346159859 conn(0x7fdd700c8850 msgr2=0x7fdd7013e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 shutdown_connections 2023-12-19T08:06:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.141+0000 7fdd753c5700 1 -- 172.21.15.111:0/3346159859 wait complete. 2023-12-19T08:06:27.166 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T08:06:27.166 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:06:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.321+0000 7f00ae40c700 1 Processor -- start 2023-12-19T08:06:27.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.321+0000 7f00ae40c700 1 -- start start 2023-12-19T08:06:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ae40c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00a8155960 0x7f00a815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ae40c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00a80c7f00 0x7f00a80aca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ae40c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a806d990 con 0x7f00a806de20 2023-12-19T08:06:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ae40c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a815a570 con 0x7f00a8155960 2023-12-19T08:06:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ae40c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a806cd80 con 0x7f00a80c7f00 2023-12-19T08:06:27.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00acc09700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f00a806de20 0x7f00a80c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33076/0 (socket says 172.21.15.111:33076) 2023-12-19T08:06:27.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00acc09700 1 -- 172.21.15.111:0/1272481845 learned_addr learned my addr 172.21.15.111:0/1272481845 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:27.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00ad40a700 1 --2- 172.21.15.111:0/1272481845 >> v2:172.21.15.111:3300/0 conn(0x7f00a8155960 0x7f00a815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 --2- 172.21.15.111:0/1272481845 >> v2:172.21.15.143:3300/0 conn(0x7f00a80c7f00 0x7f00a80aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f009ffff700 1 -- 172.21.15.111:0/1272481845 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584566751 0 0) 0x7f00a806d990 con 0x7f00a806de20 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f009ffff700 1 -- 172.21.15.111:0/1272481845 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0094003570 con 0x7f00a806de20 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 -- 172.21.15.111:0/1272481845 >> v2:172.21.15.111:3300/0 conn(0x7f00a8155960 msgr2=0x7f00a815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 --2- 172.21.15.111:0/1272481845 >> v2:172.21.15.111:3300/0 conn(0x7f00a8155960 0x7f00a815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 -- 172.21.15.111:0/1272481845 >> v1:172.21.15.111:6789/0 conn(0x7f00a806de20 legacy=0x7f00a80c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 -- 172.21.15.111:0/1272481845 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a806d060 con 0x7f00a80c7f00 2023-12-19T08:06:27.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.325+0000 7f00adc0b700 1 --2- 172.21.15.111:0/1272481845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a80c7f00 0x7f00a80aca10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f00a80c5f50 tx=0x7f00a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b79f822bb6f7bf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f009ffff700 1 -- 172.21.15.111:0/1272481845 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a000e070 con 0x7f00a80c7f00 2023-12-19T08:06:27.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f009ffff700 1 -- 172.21.15.111:0/1272481845 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a000bab0 con 0x7f00a80c7f00 2023-12-19T08:06:27.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f009ffff700 1 -- 172.21.15.111:0/1272481845 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a000be70 con 0x7f00a80c7f00 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 -- 172.21.15.111:0/1272481845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a80c7f00 msgr2=0x7f00a80aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1272481845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a80c7f00 0x7f00a80aca10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f00a80c5f50 tx=0x7f00a0009780 comp rx=0 tx=0).stop 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 -- 172.21.15.111:0/1272481845 shutdown_connections 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1272481845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a80c7f00 0x7f00a80aca10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1272481845 >> v2:172.21.15.111:3300/0 conn(0x7f00a8155960 0x7f00a815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 -- 172.21.15.111:0/1272481845 >> 172.21.15.111:0/1272481845 conn(0x7f00a8067bb0 msgr2=0x7f00a80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:27.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 -- 172.21.15.111:0/1272481845 shutdown_connections 2023-12-19T08:06:27.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.333+0000 7f00ae40c700 1 -- 172.21.15.111:0/1272481845 wait complete. 2023-12-19T08:06:27.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.337+0000 7f00ae40c700 1 Processor -- start 2023-12-19T08:06:27.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.337+0000 7f00ae40c700 1 -- start start 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 0x7f00a813c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 0x7f00a814bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 0x7f00a814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a814cf30 con 0x7f00a80c7f00 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a814dd90 con 0x7f00a8155960 2023-12-19T08:06:27.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a816a080 con 0x7f00a806de20 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 0x7f00a814bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ad40a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 0x7f00a813c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00adc0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 0x7f00a814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ad40a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 0x7f00a813c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52984/0 (socket says 172.21.15.111:52984) 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ad40a700 1 -- 172.21.15.111:0/1981058200 learned_addr learned my addr 172.21.15.111:0/1981058200 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:27.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00adc0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 0x7f00a814c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:58364/0 (socket says 172.21.15.111:58364) 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 -- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 msgr2=0x7f00a813c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 0x7f00a813c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 -- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 msgr2=0x7f00a814c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 0x7f00a814c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 -- 172.21.15.111:0/1981058200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0009a20 con 0x7f00a80c7f00 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00acc09700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 0x7f00a814bcd0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f009800c5b0 tx=0x7f0098007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098005b30 con 0x7f00a80c7f00 2023-12-19T08:06:27.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098005cb0 con 0x7f00a80c7f00 2023-12-19T08:06:27.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098007320 con 0x7f00a80c7f00 2023-12-19T08:06:27.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a816a340 con 0x7f00a80c7f00 2023-12-19T08:06:27.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.341+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a814d460 con 0x7f00a80c7f00 2023-12-19T08:06:27.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.345+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a804be40 con 0x7f00a80c7f00 2023-12-19T08:06:27.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.345+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f009801b3b0 con 0x7f00a80c7f00 2023-12-19T08:06:27.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.349+0000 7f009d7fa700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 0x7f008c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.349+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f009807c090 con 0x7f00a80c7f00 2023-12-19T08:06:27.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.349+0000 7f00ad40a700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 0x7f008c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.349+0000 7f00ad40a700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 0x7f008c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009dd0 tx=0x7f00a40091a0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.353+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f009804c210 con 0x7f00a80c7f00 2023-12-19T08:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.517+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00a806d3b0 con 0x7f008c060670 2023-12-19T08:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.537+0000 7f009d7fa700 1 -- 172.21.15.111:0/1981058200 <== mgr.7981 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286961 (secure 0 0 0) 0x7f00a806d3b0 con 0x7f008c060670 2023-12-19T08:06:27.546 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 msgr2=0x7f008c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 0x7f008c062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009dd0 tx=0x7f00a40091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 msgr2=0x7f00a814bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 0x7f00a814bcd0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f009800c5b0 tx=0x7f0098007d50 comp rx=0 tx=0).stop 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 shutdown_connections 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f008c060670 0x7f008c062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00a8155960 0x7f00a814c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a80c7f00 0x7f00a814bcd0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 --2- 172.21.15.111:0/1981058200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00a806de20 0x7f00a813c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 >> 172.21.15.111:0/1981058200 conn(0x7f00a8067bb0 msgr2=0x7f00a806ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 shutdown_connections 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.541+0000 7f00ae40c700 1 -- 172.21.15.111:0/1981058200 wait complete. 2023-12-19T08:06:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:06:27.567 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:06:26.749718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454864,"kb_used_data":15120,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752519856,"statfs":{"total":773094113280,"available":770580332544,"internally_reserved":0,"allocated":15482880,"data_stored":8677293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.489829"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307642+0000","last_change":"2023-12-19T08:06:23.301263+0000","last_active":"2023-12-19T08:06:23.307642+0000","last_peered":"2023-12-19T08:06:23.307642+0000","last_clean":"2023-12-19T08:06:23.307642+0000","last_became_active":"2023-12-19T08:06:23.299353+0000","last_became_peered":"2023-12-19T08:06:23.299353+0000","last_unstale":"2023-12-19T08:06:23.307642+0000","last_undegraded":"2023-12-19T08:06:23.307642+0000","last_fullsized":"2023-12-19T08:06:23.307642+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310658+0000","last_change":"2023-12-19T08:06:23.303294+0000","last_active":"2023-12-19T08:06:23.310658+0000","last_peered":"2023-12-19T08:06:23.310658+0000","last_clean":"2023-12-19T08:06:23.310658+0000","last_became_active":"2023-12-19T08:06:23.303019+0000","last_became_peered":"2023-12-19T08:06:23.303019+0000","last_unstale":"2023-12-19T08:06:23.310658+0000","last_undegraded":"2023-12-19T08:06:23.310658+0000","last_fullsized":"2023-12-19T08:06:23.310658+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:06.579739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.313024+0000","last_change":"2023-12-19T08:06:23.311256+0000","last_active":"2023-12-19T08:06:23.313024+0000","last_peered":"2023-12-19T08:06:23.313024+0000","last_clean":"2023-12-19T08:06:23.313024+0000","last_became_active":"2023-12-19T08:06:23.310990+0000","last_became_peered":"2023-12-19T08:06:23.310990+0000","last_unstale":"2023-12-19T08:06:23.313024+0000","last_undegraded":"2023-12-19T08:06:23.313024+0000","last_fullsized":"2023-12-19T08:06:23.313024+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:21:51.718461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306364+0000","last_change":"2023-12-19T08:06:23.306364+0000","last_active":"2023-12-19T08:06:23.306364+0000","last_peered":"2023-12-19T08:06:23.306364+0000","last_clean":"2023-12-19T08:06:23.306364+0000","last_became_active":"2023-12-19T08:06:23.305840+0000","last_became_peered":"2023-12-19T08:06:23.305840+0000","last_unstale":"2023-12-19T08:06:23.306364+0000","last_undegraded":"2023-12-19T08:06:23.306364+0000","last_fullsized":"2023-12-19T08:06:23.306364+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310736+0000","last_change":"2023-12-19T08:06:23.309825+0000","last_active":"2023-12-19T08:06:23.310736+0000","last_peered":"2023-12-19T08:06:23.310736+0000","last_clean":"2023-12-19T08:06:23.310736+0000","last_became_active":"2023-12-19T08:06:23.309500+0000","last_became_peered":"2023-12-19T08:06:23.309500+0000","last_unstale":"2023-12-19T08:06:23.310736+0000","last_undegraded":"2023-12-19T08:06:23.310736+0000","last_fullsized":"2023-12-19T08:06:23.310736+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:11.892726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.300983+0000","last_change":"2023-12-19T08:06:23.300983+0000","last_active":"2023-12-19T08:06:23.300983+0000","last_peered":"2023-12-19T08:06:23.300983+0000","last_clean":"2023-12-19T08:06:23.300983+0000","last_became_active":"2023-12-19T08:06:23.300676+0000","last_became_peered":"2023-12-19T08:06:23.300676+0000","last_unstale":"2023-12-19T08:06:23.300983+0000","last_undegraded":"2023-12-19T08:06:23.300983+0000","last_fullsized":"2023-12-19T08:06:23.300983+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:20.450021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303087+0000","last_change":"2023-12-19T08:06:23.303087+0000","last_active":"2023-12-19T08:06:23.303087+0000","last_peered":"2023-12-19T08:06:23.303087+0000","last_clean":"2023-12-19T08:06:23.303087+0000","last_became_active":"2023-12-19T08:06:23.301801+0000","last_became_peered":"2023-12-19T08:06:23.301801+0000","last_unstale":"2023-12-19T08:06:23.303087+0000","last_undegraded":"2023-12-19T08:06:23.303087+0000","last_fullsized":"2023-12-19T08:06:23.303087+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:22.145749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.311708+0000","last_change":"2023-12-19T08:06:23.305076+0000","last_active":"2023-12-19T08:06:23.311708+0000","last_peered":"2023-12-19T08:06:23.311708+0000","last_clean":"2023-12-19T08:06:23.311708+0000","last_became_active":"2023-12-19T08:06:23.304772+0000","last_became_peered":"2023-12-19T08:06:23.304772+0000","last_unstale":"2023-12-19T08:06:23.311708+0000","last_undegraded":"2023-12-19T08:06:23.311708+0000","last_fullsized":"2023-12-19T08:06:23.311708+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:30.465211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.315450+0000","last_change":"2023-12-19T08:06:23.315450+0000","last_active":"2023-12-19T08:06:23.315450+0000","last_peered":"2023-12-19T08:06:23.315450+0000","last_clean":"2023-12-19T08:06:23.315450+0000","last_became_active":"2023-12-19T08:06:23.315137+0000","last_became_peered":"2023-12-19T08:06:23.315137+0000","last_unstale":"2023-12-19T08:06:23.315450+0000","last_undegraded":"2023-12-19T08:06:23.315450+0000","last_fullsized":"2023-12-19T08:06:23.315450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301823+0000","last_change":"2023-12-19T08:06:23.295404+0000","last_active":"2023-12-19T08:06:23.301823+0000","last_peered":"2023-12-19T08:06:23.301823+0000","last_clean":"2023-12-19T08:06:23.301823+0000","last_became_active":"2023-12-19T08:06:23.295095+0000","last_became_peered":"2023-12-19T08:06:23.295095+0000","last_unstale":"2023-12-19T08:06:23.301823+0000","last_undegraded":"2023-12-19T08:06:23.301823+0000","last_fullsized":"2023-12-19T08:06:23.301823+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304128+0000","last_change":"2023-12-19T08:06:23.293699+0000","last_active":"2023-12-19T08:06:23.304128+0000","last_peered":"2023-12-19T08:06:23.304128+0000","last_clean":"2023-12-19T08:06:23.304128+0000","last_became_active":"2023-12-19T08:06:23.293396+0000","last_became_peered":"2023-12-19T08:06:23.293396+0000","last_unstale":"2023-12-19T08:06:23.304128+0000","last_undegraded":"2023-12-19T08:06:23.304128+0000","last_fullsized":"2023-12-19T08:06:23.304128+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.284556+0000","last_change":"2023-12-19T08:06:23.284556+0000","last_active":"2023-12-19T08:06:23.284556+0000","last_peered":"2023-12-19T08:06:23.284556+0000","last_clean":"2023-12-19T08:06:23.284556+0000","last_became_active":"2023-12-19T08:06:23.284242+0000","last_became_peered":"2023-12-19T08:06:23.284242+0000","last_unstale":"2023-12-19T08:06:23.284556+0000","last_undegraded":"2023-12-19T08:06:23.284556+0000","last_fullsized":"2023-12-19T08:06:23.284556+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:44.721338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303986+0000","last_change":"2023-12-19T08:06:23.297002+0000","last_active":"2023-12-19T08:06:23.303986+0000","last_peered":"2023-12-19T08:06:23.303986+0000","last_clean":"2023-12-19T08:06:23.303986+0000","last_became_active":"2023-12-19T08:06:23.296722+0000","last_became_peered":"2023-12-19T08:06:23.296722+0000","last_unstale":"2023-12-19T08:06:23.303986+0000","last_undegraded":"2023-12-19T08:06:23.303986+0000","last_fullsized":"2023-12-19T08:06:23.303986+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.312996+0000","last_change":"2023-12-19T08:06:23.310857+0000","last_active":"2023-12-19T08:06:23.312996+0000","last_peered":"2023-12-19T08:06:23.312996+0000","last_clean":"2023-12-19T08:06:23.312996+0000","last_became_active":"2023-12-19T08:06:23.310492+0000","last_became_peered":"2023-12-19T08:06:23.310492+0000","last_unstale":"2023-12-19T08:06:23.312996+0000","last_undegraded":"2023-12-19T08:06:23.312996+0000","last_fullsized":"2023-12-19T08:06:23.312996+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:09.954917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303733+0000","last_change":"2023-12-19T08:06:23.293689+0000","last_active":"2023-12-19T08:06:23.303733+0000","last_peered":"2023-12-19T08:06:23.303733+0000","last_clean":"2023-12-19T08:06:23.303733+0000","last_became_active":"2023-12-19T08:06:23.293384+0000","last_became_peered":"2023-12-19T08:06:23.293384+0000","last_unstale":"2023-12-19T08:06:23.303733+0000","last_undegraded":"2023-12-19T08:06:23.303733+0000","last_fullsized":"2023-12-19T08:06:23.303733+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:49.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.292627+0000","last_change":"2023-12-19T08:06:23.292627+0000","last_active":"2023-12-19T08:06:23.292627+0000","last_peered":"2023-12-19T08:06:23.292627+0000","last_clean":"2023-12-19T08:06:23.292627+0000","last_became_active":"2023-12-19T08:06:23.292335+0000","last_became_peered":"2023-12-19T08:06:23.292335+0000","last_unstale":"2023-12-19T08:06:23.292627+0000","last_undegraded":"2023-12-19T08:06:23.292627+0000","last_fullsized":"2023-12-19T08:06:23.292627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:59:03.204930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301148+0000","last_change":"2023-12-19T08:06:23.289203+0000","last_active":"2023-12-19T08:06:23.301148+0000","last_peered":"2023-12-19T08:06:23.301148+0000","last_clean":"2023-12-19T08:06:23.301148+0000","last_became_active":"2023-12-19T08:06:23.288827+0000","last_became_peered":"2023-12-19T08:06:23.288827+0000","last_unstale":"2023-12-19T08:06:23.301148+0000","last_undegraded":"2023-12-19T08:06:23.301148+0000","last_fullsized":"2023-12-19T08:06:23.301148+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310640+0000","last_change":"2023-12-19T08:06:23.303166+0000","last_active":"2023-12-19T08:06:23.310640+0000","last_peered":"2023-12-19T08:06:23.310640+0000","last_clean":"2023-12-19T08:06:23.310640+0000","last_became_active":"2023-12-19T08:06:23.302878+0000","last_became_peered":"2023-12-19T08:06:23.302878+0000","last_unstale":"2023-12-19T08:06:23.310640+0000","last_undegraded":"2023-12-19T08:06:23.310640+0000","last_fullsized":"2023-12-19T08:06:23.310640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:05.160858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310563+0000","last_change":"2023-12-19T08:06:23.310563+0000","last_active":"2023-12-19T08:06:23.310563+0000","last_peered":"2023-12-19T08:06:23.310563+0000","last_clean":"2023-12-19T08:06:23.310563+0000","last_became_active":"2023-12-19T08:06:23.310253+0000","last_became_peered":"2023-12-19T08:06:23.310253+0000","last_unstale":"2023-12-19T08:06:23.310563+0000","last_undegraded":"2023-12-19T08:06:23.310563+0000","last_fullsized":"2023-12-19T08:06:23.310563+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:59.013369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293911+0000","last_change":"2023-12-19T08:06:23.293911+0000","last_active":"2023-12-19T08:06:23.293911+0000","last_peered":"2023-12-19T08:06:23.293911+0000","last_clean":"2023-12-19T08:06:23.293911+0000","last_became_active":"2023-12-19T08:06:23.293608+0000","last_became_peered":"2023-12-19T08:06:23.293608+0000","last_unstale":"2023-12-19T08:06:23.293911+0000","last_undegraded":"2023-12-19T08:06:23.293911+0000","last_fullsized":"2023-12-19T08:06:23.293911+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:08.032269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301865+0000","last_change":"2023-12-19T08:06:23.301865+0000","last_active":"2023-12-19T08:06:23.301865+0000","last_peered":"2023-12-19T08:06:23.301865+0000","last_clean":"2023-12-19T08:06:23.301865+0000","last_became_active":"2023-12-19T08:06:23.301280+0000","last_became_peered":"2023-12-19T08:06:23.301280+0000","last_unstale":"2023-12-19T08:06:23.301865+0000","last_undegraded":"2023-12-19T08:06:23.301865+0000","last_fullsized":"2023-12-19T08:06:23.301865+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:01.810637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307526+0000","last_change":"2023-12-19T08:06:23.302294+0000","last_active":"2023-12-19T08:06:23.307526+0000","last_peered":"2023-12-19T08:06:23.307526+0000","last_clean":"2023-12-19T08:06:23.307526+0000","last_became_active":"2023-12-19T08:06:23.302012+0000","last_became_peered":"2023-12-19T08:06:23.302012+0000","last_unstale":"2023-12-19T08:06:23.307526+0000","last_undegraded":"2023-12-19T08:06:23.307526+0000","last_fullsized":"2023-12-19T08:06:23.307526+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324511+0000","last_change":"2023-12-19T08:06:23.324511+0000","last_active":"2023-12-19T08:06:23.324511+0000","last_peered":"2023-12-19T08:06:23.324511+0000","last_clean":"2023-12-19T08:06:23.324511+0000","last_became_active":"2023-12-19T08:06:23.323971+0000","last_became_peered":"2023-12-19T08:06:23.323971+0000","last_unstale":"2023-12-19T08:06:23.324511+0000","last_undegraded":"2023-12-19T08:06:23.324511+0000","last_fullsized":"2023-12-19T08:06:23.324511+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308679+0000","last_change":"2023-12-19T08:06:23.308679+0000","last_active":"2023-12-19T08:06:23.308679+0000","last_peered":"2023-12-19T08:06:23.308679+0000","last_clean":"2023-12-19T08:06:23.308679+0000","last_became_active":"2023-12-19T08:06:23.308221+0000","last_became_peered":"2023-12-19T08:06:23.308221+0000","last_unstale":"2023-12-19T08:06:23.308679+0000","last_undegraded":"2023-12-19T08:06:23.308679+0000","last_fullsized":"2023-12-19T08:06:23.308679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.295135+0000","last_change":"2023-12-19T08:06:23.295135+0000","last_active":"2023-12-19T08:06:23.295135+0000","last_peered":"2023-12-19T08:06:23.295135+0000","last_clean":"2023-12-19T08:06:23.295135+0000","last_became_active":"2023-12-19T08:06:23.294867+0000","last_became_peered":"2023-12-19T08:06:23.294867+0000","last_unstale":"2023-12-19T08:06:23.295135+0000","last_undegraded":"2023-12-19T08:06:23.295135+0000","last_fullsized":"2023-12-19T08:06:23.295135+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304347+0000","last_change":"2023-12-19T08:06:23.304347+0000","last_active":"2023-12-19T08:06:23.304347+0000","last_peered":"2023-12-19T08:06:23.304347+0000","last_clean":"2023-12-19T08:06:23.304347+0000","last_became_active":"2023-12-19T08:06:23.304019+0000","last_became_peered":"2023-12-19T08:06:23.304019+0000","last_unstale":"2023-12-19T08:06:23.304347+0000","last_undegraded":"2023-12-19T08:06:23.304347+0000","last_fullsized":"2023-12-19T08:06:23.304347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:04.452778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.305901+0000","last_change":"2023-12-19T08:06:23.305901+0000","last_active":"2023-12-19T08:06:23.305901+0000","last_peered":"2023-12-19T08:06:23.305901+0000","last_clean":"2023-12-19T08:06:23.305901+0000","last_became_active":"2023-12-19T08:06:23.305575+0000","last_became_peered":"2023-12-19T08:06:23.305575+0000","last_unstale":"2023-12-19T08:06:23.305901+0000","last_undegraded":"2023-12-19T08:06:23.305901+0000","last_fullsized":"2023-12-19T08:06:23.305901+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.203507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.285330+0000","last_change":"2023-12-19T08:06:23.285330+0000","last_active":"2023-12-19T08:06:23.285330+0000","last_peered":"2023-12-19T08:06:23.285330+0000","last_clean":"2023-12-19T08:06:23.285330+0000","last_became_active":"2023-12-19T08:06:23.284503+0000","last_became_peered":"2023-12-19T08:06:23.284503+0000","last_unstale":"2023-12-19T08:06:23.285330+0000","last_undegraded":"2023-12-19T08:06:23.285330+0000","last_fullsized":"2023-12-19T08:06:23.285330+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:05:55.186443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307827+0000","last_change":"2023-12-19T08:06:23.307827+0000","last_active":"2023-12-19T08:06:23.307827+0000","last_peered":"2023-12-19T08:06:23.307827+0000","last_clean":"2023-12-19T08:06:23.307827+0000","last_became_active":"2023-12-19T08:06:23.307505+0000","last_became_peered":"2023-12-19T08:06:23.307505+0000","last_unstale":"2023-12-19T08:06:23.307827+0000","last_undegraded":"2023-12-19T08:06:23.307827+0000","last_fullsized":"2023-12-19T08:06:23.307827+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:06.564383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.311836+0000","last_change":"2023-12-19T08:06:23.308049+0000","last_active":"2023-12-19T08:06:23.311836+0000","last_peered":"2023-12-19T08:06:23.311836+0000","last_clean":"2023-12-19T08:06:23.311836+0000","last_became_active":"2023-12-19T08:06:23.307730+0000","last_became_peered":"2023-12-19T08:06:23.307730+0000","last_unstale":"2023-12-19T08:06:23.311836+0000","last_undegraded":"2023-12-19T08:06:23.311836+0000","last_fullsized":"2023-12-19T08:06:23.311836+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:14.586924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.299238+0000","last_change":"2023-12-19T08:06:23.299238+0000","last_active":"2023-12-19T08:06:23.299238+0000","last_peered":"2023-12-19T08:06:23.299238+0000","last_clean":"2023-12-19T08:06:23.299238+0000","last_became_active":"2023-12-19T08:06:23.298940+0000","last_became_peered":"2023-12-19T08:06:23.298940+0000","last_unstale":"2023-12-19T08:06:23.299238+0000","last_undegraded":"2023-12-19T08:06:23.299238+0000","last_fullsized":"2023-12-19T08:06:23.299238+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308598+0000","last_change":"2023-12-19T08:06:23.308598+0000","last_active":"2023-12-19T08:06:23.308598+0000","last_peered":"2023-12-19T08:06:23.308598+0000","last_clean":"2023-12-19T08:06:23.308598+0000","last_became_active":"2023-12-19T08:06:23.308197+0000","last_became_peered":"2023-12-19T08:06:23.308197+0000","last_unstale":"2023-12-19T08:06:23.308598+0000","last_undegraded":"2023-12-19T08:06:23.308598+0000","last_fullsized":"2023-12-19T08:06:23.308598+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:49.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304068+0000","last_change":"2023-12-19T08:06:23.292316+0000","last_active":"2023-12-19T08:06:23.304068+0000","last_peered":"2023-12-19T08:06:23.304068+0000","last_clean":"2023-12-19T08:06:23.304068+0000","last_became_active":"2023-12-19T08:06:23.291982+0000","last_became_peered":"2023-12-19T08:06:23.291982+0000","last_unstale":"2023-12-19T08:06:23.304068+0000","last_undegraded":"2023-12-19T08:06:23.304068+0000","last_fullsized":"2023-12-19T08:06:23.304068+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324830+0000","last_change":"2023-12-19T08:06:22.294804+0000","last_active":"2023-12-19T08:06:23.324830+0000","last_peered":"2023-12-19T08:06:23.324830+0000","last_clean":"2023-12-19T08:06:23.324830+0000","last_became_active":"2023-12-19T08:06:22.294481+0000","last_became_peered":"2023-12-19T08:06:22.294481+0000","last_unstale":"2023-12-19T08:06:23.324830+0000","last_undegraded":"2023-12-19T08:06:23.324830+0000","last_fullsized":"2023-12-19T08:06:23.324830+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:08:11.541041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308929+0000","last_change":"2023-12-19T08:06:23.308929+0000","last_active":"2023-12-19T08:06:23.308929+0000","last_peered":"2023-12-19T08:06:23.308929+0000","last_clean":"2023-12-19T08:06:23.308929+0000","last_became_active":"2023-12-19T08:06:23.308001+0000","last_became_peered":"2023-12-19T08:06:23.308001+0000","last_unstale":"2023-12-19T08:06:23.308929+0000","last_undegraded":"2023-12-19T08:06:23.308929+0000","last_fullsized":"2023-12-19T08:06:23.308929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:44.698326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.321160+0000","last_change":"2023-12-19T08:06:22.293243+0000","last_active":"2023-12-19T08:06:23.321160+0000","last_peered":"2023-12-19T08:06:23.321160+0000","last_clean":"2023-12-19T08:06:23.321160+0000","last_became_active":"2023-12-19T08:06:22.292645+0000","last_became_peered":"2023-12-19T08:06:22.292645+0000","last_unstale":"2023-12-19T08:06:23.321160+0000","last_undegraded":"2023-12-19T08:06:23.321160+0000","last_fullsized":"2023-12-19T08:06:23.321160+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:03:58.915674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.297289+0000","last_change":"2023-12-19T08:06:23.297289+0000","last_active":"2023-12-19T08:06:23.297289+0000","last_peered":"2023-12-19T08:06:23.297289+0000","last_clean":"2023-12-19T08:06:23.297289+0000","last_became_active":"2023-12-19T08:06:23.296999+0000","last_became_peered":"2023-12-19T08:06:23.296999+0000","last_unstale":"2023-12-19T08:06:23.297289+0000","last_undegraded":"2023-12-19T08:06:23.297289+0000","last_fullsized":"2023-12-19T08:06:23.297289+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:12:49.209423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.323836+0000","last_change":"2023-12-19T08:06:22.299326+0000","last_active":"2023-12-19T08:06:23.323836+0000","last_peered":"2023-12-19T08:06:23.323836+0000","last_clean":"2023-12-19T08:06:23.323836+0000","last_became_active":"2023-12-19T08:06:22.298687+0000","last_became_peered":"2023-12-19T08:06:22.298687+0000","last_unstale":"2023-12-19T08:06:23.323836+0000","last_undegraded":"2023-12-19T08:06:23.323836+0000","last_fullsized":"2023-12-19T08:06:23.323836+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:57.999146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302251+0000","last_change":"2023-12-19T08:06:23.302251+0000","last_active":"2023-12-19T08:06:23.302251+0000","last_peered":"2023-12-19T08:06:23.302251+0000","last_clean":"2023-12-19T08:06:23.302251+0000","last_became_active":"2023-12-19T08:06:23.301492+0000","last_became_peered":"2023-12-19T08:06:23.301492+0000","last_unstale":"2023-12-19T08:06:23.302251+0000","last_undegraded":"2023-12-19T08:06:23.302251+0000","last_fullsized":"2023-12-19T08:06:23.302251+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:10.059325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.323366+0000","last_change":"2023-12-19T08:06:22.286893+0000","last_active":"2023-12-19T08:06:23.323366+0000","last_peered":"2023-12-19T08:06:23.323366+0000","last_clean":"2023-12-19T08:06:23.323366+0000","last_became_active":"2023-12-19T08:06:22.286602+0000","last_became_peered":"2023-12-19T08:06:22.286602+0000","last_unstale":"2023-12-19T08:06:23.323366+0000","last_undegraded":"2023-12-19T08:06:23.323366+0000","last_fullsized":"2023-12-19T08:06:23.323366+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:24:59.504536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308940+0000","last_change":"2023-12-19T08:06:23.304865+0000","last_active":"2023-12-19T08:06:23.308940+0000","last_peered":"2023-12-19T08:06:23.308940+0000","last_clean":"2023-12-19T08:06:23.308940+0000","last_became_active":"2023-12-19T08:06:23.304586+0000","last_became_peered":"2023-12-19T08:06:23.304586+0000","last_unstale":"2023-12-19T08:06:23.308940+0000","last_undegraded":"2023-12-19T08:06:23.308940+0000","last_fullsized":"2023-12-19T08:06:23.308940+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318694+0000","last_change":"2023-12-19T08:06:22.294550+0000","last_active":"2023-12-19T08:06:23.318694+0000","last_peered":"2023-12-19T08:06:23.318694+0000","last_clean":"2023-12-19T08:06:23.318694+0000","last_became_active":"2023-12-19T08:06:22.294227+0000","last_became_peered":"2023-12-19T08:06:22.294227+0000","last_unstale":"2023-12-19T08:06:23.318694+0000","last_undegraded":"2023-12-19T08:06:23.318694+0000","last_fullsized":"2023-12-19T08:06:23.318694+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:41.707156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307956+0000","last_change":"2023-12-19T08:06:23.303703+0000","last_active":"2023-12-19T08:06:23.307956+0000","last_peered":"2023-12-19T08:06:23.307956+0000","last_clean":"2023-12-19T08:06:23.307956+0000","last_became_active":"2023-12-19T08:06:23.303419+0000","last_became_peered":"2023-12-19T08:06:23.303419+0000","last_unstale":"2023-12-19T08:06:23.307956+0000","last_undegraded":"2023-12-19T08:06:23.307956+0000","last_fullsized":"2023-12-19T08:06:23.307956+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.274045+0000","last_change":"2023-12-19T08:06:22.292370+0000","last_active":"2023-12-19T08:06:23.274045+0000","last_peered":"2023-12-19T08:06:23.274045+0000","last_clean":"2023-12-19T08:06:23.274045+0000","last_became_active":"2023-12-19T08:06:22.292074+0000","last_became_peered":"2023-12-19T08:06:22.292074+0000","last_unstale":"2023-12-19T08:06:23.274045+0000","last_undegraded":"2023-12-19T08:06:23.274045+0000","last_fullsized":"2023-12-19T08:06:23.274045+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:44:53.952446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316122+0000","last_change":"2023-12-19T08:06:23.313825+0000","last_active":"2023-12-19T08:06:23.316122+0000","last_peered":"2023-12-19T08:06:23.316122+0000","last_clean":"2023-12-19T08:06:23.316122+0000","last_became_active":"2023-12-19T08:06:23.313551+0000","last_became_peered":"2023-12-19T08:06:23.313551+0000","last_unstale":"2023-12-19T08:06:23.316122+0000","last_undegraded":"2023-12-19T08:06:23.316122+0000","last_fullsized":"2023-12-19T08:06:23.316122+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:59:03.204930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.297374+0000","last_change":"2023-12-19T08:06:22.293568+0000","last_active":"2023-12-19T08:06:22.297374+0000","last_peered":"2023-12-19T08:06:22.297374+0000","last_clean":"2023-12-19T08:06:22.297374+0000","last_became_active":"2023-12-19T08:06:22.291910+0000","last_became_peered":"2023-12-19T08:06:22.291910+0000","last_unstale":"2023-12-19T08:06:22.297374+0000","last_undegraded":"2023-12-19T08:06:22.297374+0000","last_fullsized":"2023-12-19T08:06:22.297374+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:02.783487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306662+0000","last_change":"2023-12-19T08:06:23.301259+0000","last_active":"2023-12-19T08:06:23.306662+0000","last_peered":"2023-12-19T08:06:23.306662+0000","last_clean":"2023-12-19T08:06:23.306662+0000","last_became_active":"2023-12-19T08:06:23.300900+0000","last_became_peered":"2023-12-19T08:06:23.300900+0000","last_unstale":"2023-12-19T08:06:23.306662+0000","last_undegraded":"2023-12-19T08:06:23.306662+0000","last_fullsized":"2023-12-19T08:06:23.306662+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300230+0000","last_change":"2023-12-19T08:06:22.295514+0000","last_active":"2023-12-19T08:06:22.300230+0000","last_peered":"2023-12-19T08:06:22.300230+0000","last_clean":"2023-12-19T08:06:22.300230+0000","last_became_active":"2023-12-19T08:06:22.292978+0000","last_became_peered":"2023-12-19T08:06:22.292978+0000","last_unstale":"2023-12-19T08:06:22.300230+0000","last_undegraded":"2023-12-19T08:06:22.300230+0000","last_fullsized":"2023-12-19T08:06:22.300230+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:28.563889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303890+0000","last_change":"2023-12-19T08:06:23.303890+0000","last_active":"2023-12-19T08:06:23.303890+0000","last_peered":"2023-12-19T08:06:23.303890+0000","last_clean":"2023-12-19T08:06:23.303890+0000","last_became_active":"2023-12-19T08:06:23.303579+0000","last_became_peered":"2023-12-19T08:06:23.303579+0000","last_unstale":"2023-12-19T08:06:23.303890+0000","last_undegraded":"2023-12-19T08:06:23.303890+0000","last_fullsized":"2023-12-19T08:06:23.303890+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:34:21.718338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.294404+0000","last_change":"2023-12-19T08:06:22.292057+0000","last_active":"2023-12-19T08:06:22.294404+0000","last_peered":"2023-12-19T08:06:22.294404+0000","last_clean":"2023-12-19T08:06:22.294404+0000","last_became_active":"2023-12-19T08:06:22.291747+0000","last_became_peered":"2023-12-19T08:06:22.291747+0000","last_unstale":"2023-12-19T08:06:22.294404+0000","last_undegraded":"2023-12-19T08:06:22.294404+0000","last_fullsized":"2023-12-19T08:06:22.294404+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.190310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.315378+0000","last_change":"2023-12-19T08:06:23.315378+0000","last_active":"2023-12-19T08:06:23.315378+0000","last_peered":"2023-12-19T08:06:23.315378+0000","last_clean":"2023-12-19T08:06:23.315378+0000","last_became_active":"2023-12-19T08:06:23.315062+0000","last_became_peered":"2023-12-19T08:06:23.315062+0000","last_unstale":"2023-12-19T08:06:23.315378+0000","last_undegraded":"2023-12-19T08:06:23.315378+0000","last_fullsized":"2023-12-19T08:06:23.315378+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300095+0000","last_change":"2023-12-19T08:06:22.298313+0000","last_active":"2023-12-19T08:06:22.300095+0000","last_peered":"2023-12-19T08:06:22.300095+0000","last_clean":"2023-12-19T08:06:22.300095+0000","last_became_active":"2023-12-19T08:06:22.297980+0000","last_became_peered":"2023-12-19T08:06:22.297980+0000","last_unstale":"2023-12-19T08:06:22.300095+0000","last_undegraded":"2023-12-19T08:06:22.300095+0000","last_fullsized":"2023-12-19T08:06:22.300095+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.190310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310599+0000","last_change":"2023-12-19T08:06:23.302535+0000","last_active":"2023-12-19T08:06:23.310599+0000","last_peered":"2023-12-19T08:06:23.310599+0000","last_clean":"2023-12-19T08:06:23.310599+0000","last_became_active":"2023-12-19T08:06:23.302261+0000","last_became_peered":"2023-12-19T08:06:23.302261+0000","last_unstale":"2023-12-19T08:06:23.310599+0000","last_undegraded":"2023-12-19T08:06:23.310599+0000","last_fullsized":"2023-12-19T08:06:23.310599+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:22.145749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324336+0000","last_change":"2023-12-19T08:06:22.290518+0000","last_active":"2023-12-19T08:06:23.324336+0000","last_peered":"2023-12-19T08:06:23.324336+0000","last_clean":"2023-12-19T08:06:23.324336+0000","last_became_active":"2023-12-19T08:06:22.290234+0000","last_became_peered":"2023-12-19T08:06:22.290234+0000","last_unstale":"2023-12-19T08:06:23.324336+0000","last_undegraded":"2023-12-19T08:06:23.324336+0000","last_fullsized":"2023-12-19T08:06:23.324336+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:09.235742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.297463+0000","last_change":"2023-12-19T08:06:23.297463+0000","last_active":"2023-12-19T08:06:23.297463+0000","last_peered":"2023-12-19T08:06:23.297463+0000","last_clean":"2023-12-19T08:06:23.297463+0000","last_became_active":"2023-12-19T08:06:23.297170+0000","last_became_peered":"2023-12-19T08:06:23.297170+0000","last_unstale":"2023-12-19T08:06:23.297463+0000","last_undegraded":"2023-12-19T08:06:23.297463+0000","last_fullsized":"2023-12-19T08:06:23.297463+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.325943+0000","last_change":"2023-12-19T08:06:22.289201+0000","last_active":"2023-12-19T08:06:23.325943+0000","last_peered":"2023-12-19T08:06:23.325943+0000","last_clean":"2023-12-19T08:06:23.325943+0000","last_became_active":"2023-12-19T08:06:22.288924+0000","last_became_peered":"2023-12-19T08:06:22.288924+0000","last_unstale":"2023-12-19T08:06:23.325943+0000","last_undegraded":"2023-12-19T08:06:23.325943+0000","last_fullsized":"2023-12-19T08:06:23.325943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:28.563889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318211+0000","last_change":"2023-12-19T08:06:23.318211+0000","last_active":"2023-12-19T08:06:23.318211+0000","last_peered":"2023-12-19T08:06:23.318211+0000","last_clean":"2023-12-19T08:06:23.318211+0000","last_became_active":"2023-12-19T08:06:23.317910+0000","last_became_peered":"2023-12-19T08:06:23.317910+0000","last_unstale":"2023-12-19T08:06:23.318211+0000","last_undegraded":"2023-12-19T08:06:23.318211+0000","last_fullsized":"2023-12-19T08:06:23.318211+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:13:58.237515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.322780+0000","last_change":"2023-12-19T08:06:22.283616+0000","last_active":"2023-12-19T08:06:23.322780+0000","last_peered":"2023-12-19T08:06:23.322780+0000","last_clean":"2023-12-19T08:06:23.322780+0000","last_became_active":"2023-12-19T08:06:22.283328+0000","last_became_peered":"2023-12-19T08:06:22.283328+0000","last_unstale":"2023-12-19T08:06:23.322780+0000","last_undegraded":"2023-12-19T08:06:23.322780+0000","last_fullsized":"2023-12-19T08:06:23.322780+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:23:26.517239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307449+0000","last_change":"2023-12-19T08:06:23.298657+0000","last_active":"2023-12-19T08:06:23.307449+0000","last_peered":"2023-12-19T08:06:23.307449+0000","last_clean":"2023-12-19T08:06:23.307449+0000","last_became_active":"2023-12-19T08:06:23.298358+0000","last_became_peered":"2023-12-19T08:06:23.298358+0000","last_unstale":"2023-12-19T08:06:23.307449+0000","last_undegraded":"2023-12-19T08:06:23.307449+0000","last_fullsized":"2023-12-19T08:06:23.307449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:11.892726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.296423+0000","last_change":"2023-12-19T08:06:22.286398+0000","last_active":"2023-12-19T08:06:23.296423+0000","last_peered":"2023-12-19T08:06:23.296423+0000","last_clean":"2023-12-19T08:06:23.296423+0000","last_became_active":"2023-12-19T08:06:22.285903+0000","last_became_peered":"2023-12-19T08:06:22.285903+0000","last_unstale":"2023-12-19T08:06:23.296423+0000","last_undegraded":"2023-12-19T08:06:23.296423+0000","last_fullsized":"2023-12-19T08:06:23.296423+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:31.632186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307448+0000","last_change":"2023-12-19T08:06:23.307448+0000","last_active":"2023-12-19T08:06:23.307448+0000","last_peered":"2023-12-19T08:06:23.307448+0000","last_clean":"2023-12-19T08:06:23.307448+0000","last_became_active":"2023-12-19T08:06:23.300392+0000","last_became_peered":"2023-12-19T08:06:23.300392+0000","last_unstale":"2023-12-19T08:06:23.307448+0000","last_undegraded":"2023-12-19T08:06:23.307448+0000","last_fullsized":"2023-12-19T08:06:23.307448+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:23.712870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293810+0000","last_change":"2023-12-19T08:06:22.290323+0000","last_active":"2023-12-19T08:06:23.293810+0000","last_peered":"2023-12-19T08:06:23.293810+0000","last_clean":"2023-12-19T08:06:23.293810+0000","last_became_active":"2023-12-19T08:06:22.289996+0000","last_became_peered":"2023-12-19T08:06:22.289996+0000","last_unstale":"2023-12-19T08:06:23.293810+0000","last_undegraded":"2023-12-19T08:06:23.293810+0000","last_fullsized":"2023-12-19T08:06:23.293810+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:01.739322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.296999+0000","last_change":"2023-12-19T08:06:23.296999+0000","last_active":"2023-12-19T08:06:23.296999+0000","last_peered":"2023-12-19T08:06:23.296999+0000","last_clean":"2023-12-19T08:06:23.296999+0000","last_became_active":"2023-12-19T08:06:23.296433+0000","last_became_peered":"2023-12-19T08:06:23.296433+0000","last_unstale":"2023-12-19T08:06:23.296999+0000","last_undegraded":"2023-12-19T08:06:23.296999+0000","last_fullsized":"2023-12-19T08:06:23.296999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.321566+0000","last_change":"2023-12-19T08:06:22.289305+0000","last_active":"2023-12-19T08:06:23.321566+0000","last_peered":"2023-12-19T08:06:23.321566+0000","last_clean":"2023-12-19T08:06:23.321566+0000","last_became_active":"2023-12-19T08:06:22.287818+0000","last_became_peered":"2023-12-19T08:06:22.287818+0000","last_unstale":"2023-12-19T08:06:23.321566+0000","last_undegraded":"2023-12-19T08:06:23.321566+0000","last_fullsized":"2023-12-19T08:06:23.321566+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:10.698455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316458+0000","last_change":"2023-12-19T08:06:23.313509+0000","last_active":"2023-12-19T08:06:23.316458+0000","last_peered":"2023-12-19T08:06:23.316458+0000","last_clean":"2023-12-19T08:06:23.316458+0000","last_became_active":"2023-12-19T08:06:23.313106+0000","last_became_peered":"2023-12-19T08:06:23.313106+0000","last_unstale":"2023-12-19T08:06:23.316458+0000","last_undegraded":"2023-12-19T08:06:23.316458+0000","last_fullsized":"2023-12-19T08:06:23.316458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.317384+0000","last_change":"2023-12-19T08:06:22.298505+0000","last_active":"2023-12-19T08:06:23.317384+0000","last_peered":"2023-12-19T08:06:23.317384+0000","last_clean":"2023-12-19T08:06:23.317384+0000","last_became_active":"2023-12-19T08:06:22.297234+0000","last_became_peered":"2023-12-19T08:06:22.297234+0000","last_unstale":"2023-12-19T08:06:23.317384+0000","last_undegraded":"2023-12-19T08:06:23.317384+0000","last_fullsized":"2023-12-19T08:06:23.317384+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:31:48.619872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304727+0000","last_change":"2023-12-19T08:06:23.304727+0000","last_active":"2023-12-19T08:06:23.304727+0000","last_peered":"2023-12-19T08:06:23.304727+0000","last_clean":"2023-12-19T08:06:23.304727+0000","last_became_active":"2023-12-19T08:06:23.301454+0000","last_became_peered":"2023-12-19T08:06:23.301454+0000","last_unstale":"2023-12-19T08:06:23.304727+0000","last_undegraded":"2023-12-19T08:06:23.304727+0000","last_fullsized":"2023-12-19T08:06:23.304727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:05.160858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.292781+0000","last_change":"2023-12-19T08:06:22.298207+0000","last_active":"2023-12-19T08:06:23.292781+0000","last_peered":"2023-12-19T08:06:23.292781+0000","last_clean":"2023-12-19T08:06:23.292781+0000","last_became_active":"2023-12-19T08:06:22.297768+0000","last_became_peered":"2023-12-19T08:06:22.297768+0000","last_unstale":"2023-12-19T08:06:23.292781+0000","last_undegraded":"2023-12-19T08:06:23.292781+0000","last_fullsized":"2023-12-19T08:06:23.292781+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:19.189284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.305410+0000","last_change":"2023-12-19T08:06:23.305410+0000","last_active":"2023-12-19T08:06:23.305410+0000","last_peered":"2023-12-19T08:06:23.305410+0000","last_clean":"2023-12-19T08:06:23.305410+0000","last_became_active":"2023-12-19T08:06:23.304785+0000","last_became_peered":"2023-12-19T08:06:23.304785+0000","last_unstale":"2023-12-19T08:06:23.305410+0000","last_undegraded":"2023-12-19T08:06:23.305410+0000","last_fullsized":"2023-12-19T08:06:23.305410+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.302985+0000","last_change":"2023-12-19T08:06:22.302347+0000","last_active":"2023-12-19T08:06:22.302985+0000","last_peered":"2023-12-19T08:06:22.302985+0000","last_clean":"2023-12-19T08:06:22.302985+0000","last_became_active":"2023-12-19T08:06:22.302018+0000","last_became_peered":"2023-12-19T08:06:22.302018+0000","last_unstale":"2023-12-19T08:06:22.302985+0000","last_undegraded":"2023-12-19T08:06:22.302985+0000","last_fullsized":"2023-12-19T08:06:22.302985+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:34:20.704115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306362+0000","last_change":"2023-12-19T08:06:23.306362+0000","last_active":"2023-12-19T08:06:23.306362+0000","last_peered":"2023-12-19T08:06:23.306362+0000","last_clean":"2023-12-19T08:06:23.306362+0000","last_became_active":"2023-12-19T08:06:23.306067+0000","last_became_peered":"2023-12-19T08:06:23.306067+0000","last_unstale":"2023-12-19T08:06:23.306362+0000","last_undegraded":"2023-12-19T08:06:23.306362+0000","last_fullsized":"2023-12-19T08:06:23.306362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:06.579739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318755+0000","last_change":"2023-12-19T08:06:22.294326+0000","last_active":"2023-12-19T08:06:23.318755+0000","last_peered":"2023-12-19T08:06:23.318755+0000","last_clean":"2023-12-19T08:06:23.318755+0000","last_became_active":"2023-12-19T08:06:22.294011+0000","last_became_peered":"2023-12-19T08:06:22.294011+0000","last_unstale":"2023-12-19T08:06:23.318755+0000","last_undegraded":"2023-12-19T08:06:23.318755+0000","last_fullsized":"2023-12-19T08:06:23.318755+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:04:02.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.313938+0000","last_change":"2023-12-19T08:06:23.306531+0000","last_active":"2023-12-19T08:06:23.313938+0000","last_peered":"2023-12-19T08:06:23.313938+0000","last_clean":"2023-12-19T08:06:23.313938+0000","last_became_active":"2023-12-19T08:06:23.306229+0000","last_became_peered":"2023-12-19T08:06:23.306229+0000","last_unstale":"2023-12-19T08:06:23.313938+0000","last_undegraded":"2023-12-19T08:06:23.313938+0000","last_fullsized":"2023-12-19T08:06:23.313938+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:14.812267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.295653+0000","last_change":"2023-12-19T08:06:22.292433+0000","last_active":"2023-12-19T08:06:22.295653+0000","last_peered":"2023-12-19T08:06:22.295653+0000","last_clean":"2023-12-19T08:06:22.295653+0000","last_became_active":"2023-12-19T08:06:22.292115+0000","last_became_peered":"2023-12-19T08:06:22.292115+0000","last_unstale":"2023-12-19T08:06:22.295653+0000","last_undegraded":"2023-12-19T08:06:22.295653+0000","last_fullsized":"2023-12-19T08:06:22.295653+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:06:44.697786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310752+0000","last_change":"2023-12-19T08:06:23.302539+0000","last_active":"2023-12-19T08:06:23.310752+0000","last_peered":"2023-12-19T08:06:23.310752+0000","last_clean":"2023-12-19T08:06:23.310752+0000","last_became_active":"2023-12-19T08:06:23.301672+0000","last_became_peered":"2023-12-19T08:06:23.301672+0000","last_unstale":"2023-12-19T08:06:23.310752+0000","last_undegraded":"2023-12-19T08:06:23.310752+0000","last_fullsized":"2023-12-19T08:06:23.310752+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:44.721338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293901+0000","last_change":"2023-12-19T08:06:22.298102+0000","last_active":"2023-12-19T08:06:23.293901+0000","last_peered":"2023-12-19T08:06:23.293901+0000","last_clean":"2023-12-19T08:06:23.293901+0000","last_became_active":"2023-12-19T08:06:22.297812+0000","last_became_peered":"2023-12-19T08:06:22.297812+0000","last_unstale":"2023-12-19T08:06:23.293901+0000","last_undegraded":"2023-12-19T08:06:23.293901+0000","last_fullsized":"2023-12-19T08:06:23.293901+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:01.207523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301188+0000","last_change":"2023-12-19T08:06:23.301188+0000","last_active":"2023-12-19T08:06:23.301188+0000","last_peered":"2023-12-19T08:06:23.301188+0000","last_clean":"2023-12-19T08:06:23.301188+0000","last_became_active":"2023-12-19T08:06:23.300892+0000","last_became_peered":"2023-12-19T08:06:23.300892+0000","last_unstale":"2023-12-19T08:06:23.301188+0000","last_undegraded":"2023-12-19T08:06:23.301188+0000","last_fullsized":"2023-12-19T08:06:23.301188+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300416+0000","last_change":"2023-12-19T08:06:22.299696+0000","last_active":"2023-12-19T08:06:22.300416+0000","last_peered":"2023-12-19T08:06:22.300416+0000","last_clean":"2023-12-19T08:06:22.300416+0000","last_became_active":"2023-12-19T08:06:22.299337+0000","last_became_peered":"2023-12-19T08:06:22.299337+0000","last_unstale":"2023-12-19T08:06:22.300416+0000","last_undegraded":"2023-12-19T08:06:22.300416+0000","last_fullsized":"2023-12-19T08:06:22.300416+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:13:57.223292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.300820+0000","last_change":"2023-12-19T08:06:23.300820+0000","last_active":"2023-12-19T08:06:23.300820+0000","last_peered":"2023-12-19T08:06:23.300820+0000","last_clean":"2023-12-19T08:06:23.300820+0000","last_became_active":"2023-12-19T08:06:23.297955+0000","last_became_peered":"2023-12-19T08:06:23.297955+0000","last_unstale":"2023-12-19T08:06:23.300820+0000","last_undegraded":"2023-12-19T08:06:23.300820+0000","last_fullsized":"2023-12-19T08:06:23.300820+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:14.812267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316345+0000","last_change":"2023-12-19T08:06:22.288164+0000","last_active":"2023-12-19T08:06:23.316345+0000","last_peered":"2023-12-19T08:06:23.316345+0000","last_clean":"2023-12-19T08:06:23.316345+0000","last_became_active":"2023-12-19T08:06:22.286973+0000","last_became_peered":"2023-12-19T08:06:22.286973+0000","last_unstale":"2023-12-19T08:06:23.316345+0000","last_undegraded":"2023-12-19T08:06:23.316345+0000","last_fullsized":"2023-12-19T08:06:23.316345+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:31.632186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302314+0000","last_change":"2023-12-19T08:06:23.297341+0000","last_active":"2023-12-19T08:06:23.302314+0000","last_peered":"2023-12-19T08:06:23.302314+0000","last_clean":"2023-12-19T08:06:23.302314+0000","last_became_active":"2023-12-19T08:06:23.297056+0000","last_became_peered":"2023-12-19T08:06:23.297056+0000","last_unstale":"2023-12-19T08:06:23.302314+0000","last_undegraded":"2023-12-19T08:06:23.302314+0000","last_fullsized":"2023-12-19T08:06:23.302314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.302819+0000","last_change":"2023-12-19T08:06:22.301897+0000","last_active":"2023-12-19T08:06:22.302819+0000","last_peered":"2023-12-19T08:06:22.302819+0000","last_clean":"2023-12-19T08:06:22.302819+0000","last_became_active":"2023-12-19T08:06:22.301612+0000","last_became_peered":"2023-12-19T08:06:22.301612+0000","last_unstale":"2023-12-19T08:06:22.302819+0000","last_undegraded":"2023-12-19T08:06:22.302819+0000","last_fullsized":"2023-12-19T08:06:22.302819+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:09.235742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302848+0000","last_change":"2023-12-19T08:06:23.302848+0000","last_active":"2023-12-19T08:06:23.302848+0000","last_peered":"2023-12-19T08:06:23.302848+0000","last_clean":"2023-12-19T08:06:23.302848+0000","last_became_active":"2023-12-19T08:06:23.302558+0000","last_became_peered":"2023-12-19T08:06:23.302558+0000","last_unstale":"2023-12-19T08:06:23.302848+0000","last_undegraded":"2023-12-19T08:06:23.302848+0000","last_fullsized":"2023-12-19T08:06:23.302848+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.295024+0000","last_change":"2023-12-19T08:06:22.293668+0000","last_active":"2023-12-19T08:06:22.295024+0000","last_peered":"2023-12-19T08:06:22.295024+0000","last_clean":"2023-12-19T08:06:22.295024+0000","last_became_active":"2023-12-19T08:06:22.290530+0000","last_became_peered":"2023-12-19T08:06:22.290530+0000","last_unstale":"2023-12-19T08:06:22.295024+0000","last_undegraded":"2023-12-19T08:06:22.295024+0000","last_fullsized":"2023-12-19T08:06:22.295024+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:02.783487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.294147+0000","last_change":"2023-12-19T08:06:23.294147+0000","last_active":"2023-12-19T08:06:23.294147+0000","last_peered":"2023-12-19T08:06:23.294147+0000","last_clean":"2023-12-19T08:06:23.294147+0000","last_became_active":"2023-12-19T08:06:23.293852+0000","last_became_peered":"2023-12-19T08:06:23.293852+0000","last_unstale":"2023-12-19T08:06:23.294147+0000","last_undegraded":"2023-12-19T08:06:23.294147+0000","last_fullsized":"2023-12-19T08:06:23.294147+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:41:18.382344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300190+0000","last_change":"2023-12-19T08:06:22.295766+0000","last_active":"2023-12-19T08:06:22.300190+0000","last_peered":"2023-12-19T08:06:22.300190+0000","last_clean":"2023-12-19T08:06:22.300190+0000","last_became_active":"2023-12-19T08:06:22.293113+0000","last_became_peered":"2023-12-19T08:06:22.293113+0000","last_unstale":"2023-12-19T08:06:22.300190+0000","last_undegraded":"2023-12-19T08:06:22.300190+0000","last_fullsized":"2023-12-19T08:06:22.300190+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:41.268100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.289418+0000","last_change":"2023-12-19T08:06:23.289418+0000","last_active":"2023-12-19T08:06:23.289418+0000","last_peered":"2023-12-19T08:06:23.289418+0000","last_clean":"2023-12-19T08:06:23.289418+0000","last_became_active":"2023-12-19T08:06:23.289141+0000","last_became_peered":"2023-12-19T08:06:23.289141+0000","last_unstale":"2023-12-19T08:06:23.289418+0000","last_undegraded":"2023-12-19T08:06:23.289418+0000","last_fullsized":"2023-12-19T08:06:23.289418+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:05:55.773286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.281752+0000","last_change":"2023-12-19T08:06:22.295406+0000","last_active":"2023-12-19T08:06:23.281752+0000","last_peered":"2023-12-19T08:06:23.281752+0000","last_clean":"2023-12-19T08:06:23.281752+0000","last_became_active":"2023-12-19T08:06:22.294905+0000","last_became_peered":"2023-12-19T08:06:22.294905+0000","last_unstale":"2023-12-19T08:06:23.281752+0000","last_undegraded":"2023-12-19T08:06:23.281752+0000","last_fullsized":"2023-12-19T08:06:23.281752+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:10.698455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316572+0000","last_change":"2023-12-19T08:06:23.313228+0000","last_active":"2023-12-19T08:06:23.316572+0000","last_peered":"2023-12-19T08:06:23.316572+0000","last_clean":"2023-12-19T08:06:23.316572+0000","last_became_active":"2023-12-19T08:06:23.312882+0000","last_became_peered":"2023-12-19T08:06:23.312882+0000","last_unstale":"2023-12-19T08:06:23.316572+0000","last_undegraded":"2023-12-19T08:06:23.316572+0000","last_fullsized":"2023-12-19T08:06:23.316572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:57.748699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.281620+0000","last_change":"2023-12-19T08:06:22.292736+0000","last_active":"2023-12-19T08:06:23.281620+0000","last_peered":"2023-12-19T08:06:23.281620+0000","last_clean":"2023-12-19T08:06:23.281620+0000","last_became_active":"2023-12-19T08:06:22.290772+0000","last_became_peered":"2023-12-19T08:06:22.290772+0000","last_unstale":"2023-12-19T08:06:23.281620+0000","last_undegraded":"2023-12-19T08:06:23.281620+0000","last_fullsized":"2023-12-19T08:06:23.281620+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:03:58.915674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306287+0000","last_change":"2023-12-19T08:06:23.306287+0000","last_active":"2023-12-19T08:06:23.306287+0000","last_peered":"2023-12-19T08:06:23.306287+0000","last_clean":"2023-12-19T08:06:23.306287+0000","last_became_active":"2023-12-19T08:06:23.305987+0000","last_became_peered":"2023-12-19T08:06:23.305987+0000","last_unstale":"2023-12-19T08:06:23.306287+0000","last_undegraded":"2023-12-19T08:06:23.306287+0000","last_fullsized":"2023-12-19T08:06:23.306287+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:12:49.209423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300152+0000","last_change":"2023-12-19T08:06:22.295232+0000","last_active":"2023-12-19T08:06:22.300152+0000","last_peered":"2023-12-19T08:06:22.300152+0000","last_clean":"2023-12-19T08:06:22.300152+0000","last_became_active":"2023-12-19T08:06:22.294221+0000","last_became_peered":"2023-12-19T08:06:22.294221+0000","last_unstale":"2023-12-19T08:06:22.300152+0000","last_undegraded":"2023-12-19T08:06:22.300152+0000","last_fullsized":"2023-12-19T08:06:22.300152+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:39.101328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303964+0000","last_change":"2023-12-19T08:06:23.296502+0000","last_active":"2023-12-19T08:06:23.303964+0000","last_peered":"2023-12-19T08:06:23.303964+0000","last_clean":"2023-12-19T08:06:23.303964+0000","last_became_active":"2023-12-19T08:06:23.296123+0000","last_became_peered":"2023-12-19T08:06:23.296123+0000","last_unstale":"2023-12-19T08:06:23.303964+0000","last_undegraded":"2023-12-19T08:06:23.303964+0000","last_fullsized":"2023-12-19T08:06:23.303964+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.298149+0000","last_change":"2023-12-19T08:06:22.296251+0000","last_active":"2023-12-19T08:06:22.298149+0000","last_peered":"2023-12-19T08:06:22.298149+0000","last_clean":"2023-12-19T08:06:22.298149+0000","last_became_active":"2023-12-19T08:06:22.295941+0000","last_became_peered":"2023-12-19T08:06:22.295941+0000","last_unstale":"2023-12-19T08:06:22.298149+0000","last_undegraded":"2023-12-19T08:06:22.298149+0000","last_fullsized":"2023-12-19T08:06:22.298149+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:41.707156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302684+0000","last_change":"2023-12-19T08:06:23.302684+0000","last_active":"2023-12-19T08:06:23.302684+0000","last_peered":"2023-12-19T08:06:23.302684+0000","last_clean":"2023-12-19T08:06:23.302684+0000","last_became_active":"2023-12-19T08:06:23.302365+0000","last_became_peered":"2023-12-19T08:06:23.302365+0000","last_unstale":"2023-12-19T08:06:23.302684+0000","last_undegraded":"2023-12-19T08:06:23.302684+0000","last_fullsized":"2023-12-19T08:06:23.302684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:39:14.017630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.325313+0000","last_change":"2023-12-19T08:06:22.284277+0000","last_active":"2023-12-19T08:06:23.325313+0000","last_peered":"2023-12-19T08:06:23.325313+0000","last_clean":"2023-12-19T08:06:23.325313+0000","last_became_active":"2023-12-19T08:06:22.283995+0000","last_became_peered":"2023-12-19T08:06:22.283995+0000","last_unstale":"2023-12-19T08:06:23.325313+0000","last_undegraded":"2023-12-19T08:06:23.325313+0000","last_fullsized":"2023-12-19T08:06:23.325313+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:55.094079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'249","reported_seq":731,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.285572+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:06:22.285572+0000","last_peered":"2023-12-19T08:06:22.285572+0000","last_clean":"2023-12-19T08:06:22.285572+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:06:22.285572+0000","last_undegraded":"2023-12-19T08:06:22.285572+0000","last_fullsized":"2023-12-19T08:06:22.285572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575075,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311908,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059932,"statfs":{"total":96636764160,"available":96317370368,"internally_reserved":0,"allocated":2387968,"data_stored":1526841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.78900000000000003,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.619,"5min":1.6850000000000001,"15min":2.0459999999999998},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.2769999999999999,"15min":1.27},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7170000000000001,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.579}]},{"osd":1,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1619999999999999,"15min":1.161},"min":{"1min":0.70199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.429,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.516},{"interface":"front","average":{"1min":1.214,"5min":1.2250000000000001,"15min":1.1839999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.667,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.2849999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.169,"15min":1.161},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":1.415,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3,"15min":1.2649999999999999},"min":{"1min":1.0800000000000001,"5min":0.82999999999999996,"15min":0.70799999999999996},"max":{"1min":1.756,"5min":1.946,"15min":2.1739999999999999},"last":1.216}]},{"osd":3,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.288,"15min":1.284},"min":{"1min":0.878,"5min":0.878,"15min":0.83299999999999996},"max":{"1min":1.4610000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.137},{"interface":"front","average":{"1min":1.264,"5min":1.375,"15min":1.351},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.81799999999999995},"max":{"1min":1.605,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.385}]},{"osd":4,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3109999999999999,"15min":1.2430000000000001},"min":{"1min":0.85899999999999999,"5min":0.71899999999999997,"15min":0.67300000000000004},"max":{"1min":1.6619999999999999,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.361},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.292,"15min":1.24},"min":{"1min":0.80500000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.7070000000000001,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.458}]},{"osd":5,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.302,"15min":1.248},"min":{"1min":0.69699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.663,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.419,"15min":1.3720000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.751},"max":{"1min":1.8720000000000001,"5min":2.1219999999999999,"15min":2.2120000000000002},"last":1.496}]},{"osd":6,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.4950000000000001,"15min":1.4219999999999999},"min":{"1min":0.88,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7370000000000001,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.696},{"interface":"front","average":{"1min":1.325,"5min":1.421,"15min":1.3819999999999999},"min":{"1min":0.95799999999999996,"5min":0.85099999999999998,"15min":0.58299999999999996},"max":{"1min":1.8080000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.343}]}]},{"osd":6,"up_from":13,"seq":55834575074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1781760,"data_stored":936279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.36799999999999999},"max":{"1min":1.3959999999999999,"5min":1.5569999999999999,"15min":1.613},"last":0.747},{"interface":"front","average":{"1min":1.139,"5min":1.163,"15min":1.1359999999999999},"min":{"1min":0.81599999999999995,"5min":0.624,"15min":0.49399999999999999},"max":{"1min":1.6040000000000001,"5min":1.756,"15min":1.756},"last":1.3140000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1419999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.60799999999999998,"15min":0.53400000000000003},"max":{"1min":1.587,"5min":1.641,"15min":1.675},"last":1.76},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.113,"15min":1.1519999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.51900000000000002},"max":{"1min":1.425,"5min":1.5349999999999999,"15min":1.7470000000000001},"last":1.5629999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.21,"15min":1.23},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.59099999999999997},"max":{"1min":1.5429999999999999,"5min":1.746,"15min":1.847},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.206,"15min":1.21},"min":{"1min":0.76400000000000001,"5min":0.70799999999999996,"15min":0.58999999999999997},"max":{"1min":1.635,"5min":1.694,"15min":1.7929999999999999},"last":1.6000000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.228,"15min":1.2549999999999999},"min":{"1min":0.97299999999999998,"5min":0.77200000000000002,"15min":0.65400000000000003},"max":{"1min":1.423,"5min":1.8200000000000001,"15min":1.958},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.3069999999999999},"min":{"1min":0.95299999999999996,"5min":0.78200000000000003,"15min":0.63200000000000001},"max":{"1min":1.677,"5min":1.8540000000000001,"15min":1.9570000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.159,"15min":1.1679999999999999},"min":{"1min":0.84599999999999997,"5min":0.55500000000000005,"15min":0.47199999999999998},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8480000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":1.0029999999999999,"5min":0.72699999999999998,"15min":0.70099999999999996},"max":{"1min":1.754,"5min":1.9139999999999999,"15min":2.1440000000000001},"last":1.714}]},{"osd":5,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.276,"15min":1.28},"min":{"1min":0.90400000000000003,"5min":0.70799999999999996,"15min":0.67600000000000005},"max":{"1min":1.792,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.2869999999999999,"15min":1.2889999999999999},"min":{"1min":0.876,"5min":0.75700000000000001,"15min":0.56499999999999995},"max":{"1min":1.5920000000000001,"5min":1.7250000000000001,"15min":2.2450000000000001},"last":1.234}]},{"osd":7,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3839999999999999,"15min":1.3640000000000001},"min":{"1min":0.99399999999999999,"5min":0.73599999999999999,"15min":0.55100000000000005},"max":{"1min":1.8340000000000001,"5min":1.9239999999999999,"15min":2.04},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.431,"15min":1.4179999999999999},"min":{"1min":0.84699999999999998,"5min":0.76600000000000001,"15min":0.58099999999999996},"max":{"1min":1.976,"5min":1.976,"15min":2.137},"last":1.661}]}]},{"osd":5,"up_from":13,"seq":55834575074,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303128,"kb_used_data":1744,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068712,"statfs":{"total":96636764160,"available":96326361088,"internally_reserved":0,"allocated":1785856,"data_stored":935885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1779999999999999,"15min":1.228},"min":{"1min":0.78900000000000003,"5min":0.67900000000000005,"15min":0.53000000000000003},"max":{"1min":1.5189999999999999,"5min":1.714,"15min":2.6659999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2430000000000001,"15min":1.266},"min":{"1min":0.86499999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6599999999999999,"5min":1.8129999999999999,"15min":1.954},"last":1.3200000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1140000000000001,"15min":1.1419999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.478,"5min":1.585,"15min":1.958},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.157,"15min":1.171},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.5149999999999999,"5min":1.8620000000000001,"15min":2.7679999999999998},"last":1.173}]},{"osd":2,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.1990000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.5009999999999999,"5min":1.6739999999999999,"15min":2.0259999999999998},"last":1.216},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.175,"15min":1.2050000000000001},"min":{"1min":0.82299999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6040000000000001,"5min":1.6679999999999999,"15min":1.7490000000000001},"last":1.24}]},{"osd":3,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.22,"15min":1.2509999999999999},"min":{"1min":0.84299999999999997,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":2.9489999999999998},"last":1.446},{"interface":"front","average":{"1min":1.226,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.85199999999999998,"5min":0.79500000000000004,"15min":0.76600000000000001},"max":{"1min":1.647,"5min":1.744,"15min":2.2130000000000001},"last":1.371}]},{"osd":4,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.181,"15min":1.196},"min":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.629},"max":{"1min":1.6910000000000001,"5min":1.7749999999999999,"15min":2.1419999999999999},"last":1.073},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.268,"15min":1.272},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.59199999999999997},"max":{"1min":1.6259999999999999,"5min":1.673,"15min":2.843},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.321,"15min":1.3100000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7769999999999999,"5min":1.869,"15min":2.121},"last":1.139},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3480000000000001,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.77300000000000002},"max":{"1min":1.7869999999999999,"5min":1.96,"15min":4.1959999999999997},"last":1.3899999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.387,"15min":1.4099999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.74399999999999999},"max":{"1min":1.77,"5min":1.96,"15min":2.9929999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.478,"15min":1.456},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.91500000000000004},"max":{"1min":1.927,"5min":1.929,"15min":2.0779999999999998},"last":1.4179999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575074,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307228,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064612,"statfs":{"total":96636764160,"available":96322162688,"internally_reserved":0,"allocated":1789952,"data_stored":937261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1990000000000001,"15min":1.21},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.3109999999999999,"15min":1.3089999999999999},"min":{"1min":0.83799999999999997,"5min":0.80300000000000005,"15min":0.70299999999999996},"max":{"1min":1.796,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.397}]},{"osd":1,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.165,"15min":1.169},"min":{"1min":0.86399999999999999,"5min":0.69799999999999995,"15min":0.57299999999999995},"max":{"1min":1.7130000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.129,"15min":1.1379999999999999},"min":{"1min":0.67400000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4079999999999999,"5min":1.569,"15min":1.569},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0900000000000001,"15min":1.127},"min":{"1min":0.85699999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.538,"5min":1.538,"15min":1.746},"last":1.369},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.875,"5min":0.72399999999999998,"15min":0.69699999999999995},"max":{"1min":1.569,"5min":1.5980000000000001,"15min":1.8959999999999999},"last":1.5489999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.256,"15min":1.268},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.65900000000000003},"max":{"1min":1.5489999999999999,"5min":1.831,"15min":1.831},"last":1.274},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.4219999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.214,"15min":1.2},"min":{"1min":0.81599999999999995,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.7050000000000001,"5min":1.905,"15min":1.905},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.208,"15min":1.194},"min":{"1min":0.96399999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6679999999999999,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.95499999999999996}]},{"osd":6,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.2490000000000001,"15min":1.234},"min":{"1min":0.90300000000000002,"5min":0.73899999999999999,"15min":0.67400000000000004},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.389,"15min":1.377},"min":{"1min":0.92900000000000005,"5min":0.70899999999999996,"15min":0.67900000000000005},"max":{"1min":2.2410000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.4690000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3540000000000001,"15min":1.339},"min":{"1min":1.0009999999999999,"5min":0.58799999999999997,"15min":0.46300000000000002},"max":{"1min":1.837,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4139999999999999,"15min":1.3959999999999999},"min":{"1min":0.997,"5min":0.80200000000000005,"15min":0.70999999999999996},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.504}]}]},{"osd":3,"up_from":13,"seq":55834575074,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307232,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064608,"statfs":{"total":96636764160,"available":96322158592,"internally_reserved":0,"allocated":1794048,"data_stored":937295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.47799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3320000000000001,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.97699999999999998,"15min":0.96599999999999997},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.39200000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.72699999999999998}]},{"osd":1,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.90400000000000003,"15min":0.90600000000000003},"min":{"1min":0.66500000000000004,"5min":0.47099999999999997,"15min":0.45600000000000002},"max":{"1min":1.1559999999999999,"5min":1.609,"15min":1.609},"last":1.216},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.042,"15min":1.038},"min":{"1min":0.71799999999999997,"5min":0.47799999999999998,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":2.423,"15min":2.423},"last":0.95599999999999996}]},{"osd":2,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.042,"15min":1.0680000000000001},"min":{"1min":0.57899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.167,"15min":1.1579999999999999},"min":{"1min":0.64900000000000002,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.5800000000000001,"5min":2.48,"15min":2.48},"last":0.83399999999999996}]},{"osd":4,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.228,"15min":1.248},"min":{"1min":0.84999999999999998,"5min":0.81200000000000006,"15min":0.66800000000000004},"max":{"1min":1.887,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.012},{"interface":"front","average":{"1min":1.212,"5min":1.2310000000000001,"15min":1.2390000000000001},"min":{"1min":0.71299999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.2909999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2709999999999999,"15min":1.266},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.554,"5min":1.996,"15min":1.996},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.3300000000000001,"15min":1.331},"min":{"1min":0.85599999999999998,"5min":0.753,"15min":0.753},"max":{"1min":1.552,"5min":2.161,"15min":2.161},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.409,"15min":1.385},"min":{"1min":0.82399999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9139999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.161},{"interface":"front","average":{"1min":1.288,"5min":1.323,"15min":1.3540000000000001},"min":{"1min":0.87,"5min":0.87,"15min":0.82499999999999996},"max":{"1min":2.0019999999999998,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2709999999999999,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.397,"15min":1.411},"min":{"1min":1.0269999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.772,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.677}]}]},{"osd":2,"up_from":13,"seq":55834575074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307216,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064624,"statfs":{"total":96636764160,"available":96322174976,"internally_reserved":0,"allocated":1777664,"data_stored":935725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.89300000000000002,"15min":0.90500000000000003},"min":{"1min":0.51000000000000001,"5min":0.44600000000000001,"15min":0.437},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.776},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0129999999999999,"15min":1.0409999999999999},"min":{"1min":0.63100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5369999999999999,"5min":1.7769999999999999,"15min":1.8460000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.97199999999999998,"15min":0.95899999999999996},"min":{"1min":0.63,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.4450000000000001,"5min":1.669,"15min":1.669},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0089999999999999,"15min":1.0109999999999999},"min":{"1min":0.66500000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"max":{"1min":1.2809999999999999,"5min":1.835,"15min":1.835},"last":0.69399999999999995}]},{"osd":3,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.96299999999999997,"15min":0.97399999999999998},"min":{"1min":0.56599999999999995,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":2.3410000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1220000000000001,"15min":1.123},"min":{"1min":0.77900000000000003,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.329,"15min":1.325},"min":{"1min":0.97499999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0329999999999999,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3120000000000001,"15min":1.3},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.877,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.099}]},{"osd":5,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":1.008,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.601,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.98799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.96299999999999997}]},{"osd":6,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.288,"15min":1.296},"min":{"1min":0.94399999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.391,"15min":1.377},"min":{"1min":1.1080000000000001,"5min":0.94899999999999995,"15min":0.92100000000000004},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":1.218}]},{"osd":7,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3759999999999999,"15min":1.383},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.95899999999999996},"max":{"1min":1.661,"5min":1.9730000000000001,"15min":2.214},"last":1.175},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.377,"15min":1.373},"min":{"1min":0.99399999999999999,"5min":0.98699999999999999,"15min":0.88200000000000001},"max":{"1min":4.109,"5min":4.109,"15min":4.109},"last":1.1339999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307232,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064608,"statfs":{"total":96636764160,"available":96322158592,"internally_reserved":0,"allocated":1794048,"data_stored":942903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.86299999999999999,"15min":0.85599999999999998},"min":{"1min":0.61599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.252,"5min":1.647,"15min":1.661},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92200000000000004,"15min":0.93600000000000005},"min":{"1min":0.52900000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.421,"5min":2.2050000000000001,"15min":2.3570000000000002},"last":0.64500000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.56899999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.165,"5min":1.96,"15min":2.96},"last":0.627},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0449999999999999,"15min":1.0509999999999999},"min":{"1min":0.64100000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.548,"5min":2.323,"15min":3.1349999999999998},"last":0.77200000000000002}]},{"osd":3,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.159,"15min":1.161},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.641,"5min":2.2530000000000001,"15min":3.1800000000000002},"last":0.874},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51200000000000001},"max":{"1min":1.746,"5min":2.1299999999999999,"15min":2.407},"last":0.60199999999999998}]},{"osd":4,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1899999999999999},"min":{"1min":0.871,"5min":0.502,"15min":0.502},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.5030000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.325,"15min":1.3109999999999999},"min":{"1min":0.89500000000000002,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.875,"5min":2.2890000000000001,"15min":3.04},"last":1.171}]},{"osd":5,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2689999999999999,"15min":1.276},"min":{"1min":0.87,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.7330000000000001,"5min":1.996,"15min":2.5390000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.2849999999999999,"15min":1.286},"min":{"1min":0.98099999999999998,"5min":0.85899999999999999,"15min":0.63800000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.3450000000000002},"last":1.0740000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3740000000000001,"15min":1.3819999999999999},"min":{"1min":1.0629999999999999,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.772,"5min":2.3559999999999999,"15min":3.089},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.367,"15min":1.365},"min":{"1min":0.98899999999999999,"5min":0.76300000000000001,"15min":0.747},"max":{"1min":1.919,"5min":2.169,"15min":2.46},"last":1.135}]},{"osd":7,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4179999999999999,"15min":1.4199999999999999},"min":{"1min":0.99299999999999999,"5min":0.89300000000000002,"15min":0.88500000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.504},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":1.0309999999999999,"5min":0.81799999999999995,"15min":0.73999999999999999},"max":{"1min":2.0779999999999998,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.1539999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575074,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":2371584,"data_stored":1525104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.90800000000000003,"15min":0.91300000000000003},"min":{"1min":0.64900000000000002,"5min":0.54500000000000004,"15min":0.496},"max":{"1min":1.3700000000000001,"5min":1.466,"15min":1.5289999999999999},"last":1.147}]},{"osd":2,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99399999999999999,"15min":0.97799999999999998},"min":{"1min":0.77600000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.413,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.522},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.01,"15min":1.03},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.56899999999999995},"max":{"1min":1.3779999999999999,"5min":1.504,"15min":1.6000000000000001},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.95499999999999996,"15min":0.96299999999999997},"min":{"1min":0.59499999999999997,"5min":0.54500000000000004,"15min":0.51400000000000001},"max":{"1min":1.4590000000000001,"5min":1.5529999999999999,"15min":1.671},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.155,"5min":1.133,"15min":1.133},"min":{"1min":0.76100000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.522,"5min":1.663,"15min":1.6659999999999999},"last":1.2490000000000001}]},{"osd":4,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3360000000000001,"15min":1.3160000000000001},"min":{"1min":1.022,"5min":0.86499999999999999,"15min":0.71399999999999997},"max":{"1min":1.7,"5min":1.7,"15min":1.8620000000000001},"last":1.643},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.337,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.296}]},{"osd":5,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.353,"15min":1.353},"min":{"1min":1.0449999999999999,"5min":0.83399999999999996,"15min":0.74299999999999999},"max":{"1min":1.7849999999999999,"5min":1.9099999999999999,"15min":2.085},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.3560000000000001,"15min":1.357},"min":{"1min":0.97499999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8180000000000001,"5min":1.843,"15min":2.125},"last":1.821}]},{"osd":6,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.2689999999999999},"min":{"1min":1.036,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.6479999999999999,"5min":1.97,"15min":1.97},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.341,"15min":1.3220000000000001},"min":{"1min":0.94899999999999995,"5min":0.91900000000000004,"15min":0.75600000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.754}]},{"osd":7,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3220000000000001,"15min":1.3160000000000001},"min":{"1min":1.1140000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7090000000000001,"5min":1.8009999999999999,"15min":1.887},"last":1.599},{"interface":"front","average":{"1min":1.401,"5min":1.3839999999999999,"15min":1.369},"min":{"1min":1.121,"5min":0.88600000000000001,"15min":0.67700000000000005},"max":{"1min":1.756,"5min":1.903,"15min":2.0059999999999998},"last":1.71}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:06:27.573 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T08:06:27.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.765+0000 7f1a36243700 1 Processor -- start 2023-12-19T08:06:27.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.765+0000 7f1a36243700 1 -- start start 2023-12-19T08:06:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a36243700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 0x7f1a300c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a36243700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a3015cc20 0x7f1a3015d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a36243700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3006b470 con 0x7f1a3006b900 2023-12-19T08:06:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a36243700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3006e110 con 0x7f1a3015cc20 2023-12-19T08:06:27.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a36243700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a30161d10 con 0x7f1a300c6900 2023-12-19T08:06:27.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a3015cc20 0x7f1a3015d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a3015cc20 0x7f1a3015d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53062/0 (socket says 172.21.15.111:53062) 2023-12-19T08:06:27.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 -- 172.21.15.111:0/2854358210 learned_addr learned my addr 172.21.15.111:0/2854358210 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:27.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35241700 1 --2- 172.21.15.111:0/2854358210 >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 0x7f1a300c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 -- 172.21.15.111:0/2854358210 >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 msgr2=0x7f1a300c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 --2- 172.21.15.111:0/2854358210 >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 0x7f1a300c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 -- 172.21.15.111:0/2854358210 >> v1:172.21.15.143:6789/0 conn(0x7f1a3006b900 legacy=0x7f1a3006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 -- 172.21.15.111:0/2854358210 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3006bd70 con 0x7f1a3015cc20 2023-12-19T08:06:27.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35241700 1 --2- 172.21.15.111:0/2854358210 >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 0x7f1a300c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:27.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a35a42700 1 --2- 172.21.15.111:0/2854358210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a3015cc20 0x7f1a3015d080 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1a3006c570 tx=0x7f1a28009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef99c72cf36c5ba server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a27fff700 1 -- 172.21.15.111:0/2854358210 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2800e070 con 0x7f1a3015cc20 2023-12-19T08:06:27.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a27fff700 1 -- 172.21.15.111:0/2854358210 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2800ba50 con 0x7f1a3015cc20 2023-12-19T08:06:27.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.769+0000 7f1a27fff700 1 -- 172.21.15.111:0/2854358210 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a280092e0 con 0x7f1a3015cc20 2023-12-19T08:06:27.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- 172.21.15.111:0/2854358210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a3015cc20 msgr2=0x7f1a3015d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 --2- 172.21.15.111:0/2854358210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a3015cc20 0x7f1a3015d080 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1a3006c570 tx=0x7f1a28009840 comp rx=0 tx=0).stop 2023-12-19T08:06:27.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- 172.21.15.111:0/2854358210 shutdown_connections 2023-12-19T08:06:27.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 --2- 172.21.15.111:0/2854358210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a3015cc20 0x7f1a3015d080 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 --2- 172.21.15.111:0/2854358210 >> v2:172.21.15.143:3300/0 conn(0x7f1a300c6900 0x7f1a300c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- 172.21.15.111:0/2854358210 >> 172.21.15.111:0/2854358210 conn(0x7f1a30067bb0 msgr2=0x7f1a300cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:27.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- 172.21.15.111:0/2854358210 shutdown_connections 2023-12-19T08:06:27.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- 172.21.15.111:0/2854358210 wait complete. 2023-12-19T08:06:27.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 Processor -- start 2023-12-19T08:06:27.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.773+0000 7f1a36243700 1 -- start start 2023-12-19T08:06:27.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3006b900 0x7f1a301590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3015cc20 0x7f1a3016da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300aeb80 con 0x7f1a300c6900 2023-12-19T08:06:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300af700 con 0x7f1a3006b900 2023-12-19T08:06:27.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300aedd0 con 0x7f1a3015cc20 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53068/0 (socket says 172.21.15.111:53068) 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 -- 172.21.15.111:0/3517258369 learned_addr learned my addr 172.21.15.111:0/3517258369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a35241700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3006b900 0x7f1a301590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a35a42700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3015cc20 0x7f1a3016da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 -- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3015cc20 msgr2=0x7f1a3016da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3015cc20 0x7f1a3016da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 -- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3006b900 msgr2=0x7f1a301590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3006b900 0x7f1a301590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 -- 172.21.15.111:0/3517258369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28009ae0 con 0x7f1a300c6900 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a34a40700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c5c0 tx=0x7f1a2000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011590 con 0x7f1a300c6900 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20011710 con 0x7f1a300c6900 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011ad0 con 0x7f1a300c6900 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a300af0b0 con 0x7f1a300c6900 2023-12-19T08:06:27.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.777+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a301595f0 con 0x7f1a300c6900 2023-12-19T08:06:27.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.781+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3004be40 con 0x7f1a300c6900 2023-12-19T08:06:27.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.781+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1a2001b3b0 con 0x7f1a300c6900 2023-12-19T08:06:27.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.785+0000 7f1a257fa700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 0x7f1a14062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:27.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.785+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f1a2007c230 con 0x7f1a300c6900 2023-12-19T08:06:27.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.785+0000 7f1a35241700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 0x7f1a14062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:27.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.785+0000 7f1a35241700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 0x7f1a14062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c006f00 tx=0x7f1a2c008040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.789+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a2004c310 con 0x7f1a300c6900 2023-12-19T08:06:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.957+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a300af840 con 0x7f1a14060670 2023-12-19T08:06:27.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.977+0000 7f1a257fa700 1 -- 172.21.15.111:0/3517258369 <== mgr.7981 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286961 (secure 0 0 0) 0x7f1a300af840 con 0x7f1a14060670 2023-12-19T08:06:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 msgr2=0x7f1a14062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 0x7f1a14062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c006f00 tx=0x7f1a2c008040 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 msgr2=0x7f1a3015fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c5c0 tx=0x7f1a2000c8d0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 shutdown_connections 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f1a14060670 0x7f1a14062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1a3015cc20 0x7f1a3016da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a300c6900 0x7f1a3015fbb0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 --2- 172.21.15.111:0/3517258369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3006b900 0x7f1a301590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 >> 172.21.15.111:0/3517258369 conn(0x7f1a30067bb0 msgr2=0x7f1a30159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:27.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 shutdown_connections 2023-12-19T08:06:27.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:27.981+0000 7f1a36243700 1 -- 172.21.15.111:0/3517258369 wait complete. 2023-12-19T08:06:27.989 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T08:06:28.007 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T08:06:26.749718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454864,"kb_used_data":15120,"kb_used_omap":0,"kb_used_meta":2439680,"kb_avail":752519856,"statfs":{"total":773094113280,"available":770580332544,"internally_reserved":0,"allocated":15482880,"data_stored":8677293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2498232320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.489829"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307642+0000","last_change":"2023-12-19T08:06:23.301263+0000","last_active":"2023-12-19T08:06:23.307642+0000","last_peered":"2023-12-19T08:06:23.307642+0000","last_clean":"2023-12-19T08:06:23.307642+0000","last_became_active":"2023-12-19T08:06:23.299353+0000","last_became_peered":"2023-12-19T08:06:23.299353+0000","last_unstale":"2023-12-19T08:06:23.307642+0000","last_undegraded":"2023-12-19T08:06:23.307642+0000","last_fullsized":"2023-12-19T08:06:23.307642+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310658+0000","last_change":"2023-12-19T08:06:23.303294+0000","last_active":"2023-12-19T08:06:23.310658+0000","last_peered":"2023-12-19T08:06:23.310658+0000","last_clean":"2023-12-19T08:06:23.310658+0000","last_became_active":"2023-12-19T08:06:23.303019+0000","last_became_peered":"2023-12-19T08:06:23.303019+0000","last_unstale":"2023-12-19T08:06:23.310658+0000","last_undegraded":"2023-12-19T08:06:23.310658+0000","last_fullsized":"2023-12-19T08:06:23.310658+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:06.579739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.313024+0000","last_change":"2023-12-19T08:06:23.311256+0000","last_active":"2023-12-19T08:06:23.313024+0000","last_peered":"2023-12-19T08:06:23.313024+0000","last_clean":"2023-12-19T08:06:23.313024+0000","last_became_active":"2023-12-19T08:06:23.310990+0000","last_became_peered":"2023-12-19T08:06:23.310990+0000","last_unstale":"2023-12-19T08:06:23.313024+0000","last_undegraded":"2023-12-19T08:06:23.313024+0000","last_fullsized":"2023-12-19T08:06:23.313024+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:21:51.718461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306364+0000","last_change":"2023-12-19T08:06:23.306364+0000","last_active":"2023-12-19T08:06:23.306364+0000","last_peered":"2023-12-19T08:06:23.306364+0000","last_clean":"2023-12-19T08:06:23.306364+0000","last_became_active":"2023-12-19T08:06:23.305840+0000","last_became_peered":"2023-12-19T08:06:23.305840+0000","last_unstale":"2023-12-19T08:06:23.306364+0000","last_undegraded":"2023-12-19T08:06:23.306364+0000","last_fullsized":"2023-12-19T08:06:23.306364+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310736+0000","last_change":"2023-12-19T08:06:23.309825+0000","last_active":"2023-12-19T08:06:23.310736+0000","last_peered":"2023-12-19T08:06:23.310736+0000","last_clean":"2023-12-19T08:06:23.310736+0000","last_became_active":"2023-12-19T08:06:23.309500+0000","last_became_peered":"2023-12-19T08:06:23.309500+0000","last_unstale":"2023-12-19T08:06:23.310736+0000","last_undegraded":"2023-12-19T08:06:23.310736+0000","last_fullsized":"2023-12-19T08:06:23.310736+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:11.892726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.300983+0000","last_change":"2023-12-19T08:06:23.300983+0000","last_active":"2023-12-19T08:06:23.300983+0000","last_peered":"2023-12-19T08:06:23.300983+0000","last_clean":"2023-12-19T08:06:23.300983+0000","last_became_active":"2023-12-19T08:06:23.300676+0000","last_became_peered":"2023-12-19T08:06:23.300676+0000","last_unstale":"2023-12-19T08:06:23.300983+0000","last_undegraded":"2023-12-19T08:06:23.300983+0000","last_fullsized":"2023-12-19T08:06:23.300983+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:20.450021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303087+0000","last_change":"2023-12-19T08:06:23.303087+0000","last_active":"2023-12-19T08:06:23.303087+0000","last_peered":"2023-12-19T08:06:23.303087+0000","last_clean":"2023-12-19T08:06:23.303087+0000","last_became_active":"2023-12-19T08:06:23.301801+0000","last_became_peered":"2023-12-19T08:06:23.301801+0000","last_unstale":"2023-12-19T08:06:23.303087+0000","last_undegraded":"2023-12-19T08:06:23.303087+0000","last_fullsized":"2023-12-19T08:06:23.303087+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:22.145749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.311708+0000","last_change":"2023-12-19T08:06:23.305076+0000","last_active":"2023-12-19T08:06:23.311708+0000","last_peered":"2023-12-19T08:06:23.311708+0000","last_clean":"2023-12-19T08:06:23.311708+0000","last_became_active":"2023-12-19T08:06:23.304772+0000","last_became_peered":"2023-12-19T08:06:23.304772+0000","last_unstale":"2023-12-19T08:06:23.311708+0000","last_undegraded":"2023-12-19T08:06:23.311708+0000","last_fullsized":"2023-12-19T08:06:23.311708+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:20:30.465211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.315450+0000","last_change":"2023-12-19T08:06:23.315450+0000","last_active":"2023-12-19T08:06:23.315450+0000","last_peered":"2023-12-19T08:06:23.315450+0000","last_clean":"2023-12-19T08:06:23.315450+0000","last_became_active":"2023-12-19T08:06:23.315137+0000","last_became_peered":"2023-12-19T08:06:23.315137+0000","last_unstale":"2023-12-19T08:06:23.315450+0000","last_undegraded":"2023-12-19T08:06:23.315450+0000","last_fullsized":"2023-12-19T08:06:23.315450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301823+0000","last_change":"2023-12-19T08:06:23.295404+0000","last_active":"2023-12-19T08:06:23.301823+0000","last_peered":"2023-12-19T08:06:23.301823+0000","last_clean":"2023-12-19T08:06:23.301823+0000","last_became_active":"2023-12-19T08:06:23.295095+0000","last_became_peered":"2023-12-19T08:06:23.295095+0000","last_unstale":"2023-12-19T08:06:23.301823+0000","last_undegraded":"2023-12-19T08:06:23.301823+0000","last_fullsized":"2023-12-19T08:06:23.301823+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304128+0000","last_change":"2023-12-19T08:06:23.293699+0000","last_active":"2023-12-19T08:06:23.304128+0000","last_peered":"2023-12-19T08:06:23.304128+0000","last_clean":"2023-12-19T08:06:23.304128+0000","last_became_active":"2023-12-19T08:06:23.293396+0000","last_became_peered":"2023-12-19T08:06:23.293396+0000","last_unstale":"2023-12-19T08:06:23.304128+0000","last_undegraded":"2023-12-19T08:06:23.304128+0000","last_fullsized":"2023-12-19T08:06:23.304128+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.284556+0000","last_change":"2023-12-19T08:06:23.284556+0000","last_active":"2023-12-19T08:06:23.284556+0000","last_peered":"2023-12-19T08:06:23.284556+0000","last_clean":"2023-12-19T08:06:23.284556+0000","last_became_active":"2023-12-19T08:06:23.284242+0000","last_became_peered":"2023-12-19T08:06:23.284242+0000","last_unstale":"2023-12-19T08:06:23.284556+0000","last_undegraded":"2023-12-19T08:06:23.284556+0000","last_fullsized":"2023-12-19T08:06:23.284556+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:44.721338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303986+0000","last_change":"2023-12-19T08:06:23.297002+0000","last_active":"2023-12-19T08:06:23.303986+0000","last_peered":"2023-12-19T08:06:23.303986+0000","last_clean":"2023-12-19T08:06:23.303986+0000","last_became_active":"2023-12-19T08:06:23.296722+0000","last_became_peered":"2023-12-19T08:06:23.296722+0000","last_unstale":"2023-12-19T08:06:23.303986+0000","last_undegraded":"2023-12-19T08:06:23.303986+0000","last_fullsized":"2023-12-19T08:06:23.303986+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.312996+0000","last_change":"2023-12-19T08:06:23.310857+0000","last_active":"2023-12-19T08:06:23.312996+0000","last_peered":"2023-12-19T08:06:23.312996+0000","last_clean":"2023-12-19T08:06:23.312996+0000","last_became_active":"2023-12-19T08:06:23.310492+0000","last_became_peered":"2023-12-19T08:06:23.310492+0000","last_unstale":"2023-12-19T08:06:23.312996+0000","last_undegraded":"2023-12-19T08:06:23.312996+0000","last_fullsized":"2023-12-19T08:06:23.312996+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:24:09.954917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303733+0000","last_change":"2023-12-19T08:06:23.293689+0000","last_active":"2023-12-19T08:06:23.303733+0000","last_peered":"2023-12-19T08:06:23.303733+0000","last_clean":"2023-12-19T08:06:23.303733+0000","last_became_active":"2023-12-19T08:06:23.293384+0000","last_became_peered":"2023-12-19T08:06:23.293384+0000","last_unstale":"2023-12-19T08:06:23.303733+0000","last_undegraded":"2023-12-19T08:06:23.303733+0000","last_fullsized":"2023-12-19T08:06:23.303733+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:49.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.292627+0000","last_change":"2023-12-19T08:06:23.292627+0000","last_active":"2023-12-19T08:06:23.292627+0000","last_peered":"2023-12-19T08:06:23.292627+0000","last_clean":"2023-12-19T08:06:23.292627+0000","last_became_active":"2023-12-19T08:06:23.292335+0000","last_became_peered":"2023-12-19T08:06:23.292335+0000","last_unstale":"2023-12-19T08:06:23.292627+0000","last_undegraded":"2023-12-19T08:06:23.292627+0000","last_fullsized":"2023-12-19T08:06:23.292627+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:59:03.204930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301148+0000","last_change":"2023-12-19T08:06:23.289203+0000","last_active":"2023-12-19T08:06:23.301148+0000","last_peered":"2023-12-19T08:06:23.301148+0000","last_clean":"2023-12-19T08:06:23.301148+0000","last_became_active":"2023-12-19T08:06:23.288827+0000","last_became_peered":"2023-12-19T08:06:23.288827+0000","last_unstale":"2023-12-19T08:06:23.301148+0000","last_undegraded":"2023-12-19T08:06:23.301148+0000","last_fullsized":"2023-12-19T08:06:23.301148+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310640+0000","last_change":"2023-12-19T08:06:23.303166+0000","last_active":"2023-12-19T08:06:23.310640+0000","last_peered":"2023-12-19T08:06:23.310640+0000","last_clean":"2023-12-19T08:06:23.310640+0000","last_became_active":"2023-12-19T08:06:23.302878+0000","last_became_peered":"2023-12-19T08:06:23.302878+0000","last_unstale":"2023-12-19T08:06:23.310640+0000","last_undegraded":"2023-12-19T08:06:23.310640+0000","last_fullsized":"2023-12-19T08:06:23.310640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:05.160858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310563+0000","last_change":"2023-12-19T08:06:23.310563+0000","last_active":"2023-12-19T08:06:23.310563+0000","last_peered":"2023-12-19T08:06:23.310563+0000","last_clean":"2023-12-19T08:06:23.310563+0000","last_became_active":"2023-12-19T08:06:23.310253+0000","last_became_peered":"2023-12-19T08:06:23.310253+0000","last_unstale":"2023-12-19T08:06:23.310563+0000","last_undegraded":"2023-12-19T08:06:23.310563+0000","last_fullsized":"2023-12-19T08:06:23.310563+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:59.013369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293911+0000","last_change":"2023-12-19T08:06:23.293911+0000","last_active":"2023-12-19T08:06:23.293911+0000","last_peered":"2023-12-19T08:06:23.293911+0000","last_clean":"2023-12-19T08:06:23.293911+0000","last_became_active":"2023-12-19T08:06:23.293608+0000","last_became_peered":"2023-12-19T08:06:23.293608+0000","last_unstale":"2023-12-19T08:06:23.293911+0000","last_undegraded":"2023-12-19T08:06:23.293911+0000","last_fullsized":"2023-12-19T08:06:23.293911+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:08.032269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301865+0000","last_change":"2023-12-19T08:06:23.301865+0000","last_active":"2023-12-19T08:06:23.301865+0000","last_peered":"2023-12-19T08:06:23.301865+0000","last_clean":"2023-12-19T08:06:23.301865+0000","last_became_active":"2023-12-19T08:06:23.301280+0000","last_became_peered":"2023-12-19T08:06:23.301280+0000","last_unstale":"2023-12-19T08:06:23.301865+0000","last_undegraded":"2023-12-19T08:06:23.301865+0000","last_fullsized":"2023-12-19T08:06:23.301865+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:47:01.810637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307526+0000","last_change":"2023-12-19T08:06:23.302294+0000","last_active":"2023-12-19T08:06:23.307526+0000","last_peered":"2023-12-19T08:06:23.307526+0000","last_clean":"2023-12-19T08:06:23.307526+0000","last_became_active":"2023-12-19T08:06:23.302012+0000","last_became_peered":"2023-12-19T08:06:23.302012+0000","last_unstale":"2023-12-19T08:06:23.307526+0000","last_undegraded":"2023-12-19T08:06:23.307526+0000","last_fullsized":"2023-12-19T08:06:23.307526+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324511+0000","last_change":"2023-12-19T08:06:23.324511+0000","last_active":"2023-12-19T08:06:23.324511+0000","last_peered":"2023-12-19T08:06:23.324511+0000","last_clean":"2023-12-19T08:06:23.324511+0000","last_became_active":"2023-12-19T08:06:23.323971+0000","last_became_peered":"2023-12-19T08:06:23.323971+0000","last_unstale":"2023-12-19T08:06:23.324511+0000","last_undegraded":"2023-12-19T08:06:23.324511+0000","last_fullsized":"2023-12-19T08:06:23.324511+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308679+0000","last_change":"2023-12-19T08:06:23.308679+0000","last_active":"2023-12-19T08:06:23.308679+0000","last_peered":"2023-12-19T08:06:23.308679+0000","last_clean":"2023-12-19T08:06:23.308679+0000","last_became_active":"2023-12-19T08:06:23.308221+0000","last_became_peered":"2023-12-19T08:06:23.308221+0000","last_unstale":"2023-12-19T08:06:23.308679+0000","last_undegraded":"2023-12-19T08:06:23.308679+0000","last_fullsized":"2023-12-19T08:06:23.308679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.295135+0000","last_change":"2023-12-19T08:06:23.295135+0000","last_active":"2023-12-19T08:06:23.295135+0000","last_peered":"2023-12-19T08:06:23.295135+0000","last_clean":"2023-12-19T08:06:23.295135+0000","last_became_active":"2023-12-19T08:06:23.294867+0000","last_became_peered":"2023-12-19T08:06:23.294867+0000","last_unstale":"2023-12-19T08:06:23.295135+0000","last_undegraded":"2023-12-19T08:06:23.295135+0000","last_fullsized":"2023-12-19T08:06:23.295135+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:06:21.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304347+0000","last_change":"2023-12-19T08:06:23.304347+0000","last_active":"2023-12-19T08:06:23.304347+0000","last_peered":"2023-12-19T08:06:23.304347+0000","last_clean":"2023-12-19T08:06:23.304347+0000","last_became_active":"2023-12-19T08:06:23.304019+0000","last_became_peered":"2023-12-19T08:06:23.304019+0000","last_unstale":"2023-12-19T08:06:23.304347+0000","last_undegraded":"2023-12-19T08:06:23.304347+0000","last_fullsized":"2023-12-19T08:06:23.304347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:52:04.452778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.305901+0000","last_change":"2023-12-19T08:06:23.305901+0000","last_active":"2023-12-19T08:06:23.305901+0000","last_peered":"2023-12-19T08:06:23.305901+0000","last_clean":"2023-12-19T08:06:23.305901+0000","last_became_active":"2023-12-19T08:06:23.305575+0000","last_became_peered":"2023-12-19T08:06:23.305575+0000","last_unstale":"2023-12-19T08:06:23.305901+0000","last_undegraded":"2023-12-19T08:06:23.305901+0000","last_fullsized":"2023-12-19T08:06:23.305901+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:20.203507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.285330+0000","last_change":"2023-12-19T08:06:23.285330+0000","last_active":"2023-12-19T08:06:23.285330+0000","last_peered":"2023-12-19T08:06:23.285330+0000","last_clean":"2023-12-19T08:06:23.285330+0000","last_became_active":"2023-12-19T08:06:23.284503+0000","last_became_peered":"2023-12-19T08:06:23.284503+0000","last_unstale":"2023-12-19T08:06:23.285330+0000","last_undegraded":"2023-12-19T08:06:23.285330+0000","last_fullsized":"2023-12-19T08:06:23.285330+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:05:55.186443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307827+0000","last_change":"2023-12-19T08:06:23.307827+0000","last_active":"2023-12-19T08:06:23.307827+0000","last_peered":"2023-12-19T08:06:23.307827+0000","last_clean":"2023-12-19T08:06:23.307827+0000","last_became_active":"2023-12-19T08:06:23.307505+0000","last_became_peered":"2023-12-19T08:06:23.307505+0000","last_unstale":"2023-12-19T08:06:23.307827+0000","last_undegraded":"2023-12-19T08:06:23.307827+0000","last_fullsized":"2023-12-19T08:06:23.307827+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:39:06.564383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.311836+0000","last_change":"2023-12-19T08:06:23.308049+0000","last_active":"2023-12-19T08:06:23.311836+0000","last_peered":"2023-12-19T08:06:23.311836+0000","last_clean":"2023-12-19T08:06:23.311836+0000","last_became_active":"2023-12-19T08:06:23.307730+0000","last_became_peered":"2023-12-19T08:06:23.307730+0000","last_unstale":"2023-12-19T08:06:23.311836+0000","last_undegraded":"2023-12-19T08:06:23.311836+0000","last_fullsized":"2023-12-19T08:06:23.311836+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:14.586924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.299238+0000","last_change":"2023-12-19T08:06:23.299238+0000","last_active":"2023-12-19T08:06:23.299238+0000","last_peered":"2023-12-19T08:06:23.299238+0000","last_clean":"2023-12-19T08:06:23.299238+0000","last_became_active":"2023-12-19T08:06:23.298940+0000","last_became_peered":"2023-12-19T08:06:23.298940+0000","last_unstale":"2023-12-19T08:06:23.299238+0000","last_undegraded":"2023-12-19T08:06:23.299238+0000","last_fullsized":"2023-12-19T08:06:23.299238+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:59:06.960953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308598+0000","last_change":"2023-12-19T08:06:23.308598+0000","last_active":"2023-12-19T08:06:23.308598+0000","last_peered":"2023-12-19T08:06:23.308598+0000","last_clean":"2023-12-19T08:06:23.308598+0000","last_became_active":"2023-12-19T08:06:23.308197+0000","last_became_peered":"2023-12-19T08:06:23.308197+0000","last_unstale":"2023-12-19T08:06:23.308598+0000","last_undegraded":"2023-12-19T08:06:23.308598+0000","last_fullsized":"2023-12-19T08:06:23.308598+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:55:49.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304068+0000","last_change":"2023-12-19T08:06:23.292316+0000","last_active":"2023-12-19T08:06:23.304068+0000","last_peered":"2023-12-19T08:06:23.304068+0000","last_clean":"2023-12-19T08:06:23.304068+0000","last_became_active":"2023-12-19T08:06:23.291982+0000","last_became_peered":"2023-12-19T08:06:23.291982+0000","last_unstale":"2023-12-19T08:06:23.304068+0000","last_undegraded":"2023-12-19T08:06:23.304068+0000","last_fullsized":"2023-12-19T08:06:23.304068+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324830+0000","last_change":"2023-12-19T08:06:22.294804+0000","last_active":"2023-12-19T08:06:23.324830+0000","last_peered":"2023-12-19T08:06:23.324830+0000","last_clean":"2023-12-19T08:06:23.324830+0000","last_became_active":"2023-12-19T08:06:22.294481+0000","last_became_peered":"2023-12-19T08:06:22.294481+0000","last_unstale":"2023-12-19T08:06:23.324830+0000","last_undegraded":"2023-12-19T08:06:23.324830+0000","last_fullsized":"2023-12-19T08:06:23.324830+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:08:11.541041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308929+0000","last_change":"2023-12-19T08:06:23.308929+0000","last_active":"2023-12-19T08:06:23.308929+0000","last_peered":"2023-12-19T08:06:23.308929+0000","last_clean":"2023-12-19T08:06:23.308929+0000","last_became_active":"2023-12-19T08:06:23.308001+0000","last_became_peered":"2023-12-19T08:06:23.308001+0000","last_unstale":"2023-12-19T08:06:23.308929+0000","last_undegraded":"2023-12-19T08:06:23.308929+0000","last_fullsized":"2023-12-19T08:06:23.308929+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:31:44.698326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.321160+0000","last_change":"2023-12-19T08:06:22.293243+0000","last_active":"2023-12-19T08:06:23.321160+0000","last_peered":"2023-12-19T08:06:23.321160+0000","last_clean":"2023-12-19T08:06:23.321160+0000","last_became_active":"2023-12-19T08:06:22.292645+0000","last_became_peered":"2023-12-19T08:06:22.292645+0000","last_unstale":"2023-12-19T08:06:23.321160+0000","last_undegraded":"2023-12-19T08:06:23.321160+0000","last_fullsized":"2023-12-19T08:06:23.321160+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:03:58.915674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.297289+0000","last_change":"2023-12-19T08:06:23.297289+0000","last_active":"2023-12-19T08:06:23.297289+0000","last_peered":"2023-12-19T08:06:23.297289+0000","last_clean":"2023-12-19T08:06:23.297289+0000","last_became_active":"2023-12-19T08:06:23.296999+0000","last_became_peered":"2023-12-19T08:06:23.296999+0000","last_unstale":"2023-12-19T08:06:23.297289+0000","last_undegraded":"2023-12-19T08:06:23.297289+0000","last_fullsized":"2023-12-19T08:06:23.297289+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:12:49.209423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.323836+0000","last_change":"2023-12-19T08:06:22.299326+0000","last_active":"2023-12-19T08:06:23.323836+0000","last_peered":"2023-12-19T08:06:23.323836+0000","last_clean":"2023-12-19T08:06:23.323836+0000","last_became_active":"2023-12-19T08:06:22.298687+0000","last_became_peered":"2023-12-19T08:06:22.298687+0000","last_unstale":"2023-12-19T08:06:23.323836+0000","last_undegraded":"2023-12-19T08:06:23.323836+0000","last_fullsized":"2023-12-19T08:06:23.323836+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:57.999146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302251+0000","last_change":"2023-12-19T08:06:23.302251+0000","last_active":"2023-12-19T08:06:23.302251+0000","last_peered":"2023-12-19T08:06:23.302251+0000","last_clean":"2023-12-19T08:06:23.302251+0000","last_became_active":"2023-12-19T08:06:23.301492+0000","last_became_peered":"2023-12-19T08:06:23.301492+0000","last_unstale":"2023-12-19T08:06:23.302251+0000","last_undegraded":"2023-12-19T08:06:23.302251+0000","last_fullsized":"2023-12-19T08:06:23.302251+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:10.059325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.323366+0000","last_change":"2023-12-19T08:06:22.286893+0000","last_active":"2023-12-19T08:06:23.323366+0000","last_peered":"2023-12-19T08:06:23.323366+0000","last_clean":"2023-12-19T08:06:23.323366+0000","last_became_active":"2023-12-19T08:06:22.286602+0000","last_became_peered":"2023-12-19T08:06:22.286602+0000","last_unstale":"2023-12-19T08:06:23.323366+0000","last_undegraded":"2023-12-19T08:06:23.323366+0000","last_fullsized":"2023-12-19T08:06:23.323366+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:24:59.504536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.308940+0000","last_change":"2023-12-19T08:06:23.304865+0000","last_active":"2023-12-19T08:06:23.308940+0000","last_peered":"2023-12-19T08:06:23.308940+0000","last_clean":"2023-12-19T08:06:23.308940+0000","last_became_active":"2023-12-19T08:06:23.304586+0000","last_became_peered":"2023-12-19T08:06:23.304586+0000","last_unstale":"2023-12-19T08:06:23.308940+0000","last_undegraded":"2023-12-19T08:06:23.308940+0000","last_fullsized":"2023-12-19T08:06:23.308940+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318694+0000","last_change":"2023-12-19T08:06:22.294550+0000","last_active":"2023-12-19T08:06:23.318694+0000","last_peered":"2023-12-19T08:06:23.318694+0000","last_clean":"2023-12-19T08:06:23.318694+0000","last_became_active":"2023-12-19T08:06:22.294227+0000","last_became_peered":"2023-12-19T08:06:22.294227+0000","last_unstale":"2023-12-19T08:06:23.318694+0000","last_undegraded":"2023-12-19T08:06:23.318694+0000","last_fullsized":"2023-12-19T08:06:23.318694+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:41.707156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307956+0000","last_change":"2023-12-19T08:06:23.303703+0000","last_active":"2023-12-19T08:06:23.307956+0000","last_peered":"2023-12-19T08:06:23.307956+0000","last_clean":"2023-12-19T08:06:23.307956+0000","last_became_active":"2023-12-19T08:06:23.303419+0000","last_became_peered":"2023-12-19T08:06:23.303419+0000","last_unstale":"2023-12-19T08:06:23.307956+0000","last_undegraded":"2023-12-19T08:06:23.307956+0000","last_fullsized":"2023-12-19T08:06:23.307956+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.274045+0000","last_change":"2023-12-19T08:06:22.292370+0000","last_active":"2023-12-19T08:06:23.274045+0000","last_peered":"2023-12-19T08:06:23.274045+0000","last_clean":"2023-12-19T08:06:23.274045+0000","last_became_active":"2023-12-19T08:06:22.292074+0000","last_became_peered":"2023-12-19T08:06:22.292074+0000","last_unstale":"2023-12-19T08:06:23.274045+0000","last_undegraded":"2023-12-19T08:06:23.274045+0000","last_fullsized":"2023-12-19T08:06:23.274045+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:44:53.952446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316122+0000","last_change":"2023-12-19T08:06:23.313825+0000","last_active":"2023-12-19T08:06:23.316122+0000","last_peered":"2023-12-19T08:06:23.316122+0000","last_clean":"2023-12-19T08:06:23.316122+0000","last_became_active":"2023-12-19T08:06:23.313551+0000","last_became_peered":"2023-12-19T08:06:23.313551+0000","last_unstale":"2023-12-19T08:06:23.316122+0000","last_undegraded":"2023-12-19T08:06:23.316122+0000","last_fullsized":"2023-12-19T08:06:23.316122+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:59:03.204930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.297374+0000","last_change":"2023-12-19T08:06:22.293568+0000","last_active":"2023-12-19T08:06:22.297374+0000","last_peered":"2023-12-19T08:06:22.297374+0000","last_clean":"2023-12-19T08:06:22.297374+0000","last_became_active":"2023-12-19T08:06:22.291910+0000","last_became_peered":"2023-12-19T08:06:22.291910+0000","last_unstale":"2023-12-19T08:06:22.297374+0000","last_undegraded":"2023-12-19T08:06:22.297374+0000","last_fullsized":"2023-12-19T08:06:22.297374+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:02.783487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306662+0000","last_change":"2023-12-19T08:06:23.301259+0000","last_active":"2023-12-19T08:06:23.306662+0000","last_peered":"2023-12-19T08:06:23.306662+0000","last_clean":"2023-12-19T08:06:23.306662+0000","last_became_active":"2023-12-19T08:06:23.300900+0000","last_became_peered":"2023-12-19T08:06:23.300900+0000","last_unstale":"2023-12-19T08:06:23.306662+0000","last_undegraded":"2023-12-19T08:06:23.306662+0000","last_fullsized":"2023-12-19T08:06:23.306662+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300230+0000","last_change":"2023-12-19T08:06:22.295514+0000","last_active":"2023-12-19T08:06:22.300230+0000","last_peered":"2023-12-19T08:06:22.300230+0000","last_clean":"2023-12-19T08:06:22.300230+0000","last_became_active":"2023-12-19T08:06:22.292978+0000","last_became_peered":"2023-12-19T08:06:22.292978+0000","last_unstale":"2023-12-19T08:06:22.300230+0000","last_undegraded":"2023-12-19T08:06:22.300230+0000","last_fullsized":"2023-12-19T08:06:22.300230+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:28.563889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303890+0000","last_change":"2023-12-19T08:06:23.303890+0000","last_active":"2023-12-19T08:06:23.303890+0000","last_peered":"2023-12-19T08:06:23.303890+0000","last_clean":"2023-12-19T08:06:23.303890+0000","last_became_active":"2023-12-19T08:06:23.303579+0000","last_became_peered":"2023-12-19T08:06:23.303579+0000","last_unstale":"2023-12-19T08:06:23.303890+0000","last_undegraded":"2023-12-19T08:06:23.303890+0000","last_fullsized":"2023-12-19T08:06:23.303890+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:34:21.718338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.294404+0000","last_change":"2023-12-19T08:06:22.292057+0000","last_active":"2023-12-19T08:06:22.294404+0000","last_peered":"2023-12-19T08:06:22.294404+0000","last_clean":"2023-12-19T08:06:22.294404+0000","last_became_active":"2023-12-19T08:06:22.291747+0000","last_became_peered":"2023-12-19T08:06:22.291747+0000","last_unstale":"2023-12-19T08:06:22.294404+0000","last_undegraded":"2023-12-19T08:06:22.294404+0000","last_fullsized":"2023-12-19T08:06:22.294404+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.190310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.315378+0000","last_change":"2023-12-19T08:06:23.315378+0000","last_active":"2023-12-19T08:06:23.315378+0000","last_peered":"2023-12-19T08:06:23.315378+0000","last_clean":"2023-12-19T08:06:23.315378+0000","last_became_active":"2023-12-19T08:06:23.315062+0000","last_became_peered":"2023-12-19T08:06:23.315062+0000","last_unstale":"2023-12-19T08:06:23.315378+0000","last_undegraded":"2023-12-19T08:06:23.315378+0000","last_fullsized":"2023-12-19T08:06:23.315378+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300095+0000","last_change":"2023-12-19T08:06:22.298313+0000","last_active":"2023-12-19T08:06:22.300095+0000","last_peered":"2023-12-19T08:06:22.300095+0000","last_clean":"2023-12-19T08:06:22.300095+0000","last_became_active":"2023-12-19T08:06:22.297980+0000","last_became_peered":"2023-12-19T08:06:22.297980+0000","last_unstale":"2023-12-19T08:06:22.300095+0000","last_undegraded":"2023-12-19T08:06:22.300095+0000","last_fullsized":"2023-12-19T08:06:22.300095+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.190310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310599+0000","last_change":"2023-12-19T08:06:23.302535+0000","last_active":"2023-12-19T08:06:23.310599+0000","last_peered":"2023-12-19T08:06:23.310599+0000","last_clean":"2023-12-19T08:06:23.310599+0000","last_became_active":"2023-12-19T08:06:23.302261+0000","last_became_peered":"2023-12-19T08:06:23.302261+0000","last_unstale":"2023-12-19T08:06:23.310599+0000","last_undegraded":"2023-12-19T08:06:23.310599+0000","last_fullsized":"2023-12-19T08:06:23.310599+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:22.145749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.324336+0000","last_change":"2023-12-19T08:06:22.290518+0000","last_active":"2023-12-19T08:06:23.324336+0000","last_peered":"2023-12-19T08:06:23.324336+0000","last_clean":"2023-12-19T08:06:23.324336+0000","last_became_active":"2023-12-19T08:06:22.290234+0000","last_became_peered":"2023-12-19T08:06:22.290234+0000","last_unstale":"2023-12-19T08:06:23.324336+0000","last_undegraded":"2023-12-19T08:06:23.324336+0000","last_fullsized":"2023-12-19T08:06:23.324336+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:09.235742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.297463+0000","last_change":"2023-12-19T08:06:23.297463+0000","last_active":"2023-12-19T08:06:23.297463+0000","last_peered":"2023-12-19T08:06:23.297463+0000","last_clean":"2023-12-19T08:06:23.297463+0000","last_became_active":"2023-12-19T08:06:23.297170+0000","last_became_peered":"2023-12-19T08:06:23.297170+0000","last_unstale":"2023-12-19T08:06:23.297463+0000","last_undegraded":"2023-12-19T08:06:23.297463+0000","last_fullsized":"2023-12-19T08:06:23.297463+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.325943+0000","last_change":"2023-12-19T08:06:22.289201+0000","last_active":"2023-12-19T08:06:23.325943+0000","last_peered":"2023-12-19T08:06:23.325943+0000","last_clean":"2023-12-19T08:06:23.325943+0000","last_became_active":"2023-12-19T08:06:22.288924+0000","last_became_peered":"2023-12-19T08:06:22.288924+0000","last_unstale":"2023-12-19T08:06:23.325943+0000","last_undegraded":"2023-12-19T08:06:23.325943+0000","last_fullsized":"2023-12-19T08:06:23.325943+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:45:28.563889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318211+0000","last_change":"2023-12-19T08:06:23.318211+0000","last_active":"2023-12-19T08:06:23.318211+0000","last_peered":"2023-12-19T08:06:23.318211+0000","last_clean":"2023-12-19T08:06:23.318211+0000","last_became_active":"2023-12-19T08:06:23.317910+0000","last_became_peered":"2023-12-19T08:06:23.317910+0000","last_unstale":"2023-12-19T08:06:23.318211+0000","last_undegraded":"2023-12-19T08:06:23.318211+0000","last_fullsized":"2023-12-19T08:06:23.318211+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:13:58.237515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.322780+0000","last_change":"2023-12-19T08:06:22.283616+0000","last_active":"2023-12-19T08:06:23.322780+0000","last_peered":"2023-12-19T08:06:23.322780+0000","last_clean":"2023-12-19T08:06:23.322780+0000","last_became_active":"2023-12-19T08:06:22.283328+0000","last_became_peered":"2023-12-19T08:06:22.283328+0000","last_unstale":"2023-12-19T08:06:23.322780+0000","last_undegraded":"2023-12-19T08:06:23.322780+0000","last_fullsized":"2023-12-19T08:06:23.322780+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:23:26.517239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307449+0000","last_change":"2023-12-19T08:06:23.298657+0000","last_active":"2023-12-19T08:06:23.307449+0000","last_peered":"2023-12-19T08:06:23.307449+0000","last_clean":"2023-12-19T08:06:23.307449+0000","last_became_active":"2023-12-19T08:06:23.298358+0000","last_became_peered":"2023-12-19T08:06:23.298358+0000","last_unstale":"2023-12-19T08:06:23.307449+0000","last_undegraded":"2023-12-19T08:06:23.307449+0000","last_fullsized":"2023-12-19T08:06:23.307449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:11.892726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.296423+0000","last_change":"2023-12-19T08:06:22.286398+0000","last_active":"2023-12-19T08:06:23.296423+0000","last_peered":"2023-12-19T08:06:23.296423+0000","last_clean":"2023-12-19T08:06:23.296423+0000","last_became_active":"2023-12-19T08:06:22.285903+0000","last_became_peered":"2023-12-19T08:06:22.285903+0000","last_unstale":"2023-12-19T08:06:23.296423+0000","last_undegraded":"2023-12-19T08:06:23.296423+0000","last_fullsized":"2023-12-19T08:06:23.296423+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:31.632186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.307448+0000","last_change":"2023-12-19T08:06:23.307448+0000","last_active":"2023-12-19T08:06:23.307448+0000","last_peered":"2023-12-19T08:06:23.307448+0000","last_clean":"2023-12-19T08:06:23.307448+0000","last_became_active":"2023-12-19T08:06:23.300392+0000","last_became_peered":"2023-12-19T08:06:23.300392+0000","last_unstale":"2023-12-19T08:06:23.307448+0000","last_undegraded":"2023-12-19T08:06:23.307448+0000","last_fullsized":"2023-12-19T08:06:23.307448+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:23.712870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293810+0000","last_change":"2023-12-19T08:06:22.290323+0000","last_active":"2023-12-19T08:06:23.293810+0000","last_peered":"2023-12-19T08:06:23.293810+0000","last_clean":"2023-12-19T08:06:23.293810+0000","last_became_active":"2023-12-19T08:06:22.289996+0000","last_became_peered":"2023-12-19T08:06:22.289996+0000","last_unstale":"2023-12-19T08:06:23.293810+0000","last_undegraded":"2023-12-19T08:06:23.293810+0000","last_fullsized":"2023-12-19T08:06:23.293810+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:01.739322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.296999+0000","last_change":"2023-12-19T08:06:23.296999+0000","last_active":"2023-12-19T08:06:23.296999+0000","last_peered":"2023-12-19T08:06:23.296999+0000","last_clean":"2023-12-19T08:06:23.296999+0000","last_became_active":"2023-12-19T08:06:23.296433+0000","last_became_peered":"2023-12-19T08:06:23.296433+0000","last_unstale":"2023-12-19T08:06:23.296999+0000","last_undegraded":"2023-12-19T08:06:23.296999+0000","last_fullsized":"2023-12-19T08:06:23.296999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:55:19.643345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.321566+0000","last_change":"2023-12-19T08:06:22.289305+0000","last_active":"2023-12-19T08:06:23.321566+0000","last_peered":"2023-12-19T08:06:23.321566+0000","last_clean":"2023-12-19T08:06:23.321566+0000","last_became_active":"2023-12-19T08:06:22.287818+0000","last_became_peered":"2023-12-19T08:06:22.287818+0000","last_unstale":"2023-12-19T08:06:23.321566+0000","last_undegraded":"2023-12-19T08:06:23.321566+0000","last_fullsized":"2023-12-19T08:06:23.321566+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:10.698455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316458+0000","last_change":"2023-12-19T08:06:23.313509+0000","last_active":"2023-12-19T08:06:23.316458+0000","last_peered":"2023-12-19T08:06:23.316458+0000","last_clean":"2023-12-19T08:06:23.316458+0000","last_became_active":"2023-12-19T08:06:23.313106+0000","last_became_peered":"2023-12-19T08:06:23.313106+0000","last_unstale":"2023-12-19T08:06:23.316458+0000","last_undegraded":"2023-12-19T08:06:23.316458+0000","last_fullsized":"2023-12-19T08:06:23.316458+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.317384+0000","last_change":"2023-12-19T08:06:22.298505+0000","last_active":"2023-12-19T08:06:23.317384+0000","last_peered":"2023-12-19T08:06:23.317384+0000","last_clean":"2023-12-19T08:06:23.317384+0000","last_became_active":"2023-12-19T08:06:22.297234+0000","last_became_peered":"2023-12-19T08:06:22.297234+0000","last_unstale":"2023-12-19T08:06:23.317384+0000","last_undegraded":"2023-12-19T08:06:23.317384+0000","last_fullsized":"2023-12-19T08:06:23.317384+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:31:48.619872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.304727+0000","last_change":"2023-12-19T08:06:23.304727+0000","last_active":"2023-12-19T08:06:23.304727+0000","last_peered":"2023-12-19T08:06:23.304727+0000","last_clean":"2023-12-19T08:06:23.304727+0000","last_became_active":"2023-12-19T08:06:23.301454+0000","last_became_peered":"2023-12-19T08:06:23.301454+0000","last_unstale":"2023-12-19T08:06:23.304727+0000","last_undegraded":"2023-12-19T08:06:23.304727+0000","last_fullsized":"2023-12-19T08:06:23.304727+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:05.160858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.292781+0000","last_change":"2023-12-19T08:06:22.298207+0000","last_active":"2023-12-19T08:06:23.292781+0000","last_peered":"2023-12-19T08:06:23.292781+0000","last_clean":"2023-12-19T08:06:23.292781+0000","last_became_active":"2023-12-19T08:06:22.297768+0000","last_became_peered":"2023-12-19T08:06:22.297768+0000","last_unstale":"2023-12-19T08:06:23.292781+0000","last_undegraded":"2023-12-19T08:06:23.292781+0000","last_fullsized":"2023-12-19T08:06:23.292781+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:44:19.189284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.305410+0000","last_change":"2023-12-19T08:06:23.305410+0000","last_active":"2023-12-19T08:06:23.305410+0000","last_peered":"2023-12-19T08:06:23.305410+0000","last_clean":"2023-12-19T08:06:23.305410+0000","last_became_active":"2023-12-19T08:06:23.304785+0000","last_became_peered":"2023-12-19T08:06:23.304785+0000","last_unstale":"2023-12-19T08:06:23.305410+0000","last_undegraded":"2023-12-19T08:06:23.305410+0000","last_fullsized":"2023-12-19T08:06:23.305410+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:59:09.967118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.302985+0000","last_change":"2023-12-19T08:06:22.302347+0000","last_active":"2023-12-19T08:06:22.302985+0000","last_peered":"2023-12-19T08:06:22.302985+0000","last_clean":"2023-12-19T08:06:22.302985+0000","last_became_active":"2023-12-19T08:06:22.302018+0000","last_became_peered":"2023-12-19T08:06:22.302018+0000","last_unstale":"2023-12-19T08:06:22.302985+0000","last_undegraded":"2023-12-19T08:06:22.302985+0000","last_fullsized":"2023-12-19T08:06:22.302985+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:34:20.704115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306362+0000","last_change":"2023-12-19T08:06:23.306362+0000","last_active":"2023-12-19T08:06:23.306362+0000","last_peered":"2023-12-19T08:06:23.306362+0000","last_clean":"2023-12-19T08:06:23.306362+0000","last_became_active":"2023-12-19T08:06:23.306067+0000","last_became_peered":"2023-12-19T08:06:23.306067+0000","last_unstale":"2023-12-19T08:06:23.306362+0000","last_undegraded":"2023-12-19T08:06:23.306362+0000","last_fullsized":"2023-12-19T08:06:23.306362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:06.579739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.318755+0000","last_change":"2023-12-19T08:06:22.294326+0000","last_active":"2023-12-19T08:06:23.318755+0000","last_peered":"2023-12-19T08:06:23.318755+0000","last_clean":"2023-12-19T08:06:23.318755+0000","last_became_active":"2023-12-19T08:06:22.294011+0000","last_became_peered":"2023-12-19T08:06:22.294011+0000","last_unstale":"2023-12-19T08:06:23.318755+0000","last_undegraded":"2023-12-19T08:06:23.318755+0000","last_fullsized":"2023-12-19T08:06:23.318755+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:04:02.999478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.313938+0000","last_change":"2023-12-19T08:06:23.306531+0000","last_active":"2023-12-19T08:06:23.313938+0000","last_peered":"2023-12-19T08:06:23.313938+0000","last_clean":"2023-12-19T08:06:23.313938+0000","last_became_active":"2023-12-19T08:06:23.306229+0000","last_became_peered":"2023-12-19T08:06:23.306229+0000","last_unstale":"2023-12-19T08:06:23.313938+0000","last_undegraded":"2023-12-19T08:06:23.313938+0000","last_fullsized":"2023-12-19T08:06:23.313938+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:14.812267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.295653+0000","last_change":"2023-12-19T08:06:22.292433+0000","last_active":"2023-12-19T08:06:22.295653+0000","last_peered":"2023-12-19T08:06:22.295653+0000","last_clean":"2023-12-19T08:06:22.295653+0000","last_became_active":"2023-12-19T08:06:22.292115+0000","last_became_peered":"2023-12-19T08:06:22.292115+0000","last_unstale":"2023-12-19T08:06:22.295653+0000","last_undegraded":"2023-12-19T08:06:22.295653+0000","last_fullsized":"2023-12-19T08:06:22.295653+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T08:06:44.697786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.310752+0000","last_change":"2023-12-19T08:06:23.302539+0000","last_active":"2023-12-19T08:06:23.310752+0000","last_peered":"2023-12-19T08:06:23.310752+0000","last_clean":"2023-12-19T08:06:23.310752+0000","last_became_active":"2023-12-19T08:06:23.301672+0000","last_became_peered":"2023-12-19T08:06:23.301672+0000","last_unstale":"2023-12-19T08:06:23.310752+0000","last_undegraded":"2023-12-19T08:06:23.310752+0000","last_fullsized":"2023-12-19T08:06:23.310752+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:44.721338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.293901+0000","last_change":"2023-12-19T08:06:22.298102+0000","last_active":"2023-12-19T08:06:23.293901+0000","last_peered":"2023-12-19T08:06:23.293901+0000","last_clean":"2023-12-19T08:06:23.293901+0000","last_became_active":"2023-12-19T08:06:22.297812+0000","last_became_peered":"2023-12-19T08:06:22.297812+0000","last_unstale":"2023-12-19T08:06:23.293901+0000","last_undegraded":"2023-12-19T08:06:23.293901+0000","last_fullsized":"2023-12-19T08:06:23.293901+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:55:01.207523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.301188+0000","last_change":"2023-12-19T08:06:23.301188+0000","last_active":"2023-12-19T08:06:23.301188+0000","last_peered":"2023-12-19T08:06:23.301188+0000","last_clean":"2023-12-19T08:06:23.301188+0000","last_became_active":"2023-12-19T08:06:23.300892+0000","last_became_peered":"2023-12-19T08:06:23.300892+0000","last_unstale":"2023-12-19T08:06:23.301188+0000","last_undegraded":"2023-12-19T08:06:23.301188+0000","last_fullsized":"2023-12-19T08:06:23.301188+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:36:39.495228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300416+0000","last_change":"2023-12-19T08:06:22.299696+0000","last_active":"2023-12-19T08:06:22.300416+0000","last_peered":"2023-12-19T08:06:22.300416+0000","last_clean":"2023-12-19T08:06:22.300416+0000","last_became_active":"2023-12-19T08:06:22.299337+0000","last_became_peered":"2023-12-19T08:06:22.299337+0000","last_unstale":"2023-12-19T08:06:22.300416+0000","last_undegraded":"2023-12-19T08:06:22.300416+0000","last_fullsized":"2023-12-19T08:06:22.300416+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:13:57.223292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.300820+0000","last_change":"2023-12-19T08:06:23.300820+0000","last_active":"2023-12-19T08:06:23.300820+0000","last_peered":"2023-12-19T08:06:23.300820+0000","last_clean":"2023-12-19T08:06:23.300820+0000","last_became_active":"2023-12-19T08:06:23.297955+0000","last_became_peered":"2023-12-19T08:06:23.297955+0000","last_unstale":"2023-12-19T08:06:23.300820+0000","last_undegraded":"2023-12-19T08:06:23.300820+0000","last_fullsized":"2023-12-19T08:06:23.300820+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:53:14.812267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316345+0000","last_change":"2023-12-19T08:06:22.288164+0000","last_active":"2023-12-19T08:06:23.316345+0000","last_peered":"2023-12-19T08:06:23.316345+0000","last_clean":"2023-12-19T08:06:23.316345+0000","last_became_active":"2023-12-19T08:06:22.286973+0000","last_became_peered":"2023-12-19T08:06:22.286973+0000","last_unstale":"2023-12-19T08:06:23.316345+0000","last_undegraded":"2023-12-19T08:06:23.316345+0000","last_fullsized":"2023-12-19T08:06:23.316345+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:31.632186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302314+0000","last_change":"2023-12-19T08:06:23.297341+0000","last_active":"2023-12-19T08:06:23.302314+0000","last_peered":"2023-12-19T08:06:23.302314+0000","last_clean":"2023-12-19T08:06:23.302314+0000","last_became_active":"2023-12-19T08:06:23.297056+0000","last_became_peered":"2023-12-19T08:06:23.297056+0000","last_unstale":"2023-12-19T08:06:23.302314+0000","last_undegraded":"2023-12-19T08:06:23.302314+0000","last_fullsized":"2023-12-19T08:06:23.302314+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:10:26.773952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.302819+0000","last_change":"2023-12-19T08:06:22.301897+0000","last_active":"2023-12-19T08:06:22.302819+0000","last_peered":"2023-12-19T08:06:22.302819+0000","last_clean":"2023-12-19T08:06:22.302819+0000","last_became_active":"2023-12-19T08:06:22.301612+0000","last_became_peered":"2023-12-19T08:06:22.301612+0000","last_unstale":"2023-12-19T08:06:22.302819+0000","last_undegraded":"2023-12-19T08:06:22.302819+0000","last_fullsized":"2023-12-19T08:06:22.302819+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:09.235742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302848+0000","last_change":"2023-12-19T08:06:23.302848+0000","last_active":"2023-12-19T08:06:23.302848+0000","last_peered":"2023-12-19T08:06:23.302848+0000","last_clean":"2023-12-19T08:06:23.302848+0000","last_became_active":"2023-12-19T08:06:23.302558+0000","last_became_peered":"2023-12-19T08:06:23.302558+0000","last_unstale":"2023-12-19T08:06:23.302848+0000","last_undegraded":"2023-12-19T08:06:23.302848+0000","last_fullsized":"2023-12-19T08:06:23.302848+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:27:52.364078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.295024+0000","last_change":"2023-12-19T08:06:22.293668+0000","last_active":"2023-12-19T08:06:22.295024+0000","last_peered":"2023-12-19T08:06:22.295024+0000","last_clean":"2023-12-19T08:06:22.295024+0000","last_became_active":"2023-12-19T08:06:22.290530+0000","last_became_peered":"2023-12-19T08:06:22.290530+0000","last_unstale":"2023-12-19T08:06:22.295024+0000","last_undegraded":"2023-12-19T08:06:22.295024+0000","last_fullsized":"2023-12-19T08:06:22.295024+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:02.783487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.294147+0000","last_change":"2023-12-19T08:06:23.294147+0000","last_active":"2023-12-19T08:06:23.294147+0000","last_peered":"2023-12-19T08:06:23.294147+0000","last_clean":"2023-12-19T08:06:23.294147+0000","last_became_active":"2023-12-19T08:06:23.293852+0000","last_became_peered":"2023-12-19T08:06:23.293852+0000","last_unstale":"2023-12-19T08:06:23.294147+0000","last_undegraded":"2023-12-19T08:06:23.294147+0000","last_fullsized":"2023-12-19T08:06:23.294147+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:41:18.382344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300190+0000","last_change":"2023-12-19T08:06:22.295766+0000","last_active":"2023-12-19T08:06:22.300190+0000","last_peered":"2023-12-19T08:06:22.300190+0000","last_clean":"2023-12-19T08:06:22.300190+0000","last_became_active":"2023-12-19T08:06:22.293113+0000","last_became_peered":"2023-12-19T08:06:22.293113+0000","last_unstale":"2023-12-19T08:06:22.300190+0000","last_undegraded":"2023-12-19T08:06:22.300190+0000","last_fullsized":"2023-12-19T08:06:22.300190+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:22:41.268100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.289418+0000","last_change":"2023-12-19T08:06:23.289418+0000","last_active":"2023-12-19T08:06:23.289418+0000","last_peered":"2023-12-19T08:06:23.289418+0000","last_clean":"2023-12-19T08:06:23.289418+0000","last_became_active":"2023-12-19T08:06:23.289141+0000","last_became_peered":"2023-12-19T08:06:23.289141+0000","last_unstale":"2023-12-19T08:06:23.289418+0000","last_undegraded":"2023-12-19T08:06:23.289418+0000","last_fullsized":"2023-12-19T08:06:23.289418+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:05:55.773286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.281752+0000","last_change":"2023-12-19T08:06:22.295406+0000","last_active":"2023-12-19T08:06:23.281752+0000","last_peered":"2023-12-19T08:06:23.281752+0000","last_clean":"2023-12-19T08:06:23.281752+0000","last_became_active":"2023-12-19T08:06:22.294905+0000","last_became_peered":"2023-12-19T08:06:22.294905+0000","last_unstale":"2023-12-19T08:06:23.281752+0000","last_undegraded":"2023-12-19T08:06:23.281752+0000","last_fullsized":"2023-12-19T08:06:23.281752+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:07:10.698455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.316572+0000","last_change":"2023-12-19T08:06:23.313228+0000","last_active":"2023-12-19T08:06:23.316572+0000","last_peered":"2023-12-19T08:06:23.316572+0000","last_clean":"2023-12-19T08:06:23.316572+0000","last_became_active":"2023-12-19T08:06:23.312882+0000","last_became_peered":"2023-12-19T08:06:23.312882+0000","last_unstale":"2023-12-19T08:06:23.316572+0000","last_undegraded":"2023-12-19T08:06:23.316572+0000","last_fullsized":"2023-12-19T08:06:23.316572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:57.748699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.281620+0000","last_change":"2023-12-19T08:06:22.292736+0000","last_active":"2023-12-19T08:06:23.281620+0000","last_peered":"2023-12-19T08:06:23.281620+0000","last_clean":"2023-12-19T08:06:23.281620+0000","last_became_active":"2023-12-19T08:06:22.290772+0000","last_became_peered":"2023-12-19T08:06:22.290772+0000","last_unstale":"2023-12-19T08:06:23.281620+0000","last_undegraded":"2023-12-19T08:06:23.281620+0000","last_fullsized":"2023-12-19T08:06:23.281620+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:03:58.915674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.306287+0000","last_change":"2023-12-19T08:06:23.306287+0000","last_active":"2023-12-19T08:06:23.306287+0000","last_peered":"2023-12-19T08:06:23.306287+0000","last_clean":"2023-12-19T08:06:23.306287+0000","last_became_active":"2023-12-19T08:06:23.305987+0000","last_became_peered":"2023-12-19T08:06:23.305987+0000","last_unstale":"2023-12-19T08:06:23.306287+0000","last_undegraded":"2023-12-19T08:06:23.306287+0000","last_fullsized":"2023-12-19T08:06:23.306287+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:12:49.209423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.300152+0000","last_change":"2023-12-19T08:06:22.295232+0000","last_active":"2023-12-19T08:06:22.300152+0000","last_peered":"2023-12-19T08:06:22.300152+0000","last_clean":"2023-12-19T08:06:22.300152+0000","last_became_active":"2023-12-19T08:06:22.294221+0000","last_became_peered":"2023-12-19T08:06:22.294221+0000","last_unstale":"2023-12-19T08:06:22.300152+0000","last_undegraded":"2023-12-19T08:06:22.300152+0000","last_fullsized":"2023-12-19T08:06:22.300152+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:39.101328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.303964+0000","last_change":"2023-12-19T08:06:23.296502+0000","last_active":"2023-12-19T08:06:23.303964+0000","last_peered":"2023-12-19T08:06:23.303964+0000","last_clean":"2023-12-19T08:06:23.303964+0000","last_became_active":"2023-12-19T08:06:23.296123+0000","last_became_peered":"2023-12-19T08:06:23.296123+0000","last_unstale":"2023-12-19T08:06:23.303964+0000","last_undegraded":"2023-12-19T08:06:23.303964+0000","last_fullsized":"2023-12-19T08:06:23.303964+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:59.641897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.298149+0000","last_change":"2023-12-19T08:06:22.296251+0000","last_active":"2023-12-19T08:06:22.298149+0000","last_peered":"2023-12-19T08:06:22.298149+0000","last_clean":"2023-12-19T08:06:22.298149+0000","last_became_active":"2023-12-19T08:06:22.295941+0000","last_became_peered":"2023-12-19T08:06:22.295941+0000","last_unstale":"2023-12-19T08:06:22.298149+0000","last_undegraded":"2023-12-19T08:06:22.298149+0000","last_fullsized":"2023-12-19T08:06:22.298149+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:41.707156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":8,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.302684+0000","last_change":"2023-12-19T08:06:23.302684+0000","last_active":"2023-12-19T08:06:23.302684+0000","last_peered":"2023-12-19T08:06:23.302684+0000","last_clean":"2023-12-19T08:06:23.302684+0000","last_became_active":"2023-12-19T08:06:23.302365+0000","last_became_peered":"2023-12-19T08:06:23.302365+0000","last_unstale":"2023-12-19T08:06:23.302684+0000","last_undegraded":"2023-12-19T08:06:23.302684+0000","last_fullsized":"2023-12-19T08:06:23.302684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:22.250137+0000","last_clean_scrub_stamp":"2023-12-19T08:06:22.250137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:39:14.017630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-12-19T08:06:23.325313+0000","last_change":"2023-12-19T08:06:22.284277+0000","last_active":"2023-12-19T08:06:23.325313+0000","last_peered":"2023-12-19T08:06:23.325313+0000","last_clean":"2023-12-19T08:06:23.325313+0000","last_became_active":"2023-12-19T08:06:22.283995+0000","last_became_peered":"2023-12-19T08:06:22.283995+0000","last_unstale":"2023-12-19T08:06:23.325313+0000","last_undegraded":"2023-12-19T08:06:23.325313+0000","last_fullsized":"2023-12-19T08:06:23.325313+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T08:06:21.235914+0000","last_clean_scrub_stamp":"2023-12-19T08:06:21.235914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:42:55.094079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'249","reported_seq":731,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-12-19T08:06:22.285572+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T08:06:22.285572+0000","last_peered":"2023-12-19T08:06:22.285572+0000","last_clean":"2023-12-19T08:06:22.285572+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T08:06:22.285572+0000","last_undegraded":"2023-12-19T08:06:22.285572+0000","last_fullsized":"2023-12-19T08:06:22.285572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575075,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311908,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059932,"statfs":{"total":96636764160,"available":96317370368,"internally_reserved":0,"allocated":2387968,"data_stored":1526841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2230000000000001,"15min":1.25},"min":{"1min":0.78900000000000003,"5min":0.69699999999999995,"15min":0.67600000000000005},"max":{"1min":1.619,"5min":1.6850000000000001,"15min":2.0459999999999998},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.2769999999999999,"15min":1.27},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7170000000000001,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.579}]},{"osd":1,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1619999999999999,"15min":1.161},"min":{"1min":0.70199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.429,"5min":4.9160000000000004,"15min":4.9160000000000004},"last":1.516},{"interface":"front","average":{"1min":1.214,"5min":1.2250000000000001,"15min":1.1839999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.667,"5min":3.8969999999999998,"15min":3.8969999999999998},"last":1.2849999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.169,"15min":1.161},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":1.415,"5min":4.9409999999999998,"15min":4.9409999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3,"15min":1.2649999999999999},"min":{"1min":1.0800000000000001,"5min":0.82999999999999996,"15min":0.70799999999999996},"max":{"1min":1.756,"5min":1.946,"15min":2.1739999999999999},"last":1.216}]},{"osd":3,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.288,"15min":1.284},"min":{"1min":0.878,"5min":0.878,"15min":0.83299999999999996},"max":{"1min":1.4610000000000001,"5min":4.8220000000000001,"15min":4.8220000000000001},"last":1.137},{"interface":"front","average":{"1min":1.264,"5min":1.375,"15min":1.351},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.81799999999999995},"max":{"1min":1.605,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":1.385}]},{"osd":4,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3109999999999999,"15min":1.2430000000000001},"min":{"1min":0.85899999999999999,"5min":0.71899999999999997,"15min":0.67300000000000004},"max":{"1min":1.6619999999999999,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":1.361},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.292,"15min":1.24},"min":{"1min":0.80500000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.7070000000000001,"5min":6.8360000000000003,"15min":6.8360000000000003},"last":1.458}]},{"osd":5,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.302,"15min":1.248},"min":{"1min":0.69699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.663,"5min":4.5679999999999996,"15min":4.5679999999999996},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.419,"15min":1.3720000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.751},"max":{"1min":1.8720000000000001,"5min":2.1219999999999999,"15min":2.2120000000000002},"last":1.496}]},{"osd":6,"last update":"Tue Dec 19 08:05:36 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.4950000000000001,"15min":1.4219999999999999},"min":{"1min":0.88,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7370000000000001,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.696},{"interface":"front","average":{"1min":1.325,"5min":1.421,"15min":1.3819999999999999},"min":{"1min":0.95799999999999996,"5min":0.85099999999999998,"15min":0.58299999999999996},"max":{"1min":1.8080000000000001,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.343}]}]},{"osd":6,"up_from":13,"seq":55834575074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303124,"kb_used_data":1740,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068716,"statfs":{"total":96636764160,"available":96326365184,"internally_reserved":0,"allocated":1781760,"data_stored":936279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1000000000000001,"15min":1.0980000000000001},"min":{"1min":0.73999999999999999,"5min":0.70999999999999996,"15min":0.36799999999999999},"max":{"1min":1.3959999999999999,"5min":1.5569999999999999,"15min":1.613},"last":0.747},{"interface":"front","average":{"1min":1.139,"5min":1.163,"15min":1.1359999999999999},"min":{"1min":0.81599999999999995,"5min":0.624,"15min":0.49399999999999999},"max":{"1min":1.6040000000000001,"5min":1.756,"15min":1.756},"last":1.3140000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1419999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.60799999999999998,"15min":0.53400000000000003},"max":{"1min":1.587,"5min":1.641,"15min":1.675},"last":1.76},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.113,"15min":1.1519999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.51900000000000002},"max":{"1min":1.425,"5min":1.5349999999999999,"15min":1.7470000000000001},"last":1.5629999999999999}]},{"osd":2,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.21,"15min":1.23},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.59099999999999997},"max":{"1min":1.5429999999999999,"5min":1.746,"15min":1.847},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.206,"15min":1.21},"min":{"1min":0.76400000000000001,"5min":0.70799999999999996,"15min":0.58999999999999997},"max":{"1min":1.635,"5min":1.694,"15min":1.7929999999999999},"last":1.6000000000000001}]},{"osd":3,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.228,"15min":1.2549999999999999},"min":{"1min":0.97299999999999998,"5min":0.77200000000000002,"15min":0.65400000000000003},"max":{"1min":1.423,"5min":1.8200000000000001,"15min":1.958},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2909999999999999,"15min":1.3069999999999999},"min":{"1min":0.95299999999999996,"5min":0.78200000000000003,"15min":0.63200000000000001},"max":{"1min":1.677,"5min":1.8540000000000001,"15min":1.9570000000000001},"last":1.5109999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.159,"15min":1.1679999999999999},"min":{"1min":0.84599999999999997,"5min":0.55500000000000005,"15min":0.47199999999999998},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.8480000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.3740000000000001,"15min":1.391},"min":{"1min":1.0029999999999999,"5min":0.72699999999999998,"15min":0.70099999999999996},"max":{"1min":1.754,"5min":1.9139999999999999,"15min":2.1440000000000001},"last":1.714}]},{"osd":5,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.276,"15min":1.28},"min":{"1min":0.90400000000000003,"5min":0.70799999999999996,"15min":0.67600000000000005},"max":{"1min":1.792,"5min":1.9490000000000001,"15min":2.1989999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.2869999999999999,"15min":1.2889999999999999},"min":{"1min":0.876,"5min":0.75700000000000001,"15min":0.56499999999999995},"max":{"1min":1.5920000000000001,"5min":1.7250000000000001,"15min":2.2450000000000001},"last":1.234}]},{"osd":7,"last update":"Tue Dec 19 08:05:47 2023","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3839999999999999,"15min":1.3640000000000001},"min":{"1min":0.99399999999999999,"5min":0.73599999999999999,"15min":0.55100000000000005},"max":{"1min":1.8340000000000001,"5min":1.9239999999999999,"15min":2.04},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.431,"15min":1.4179999999999999},"min":{"1min":0.84699999999999998,"5min":0.76600000000000001,"15min":0.58099999999999996},"max":{"1min":1.976,"5min":1.976,"15min":2.137},"last":1.661}]}]},{"osd":5,"up_from":13,"seq":55834575074,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303128,"kb_used_data":1744,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068712,"statfs":{"total":96636764160,"available":96326361088,"internally_reserved":0,"allocated":1785856,"data_stored":935885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1779999999999999,"15min":1.228},"min":{"1min":0.78900000000000003,"5min":0.67900000000000005,"15min":0.53000000000000003},"max":{"1min":1.5189999999999999,"5min":1.714,"15min":2.6659999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2430000000000001,"15min":1.266},"min":{"1min":0.86499999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6599999999999999,"5min":1.8129999999999999,"15min":1.954},"last":1.3200000000000001}]},{"osd":1,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1140000000000001,"15min":1.1419999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.478,"5min":1.585,"15min":1.958},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.157,"15min":1.171},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.5149999999999999,"5min":1.8620000000000001,"15min":2.7679999999999998},"last":1.173}]},{"osd":2,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.1990000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.5009999999999999,"5min":1.6739999999999999,"15min":2.0259999999999998},"last":1.216},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.175,"15min":1.2050000000000001},"min":{"1min":0.82299999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6040000000000001,"5min":1.6679999999999999,"15min":1.7490000000000001},"last":1.24}]},{"osd":3,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.22,"15min":1.2509999999999999},"min":{"1min":0.84299999999999997,"5min":0.76200000000000001,"15min":0.748},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":2.9489999999999998},"last":1.446},{"interface":"front","average":{"1min":1.226,"5min":1.304,"15min":1.3069999999999999},"min":{"1min":0.85199999999999998,"5min":0.79500000000000004,"15min":0.76600000000000001},"max":{"1min":1.647,"5min":1.744,"15min":2.2130000000000001},"last":1.371}]},{"osd":4,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.181,"15min":1.196},"min":{"1min":0.69399999999999995,"5min":0.69199999999999995,"15min":0.629},"max":{"1min":1.6910000000000001,"5min":1.7749999999999999,"15min":2.1419999999999999},"last":1.073},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.268,"15min":1.272},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.59199999999999997},"max":{"1min":1.6259999999999999,"5min":1.673,"15min":2.843},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.321,"15min":1.3100000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.7769999999999999,"5min":1.869,"15min":2.121},"last":1.139},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3480000000000001,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.77300000000000002},"max":{"1min":1.7869999999999999,"5min":1.96,"15min":4.1959999999999997},"last":1.3899999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.387,"15min":1.4099999999999999},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.74399999999999999},"max":{"1min":1.77,"5min":1.96,"15min":2.9929999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.478,"15min":1.456},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.91500000000000004},"max":{"1min":1.927,"5min":1.929,"15min":2.0779999999999998},"last":1.4179999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575074,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307228,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064612,"statfs":{"total":96636764160,"available":96322162688,"internally_reserved":0,"allocated":1789952,"data_stored":937261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1990000000000001,"15min":1.21},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"max":{"1min":2.0190000000000001,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.3109999999999999,"15min":1.3089999999999999},"min":{"1min":0.83799999999999997,"5min":0.80300000000000005,"15min":0.70299999999999996},"max":{"1min":1.796,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.397}]},{"osd":1,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.165,"15min":1.169},"min":{"1min":0.86399999999999999,"5min":0.69799999999999995,"15min":0.57299999999999995},"max":{"1min":1.7130000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.129,"15min":1.1379999999999999},"min":{"1min":0.67400000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4079999999999999,"5min":1.569,"15min":1.569},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0900000000000001,"15min":1.127},"min":{"1min":0.85699999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.538,"5min":1.538,"15min":1.746},"last":1.369},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2110000000000001,"15min":1.2290000000000001},"min":{"1min":0.875,"5min":0.72399999999999998,"15min":0.69699999999999995},"max":{"1min":1.569,"5min":1.5980000000000001,"15min":1.8959999999999999},"last":1.5489999999999999}]},{"osd":3,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.256,"15min":1.268},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.65900000000000003},"max":{"1min":1.5489999999999999,"5min":1.831,"15min":1.831},"last":1.274},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.292,"15min":1.321},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.4219999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.214,"15min":1.2},"min":{"1min":0.81599999999999995,"5min":0.58299999999999996,"15min":0.53200000000000003},"max":{"1min":1.7050000000000001,"5min":1.905,"15min":1.905},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.208,"15min":1.194},"min":{"1min":0.96399999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.6679999999999999,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.95499999999999996}]},{"osd":6,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.2490000000000001,"15min":1.234},"min":{"1min":0.90300000000000002,"5min":0.73899999999999999,"15min":0.67400000000000004},"max":{"1min":2.3599999999999999,"5min":2.3599999999999999,"15min":2.3599999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.389,"15min":1.377},"min":{"1min":0.92900000000000005,"5min":0.70899999999999996,"15min":0.67900000000000005},"max":{"1min":2.2410000000000001,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.4690000000000001}]},{"osd":7,"last update":"Tue Dec 19 08:05:41 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3540000000000001,"15min":1.339},"min":{"1min":1.0009999999999999,"5min":0.58799999999999997,"15min":0.46300000000000002},"max":{"1min":1.837,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.4139999999999999,"15min":1.3959999999999999},"min":{"1min":0.997,"5min":0.80200000000000005,"15min":0.70999999999999996},"max":{"1min":2.4729999999999999,"5min":2.4729999999999999,"15min":2.4729999999999999},"last":1.504}]}]},{"osd":3,"up_from":13,"seq":55834575074,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307232,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064608,"statfs":{"total":96636764160,"available":96322158592,"internally_reserved":0,"allocated":1794048,"data_stored":937295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.47799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.3320000000000001,"5min":2.5539999999999998,"15min":2.5539999999999998},"last":0.996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.97699999999999998,"15min":0.96599999999999997},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.39200000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.72699999999999998}]},{"osd":1,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.90400000000000003,"15min":0.90600000000000003},"min":{"1min":0.66500000000000004,"5min":0.47099999999999997,"15min":0.45600000000000002},"max":{"1min":1.1559999999999999,"5min":1.609,"15min":1.609},"last":1.216},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.042,"15min":1.038},"min":{"1min":0.71799999999999997,"5min":0.47799999999999998,"15min":0.442},"max":{"1min":1.4970000000000001,"5min":2.423,"15min":2.423},"last":0.95599999999999996}]},{"osd":2,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.042,"15min":1.0680000000000001},"min":{"1min":0.57899999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.167,"15min":1.1579999999999999},"min":{"1min":0.64900000000000002,"5min":0.57299999999999995,"15min":0.53800000000000003},"max":{"1min":1.5800000000000001,"5min":2.48,"15min":2.48},"last":0.83399999999999996}]},{"osd":4,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.228,"15min":1.248},"min":{"1min":0.84999999999999998,"5min":0.81200000000000006,"15min":0.66800000000000004},"max":{"1min":1.887,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.012},{"interface":"front","average":{"1min":1.212,"5min":1.2310000000000001,"15min":1.2390000000000001},"min":{"1min":0.71299999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.2909999999999999}]},{"osd":5,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2709999999999999,"15min":1.266},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.554,"5min":1.996,"15min":1.996},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.3300000000000001,"15min":1.331},"min":{"1min":0.85599999999999998,"5min":0.753,"15min":0.753},"max":{"1min":1.552,"5min":2.161,"15min":2.161},"last":1.123}]},{"osd":6,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.409,"15min":1.385},"min":{"1min":0.82399999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9139999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.161},{"interface":"front","average":{"1min":1.288,"5min":1.323,"15min":1.3540000000000001},"min":{"1min":0.87,"5min":0.87,"15min":0.82499999999999996},"max":{"1min":2.0019999999999998,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.2709999999999999,"5min":2.7530000000000001,"15min":2.7530000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.397,"15min":1.411},"min":{"1min":1.0269999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.772,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.677}]}]},{"osd":2,"up_from":13,"seq":55834575074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307216,"kb_used_data":1736,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064624,"statfs":{"total":96636764160,"available":96322174976,"internally_reserved":0,"allocated":1777664,"data_stored":935725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.89300000000000002,"15min":0.90500000000000003},"min":{"1min":0.51000000000000001,"5min":0.44600000000000001,"15min":0.437},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.776},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0129999999999999,"15min":1.0409999999999999},"min":{"1min":0.63100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5369999999999999,"5min":1.7769999999999999,"15min":1.8460000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.97199999999999998,"15min":0.95899999999999996},"min":{"1min":0.63,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.4450000000000001,"5min":1.669,"15min":1.669},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0089999999999999,"15min":1.0109999999999999},"min":{"1min":0.66500000000000004,"5min":0.53500000000000003,"15min":0.53100000000000003},"max":{"1min":1.2809999999999999,"5min":1.835,"15min":1.835},"last":0.69399999999999995}]},{"osd":3,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.96299999999999997,"15min":0.97399999999999998},"min":{"1min":0.56599999999999995,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":2.3410000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1220000000000001,"15min":1.123},"min":{"1min":0.77900000000000003,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.329,"15min":1.325},"min":{"1min":0.97499999999999998,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0329999999999999,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3120000000000001,"15min":1.3},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.877,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.099}]},{"osd":5,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3280000000000001,"15min":1.3360000000000001},"min":{"1min":1.008,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.601,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.268,"15min":1.2629999999999999},"min":{"1min":0.98799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":0.96299999999999997}]},{"osd":6,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.288,"15min":1.296},"min":{"1min":0.94399999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.391,"15min":1.377},"min":{"1min":1.1080000000000001,"5min":0.94899999999999995,"15min":0.92100000000000004},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":1.218}]},{"osd":7,"last update":"Tue Dec 19 08:05:40 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3759999999999999,"15min":1.383},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.95899999999999996},"max":{"1min":1.661,"5min":1.9730000000000001,"15min":2.214},"last":1.175},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.377,"15min":1.373},"min":{"1min":0.99399999999999999,"5min":0.98699999999999999,"15min":0.88200000000000001},"max":{"1min":4.109,"5min":4.109,"15min":4.109},"last":1.1339999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307232,"kb_used_data":1752,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064608,"statfs":{"total":96636764160,"available":96322158592,"internally_reserved":0,"allocated":1794048,"data_stored":942903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.86299999999999999,"15min":0.85599999999999998},"min":{"1min":0.61599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.252,"5min":1.647,"15min":1.661},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92200000000000004,"15min":0.93600000000000005},"min":{"1min":0.52900000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.421,"5min":2.2050000000000001,"15min":2.3570000000000002},"last":0.64500000000000002}]},{"osd":2,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.56899999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.165,"5min":1.96,"15min":2.96},"last":0.627},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0449999999999999,"15min":1.0509999999999999},"min":{"1min":0.64100000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.548,"5min":2.323,"15min":3.1349999999999998},"last":0.77200000000000002}]},{"osd":3,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.159,"15min":1.161},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.641,"5min":2.2530000000000001,"15min":3.1800000000000002},"last":0.874},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51200000000000001},"max":{"1min":1.746,"5min":2.1299999999999999,"15min":2.407},"last":0.60199999999999998}]},{"osd":4,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1779999999999999,"15min":1.1899999999999999},"min":{"1min":0.871,"5min":0.502,"15min":0.502},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.5030000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.325,"15min":1.3109999999999999},"min":{"1min":0.89500000000000002,"5min":0.68200000000000005,"15min":0.64400000000000002},"max":{"1min":1.875,"5min":2.2890000000000001,"15min":3.04},"last":1.171}]},{"osd":5,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2689999999999999,"15min":1.276},"min":{"1min":0.87,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.7330000000000001,"5min":1.996,"15min":2.5390000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.2849999999999999,"15min":1.286},"min":{"1min":0.98099999999999998,"5min":0.85899999999999999,"15min":0.63800000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.3450000000000002},"last":1.0740000000000001}]},{"osd":6,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3740000000000001,"15min":1.3819999999999999},"min":{"1min":1.0629999999999999,"5min":0.79000000000000004,"15min":0.72799999999999998},"max":{"1min":1.772,"5min":2.3559999999999999,"15min":3.089},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.367,"15min":1.365},"min":{"1min":0.98899999999999999,"5min":0.76300000000000001,"15min":0.747},"max":{"1min":1.919,"5min":2.169,"15min":2.46},"last":1.135}]},{"osd":7,"last update":"Tue Dec 19 08:05:38 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.4179999999999999,"15min":1.4199999999999999},"min":{"1min":0.99299999999999999,"5min":0.89300000000000002,"15min":0.88500000000000001},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.504},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3939999999999999,"15min":1.389},"min":{"1min":1.0309999999999999,"5min":0.81799999999999995,"15min":0.73999999999999999},"max":{"1min":2.0779999999999998,"5min":2.7290000000000001,"15min":2.7290000000000001},"last":1.1539999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575074,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307796,"kb_used_data":2316,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064044,"statfs":{"total":96636764160,"available":96321581056,"internally_reserved":0,"allocated":2371584,"data_stored":1525104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.51900000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.90800000000000003,"15min":0.91300000000000003},"min":{"1min":0.64900000000000002,"5min":0.54500000000000004,"15min":0.496},"max":{"1min":1.3700000000000001,"5min":1.466,"15min":1.5289999999999999},"last":1.147}]},{"osd":2,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99399999999999999,"15min":0.97799999999999998},"min":{"1min":0.77600000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.413,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.522},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.01,"15min":1.03},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.56899999999999995},"max":{"1min":1.3779999999999999,"5min":1.504,"15min":1.6000000000000001},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.95499999999999996,"15min":0.96299999999999997},"min":{"1min":0.59499999999999997,"5min":0.54500000000000004,"15min":0.51400000000000001},"max":{"1min":1.4590000000000001,"5min":1.5529999999999999,"15min":1.671},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.155,"5min":1.133,"15min":1.133},"min":{"1min":0.76100000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.522,"5min":1.663,"15min":1.6659999999999999},"last":1.2490000000000001}]},{"osd":4,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3360000000000001,"15min":1.3160000000000001},"min":{"1min":1.022,"5min":0.86499999999999999,"15min":0.71399999999999997},"max":{"1min":1.7,"5min":1.7,"15min":1.8620000000000001},"last":1.643},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.337,"15min":1.325},"min":{"1min":0.96999999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.296}]},{"osd":5,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.353,"15min":1.353},"min":{"1min":1.0449999999999999,"5min":0.83399999999999996,"15min":0.74299999999999999},"max":{"1min":1.7849999999999999,"5min":1.9099999999999999,"15min":2.085},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.3560000000000001,"15min":1.357},"min":{"1min":0.97499999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.8180000000000001,"5min":1.843,"15min":2.125},"last":1.821}]},{"osd":6,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.2689999999999999},"min":{"1min":1.036,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.6479999999999999,"5min":1.97,"15min":1.97},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.341,"15min":1.3220000000000001},"min":{"1min":0.94899999999999995,"5min":0.91900000000000004,"15min":0.75600000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.754}]},{"osd":7,"last update":"Tue Dec 19 08:05:52 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3220000000000001,"15min":1.3160000000000001},"min":{"1min":1.1140000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.7090000000000001,"5min":1.8009999999999999,"15min":1.887},"last":1.599},{"interface":"front","average":{"1min":1.401,"5min":1.3839999999999999,"15min":1.369},"min":{"1min":1.121,"5min":0.88600000000000001,"15min":0.67700000000000005},"max":{"1min":1.756,"5min":1.903,"15min":2.0059999999999998},"last":1.71}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T08:06:28.012 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T08:06:28.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T08:06:28.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.217+0000 7ff4af59e700 1 Processor -- start 2023-12-19T08:06:28.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.217+0000 7ff4af59e700 1 -- start start 2023-12-19T08:06:28.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b006aeb0 0x7ff4b0159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4af59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff4b01595a0 0x7ff4b0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4af59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006c850 con 0x7ff4b0158c90 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4af59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b00cb650 con 0x7ff4b01595a0 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b006b470 con 0x7ff4b006aeb0 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4ae59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff4b0158c90 0x7ff4b00caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33092/0 (socket says 172.21.15.111:33092) 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4ae59c700 1 -- 172.21.15.111:0/1624328916 learned_addr learned my addr 172.21.15.111:0/1624328916 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:28.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.221+0000 7ff4aed9d700 1 --2- 172.21.15.111:0/1624328916 >> v2:172.21.15.111:3300/0 conn(0x7ff4b01595a0 0x7ff4b0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4add9b700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b006aeb0 0x7ff4b0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4ad59a700 1 -- 172.21.15.111:0/1624328916 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274837188 0 0) 0x7ff4b006c850 con 0x7ff4b0158c90 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4ad59a700 1 -- 172.21.15.111:0/1624328916 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003570 con 0x7ff4b0158c90 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4aed9d700 1 -- 172.21.15.111:0/1624328916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b006aeb0 msgr2=0x7ff4b0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4aed9d700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b006aeb0 0x7ff4b0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4aed9d700 1 -- 172.21.15.111:0/1624328916 >> v1:172.21.15.111:6789/0 conn(0x7ff4b0158c90 legacy=0x7ff4b00caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4aed9d700 1 -- 172.21.15.111:0/1624328916 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b006b750 con 0x7ff4b01595a0 2023-12-19T08:06:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4aed9d700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b01595a0 0x7ff4b0159a20 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009e60 tx=0x7ff4a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c99b69c4ce2f8db server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4ad59a700 1 -- 172.21.15.111:0/1624328916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a800e070 con 0x7ff4b01595a0 2023-12-19T08:06:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4ad59a700 1 -- 172.21.15.111:0/1624328916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a800bab0 con 0x7ff4b01595a0 2023-12-19T08:06:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4ad59a700 1 -- 172.21.15.111:0/1624328916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a800be70 con 0x7ff4b01595a0 2023-12-19T08:06:28.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 -- 172.21.15.111:0/1624328916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b01595a0 msgr2=0x7ff4b0159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b01595a0 0x7ff4b0159a20 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009e60 tx=0x7ff4a8009780 comp rx=0 tx=0).stop 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 -- 172.21.15.111:0/1624328916 shutdown_connections 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b01595a0 0x7ff4b0159a20 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 --2- 172.21.15.111:0/1624328916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b006aeb0 0x7ff4b0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 -- 172.21.15.111:0/1624328916 >> 172.21.15.111:0/1624328916 conn(0x7ff4b0067bb0 msgr2=0x7ff4b00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 -- 172.21.15.111:0/1624328916 shutdown_connections 2023-12-19T08:06:28.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.225+0000 7ff4af59e700 1 -- 172.21.15.111:0/1624328916 wait complete. 2023-12-19T08:06:28.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 Processor -- start 2023-12-19T08:06:28.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 -- start start 2023-12-19T08:06:28.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b0158c90 0x7ff4b00b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4b01595a0 0x7ff4b00b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b015bd40 con 0x7ff4b006aeb0 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b015cba0 con 0x7ff4b01595a0 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4af59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b015bf90 con 0x7ff4b0158c90 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53076/0 (socket says 172.21.15.111:53076) 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 -- 172.21.15.111:0/2939552586 learned_addr learned my addr 172.21.15.111:0/2939552586 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4aed9d700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4b01595a0 0x7ff4b00b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4add9b700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b0158c90 0x7ff4b00b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 -- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b0158c90 msgr2=0x7ff4b00b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b0158c90 0x7ff4b00b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 -- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4b01595a0 msgr2=0x7ff4b00b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4b01595a0 0x7ff4b00b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 -- 172.21.15.111:0/2939552586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8009a20 con 0x7ff4b006aeb0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.229+0000 7ff4ae59c700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c7f0 tx=0x7ff4a0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0005cd0 con 0x7ff4b006aeb0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0005e50 con 0x7ff4b006aeb0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b015c270 con 0x7ff4b006aeb0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0007320 con 0x7ff4b006aeb0 2023-12-19T08:06:28.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b015c830 con 0x7ff4b006aeb0 2023-12-19T08:06:28.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b004be40 con 0x7ff4b006aeb0 2023-12-19T08:06:28.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.233+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff4a001b3b0 con 0x7ff4b006aeb0 2023-12-19T08:06:28.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.237+0000 7ff49e7fc700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 0x7ff490062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.237+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7ff4a007c320 con 0x7ff4b006aeb0 2023-12-19T08:06:28.242 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-12-19T08:06:28.237+0000 7fdcf8e71700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T08:06:28.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.237+0000 7ff4add9b700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 0x7ff490062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.237+0000 7ff4add9b700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 0x7ff490062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4b015e460 tx=0x7ff4a40091a0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.241+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff4a004c4a0 con 0x7ff4b006aeb0 2023-12-19T08:06:28.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.469+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff4b00b2670 con 0x7ff4b006aeb0 2023-12-19T08:06:28.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.481+0000 7ff49e7fc700 1 -- 172.21.15.111:0/2939552586 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff4a004fac0 con 0x7ff4b006aeb0 2023-12-19T08:06:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 msgr2=0x7ff490062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 0x7ff490062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4b015e460 tx=0x7ff4a40091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 msgr2=0x7ff4b01433b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c7f0 tx=0x7ff4a0007e00 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 shutdown_connections 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff490060670 0x7ff490062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4b01595a0 0x7ff4b00b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4b0158c90 0x7ff4b00b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 --2- 172.21.15.111:0/2939552586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b006aeb0 0x7ff4b01433b0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 >> 172.21.15.111:0/2939552586 conn(0x7ff4b0067bb0 msgr2=0x7ff4b0157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 shutdown_connections 2023-12-19T08:06:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.485+0000 7ff4af59e700 1 -- 172.21.15.111:0/2939552586 wait complete. 2023-12-19T08:06:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T08:06:28.508 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T08:06:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 Processor -- start 2023-12-19T08:06:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 -- start start 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 0x7ffb3c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb3c06de20 0x7ffb3c0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3c06d990 con 0x7ffb3c06c7c0 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3c15a570 con 0x7ffb3c155960 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb41323700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3c06cd80 con 0x7ffb3c06de20 2023-12-19T08:06:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb3bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 0x7ffb3c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb3bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 0x7ffb3c15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53078/0 (socket says 172.21.15.111:53078) 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb3bfff700 1 -- 172.21.15.111:0/3120376355 learned_addr learned my addr 172.21.15.111:0/3120376355 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.665+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3120376355 >> v2:172.21.15.143:3300/0 conn(0x7ffb3c06de20 0x7ffb3c0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3120376355 >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 msgr2=0x7ffb3c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3120376355 >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 0x7ffb3c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3120376355 >> v1:172.21.15.143:6789/0 conn(0x7ffb3c06c7c0 legacy=0x7ffb3c0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3120376355 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c06d060 con 0x7ffb3c06de20 2023-12-19T08:06:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3120376355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06de20 0x7ffb3c0c77f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c06bf60 tx=0x7ffb300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cee000426d0bb51 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3affd700 1 -- 172.21.15.111:0/3120376355 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30011630 con 0x7ffb3c06de20 2023-12-19T08:06:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3affd700 1 -- 172.21.15.111:0/3120376355 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30011790 con 0x7ffb3c06de20 2023-12-19T08:06:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb3affd700 1 -- 172.21.15.111:0/3120376355 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30011b30 con 0x7ffb3c06de20 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 -- 172.21.15.111:0/3120376355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06de20 msgr2=0x7ffb3c0c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 --2- 172.21.15.111:0/3120376355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06de20 0x7ffb3c0c77f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c06bf60 tx=0x7ffb300091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 -- 172.21.15.111:0/3120376355 shutdown_connections 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 --2- 172.21.15.111:0/3120376355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06de20 0x7ffb3c0c77f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 --2- 172.21.15.111:0/3120376355 >> v2:172.21.15.111:3300/0 conn(0x7ffb3c155960 0x7ffb3c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 -- 172.21.15.111:0/3120376355 >> 172.21.15.111:0/3120376355 conn(0x7ffb3c067bb0 msgr2=0x7ffb3c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 -- 172.21.15.111:0/3120376355 shutdown_connections 2023-12-19T08:06:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.669+0000 7ffb41323700 1 -- 172.21.15.111:0/3120376355 wait complete. 2023-12-19T08:06:28.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.673+0000 7ffb41323700 1 Processor -- start 2023-12-19T08:06:28.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.673+0000 7ffb41323700 1 -- start start 2023-12-19T08:06:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06c7c0 0x7ffb3c1465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb3c155960 0x7ffb3c0adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c16a610 con 0x7ffb3c06de20 2023-12-19T08:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0c61e0 con 0x7ffb3c06c7c0 2023-12-19T08:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c0c64c0 con 0x7ffb3c155960 2023-12-19T08:06:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53084/0 (socket says 172.21.15.111:53084) 2023-12-19T08:06:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3244984957 learned_addr learned my addr 172.21.15.111:0/3244984957 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3bfff700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06c7c0 0x7ffb3c1465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb40b22700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb3c155960 0x7ffb3c0adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb3c155960 msgr2=0x7ffb3c0adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb3c155960 0x7ffb3c0adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06c7c0 msgr2=0x7ffb3c1465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06c7c0 0x7ffb3c1465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 -- 172.21.15.111:0/3244984957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30009420 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb3b7fe700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c06cec0 tx=0x7ffb30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30000d10 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30011900 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c0c67a0 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30025420 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.677+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c146b20 con 0x7ffb3c06de20 2023-12-19T08:06:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.681+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3c04be40 con 0x7ffb3c06de20 2023-12-19T08:06:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.681+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ffb30023070 con 0x7ffb3c06de20 2023-12-19T08:06:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.681+0000 7ffb23fff700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 0x7ffb28062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.685+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7ffb3007e230 con 0x7ffb3c06de20 2023-12-19T08:06:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.685+0000 7ffb3bfff700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 0x7ffb28062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.685+0000 7ffb3bfff700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 0x7ffb28062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c006fd0 tx=0x7ffb2c009250 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.689+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb3004e3b0 con 0x7ffb3c06de20 2023-12-19T08:06:28.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.897+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb3c0cc350 con 0x7ffb3c06de20 2023-12-19T08:06:28.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.897+0000 7ffb23fff700 1 -- 172.21.15.111:0/3244984957 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7ffb300519d0 con 0x7ffb3c06de20 2023-12-19T08:06:28.904 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:28.904 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 msgr2=0x7ffb28062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 0x7ffb28062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c006fd0 tx=0x7ffb2c009250 comp rx=0 tx=0).stop 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 msgr2=0x7ffb3c1487b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c06cec0 tx=0x7ffb30017000 comp rx=0 tx=0).stop 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 shutdown_connections 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb28060670 0x7ffb28062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb3c155960 0x7ffb3c0adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb3c06de20 0x7ffb3c1487b0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 --2- 172.21.15.111:0/3244984957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb3c06c7c0 0x7ffb3c1465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:28.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 >> 172.21.15.111:0/3244984957 conn(0x7ffb3c067bb0 msgr2=0x7ffb3c0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:28.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 shutdown_connections 2023-12-19T08:06:28.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:28.901+0000 7ffb41323700 1 -- 172.21.15.111:0/3244984957 wait complete. 2023-12-19T08:06:28.908 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T08:06:29.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T08:06:22.717187+0000', 'modified': '2023-12-19T08:06:26.311289+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8394}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8394': {'gid': 8394, 'name': 'c', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/794754877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 794754877}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 794754877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T08:06:29.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T08:06:29.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-19T08:06:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.493+0000 7f01698cd700 1 Processor -- start 2023-12-19T08:06:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.493+0000 7f01698cd700 1 -- start start 2023-12-19T08:06:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01698cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 0x7f016415b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01698cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016406d7d0 con 0x7f01640b13e0 2023-12-19T08:06:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01698cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016406cc50 con 0x7f016406c7c0 2023-12-19T08:06:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01698cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01640ae280 con 0x7f0164140420 2023-12-19T08:06:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01637fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f01640b13e0 0x7f01640ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33106/0 (socket says 172.21.15.111:33106) 2023-12-19T08:06:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01637fe700 1 -- 172.21.15.111:0/3799357427 learned_addr learned my addr 172.21.15.111:0/3799357427 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f0162ffd700 1 --2- 172.21.15.111:0/3799357427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 0x7f016415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007035649 0 0) 0x7f016406d7d0 con 0x7f01640b13e0 2023-12-19T08:06:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003570 con 0x7f01640b13e0 2023-12-19T08:06:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669618727 0 0) 0x7f016406cc50 con 0x7f016406c7c0 2023-12-19T08:06:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f016406d7d0 con 0x7f016406c7c0 2023-12-19T08:06:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f0162ffd700 1 -- 172.21.15.111:0/3799357427 >> v1:172.21.15.143:6789/0 conn(0x7f016406c7c0 legacy=0x7f01640b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f0162ffd700 1 -- 172.21.15.111:0/3799357427 >> v1:172.21.15.111:6789/0 conn(0x7f01640b13e0 legacy=0x7f01640ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689349968 0 0) 0x7f0144003570 con 0x7f01640b13e0 2023-12-19T08:06:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.497+0000 7f0162ffd700 1 -- 172.21.15.111:0/3799357427 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016406cf30 con 0x7f0164140420 2023-12-19T08:06:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f0162ffd700 1 --2- 172.21.15.111:0/3799357427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 0x7f016415b220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0154009910 tx=0x7f01540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55f1b50987ade00 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c0017f0 con 0x7f0164140420 2023-12-19T08:06:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154011420 con 0x7f0164140420 2023-12-19T08:06:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01627fc700 1 -- 172.21.15.111:0/3799357427 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540117c0 con 0x7f0164140420 2023-12-19T08:06:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 -- 172.21.15.111:0/3799357427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 msgr2=0x7f016415b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 --2- 172.21.15.111:0/3799357427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 0x7f016415b220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0154009910 tx=0x7f01540091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 -- 172.21.15.111:0/3799357427 shutdown_connections 2023-12-19T08:06:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 --2- 172.21.15.111:0/3799357427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0164140420 0x7f016415b220 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 -- 172.21.15.111:0/3799357427 >> 172.21.15.111:0/3799357427 conn(0x7f0164067b60 msgr2=0x7f01640ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 -- 172.21.15.111:0/3799357427 shutdown_connections 2023-12-19T08:06:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.501+0000 7f01698cd700 1 -- 172.21.15.111:0/3799357427 wait complete. 2023-12-19T08:06:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 Processor -- start 2023-12-19T08:06:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 -- start start 2023-12-19T08:06:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f016406c7c0 0x7f01641365a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01640b13e0 0x7f01640af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016415c510 con 0x7f0164140420 2023-12-19T08:06:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0164136ae0 con 0x7f01640b13e0 2023-12-19T08:06:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01698cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016415b900 con 0x7f016406c7c0 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53088/0 (socket says 172.21.15.111:53088) 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 -- 172.21.15.111:0/1064317535 learned_addr learned my addr 172.21.15.111:0/1064317535 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f016406c7c0 0x7f01641365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f015a7fc700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01640b13e0 0x7f01640af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 -- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f016406c7c0 msgr2=0x7f01641365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f016406c7c0 0x7f01641365a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 -- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01640b13e0 msgr2=0x7f01640af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01640b13e0 0x7f01640af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.505+0000 7f01637fe700 1 -- 172.21.15.111:0/1064317535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154009420 con 0x7f0164140420 2023-12-19T08:06:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f01637fe700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f015c00cb70 tx=0x7f015c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f015b7fe700 1 -- 172.21.15.111:0/1064317535 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c004dc0 con 0x7f0164140420 2023-12-19T08:06:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f015b7fe700 1 -- 172.21.15.111:0/1064317535 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c00a700 con 0x7f0164140420 2023-12-19T08:06:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016415bbe0 con 0x7f0164140420 2023-12-19T08:06:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f015b7fe700 1 -- 172.21.15.111:0/1064317535 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c00aac0 con 0x7f0164140420 2023-12-19T08:06:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016415c1a0 con 0x7f0164140420 2023-12-19T08:06:29.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.509+0000 7f015b7fe700 1 -- 172.21.15.111:0/1064317535 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f015c007150 con 0x7f0164140420 2023-12-19T08:06:29.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.513+0000 7f015b7fe700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 0x7f0148062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.513+0000 7f015b7fe700 1 -- 172.21.15.111:0/1064317535 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f015c07c650 con 0x7f0164140420 2023-12-19T08:06:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.513+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 0x7f0148062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.513+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 0x7f0148062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01540093f0 tx=0x7f0154016000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 Processor -- start 2023-12-19T08:06:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 -- start start 2023-12-19T08:06:29.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394710 0x2612440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x23df790 0x2612980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x253ff30 con 0x2394ae0 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x253bed0 con 0x23df790 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x25020b0 con 0x2394710 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53094/0 (socket says 172.21.15.111:53094) 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 -- 172.21.15.111:0/4100282828 learned_addr learned my addr 172.21.15.111:0/4100282828 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f01637fe700 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x23df790 0x2612980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f015a7fc700 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394710 0x2612440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 -- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394710 msgr2=0x2612440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394710 0x2612440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 -- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x23df790 msgr2=0x2612980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x23df790 0x2612980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.517+0000 7f0162ffd700 1 -- 172.21.15.111:0/4100282828 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23e09e0 con 0x2394ae0 2023-12-19T08:06:29.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f0162ffd700 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f0154006d80 tx=0x7f0154021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b944d36fc61a399b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f0153fff700 1 -- 172.21.15.111:0/4100282828 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154021070 con 0x2394ae0 2023-12-19T08:06:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f0153fff700 1 -- 172.21.15.111:0/4100282828 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154033480 con 0x2394ae0 2023-12-19T08:06:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f0153fff700 1 -- 172.21.15.111:0/4100282828 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154033670 con 0x2394ae0 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- 172.21.15.111:0/4100282828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 msgr2=0x2611f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f0154006d80 tx=0x7f0154021000 comp rx=0 tx=0).stop 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- 172.21.15.111:0/4100282828 shutdown_connections 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x23df790 0x2612980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394710 0x2612440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 --2- 172.21.15.111:0/4100282828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2611f00 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- 172.21.15.111:0/4100282828 >> 172.21.15.111:0/4100282828 conn(0x2394eb0 msgr2=0x25ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- 172.21.15.111:0/4100282828 shutdown_connections 2023-12-19T08:06:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- 172.21.15.111:0/4100282828 wait complete. 2023-12-19T08:06:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.521+0000 7f016bb91740 1 -- start start 2023-12-19T08:06:29.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 Processor -- start 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2616510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x2616a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2501d50 con 0x23df790 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x24da970 con 0x2394710 2023-12-19T08:06:29.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x2504920 con 0x2394ae0 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f016bb91740 10 client.0 _ll_register_callbacks cb 0x25856a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53100/0 (socket says 172.21.15.111:53100) 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f01637fe700 1 -- 172.21.15.111:0/2212001501 learned_addr learned my addr 172.21.15.111:0/2212001501 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f015a7fc700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x2616a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f0162ffd700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2616510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f01637fe700 1 -- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 msgr2=0x2616a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.525+0000 7f01637fe700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x2616a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01637fe700 1 -- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 msgr2=0x2616510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01637fe700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2616510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01637fe700 1 -- 172.21.15.111:0/2212001501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016406cf30 con 0x23df790 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01637fe700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f015c07a780 tx=0x7f015c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3885b28ac3ca54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c04fa00 con 0x23df790 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f015c074e40 con 0x23df790 2023-12-19T08:06:29.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x24dacc0 con 0x23df790 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f015c057060 con 0x23df790 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 10 client.8538 fetch_fsmap learned FSMap version 63 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x250a370 con 0x23df790 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x24dacc0 con 0x23df790 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 63) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f015c04c300 con 0x23df790 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 10 client.8538 fetch_fsmap finished waiting for FSMap version 63 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 10 client.8538 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 20 client.8538 populate_metadata read hostname 'smithi111' 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x25f9220 0x26216b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 4 client.8538 mds_command: new command op to 8394 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 --> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x24de3e0 con 0x25f9220 2023-12-19T08:06:29.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f0162ffd700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x25f9220 0x26216b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f0162ffd700 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x25f9220 0x26216b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a42a2beb08d31fd server_cookie=4c6e3aa7ffe2418f in_seq=0 out_seq=0 2023-12-19T08:06:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.529+0000 7f01517fa700 10 client.8538 ms_handle_connect on v2:172.21.15.111:6834/794754877 2023-12-19T08:06:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f01517fa700 1 -- 172.21.15.111:0/2212001501 <== mds.0 v2:172.21.15.111:6834/794754877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x24de3e0 con 0x25f9220 2023-12-19T08:06:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f01517fa700 10 client.8538 handle_command_reply: tid=0 2023-12-19T08:06:29.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 client.8538 shutdown 2023-12-19T08:06:29.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 msgr2=0x25f7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f015c07a780 tx=0x7f015c009040 comp rx=0 tx=0).stop 2023-12-19T08:06:29.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 shutdown_connections 2023-12-19T08:06:29.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x25f9220 0x26216b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x25f7cb0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x2616a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 --2- 172.21.15.111:0/2212001501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2616510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f01517fa700 0 client.8538 ms_handle_reset on v2:172.21.15.111:6834/794754877 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 >> 172.21.15.111:0/2212001501 conn(0x2394eb0 msgr2=0x25c6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 shutdown_connections 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 1 -- 172.21.15.111:0/2212001501 wait complete. 2023-12-19T08:06:29.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.533+0000 7f016bb91740 20 client.8538 trim_cache size 0 max 16384 2023-12-19T08:06:29.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 Processor -- start 2023-12-19T08:06:29.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 -- start start 2023-12-19T08:06:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x25c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x25c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x24dcb80 con 0x23df790 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x250f040 con 0x2394710 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x2540ef0 con 0x2394ae0 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f01637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53112/0 (socket says 172.21.15.111:53112) 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f01637fe700 1 -- 172.21.15.111:0/698727142 learned_addr learned my addr 172.21.15.111:0/698727142 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f015a7fc700 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x25c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.553+0000 7f0162ffd700 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x25c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 -- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 msgr2=0x25c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x25c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 -- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 msgr2=0x25c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x25c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 -- 172.21.15.111:0/698727142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24f0d80 con 0x23df790 2023-12-19T08:06:29.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f01637fe700 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f015c007f00 tx=0x7f015c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74b43745f499aea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f0158ff9700 1 -- 172.21.15.111:0/698727142 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c04fa00 con 0x23df790 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f0158ff9700 1 -- 172.21.15.111:0/698727142 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c050050 con 0x23df790 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f0158ff9700 1 -- 172.21.15.111:0/698727142 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c04c680 con 0x23df790 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- 172.21.15.111:0/698727142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 msgr2=0x26116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f015c007f00 tx=0x7f015c013000 comp rx=0 tx=0).stop 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- 172.21.15.111:0/698727142 shutdown_connections 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x23df790 0x26116b0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2394ae0 0x25c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 --2- 172.21.15.111:0/698727142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x25c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- 172.21.15.111:0/698727142 >> 172.21.15.111:0/698727142 conn(0x2394eb0 msgr2=0x25c81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- 172.21.15.111:0/698727142 shutdown_connections 2023-12-19T08:06:29.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- 172.21.15.111:0/698727142 wait complete. 2023-12-19T08:06:29.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 -- start start 2023-12-19T08:06:29.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.557+0000 7f016bb91740 1 Processor -- start 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2674c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 0x25cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2537b50 con 0x2394ae0 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x253f9d0 con 0x2394710 2023-12-19T08:06:29.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x24de3e0 con 0x23df790 2023-12-19T08:06:29.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f016bb91740 10 client.0 _ll_register_callbacks cb 0x23609d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:58450/0 (socket says 172.21.15.111:58450) 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 -- 172.21.15.111:0/1077434241 learned_addr learned my addr 172.21.15.111:0/1077434241 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f015a7fc700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2674c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f01637fe700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 0x25cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 -- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 msgr2=0x25cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 0x25cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 -- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 msgr2=0x2674c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2674c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 -- 172.21.15.111:0/1077434241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x250a370 con 0x2394710 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f01637fe700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 0x25cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f0162ffd700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0154003010 tx=0x7f015402b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da93dd1e58e7f703 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.561+0000 7f01517fa700 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2023-12-19T08:06:29.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015403a040 con 0x2394710 2023-12-19T08:06:29.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0154040490 con 0x2394710 2023-12-19T08:06:29.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0154033480 con 0x2394710 2023-12-19T08:06:29.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0154021040 con 0x2394710 2023-12-19T08:06:29.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 10 client.8317 fetch_fsmap learned FSMap version 63 2023-12-19T08:06:29.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x24e5130 con 0x2394710 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154033480 con 0x2394710 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 63) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0154040880 con 0x2394710 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 10 client.8317 fetch_fsmap finished waiting for FSMap version 63 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 10 client.8317 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 20 client.8317 populate_metadata read hostname 'smithi111' 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x267b550 0x267d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 4 client.8317 mds_command: new command op to 8394 tid=0[{"prefix": "status"}] 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.565+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 --> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] -- command(tid 0: {"prefix": "status"}) v1 -- 0x2510410 con 0x267b550 2023-12-19T08:06:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f015a7fc700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x267b550 0x267d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f015a7fc700 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x267b550 0x267d930 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=555e150c51f8d947 server_cookie=82204ae323aa1fa3 in_seq=0 out_seq=0 2023-12-19T08:06:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01517fa700 10 client.8317 ms_handle_connect on v2:172.21.15.111:6834/794754877 2023-12-19T08:06:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01517fa700 1 -- 172.21.15.111:0/1077434241 <== mds.0 v2:172.21.15.111:6834/794754877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x2510410 con 0x267b550 2023-12-19T08:06:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01517fa700 10 client.8317 handle_command_reply: tid=0 2023-12-19T08:06:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 client.8317 shutdown 2023-12-19T08:06:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 msgr2=0x2674740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0154003010 tx=0x7f015402b040 comp rx=0 tx=0).stop 2023-12-19T08:06:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 shutdown_connections 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:6834/794754877,v1:172.21.15.111:6835/794754877] conn(0x267b550 0x267d930 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x23df790 0x25cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2394ae0 0x2674c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 --2- 172.21.15.111:0/1077434241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2394710 0x2674740 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 >> 172.21.15.111:0/1077434241 conn(0x2394eb0 msgr2=0x25f5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01517fa700 0 client.8317 ms_handle_reset on v2:172.21.15.111:6834/794754877 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 shutdown_connections 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 1 -- 172.21.15.111:0/1077434241 wait complete. 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f016bb91740 20 client.8317 trim_cache size 0 max 16384 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:06:29.575 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 8394, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2785605569999996, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 63, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 144, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 144, 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.220917155 2023-12-19T08:06:29.576 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 msgr2=0x7f0148062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 0x7f0148062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01540093f0 tx=0x7f0154016000 comp rx=0 tx=0).stop 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 msgr2=0x7f01640afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f015c00cb70 tx=0x7f015c007c70 comp rx=0 tx=0).stop 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 shutdown_connections 2023-12-19T08:06:29.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0148060670 0x7f0148062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01640b13e0 0x7f01640af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0164140420 0x7f01640afb80 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 --2- 172.21.15.111:0/1064317535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f016406c7c0 0x7f01641365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 >> 172.21.15.111:0/1064317535 conn(0x7f0164067b60 msgr2=0x7f01640cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.569+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 shutdown_connections 2023-12-19T08:06:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.573+0000 7f01698cd700 1 -- 172.21.15.111:0/1064317535 wait complete. 2023-12-19T08:06:29.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T08:06:29.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.753+0000 7f09f11cc700 1 Processor -- start 2023-12-19T08:06:29.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.753+0000 7f09f11cc700 1 -- start start 2023-12-19T08:06:29.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec0b1410 0x7f09ec0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec06a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec06c7c0 con 0x7f09ec15a860 2023-12-19T08:06:29.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec06cd80 con 0x7f09ec0b1410 2023-12-19T08:06:29.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec06af10 con 0x7f09ec06a580 2023-12-19T08:06:29.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec0b1410 0x7f09ec0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ec0b1410 0x7f09ec0ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53130/0 (socket says 172.21.15.111:53130) 2023-12-19T08:06:29.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 -- 172.21.15.111:0/2275663455 learned_addr learned my addr 172.21.15.111:0/2275663455 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09eb7fe700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec06a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 -- 172.21.15.111:0/2275663455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 msgr2=0x7f09ec06a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:29.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec06a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 -- 172.21.15.111:0/2275663455 >> v1:172.21.15.143:6789/0 conn(0x7f09ec15a860 legacy=0x7f09ec15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 -- 172.21.15.111:0/2275663455 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec06b8f0 con 0x7f09ec0b1410 2023-12-19T08:06:29.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09ebfff700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec0ab440 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0099b0 tx=0x7f09dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9f1e9d1caae912 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09eaffd700 1 -- 172.21.15.111:0/2275663455 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc009ea0 con 0x7f09ec0b1410 2023-12-19T08:06:29.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09eaffd700 1 -- 172.21.15.111:0/2275663455 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc00b860 con 0x7f09ec0b1410 2023-12-19T08:06:29.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09eaffd700 1 -- 172.21.15.111:0/2275663455 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00bc00 con 0x7f09ec0b1410 2023-12-19T08:06:29.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 -- 172.21.15.111:0/2275663455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 msgr2=0x7f09ec0ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec0ab440 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0099b0 tx=0x7f09dc009250 comp rx=0 tx=0).stop 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 -- 172.21.15.111:0/2275663455 shutdown_connections 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec06a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.757+0000 7f09f11cc700 1 --2- 172.21.15.111:0/2275663455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec0ab440 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.761+0000 7f09f11cc700 1 -- 172.21.15.111:0/2275663455 >> 172.21.15.111:0/2275663455 conn(0x7f09ec067bb0 msgr2=0x7f09ec0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.761+0000 7f09f11cc700 1 -- 172.21.15.111:0/2275663455 shutdown_connections 2023-12-19T08:06:29.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.761+0000 7f09f11cc700 1 -- 172.21.15.111:0/2275663455 wait complete. 2023-12-19T08:06:29.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.761+0000 7f09f11cc700 1 Processor -- start 2023-12-19T08:06:29.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.761+0000 7f09f11cc700 1 -- start start 2023-12-19T08:06:29.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec15a860 0x7f09ec155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1582e0 con 0x7f09ec0b1410 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec158530 con 0x7f09ec15a860 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec158810 con 0x7f09ec06a580 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53138/0 (socket says 172.21.15.111:53138) 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 -- 172.21.15.111:0/1741778795 learned_addr learned my addr 172.21.15.111:0/1741778795 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09ebfff700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f09cb700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec15a860 0x7f09ec155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 -- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 msgr2=0x7f09ec0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec0c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 -- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec15a860 msgr2=0x7f09ec155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec15a860 0x7f09ec155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 -- 172.21.15.111:0/1741778795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc0094f0 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09ebfff700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09eb7fe700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f09e000e930 tx=0x7f09e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000eed0 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e000c850 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec158af0 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000cbf0 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec0c87a0 con 0x7f09ec0b1410 2023-12-19T08:06:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.765+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec04be40 con 0x7f09ec0b1410 2023-12-19T08:06:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.769+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f09e0013470 con 0x7f09ec0b1410 2023-12-19T08:06:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.769+0000 7f09d3fff700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 0x7f09d8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.769+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f09e007cec0 con 0x7f09ec0b1410 2023-12-19T08:06:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.769+0000 7f09ebfff700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 0x7f09d8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.773+0000 7f09ebfff700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 0x7f09d8062b50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0099b0 tx=0x7f09dc017000 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:29.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.777+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f09e004cfa0 con 0x7f09ec0b1410 2023-12-19T08:06:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.981+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f09ec0cc340 con 0x7f09ec0b1410 2023-12-19T08:06:29.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.981+0000 7f09d3fff700 1 -- 172.21.15.111:0/1741778795 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7f09e00505c0 con 0x7f09ec0b1410 2023-12-19T08:06:29.985 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 msgr2=0x7f09d8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 0x7f09d8062b50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0099b0 tx=0x7f09dc017000 comp rx=0 tx=0).stop 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 msgr2=0x7f09ec154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f09e000e930 tx=0x7f09e00091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 shutdown_connections 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f09d80606a0 0x7f09d8062b50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09ec15a860 0x7f09ec155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec0b1410 0x7f09ec154af0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 --2- 172.21.15.111:0/1741778795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09ec06a580 0x7f09ec0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 >> 172.21.15.111:0/1741778795 conn(0x7f09ec067bb0 msgr2=0x7f09ec0ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 shutdown_connections 2023-12-19T08:06:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:29.985+0000 7f09f11cc700 1 -- 172.21.15.111:0/1741778795 wait complete. 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T08:06:30.012 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:06:30.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.013 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T08:06:30.029 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T08:06:30.029 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T08:06:30.029 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T08:06:30.030 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41268sec preferred_lft 41268sec 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:06:30.031 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T08:06:30.032 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T08:06:30.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.033 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T08:06:30.112 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T08:06:30.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.114 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T08:06:30.176 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T08:06:30.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.177 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T08:06:30.177 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T08:06:30.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T08:06:30.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T08:06:30.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T08:06:30.178 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T08:06:30.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.306 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T08:06:30.326 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T08:06:30.326 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T08:06:30.327 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:06:30.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T08:06:30.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T08:06:30.409 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T08:06:30.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:06:30.409 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:30.471 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T08:06:30.471 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T08:06:30.471 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:30.530 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:06:30.615 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:06:30.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:06:30.690 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:06:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:06:30.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:30.693 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:06:30.756 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T08:06:30.757 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T08:06:30.806 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T08:06:30.841 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T08:06:30.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T08:06:30.921 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T08:06:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T08:06:30.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:30.925 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T08:06:30.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T08:06:30.933+0000 7f46c058b380 -1 init, newargv = 0x557990581a30 newargc=15 2023-12-19T08:06:30.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[55743]: starting ceph client 2023-12-19T08:06:30.947 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[55743]: starting fuse 2023-12-19T08:06:30.951 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T08:06:30.951 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:06:30.951 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T08:06:30.952 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T08:06:30.953 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T08:06:30.953 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T08:06:31.104 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.55743.asok 2023-12-19T08:06:31.291 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55743.asok 2023-12-19T08:06:31.291 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:31.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55743.asok status 2023-12-19T08:06:31.475 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T08:06:31.475 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T08:06:31.475 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T08:06:31.475 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "55743", 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 8556, 2023-12-19T08:06:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 8556 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3019632204 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T08:06:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 3019632204 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.8556 192.168.0.1:0/3019632204", 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/3019632204", 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 63, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 144, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T08:06:31.478 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T08:06:31.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:31.495 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T08:06:31.496 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:31.512 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T08:06:31.512 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:31.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:31.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T08:06:31.595 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T08:06:31.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 Processor -- start 2023-12-19T08:06:31.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 -- start start 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53cc156ad0 0x7f53cc0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53cc0b37a0 0x7f53cc0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0b3ca0 con 0x7f53cc156ea0 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53cc15b870 con 0x7f53cc0b37a0 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.809+0000 7f53d0878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53cc06d300 con 0x7f53cc156ad0 2023-12-19T08:06:31.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c9d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f53cc156ea0 0x7f53cc0cc720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48628/0 (socket says 172.21.15.111:48628) 2023-12-19T08:06:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c9d9b700 1 -- 172.21.15.111:0/2283011713 learned_addr learned my addr 172.21.15.111:0/2283011713 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 --2- 172.21.15.111:0/2283011713 >> v2:172.21.15.111:3300/0 conn(0x7f53cc0b37a0 0x7f53cc0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53ca59c700 1 --2- 172.21.15.111:0/2283011713 >> v2:172.21.15.143:3300/0 conn(0x7f53cc156ad0 0x7f53cc0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c959a700 1 -- 172.21.15.111:0/2283011713 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184263031 0 0) 0x7f53cc0b3ca0 con 0x7f53cc156ea0 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c959a700 1 -- 172.21.15.111:0/2283011713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8003570 con 0x7f53cc156ea0 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 -- 172.21.15.111:0/2283011713 >> v2:172.21.15.143:3300/0 conn(0x7f53cc156ad0 msgr2=0x7f53cc0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 --2- 172.21.15.111:0/2283011713 >> v2:172.21.15.143:3300/0 conn(0x7f53cc156ad0 0x7f53cc0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 -- 172.21.15.111:0/2283011713 >> v1:172.21.15.111:6789/0 conn(0x7f53cc156ea0 legacy=0x7f53cc0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 -- 172.21.15.111:0/2283011713 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc06d5e0 con 0x7f53cc0b37a0 2023-12-19T08:06:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53cad9d700 1 --2- 172.21.15.111:0/2283011713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc0b15e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0ceeb0 tx=0x7f53c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6858823d743cd18e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c959a700 1 -- 172.21.15.111:0/2283011713 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c400e070 con 0x7f53cc0b37a0 2023-12-19T08:06:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c959a700 1 -- 172.21.15.111:0/2283011713 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c400bab0 con 0x7f53cc0b37a0 2023-12-19T08:06:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53c959a700 1 -- 172.21.15.111:0/2283011713 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c400be70 con 0x7f53cc0b37a0 2023-12-19T08:06:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 -- 172.21.15.111:0/2283011713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 msgr2=0x7f53cc0b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 --2- 172.21.15.111:0/2283011713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc0b15e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0ceeb0 tx=0x7f53c4009780 comp rx=0 tx=0).stop 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 -- 172.21.15.111:0/2283011713 shutdown_connections 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 --2- 172.21.15.111:0/2283011713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc0b15e0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 --2- 172.21.15.111:0/2283011713 >> v2:172.21.15.143:3300/0 conn(0x7f53cc156ad0 0x7f53cc0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 -- 172.21.15.111:0/2283011713 >> 172.21.15.111:0/2283011713 conn(0x7f53cc0cdda0 msgr2=0x7f53cc0ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 -- 172.21.15.111:0/2283011713 shutdown_connections 2023-12-19T08:06:31.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.813+0000 7f53d0878700 1 -- 172.21.15.111:0/2283011713 wait complete. 2023-12-19T08:06:31.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 Processor -- start 2023-12-19T08:06:31.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 -- start start 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 0x7f53cc0c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 0x7f53cc0c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc16c0a0 con 0x7f53cc0b37a0 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc0c3040 con 0x7f53cc156ea0 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53d0878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc0c3320 con 0x7f53cc156ad0 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59030/0 (socket says 172.21.15.111:59030) 2023-12-19T08:06:31.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 -- 172.21.15.111:0/3623153831 learned_addr learned my addr 172.21.15.111:0/3623153831 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53cad9d700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 0x7f53cc0c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 -- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 msgr2=0x7f53cc0c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53c9d9b700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 0x7f53cc0c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 0x7f53cc0c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 -- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 msgr2=0x7f53cc0c41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 0x7f53cc0c41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53ca59c700 1 -- 172.21.15.111:0/3623153831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4009a20 con 0x7f53cc0b37a0 2023-12-19T08:06:31.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53cad9d700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 0x7f53cc0c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.817+0000 7f53c9d9b700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 0x7f53cc0c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53ca59c700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00ec80 tx=0x7f53bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc00c9d0 con 0x7f53cc0b37a0 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53bc00cb30 con 0x7f53cc0b37a0 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc013470 con 0x7f53cc0b37a0 2023-12-19T08:06:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc0c3600 con 0x7f53cc0b37a0 2023-12-19T08:06:31.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc15da90 con 0x7f53cc0b37a0 2023-12-19T08:06:31.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc071120 con 0x7f53cc0b37a0 2023-12-19T08:06:31.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.821+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f53bc013820 con 0x7f53cc0b37a0 2023-12-19T08:06:31.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.825+0000 7f53c27fc700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 0x7f53ac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:31.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.825+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f53bc07d0f0 con 0x7f53cc0b37a0 2023-12-19T08:06:31.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.825+0000 7f53c9d9b700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 0x7f53ac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:31.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.825+0000 7f53c9d9b700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 0x7f53ac062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0afab0 tx=0x7f53b4009040 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:31.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:31.829+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f53bc04d1d0 con 0x7f53cc0b37a0 2023-12-19T08:06:32.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.025+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f53cc06d8c0 con 0x7f53ac060670 2023-12-19T08:06:32.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53c27fc700 1 -- 172.21.15.111:0/3623153831 <== mgr.7981 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f53cc06d8c0 con 0x7f53ac060670 2023-12-19T08:06:32.033 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T08:06:32.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T08:06:32.033 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T08:06:32.033 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T08:06:32.033 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 msgr2=0x7f53ac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 0x7f53ac062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0afab0 tx=0x7f53b4009040 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 msgr2=0x7f53cc15d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00ec80 tx=0x7f53bc0091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 shutdown_connections 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f53ac060670 0x7f53ac062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc156ea0 0x7f53cc0c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc156ad0 0x7f53cc0c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 --2- 172.21.15.111:0/3623153831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53cc0b37a0 0x7f53cc15d1c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 >> 172.21.15.111:0/3623153831 conn(0x7f53cc0cdda0 msgr2=0x7f53cc153850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 shutdown_connections 2023-12-19T08:06:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.029+0000 7f53d0878700 1 -- 172.21.15.111:0/3623153831 wait complete. 2023-12-19T08:06:32.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T08:06:32.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.209+0000 7f4b6a745700 1 Processor -- start 2023-12-19T08:06:32.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.209+0000 7f4b6a745700 1 -- start start 2023-12-19T08:06:32.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b6a745700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b64158e20 0x7f4b640ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b6a745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b6a745700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640cbdf0 con 0x7f4b640cb970 2023-12-19T08:06:32.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b6a745700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b640cc0d0 con 0x7f4b64158e20 2023-12-19T08:06:32.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b6a745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640cc3b0 con 0x7f4b640ca2c0 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b5bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4b640cb970 0x7f4b6406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48644/0 (socket says 172.21.15.111:48644) 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b5bfff700 1 -- 172.21.15.111:0/151038972 learned_addr learned my addr 172.21.15.111:0/151038972 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b63fff700 1 --2- 172.21.15.111:0/151038972 >> v2:172.21.15.143:3300/0 conn(0x7f4b64158e20 0x7f4b640ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.213+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/151038972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888052534 0 0) 0x7f4b640cbdf0 con 0x7f4b640cb970 2023-12-19T08:06:32.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b50003570 con 0x7f4b640cb970 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b68ce2700 1 -- 172.21.15.111:0/151038972 >> v2:172.21.15.143:3300/0 conn(0x7f4b64158e20 msgr2=0x7f4b640ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813972798 0 0) 0x7f4b50003570 con 0x7f4b640cb970 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/151038972 >> v2:172.21.15.143:3300/0 conn(0x7f4b64158e20 0x7f4b640ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b68ce2700 1 -- 172.21.15.111:0/151038972 >> v1:172.21.15.111:6789/0 conn(0x7f4b640cb970 legacy=0x7f4b6406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b68ce2700 1 -- 172.21.15.111:0/151038972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640cc690 con 0x7f4b640ca2c0 2023-12-19T08:06:32.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/151038972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640ceb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009e90 tx=0x7f4b5c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38488e0865a824fd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00e070 con 0x7f4b640ca2c0 2023-12-19T08:06:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c00b870 con 0x7f4b640ca2c0 2023-12-19T08:06:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b637fe700 1 -- 172.21.15.111:0/151038972 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c001470 con 0x7f4b640ca2c0 2023-12-19T08:06:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 -- 172.21.15.111:0/151038972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 msgr2=0x7f4b640ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 --2- 172.21.15.111:0/151038972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640ceb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009e90 tx=0x7f4b5c009780 comp rx=0 tx=0).stop 2023-12-19T08:06:32.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 -- 172.21.15.111:0/151038972 shutdown_connections 2023-12-19T08:06:32.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 --2- 172.21.15.111:0/151038972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640ceb00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 --2- 172.21.15.111:0/151038972 >> v2:172.21.15.143:3300/0 conn(0x7f4b64158e20 0x7f4b640ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 -- 172.21.15.111:0/151038972 >> 172.21.15.111:0/151038972 conn(0x7f4b64067bb0 msgr2=0x7f4b640d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:32.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 -- 172.21.15.111:0/151038972 shutdown_connections 2023-12-19T08:06:32.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.217+0000 7f4b6a745700 1 -- 172.21.15.111:0/151038972 wait complete. 2023-12-19T08:06:32.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 Processor -- start 2023-12-19T08:06:32.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 -- start start 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b64158e20 0x7f4b640b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640b5950 con 0x7f4b6414fe90 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640b4da0 con 0x7f4b64158e20 2023-12-19T08:06:32.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b6a745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640b43d0 con 0x7f4b640ca2c0 2023-12-19T08:06:32.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b68ce2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b68ce2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59036/0 (socket says 172.21.15.111:59036) 2023-12-19T08:06:32.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b68ce2700 1 -- 172.21.15.111:0/910803436 learned_addr learned my addr 172.21.15.111:0/910803436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:32.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.221+0000 7f4b63fff700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b5bfff700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b64158e20 0x7f4b640b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 -- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 msgr2=0x7f4b640b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 -- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b64158e20 msgr2=0x7f4b640b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b64158e20 0x7f4b640b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 -- 172.21.15.111:0/910803436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c009a20 con 0x7f4b6414fe90 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b68ce2700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0099f0 tx=0x7f4b5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c012c20 con 0x7f4b6414fe90 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c012dc0 con 0x7f4b6414fe90 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c01e6a0 con 0x7f4b6414fe90 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b640b3550 con 0x7f4b6414fe90 2023-12-19T08:06:32.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b640b3d80 con 0x7f4b6414fe90 2023-12-19T08:06:32.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.225+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b640b4560 con 0x7f4b6414fe90 2023-12-19T08:06:32.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.229+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 59) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4b5c01ea90 con 0x7f4b6414fe90 2023-12-19T08:06:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.229+0000 7f4b60ff9700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 0x7f4b44062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.229+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f4b5c07ef60 con 0x7f4b6414fe90 2023-12-19T08:06:32.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.233+0000 7f4b63fff700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 0x7f4b44062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.233+0000 7f4b63fff700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 0x7f4b44062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009d10 tx=0x7f4b540091a0 comp rx=0 tx=0).ready entity=mgr.7981 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:32.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.237+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4b5c0529c0 con 0x7f4b6414fe90 2023-12-19T08:06:32.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.445+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4b6406a7a0 con 0x7f4b6414fe90 2023-12-19T08:06:32.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.593+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 60) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f4b5c04f300 con 0x7f4b6414fe90 2023-12-19T08:06:32.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.593+0000 7f4b60ff9700 1 -- 172.21.15.111:0/910803436 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v60) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4b5c04c4a0 con 0x7f4b6414fe90 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 msgr2=0x7f4b44062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 0x7f4b44062b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009d10 tx=0x7f4b540091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 msgr2=0x7f4b6414ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0099f0 tx=0x7f4b5c017000 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 shutdown_connections 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4b440606a0 0x7f4b44062b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b64158e20 0x7f4b640b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b6414fe90 0x7f4b6414ab00 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 --2- 172.21.15.111:0/910803436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b640ca2c0 0x7f4b640b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 >> 172.21.15.111:0/910803436 conn(0x7f4b64067bb0 msgr2=0x7f4b64155ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:32.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 shutdown_connections 2023-12-19T08:06:32.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.597+0000 7f4b6a745700 1 -- 172.21.15.111:0/910803436 wait complete. 2023-12-19T08:06:32.621 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T08:06:32.783 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:32.777+0000 7f36de6fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:06:32.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.781+0000 7f82e9af0700 1 Processor -- start 2023-12-19T08:06:32.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.781+0000 7f82e9af0700 1 -- start start 2023-12-19T08:06:32.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e9af0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82e4069a20 0x7f82e406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e9af0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40ab6c0 con 0x7f82e41591e0 2023-12-19T08:06:32.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e9af0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40ac240 con 0x7f82e406cfa0 2023-12-19T08:06:32.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e9af0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40ab910 con 0x7f82e4069a20 2023-12-19T08:06:32.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82e4069a20 0x7f82e406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82e4069a20 0x7f82e406d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59058/0 (socket says 172.21.15.111:59058) 2023-12-19T08:06:32.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e3fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f82e41591e0 0x7f82e40adac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48664/0 (socket says 172.21.15.111:48664) 2023-12-19T08:06:32.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 -- 172.21.15.111:0/2216997844 learned_addr learned my addr 172.21.15.111:0/2216997844 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101436516 0 0) 0x7f82e40ac240 con 0x7f82e406cfa0 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82cc003570 con 0x7f82e406cfa0 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 -- 172.21.15.111:0/2216997844 >> v1:172.21.15.143:6789/0 conn(0x7f82e406cfa0 legacy=0x7f82e4158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325663650 0 0) 0x7f82e40ab6c0 con 0x7f82e41591e0 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 -- 172.21.15.111:0/2216997844 >> v1:172.21.15.111:6789/0 conn(0x7f82e41591e0 legacy=0x7f82e40adac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 -- 172.21.15.111:0/2216997844 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e40abbf0 con 0x7f82e4069a20 2023-12-19T08:06:32.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.785+0000 7f82e2ffd700 1 --2- 172.21.15.111:0/2216997844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4069a20 0x7f82e406d370 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f82d80099b0 tx=0x7f82d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4454ae9c95d5db6d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:32.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8009ea0 con 0x7f82e4069a20 2023-12-19T08:06:32.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d800b860 con 0x7f82e4069a20 2023-12-19T08:06:32.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e27fc700 1 -- 172.21.15.111:0/2216997844 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800bc00 con 0x7f82e4069a20 2023-12-19T08:06:32.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 -- 172.21.15.111:0/2216997844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4069a20 msgr2=0x7f82e406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 --2- 172.21.15.111:0/2216997844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4069a20 0x7f82e406d370 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f82d80099b0 tx=0x7f82d8009250 comp rx=0 tx=0).stop 2023-12-19T08:06:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 -- 172.21.15.111:0/2216997844 shutdown_connections 2023-12-19T08:06:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 --2- 172.21.15.111:0/2216997844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4069a20 0x7f82e406d370 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 -- 172.21.15.111:0/2216997844 >> 172.21.15.111:0/2216997844 conn(0x7f82e40cd870 msgr2=0x7f82e4069310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 -- 172.21.15.111:0/2216997844 shutdown_connections 2023-12-19T08:06:32.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.789+0000 7f82e9af0700 1 -- 172.21.15.111:0/2216997844 wait complete. 2023-12-19T08:06:32.796 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:32.796+0000 7f98d4b61000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T08:06:32.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 Processor -- start 2023-12-19T08:06:32.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- start start 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e4069a20 0x7f82e40c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e406cfa0 0x7f82e414a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40c7f60 con 0x7f82e41591e0 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40c7350 con 0x7f82e4069a20 2023-12-19T08:06:32.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40c7630 con 0x7f82e406cfa0 2023-12-19T08:06:32.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59062/0 (socket says 172.21.15.111:59062) 2023-12-19T08:06:32.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 -- 172.21.15.111:0/3576120321 learned_addr learned my addr 172.21.15.111:0/3576120321 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e2ffd700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e406cfa0 0x7f82e414a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e37fe700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e4069a20 0x7f82e40c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 -- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e406cfa0 msgr2=0x7f82e414a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e406cfa0 0x7f82e414a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 -- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e4069a20 msgr2=0x7f82e40c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e4069a20 0x7f82e40c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:32.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 -- 172.21.15.111:0/3576120321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d80094f0 con 0x7f82e41591e0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e3fff700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00ccc0 tx=0x7f82dc001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc005df0 con 0x7f82e41591e0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc00a600 con 0x7f82e41591e0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e40c7910 con 0x7f82e41591e0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00a9c0 con 0x7f82e41591e0 2023-12-19T08:06:32.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.793+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e414adc0 con 0x7f82e41591e0 2023-12-19T08:06:32.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.797+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e40c80d0 con 0x7f82e41591e0 2023-12-19T08:06:32.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.797+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 60) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f82dc0070f0 con 0x7f82e41591e0 2023-12-19T08:06:32.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.797+0000 7f82cb7fe700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 0x7f82d0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:32.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.797+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f82dc07cde0 con 0x7f82e41591e0 2023-12-19T08:06:32.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.801+0000 7f82e37fe700 1 -- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 msgr2=0x7f82d0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:32.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.801+0000 7f82e37fe700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 0x7f82d0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:32.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.805+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f82dc04cf40 con 0x7f82e41591e0 2023-12-19T08:06:32.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.981+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f82e4071120 con 0x7f82e41591e0 2023-12-19T08:06:33.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.993+0000 7f82cb7fe700 1 -- 172.21.15.111:0/3576120321 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f82dc050560 con 0x7f82e41591e0 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 msgr2=0x7f82d0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 0x7f82d0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 msgr2=0x7f82e40c6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00ccc0 tx=0x7f82dc001a70 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 shutdown_connections 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f82d00606c0 0x7f82d0062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41591e0 0x7f82e40c6a20 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82e406cfa0 0x7f82e414a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 --2- 172.21.15.111:0/3576120321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82e4069a20 0x7f82e40c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 >> 172.21.15.111:0/3576120321 conn(0x7f82e40cd870 msgr2=0x7f82e40af090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 shutdown_connections 2023-12-19T08:06:33.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:32.997+0000 7f82e9af0700 1 -- 172.21.15.111:0/3576120321 wait complete. 2023-12-19T08:06:33.067 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T08:06:33.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.233+0000 7f51c8904700 1 Processor -- start 2023-12-19T08:06:33.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.233+0000 7f51c8904700 1 -- start start 2023-12-19T08:06:33.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c8904700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f51c406aeb0 0x7f51c4159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c8904700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 0x7f51c41599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c8904700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c406c850 con 0x7f51c4158c90 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c8904700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c40cb650 con 0x7f51c406aeb0 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c8904700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c406b470 con 0x7f51c41595a0 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f51c406aeb0 0x7f51c4159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f51c406aeb0 0x7f51c4159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59076/0 (socket says 172.21.15.111:59076) 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 -- 172.21.15.111:0/3457579292 learned_addr learned my addr 172.21.15.111:0/3457579292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c359e700 1 --2- 172.21.15.111:0/3457579292 >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 0x7f51c41599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 -- 172.21.15.111:0/3457579292 >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 msgr2=0x7f51c41599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 --2- 172.21.15.111:0/3457579292 >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 0x7f51c41599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 -- 172.21.15.111:0/3457579292 >> v1:172.21.15.143:6789/0 conn(0x7f51c4158c90 legacy=0x7f51c40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 -- 172.21.15.111:0/3457579292 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c406b750 con 0x7f51c406aeb0 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c359e700 1 --2- 172.21.15.111:0/3457579292 >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 0x7f51c41599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T08:06:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c259c700 1 --2- 172.21.15.111:0/3457579292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406aeb0 0x7f51c4159060 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f51c40cc7f0 tx=0x7f51b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93865d59669e2b3d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c1d9b700 1 -- 172.21.15.111:0/3457579292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800b860 con 0x7f51c406aeb0 2023-12-19T08:06:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c1d9b700 1 -- 172.21.15.111:0/3457579292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b800b9c0 con 0x7f51c406aeb0 2023-12-19T08:06:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.237+0000 7f51c1d9b700 1 -- 172.21.15.111:0/3457579292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800bd60 con 0x7f51c406aeb0 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 -- 172.21.15.111:0/3457579292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406aeb0 msgr2=0x7f51c4159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 --2- 172.21.15.111:0/3457579292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406aeb0 0x7f51c4159060 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f51c40cc7f0 tx=0x7f51b8009250 comp rx=0 tx=0).stop 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 -- 172.21.15.111:0/3457579292 shutdown_connections 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 --2- 172.21.15.111:0/3457579292 >> v2:172.21.15.143:3300/0 conn(0x7f51c41595a0 0x7f51c41599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 --2- 172.21.15.111:0/3457579292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c406aeb0 0x7f51c4159060 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 -- 172.21.15.111:0/3457579292 >> 172.21.15.111:0/3457579292 conn(0x7f51c4067bb0 msgr2=0x7f51c40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 -- 172.21.15.111:0/3457579292 shutdown_connections 2023-12-19T08:06:33.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.241+0000 7f51c8904700 1 -- 172.21.15.111:0/3457579292 wait complete. 2023-12-19T08:06:33.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 Processor -- start 2023-12-19T08:06:33.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 -- start start 2023-12-19T08:06:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 0x7f51c41690f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c41595a0 0x7f51c40c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40b4620 con 0x7f51c4158c90 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4169630 con 0x7f51c406aeb0 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.245+0000 7f51c8904700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c40b3d30 con 0x7f51c41595a0 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59084/0 (socket says 172.21.15.111:59084) 2023-12-19T08:06:33.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 -- 172.21.15.111:0/1363386537 learned_addr learned my addr 172.21.15.111:0/1363386537 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 -- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c41595a0 msgr2=0x7f51c40c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c41595a0 0x7f51c40c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 -- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 msgr2=0x7f51c41690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c2d9d700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 0x7f51c41690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 0x7f51c41690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 -- 172.21.15.111:0/1363386537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b80094f0 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c2d9d700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 0x7f51c41690f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c259c700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f51b80090b0 tx=0x7f51b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8011e20 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b801f070 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4158140 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b801a890 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c40b3fd0 con 0x7f51c4158c90 2023-12-19T08:06:33.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.249+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c40b47d0 con 0x7f51c4158c90 2023-12-19T08:06:33.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.253+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 60) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f51b801ac00 con 0x7f51c4158c90 2023-12-19T08:06:33.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.253+0000 7f51aaffd700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 0x7f51b0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.253+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f51b8021070 con 0x7f51c4158c90 2023-12-19T08:06:33.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.253+0000 7f51c2d9d700 1 -- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 msgr2=0x7f51b0062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:33.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.253+0000 7f51c2d9d700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 0x7f51b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:33.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.257+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f51c40b47d0 con 0x7f51c4158c90 2023-12-19T08:06:33.344 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:33.341+0000 7f36de6fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:06:33.349 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:33.352+0000 7f98d4b61000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T08:06:33.419 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:33.413+0000 7f36de6fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:06:33.423 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:33.424+0000 7f98d4b61000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T08:06:33.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.425+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f51c40b47d0 con 0x7f51c4158c90 2023-12-19T08:06:33.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.429+0000 7f51aaffd700 1 -- 172.21.15.111:0/1363386537 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f51c40b47d0 con 0x7f51c4158c90 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 msgr2=0x7f51b0062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 0x7f51b0062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 msgr2=0x7f51c4157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f51b80090b0 tx=0x7f51b8017000 comp rx=0 tx=0).stop 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 shutdown_connections 2023-12-19T08:06:33.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f51b00606c0 0x7f51b0062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51c41595a0 0x7f51c40c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c4158c90 0x7f51c4157c00 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 --2- 172.21.15.111:0/1363386537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51c406aeb0 0x7f51c41690f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 >> 172.21.15.111:0/1363386537 conn(0x7f51c4067bb0 msgr2=0x7f51c40afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 shutdown_connections 2023-12-19T08:06:33.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.433+0000 7f51c8904700 1 -- 172.21.15.111:0/1363386537 wait complete. 2023-12-19T08:06:33.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T08:06:33.459 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T08:06:33.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1 1M 2023-12-19T08:06:33.571 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:33.572+0000 7f98d4b61000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:33.575 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:33.569+0000 7f36de6fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.697+0000 7f8a9e20c700 1 Processor -- start 2023-12-19T08:06:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.697+0000 7f8a9e20c700 1 -- start start 2023-12-19T08:06:33.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a980aedc0 0x7f8a980af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a980cac50 0x7f8a980af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9806b3d0 con 0x7f8a9806af50 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980cc140 con 0x7f8a980aedc0 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980cb3f0 con 0x7f8a980cac50 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a980aedc0 0x7f8a980af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9ca09700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8a9806af50 0x7f8a9806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:48676/0 (socket says 172.21.15.111:48676) 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a980aedc0 0x7f8a980af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59098/0 (socket says 172.21.15.111:59098) 2023-12-19T08:06:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9ca09700 1 -- 172.21.15.111:0/2271011420 learned_addr learned my addr 172.21.15.111:0/2271011420 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T08:06:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9da0b700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a980cac50 0x7f8a980af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 -- 172.21.15.111:0/2271011420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a980cac50 msgr2=0x7f8a980af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a980cac50 0x7f8a980af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 -- 172.21.15.111:0/2271011420 >> v1:172.21.15.111:6789/0 conn(0x7f8a9806af50 legacy=0x7f8a9806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a8ffff700 1 -- 172.21.15.111:0/2271011420 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227673481 0 0) 0x7f8a88001110 con 0x7f8a9806af50 2023-12-19T08:06:33.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 -- 172.21.15.111:0/2271011420 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9806d220 con 0x7f8a980aedc0 2023-12-19T08:06:33.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980aedc0 0x7f8a980af190 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8a940098e0 tx=0x7f8a940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272105f33597bcad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a8ffff700 1 -- 172.21.15.111:0/2271011420 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400b640 con 0x7f8a980aedc0 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a8ffff700 1 -- 172.21.15.111:0/2271011420 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400b7a0 con 0x7f8a980aedc0 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a8ffff700 1 -- 172.21.15.111:0/2271011420 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400bb40 con 0x7f8a980aedc0 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- 172.21.15.111:0/2271011420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980aedc0 msgr2=0x7f8a980af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980aedc0 0x7f8a980af190 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8a940098e0 tx=0x7f8a940091a0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- 172.21.15.111:0/2271011420 shutdown_connections 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a980cac50 0x7f8a980af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/2271011420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980aedc0 0x7f8a980af190 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.701+0000 7f8a9e20c700 1 -- 172.21.15.111:0/2271011420 >> 172.21.15.111:0/2271011420 conn(0x7f8a980cc760 msgr2=0x7f8a980ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- 172.21.15.111:0/2271011420 shutdown_connections 2023-12-19T08:06:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- 172.21.15.111:0/2271011420 wait complete. 2023-12-19T08:06:33.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 Processor -- start 2023-12-19T08:06:33.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- start start 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a9806af50 0x7f8a9814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a980aedc0 0x7f8a9815bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980c3e40 con 0x7f8a980cac50 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9814c670 con 0x7f8a980aedc0 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9e20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b5710 con 0x7f8a9806af50 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9da0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9da0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59106/0 (socket says 172.21.15.111:59106) 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9da0b700 1 -- 172.21.15.111:0/3115749579 learned_addr learned my addr 172.21.15.111:0/3115749579 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a9806af50 0x7f8a9814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.705+0000 7f8a9ca09700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a980aedc0 0x7f8a9815bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a9806af50 msgr2=0x7f8a9814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a9806af50 0x7f8a9814c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a980aedc0 msgr2=0x7f8a9815bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a980aedc0 0x7f8a9815bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009420 con 0x7f8a980cac50 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9da0b700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e90 tx=0x7f8a9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000cc20 con 0x7f8a980cac50 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9000f040 con 0x7f8a980cac50 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90014480 con 0x7f8a980cac50 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a980b59f0 con 0x7f8a980cac50 2023-12-19T08:06:33.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a980c3ad0 con 0x7f8a980cac50 2023-12-19T08:06:33.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9806a420 con 0x7f8a980cac50 2023-12-19T08:06:33.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.709+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 60) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f8a90014830 con 0x7f8a980cac50 2023-12-19T08:06:33.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.713+0000 7f8a8d7fa700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:33.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.713+0000 7f8a9d20a700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:33.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.713+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f8a9007db40 con 0x7f8a980cac50 2023-12-19T08:06:33.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.713+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T08:06:33.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.717+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a9806a420 con 0x7f8a980cac50 2023-12-19T08:06:33.847 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:33.848+0000 7f98d4b61000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:06:33.852 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:33.849+0000 7f36de6fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T08:06:33.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.913+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a9806a420 con 0x7f8a7c0606c0 2023-12-19T08:06:33.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.917+0000 7f8a9d20a700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:33.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:33.917+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T08:06:33.922 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:33.924+0000 7f98d4b61000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:33.926 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:33.921+0000 7f36de6fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:34.145 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:34.148+0000 7f98d4b61000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:06:34.151 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:34.145+0000 7f36de6fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T08:06:34.231 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:34.232+0000 7f98d4b61000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:06:34.233 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:34.229+0000 7f36de6fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T08:06:34.322 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:34.317+0000 7f36de6fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:06:34.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:34.321+0000 7f8a9d20a700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:34.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:34.321+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T08:06:34.325 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:34.328+0000 7f98d4b61000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T08:06:34.725 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:34.721+0000 7f36de6fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:06:34.736 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:34.736+0000 7f98d4b61000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T08:06:34.892 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:34.892+0000 7f98d4b61000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:06:34.899 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:34.893+0000 7f36de6fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T08:06:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:35.125+0000 7f8a9d20a700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:35.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:35.125+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T08:06:35.264 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.264+0000 7f98d4b61000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:06:35.276 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:35.273+0000 7f36de6fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T08:06:35.483 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.484+0000 7f98d4b61000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:35.504 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:35.501+0000 7f36de6fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T08:06:35.576 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.576+0000 7f98d4b61000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:06:35.598 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:35.593+0000 7f36de6fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T08:06:35.654 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.656+0000 7f98d4b61000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:06:35.674 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:35.669+0000 7f36de6fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T08:06:35.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.732+0000 7f98d4b61000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:06:35.760 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:35.757+0000 7f36de6fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T08:06:35.947 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:35.948+0000 7f98d4b61000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:36.004 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:36.001+0000 7f36de6fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T08:06:36.029 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:36.032+0000 7f98d4b61000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:06:36.088 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:36.085+0000 7f36de6fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T08:06:36.112 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:36.112+0000 7f98d4b61000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:06:36.181 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:36.177+0000 7f36de6fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T08:06:36.298 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:36.300+0000 7f98d4b61000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:06:36.381 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:36.377+0000 7f36de6fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T08:06:36.382 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:36.384+0000 7f98d4b61000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:06:36.476 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:36.469+0000 7f36de6fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T08:06:36.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:36.729+0000 7f8a9d20a700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T08:06:36.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:36.729+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T08:06:36.959 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:06:36.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:06:37.092 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.092+0000 7f98d4b61000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:06:37.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:37.105+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 61) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f8a9007bcf0 con 0x7f8a980cac50 2023-12-19T08:06:37.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:37.105+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T08:06:37.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:37.105+0000 7f8a8d7fa700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:37.216 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T08:06:37.213+0000 7f36de6fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T08:06:37.289 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.288+0000 7f98966bd700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.289 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.288+0000 7f98966bd700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.289 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.288+0000 7f98966bd700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.289 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.288+0000 7f98966bd700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.289 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.288+0000 7f98966bd700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.292+0000 7f98936b7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.292+0000 7f98936b7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.292+0000 7f98936b7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.292+0000 7f98936b7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:37.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:06:37.292+0000 7f98936b7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:06:38.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.117+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 62) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f8a9004df80 con 0x7f8a980cac50 2023-12-19T08:06:38.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.117+0000 7f8a8d7fa700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.117+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a9806a420 con 0x7f8a7c0606c0 2023-12-19T08:06:38.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.121+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.129+0000 7f8a9d20a700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a940093f0 tx=0x7f8a94017000 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.225+0000 7f8a8d7fa700 1 -- 172.21.15.111:0/3115749579 <== mgr.7409 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f8a9806a420 con 0x7f8a7c0606c0 2023-12-19T08:06:38.230 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1 2023-12-19T08:06:38.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 msgr2=0x7f8a7c062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8a940093f0 tx=0x7f8a94017000 comp rx=0 tx=0).stop 2023-12-19T08:06:38.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 msgr2=0x7f8a9815c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009e90 tx=0x7f8a9000b040 comp rx=0 tx=0).stop 2023-12-19T08:06:38.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 shutdown_connections 2023-12-19T08:06:38.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8a7c0606c0 0x7f8a7c062af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a980aedc0 0x7f8a9815bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a980cac50 0x7f8a9815c310 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 --2- 172.21.15.111:0/3115749579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a9806af50 0x7f8a9814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 >> 172.21.15.111:0/3115749579 conn(0x7f8a980cc760 msgr2=0x7f8a9815e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 shutdown_connections 2023-12-19T08:06:38.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.229+0000 7f8a9e20c700 1 -- 172.21.15.111:0/3115749579 wait complete. 2023-12-19T08:06:38.262 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-12-19T08:06:38.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs /snap_test_dir1 55n 2023-12-19T08:06:38.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 Processor -- start 2023-12-19T08:06:38.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 -- start start 2023-12-19T08:06:38.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb848158c90 0x7fb8480caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb84806aeb0 0x7fb848159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84806c850 con 0x7fb8481595a0 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84806b470 con 0x7fb84806aeb0 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb84d84c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84806b750 con 0x7fb848158c90 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb83ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb84806aeb0 0x7fb848159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb83ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb84806aeb0 0x7fb848159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59108/0 (socket says 172.21.15.111:59108) 2023-12-19T08:06:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.425+0000 7fb83ffff700 1 -- 172.21.15.111:0/747883067 learned_addr learned my addr 172.21.15.111:0/747883067 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84c84a700 1 --2- 172.21.15.111:0/747883067 >> v2:172.21.15.143:3300/0 conn(0x7fb848158c90 0x7fb8480caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83ffff700 1 -- 172.21.15.111:0/747883067 >> v2:172.21.15.143:3300/0 conn(0x7fb848158c90 msgr2=0x7fb8480caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83ffff700 1 --2- 172.21.15.111:0/747883067 >> v2:172.21.15.143:3300/0 conn(0x7fb848158c90 0x7fb8480caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83ffff700 1 -- 172.21.15.111:0/747883067 >> v1:172.21.15.143:6789/0 conn(0x7fb8481595a0 legacy=0x7fb84815c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83ffff700 1 -- 172.21.15.111:0/747883067 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84806ba30 con 0x7fb84806aeb0 2023-12-19T08:06:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83ffff700 1 --2- 172.21.15.111:0/747883067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848159060 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb8480698f0 tx=0x7fb838009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16a1a31741557ec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83f7fe700 1 -- 172.21.15.111:0/747883067 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83800b860 con 0x7fb84806aeb0 2023-12-19T08:06:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83f7fe700 1 -- 172.21.15.111:0/747883067 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83800b9c0 con 0x7fb84806aeb0 2023-12-19T08:06:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb83f7fe700 1 -- 172.21.15.111:0/747883067 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83800bd60 con 0x7fb84806aeb0 2023-12-19T08:06:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 -- 172.21.15.111:0/747883067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 msgr2=0x7fb848159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 --2- 172.21.15.111:0/747883067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848159060 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb8480698f0 tx=0x7fb838009250 comp rx=0 tx=0).stop 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 -- 172.21.15.111:0/747883067 shutdown_connections 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 --2- 172.21.15.111:0/747883067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848159060 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 --2- 172.21.15.111:0/747883067 >> v2:172.21.15.143:3300/0 conn(0x7fb848158c90 0x7fb8480caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 -- 172.21.15.111:0/747883067 >> 172.21.15.111:0/747883067 conn(0x7fb848067bb0 msgr2=0x7fb8480cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 -- 172.21.15.111:0/747883067 shutdown_connections 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 -- 172.21.15.111:0/747883067 wait complete. 2023-12-19T08:06:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.429+0000 7fb84d84c700 1 Processor -- start 2023-12-19T08:06:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- start start 2023-12-19T08:06:38.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb848158c90 0x7fb8480c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8481595a0 0x7fb848156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480c8220 con 0x7fb84806aeb0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb848155b30 con 0x7fb8481595a0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb848155d80 con 0x7fb848158c90 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59124/0 (socket says 172.21.15.111:59124) 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 -- 172.21.15.111:0/3287830758 learned_addr learned my addr 172.21.15.111:0/3287830758 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb83ffff700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb848158c90 0x7fb8480c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d04b700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8481595a0 0x7fb848156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 -- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb848158c90 msgr2=0x7fb8480c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb848158c90 0x7fb8480c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 -- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8481595a0 msgr2=0x7fb848156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8481595a0 0x7fb848156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 -- 172.21.15.111:0/3287830758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8380094f0 con 0x7fb84806aeb0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84c84a700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb84400e190 tx=0x7fb844009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84400ee30 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844014420 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb848156060 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8440147e0 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.433+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb848154190 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.437+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb848156250 con 0x7fb84806aeb0 2023-12-19T08:06:38.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.437+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 62) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fb844014b90 con 0x7fb84806aeb0 2023-12-19T08:06:38.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.437+0000 7fb83cff9700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 0x7fb8340628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.437+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fb84407d270 con 0x7fb84806aeb0 2023-12-19T08:06:38.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.437+0000 7fb83ffff700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 0x7fb8340628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.441+0000 7fb83ffff700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 0x7fb8340628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb8480698f0 tx=0x7fb838017000 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.441+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb84404d580 con 0x7fb84806aeb0 2023-12-19T08:06:38.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.633+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7fb848156450 con 0x7fb8340603f0 2023-12-19T08:06:38.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.733+0000 7fb83cff9700 1 -- 172.21.15.111:0/3287830758 <== mgr.7409 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7fb848156450 con 0x7fb8340603f0 2023-12-19T08:06:38.739 INFO:teuthology.orchestra.run.smithi111.stdout:Retention added to path /snap_test_dir1 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 msgr2=0x7fb8340628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 0x7fb8340628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb8480698f0 tx=0x7fb838017000 comp rx=0 tx=0).stop 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 msgr2=0x7fb848153c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb84400e190 tx=0x7fb844009920 comp rx=0 tx=0).stop 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 shutdown_connections 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8340603f0 0x7fb8340628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8481595a0 0x7fb848156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb848158c90 0x7fb8480c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 --2- 172.21.15.111:0/3287830758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb84806aeb0 0x7fb848153c50 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 >> 172.21.15.111:0/3287830758 conn(0x7fb848067bb0 msgr2=0x7fb84806a660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 shutdown_connections 2023-12-19T08:06:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.737+0000 7fb84d84c700 1 -- 172.21.15.111:0/3287830758 wait complete. 2023-12-19T08:06:38.767 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-12-19T08:06:38.768 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [] 2023-12-19T08:06:38.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json /snap_test_dir1 2023-12-19T08:06:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.937+0000 7f16364f0700 1 Processor -- start 2023-12-19T08:06:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.937+0000 7f16364f0700 1 -- start start 2023-12-19T08:06:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16364f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300cb130 0x7f16300cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16364f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f163006a6a0 0x7f163006aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16364f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16300b0ac0 con 0x7f163015c220 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16364f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163006b030 con 0x7f163006a6a0 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16364f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163006b210 con 0x7f16300cb130 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f163006a6a0 0x7f163006aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f163006a6a0 0x7f163006aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59130/0 (socket says 172.21.15.111:59130) 2023-12-19T08:06:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 -- 172.21.15.111:0/3843462087 learned_addr learned my addr 172.21.15.111:0/3843462087 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f162ffff700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300cb130 0x7f16300cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 -- 172.21.15.111:0/3843462087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300cb130 msgr2=0x7f16300cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300cb130 0x7f16300cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 -- 172.21.15.111:0/3843462087 >> v1:172.21.15.143:6789/0 conn(0x7f163015c220 legacy=0x7f163006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 -- 172.21.15.111:0/3843462087 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300ca810 con 0x7f163006a6a0 2023-12-19T08:06:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.941+0000 7f16277fe700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163006aaf0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f16180099b0 tx=0x7f1618009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed581eda33463e9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f162f7fe700 1 -- 172.21.15.111:0/3843462087 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161800b820 con 0x7f163006a6a0 2023-12-19T08:06:38.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f162f7fe700 1 -- 172.21.15.111:0/3843462087 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161800b980 con 0x7f163006a6a0 2023-12-19T08:06:38.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f162f7fe700 1 -- 172.21.15.111:0/3843462087 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161800bd20 con 0x7f163006a6a0 2023-12-19T08:06:38.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 -- 172.21.15.111:0/3843462087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 msgr2=0x7f163006aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163006aaf0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f16180099b0 tx=0x7f1618009250 comp rx=0 tx=0).stop 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 -- 172.21.15.111:0/3843462087 shutdown_connections 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163006aaf0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 --2- 172.21.15.111:0/3843462087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300cb130 0x7f16300cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 -- 172.21.15.111:0/3843462087 >> 172.21.15.111:0/3843462087 conn(0x7f1630067bb0 msgr2=0x7f16300cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 -- 172.21.15.111:0/3843462087 shutdown_connections 2023-12-19T08:06:38.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.945+0000 7f16364f0700 1 -- 172.21.15.111:0/3843462087 wait complete. 2023-12-19T08:06:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 Processor -- start 2023-12-19T08:06:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 -- start start 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16300cb130 0x7f16300c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163015c220 0x7f1630148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630148ba0 con 0x7f163006a6a0 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630148de0 con 0x7f16300cb130 2023-12-19T08:06:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f16364f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16301490c0 con 0x7f163015c220 2023-12-19T08:06:38.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f162ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f162ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59146/0 (socket says 172.21.15.111:59146) 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.949+0000 7f162ffff700 1 -- 172.21.15.111:0/1126926329 learned_addr learned my addr 172.21.15.111:0/1126926329 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f16277fe700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16300cb130 0x7f16300c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f1634a8d700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163015c220 0x7f1630148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 -- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163015c220 msgr2=0x7f1630148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163015c220 0x7f1630148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 -- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16300cb130 msgr2=0x7f16300c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16300cb130 0x7f16300c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 -- 172.21.15.111:0/1126926329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180094f0 con 0x7f163006a6a0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162ffff700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f162000d890 tx=0x7f1620008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16200125f0 con 0x7f163006a6a0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1620012770 con 0x7f163006a6a0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16301493a0 con 0x7f163006a6a0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620012b30 con 0x7f163006a6a0 2023-12-19T08:06:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630149960 con 0x7f163006a6a0 2023-12-19T08:06:38.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.953+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1630071120 con 0x7f163006a6a0 2023-12-19T08:06:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.957+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 62) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f162001e460 con 0x7f163006a6a0 2023-12-19T08:06:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.957+0000 7f162cff9700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 0x7f16100628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T08:06:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.957+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f162007d0d0 con 0x7f163006a6a0 2023-12-19T08:06:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.957+0000 7f16277fe700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 0x7f16100628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T08:06:38.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.961+0000 7f16277fe700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 0x7f16100628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f16180099b0 tx=0x7f1618016000 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T08:06:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:38.965+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f162004d3e0 con 0x7f163006a6a0 2023-12-19T08:06:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.193+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16300cb500 con 0x7f16100603f0 2023-12-19T08:06:39.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.209+0000 7f162cff9700 1 -- 172.21.15.111:0/1126926329 <== mgr.7409 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f16300cb500 con 0x7f16100603f0 2023-12-19T08:06:39.215 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T08:06:39.215 INFO:teuthology.orchestra.run.smithi111.stdout:{"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-12-19T08:06:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 msgr2=0x7f16100628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 0x7f16100628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f16180099b0 tx=0x7f1618016000 comp rx=0 tx=0).stop 2023-12-19T08:06:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 msgr2=0x7f163014b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f162000d890 tx=0x7f1620008040 comp rx=0 tx=0).stop 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 shutdown_connections 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f16100603f0 0x7f16100628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16300cb130 0x7f16300c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163015c220 0x7f1630148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 --2- 172.21.15.111:0/1126926329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f163006a6a0 0x7f163014b180 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 >> 172.21.15.111:0/1126926329 conn(0x7f1630067bb0 msgr2=0x7f16300b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 shutdown_connections 2023-12-19T08:06:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T08:06:39.213+0000 7f16364f0700 1 -- 172.21.15.111:0/1126926329 wait complete. 2023-12-19T08:06:39.242 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1M"], "retention": [{"n": 55}]} 2023-12-19T08:06:39.243 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1M'], 'retention': [{'n': 55}]} 2023-12-19T08:07:00.760 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:07:00.759+0000 7f988569b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:07:00.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:07:00.759+0000 7f988569b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:07:00.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:07:00.759+0000 7f988569b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:07:00.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:07:00.759+0000 7f988569b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:07:00.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T08:07:00.759+0000 7f988569b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T08:07:07.040 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:07:07.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:07:37.122 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:07:37.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:08:07.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:08:07.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:08:37.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:08:37.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:09:07.375 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:09:07.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:09:37.468 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:09:37.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:10:07.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:10:07.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:10:37.648 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:10:37.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:11:07.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:11:07.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:11:37.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:11:37.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:12:07.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:12:07.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:12:37.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:12:38.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:13:08.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:13:08.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:13:38.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:13:38.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:14:08.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:14:08.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:14:38.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:14:38.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:15:08.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:15:08.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:15:38.677 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:15:38.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:16:08.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:16:08.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:16:38.922 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:16:38.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:17:08.998 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:17:09.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:17:39.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:17:39.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:18:09.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:18:09.173 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:18:39.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:18:39.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:19:09.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:19:09.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:19:39.492 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:19:39.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:20:09.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:20:09.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:20:39.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:20:39.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:21:09.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:21:09.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:21:40.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:21:40.051 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:22:10.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:22:10.198 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:22:40.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:22:40.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:23:10.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:23:10.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:23:40.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:23:40.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:24:10.806 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:24:10.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:24:40.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:24:40.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:25:11.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:25:11.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:25:41.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:25:41.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:26:11.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:26:11.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:26:41.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:26:41.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:27:11.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:27:11.811 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:27:41.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:27:41.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:28:12.047 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:28:12.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:28:42.203 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:28:42.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:29:12.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:29:12.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:29:42.504 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:29:42.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:30:12.677 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:30:12.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:30:42.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:30:42.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:31:12.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:31:12.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:31:43.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:31:43.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:32:13.257 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:32:13.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:32:43.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:32:43.424 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:33:13.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:33:13.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:33:43.652 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:33:43.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:34:13.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:34:13.808 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:34:43.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:34:43.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:35:14.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:35:14.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:35:44.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:35:44.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:36:14.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:36:14.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:36:44.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:36:44.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:37:14.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:37:14.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:37:45.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:37:45.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:38:15.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:38:15.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:38:45.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:38:45.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:39:15.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:39:15.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:39:45.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:39:45.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:40:15.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:40:15.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:40:45.886 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:40:45.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:41:16.024 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:41:16.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:41:46.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:41:46.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:42:16.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:42:16.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:42:46.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:42:46.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:43:16.556 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:43:16.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:43:46.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:43:46.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:44:16.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:44:16.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:44:46.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:44:46.955 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:45:17.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:45:17.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:45:47.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:45:47.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:46:17.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:46:17.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:46:47.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:46:47.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:47:17.573 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:47:17.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:47:47.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:47:47.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:48:17.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:48:17.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:48:48.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:48:48.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:49:18.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:49:18.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:49:48.243 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:49:48.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:50:18.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:50:18.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:50:48.718 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:50:48.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:51:18.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:51:18.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:51:48.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:51:48.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:52:19.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:52:19.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:52:49.265 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:52:49.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:53:19.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:53:19.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:53:49.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:53:49.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:54:19.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:54:19.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:54:49.904 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:54:49.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:55:20.077 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:55:20.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:55:50.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:55:50.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:56:20.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:56:20.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:56:50.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:56:50.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:57:20.733 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:57:20.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:57:50.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:57:50.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:58:21.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:58:21.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:58:51.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:58:51.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:59:21.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:59:21.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:59:51.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T08:59:51.450 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:00:21.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:00:21.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:00:51.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:00:51.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:01:21.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:01:21.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:01:52.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:01:52.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:02:16.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:16.244 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_08_00_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_09_01_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_10_00_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_11_00_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_12_00_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_13_00_UTC 2023-12-19T09:02:16.283 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_14_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_15_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_16_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_17_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_18_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_19_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_20_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_21_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_22_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_23_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_24_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_25_00_UTC 2023-12-19T09:02:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_26_01_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_27_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_28_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_29_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_30_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_31_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_32_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_33_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_34_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_35_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_36_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_37_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_38_00_UTC 2023-12-19T09:02:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_39_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_40_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_41_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_42_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_43_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_44_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_45_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_46_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_47_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_48_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_49_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_50_00_UTC 2023-12-19T09:02:16.286 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_51_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_52_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_53_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_54_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_55_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_56_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_57_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_58_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_59_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_00_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_01_00_UTC 2023-12-19T09:02:16.287 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_02_00_UTC 2023-12-19T09:02:16.288 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-08_08_00_UTC', 'scheduled-2023-12-19-08_09_01_UTC', 'scheduled-2023-12-19-08_10_00_UTC', 'scheduled-2023-12-19-08_11_00_UTC', 'scheduled-2023-12-19-08_12_00_UTC', 'scheduled-2023-12-19-08_13_00_UTC', 'scheduled-2023-12-19-08_14_00_UTC', 'scheduled-2023-12-19-08_15_00_UTC', 'scheduled-2023-12-19-08_16_00_UTC', 'scheduled-2023-12-19-08_17_00_UTC', 'scheduled-2023-12-19-08_18_00_UTC', 'scheduled-2023-12-19-08_19_00_UTC', 'scheduled-2023-12-19-08_20_00_UTC', 'scheduled-2023-12-19-08_21_00_UTC', 'scheduled-2023-12-19-08_22_00_UTC', 'scheduled-2023-12-19-08_23_00_UTC', 'scheduled-2023-12-19-08_24_00_UTC', 'scheduled-2023-12-19-08_25_00_UTC', 'scheduled-2023-12-19-08_26_01_UTC', 'scheduled-2023-12-19-08_27_00_UTC', 'scheduled-2023-12-19-08_28_00_UTC', 'scheduled-2023-12-19-08_29_00_UTC', 'scheduled-2023-12-19-08_30_00_UTC', 'scheduled-2023-12-19-08_31_00_UTC', 'scheduled-2023-12-19-08_32_00_UTC', 'scheduled-2023-12-19-08_33_00_UTC', 'scheduled-2023-12-19-08_34_00_UTC', 'scheduled-2023-12-19-08_35_00_UTC', 'scheduled-2023-12-19-08_36_00_UTC', 'scheduled-2023-12-19-08_37_00_UTC', 'scheduled-2023-12-19-08_38_00_UTC', 'scheduled-2023-12-19-08_39_00_UTC', 'scheduled-2023-12-19-08_40_00_UTC', 'scheduled-2023-12-19-08_41_00_UTC', 'scheduled-2023-12-19-08_42_00_UTC', 'scheduled-2023-12-19-08_43_00_UTC', 'scheduled-2023-12-19-08_44_00_UTC', 'scheduled-2023-12-19-08_45_00_UTC', 'scheduled-2023-12-19-08_46_00_UTC', 'scheduled-2023-12-19-08_47_00_UTC', 'scheduled-2023-12-19-08_48_00_UTC', 'scheduled-2023-12-19-08_49_00_UTC', 'scheduled-2023-12-19-08_50_00_UTC', 'scheduled-2023-12-19-08_51_00_UTC', 'scheduled-2023-12-19-08_52_00_UTC', 'scheduled-2023-12-19-08_53_00_UTC', 'scheduled-2023-12-19-08_54_00_UTC', 'scheduled-2023-12-19-08_55_00_UTC', 'scheduled-2023-12-19-08_56_00_UTC', 'scheduled-2023-12-19-08_57_00_UTC', 'scheduled-2023-12-19-08_58_00_UTC', 'scheduled-2023-12-19-08_59_00_UTC', 'scheduled-2023-12-19-09_00_00_UTC', 'scheduled-2023-12-19-09_01_00_UTC', 'scheduled-2023-12-19-09_02_00_UTC'] 2023-12-19T09:02:16.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1 2023-12-19T09:02:16.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 Processor -- start 2023-12-19T09:02:16.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 -- start start 2023-12-19T09:02:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 0x7f06f006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f006d7b0 con 0x7f06f00ac650 2023-12-19T09:02:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f00ae570 con 0x7f06f006b060 2023-12-19T09:02:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f7d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f006e180 con 0x7f06f0156e70 2023-12-19T09:02:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.488+0000 7f06f62e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f06f00ac650 0x7f06f00aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:52698/0 (socket says 172.21.15.111:52698) 2023-12-19T09:02:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f62e2700 1 -- 172.21.15.111:0/3589556316 learned_addr learned my addr 172.21.15.111:0/3589556316 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f5ae1700 1 --2- 172.21.15.111:0/3589556316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 0x7f06f006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f4adf700 1 -- 172.21.15.111:0/3589556316 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936930299 0 0) 0x7f06f006d7b0 con 0x7f06f00ac650 2023-12-19T09:02:16.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f5ae1700 1 -- 172.21.15.111:0/3589556316 >> v1:172.21.15.143:6789/0 conn(0x7f06f006b060 legacy=0x7f06f00abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f5ae1700 1 -- 172.21.15.111:0/3589556316 >> v1:172.21.15.111:6789/0 conn(0x7f06f00ac650 legacy=0x7f06f00aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f5ae1700 1 -- 172.21.15.111:0/3589556316 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f006cba0 con 0x7f06f0156e70 2023-12-19T09:02:16.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f5ae1700 1 --2- 172.21.15.111:0/3589556316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 0x7f06f006dc40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009e10 tx=0x7f06ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8462b22845b44ae6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f4adf700 1 -- 172.21.15.111:0/3589556316 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00e070 con 0x7f06f0156e70 2023-12-19T09:02:16.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f4adf700 1 -- 172.21.15.111:0/3589556316 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec00bbb0 con 0x7f06f0156e70 2023-12-19T09:02:16.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f4adf700 1 -- 172.21.15.111:0/3589556316 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec012430 con 0x7f06f0156e70 2023-12-19T09:02:16.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 -- 172.21.15.111:0/3589556316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 msgr2=0x7f06f006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 --2- 172.21.15.111:0/3589556316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 0x7f06f006dc40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009e10 tx=0x7f06ec009780 comp rx=0 tx=0).stop 2023-12-19T09:02:16.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 -- 172.21.15.111:0/3589556316 shutdown_connections 2023-12-19T09:02:16.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 --2- 172.21.15.111:0/3589556316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f0156e70 0x7f06f006dc40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 -- 172.21.15.111:0/3589556316 >> 172.21.15.111:0/3589556316 conn(0x7f06f0067bb0 msgr2=0x7f06f00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:16.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 -- 172.21.15.111:0/3589556316 shutdown_connections 2023-12-19T09:02:16.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.492+0000 7f06f7d45700 1 -- 172.21.15.111:0/3589556316 wait complete. 2023-12-19T09:02:16.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.496+0000 7f06f7d45700 1 Processor -- start 2023-12-19T09:02:16.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.496+0000 7f06f7d45700 1 -- start start 2023-12-19T09:02:16.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 0x7f06f014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f0156e70 0x7f06f0157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f0155d90 con 0x7f06f006b060 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f014c7b0 con 0x7f06f0156e70 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f01547e0 con 0x7f06f00ac650 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52112/0 (socket says 172.21.15.111:52112) 2023-12-19T09:02:16.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 -- 172.21.15.111:0/1430170895 learned_addr learned my addr 172.21.15.111:0/1430170895 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:16.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f62e2700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f0156e70 0x7f06f0157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f52e0700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 0x7f06f014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 -- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 msgr2=0x7f06f014c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 0x7f06f014c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 -- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f0156e70 msgr2=0x7f06f0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f0156e70 0x7f06f0157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 -- 172.21.15.111:0/1430170895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec009a20 con 0x7f06f006b060 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f52e0700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 0x7f06f014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f5ae1700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00b6c0 tx=0x7f06ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00be70 con 0x7f06f006b060 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec01d630 con 0x7f06f006b060 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec01d9f0 con 0x7f06f006b060 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f0154af0 con 0x7f06f006b060 2023-12-19T09:02:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.500+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f0155330 con 0x7f06f006b060 2023-12-19T09:02:16.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.504+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f0154c80 con 0x7f06f006b060 2023-12-19T09:02:16.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.504+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 65) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f06ec01b3b0 con 0x7f06f006b060 2023-12-19T09:02:16.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.504+0000 7f06e5ffb700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 0x7f06d4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.504+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f06ec07f530 con 0x7f06f006b060 2023-12-19T09:02:16.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.508+0000 7f06f52e0700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 0x7f06d4062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.508+0000 7f06f52e0700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 0x7f06d4062b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06f006e110 tx=0x7f06e0009040 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.512+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f06ec04f370 con 0x7f06f006b060 2023-12-19T09:02:16.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.708+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06f00711b0 con 0x7f06d40606c0 2023-12-19T09:02:16.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.724+0000 7f06e5ffb700 1 -- 172.21.15.111:0/1430170895 <== mgr.7409 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f06e000d5c0 con 0x7f06d40606c0 2023-12-19T09:02:16.729 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:16.729 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:06:38", "first": "2023-12-19T08:07:00", "last": "2023-12-19T09:02:00", "last_pruned": "2023-12-19T09:02:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-12-19T09:02:16.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 msgr2=0x7f06d4062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 0x7f06d4062b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06f006e110 tx=0x7f06e0009040 comp rx=0 tx=0).stop 2023-12-19T09:02:16.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 msgr2=0x7f06f0155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00b6c0 tx=0x7f06ec018000 comp rx=0 tx=0).stop 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 shutdown_connections 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06d40606c0 0x7f06d4062b70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f0156e70 0x7f06f0157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f00ac650 0x7f06f014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 --2- 172.21.15.111:0/1430170895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f006b060 0x7f06f0155850 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 >> 172.21.15.111:0/1430170895 conn(0x7f06f0067bb0 msgr2=0x7f06f00aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 shutdown_connections 2023-12-19T09:02:16.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.728+0000 7f06f7d45700 1 -- 172.21.15.111:0/1430170895 wait complete. 2023-12-19T09:02:16.752 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1M", "retention": {"n": 55}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T08:06:38", "first": "2023-12-19T08:07:00", "last": "2023-12-19T09:02:00", "last_pruned": "2023-12-19T09:02:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-12-19T09:02:16.752 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1M', 'retention': {'n': 55}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T08:06:38', 'first': '2023-12-19T08:07:00', 'last': '2023-12-19T09:02:00', 'last_pruned': '2023-12-19T09:02:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-12-19T09:02:16.752 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-12-19T09:02:16.752 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-12-19T09:02:16.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1 2023-12-19T09:02:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 Processor -- start 2023-12-19T09:02:16.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 -- start start 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4fc158da0 0x7fc4fc06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc06df10 0x7fc4fc06ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc0ca290 con 0x7fc4fc0caca0 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc15d720 con 0x7fc4fc158da0 2023-12-19T09:02:16.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.912+0000 7fc503016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0ca4a0 con 0x7fc4fc06df10 2023-12-19T09:02:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc501813700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc06df10 0x7fc4fc06ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52122/0 (socket says 172.21.15.111:52122) 2023-12-19T09:02:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 -- 172.21.15.111:0/545524731 learned_addr learned my addr 172.21.15.111:0/545524731 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:16.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502014700 1 --2- 172.21.15.111:0/545524731 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc158da0 0x7fc4fc06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 -- 172.21.15.111:0/545524731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc06df10 msgr2=0x7fc4fc06ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 --2- 172.21.15.111:0/545524731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc06df10 0x7fc4fc06ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 -- 172.21.15.111:0/545524731 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc158da0 msgr2=0x7fc4fc06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 --2- 172.21.15.111:0/545524731 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc158da0 0x7fc4fc06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 -- 172.21.15.111:0/545524731 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0ca780 con 0x7fc4fc0caca0 2023-12-19T09:02:16.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc502815700 1 --2- 172.21.15.111:0/545524731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc06a310 tx=0x7fc4f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc67ca8012a31918 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc501012700 1 -- 172.21.15.111:0/545524731 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f400e070 con 0x7fc4fc0caca0 2023-12-19T09:02:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc501012700 1 -- 172.21.15.111:0/545524731 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f400b970 con 0x7fc4fc0caca0 2023-12-19T09:02:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc501012700 1 -- 172.21.15.111:0/545524731 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f40092e0 con 0x7fc4fc0caca0 2023-12-19T09:02:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc503016700 1 -- 172.21.15.111:0/545524731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 msgr2=0x7fc4fc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.916+0000 7fc503016700 1 --2- 172.21.15.111:0/545524731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc06a310 tx=0x7fc4f4009840 comp rx=0 tx=0).stop 2023-12-19T09:02:16.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 -- 172.21.15.111:0/545524731 shutdown_connections 2023-12-19T09:02:16.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 --2- 172.21.15.111:0/545524731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc06a7d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 --2- 172.21.15.111:0/545524731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc06df10 0x7fc4fc06ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 --2- 172.21.15.111:0/545524731 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc158da0 0x7fc4fc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 -- 172.21.15.111:0/545524731 >> 172.21.15.111:0/545524731 conn(0x7fc4fc0cc760 msgr2=0x7fc4fc0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:16.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 -- 172.21.15.111:0/545524731 shutdown_connections 2023-12-19T09:02:16.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 -- 172.21.15.111:0/545524731 wait complete. 2023-12-19T09:02:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 Processor -- start 2023-12-19T09:02:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.920+0000 7fc503016700 1 -- start start 2023-12-19T09:02:16.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 0x7fc4fc0c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 0x7fc4fc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0c8310 con 0x7fc4fc0caca0 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0c85f0 con 0x7fc4fc06df10 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0c62d0 con 0x7fc4fc158da0 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52124/0 (socket says 172.21.15.111:52124) 2023-12-19T09:02:16.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 -- 172.21.15.111:0/665171556 learned_addr learned my addr 172.21.15.111:0/665171556 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc502815700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 0x7fc4fc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc502014700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 0x7fc4fc0c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 -- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 msgr2=0x7fc4fc1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 0x7fc4fc1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 -- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 msgr2=0x7fc4fc0c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 0x7fc4fc0c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 -- 172.21.15.111:0/665171556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f4009ae0 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc502815700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 0x7fc4fc1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc502014700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 0x7fc4fc0c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc501813700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00ecb0 tx=0x7fc4ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00c9d0 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec00cb50 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc0c65b0 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec013470 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc15d060 con 0x7fc4fc0caca0 2023-12-19T09:02:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.924+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc04be40 con 0x7fc4fc0caca0 2023-12-19T09:02:16.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.928+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 65) v1 ==== 80092+0+0 (secure 0 0 0) 0x7fc4ec013860 con 0x7fc4fc0caca0 2023-12-19T09:02:16.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.928+0000 7fc4f27fc700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 0x7fc4e0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:16.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.928+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fc4ec07d230 con 0x7fc4fc0caca0 2023-12-19T09:02:16.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.932+0000 7fc502014700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 0x7fc4e0062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:16.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.932+0000 7fc502014700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 0x7fc4e0062b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006f00 tx=0x7fc4f8008040 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:16.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:16.936+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc4ec04d220 con 0x7fc4fc0caca0 2023-12-19T09:02:17.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.124+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4fc158950 con 0x7fc4e00606c0 2023-12-19T09:02:17.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.244+0000 7fc4f27fc700 1 -- 172.21.15.111:0/665171556 <== mgr.7409 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7fc4fc158950 con 0x7fc4e00606c0 2023-12-19T09:02:17.250 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 msgr2=0x7fc4e0062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 0x7fc4e0062b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006f00 tx=0x7fc4f8008040 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 msgr2=0x7fc4fc15cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00ecb0 tx=0x7fc4ec0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 shutdown_connections 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc4e00606c0 0x7fc4e0062b70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc158da0 0x7fc4fc1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4fc0caca0 0x7fc4fc15cb20 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 --2- 172.21.15.111:0/665171556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc06df10 0x7fc4fc0c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 >> 172.21.15.111:0/665171556 conn(0x7fc4fc0cc760 msgr2=0x7fc4fc0af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:17.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 shutdown_connections 2023-12-19T09:02:17.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:17.248+0000 7fc503016700 1 -- 172.21.15.111:0/665171556 wait complete. 2023-12-19T09:02:17.277 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-12-19T09:02:17.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.277 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_08_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_09_01_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_10_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_11_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_12_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_13_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_14_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_15_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_16_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_17_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_18_00_UTC 2023-12-19T09:02:17.312 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_19_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_20_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_21_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_22_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_23_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_24_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_25_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_26_01_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_27_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_28_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_29_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_30_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_31_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_32_00_UTC 2023-12-19T09:02:17.313 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_33_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_34_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_35_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_36_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_37_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_38_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_39_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_40_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_41_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_42_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_43_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_44_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_45_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_46_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_47_00_UTC 2023-12-19T09:02:17.314 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_48_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_49_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_50_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_51_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_52_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_53_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_54_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_55_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_56_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_57_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_58_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-08_59_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_00_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_01_00_UTC 2023-12-19T09:02:17.315 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_02_00_UTC 2023-12-19T09:02:17.316 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_08_00_UTC 2023-12-19T09:02:17.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.317 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_08_00_UTC) 2023-12-19T09:02:17.405 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_09_01_UTC 2023-12-19T09:02:17.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.405 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_09_01_UTC) 2023-12-19T09:02:17.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_10_00_UTC 2023-12-19T09:02:17.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.490 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_10_00_UTC) 2023-12-19T09:02:17.571 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_11_00_UTC 2023-12-19T09:02:17.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.571 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_11_00_UTC) 2023-12-19T09:02:17.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_12_00_UTC 2023-12-19T09:02:17.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.654 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_12_00_UTC) 2023-12-19T09:02:17.737 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_13_00_UTC 2023-12-19T09:02:17.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.738 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_13_00_UTC) 2023-12-19T09:02:17.818 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_14_00_UTC 2023-12-19T09:02:17.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.819 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_14_00_UTC) 2023-12-19T09:02:17.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_15_00_UTC 2023-12-19T09:02:17.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.904 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_15_00_UTC) 2023-12-19T09:02:17.984 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_16_00_UTC 2023-12-19T09:02:17.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:17.985 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_16_00_UTC) 2023-12-19T09:02:18.067 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_17_00_UTC 2023-12-19T09:02:18.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.067 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_17_00_UTC) 2023-12-19T09:02:18.146 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_18_00_UTC 2023-12-19T09:02:18.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.147 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_18_00_UTC) 2023-12-19T09:02:18.230 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_19_00_UTC 2023-12-19T09:02:18.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.231 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_19_00_UTC) 2023-12-19T09:02:18.308 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_20_00_UTC 2023-12-19T09:02:18.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.309 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_20_00_UTC) 2023-12-19T09:02:18.391 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_21_00_UTC 2023-12-19T09:02:18.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.391 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_21_00_UTC) 2023-12-19T09:02:18.469 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_22_00_UTC 2023-12-19T09:02:18.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.470 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_22_00_UTC) 2023-12-19T09:02:18.550 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_23_00_UTC 2023-12-19T09:02:18.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.550 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_23_00_UTC) 2023-12-19T09:02:18.631 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_24_00_UTC 2023-12-19T09:02:18.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.631 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_24_00_UTC) 2023-12-19T09:02:18.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_25_00_UTC 2023-12-19T09:02:18.709 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.709 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_25_00_UTC) 2023-12-19T09:02:18.790 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_26_01_UTC 2023-12-19T09:02:18.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.790 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_26_01_UTC) 2023-12-19T09:02:18.872 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_27_00_UTC 2023-12-19T09:02:18.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.872 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_27_00_UTC) 2023-12-19T09:02:18.954 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_28_00_UTC 2023-12-19T09:02:18.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:18.954 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_28_00_UTC) 2023-12-19T09:02:19.035 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_29_00_UTC 2023-12-19T09:02:19.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.035 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_29_00_UTC) 2023-12-19T09:02:19.113 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_30_00_UTC 2023-12-19T09:02:19.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.113 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_30_00_UTC) 2023-12-19T09:02:19.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_31_00_UTC 2023-12-19T09:02:19.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.195 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_31_00_UTC) 2023-12-19T09:02:19.280 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_32_00_UTC 2023-12-19T09:02:19.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.280 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_32_00_UTC) 2023-12-19T09:02:19.363 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_33_00_UTC 2023-12-19T09:02:19.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.364 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_33_00_UTC) 2023-12-19T09:02:19.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_34_00_UTC 2023-12-19T09:02:19.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.452 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_34_00_UTC) 2023-12-19T09:02:19.536 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_35_00_UTC 2023-12-19T09:02:19.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.536 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_35_00_UTC) 2023-12-19T09:02:19.616 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_36_00_UTC 2023-12-19T09:02:19.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.616 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_36_00_UTC) 2023-12-19T09:02:19.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_37_00_UTC 2023-12-19T09:02:19.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.704 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_37_00_UTC) 2023-12-19T09:02:19.786 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_38_00_UTC 2023-12-19T09:02:19.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.786 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_38_00_UTC) 2023-12-19T09:02:19.871 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_39_00_UTC 2023-12-19T09:02:19.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.871 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_39_00_UTC) 2023-12-19T09:02:19.955 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_40_00_UTC 2023-12-19T09:02:19.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:19.956 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_40_00_UTC) 2023-12-19T09:02:20.033 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_41_00_UTC 2023-12-19T09:02:20.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.033 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_41_00_UTC) 2023-12-19T09:02:20.115 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_42_00_UTC 2023-12-19T09:02:20.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.116 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_42_00_UTC) 2023-12-19T09:02:20.196 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_43_00_UTC 2023-12-19T09:02:20.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.197 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_43_00_UTC) 2023-12-19T09:02:20.279 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_44_00_UTC 2023-12-19T09:02:20.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.279 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_44_00_UTC) 2023-12-19T09:02:20.357 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_45_00_UTC 2023-12-19T09:02:20.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.357 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_45_00_UTC) 2023-12-19T09:02:20.439 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_46_00_UTC 2023-12-19T09:02:20.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.439 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_46_00_UTC) 2023-12-19T09:02:20.522 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_47_00_UTC 2023-12-19T09:02:20.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.522 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_47_00_UTC) 2023-12-19T09:02:20.606 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_48_00_UTC 2023-12-19T09:02:20.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.606 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_48_00_UTC) 2023-12-19T09:02:20.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_49_00_UTC 2023-12-19T09:02:20.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.692 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_49_00_UTC) 2023-12-19T09:02:20.777 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_50_00_UTC 2023-12-19T09:02:20.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.777 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_50_00_UTC) 2023-12-19T09:02:20.860 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_51_00_UTC 2023-12-19T09:02:20.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.860 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_51_00_UTC) 2023-12-19T09:02:20.946 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_52_00_UTC 2023-12-19T09:02:20.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:20.947 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_52_00_UTC) 2023-12-19T09:02:21.028 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_53_00_UTC 2023-12-19T09:02:21.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.028 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_53_00_UTC) 2023-12-19T09:02:21.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_54_00_UTC 2023-12-19T09:02:21.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.116 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_54_00_UTC) 2023-12-19T09:02:21.196 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_55_00_UTC 2023-12-19T09:02:21.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.196 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_55_00_UTC) 2023-12-19T09:02:21.278 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_56_00_UTC 2023-12-19T09:02:21.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.278 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_56_00_UTC) 2023-12-19T09:02:21.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_57_00_UTC 2023-12-19T09:02:21.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.362 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_57_00_UTC) 2023-12-19T09:02:21.442 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_58_00_UTC 2023-12-19T09:02:21.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.442 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_58_00_UTC) 2023-12-19T09:02:21.523 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-08_59_00_UTC 2023-12-19T09:02:21.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.523 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-08_59_00_UTC) 2023-12-19T09:02:21.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-09_00_00_UTC 2023-12-19T09:02:21.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.611 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-09_00_00_UTC) 2023-12-19T09:02:21.691 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-09_01_00_UTC 2023-12-19T09:02:21.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.691 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-09_01_00_UTC) 2023-12-19T09:02:21.770 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-09_02_00_UTC 2023-12-19T09:02:21.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.770 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-09_02_00_UTC) 2023-12-19T09:02:21.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:21.848 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T09:02:21.926 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T09:02:22.107 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:02:22.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:02:22.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 Processor -- start 2023-12-19T09:02:22.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 -- start start 2023-12-19T09:02:22.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0810158b00 0x7f0810158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f081006a4f0 0x7f08100cab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081006b6c0 con 0x7f0810158b00 2023-12-19T09:02:22.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f081015d880 con 0x7f081006a4f0 2023-12-19T09:02:22.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.132+0000 7f080f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081006aab0 con 0x7f081006bb50 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0810158b00 0x7f0810158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f0805d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f0805d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:36568/0 (socket says 172.21.15.111:36568) 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f0805d9b700 1 -- 172.21.15.111:0/3796007933 learned_addr learned my addr 172.21.15.111:0/3796007933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080ed9d700 1 --2- 172.21.15.111:0/3796007933 >> v2:172.21.15.143:3300/0 conn(0x7f081006a4f0 0x7f08100cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 -- 172.21.15.111:0/3796007933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 msgr2=0x7f081015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:22.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 --2- 172.21.15.111:0/3796007933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 -- 172.21.15.111:0/3796007933 >> v2:172.21.15.143:3300/0 conn(0x7f081006a4f0 msgr2=0x7f08100cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:22.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 --2- 172.21.15.111:0/3796007933 >> v2:172.21.15.143:3300/0 conn(0x7f081006a4f0 0x7f08100cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 -- 172.21.15.111:0/3796007933 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081006ad90 con 0x7f0810158b00 2023-12-19T09:02:22.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080e59c700 1 --2- 172.21.15.111:0/3796007933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f0810158ed0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0810068510 tx=0x7f08000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105a68297d6faaec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:22.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080dd9b700 1 -- 172.21.15.111:0/3796007933 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800011630 con 0x7f0810158b00 2023-12-19T09:02:22.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080dd9b700 1 -- 172.21.15.111:0/3796007933 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000117b0 con 0x7f0810158b00 2023-12-19T09:02:22.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080dd9b700 1 -- 172.21.15.111:0/3796007933 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800011b70 con 0x7f0810158b00 2023-12-19T09:02:22.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 -- 172.21.15.111:0/3796007933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 msgr2=0x7f0810158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:22.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 --2- 172.21.15.111:0/3796007933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f0810158ed0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0810068510 tx=0x7f08000091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 -- 172.21.15.111:0/3796007933 shutdown_connections 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 --2- 172.21.15.111:0/3796007933 >> v2:172.21.15.143:3300/0 conn(0x7f081006a4f0 0x7f08100cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 --2- 172.21.15.111:0/3796007933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 --2- 172.21.15.111:0/3796007933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f0810158ed0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 -- 172.21.15.111:0/3796007933 >> 172.21.15.111:0/3796007933 conn(0x7f0810067bb0 msgr2=0x7f08100cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 -- 172.21.15.111:0/3796007933 shutdown_connections 2023-12-19T09:02:22.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.136+0000 7f080f59e700 1 -- 172.21.15.111:0/3796007933 wait complete. 2023-12-19T09:02:22.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 Processor -- start 2023-12-19T09:02:22.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 -- start start 2023-12-19T09:02:22.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f081006a4f0 0x7f081015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081015a6d0 con 0x7f0810158b00 2023-12-19T09:02:22.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0810159ac0 con 0x7f081006a4f0 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0810159da0 con 0x7f081006bb50 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59720/0 (socket says 172.21.15.111:59720) 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 -- 172.21.15.111:0/2056663476 learned_addr learned my addr 172.21.15.111:0/2056663476 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080e59c700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f081006a4f0 0x7f081015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f0805d9b700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 -- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 msgr2=0x7f081015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 -- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f081006a4f0 msgr2=0x7f081015ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f081006a4f0 0x7f081015ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 -- 172.21.15.111:0/2056663476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800009420 con 0x7f0810158b00 2023-12-19T09:02:22.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.140+0000 7f080ed9d700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0808009e90 tx=0x7f080800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:22.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800cc20 con 0x7f0810158b00 2023-12-19T09:02:22.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080800f040 con 0x7f0810158b00 2023-12-19T09:02:22.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081015a080 con 0x7f0810158b00 2023-12-19T09:02:22.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808014480 con 0x7f0810158b00 2023-12-19T09:02:22.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0810156f30 con 0x7f0810158b00 2023-12-19T09:02:22.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081015a880 con 0x7f0810158b00 2023-12-19T09:02:22.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.144+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 65) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f0808014830 con 0x7f0810158b00 2023-12-19T09:02:22.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.148+0000 7f0806ffd700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 0x7f07f0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:22.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.148+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f080807dcd0 con 0x7f0810158b00 2023-12-19T09:02:22.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.148+0000 7f080e59c700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 0x7f07f0062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:22.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.148+0000 7f080e59c700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 0x7f07f0062b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0810068510 tx=0x7f0800017000 comp rx=0 tx=0).ready entity=mgr.7409 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:22.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.152+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f081015a880 con 0x7f0810158b00 2023-12-19T09:02:22.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.356+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f081004be40 con 0x7f0810158b00 2023-12-19T09:02:22.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.420+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f080807bcf0 con 0x7f0810158b00 2023-12-19T09:02:22.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f0806ffd700 1 -- 172.21.15.111:0/2056663476 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v66) v1 ==== 93+0+0 (secure 0 0 0) 0x7f08080513b0 con 0x7f0810158b00 2023-12-19T09:02:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 msgr2=0x7f07f0062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 0x7f07f0062b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0810068510 tx=0x7f0800017000 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 msgr2=0x7f08101593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0808009e90 tx=0x7f080800b040 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 shutdown_connections 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f07f00606c0 0x7f07f0062b70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0810158b00 0x7f08101593c0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f081006bb50 0x7f081015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 --2- 172.21.15.111:0/2056663476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f081006a4f0 0x7f081015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 >> 172.21.15.111:0/2056663476 conn(0x7f0810067bb0 msgr2=0x7f08100cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 shutdown_connections 2023-12-19T09:02:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:22.424+0000 7f080f59e700 1 -- 172.21.15.111:0/2056663476 wait complete. 2023-12-19T09:02:22.453 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:02:22.453 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:22.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:22.528 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:02:22.529 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:02:22.529 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:02:22.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:22.565 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[55743]: fuse finished with error 0 and tester_r 0 2023-12-19T09:02:22.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:02:22.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:22.569 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:02:22.629 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:22.624+0000 7f35b8fb1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:22.633 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:22.627+0000 7fd3a4327000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:22.757 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:02:22.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:22.757 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T09:02:22.778 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T09:02:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:22.779 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:02:22.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:22.779 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T09:02:22.818 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T09:02:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:22.820 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:02:22.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:22.821 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T09:02:22.842 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T09:02:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:22.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.012+0000 7ff7b982e700 1 Processor -- start 2023-12-19T09:02:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.012+0000 7ff7b982e700 1 -- start start 2023-12-19T09:02:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7b982e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7b4068800 0x7ff7b40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7b982e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b406c9f0 con 0x7ff7b40c2980 2023-12-19T09:02:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7b982e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b40c7f20 con 0x7ff7b413f890 2023-12-19T09:02:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7b982e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b40c1c50 con 0x7ff7b4068800 2023-12-19T09:02:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7aa7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff7b40c2980 0x7ff7b40680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58138/0 (socket says 172.21.15.111:58138) 2023-12-19T09:02:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7aa7fc700 1 -- 172.21.15.111:0/776173628 learned_addr learned my addr 172.21.15.111:0/776173628 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.016+0000 7ff7b37fe700 1 --2- 172.21.15.111:0/776173628 >> v2:172.21.15.143:3300/0 conn(0x7ff7b4068800 0x7ff7b40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519324682 0 0) 0x7ff7b406c9f0 con 0x7ff7b40c2980 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0003570 con 0x7ff7b40c2980 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091491754 0 0) 0x7ff7b40c7f20 con 0x7ff7b413f890 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b37fe700 1 -- 172.21.15.111:0/776173628 >> v1:172.21.15.143:6789/0 conn(0x7ff7b413f890 legacy=0x7ff7b40c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b37fe700 1 -- 172.21.15.111:0/776173628 >> v1:172.21.15.111:6789/0 conn(0x7ff7b40c2980 legacy=0x7ff7b40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b37fe700 1 -- 172.21.15.111:0/776173628 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40c8190 con 0x7ff7b4068800 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b37fe700 1 --2- 172.21.15.111:0/776173628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b4068800 0x7ff7b40aeba0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009e60 tx=0x7ff7ac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582fc3e46f0dee9b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac00e070 con 0x7ff7b4068800 2023-12-19T09:02:23.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac00bb60 con 0x7ff7b4068800 2023-12-19T09:02:23.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b27fc700 1 -- 172.21.15.111:0/776173628 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac012430 con 0x7ff7b4068800 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 -- 172.21.15.111:0/776173628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b4068800 msgr2=0x7ff7b40aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 --2- 172.21.15.111:0/776173628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b4068800 0x7ff7b40aeba0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009e60 tx=0x7ff7ac009780 comp rx=0 tx=0).stop 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 -- 172.21.15.111:0/776173628 shutdown_connections 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 --2- 172.21.15.111:0/776173628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b4068800 0x7ff7b40aeba0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 -- 172.21.15.111:0/776173628 >> 172.21.15.111:0/776173628 conn(0x7ff7b40c8850 msgr2=0x7ff7b40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 -- 172.21.15.111:0/776173628 shutdown_connections 2023-12-19T09:02:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.020+0000 7ff7b982e700 1 -- 172.21.15.111:0/776173628 wait complete. 2023-12-19T09:02:23.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 Processor -- start 2023-12-19T09:02:23.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 -- start start 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b40c2980 0x7ff7b4130570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7b413f890 0x7ff7b40aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40ac840 con 0x7ff7b4068800 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b413d8b0 con 0x7ff7b40c2980 2023-12-19T09:02:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b982e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b413db90 con 0x7ff7b413f890 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7b413f890 0x7ff7b40aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59754/0 (socket says 172.21.15.111:59754) 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7b2ffd700 1 -- 172.21.15.111:0/2045627038 learned_addr learned my addr 172.21.15.111:0/2045627038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.024+0000 7ff7aa7fc700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b40c2980 0x7ff7b4130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7b413f890 msgr2=0x7ff7b40aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7b413f890 0x7ff7b40aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b40c2980 msgr2=0x7ff7b4130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b40c2980 0x7ff7b4130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 -- 172.21.15.111:0/2045627038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac009a20 con 0x7ff7b4068800 2023-12-19T09:02:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b2ffd700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c530 tx=0x7ff7a400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4011590 con 0x7ff7b4068800 2023-12-19T09:02:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a4011710 con 0x7ff7b4068800 2023-12-19T09:02:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40ac4d0 con 0x7ff7b4068800 2023-12-19T09:02:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4011ad0 con 0x7ff7b4068800 2023-12-19T09:02:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b413de70 con 0x7ff7b4068800 2023-12-19T09:02:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.028+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b40ac330 con 0x7ff7b4068800 2023-12-19T09:02:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.032+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7ff7a401b3b0 con 0x7ff7b4068800 2023-12-19T09:02:23.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.032+0000 7ff7ab7fe700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 0x7ff794062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.032+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff7a407c2b0 con 0x7ff7b4068800 2023-12-19T09:02:23.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.032+0000 7ff7aa7fc700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 msgr2=0x7ff794062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.032+0000 7ff7aa7fc700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 0x7ff794062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.036+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff7b40ac330 con 0x7ff7b4068800 2023-12-19T09:02:23.188 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:23.183+0000 7fd3a4327000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:23.228 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:23.224+0000 7f35b8fb1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:23.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.236+0000 7ff7aa7fc700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 msgr2=0x7ff794062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:23.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.236+0000 7ff7aa7fc700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 0x7ff794062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:23.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.256+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7b40ac330 con 0x7ff7b4068800 2023-12-19T09:02:23.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7ab7fe700 1 -- 172.21.15.111:0/2045627038 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7ff7b40ac330 con 0x7ff7b4068800 2023-12-19T09:02:23.265 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:23.265 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 msgr2=0x7ff794062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 0x7ff794062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 msgr2=0x7ff7b40a9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400c530 tx=0x7ff7a400c840 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 shutdown_connections 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff794060670 0x7ff794062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7b40c2980 0x7ff7b4130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7b413f890 0x7ff7b40aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 --2- 172.21.15.111:0/2045627038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7b4068800 0x7ff7b40a9260 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 >> 172.21.15.111:0/2045627038 conn(0x7ff7b40c8850 msgr2=0x7ff7b413e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.260+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 shutdown_connections 2023-12-19T09:02:23.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.264+0000 7ff7b982e700 1 -- 172.21.15.111:0/2045627038 wait complete. 2023-12-19T09:02:23.270 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T09:02:23.279 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:23.275+0000 7fd3a4327000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:23.286 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:23.309 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:23.304+0000 7f35b8fb1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:23.426 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:23.423+0000 7fd3a4327000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.440+0000 7f785d425700 1 Processor -- start 2023-12-19T09:02:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.440+0000 7f785d425700 1 -- start start 2023-12-19T09:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f785d425700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f785813f890 0x7f78580aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f785d425700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f785d425700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785806c9f0 con 0x7f78580c1a80 2023-12-19T09:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f785d425700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78580c7f20 con 0x7f785813f890 2023-12-19T09:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f785d425700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78580c8190 con 0x7f78580c2980 2023-12-19T09:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f785813f890 0x7f78580aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f78577fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f78580c1a80 0x7f78580680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58144/0 (socket says 172.21.15.111:58144) 2023-12-19T09:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f78577fe700 1 -- 172.21.15.111:0/3115342862 learned_addr learned my addr 172.21.15.111:0/3115342862 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f78567fc700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 -- 172.21.15.111:0/3115342862 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 msgr2=0x7f78580c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 -- 172.21.15.111:0/3115342862 >> v1:172.21.15.111:6789/0 conn(0x7f78580c1a80 legacy=0x7f78580680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 -- 172.21.15.111:0/3115342862 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580c8470 con 0x7f785813f890 2023-12-19T09:02:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.444+0000 7f7856ffd700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580aeba0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f78480098e0 tx=0x7f78480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3d4119ac0840e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:23.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f7855ffb700 1 -- 172.21.15.111:0/3115342862 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784800b640 con 0x7f785813f890 2023-12-19T09:02:23.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f7855ffb700 1 -- 172.21.15.111:0/3115342862 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784800b7a0 con 0x7f785813f890 2023-12-19T09:02:23.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f7855ffb700 1 -- 172.21.15.111:0/3115342862 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784800bb40 con 0x7f785813f890 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 -- 172.21.15.111:0/3115342862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 msgr2=0x7f78580aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580aeba0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f78480098e0 tx=0x7f78480091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 -- 172.21.15.111:0/3115342862 shutdown_connections 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 --2- 172.21.15.111:0/3115342862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580aeba0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 -- 172.21.15.111:0/3115342862 >> 172.21.15.111:0/3115342862 conn(0x7f78580c8850 msgr2=0x7f78580c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 -- 172.21.15.111:0/3115342862 shutdown_connections 2023-12-19T09:02:23.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.448+0000 7f785d425700 1 -- 172.21.15.111:0/3115342862 wait complete. 2023-12-19T09:02:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.452+0000 7f785d425700 1 Processor -- start 2023-12-19T09:02:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.452+0000 7f785d425700 1 -- start start 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78580c5330 0x7f7858163740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785816e900 con 0x7f785813f890 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78580c5770 con 0x7f78580c5330 2023-12-19T09:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785816dcf0 con 0x7f78580c2980 2023-12-19T09:02:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59768/0 (socket says 172.21.15.111:59768) 2023-12-19T09:02:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 -- 172.21.15.111:0/3126467768 learned_addr learned my addr 172.21.15.111:0/3126467768 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f7856ffd700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 msgr2=0x7f78580addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78580c5330 msgr2=0x7f7858163740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78580c5330 0x7f7858163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 -- 172.21.15.111:0/3126467768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7848009420 con 0x7f785813f890 2023-12-19T09:02:23.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f78567fc700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f784c00ea70 tx=0x7f784c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:23.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c00c850 con 0x7f785813f890 2023-12-19T09:02:23.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c00c9d0 con 0x7f785813f890 2023-12-19T09:02:23.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c00cd90 con 0x7f785813f890 2023-12-19T09:02:23.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785816dfd0 con 0x7f785813f890 2023-12-19T09:02:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.456+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785816e590 con 0x7f785813f890 2023-12-19T09:02:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.460+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785816eab0 con 0x7f785813f890 2023-12-19T09:02:23.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.460+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f784c013680 con 0x7f785813f890 2023-12-19T09:02:23.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.464+0000 7f783effd700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 0x7f7844062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.464+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f784c07d170 con 0x7f785813f890 2023-12-19T09:02:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.464+0000 7f7856ffd700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 msgr2=0x7f7844062bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.464+0000 7f7856ffd700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 0x7f7844062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:23.474 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:23.468+0000 7f35b8fb1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:23.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.472+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f785816eab0 con 0x7f785813f890 2023-12-19T09:02:23.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.668+0000 7f7856ffd700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 msgr2=0x7f7844062bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:23.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.668+0000 7f7856ffd700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 0x7f7844062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:23.865 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:23.859+0000 7fd3a4327000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:02:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.876+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7858071120 con 0x7f785813f890 2023-12-19T09:02:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.880+0000 7f783effd700 1 -- 172.21.15.111:0/3126467768 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+17115 (secure 0 0 0) 0x7f784c04d120 con 0x7f785813f890 2023-12-19T09:02:23.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:23.889 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":148,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:23.440817+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":148,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3383118528":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/4020762991":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/3429689930":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/415557684":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2445077517":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6833/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6832/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2762275584":"2023-12-20T08:06:37.094770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":18,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 msgr2=0x7f7844062bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 0x7f7844062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 msgr2=0x7f78580c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f784c00ea70 tx=0x7f784c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 shutdown_connections 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7844060740 0x7f7844062bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78580c5330 0x7f7858163740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f785813f890 0x7f78580c4df0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 --2- 172.21.15.111:0/3126467768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78580c2980 0x7f78580addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 >> 172.21.15.111:0/3126467768 conn(0x7f78580c8850 msgr2=0x7f78580c1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 shutdown_connections 2023-12-19T09:02:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:23.884+0000 7f785d425700 1 -- 172.21.15.111:0/3126467768 wait complete. 2023-12-19T09:02:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T08:06:20.716528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T08:06:21.707600+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 148, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T09:02:23.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:02:23.945 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:23.932+0000 7f35b8fb1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:02:23.992 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:23.987+0000 7fd3a4327000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:24.086 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:24.080+0000 7f35b8fb1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:24.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.180+0000 7fa677323700 1 Processor -- start 2023-12-19T09:02:24.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.180+0000 7fa677323700 1 -- start start 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.184+0000 7fa677323700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa67006bad0 0x7fa67006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.184+0000 7fa677323700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa6700cb500 0x7fa6700c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.184+0000 7fa677323700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67006cec0 con 0x7fa6701580b0 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.184+0000 7fa677323700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6700c7330 con 0x7fa6700cb500 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.184+0000 7fa677323700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67006d1a0 con 0x7fa67006bad0 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa67006bad0 0x7fa67006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa67006bad0 0x7fa67006c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38216/0 (socket says 172.21.15.111:38216) 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 -- 172.21.15.111:0/3471672278 learned_addr learned my addr 172.21.15.111:0/3471672278 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:24.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa676b22700 1 --2- 172.21.15.111:0/3471672278 >> v2:172.21.15.111:3300/0 conn(0x7fa6700cb500 0x7fa6700c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 -- 172.21.15.111:0/3471672278 >> v2:172.21.15.111:3300/0 conn(0x7fa6700cb500 msgr2=0x7fa6700c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 --2- 172.21.15.111:0/3471672278 >> v2:172.21.15.111:3300/0 conn(0x7fa6700cb500 0x7fa6700c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 -- 172.21.15.111:0/3471672278 >> v1:172.21.15.111:6789/0 conn(0x7fa6701580b0 legacy=0x7fa67006b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 -- 172.21.15.111:0/3471672278 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67006d480 con 0x7fa67006bad0 2023-12-19T09:02:24.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa675b20700 1 --2- 172.21.15.111:0/3471672278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa67006bad0 0x7fa67006c7c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa67006c530 tx=0x7fa664009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efa30440b0500416 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.188+0000 7fa67531f700 1 -- 172.21.15.111:0/3471672278 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664009e70 con 0x7fa67006bad0 2023-12-19T09:02:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa67531f700 1 -- 172.21.15.111:0/3471672278 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66400b860 con 0x7fa67006bad0 2023-12-19T09:02:24.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa67531f700 1 -- 172.21.15.111:0/3471672278 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66400bc00 con 0x7fa67006bad0 2023-12-19T09:02:24.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 -- 172.21.15.111:0/3471672278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa67006bad0 msgr2=0x7fa67006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 --2- 172.21.15.111:0/3471672278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa67006bad0 0x7fa67006c7c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa67006c530 tx=0x7fa664009250 comp rx=0 tx=0).stop 2023-12-19T09:02:24.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 -- 172.21.15.111:0/3471672278 shutdown_connections 2023-12-19T09:02:24.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 --2- 172.21.15.111:0/3471672278 >> v2:172.21.15.111:3300/0 conn(0x7fa6700cb500 0x7fa6700c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 --2- 172.21.15.111:0/3471672278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa67006bad0 0x7fa67006c7c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 -- 172.21.15.111:0/3471672278 >> 172.21.15.111:0/3471672278 conn(0x7fa670067bb0 msgr2=0x7fa6700cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:24.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 -- 172.21.15.111:0/3471672278 shutdown_connections 2023-12-19T09:02:24.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.192+0000 7fa677323700 1 -- 172.21.15.111:0/3471672278 wait complete. 2023-12-19T09:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 Processor -- start 2023-12-19T09:02:24.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 -- start start 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6700cb500 0x7fa670154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 0x7fa6700ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700b0320 con 0x7fa67006bad0 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700b1180 con 0x7fa6700cb500 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa677323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700b0570 con 0x7fa6701580b0 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa675b20700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6700cb500 0x7fa670154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59778/0 (socket says 172.21.15.111:59778) 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 -- 172.21.15.111:0/2376171545 learned_addr learned my addr 172.21.15.111:0/2376171545 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:24.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676b22700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 0x7fa6700ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 -- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 msgr2=0x7fa6700ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 0x7fa6700ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 -- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6700cb500 msgr2=0x7fa670154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6700cb500 0x7fa670154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676321700 1 -- 172.21.15.111:0/2376171545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640094f0 con 0x7fa67006bad0 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.196+0000 7fa676b22700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 0x7fa6700ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:24.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa676321700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c5c0 tx=0x7fa66c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c011590 con 0x7fa67006bad0 2023-12-19T09:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c011710 con 0x7fa67006bad0 2023-12-19T09:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c011ad0 con 0x7fa67006bad0 2023-12-19T09:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6700b0850 con 0x7fa67006bad0 2023-12-19T09:02:24.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6700b0e10 con 0x7fa67006bad0 2023-12-19T09:02:24.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.200+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6700c8200 con 0x7fa67006bad0 2023-12-19T09:02:24.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.212+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fa66c01b3b0 con 0x7fa67006bad0 2023-12-19T09:02:24.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.216+0000 7fa6627fc700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 0x7fa654062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.216+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fa66c07c800 con 0x7fa67006bad0 2023-12-19T09:02:24.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.216+0000 7fa675b20700 1 -- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 msgr2=0x7fa654062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:24.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.216+0000 7fa675b20700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 0x7fa654062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:24.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.220+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa66c04c7b0 con 0x7fa67006bad0 2023-12-19T09:02:24.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:24.287+0000 7fd3a4327000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:02:24.357 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:24.352+0000 7f35b8fb1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:02:24.368 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:24.363+0000 7fd3a4327000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:02:24.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.396+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa67004be40 con 0x7fa67006bad0 2023-12-19T09:02:24.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.396+0000 7fa6627fc700 1 -- 172.21.15.111:0/2376171545 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa66c0170a0 con 0x7fa67006bad0 2023-12-19T09:02:24.403 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:02:24.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 msgr2=0x7fa654062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 0x7fa654062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 msgr2=0x7fa6700b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00c5c0 tx=0x7fa66c00c8d0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 shutdown_connections 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa6540606f0 0x7fa654062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6700cb500 0x7fa670154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6701580b0 0x7fa6700ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 --2- 172.21.15.111:0/2376171545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa67006bad0 0x7fa6700b1610 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 >> 172.21.15.111:0/2376171545 conn(0x7fa670067bb0 msgr2=0x7fa6700c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 shutdown_connections 2023-12-19T09:02:24.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.400+0000 7fa677323700 1 -- 172.21.15.111:0/2376171545 wait complete. 2023-12-19T09:02:24.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T09:02:24.445 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:24.440+0000 7f35b8fb1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:02:24.451 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:24.447+0000 7fd3a4327000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:02:24.542 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:24.536+0000 7f35b8fb1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:02:24.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.568+0000 7fa66182e700 1 Processor -- start 2023-12-19T09:02:24.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.568+0000 7fa66182e700 1 -- start start 2023-12-19T09:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa66182e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa65c156a10 0x7fa65c0cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa66182e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa66182e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c0ac620 con 0x7fa65c156de0 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa66182e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c0ac060 con 0x7fa65c156a10 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa66182e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0ac340 con 0x7fa65c06ac90 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa65b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa65affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa65c156a10 0x7fa65c0cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa65b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:36606/0 (socket says 172.21.15.111:36606) 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa65b7fe700 1 -- 172.21.15.111:0/2408387886 learned_addr learned my addr 172.21.15.111:0/2408387886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa65b7fe700 1 -- 172.21.15.111:0/2408387886 >> v2:172.21.15.143:3300/0 conn(0x7fa65c156a10 msgr2=0x7fa65c0cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.572+0000 7fa659ffb700 1 -- 172.21.15.111:0/2408387886 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295269464 0 0) 0x7fa65c0ac620 con 0x7fa65c156de0 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa65b7fe700 1 --2- 172.21.15.111:0/2408387886 >> v2:172.21.15.143:3300/0 conn(0x7fa65c156a10 0x7fa65c0cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa65b7fe700 1 -- 172.21.15.111:0/2408387886 >> v1:172.21.15.143:6789/0 conn(0x7fa65c156de0 legacy=0x7fa65c06a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa65b7fe700 1 -- 172.21.15.111:0/2408387886 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c06c860 con 0x7fa65c06ac90 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa65b7fe700 1 --2- 172.21.15.111:0/2408387886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa654009f20 tx=0x7fa654009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db490e5bc57aac04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa659ffb700 1 -- 172.21.15.111:0/2408387886 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65400e070 con 0x7fa65c06ac90 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa659ffb700 1 -- 172.21.15.111:0/2408387886 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65400b9d0 con 0x7fa65c06ac90 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa659ffb700 1 -- 172.21.15.111:0/2408387886 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540092e0 con 0x7fa65c06ac90 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 -- 172.21.15.111:0/2408387886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 msgr2=0x7fa65c06d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 --2- 172.21.15.111:0/2408387886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa654009f20 tx=0x7fa654009840 comp rx=0 tx=0).stop 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 -- 172.21.15.111:0/2408387886 shutdown_connections 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 --2- 172.21.15.111:0/2408387886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c06ac90 0x7fa65c06d060 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 --2- 172.21.15.111:0/2408387886 >> v2:172.21.15.143:3300/0 conn(0x7fa65c156a10 0x7fa65c0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 -- 172.21.15.111:0/2408387886 >> 172.21.15.111:0/2408387886 conn(0x7fa65c067bb0 msgr2=0x7fa65c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 -- 172.21.15.111:0/2408387886 shutdown_connections 2023-12-19T09:02:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.576+0000 7fa66182e700 1 -- 172.21.15.111:0/2408387886 wait complete. 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 Processor -- start 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 -- start start 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa65c156a10 0x7fa65c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 0x7fa65c0c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c151fa0 con 0x7fa65c156a10 2023-12-19T09:02:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0a4440 con 0x7fa65c06ac90 2023-12-19T09:02:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa66182e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c169bb0 con 0x7fa65c0c6a30 2023-12-19T09:02:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa65affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa65affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38246/0 (socket says 172.21.15.111:38246) 2023-12-19T09:02:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa65affd700 1 -- 172.21.15.111:0/2277265348 learned_addr learned my addr 172.21.15.111:0/2277265348 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa65b7fe700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 0x7fa65c0c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.580+0000 7fa65a7fc700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa65c156a10 0x7fa65c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 -- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 msgr2=0x7fa65c0c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 0x7fa65c0c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 -- 172.21.15.111:0/2277265348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa65c156a10 msgr2=0x7fa65c151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa65c156a10 0x7fa65c151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 -- 172.21.15.111:0/2277265348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa654009ae0 con 0x7fa65c06ac90 2023-12-19T09:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65b7fe700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 0x7fa65c0c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa65affd700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00ecb0 tx=0x7fa64c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00c850 con 0x7fa65c06ac90 2023-12-19T09:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c00c9b0 con 0x7fa65c06ac90 2023-12-19T09:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00cd50 con 0x7fa65c06ac90 2023-12-19T09:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c0a4690 con 0x7fa65c06ac90 2023-12-19T09:02:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.584+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c0c6ed0 con 0x7fa65c06ac90 2023-12-19T09:02:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c04be40 con 0x7fa65c06ac90 2023-12-19T09:02:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fa64c013680 con 0x7fa65c06ac90 2023-12-19T09:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa642ffd700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 0x7fa648062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fa64c07cf60 con 0x7fa65c06ac90 2023-12-19T09:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa65a7fc700 1 -- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 msgr2=0x7fa648062bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.588+0000 7fa65a7fc700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 0x7fa648062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:24.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.596+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa64c04cfb0 con 0x7fa65c06ac90 2023-12-19T09:02:24.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.776+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa65c06cb40 con 0x7fa65c06ac90 2023-12-19T09:02:24.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.776+0000 7fa642ffd700 1 -- 172.21.15.111:0/2277265348 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa64c0505d0 con 0x7fa65c06ac90 2023-12-19T09:02:24.782 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T09:02:24.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 msgr2=0x7fa648062bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:24.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 0x7fa648062bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 msgr2=0x7fa65c159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:24.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00ecb0 tx=0x7fa64c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 shutdown_connections 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa648060740 0x7fa648062bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa65c0c6a30 0x7fa65c0c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa65c156a10 0x7fa65c151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 --2- 172.21.15.111:0/2277265348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa65c06ac90 0x7fa65c159710 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:24.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 >> 172.21.15.111:0/2277265348 conn(0x7fa65c067bb0 msgr2=0x7fa65c155a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:24.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 shutdown_connections 2023-12-19T09:02:24.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:24.780+0000 7fa66182e700 1 -- 172.21.15.111:0/2277265348 wait complete. 2023-12-19T09:02:24.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T09:02:24.829 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:24.823+0000 7fd3a4327000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:02:24.931 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:24.924+0000 7f35b8fb1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:02:24.991 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:24.987+0000 7fd3a4327000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:02:25.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 Processor -- start 2023-12-19T09:02:25.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 -- start start 2023-12-19T09:02:25.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2e8159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e806c850 con 0x7fe2e806aeb0 2023-12-19T09:02:25.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e80cb650 con 0x7fe2e8158c90 2023-12-19T09:02:25.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ed06b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e806b470 con 0x7fe2e806dcf0 2023-12-19T09:02:25.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2df7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe2e806aeb0 0x7fe2e8159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58156/0 (socket says 172.21.15.111:58156) 2023-12-19T09:02:25.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2df7fe700 1 -- 172.21.15.111:0/145826542 learned_addr learned my addr 172.21.15.111:0/145826542 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:25.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977236617 0 0) 0x7fe2e806c850 con 0x7fe2e806aeb0 2023-12-19T09:02:25.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d4003570 con 0x7fe2e806aeb0 2023-12-19T09:02:25.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2ec86a700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2e8159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075184071 0 0) 0x7fe2d4003570 con 0x7fe2e806aeb0 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 msgr2=0x7fe2e8159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2e8159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 >> v1:172.21.15.143:6789/0 conn(0x7fe2e8158c90 legacy=0x7fe2e80caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e806b750 con 0x7fe2e806aeb0 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fe2d0001470 con 0x7fe2e806aeb0 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 >> v1:172.21.15.111:6789/0 conn(0x7fe2e806aeb0 legacy=0x7fe2e8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2d4003a90 0x7fe2d4003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4004810 0x7fe2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2d4005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e806b470 con 0x7fe2d4003a90 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d400bc20 con 0x7fe2d4004810 2023-12-19T09:02:25.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d400bde0 con 0x7fe2e806dcf0 2023-12-19T09:02:25.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.012+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2d4003a90 0x7fe2d4003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4004810 0x7fe2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ec86a700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2d4005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 -- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 msgr2=0x7fe2d4005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e806dcf0 0x7fe2d4005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 -- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4004810 msgr2=0x7fe2d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4004810 0x7fe2d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 -- 172.21.15.111:0/145826542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d4003570 con 0x7fe2d4003a90 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4004810 0x7fe2d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2d4003a90 0x7fe2d4003ea0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001880 tx=0x7fe2d8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccc934c22863d00 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8012420 con 0x7fe2d4003a90 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8012580 con 0x7fe2d4003a90 2023-12-19T09:02:25.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2e77fe700 1 -- 172.21.15.111:0/145826542 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8012920 con 0x7fe2d4003a90 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 -- 172.21.15.111:0/145826542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2d4003a90 msgr2=0x7fe2d4003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/145826542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2d4003a90 0x7fe2d4003ea0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001880 tx=0x7fe2d8007d30 comp rx=0 tx=0).stop 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ec86a700 1 -- 172.21.15.111:0/145826542 reap_dead start 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 -- 172.21.15.111:0/145826542 shutdown_connections 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 -- 172.21.15.111:0/145826542 >> 172.21.15.111:0/145826542 conn(0x7fe2e8067bb0 msgr2=0x7fe2e80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 -- 172.21.15.111:0/145826542 shutdown_connections 2023-12-19T09:02:25.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.016+0000 7fe2ed06b700 1 -- 172.21.15.111:0/145826542 wait complete. 2023-12-19T09:02:25.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 Processor -- start 2023-12-19T09:02:25.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 -- start start 2023-12-19T09:02:25.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4003a90 0x7fe2e80c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 0x7fe2e8152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e81561f0 con 0x7fe2e80c9290 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80c7710 con 0x7fe2d4003a90 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ed06b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80c79b0 con 0x7fe2e81536c0 2023-12-19T09:02:25.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ec86a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ec86a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59812/0 (socket says 172.21.15.111:59812) 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4003a90 0x7fe2e80c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2ec86a700 1 -- 172.21.15.111:0/3118252882 learned_addr learned my addr 172.21.15.111:0/3118252882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.020+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 0x7fe2e8152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 -- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 msgr2=0x7fe2e8152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 0x7fe2e8152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 -- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4003a90 msgr2=0x7fe2e80c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4003a90 0x7fe2e80c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 -- 172.21.15.111:0/3118252882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80c7c90 con 0x7fe2e80c9290 2023-12-19T09:02:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 0x7fe2e8152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ec86a700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe2e806e830 tx=0x7fe2e0009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0002ce0 con 0x7fe2e80c9290 2023-12-19T09:02:25.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e000f050 con 0x7fe2e80c9290 2023-12-19T09:02:25.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e8155ba0 con 0x7fe2e80c9290 2023-12-19T09:02:25.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0014660 con 0x7fe2e80c9290 2023-12-19T09:02:25.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e80c9c40 con 0x7fe2e80c9290 2023-12-19T09:02:25.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.024+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e81563a0 con 0x7fe2e80c9290 2023-12-19T09:02:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.028+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fe2e0003040 con 0x7fe2e80c9290 2023-12-19T09:02:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.028+0000 7fe2e4ff9700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 0x7fe2c8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.028+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fe2e007d210 con 0x7fe2e80c9290 2023-12-19T09:02:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.028+0000 7fe2e7fff700 1 -- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 msgr2=0x7fe2c8062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:25.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.028+0000 7fe2e7fff700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 0x7fe2c8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:25.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.036+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2e81563a0 con 0x7fe2e80c9290 2023-12-19T09:02:25.089 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:25.084+0000 7f35b8fb1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:02:25.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.196+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2e80c7e20 con 0x7fe2e80c9290 2023-12-19T09:02:25.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.200+0000 7fe2e4ff9700 1 -- 172.21.15.111:0/3118252882 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2e80c7e20 con 0x7fe2e80c9290 2023-12-19T09:02:25.206 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T09:02:25.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 msgr2=0x7fe2c8062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:25.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 0x7fe2c8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 msgr2=0x7fe2e80c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe2e806e830 tx=0x7fe2e0009870 comp rx=0 tx=0).stop 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 shutdown_connections 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2c80606a0 0x7fe2c8062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e80c9290 0x7fe2e80c9700 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e81536c0 0x7fe2e8152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 --2- 172.21.15.111:0/3118252882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2d4003a90 0x7fe2e80c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 >> 172.21.15.111:0/3118252882 conn(0x7fe2e8067bb0 msgr2=0x7fe2e80cf1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 shutdown_connections 2023-12-19T09:02:25.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.204+0000 7fe2ed06b700 1 -- 172.21.15.111:0/3118252882 wait complete. 2023-12-19T09:02:25.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:25.337 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:25.331+0000 7fd3a4327000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:02:25.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.384+0000 7f376a4e3700 1 Processor -- start 2023-12-19T09:02:25.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.384+0000 7f376a4e3700 1 -- start start 2023-12-19T09:02:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f376a4e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f376415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f376a4e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f37640c7f00 0x7f37640aca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f376a4e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376406d990 con 0x7f376406de20 2023-12-19T09:02:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f376a4e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376415a570 con 0x7f37640c7f00 2023-12-19T09:02:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f376a4e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376406cd80 con 0x7f3764155960 2023-12-19T09:02:25.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3768ce0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f376406de20 0x7f37640c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58164/0 (socket says 172.21.15.111:58164) 2023-12-19T09:02:25.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3768ce0700 1 -- 172.21.15.111:0/24790204 learned_addr learned my addr 172.21.15.111:0/24790204 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:25.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3769ce2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f37640c7f00 0x7f37640aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f37694e1700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f376415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3769ce2700 1 -- 172.21.15.111:0/24790204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 msgr2=0x7f376415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3769ce2700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f376415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3769ce2700 1 -- 172.21.15.111:0/24790204 >> v1:172.21.15.111:6789/0 conn(0x7f376406de20 legacy=0x7f37640c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.388+0000 7f3769ce2700 1 -- 172.21.15.111:0/24790204 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376406d060 con 0x7f37640c7f00 2023-12-19T09:02:25.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f3769ce2700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f37640aca10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f375c00b9a0 tx=0x7f375c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31f5ff4352f53a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f3763fff700 1 -- 172.21.15.111:0/24790204 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c00be90 con 0x7f37640c7f00 2023-12-19T09:02:25.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f3763fff700 1 -- 172.21.15.111:0/24790204 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c00ca60 con 0x7f37640c7f00 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f3763fff700 1 -- 172.21.15.111:0/24790204 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c00ce20 con 0x7f37640c7f00 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- 172.21.15.111:0/24790204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 msgr2=0x7f37640aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f37640aca10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f375c00b9a0 tx=0x7f375c0090f0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- 172.21.15.111:0/24790204 shutdown_connections 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f37640aca10 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 --2- 172.21.15.111:0/24790204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f376415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- 172.21.15.111:0/24790204 >> 172.21.15.111:0/24790204 conn(0x7f3764067bb0 msgr2=0x7f37640cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- 172.21.15.111:0/24790204 shutdown_connections 2023-12-19T09:02:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- 172.21.15.111:0/24790204 wait complete. 2023-12-19T09:02:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 Processor -- start 2023-12-19T09:02:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.392+0000 7f376a4e3700 1 -- start start 2023-12-19T09:02:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376406de20 0x7f37641524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f37640afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376414de20 con 0x7f37640c7f00 2023-12-19T09:02:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764169a70 con 0x7f376406de20 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376414c830 con 0x7f3764155960 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59830/0 (socket says 172.21.15.111:59830) 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 -- 172.21.15.111:0/2797471106 learned_addr learned my addr 172.21.15.111:0/2797471106 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3769ce2700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f37640afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f37694e1700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376406de20 0x7f37641524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 msgr2=0x7f37640afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f37640afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376406de20 msgr2=0x7f37641524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376406de20 0x7f37641524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 -- 172.21.15.111:0/2797471106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c00b4e0 con 0x7f37640c7f00 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f3768ce0700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f375000c7c0 tx=0x7f3750007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750005b30 con 0x7f37640c7f00 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3750005c90 con 0x7f37640c7f00 2023-12-19T09:02:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750004a80 con 0x7f37640c7f00 2023-12-19T09:02:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376414cb10 con 0x7f37640c7f00 2023-12-19T09:02:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376414d0d0 con 0x7f37640c7f00 2023-12-19T09:02:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.396+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376414d2f0 con 0x7f37640c7f00 2023-12-19T09:02:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.400+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f37500070d0 con 0x7f37640c7f00 2023-12-19T09:02:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.400+0000 7f37617fa700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 0x7f3748062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.400+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7f375007c280 con 0x7f37640c7f00 2023-12-19T09:02:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.400+0000 7f37694e1700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 msgr2=0x7f3748062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.400+0000 7f37694e1700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 0x7f3748062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:25.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.404+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f375004c1b0 con 0x7f37640c7f00 2023-12-19T09:02:25.486 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:25.480+0000 7f35b8fb1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:02:25.561 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:25.555+0000 7fd3a4327000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.604+0000 7f37694e1700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 msgr2=0x7f3748062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.604+0000 7f37694e1700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 0x7f3748062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:25.649 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:25.643+0000 7fd3a4327000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:02:25.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.696+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f376404be40 con 0x7f37640c7f00 2023-12-19T09:02:25.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.700+0000 7f37617fa700 1 -- 172.21.15.111:0/2797471106 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7f375004f7d0 con 0x7f37640c7f00 2023-12-19T09:02:25.705 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:25.705 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T09:02:25.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 msgr2=0x7f3748062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:25.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 0x7f3748062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 msgr2=0x7f3764169530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f375000c7c0 tx=0x7f3750007f70 comp rx=0 tx=0).stop 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 shutdown_connections 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3748060670 0x7f3748062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3764155960 0x7f37640afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37640c7f00 0x7f3764169530 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 --2- 172.21.15.111:0/2797471106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f376406de20 0x7f37641524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 >> 172.21.15.111:0/2797471106 conn(0x7f3764067bb0 msgr2=0x7f376406a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 shutdown_connections 2023-12-19T09:02:25.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.704+0000 7f376a4e3700 1 -- 172.21.15.111:0/2797471106 wait complete. 2023-12-19T09:02:25.711 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T09:02:25.721 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:25.715+0000 7fd3a4327000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:02:25.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:25.768 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:25.764+0000 7f35b8fb1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:25.798 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:25.795+0000 7fd3a4327000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:02:25.868 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:25.864+0000 7f35b8fb1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:02:25.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 Processor -- start 2023-12-19T09:02:25.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 -- start start 2023-12-19T09:02:25.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 0x7ffb680c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb680afc80 0x7ffb680c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680ca650 con 0x7ffb680c8930 2023-12-19T09:02:25.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb680c8db0 con 0x7ffb680afc80 2023-12-19T09:02:25.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb6ec2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68156700 con 0x7ffb68153860 2023-12-19T09:02:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb67fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb680afc80 0x7ffb680c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb67fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb680afc80 0x7ffb680c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59836/0 (socket says 172.21.15.111:59836) 2023-12-19T09:02:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.888+0000 7ffb67fff700 1 -- 172.21.15.111:0/1034500943 learned_addr learned my addr 172.21.15.111:0/1034500943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:25.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6c9ca700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 0x7ffb680c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb67fff700 1 -- 172.21.15.111:0/1034500943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 msgr2=0x7ffb680c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb67fff700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 0x7ffb680c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb67fff700 1 -- 172.21.15.111:0/1034500943 >> v1:172.21.15.143:6789/0 conn(0x7ffb680c8930 legacy=0x7ffb681599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb67fff700 1 -- 172.21.15.111:0/1034500943 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb681569e0 con 0x7ffb680afc80 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6c9ca700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 0x7ffb680c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:25.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb67fff700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680afc80 0x7ffb680c83f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ffb540099b0 tx=0x7ffb54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d73e70d92bebcfa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb677fe700 1 -- 172.21.15.111:0/1034500943 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400b820 con 0x7ffb680afc80 2023-12-19T09:02:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb677fe700 1 -- 172.21.15.111:0/1034500943 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5400b980 con 0x7ffb680afc80 2023-12-19T09:02:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb677fe700 1 -- 172.21.15.111:0/1034500943 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400bd20 con 0x7ffb680afc80 2023-12-19T09:02:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/1034500943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680afc80 msgr2=0x7ffb680c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680afc80 0x7ffb680c83f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ffb540099b0 tx=0x7ffb54009250 comp rx=0 tx=0).stop 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/1034500943 shutdown_connections 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb680afc80 0x7ffb680c83f0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/1034500943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb68153860 0x7ffb680c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/1034500943 >> 172.21.15.111:0/1034500943 conn(0x7ffb680cd8e0 msgr2=0x7ffb680c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/1034500943 shutdown_connections 2023-12-19T09:02:25.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.892+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/1034500943 wait complete. 2023-12-19T09:02:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 Processor -- start 2023-12-19T09:02:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 -- start start 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb680afc80 0x7ffb68158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 0x7ffb6816dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68159210 con 0x7ffb68153860 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68157c30 con 0x7ffb680c8930 2023-12-19T09:02:25.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6ec2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68157f10 con 0x7ffb680afc80 2023-12-19T09:02:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6d1cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb67fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 0x7ffb6816dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6c9ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb680afc80 0x7ffb68158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6d1cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59838/0 (socket says 172.21.15.111:59838) 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.896+0000 7ffb6d1cb700 1 -- 172.21.15.111:0/2270627778 learned_addr learned my addr 172.21.15.111:0/2270627778 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 -- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb680afc80 msgr2=0x7ffb68158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb680afc80 0x7ffb68158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 -- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 msgr2=0x7ffb6816dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 0x7ffb6816dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 -- 172.21.15.111:0/2270627778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb540094f0 con 0x7ffb68153860 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb67fff700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 0x7ffb6816dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6d1cb700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c930 tx=0x7ffb600026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60012630 con 0x7ffb68153860 2023-12-19T09:02:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb600127b0 con 0x7ffb68153860 2023-12-19T09:02:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681581f0 con 0x7ffb68153860 2023-12-19T09:02:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60012b70 con 0x7ffb68153860 2023-12-19T09:02:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681587b0 con 0x7ffb68153860 2023-12-19T09:02:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.900+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68150230 con 0x7ffb68153860 2023-12-19T09:02:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.904+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7ffb6001b400 con 0x7ffb68153860 2023-12-19T09:02:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.904+0000 7ffb64ff9700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 0x7ffb4c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.904+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7ffb6007cce0 con 0x7ffb68153860 2023-12-19T09:02:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.904+0000 7ffb6c9ca700 1 -- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 msgr2=0x7ffb4c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.904+0000 7ffb6c9ca700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 0x7ffb4c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:25.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:25.908+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb68150230 con 0x7ffb68153860 2023-12-19T09:02:25.956 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:25.948+0000 7f35b8fb1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:02:26.014 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:26.011+0000 7fd3a4327000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:26.042 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.036+0000 7f35b8fb1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:02:26.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.084+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb68150230 con 0x7ffb68153860 2023-12-19T09:02:26.106 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:26.087+0000 7fd3a4327000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:02:26.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb64ff9700 1 -- 172.21.15.111:0/2270627778 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+17115 (secure 0 0 0) 0x7ffb68150230 con 0x7ffb68153860 2023-12-19T09:02:26.107 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":149,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:24.446422+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":149,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3383118528":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/4020762991":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/3429689930":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/415557684":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2445077517":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6833/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6832/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2762275584":"2023-12-20T08:06:37.094770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":18,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 msgr2=0x7ffb4c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 0x7ffb4c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 msgr2=0x7ffb6814e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c930 tx=0x7ffb600026e0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 shutdown_connections 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ffb4c060670 0x7ffb4c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb680c8930 0x7ffb6816dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68153860 0x7ffb6814e2c0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 --2- 172.21.15.111:0/2270627778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb680afc80 0x7ffb68158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 >> 172.21.15.111:0/2270627778 conn(0x7ffb680cd8e0 msgr2=0x7ffb68150690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.088+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 shutdown_connections 2023-12-19T09:02:26.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.092+0000 7ffb6ec2e700 1 -- 172.21.15.111:0/2270627778 wait complete. 2023-12-19T09:02:26.118 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:02:26.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:26.177 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:26.171+0000 7fd3a4327000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:02:26.283 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.276+0000 7f35b8fb1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:26.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 Processor -- start 2023-12-19T09:02:26.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 -- start start 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea800c7190 con 0x7fea800aeba0 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea800c1b50 con 0x7fea8015d320 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea87c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea800c1e30 con 0x7fea800c15b0 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea86215700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fea800aeba0 0x7fea800680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58178/0 (socket says 172.21.15.111:58178) 2023-12-19T09:02:26.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea86215700 1 -- 172.21.15.111:0/3147462499 learned_addr learned my addr 172.21.15.111:0/3147462499 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:26.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.324+0000 7fea85213700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90290620 0 0) 0x7fea800c7190 con 0x7fea800aeba0 2023-12-19T09:02:26.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003570 con 0x7fea800aeba0 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281720185 0 0) 0x7fea6c003570 con 0x7fea800aeba0 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 msgr2=0x7fea8015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea8015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 >> v1:172.21.15.143:6789/0 conn(0x7fea8015d320 legacy=0x7fea8006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800c6e20 con 0x7fea800aeba0 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fea780017f0 con 0x7fea800aeba0 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 >> v1:172.21.15.111:6789/0 conn(0x7fea800aeba0 legacy=0x7fea800680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c003a90 0x7fea6c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea6c004810 0x7fea6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea800c1e30 con 0x7fea6c003a90 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c00bb10 con 0x7fea6c004810 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c00bcd0 con 0x7fea800c15b0 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c003a90 0x7fea6c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85213700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea6c004810 0x7fea6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea86215700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 -- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 msgr2=0x7fea6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 -- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea6c004810 msgr2=0x7fea6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea6c004810 0x7fea6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 -- 172.21.15.111:0/3147462499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c003570 con 0x7fea6c003a90 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea86215700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea800c15b0 0x7fea6c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:26.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea85a14700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c003a90 0x7fea6c003e60 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0019a0 tx=0x7fea7c007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee6bd5e419df06b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:26.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00c5d0 con 0x7fea6c003a90 2023-12-19T09:02:26.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c00c730 con 0x7fea6c003a90 2023-12-19T09:02:26.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea84a12700 1 -- 172.21.15.111:0/3147462499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00cad0 con 0x7fea6c003a90 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea87c78700 1 -- 172.21.15.111:0/3147462499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c003a90 msgr2=0x7fea6c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea87c78700 1 --2- 172.21.15.111:0/3147462499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c003a90 0x7fea6c003e60 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0019a0 tx=0x7fea7c007d10 comp rx=0 tx=0).stop 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea86215700 1 -- 172.21.15.111:0/3147462499 reap_dead start 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea87c78700 1 -- 172.21.15.111:0/3147462499 shutdown_connections 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.328+0000 7fea87c78700 1 -- 172.21.15.111:0/3147462499 >> 172.21.15.111:0/3147462499 conn(0x7fea800c8850 msgr2=0x7fea800c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.332+0000 7fea87c78700 1 -- 172.21.15.111:0/3147462499 shutdown_connections 2023-12-19T09:02:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.332+0000 7fea87c78700 1 -- 172.21.15.111:0/3147462499 wait complete. 2023-12-19T09:02:26.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.332+0000 7fea87c78700 1 Processor -- start 2023-12-19T09:02:26.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.332+0000 7fea87c78700 1 -- start start 2023-12-19T09:02:26.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea6c003a90 0x7fea8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea8015c3b0 0x7fea80153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8015dc30 con 0x7fea8015bf40 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea800b18b0 con 0x7fea8015c3b0 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea800c4300 con 0x7fea6c003a90 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59858/0 (socket says 172.21.15.111:59858) 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 -- 172.21.15.111:0/734484308 learned_addr learned my addr 172.21.15.111:0/734484308 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:26.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85a14700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea6c003a90 0x7fea8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea86215700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea8015c3b0 0x7fea80153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea6c003a90 msgr2=0x7fea8015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea6c003a90 0x7fea8015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea8015c3b0 msgr2=0x7fea80153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea8015c3b0 0x7fea80153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 -- 172.21.15.111:0/734484308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800c45e0 con 0x7fea8015bf40 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea85213700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fea70011990 tx=0x7fea70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:26.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70015670 con 0x7fea8015bf40 2023-12-19T09:02:26.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea700157d0 con 0x7fea8015bf40 2023-12-19T09:02:26.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea800c48c0 con 0x7fea8015bf40 2023-12-19T09:02:26.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70015b90 con 0x7fea8015bf40 2023-12-19T09:02:26.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.336+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea800b0b40 con 0x7fea8015bf40 2023-12-19T09:02:26.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea80071120 con 0x7fea8015bf40 2023-12-19T09:02:26.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fea70013410 con 0x7fea8015bf40 2023-12-19T09:02:26.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea75ffb700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 0x7fea64062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fea7007d460 con 0x7fea8015bf40 2023-12-19T09:02:26.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea85a14700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 msgr2=0x7fea64062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:26.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.340+0000 7fea85a14700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 0x7fea64062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:26.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.352+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fea7004d430 con 0x7fea8015bf40 2023-12-19T09:02:26.360 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:26.355+0000 7fd3a4327000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:02:26.364 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.360+0000 7f35b8fb1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:02:26.442 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:26.439+0000 7fd3a4327000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:02:26.462 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.456+0000 7f35b8fb1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:02:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.544+0000 7fea85a14700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 msgr2=0x7fea64062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.544+0000 7fea85a14700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 0x7fea64062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.564+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea800642a0 con 0x7fea8015bf40 2023-12-19T09:02:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.564+0000 7fea75ffb700 1 -- 172.21.15.111:0/734484308 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7fea70050a50 con 0x7fea8015bf40 2023-12-19T09:02:26.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:26.570 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T09:02:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 msgr2=0x7fea64062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 0x7fea64062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 msgr2=0x7fea801537d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fea70011990 tx=0x7fea70008040 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 shutdown_connections 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fea640606a0 0x7fea64062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea8015c3b0 0x7fea80153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea8015bf40 0x7fea801537d0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 --2- 172.21.15.111:0/734484308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea6c003a90 0x7fea8015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 >> 172.21.15.111:0/734484308 conn(0x7fea800c8850 msgr2=0x7fea80160930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 shutdown_connections 2023-12-19T09:02:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.568+0000 7fea87c78700 1 -- 172.21.15.111:0/734484308 wait complete. 2023-12-19T09:02:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T09:02:26.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:26.667 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.660+0000 7f35b8fb1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:02:26.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.748+0000 7fe2efaae700 1 Processor -- start 2023-12-19T09:02:26.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.748+0000 7fe2efaae700 1 -- start start 2023-12-19T09:02:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2efaae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2e8155b00 0x7fe2e806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2efaae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 0x7fe2e815a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2efaae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e80ca3d0 con 0x7fe2e8155ed0 2023-12-19T09:02:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2efaae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e806e4c0 con 0x7fe2e8155b00 2023-12-19T09:02:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2efaae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e806e7a0 con 0x7fe2e80c87b0 2023-12-19T09:02:26.758 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:26.752+0000 7f35b8fb1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed049700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe2e8155ed0 0x7fe2e80c80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58186/0 (socket says 172.21.15.111:58186) 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed049700 1 -- 172.21.15.111:0/413937064 learned_addr learned my addr 172.21.15.111:0/413937064 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/413937064 >> v2:172.21.15.143:3300/0 conn(0x7fe2e8155b00 0x7fe2e806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ee04b700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 0x7fe2e815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ec848700 1 -- 172.21.15.111:0/413937064 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213571975 0 0) 0x7fe2e80ca3d0 con 0x7fe2e8155ed0 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ec848700 1 -- 172.21.15.111:0/413937064 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d0003570 con 0x7fe2e8155ed0 2023-12-19T09:02:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 -- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 msgr2=0x7fe2e815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 0x7fe2e815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 -- 172.21.15.111:0/413937064 >> v1:172.21.15.111:6789/0 conn(0x7fe2e8155ed0 legacy=0x7fe2e80c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 -- 172.21.15.111:0/413937064 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e806e0e0 con 0x7fe2e8155b00 2023-12-19T09:02:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ee04b700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 0x7fe2e815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:26.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e8155b00 0x7fe2e806c7c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80098e0 tx=0x7fe2d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7c17eea81e94e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ec848700 1 -- 172.21.15.111:0/413937064 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8011630 con 0x7fe2e8155b00 2023-12-19T09:02:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ec848700 1 -- 172.21.15.111:0/413937064 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8011790 con 0x7fe2e8155b00 2023-12-19T09:02:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.752+0000 7fe2ec848700 1 -- 172.21.15.111:0/413937064 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8011b30 con 0x7fe2e8155b00 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e8155b00 msgr2=0x7fe2e806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e8155b00 0x7fe2e806c7c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80098e0 tx=0x7fe2d80091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- 172.21.15.111:0/413937064 shutdown_connections 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e80c87b0 0x7fe2e815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 --2- 172.21.15.111:0/413937064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e8155b00 0x7fe2e806c7c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- 172.21.15.111:0/413937064 >> 172.21.15.111:0/413937064 conn(0x7fe2e80c90f0 msgr2=0x7fe2e80c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- 172.21.15.111:0/413937064 shutdown_connections 2023-12-19T09:02:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- 172.21.15.111:0/413937064 wait complete. 2023-12-19T09:02:26.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 Processor -- start 2023-12-19T09:02:26.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.756+0000 7fe2efaae700 1 -- start start 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 0x7fe2e8157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e8155b00 0x7fe2e81601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8157c50 con 0x7fe2e8155ed0 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8157f30 con 0x7fe2e80c87b0 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8158210 con 0x7fe2e8155b00 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59862/0 (socket says 172.21.15.111:59862) 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 -- 172.21.15.111:0/1695045147 learned_addr learned my addr 172.21.15.111:0/1695045147 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ed049700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e8155b00 0x7fe2e81601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 0x7fe2e8157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e8155b00 msgr2=0x7fe2e81601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e8155b00 0x7fe2e81601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 msgr2=0x7fe2e8157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 0x7fe2e8157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 -- 172.21.15.111:0/1695045147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e0009a20 con 0x7fe2e8155ed0 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 0x7fe2e8157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2ee04b700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009f60 tx=0x7fe2e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000ccd0 con 0x7fe2e8155ed0 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e000f040 con 0x7fe2e8155ed0 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d8009420 con 0x7fe2e8155ed0 2023-12-19T09:02:26.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00146d0 con 0x7fe2e8155ed0 2023-12-19T09:02:26.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.760+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e81587d0 con 0x7fe2e8155ed0 2023-12-19T09:02:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8169100 con 0x7fe2e8155ed0 2023-12-19T09:02:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 66) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fe2e0014a40 con 0x7fe2e8155ed0 2023-12-19T09:02:26.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2e5ffb700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 0x7fe2d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 7005+0+0 (secure 0 0 0) 0x7fe2e007de30 con 0x7fe2e8155ed0 2023-12-19T09:02:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2ed84a700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 msgr2=0x7fe2d4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:26.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.764+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 0x7fe2d4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:26.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.772+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2e004dc50 con 0x7fe2e8155ed0 2023-12-19T09:02:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.968+0000 7fe2ed84a700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 msgr2=0x7fe2d4062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.968+0000 7fe2ed84a700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 0x7fe2d4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:26.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.984+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2e8071120 con 0x7fe2e8155ed0 2023-12-19T09:02:26.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.984+0000 7fe2e5ffb700 1 -- 172.21.15.111:0/1695045147 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4608 (secure 0 0 0) 0x7fe2e0018020 con 0x7fe2e8155ed0 2023-12-19T09:02:26.990 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:26.990 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T08:06:22.717187+0000","modified":"2023-12-19T08:06:26.311289+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8394":{"gid":8394,"name":"c","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/794754877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":794754877},{"type":"v1","addr":"172.21.15.111:6835","nonce":794754877}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 msgr2=0x7fe2d4062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 0x7fe2d4062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 msgr2=0x7fe2e8145ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009f60 tx=0x7fe2e000b040 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 shutdown_connections 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe2d4060670 0x7fe2d4062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e8155ed0 0x7fe2e8145ba0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2e8155b00 0x7fe2e81601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 --2- 172.21.15.111:0/1695045147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2e80c87b0 0x7fe2e8157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 >> 172.21.15.111:0/1695045147 conn(0x7fe2e80c90f0 msgr2=0x7fe2e80adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:26.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 shutdown_connections 2023-12-19T09:02:26.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:26.988+0000 7fe2efaae700 1 -- 172.21.15.111:0/1695045147 wait complete. 2023-12-19T09:02:26.995 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 63 2023-12-19T09:02:27.010 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:27.147 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.143+0000 7fd3a4327000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:02:27.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 Processor -- start 2023-12-19T09:02:27.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 -- start start 2023-12-19T09:02:27.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9094158e20 0x7f909406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909406b730 con 0x7f909415ca80 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909406ab20 con 0x7f9094158e20 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909ab23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909406ae00 con 0x7f90941591f0 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9094158e20 0x7f909406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9094158e20 0x7f909406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59866/0 (socket says 172.21.15.111:59866) 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 -- 172.21.15.111:0/337295660 learned_addr learned my addr 172.21.15.111:0/337295660 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f909a322700 1 --1- 172.21.15.111:0/337295660 >> v1:172.21.15.111:6789/0 conn(0x7f909415ca80 0x7f909406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58188/0 (socket says 172.21.15.111:58188) 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099320700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 -- 172.21.15.111:0/337295660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 msgr2=0x7f90940ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 -- 172.21.15.111:0/337295660 >> v1:172.21.15.111:6789/0 conn(0x7f909415ca80 legacy=0x7f909406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 -- 172.21.15.111:0/337295660 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909406b0e0 con 0x7f9094158e20 2023-12-19T09:02:27.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9098b1f700 1 -- 172.21.15.111:0/337295660 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256248045 0 0) 0x7f909406b730 con 0x7f909415ca80 2023-12-19T09:02:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099320700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9099b21700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9094158e20 0x7f909406a540 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f90900099b0 tx=0x7f9090009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ca002b8c7e1a40 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9098b1f700 1 -- 172.21.15.111:0/337295660 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000b860 con 0x7f9094158e20 2023-12-19T09:02:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9098b1f700 1 -- 172.21.15.111:0/337295660 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000b9c0 con 0x7f9094158e20 2023-12-19T09:02:27.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.168+0000 7f9098b1f700 1 -- 172.21.15.111:0/337295660 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000bd60 con 0x7f9094158e20 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- 172.21.15.111:0/337295660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9094158e20 msgr2=0x7f909406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9094158e20 0x7f909406a540 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f90900099b0 tx=0x7f9090009250 comp rx=0 tx=0).stop 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- 172.21.15.111:0/337295660 shutdown_connections 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 --2- 172.21.15.111:0/337295660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9094158e20 0x7f909406a540 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- 172.21.15.111:0/337295660 >> 172.21.15.111:0/337295660 conn(0x7f9094067bb0 msgr2=0x7f90940cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- 172.21.15.111:0/337295660 shutdown_connections 2023-12-19T09:02:27.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- 172.21.15.111:0/337295660 wait complete. 2023-12-19T09:02:27.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 Processor -- start 2023-12-19T09:02:27.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 -- start start 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.172+0000 7f909ab23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9094158e20 0x7f90940ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940c9500 con 0x7f909415ca80 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9094155260 con 0x7f9094158e20 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940c88f0 con 0x7f90941591f0 2023-12-19T09:02:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59868/0 (socket says 172.21.15.111:59868) 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 -- 172.21.15.111:0/2252363501 learned_addr learned my addr 172.21.15.111:0/2252363501 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f9099b21700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9094158e20 0x7f90940ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f9099320700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 -- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 msgr2=0x7f90940c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 -- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9094158e20 msgr2=0x7f90940ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9094158e20 0x7f90940ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 -- 172.21.15.111:0/2252363501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900094f0 con 0x7f909415ca80 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909a322700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f908c00ccc0 tx=0x7f908c001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c005df0 con 0x7f909415ca80 2023-12-19T09:02:27.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c00a600 con 0x7f909415ca80 2023-12-19T09:02:27.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90940c8bd0 con 0x7f909415ca80 2023-12-19T09:02:27.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c00a9c0 con 0x7f909415ca80 2023-12-19T09:02:27.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90940c8f60 con 0x7f909415ca80 2023-12-19T09:02:27.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.176+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909404be40 con 0x7f909415ca80 2023-12-19T09:02:27.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.180+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 67) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f908c0070f0 con 0x7f909415ca80 2023-12-19T09:02:27.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.180+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f908c07c870 con 0x7f909415ca80 2023-12-19T09:02:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.188+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f908c04ccb0 con 0x7f909415ca80 2023-12-19T09:02:27.248 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd36da84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.248 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd36da84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.248 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd36da84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.248 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd36da84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.248 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd36da84700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd370289700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd370289700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd370289700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd370289700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:27.243+0000 7fd370289700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:27.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.364+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f909406b3c0 con 0x7f909415ca80 2023-12-19T09:02:27.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f9089ffb700 1 -- 172.21.15.111:0/2252363501 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+17420 (secure 0 0 0) 0x7f908c0502d0 con 0x7f909415ca80 2023-12-19T09:02:27.373 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:27.373 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":150,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:27.149938+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-12-19T08:06:20.716528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-12-19T08:06:21.707600+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":149,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/4229900533":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/125278603":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/2780637323":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/789358368":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/3383118528":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/4020762991":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/3548009647":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/3429689930":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/415557684":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2445077517":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6833/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/375201899":"2023-12-20T09:02:27.149884+0000","172.21.15.143:6832/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2762275584":"2023-12-20T08:06:37.094770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":18,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 msgr2=0x7f9094151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f908c00ccc0 tx=0x7f908c001a70 comp rx=0 tx=0).stop 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 shutdown_connections 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f909415ca80 0x7f9094151850 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90941591f0 0x7f90940c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 --2- 172.21.15.111:0/2252363501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9094158e20 0x7f90940ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.368+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 >> 172.21.15.111:0/2252363501 conn(0x7f9094067bb0 msgr2=0x7f9094152a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.372+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 shutdown_connections 2023-12-19T09:02:27.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.372+0000 7f909ab23700 1 -- 172.21.15.111:0/2252363501 wait complete. 2023-12-19T09:02:27.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T09:02:27.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.564+0000 7ff605dcc700 1 Processor -- start 2023-12-19T09:02:27.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.564+0000 7ff605dcc700 1 -- start start 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6000b0810 0x7ff6000b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff60006be10 0x7ff600158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6000af9c0 0x7ff600154b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6000af530 con 0x7ff6000af9c0 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff600159660 con 0x7ff60006be10 2023-12-19T09:02:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff600159940 con 0x7ff6000b0810 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6000af9c0 0x7ff600154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6000af9c0 0x7ff600154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59882/0 (socket says 172.21.15.111:59882) 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 -- 172.21.15.111:0/3564949653 learned_addr learned my addr 172.21.15.111:0/3564949653 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff5fffff700 1 --2- 172.21.15.111:0/3564949653 >> v2:172.21.15.143:3300/0 conn(0x7ff60006be10 0x7ff600158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff604dca700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6000b0810 0x7ff6000b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 -- 172.21.15.111:0/3564949653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6000b0810 msgr2=0x7ff6000b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6000b0810 0x7ff6000b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 -- 172.21.15.111:0/3564949653 >> v2:172.21.15.143:3300/0 conn(0x7ff60006be10 msgr2=0x7ff600158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:27.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3564949653 >> v2:172.21.15.143:3300/0 conn(0x7ff60006be10 0x7ff600158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 -- 172.21.15.111:0/3564949653 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600159c20 con 0x7ff6000af9c0 2023-12-19T09:02:27.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000af9c0 0x7ff600154b20 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009f20 tx=0x7ff5f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191e12bea64983a1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff5ff7fe700 1 -- 172.21.15.111:0/3564949653 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800e070 con 0x7ff6000af9c0 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff5ff7fe700 1 -- 172.21.15.111:0/3564949653 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f800bad0 con 0x7ff6000af9c0 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff5ff7fe700 1 -- 172.21.15.111:0/3564949653 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80092e0 con 0x7ff6000af9c0 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 -- 172.21.15.111:0/3564949653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000af9c0 msgr2=0x7ff600154b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.568+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000af9c0 0x7ff600154b20 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009f20 tx=0x7ff5f8009840 comp rx=0 tx=0).stop 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- 172.21.15.111:0/3564949653 shutdown_connections 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000af9c0 0x7ff600154b20 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3564949653 >> v2:172.21.15.143:3300/0 conn(0x7ff60006be10 0x7ff600158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3564949653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6000b0810 0x7ff6000b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- 172.21.15.111:0/3564949653 >> 172.21.15.111:0/3564949653 conn(0x7ff600067bb0 msgr2=0x7ff6000add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- 172.21.15.111:0/3564949653 shutdown_connections 2023-12-19T09:02:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- 172.21.15.111:0/3564949653 wait complete. 2023-12-19T09:02:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 Processor -- start 2023-12-19T09:02:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- start start 2023-12-19T09:02:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 0x7ff600149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6000af9c0 0x7ff6000cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014ef90 con 0x7ff6000b0810 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014a350 con 0x7ff6000af9c0 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff605dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000cc850 con 0x7ff60006be10 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff6055cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff6055cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59886/0 (socket says 172.21.15.111:59886) 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff6055cb700 1 -- 172.21.15.111:0/3315670891 learned_addr learned my addr 172.21.15.111:0/3315670891 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.572+0000 7ff604dca700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 0x7ff600149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff5fffff700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6000af9c0 0x7ff6000cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 -- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 msgr2=0x7ff600149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 0x7ff600149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 -- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6000af9c0 msgr2=0x7ff6000cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6000af9c0 0x7ff6000cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 -- 172.21.15.111:0/3315670891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8009ae0 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff604dca700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 0x7ff600149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff6055cb700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff600069fb0 tx=0x7ff5f8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80032f0 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8008d10 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6000ccb30 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f801d650 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60014f530 con 0x7ff6000b0810 2023-12-19T09:02:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.576+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60014f720 con 0x7ff6000b0810 2023-12-19T09:02:27.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.580+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 67) v1 ==== 79672+0+0 (secure 0 0 0) 0x7ff5f8003040 con 0x7ff6000b0810 2023-12-19T09:02:27.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.580+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 7219+0+0 (secure 0 0 0) 0x7ff5f807e5e0 con 0x7ff6000b0810 2023-12-19T09:02:27.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.584+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff5f804e8b0 con 0x7ff6000b0810 2023-12-19T09:02:27.650 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:27.644+0000 7f35b8fb1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:02:27.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.800+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff60004be40 con 0x7ff6000b0810 2023-12-19T09:02:27.806 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T09:02:27.800+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T09:02:27.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.812+0000 7ff5fcff9700 1 -- 172.21.15.111:0/3315670891 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v64) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff5f8051ed0 con 0x7ff6000b0810 2023-12-19T09:02:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 msgr2=0x7ff600155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff600069fb0 tx=0x7ff5f8018000 comp rx=0 tx=0).stop 2023-12-19T09:02:27.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 shutdown_connections 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6000af9c0 0x7ff6000cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6000b0810 0x7ff600155ae0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 --2- 172.21.15.111:0/3315670891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff60006be10 0x7ff600149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 >> 172.21.15.111:0/3315670891 conn(0x7ff600067bb0 msgr2=0x7ff600152d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 shutdown_connections 2023-12-19T09:02:27.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:27.816+0000 7ff605dcc700 1 -- 172.21.15.111:0/3315670891 wait complete. 2023-12-19T09:02:27.822 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T09:02:27.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T09:02:27.883 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T09:02:28.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.000+0000 7f9ca1145700 1 Processor -- start 2023-12-19T09:02:28.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 -- start start 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 0x7f9c9c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 0x7f9c9c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c06a7a0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c0c7630 con 0x7f9c9c06c7c0 2023-12-19T09:02:28.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9ca1145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c06d4b0 con 0x7f9c9c15aa30 2023-12-19T09:02:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59904/0 (socket says 172.21.15.111:59904) 2023-12-19T09:02:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 -- 172.21.15.111:0/2312528106 learned_addr learned my addr 172.21.15.111:0/2312528106 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c93fff700 1 --2- 172.21.15.111:0/2312528106 >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 0x7f9c9c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9b59e700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 0x7f9c9c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 -- 172.21.15.111:0/2312528106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 msgr2=0x7f9c9c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 0x7f9c9c15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 -- 172.21.15.111:0/2312528106 >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 msgr2=0x7f9c9c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/2312528106 >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 0x7f9c9c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 -- 172.21.15.111:0/2312528106 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c06d790 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c93fff700 1 --2- 172.21.15.111:0/2312528106 >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 0x7f9c9c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9b59e700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 0x7f9c9c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.004+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0c86f0 tx=0x7f9c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a682b34b05655d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9c9a59c700 1 -- 172.21.15.111:0/2312528106 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c00b860 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9c9a59c700 1 -- 172.21.15.111:0/2312528106 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8c00b9c0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9c9a59c700 1 -- 172.21.15.111:0/2312528106 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c00bd60 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- 172.21.15.111:0/2312528106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 msgr2=0x7f9c9c0c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0c86f0 tx=0x7f9c8c009250 comp rx=0 tx=0).stop 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- 172.21.15.111:0/2312528106 shutdown_connections 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c15aa30 0x7f9c9c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 --2- 172.21.15.111:0/2312528106 >> v2:172.21.15.143:3300/0 conn(0x7f9c9c06c7c0 0x7f9c9c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 --2- 172.21.15.111:0/2312528106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0c70f0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- 172.21.15.111:0/2312528106 >> 172.21.15.111:0/2312528106 conn(0x7f9c9c067bb0 msgr2=0x7f9c9c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- 172.21.15.111:0/2312528106 shutdown_connections 2023-12-19T09:02:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- 172.21.15.111:0/2312528106 wait complete. 2023-12-19T09:02:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 Processor -- start 2023-12-19T09:02:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.008+0000 7f9ca1145700 1 -- start start 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c06c7c0 0x7f9c9c0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 0x7f9c9c148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0ab280 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0aa670 con 0x7f9c9c15aa30 2023-12-19T09:02:28.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0aa950 con 0x7f9c9c06c7c0 2023-12-19T09:02:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59920/0 (socket says 172.21.15.111:59920) 2023-12-19T09:02:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 -- 172.21.15.111:0/3459975021 learned_addr learned my addr 172.21.15.111:0/3459975021 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:28.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c06c7c0 0x7f9c9c0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c9b59e700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 0x7f9c9c148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 -- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c06c7c0 msgr2=0x7f9c9c0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c06c7c0 0x7f9c9c0ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 -- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 msgr2=0x7f9c9c148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 0x7f9c9c148440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 -- 172.21.15.111:0/3459975021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c0094f0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c9b59e700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 0x7f9c9c148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c93fff700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c880 tx=0x7f9c84008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011600 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c84011760 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011ae0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c0aac30 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.012+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c0ace20 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.016+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c154cc0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.016+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 67) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f9c84010350 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.016+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f9c8407c8a0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.024+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c9c154cc0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.168+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 68) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f9c8404c6c0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.168+0000 7f9c92ffd700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 0x7f9c7c068d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:28.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.176+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 0x7f9c7c068d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:28.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.176+0000 7f9c9ad9d700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 0x7f9c7c068d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0c86f0 tx=0x7f9c8c017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:28.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.240+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9c9c154cc0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.816+0000 7f9c92ffd700 1 -- 172.21.15.111:0/3459975021 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9c9c154cc0 con 0x7f9c9c0c6d20 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 msgr2=0x7f9c7c068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 0x7f9c7c068d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0c86f0 tx=0x7f9c8c017000 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 msgr2=0x7f9c9c0a9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c880 tx=0x7f9c84008040 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 shutdown_connections 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9c7c066990 0x7f9c7c068d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c9c15aa30 0x7f9c9c148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c0c6d20 0x7f9c9c0a9c70 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 --2- 172.21.15.111:0/3459975021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c9c06c7c0 0x7f9c9c0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 >> 172.21.15.111:0/3459975021 conn(0x7f9c9c067bb0 msgr2=0x7f9c9c153330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:28.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 shutdown_connections 2023-12-19T09:02:28.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:28.820+0000 7f9ca1145700 1 -- 172.21.15.111:0/3459975021 wait complete. 2023-12-19T09:02:28.848 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T09:02:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.068+0000 7facd122e700 1 Processor -- start 2023-12-19T09:02:29.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.068+0000 7facd122e700 1 -- start start 2023-12-19T09:02:29.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facd122e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facd122e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faccc161da0 0x7faccc06a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facd122e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0c6560 con 0x7faccc15d720 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facd122e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faccc0c70e0 con 0x7faccc161da0 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facd122e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0c67b0 con 0x7faccc15d350 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faccc161da0 0x7faccc06a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faccc161da0 0x7faccc06a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59936/0 (socket says 172.21.15.111:59936) 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 -- 172.21.15.111:0/2584327480 learned_addr learned my addr 172.21.15.111:0/2584327480 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:29.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7faccad9d700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 -- 172.21.15.111:0/2584327480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 msgr2=0x7faccc0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc0cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 -- 172.21.15.111:0/2584327480 >> v1:172.21.15.143:6789/0 conn(0x7faccc15d720 legacy=0x7faccc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.072+0000 7facca59c700 1 -- 172.21.15.111:0/2584327480 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0c6a90 con 0x7faccc161da0 2023-12-19T09:02:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facca59c700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc06a6a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7facc00099b0 tx=0x7facc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2e4b908a0f8920 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facc9d9b700 1 -- 172.21.15.111:0/2584327480 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc000b860 con 0x7faccc161da0 2023-12-19T09:02:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facc9d9b700 1 -- 172.21.15.111:0/2584327480 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc000b9c0 con 0x7faccc161da0 2023-12-19T09:02:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facc9d9b700 1 -- 172.21.15.111:0/2584327480 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc000bd60 con 0x7faccc161da0 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 -- 172.21.15.111:0/2584327480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 msgr2=0x7faccc06a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc06a6a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7facc00099b0 tx=0x7facc0009250 comp rx=0 tx=0).stop 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 -- 172.21.15.111:0/2584327480 shutdown_connections 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc06a6a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 --2- 172.21.15.111:0/2584327480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc0cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 -- 172.21.15.111:0/2584327480 >> 172.21.15.111:0/2584327480 conn(0x7faccc067bb0 msgr2=0x7faccc0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 -- 172.21.15.111:0/2584327480 shutdown_connections 2023-12-19T09:02:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.076+0000 7facd122e700 1 -- 172.21.15.111:0/2584327480 wait complete. 2023-12-19T09:02:29.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 Processor -- start 2023-12-19T09:02:29.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 -- start start 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc15d720 0x7faccc15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0c9670 con 0x7faccc161da0 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc0c9bc0 con 0x7faccc15d720 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facd122e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faccc15bd70 con 0x7faccc15d350 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59938/0 (socket says 172.21.15.111:59938) 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 -- 172.21.15.111:0/1967228863 learned_addr learned my addr 172.21.15.111:0/1967228863 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facca59c700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc15d720 0x7faccc15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 -- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 msgr2=0x7faccc155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 -- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc15d720 msgr2=0x7faccc15b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc15d720 0x7faccc15b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 -- 172.21.15.111:0/1967228863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00094f0 con 0x7faccc161da0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7faccb59e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7facc400c910 tx=0x7facc4001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.080+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4012630 con 0x7faccc161da0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc4012790 con 0x7faccc161da0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc15c050 con 0x7faccc161da0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4012b50 con 0x7faccc161da0 2023-12-19T09:02:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc156620 con 0x7faccc161da0 2023-12-19T09:02:29.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc15c290 con 0x7faccc161da0 2023-12-19T09:02:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.084+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 68) v1 ==== 79799+0+0 (secure 0 0 0) 0x7facc401a3a0 con 0x7faccc161da0 2023-12-19T09:02:29.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.088+0000 7facb2ffd700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 0x7facb8062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:29.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.088+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 7305+0+0 (secure 0 0 0) 0x7facc407cd80 con 0x7faccc161da0 2023-12-19T09:02:29.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.088+0000 7faccad9d700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 0x7facb8062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.088+0000 7faccad9d700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 0x7facb8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7facbc006fd0 tx=0x7facbc008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.096+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faccc15c290 con 0x7faccc161da0 2023-12-19T09:02:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.272+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faccc0c9d80 con 0x7faccc161da0 2023-12-19T09:02:29.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:29.824+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7facc404d2e0 con 0x7faccc161da0 2023-12-19T09:02:30.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.136+0000 7facb2ffd700 1 -- 172.21.15.111:0/1967228863 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v152) v1 ==== 181+0+0 (secure 0 0 0) 0x7facc40509a0 con 0x7faccc161da0 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 msgr2=0x7facb8062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 0x7facb8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7facbc006fd0 tx=0x7facbc008040 comp rx=0 tx=0).stop 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 msgr2=0x7faccc15b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7facc400c910 tx=0x7facc4001710 comp rx=0 tx=0).stop 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 shutdown_connections 2023-12-19T09:02:30.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7facb80603a0 0x7facb8062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc15d720 0x7faccc15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faccc161da0 0x7faccc15b6b0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 --2- 172.21.15.111:0/1967228863 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faccc15d350 0x7faccc155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 >> 172.21.15.111:0/1967228863 conn(0x7faccc067bb0 msgr2=0x7faccc156a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 shutdown_connections 2023-12-19T09:02:30.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.140+0000 7facd122e700 1 -- 172.21.15.111:0/1967228863 wait complete. 2023-12-19T09:02:30.150 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T09:02:30.174 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T09:02:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.332+0000 7fb5320cd700 1 Processor -- start 2023-12-19T09:02:30.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.332+0000 7fb5320cd700 1 -- start start 2023-12-19T09:02:30.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 0x7fb52c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:30.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c1572d0 con 0x7fb52c1538c0 2023-12-19T09:02:30.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c156a60 con 0x7fb52c0c6120 2023-12-19T09:02:30.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c156d10 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 0x7fb52c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:30.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb52c1538c0 0x7fb52c156350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:58208/0 (socket says 172.21.15.111:58208) 2023-12-19T09:02:30.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52b7fe700 1 -- 172.21.15.111:0/2650001157 learned_addr learned my addr 172.21.15.111:0/2650001157 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:30.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52affd700 1 -- 172.21.15.111:0/2650001157 >> v1:172.21.15.143:6789/0 conn(0x7fb52c0c6120 legacy=0x7fb52c0ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52affd700 1 -- 172.21.15.111:0/2650001157 >> v1:172.21.15.111:6789/0 conn(0x7fb52c1538c0 legacy=0x7fb52c156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52affd700 1 -- 172.21.15.111:0/2650001157 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c156ff0 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52a7fc700 1 -- 172.21.15.111:0/2650001157 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284372966 0 0) 0x7fb52c1572d0 con 0x7fb52c1538c0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52affd700 1 --2- 172.21.15.111:0/2650001157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 0x7fb52c06c7c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb520009910 tx=0x7fb5200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd3ce9155f0fc84 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52a7fc700 1 -- 172.21.15.111:0/2650001157 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000b860 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52a7fc700 1 -- 172.21.15.111:0/2650001157 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52000b9c0 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb52a7fc700 1 -- 172.21.15.111:0/2650001157 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000bd60 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 -- 172.21.15.111:0/2650001157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 msgr2=0x7fb52c06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.336+0000 7fb5320cd700 1 --2- 172.21.15.111:0/2650001157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 0x7fb52c06c7c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb520009910 tx=0x7fb5200091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.340+0000 7fb5320cd700 1 -- 172.21.15.111:0/2650001157 shutdown_connections 2023-12-19T09:02:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.340+0000 7fb5320cd700 1 --2- 172.21.15.111:0/2650001157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c0cc4e0 0x7fb52c06c7c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.340+0000 7fb5320cd700 1 -- 172.21.15.111:0/2650001157 >> 172.21.15.111:0/2650001157 conn(0x7fb52c067bb0 msgr2=0x7fb52c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.340+0000 7fb5320cd700 1 -- 172.21.15.111:0/2650001157 shutdown_connections 2023-12-19T09:02:30.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.340+0000 7fb5320cd700 1 -- 172.21.15.111:0/2650001157 wait complete. 2023-12-19T09:02:30.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 Processor -- start 2023-12-19T09:02:30.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 -- start start 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52c0c6120 0x7fb52c16db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 0x7fb52c0c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0a4790 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c16adc0 con 0x7fb52c0c6120 2023-12-19T09:02:30.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb5320cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c16a160 con 0x7fb52c1538c0 2023-12-19T09:02:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59942/0 (socket says 172.21.15.111:59942) 2023-12-19T09:02:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 -- 172.21.15.111:0/3164792449 learned_addr learned my addr 172.21.15.111:0/3164792449 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52b7fe700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52c0c6120 0x7fb52c16db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:30.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52bfff700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 0x7fb52c0c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 -- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 msgr2=0x7fb52c0c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 0x7fb52c0c4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 -- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52c0c6120 msgr2=0x7fb52c16db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:30.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52c0c6120 0x7fb52c16db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 -- 172.21.15.111:0/3164792449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb520009420 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52bfff700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 0x7fb52c0c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.344+0000 7fb52affd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fb520013000 tx=0x7fb520017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52001f070 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520002f70 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520025420 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c16a440 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c16aa00 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.348+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c16af30 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.352+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb520023070 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.352+0000 7fb5137fe700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 0x7fb518062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.352+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fb52007e040 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.352+0000 7fb52b7fe700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 0x7fb518062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:30.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.356+0000 7fb52b7fe700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 0x7fb518062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb51c000c00 tx=0x7fb51c007f30 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:30.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.360+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb52004e090 con 0x7fb52c0cc4e0 2023-12-19T09:02:30.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:30.532+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb52c071120 con 0x7fb52c0cc4e0 2023-12-19T09:02:31.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.484+0000 7fb5137fe700 1 -- 172.21.15.111:0/3164792449 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v153) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb5200516b0 con 0x7fb52c0cc4e0 2023-12-19T09:02:31.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 msgr2=0x7fb518062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 0x7fb518062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb51c000c00 tx=0x7fb51c007f30 comp rx=0 tx=0).stop 2023-12-19T09:02:31.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 msgr2=0x7fb52c16c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fb520013000 tx=0x7fb520017000 comp rx=0 tx=0).stop 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 shutdown_connections 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb5180606a0 0x7fb518062b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52c1538c0 0x7fb52c0c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb52c0cc4e0 0x7fb52c16c950 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 --2- 172.21.15.111:0/3164792449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52c0c6120 0x7fb52c16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 >> 172.21.15.111:0/3164792449 conn(0x7fb52c067bb0 msgr2=0x7fb52c06a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 shutdown_connections 2023-12-19T09:02:31.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.488+0000 7fb5320cd700 1 -- 172.21.15.111:0/3164792449 wait complete. 2023-12-19T09:02:31.497 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T09:02:31.516 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T09:02:31.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.672+0000 7f970deae700 1 Processor -- start 2023-12-19T09:02:31.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.672+0000 7f970deae700 1 -- start start 2023-12-19T09:02:31.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080cab10 0x7f97080ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806b1b0 0x7f970806b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97080c7450 con 0x7f970815d410 2023-12-19T09:02:31.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970806bea0 con 0x7f970806b1b0 2023-12-19T09:02:31.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c6e00 con 0x7f97080cab10 2023-12-19T09:02:31.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806b1b0 0x7f970806b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f970806b1b0 0x7f970806b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59956/0 (socket says 172.21.15.111:59956) 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 -- 172.21.15.111:0/1815025684 learned_addr learned my addr 172.21.15.111:0/1815025684 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9706ffd700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080cab10 0x7f97080ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 -- 172.21.15.111:0/1815025684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080cab10 msgr2=0x7f97080ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080cab10 0x7f97080ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 -- 172.21.15.111:0/1815025684 >> v1:172.21.15.111:6789/0 conn(0x7f970815d410 legacy=0x7f97080ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 -- 172.21.15.111:0/1815025684 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97080c70e0 con 0x7f970806b1b0 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f9707fff700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806b1b0 0x7f970806b650 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9700009f20 tx=0x7f9700009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9511a1329c97d48c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f97067fc700 1 -- 172.21.15.111:0/1815025684 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000e070 con 0x7f970806b1b0 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f97067fc700 1 -- 172.21.15.111:0/1815025684 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000b9d0 con 0x7f970806b1b0 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f97067fc700 1 -- 172.21.15.111:0/1815025684 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97000092e0 con 0x7f970806b1b0 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 -- 172.21.15.111:0/1815025684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806b1b0 msgr2=0x7f970806b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.676+0000 7f970deae700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806b1b0 0x7f970806b650 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f9700009f20 tx=0x7f9700009840 comp rx=0 tx=0).stop 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 -- 172.21.15.111:0/1815025684 shutdown_connections 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970806b1b0 0x7f970806b650 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 --2- 172.21.15.111:0/1815025684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080cab10 0x7f97080ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 -- 172.21.15.111:0/1815025684 >> 172.21.15.111:0/1815025684 conn(0x7f9708067bb0 msgr2=0x7f97080cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 -- 172.21.15.111:0/1815025684 shutdown_connections 2023-12-19T09:02:31.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 -- 172.21.15.111:0/1815025684 wait complete. 2023-12-19T09:02:31.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 Processor -- start 2023-12-19T09:02:31.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.680+0000 7f970deae700 1 -- start start 2023-12-19T09:02:31.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f970806b1b0 0x7f97080c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97080cab10 0x7f97080afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c3ec0 con 0x7f970815d410 2023-12-19T09:02:31.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c4a20 con 0x7f97080cab10 2023-12-19T09:02:31.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c40f0 con 0x7f970806b1b0 2023-12-19T09:02:31.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59964/0 (socket says 172.21.15.111:59964) 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 -- 172.21.15.111:0/3364162351 learned_addr learned my addr 172.21.15.111:0/3364162351 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f97077fe700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f970806b1b0 0x7f97080c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9706ffd700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97080cab10 0x7f97080afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 -- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f970806b1b0 msgr2=0x7f97080c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f970806b1b0 0x7f97080c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 -- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97080cab10 msgr2=0x7f97080afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97080cab10 0x7f97080afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 -- 172.21.15.111:0/3364162351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700009ae0 con 0x7f970815d410 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f9707fff700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f970000b610 tx=0x7f9700018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700008ad0 con 0x7f970815d410 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9700008c30 con 0x7f970815d410 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97080c43d0 con 0x7f970815d410 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.684+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700025460 con 0x7f970815d410 2023-12-19T09:02:31.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.688+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9708163d90 con 0x7f970815d410 2023-12-19T09:02:31.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.688+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97080c4bd0 con 0x7f970815d410 2023-12-19T09:02:31.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.688+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9700003040 con 0x7f970815d410 2023-12-19T09:02:31.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.692+0000 7f96ef7fe700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 0x7f96f4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:31.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.692+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f970007e8a0 con 0x7f970815d410 2023-12-19T09:02:31.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.692+0000 7f97077fe700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 0x7f96f4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:31.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.692+0000 7f97077fe700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 0x7f96f4062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96f8003a80 tx=0x7f96f8007ec0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:31.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.696+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f97080c4bd0 con 0x7f970815d410 2023-12-19T09:02:31.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.868+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f97080c4b90 con 0x7f970815d410 2023-12-19T09:02:31.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.876+0000 7f96ef7fe700 1 -- 172.21.15.111:0/3364162351 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v49) v1 ==== 120+0+0 (secure 0 0 0) 0x7f970004e910 con 0x7f970815d410 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 msgr2=0x7f96f4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 0x7f96f4062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f96f8003a80 tx=0x7f96f8007ec0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 msgr2=0x7f97080b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f970000b610 tx=0x7f9700018000 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 shutdown_connections 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f96f4060670 0x7f96f4062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97080cab10 0x7f97080afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f970815d410 0x7f97080b00b0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 --2- 172.21.15.111:0/3364162351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f970806b1b0 0x7f97080c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 >> 172.21.15.111:0/3364162351 conn(0x7f9708067bb0 msgr2=0x7f970806a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 shutdown_connections 2023-12-19T09:02:31.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:31.880+0000 7f970deae700 1 -- 172.21.15.111:0/3364162351 wait complete. 2023-12-19T09:02:31.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T09:02:32.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.064+0000 7fceae7cc700 1 Processor -- start 2023-12-19T09:02:32.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.068+0000 7fceae7cc700 1 -- start start 2023-12-19T09:02:32.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 0x7fcea8159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcea81595a0 0x7fcea81599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea806c850 con 0x7fcea8158c90 2023-12-19T09:02:32.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea80cb650 con 0x7fcea81595a0 2023-12-19T09:02:32.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea806b470 con 0x7fcea806aeb0 2023-12-19T09:02:32.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fcead7ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcea8158c90 0x7fcea80caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54992/0 (socket says 172.21.15.111:54992) 2023-12-19T09:02:32.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fcead7ca700 1 -- 172.21.15.111:0/2833221631 learned_addr learned my addr 172.21.15.111:0/2833221631 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:32.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2833221631 >> v2:172.21.15.111:3300/0 conn(0x7fcea81595a0 0x7fcea81599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceacfc9700 1 --2- 172.21.15.111:0/2833221631 >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 0x7fcea8159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fce9ffff700 1 -- 172.21.15.111:0/2833221631 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139531849 0 0) 0x7fcea806c850 con 0x7fcea8158c90 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fce9ffff700 1 -- 172.21.15.111:0/2833221631 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce94003570 con 0x7fcea8158c90 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 -- 172.21.15.111:0/2833221631 >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 msgr2=0x7fcea8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2833221631 >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 0x7fcea8159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 -- 172.21.15.111:0/2833221631 >> v1:172.21.15.111:6789/0 conn(0x7fcea8158c90 legacy=0x7fcea80caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 -- 172.21.15.111:0/2833221631 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea806b750 con 0x7fcea81595a0 2023-12-19T09:02:32.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceacfc9700 1 --2- 172.21.15.111:0/2833221631 >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 0x7fcea8159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:32.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2833221631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea81599c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcea8068000 tx=0x7fcea0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a001138ecd77e93 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fce9ffff700 1 -- 172.21.15.111:0/2833221631 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000e070 con 0x7fcea81595a0 2023-12-19T09:02:32.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fce9ffff700 1 -- 172.21.15.111:0/2833221631 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea000bab0 con 0x7fcea81595a0 2023-12-19T09:02:32.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fce9ffff700 1 -- 172.21.15.111:0/2833221631 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000be70 con 0x7fcea81595a0 2023-12-19T09:02:32.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 -- 172.21.15.111:0/2833221631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 msgr2=0x7fcea81599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.072+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2833221631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea81599c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcea8068000 tx=0x7fcea0009780 comp rx=0 tx=0).stop 2023-12-19T09:02:32.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- 172.21.15.111:0/2833221631 shutdown_connections 2023-12-19T09:02:32.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2833221631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea81599c0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2833221631 >> v2:172.21.15.143:3300/0 conn(0x7fcea806aeb0 0x7fcea8159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- 172.21.15.111:0/2833221631 >> 172.21.15.111:0/2833221631 conn(0x7fcea8067bb0 msgr2=0x7fcea80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:32.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- 172.21.15.111:0/2833221631 shutdown_connections 2023-12-19T09:02:32.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- 172.21.15.111:0/2833221631 wait complete. 2023-12-19T09:02:32.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 Processor -- start 2023-12-19T09:02:32.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- start start 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 0x7fcea8155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 0x7fcea815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea8155f80 con 0x7fcea81595a0 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea815ce10 con 0x7fcea806aeb0 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.076+0000 7fceae7cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea815d0f0 con 0x7fcea8158c90 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42130/0 (socket says 172.21.15.111:42130) 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 -- 172.21.15.111:0/2260130519 learned_addr learned my addr 172.21.15.111:0/2260130519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fcead7ca700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 0x7fcea8155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceacfc9700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 0x7fcea815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 -- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 msgr2=0x7fcea815c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 0x7fcea815c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 -- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 msgr2=0x7fcea8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 0x7fcea8155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 -- 172.21.15.111:0/2260130519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98009420 con 0x7fcea81595a0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fcead7ca700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 0x7fcea8155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceacfc9700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 0x7fcea815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceadfcb700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcea8069b70 tx=0x7fcea0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0027470 con 0x7fcea81595a0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea00275d0 con 0x7fcea81595a0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0009a20 con 0x7fcea81595a0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0027990 con 0x7fcea81595a0 2023-12-19T09:02:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea815d6b0 con 0x7fcea81595a0 2023-12-19T09:02:32.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.080+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea804be40 con 0x7fcea81595a0 2023-12-19T09:02:32.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.084+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcea0024070 con 0x7fcea81595a0 2023-12-19T09:02:32.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.084+0000 7fce9d7fa700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 0x7fce8c062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.084+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fcea007f680 con 0x7fcea81595a0 2023-12-19T09:02:32.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.084+0000 7fcead7ca700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 0x7fce8c062ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.084+0000 7fcead7ca700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 0x7fce8c062ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b5d20 tx=0x7fcea4009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.088+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcea004f800 con 0x7fcea81595a0 2023-12-19T09:02:32.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.288+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcea806ba30 con 0x7fcea81595a0 2023-12-19T09:02:32.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.300+0000 7fce9d7fa700 1 -- 172.21.15.111:0/2260130519 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcea0052e20 con 0x7fcea81595a0 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 msgr2=0x7fce8c062ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 0x7fce8c062ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcea80b5d20 tx=0x7fcea4009040 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 msgr2=0x7fcea8147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcea8069b70 tx=0x7fcea0019000 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 shutdown_connections 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fce8c0606f0 0x7fce8c062ba0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcea81595a0 0x7fcea8147190 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea8158c90 0x7fcea815c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 --2- 172.21.15.111:0/2260130519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea806aeb0 0x7fcea8155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 >> 172.21.15.111:0/2260130519 conn(0x7fcea8067bb0 msgr2=0x7fcea8157930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 shutdown_connections 2023-12-19T09:02:32.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.304+0000 7fceae7cc700 1 -- 172.21.15.111:0/2260130519 wait complete. 2023-12-19T09:02:32.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T09:02:32.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 Processor -- start 2023-12-19T09:02:32.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 -- start start 2023-12-19T09:02:32.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc0b8158e20 0x7fc0b80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc0b80cb970 0x7fc0b80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b80ca2c0 0x7fc0b806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80cbdf0 con 0x7fc0b8158e20 2023-12-19T09:02:32.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80cc0d0 con 0x7fc0b80cb970 2023-12-19T09:02:32.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0bca78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80cc3b0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0b659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc0b8158e20 0x7fc0b80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0b659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc0b8158e20 0x7fc0b80ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42144/0 (socket says 172.21.15.111:42144) 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0b659c700 1 -- 172.21.15.111:0/1925856836 learned_addr learned my addr 172.21.15.111:0/1925856836 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0b5d9b700 1 --2- 172.21.15.111:0/1925856836 >> v2:172.21.15.143:3300/0 conn(0x7fc0b80cb970 0x7fc0b80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.484+0000 7fc0b6d9d700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b80ca2c0 0x7fc0b806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 -- 172.21.15.111:0/1925856836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b80ca2c0 msgr2=0x7fc0b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:32.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b80ca2c0 0x7fc0b806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 -- 172.21.15.111:0/1925856836 >> v2:172.21.15.143:3300/0 conn(0x7fc0b80cb970 msgr2=0x7fc0b80ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:32.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 --2- 172.21.15.111:0/1925856836 >> v2:172.21.15.143:3300/0 conn(0x7fc0b80cb970 0x7fc0b80ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 -- 172.21.15.111:0/1925856836 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80cc690 con 0x7fc0b8158e20 2023-12-19T09:02:32.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.488+0000 7fc0b659c700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b8158e20 0x7fc0b80ce5c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80099b0 tx=0x7fc0a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6f36beff2d05e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0b559a700 1 -- 172.21.15.111:0/1925856836 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8009ea0 con 0x7fc0b8158e20 2023-12-19T09:02:32.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0b559a700 1 -- 172.21.15.111:0/1925856836 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a800b820 con 0x7fc0b8158e20 2023-12-19T09:02:32.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0b559a700 1 -- 172.21.15.111:0/1925856836 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800bbc0 con 0x7fc0b8158e20 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- 172.21.15.111:0/1925856836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b8158e20 msgr2=0x7fc0b80ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b8158e20 0x7fc0b80ce5c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80099b0 tx=0x7fc0a8009250 comp rx=0 tx=0).stop 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- 172.21.15.111:0/1925856836 shutdown_connections 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b80ca2c0 0x7fc0b806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 --2- 172.21.15.111:0/1925856836 >> v2:172.21.15.143:3300/0 conn(0x7fc0b80cb970 0x7fc0b80ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 --2- 172.21.15.111:0/1925856836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b8158e20 0x7fc0b80ce5c0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- 172.21.15.111:0/1925856836 >> 172.21.15.111:0/1925856836 conn(0x7fc0b8067bb0 msgr2=0x7fc0b80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- 172.21.15.111:0/1925856836 shutdown_connections 2023-12-19T09:02:32.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- 172.21.15.111:0/1925856836 wait complete. 2023-12-19T09:02:32.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 Processor -- start 2023-12-19T09:02:32.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.492+0000 7fc0bca78700 1 -- start start 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b8158e20 0x7fc0b814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b814f5c0 0x7fc0b815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b815ec30 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80b5a40 con 0x7fc0b814f5c0 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b815d170 con 0x7fc0b8158e20 2023-12-19T09:02:32.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42148/0 (socket says 172.21.15.111:42148) 2023-12-19T09:02:32.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 -- 172.21.15.111:0/3245350403 learned_addr learned my addr 172.21.15.111:0/3245350403 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:32.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b5d9b700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b8158e20 0x7fc0b814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b6d9d700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b814f5c0 0x7fc0b815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 -- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b8158e20 msgr2=0x7fc0b814ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b8158e20 0x7fc0b814ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 -- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b814f5c0 msgr2=0x7fc0b815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b814f5c0 0x7fc0b815d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 -- 172.21.15.111:0/3245350403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80094f0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0b659c700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8069a80 tx=0x7fc0a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800bba0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a8011420 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b815cb60 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a80117c0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.496+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b80b5f70 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.500+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b8071120 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.500+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc0a8011b30 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.500+0000 7fc0a67fc700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 0x7fc098062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.500+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fc0a807db60 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.500+0000 7fc0b5d9b700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 0x7fc098062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.504+0000 7fc0b5d9b700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 0x7fc098062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80ced80 tx=0x7fc0ac008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.504+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc0a804dce0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.672+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc0b806a7a0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.680+0000 7fc0a67fc700 1 -- 172.21.15.111:0/3245350403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v51) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc0a80150b0 con 0x7fc0b80ca2c0 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 msgr2=0x7fc098062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 0x7fc098062b50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80ced80 tx=0x7fc0ac008040 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 msgr2=0x7fc0b815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8069a80 tx=0x7fc0a8017000 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 shutdown_connections 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc0980606a0 0x7fc098062b50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0b814f5c0 0x7fc0b815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0b8158e20 0x7fc0b814ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 --2- 172.21.15.111:0/3245350403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b80ca2c0 0x7fc0b815e6f0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 >> 172.21.15.111:0/3245350403 conn(0x7fc0b8067bb0 msgr2=0x7fc0b80c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 shutdown_connections 2023-12-19T09:02:32.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.684+0000 7fc0bca78700 1 -- 172.21.15.111:0/3245350403 wait complete. 2023-12-19T09:02:32.713 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-12-19T09:02:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.868+0000 7f75ed4f0700 1 Processor -- start 2023-12-19T09:02:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 -- start start 2023-12-19T09:02:32.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e8140710 0x7f75e80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f75e8068800 0x7f75e80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e806e150 con 0x7f75e8140ae0 2023-12-19T09:02:32.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e806d540 con 0x7f75e8140710 2023-12-19T09:02:32.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75ed4f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e806d820 con 0x7f75e8068800 2023-12-19T09:02:32.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e8140710 0x7f75e80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e8140710 0x7f75e80c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42156/0 (socket says 172.21.15.111:42156) 2023-12-19T09:02:32.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 -- 172.21.15.111:0/2755848872 learned_addr learned my addr 172.21.15.111:0/2755848872 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:32.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e77fe700 1 --2- 172.21.15.111:0/2755848872 >> v2:172.21.15.143:3300/0 conn(0x7f75e8068800 0x7f75e80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 -- 172.21.15.111:0/2755848872 >> v2:172.21.15.143:3300/0 conn(0x7f75e8068800 msgr2=0x7f75e80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:32.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 --2- 172.21.15.111:0/2755848872 >> v2:172.21.15.143:3300/0 conn(0x7f75e8068800 0x7f75e80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 -- 172.21.15.111:0/2755848872 >> v1:172.21.15.143:6789/0 conn(0x7f75e8140ae0 legacy=0x7f75e80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.872+0000 7f75e6ffd700 1 -- 172.21.15.111:0/2755848872 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e806db00 con 0x7f75e8140710 2023-12-19T09:02:32.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75e6ffd700 1 --2- 172.21.15.111:0/2755848872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140710 0x7f75e80c70a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f75e806bfc0 tx=0x7f75d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edc51c9a98c70e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75e5ffb700 1 -- 172.21.15.111:0/2755848872 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d800e070 con 0x7f75e8140710 2023-12-19T09:02:32.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75e5ffb700 1 -- 172.21.15.111:0/2755848872 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d800ba50 con 0x7f75e8140710 2023-12-19T09:02:32.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75e5ffb700 1 -- 172.21.15.111:0/2755848872 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d80092e0 con 0x7f75e8140710 2023-12-19T09:02:32.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- 172.21.15.111:0/2755848872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140710 msgr2=0x7f75e80c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/2755848872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140710 0x7f75e80c70a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f75e806bfc0 tx=0x7f75d8009840 comp rx=0 tx=0).stop 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- 172.21.15.111:0/2755848872 shutdown_connections 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/2755848872 >> v2:172.21.15.143:3300/0 conn(0x7f75e8068800 0x7f75e80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/2755848872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140710 0x7f75e80c70a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- 172.21.15.111:0/2755848872 >> 172.21.15.111:0/2755848872 conn(0x7f75e80c8850 msgr2=0x7f75e80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- 172.21.15.111:0/2755848872 shutdown_connections 2023-12-19T09:02:32.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- 172.21.15.111:0/2755848872 wait complete. 2023-12-19T09:02:32.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 Processor -- start 2023-12-19T09:02:32.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.876+0000 7f75ed4f0700 1 -- start start 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75e8068800 0x7f75e815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 0x7f75e8166740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e815bad0 con 0x7f75e8140ae0 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e815ac00 con 0x7f75e8140710 2023-12-19T09:02:32.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e815ae50 con 0x7f75e8068800 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75e8068800 0x7f75e815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42170/0 (socket says 172.21.15.111:42170) 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 -- 172.21.15.111:0/724063293 learned_addr learned my addr 172.21.15.111:0/724063293 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e67fc700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 0x7f75e8166740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 -- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75e8068800 msgr2=0x7f75e815bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75e8068800 0x7f75e815bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 -- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 msgr2=0x7f75e8166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:32.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 0x7f75e8166740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:32.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 -- 172.21.15.111:0/724063293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8009ae0 con 0x7f75e8140ae0 2023-12-19T09:02:32.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e67fc700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 0x7f75e8166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:32.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75e77fe700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f75e000d890 tx=0x7f75e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0018070 con 0x7f75e8140ae0 2023-12-19T09:02:32.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e00125f0 con 0x7f75e8140ae0 2023-12-19T09:02:32.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0012990 con 0x7f75e8140ae0 2023-12-19T09:02:32.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e815b130 con 0x7f75e8140ae0 2023-12-19T09:02:32.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.880+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e815c430 con 0x7f75e8140ae0 2023-12-19T09:02:32.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.884+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e815bc10 con 0x7f75e8140ae0 2023-12-19T09:02:32.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.884+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f75e0014430 con 0x7f75e8140ae0 2023-12-19T09:02:32.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.888+0000 7f75ceffd700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 0x7f75d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:32.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.888+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f75e007ccc0 con 0x7f75e8140ae0 2023-12-19T09:02:32.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.888+0000 7f75e6ffd700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 0x7f75d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:32.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.888+0000 7f75e6ffd700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 0x7f75d4062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009880 tx=0x7f75d8018000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:32.892+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f75e815bc10 con 0x7f75e8140ae0 2023-12-19T09:02:33.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.112+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f75e815bc10 con 0x7f75e8140ae0 2023-12-19T09:02:33.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.496+0000 7f75ceffd700 1 -- 172.21.15.111:0/724063293 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2430) v1 ==== 178+0+0 (secure 0 0 0) 0x7f75e815bc10 con 0x7f75e8140ae0 2023-12-19T09:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 msgr2=0x7f75d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 0x7f75d4062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009880 tx=0x7f75d8018000 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 msgr2=0x7f75e80c3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f75e000d890 tx=0x7f75e0008040 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 shutdown_connections 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f75d4060670 0x7f75d4062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8140710 0x7f75e8166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e8140ae0 0x7f75e80c3950 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 --2- 172.21.15.111:0/724063293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75e8068800 0x7f75e815bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 >> 172.21.15.111:0/724063293 conn(0x7f75e80c8850 msgr2=0x7f75e80c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 shutdown_connections 2023-12-19T09:02:33.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.500+0000 7f75ed4f0700 1 -- 172.21.15.111:0/724063293 wait complete. 2023-12-19T09:02:33.525 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-12-19T09:02:33.526 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-12-19T09:02:33.526 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T09:02:33.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:33.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.684+0000 7f4be16ae700 1 Processor -- start 2023-12-19T09:02:33.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.684+0000 7f4be16ae700 1 -- start start 2023-12-19T09:02:33.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4bdc15a210 0x7f4bdc06b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 0x7f4bdc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0cb030 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc1599a0 con 0x7f4bdc15a210 2023-12-19T09:02:33.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cb290 con 0x7f4bdc15a5e0 2023-12-19T09:02:33.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4bdc15a210 0x7f4bdc06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4bdc15a210 0x7f4bdc06b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42180/0 (socket says 172.21.15.111:42180) 2023-12-19T09:02:33.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bda7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 0x7f4bdc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 -- 172.21.15.111:0/2048178960 learned_addr learned my addr 172.21.15.111:0/2048178960 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 -- 172.21.15.111:0/2048178960 >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 msgr2=0x7f4bdc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 --2- 172.21.15.111:0/2048178960 >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 0x7f4bdc06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 -- 172.21.15.111:0/2048178960 >> v1:172.21.15.143:6789/0 conn(0x7f4bdc0c90b0 legacy=0x7f4bdc0c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 -- 172.21.15.111:0/2048178960 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0cb570 con 0x7f4bdc15a210 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bda7fc700 1 --2- 172.21.15.111:0/2048178960 >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 0x7f4bdc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:33.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bdaffd700 1 --2- 172.21.15.111:0/2048178960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc15a210 0x7f4bdc06b280 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc06c4c0 tx=0x7f4bcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adb79d592cadc02 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:33.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bd9ffb700 1 -- 172.21.15.111:0/2048178960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc009e70 con 0x7f4bdc15a210 2023-12-19T09:02:33.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bd9ffb700 1 -- 172.21.15.111:0/2048178960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc00b860 con 0x7f4bdc15a210 2023-12-19T09:02:33.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4bd9ffb700 1 -- 172.21.15.111:0/2048178960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00bc00 con 0x7f4bdc15a210 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- 172.21.15.111:0/2048178960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc15a210 msgr2=0x7f4bdc06b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 --2- 172.21.15.111:0/2048178960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc15a210 0x7f4bdc06b280 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc06c4c0 tx=0x7f4bcc009250 comp rx=0 tx=0).stop 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- 172.21.15.111:0/2048178960 shutdown_connections 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 --2- 172.21.15.111:0/2048178960 >> v2:172.21.15.143:3300/0 conn(0x7f4bdc15a5e0 0x7f4bdc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 --2- 172.21.15.111:0/2048178960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc15a210 0x7f4bdc06b280 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.688+0000 7f4be16ae700 1 -- 172.21.15.111:0/2048178960 >> 172.21.15.111:0/2048178960 conn(0x7f4bdc067410 msgr2=0x7f4bdc0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- 172.21.15.111:0/2048178960 shutdown_connections 2023-12-19T09:02:33.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- 172.21.15.111:0/2048178960 wait complete. 2023-12-19T09:02:33.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 Processor -- start 2023-12-19T09:02:33.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- start start 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bdc15a210 0x7f4bdc152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bdc15a5e0 0x7f4bdc155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc1463d0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc154c50 con 0x7f4bdc15a5e0 2023-12-19T09:02:33.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4be16ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc152c50 con 0x7f4bdc15a210 2023-12-19T09:02:33.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4bdaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4bdaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42184/0 (socket says 172.21.15.111:42184) 2023-12-19T09:02:33.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4bdaffd700 1 -- 172.21.15.111:0/1969795423 learned_addr learned my addr 172.21.15.111:0/1969795423 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:33.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4bda7fc700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bdc15a210 0x7f4bdc152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.692+0000 7f4bdb7fe700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bdc15a5e0 0x7f4bdc155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 -- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bdc15a210 msgr2=0x7f4bdc152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bdc15a210 0x7f4bdc152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 -- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bdc15a5e0 msgr2=0x7f4bdc155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bdc15a5e0 0x7f4bdc155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 -- 172.21.15.111:0/1969795423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0009420 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bdaffd700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc069c10 tx=0x7f4bcc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00bbe0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc011420 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc0094f0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0117c0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc1548e0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.696+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc0cbba0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.700+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4bcc011bf0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.700+0000 7f4bc2ffd700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 0x7f4bc8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:33.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.700+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f4bcc07e090 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.700+0000 7f4bda7fc700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 0x7f4bc8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:33.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.704+0000 7f4bda7fc700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 0x7f4bc8062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc154030 tx=0x7f4bd00091a0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:33.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.708+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4bdc0cbba0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.876+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bdc0cbba0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4bc2ffd700 1 -- 172.21.15.111:0/1969795423 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14234 (secure 0 0 0) 0x7f4bdc0cbba0 con 0x7f4bdc0c90b0 2023-12-19T09:02:33.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:33.886 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":153,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:31.477183+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6835/794754877":"2023-12-20T09:02:27.804383+0000","172.21.15.143:0/4229900533":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/125278603":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/2780637323":"2023-12-20T09:02:27.149884+0000","172.21.15.111:6834/794754877":"2023-12-20T09:02:27.804383+0000","172.21.15.143:0/789358368":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/3383118528":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/4020762991":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/3548009647":"2023-12-20T09:02:27.149884+0000","172.21.15.143:0/3429689930":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/415557684":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2445077517":"2023-12-20T08:06:37.094770+0000","172.21.15.143:6833/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/375201899":"2023-12-20T09:02:27.149884+0000","172.21.15.143:6832/33796":"2023-12-20T08:06:37.094770+0000","172.21.15.143:0/2762275584":"2023-12-20T08:06:37.094770+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:33.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 msgr2=0x7f4bc8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 0x7f4bc8062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc154030 tx=0x7f4bd00091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 msgr2=0x7f4bdc145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:33.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc069c10 tx=0x7f4bcc017000 comp rx=0 tx=0).stop 2023-12-19T09:02:33.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 shutdown_connections 2023-12-19T09:02:33.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4bc8060670 0x7f4bc8062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bdc15a5e0 0x7f4bdc155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bdc15a210 0x7f4bdc152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 --2- 172.21.15.111:0/1969795423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bdc0c90b0 0x7f4bdc145e90 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:33.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.880+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 >> 172.21.15.111:0/1969795423 conn(0x7f4bdc067410 msgr2=0x7f4bdc156ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:33.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.884+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 shutdown_connections 2023-12-19T09:02:33.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:33.884+0000 7f4be16ae700 1 -- 172.21.15.111:0/1969795423 wait complete. 2023-12-19T09:02:33.909 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:02:33.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:02:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 Processor -- start 2023-12-19T09:02:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 -- start start 2023-12-19T09:02:34.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f693815d410 0x7f69380ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69380ca400 0x7f69380ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380c7450 con 0x7f693806b1b0 2023-12-19T09:02:34.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380cad50 con 0x7f693815d410 2023-12-19T09:02:34.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.076+0000 7f693eef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693806bea0 con 0x7f69380ca400 2023-12-19T09:02:34.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f693815d410 0x7f69380ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f693815d410 0x7f69380ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42196/0 (socket says 172.21.15.111:42196) 2023-12-19T09:02:34.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 -- 172.21.15.111:0/1637045232 learned_addr learned my addr 172.21.15.111:0/1637045232 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:34.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f6937fff700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69380ca400 0x7f69380ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 -- 172.21.15.111:0/1637045232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69380ca400 msgr2=0x7f69380ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:34.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69380ca400 0x7f69380ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 -- 172.21.15.111:0/1637045232 >> v1:172.21.15.111:6789/0 conn(0x7f693806b1b0 legacy=0x7f693806b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 -- 172.21.15.111:0/1637045232 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69380c6e00 con 0x7f693815d410 2023-12-19T09:02:34.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693cc8c700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f693815d410 0x7f69380ab6c0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f692c009f20 tx=0x7f692c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9141d6f4af75c6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f69377fe700 1 -- 172.21.15.111:0/1637045232 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c00e070 con 0x7f693815d410 2023-12-19T09:02:34.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f69377fe700 1 -- 172.21.15.111:0/1637045232 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c00b9d0 con 0x7f693815d410 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f69377fe700 1 -- 172.21.15.111:0/1637045232 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c0092e0 con 0x7f693815d410 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 -- 172.21.15.111:0/1637045232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f693815d410 msgr2=0x7f69380ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f693815d410 0x7f69380ab6c0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f692c009f20 tx=0x7f692c009840 comp rx=0 tx=0).stop 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 -- 172.21.15.111:0/1637045232 shutdown_connections 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69380ca400 0x7f69380ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 --2- 172.21.15.111:0/1637045232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f693815d410 0x7f69380ab6c0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 -- 172.21.15.111:0/1637045232 >> 172.21.15.111:0/1637045232 conn(0x7f6938067bb0 msgr2=0x7f69380cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 -- 172.21.15.111:0/1637045232 shutdown_connections 2023-12-19T09:02:34.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.080+0000 7f693eef0700 1 -- 172.21.15.111:0/1637045232 wait complete. 2023-12-19T09:02:34.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.084+0000 7f693eef0700 1 Processor -- start 2023-12-19T09:02:34.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.084+0000 7f693eef0700 1 -- start start 2023-12-19T09:02:34.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f693806b1b0 0x7f6938159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f693815d410 0x7f69380c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693814a200 con 0x7f69380ca400 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69380b1040 con 0x7f693806b1b0 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69380b1320 con 0x7f693815d410 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42204/0 (socket says 172.21.15.111:42204) 2023-12-19T09:02:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 -- 172.21.15.111:0/943971512 learned_addr learned my addr 172.21.15.111:0/943971512 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:34.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693cc8c700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f693806b1b0 0x7f6938159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693d48d700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f693815d410 0x7f69380c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 -- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f693815d410 msgr2=0x7f69380c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f693815d410 0x7f69380c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 -- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f693806b1b0 msgr2=0x7f6938159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f693806b1b0 0x7f6938159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 -- 172.21.15.111:0/943971512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c009ae0 con 0x7f69380ca400 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6937fff700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f692400e930 tx=0x7f69240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924013680 con 0x7f69380ca400 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69240137e0 con 0x7f69380ca400 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6938149e90 con 0x7f69380ca400 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924013b80 con 0x7f69380ca400 2023-12-19T09:02:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.088+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6938159b50 con 0x7f69380ca400 2023-12-19T09:02:34.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.092+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693814a3b0 con 0x7f69380ca400 2023-12-19T09:02:34.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.092+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f692400c460 con 0x7f69380ca400 2023-12-19T09:02:34.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.092+0000 7f6934ff9700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 0x7f691c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.092+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f692407cd80 con 0x7f69380ca400 2023-12-19T09:02:34.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.092+0000 7f693cc8c700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 0x7f691c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.096+0000 7f693cc8c700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 0x7f691c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f692c009880 tx=0x7f692c018000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.100+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f693814a3b0 con 0x7f69380ca400 2023-12-19T09:02:34.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.268+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6938071120 con 0x7f69380ca400 2023-12-19T09:02:34.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.268+0000 7f6934ff9700 1 -- 172.21.15.111:0/943971512 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v153) v1 ==== 93+0+10 (secure 0 0 0) 0x7f692404cf00 con 0x7f69380ca400 2023-12-19T09:02:34.273 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 msgr2=0x7f691c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 0x7f691c062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f692c009880 tx=0x7f692c018000 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 msgr2=0x7f69380b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f692400e930 tx=0x7f69240091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 shutdown_connections 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f691c060670 0x7f691c062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f693815d410 0x7f69380c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69380ca400 0x7f69380b0530 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 --2- 172.21.15.111:0/943971512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f693806b1b0 0x7f6938159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 >> 172.21.15.111:0/943971512 conn(0x7f6938067bb0 msgr2=0x7f69380c9780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 shutdown_connections 2023-12-19T09:02:34.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.272+0000 7f693eef0700 1 -- 172.21.15.111:0/943971512 wait complete. 2023-12-19T09:02:34.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-12-19T09:02:34.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.452+0000 7fc2b759e700 1 Processor -- start 2023-12-19T09:02:34.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.452+0000 7fc2b759e700 1 -- start start 2023-12-19T09:02:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2b8158e20 0x7fc2b806a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2b81591f0 0x7fc2b80ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b806b730 con 0x7fc2b815ca80 2023-12-19T09:02:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b806ab20 con 0x7fc2b81591f0 2023-12-19T09:02:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b806ae00 con 0x7fc2b8158e20 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2b81591f0 0x7fc2b80ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2b81591f0 0x7fc2b80ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42206/0 (socket says 172.21.15.111:42206) 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 -- 172.21.15.111:0/2747741988 learned_addr learned my addr 172.21.15.111:0/2747741988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b659c700 1 --2- 172.21.15.111:0/2747741988 >> v2:172.21.15.143:3300/0 conn(0x7fc2b8158e20 0x7fc2b806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 -- 172.21.15.111:0/2747741988 >> v2:172.21.15.143:3300/0 conn(0x7fc2b8158e20 msgr2=0x7fc2b806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 --2- 172.21.15.111:0/2747741988 >> v2:172.21.15.143:3300/0 conn(0x7fc2b8158e20 0x7fc2b806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 -- 172.21.15.111:0/2747741988 >> v1:172.21.15.143:6789/0 conn(0x7fc2b815ca80 legacy=0x7fc2b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 -- 172.21.15.111:0/2747741988 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b806b0e0 con 0x7fc2b81591f0 2023-12-19T09:02:34.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2add9b700 1 --2- 172.21.15.111:0/2747741988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b81591f0 0x7fc2b80ce100 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806e510 tx=0x7fc2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6168edfcda1eaca6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b5d9b700 1 -- 172.21.15.111:0/2747741988 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000b860 con 0x7fc2b81591f0 2023-12-19T09:02:34.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b5d9b700 1 -- 172.21.15.111:0/2747741988 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a000b9c0 con 0x7fc2b81591f0 2023-12-19T09:02:34.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b5d9b700 1 -- 172.21.15.111:0/2747741988 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000bd60 con 0x7fc2b81591f0 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 -- 172.21.15.111:0/2747741988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b81591f0 msgr2=0x7fc2b80ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 --2- 172.21.15.111:0/2747741988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b81591f0 0x7fc2b80ce100 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806e510 tx=0x7fc2a0009250 comp rx=0 tx=0).stop 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 -- 172.21.15.111:0/2747741988 shutdown_connections 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.456+0000 7fc2b759e700 1 --2- 172.21.15.111:0/2747741988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b81591f0 0x7fc2b80ce100 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 --2- 172.21.15.111:0/2747741988 >> v2:172.21.15.143:3300/0 conn(0x7fc2b8158e20 0x7fc2b806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- 172.21.15.111:0/2747741988 >> 172.21.15.111:0/2747741988 conn(0x7fc2b8067bb0 msgr2=0x7fc2b80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- 172.21.15.111:0/2747741988 shutdown_connections 2023-12-19T09:02:34.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- 172.21.15.111:0/2747741988 wait complete. 2023-12-19T09:02:34.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 Processor -- start 2023-12-19T09:02:34.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- start start 2023-12-19T09:02:34.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 0x7fc2b814a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2b815ca80 0x7fc2b816db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b814aeb0 con 0x7fc2b8158e20 2023-12-19T09:02:34.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b816e090 con 0x7fc2b815ca80 2023-12-19T09:02:34.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.460+0000 7fc2b759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b81561e0 con 0x7fc2b81591f0 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42212/0 (socket says 172.21.15.111:42212) 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 -- 172.21.15.111:0/1009059874 learned_addr learned my addr 172.21.15.111:0/1009059874 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2add9b700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 0x7fc2b814a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b6d9d700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2b815ca80 0x7fc2b816db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 -- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 msgr2=0x7fc2b814a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 0x7fc2b814a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 -- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2b815ca80 msgr2=0x7fc2b816db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:34.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2b815ca80 0x7fc2b816db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 -- 172.21.15.111:0/1009059874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00094f0 con 0x7fc2b8158e20 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2add9b700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 0x7fc2b814a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b659c700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800d890 tx=0x7fc2a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a800b600 con 0x7fc2b8158e20 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a800b760 con 0x7fc2b8158e20 2023-12-19T09:02:34.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a800bae0 con 0x7fc2b8158e20 2023-12-19T09:02:34.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b81564c0 con 0x7fc2b8158e20 2023-12-19T09:02:34.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b8156a80 con 0x7fc2b8158e20 2023-12-19T09:02:34.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.464+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b806a5a0 con 0x7fc2b8158e20 2023-12-19T09:02:34.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.468+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc2a801e460 con 0x7fc2b8158e20 2023-12-19T09:02:34.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.468+0000 7fc2aeffd700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 0x7fc298062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:34.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.468+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7fc2a807d360 con 0x7fc2b8158e20 2023-12-19T09:02:34.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.468+0000 7fc2add9b700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 0x7fc298062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:34.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.472+0000 7fc2add9b700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 0x7fc298062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806e510 tx=0x7fc2a0017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:34.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.476+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2b806a5a0 con 0x7fc2b8158e20 2023-12-19T09:02:34.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:34.704+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fc2b806a5a0 con 0x7fc2b8158e20 2023-12-19T09:02:35.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.504+0000 7fc2aeffd700 1 -- 172.21.15.111:0/1009059874 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2432) v1 ==== 177+0+0 (secure 0 0 0) 0x7fc2b806a5a0 con 0x7fc2b8158e20 2023-12-19T09:02:35.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 msgr2=0x7fc298062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 0x7fc298062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2b806e510 tx=0x7fc2a0017000 comp rx=0 tx=0).stop 2023-12-19T09:02:35.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 msgr2=0x7fc2b8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800d890 tx=0x7fc2a8008040 comp rx=0 tx=0).stop 2023-12-19T09:02:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 shutdown_connections 2023-12-19T09:02:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc298060670 0x7fc298062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2b815ca80 0x7fc2b816db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2b81591f0 0x7fc2b814a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 --2- 172.21.15.111:0/1009059874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2b8158e20 0x7fc2b8155be0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 >> 172.21.15.111:0/1009059874 conn(0x7fc2b8067bb0 msgr2=0x7fc2b80c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:35.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 shutdown_connections 2023-12-19T09:02:35.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.508+0000 7fc2b759e700 1 -- 172.21.15.111:0/1009059874 wait complete. 2023-12-19T09:02:35.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T09:02:35.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.700+0000 7f9a71945700 1 Processor -- start 2023-12-19T09:02:35.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.700+0000 7f9a71945700 1 -- start start 2023-12-19T09:02:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a71945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157150 0x7f9a6c0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a71945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ca660 0x7f9a6c157520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a71945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6c0cd980 con 0x7f9a6c157a60 2023-12-19T09:02:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a71945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c06cc60 con 0x7f9a6c0ca660 2023-12-19T09:02:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a71945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c15a4a0 con 0x7f9a6c157150 2023-12-19T09:02:35.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ca660 0x7f9a6c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ca660 0x7f9a6c157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45646/0 (socket says 172.21.15.111:45646) 2023-12-19T09:02:35.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/345255789 learned_addr learned my addr 172.21.15.111:0/345255789 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:35.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6affd700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157150 0x7f9a6c0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6b7fe700 1 --1- 172.21.15.111:0/345255789 >> v1:172.21.15.143:6789/0 conn(0x7f9a6c157a60 0x7f9a6c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:47026/0 (socket says 172.21.15.111:47026) 2023-12-19T09:02:35.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157150 msgr2=0x7f9a6c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:35.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157150 0x7f9a6c0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/345255789 >> v1:172.21.15.143:6789/0 conn(0x7f9a6c157a60 legacy=0x7f9a6c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/345255789 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0cd3c0 con 0x7f9a6c0ca660 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a6a7fc700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c157520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9a600099e0 tx=0x7f9a60009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=900a919c8b938c08 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a69ffb700 1 -- 172.21.15.111:0/345255789 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000b820 con 0x7f9a6c0ca660 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a69ffb700 1 -- 172.21.15.111:0/345255789 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6000b980 con 0x7f9a6c0ca660 2023-12-19T09:02:35.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.704+0000 7f9a69ffb700 1 -- 172.21.15.111:0/345255789 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000bd40 con 0x7f9a6c0ca660 2023-12-19T09:02:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 msgr2=0x7f9a6c157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c157520 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9a600099e0 tx=0x7f9a60009250 comp rx=0 tx=0).stop 2023-12-19T09:02:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- 172.21.15.111:0/345255789 shutdown_connections 2023-12-19T09:02:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c157520 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 --2- 172.21.15.111:0/345255789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157150 0x7f9a6c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- 172.21.15.111:0/345255789 >> 172.21.15.111:0/345255789 conn(0x7f9a6c0cdc60 msgr2=0x7f9a6c0ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:35.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- 172.21.15.111:0/345255789 shutdown_connections 2023-12-19T09:02:35.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- 172.21.15.111:0/345255789 wait complete. 2023-12-19T09:02:35.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 Processor -- start 2023-12-19T09:02:35.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.708+0000 7f9a71945700 1 -- start start 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157a60 0x7f9a6c0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0ada00 con 0x7f9a6c157150 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0adce0 con 0x7f9a6c0ca660 2023-12-19T09:02:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a71945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0adfc0 con 0x7f9a6c157a60 2023-12-19T09:02:35.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157a60 0x7f9a6c0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42214/0 (socket says 172.21.15.111:42214) 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/3952560324 learned_addr learned my addr 172.21.15.111:0/3952560324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6affd700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157a60 msgr2=0x7f9a6c0ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157a60 0x7f9a6c0ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 msgr2=0x7f9a6c14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c14c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.712+0000 7f9a6a7fc700 1 -- 172.21.15.111:0/3952560324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a600094f0 con 0x7f9a6c157150 2023-12-19T09:02:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a6a7fc700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f9a60006fd0 tx=0x7f9a60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:35.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6001f070 con 0x7f9a6c157150 2023-12-19T09:02:35.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60011d30 con 0x7f9a6c157150 2023-12-19T09:02:35.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c0ae2a0 con 0x7f9a6c157150 2023-12-19T09:02:35.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000b820 con 0x7f9a6c157150 2023-12-19T09:02:35.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c1696f0 con 0x7f9a6c157150 2023-12-19T09:02:35.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.716+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c04be40 con 0x7f9a6c157150 2023-12-19T09:02:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.720+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9a6001a8c0 con 0x7f9a6c157150 2023-12-19T09:02:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.720+0000 7f9a52ffd700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 0x7f9a58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.720+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f9a6007e030 con 0x7f9a6c157150 2023-12-19T09:02:35.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.720+0000 7f9a6affd700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 0x7f9a58062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:35.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.720+0000 7f9a6affd700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 0x7f9a58062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0b04b0 tx=0x7f9a5c0091a0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:35.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.728+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9a6004e2d0 con 0x7f9a6c157150 2023-12-19T09:02:35.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.900+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9a6c06a400 con 0x7f9a6c157150 2023-12-19T09:02:35.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.908+0000 7f9a52ffd700 1 -- 172.21.15.111:0/3952560324 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9a60006cb0 con 0x7f9a6c157150 2023-12-19T09:02:35.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 msgr2=0x7f9a58062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 0x7f9a58062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0b04b0 tx=0x7f9a5c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 msgr2=0x7f9a6c1691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:35.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f9a60006fd0 tx=0x7f9a60017000 comp rx=0 tx=0).stop 2023-12-19T09:02:35.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 shutdown_connections 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9a58060670 0x7f9a58062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c157a60 0x7f9a6c0ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a6c157150 0x7f9a6c1691b0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 --2- 172.21.15.111:0/3952560324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ca660 0x7f9a6c14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 >> 172.21.15.111:0/3952560324 conn(0x7f9a6c0cdc60 msgr2=0x7f9a6c0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 shutdown_connections 2023-12-19T09:02:35.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:35.912+0000 7f9a71945700 1 -- 172.21.15.111:0/3952560324 wait complete. 2023-12-19T09:02:35.938 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:02:35.938 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:35.984 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:36.029 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:02:36.029 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T09:02:36.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 Processor -- start 2023-12-19T09:02:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 -- start start 2023-12-19T09:02:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7b580b1410 0x7f7b580ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b5806a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5806c7c0 con 0x7f7b5815a860 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5806cd80 con 0x7f7b580b1410 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.188+0000 7f7b5e28e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5806af10 con 0x7f7b5806a580 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5da8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7b5815a860 0x7f7b5815aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54998/0 (socket says 172.21.15.111:54998) 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5da8d700 1 -- 172.21.15.111:0/1047276997 learned_addr learned my addr 172.21.15.111:0/1047276997 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b5806a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5d28c700 1 --2- 172.21.15.111:0/1047276997 >> v2:172.21.15.143:3300/0 conn(0x7f7b580b1410 0x7f7b580ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945286847 0 0) 0x7f7b5806c7c0 con 0x7f7b5815a860 2023-12-19T09:02:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b44003570 con 0x7f7b5815a860 2023-12-19T09:02:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068201003 0 0) 0x7f7b44003570 con 0x7f7b5815a860 2023-12-19T09:02:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 msgr2=0x7f7b5806a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b5806a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 >> v2:172.21.15.143:3300/0 conn(0x7f7b580b1410 msgr2=0x7f7b580ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 --2- 172.21.15.111:0/1047276997 >> v2:172.21.15.143:3300/0 conn(0x7f7b580b1410 0x7f7b580ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5806b8f0 con 0x7f7b5815a860 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f7b500017f0 con 0x7f7b5815a860 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 >> v1:172.21.15.111:6789/0 conn(0x7f7b5815a860 legacy=0x7f7b5815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b44003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b44003fe0 0x7f7b44004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44004d50 0x7f7b440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5806cd80 con 0x7f7b44003fe0 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5806af10 con 0x7f7b44004d50 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4400ba90 con 0x7f7b5806a580 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b44003fe0 0x7f7b44004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5d28c700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b44003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5da8d700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44004d50 0x7f7b440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 msgr2=0x7f7b44003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5806a580 0x7f7b44003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44004d50 msgr2=0x7f7b440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44004d50 0x7f7b440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/1047276997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44003570 con 0x7f7b44003fe0 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.192+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b44003fe0 0x7f7b44004810 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009c00 tx=0x7f7b480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953b4ab9fa632c62 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800e6b0 con 0x7f7b44003fe0 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4800e810 con 0x7f7b44003fe0 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b4ffff700 1 -- 172.21.15.111:0/1047276997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800ebd0 con 0x7f7b44003fe0 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 -- 172.21.15.111:0/1047276997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b44003fe0 msgr2=0x7f7b44004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/1047276997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b44003fe0 0x7f7b44004810 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009c00 tx=0x7f7b480091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5da8d700 1 -- 172.21.15.111:0/1047276997 reap_dead start 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 -- 172.21.15.111:0/1047276997 shutdown_connections 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 -- 172.21.15.111:0/1047276997 >> 172.21.15.111:0/1047276997 conn(0x7f7b58067bb0 msgr2=0x7f7b580cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 -- 172.21.15.111:0/1047276997 shutdown_connections 2023-12-19T09:02:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.196+0000 7f7b5e28e700 1 -- 172.21.15.111:0/1047276997 wait complete. 2023-12-19T09:02:36.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 Processor -- start 2023-12-19T09:02:36.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 -- start start 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44003fe0 0x7f7b5816a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b581422b0 0x7f7b58157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5816a5e0 con 0x7f7b581592d0 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58159a20 con 0x7f7b44003fe0 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5e28e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58159d00 con 0x7f7b581422b0 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42238/0 (socket says 172.21.15.111:42238) 2023-12-19T09:02:36.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/931950572 learned_addr learned my addr 172.21.15.111:0/931950572 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5da8d700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b581422b0 0x7f7b58157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5d28c700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44003fe0 0x7f7b5816a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b581422b0 msgr2=0x7f7b58157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b581422b0 0x7f7b58157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44003fe0 msgr2=0x7f7b5816a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44003fe0 0x7f7b5816a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 -- 172.21.15.111:0/931950572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58159fe0 con 0x7f7b581592d0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.200+0000 7f7b5ca8b700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7b44004730 tx=0x7f7b4801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48022070 con 0x7f7b581592d0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4800e980 con 0x7f7b581592d0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5815a2c0 con 0x7f7b581592d0 2023-12-19T09:02:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48027470 con 0x7f7b581592d0 2023-12-19T09:02:36.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b58157860 con 0x7f7b581592d0 2023-12-19T09:02:36.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5804be40 con 0x7f7b581592d0 2023-12-19T09:02:36.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.204+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f7b4801c450 con 0x7f7b581592d0 2023-12-19T09:02:36.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.208+0000 7f7b4d7fa700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 0x7f7b3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.208+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f7b4807e790 con 0x7f7b581592d0 2023-12-19T09:02:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.208+0000 7f7b5d28c700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 0x7f7b3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.208+0000 7f7b5d28c700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 0x7f7b3c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7b54006f00 tx=0x7f7b54008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.212+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7b480523a0 con 0x7f7b581592d0 2023-12-19T09:02:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.420+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b580cc340 con 0x7f7b581592d0 2023-12-19T09:02:36.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b4d7fa700 1 -- 172.21.15.111:0/931950572 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f7b48017df0 con 0x7f7b581592d0 2023-12-19T09:02:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:36.429 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":65,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8185,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/1355508345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1355508345},{"type":"v1","addr":"172.21.15.143:6836","nonce":1355508345}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8188,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2868499513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2868499513},{"type":"v1","addr":"172.21.15.143:6837","nonce":2868499513}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8391,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2029917941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2029917941},{"type":"v1","addr":"172.21.15.111:6833","nonce":2029917941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":8811,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2582072968","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2582072968},{"type":"v1","addr":"172.21.15.111:6835","nonce":2582072968}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[]} 2023-12-19T09:02:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 msgr2=0x7f7b3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 0x7f7b3c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7b54006f00 tx=0x7f7b54008040 comp rx=0 tx=0).stop 2023-12-19T09:02:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 msgr2=0x7f7b58141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7b44004730 tx=0x7f7b4801a000 comp rx=0 tx=0).stop 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 shutdown_connections 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7b3c060670 0x7f7b3c062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b581422b0 0x7f7b58157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b581592d0 0x7f7b58141d70 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 --2- 172.21.15.111:0/931950572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b44003fe0 0x7f7b5816a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 >> 172.21.15.111:0/931950572 conn(0x7f7b58067bb0 msgr2=0x7f7b580b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 shutdown_connections 2023-12-19T09:02:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.424+0000 7f7b5e28e700 1 -- 172.21.15.111:0/931950572 wait complete. 2023-12-19T09:02:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 65 2023-12-19T09:02:36.455 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T09:02:36.455 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T09:02:36.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T09:02:36.455 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:02:36.455 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T09:02:36.455 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T09:02:36.456 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T09:02:36.456 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:02:36.456 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T09:02:36.456 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T09:02:36.456 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T09:02:36.456 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:02:36.456 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T09:02:36.456 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T09:02:36.457 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T09:02:36.457 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:02:36.476 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T09:02:36.476 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T09:02:36.477 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T09:02:36.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T09:02:36.478 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T09:02:36.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T09:02:36.481 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T09:02:36.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T09:02:36.530 INFO:tasks.ceph.mds.a:Started 2023-12-19T09:02:36.530 DEBUG:teuthology.parallel:result is None 2023-12-19T09:02:36.531 INFO:tasks.ceph.mds.d:Started 2023-12-19T09:02:36.532 DEBUG:teuthology.parallel:result is None 2023-12-19T09:02:36.535 INFO:tasks.ceph.mds.c:Started 2023-12-19T09:02:36.535 DEBUG:teuthology.parallel:result is None 2023-12-19T09:02:36.537 INFO:tasks.ceph.mds.b:Started 2023-12-19T09:02:36.537 DEBUG:teuthology.parallel:result is None 2023-12-19T09:02:36.537 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T09:02:36.641 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:02:36.641 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T09:02:36.642 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:02:36.643 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T09:02:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.684+0000 7f46964f0700 1 Processor -- start 2023-12-19T09:02:36.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.684+0000 7f46964f0700 1 -- start start 2023-12-19T09:02:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469015b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469006d7d0 con 0x7f469006c7c0 2023-12-19T09:02:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469006cc50 con 0x7f46900b13e0 2023-12-19T09:02:36.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46900ae280 con 0x7f4690140420 2023-12-19T09:02:36.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f469006c7c0 0x7f46900b0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55014/0 (socket says 172.21.15.111:55014) 2023-12-19T09:02:36.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468f7fe700 1 -- 172.21.15.111:0/1916489678 learned_addr learned my addr 172.21.15.111:0/1916489678 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:36.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f4694a8d700 1 --1- 172.21.15.111:0/1916489678 >> v1:172.21.15.143:6789/0 conn(0x7f46900b13e0 0x7f46900ca8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:47040/0 (socket says 172.21.15.111:47040) 2023-12-19T09:02:36.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468ffff700 1 --2- 172.21.15.111:0/1916489678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427142228 0 0) 0x7f469006cc50 con 0x7f46900b13e0 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678003570 con 0x7f46900b13e0 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729244951 0 0) 0x7f469006d7d0 con 0x7f469006c7c0 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469006cc50 con 0x7f469006c7c0 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468ffff700 1 -- 172.21.15.111:0/1916489678 >> v1:172.21.15.143:6789/0 conn(0x7f46900b13e0 legacy=0x7f46900ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468ffff700 1 -- 172.21.15.111:0/1916489678 >> v1:172.21.15.111:6789/0 conn(0x7f469006c7c0 legacy=0x7f46900b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468ffff700 1 -- 172.21.15.111:0/1916489678 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469006cf30 con 0x7f4690140420 2023-12-19T09:02:36.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468ffff700 1 --2- 172.21.15.111:0/1916489678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469015b220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4680009910 tx=0x7f46800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=545e050f7fc5c21e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680011630 con 0x7f4690140420 2023-12-19T09:02:36.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680011790 con 0x7f4690140420 2023-12-19T09:02:36.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f468effd700 1 -- 172.21.15.111:0/1916489678 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680011b30 con 0x7f4690140420 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 -- 172.21.15.111:0/1916489678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 msgr2=0x7f469015b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.688+0000 7f46964f0700 1 --2- 172.21.15.111:0/1916489678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469015b220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4680009910 tx=0x7f46800091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 -- 172.21.15.111:0/1916489678 shutdown_connections 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 --2- 172.21.15.111:0/1916489678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469015b220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 -- 172.21.15.111:0/1916489678 >> 172.21.15.111:0/1916489678 conn(0x7f4690067b60 msgr2=0x7f46900ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 -- 172.21.15.111:0/1916489678 shutdown_connections 2023-12-19T09:02:36.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 -- 172.21.15.111:0/1916489678 wait complete. 2023-12-19T09:02:36.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 Processor -- start 2023-12-19T09:02:36.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.692+0000 7f46964f0700 1 -- start start 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46900b13e0 0x7f4690158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469013baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4690169780 con 0x7f469006c7c0 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4690169a60 con 0x7f46900b13e0 2023-12-19T09:02:36.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4690169d40 con 0x7f4690140420 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42264/0 (socket says 172.21.15.111:42264) 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 -- 172.21.15.111:0/2351523699 learned_addr learned my addr 172.21.15.111:0/2351523699 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468f7fe700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46900b13e0 0x7f4690158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f4694a8d700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469013baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 -- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 msgr2=0x7f469013baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469013baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 -- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46900b13e0 msgr2=0x7f4690158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46900b13e0 0x7f4690158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 -- 172.21.15.111:0/2351523699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680009420 con 0x7f469006c7c0 2023-12-19T09:02:36.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f4694a8d700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469013baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f468ffff700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4690069d50 tx=0x7f4680017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468001f070 con 0x7f469006c7c0 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4680000bc0 con 0x7f469006c7c0 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469016a020 con 0x7f469006c7c0 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680025420 con 0x7f469006c7c0 2023-12-19T09:02:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.696+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4690153c70 con 0x7f469006c7c0 2023-12-19T09:02:36.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4690071120 con 0x7f469006c7c0 2023-12-19T09:02:36.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f4680023070 con 0x7f469006c7c0 2023-12-19T09:02:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f4677fff700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 0x7f467c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6409+0+0 (secure 0 0 0) 0x7f468007ddd0 con 0x7f469006c7c0 2023-12-19T09:02:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f468f7fe700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 0x7f467c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.700+0000 7f468f7fe700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 0x7f467c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f469006a230 tx=0x7f4684009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.704+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f468004e0a0 con 0x7f469006c7c0 2023-12-19T09:02:36.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:36.884+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f46900aedb0 con 0x7f469006c7c0 2023-12-19T09:02:37.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.528+0000 7f4677fff700 1 -- 172.21.15.111:0/2351523699 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v155) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4680015080 con 0x7f469006c7c0 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 msgr2=0x7f467c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 0x7f467c062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f469006a230 tx=0x7f4684009040 comp rx=0 tx=0).stop 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 msgr2=0x7f4690157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4690069d50 tx=0x7f4680017000 comp rx=0 tx=0).stop 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 shutdown_connections 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f467c060670 0x7f467c062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46900b13e0 0x7f4690158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4690140420 0x7f469013baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 --2- 172.21.15.111:0/2351523699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469006c7c0 0x7f4690157b30 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 >> 172.21.15.111:0/2351523699 conn(0x7f4690067b60 msgr2=0x7f4690158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:37.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 shutdown_connections 2023-12-19T09:02:37.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.536+0000 7f46964f0700 1 -- 172.21.15.111:0/2351523699 wait complete. 2023-12-19T09:02:37.543 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T09:02:37.564 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T09:02:37.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.712+0000 7f9c15dcc700 1 Processor -- start 2023-12-19T09:02:37.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.712+0000 7f9c15dcc700 1 -- start start 2023-12-19T09:02:37.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c15dcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c1006d760 0x7f9c101591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:37.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c15dcc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c10159f90 con 0x7f9c10159700 2023-12-19T09:02:37.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c15dcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1015a270 con 0x7f9c10158df0 2023-12-19T09:02:37.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c15dcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100c7930 con 0x7f9c1006d760 2023-12-19T09:02:37.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c0ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c1006d760 0x7f9c101591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:37.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c155cb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9c10159700 0x7f9c100c8130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55026/0 (socket says 172.21.15.111:55026) 2023-12-19T09:02:37.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c155cb700 1 -- 172.21.15.111:0/1879085913 learned_addr learned my addr 172.21.15.111:0/1879085913 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:37.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c0ffff700 1 -- 172.21.15.111:0/1879085913 >> v1:172.21.15.143:6789/0 conn(0x7f9c10158df0 legacy=0x7f9c1006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c0ffff700 1 -- 172.21.15.111:0/1879085913 >> v1:172.21.15.111:6789/0 conn(0x7f9c10159700 legacy=0x7f9c100c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.716+0000 7f9c0ffff700 1 -- 172.21.15.111:0/1879085913 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100c7c10 con 0x7f9c1006d760 2023-12-19T09:02:37.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c0ffff700 1 --2- 172.21.15.111:0/1879085913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c1006d760 0x7f9c101591c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0098e0 tx=0x7f9bfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8503250d0f1cf42 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:37.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c0f7fe700 1 -- 172.21.15.111:0/1879085913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc009e80 con 0x7f9c1006d760 2023-12-19T09:02:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c0f7fe700 1 -- 172.21.15.111:0/1879085913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc00b860 con 0x7f9c1006d760 2023-12-19T09:02:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c0f7fe700 1 -- 172.21.15.111:0/1879085913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00bbe0 con 0x7f9c1006d760 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 -- 172.21.15.111:0/1879085913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c1006d760 msgr2=0x7f9c101591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/1879085913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c1006d760 0x7f9c101591c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0098e0 tx=0x7f9bfc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 -- 172.21.15.111:0/1879085913 shutdown_connections 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/1879085913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c1006d760 0x7f9c101591c0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 -- 172.21.15.111:0/1879085913 >> 172.21.15.111:0/1879085913 conn(0x7f9c1006c080 msgr2=0x7f9c100ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 -- 172.21.15.111:0/1879085913 shutdown_connections 2023-12-19T09:02:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.720+0000 7f9c15dcc700 1 -- 172.21.15.111:0/1879085913 wait complete. 2023-12-19T09:02:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 Processor -- start 2023-12-19T09:02:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 -- start start 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 0x7f9c101549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c10159700 0x7f9c100678b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.724+0000 7f9c15dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ad920 con 0x7f9c10158df0 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c15dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10154ee0 con 0x7f9c10159700 2023-12-19T09:02:37.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c15dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100acd10 con 0x7f9c1006d760 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42274/0 (socket says 172.21.15.111:42274) 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 -- 172.21.15.111:0/409914348 learned_addr learned my addr 172.21.15.111:0/409914348 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c155cb700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c10159700 0x7f9c100678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c14dca700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 0x7f9c101549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 -- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 msgr2=0x7f9c101549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 0x7f9c101549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 -- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c10159700 msgr2=0x7f9c100678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c10159700 0x7f9c100678b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 -- 172.21.15.111:0/409914348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc009420 con 0x7f9c10158df0 2023-12-19T09:02:37.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c14dca700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 0x7f9c101549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:37.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0ffff700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013000 tx=0x7f9bfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:37.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00bbe0 con 0x7f9c10158df0 2023-12-19T09:02:37.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100acff0 con 0x7f9c10158df0 2023-12-19T09:02:37.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc011420 con 0x7f9c10158df0 2023-12-19T09:02:37.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0117e0 con 0x7f9c10158df0 2023-12-19T09:02:37.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.728+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c100ad5b0 con 0x7f9c10158df0 2023-12-19T09:02:37.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.732+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1004be40 con 0x7f9c10158df0 2023-12-19T09:02:37.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.732+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9bfc011c10 con 0x7f9c10158df0 2023-12-19T09:02:37.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.732+0000 7f9c0cff9700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 0x7f9bf4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:37.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.732+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f9bfc07e380 con 0x7f9c10158df0 2023-12-19T09:02:37.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.736+0000 7f9c14dca700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 0x7f9bf4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:37.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.736+0000 7f9c14dca700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 0x7f9bf4062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c04007fd0 tx=0x7f9c04009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:37.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.740+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9bfc04e7f0 con 0x7f9c10158df0 2023-12-19T09:02:37.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.968+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9c100cd1f0 con 0x7f9c10158df0 2023-12-19T09:02:37.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.968+0000 7f9c0cff9700 1 -- 172.21.15.111:0/409914348 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v39) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9bfc051dd0 con 0x7f9c10158df0 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:02:37.974 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T09:02:37.975 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.977 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T09:02:37.978 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.979 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T09:02:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T09:02:37.982 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:02:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T09:02:37.985 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:02:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 msgr2=0x7f9bf4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 0x7f9bf4062b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c04007fd0 tx=0x7f9c04009040 comp rx=0 tx=0).stop 2023-12-19T09:02:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 msgr2=0x7f9c100ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:37.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc013000 tx=0x7f9bfc017000 comp rx=0 tx=0).stop 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 shutdown_connections 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9bf4060670 0x7f9bf4062b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c10159700 0x7f9c100678b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c10158df0 0x7f9c100ac3a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 --2- 172.21.15.111:0/409914348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1006d760 0x7f9c101549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 >> 172.21.15.111:0/409914348 conn(0x7f9c1006c080 msgr2=0x7f9c100cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 shutdown_connections 2023-12-19T09:02:37.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:37.972+0000 7f9c15dcc700 1 -- 172.21.15.111:0/409914348 wait complete. 2023-12-19T09:02:37.998 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:38.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.156+0000 7f23eab63700 1 Processor -- start 2023-12-19T09:02:38.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.156+0000 7f23eab63700 1 -- start start 2023-12-19T09:02:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23eab63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e40c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23eab63700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23e415cc20 0x7f23e415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23eab63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e406b470 con 0x7f23e406b900 2023-12-19T09:02:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23eab63700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e406e110 con 0x7f23e415cc20 2023-12-19T09:02:38.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23eab63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4161d10 con 0x7f23e40c6900 2023-12-19T09:02:38.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23e415cc20 0x7f23e415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23e415cc20 0x7f23e415d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45726/0 (socket says 172.21.15.111:45726) 2023-12-19T09:02:38.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 -- 172.21.15.111:0/3854999680 learned_addr learned my addr 172.21.15.111:0/3854999680 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:38.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23e9b61700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e40c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 -- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 msgr2=0x7f23e40c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e40c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 -- 172.21.15.111:0/3854999680 >> v1:172.21.15.143:6789/0 conn(0x7f23e406b900 legacy=0x7f23e406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 -- 172.21.15.111:0/3854999680 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e406bd70 con 0x7f23e415cc20 2023-12-19T09:02:38.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.160+0000 7f23ea362700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e415d0a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009f50 tx=0x7f23dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46aa61bd1d4546b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23e8b5f700 1 -- 172.21.15.111:0/3854999680 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc00e070 con 0x7f23e415cc20 2023-12-19T09:02:38.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23e8b5f700 1 -- 172.21.15.111:0/3854999680 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc00b9c0 con 0x7f23e415cc20 2023-12-19T09:02:38.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23e8b5f700 1 -- 172.21.15.111:0/3854999680 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0092e0 con 0x7f23e415cc20 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 -- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 msgr2=0x7f23e415d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e415d0a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009f50 tx=0x7f23dc009840 comp rx=0 tx=0).stop 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 -- 172.21.15.111:0/3854999680 shutdown_connections 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e415d0a0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 --2- 172.21.15.111:0/3854999680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e40c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 -- 172.21.15.111:0/3854999680 >> 172.21.15.111:0/3854999680 conn(0x7f23e4067bb0 msgr2=0x7f23e40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:38.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 -- 172.21.15.111:0/3854999680 shutdown_connections 2023-12-19T09:02:38.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.164+0000 7f23eab63700 1 -- 172.21.15.111:0/3854999680 wait complete. 2023-12-19T09:02:38.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 Processor -- start 2023-12-19T09:02:38.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 -- start start 2023-12-19T09:02:38.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e416d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e416dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e4159820 con 0x7f23e406b900 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e415e8b0 con 0x7f23e415cc20 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23eab63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e415eb90 con 0x7f23e40c6900 2023-12-19T09:02:38.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42278/0 (socket says 172.21.15.111:42278) 2023-12-19T09:02:38.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 -- 172.21.15.111:0/4153384035 learned_addr learned my addr 172.21.15.111:0/4153384035 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:38.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9360700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e416d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23ea362700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e416dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 -- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 msgr2=0x7f23e416d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e416d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 -- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 msgr2=0x7f23e416dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e416dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 -- 172.21.15.111:0/4153384035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc009ae0 con 0x7f23e406b900 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.168+0000 7f23e9b61700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f23e000ec80 tx=0x7f23e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e000c850 con 0x7f23e406b900 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e000c9b0 con 0x7f23e406b900 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e000cd50 con 0x7f23e406b900 2023-12-19T09:02:38.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e415ee70 con 0x7f23e406b900 2023-12-19T09:02:38.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e415f430 con 0x7f23e406b900 2023-12-19T09:02:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e404be40 con 0x7f23e406b900 2023-12-19T09:02:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.172+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f23e0013680 con 0x7f23e406b900 2023-12-19T09:02:38.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.176+0000 7f23d9ffb700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 0x7f23c8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.176+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f23e007cba0 con 0x7f23e406b900 2023-12-19T09:02:38.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.176+0000 7f23e9360700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 0x7f23c8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.176+0000 7f23e9360700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 0x7f23c8062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23e40abfc0 tx=0x7f23d4007ec0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.180+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23e004d070 con 0x7f23e406b900 2023-12-19T09:02:38.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.360+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23e406a580 con 0x7f23e406b900 2023-12-19T09:02:38.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23d9ffb700 1 -- 172.21.15.111:0/4153384035 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+13231 (secure 0 0 0) 0x7f23e0050690 con 0x7f23e406b900 2023-12-19T09:02:38.368 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:38.368 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":157,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:37.543746+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 msgr2=0x7f23c8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 0x7f23c8062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23e40abfc0 tx=0x7f23d4007ec0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 msgr2=0x7f23e415fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f23e000ec80 tx=0x7f23e00091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 shutdown_connections 2023-12-19T09:02:38.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23c8060670 0x7f23c8062b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23e415cc20 0x7f23e416dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23e40c6900 0x7f23e416d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 --2- 172.21.15.111:0/4153384035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e406b900 0x7f23e415fe50 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 >> 172.21.15.111:0/4153384035 conn(0x7f23e4067bb0 msgr2=0x7f23e4159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 shutdown_connections 2023-12-19T09:02:38.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.364+0000 7f23eab63700 1 -- 172.21.15.111:0/4153384035 wait complete. 2023-12-19T09:02:38.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:02:38.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:02:38.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.552+0000 7f9bb359e700 1 Processor -- start 2023-12-19T09:02:38.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.552+0000 7f9bb359e700 1 -- start start 2023-12-19T09:02:38.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bb406dcf0 0x7f9bb4159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb406c850 con 0x7f9bb4158c90 2023-12-19T09:02:38.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40cb650 con 0x7f9bb406aeb0 2023-12-19T09:02:38.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb406b470 con 0x7f9bb406dcf0 2023-12-19T09:02:38.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9bb4158c90 0x7f9bb40caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55032/0 (socket says 172.21.15.111:55032) 2023-12-19T09:02:38.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb259c700 1 -- 172.21.15.111:0/2442174052 learned_addr learned my addr 172.21.15.111:0/2442174052 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:38.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb2d9d700 1 --2- 172.21.15.111:0/2442174052 >> v2:172.21.15.143:3300/0 conn(0x7f9bb406dcf0 0x7f9bb4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700473541 0 0) 0x7f9bb406c850 con 0x7f9bb4158c90 2023-12-19T09:02:38.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b9c003570 con 0x7f9bb4158c90 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744009437 0 0) 0x7f9b9c003570 con 0x7f9bb4158c90 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 >> v2:172.21.15.143:3300/0 conn(0x7f9bb406dcf0 msgr2=0x7f9bb4159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 --2- 172.21.15.111:0/2442174052 >> v2:172.21.15.143:3300/0 conn(0x7f9bb406dcf0 0x7f9bb4159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 >> v1:172.21.15.143:6789/0 conn(0x7f9bb406aeb0 legacy=0x7f9bb4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb406b750 con 0x7f9bb4158c90 2023-12-19T09:02:38.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f9ba4001470 con 0x7f9bb4158c90 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 >> v1:172.21.15.111:6789/0 conn(0x7f9bb4158c90 legacy=0x7f9bb40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9b9c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b9c004810 0x7f9b9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b9c0051c0 0x7f9b9c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb406b470 con 0x7f9b9c003a60 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c00c0d0 con 0x7f9b9c0051c0 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c00c290 con 0x7f9b9c004810 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb259c700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9b9c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb1d9b700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b9c004810 0x7f9b9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.556+0000 7f9bb2d9d700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b9c0051c0 0x7f9b9c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 -- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b9c004810 msgr2=0x7f9b9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b9c004810 0x7f9b9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 -- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b9c0051c0 msgr2=0x7f9b9c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b9c0051c0 0x7f9b9c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 -- 172.21.15.111:0/2442174052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c003570 con 0x7f9b9c003a60 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb2d9d700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b9c0051c0 0x7f9b9c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb2d9d700 1 -- 172.21.15.111:0/2442174052 reap_dead start 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb259c700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9b9c003e70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c3c0 tx=0x7f9ba4001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2090871a5eea0037 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400cc90 con 0x7f9b9c003a60 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba400cdf0 con 0x7f9b9c003a60 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb159a700 1 -- 172.21.15.111:0/2442174052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4005b70 con 0x7f9b9c003a60 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 -- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 msgr2=0x7f9b9c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9b9c003e70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c3c0 tx=0x7f9ba4001e50 comp rx=0 tx=0).stop 2023-12-19T09:02:38.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 -- 172.21.15.111:0/2442174052 shutdown_connections 2023-12-19T09:02:38.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 --2- 172.21.15.111:0/2442174052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9b9c003e70 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 -- 172.21.15.111:0/2442174052 >> 172.21.15.111:0/2442174052 conn(0x7f9bb4067bb0 msgr2=0x7f9bb40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:38.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 -- 172.21.15.111:0/2442174052 shutdown_connections 2023-12-19T09:02:38.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.560+0000 7f9bb359e700 1 -- 172.21.15.111:0/2442174052 wait complete. 2023-12-19T09:02:38.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 Processor -- start 2023-12-19T09:02:38.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 -- start start 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 0x7f9bb41575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 0x7f9bb40c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4157b20 con 0x7f9b9c003a60 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4154a20 con 0x7f9bb4157170 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c90c0 con 0x7f9bb40c8390 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42302/0 (socket says 172.21.15.111:42302) 2023-12-19T09:02:38.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 -- 172.21.15.111:0/4235298903 learned_addr learned my addr 172.21.15.111:0/4235298903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:38.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb2d9d700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 0x7f9bb40c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 -- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 msgr2=0x7f9bb40c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb1d9b700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 0x7f9bb41575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 0x7f9bb40c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 -- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 msgr2=0x7f9bb41575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 0x7f9bb41575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb259c700 1 -- 172.21.15.111:0/4235298903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40c93a0 con 0x7f9b9c003a60 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb2d9d700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 0x7f9bb40c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.564+0000 7f9bb1d9b700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 0x7f9bb41575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9bb259c700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4001e90 tx=0x7f9ba4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40070d0 con 0x7f9b9c003a60 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba4007250 con 0x7f9b9c003a60 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb40c9680 con 0x7f9b9c003a60 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4007610 con 0x7f9b9c003a60 2023-12-19T09:02:38.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb4153660 con 0x7f9b9c003a60 2023-12-19T09:02:38.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb40c9f40 con 0x7f9b9c003a60 2023-12-19T09:02:38.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.568+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9ba4028460 con 0x7f9b9c003a60 2023-12-19T09:02:38.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.572+0000 7f9ba27fc700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 0x7f9b94062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.572+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f9ba407dca0 con 0x7f9b9c003a60 2023-12-19T09:02:38.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.572+0000 7f9bb1d9b700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 0x7f9b94062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.572+0000 7f9bb1d9b700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 0x7f9b94062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000c00 tx=0x7f9ba8009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.576+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9bb40c9f40 con 0x7f9b9c003a60 2023-12-19T09:02:38.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.744+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bb40c9f40 con 0x7f9b9c003a60 2023-12-19T09:02:38.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.744+0000 7f9ba27fc700 1 -- 172.21.15.111:0/4235298903 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bb40c9f40 con 0x7f9b9c003a60 2023-12-19T09:02:38.751 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:02:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 msgr2=0x7f9b94062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 0x7f9b94062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8000c00 tx=0x7f9ba8009040 comp rx=0 tx=0).stop 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 msgr2=0x7f9bb4153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4001e90 tx=0x7f9ba4015000 comp rx=0 tx=0).stop 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 shutdown_connections 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9b94060670 0x7f9b94062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bb4157170 0x7f9bb41575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bb40c8390 0x7f9bb40c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 --2- 172.21.15.111:0/4235298903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b9c003a60 0x7f9bb4153120 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 >> 172.21.15.111:0/4235298903 conn(0x7f9bb4067bb0 msgr2=0x7f9bb40ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 shutdown_connections 2023-12-19T09:02:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.748+0000 7f9bb359e700 1 -- 172.21.15.111:0/4235298903 wait complete. 2023-12-19T09:02:38.778 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T09:02:38.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.932+0000 7f04d1385700 1 Processor -- start 2023-12-19T09:02:38.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.932+0000 7f04d1385700 1 -- start start 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f04cc156880 0x7f04cc0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f04cc0c6dc0 0x7f04cc0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 0x7f04cc0adc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc0c7300 con 0x7f04cc156880 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04cc06d1d0 con 0x7f04cc0c6dc0 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04d1385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc06dd10 con 0x7f04cc156c50 2023-12-19T09:02:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f04cc156880 0x7f04cc0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f04cc156880 0x7f04cc0c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42318/0 (socket says 172.21.15.111:42318) 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 -- 172.21.15.111:0/3194810108 learned_addr learned my addr 172.21.15.111:0/3194810108 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04c27fc700 1 --2- 172.21.15.111:0/3194810108 >> v2:172.21.15.143:3300/0 conn(0x7f04cc0c6dc0 0x7f04cc0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04cb7fe700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 0x7f04cc0adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 -- 172.21.15.111:0/3194810108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 msgr2=0x7f04cc0adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 0x7f04cc0adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 -- 172.21.15.111:0/3194810108 >> v2:172.21.15.143:3300/0 conn(0x7f04cc0c6dc0 msgr2=0x7f04cc0ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 --2- 172.21.15.111:0/3194810108 >> v2:172.21.15.143:3300/0 conn(0x7f04cc0c6dc0 0x7f04cc0ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 -- 172.21.15.111:0/3194810108 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc06d3e0 con 0x7f04cc156880 2023-12-19T09:02:38.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04cb7fe700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 0x7f04cc0adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:38.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04caffd700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156880 0x7f04cc0c6880 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f04cc06a380 tx=0x7f04bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40decd56c3b2038 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04ca7fc700 1 -- 172.21.15.111:0/3194810108 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00b860 con 0x7f04cc156880 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04ca7fc700 1 -- 172.21.15.111:0/3194810108 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc00b9c0 con 0x7f04cc156880 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.936+0000 7f04ca7fc700 1 -- 172.21.15.111:0/3194810108 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00bd60 con 0x7f04cc156880 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- 172.21.15.111:0/3194810108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156880 msgr2=0x7f04cc0c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156880 0x7f04cc0c6880 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f04cc06a380 tx=0x7f04bc009250 comp rx=0 tx=0).stop 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- 172.21.15.111:0/3194810108 shutdown_connections 2023-12-19T09:02:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc156c50 0x7f04cc0adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 --2- 172.21.15.111:0/3194810108 >> v2:172.21.15.143:3300/0 conn(0x7f04cc0c6dc0 0x7f04cc0ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 --2- 172.21.15.111:0/3194810108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156880 0x7f04cc0c6880 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- 172.21.15.111:0/3194810108 >> 172.21.15.111:0/3194810108 conn(0x7f04cc067bb0 msgr2=0x7f04cc0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- 172.21.15.111:0/3194810108 shutdown_connections 2023-12-19T09:02:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- 172.21.15.111:0/3194810108 wait complete. 2023-12-19T09:02:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 Processor -- start 2023-12-19T09:02:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.940+0000 7f04d1385700 1 -- start start 2023-12-19T09:02:38.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc0c6dc0 0x7f04cc153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04cc156880 0x7f04cc0b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc14da10 con 0x7f04cc156c50 2023-12-19T09:02:38.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0b00d0 con 0x7f04cc156880 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0af4c0 con 0x7f04cc0c6dc0 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42322/0 (socket says 172.21.15.111:42322) 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 -- 172.21.15.111:0/1542004178 learned_addr learned my addr 172.21.15.111:0/1542004178 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04caffd700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc0c6dc0 0x7f04cc153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04c27fc700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04cc156880 0x7f04cc0b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 -- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc0c6dc0 msgr2=0x7f04cc153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc0c6dc0 0x7f04cc153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 -- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04cc156880 msgr2=0x7f04cc0b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04cc156880 0x7f04cc0b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 -- 172.21.15.111:0/1542004178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0094f0 con 0x7f04cc156c50 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04cb7fe700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f04c400a380 tx=0x7f04c4007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4015440 con 0x7f04cc156c50 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c40155a0 con 0x7f04cc156c50 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4015960 con 0x7f04cc156c50 2023-12-19T09:02:38.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc0af7a0 con 0x7f04cc156c50 2023-12-19T09:02:38.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.944+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc0afd60 con 0x7f04cc156c50 2023-12-19T09:02:38.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.948+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc0b0260 con 0x7f04cc156c50 2023-12-19T09:02:38.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.948+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f04c4015d50 con 0x7f04cc156c50 2023-12-19T09:02:38.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.948+0000 7f04c37fe700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 0x7f04b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.948+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f04c407d200 con 0x7f04cc156c50 2023-12-19T09:02:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.948+0000 7f04caffd700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 0x7f04b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.952+0000 7f04caffd700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 0x7f04b0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f04cc06a380 tx=0x7f04bc017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:38.956+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f04c404d630 con 0x7f04cc156c50 2023-12-19T09:02:39.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.128+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f04cc04be40 con 0x7f04cc156c50 2023-12-19T09:02:39.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.128+0000 7f04c37fe700 1 -- 172.21.15.111:0/1542004178 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+65 (secure 0 0 0) 0x7f04c401a060 con 0x7f04cc156c50 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:39.135 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 msgr2=0x7f04b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 0x7f04b0062b20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f04cc06a380 tx=0x7f04bc017000 comp rx=0 tx=0).stop 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 msgr2=0x7f04cc14d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f04c400a380 tx=0x7f04c4007900 comp rx=0 tx=0).stop 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 shutdown_connections 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f04b0060670 0x7f04b0062b20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04cc156880 0x7f04cc0b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04cc156c50 0x7f04cc14d4d0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 --2- 172.21.15.111:0/1542004178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04cc0c6dc0 0x7f04cc153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 >> 172.21.15.111:0/1542004178 conn(0x7f04cc067bb0 msgr2=0x7f04cc155030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 shutdown_connections 2023-12-19T09:02:39.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.132+0000 7f04d1385700 1 -- 172.21.15.111:0/1542004178 wait complete. 2023-12-19T09:02:39.161 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T09:02:39.161 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T09:02:39.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.324+0000 7f98f1c0c700 1 Processor -- start 2023-12-19T09:02:39.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.324+0000 7f98f1c0c700 1 -- start start 2023-12-19T09:02:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.324+0000 7f98f1c0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f98ec0addc0 0x7f98ec0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec06da90 con 0x7f98ec15b140 2023-12-19T09:02:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ec06a620 con 0x7f98ec0ad9f0 2023-12-19T09:02:39.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ec06dd30 con 0x7f98ec0addc0 2023-12-19T09:02:39.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98ebfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f98ec0addc0 0x7f98ec0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f140b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f98ec15b140 0x7f98ec0b0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55042/0 (socket says 172.21.15.111:55042) 2023-12-19T09:02:39.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f140b700 1 -- 172.21.15.111:0/3933143692 learned_addr learned my addr 172.21.15.111:0/3933143692 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98ebfff700 1 -- 172.21.15.111:0/3933143692 >> v1:172.21.15.143:6789/0 conn(0x7f98ec0ad9f0 legacy=0x7f98ec06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98ebfff700 1 -- 172.21.15.111:0/3933143692 >> v1:172.21.15.111:6789/0 conn(0x7f98ec15b140 legacy=0x7f98ec0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98ebfff700 1 -- 172.21.15.111:0/3933143692 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec06a8c0 con 0x7f98ec0addc0 2023-12-19T09:02:39.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98ebfff700 1 --2- 172.21.15.111:0/3933143692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0c8db0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009e60 tx=0x7f98dc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a92c8fb6b4880c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98eb7fe700 1 -- 172.21.15.111:0/3933143692 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00e070 con 0x7f98ec0addc0 2023-12-19T09:02:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98eb7fe700 1 -- 172.21.15.111:0/3933143692 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc00bc30 con 0x7f98ec0addc0 2023-12-19T09:02:39.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98eb7fe700 1 -- 172.21.15.111:0/3933143692 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc012430 con 0x7f98ec0addc0 2023-12-19T09:02:39.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- 172.21.15.111:0/3933143692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 msgr2=0x7f98ec0c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/3933143692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0c8db0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009e60 tx=0x7f98dc009780 comp rx=0 tx=0).stop 2023-12-19T09:02:39.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- 172.21.15.111:0/3933143692 shutdown_connections 2023-12-19T09:02:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/3933143692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0c8db0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.328+0000 7f98f1c0c700 1 -- 172.21.15.111:0/3933143692 >> 172.21.15.111:0/3933143692 conn(0x7f98ec067bb0 msgr2=0x7f98ec0c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.332+0000 7f98f1c0c700 1 -- 172.21.15.111:0/3933143692 shutdown_connections 2023-12-19T09:02:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.332+0000 7f98f1c0c700 1 -- 172.21.15.111:0/3933143692 wait complete. 2023-12-19T09:02:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.332+0000 7f98f1c0c700 1 Processor -- start 2023-12-19T09:02:39.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.332+0000 7f98f1c0c700 1 -- start start 2023-12-19T09:02:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98ec0ad9f0 0x7f98ec0ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0c0f30 con 0x7f98ec15b140 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0ab810 con 0x7f98ec0addc0 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec0c8000 con 0x7f98ec0ad9f0 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42326/0 (socket says 172.21.15.111:42326) 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 -- 172.21.15.111:0/738273271 learned_addr learned my addr 172.21.15.111:0/738273271 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98ebfff700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f0c0a700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98ec0ad9f0 0x7f98ec0ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 -- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98ec0ad9f0 msgr2=0x7f98ec0ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98ec0ad9f0 0x7f98ec0ad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 -- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 msgr2=0x7f98ec0ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0ab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 -- 172.21.15.111:0/738273271 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc009a20 con 0x7f98ec15b140 2023-12-19T09:02:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f140b700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c4a0 tx=0x7f98e4007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e400ce50 con 0x7f98ec15b140 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e4005b30 con 0x7f98ec15b140 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e40070d0 con 0x7f98ec15b140 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec0c0970 con 0x7f98ec15b140 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.336+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec0c1290 con 0x7f98ec15b140 2023-12-19T09:02:39.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.340+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec04be40 con 0x7f98ec15b140 2023-12-19T09:02:39.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.340+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f98e40074c0 con 0x7f98ec15b140 2023-12-19T09:02:39.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.340+0000 7f98e8ff9700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 0x7f98cc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.340+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f98e407bc50 con 0x7f98ec15b140 2023-12-19T09:02:39.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.344+0000 7f98f0c0a700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 0x7f98cc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.344+0000 7f98f0c0a700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 0x7f98cc062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f98d4003aa0 tx=0x7f98d4000bc0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.348+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f98e404c080 con 0x7f98ec15b140 2023-12-19T09:02:39.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.516+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f98ec0cb370 con 0x7f98ec15b140 2023-12-19T09:02:39.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.640+0000 7f98e8ff9700 1 -- 172.21.15.111:0/738273271 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v158) v1 ==== 138+0+0 (secure 0 0 0) 0x7f98e404f6a0 con 0x7f98ec15b140 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 msgr2=0x7f98cc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 0x7f98cc062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f98d4003aa0 tx=0x7f98d4000bc0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 msgr2=0x7f98ec0c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c4a0 tx=0x7f98e4007f10 comp rx=0 tx=0).stop 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 shutdown_connections 2023-12-19T09:02:39.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f98cc060670 0x7f98cc062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98ec0addc0 0x7f98ec0ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec15b140 0x7f98ec0c7ac0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 --2- 172.21.15.111:0/738273271 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98ec0ad9f0 0x7f98ec0ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 >> 172.21.15.111:0/738273271 conn(0x7f98ec067bb0 msgr2=0x7f98ec06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 shutdown_connections 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.644+0000 7f98f1c0c700 1 -- 172.21.15.111:0/738273271 wait complete. 2023-12-19T09:02:39.652 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T09:02:39.669 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T09:02:39.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.836+0000 7ff7c4d45700 1 Processor -- start 2023-12-19T09:02:39.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.836+0000 7ff7c4d45700 1 -- start start 2023-12-19T09:02:39.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7c4d45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7c0159790 0x7ff7c00684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7c4d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b1ed0 0x7ff7c0159b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7c4d45700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0067790 con 0x7ff7c00b0cb0 2023-12-19T09:02:39.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7c4d45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c0068030 con 0x7ff7c0159790 2023-12-19T09:02:39.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7c4d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00b1200 con 0x7ff7c00b1ed0 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bdd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff7c00b0cb0 0x7ff7c00b17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55050/0 (socket says 172.21.15.111:55050) 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bdd9b700 1 -- 172.21.15.111:0/3741915474 learned_addr learned my addr 172.21.15.111:0/3741915474 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 --2- 172.21.15.111:0/3741915474 >> v2:172.21.15.111:3300/0 conn(0x7ff7c0159790 0x7ff7c00684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bed9d700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b1ed0 0x7ff7c0159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bd59a700 1 -- 172.21.15.111:0/3741915474 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433995484 0 0) 0x7ff7c0067790 con 0x7ff7c00b0cb0 2023-12-19T09:02:39.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bd59a700 1 -- 172.21.15.111:0/3741915474 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003570 con 0x7ff7c00b0cb0 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 -- 172.21.15.111:0/3741915474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b1ed0 msgr2=0x7ff7c0159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b1ed0 0x7ff7c0159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 -- 172.21.15.111:0/3741915474 >> v1:172.21.15.111:6789/0 conn(0x7ff7c00b0cb0 legacy=0x7ff7c00b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 -- 172.21.15.111:0/3741915474 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c00679e0 con 0x7ff7c0159790 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7be59c700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c00684c0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00098e0 tx=0x7ff7b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8b69ed3783207 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bd59a700 1 -- 172.21.15.111:0/3741915474 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0011630 con 0x7ff7c0159790 2023-12-19T09:02:39.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bd59a700 1 -- 172.21.15.111:0/3741915474 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b0011790 con 0x7ff7c0159790 2023-12-19T09:02:39.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.840+0000 7ff7bd59a700 1 -- 172.21.15.111:0/3741915474 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0011b30 con 0x7ff7c0159790 2023-12-19T09:02:39.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- 172.21.15.111:0/3741915474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 msgr2=0x7ff7c00684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c00684c0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00098e0 tx=0x7ff7b00091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- 172.21.15.111:0/3741915474 shutdown_connections 2023-12-19T09:02:39.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b1ed0 0x7ff7c0159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/3741915474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c00684c0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- 172.21.15.111:0/3741915474 >> 172.21.15.111:0/3741915474 conn(0x7ff7c00cdef0 msgr2=0x7ff7c00ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- 172.21.15.111:0/3741915474 shutdown_connections 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- 172.21.15.111:0/3741915474 wait complete. 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 Processor -- start 2023-12-19T09:02:39.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- start start 2023-12-19T09:02:39.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 0x7ff7c00c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7c00b1ed0 0x7ff7c015e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c016a290 con 0x7ff7c0159790 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c015dc50 con 0x7ff7c00b1ed0 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.844+0000 7ff7c4d45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c015d640 con 0x7ff7c00b0cb0 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7be59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 0x7ff7c00c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42348/0 (socket says 172.21.15.111:42348) 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 -- 172.21.15.111:0/2295903175 learned_addr learned my addr 172.21.15.111:0/2295903175 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:39.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bdd9b700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7c00b1ed0 0x7ff7c015e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 -- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 msgr2=0x7ff7c00c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 0x7ff7c00c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 -- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7c00b1ed0 msgr2=0x7ff7c015e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:39.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7c00b1ed0 0x7ff7c015e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 -- 172.21.15.111:0/2295903175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0009420 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7be59c700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 0x7ff7c00c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7bed9d700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009ec0 tx=0x7ff7b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b800cb70 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b800f040 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c00c9ec0 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8014480 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c016a030 con 0x7ff7c0159790 2023-12-19T09:02:39.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.848+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c015da00 con 0x7ff7c0159790 2023-12-19T09:02:39.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.852+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff7b80147f0 con 0x7ff7c0159790 2023-12-19T09:02:39.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.852+0000 7ff7ae7fc700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 0x7ff7a0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:39.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.852+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6091+0+0 (secure 0 0 0) 0x7ff7b807ddb0 con 0x7ff7c0159790 2023-12-19T09:02:39.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.852+0000 7ff7be59c700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 0x7ff7a0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:39.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.856+0000 7ff7be59c700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 0x7ff7a0062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00091e0 tx=0x7ff7b0017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:39.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:39.860+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff7b804dfe0 con 0x7ff7c0159790 2023-12-19T09:02:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.032+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff7c0071120 con 0x7ff7c0159790 2023-12-19T09:02:40.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.648+0000 7ff7ae7fc700 1 -- 172.21.15.111:0/2295903175 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v159) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff7b8051600 con 0x7ff7c0159790 2023-12-19T09:02:40.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 msgr2=0x7ff7a0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 0x7ff7a0062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00091e0 tx=0x7ff7b0017000 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 msgr2=0x7ff7c0169280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009ec0 tx=0x7ff7b800b040 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 shutdown_connections 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff7a0060670 0x7ff7a0062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7c00b1ed0 0x7ff7c015e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7c0159790 0x7ff7c0169280 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 --2- 172.21.15.111:0/2295903175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7c00b0cb0 0x7ff7c00c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.652+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 >> 172.21.15.111:0/2295903175 conn(0x7ff7c00cdef0 msgr2=0x7ff7c0155030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:40.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.660+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 shutdown_connections 2023-12-19T09:02:40.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.664+0000 7ff7c4d45700 1 -- 172.21.15.111:0/2295903175 wait complete. 2023-12-19T09:02:40.678 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T09:02:40.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T09:02:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 Processor -- start 2023-12-19T09:02:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 -- start start 2023-12-19T09:02:40.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 0x7fcef006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcef00cecd0 0x7fcef006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef00cbdf0 con 0x7fcef0158e20 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef00cc0d0 con 0x7fcef00cecd0 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.860+0000 7fcef5185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef00cc3b0 con 0x7fcef00cb970 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcef00cecd0 0x7fcef006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcef00cecd0 0x7fcef006b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42350/0 (socket says 172.21.15.111:42350) 2023-12-19T09:02:40.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 -- 172.21.15.111:0/3443501436 learned_addr learned my addr 172.21.15.111:0/3443501436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:40.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceee59c700 1 --2- 172.21.15.111:0/3443501436 >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 0x7fcef006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 -- 172.21.15.111:0/3443501436 >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 msgr2=0x7fcef006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 --2- 172.21.15.111:0/3443501436 >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 0x7fcef006acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 -- 172.21.15.111:0/3443501436 >> v1:172.21.15.111:6789/0 conn(0x7fcef0158e20 legacy=0x7fcef00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 -- 172.21.15.111:0/3443501436 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00cc690 con 0x7fcef00cecd0 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceee59c700 1 --2- 172.21.15.111:0/3443501436 >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 0x7fcef006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceef59e700 1 --2- 172.21.15.111:0/3443501436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef00cecd0 0x7fcef006b200 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009f20 tx=0x7fcee8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8224ef530526f808 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceedd9b700 1 -- 172.21.15.111:0/3443501436 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee800e070 con 0x7fcef00cecd0 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceedd9b700 1 -- 172.21.15.111:0/3443501436 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee800ba40 con 0x7fcef00cecd0 2023-12-19T09:02:40.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fceedd9b700 1 -- 172.21.15.111:0/3443501436 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80092e0 con 0x7fcef00cecd0 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 -- 172.21.15.111:0/3443501436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef00cecd0 msgr2=0x7fcef006b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 --2- 172.21.15.111:0/3443501436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef00cecd0 0x7fcef006b200 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009f20 tx=0x7fcee8009840 comp rx=0 tx=0).stop 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 -- 172.21.15.111:0/3443501436 shutdown_connections 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 --2- 172.21.15.111:0/3443501436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef00cecd0 0x7fcef006b200 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 --2- 172.21.15.111:0/3443501436 >> v2:172.21.15.143:3300/0 conn(0x7fcef00cb970 0x7fcef006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 -- 172.21.15.111:0/3443501436 >> 172.21.15.111:0/3443501436 conn(0x7fcef0067bb0 msgr2=0x7fcef00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 -- 172.21.15.111:0/3443501436 shutdown_connections 2023-12-19T09:02:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.864+0000 7fcef5185700 1 -- 172.21.15.111:0/3443501436 wait complete. 2023-12-19T09:02:40.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 Processor -- start 2023-12-19T09:02:40.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 -- start start 2023-12-19T09:02:40.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcef00cecd0 0x7fcef016df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcef0167190 0x7fcef00d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef00b20e0 con 0x7fcef0158e20 2023-12-19T09:02:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef00c85e0 con 0x7fcef0167190 2023-12-19T09:02:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fcef5185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef00b15a0 con 0x7fcef00cecd0 2023-12-19T09:02:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42366/0 (socket says 172.21.15.111:42366) 2023-12-19T09:02:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 -- 172.21.15.111:0/3733620089 learned_addr learned my addr 172.21.15.111:0/3733620089 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceef59e700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcef0167190 0x7fcef00d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 -- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcef00cecd0 msgr2=0x7fcef016df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcef00cecd0 0x7fcef016df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 -- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcef0167190 msgr2=0x7fcef00d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcef0167190 0x7fcef00d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 -- 172.21.15.111:0/3733620089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8009ae0 con 0x7fcef0158e20 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.868+0000 7fceee59c700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fcee400d890 tx=0x7fcee4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee40125f0 con 0x7fcef0158e20 2023-12-19T09:02:40.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4012750 con 0x7fcef0158e20 2023-12-19T09:02:40.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef00b1840 con 0x7fcef0158e20 2023-12-19T09:02:40.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4012af0 con 0x7fcef0158e20 2023-12-19T09:02:40.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef00b1e00 con 0x7fcef0158e20 2023-12-19T09:02:40.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.872+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef00c83b0 con 0x7fcef0158e20 2023-12-19T09:02:40.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.876+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fcee4027430 con 0x7fcef0158e20 2023-12-19T09:02:40.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.876+0000 7fced6ffd700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 0x7fcedc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:40.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.876+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fcee407da10 con 0x7fcef0158e20 2023-12-19T09:02:40.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.876+0000 7fceeed9d700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 0x7fcedc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:40.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.880+0000 7fceeed9d700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 0x7fcedc062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcef006b8d0 tx=0x7fcee0007b40 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:40.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:40.884+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcee404dbf0 con 0x7fcef0158e20 2023-12-19T09:02:41.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.088+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcef00c83b0 con 0x7fcef0158e20 2023-12-19T09:02:41.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.668+0000 7fced6ffd700 1 -- 172.21.15.111:0/3733620089 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v68) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcef00c83b0 con 0x7fcef0158e20 2023-12-19T09:02:41.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.680+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 msgr2=0x7fcedc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.680+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 0x7fcedc062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcef006b8d0 tx=0x7fcee0007b40 comp rx=0 tx=0).stop 2023-12-19T09:02:41.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.680+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 msgr2=0x7fcef00b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.680+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fcee400d890 tx=0x7fcee4008040 comp rx=0 tx=0).stop 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 shutdown_connections 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fcedc060670 0x7fcedc062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcef0167190 0x7fcef00d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef0158e20 0x7fcef00b1060 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 --2- 172.21.15.111:0/3733620089 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcef00cecd0 0x7fcef016df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.684+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 >> 172.21.15.111:0/3733620089 conn(0x7fcef0067bb0 msgr2=0x7fcef014fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:41.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.688+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 shutdown_connections 2023-12-19T09:02:41.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.688+0000 7fcef5185700 1 -- 172.21.15.111:0/3733620089 wait complete. 2023-12-19T09:02:41.700 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 19 and data pool 20 2023-12-19T09:02:41.718 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:41.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 Processor -- start 2023-12-19T09:02:41.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 -- start start 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ea40cd580 0x7f2ea4161770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea40b1d10 con 0x7f2ea40cc1a0 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea406c940 con 0x7f2ea40b0d50 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea926e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea406cc20 con 0x7f2ea40cd580 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea2ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2ea40cc1a0 0x7f2ea40b1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46916/0 (socket says 172.21.15.111:46916) 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea2ffd700 1 -- 172.21.15.111:0/2095572286 learned_addr learned my addr 172.21.15.111:0/2095572286 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:41.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.880+0000 7f2ea37fe700 1 --2- 172.21.15.111:0/2095572286 >> v2:172.21.15.111:3300/0 conn(0x7f2ea40cd580 0x7f2ea4161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674795453 0 0) 0x7f2ea40b1d10 con 0x7f2ea40cc1a0 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c003570 con 0x7f2ea40cc1a0 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983144565 0 0) 0x7f2ea406c940 con 0x7f2ea40b0d50 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea37fe700 1 -- 172.21.15.111:0/2095572286 >> v1:172.21.15.143:6789/0 conn(0x7f2ea40b0d50 legacy=0x7f2ea406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea37fe700 1 -- 172.21.15.111:0/2095572286 >> v1:172.21.15.111:6789/0 conn(0x7f2ea40cc1a0 legacy=0x7f2ea40b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea37fe700 1 -- 172.21.15.111:0/2095572286 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea406cf00 con 0x7f2ea40cd580 2023-12-19T09:02:41.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea37fe700 1 --2- 172.21.15.111:0/2095572286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4161770 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009e60 tx=0x7f2e9c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d632c94de88231 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:41.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c00e070 con 0x7f2ea40cd580 2023-12-19T09:02:41.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c00bb60 con 0x7f2ea40cd580 2023-12-19T09:02:41.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea1ffb700 1 -- 172.21.15.111:0/2095572286 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c012430 con 0x7f2ea40cd580 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 -- 172.21.15.111:0/2095572286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 msgr2=0x7f2ea4161770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 --2- 172.21.15.111:0/2095572286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4161770 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009e60 tx=0x7f2e9c009780 comp rx=0 tx=0).stop 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 -- 172.21.15.111:0/2095572286 shutdown_connections 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 --2- 172.21.15.111:0/2095572286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4161770 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 -- 172.21.15.111:0/2095572286 >> 172.21.15.111:0/2095572286 conn(0x7f2ea40cdc20 msgr2=0x7f2ea40ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 -- 172.21.15.111:0/2095572286 shutdown_connections 2023-12-19T09:02:41.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.884+0000 7f2ea926e700 1 -- 172.21.15.111:0/2095572286 wait complete. 2023-12-19T09:02:41.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 Processor -- start 2023-12-19T09:02:41.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 -- start start 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 0x7f2ea414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ea40cc1a0 0x7f2ea40ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40c9680 con 0x7f2ea40cd580 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40af250 con 0x7f2ea40b0d50 2023-12-19T09:02:41.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.888+0000 7f2ea926e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40b00b0 con 0x7f2ea40cc1a0 2023-12-19T09:02:41.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:41.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 0x7f2ea414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:41.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52418/0 (socket says 172.21.15.111:52418) 2023-12-19T09:02:41.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 -- 172.21.15.111:0/170072875 learned_addr learned my addr 172.21.15.111:0/170072875 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:41.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea27fc700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ea40cc1a0 0x7f2ea40ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 -- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ea40cc1a0 msgr2=0x7f2ea40ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ea40cc1a0 0x7f2ea40ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 -- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 msgr2=0x7f2ea414ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 0x7f2ea414ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 -- 172.21.15.111:0/170072875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c009a20 con 0x7f2ea40cd580 2023-12-19T09:02:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea2ffd700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 0x7f2ea414ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:41.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea37fe700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00b6c0 tx=0x7f2e9c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:41.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c026480 con 0x7f2ea40cd580 2023-12-19T09:02:41.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c026600 con 0x7f2ea40cd580 2023-12-19T09:02:41.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea40af4a0 con 0x7f2ea40cd580 2023-12-19T09:02:41.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0269c0 con 0x7f2ea40cd580 2023-12-19T09:02:41.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea40afa60 con 0x7f2ea40cd580 2023-12-19T09:02:41.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.892+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea404be40 con 0x7f2ea40cd580 2023-12-19T09:02:41.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.896+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2e9c01c400 con 0x7f2ea40cd580 2023-12-19T09:02:41.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.896+0000 7f2e8affd700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 0x7f2e90062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:41.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.896+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f2e9c07f0b0 con 0x7f2ea40cd580 2023-12-19T09:02:41.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.900+0000 7f2ea2ffd700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 0x7f2e90062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:41.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.900+0000 7f2ea2ffd700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 0x7f2e90062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2e94007fd0 tx=0x7f2e94009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:41.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:41.904+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e9c04f080 con 0x7f2ea40cd580 2023-12-19T09:02:42.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.072+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ea40b01f0 con 0x7f2ea40cd580 2023-12-19T09:02:42.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2e8affd700 1 -- 172.21.15.111:0/170072875 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16464 (secure 0 0 0) 0x7f2e9c0526a0 con 0x7f2ea40cd580 2023-12-19T09:02:42.081 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:42.081 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":160,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:41.092663+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 msgr2=0x7f2e90062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 0x7f2e90062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2e94007fd0 tx=0x7f2e94009040 comp rx=0 tx=0).stop 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 msgr2=0x7f2ea4151ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00b6c0 tx=0x7f2e9c019000 comp rx=0 tx=0).stop 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 shutdown_connections 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e90060670 0x7f2e90062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ea40cd580 0x7f2ea4151ae0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ea40cc1a0 0x7f2ea40ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 --2- 172.21.15.111:0/170072875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ea40b0d50 0x7f2ea414ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 >> 172.21.15.111:0/170072875 conn(0x7f2ea40cdc20 msgr2=0x7f2ea41501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 shutdown_connections 2023-12-19T09:02:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.076+0000 7f2ea926e700 1 -- 172.21.15.111:0/170072875 wait complete. 2023-12-19T09:02:42.106 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:42.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 Processor -- start 2023-12-19T09:02:42.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 -- start start 2023-12-19T09:02:42.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2284155960 0x7f228415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f22840c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228406d990 con 0x7f228406c7c0 2023-12-19T09:02:42.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228415a570 con 0x7f2284155960 2023-12-19T09:02:42.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2289403700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228406cd80 con 0x7f228406de20 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2283fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2284155960 0x7f228415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2283fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2284155960 0x7f228415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49280/0 (socket says 172.21.15.111:49280) 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f2283fff700 1 -- 172.21.15.111:0/3244585406 learned_addr learned my addr 172.21.15.111:0/3244585406 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.260+0000 7f227b7fe700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f22840c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2283fff700 1 -- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 msgr2=0x7f22840c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2283fff700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f22840c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2283fff700 1 -- 172.21.15.111:0/3244585406 >> v1:172.21.15.143:6789/0 conn(0x7f228406c7c0 legacy=0x7f22840c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2283fff700 1 -- 172.21.15.111:0/3244585406 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228406d060 con 0x7f2284155960 2023-12-19T09:02:42.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2283fff700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f228415a030 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2274009f20 tx=0x7f2274009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6432f5e50a90f238 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f22837fe700 1 -- 172.21.15.111:0/3244585406 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227400e070 con 0x7f2284155960 2023-12-19T09:02:42.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f22837fe700 1 -- 172.21.15.111:0/3244585406 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400ba80 con 0x7f2284155960 2023-12-19T09:02:42.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f22837fe700 1 -- 172.21.15.111:0/3244585406 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740092e0 con 0x7f2284155960 2023-12-19T09:02:42.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 -- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 msgr2=0x7f228415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f228415a030 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2274009f20 tx=0x7f2274009840 comp rx=0 tx=0).stop 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 -- 172.21.15.111:0/3244585406 shutdown_connections 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f22840c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 --2- 172.21.15.111:0/3244585406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f228415a030 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 -- 172.21.15.111:0/3244585406 >> 172.21.15.111:0/3244585406 conn(0x7f2284067bb0 msgr2=0x7f22840cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 -- 172.21.15.111:0/3244585406 shutdown_connections 2023-12-19T09:02:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.264+0000 7f2289403700 1 -- 172.21.15.111:0/3244585406 wait complete. 2023-12-19T09:02:42.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 Processor -- start 2023-12-19T09:02:42.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 -- start start 2023-12-19T09:02:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f2284148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f2284158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2284156dc0 con 0x7f228406c7c0 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2284149510 con 0x7f2284155960 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2289403700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2284149780 con 0x7f228406de20 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.268+0000 7f2283fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52420/0 (socket says 172.21.15.111:52420) 2023-12-19T09:02:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 -- 172.21.15.111:0/3096718344 learned_addr learned my addr 172.21.15.111:0/3096718344 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f227b7fe700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f2284148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2288c02700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f2284158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 -- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 msgr2=0x7f2284148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f2284148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 -- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 msgr2=0x7f2284158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f2284158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 -- 172.21.15.111:0/3096718344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2274009ae0 con 0x7f228406c7c0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2288c02700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f2284158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2283fff700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f2274009430 tx=0x7f2274018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274008b70 con 0x7f228406c7c0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2274008cd0 con 0x7f228406c7c0 2023-12-19T09:02:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22740266b0 con 0x7f228406c7c0 2023-12-19T09:02:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2284149a60 con 0x7f228406c7c0 2023-12-19T09:02:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228414a020 con 0x7f228406c7c0 2023-12-19T09:02:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.272+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2284156b80 con 0x7f228406c7c0 2023-12-19T09:02:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.276+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2274003040 con 0x7f228406c7c0 2023-12-19T09:02:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.276+0000 7f2280ff9700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 0x7f2264062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.276+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f227407ea90 con 0x7f228406c7c0 2023-12-19T09:02:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.280+0000 7f227b7fe700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 0x7f2264062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.280+0000 7f227b7fe700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 0x7f2264062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f226c006fd0 tx=0x7f226c008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.284+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f227404eb10 con 0x7f228406c7c0 2023-12-19T09:02:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.460+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f228404be40 con 0x7f228406c7c0 2023-12-19T09:02:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.460+0000 7f2280ff9700 1 -- 172.21.15.111:0/3096718344 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16464 (secure 0 0 0) 0x7f2274052130 con 0x7f228406c7c0 2023-12-19T09:02:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:42.467 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":160,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:41.092663+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 msgr2=0x7f2264062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 0x7f2264062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f226c006fd0 tx=0x7f226c008040 comp rx=0 tx=0).stop 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 msgr2=0x7f22841564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f2274009430 tx=0x7f2274018000 comp rx=0 tx=0).stop 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 shutdown_connections 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f22640606a0 0x7f2264062b50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2284155960 0x7f2284158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f228406de20 0x7f2284148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 --2- 172.21.15.111:0/3096718344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f228406c7c0 0x7f22841564d0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 >> 172.21.15.111:0/3096718344 conn(0x7f2284067bb0 msgr2=0x7f22840b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 shutdown_connections 2023-12-19T09:02:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.464+0000 7f2289403700 1 -- 172.21.15.111:0/3096718344 wait complete. 2023-12-19T09:02:42.490 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T09:02:42.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.652+0000 7f2e7562e700 1 Processor -- start 2023-12-19T09:02:42.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.652+0000 7f2e7562e700 1 -- start start 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e70158fa0 0x7f2e7006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e7006ced0 0x7f2e700b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700b0280 con 0x7f2e7006d8b0 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7006b4e0 con 0x7f2e7006ced0 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7006b7c0 con 0x7f2e70158fa0 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e7006ced0 0x7f2e700b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e7006ced0 0x7f2e700b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52432/0 (socket says 172.21.15.111:52432) 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 -- 172.21.15.111:0/727960405 learned_addr learned my addr 172.21.15.111:0/727960405 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6effd700 1 --2- 172.21.15.111:0/727960405 >> v2:172.21.15.143:3300/0 conn(0x7f2e70158fa0 0x7f2e7006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 -- 172.21.15.111:0/727960405 >> v2:172.21.15.143:3300/0 conn(0x7f2e70158fa0 msgr2=0x7f2e7006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 --2- 172.21.15.111:0/727960405 >> v2:172.21.15.143:3300/0 conn(0x7f2e70158fa0 0x7f2e7006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 -- 172.21.15.111:0/727960405 >> v1:172.21.15.143:6789/0 conn(0x7f2e7006d8b0 legacy=0x7f2e7006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 -- 172.21.15.111:0/727960405 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7006baa0 con 0x7f2e7006ced0 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6f7fe700 1 --2- 172.21.15.111:0/727960405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006ced0 0x7f2e700b06f0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2e7006f8c0 tx=0x7f2e68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c045930e2e6001c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6dffb700 1 -- 172.21.15.111:0/727960405 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800e070 con 0x7f2e7006ced0 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6dffb700 1 -- 172.21.15.111:0/727960405 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800ba10 con 0x7f2e7006ced0 2023-12-19T09:02:42.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e6dffb700 1 -- 172.21.15.111:0/727960405 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e680092e0 con 0x7f2e7006ced0 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- 172.21.15.111:0/727960405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006ced0 msgr2=0x7f2e700b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 --2- 172.21.15.111:0/727960405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006ced0 0x7f2e700b06f0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2e7006f8c0 tx=0x7f2e68009840 comp rx=0 tx=0).stop 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- 172.21.15.111:0/727960405 shutdown_connections 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 --2- 172.21.15.111:0/727960405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006ced0 0x7f2e700b06f0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 --2- 172.21.15.111:0/727960405 >> v2:172.21.15.143:3300/0 conn(0x7f2e70158fa0 0x7f2e7006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.656+0000 7f2e7562e700 1 -- 172.21.15.111:0/727960405 >> 172.21.15.111:0/727960405 conn(0x7f2e700d0b90 msgr2=0x7f2e700ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.660+0000 7f2e7562e700 1 -- 172.21.15.111:0/727960405 shutdown_connections 2023-12-19T09:02:42.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.660+0000 7f2e7562e700 1 -- 172.21.15.111:0/727960405 wait complete. 2023-12-19T09:02:42.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.660+0000 7f2e7562e700 1 Processor -- start 2023-12-19T09:02:42.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.660+0000 7f2e7562e700 1 -- start start 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e70158fa0 0x7f2e700c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700cb490 con 0x7f2e7006d8b0 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700c7820 con 0x7f2e7006ced0 2023-12-19T09:02:42.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e7562e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70157e10 con 0x7f2e70158fa0 2023-12-19T09:02:42.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49320/0 (socket says 172.21.15.111:49320) 2023-12-19T09:02:42.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52448/0 (socket says 172.21.15.111:52448) 2023-12-19T09:02:42.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6effd700 1 -- 172.21.15.111:0/3933373889 learned_addr learned my addr 172.21.15.111:0/3933373889 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:42.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6f7fe700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e70158fa0 0x7f2e700c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 -- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e70158fa0 msgr2=0x7f2e700c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e70158fa0 0x7f2e700c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 -- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 msgr2=0x7f2e700caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:42.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:42.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 -- 172.21.15.111:0/3933373889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009ae0 con 0x7f2e7006d8b0 2023-12-19T09:02:42.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6effd700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:42.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.664+0000 7f2e6e7fc700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2e7006dcb0 tx=0x7f2e64007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400ce60 con 0x7f2e7006d8b0 2023-12-19T09:02:42.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64011590 con 0x7f2e7006d8b0 2023-12-19T09:02:42.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70169be0 con 0x7f2e7006d8b0 2023-12-19T09:02:42.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64011930 con 0x7f2e7006d8b0 2023-12-19T09:02:42.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e700c71d0 con 0x7f2e7006d8b0 2023-12-19T09:02:42.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.668+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e700c79b0 con 0x7f2e7006d8b0 2023-12-19T09:02:42.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.672+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2e6401c460 con 0x7f2e7006d8b0 2023-12-19T09:02:42.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.672+0000 7f2e56ffd700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 0x7f2e5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:42.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.672+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f2e6407c800 con 0x7f2e7006d8b0 2023-12-19T09:02:42.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.672+0000 7f2e6effd700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 0x7f2e5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:42.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.676+0000 7f2e6effd700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 0x7f2e5c062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006fd0 tx=0x7f2e60008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:42.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.680+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e6404c8e0 con 0x7f2e7006d8b0 2023-12-19T09:02:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:42.888+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2e70071120 con 0x7f2e7006d8b0 2023-12-19T09:02:43.537 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T09:02:43.532+0000 7fc8a43f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:02:43.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.688+0000 7f2e56ffd700 1 -- 172.21.15.111:0/3933373889 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v71) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2e6404ff00 con 0x7f2e7006d8b0 2023-12-19T09:02:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 msgr2=0x7f2e5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 0x7f2e5c062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006fd0 tx=0x7f2e60008040 comp rx=0 tx=0).stop 2023-12-19T09:02:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 msgr2=0x7f2e700c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2e7006dcb0 tx=0x7f2e64007ea0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 shutdown_connections 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2e5c060670 0x7f2e5c062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e70158fa0 0x7f2e700c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e7006d8b0 0x7f2e700c7cb0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 --2- 172.21.15.111:0/3933373889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e7006ced0 0x7f2e700caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 >> 172.21.15.111:0/3933373889 conn(0x7f2e700d0b90 msgr2=0x7f2e70158350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 shutdown_connections 2023-12-19T09:02:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.692+0000 7f2e7562e700 1 -- 172.21.15.111:0/3933373889 wait complete. 2023-12-19T09:02:43.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T09:02:43.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.888+0000 7f38852ae700 1 Processor -- start 2023-12-19T09:02:43.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.888+0000 7f38852ae700 1 -- start start 2023-12-19T09:02:43.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f38852ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f38800ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f38852ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38800cb970 0x7f38800ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f38852ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38800cbdf0 con 0x7f38800ca2c0 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f38852ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38800cc0d0 con 0x7f38800cb970 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f38852ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800cc3b0 con 0x7f3880158e20 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f38800ca2c0 0x7f388006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46922/0 (socket says 172.21.15.111:46922) 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387f7fe700 1 -- 172.21.15.111:0/3882958793 learned_addr learned my addr 172.21.15.111:0/3882958793 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387effd700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f38800ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387e7fc700 1 --2- 172.21.15.111:0/3882958793 >> v2:172.21.15.143:3300/0 conn(0x7f38800cb970 0x7f38800ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079541252 0 0) 0x7f38800cbdf0 con 0x7f38800ca2c0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868003570 con 0x7f38800ca2c0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223976980 0 0) 0x7f3868003570 con 0x7f38800ca2c0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 msgr2=0x7f38800ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f38800ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 >> v2:172.21.15.143:3300/0 conn(0x7f38800cb970 msgr2=0x7f38800ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 --2- 172.21.15.111:0/3882958793 >> v2:172.21.15.143:3300/0 conn(0x7f38800cb970 0x7f38800ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800cc690 con 0x7f38800ca2c0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3878001810 con 0x7f38800ca2c0 2023-12-19T09:02:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 >> v1:172.21.15.111:6789/0 conn(0x7f38800ca2c0 legacy=0x7f388006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f3868003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3868003fe0 0x7f3868004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3868004d50 0x7f38680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800cc0d0 con 0x7f3868003fe0 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800cc3b0 con 0x7f3868004d50 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386800ba90 con 0x7f3880158e20 2023-12-19T09:02:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387e7fc700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3868003fe0 0x7f3868004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387effd700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f3868003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.892+0000 7f387f7fe700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3868004d50 0x7f38680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 -- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 msgr2=0x7f3868003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3880158e20 0x7f3868003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 -- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3868004d50 msgr2=0x7f38680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3868004d50 0x7f38680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 -- 172.21.15.111:0/3882958793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868003570 con 0x7f3868003fe0 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387f7fe700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3868004d50 0x7f38680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387e7fc700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3868003fe0 0x7f3868004810 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3874009c00 tx=0x7f38740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94abfed6fd24e56 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387400e6b0 con 0x7f3868003fe0 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387400e810 con 0x7f3868003fe0 2023-12-19T09:02:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387dffb700 1 -- 172.21.15.111:0/3882958793 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387400ebd0 con 0x7f3868003fe0 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 -- 172.21.15.111:0/3882958793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3868003fe0 msgr2=0x7f3868004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 --2- 172.21.15.111:0/3882958793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3868003fe0 0x7f3868004810 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3874009c00 tx=0x7f38740091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f387f7fe700 1 -- 172.21.15.111:0/3882958793 reap_dead start 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 -- 172.21.15.111:0/3882958793 shutdown_connections 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 -- 172.21.15.111:0/3882958793 >> 172.21.15.111:0/3882958793 conn(0x7f3880067bb0 msgr2=0x7f38800d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 -- 172.21.15.111:0/3882958793 shutdown_connections 2023-12-19T09:02:43.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.896+0000 7f38852ae700 1 -- 172.21.15.111:0/3882958793 wait complete. 2023-12-19T09:02:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 Processor -- start 2023-12-19T09:02:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 -- start start 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3868003fe0 0x7f3880169b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3880154db0 0x7f3880155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388015c020 con 0x7f388016c560 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388016d0e0 con 0x7f3880154db0 2023-12-19T09:02:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f38852ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388016d3c0 con 0x7f3868003fe0 2023-12-19T09:02:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.900+0000 7f387f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52462/0 (socket says 172.21.15.111:52462) 2023-12-19T09:02:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 -- 172.21.15.111:0/1643076624 learned_addr learned my addr 172.21.15.111:0/1643076624 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387e7fc700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3880154db0 0x7f3880155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387effd700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3868003fe0 0x7f3880169b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 -- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3868003fe0 msgr2=0x7f3880169b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3868003fe0 0x7f3880169b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 -- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3880154db0 msgr2=0x7f3880155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3880154db0 0x7f3880155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 -- 172.21.15.111:0/1643076624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880169070 con 0x7f388016c560 2023-12-19T09:02:43.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f387f7fe700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3878001460 tx=0x7f3878001280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800eb10 con 0x7f388016c560 2023-12-19T09:02:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387800ec70 con 0x7f388016c560 2023-12-19T09:02:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3880169350 con 0x7f388016c560 2023-12-19T09:02:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878013420 con 0x7f388016c560 2023-12-19T09:02:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38800b5290 con 0x7f388016c560 2023-12-19T09:02:43.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.904+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38801697d0 con 0x7f388016c560 2023-12-19T09:02:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.908+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3878013870 con 0x7f388016c560 2023-12-19T09:02:43.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.908+0000 7f3866ffd700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 0x7f386c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:43.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.908+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f387807d200 con 0x7f388016c560 2023-12-19T09:02:43.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.908+0000 7f387effd700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 0x7f386c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:43.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.912+0000 7f387effd700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 0x7f386c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3870006f00 tx=0x7f38700091a0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:43.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:43.916+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f387804d2e0 con 0x7f388016c560 2023-12-19T09:02:44.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.128+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3880071120 con 0x7f388016c560 2023-12-19T09:02:44.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.692+0000 7f3866ffd700 1 -- 172.21.15.111:0/1643076624 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v72) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3878050900 con 0x7f388016c560 2023-12-19T09:02:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 msgr2=0x7f386c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 0x7f386c062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3870006f00 tx=0x7f38700091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 msgr2=0x7f388016c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3878001460 tx=0x7f3878001280 comp rx=0 tx=0).stop 2023-12-19T09:02:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 shutdown_connections 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f386c060670 0x7f386c062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3880154db0 0x7f3880155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f388016c560 0x7f388016c9e0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 --2- 172.21.15.111:0/1643076624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3868003fe0 0x7f3880169b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 >> 172.21.15.111:0/1643076624 conn(0x7f3880067bb0 msgr2=0x7f38800cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 shutdown_connections 2023-12-19T09:02:44.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.696+0000 7f38852ae700 1 -- 172.21.15.111:0/1643076624 wait complete. 2023-12-19T09:02:44.725 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:44.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 Processor -- start 2023-12-19T09:02:44.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 -- start start 2023-12-19T09:02:44.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdaf8140ae0 0x7fdaf80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdaf8068630 0x7fdaf80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf806e150 con 0x7fdaf8140710 2023-12-19T09:02:44.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf8068a70 con 0x7fdaf8140ae0 2023-12-19T09:02:44.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdafe785700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf806d540 con 0x7fdaf8068630 2023-12-19T09:02:44.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.888+0000 7fdaef7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdaf8140ae0 0x7fdaf80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdaf8140ae0 0x7fdaf80680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52464/0 (socket says 172.21.15.111:52464) 2023-12-19T09:02:44.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 -- 172.21.15.111:0/1787513145 learned_addr learned my addr 172.21.15.111:0/1787513145 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:44.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafcd22700 1 --2- 172.21.15.111:0/1787513145 >> v2:172.21.15.143:3300/0 conn(0x7fdaf8068630 0x7fdaf80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 -- 172.21.15.111:0/1787513145 >> v2:172.21.15.143:3300/0 conn(0x7fdaf8068630 msgr2=0x7fdaf80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:44.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/1787513145 >> v2:172.21.15.143:3300/0 conn(0x7fdaf8068630 0x7fdaf80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 -- 172.21.15.111:0/1787513145 >> v1:172.21.15.143:6789/0 conn(0x7fdaf8140710 legacy=0x7fdaf80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 -- 172.21.15.111:0/1787513145 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf806d820 con 0x7fdaf8140ae0 2023-12-19T09:02:44.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/1787513145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8140ae0 0x7fdaf80680f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdaf806bfc0 tx=0x7fdae0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7447a06243ca0d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:44.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaf77fe700 1 -- 172.21.15.111:0/1787513145 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0009e70 con 0x7fdaf8140ae0 2023-12-19T09:02:44.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaf77fe700 1 -- 172.21.15.111:0/1787513145 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae000b860 con 0x7fdaf8140ae0 2023-12-19T09:02:44.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdaf77fe700 1 -- 172.21.15.111:0/1787513145 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae000bc00 con 0x7fdaf8140ae0 2023-12-19T09:02:44.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 -- 172.21.15.111:0/1787513145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8140ae0 msgr2=0x7fdaf80680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 --2- 172.21.15.111:0/1787513145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8140ae0 0x7fdaf80680f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdaf806bfc0 tx=0x7fdae0009250 comp rx=0 tx=0).stop 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 -- 172.21.15.111:0/1787513145 shutdown_connections 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 --2- 172.21.15.111:0/1787513145 >> v2:172.21.15.143:3300/0 conn(0x7fdaf8068630 0x7fdaf80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 --2- 172.21.15.111:0/1787513145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8140ae0 0x7fdaf80680f0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 -- 172.21.15.111:0/1787513145 >> 172.21.15.111:0/1787513145 conn(0x7fdaf80c8850 msgr2=0x7fdaf80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 -- 172.21.15.111:0/1787513145 shutdown_connections 2023-12-19T09:02:44.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.892+0000 7fdafe785700 1 -- 172.21.15.111:0/1787513145 wait complete. 2023-12-19T09:02:44.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 Processor -- start 2023-12-19T09:02:44.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 -- start start 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 0x7fdaf80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf8140ae0 0x7fdaf816a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf816a580 con 0x7fdaf8068630 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80afb20 con 0x7fdaf8140710 2023-12-19T09:02:44.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdafe785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf8157f90 con 0x7fdaf8140ae0 2023-12-19T09:02:44.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.896+0000 7fdaf7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52476/0 (socket says 172.21.15.111:52476) 2023-12-19T09:02:44.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf7fff700 1 -- 172.21.15.111:0/3805160402 learned_addr learned my addr 172.21.15.111:0/3805160402 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:44.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 0x7fdaf80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdafcd22700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf8140ae0 0x7fdaf816a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 -- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf8140ae0 msgr2=0x7fdaf816a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf8140ae0 0x7fdaf816a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 -- 172.21.15.111:0/3805160402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 msgr2=0x7fdaf816aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 -- 172.21.15.111:0/3805160402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00094f0 con 0x7fdaf8140710 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf7fff700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:02:44.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaef7fe700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 0x7fdaf80aeba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8068c30 tx=0x7fdae0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:44.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae000bbe0 con 0x7fdaf8140710 2023-12-19T09:02:44.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae0000c90 con 0x7fdaf8140710 2023-12-19T09:02:44.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0011650 con 0x7fdaf8140710 2023-12-19T09:02:44.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf80af280 con 0x7fdaf8140710 2023-12-19T09:02:44.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.900+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf80af840 con 0x7fdaf8140710 2023-12-19T09:02:44.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.904+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf8071120 con 0x7fdaf8140710 2023-12-19T09:02:44.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.904+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fdae0011a40 con 0x7fdaf8140710 2023-12-19T09:02:44.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.904+0000 7fdaf4ff9700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 0x7fdad8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:44.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.904+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fdae007e460 con 0x7fdaf8140710 2023-12-19T09:02:44.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.904+0000 7fdaf7fff700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 0x7fdad8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:44.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.908+0000 7fdaf7fff700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 0x7fdad8062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdae8001df0 tx=0x7fdae8009040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:44.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:44.912+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdae004e540 con 0x7fdaf8140710 2023-12-19T09:02:45.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.120+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaf80642a0 con 0x7fdaf8140710 2023-12-19T09:02:45.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.120+0000 7fdaf4ff9700 1 -- 172.21.15.111:0/3805160402 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fdae0051b60 con 0x7fdaf8140710 2023-12-19T09:02:45.127 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:45.127 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:02:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 msgr2=0x7fdad8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 0x7fdad8062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdae8001df0 tx=0x7fdae8009040 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 msgr2=0x7fdaf80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 0x7fdaf80aeba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8068c30 tx=0x7fdae0017000 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 shutdown_connections 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdad8060670 0x7fdad8062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf8140710 0x7fdaf80aeba0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf8140ae0 0x7fdaf816a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 --2- 172.21.15.111:0/3805160402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaf8068630 0x7fdaf816aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 >> 172.21.15.111:0/3805160402 conn(0x7fdaf80c8850 msgr2=0x7fdaf80cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:45.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 shutdown_connections 2023-12-19T09:02:45.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.124+0000 7fdafe785700 1 -- 172.21.15.111:0/3805160402 wait complete. 2023-12-19T09:02:45.132 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:02:45.150 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:45.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.304+0000 7f8e37123700 1 Processor -- start 2023-12-19T09:02:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.304+0000 7f8e37123700 1 -- start start 2023-12-19T09:02:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e37123700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e30158d60 0x7f8e3015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e37123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3006b7a0 0x7f8e300c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e37123700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e300c5610 con 0x7f8e300c51b0 2023-12-19T09:02:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e37123700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3006c7c0 con 0x7f8e30158d60 2023-12-19T09:02:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e37123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3006caa0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e35920700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3006b7a0 0x7f8e300c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e30158d60 0x7f8e3015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e30158d60 0x7f8e3015c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49346/0 (socket says 172.21.15.111:49346) 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 -- 172.21.15.111:0/1554986167 learned_addr learned my addr 172.21.15.111:0/1554986167 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36922700 1 --1- 172.21.15.111:0/1554986167 >> v1:172.21.15.143:6789/0 conn(0x7f8e300c51b0 0x7f8e3006a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:36706/0 (socket says 172.21.15.111:36706) 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 -- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3006b7a0 msgr2=0x7f8e300c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:45.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 --2- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3006b7a0 0x7f8e300c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 -- 172.21.15.111:0/1554986167 >> v1:172.21.15.143:6789/0 conn(0x7f8e300c51b0 legacy=0x7f8e3006a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.308+0000 7f8e36121700 1 -- 172.21.15.111:0/1554986167 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3015ce60 con 0x7f8e30158d60 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e36121700 1 --2- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e3015c920 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0099b0 tx=0x7f8e2c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70a2f61473095b35 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e3511f700 1 -- 172.21.15.111:0/1554986167 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c009ea0 con 0x7f8e30158d60 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e3511f700 1 -- 172.21.15.111:0/1554986167 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c00b860 con 0x7f8e30158d60 2023-12-19T09:02:45.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e3511f700 1 -- 172.21.15.111:0/1554986167 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00bc20 con 0x7f8e30158d60 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 -- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 msgr2=0x7f8e3015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 --2- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e3015c920 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0099b0 tx=0x7f8e2c009250 comp rx=0 tx=0).stop 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 -- 172.21.15.111:0/1554986167 shutdown_connections 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 --2- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3006b7a0 0x7f8e300c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 --2- 172.21.15.111:0/1554986167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e3015c920 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 -- 172.21.15.111:0/1554986167 >> 172.21.15.111:0/1554986167 conn(0x7f8e30067bb0 msgr2=0x7f8e300cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 -- 172.21.15.111:0/1554986167 shutdown_connections 2023-12-19T09:02:45.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.312+0000 7f8e37123700 1 -- 172.21.15.111:0/1554986167 wait complete. 2023-12-19T09:02:45.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 Processor -- start 2023-12-19T09:02:45.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 -- start start 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e300c51b0 0x7f8e3014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e301580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3014f890 con 0x7f8e3006b7a0 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30150360 con 0x7f8e30158d60 2023-12-19T09:02:45.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e37123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e301592f0 con 0x7f8e300c51b0 2023-12-19T09:02:45.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52486/0 (socket says 172.21.15.111:52486) 2023-12-19T09:02:45.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 -- 172.21.15.111:0/314349164 learned_addr learned my addr 172.21.15.111:0/314349164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:45.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e35920700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e300c51b0 0x7f8e3014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36922700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e301580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 -- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e300c51b0 msgr2=0x7f8e3014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e300c51b0 0x7f8e3014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 -- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 msgr2=0x7f8e301580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e301580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.316+0000 7f8e36121700 1 -- 172.21.15.111:0/314349164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c0094f0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e36121700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00b3c0 tx=0x7f8e2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c01f070 con 0x7f8e3006b7a0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c006b10 con 0x7f8e3006b7a0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00bc20 con 0x7f8e3006b7a0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e301585e0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3014fda0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3004be40 con 0x7f8e3006b7a0 2023-12-19T09:02:45.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.320+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8e2c011630 con 0x7f8e3006b7a0 2023-12-19T09:02:45.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.324+0000 7f8e267fc700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 0x7f8e14062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.324+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f8e2c07dea0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.324+0000 7f8e35920700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 0x7f8e14062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.324+0000 7f8e35920700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 0x7f8e14062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e3014f230 tx=0x7f8e200091a0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.328+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e2c04e0a0 con 0x7f8e3006b7a0 2023-12-19T09:02:45.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.496+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e3015d130 con 0x7f8e3006b7a0 2023-12-19T09:02:45.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e267fc700 1 -- 172.21.15.111:0/314349164 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16451 (secure 0 0 0) 0x7f8e2c015090 con 0x7f8e3006b7a0 2023-12-19T09:02:45.505 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:45.505 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":161,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:42.665425+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 msgr2=0x7f8e14062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 0x7f8e14062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8e3014f230 tx=0x7f8e200091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 msgr2=0x7f8e3014e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00b3c0 tx=0x7f8e2c017000 comp rx=0 tx=0).stop 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 shutdown_connections 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8e14060670 0x7f8e14062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e30158d60 0x7f8e301580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e300c51b0 0x7f8e3014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 --2- 172.21.15.111:0/314349164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3006b7a0 0x7f8e3014e320 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 >> 172.21.15.111:0/314349164 conn(0x7f8e30067bb0 msgr2=0x7f8e300cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:45.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 shutdown_connections 2023-12-19T09:02:45.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.500+0000 7f8e37123700 1 -- 172.21.15.111:0/314349164 wait complete. 2023-12-19T09:02:45.526 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T09:02:45.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:02:45.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.680+0000 7f0199af0700 1 Processor -- start 2023-12-19T09:02:45.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.680+0000 7f0199af0700 1 -- start start 2023-12-19T09:02:45.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.680+0000 7f0199af0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01940c2980 0x7f01940aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0199af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 0x7f0194068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0199af0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019406c9f0 con 0x7f019413f890 2023-12-19T09:02:45.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0199af0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01940c7f20 con 0x7f01940c2980 2023-12-19T09:02:45.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0199af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0194068a50 con 0x7f01940680f0 2023-12-19T09:02:45.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f01937fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f019413f890 0x7f01940c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46938/0 (socket says 172.21.15.111:46938) 2023-12-19T09:02:45.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f01937fe700 1 -- 172.21.15.111:0/4237157504 learned_addr learned my addr 172.21.15.111:0/4237157504 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 --2- 172.21.15.111:0/4237157504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 0x7f0194068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f018affd700 1 --2- 172.21.15.111:0/4237157504 >> v2:172.21.15.143:3300/0 conn(0x7f01940c2980 0x7f01940aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0192ffd700 1 -- 172.21.15.111:0/4237157504 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689784359 0 0) 0x7f019406c9f0 con 0x7f019413f890 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0192ffd700 1 -- 172.21.15.111:0/4237157504 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003570 con 0x7f019413f890 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 -- 172.21.15.111:0/4237157504 >> v2:172.21.15.143:3300/0 conn(0x7f01940c2980 msgr2=0x7f01940aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 --2- 172.21.15.111:0/4237157504 >> v2:172.21.15.143:3300/0 conn(0x7f01940c2980 0x7f01940aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 -- 172.21.15.111:0/4237157504 >> v1:172.21.15.111:6789/0 conn(0x7f019413f890 legacy=0x7f01940c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 -- 172.21.15.111:0/4237157504 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940c1c50 con 0x7f01940680f0 2023-12-19T09:02:45.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0193fff700 1 --2- 172.21.15.111:0/4237157504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 0x7f0194068510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098e0 tx=0x7f018c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde5a1f966475531 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0192ffd700 1 -- 172.21.15.111:0/4237157504 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c011630 con 0x7f01940680f0 2023-12-19T09:02:45.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0192ffd700 1 -- 172.21.15.111:0/4237157504 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c011790 con 0x7f01940680f0 2023-12-19T09:02:45.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.684+0000 7f0192ffd700 1 -- 172.21.15.111:0/4237157504 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c011b30 con 0x7f01940680f0 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- 172.21.15.111:0/4237157504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 msgr2=0x7f0194068510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 --2- 172.21.15.111:0/4237157504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 0x7f0194068510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098e0 tx=0x7f018c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- 172.21.15.111:0/4237157504 shutdown_connections 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 --2- 172.21.15.111:0/4237157504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940680f0 0x7f0194068510 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 --2- 172.21.15.111:0/4237157504 >> v2:172.21.15.143:3300/0 conn(0x7f01940c2980 0x7f01940aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- 172.21.15.111:0/4237157504 >> 172.21.15.111:0/4237157504 conn(0x7f01940c8850 msgr2=0x7f01940c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- 172.21.15.111:0/4237157504 shutdown_connections 2023-12-19T09:02:45.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- 172.21.15.111:0/4237157504 wait complete. 2023-12-19T09:02:45.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 Processor -- start 2023-12-19T09:02:45.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.688+0000 7f0199af0700 1 -- start start 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940c2980 0x7f0194155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f019413f890 0x7f01941578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0194158ed0 con 0x7f01940680f0 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0194155d50 con 0x7f019413f890 2023-12-19T09:02:45.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0199af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01941582c0 con 0x7f01940c2980 2023-12-19T09:02:45.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52492/0 (socket says 172.21.15.111:52492) 2023-12-19T09:02:45.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 -- 172.21.15.111:0/2766003279 learned_addr learned my addr 172.21.15.111:0/2766003279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f0193fff700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f019413f890 0x7f01941578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f018affd700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940c2980 0x7f0194155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 -- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940c2980 msgr2=0x7f0194155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940c2980 0x7f0194155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 -- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f019413f890 msgr2=0x7f01941578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f019413f890 0x7f01941578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 -- 172.21.15.111:0/2766003279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c009420 con 0x7f01940680f0 2023-12-19T09:02:45.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.692+0000 7f01937fe700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f018400cc40 tx=0x7f0184001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.696+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018400a8b0 con 0x7f01940680f0 2023-12-19T09:02:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.696+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018400aa30 con 0x7f01940680f0 2023-12-19T09:02:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.696+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018400adf0 con 0x7f01940680f0 2023-12-19T09:02:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.696+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01941585a0 con 0x7f01940680f0 2023-12-19T09:02:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.696+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0194158b60 con 0x7f01940680f0 2023-12-19T09:02:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.700+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194071120 con 0x7f01940680f0 2023-12-19T09:02:45.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.700+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0184007300 con 0x7f01940680f0 2023-12-19T09:02:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.700+0000 7f018bfff700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 0x7f0174062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.700+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f018407cfc0 con 0x7f01940680f0 2023-12-19T09:02:45.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.700+0000 7f018affd700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 0x7f0174062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:45.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.704+0000 7f018affd700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 0x7f0174062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01940c6800 tx=0x7f017c0091a0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:45.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.704+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f018404d270 con 0x7f01940680f0 2023-12-19T09:02:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.868+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0194159010 con 0x7f0174060670 2023-12-19T09:02:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.892+0000 7f018bfff700 1 -- 172.21.15.111:0/2766003279 <== mgr.7493 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286590 (secure 0 0 0) 0x7f017c011a20 con 0x7f0174060670 2023-12-19T09:02:45.903 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 msgr2=0x7f0174062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 0x7f0174062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01940c6800 tx=0x7f017c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 msgr2=0x7f0194129f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f018400cc40 tx=0x7f0184001a70 comp rx=0 tx=0).stop 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 shutdown_connections 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0174060670 0x7f0174062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f019413f890 0x7f01941578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01940c2980 0x7f0194155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 --2- 172.21.15.111:0/2766003279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01940680f0 0x7f0194129f10 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 >> 172.21.15.111:0/2766003279 conn(0x7f01940c8850 msgr2=0x7f01940c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 shutdown_connections 2023-12-19T09:02:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:45.900+0000 7f0199af0700 1 -- 172.21.15.111:0/2766003279 wait complete. 2023-12-19T09:02:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:02:45.923 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T09:02:45.179423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460668,"kb_used_data":16828,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752514052,"statfs":{"total":773094113280,"available":770574389248,"internally_reserved":0,"allocated":17231872,"data_stored":9651961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.536804"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695726+0000","last_change":"2023-12-19T09:02:41.688015+0000","last_active":"2023-12-19T09:02:41.695726+0000","last_peered":"2023-12-19T09:02:41.695726+0000","last_clean":"2023-12-19T09:02:41.695726+0000","last_became_active":"2023-12-19T09:02:41.687767+0000","last_became_peered":"2023-12-19T09:02:41.687767+0000","last_unstale":"2023-12-19T09:02:41.695726+0000","last_undegraded":"2023-12-19T09:02:41.695726+0000","last_fullsized":"2023-12-19T09:02:41.695726+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:27:54.900814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686131+0000","last_change":"2023-12-19T09:02:41.681865+0000","last_active":"2023-12-19T09:02:41.686131+0000","last_peered":"2023-12-19T09:02:41.686131+0000","last_clean":"2023-12-19T09:02:41.686131+0000","last_became_active":"2023-12-19T09:02:41.681354+0000","last_became_peered":"2023-12-19T09:02:41.681354+0000","last_unstale":"2023-12-19T09:02:41.686131+0000","last_undegraded":"2023-12-19T09:02:41.686131+0000","last_fullsized":"2023-12-19T09:02:41.686131+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:39.353297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695137+0000","last_change":"2023-12-19T09:02:41.695137+0000","last_active":"2023-12-19T09:02:41.695137+0000","last_peered":"2023-12-19T09:02:41.695137+0000","last_clean":"2023-12-19T09:02:41.695137+0000","last_became_active":"2023-12-19T09:02:41.694795+0000","last_became_peered":"2023-12-19T09:02:41.694795+0000","last_unstale":"2023-12-19T09:02:41.695137+0000","last_undegraded":"2023-12-19T09:02:41.695137+0000","last_fullsized":"2023-12-19T09:02:41.695137+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:39.732982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695957+0000","last_change":"2023-12-19T09:02:41.693959+0000","last_active":"2023-12-19T09:02:41.695957+0000","last_peered":"2023-12-19T09:02:41.695957+0000","last_clean":"2023-12-19T09:02:41.695957+0000","last_became_active":"2023-12-19T09:02:41.693672+0000","last_became_peered":"2023-12-19T09:02:41.693672+0000","last_unstale":"2023-12-19T09:02:41.695957+0000","last_undegraded":"2023-12-19T09:02:41.695957+0000","last_fullsized":"2023-12-19T09:02:41.695957+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:05.643249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691606+0000","last_change":"2023-12-19T09:02:41.683890+0000","last_active":"2023-12-19T09:02:41.691606+0000","last_peered":"2023-12-19T09:02:41.691606+0000","last_clean":"2023-12-19T09:02:41.691606+0000","last_became_active":"2023-12-19T09:02:41.683573+0000","last_became_peered":"2023-12-19T09:02:41.683573+0000","last_unstale":"2023-12-19T09:02:41.691606+0000","last_undegraded":"2023-12-19T09:02:41.691606+0000","last_fullsized":"2023-12-19T09:02:41.691606+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:56.665507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.694176+0000","last_change":"2023-12-19T09:02:41.694176+0000","last_active":"2023-12-19T09:02:41.694176+0000","last_peered":"2023-12-19T09:02:41.694176+0000","last_clean":"2023-12-19T09:02:41.694176+0000","last_became_active":"2023-12-19T09:02:41.693890+0000","last_became_peered":"2023-12-19T09:02:41.693890+0000","last_unstale":"2023-12-19T09:02:41.694176+0000","last_undegraded":"2023-12-19T09:02:41.694176+0000","last_fullsized":"2023-12-19T09:02:41.694176+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:54.108905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697884+0000","last_change":"2023-12-19T09:02:41.689556+0000","last_active":"2023-12-19T09:02:41.697884+0000","last_peered":"2023-12-19T09:02:41.697884+0000","last_clean":"2023-12-19T09:02:41.697884+0000","last_became_active":"2023-12-19T09:02:41.689214+0000","last_became_peered":"2023-12-19T09:02:41.689214+0000","last_unstale":"2023-12-19T09:02:41.697884+0000","last_undegraded":"2023-12-19T09:02:41.697884+0000","last_fullsized":"2023-12-19T09:02:41.697884+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:46:06.299337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678924+0000","last_change":"2023-12-19T09:02:41.678924+0000","last_active":"2023-12-19T09:02:41.678924+0000","last_peered":"2023-12-19T09:02:41.678924+0000","last_clean":"2023-12-19T09:02:41.678924+0000","last_became_active":"2023-12-19T09:02:41.678628+0000","last_became_peered":"2023-12-19T09:02:41.678628+0000","last_unstale":"2023-12-19T09:02:41.678924+0000","last_undegraded":"2023-12-19T09:02:41.678924+0000","last_fullsized":"2023-12-19T09:02:41.678924+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:21.404408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687870+0000","last_change":"2023-12-19T09:02:41.687870+0000","last_active":"2023-12-19T09:02:41.687870+0000","last_peered":"2023-12-19T09:02:41.687870+0000","last_clean":"2023-12-19T09:02:41.687870+0000","last_became_active":"2023-12-19T09:02:41.687532+0000","last_became_peered":"2023-12-19T09:02:41.687532+0000","last_unstale":"2023-12-19T09:02:41.687870+0000","last_undegraded":"2023-12-19T09:02:41.687870+0000","last_fullsized":"2023-12-19T09:02:41.687870+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:07:44.794051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.668389+0000","last_change":"2023-12-19T09:02:41.668389+0000","last_active":"2023-12-19T09:02:41.668389+0000","last_peered":"2023-12-19T09:02:41.668389+0000","last_clean":"2023-12-19T09:02:41.668389+0000","last_became_active":"2023-12-19T09:02:41.668075+0000","last_became_peered":"2023-12-19T09:02:41.668075+0000","last_unstale":"2023-12-19T09:02:41.668389+0000","last_undegraded":"2023-12-19T09:02:41.668389+0000","last_fullsized":"2023-12-19T09:02:41.668389+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:01.696144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688648+0000","last_change":"2023-12-19T09:02:41.687249+0000","last_active":"2023-12-19T09:02:41.688648+0000","last_peered":"2023-12-19T09:02:41.688648+0000","last_clean":"2023-12-19T09:02:41.688648+0000","last_became_active":"2023-12-19T09:02:41.686971+0000","last_became_peered":"2023-12-19T09:02:41.686971+0000","last_unstale":"2023-12-19T09:02:41.688648+0000","last_undegraded":"2023-12-19T09:02:41.688648+0000","last_fullsized":"2023-12-19T09:02:41.688648+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:42.638629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690440+0000","last_change":"2023-12-19T09:02:41.684465+0000","last_active":"2023-12-19T09:02:41.690440+0000","last_peered":"2023-12-19T09:02:41.690440+0000","last_clean":"2023-12-19T09:02:41.690440+0000","last_became_active":"2023-12-19T09:02:41.684114+0000","last_became_peered":"2023-12-19T09:02:41.684114+0000","last_unstale":"2023-12-19T09:02:41.690440+0000","last_undegraded":"2023-12-19T09:02:41.690440+0000","last_fullsized":"2023-12-19T09:02:41.690440+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:38.150197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686942+0000","last_change":"2023-12-19T09:02:41.686942+0000","last_active":"2023-12-19T09:02:41.686942+0000","last_peered":"2023-12-19T09:02:41.686942+0000","last_clean":"2023-12-19T09:02:41.686942+0000","last_became_active":"2023-12-19T09:02:41.684704+0000","last_became_peered":"2023-12-19T09:02:41.684704+0000","last_unstale":"2023-12-19T09:02:41.686942+0000","last_undegraded":"2023-12-19T09:02:41.686942+0000","last_fullsized":"2023-12-19T09:02:41.686942+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:36.506687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691284+0000","last_change":"2023-12-19T09:02:41.681925+0000","last_active":"2023-12-19T09:02:41.691284+0000","last_peered":"2023-12-19T09:02:41.691284+0000","last_clean":"2023-12-19T09:02:41.691284+0000","last_became_active":"2023-12-19T09:02:41.681260+0000","last_became_peered":"2023-12-19T09:02:41.681260+0000","last_unstale":"2023-12-19T09:02:41.691284+0000","last_undegraded":"2023-12-19T09:02:41.691284+0000","last_fullsized":"2023-12-19T09:02:41.691284+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:15.669943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690352+0000","last_change":"2023-12-19T09:02:41.681962+0000","last_active":"2023-12-19T09:02:41.690352+0000","last_peered":"2023-12-19T09:02:41.690352+0000","last_clean":"2023-12-19T09:02:41.690352+0000","last_became_active":"2023-12-19T09:02:41.681674+0000","last_became_peered":"2023-12-19T09:02:41.681674+0000","last_unstale":"2023-12-19T09:02:41.690352+0000","last_undegraded":"2023-12-19T09:02:41.690352+0000","last_fullsized":"2023-12-19T09:02:41.690352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:03.605824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.689728+0000","last_change":"2023-12-19T09:02:41.689728+0000","last_active":"2023-12-19T09:02:41.689728+0000","last_peered":"2023-12-19T09:02:41.689728+0000","last_clean":"2023-12-19T09:02:41.689728+0000","last_became_active":"2023-12-19T09:02:41.685378+0000","last_became_peered":"2023-12-19T09:02:41.685378+0000","last_unstale":"2023-12-19T09:02:41.689728+0000","last_undegraded":"2023-12-19T09:02:41.689728+0000","last_fullsized":"2023-12-19T09:02:41.689728+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:36:24.790523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697105+0000","last_change":"2023-12-19T09:02:41.697105+0000","last_active":"2023-12-19T09:02:41.697105+0000","last_peered":"2023-12-19T09:02:41.697105+0000","last_clean":"2023-12-19T09:02:41.697105+0000","last_became_active":"2023-12-19T09:02:41.695330+0000","last_became_peered":"2023-12-19T09:02:41.695330+0000","last_unstale":"2023-12-19T09:02:41.697105+0000","last_undegraded":"2023-12-19T09:02:41.697105+0000","last_fullsized":"2023-12-19T09:02:41.697105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:37.885338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691850+0000","last_change":"2023-12-19T09:02:41.681671+0000","last_active":"2023-12-19T09:02:41.691850+0000","last_peered":"2023-12-19T09:02:41.691850+0000","last_clean":"2023-12-19T09:02:41.691850+0000","last_became_active":"2023-12-19T09:02:41.681368+0000","last_became_peered":"2023-12-19T09:02:41.681368+0000","last_unstale":"2023-12-19T09:02:41.691850+0000","last_undegraded":"2023-12-19T09:02:41.691850+0000","last_fullsized":"2023-12-19T09:02:41.691850+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:21:55.379680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678036+0000","last_change":"2023-12-19T09:02:41.673557+0000","last_active":"2023-12-19T09:02:41.678036+0000","last_peered":"2023-12-19T09:02:41.678036+0000","last_clean":"2023-12-19T09:02:41.678036+0000","last_became_active":"2023-12-19T09:02:41.673225+0000","last_became_peered":"2023-12-19T09:02:41.673225+0000","last_unstale":"2023-12-19T09:02:41.678036+0000","last_undegraded":"2023-12-19T09:02:41.678036+0000","last_fullsized":"2023-12-19T09:02:41.678036+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:39.987960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690829+0000","last_change":"2023-12-19T09:02:41.681001+0000","last_active":"2023-12-19T09:02:41.690829+0000","last_peered":"2023-12-19T09:02:41.690829+0000","last_clean":"2023-12-19T09:02:41.690829+0000","last_became_active":"2023-12-19T09:02:41.680693+0000","last_became_peered":"2023-12-19T09:02:41.680693+0000","last_unstale":"2023-12-19T09:02:41.690829+0000","last_undegraded":"2023-12-19T09:02:41.690829+0000","last_fullsized":"2023-12-19T09:02:41.690829+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:57.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.692064+0000","last_change":"2023-12-19T09:02:41.691014+0000","last_active":"2023-12-19T09:02:41.692064+0000","last_peered":"2023-12-19T09:02:41.692064+0000","last_clean":"2023-12-19T09:02:41.692064+0000","last_became_active":"2023-12-19T09:02:41.690707+0000","last_became_peered":"2023-12-19T09:02:41.690707+0000","last_unstale":"2023-12-19T09:02:41.692064+0000","last_undegraded":"2023-12-19T09:02:41.692064+0000","last_fullsized":"2023-12-19T09:02:41.692064+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:48.584202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.683278+0000","last_change":"2023-12-19T09:02:41.675799+0000","last_active":"2023-12-19T09:02:41.683278+0000","last_peered":"2023-12-19T09:02:41.683278+0000","last_clean":"2023-12-19T09:02:41.683278+0000","last_became_active":"2023-12-19T09:02:41.675477+0000","last_became_peered":"2023-12-19T09:02:41.675477+0000","last_unstale":"2023-12-19T09:02:41.683278+0000","last_undegraded":"2023-12-19T09:02:41.683278+0000","last_fullsized":"2023-12-19T09:02:41.683278+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:21:06.946197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.685157+0000","last_change":"2023-12-19T09:02:41.681347+0000","last_active":"2023-12-19T09:02:41.685157+0000","last_peered":"2023-12-19T09:02:41.685157+0000","last_clean":"2023-12-19T09:02:41.685157+0000","last_became_active":"2023-12-19T09:02:41.681053+0000","last_became_peered":"2023-12-19T09:02:41.681053+0000","last_unstale":"2023-12-19T09:02:41.685157+0000","last_undegraded":"2023-12-19T09:02:41.685157+0000","last_fullsized":"2023-12-19T09:02:41.685157+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:27.797838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690587+0000","last_change":"2023-12-19T09:02:41.690587+0000","last_active":"2023-12-19T09:02:41.690587+0000","last_peered":"2023-12-19T09:02:41.690587+0000","last_clean":"2023-12-19T09:02:41.690587+0000","last_became_active":"2023-12-19T09:02:41.685638+0000","last_became_peered":"2023-12-19T09:02:41.685638+0000","last_unstale":"2023-12-19T09:02:41.690587+0000","last_undegraded":"2023-12-19T09:02:41.690587+0000","last_fullsized":"2023-12-19T09:02:41.690587+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:42.638629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.699620+0000","last_change":"2023-12-19T09:02:41.699620+0000","last_active":"2023-12-19T09:02:41.699620+0000","last_peered":"2023-12-19T09:02:41.699620+0000","last_clean":"2023-12-19T09:02:41.699620+0000","last_became_active":"2023-12-19T09:02:41.699272+0000","last_became_peered":"2023-12-19T09:02:41.699272+0000","last_unstale":"2023-12-19T09:02:41.699620+0000","last_undegraded":"2023-12-19T09:02:41.699620+0000","last_fullsized":"2023-12-19T09:02:41.699620+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.924437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686325+0000","last_change":"2023-12-19T09:02:41.686325+0000","last_active":"2023-12-19T09:02:41.686325+0000","last_peered":"2023-12-19T09:02:41.686325+0000","last_clean":"2023-12-19T09:02:41.686325+0000","last_became_active":"2023-12-19T09:02:41.686014+0000","last_became_peered":"2023-12-19T09:02:41.686014+0000","last_unstale":"2023-12-19T09:02:41.686325+0000","last_undegraded":"2023-12-19T09:02:41.686325+0000","last_fullsized":"2023-12-19T09:02:41.686325+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:52.355309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700220+0000","last_change":"2023-12-19T09:02:41.698088+0000","last_active":"2023-12-19T09:02:41.700220+0000","last_peered":"2023-12-19T09:02:41.700220+0000","last_clean":"2023-12-19T09:02:41.700220+0000","last_became_active":"2023-12-19T09:02:41.697754+0000","last_became_peered":"2023-12-19T09:02:41.697754+0000","last_unstale":"2023-12-19T09:02:41.700220+0000","last_undegraded":"2023-12-19T09:02:41.700220+0000","last_fullsized":"2023-12-19T09:02:41.700220+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:12.018813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671920+0000","last_change":"2023-12-19T09:02:41.671920+0000","last_active":"2023-12-19T09:02:41.671920+0000","last_peered":"2023-12-19T09:02:41.671920+0000","last_clean":"2023-12-19T09:02:41.671920+0000","last_became_active":"2023-12-19T09:02:41.671431+0000","last_became_peered":"2023-12-19T09:02:41.671431+0000","last_unstale":"2023-12-19T09:02:41.671920+0000","last_undegraded":"2023-12-19T09:02:41.671920+0000","last_fullsized":"2023-12-19T09:02:41.671920+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:46:06.299337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686572+0000","last_change":"2023-12-19T09:02:41.686572+0000","last_active":"2023-12-19T09:02:41.686572+0000","last_peered":"2023-12-19T09:02:41.686572+0000","last_clean":"2023-12-19T09:02:41.686572+0000","last_became_active":"2023-12-19T09:02:41.685504+0000","last_became_peered":"2023-12-19T09:02:41.685504+0000","last_unstale":"2023-12-19T09:02:41.686572+0000","last_undegraded":"2023-12-19T09:02:41.686572+0000","last_fullsized":"2023-12-19T09:02:41.686572+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:20.293179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686568+0000","last_change":"2023-12-19T09:02:41.686568+0000","last_active":"2023-12-19T09:02:41.686568+0000","last_peered":"2023-12-19T09:02:41.686568+0000","last_clean":"2023-12-19T09:02:41.686568+0000","last_became_active":"2023-12-19T09:02:41.684935+0000","last_became_peered":"2023-12-19T09:02:41.684935+0000","last_unstale":"2023-12-19T09:02:41.686568+0000","last_undegraded":"2023-12-19T09:02:41.686568+0000","last_fullsized":"2023-12-19T09:02:41.686568+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:31:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697714+0000","last_change":"2023-12-19T09:02:41.697714+0000","last_active":"2023-12-19T09:02:41.697714+0000","last_peered":"2023-12-19T09:02:41.697714+0000","last_clean":"2023-12-19T09:02:41.697714+0000","last_became_active":"2023-12-19T09:02:41.697352+0000","last_became_peered":"2023-12-19T09:02:41.697352+0000","last_unstale":"2023-12-19T09:02:41.697714+0000","last_undegraded":"2023-12-19T09:02:41.697714+0000","last_fullsized":"2023-12-19T09:02:41.697714+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:34.267613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691926+0000","last_change":"2023-12-19T09:02:41.691926+0000","last_active":"2023-12-19T09:02:41.691926+0000","last_peered":"2023-12-19T09:02:41.691926+0000","last_clean":"2023-12-19T09:02:41.691926+0000","last_became_active":"2023-12-19T09:02:41.691568+0000","last_became_peered":"2023-12-19T09:02:41.691568+0000","last_unstale":"2023-12-19T09:02:41.691926+0000","last_undegraded":"2023-12-19T09:02:41.691926+0000","last_fullsized":"2023-12-19T09:02:41.691926+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:39.987960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678358+0000","last_change":"2023-12-19T09:02:41.678358+0000","last_active":"2023-12-19T09:02:41.678358+0000","last_peered":"2023-12-19T09:02:41.678358+0000","last_clean":"2023-12-19T09:02:41.678358+0000","last_became_active":"2023-12-19T09:02:41.678069+0000","last_became_peered":"2023-12-19T09:02:41.678069+0000","last_unstale":"2023-12-19T09:02:41.678358+0000","last_undegraded":"2023-12-19T09:02:41.678358+0000","last_fullsized":"2023-12-19T09:02:41.678358+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:34.297944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"160'4","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700820+0000","last_change":"2023-12-19T09:02:40.679315+0000","last_active":"2023-12-19T09:02:41.700820+0000","last_peered":"2023-12-19T09:02:41.700820+0000","last_clean":"2023-12-19T09:02:41.700820+0000","last_became_active":"2023-12-19T09:02:40.679009+0000","last_became_peered":"2023-12-19T09:02:40.679009+0000","last_unstale":"2023-12-19T09:02:41.700820+0000","last_undegraded":"2023-12-19T09:02:41.700820+0000","last_fullsized":"2023-12-19T09:02:41.700820+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.287384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.682897+0000","last_change":"2023-12-19T09:02:41.676211+0000","last_active":"2023-12-19T09:02:41.682897+0000","last_peered":"2023-12-19T09:02:41.682897+0000","last_clean":"2023-12-19T09:02:41.682897+0000","last_became_active":"2023-12-19T09:02:41.675473+0000","last_became_peered":"2023-12-19T09:02:41.675473+0000","last_unstale":"2023-12-19T09:02:41.682897+0000","last_undegraded":"2023-12-19T09:02:41.682897+0000","last_fullsized":"2023-12-19T09:02:41.682897+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:05:01.925670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691878+0000","last_change":"2023-12-19T09:02:40.676355+0000","last_active":"2023-12-19T09:02:41.691878+0000","last_peered":"2023-12-19T09:02:41.691878+0000","last_clean":"2023-12-19T09:02:41.691878+0000","last_became_active":"2023-12-19T09:02:40.676055+0000","last_became_peered":"2023-12-19T09:02:40.676055+0000","last_unstale":"2023-12-19T09:02:41.691878+0000","last_undegraded":"2023-12-19T09:02:41.691878+0000","last_fullsized":"2023-12-19T09:02:41.691878+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:16.454878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686468+0000","last_change":"2023-12-19T09:02:41.681345+0000","last_active":"2023-12-19T09:02:41.686468+0000","last_peered":"2023-12-19T09:02:41.686468+0000","last_clean":"2023-12-19T09:02:41.686468+0000","last_became_active":"2023-12-19T09:02:41.681027+0000","last_became_peered":"2023-12-19T09:02:41.681027+0000","last_unstale":"2023-12-19T09:02:41.686468+0000","last_undegraded":"2023-12-19T09:02:41.686468+0000","last_fullsized":"2023-12-19T09:02:41.686468+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:58.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.703125+0000","last_change":"2023-12-19T09:02:40.681079+0000","last_active":"2023-12-19T09:02:41.703125+0000","last_peered":"2023-12-19T09:02:41.703125+0000","last_clean":"2023-12-19T09:02:41.703125+0000","last_became_active":"2023-12-19T09:02:40.680802+0000","last_became_peered":"2023-12-19T09:02:40.680802+0000","last_unstale":"2023-12-19T09:02:41.703125+0000","last_undegraded":"2023-12-19T09:02:41.703125+0000","last_fullsized":"2023-12-19T09:02:41.703125+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:26.793552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686113+0000","last_change":"2023-12-19T09:02:41.678854+0000","last_active":"2023-12-19T09:02:41.686113+0000","last_peered":"2023-12-19T09:02:41.686113+0000","last_clean":"2023-12-19T09:02:41.686113+0000","last_became_active":"2023-12-19T09:02:41.678496+0000","last_became_peered":"2023-12-19T09:02:41.678496+0000","last_unstale":"2023-12-19T09:02:41.686113+0000","last_undegraded":"2023-12-19T09:02:41.686113+0000","last_fullsized":"2023-12-19T09:02:41.686113+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:06:56.210764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.701694+0000","last_change":"2023-12-19T09:02:40.679588+0000","last_active":"2023-12-19T09:02:41.701694+0000","last_peered":"2023-12-19T09:02:41.701694+0000","last_clean":"2023-12-19T09:02:41.701694+0000","last_became_active":"2023-12-19T09:02:40.679300+0000","last_became_peered":"2023-12-19T09:02:40.679300+0000","last_unstale":"2023-12-19T09:02:41.701694+0000","last_undegraded":"2023-12-19T09:02:41.701694+0000","last_fullsized":"2023-12-19T09:02:41.701694+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:47.579915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686279+0000","last_change":"2023-12-19T09:02:41.686279+0000","last_active":"2023-12-19T09:02:41.686279+0000","last_peered":"2023-12-19T09:02:41.686279+0000","last_clean":"2023-12-19T09:02:41.686279+0000","last_became_active":"2023-12-19T09:02:41.686011+0000","last_became_peered":"2023-12-19T09:02:41.686011+0000","last_unstale":"2023-12-19T09:02:41.686279+0000","last_undegraded":"2023-12-19T09:02:41.686279+0000","last_fullsized":"2023-12-19T09:02:41.686279+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:19.645398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"160'1","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.660901+0000","last_change":"2023-12-19T09:02:40.669273+0000","last_active":"2023-12-19T09:02:41.660901+0000","last_peered":"2023-12-19T09:02:41.660901+0000","last_clean":"2023-12-19T09:02:41.660901+0000","last_became_active":"2023-12-19T09:02:40.668687+0000","last_became_peered":"2023-12-19T09:02:40.668687+0000","last_unstale":"2023-12-19T09:02:41.660901+0000","last_undegraded":"2023-12-19T09:02:41.660901+0000","last_fullsized":"2023-12-19T09:02:41.660901+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:42.920150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.674412+0000","last_change":"2023-12-19T09:02:41.674412+0000","last_active":"2023-12-19T09:02:41.674412+0000","last_peered":"2023-12-19T09:02:41.674412+0000","last_clean":"2023-12-19T09:02:41.674412+0000","last_became_active":"2023-12-19T09:02:41.674133+0000","last_became_peered":"2023-12-19T09:02:41.674133+0000","last_unstale":"2023-12-19T09:02:41.674412+0000","last_undegraded":"2023-12-19T09:02:41.674412+0000","last_fullsized":"2023-12-19T09:02:41.674412+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:18.930961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690611+0000","last_change":"2023-12-19T09:02:40.667693+0000","last_active":"2023-12-19T09:02:41.690611+0000","last_peered":"2023-12-19T09:02:41.690611+0000","last_clean":"2023-12-19T09:02:41.690611+0000","last_became_active":"2023-12-19T09:02:40.667395+0000","last_became_peered":"2023-12-19T09:02:40.667395+0000","last_unstale":"2023-12-19T09:02:41.690611+0000","last_undegraded":"2023-12-19T09:02:41.690611+0000","last_fullsized":"2023-12-19T09:02:41.690611+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:38.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690037+0000","last_change":"2023-12-19T09:02:41.676813+0000","last_active":"2023-12-19T09:02:41.690037+0000","last_peered":"2023-12-19T09:02:41.690037+0000","last_clean":"2023-12-19T09:02:41.690037+0000","last_became_active":"2023-12-19T09:02:41.676513+0000","last_became_peered":"2023-12-19T09:02:41.676513+0000","last_unstale":"2023-12-19T09:02:41.690037+0000","last_undegraded":"2023-12-19T09:02:41.690037+0000","last_fullsized":"2023-12-19T09:02:41.690037+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:32.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.681791+0000","last_change":"2023-12-19T09:02:40.680571+0000","last_active":"2023-12-19T09:02:40.681791+0000","last_peered":"2023-12-19T09:02:40.681791+0000","last_clean":"2023-12-19T09:02:40.681791+0000","last_became_active":"2023-12-19T09:02:40.680271+0000","last_became_peered":"2023-12-19T09:02:40.680271+0000","last_unstale":"2023-12-19T09:02:40.681791+0000","last_undegraded":"2023-12-19T09:02:40.681791+0000","last_fullsized":"2023-12-19T09:02:40.681791+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:34.307881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697937+0000","last_change":"2023-12-19T09:02:41.691924+0000","last_active":"2023-12-19T09:02:41.697937+0000","last_peered":"2023-12-19T09:02:41.697937+0000","last_clean":"2023-12-19T09:02:41.697937+0000","last_became_active":"2023-12-19T09:02:41.691594+0000","last_became_peered":"2023-12-19T09:02:41.691594+0000","last_unstale":"2023-12-19T09:02:41.697937+0000","last_undegraded":"2023-12-19T09:02:41.697937+0000","last_fullsized":"2023-12-19T09:02:41.697937+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:09.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.683958+0000","last_change":"2023-12-19T09:02:40.682406+0000","last_active":"2023-12-19T09:02:40.683958+0000","last_peered":"2023-12-19T09:02:40.683958+0000","last_clean":"2023-12-19T09:02:40.683958+0000","last_became_active":"2023-12-19T09:02:40.682140+0000","last_became_peered":"2023-12-19T09:02:40.682140+0000","last_unstale":"2023-12-19T09:02:40.683958+0000","last_undegraded":"2023-12-19T09:02:40.683958+0000","last_fullsized":"2023-12-19T09:02:40.683958+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:53.542618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695606+0000","last_change":"2023-12-19T09:02:41.687001+0000","last_active":"2023-12-19T09:02:41.695606+0000","last_peered":"2023-12-19T09:02:41.695606+0000","last_clean":"2023-12-19T09:02:41.695606+0000","last_became_active":"2023-12-19T09:02:41.686681+0000","last_became_peered":"2023-12-19T09:02:41.686681+0000","last_unstale":"2023-12-19T09:02:41.695606+0000","last_undegraded":"2023-12-19T09:02:41.695606+0000","last_fullsized":"2023-12-19T09:02:41.695606+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:44.338604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671837+0000","last_change":"2023-12-19T09:02:40.680017+0000","last_active":"2023-12-19T09:02:41.671837+0000","last_peered":"2023-12-19T09:02:41.671837+0000","last_clean":"2023-12-19T09:02:41.671837+0000","last_became_active":"2023-12-19T09:02:40.679719+0000","last_became_peered":"2023-12-19T09:02:40.679719+0000","last_unstale":"2023-12-19T09:02:41.671837+0000","last_undegraded":"2023-12-19T09:02:41.671837+0000","last_fullsized":"2023-12-19T09:02:41.671837+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:12:42.129093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686175+0000","last_change":"2023-12-19T09:02:41.686175+0000","last_active":"2023-12-19T09:02:41.686175+0000","last_peered":"2023-12-19T09:02:41.686175+0000","last_clean":"2023-12-19T09:02:41.686175+0000","last_became_active":"2023-12-19T09:02:41.682676+0000","last_became_peered":"2023-12-19T09:02:41.682676+0000","last_unstale":"2023-12-19T09:02:41.686175+0000","last_undegraded":"2023-12-19T09:02:41.686175+0000","last_fullsized":"2023-12-19T09:02:41.686175+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:26.456697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664196+0000","last_change":"2023-12-19T09:02:40.670863+0000","last_active":"2023-12-19T09:02:41.664196+0000","last_peered":"2023-12-19T09:02:41.664196+0000","last_clean":"2023-12-19T09:02:41.664196+0000","last_became_active":"2023-12-19T09:02:40.669522+0000","last_became_peered":"2023-12-19T09:02:40.669522+0000","last_unstale":"2023-12-19T09:02:41.664196+0000","last_undegraded":"2023-12-19T09:02:41.664196+0000","last_fullsized":"2023-12-19T09:02:41.664196+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:32:16.204350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.684642+0000","last_change":"2023-12-19T09:02:41.684642+0000","last_active":"2023-12-19T09:02:41.684642+0000","last_peered":"2023-12-19T09:02:41.684642+0000","last_clean":"2023-12-19T09:02:41.684642+0000","last_became_active":"2023-12-19T09:02:41.684303+0000","last_became_peered":"2023-12-19T09:02:41.684303+0000","last_unstale":"2023-12-19T09:02:41.684642+0000","last_undegraded":"2023-12-19T09:02:41.684642+0000","last_fullsized":"2023-12-19T09:02:41.684642+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:14.349193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.702490+0000","last_change":"2023-12-19T09:02:40.682208+0000","last_active":"2023-12-19T09:02:41.702490+0000","last_peered":"2023-12-19T09:02:41.702490+0000","last_clean":"2023-12-19T09:02:41.702490+0000","last_became_active":"2023-12-19T09:02:40.681893+0000","last_became_peered":"2023-12-19T09:02:40.681893+0000","last_unstale":"2023-12-19T09:02:41.702490+0000","last_undegraded":"2023-12-19T09:02:41.702490+0000","last_fullsized":"2023-12-19T09:02:41.702490+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:38.349011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.693169+0000","last_change":"2023-12-19T09:02:41.693169+0000","last_active":"2023-12-19T09:02:41.693169+0000","last_peered":"2023-12-19T09:02:41.693169+0000","last_clean":"2023-12-19T09:02:41.693169+0000","last_became_active":"2023-12-19T09:02:41.692820+0000","last_became_peered":"2023-12-19T09:02:41.692820+0000","last_unstale":"2023-12-19T09:02:41.693169+0000","last_undegraded":"2023-12-19T09:02:41.693169+0000","last_fullsized":"2023-12-19T09:02:41.693169+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:54:52.261512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":16,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.704762+0000","last_change":"2023-12-19T09:02:40.680150+0000","last_active":"2023-12-19T09:02:41.704762+0000","last_peered":"2023-12-19T09:02:41.704762+0000","last_clean":"2023-12-19T09:02:41.704762+0000","last_became_active":"2023-12-19T09:02:40.679876+0000","last_became_peered":"2023-12-19T09:02:40.679876+0000","last_unstale":"2023-12-19T09:02:41.704762+0000","last_undegraded":"2023-12-19T09:02:41.704762+0000","last_fullsized":"2023-12-19T09:02:41.704762+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:35.957548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671706+0000","last_change":"2023-12-19T09:02:41.671706+0000","last_active":"2023-12-19T09:02:41.671706+0000","last_peered":"2023-12-19T09:02:41.671706+0000","last_clean":"2023-12-19T09:02:41.671706+0000","last_became_active":"2023-12-19T09:02:41.671044+0000","last_became_peered":"2023-12-19T09:02:41.671044+0000","last_unstale":"2023-12-19T09:02:41.671706+0000","last_undegraded":"2023-12-19T09:02:41.671706+0000","last_fullsized":"2023-12-19T09:02:41.671706+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:17.459164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.696824+0000","last_change":"2023-12-19T09:02:40.670168+0000","last_active":"2023-12-19T09:02:41.696824+0000","last_peered":"2023-12-19T09:02:41.696824+0000","last_clean":"2023-12-19T09:02:41.696824+0000","last_became_active":"2023-12-19T09:02:40.669843+0000","last_became_peered":"2023-12-19T09:02:40.669843+0000","last_unstale":"2023-12-19T09:02:41.696824+0000","last_undegraded":"2023-12-19T09:02:41.696824+0000","last_fullsized":"2023-12-19T09:02:41.696824+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:53.017216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.682238+0000","last_change":"2023-12-19T09:02:41.679662+0000","last_active":"2023-12-19T09:02:41.682238+0000","last_peered":"2023-12-19T09:02:41.682238+0000","last_clean":"2023-12-19T09:02:41.682238+0000","last_became_active":"2023-12-19T09:02:41.679255+0000","last_became_peered":"2023-12-19T09:02:41.679255+0000","last_unstale":"2023-12-19T09:02:41.682238+0000","last_undegraded":"2023-12-19T09:02:41.682238+0000","last_fullsized":"2023-12-19T09:02:41.682238+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:17:00.972380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.680582+0000","last_change":"2023-12-19T09:02:40.677683+0000","last_active":"2023-12-19T09:02:40.680582+0000","last_peered":"2023-12-19T09:02:40.680582+0000","last_clean":"2023-12-19T09:02:40.680582+0000","last_became_active":"2023-12-19T09:02:40.677345+0000","last_became_peered":"2023-12-19T09:02:40.677345+0000","last_unstale":"2023-12-19T09:02:40.680582+0000","last_undegraded":"2023-12-19T09:02:40.680582+0000","last_fullsized":"2023-12-19T09:02:40.680582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:24.298166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.693434+0000","last_change":"2023-12-19T09:02:41.691385+0000","last_active":"2023-12-19T09:02:41.693434+0000","last_peered":"2023-12-19T09:02:41.693434+0000","last_clean":"2023-12-19T09:02:41.693434+0000","last_became_active":"2023-12-19T09:02:41.690473+0000","last_became_peered":"2023-12-19T09:02:41.690473+0000","last_unstale":"2023-12-19T09:02:41.693434+0000","last_undegraded":"2023-12-19T09:02:41.693434+0000","last_fullsized":"2023-12-19T09:02:41.693434+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:29.217800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":17,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.696668+0000","last_change":"2023-12-19T09:02:40.678650+0000","last_active":"2023-12-19T09:02:41.696668+0000","last_peered":"2023-12-19T09:02:41.696668+0000","last_clean":"2023-12-19T09:02:41.696668+0000","last_became_active":"2023-12-19T09:02:40.678339+0000","last_became_peered":"2023-12-19T09:02:40.678339+0000","last_unstale":"2023-12-19T09:02:41.696668+0000","last_undegraded":"2023-12-19T09:02:41.696668+0000","last_fullsized":"2023-12-19T09:02:41.696668+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:12:42.129093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695536+0000","last_change":"2023-12-19T09:02:41.695536+0000","last_active":"2023-12-19T09:02:41.695536+0000","last_peered":"2023-12-19T09:02:41.695536+0000","last_clean":"2023-12-19T09:02:41.695536+0000","last_became_active":"2023-12-19T09:02:41.695029+0000","last_became_peered":"2023-12-19T09:02:41.695029+0000","last_unstale":"2023-12-19T09:02:41.695536+0000","last_undegraded":"2023-12-19T09:02:41.695536+0000","last_fullsized":"2023-12-19T09:02:41.695536+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:29:38.972931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688643+0000","last_change":"2023-12-19T09:02:40.676639+0000","last_active":"2023-12-19T09:02:41.688643+0000","last_peered":"2023-12-19T09:02:41.688643+0000","last_clean":"2023-12-19T09:02:41.688643+0000","last_became_active":"2023-12-19T09:02:40.675499+0000","last_became_peered":"2023-12-19T09:02:40.675499+0000","last_unstale":"2023-12-19T09:02:41.688643+0000","last_undegraded":"2023-12-19T09:02:41.688643+0000","last_fullsized":"2023-12-19T09:02:41.688643+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687973+0000","last_change":"2023-12-19T09:02:41.682080+0000","last_active":"2023-12-19T09:02:41.687973+0000","last_peered":"2023-12-19T09:02:41.687973+0000","last_clean":"2023-12-19T09:02:41.687973+0000","last_became_active":"2023-12-19T09:02:41.681581+0000","last_became_peered":"2023-12-19T09:02:41.681581+0000","last_unstale":"2023-12-19T09:02:41.687973+0000","last_undegraded":"2023-12-19T09:02:41.687973+0000","last_fullsized":"2023-12-19T09:02:41.687973+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:19.953419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"160'2","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700539+0000","last_change":"2023-12-19T09:02:40.681535+0000","last_active":"2023-12-19T09:02:41.700539+0000","last_peered":"2023-12-19T09:02:41.700539+0000","last_clean":"2023-12-19T09:02:41.700539+0000","last_became_active":"2023-12-19T09:02:40.681249+0000","last_became_peered":"2023-12-19T09:02:40.681249+0000","last_unstale":"2023-12-19T09:02:41.700539+0000","last_undegraded":"2023-12-19T09:02:41.700539+0000","last_fullsized":"2023-12-19T09:02:41.700539+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:44:31.372754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698227+0000","last_change":"2023-12-19T09:02:41.692306+0000","last_active":"2023-12-19T09:02:41.698227+0000","last_peered":"2023-12-19T09:02:41.698227+0000","last_clean":"2023-12-19T09:02:41.698227+0000","last_became_active":"2023-12-19T09:02:41.691997+0000","last_became_peered":"2023-12-19T09:02:41.691997+0000","last_unstale":"2023-12-19T09:02:41.698227+0000","last_undegraded":"2023-12-19T09:02:41.698227+0000","last_fullsized":"2023-12-19T09:02:41.698227+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:58:43.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.669861+0000","last_change":"2023-12-19T09:02:40.677208+0000","last_active":"2023-12-19T09:02:41.669861+0000","last_peered":"2023-12-19T09:02:41.669861+0000","last_clean":"2023-12-19T09:02:41.669861+0000","last_became_active":"2023-12-19T09:02:40.674996+0000","last_became_peered":"2023-12-19T09:02:40.674996+0000","last_unstale":"2023-12-19T09:02:41.669861+0000","last_undegraded":"2023-12-19T09:02:41.669861+0000","last_fullsized":"2023-12-19T09:02:41.669861+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687068+0000","last_change":"2023-12-19T09:02:41.687068+0000","last_active":"2023-12-19T09:02:41.687068+0000","last_peered":"2023-12-19T09:02:41.687068+0000","last_clean":"2023-12-19T09:02:41.687068+0000","last_became_active":"2023-12-19T09:02:41.686750+0000","last_became_peered":"2023-12-19T09:02:41.686750+0000","last_unstale":"2023-12-19T09:02:41.687068+0000","last_undegraded":"2023-12-19T09:02:41.687068+0000","last_fullsized":"2023-12-19T09:02:41.687068+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:17:00.972380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.669437+0000","last_change":"2023-12-19T09:02:40.674613+0000","last_active":"2023-12-19T09:02:41.669437+0000","last_peered":"2023-12-19T09:02:41.669437+0000","last_clean":"2023-12-19T09:02:41.669437+0000","last_became_active":"2023-12-19T09:02:40.674262+0000","last_became_peered":"2023-12-19T09:02:40.674262+0000","last_unstale":"2023-12-19T09:02:41.669437+0000","last_undegraded":"2023-12-19T09:02:41.669437+0000","last_fullsized":"2023-12-19T09:02:41.669437+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:18.949132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.692143+0000","last_change":"2023-12-19T09:02:41.686311+0000","last_active":"2023-12-19T09:02:41.692143+0000","last_peered":"2023-12-19T09:02:41.692143+0000","last_clean":"2023-12-19T09:02:41.692143+0000","last_became_active":"2023-12-19T09:02:41.685951+0000","last_became_peered":"2023-12-19T09:02:41.685951+0000","last_unstale":"2023-12-19T09:02:41.692143+0000","last_undegraded":"2023-12-19T09:02:41.692143+0000","last_fullsized":"2023-12-19T09:02:41.692143+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:28:46.809959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"160'2","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.701556+0000","last_change":"2023-12-19T09:02:40.673205+0000","last_active":"2023-12-19T09:02:41.701556+0000","last_peered":"2023-12-19T09:02:41.701556+0000","last_clean":"2023-12-19T09:02:41.701556+0000","last_became_active":"2023-12-19T09:02:40.672882+0000","last_became_peered":"2023-12-19T09:02:40.672882+0000","last_unstale":"2023-12-19T09:02:41.701556+0000","last_undegraded":"2023-12-19T09:02:41.701556+0000","last_fullsized":"2023-12-19T09:02:41.701556+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:19.288893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687204+0000","last_change":"2023-12-19T09:02:41.678873+0000","last_active":"2023-12-19T09:02:41.687204+0000","last_peered":"2023-12-19T09:02:41.687204+0000","last_clean":"2023-12-19T09:02:41.687204+0000","last_became_active":"2023-12-19T09:02:41.678533+0000","last_became_peered":"2023-12-19T09:02:41.678533+0000","last_unstale":"2023-12-19T09:02:41.687204+0000","last_undegraded":"2023-12-19T09:02:41.687204+0000","last_fullsized":"2023-12-19T09:02:41.687204+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:23.778315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663987+0000","last_change":"2023-12-19T09:02:40.667609+0000","last_active":"2023-12-19T09:02:41.663987+0000","last_peered":"2023-12-19T09:02:41.663987+0000","last_clean":"2023-12-19T09:02:41.663987+0000","last_became_active":"2023-12-19T09:02:40.667322+0000","last_became_peered":"2023-12-19T09:02:40.667322+0000","last_unstale":"2023-12-19T09:02:41.663987+0000","last_undegraded":"2023-12-19T09:02:41.663987+0000","last_fullsized":"2023-12-19T09:02:41.663987+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:39.411131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697411+0000","last_change":"2023-12-19T09:02:41.694968+0000","last_active":"2023-12-19T09:02:41.697411+0000","last_peered":"2023-12-19T09:02:41.697411+0000","last_clean":"2023-12-19T09:02:41.697411+0000","last_became_active":"2023-12-19T09:02:41.694556+0000","last_became_peered":"2023-12-19T09:02:41.694556+0000","last_unstale":"2023-12-19T09:02:41.697411+0000","last_undegraded":"2023-12-19T09:02:41.697411+0000","last_fullsized":"2023-12-19T09:02:41.697411+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:44.056173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664289+0000","last_change":"2023-12-19T09:02:40.681138+0000","last_active":"2023-12-19T09:02:41.664289+0000","last_peered":"2023-12-19T09:02:41.664289+0000","last_clean":"2023-12-19T09:02:41.664289+0000","last_became_active":"2023-12-19T09:02:40.680701+0000","last_became_peered":"2023-12-19T09:02:40.680701+0000","last_unstale":"2023-12-19T09:02:41.664289+0000","last_undegraded":"2023-12-19T09:02:41.664289+0000","last_fullsized":"2023-12-19T09:02:41.664289+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:09:47.280420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695105+0000","last_change":"2023-12-19T09:02:41.692829+0000","last_active":"2023-12-19T09:02:41.695105+0000","last_peered":"2023-12-19T09:02:41.695105+0000","last_clean":"2023-12-19T09:02:41.695105+0000","last_became_active":"2023-12-19T09:02:41.692489+0000","last_became_peered":"2023-12-19T09:02:41.692489+0000","last_unstale":"2023-12-19T09:02:41.695105+0000","last_undegraded":"2023-12-19T09:02:41.695105+0000","last_fullsized":"2023-12-19T09:02:41.695105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:26.456697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.676897+0000","last_change":"2023-12-19T09:02:40.674065+0000","last_active":"2023-12-19T09:02:40.676897+0000","last_peered":"2023-12-19T09:02:40.676897+0000","last_clean":"2023-12-19T09:02:40.676897+0000","last_became_active":"2023-12-19T09:02:40.673072+0000","last_became_peered":"2023-12-19T09:02:40.673072+0000","last_unstale":"2023-12-19T09:02:40.676897+0000","last_undegraded":"2023-12-19T09:02:40.676897+0000","last_fullsized":"2023-12-19T09:02:40.676897+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:33.293657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.689672+0000","last_change":"2023-12-19T09:02:41.674870+0000","last_active":"2023-12-19T09:02:41.689672+0000","last_peered":"2023-12-19T09:02:41.689672+0000","last_clean":"2023-12-19T09:02:41.689672+0000","last_became_active":"2023-12-19T09:02:41.674467+0000","last_became_peered":"2023-12-19T09:02:41.674467+0000","last_unstale":"2023-12-19T09:02:41.689672+0000","last_undegraded":"2023-12-19T09:02:41.689672+0000","last_fullsized":"2023-12-19T09:02:41.689672+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:46.813624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698738+0000","last_change":"2023-12-19T09:02:40.679348+0000","last_active":"2023-12-19T09:02:41.698738+0000","last_peered":"2023-12-19T09:02:41.698738+0000","last_clean":"2023-12-19T09:02:41.698738+0000","last_became_active":"2023-12-19T09:02:40.678397+0000","last_became_peered":"2023-12-19T09:02:40.678397+0000","last_unstale":"2023-12-19T09:02:41.698738+0000","last_undegraded":"2023-12-19T09:02:41.698738+0000","last_fullsized":"2023-12-19T09:02:41.698738+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:35.667111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.680120+0000","last_change":"2023-12-19T09:02:41.680120+0000","last_active":"2023-12-19T09:02:41.680120+0000","last_peered":"2023-12-19T09:02:41.680120+0000","last_clean":"2023-12-19T09:02:41.680120+0000","last_became_active":"2023-12-19T09:02:41.679824+0000","last_became_peered":"2023-12-19T09:02:41.679824+0000","last_unstale":"2023-12-19T09:02:41.680120+0000","last_undegraded":"2023-12-19T09:02:41.680120+0000","last_fullsized":"2023-12-19T09:02:41.680120+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:18.511639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.679113+0000","last_change":"2023-12-19T09:02:40.676423+0000","last_active":"2023-12-19T09:02:40.679113+0000","last_peered":"2023-12-19T09:02:40.679113+0000","last_clean":"2023-12-19T09:02:40.679113+0000","last_became_active":"2023-12-19T09:02:40.676103+0000","last_became_peered":"2023-12-19T09:02:40.676103+0000","last_unstale":"2023-12-19T09:02:40.679113+0000","last_undegraded":"2023-12-19T09:02:40.679113+0000","last_fullsized":"2023-12-19T09:02:40.679113+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:29:37.968645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698249+0000","last_change":"2023-12-19T09:02:41.690109+0000","last_active":"2023-12-19T09:02:41.698249+0000","last_peered":"2023-12-19T09:02:41.698249+0000","last_clean":"2023-12-19T09:02:41.698249+0000","last_became_active":"2023-12-19T09:02:41.688198+0000","last_became_peered":"2023-12-19T09:02:41.688198+0000","last_unstale":"2023-12-19T09:02:41.698249+0000","last_undegraded":"2023-12-19T09:02:41.698249+0000","last_fullsized":"2023-12-19T09:02:41.698249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:17.459164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.677954+0000","last_change":"2023-12-19T09:02:40.673689+0000","last_active":"2023-12-19T09:02:40.677954+0000","last_peered":"2023-12-19T09:02:40.677954+0000","last_clean":"2023-12-19T09:02:40.677954+0000","last_became_active":"2023-12-19T09:02:40.671125+0000","last_became_peered":"2023-12-19T09:02:40.671125+0000","last_unstale":"2023-12-19T09:02:40.677954+0000","last_undegraded":"2023-12-19T09:02:40.677954+0000","last_fullsized":"2023-12-19T09:02:40.677954+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:17.507353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664013+0000","last_change":"2023-12-19T09:02:41.664013+0000","last_active":"2023-12-19T09:02:41.664013+0000","last_peered":"2023-12-19T09:02:41.664013+0000","last_clean":"2023-12-19T09:02:41.664013+0000","last_became_active":"2023-12-19T09:02:41.663710+0000","last_became_peered":"2023-12-19T09:02:41.663710+0000","last_unstale":"2023-12-19T09:02:41.664013+0000","last_undegraded":"2023-12-19T09:02:41.664013+0000","last_fullsized":"2023-12-19T09:02:41.664013+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:22:23.800325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.677065+0000","last_change":"2023-12-19T09:02:40.673552+0000","last_active":"2023-12-19T09:02:40.677065+0000","last_peered":"2023-12-19T09:02:40.677065+0000","last_clean":"2023-12-19T09:02:40.677065+0000","last_became_active":"2023-12-19T09:02:40.673224+0000","last_became_peered":"2023-12-19T09:02:40.673224+0000","last_unstale":"2023-12-19T09:02:40.677065+0000","last_undegraded":"2023-12-19T09:02:40.677065+0000","last_fullsized":"2023-12-19T09:02:40.677065+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:11.665711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.683944+0000","last_change":"2023-12-19T09:02:41.683944+0000","last_active":"2023-12-19T09:02:41.683944+0000","last_peered":"2023-12-19T09:02:41.683944+0000","last_clean":"2023-12-19T09:02:41.683944+0000","last_became_active":"2023-12-19T09:02:41.683605+0000","last_became_peered":"2023-12-19T09:02:41.683605+0000","last_unstale":"2023-12-19T09:02:41.683944+0000","last_undegraded":"2023-12-19T09:02:41.683944+0000","last_fullsized":"2023-12-19T09:02:41.683944+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:58.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671545+0000","last_change":"2023-12-19T09:02:40.672906+0000","last_active":"2023-12-19T09:02:41.671545+0000","last_peered":"2023-12-19T09:02:41.671545+0000","last_clean":"2023-12-19T09:02:41.671545+0000","last_became_active":"2023-12-19T09:02:40.672617+0000","last_became_peered":"2023-12-19T09:02:40.672617+0000","last_unstale":"2023-12-19T09:02:41.671545+0000","last_undegraded":"2023-12-19T09:02:41.671545+0000","last_fullsized":"2023-12-19T09:02:41.671545+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:34.307881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690160+0000","last_change":"2023-12-19T09:02:41.690160+0000","last_active":"2023-12-19T09:02:41.690160+0000","last_peered":"2023-12-19T09:02:41.690160+0000","last_clean":"2023-12-19T09:02:41.690160+0000","last_became_active":"2023-12-19T09:02:41.689873+0000","last_became_peered":"2023-12-19T09:02:41.689873+0000","last_unstale":"2023-12-19T09:02:41.690160+0000","last_undegraded":"2023-12-19T09:02:41.690160+0000","last_fullsized":"2023-12-19T09:02:41.690160+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:39.353297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.665923+0000","last_change":"2023-12-19T09:02:40.678865+0000","last_active":"2023-12-19T09:02:41.665923+0000","last_peered":"2023-12-19T09:02:41.665923+0000","last_clean":"2023-12-19T09:02:41.665923+0000","last_became_active":"2023-12-19T09:02:40.678342+0000","last_became_peered":"2023-12-19T09:02:40.678342+0000","last_unstale":"2023-12-19T09:02:41.665923+0000","last_undegraded":"2023-12-19T09:02:41.665923+0000","last_fullsized":"2023-12-19T09:02:41.665923+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:04.638962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691360+0000","last_change":"2023-12-19T09:02:41.691360+0000","last_active":"2023-12-19T09:02:41.691360+0000","last_peered":"2023-12-19T09:02:41.691360+0000","last_clean":"2023-12-19T09:02:41.691360+0000","last_became_active":"2023-12-19T09:02:41.684652+0000","last_became_peered":"2023-12-19T09:02:41.684652+0000","last_unstale":"2023-12-19T09:02:41.691360+0000","last_undegraded":"2023-12-19T09:02:41.691360+0000","last_fullsized":"2023-12-19T09:02:41.691360+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.700194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.681605+0000","last_change":"2023-12-19T09:02:40.679845+0000","last_active":"2023-12-19T09:02:40.681605+0000","last_peered":"2023-12-19T09:02:40.681605+0000","last_clean":"2023-12-19T09:02:40.681605+0000","last_became_active":"2023-12-19T09:02:40.679189+0000","last_became_peered":"2023-12-19T09:02:40.679189+0000","last_unstale":"2023-12-19T09:02:40.681605+0000","last_undegraded":"2023-12-19T09:02:40.681605+0000","last_fullsized":"2023-12-19T09:02:40.681605+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:58:48.569570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697959+0000","last_change":"2023-12-19T09:02:41.691978+0000","last_active":"2023-12-19T09:02:41.697959+0000","last_peered":"2023-12-19T09:02:41.697959+0000","last_clean":"2023-12-19T09:02:41.697959+0000","last_became_active":"2023-12-19T09:02:41.691679+0000","last_became_peered":"2023-12-19T09:02:41.691679+0000","last_unstale":"2023-12-19T09:02:41.697959+0000","last_undegraded":"2023-12-19T09:02:41.697959+0000","last_fullsized":"2023-12-19T09:02:41.697959+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:01.696144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663614+0000","last_change":"2023-12-19T09:02:40.672742+0000","last_active":"2023-12-19T09:02:41.663614+0000","last_peered":"2023-12-19T09:02:41.663614+0000","last_clean":"2023-12-19T09:02:41.663614+0000","last_became_active":"2023-12-19T09:02:40.672417+0000","last_became_peered":"2023-12-19T09:02:40.672417+0000","last_unstale":"2023-12-19T09:02:41.663614+0000","last_undegraded":"2023-12-19T09:02:41.663614+0000","last_fullsized":"2023-12-19T09:02:41.663614+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:16:59.968094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688498+0000","last_change":"2023-12-19T09:02:41.688498+0000","last_active":"2023-12-19T09:02:41.688498+0000","last_peered":"2023-12-19T09:02:41.688498+0000","last_clean":"2023-12-19T09:02:41.688498+0000","last_became_active":"2023-12-19T09:02:41.685706+0000","last_became_peered":"2023-12-19T09:02:41.685706+0000","last_unstale":"2023-12-19T09:02:41.688498+0000","last_undegraded":"2023-12-19T09:02:41.688498+0000","last_fullsized":"2023-12-19T09:02:41.688498+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:50:43.700709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663237+0000","last_change":"2023-12-19T09:02:40.682057+0000","last_active":"2023-12-19T09:02:41.663237+0000","last_peered":"2023-12-19T09:02:41.663237+0000","last_clean":"2023-12-19T09:02:41.663237+0000","last_became_active":"2023-12-19T09:02:40.681531+0000","last_became_peered":"2023-12-19T09:02:40.681531+0000","last_unstale":"2023-12-19T09:02:41.663237+0000","last_undegraded":"2023-12-19T09:02:41.663237+0000","last_fullsized":"2023-12-19T09:02:41.663237+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:57.227017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'271","reported_seq":825,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.666129+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:02:41.666129+0000","last_peered":"2023-12-19T09:02:41.666129+0000","last_clean":"2023-12-19T09:02:41.666129+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:02:41.666129+0000","last_undegraded":"2023-12-19T09:02:41.666129+0000","last_fullsized":"2023-12-19T09:02:41.666129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575749,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299832,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072008,"statfs":{"total":96636764160,"available":96329736192,"internally_reserved":0,"allocated":2605056,"data_stored":1648658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3500000000000001,"15min":1.3069999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.54400000000000004},"max":{"1min":1.649,"5min":1.712,"15min":1.712},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.98099999999999998,"5min":0.74199999999999999,"15min":0.66800000000000004},"max":{"1min":1.4610000000000001,"5min":1.6559999999999999,"15min":1.788},"last":1.4319999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.878,"5min":0.69799999999999995,"15min":0.64000000000000001},"max":{"1min":1.6020000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1859999999999999,"15min":1.1899999999999999},"min":{"1min":0.874,"5min":0.73599999999999999,"15min":0.64500000000000002},"max":{"1min":1.4239999999999999,"5min":1.5880000000000001,"15min":1.784},"last":1.0429999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.177,"15min":1.1719999999999999},"min":{"1min":0.92900000000000005,"5min":0.70599999999999996,"15min":0.57199999999999995},"max":{"1min":1.504,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.454},{"interface":"front","average":{"1min":1.333,"5min":1.302,"15min":1.28},"min":{"1min":0.84699999999999998,"5min":0.79000000000000004,"15min":0.60899999999999999},"max":{"1min":1.5860000000000001,"5min":1.776,"15min":1.792},"last":1.2669999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.323},"min":{"1min":1.0780000000000001,"5min":1.034,"15min":0.625},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.778},"last":1.538},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3580000000000001,"15min":1.351},"min":{"1min":1.093,"5min":0.88900000000000001,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.659,"15min":1.774},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2430000000000001,"15min":1.218},"min":{"1min":0.84499999999999997,"5min":0.76400000000000001,"15min":0.56799999999999995},"max":{"1min":1.694,"5min":1.847,"15min":1.847},"last":1.173},{"interface":"front","average":{"1min":1.282,"5min":1.248,"15min":1.2210000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.42299999999999999},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.21}]},{"osd":5,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2589999999999999,"15min":1.2330000000000001},"min":{"1min":1.002,"5min":0.85899999999999999,"15min":0.35599999999999998},"max":{"1min":1.7350000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.097},{"interface":"front","average":{"1min":1.496,"5min":1.413,"15min":1.365},"min":{"1min":1.0840000000000001,"5min":0.65300000000000002,"15min":0.498},"max":{"1min":1.7789999999999999,"5min":1.9099999999999999,"15min":1.97},"last":1.623}]},{"osd":6,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4139999999999999,"15min":1.3819999999999999},"min":{"1min":1.032,"5min":0.80900000000000005,"15min":0.51600000000000001},"max":{"1min":1.841,"5min":1.841,"15min":1.8540000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3700000000000001,"15min":1.3660000000000001},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.47899999999999998},"max":{"1min":1.7569999999999999,"5min":1.804,"15min":1.859},"last":1.341}]}]},{"osd":6,"up_from":13,"seq":55834575748,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311532,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060308,"statfs":{"total":96636764160,"available":96317755392,"internally_reserved":0,"allocated":2002944,"data_stored":1057308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.145,"15min":1.1299999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.51500000000000001},"max":{"1min":1.5149999999999999,"5min":1.5609999999999999,"15min":1.6200000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.214,"5min":1.1930000000000001,"15min":1.1699999999999999},"min":{"1min":0.90800000000000003,"5min":0.81599999999999995,"15min":0.61399999999999999},"max":{"1min":1.5980000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.21}]},{"osd":1,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.238,"15min":1.196},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.591,"5min":1.696,"15min":1.696},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1719999999999999,"15min":1.181},"min":{"1min":0.83899999999999997,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.0209999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2609999999999999,"15min":1.244},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6140000000000001,"5min":1.806,"15min":1.806},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2490000000000001,"15min":1.2390000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.63400000000000001},"max":{"1min":1.5700000000000001,"5min":1.738,"15min":1.796},"last":0.96499999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3080000000000001,"15min":1.2709999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76300000000000001},"max":{"1min":1.5609999999999999,"5min":1.6879999999999999,"15min":1.7889999999999999},"last":1.111},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3140000000000001,"15min":1.327},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.7729999999999999},"last":1.359}]},{"osd":4,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1639999999999999,"15min":1.1870000000000001},"min":{"1min":0.73899999999999999,"5min":0.66800000000000004,"15min":0.63400000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4350000000000001,"15min":1.415},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.51100000000000001},"max":{"1min":1.669,"5min":1.8129999999999999,"15min":1.849},"last":1.327}]},{"osd":5,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2829999999999999,"15min":1.2869999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":1.784,"15min":1.994},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.27,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67700000000000005,"15min":0.56399999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.288}]},{"osd":7,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3620000000000001,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.56100000000000005},"max":{"1min":1.7050000000000001,"5min":1.7949999999999999,"15min":1.9299999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7649999999999999,"5min":2.165,"15min":2.165},"last":1.1679999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303348,"kb_used_data":1964,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068492,"statfs":{"total":96636764160,"available":96326135808,"internally_reserved":0,"allocated":2011136,"data_stored":1063842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2769999999999999,"15min":1.304},"min":{"1min":0.73499999999999999,"5min":0.626,"15min":0.57999999999999996},"max":{"1min":1.546,"5min":1.6479999999999999,"15min":1.7450000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3080000000000001,"15min":1.3160000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.448}]},{"osd":1,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.161,"15min":1.169},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.57199999999999995},"max":{"1min":1.6419999999999999,"5min":1.7230000000000001,"15min":1.748},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.82499999999999996,"5min":0.80400000000000005,"15min":0.72499999999999998},"max":{"1min":1.5860000000000001,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.319}]},{"osd":2,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2290000000000001,"15min":1.2150000000000001},"min":{"1min":0.89200000000000002,"5min":0.73999999999999999,"15min":0.55300000000000005},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.595},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.2450000000000001},"min":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.4530000000000001,"5min":1.726,"15min":1.726},"last":1.4990000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.3049999999999999,"15min":1.2909999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5569999999999999,"5min":1.7609999999999999,"15min":1.891},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.306},"min":{"1min":0.79300000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.6930000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.94799999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5980000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.321,"15min":1.3069999999999999},"min":{"1min":0.92900000000000005,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6220000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2350000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3340000000000001,"15min":1.2969999999999999},"min":{"1min":0.98399999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.7170000000000001,"5min":1.833,"15min":1.833},"last":1.778},{"interface":"front","average":{"1min":1.403,"5min":1.3819999999999999,"15min":1.3600000000000001},"min":{"1min":1.0349999999999999,"5min":0.79100000000000004,"15min":0.76800000000000002},"max":{"1min":1.7390000000000001,"5min":1.823,"15min":1.823},"last":1.5669999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.468,"15min":1.4350000000000001},"min":{"1min":0.98899999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.889,"5min":1.96,"15min":1.986},"last":1.409},{"interface":"front","average":{"1min":1.554,"5min":1.486,"15min":1.4650000000000001},"min":{"1min":0.97699999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.9530000000000001,"5min":1.99,"15min":1.99},"last":1.869}]}]},{"osd":4,"up_from":13,"seq":55834575748,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311524,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060316,"statfs":{"total":96636764160,"available":96317763584,"internally_reserved":0,"allocated":1994752,"data_stored":1056664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.622},"max":{"1min":1.5669999999999999,"5min":1.6659999999999999,"15min":1.7270000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.78300000000000003},"max":{"1min":1.6180000000000001,"5min":1.7270000000000001,"15min":1.768},"last":0.89000000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.21,"15min":1.202},"min":{"1min":0.85799999999999998,"5min":0.80700000000000005,"15min":0.54500000000000004},"max":{"1min":1.5389999999999999,"5min":1.7869999999999999,"15min":1.8959999999999999},"last":1.004},{"interface":"front","average":{"1min":1.159,"5min":1.171,"15min":1.149},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.59999999999999998},"max":{"1min":1.389,"5min":1.712,"15min":1.712},"last":1.163}]},{"osd":2,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1890000000000001,"15min":1.173},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.66100000000000003},"max":{"1min":1.5409999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6200000000000001,"5min":1.7290000000000001,"15min":1.8149999999999999},"last":1.0960000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.69199999999999995},"max":{"1min":1.575,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.282},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.377,"15min":1.3779999999999999},"min":{"1min":0.95399999999999996,"5min":0.90200000000000002,"15min":0.77800000000000002},"max":{"1min":1.673,"5min":1.7589999999999999,"15min":1.863},"last":1.359}]},{"osd":5,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.258,"15min":1.228},"min":{"1min":0.92000000000000004,"5min":0.875,"15min":0.623},"max":{"1min":1.5660000000000001,"5min":1.796,"15min":1.962},"last":1.198},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.82799999999999996,"5min":0.69899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6879999999999999,"5min":1.823,"15min":1.823},"last":1.0580000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.266,"15min":1.238},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.59399999999999997},"max":{"1min":1.899,"5min":1.909,"15min":1.909},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.419,"15min":1.405},"min":{"1min":1.0700000000000001,"5min":1.034,"15min":0.64200000000000002},"max":{"1min":1.677,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.296}]},{"osd":7,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.337,"15min":1.3340000000000001},"min":{"1min":0.95199999999999996,"5min":0.86499999999999999,"15min":0.71799999999999997},"max":{"1min":1.851,"5min":1.851,"15min":1.919},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3999999999999999,"15min":1.393},"min":{"1min":1.0860000000000001,"5min":0.84099999999999997,"15min":0.65100000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.141}]}]},{"osd":3,"up_from":13,"seq":55834575748,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307456,"kb_used_data":1976,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064384,"statfs":{"total":96636764160,"available":96321929216,"internally_reserved":0,"allocated":2023424,"data_stored":1065426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.85699999999999998,"15min":0.86399999999999999},"min":{"1min":0.66000000000000003,"5min":0.60999999999999999,"15min":0.60399999999999998},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.998,"15min":0.98999999999999999},"min":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.59299999999999997},"max":{"1min":1.353,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.85899999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94199999999999995,"15min":0.94499999999999995},"min":{"1min":0.72399999999999998,"5min":0.66100000000000003,"15min":0.63},"max":{"1min":1.0940000000000001,"5min":1.528,"15min":2.456},"last":0.752},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.79300000000000004,"5min":0.70899999999999996,"15min":0.69699999999999995},"max":{"1min":1.337,"5min":1.538,"15min":1.694},"last":1.0960000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.083,"15min":1.095},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.75900000000000001},"max":{"1min":1.4570000000000001,"5min":1.534,"15min":2.0150000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.84299999999999997,"15min":0.81699999999999995},"max":{"1min":1.4399999999999999,"5min":1.6539999999999999,"15min":1.681},"last":1.331}]},{"osd":4,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2809999999999999,"15min":1.2829999999999999},"min":{"1min":1.0469999999999999,"5min":1.0229999999999999,"15min":0.94099999999999995},"max":{"1min":1.5409999999999999,"5min":1.6120000000000001,"15min":1.6679999999999999},"last":1.272},{"interface":"front","average":{"1min":1.294,"5min":1.264,"15min":1.254},"min":{"1min":1.0900000000000001,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.5580000000000001,"5min":1.742,"15min":2.6520000000000001},"last":1.143}]},{"osd":5,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.3169999999999999,"15min":1.296},"min":{"1min":1.071,"5min":0.97099999999999997,"15min":0.85399999999999998},"max":{"1min":1.4850000000000001,"5min":1.7589999999999999,"15min":2.7519999999999998},"last":1.405},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.395,"15min":1.393},"min":{"1min":1.101,"5min":1.0660000000000001,"15min":1.0129999999999999},"max":{"1min":1.8140000000000001,"5min":1.8759999999999999,"15min":2.5710000000000002},"last":1.448}]},{"osd":6,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.431,"15min":1.423},"min":{"1min":1.216,"5min":1.113,"15min":1.0389999999999999},"max":{"1min":1.837,"5min":1.9370000000000001,"15min":1.9610000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.427,"15min":1.4039999999999999},"min":{"1min":1.2110000000000001,"5min":1.0169999999999999,"15min":0.88800000000000001},"max":{"1min":1.6699999999999999,"5min":1.925,"15min":2.2330000000000001},"last":1.4990000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":1.163,"5min":1.163,"15min":1.1160000000000001},"max":{"1min":1.7769999999999999,"5min":2.0379999999999998,"15min":2.202},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.139,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":1.7090000000000001,"5min":1.8400000000000001,"15min":2.8620000000000001},"last":1.53}]}]},{"osd":2,"up_from":13,"seq":55834575748,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307436,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064404,"statfs":{"total":96636764160,"available":96321949696,"internally_reserved":0,"allocated":2002944,"data_stored":1057252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.94199999999999995,"15min":0.93300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.085,"15min":1.0760000000000001},"min":{"1min":0.81799999999999995,"5min":0.81100000000000005,"15min":0.60799999999999998},"max":{"1min":1.367,"5min":1.4650000000000001,"15min":1.744},"last":0.997}]},{"osd":1,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70599999999999996,"5min":0.66800000000000004,"15min":0.56399999999999995},"max":{"1min":1.3939999999999999,"5min":1.4119999999999999,"15min":1.948},"last":1.093},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.42699999999999999},"max":{"1min":1.538,"5min":1.6200000000000001,"15min":1.6499999999999999},"last":0.85399999999999998}]},{"osd":3,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.628},"max":{"1min":1.1579999999999999,"5min":1.6140000000000001,"15min":1.8009999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.1639999999999999,"15min":1.1539999999999999},"min":{"1min":0.71499999999999997,"5min":0.63600000000000001,"15min":0.57899999999999996},"max":{"1min":1.4199999999999999,"5min":1.49,"15min":2.1760000000000002},"last":1.3009999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.327,"15min":1.3320000000000001},"min":{"1min":1.012,"5min":1.0109999999999999,"15min":0.84899999999999998},"max":{"1min":1.6040000000000001,"5min":1.6719999999999999,"15min":2.0590000000000002},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.325,"15min":1.323},"min":{"1min":0.93200000000000005,"5min":0.84099999999999997,"15min":0.82899999999999996},"max":{"1min":1.7010000000000001,"5min":1.714,"15min":3.1030000000000002},"last":1.1419999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3720000000000001,"15min":1.363},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.571,"5min":1.7270000000000001,"15min":1.9099999999999999},"last":1.262},{"interface":"front","average":{"1min":1.22,"5min":1.286,"15min":1.2949999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.548,"5min":2.0830000000000002,"15min":2.2959999999999998},"last":1.4039999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3440000000000001,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.72599999999999998},"max":{"1min":1.524,"5min":1.9199999999999999,"15min":2.3940000000000001},"last":1.522},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.415,"15min":1.397},"min":{"1min":1.016,"5min":1.016,"15min":0.89900000000000002},"max":{"1min":1.6930000000000001,"5min":1.8520000000000001,"15min":1.869},"last":1.4650000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.0900000000000001,"5min":1.03,"15min":0.60199999999999998},"max":{"1min":1.6839999999999999,"5min":1.8340000000000001,"15min":2.1680000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4259999999999999,"15min":1.4059999999999999},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.82199999999999995},"max":{"1min":1.722,"5min":2.0859999999999999,"15min":2.7109999999999999},"last":1.5900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1056734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86099999999999999,"15min":0.85399999999999998},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.34799999999999998},"max":{"1min":1.2,"5min":1.54,"15min":1.54},"last":1.107},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.72299999999999998,"5min":0.63700000000000001,"15min":0.42099999999999999},"max":{"1min":1.2849999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.97699999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93400000000000005,"15min":0.92900000000000005},"min":{"1min":0.78900000000000003,"5min":0.66600000000000004,"15min":0.36499999999999999},"max":{"1min":1.2470000000000001,"5min":1.4359999999999999,"15min":1.7430000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0960000000000001,"15min":1.0800000000000001},"min":{"1min":0.82499999999999996,"5min":0.64800000000000002,"15min":0.503},"max":{"1min":1.454,"5min":1.6240000000000001,"15min":1.6259999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.1870000000000001,"15min":1.1819999999999999},"min":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.55600000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.097,"15min":1.0880000000000001},"min":{"1min":0.90900000000000003,"5min":0.60499999999999998,"15min":0.45100000000000001},"max":{"1min":1.4770000000000001,"5min":1.706,"15min":1.706},"last":1.0760000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.242,"15min":1.2250000000000001},"min":{"1min":1.0009999999999999,"5min":0.82799999999999996,"15min":0.60699999999999998},"max":{"1min":1.444,"5min":1.544,"15min":1.5900000000000001},"last":1.478},{"interface":"front","average":{"1min":1.355,"5min":1.345,"15min":1.331},"min":{"1min":1.0429999999999999,"5min":0.875,"15min":0.83499999999999996},"max":{"1min":1.5960000000000001,"5min":1.958,"15min":2.024},"last":1.181}]},{"osd":5,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3140000000000001,"15min":1.3200000000000001},"min":{"1min":1.0920000000000001,"5min":0.97899999999999998,"15min":0.78400000000000003},"max":{"1min":1.554,"5min":2.0569999999999999,"15min":2.0699999999999998},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":1.0740000000000001,"5min":0.90200000000000002,"15min":0.72199999999999998},"max":{"1min":1.5529999999999999,"5min":1.96,"15min":2.052},"last":1.534}]},{"osd":6,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":1.1100000000000001,"5min":0.93799999999999994,"15min":0.92400000000000004},"max":{"1min":1.9750000000000001,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.385},"min":{"1min":1.113,"5min":0.85599999999999998,"15min":0.81499999999999995},"max":{"1min":1.77,"5min":1.881,"15min":2.0800000000000001},"last":1.8779999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.425,"15min":1.427},"min":{"1min":1.091,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.861,"5min":2.117,"15min":2.117},"last":1.9750000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.401,"15min":1.401},"min":{"1min":1.141,"5min":0.95899999999999996,"15min":0.78800000000000003},"max":{"1min":1.643,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.577}]}]},{"osd":0,"up_from":13,"seq":55834575748,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312108,"kb_used_data":2532,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059732,"statfs":{"total":96636764160,"available":96317165568,"internally_reserved":0,"allocated":2592768,"data_stored":1646077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85099999999999998,"15min":0.84799999999999998},"min":{"1min":0.66100000000000003,"5min":0.58299999999999996,"15min":0.41299999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.512},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.97199999999999998,"15min":0.97299999999999998},"min":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.48599999999999999},"max":{"1min":1.242,"5min":1.464,"15min":1.488},"last":0.98599999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0149999999999999,"15min":0.995},"min":{"1min":0.82799999999999996,"5min":0.66200000000000003,"15min":0.51900000000000002},"max":{"1min":1.3360000000000001,"5min":1.4850000000000001,"15min":1.625},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.079,"15min":1.0660000000000001},"min":{"1min":0.97499999999999998,"5min":0.79600000000000004,"15min":0.498},"max":{"1min":1.286,"5min":1.4299999999999999,"15min":1.6499999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.05,"15min":1.024},"min":{"1min":0.79300000000000004,"5min":0.56000000000000005,"15min":0.47399999999999998},"max":{"1min":1.444,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.224,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.96199999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.4970000000000001,"5min":1.5649999999999999,"15min":1.698},"last":0.75700000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3839999999999999,"15min":1.383},"min":{"1min":1.2130000000000001,"5min":1.0089999999999999,"15min":0.747},"max":{"1min":1.5549999999999999,"5min":1.752,"15min":1.752},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.371,"15min":1.363},"min":{"1min":1.264,"5min":0.97399999999999998,"15min":0.93799999999999994},"max":{"1min":1.599,"5min":1.792,"15min":1.792},"last":0.93999999999999995}]},{"osd":5,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.411,"15min":1.387},"min":{"1min":1.2050000000000001,"5min":0.93500000000000005,"15min":0.85099999999999998},"max":{"1min":1.694,"5min":1.863,"15min":1.863},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.446,"15min":1.431},"min":{"1min":1.173,"5min":1.137,"15min":0.88200000000000001},"max":{"1min":1.589,"5min":1.7470000000000001,"15min":1.7589999999999999},"last":1.165}]},{"osd":6,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3220000000000001,"15min":1.296},"min":{"1min":1.0569999999999999,"5min":0.96799999999999997,"15min":0.89800000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.79},"last":1.028},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4039999999999999,"15min":1.3799999999999999},"min":{"1min":1.0920000000000001,"5min":1.091,"15min":0.99299999999999999},"max":{"1min":1.659,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.0900000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.395,"15min":1.3700000000000001},"min":{"1min":1.1279999999999999,"5min":1.032,"15min":0.97699999999999998},"max":{"1min":1.7529999999999999,"5min":1.865,"15min":1.9590000000000001},"last":1.302},{"interface":"front","average":{"1min":1.532,"5min":1.4410000000000001,"15min":1.409},"min":{"1min":1.2010000000000001,"5min":1.0620000000000001,"15min":1.0229999999999999},"max":{"1min":1.931,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.1180000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:02:45.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:02:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 Processor -- start 2023-12-19T09:02:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 -- start start 2023-12-19T09:02:46.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5118158e70 0x7f51180afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f511806a540 0x7f511806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 0x7f51180cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51180ca5a0 con 0x7f511806a540 2023-12-19T09:02:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511806dac0 con 0x7f5118158e70 2023-12-19T09:02:46.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511de0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180b0260 con 0x7f51180cc470 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f5117fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f511806a540 0x7f511806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f5117fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f511806a540 0x7f511806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52508/0 (socket says 172.21.15.111:52508) 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f5117fff700 1 -- 172.21.15.111:0/2070075115 learned_addr learned my addr 172.21.15.111:0/2070075115 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511d60b700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 0x7f51180cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.124+0000 7f511ce0a700 1 --2- 172.21.15.111:0/2070075115 >> v2:172.21.15.143:3300/0 conn(0x7f5118158e70 0x7f51180afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 -- 172.21.15.111:0/2070075115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 msgr2=0x7f51180cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 0x7f51180cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 -- 172.21.15.111:0/2070075115 >> v2:172.21.15.143:3300/0 conn(0x7f5118158e70 msgr2=0x7f51180afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 --2- 172.21.15.111:0/2070075115 >> v2:172.21.15.143:3300/0 conn(0x7f5118158e70 0x7f51180afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 -- 172.21.15.111:0/2070075115 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511806d220 con 0x7f511806a540 2023-12-19T09:02:46.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511d60b700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 0x7f51180cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:46.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f5117fff700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f511806a540 0x7f511806c9d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f511806a210 tx=0x7f5104009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161d2ea0534bf43b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f51177fe700 1 -- 172.21.15.111:0/2070075115 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400b860 con 0x7f511806a540 2023-12-19T09:02:46.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f51177fe700 1 -- 172.21.15.111:0/2070075115 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510400b9c0 con 0x7f511806a540 2023-12-19T09:02:46.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f51177fe700 1 -- 172.21.15.111:0/2070075115 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400bd80 con 0x7f511806a540 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 -- 172.21.15.111:0/2070075115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f511806a540 msgr2=0x7f511806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f511806a540 0x7f511806c9d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f511806a210 tx=0x7f5104009250 comp rx=0 tx=0).stop 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 -- 172.21.15.111:0/2070075115 shutdown_connections 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51180cc470 0x7f51180cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 --2- 172.21.15.111:0/2070075115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f511806a540 0x7f511806c9d0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 --2- 172.21.15.111:0/2070075115 >> v2:172.21.15.143:3300/0 conn(0x7f5118158e70 0x7f51180afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 -- 172.21.15.111:0/2070075115 >> 172.21.15.111:0/2070075115 conn(0x7f51180670e0 msgr2=0x7f51180674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:46.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 -- 172.21.15.111:0/2070075115 shutdown_connections 2023-12-19T09:02:46.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.128+0000 7f511de0c700 1 -- 172.21.15.111:0/2070075115 wait complete. 2023-12-19T09:02:46.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 Processor -- start 2023-12-19T09:02:46.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 -- start start 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f511806a540 0x7f51180c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 0x7f511815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180c6e70 con 0x7f51180cc470 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180c70c0 con 0x7f511806a540 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.132+0000 7f511de0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180c73a0 con 0x7f5118158e70 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52516/0 (socket says 172.21.15.111:52516) 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 -- 172.21.15.111:0/564186779 learned_addr learned my addr 172.21.15.111:0/564186779 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:46.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f511d60b700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 0x7f511815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f511ce0a700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f511806a540 0x7f51180c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 -- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 msgr2=0x7f511815a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 0x7f511815a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 -- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f511806a540 msgr2=0x7f51180c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f511806a540 0x7f51180c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 -- 172.21.15.111:0/564186779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51040094f0 con 0x7f51180cc470 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f511d60b700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 0x7f511815a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:46.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.136+0000 7f5117fff700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f51180cfb30 tx=0x7f5104017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104011e20 con 0x7f51180cc470 2023-12-19T09:02:46.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104026450 con 0x7f51180cc470 2023-12-19T09:02:46.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51180c7680 con 0x7f51180cc470 2023-12-19T09:02:46.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104026810 con 0x7f51180cc470 2023-12-19T09:02:46.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51180c7c40 con 0x7f51180cc470 2023-12-19T09:02:46.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.140+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51180c7e60 con 0x7f51180cc470 2023-12-19T09:02:46.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.144+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5104026bc0 con 0x7f51180cc470 2023-12-19T09:02:46.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.144+0000 7f5114ff9700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 0x7f50fc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.144+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f510407e750 con 0x7f51180cc470 2023-12-19T09:02:46.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.144+0000 7f511ce0a700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 0x7f50fc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.144+0000 7f511ce0a700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 0x7f50fc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f510c006fd0 tx=0x7f510c008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.148+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f510404e950 con 0x7f51180cc470 2023-12-19T09:02:46.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.316+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f511804be40 con 0x7f50fc060670 2023-12-19T09:02:46.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.336+0000 7f5114ff9700 1 -- 172.21.15.111:0/564186779 <== mgr.7493 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286590 (secure 0 0 0) 0x7f511804be40 con 0x7f50fc060670 2023-12-19T09:02:46.347 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 msgr2=0x7f50fc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 0x7f50fc062b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f510c006fd0 tx=0x7f510c008040 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 msgr2=0x7f5118159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f51180cfb30 tx=0x7f5104017000 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 shutdown_connections 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f50fc060670 0x7f50fc062b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5118158e70 0x7f511815a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51180cc470 0x7f5118159c70 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 --2- 172.21.15.111:0/564186779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f511806a540 0x7f51180c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 >> 172.21.15.111:0/564186779 conn(0x7f51180670e0 msgr2=0x7f51180674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 shutdown_connections 2023-12-19T09:02:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.344+0000 7f511de0c700 1 -- 172.21.15.111:0/564186779 wait complete. 2023-12-19T09:02:46.353 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:02:46.372 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T09:02:45.179423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2460668,"kb_used_data":16828,"kb_used_omap":0,"kb_used_meta":2443776,"kb_avail":752514052,"statfs":{"total":773094113280,"available":770574389248,"internally_reserved":0,"allocated":17231872,"data_stored":9651961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2502426624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.536804"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695726+0000","last_change":"2023-12-19T09:02:41.688015+0000","last_active":"2023-12-19T09:02:41.695726+0000","last_peered":"2023-12-19T09:02:41.695726+0000","last_clean":"2023-12-19T09:02:41.695726+0000","last_became_active":"2023-12-19T09:02:41.687767+0000","last_became_peered":"2023-12-19T09:02:41.687767+0000","last_unstale":"2023-12-19T09:02:41.695726+0000","last_undegraded":"2023-12-19T09:02:41.695726+0000","last_fullsized":"2023-12-19T09:02:41.695726+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:27:54.900814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686131+0000","last_change":"2023-12-19T09:02:41.681865+0000","last_active":"2023-12-19T09:02:41.686131+0000","last_peered":"2023-12-19T09:02:41.686131+0000","last_clean":"2023-12-19T09:02:41.686131+0000","last_became_active":"2023-12-19T09:02:41.681354+0000","last_became_peered":"2023-12-19T09:02:41.681354+0000","last_unstale":"2023-12-19T09:02:41.686131+0000","last_undegraded":"2023-12-19T09:02:41.686131+0000","last_fullsized":"2023-12-19T09:02:41.686131+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:39.353297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695137+0000","last_change":"2023-12-19T09:02:41.695137+0000","last_active":"2023-12-19T09:02:41.695137+0000","last_peered":"2023-12-19T09:02:41.695137+0000","last_clean":"2023-12-19T09:02:41.695137+0000","last_became_active":"2023-12-19T09:02:41.694795+0000","last_became_peered":"2023-12-19T09:02:41.694795+0000","last_unstale":"2023-12-19T09:02:41.695137+0000","last_undegraded":"2023-12-19T09:02:41.695137+0000","last_fullsized":"2023-12-19T09:02:41.695137+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:29:39.732982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695957+0000","last_change":"2023-12-19T09:02:41.693959+0000","last_active":"2023-12-19T09:02:41.695957+0000","last_peered":"2023-12-19T09:02:41.695957+0000","last_clean":"2023-12-19T09:02:41.695957+0000","last_became_active":"2023-12-19T09:02:41.693672+0000","last_became_peered":"2023-12-19T09:02:41.693672+0000","last_unstale":"2023-12-19T09:02:41.695957+0000","last_undegraded":"2023-12-19T09:02:41.695957+0000","last_fullsized":"2023-12-19T09:02:41.695957+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:05.643249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691606+0000","last_change":"2023-12-19T09:02:41.683890+0000","last_active":"2023-12-19T09:02:41.691606+0000","last_peered":"2023-12-19T09:02:41.691606+0000","last_clean":"2023-12-19T09:02:41.691606+0000","last_became_active":"2023-12-19T09:02:41.683573+0000","last_became_peered":"2023-12-19T09:02:41.683573+0000","last_unstale":"2023-12-19T09:02:41.691606+0000","last_undegraded":"2023-12-19T09:02:41.691606+0000","last_fullsized":"2023-12-19T09:02:41.691606+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:56.665507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.694176+0000","last_change":"2023-12-19T09:02:41.694176+0000","last_active":"2023-12-19T09:02:41.694176+0000","last_peered":"2023-12-19T09:02:41.694176+0000","last_clean":"2023-12-19T09:02:41.694176+0000","last_became_active":"2023-12-19T09:02:41.693890+0000","last_became_peered":"2023-12-19T09:02:41.693890+0000","last_unstale":"2023-12-19T09:02:41.694176+0000","last_undegraded":"2023-12-19T09:02:41.694176+0000","last_fullsized":"2023-12-19T09:02:41.694176+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:54.108905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697884+0000","last_change":"2023-12-19T09:02:41.689556+0000","last_active":"2023-12-19T09:02:41.697884+0000","last_peered":"2023-12-19T09:02:41.697884+0000","last_clean":"2023-12-19T09:02:41.697884+0000","last_became_active":"2023-12-19T09:02:41.689214+0000","last_became_peered":"2023-12-19T09:02:41.689214+0000","last_unstale":"2023-12-19T09:02:41.697884+0000","last_undegraded":"2023-12-19T09:02:41.697884+0000","last_fullsized":"2023-12-19T09:02:41.697884+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:46:06.299337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678924+0000","last_change":"2023-12-19T09:02:41.678924+0000","last_active":"2023-12-19T09:02:41.678924+0000","last_peered":"2023-12-19T09:02:41.678924+0000","last_clean":"2023-12-19T09:02:41.678924+0000","last_became_active":"2023-12-19T09:02:41.678628+0000","last_became_peered":"2023-12-19T09:02:41.678628+0000","last_unstale":"2023-12-19T09:02:41.678924+0000","last_undegraded":"2023-12-19T09:02:41.678924+0000","last_fullsized":"2023-12-19T09:02:41.678924+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:21.404408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687870+0000","last_change":"2023-12-19T09:02:41.687870+0000","last_active":"2023-12-19T09:02:41.687870+0000","last_peered":"2023-12-19T09:02:41.687870+0000","last_clean":"2023-12-19T09:02:41.687870+0000","last_became_active":"2023-12-19T09:02:41.687532+0000","last_became_peered":"2023-12-19T09:02:41.687532+0000","last_unstale":"2023-12-19T09:02:41.687870+0000","last_undegraded":"2023-12-19T09:02:41.687870+0000","last_fullsized":"2023-12-19T09:02:41.687870+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:07:44.794051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.668389+0000","last_change":"2023-12-19T09:02:41.668389+0000","last_active":"2023-12-19T09:02:41.668389+0000","last_peered":"2023-12-19T09:02:41.668389+0000","last_clean":"2023-12-19T09:02:41.668389+0000","last_became_active":"2023-12-19T09:02:41.668075+0000","last_became_peered":"2023-12-19T09:02:41.668075+0000","last_unstale":"2023-12-19T09:02:41.668389+0000","last_undegraded":"2023-12-19T09:02:41.668389+0000","last_fullsized":"2023-12-19T09:02:41.668389+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:01.696144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688648+0000","last_change":"2023-12-19T09:02:41.687249+0000","last_active":"2023-12-19T09:02:41.688648+0000","last_peered":"2023-12-19T09:02:41.688648+0000","last_clean":"2023-12-19T09:02:41.688648+0000","last_became_active":"2023-12-19T09:02:41.686971+0000","last_became_peered":"2023-12-19T09:02:41.686971+0000","last_unstale":"2023-12-19T09:02:41.688648+0000","last_undegraded":"2023-12-19T09:02:41.688648+0000","last_fullsized":"2023-12-19T09:02:41.688648+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:42.638629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690440+0000","last_change":"2023-12-19T09:02:41.684465+0000","last_active":"2023-12-19T09:02:41.690440+0000","last_peered":"2023-12-19T09:02:41.690440+0000","last_clean":"2023-12-19T09:02:41.690440+0000","last_became_active":"2023-12-19T09:02:41.684114+0000","last_became_peered":"2023-12-19T09:02:41.684114+0000","last_unstale":"2023-12-19T09:02:41.690440+0000","last_undegraded":"2023-12-19T09:02:41.690440+0000","last_fullsized":"2023-12-19T09:02:41.690440+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:55:38.150197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686942+0000","last_change":"2023-12-19T09:02:41.686942+0000","last_active":"2023-12-19T09:02:41.686942+0000","last_peered":"2023-12-19T09:02:41.686942+0000","last_clean":"2023-12-19T09:02:41.686942+0000","last_became_active":"2023-12-19T09:02:41.684704+0000","last_became_peered":"2023-12-19T09:02:41.684704+0000","last_unstale":"2023-12-19T09:02:41.686942+0000","last_undegraded":"2023-12-19T09:02:41.686942+0000","last_fullsized":"2023-12-19T09:02:41.686942+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:36.506687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691284+0000","last_change":"2023-12-19T09:02:41.681925+0000","last_active":"2023-12-19T09:02:41.691284+0000","last_peered":"2023-12-19T09:02:41.691284+0000","last_clean":"2023-12-19T09:02:41.691284+0000","last_became_active":"2023-12-19T09:02:41.681260+0000","last_became_peered":"2023-12-19T09:02:41.681260+0000","last_unstale":"2023-12-19T09:02:41.691284+0000","last_undegraded":"2023-12-19T09:02:41.691284+0000","last_fullsized":"2023-12-19T09:02:41.691284+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:15.669943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690352+0000","last_change":"2023-12-19T09:02:41.681962+0000","last_active":"2023-12-19T09:02:41.690352+0000","last_peered":"2023-12-19T09:02:41.690352+0000","last_clean":"2023-12-19T09:02:41.690352+0000","last_became_active":"2023-12-19T09:02:41.681674+0000","last_became_peered":"2023-12-19T09:02:41.681674+0000","last_unstale":"2023-12-19T09:02:41.690352+0000","last_undegraded":"2023-12-19T09:02:41.690352+0000","last_fullsized":"2023-12-19T09:02:41.690352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:03.605824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.689728+0000","last_change":"2023-12-19T09:02:41.689728+0000","last_active":"2023-12-19T09:02:41.689728+0000","last_peered":"2023-12-19T09:02:41.689728+0000","last_clean":"2023-12-19T09:02:41.689728+0000","last_became_active":"2023-12-19T09:02:41.685378+0000","last_became_peered":"2023-12-19T09:02:41.685378+0000","last_unstale":"2023-12-19T09:02:41.689728+0000","last_undegraded":"2023-12-19T09:02:41.689728+0000","last_fullsized":"2023-12-19T09:02:41.689728+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:36:24.790523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697105+0000","last_change":"2023-12-19T09:02:41.697105+0000","last_active":"2023-12-19T09:02:41.697105+0000","last_peered":"2023-12-19T09:02:41.697105+0000","last_clean":"2023-12-19T09:02:41.697105+0000","last_became_active":"2023-12-19T09:02:41.695330+0000","last_became_peered":"2023-12-19T09:02:41.695330+0000","last_unstale":"2023-12-19T09:02:41.697105+0000","last_undegraded":"2023-12-19T09:02:41.697105+0000","last_fullsized":"2023-12-19T09:02:41.697105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:37.885338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691850+0000","last_change":"2023-12-19T09:02:41.681671+0000","last_active":"2023-12-19T09:02:41.691850+0000","last_peered":"2023-12-19T09:02:41.691850+0000","last_clean":"2023-12-19T09:02:41.691850+0000","last_became_active":"2023-12-19T09:02:41.681368+0000","last_became_peered":"2023-12-19T09:02:41.681368+0000","last_unstale":"2023-12-19T09:02:41.691850+0000","last_undegraded":"2023-12-19T09:02:41.691850+0000","last_fullsized":"2023-12-19T09:02:41.691850+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:21:55.379680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678036+0000","last_change":"2023-12-19T09:02:41.673557+0000","last_active":"2023-12-19T09:02:41.678036+0000","last_peered":"2023-12-19T09:02:41.678036+0000","last_clean":"2023-12-19T09:02:41.678036+0000","last_became_active":"2023-12-19T09:02:41.673225+0000","last_became_peered":"2023-12-19T09:02:41.673225+0000","last_unstale":"2023-12-19T09:02:41.678036+0000","last_undegraded":"2023-12-19T09:02:41.678036+0000","last_fullsized":"2023-12-19T09:02:41.678036+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:39.987960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690829+0000","last_change":"2023-12-19T09:02:41.681001+0000","last_active":"2023-12-19T09:02:41.690829+0000","last_peered":"2023-12-19T09:02:41.690829+0000","last_clean":"2023-12-19T09:02:41.690829+0000","last_became_active":"2023-12-19T09:02:41.680693+0000","last_became_peered":"2023-12-19T09:02:41.680693+0000","last_unstale":"2023-12-19T09:02:41.690829+0000","last_undegraded":"2023-12-19T09:02:41.690829+0000","last_fullsized":"2023-12-19T09:02:41.690829+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:11:57.379165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.692064+0000","last_change":"2023-12-19T09:02:41.691014+0000","last_active":"2023-12-19T09:02:41.692064+0000","last_peered":"2023-12-19T09:02:41.692064+0000","last_clean":"2023-12-19T09:02:41.692064+0000","last_became_active":"2023-12-19T09:02:41.690707+0000","last_became_peered":"2023-12-19T09:02:41.690707+0000","last_unstale":"2023-12-19T09:02:41.692064+0000","last_undegraded":"2023-12-19T09:02:41.692064+0000","last_fullsized":"2023-12-19T09:02:41.692064+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:48.584202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.683278+0000","last_change":"2023-12-19T09:02:41.675799+0000","last_active":"2023-12-19T09:02:41.683278+0000","last_peered":"2023-12-19T09:02:41.683278+0000","last_clean":"2023-12-19T09:02:41.683278+0000","last_became_active":"2023-12-19T09:02:41.675477+0000","last_became_peered":"2023-12-19T09:02:41.675477+0000","last_unstale":"2023-12-19T09:02:41.683278+0000","last_undegraded":"2023-12-19T09:02:41.683278+0000","last_fullsized":"2023-12-19T09:02:41.683278+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:21:06.946197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.685157+0000","last_change":"2023-12-19T09:02:41.681347+0000","last_active":"2023-12-19T09:02:41.685157+0000","last_peered":"2023-12-19T09:02:41.685157+0000","last_clean":"2023-12-19T09:02:41.685157+0000","last_became_active":"2023-12-19T09:02:41.681053+0000","last_became_peered":"2023-12-19T09:02:41.681053+0000","last_unstale":"2023-12-19T09:02:41.685157+0000","last_undegraded":"2023-12-19T09:02:41.685157+0000","last_fullsized":"2023-12-19T09:02:41.685157+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:27.797838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690587+0000","last_change":"2023-12-19T09:02:41.690587+0000","last_active":"2023-12-19T09:02:41.690587+0000","last_peered":"2023-12-19T09:02:41.690587+0000","last_clean":"2023-12-19T09:02:41.690587+0000","last_became_active":"2023-12-19T09:02:41.685638+0000","last_became_peered":"2023-12-19T09:02:41.685638+0000","last_unstale":"2023-12-19T09:02:41.690587+0000","last_undegraded":"2023-12-19T09:02:41.690587+0000","last_fullsized":"2023-12-19T09:02:41.690587+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:52:42.638629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.699620+0000","last_change":"2023-12-19T09:02:41.699620+0000","last_active":"2023-12-19T09:02:41.699620+0000","last_peered":"2023-12-19T09:02:41.699620+0000","last_clean":"2023-12-19T09:02:41.699620+0000","last_became_active":"2023-12-19T09:02:41.699272+0000","last_became_peered":"2023-12-19T09:02:41.699272+0000","last_unstale":"2023-12-19T09:02:41.699620+0000","last_undegraded":"2023-12-19T09:02:41.699620+0000","last_fullsized":"2023-12-19T09:02:41.699620+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:43.924437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686325+0000","last_change":"2023-12-19T09:02:41.686325+0000","last_active":"2023-12-19T09:02:41.686325+0000","last_peered":"2023-12-19T09:02:41.686325+0000","last_clean":"2023-12-19T09:02:41.686325+0000","last_became_active":"2023-12-19T09:02:41.686014+0000","last_became_peered":"2023-12-19T09:02:41.686014+0000","last_unstale":"2023-12-19T09:02:41.686325+0000","last_undegraded":"2023-12-19T09:02:41.686325+0000","last_fullsized":"2023-12-19T09:02:41.686325+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:10:52.355309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700220+0000","last_change":"2023-12-19T09:02:41.698088+0000","last_active":"2023-12-19T09:02:41.700220+0000","last_peered":"2023-12-19T09:02:41.700220+0000","last_clean":"2023-12-19T09:02:41.700220+0000","last_became_active":"2023-12-19T09:02:41.697754+0000","last_became_peered":"2023-12-19T09:02:41.697754+0000","last_unstale":"2023-12-19T09:02:41.700220+0000","last_undegraded":"2023-12-19T09:02:41.700220+0000","last_fullsized":"2023-12-19T09:02:41.700220+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:12.018813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671920+0000","last_change":"2023-12-19T09:02:41.671920+0000","last_active":"2023-12-19T09:02:41.671920+0000","last_peered":"2023-12-19T09:02:41.671920+0000","last_clean":"2023-12-19T09:02:41.671920+0000","last_became_active":"2023-12-19T09:02:41.671431+0000","last_became_peered":"2023-12-19T09:02:41.671431+0000","last_unstale":"2023-12-19T09:02:41.671920+0000","last_undegraded":"2023-12-19T09:02:41.671920+0000","last_fullsized":"2023-12-19T09:02:41.671920+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:46:06.299337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686572+0000","last_change":"2023-12-19T09:02:41.686572+0000","last_active":"2023-12-19T09:02:41.686572+0000","last_peered":"2023-12-19T09:02:41.686572+0000","last_clean":"2023-12-19T09:02:41.686572+0000","last_became_active":"2023-12-19T09:02:41.685504+0000","last_became_peered":"2023-12-19T09:02:41.685504+0000","last_unstale":"2023-12-19T09:02:41.686572+0000","last_undegraded":"2023-12-19T09:02:41.686572+0000","last_fullsized":"2023-12-19T09:02:41.686572+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:20.293179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686568+0000","last_change":"2023-12-19T09:02:41.686568+0000","last_active":"2023-12-19T09:02:41.686568+0000","last_peered":"2023-12-19T09:02:41.686568+0000","last_clean":"2023-12-19T09:02:41.686568+0000","last_became_active":"2023-12-19T09:02:41.684935+0000","last_became_peered":"2023-12-19T09:02:41.684935+0000","last_unstale":"2023-12-19T09:02:41.686568+0000","last_undegraded":"2023-12-19T09:02:41.686568+0000","last_fullsized":"2023-12-19T09:02:41.686568+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:31:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697714+0000","last_change":"2023-12-19T09:02:41.697714+0000","last_active":"2023-12-19T09:02:41.697714+0000","last_peered":"2023-12-19T09:02:41.697714+0000","last_clean":"2023-12-19T09:02:41.697714+0000","last_became_active":"2023-12-19T09:02:41.697352+0000","last_became_peered":"2023-12-19T09:02:41.697352+0000","last_unstale":"2023-12-19T09:02:41.697714+0000","last_undegraded":"2023-12-19T09:02:41.697714+0000","last_fullsized":"2023-12-19T09:02:41.697714+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:34.267613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691926+0000","last_change":"2023-12-19T09:02:41.691926+0000","last_active":"2023-12-19T09:02:41.691926+0000","last_peered":"2023-12-19T09:02:41.691926+0000","last_clean":"2023-12-19T09:02:41.691926+0000","last_became_active":"2023-12-19T09:02:41.691568+0000","last_became_peered":"2023-12-19T09:02:41.691568+0000","last_unstale":"2023-12-19T09:02:41.691926+0000","last_undegraded":"2023-12-19T09:02:41.691926+0000","last_fullsized":"2023-12-19T09:02:41.691926+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:39.987960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.678358+0000","last_change":"2023-12-19T09:02:41.678358+0000","last_active":"2023-12-19T09:02:41.678358+0000","last_peered":"2023-12-19T09:02:41.678358+0000","last_clean":"2023-12-19T09:02:41.678358+0000","last_became_active":"2023-12-19T09:02:41.678069+0000","last_became_peered":"2023-12-19T09:02:41.678069+0000","last_unstale":"2023-12-19T09:02:41.678358+0000","last_undegraded":"2023-12-19T09:02:41.678358+0000","last_fullsized":"2023-12-19T09:02:41.678358+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:34.297944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"160'4","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700820+0000","last_change":"2023-12-19T09:02:40.679315+0000","last_active":"2023-12-19T09:02:41.700820+0000","last_peered":"2023-12-19T09:02:41.700820+0000","last_clean":"2023-12-19T09:02:41.700820+0000","last_became_active":"2023-12-19T09:02:40.679009+0000","last_became_peered":"2023-12-19T09:02:40.679009+0000","last_unstale":"2023-12-19T09:02:41.700820+0000","last_undegraded":"2023-12-19T09:02:41.700820+0000","last_fullsized":"2023-12-19T09:02:41.700820+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:01:54.287384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.682897+0000","last_change":"2023-12-19T09:02:41.676211+0000","last_active":"2023-12-19T09:02:41.682897+0000","last_peered":"2023-12-19T09:02:41.682897+0000","last_clean":"2023-12-19T09:02:41.682897+0000","last_became_active":"2023-12-19T09:02:41.675473+0000","last_became_peered":"2023-12-19T09:02:41.675473+0000","last_unstale":"2023-12-19T09:02:41.682897+0000","last_undegraded":"2023-12-19T09:02:41.682897+0000","last_fullsized":"2023-12-19T09:02:41.682897+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:05:01.925670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691878+0000","last_change":"2023-12-19T09:02:40.676355+0000","last_active":"2023-12-19T09:02:41.691878+0000","last_peered":"2023-12-19T09:02:41.691878+0000","last_clean":"2023-12-19T09:02:41.691878+0000","last_became_active":"2023-12-19T09:02:40.676055+0000","last_became_peered":"2023-12-19T09:02:40.676055+0000","last_unstale":"2023-12-19T09:02:41.691878+0000","last_undegraded":"2023-12-19T09:02:41.691878+0000","last_fullsized":"2023-12-19T09:02:41.691878+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:16.454878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686468+0000","last_change":"2023-12-19T09:02:41.681345+0000","last_active":"2023-12-19T09:02:41.686468+0000","last_peered":"2023-12-19T09:02:41.686468+0000","last_clean":"2023-12-19T09:02:41.686468+0000","last_became_active":"2023-12-19T09:02:41.681027+0000","last_became_peered":"2023-12-19T09:02:41.681027+0000","last_unstale":"2023-12-19T09:02:41.686468+0000","last_undegraded":"2023-12-19T09:02:41.686468+0000","last_fullsized":"2023-12-19T09:02:41.686468+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:58.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.703125+0000","last_change":"2023-12-19T09:02:40.681079+0000","last_active":"2023-12-19T09:02:41.703125+0000","last_peered":"2023-12-19T09:02:41.703125+0000","last_clean":"2023-12-19T09:02:41.703125+0000","last_became_active":"2023-12-19T09:02:40.680802+0000","last_became_peered":"2023-12-19T09:02:40.680802+0000","last_unstale":"2023-12-19T09:02:41.703125+0000","last_undegraded":"2023-12-19T09:02:41.703125+0000","last_fullsized":"2023-12-19T09:02:41.703125+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:26.793552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686113+0000","last_change":"2023-12-19T09:02:41.678854+0000","last_active":"2023-12-19T09:02:41.686113+0000","last_peered":"2023-12-19T09:02:41.686113+0000","last_clean":"2023-12-19T09:02:41.686113+0000","last_became_active":"2023-12-19T09:02:41.678496+0000","last_became_peered":"2023-12-19T09:02:41.678496+0000","last_unstale":"2023-12-19T09:02:41.686113+0000","last_undegraded":"2023-12-19T09:02:41.686113+0000","last_fullsized":"2023-12-19T09:02:41.686113+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:06:56.210764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.701694+0000","last_change":"2023-12-19T09:02:40.679588+0000","last_active":"2023-12-19T09:02:41.701694+0000","last_peered":"2023-12-19T09:02:41.701694+0000","last_clean":"2023-12-19T09:02:41.701694+0000","last_became_active":"2023-12-19T09:02:40.679300+0000","last_became_peered":"2023-12-19T09:02:40.679300+0000","last_unstale":"2023-12-19T09:02:41.701694+0000","last_undegraded":"2023-12-19T09:02:41.701694+0000","last_fullsized":"2023-12-19T09:02:41.701694+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:43:47.579915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686279+0000","last_change":"2023-12-19T09:02:41.686279+0000","last_active":"2023-12-19T09:02:41.686279+0000","last_peered":"2023-12-19T09:02:41.686279+0000","last_clean":"2023-12-19T09:02:41.686279+0000","last_became_active":"2023-12-19T09:02:41.686011+0000","last_became_peered":"2023-12-19T09:02:41.686011+0000","last_unstale":"2023-12-19T09:02:41.686279+0000","last_undegraded":"2023-12-19T09:02:41.686279+0000","last_fullsized":"2023-12-19T09:02:41.686279+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:19.645398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"160'1","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.660901+0000","last_change":"2023-12-19T09:02:40.669273+0000","last_active":"2023-12-19T09:02:41.660901+0000","last_peered":"2023-12-19T09:02:41.660901+0000","last_clean":"2023-12-19T09:02:41.660901+0000","last_became_active":"2023-12-19T09:02:40.668687+0000","last_became_peered":"2023-12-19T09:02:40.668687+0000","last_unstale":"2023-12-19T09:02:41.660901+0000","last_undegraded":"2023-12-19T09:02:41.660901+0000","last_fullsized":"2023-12-19T09:02:41.660901+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:45:42.920150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.674412+0000","last_change":"2023-12-19T09:02:41.674412+0000","last_active":"2023-12-19T09:02:41.674412+0000","last_peered":"2023-12-19T09:02:41.674412+0000","last_clean":"2023-12-19T09:02:41.674412+0000","last_became_active":"2023-12-19T09:02:41.674133+0000","last_became_peered":"2023-12-19T09:02:41.674133+0000","last_unstale":"2023-12-19T09:02:41.674412+0000","last_undegraded":"2023-12-19T09:02:41.674412+0000","last_fullsized":"2023-12-19T09:02:41.674412+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:18.930961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690611+0000","last_change":"2023-12-19T09:02:40.667693+0000","last_active":"2023-12-19T09:02:41.690611+0000","last_peered":"2023-12-19T09:02:41.690611+0000","last_clean":"2023-12-19T09:02:41.690611+0000","last_became_active":"2023-12-19T09:02:40.667395+0000","last_became_peered":"2023-12-19T09:02:40.667395+0000","last_unstale":"2023-12-19T09:02:41.690611+0000","last_undegraded":"2023-12-19T09:02:41.690611+0000","last_fullsized":"2023-12-19T09:02:41.690611+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:42:38.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690037+0000","last_change":"2023-12-19T09:02:41.676813+0000","last_active":"2023-12-19T09:02:41.690037+0000","last_peered":"2023-12-19T09:02:41.690037+0000","last_clean":"2023-12-19T09:02:41.690037+0000","last_became_active":"2023-12-19T09:02:41.676513+0000","last_became_peered":"2023-12-19T09:02:41.676513+0000","last_unstale":"2023-12-19T09:02:41.690037+0000","last_undegraded":"2023-12-19T09:02:41.690037+0000","last_fullsized":"2023-12-19T09:02:41.690037+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:32.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.681791+0000","last_change":"2023-12-19T09:02:40.680571+0000","last_active":"2023-12-19T09:02:40.681791+0000","last_peered":"2023-12-19T09:02:40.681791+0000","last_clean":"2023-12-19T09:02:40.681791+0000","last_became_active":"2023-12-19T09:02:40.680271+0000","last_became_peered":"2023-12-19T09:02:40.680271+0000","last_unstale":"2023-12-19T09:02:40.681791+0000","last_undegraded":"2023-12-19T09:02:40.681791+0000","last_fullsized":"2023-12-19T09:02:40.681791+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:34.307881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697937+0000","last_change":"2023-12-19T09:02:41.691924+0000","last_active":"2023-12-19T09:02:41.697937+0000","last_peered":"2023-12-19T09:02:41.697937+0000","last_clean":"2023-12-19T09:02:41.697937+0000","last_became_active":"2023-12-19T09:02:41.691594+0000","last_became_peered":"2023-12-19T09:02:41.691594+0000","last_unstale":"2023-12-19T09:02:41.697937+0000","last_undegraded":"2023-12-19T09:02:41.697937+0000","last_fullsized":"2023-12-19T09:02:41.697937+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:09.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.683958+0000","last_change":"2023-12-19T09:02:40.682406+0000","last_active":"2023-12-19T09:02:40.683958+0000","last_peered":"2023-12-19T09:02:40.683958+0000","last_clean":"2023-12-19T09:02:40.683958+0000","last_became_active":"2023-12-19T09:02:40.682140+0000","last_became_peered":"2023-12-19T09:02:40.682140+0000","last_unstale":"2023-12-19T09:02:40.683958+0000","last_undegraded":"2023-12-19T09:02:40.683958+0000","last_fullsized":"2023-12-19T09:02:40.683958+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:11:53.542618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695606+0000","last_change":"2023-12-19T09:02:41.687001+0000","last_active":"2023-12-19T09:02:41.695606+0000","last_peered":"2023-12-19T09:02:41.695606+0000","last_clean":"2023-12-19T09:02:41.695606+0000","last_became_active":"2023-12-19T09:02:41.686681+0000","last_became_peered":"2023-12-19T09:02:41.686681+0000","last_unstale":"2023-12-19T09:02:41.695606+0000","last_undegraded":"2023-12-19T09:02:41.695606+0000","last_fullsized":"2023-12-19T09:02:41.695606+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:44.338604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671837+0000","last_change":"2023-12-19T09:02:40.680017+0000","last_active":"2023-12-19T09:02:41.671837+0000","last_peered":"2023-12-19T09:02:41.671837+0000","last_clean":"2023-12-19T09:02:41.671837+0000","last_became_active":"2023-12-19T09:02:40.679719+0000","last_became_peered":"2023-12-19T09:02:40.679719+0000","last_unstale":"2023-12-19T09:02:41.671837+0000","last_undegraded":"2023-12-19T09:02:41.671837+0000","last_fullsized":"2023-12-19T09:02:41.671837+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:12:42.129093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.686175+0000","last_change":"2023-12-19T09:02:41.686175+0000","last_active":"2023-12-19T09:02:41.686175+0000","last_peered":"2023-12-19T09:02:41.686175+0000","last_clean":"2023-12-19T09:02:41.686175+0000","last_became_active":"2023-12-19T09:02:41.682676+0000","last_became_peered":"2023-12-19T09:02:41.682676+0000","last_unstale":"2023-12-19T09:02:41.686175+0000","last_undegraded":"2023-12-19T09:02:41.686175+0000","last_fullsized":"2023-12-19T09:02:41.686175+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:26.456697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664196+0000","last_change":"2023-12-19T09:02:40.670863+0000","last_active":"2023-12-19T09:02:41.664196+0000","last_peered":"2023-12-19T09:02:41.664196+0000","last_clean":"2023-12-19T09:02:41.664196+0000","last_became_active":"2023-12-19T09:02:40.669522+0000","last_became_peered":"2023-12-19T09:02:40.669522+0000","last_unstale":"2023-12-19T09:02:41.664196+0000","last_undegraded":"2023-12-19T09:02:41.664196+0000","last_fullsized":"2023-12-19T09:02:41.664196+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:32:16.204350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.684642+0000","last_change":"2023-12-19T09:02:41.684642+0000","last_active":"2023-12-19T09:02:41.684642+0000","last_peered":"2023-12-19T09:02:41.684642+0000","last_clean":"2023-12-19T09:02:41.684642+0000","last_became_active":"2023-12-19T09:02:41.684303+0000","last_became_peered":"2023-12-19T09:02:41.684303+0000","last_unstale":"2023-12-19T09:02:41.684642+0000","last_undegraded":"2023-12-19T09:02:41.684642+0000","last_fullsized":"2023-12-19T09:02:41.684642+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:14.349193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":13,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.702490+0000","last_change":"2023-12-19T09:02:40.682208+0000","last_active":"2023-12-19T09:02:41.702490+0000","last_peered":"2023-12-19T09:02:41.702490+0000","last_clean":"2023-12-19T09:02:41.702490+0000","last_became_active":"2023-12-19T09:02:40.681893+0000","last_became_peered":"2023-12-19T09:02:40.681893+0000","last_unstale":"2023-12-19T09:02:41.702490+0000","last_undegraded":"2023-12-19T09:02:41.702490+0000","last_fullsized":"2023-12-19T09:02:41.702490+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:38.349011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.693169+0000","last_change":"2023-12-19T09:02:41.693169+0000","last_active":"2023-12-19T09:02:41.693169+0000","last_peered":"2023-12-19T09:02:41.693169+0000","last_clean":"2023-12-19T09:02:41.693169+0000","last_became_active":"2023-12-19T09:02:41.692820+0000","last_became_peered":"2023-12-19T09:02:41.692820+0000","last_unstale":"2023-12-19T09:02:41.693169+0000","last_undegraded":"2023-12-19T09:02:41.693169+0000","last_fullsized":"2023-12-19T09:02:41.693169+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:54:52.261512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":16,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.704762+0000","last_change":"2023-12-19T09:02:40.680150+0000","last_active":"2023-12-19T09:02:41.704762+0000","last_peered":"2023-12-19T09:02:41.704762+0000","last_clean":"2023-12-19T09:02:41.704762+0000","last_became_active":"2023-12-19T09:02:40.679876+0000","last_became_peered":"2023-12-19T09:02:40.679876+0000","last_unstale":"2023-12-19T09:02:41.704762+0000","last_undegraded":"2023-12-19T09:02:41.704762+0000","last_fullsized":"2023-12-19T09:02:41.704762+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:35.957548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671706+0000","last_change":"2023-12-19T09:02:41.671706+0000","last_active":"2023-12-19T09:02:41.671706+0000","last_peered":"2023-12-19T09:02:41.671706+0000","last_clean":"2023-12-19T09:02:41.671706+0000","last_became_active":"2023-12-19T09:02:41.671044+0000","last_became_peered":"2023-12-19T09:02:41.671044+0000","last_unstale":"2023-12-19T09:02:41.671706+0000","last_undegraded":"2023-12-19T09:02:41.671706+0000","last_fullsized":"2023-12-19T09:02:41.671706+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:17.459164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.696824+0000","last_change":"2023-12-19T09:02:40.670168+0000","last_active":"2023-12-19T09:02:41.696824+0000","last_peered":"2023-12-19T09:02:41.696824+0000","last_clean":"2023-12-19T09:02:41.696824+0000","last_became_active":"2023-12-19T09:02:40.669843+0000","last_became_peered":"2023-12-19T09:02:40.669843+0000","last_unstale":"2023-12-19T09:02:41.696824+0000","last_undegraded":"2023-12-19T09:02:41.696824+0000","last_fullsized":"2023-12-19T09:02:41.696824+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:53.017216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.682238+0000","last_change":"2023-12-19T09:02:41.679662+0000","last_active":"2023-12-19T09:02:41.682238+0000","last_peered":"2023-12-19T09:02:41.682238+0000","last_clean":"2023-12-19T09:02:41.682238+0000","last_became_active":"2023-12-19T09:02:41.679255+0000","last_became_peered":"2023-12-19T09:02:41.679255+0000","last_unstale":"2023-12-19T09:02:41.682238+0000","last_undegraded":"2023-12-19T09:02:41.682238+0000","last_fullsized":"2023-12-19T09:02:41.682238+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:17:00.972380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.680582+0000","last_change":"2023-12-19T09:02:40.677683+0000","last_active":"2023-12-19T09:02:40.680582+0000","last_peered":"2023-12-19T09:02:40.680582+0000","last_clean":"2023-12-19T09:02:40.680582+0000","last_became_active":"2023-12-19T09:02:40.677345+0000","last_became_peered":"2023-12-19T09:02:40.677345+0000","last_unstale":"2023-12-19T09:02:40.680582+0000","last_undegraded":"2023-12-19T09:02:40.680582+0000","last_fullsized":"2023-12-19T09:02:40.680582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:24.298166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.693434+0000","last_change":"2023-12-19T09:02:41.691385+0000","last_active":"2023-12-19T09:02:41.693434+0000","last_peered":"2023-12-19T09:02:41.693434+0000","last_clean":"2023-12-19T09:02:41.693434+0000","last_became_active":"2023-12-19T09:02:41.690473+0000","last_became_peered":"2023-12-19T09:02:41.690473+0000","last_unstale":"2023-12-19T09:02:41.693434+0000","last_undegraded":"2023-12-19T09:02:41.693434+0000","last_fullsized":"2023-12-19T09:02:41.693434+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:29.217800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":17,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.696668+0000","last_change":"2023-12-19T09:02:40.678650+0000","last_active":"2023-12-19T09:02:41.696668+0000","last_peered":"2023-12-19T09:02:41.696668+0000","last_clean":"2023-12-19T09:02:41.696668+0000","last_became_active":"2023-12-19T09:02:40.678339+0000","last_became_peered":"2023-12-19T09:02:40.678339+0000","last_unstale":"2023-12-19T09:02:41.696668+0000","last_undegraded":"2023-12-19T09:02:41.696668+0000","last_fullsized":"2023-12-19T09:02:41.696668+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:12:42.129093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695536+0000","last_change":"2023-12-19T09:02:41.695536+0000","last_active":"2023-12-19T09:02:41.695536+0000","last_peered":"2023-12-19T09:02:41.695536+0000","last_clean":"2023-12-19T09:02:41.695536+0000","last_became_active":"2023-12-19T09:02:41.695029+0000","last_became_peered":"2023-12-19T09:02:41.695029+0000","last_unstale":"2023-12-19T09:02:41.695536+0000","last_undegraded":"2023-12-19T09:02:41.695536+0000","last_fullsized":"2023-12-19T09:02:41.695536+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:29:38.972931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"160'1","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688643+0000","last_change":"2023-12-19T09:02:40.676639+0000","last_active":"2023-12-19T09:02:41.688643+0000","last_peered":"2023-12-19T09:02:41.688643+0000","last_clean":"2023-12-19T09:02:41.688643+0000","last_became_active":"2023-12-19T09:02:40.675499+0000","last_became_peered":"2023-12-19T09:02:40.675499+0000","last_unstale":"2023-12-19T09:02:41.688643+0000","last_undegraded":"2023-12-19T09:02:41.688643+0000","last_fullsized":"2023-12-19T09:02:41.688643+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687973+0000","last_change":"2023-12-19T09:02:41.682080+0000","last_active":"2023-12-19T09:02:41.687973+0000","last_peered":"2023-12-19T09:02:41.687973+0000","last_clean":"2023-12-19T09:02:41.687973+0000","last_became_active":"2023-12-19T09:02:41.681581+0000","last_became_peered":"2023-12-19T09:02:41.681581+0000","last_unstale":"2023-12-19T09:02:41.687973+0000","last_undegraded":"2023-12-19T09:02:41.687973+0000","last_fullsized":"2023-12-19T09:02:41.687973+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:19.953419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"160'2","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.700539+0000","last_change":"2023-12-19T09:02:40.681535+0000","last_active":"2023-12-19T09:02:41.700539+0000","last_peered":"2023-12-19T09:02:41.700539+0000","last_clean":"2023-12-19T09:02:41.700539+0000","last_became_active":"2023-12-19T09:02:40.681249+0000","last_became_peered":"2023-12-19T09:02:40.681249+0000","last_unstale":"2023-12-19T09:02:41.700539+0000","last_undegraded":"2023-12-19T09:02:41.700539+0000","last_fullsized":"2023-12-19T09:02:41.700539+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:44:31.372754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698227+0000","last_change":"2023-12-19T09:02:41.692306+0000","last_active":"2023-12-19T09:02:41.698227+0000","last_peered":"2023-12-19T09:02:41.698227+0000","last_clean":"2023-12-19T09:02:41.698227+0000","last_became_active":"2023-12-19T09:02:41.691997+0000","last_became_peered":"2023-12-19T09:02:41.691997+0000","last_unstale":"2023-12-19T09:02:41.698227+0000","last_undegraded":"2023-12-19T09:02:41.698227+0000","last_fullsized":"2023-12-19T09:02:41.698227+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:58:43.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.669861+0000","last_change":"2023-12-19T09:02:40.677208+0000","last_active":"2023-12-19T09:02:41.669861+0000","last_peered":"2023-12-19T09:02:41.669861+0000","last_clean":"2023-12-19T09:02:41.669861+0000","last_became_active":"2023-12-19T09:02:40.674996+0000","last_became_peered":"2023-12-19T09:02:40.674996+0000","last_unstale":"2023-12-19T09:02:41.669861+0000","last_undegraded":"2023-12-19T09:02:41.669861+0000","last_fullsized":"2023-12-19T09:02:41.669861+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:18:17.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687068+0000","last_change":"2023-12-19T09:02:41.687068+0000","last_active":"2023-12-19T09:02:41.687068+0000","last_peered":"2023-12-19T09:02:41.687068+0000","last_clean":"2023-12-19T09:02:41.687068+0000","last_became_active":"2023-12-19T09:02:41.686750+0000","last_became_peered":"2023-12-19T09:02:41.686750+0000","last_unstale":"2023-12-19T09:02:41.687068+0000","last_undegraded":"2023-12-19T09:02:41.687068+0000","last_fullsized":"2023-12-19T09:02:41.687068+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:17:00.972380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.669437+0000","last_change":"2023-12-19T09:02:40.674613+0000","last_active":"2023-12-19T09:02:41.669437+0000","last_peered":"2023-12-19T09:02:41.669437+0000","last_clean":"2023-12-19T09:02:41.669437+0000","last_became_active":"2023-12-19T09:02:40.674262+0000","last_became_peered":"2023-12-19T09:02:40.674262+0000","last_unstale":"2023-12-19T09:02:41.669437+0000","last_undegraded":"2023-12-19T09:02:41.669437+0000","last_fullsized":"2023-12-19T09:02:41.669437+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:46:18.949132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.692143+0000","last_change":"2023-12-19T09:02:41.686311+0000","last_active":"2023-12-19T09:02:41.692143+0000","last_peered":"2023-12-19T09:02:41.692143+0000","last_clean":"2023-12-19T09:02:41.692143+0000","last_became_active":"2023-12-19T09:02:41.685951+0000","last_became_peered":"2023-12-19T09:02:41.685951+0000","last_unstale":"2023-12-19T09:02:41.692143+0000","last_undegraded":"2023-12-19T09:02:41.692143+0000","last_fullsized":"2023-12-19T09:02:41.692143+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:28:46.809959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"160'2","reported_seq":11,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.701556+0000","last_change":"2023-12-19T09:02:40.673205+0000","last_active":"2023-12-19T09:02:41.701556+0000","last_peered":"2023-12-19T09:02:41.701556+0000","last_clean":"2023-12-19T09:02:41.701556+0000","last_became_active":"2023-12-19T09:02:40.672882+0000","last_became_peered":"2023-12-19T09:02:40.672882+0000","last_unstale":"2023-12-19T09:02:41.701556+0000","last_undegraded":"2023-12-19T09:02:41.701556+0000","last_fullsized":"2023-12-19T09:02:41.701556+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:02:19.288893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.687204+0000","last_change":"2023-12-19T09:02:41.678873+0000","last_active":"2023-12-19T09:02:41.687204+0000","last_peered":"2023-12-19T09:02:41.687204+0000","last_clean":"2023-12-19T09:02:41.687204+0000","last_became_active":"2023-12-19T09:02:41.678533+0000","last_became_peered":"2023-12-19T09:02:41.678533+0000","last_unstale":"2023-12-19T09:02:41.687204+0000","last_undegraded":"2023-12-19T09:02:41.687204+0000","last_fullsized":"2023-12-19T09:02:41.687204+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:09:23.778315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663987+0000","last_change":"2023-12-19T09:02:40.667609+0000","last_active":"2023-12-19T09:02:41.663987+0000","last_peered":"2023-12-19T09:02:41.663987+0000","last_clean":"2023-12-19T09:02:41.663987+0000","last_became_active":"2023-12-19T09:02:40.667322+0000","last_became_peered":"2023-12-19T09:02:40.667322+0000","last_unstale":"2023-12-19T09:02:41.663987+0000","last_undegraded":"2023-12-19T09:02:41.663987+0000","last_fullsized":"2023-12-19T09:02:41.663987+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:07:39.411131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697411+0000","last_change":"2023-12-19T09:02:41.694968+0000","last_active":"2023-12-19T09:02:41.697411+0000","last_peered":"2023-12-19T09:02:41.697411+0000","last_clean":"2023-12-19T09:02:41.697411+0000","last_became_active":"2023-12-19T09:02:41.694556+0000","last_became_peered":"2023-12-19T09:02:41.694556+0000","last_unstale":"2023-12-19T09:02:41.697411+0000","last_undegraded":"2023-12-19T09:02:41.697411+0000","last_fullsized":"2023-12-19T09:02:41.697411+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:40:44.056173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664289+0000","last_change":"2023-12-19T09:02:40.681138+0000","last_active":"2023-12-19T09:02:41.664289+0000","last_peered":"2023-12-19T09:02:41.664289+0000","last_clean":"2023-12-19T09:02:41.664289+0000","last_became_active":"2023-12-19T09:02:40.680701+0000","last_became_peered":"2023-12-19T09:02:40.680701+0000","last_unstale":"2023-12-19T09:02:41.664289+0000","last_undegraded":"2023-12-19T09:02:41.664289+0000","last_fullsized":"2023-12-19T09:02:41.664289+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:09:47.280420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.695105+0000","last_change":"2023-12-19T09:02:41.692829+0000","last_active":"2023-12-19T09:02:41.695105+0000","last_peered":"2023-12-19T09:02:41.695105+0000","last_clean":"2023-12-19T09:02:41.695105+0000","last_became_active":"2023-12-19T09:02:41.692489+0000","last_became_peered":"2023-12-19T09:02:41.692489+0000","last_unstale":"2023-12-19T09:02:41.695105+0000","last_undegraded":"2023-12-19T09:02:41.695105+0000","last_fullsized":"2023-12-19T09:02:41.695105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:09:26.456697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.676897+0000","last_change":"2023-12-19T09:02:40.674065+0000","last_active":"2023-12-19T09:02:40.676897+0000","last_peered":"2023-12-19T09:02:40.676897+0000","last_clean":"2023-12-19T09:02:40.676897+0000","last_became_active":"2023-12-19T09:02:40.673072+0000","last_became_peered":"2023-12-19T09:02:40.673072+0000","last_unstale":"2023-12-19T09:02:40.676897+0000","last_undegraded":"2023-12-19T09:02:40.676897+0000","last_fullsized":"2023-12-19T09:02:40.676897+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:16:33.293657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.689672+0000","last_change":"2023-12-19T09:02:41.674870+0000","last_active":"2023-12-19T09:02:41.689672+0000","last_peered":"2023-12-19T09:02:41.689672+0000","last_clean":"2023-12-19T09:02:41.689672+0000","last_became_active":"2023-12-19T09:02:41.674467+0000","last_became_peered":"2023-12-19T09:02:41.674467+0000","last_unstale":"2023-12-19T09:02:41.689672+0000","last_undegraded":"2023-12-19T09:02:41.689672+0000","last_fullsized":"2023-12-19T09:02:41.689672+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:24:46.813624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"160'2","reported_seq":12,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698738+0000","last_change":"2023-12-19T09:02:40.679348+0000","last_active":"2023-12-19T09:02:41.698738+0000","last_peered":"2023-12-19T09:02:41.698738+0000","last_clean":"2023-12-19T09:02:41.698738+0000","last_became_active":"2023-12-19T09:02:40.678397+0000","last_became_peered":"2023-12-19T09:02:40.678397+0000","last_unstale":"2023-12-19T09:02:41.698738+0000","last_undegraded":"2023-12-19T09:02:41.698738+0000","last_fullsized":"2023-12-19T09:02:41.698738+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:35.667111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.680120+0000","last_change":"2023-12-19T09:02:41.680120+0000","last_active":"2023-12-19T09:02:41.680120+0000","last_peered":"2023-12-19T09:02:41.680120+0000","last_clean":"2023-12-19T09:02:41.680120+0000","last_became_active":"2023-12-19T09:02:41.679824+0000","last_became_peered":"2023-12-19T09:02:41.679824+0000","last_unstale":"2023-12-19T09:02:41.680120+0000","last_undegraded":"2023-12-19T09:02:41.680120+0000","last_fullsized":"2023-12-19T09:02:41.680120+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:18.511639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.679113+0000","last_change":"2023-12-19T09:02:40.676423+0000","last_active":"2023-12-19T09:02:40.679113+0000","last_peered":"2023-12-19T09:02:40.679113+0000","last_clean":"2023-12-19T09:02:40.679113+0000","last_became_active":"2023-12-19T09:02:40.676103+0000","last_became_peered":"2023-12-19T09:02:40.676103+0000","last_unstale":"2023-12-19T09:02:40.679113+0000","last_undegraded":"2023-12-19T09:02:40.679113+0000","last_fullsized":"2023-12-19T09:02:40.679113+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:29:37.968645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.698249+0000","last_change":"2023-12-19T09:02:41.690109+0000","last_active":"2023-12-19T09:02:41.698249+0000","last_peered":"2023-12-19T09:02:41.698249+0000","last_clean":"2023-12-19T09:02:41.698249+0000","last_became_active":"2023-12-19T09:02:41.688198+0000","last_became_peered":"2023-12-19T09:02:41.688198+0000","last_unstale":"2023-12-19T09:02:41.698249+0000","last_undegraded":"2023-12-19T09:02:41.698249+0000","last_fullsized":"2023-12-19T09:02:41.698249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:36:17.459164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.677954+0000","last_change":"2023-12-19T09:02:40.673689+0000","last_active":"2023-12-19T09:02:40.677954+0000","last_peered":"2023-12-19T09:02:40.677954+0000","last_clean":"2023-12-19T09:02:40.677954+0000","last_became_active":"2023-12-19T09:02:40.671125+0000","last_became_peered":"2023-12-19T09:02:40.671125+0000","last_unstale":"2023-12-19T09:02:40.677954+0000","last_undegraded":"2023-12-19T09:02:40.677954+0000","last_fullsized":"2023-12-19T09:02:40.677954+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:28:17.507353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.664013+0000","last_change":"2023-12-19T09:02:41.664013+0000","last_active":"2023-12-19T09:02:41.664013+0000","last_peered":"2023-12-19T09:02:41.664013+0000","last_clean":"2023-12-19T09:02:41.664013+0000","last_became_active":"2023-12-19T09:02:41.663710+0000","last_became_peered":"2023-12-19T09:02:41.663710+0000","last_unstale":"2023-12-19T09:02:41.664013+0000","last_undegraded":"2023-12-19T09:02:41.664013+0000","last_fullsized":"2023-12-19T09:02:41.664013+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:22:23.800325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.677065+0000","last_change":"2023-12-19T09:02:40.673552+0000","last_active":"2023-12-19T09:02:40.677065+0000","last_peered":"2023-12-19T09:02:40.677065+0000","last_clean":"2023-12-19T09:02:40.677065+0000","last_became_active":"2023-12-19T09:02:40.673224+0000","last_became_peered":"2023-12-19T09:02:40.673224+0000","last_unstale":"2023-12-19T09:02:40.677065+0000","last_undegraded":"2023-12-19T09:02:40.677065+0000","last_fullsized":"2023-12-19T09:02:40.677065+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:29:11.665711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.683944+0000","last_change":"2023-12-19T09:02:41.683944+0000","last_active":"2023-12-19T09:02:41.683944+0000","last_peered":"2023-12-19T09:02:41.683944+0000","last_clean":"2023-12-19T09:02:41.683944+0000","last_became_active":"2023-12-19T09:02:41.683605+0000","last_became_peered":"2023-12-19T09:02:41.683605+0000","last_unstale":"2023-12-19T09:02:41.683944+0000","last_undegraded":"2023-12-19T09:02:41.683944+0000","last_fullsized":"2023-12-19T09:02:41.683944+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:58.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.671545+0000","last_change":"2023-12-19T09:02:40.672906+0000","last_active":"2023-12-19T09:02:41.671545+0000","last_peered":"2023-12-19T09:02:41.671545+0000","last_clean":"2023-12-19T09:02:41.671545+0000","last_became_active":"2023-12-19T09:02:40.672617+0000","last_became_peered":"2023-12-19T09:02:40.672617+0000","last_unstale":"2023-12-19T09:02:41.671545+0000","last_undegraded":"2023-12-19T09:02:41.671545+0000","last_fullsized":"2023-12-19T09:02:41.671545+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:34.307881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.690160+0000","last_change":"2023-12-19T09:02:41.690160+0000","last_active":"2023-12-19T09:02:41.690160+0000","last_peered":"2023-12-19T09:02:41.690160+0000","last_clean":"2023-12-19T09:02:41.690160+0000","last_became_active":"2023-12-19T09:02:41.689873+0000","last_became_peered":"2023-12-19T09:02:41.689873+0000","last_unstale":"2023-12-19T09:02:41.690160+0000","last_undegraded":"2023-12-19T09:02:41.690160+0000","last_fullsized":"2023-12-19T09:02:41.690160+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:09:39.353297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.665923+0000","last_change":"2023-12-19T09:02:40.678865+0000","last_active":"2023-12-19T09:02:41.665923+0000","last_peered":"2023-12-19T09:02:41.665923+0000","last_clean":"2023-12-19T09:02:41.665923+0000","last_became_active":"2023-12-19T09:02:40.678342+0000","last_became_peered":"2023-12-19T09:02:40.678342+0000","last_unstale":"2023-12-19T09:02:41.665923+0000","last_undegraded":"2023-12-19T09:02:41.665923+0000","last_fullsized":"2023-12-19T09:02:41.665923+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:12:04.638962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.691360+0000","last_change":"2023-12-19T09:02:41.691360+0000","last_active":"2023-12-19T09:02:41.691360+0000","last_peered":"2023-12-19T09:02:41.691360+0000","last_clean":"2023-12-19T09:02:41.691360+0000","last_became_active":"2023-12-19T09:02:41.684652+0000","last_became_peered":"2023-12-19T09:02:41.684652+0000","last_unstale":"2023-12-19T09:02:41.691360+0000","last_undegraded":"2023-12-19T09:02:41.691360+0000","last_fullsized":"2023-12-19T09:02:41.691360+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:40:45.700194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-12-19T09:02:40.681605+0000","last_change":"2023-12-19T09:02:40.679845+0000","last_active":"2023-12-19T09:02:40.681605+0000","last_peered":"2023-12-19T09:02:40.681605+0000","last_clean":"2023-12-19T09:02:40.681605+0000","last_became_active":"2023-12-19T09:02:40.679189+0000","last_became_peered":"2023-12-19T09:02:40.679189+0000","last_unstale":"2023-12-19T09:02:40.681605+0000","last_undegraded":"2023-12-19T09:02:40.681605+0000","last_fullsized":"2023-12-19T09:02:40.681605+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:58:48.569570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.697959+0000","last_change":"2023-12-19T09:02:41.691978+0000","last_active":"2023-12-19T09:02:41.697959+0000","last_peered":"2023-12-19T09:02:41.697959+0000","last_clean":"2023-12-19T09:02:41.697959+0000","last_became_active":"2023-12-19T09:02:41.691679+0000","last_became_peered":"2023-12-19T09:02:41.691679+0000","last_unstale":"2023-12-19T09:02:41.697959+0000","last_undegraded":"2023-12-19T09:02:41.697959+0000","last_fullsized":"2023-12-19T09:02:41.697959+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:27:01.696144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663614+0000","last_change":"2023-12-19T09:02:40.672742+0000","last_active":"2023-12-19T09:02:41.663614+0000","last_peered":"2023-12-19T09:02:41.663614+0000","last_clean":"2023-12-19T09:02:41.663614+0000","last_became_active":"2023-12-19T09:02:40.672417+0000","last_became_peered":"2023-12-19T09:02:40.672417+0000","last_unstale":"2023-12-19T09:02:41.663614+0000","last_undegraded":"2023-12-19T09:02:41.663614+0000","last_fullsized":"2023-12-19T09:02:41.663614+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:16:59.968094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":8,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.688498+0000","last_change":"2023-12-19T09:02:41.688498+0000","last_active":"2023-12-19T09:02:41.688498+0000","last_peered":"2023-12-19T09:02:41.688498+0000","last_clean":"2023-12-19T09:02:41.688498+0000","last_became_active":"2023-12-19T09:02:41.685706+0000","last_became_peered":"2023-12-19T09:02:41.685706+0000","last_unstale":"2023-12-19T09:02:41.688498+0000","last_undegraded":"2023-12-19T09:02:41.688498+0000","last_fullsized":"2023-12-19T09:02:41.688498+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:40.639124+0000","last_clean_scrub_stamp":"2023-12-19T09:02:40.639124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:50:43.700709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.663237+0000","last_change":"2023-12-19T09:02:40.682057+0000","last_active":"2023-12-19T09:02:41.663237+0000","last_peered":"2023-12-19T09:02:41.663237+0000","last_clean":"2023-12-19T09:02:41.663237+0000","last_became_active":"2023-12-19T09:02:40.681531+0000","last_became_peered":"2023-12-19T09:02:40.681531+0000","last_unstale":"2023-12-19T09:02:41.663237+0000","last_undegraded":"2023-12-19T09:02:41.663237+0000","last_fullsized":"2023-12-19T09:02:41.663237+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:02:39.634838+0000","last_clean_scrub_stamp":"2023-12-19T09:02:39.634838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:14:57.227017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'271","reported_seq":825,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-12-19T09:02:41.666129+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:02:41.666129+0000","last_peered":"2023-12-19T09:02:41.666129+0000","last_clean":"2023-12-19T09:02:41.666129+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:02:41.666129+0000","last_undegraded":"2023-12-19T09:02:41.666129+0000","last_fullsized":"2023-12-19T09:02:41.666129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575749,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299832,"kb_used_data":2544,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94072008,"statfs":{"total":96636764160,"available":96329736192,"internally_reserved":0,"allocated":2605056,"data_stored":1648658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3500000000000001,"15min":1.3069999999999999},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.54400000000000004},"max":{"1min":1.649,"5min":1.712,"15min":1.712},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.302,"15min":1.3129999999999999},"min":{"1min":0.98099999999999998,"5min":0.74199999999999999,"15min":0.66800000000000004},"max":{"1min":1.4610000000000001,"5min":1.6559999999999999,"15min":1.788},"last":1.4319999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.878,"5min":0.69799999999999995,"15min":0.64000000000000001},"max":{"1min":1.6020000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1859999999999999,"15min":1.1899999999999999},"min":{"1min":0.874,"5min":0.73599999999999999,"15min":0.64500000000000002},"max":{"1min":1.4239999999999999,"5min":1.5880000000000001,"15min":1.784},"last":1.0429999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.177,"15min":1.1719999999999999},"min":{"1min":0.92900000000000005,"5min":0.70599999999999996,"15min":0.57199999999999995},"max":{"1min":1.504,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.454},{"interface":"front","average":{"1min":1.333,"5min":1.302,"15min":1.28},"min":{"1min":0.84699999999999998,"5min":0.79000000000000004,"15min":0.60899999999999999},"max":{"1min":1.5860000000000001,"5min":1.776,"15min":1.792},"last":1.2669999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.331,"15min":1.323},"min":{"1min":1.0780000000000001,"5min":1.034,"15min":0.625},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.778},"last":1.538},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3580000000000001,"15min":1.351},"min":{"1min":1.093,"5min":0.88900000000000001,"15min":0.61099999999999999},"max":{"1min":1.601,"5min":1.659,"15min":1.774},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2430000000000001,"15min":1.218},"min":{"1min":0.84499999999999997,"5min":0.76400000000000001,"15min":0.56799999999999995},"max":{"1min":1.694,"5min":1.847,"15min":1.847},"last":1.173},{"interface":"front","average":{"1min":1.282,"5min":1.248,"15min":1.2210000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.42299999999999999},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":1.21}]},{"osd":5,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2589999999999999,"15min":1.2330000000000001},"min":{"1min":1.002,"5min":0.85899999999999999,"15min":0.35599999999999998},"max":{"1min":1.7350000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.097},{"interface":"front","average":{"1min":1.496,"5min":1.413,"15min":1.365},"min":{"1min":1.0840000000000001,"5min":0.65300000000000002,"15min":0.498},"max":{"1min":1.7789999999999999,"5min":1.9099999999999999,"15min":1.97},"last":1.623}]},{"osd":6,"last update":"Tue Dec 19 09:02:12 2023","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4139999999999999,"15min":1.3819999999999999},"min":{"1min":1.032,"5min":0.80900000000000005,"15min":0.51600000000000001},"max":{"1min":1.841,"5min":1.841,"15min":1.8540000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3700000000000001,"15min":1.3660000000000001},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.47899999999999998},"max":{"1min":1.7569999999999999,"5min":1.804,"15min":1.859},"last":1.341}]}]},{"osd":6,"up_from":13,"seq":55834575748,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311532,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060308,"statfs":{"total":96636764160,"available":96317755392,"internally_reserved":0,"allocated":2002944,"data_stored":1057308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.145,"15min":1.1299999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.51500000000000001},"max":{"1min":1.5149999999999999,"5min":1.5609999999999999,"15min":1.6200000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.214,"5min":1.1930000000000001,"15min":1.1699999999999999},"min":{"1min":0.90800000000000003,"5min":0.81599999999999995,"15min":0.61399999999999999},"max":{"1min":1.5980000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.21}]},{"osd":1,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.238,"15min":1.196},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.591,"5min":1.696,"15min":1.696},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1719999999999999,"15min":1.181},"min":{"1min":0.83899999999999997,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.0209999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2609999999999999,"15min":1.244},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.6140000000000001,"5min":1.806,"15min":1.806},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2490000000000001,"15min":1.2390000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.63400000000000001},"max":{"1min":1.5700000000000001,"5min":1.738,"15min":1.796},"last":0.96499999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3080000000000001,"15min":1.2709999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.76300000000000001},"max":{"1min":1.5609999999999999,"5min":1.6879999999999999,"15min":1.7889999999999999},"last":1.111},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3140000000000001,"15min":1.327},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.7729999999999999},"last":1.359}]},{"osd":4,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1639999999999999,"15min":1.1870000000000001},"min":{"1min":0.73899999999999999,"5min":0.66800000000000004,"15min":0.63400000000000001},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4350000000000001,"15min":1.415},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.51100000000000001},"max":{"1min":1.669,"5min":1.8129999999999999,"15min":1.849},"last":1.327}]},{"osd":5,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2829999999999999,"15min":1.2869999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.756,"5min":1.784,"15min":1.994},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.27,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67700000000000005,"15min":0.56399999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.288}]},{"osd":7,"last update":"Tue Dec 19 09:02:18 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.3620000000000001,"15min":1.335},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.56100000000000005},"max":{"1min":1.7050000000000001,"5min":1.7949999999999999,"15min":1.9299999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4279999999999999,"15min":1.415},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7649999999999999,"5min":2.165,"15min":2.165},"last":1.1679999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303348,"kb_used_data":1964,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068492,"statfs":{"total":96636764160,"available":96326135808,"internally_reserved":0,"allocated":2011136,"data_stored":1063842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2769999999999999,"15min":1.304},"min":{"1min":0.73499999999999999,"5min":0.626,"15min":0.57999999999999996},"max":{"1min":1.546,"5min":1.6479999999999999,"15min":1.7450000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3080000000000001,"15min":1.3160000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.448}]},{"osd":1,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.161,"15min":1.169},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.57199999999999995},"max":{"1min":1.6419999999999999,"5min":1.7230000000000001,"15min":1.748},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.82499999999999996,"5min":0.80400000000000005,"15min":0.72499999999999998},"max":{"1min":1.5860000000000001,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.319}]},{"osd":2,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2290000000000001,"15min":1.2150000000000001},"min":{"1min":0.89200000000000002,"5min":0.73999999999999999,"15min":0.55300000000000005},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.595},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.224,"15min":1.2450000000000001},"min":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.4530000000000001,"5min":1.726,"15min":1.726},"last":1.4990000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.3049999999999999,"15min":1.2909999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5569999999999999,"5min":1.7609999999999999,"15min":1.891},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3160000000000001,"15min":1.306},"min":{"1min":0.79300000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.6930000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2010000000000001,"15min":1.196},"min":{"1min":0.94799999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5980000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.321,"15min":1.3069999999999999},"min":{"1min":0.92900000000000005,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.6220000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2350000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3340000000000001,"15min":1.2969999999999999},"min":{"1min":0.98399999999999999,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.7170000000000001,"5min":1.833,"15min":1.833},"last":1.778},{"interface":"front","average":{"1min":1.403,"5min":1.3819999999999999,"15min":1.3600000000000001},"min":{"1min":1.0349999999999999,"5min":0.79100000000000004,"15min":0.76800000000000002},"max":{"1min":1.7390000000000001,"5min":1.823,"15min":1.823},"last":1.5669999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:01:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.468,"15min":1.4350000000000001},"min":{"1min":0.98899999999999999,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.889,"5min":1.96,"15min":1.986},"last":1.409},{"interface":"front","average":{"1min":1.554,"5min":1.486,"15min":1.4650000000000001},"min":{"1min":0.97699999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.9530000000000001,"5min":1.99,"15min":1.99},"last":1.869}]}]},{"osd":4,"up_from":13,"seq":55834575748,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311524,"kb_used_data":1948,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060316,"statfs":{"total":96636764160,"available":96317763584,"internally_reserved":0,"allocated":1994752,"data_stored":1056664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.622},"max":{"1min":1.5669999999999999,"5min":1.6659999999999999,"15min":1.7270000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.78300000000000003},"max":{"1min":1.6180000000000001,"5min":1.7270000000000001,"15min":1.768},"last":0.89000000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.21,"15min":1.202},"min":{"1min":0.85799999999999998,"5min":0.80700000000000005,"15min":0.54500000000000004},"max":{"1min":1.5389999999999999,"5min":1.7869999999999999,"15min":1.8959999999999999},"last":1.004},{"interface":"front","average":{"1min":1.159,"5min":1.171,"15min":1.149},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.59999999999999998},"max":{"1min":1.389,"5min":1.712,"15min":1.712},"last":1.163}]},{"osd":2,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1890000000000001,"15min":1.173},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.66100000000000003},"max":{"1min":1.5409999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6200000000000001,"5min":1.7290000000000001,"15min":1.8149999999999999},"last":1.0960000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.69199999999999995},"max":{"1min":1.575,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":1.282},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.377,"15min":1.3779999999999999},"min":{"1min":0.95399999999999996,"5min":0.90200000000000002,"15min":0.77800000000000002},"max":{"1min":1.673,"5min":1.7589999999999999,"15min":1.863},"last":1.359}]},{"osd":5,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.258,"15min":1.228},"min":{"1min":0.92000000000000004,"5min":0.875,"15min":0.623},"max":{"1min":1.5660000000000001,"5min":1.796,"15min":1.962},"last":1.198},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.82799999999999996,"5min":0.69899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6879999999999999,"5min":1.823,"15min":1.823},"last":1.0580000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.266,"15min":1.238},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.59399999999999997},"max":{"1min":1.899,"5min":1.909,"15min":1.909},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.419,"15min":1.405},"min":{"1min":1.0700000000000001,"5min":1.034,"15min":0.64200000000000002},"max":{"1min":1.677,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.296}]},{"osd":7,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.337,"15min":1.3340000000000001},"min":{"1min":0.95199999999999996,"5min":0.86499999999999999,"15min":0.71799999999999997},"max":{"1min":1.851,"5min":1.851,"15min":1.919},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3999999999999999,"15min":1.393},"min":{"1min":1.0860000000000001,"5min":0.84099999999999997,"15min":0.65100000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.141}]}]},{"osd":3,"up_from":13,"seq":55834575748,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307456,"kb_used_data":1976,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064384,"statfs":{"total":96636764160,"available":96321929216,"internally_reserved":0,"allocated":2023424,"data_stored":1065426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.85699999999999998,"15min":0.86399999999999999},"min":{"1min":0.66000000000000003,"5min":0.60999999999999999,"15min":0.60399999999999998},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.998,"15min":0.98999999999999999},"min":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.59299999999999997},"max":{"1min":1.353,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.85899999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94199999999999995,"15min":0.94499999999999995},"min":{"1min":0.72399999999999998,"5min":0.66100000000000003,"15min":0.63},"max":{"1min":1.0940000000000001,"5min":1.528,"15min":2.456},"last":0.752},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0860000000000001,"15min":1.087},"min":{"1min":0.79300000000000004,"5min":0.70899999999999996,"15min":0.69699999999999995},"max":{"1min":1.337,"5min":1.538,"15min":1.694},"last":1.0960000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.083,"15min":1.095},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.75900000000000001},"max":{"1min":1.4570000000000001,"5min":1.534,"15min":2.0150000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.1930000000000001},"min":{"1min":0.91000000000000003,"5min":0.84299999999999997,"15min":0.81699999999999995},"max":{"1min":1.4399999999999999,"5min":1.6539999999999999,"15min":1.681},"last":1.331}]},{"osd":4,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2809999999999999,"15min":1.2829999999999999},"min":{"1min":1.0469999999999999,"5min":1.0229999999999999,"15min":0.94099999999999995},"max":{"1min":1.5409999999999999,"5min":1.6120000000000001,"15min":1.6679999999999999},"last":1.272},{"interface":"front","average":{"1min":1.294,"5min":1.264,"15min":1.254},"min":{"1min":1.0900000000000001,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.5580000000000001,"5min":1.742,"15min":2.6520000000000001},"last":1.143}]},{"osd":5,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.3169999999999999,"15min":1.296},"min":{"1min":1.071,"5min":0.97099999999999997,"15min":0.85399999999999998},"max":{"1min":1.4850000000000001,"5min":1.7589999999999999,"15min":2.7519999999999998},"last":1.405},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.395,"15min":1.393},"min":{"1min":1.101,"5min":1.0660000000000001,"15min":1.0129999999999999},"max":{"1min":1.8140000000000001,"5min":1.8759999999999999,"15min":2.5710000000000002},"last":1.448}]},{"osd":6,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.431,"15min":1.423},"min":{"1min":1.216,"5min":1.113,"15min":1.0389999999999999},"max":{"1min":1.837,"5min":1.9370000000000001,"15min":1.9610000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.427,"15min":1.4039999999999999},"min":{"1min":1.2110000000000001,"5min":1.0169999999999999,"15min":0.88800000000000001},"max":{"1min":1.6699999999999999,"5min":1.925,"15min":2.2330000000000001},"last":1.4990000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:02:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":1.163,"5min":1.163,"15min":1.1160000000000001},"max":{"1min":1.7769999999999999,"5min":2.0379999999999998,"15min":2.202},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.139,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":1.7090000000000001,"5min":1.8400000000000001,"15min":2.8620000000000001},"last":1.53}]}]},{"osd":2,"up_from":13,"seq":55834575748,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307436,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064404,"statfs":{"total":96636764160,"available":96321949696,"internally_reserved":0,"allocated":2002944,"data_stored":1057252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.94199999999999995,"15min":0.93300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.085,"15min":1.0760000000000001},"min":{"1min":0.81799999999999995,"5min":0.81100000000000005,"15min":0.60799999999999998},"max":{"1min":1.367,"5min":1.4650000000000001,"15min":1.744},"last":0.997}]},{"osd":1,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70599999999999996,"5min":0.66800000000000004,"15min":0.56399999999999995},"max":{"1min":1.3939999999999999,"5min":1.4119999999999999,"15min":1.948},"last":1.093},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.42699999999999999},"max":{"1min":1.538,"5min":1.6200000000000001,"15min":1.6499999999999999},"last":0.85399999999999998}]},{"osd":3,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.628},"max":{"1min":1.1579999999999999,"5min":1.6140000000000001,"15min":1.8009999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.1639999999999999,"15min":1.1539999999999999},"min":{"1min":0.71499999999999997,"5min":0.63600000000000001,"15min":0.57899999999999996},"max":{"1min":1.4199999999999999,"5min":1.49,"15min":2.1760000000000002},"last":1.3009999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.327,"15min":1.3320000000000001},"min":{"1min":1.012,"5min":1.0109999999999999,"15min":0.84899999999999998},"max":{"1min":1.6040000000000001,"5min":1.6719999999999999,"15min":2.0590000000000002},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.325,"15min":1.323},"min":{"1min":0.93200000000000005,"5min":0.84099999999999997,"15min":0.82899999999999996},"max":{"1min":1.7010000000000001,"5min":1.714,"15min":3.1030000000000002},"last":1.1419999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3720000000000001,"15min":1.363},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.571,"5min":1.7270000000000001,"15min":1.9099999999999999},"last":1.262},{"interface":"front","average":{"1min":1.22,"5min":1.286,"15min":1.2949999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.548,"5min":2.0830000000000002,"15min":2.2959999999999998},"last":1.4039999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3440000000000001,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.72599999999999998},"max":{"1min":1.524,"5min":1.9199999999999999,"15min":2.3940000000000001},"last":1.522},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.415,"15min":1.397},"min":{"1min":1.016,"5min":1.016,"15min":0.89900000000000002},"max":{"1min":1.6930000000000001,"5min":1.8520000000000001,"15min":1.869},"last":1.4650000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.0900000000000001,"5min":1.03,"15min":0.60199999999999998},"max":{"1min":1.6839999999999999,"5min":1.8340000000000001,"15min":2.1680000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4259999999999999,"15min":1.4059999999999999},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.82199999999999995},"max":{"1min":1.722,"5min":2.0859999999999999,"15min":2.7109999999999999},"last":1.5900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307432,"kb_used_data":1952,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064408,"statfs":{"total":96636764160,"available":96321953792,"internally_reserved":0,"allocated":1998848,"data_stored":1056734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86099999999999999,"15min":0.85399999999999998},"min":{"1min":0.624,"5min":0.54300000000000004,"15min":0.34799999999999998},"max":{"1min":1.2,"5min":1.54,"15min":1.54},"last":1.107},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.72299999999999998,"5min":0.63700000000000001,"15min":0.42099999999999999},"max":{"1min":1.2849999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.97699999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.93400000000000005,"15min":0.92900000000000005},"min":{"1min":0.78900000000000003,"5min":0.66600000000000004,"15min":0.36499999999999999},"max":{"1min":1.2470000000000001,"5min":1.4359999999999999,"15min":1.7430000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0960000000000001,"15min":1.0800000000000001},"min":{"1min":0.82499999999999996,"5min":0.64800000000000002,"15min":0.503},"max":{"1min":1.454,"5min":1.6240000000000001,"15min":1.6259999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.1870000000000001,"15min":1.1819999999999999},"min":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.55600000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.097,"15min":1.0880000000000001},"min":{"1min":0.90900000000000003,"5min":0.60499999999999998,"15min":0.45100000000000001},"max":{"1min":1.4770000000000001,"5min":1.706,"15min":1.706},"last":1.0760000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.242,"15min":1.2250000000000001},"min":{"1min":1.0009999999999999,"5min":0.82799999999999996,"15min":0.60699999999999998},"max":{"1min":1.444,"5min":1.544,"15min":1.5900000000000001},"last":1.478},{"interface":"front","average":{"1min":1.355,"5min":1.345,"15min":1.331},"min":{"1min":1.0429999999999999,"5min":0.875,"15min":0.83499999999999996},"max":{"1min":1.5960000000000001,"5min":1.958,"15min":2.024},"last":1.181}]},{"osd":5,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3140000000000001,"15min":1.3200000000000001},"min":{"1min":1.0920000000000001,"5min":0.97899999999999998,"15min":0.78400000000000003},"max":{"1min":1.554,"5min":2.0569999999999999,"15min":2.0699999999999998},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":1.0740000000000001,"5min":0.90200000000000002,"15min":0.72199999999999998},"max":{"1min":1.5529999999999999,"5min":1.96,"15min":2.052},"last":1.534}]},{"osd":6,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.387,"15min":1.3899999999999999},"min":{"1min":1.1100000000000001,"5min":0.93799999999999994,"15min":0.92400000000000004},"max":{"1min":1.9750000000000001,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3959999999999999,"15min":1.385},"min":{"1min":1.113,"5min":0.85599999999999998,"15min":0.81499999999999995},"max":{"1min":1.77,"5min":1.881,"15min":2.0800000000000001},"last":1.8779999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:02:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.425,"15min":1.427},"min":{"1min":1.091,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.861,"5min":2.117,"15min":2.117},"last":1.9750000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.401,"15min":1.401},"min":{"1min":1.141,"5min":0.95899999999999996,"15min":0.78800000000000003},"max":{"1min":1.643,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.577}]}]},{"osd":0,"up_from":13,"seq":55834575748,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312108,"kb_used_data":2532,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059732,"statfs":{"total":96636764160,"available":96317165568,"internally_reserved":0,"allocated":2592768,"data_stored":1646077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85099999999999998,"15min":0.84799999999999998},"min":{"1min":0.66100000000000003,"5min":0.58299999999999996,"15min":0.41299999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.512},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.97199999999999998,"15min":0.97299999999999998},"min":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.48599999999999999},"max":{"1min":1.242,"5min":1.464,"15min":1.488},"last":0.98599999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0149999999999999,"15min":0.995},"min":{"1min":0.82799999999999996,"5min":0.66200000000000003,"15min":0.51900000000000002},"max":{"1min":1.3360000000000001,"5min":1.4850000000000001,"15min":1.625},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.079,"15min":1.0660000000000001},"min":{"1min":0.97499999999999998,"5min":0.79600000000000004,"15min":0.498},"max":{"1min":1.286,"5min":1.4299999999999999,"15min":1.6499999999999999},"last":1.234}]},{"osd":3,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.05,"15min":1.024},"min":{"1min":0.79300000000000004,"5min":0.56000000000000005,"15min":0.47399999999999998},"max":{"1min":1.444,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.224,"5min":1.169,"15min":1.1519999999999999},"min":{"1min":0.96199999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.4970000000000001,"5min":1.5649999999999999,"15min":1.698},"last":0.75700000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3839999999999999,"15min":1.383},"min":{"1min":1.2130000000000001,"5min":1.0089999999999999,"15min":0.747},"max":{"1min":1.5549999999999999,"5min":1.752,"15min":1.752},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.371,"15min":1.363},"min":{"1min":1.264,"5min":0.97399999999999998,"15min":0.93799999999999994},"max":{"1min":1.599,"5min":1.792,"15min":1.792},"last":0.93999999999999995}]},{"osd":5,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.411,"15min":1.387},"min":{"1min":1.2050000000000001,"5min":0.93500000000000005,"15min":0.85099999999999998},"max":{"1min":1.694,"5min":1.863,"15min":1.863},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.446,"15min":1.431},"min":{"1min":1.173,"5min":1.137,"15min":0.88200000000000001},"max":{"1min":1.589,"5min":1.7470000000000001,"15min":1.7589999999999999},"last":1.165}]},{"osd":6,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3220000000000001,"15min":1.296},"min":{"1min":1.0569999999999999,"5min":0.96799999999999997,"15min":0.89800000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.79},"last":1.028},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4039999999999999,"15min":1.3799999999999999},"min":{"1min":1.0920000000000001,"5min":1.091,"15min":0.99299999999999999},"max":{"1min":1.659,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.0900000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:01:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.395,"15min":1.3700000000000001},"min":{"1min":1.1279999999999999,"5min":1.032,"15min":0.97699999999999998},"max":{"1min":1.7529999999999999,"5min":1.865,"15min":1.9590000000000001},"last":1.302},{"interface":"front","average":{"1min":1.532,"5min":1.4410000000000001,"15min":1.409},"min":{"1min":1.2010000000000001,"5min":1.0620000000000001,"15min":1.0229999999999999},"max":{"1min":1.931,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.1180000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:02:46.379 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T09:02:46.379 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T09:02:46.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.584+0000 7f2c9da2e700 1 Processor -- start 2023-12-19T09:02:46.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.584+0000 7f2c9da2e700 1 -- start start 2023-12-19T09:02:46.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2c98155b20 0x7f2c980c7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c980c7cc0 0x7f2c9806d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c980c8120 con 0x7f2c98155750 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c980af0d0 con 0x7f2c98155b20 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9806c960 con 0x7f2c980c7cc0 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2c98155b20 0x7f2c980c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2c98155b20 0x7f2c980c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52528/0 (socket says 172.21.15.111:52528) 2023-12-19T09:02:46.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 -- 172.21.15.111:0/1054283611 learned_addr learned my addr 172.21.15.111:0/1054283611 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:46.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 -- 172.21.15.111:0/1054283611 >> v2:172.21.15.143:3300/0 conn(0x7f2c980c7cc0 msgr2=0x7f2c9806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:46.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 --2- 172.21.15.111:0/1054283611 >> v2:172.21.15.143:3300/0 conn(0x7f2c980c7cc0 0x7f2c9806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 -- 172.21.15.111:0/1054283611 >> v1:172.21.15.111:6789/0 conn(0x7f2c98155750 legacy=0x7f2c980ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:46.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 -- 172.21.15.111:0/1054283611 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9806cc40 con 0x7f2c98155b20 2023-12-19T09:02:46.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c967fc700 1 --2- 172.21.15.111:0/1054283611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155b20 0x7f2c980c7780 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2c980ac590 tx=0x7f2c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384d0fc8b55b2827 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c95ffb700 1 -- 172.21.15.111:0/1054283611 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00b820 con 0x7f2c98155b20 2023-12-19T09:02:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c95ffb700 1 -- 172.21.15.111:0/1054283611 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c00b980 con 0x7f2c98155b20 2023-12-19T09:02:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c95ffb700 1 -- 172.21.15.111:0/1054283611 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00bd20 con 0x7f2c98155b20 2023-12-19T09:02:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 -- 172.21.15.111:0/1054283611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155b20 msgr2=0x7f2c980c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.588+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/1054283611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155b20 0x7f2c980c7780 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2c980ac590 tx=0x7f2c8c009250 comp rx=0 tx=0).stop 2023-12-19T09:02:46.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 -- 172.21.15.111:0/1054283611 shutdown_connections 2023-12-19T09:02:46.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/1054283611 >> v2:172.21.15.143:3300/0 conn(0x7f2c980c7cc0 0x7f2c9806d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/1054283611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155b20 0x7f2c980c7780 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 -- 172.21.15.111:0/1054283611 >> 172.21.15.111:0/1054283611 conn(0x7f2c98067bb0 msgr2=0x7f2c980cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:46.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 -- 172.21.15.111:0/1054283611 shutdown_connections 2023-12-19T09:02:46.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 -- 172.21.15.111:0/1054283611 wait complete. 2023-12-19T09:02:46.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 Processor -- start 2023-12-19T09:02:46.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.592+0000 7f2c9da2e700 1 -- start start 2023-12-19T09:02:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980c7cc0 0x7f2c98158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c98155b20 0x7f2c980c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98169e70 con 0x7f2c98155750 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980c9b80 con 0x7f2c980c7cc0 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c9da2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980c9e60 con 0x7f2c98155b20 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52540/0 (socket says 172.21.15.111:52540) 2023-12-19T09:02:46.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 -- 172.21.15.111:0/4286642719 learned_addr learned my addr 172.21.15.111:0/4286642719 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c96ffd700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980c7cc0 0x7f2c98158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c977fe700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c98155b20 0x7f2c980c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 -- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c98155b20 msgr2=0x7f2c980c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c98155b20 0x7f2c980c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 -- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980c7cc0 msgr2=0x7f2c98158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980c7cc0 0x7f2c98158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 -- 172.21.15.111:0/4286642719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0094f0 con 0x7f2c98155750 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c967fc700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c003080 tx=0x7f2c8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c010e70 con 0x7f2c98155750 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.596+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c018420 con 0x7f2c98155750 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.600+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c980ca140 con 0x7f2c98155750 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.600+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c018800 con 0x7f2c98155750 2023-12-19T09:02:46.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.600+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98158eb0 con 0x7f2c98155750 2023-12-19T09:02:46.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.600+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c980ca640 con 0x7f2c98155750 2023-12-19T09:02:46.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.600+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f2c8c018bb0 con 0x7f2c98155750 2023-12-19T09:02:46.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.604+0000 7f2c7effd700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 0x7f2c84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:46.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.604+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f2c8c014070 con 0x7f2c98155750 2023-12-19T09:02:46.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.604+0000 7f2c96ffd700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 0x7f2c84062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:46.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.604+0000 7f2c96ffd700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 0x7f2c84062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c980a61e0 tx=0x7f2c88007ea0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:46.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.608+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c8c04dcf0 con 0x7f2c98155750 2023-12-19T09:02:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.836+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2c980c8120 con 0x7f2c98155750 2023-12-19T09:02:46.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.848+0000 7f2c7effd700 1 -- 172.21.15.111:0/4286642719 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v40) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2c980c8120 con 0x7f2c98155750 2023-12-19T09:02:46.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 msgr2=0x7f2c84062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 0x7f2c84062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c980a61e0 tx=0x7f2c88007ea0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 msgr2=0x7f2c980c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c003080 tx=0x7f2c8c018000 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 shutdown_connections 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f2c84060670 0x7f2c84062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c98155b20 0x7f2c980c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c98155750 0x7f2c980c4d50 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 --2- 172.21.15.111:0/4286642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980c7cc0 0x7f2c98158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 >> 172.21.15.111:0/4286642719 conn(0x7f2c98067bb0 msgr2=0x7f2c980b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 shutdown_connections 2023-12-19T09:02:46.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:46.852+0000 7f2c9da2e700 1 -- 172.21.15.111:0/4286642719 wait complete. 2023-12-19T09:02:46.858 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T09:02:46.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:47.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.040+0000 7fd560923700 1 Processor -- start 2023-12-19T09:02:47.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.040+0000 7fd560923700 1 -- start start 2023-12-19T09:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd560923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c158e20 0x7fd55c06a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd560923700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd55c1591f0 0x7fd55c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd560923700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c06b730 con 0x7fd55c15ca80 2023-12-19T09:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd560923700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c06ab20 con 0x7fd55c1591f0 2023-12-19T09:02:47.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd560923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c06ae00 con 0x7fd55c158e20 2023-12-19T09:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd55c1591f0 0x7fd55c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd55c1591f0 0x7fd55c0ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52556/0 (socket says 172.21.15.111:52556) 2023-12-19T09:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 -- 172.21.15.111:0/2568315091 learned_addr learned my addr 172.21.15.111:0/2568315091 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd55b7fe700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c158e20 0x7fd55c06a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 -- 172.21.15.111:0/2568315091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c158e20 msgr2=0x7fd55c06a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c158e20 0x7fd55c06a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 -- 172.21.15.111:0/2568315091 >> v1:172.21.15.143:6789/0 conn(0x7fd55c15ca80 legacy=0x7fd55c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 -- 172.21.15.111:0/2568315091 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c06b0e0 con 0x7fd55c1591f0 2023-12-19T09:02:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd55affd700 1 -- 172.21.15.111:0/2568315091 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524481358 0 0) 0x7fd55c06b730 con 0x7fd55c15ca80 2023-12-19T09:02:47.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd552ffd700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c1591f0 0x7fd55c0ce100 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd5440099b0 tx=0x7fd544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf13a534e93f5fa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd55affd700 1 -- 172.21.15.111:0/2568315091 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544009ea0 con 0x7fd55c1591f0 2023-12-19T09:02:47.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd55affd700 1 -- 172.21.15.111:0/2568315091 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400b860 con 0x7fd55c1591f0 2023-12-19T09:02:47.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.044+0000 7fd55affd700 1 -- 172.21.15.111:0/2568315091 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400bc00 con 0x7fd55c1591f0 2023-12-19T09:02:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 -- 172.21.15.111:0/2568315091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c1591f0 msgr2=0x7fd55c0ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c1591f0 0x7fd55c0ce100 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd5440099b0 tx=0x7fd544009250 comp rx=0 tx=0).stop 2023-12-19T09:02:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 -- 172.21.15.111:0/2568315091 shutdown_connections 2023-12-19T09:02:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c1591f0 0x7fd55c0ce100 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 --2- 172.21.15.111:0/2568315091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c158e20 0x7fd55c06a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 -- 172.21.15.111:0/2568315091 >> 172.21.15.111:0/2568315091 conn(0x7fd55c067bb0 msgr2=0x7fd55c0cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 -- 172.21.15.111:0/2568315091 shutdown_connections 2023-12-19T09:02:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.048+0000 7fd560923700 1 -- 172.21.15.111:0/2568315091 wait complete. 2023-12-19T09:02:47.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 Processor -- start 2023-12-19T09:02:47.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 -- start start 2023-12-19T09:02:47.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c1591f0 0x7fd55c170a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c15ca80 0x7fd55c0cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0c7be0 con 0x7fd55c158e20 2023-12-19T09:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0c7010 con 0x7fd55c15ca80 2023-12-19T09:02:47.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd560923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0c72b0 con 0x7fd55c1591f0 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52572/0 (socket says 172.21.15.111:52572) 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 -- 172.21.15.111:0/643792298 learned_addr learned my addr 172.21.15.111:0/643792298 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55bfff700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c15ca80 0x7fd55c0cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd552ffd700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c1591f0 0x7fd55c170a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 -- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c1591f0 msgr2=0x7fd55c170a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c1591f0 0x7fd55c170a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 -- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c15ca80 msgr2=0x7fd55c0cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c15ca80 0x7fd55c0cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 -- 172.21.15.111:0/643792298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5440094f0 con 0x7fd55c158e20 2023-12-19T09:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.052+0000 7fd55b7fe700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00ec80 tx=0x7fd54c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c00c850 con 0x7fd55c158e20 2023-12-19T09:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c00c9b0 con 0x7fd55c158e20 2023-12-19T09:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c0c7590 con 0x7fd55c158e20 2023-12-19T09:02:47.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c00cd50 con 0x7fd55c158e20 2023-12-19T09:02:47.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c170f40 con 0x7fd55c158e20 2023-12-19T09:02:47.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c04be40 con 0x7fd55c158e20 2023-12-19T09:02:47.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.056+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd54c013680 con 0x7fd55c158e20 2023-12-19T09:02:47.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.060+0000 7fd553fff700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 0x7fd53c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.060+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fd54c07cf90 con 0x7fd55c158e20 2023-12-19T09:02:47.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.060+0000 7fd552ffd700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 0x7fd53c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.064+0000 7fd552ffd700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 0x7fd53c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd5440099b0 tx=0x7fd544017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.064+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd54c04d070 con 0x7fd55c158e20 2023-12-19T09:02:47.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.272+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd55c06b3c0 con 0x7fd55c158e20 2023-12-19T09:02:47.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.272+0000 7fd553fff700 1 -- 172.21.15.111:0/643792298 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7fd54c050690 con 0x7fd55c158e20 2023-12-19T09:02:47.278 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:47.278 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 msgr2=0x7fd53c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 0x7fd53c062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd5440099b0 tx=0x7fd544017000 comp rx=0 tx=0).stop 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 msgr2=0x7fd55c16dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00ec80 tx=0x7fd54c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 shutdown_connections 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd53c060670 0x7fd53c062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd55c15ca80 0x7fd55c0cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd55c1591f0 0x7fd55c170a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 --2- 172.21.15.111:0/643792298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd55c158e20 0x7fd55c16dc10 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 >> 172.21.15.111:0/643792298 conn(0x7fd55c067bb0 msgr2=0x7fd55c153b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 shutdown_connections 2023-12-19T09:02:47.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.276+0000 7fd560923700 1 -- 172.21.15.111:0/643792298 wait complete. 2023-12-19T09:02:47.284 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:02:47.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T09:02:41.093309+0000', 'modified': '2023-12-19T09:02:44.691422+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8910': {'gid': 8910, 'name': 'a', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6834/4167714642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6832', 'nonce': 4167714642}, {'type': 'v1', 'addr': '172.21.15.111:6834', 'nonce': 4167714642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T09:02:47.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T09:02:47.704 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T09:02:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.848+0000 7f872106b700 1 Processor -- start 2023-12-19T09:02:47.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.848+0000 7f872106b700 1 -- start start 2023-12-19T09:02:47.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f872106b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c155960 0x7f871c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f872106b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f871c06de20 0x7f871c0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f872106b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c06d990 con 0x7f871c06c7c0 2023-12-19T09:02:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f872106b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c15a570 con 0x7f871c06de20 2023-12-19T09:02:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f872106b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c06cd80 con 0x7f871c155960 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f8713fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f871c06de20 0x7f871c0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f8713fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f871c06de20 0x7f871c0c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49428/0 (socket says 172.21.15.111:49428) 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f8713fff700 1 -- 172.21.15.111:0/1157763078 learned_addr learned my addr 172.21.15.111:0/1157763078 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.852+0000 7f871bfff700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c155960 0x7f871c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f8713fff700 1 -- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c155960 msgr2=0x7f871c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f8713fff700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c155960 0x7f871c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f8713fff700 1 -- 172.21.15.111:0/1157763078 >> v1:172.21.15.143:6789/0 conn(0x7f871c06c7c0 legacy=0x7f871c0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f8713fff700 1 -- 172.21.15.111:0/1157763078 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c06d060 con 0x7f871c06de20 2023-12-19T09:02:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f8713fff700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c06de20 0x7f871c0c77f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f87040099b0 tx=0x7f8704009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c517aa1394e27ecb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f871b7fe700 1 -- 172.21.15.111:0/1157763078 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704009ea0 con 0x7f871c06de20 2023-12-19T09:02:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f871b7fe700 1 -- 172.21.15.111:0/1157763078 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870400b860 con 0x7f871c06de20 2023-12-19T09:02:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f871b7fe700 1 -- 172.21.15.111:0/1157763078 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870400bc00 con 0x7f871c06de20 2023-12-19T09:02:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c06de20 msgr2=0x7f871c0c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c06de20 0x7f871c0c77f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f87040099b0 tx=0x7f8704009250 comp rx=0 tx=0).stop 2023-12-19T09:02:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- 172.21.15.111:0/1157763078 shutdown_connections 2023-12-19T09:02:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c06de20 0x7f871c0c77f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 --2- 172.21.15.111:0/1157763078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c155960 0x7f871c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- 172.21.15.111:0/1157763078 >> 172.21.15.111:0/1157763078 conn(0x7f871c067bb0 msgr2=0x7f871c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- 172.21.15.111:0/1157763078 shutdown_connections 2023-12-19T09:02:47.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- 172.21.15.111:0/1157763078 wait complete. 2023-12-19T09:02:47.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 Processor -- start 2023-12-19T09:02:47.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.856+0000 7f872106b700 1 -- start start 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c06c7c0 0x7f871c14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c155960 0x7f871c149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c15aea0 con 0x7f871c06de20 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c14c5c0 con 0x7f871c155960 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c15a850 con 0x7f871c06c7c0 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52588/0 (socket says 172.21.15.111:52588) 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 -- 172.21.15.111:0/1717292807 learned_addr learned my addr 172.21.15.111:0/1717292807 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f871bfff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c06c7c0 0x7f871c14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872086a700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c155960 0x7f871c149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 -- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c06c7c0 msgr2=0x7f871c14ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c06c7c0 0x7f871c14ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 -- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c155960 msgr2=0x7f871c149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c155960 0x7f871c149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 -- 172.21.15.111:0/1717292807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040094f0 con 0x7f871c06de20 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8713fff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f871c064040 tx=0x7f8704017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8718ff9700 1 -- 172.21.15.111:0/1717292807 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870400bbe0 con 0x7f871c06de20 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8718ff9700 1 -- 172.21.15.111:0/1717292807 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704011420 con 0x7f871c06de20 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c15ab30 con 0x7f871c06de20 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f8718ff9700 1 -- 172.21.15.111:0/1717292807 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040117c0 con 0x7f871c06de20 2023-12-19T09:02:47.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.860+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c14c200 con 0x7f871c06de20 2023-12-19T09:02:47.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.864+0000 7f8718ff9700 1 -- 172.21.15.111:0/1717292807 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f8704011bf0 con 0x7f871c06de20 2023-12-19T09:02:47.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.864+0000 7f8718ff9700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 0x7f86fc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8718ff9700 1 -- 172.21.15.111:0/1717292807 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f870407e690 con 0x7f871c06de20 2023-12-19T09:02:47.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 0x7f86fc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 0x7f86fc062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d40 tx=0x7f870c008040 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 Processor -- start 2023-12-19T09:02:47.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 -- start start 2023-12-19T09:02:47.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1de6710 0x2064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e31790 0x2064980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f91f30 con 0x1de6ae0 2023-12-19T09:02:47.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f8ded0 con 0x1e31790 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f540b0 con 0x1de6710 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52598/0 (socket says 172.21.15.111:52598) 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 -- 172.21.15.111:0/3868480352 learned_addr learned my addr 172.21.15.111:0/3868480352 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f8713fff700 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1de6710 0x2064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 -- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1de6710 msgr2=0x2064440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1de6710 0x2064440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 -- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e31790 msgr2=0x2064980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e31790 0x2064980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 -- 172.21.15.111:0/3868480352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1e329e0 con 0x1de6ae0 2023-12-19T09:02:47.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.868+0000 7f871bfff700 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f870c009620 tx=0x7f870c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa35f250cec3c54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f87117fa700 1 -- 172.21.15.111:0/3868480352 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c011e60 con 0x1de6ae0 2023-12-19T09:02:47.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f87117fa700 1 -- 172.21.15.111:0/3868480352 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c015c60 con 0x1de6ae0 2023-12-19T09:02:47.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f87117fa700 1 -- 172.21.15.111:0/3868480352 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c021650 con 0x1de6ae0 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- 172.21.15.111:0/3868480352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 msgr2=0x2063f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f870c009620 tx=0x7f870c018000 comp rx=0 tx=0).stop 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- 172.21.15.111:0/3868480352 shutdown_connections 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1e31790 0x2064980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1de6710 0x2064440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- 172.21.15.111:0/3868480352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063f00 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- 172.21.15.111:0/3868480352 >> 172.21.15.111:0/3868480352 conn(0x1de6eb0 msgr2=0x2020930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- 172.21.15.111:0/3868480352 shutdown_connections 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- 172.21.15.111:0/3868480352 wait complete. 2023-12-19T09:02:47.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- start start 2023-12-19T09:02:47.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 Processor -- start 2023-12-19T09:02:47.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6ae0 0x2068ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x2049cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f53d50 con 0x1de6710 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f2c970 con 0x1de6ae0 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f56420 con 0x1e31790 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f8724584740 10 client.0 _ll_register_callbacks cb 0x1fd76a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f871bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f871bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52614/0 (socket says 172.21.15.111:52614) 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f871bfff700 1 -- 172.21.15.111:0/3149192449 learned_addr learned my addr 172.21.15.111:0/3149192449 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.872+0000 7f872086a700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x2049cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8713fff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6ae0 0x2068ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 -- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 msgr2=0x2049cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x2049cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 -- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6ae0 msgr2=0x2068ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6ae0 0x2068ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 -- 172.21.15.111:0/3149192449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c011a10 con 0x1de6710 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f871bfff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f870c008010 tx=0x7f870c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7efd812c6fa167 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c021e30 con 0x1de6710 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1f2ccc0 con 0x1de6710 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f870c02a420 con 0x1de6710 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f870c02a7f0 con 0x1de6710 2023-12-19T09:02:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 10 client.9036 fetch_fsmap learned FSMap version 72 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x1f5c370 con 0x1de6710 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1f2ccc0 con 0x1de6710 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 72) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f870c02a950 con 0x1de6710 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 10 client.9036 fetch_fsmap finished waiting for FSMap version 72 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 10 client.9036 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 20 client.9036 populate_metadata read hostname 'smithi111' 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x204b380 0x20736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 4 client.9036 mds_command: new command op to 8910 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T09:02:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 --> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1f303e0 con 0x204b380 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8713fff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x204b380 0x20736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f8713fff700 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x204b380 0x20736b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46942701b6ca59ed server_cookie=ec943d3c3e32674 in_seq=0 out_seq=0 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.876+0000 7f86fa7fc700 10 client.9036 ms_handle_connect on v2:172.21.15.111:6832/4167714642 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3149192449 <== mds.0 v2:172.21.15.111:6832/4167714642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x1f303e0 con 0x204b380 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f86fa7fc700 10 client.9036 handle_command_reply: tid=0 2023-12-19T09:02:47.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 client.9036 shutdown 2023-12-19T09:02:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 msgr2=0x2068590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f870c008010 tx=0x7f870c01a000 comp rx=0 tx=0).stop 2023-12-19T09:02:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 shutdown_connections 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x204b380 0x20736b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6ae0 0x2068ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x2049cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 --2- 172.21.15.111:0/3149192449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6710 0x2068590 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 >> 172.21.15.111:0/3149192449 conn(0x1de6eb0 msgr2=0x2018f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f86fa7fc700 0 client.9036 ms_handle_reset on v2:172.21.15.111:6832/4167714642 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 shutdown_connections 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 1 -- 172.21.15.111:0/3149192449 wait complete. 2023-12-19T09:02:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.880+0000 7f8724584740 20 client.9036 trim_cache size 0 max 16384 2023-12-19T09:02:47.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 Processor -- start 2023-12-19T09:02:47.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 -- start start 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2063440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f8b890 con 0x1de6ae0 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f61040 con 0x1de6710 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f8cd60 con 0x1e31790 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52620/0 (socket says 172.21.15.111:52620) 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 -- 172.21.15.111:0/3014257338 learned_addr learned my addr 172.21.15.111:0/3014257338 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f872086a700 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 -- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 msgr2=0x201aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 -- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 msgr2=0x2063440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2063440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 -- 172.21.15.111:0/3014257338 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1f42d80 con 0x1de6ae0 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f872086a700 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:47.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8713fff700 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8704009290 tx=0x7f870407e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2880a5d80a0e2548 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8711ffb700 1 -- 172.21.15.111:0/3014257338 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704011ea0 con 0x1de6ae0 2023-12-19T09:02:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8711ffb700 1 -- 172.21.15.111:0/3014257338 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704052430 con 0x1de6ae0 2023-12-19T09:02:47.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.900+0000 7f8711ffb700 1 -- 172.21.15.111:0/3014257338 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040507d0 con 0x1de6ae0 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- 172.21.15.111:0/3014257338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 msgr2=0x2063980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8704009290 tx=0x7f870407e040 comp rx=0 tx=0).stop 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- 172.21.15.111:0/3014257338 shutdown_connections 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x2063980 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- 172.21.15.111:0/3014257338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2063440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- 172.21.15.111:0/3014257338 >> 172.21.15.111:0/3014257338 conn(0x1de6eb0 msgr2=0x2049570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- 172.21.15.111:0/3014257338 shutdown_connections 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- 172.21.15.111:0/3014257338 wait complete. 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- start start 2023-12-19T09:02:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 Processor -- start 2023-12-19T09:02:47.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2019cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x201d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f90590 con 0x1de6ae0 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f919d0 con 0x1de6710 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e5f7c0 con 0x1e31790 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8724584740 10 client.0 _ll_register_callbacks cb 0x1db29d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x201d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8713fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x201d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52624/0 (socket says 172.21.15.111:52624) 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f8713fff700 1 -- 172.21.15.111:0/3225906880 learned_addr learned my addr 172.21.15.111:0/3225906880 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f872086a700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.904+0000 7f871bfff700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2019cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 -- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 msgr2=0x2019cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2019cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 -- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 msgr2=0x201d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:47.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x201d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 -- 172.21.15.111:0/3225906880 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c14c200 con 0x1e31790 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f872086a700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201da00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8714007fd0 tx=0x7f8714009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e0278993097f88 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.143:3301/0 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871401b030 con 0x1e31790 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mon.2 v2:172.21.15.143:3301/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f871401c070 con 0x1e31790 2023-12-19T09:02:47.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f870c015c60 con 0x1e31790 2023-12-19T09:02:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8714019d00 con 0x1e31790 2023-12-19T09:02:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 10 client.7760 fetch_fsmap learned FSMap version 72 2023-12-19T09:02:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x1f37130 con 0x1e31790 2023-12-19T09:02:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mon.2 v2:172.21.15.143:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c015c60 con 0x1e31790 2023-12-19T09:02:47.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mon.2 v2:172.21.15.143:3301/0 5 ==== fsmap(e 72) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f8714015560 con 0x1e31790 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 10 client.7760 fetch_fsmap finished waiting for FSMap version 72 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 10 client.7760 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 20 client.7760 populate_metadata read hostname 'smithi111' 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x201df40 0x20bd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 4 client.7760 mds_command: new command op to 8910 tid=0[{"prefix": "status"}] 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 --> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1f62410 con 0x201df40 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f871bfff700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x201df40 0x20bd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 10 client.7760 ms_handle_connect on v2:172.21.15.111:6832/4167714642 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f871bfff700 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x201df40 0x20bd320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d5997e734a37a4e server_cookie=b525a7ea21d20ea4 in_seq=0 out_seq=0 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 1 -- 172.21.15.111:0/3225906880 <== mds.0 v2:172.21.15.111:6832/4167714642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x1f62410 con 0x201df40 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f86fa7fc700 10 client.7760 handle_command_reply: tid=0 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.908+0000 7f8724584740 1 client.7760 shutdown 2023-12-19T09:02:47.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 msgr2=0x201da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201da00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8714007fd0 tx=0x7f8714009040 comp rx=0 tx=0).stop 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 shutdown_connections 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:6832/4167714642,v1:172.21.15.111:6834/4167714642] conn(0x201df40 0x20bd320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x1e31790 0x201da00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1de6ae0 0x201d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 --2- 172.21.15.111:0/3225906880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x1de6710 0x2019cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f86fa7fc700 0 client.7760 ms_handle_reset on v2:172.21.15.111:6832/4167714642 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 >> 172.21.15.111:0/3225906880 conn(0x1de6eb0 msgr2=0x2047780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 shutdown_connections 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 1 -- 172.21.15.111:0/3225906880 wait complete. 2023-12-19T09:02:47.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f8724584740 20 client.7760 trim_cache size 0 max 16384 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 8910, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2468118930000003, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 72, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 161, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 161, 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 11.268996717 2023-12-19T09:02:47.919 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 msgr2=0x7f86fc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 0x7f86fc062b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d40 tx=0x7f870c008040 comp rx=0 tx=0).stop 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 msgr2=0x7f871c14bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f871c064040 tx=0x7f8704017000 comp rx=0 tx=0).stop 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 shutdown_connections 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86fc0606a0 0x7f86fc062b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f871c155960 0x7f871c149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f871c06de20 0x7f871c14bcc0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 --2- 172.21.15.111:0/1717292807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f871c06c7c0 0x7f871c14ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 >> 172.21.15.111:0/1717292807 conn(0x7f871c067bb0 msgr2=0x7f871c15ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 shutdown_connections 2023-12-19T09:02:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:47.912+0000 7f872106b700 1 -- 172.21.15.111:0/1717292807 wait complete. 2023-12-19T09:02:47.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T09:02:48.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 Processor -- start 2023-12-19T09:02:48.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 -- start start 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f78cc155960 0x7f78cc15a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06de20 0x7f78cc0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78cc06d990 con 0x7f78cc06c7c0 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78cc15a570 con 0x7f78cc155960 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d2c6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc06cd80 con 0x7f78cc06de20 2023-12-19T09:02:48.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d246a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f78cc06c7c0 0x7f78cc0c5fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46944/0 (socket says 172.21.15.111:46944) 2023-12-19T09:02:48.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d246a700 1 -- 172.21.15.111:0/4226213582 learned_addr learned my addr 172.21.15.111:0/4226213582 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:48.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d1c69700 1 --2- 172.21.15.111:0/4226213582 >> v2:172.21.15.111:3300/0 conn(0x7f78cc155960 0x7f78cc15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d1468700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06de20 0x7f78cc0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.072+0000 7f78d0c67700 1 -- 172.21.15.111:0/4226213582 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188731377 0 0) 0x7f78cc06d990 con 0x7f78cc06c7c0 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d0c67700 1 -- 172.21.15.111:0/4226213582 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b8003570 con 0x7f78cc06c7c0 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d1c69700 1 -- 172.21.15.111:0/4226213582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06de20 msgr2=0x7f78cc0c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d1c69700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06de20 0x7f78cc0c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d1c69700 1 -- 172.21.15.111:0/4226213582 >> v1:172.21.15.111:6789/0 conn(0x7f78cc06c7c0 legacy=0x7f78cc0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d1c69700 1 -- 172.21.15.111:0/4226213582 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc06d060 con 0x7f78cc155960 2023-12-19T09:02:48.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d1c69700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a030 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f78c80098e0 tx=0x7f78c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a1af8a3393018a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:48.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d0c67700 1 -- 172.21.15.111:0/4226213582 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c800b640 con 0x7f78cc155960 2023-12-19T09:02:48.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d0c67700 1 -- 172.21.15.111:0/4226213582 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800b7a0 con 0x7f78cc155960 2023-12-19T09:02:48.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d0c67700 1 -- 172.21.15.111:0/4226213582 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c800bb40 con 0x7f78cc155960 2023-12-19T09:02:48.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 -- 172.21.15.111:0/4226213582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 msgr2=0x7f78cc15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a030 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f78c80098e0 tx=0x7f78c80091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 -- 172.21.15.111:0/4226213582 shutdown_connections 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06de20 0x7f78cc0c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/4226213582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a030 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 -- 172.21.15.111:0/4226213582 >> 172.21.15.111:0/4226213582 conn(0x7f78cc067bb0 msgr2=0x7f78cc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 -- 172.21.15.111:0/4226213582 shutdown_connections 2023-12-19T09:02:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 -- 172.21.15.111:0/4226213582 wait complete. 2023-12-19T09:02:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.076+0000 7f78d2c6b700 1 Processor -- start 2023-12-19T09:02:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- start start 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06c7c0 0x7f78cc16d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78cc06de20 0x7f78cc15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc16cd20 con 0x7f78cc155960 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc15d820 con 0x7f78cc06de20 2023-12-19T09:02:48.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc15cfc0 con 0x7f78cc06c7c0 2023-12-19T09:02:48.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52636/0 (socket says 172.21.15.111:52636) 2023-12-19T09:02:48.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 -- 172.21.15.111:0/2201856890 learned_addr learned my addr 172.21.15.111:0/2201856890 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:48.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d1c69700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06c7c0 0x7f78cc16d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d1468700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78cc06de20 0x7f78cc15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 -- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06c7c0 msgr2=0x7f78cc16d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06c7c0 0x7f78cc16d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 -- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78cc06de20 msgr2=0x7f78cc15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78cc06de20 0x7f78cc15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 -- 172.21.15.111:0/2201856890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009420 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d246a700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f78c4001460 tx=0x7f78c4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400a8a0 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c400aa00 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400ad80 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc15d260 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.080+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc16c940 con 0x7f78cc155960 2023-12-19T09:02:48.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.084+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc16ce60 con 0x7f78cc155960 2023-12-19T09:02:48.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.084+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f78c4007300 con 0x7f78cc155960 2023-12-19T09:02:48.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.084+0000 7f78c1ffb700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 0x7f78b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:48.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.084+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f78c407d000 con 0x7f78cc155960 2023-12-19T09:02:48.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.084+0000 7f78d1c69700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 0x7f78b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:48.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.088+0000 7f78d1c69700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 0x7f78b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78c80093f0 tx=0x7f78c8017000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:48.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.092+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f78cc16ce60 con 0x7f78cc155960 2023-12-19T09:02:48.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.296+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f78cc16ce60 con 0x7f78cc155960 2023-12-19T09:02:48.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.296+0000 7f78c1ffb700 1 -- 172.21.15.111:0/2201856890 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) v1 ==== 53+0+73 (secure 0 0 0) 0x7f78cc16ce60 con 0x7f78cc155960 2023-12-19T09:02:48.302 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T09:02:48.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 msgr2=0x7f78b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 0x7f78b0062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f78c80093f0 tx=0x7f78c8017000 comp rx=0 tx=0).stop 2023-12-19T09:02:48.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 msgr2=0x7f78cc15a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f78c4001460 tx=0x7f78c4001a40 comp rx=0 tx=0).stop 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 shutdown_connections 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78b0060670 0x7f78b0062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78cc06de20 0x7f78cc15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78cc155960 0x7f78cc15a6e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 --2- 172.21.15.111:0/2201856890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78cc06c7c0 0x7f78cc16d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 >> 172.21.15.111:0/2201856890 conn(0x7f78cc067bb0 msgr2=0x7f78cc0c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 shutdown_connections 2023-12-19T09:02:48.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:48.300+0000 7f78d2c6b700 1 -- 172.21.15.111:0/2201856890 wait complete. 2023-12-19T09:02:48.325 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T09:02:48.325 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T09:02:48.325 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T09:02:48.326 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:02:48.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.326 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T09:02:48.346 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T09:02:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T09:02:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T09:02:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 37889sec preferred_lft 37889sec 2023-12-19T09:02:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T09:02:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.349 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:02:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:02:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T09:02:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.349 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:02:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:02:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T09:02:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:02:48.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.350 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T09:02:48.429 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T09:02:48.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.432 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T09:02:48.501 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T09:02:48.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.502 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T09:02:48.503 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T09:02:48.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T09:02:48.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T09:02:48.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T09:02:48.503 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T09:02:48.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.634 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:02:48.650 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:02:48.650 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:02:48.651 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:02:48.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:48.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T09:02:48.729 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T09:02:48.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:02:48.729 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:02:48.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:02:48.793 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:48.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:02:48.942 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:02:48.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:02:49.018 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:02:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:02:49.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:49.023 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:02:49.086 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T09:02:49.086 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T09:02:49.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:02:49.169 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:02:49.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:02:49.245 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:02:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:02:49.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:49.249 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:02:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T09:02:49.263+0000 7f3c7c8f8380 -1 init, newargv = 0x55c2e9089db0 newargc=15 2023-12-19T09:02:49.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[57873]: starting ceph client 2023-12-19T09:02:49.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[57873]: starting fuse 2023-12-19T09:02:49.283 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T09:02:49.283 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T09:02:49.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:02:49.284 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T09:02:49.285 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T09:02:49.452 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.57873.asok 2023-12-19T09:02:49.635 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57873.asok 2023-12-19T09:02:49.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:49.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57873.asok status 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T09:02:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "57873", 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 9060, 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 9060 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:02:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1553868504 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1553868504 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.9060 192.168.0.1:0/1553868504", 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/1553868504", 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 72, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 161, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T09:02:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T09:02:49.815 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:02:49.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:49.825 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T09:02:49.825 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:49.849 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T09:02:49.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:49.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:49.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:49.930 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T09:02:50.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.135+0000 7fca40945700 1 Processor -- start 2023-12-19T09:02:50.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.135+0000 7fca40945700 1 -- start start 2023-12-19T09:02:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca40945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 0x7fca3c06b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca40945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0af8c0 con 0x7fca3c155450 2023-12-19T09:02:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca40945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c159ad0 con 0x7fca3c0abb20 2023-12-19T09:02:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca40945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c159db0 con 0x7fca3c155820 2023-12-19T09:02:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3a59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fca3c155450 0x7fca3c159280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46956/0 (socket says 172.21.15.111:46956) 2023-12-19T09:02:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3a59c700 1 -- 172.21.15.111:0/2487934870 learned_addr learned my addr 172.21.15.111:0/2487934870 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3ad9d700 1 --2- 172.21.15.111:0/2487934870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 0x7fca3c06b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3959a700 1 -- 172.21.15.111:0/2487934870 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865749275 0 0) 0x7fca3c0af8c0 con 0x7fca3c155450 2023-12-19T09:02:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3959a700 1 -- 172.21.15.111:0/2487934870 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca24003570 con 0x7fca3c155450 2023-12-19T09:02:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3ad9d700 1 -- 172.21.15.111:0/2487934870 >> v1:172.21.15.143:6789/0 conn(0x7fca3c0abb20 legacy=0x7fca3c0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3ad9d700 1 -- 172.21.15.111:0/2487934870 >> v1:172.21.15.111:6789/0 conn(0x7fca3c155450 legacy=0x7fca3c159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3ad9d700 1 -- 172.21.15.111:0/2487934870 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c15a090 con 0x7fca3c155820 2023-12-19T09:02:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.139+0000 7fca3ad9d700 1 --2- 172.21.15.111:0/2487934870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 0x7fca3c06b310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca34009e90 tx=0x7fca34009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0ff4b4dad7c8ff server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca3959a700 1 -- 172.21.15.111:0/2487934870 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3400e070 con 0x7fca3c155820 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca3959a700 1 -- 172.21.15.111:0/2487934870 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3400bc30 con 0x7fca3c155820 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca3959a700 1 -- 172.21.15.111:0/2487934870 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34012430 con 0x7fca3c155820 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- 172.21.15.111:0/2487934870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 msgr2=0x7fca3c06b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 --2- 172.21.15.111:0/2487934870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 0x7fca3c06b310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca34009e90 tx=0x7fca34009780 comp rx=0 tx=0).stop 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- 172.21.15.111:0/2487934870 shutdown_connections 2023-12-19T09:02:50.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 --2- 172.21.15.111:0/2487934870 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155820 0x7fca3c06b310 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- 172.21.15.111:0/2487934870 >> 172.21.15.111:0/2487934870 conn(0x7fca3c067540 msgr2=0x7fca3c067910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:50.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- 172.21.15.111:0/2487934870 shutdown_connections 2023-12-19T09:02:50.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- 172.21.15.111:0/2487934870 wait complete. 2023-12-19T09:02:50.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 Processor -- start 2023-12-19T09:02:50.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.143+0000 7fca40945700 1 -- start start 2023-12-19T09:02:50.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155450 0x7fca3c0a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca3c155820 0x7fca3c0aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0c2fc0 con 0x7fca3c155820 2023-12-19T09:02:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0cc110 con 0x7fca3c0abb20 2023-12-19T09:02:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0c3450 con 0x7fca3c155450 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca3c155820 0x7fca3c0aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca39d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155450 0x7fca3c0a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49488/0 (socket says 172.21.15.111:49488) 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 -- 172.21.15.111:0/4194041269 learned_addr learned my addr 172.21.15.111:0/4194041269 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 -- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155450 msgr2=0x7fca3c0a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155450 0x7fca3c0a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 -- 172.21.15.111:0/4194041269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca3c155820 msgr2=0x7fca3c0aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca3c155820 0x7fca3c0aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 -- 172.21.15.111:0/4194041269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34009a20 con 0x7fca3c0abb20 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca3a59c700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c7c0 tx=0x7fca2c007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c005cd0 con 0x7fca3c0abb20 2023-12-19T09:02:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c005e30 con 0x7fca3c0abb20 2023-12-19T09:02:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c0c23b0 con 0x7fca3c0abb20 2023-12-19T09:02:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c007320 con 0x7fca3c0abb20 2023-12-19T09:02:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.147+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c0c2970 con 0x7fca3c0abb20 2023-12-19T09:02:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.151+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c071120 con 0x7fca3c0abb20 2023-12-19T09:02:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.151+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fca2c01b3b0 con 0x7fca3c0abb20 2023-12-19T09:02:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.151+0000 7fca2a7fc700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 0x7fca1c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.151+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fca2c07c2e0 con 0x7fca3c0abb20 2023-12-19T09:02:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.151+0000 7fca39d9b700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 0x7fca1c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.155+0000 7fca39d9b700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 0x7fca1c062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca3c1494c0 tx=0x7fca30007ec0 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.163+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca2c04c460 con 0x7fca3c0abb20 2023-12-19T09:02:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.363+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fca3c06c240 con 0x7fca1c060670 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca2a7fc700 1 -- 172.21.15.111:0/4194041269 <== mgr.7493 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fca3c06c240 con 0x7fca1c060670 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:02:50.372 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 msgr2=0x7fca1c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 0x7fca1c062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca3c1494c0 tx=0x7fca30007ec0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 msgr2=0x7fca3c145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c7c0 tx=0x7fca2c007e00 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 shutdown_connections 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fca1c060670 0x7fca1c062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca3c155820 0x7fca3c0aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca3c155450 0x7fca3c0a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 --2- 172.21.15.111:0/4194041269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca3c0abb20 0x7fca3c145970 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 >> 172.21.15.111:0/4194041269 conn(0x7fca3c067540 msgr2=0x7fca3c06b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 shutdown_connections 2023-12-19T09:02:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.367+0000 7fca40945700 1 -- 172.21.15.111:0/4194041269 wait complete. 2023-12-19T09:02:50.396 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T09:02:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 Processor -- start 2023-12-19T09:02:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 -- start start 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08a00caa10 0x7f08a015d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08a00cc040 0x7f08a0158900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a015d960 con 0x7f08a00caa10 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00cbbb0 con 0x7f08a00cc040 2023-12-19T09:02:50.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.543+0000 7f08a4a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a006dc90 con 0x7f08a00ce2d0 2023-12-19T09:02:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08a00caa10 0x7f08a015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08a00caa10 0x7f08a015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52642/0 (socket says 172.21.15.111:52642) 2023-12-19T09:02:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 -- 172.21.15.111:0/2656151958 learned_addr learned my addr 172.21.15.111:0/2656151958 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089f59e700 1 --2- 172.21.15.111:0/2656151958 >> v2:172.21.15.143:3300/0 conn(0x7f08a00cc040 0x7f08a0158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 -- 172.21.15.111:0/2656151958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 msgr2=0x7f08a00ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 --2- 172.21.15.111:0/2656151958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 -- 172.21.15.111:0/2656151958 >> v2:172.21.15.143:3300/0 conn(0x7f08a00cc040 msgr2=0x7f08a0158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 --2- 172.21.15.111:0/2656151958 >> v2:172.21.15.143:3300/0 conn(0x7f08a00cc040 0x7f08a0158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 -- 172.21.15.111:0/2656151958 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00cafa0 con 0x7f08a00caa10 2023-12-19T09:02:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089659c700 1 --2- 172.21.15.111:0/2656151958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a015d420 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f08a00b2ab0 tx=0x7f0888009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d490da3591b08a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089e59c700 1 -- 172.21.15.111:0/2656151958 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800e070 con 0x7f08a00caa10 2023-12-19T09:02:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089e59c700 1 -- 172.21.15.111:0/2656151958 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088800b970 con 0x7f08a00caa10 2023-12-19T09:02:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f089e59c700 1 -- 172.21.15.111:0/2656151958 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08880092e0 con 0x7f08a00caa10 2023-12-19T09:02:50.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 -- 172.21.15.111:0/2656151958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 msgr2=0x7f08a015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/2656151958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a015d420 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f08a00b2ab0 tx=0x7f0888009840 comp rx=0 tx=0).stop 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 -- 172.21.15.111:0/2656151958 shutdown_connections 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/2656151958 >> v2:172.21.15.143:3300/0 conn(0x7f08a00cc040 0x7f08a0158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/2656151958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a015d420 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/2656151958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 -- 172.21.15.111:0/2656151958 >> 172.21.15.111:0/2656151958 conn(0x7f08a0067bb0 msgr2=0x7f08a00cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 -- 172.21.15.111:0/2656151958 shutdown_connections 2023-12-19T09:02:50.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.547+0000 7f08a4a2d700 1 -- 172.21.15.111:0/2656151958 wait complete. 2023-12-19T09:02:50.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.551+0000 7f08a4a2d700 1 Processor -- start 2023-12-19T09:02:50.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.551+0000 7f08a4a2d700 1 -- start start 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 0x7f08a00c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00c9270 con 0x7f08a00caa10 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00b4290 con 0x7f08a00cc040 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a0153c40 con 0x7f08a00ce2d0 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52656/0 (socket says 172.21.15.111:52656) 2023-12-19T09:02:50.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 -- 172.21.15.111:0/1576942576 learned_addr learned my addr 172.21.15.111:0/1576942576 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:50.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089659c700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 0x7f08a00c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 -- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 msgr2=0x7f08a00c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089f59e700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 -- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 msgr2=0x7f08a00c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 0x7f08a00c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 -- 172.21.15.111:0/1576942576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888009ae0 con 0x7f08a00caa10 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089659c700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 0x7f08a00c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f089ed9d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f089000e950 tx=0x7f08900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000c850 con 0x7f08a00caa10 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089000c9b0 con 0x7f08a00caa10 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a00b4b10 con 0x7f08a00caa10 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000cd70 con 0x7f08a00caa10 2023-12-19T09:02:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.555+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a0153960 con 0x7f08a00caa10 2023-12-19T09:02:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.559+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a004be40 con 0x7f08a00caa10 2023-12-19T09:02:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.559+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 69) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0890013680 con 0x7f08a00caa10 2023-12-19T09:02:50.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.559+0000 7f08977fe700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 0x7f0880062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:50.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.559+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f089007ce70 con 0x7f08a00caa10 2023-12-19T09:02:50.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.559+0000 7f089659c700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 0x7f0880062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:50.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.563+0000 7f089659c700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 0x7f0880062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0888009ab0 tx=0x7f0888018000 comp rx=0 tx=0).ready entity=mgr.7493 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:50.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.567+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f089004cff0 con 0x7f08a00caa10 2023-12-19T09:02:50.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:50.771+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f08a00cb280 con 0x7f08a00caa10 2023-12-19T09:02:51.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.187+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 70) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f089004d3c0 con 0x7f08a00caa10 2023-12-19T09:02:51.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.187+0000 7f08977fe700 1 -- 172.21.15.111:0/1576942576 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v70) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0890050a80 con 0x7f08a00caa10 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 msgr2=0x7f0880062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 0x7f0880062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0888009ab0 tx=0x7f0888018000 comp rx=0 tx=0).stop 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 msgr2=0x7f08a00c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f089000e950 tx=0x7f08900091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 shutdown_connections 2023-12-19T09:02:51.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0880060670 0x7f0880062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08a00cc040 0x7f08a00c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08a00ce2d0 0x7f08a00c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 --2- 172.21.15.111:0/1576942576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08a00caa10 0x7f08a00c8d30 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 >> 172.21.15.111:0/1576942576 conn(0x7f08a0067bb0 msgr2=0x7f08a00c4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 shutdown_connections 2023-12-19T09:02:51.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.191+0000 7f08a4a2d700 1 -- 172.21.15.111:0/1576942576 wait complete. 2023-12-19T09:02:51.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T09:02:51.375 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:51.367+0000 7f715795b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:51.388 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:51.383+0000 7f64bcf10000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:51.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.387+0000 7f5505dcc700 1 Processor -- start 2023-12-19T09:02:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.387+0000 7f5505dcc700 1 -- start start 2023-12-19T09:02:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f550006b050 0x7f550006b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006c7c0 0x7f550015d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55001594b0 0x7f55001598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550015da20 con 0x7f550006b050 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550006da40 con 0x7f55001594b0 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5505dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550006cc10 con 0x7f550006c7c0 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f550006b050 0x7f550006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f550006b050 0x7f550006b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52680/0 (socket says 172.21.15.111:52680) 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 -- 172.21.15.111:0/2341357018 learned_addr learned my addr 172.21.15.111:0/2341357018 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f54f7fff700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006c7c0 0x7f550015d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f55055cb700 1 --2- 172.21.15.111:0/2341357018 >> v2:172.21.15.143:3300/0 conn(0x7f55001594b0 0x7f55001598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 -- 172.21.15.111:0/2341357018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006c7c0 msgr2=0x7f550015d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006c7c0 0x7f550015d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 -- 172.21.15.111:0/2341357018 >> v2:172.21.15.143:3300/0 conn(0x7f55001594b0 msgr2=0x7f55001598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 --2- 172.21.15.111:0/2341357018 >> v2:172.21.15.143:3300/0 conn(0x7f55001594b0 0x7f55001598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 -- 172.21.15.111:0/2341357018 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550006cec0 con 0x7f550006b050 2023-12-19T09:02:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f5504dca700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006b050 0x7f550006b420 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f55000aee10 tx=0x7f54fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f923e9a2e4f3c3cf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f54f77fe700 1 -- 172.21.15.111:0/2341357018 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00b860 con 0x7f550006b050 2023-12-19T09:02:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f54f77fe700 1 -- 172.21.15.111:0/2341357018 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54fc00b9c0 con 0x7f550006b050 2023-12-19T09:02:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.391+0000 7f54f77fe700 1 -- 172.21.15.111:0/2341357018 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00bd60 con 0x7f550006b050 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 -- 172.21.15.111:0/2341357018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006b050 msgr2=0x7f550006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006b050 0x7f550006b420 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f55000aee10 tx=0x7f54fc009250 comp rx=0 tx=0).stop 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 -- 172.21.15.111:0/2341357018 shutdown_connections 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 --2- 172.21.15.111:0/2341357018 >> v2:172.21.15.143:3300/0 conn(0x7f55001594b0 0x7f55001598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006c7c0 0x7f550015d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 --2- 172.21.15.111:0/2341357018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006b050 0x7f550006b420 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 -- 172.21.15.111:0/2341357018 >> 172.21.15.111:0/2341357018 conn(0x7f55000c8cd0 msgr2=0x7f55000c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 -- 172.21.15.111:0/2341357018 shutdown_connections 2023-12-19T09:02:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 -- 172.21.15.111:0/2341357018 wait complete. 2023-12-19T09:02:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.395+0000 7f5505dcc700 1 Processor -- start 2023-12-19T09:02:51.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 -- start start 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006b050 0x7f55000b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55001594b0 0x7f55000ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001504d0 con 0x7f550006c7c0 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5500155c10 con 0x7f55001594b0 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55000b0e80 con 0x7f550006b050 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52694/0 (socket says 172.21.15.111:52694) 2023-12-19T09:02:51.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 -- 172.21.15.111:0/3634994570 learned_addr learned my addr 172.21.15.111:0/3634994570 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:51.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5504dca700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006b050 0x7f55000b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f55055cb700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55001594b0 0x7f55000ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 -- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006b050 msgr2=0x7f55000b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006b050 0x7f55000b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 -- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55001594b0 msgr2=0x7f55000ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55001594b0 0x7f55000ac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 -- 172.21.15.111:0/3634994570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0094f0 con 0x7f550006c7c0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f7fff700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f54f000d8c0 tx=0x7f54f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00125f0 con 0x7f550006c7c0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f0012770 con 0x7f550006c7c0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55000b00c0 con 0x7f550006c7c0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.399+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0012b30 con 0x7f550006c7c0 2023-12-19T09:02:51.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.403+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001555c0 con 0x7f550006c7c0 2023-12-19T09:02:51.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.403+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550004be40 con 0x7f550006c7c0 2023-12-19T09:02:51.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.403+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 70) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f54f001e460 con 0x7f550006c7c0 2023-12-19T09:02:51.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.407+0000 7f54f4ff9700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 0x7f54ec062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.407+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f54f007d360 con 0x7f550006c7c0 2023-12-19T09:02:51.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.407+0000 7f5504dca700 1 -- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 msgr2=0x7f54ec062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:51.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.407+0000 7f5504dca700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 0x7f54ec062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:51.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.411+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54f0050ed0 con 0x7f550006c7c0 2023-12-19T09:02:51.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.587+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f550006d230 con 0x7f550006c7c0 2023-12-19T09:02:51.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.595+0000 7f54f4ff9700 1 -- 172.21.15.111:0/3634994570 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f54f0050a60 con 0x7f550006c7c0 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 msgr2=0x7f54ec062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 0x7f54ec062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 msgr2=0x7f55000a89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f54f000d8c0 tx=0x7f54f0008040 comp rx=0 tx=0).stop 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 shutdown_connections 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f54ec0606f0 0x7f54ec062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55001594b0 0x7f55000ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f550006c7c0 0x7f55000a89f0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 --2- 172.21.15.111:0/3634994570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f550006b050 0x7f55000b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 >> 172.21.15.111:0/3634994570 conn(0x7f55000c8cd0 msgr2=0x7f55000c90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 shutdown_connections 2023-12-19T09:02:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.599+0000 7f5505dcc700 1 -- 172.21.15.111:0/3634994570 wait complete. 2023-12-19T09:02:51.622 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T09:02:51.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 Processor -- start 2023-12-19T09:02:51.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 -- start start 2023-12-19T09:02:51.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 0x7fe2b006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b00cecd0 0x7fe2b006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b00cbdf0 con 0x7fe2b0158e20 2023-12-19T09:02:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00cc0d0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b722e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00cc3b0 con 0x7fe2b00cb970 2023-12-19T09:02:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b57cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b00cecd0 0x7fe2b006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b57cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2b00cecd0 0x7fe2b006b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52052/0 (socket says 172.21.15.111:52052) 2023-12-19T09:02:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b57cb700 1 -- 172.21.15.111:0/299995288 learned_addr learned my addr 172.21.15.111:0/299995288 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2b4fca700 1 --1- 172.21.15.111:0/299995288 >> v1:172.21.15.111:6789/0 conn(0x7fe2b0158e20 0x7fe2b00ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46134/0 (socket says 172.21.15.111:46134) 2023-12-19T09:02:51.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.779+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/299995288 >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 0x7fe2b006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b57cb700 1 -- 172.21.15.111:0/299995288 >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 msgr2=0x7fe2b006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b57cb700 1 --2- 172.21.15.111:0/299995288 >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 0x7fe2b006acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b57cb700 1 -- 172.21.15.111:0/299995288 >> v1:172.21.15.111:6789/0 conn(0x7fe2b0158e20 legacy=0x7fe2b00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b57cb700 1 -- 172.21.15.111:0/299995288 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00cc690 con 0x7fe2b00cecd0 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/299995288 >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 0x7fe2b006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b57cb700 1 --2- 172.21.15.111:0/299995288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b006b200 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009f20 tx=0x7fe2a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d89192ed1850698 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2a77fe700 1 -- 172.21.15.111:0/299995288 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a800e070 con 0x7fe2b00cecd0 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2a77fe700 1 -- 172.21.15.111:0/299995288 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a800ba40 con 0x7fe2b00cecd0 2023-12-19T09:02:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2a77fe700 1 -- 172.21.15.111:0/299995288 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a80092e0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 -- 172.21.15.111:0/299995288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 msgr2=0x7fe2b006b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 --2- 172.21.15.111:0/299995288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b006b200 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009f20 tx=0x7fe2a8009840 comp rx=0 tx=0).stop 2023-12-19T09:02:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 -- 172.21.15.111:0/299995288 shutdown_connections 2023-12-19T09:02:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 --2- 172.21.15.111:0/299995288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b006b200 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 --2- 172.21.15.111:0/299995288 >> v2:172.21.15.143:3300/0 conn(0x7fe2b00cb970 0x7fe2b006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 -- 172.21.15.111:0/299995288 >> 172.21.15.111:0/299995288 conn(0x7fe2b0067bb0 msgr2=0x7fe2b00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:51.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 -- 172.21.15.111:0/299995288 shutdown_connections 2023-12-19T09:02:51.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.783+0000 7fe2b722e700 1 -- 172.21.15.111:0/299995288 wait complete. 2023-12-19T09:02:51.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 Processor -- start 2023-12-19T09:02:51.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 -- start start 2023-12-19T09:02:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0158e20 0x7fe2b00b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b00b5710 0x7fe2b0152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0151ef0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b00b3c20 con 0x7fe2b00b5710 2023-12-19T09:02:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b722e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b01512e0 con 0x7fe2b0158e20 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.787+0000 7fe2b4fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52056/0 (socket says 172.21.15.111:52056) 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 -- 172.21.15.111:0/1958831404 learned_addr learned my addr 172.21.15.111:0/1958831404 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0158e20 0x7fe2b00b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b57cb700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b00b5710 0x7fe2b0152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 -- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0158e20 msgr2=0x7fe2b00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0158e20 0x7fe2b00b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 -- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b00b5710 msgr2=0x7fe2b0152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b00b5710 0x7fe2b0152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 -- 172.21.15.111:0/1958831404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8009ae0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b4fca700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00c7c0 tx=0x7fe2ac007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac005a70 con 0x7fe2b00cecd0 2023-12-19T09:02:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac005bd0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b01515c0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0070d0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b0151b80 con 0x7fe2b00cecd0 2023-12-19T09:02:51.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.791+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b006b760 con 0x7fe2b00cecd0 2023-12-19T09:02:51.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.795+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 70) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fe2ac007480 con 0x7fe2b00cecd0 2023-12-19T09:02:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.795+0000 7fe2a4ff9700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 0x7fe29c062ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.795+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fe2ac07c0e0 con 0x7fe2b00cecd0 2023-12-19T09:02:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.795+0000 7fe2a7fff700 1 -- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 msgr2=0x7fe29c062ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.795+0000 7fe2a7fff700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 0x7fe29c062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:51.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.803+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe2b006b760 con 0x7fe2b00cecd0 2023-12-19T09:02:51.942 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:51.939+0000 7f64bcf10000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:51.944 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:51.939+0000 7f715795b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:51.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.971+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe2b006b760 con 0x7fe2b00cecd0 2023-12-19T09:02:51.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.979+0000 7fe2a4ff9700 1 -- 172.21.15.111:0/1958831404 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe2b006b760 con 0x7fe2b00cecd0 2023-12-19T09:02:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 msgr2=0x7fe29c062ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 0x7fe29c062ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 msgr2=0x7fe2b014ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00c7c0 tx=0x7fe2ac007f70 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 shutdown_connections 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fe29c0606f0 0x7fe29c062ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b00b5710 0x7fe2b0152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2b0158e20 0x7fe2b00b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 --2- 172.21.15.111:0/1958831404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2b00cecd0 0x7fe2b014ca90 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 >> 172.21.15.111:0/1958831404 conn(0x7fe2b0067bb0 msgr2=0x7fe2b006b050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 shutdown_connections 2023-12-19T09:02:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:51.983+0000 7fe2b722e700 1 -- 172.21.15.111:0/1958831404 wait complete. 2023-12-19T09:02:52.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:52.013 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T09:02:52.019 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.015+0000 7f64bcf10000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:52.023 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.015+0000 7f715795b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:52.043 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1h 2023-12-19T09:02:52.163 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.159+0000 7f64bcf10000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:52.180 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.175+0000 7f715795b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:52.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:02:52.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:02:52.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 Processor -- start 2023-12-19T09:02:52.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 -- start start 2023-12-19T09:02:52.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f76f00866c0 0x7f76f00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f76f0086a90 0x7f76f00c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f00ac820 con 0x7f76f006a6a0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f00abef0 con 0x7f76f0086a90 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f8185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f00ac1d0 con 0x7f76f00866c0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f6722700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f76f006a6a0 0x7f76f006ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46138/0 (socket says 172.21.15.111:46138) 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f6722700 1 -- 172.21.15.111:0/293832175 learned_addr learned my addr 172.21.15.111:0/293832175 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f5720700 1 --2- 172.21.15.111:0/293832175 >> v2:172.21.15.111:3300/0 conn(0x7f76f0086a90 0x7f76f00c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435763751 0 0) 0x7f76f00ac820 con 0x7f76f006a6a0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.243+0000 7f76f5f21700 1 --2- 172.21.15.111:0/293832175 >> v2:172.21.15.143:3300/0 conn(0x7f76f00866c0 0x7f76f00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc003550 con 0x7f76f006a6a0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4197387412 0 0) 0x7f76dc003550 con 0x7f76f006a6a0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f5720700 1 -- 172.21.15.111:0/293832175 >> v2:172.21.15.143:3300/0 conn(0x7f76f00866c0 msgr2=0x7f76f00c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f5720700 1 --2- 172.21.15.111:0/293832175 >> v2:172.21.15.143:3300/0 conn(0x7f76f00866c0 0x7f76f00c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f5720700 1 -- 172.21.15.111:0/293832175 >> v1:172.21.15.111:6789/0 conn(0x7f76f006a6a0 legacy=0x7f76f006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f5720700 1 -- 172.21.15.111:0/293832175 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00ac4b0 con 0x7f76f0086a90 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f5720700 1 --2- 172.21.15.111:0/293832175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f0086a90 0x7f76f00c7010 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f76f00c68e0 tx=0x7f76e0007200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37bb2d82e4a0653 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:52.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e000e070 con 0x7f76f0086a90 2023-12-19T09:02:52.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e00090d0 con 0x7f76f0086a90 2023-12-19T09:02:52.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f4f1f700 1 -- 172.21.15.111:0/293832175 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e80017f0 con 0x7f76f0086a90 2023-12-19T09:02:52.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 -- 172.21.15.111:0/293832175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f0086a90 msgr2=0x7f76f00c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:52.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 --2- 172.21.15.111:0/293832175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f0086a90 0x7f76f00c7010 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f76f00c68e0 tx=0x7f76e0007200 comp rx=0 tx=0).stop 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 -- 172.21.15.111:0/293832175 shutdown_connections 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 --2- 172.21.15.111:0/293832175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f0086a90 0x7f76f00c7010 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 --2- 172.21.15.111:0/293832175 >> v2:172.21.15.143:3300/0 conn(0x7f76f00866c0 0x7f76f00c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 -- 172.21.15.111:0/293832175 >> 172.21.15.111:0/293832175 conn(0x7f76f0067bb0 msgr2=0x7f76f00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 -- 172.21.15.111:0/293832175 shutdown_connections 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.247+0000 7f76f8185700 1 -- 172.21.15.111:0/293832175 wait complete. 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 Processor -- start 2023-12-19T09:02:52.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- start start 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76f006a6a0 0x7f76f0169090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f0086a90 0x7f76f0167710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0167ff0 con 0x7f76f00866c0 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f01682d0 con 0x7f76f0086a90 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f01685b0 con 0x7f76f006a6a0 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52066/0 (socket says 172.21.15.111:52066) 2023-12-19T09:02:52.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 -- 172.21.15.111:0/2020686836 learned_addr learned my addr 172.21.15.111:0/2020686836 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f6722700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f0086a90 0x7f76f0167710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76f006a6a0 0x7f76f0169090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76f006a6a0 msgr2=0x7f76f0169090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76f006a6a0 0x7f76f0169090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f0086a90 msgr2=0x7f76f0167710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f0086a90 0x7f76f0167710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00094d0 con 0x7f76f00866c0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f5720700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f76e0002000 tx=0x7f76e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0012640 con 0x7f76f00866c0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e00127a0 con 0x7f76f00866c0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0012b60 con 0x7f76f00866c0 2023-12-19T09:02:52.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f0168890 con 0x7f76f00866c0 2023-12-19T09:02:52.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.251+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f0165910 con 0x7f76f00866c0 2023-12-19T09:02:52.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0168d90 con 0x7f76f00866c0 2023-12-19T09:02:52.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 70) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f76e001c400 con 0x7f76f00866c0 2023-12-19T09:02:52.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76e5ffb700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:52.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f76e007d950 con 0x7f76f00866c0 2023-12-19T09:02:52.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76f5f21700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:52.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.255+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:52.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.259+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f76e004d960 con 0x7f76f00866c0 2023-12-19T09:02:52.446 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.443+0000 7f64bcf10000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:02:52.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.451+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f0071120 con 0x7f76d40606c0 2023-12-19T09:02:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.459+0000 7f76f5f21700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.459+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:52.471 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.463+0000 7f715795b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:02:52.523 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.519+0000 7f64bcf10000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:52.545 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.539+0000 7f715795b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:52.766 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.763+0000 7f64bcf10000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:02:52.766 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.759+0000 7f715795b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:02:52.847 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.843+0000 7f64bcf10000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:02:52.852 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.847+0000 7f715795b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:02:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.863+0000 7f76f5f21700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:52.863+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T09:02:52.937 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:52.935+0000 7f64bcf10000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:02:52.942 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:52.935+0000 7f715795b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:02:53.345 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:53.339+0000 7f64bcf10000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:02:53.355 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:53.347+0000 7f715795b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:02:53.519 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:53.515+0000 7f64bcf10000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:02:53.525 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:53.519+0000 7f715795b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:02:53.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:53.667+0000 7f76f5f21700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:53.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:53.667+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T09:02:53.894 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:53.891+0000 7f64bcf10000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:02:53.901 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:53.895+0000 7f715795b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:02:54.123 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.119+0000 7f64bcf10000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:54.126 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.119+0000 7f715795b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:54.218 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.211+0000 7f715795b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:02:54.221 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.219+0000 7f64bcf10000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:02:54.289 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.283+0000 7f715795b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:02:54.313 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.307+0000 7f64bcf10000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:02:54.375 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.367+0000 7f715795b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:02:54.398 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.391+0000 7f64bcf10000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:02:54.587 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.579+0000 7f715795b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:54.639 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.635+0000 7f64bcf10000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:02:54.665 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.659+0000 7f715795b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:02:54.719 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.715+0000 7f64bcf10000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:02:54.748 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.743+0000 7f715795b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:02:54.813 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:54.807+0000 7f64bcf10000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:02:54.931 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:54.923+0000 7f715795b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:02:55.012 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:55.007+0000 7f64bcf10000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:02:55.016 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:55.011+0000 7f715795b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:02:55.102 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:55.099+0000 7f64bcf10000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:02:55.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:55.271+0000 7f76f5f21700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:55.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:55.271+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T09:02:55.749 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:55.743+0000 7f715795b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:02:55.850 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:55.847+0000 7f64bcf10000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:02:55.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:55.863+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 71) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f76e004dd30 con 0x7f76f00866c0 2023-12-19T09:02:55.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:55.863+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:55.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:55.863+0000 7f76e5ffb700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:56.045 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.038+0000 7f6480625700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.046 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.038+0000 7f6480625700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.046 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.038+0000 7f6480625700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.046 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.038+0000 7f6480625700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.046 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.038+0000 7f6480625700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.047 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.042+0000 7f647d61f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.047 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.042+0000 7f647d61f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.047 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.042+0000 7f647d61f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.047 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.042+0000 7f647d61f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.048 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:56.042+0000 7f647d61f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:02:56.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.871+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 72) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f76e00076a0 con 0x7f76f00866c0 2023-12-19T09:02:56.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.871+0000 7f76e5ffb700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:56.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.871+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f0071120 con 0x7f76d40606c0 2023-12-19T09:02:56.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.879+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:56.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.883+0000 7f76f5f21700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062af0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76ec007fd0 tx=0x7f76ec00f040 comp rx=0 tx=0).ready entity=mgr.8788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:56.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.979+0000 7f76e5ffb700 1 -- 172.21.15.111:0/2020686836 <== mgr.8788 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f76f0071120 con 0x7f76d40606c0 2023-12-19T09:02:56.986 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T09:02:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 msgr2=0x7f76d4062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062af0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f76ec007fd0 tx=0x7f76ec00f040 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 msgr2=0x7f76f01653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f76e0002000 tx=0x7f76e0017000 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 shutdown_connections 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f76d40606c0 0x7f76d4062af0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f0086a90 0x7f76f0167710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f00866c0 0x7f76f01653d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 --2- 172.21.15.111:0/2020686836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76f006a6a0 0x7f76f0169090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 >> 172.21.15.111:0/2020686836 conn(0x7f76f0067bb0 msgr2=0x7f76f00c9650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 shutdown_connections 2023-12-19T09:02:56.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:56.983+0000 7f76f8185700 1 -- 172.21.15.111:0/2020686836 wait complete. 2023-12-19T09:02:57.016 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T09:02:57.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T09:02:57.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.175+0000 7fdc1a22e700 1 Processor -- start 2023-12-19T09:02:57.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.175+0000 7fdc1a22e700 1 -- start start 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.179+0000 7fdc1a22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc140aea30 0x7fdc140aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.179+0000 7fdc1a22e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdc140c7780 0x7fdc140c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.179+0000 7fdc1a22e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140caf00 con 0x7fdc1413f830 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.179+0000 7fdc1a22e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc140cc200 con 0x7fdc140c7780 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.179+0000 7fdc1a22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc140cb160 con 0x7fdc140aea30 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc137fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc1413f830 0x7fdc140ca7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46148/0 (socket says 172.21.15.111:46148) 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc137fe700 1 -- 172.21.15.111:0/2092506218 learned_addr learned my addr 172.21.15.111:0/2092506218 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 --2- 172.21.15.111:0/2092506218 >> v2:172.21.15.111:3300/0 conn(0x7fdc140c7780 0x7fdc140c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc12ffd700 1 --2- 172.21.15.111:0/2092506218 >> v2:172.21.15.143:3300/0 conn(0x7fdc140aea30 0x7fdc140aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc127fc700 1 -- 172.21.15.111:0/2092506218 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126974225 0 0) 0x7fdc140caf00 con 0x7fdc1413f830 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc127fc700 1 -- 172.21.15.111:0/2092506218 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbfc003570 con 0x7fdc1413f830 2023-12-19T09:02:57.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 -- 172.21.15.111:0/2092506218 >> v2:172.21.15.143:3300/0 conn(0x7fdc140aea30 msgr2=0x7fdc140aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 --2- 172.21.15.111:0/2092506218 >> v2:172.21.15.143:3300/0 conn(0x7fdc140aea30 0x7fdc140aeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 -- 172.21.15.111:0/2092506218 >> v1:172.21.15.111:6789/0 conn(0x7fdc1413f830 legacy=0x7fdc140ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 -- 172.21.15.111:0/2092506218 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140cb440 con 0x7fdc140c7780 2023-12-19T09:02:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc13fff700 1 --2- 172.21.15.111:0/2092506218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140c7780 0x7fdc140c7be0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdc140ad740 tx=0x7fdc0c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3969e6f849ceab86 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc127fc700 1 -- 172.21.15.111:0/2092506218 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00e070 con 0x7fdc140c7780 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc127fc700 1 -- 172.21.15.111:0/2092506218 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c00bab0 con 0x7fdc140c7780 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc127fc700 1 -- 172.21.15.111:0/2092506218 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00be70 con 0x7fdc140c7780 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 -- 172.21.15.111:0/2092506218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140c7780 msgr2=0x7fdc140c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/2092506218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140c7780 0x7fdc140c7be0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdc140ad740 tx=0x7fdc0c009780 comp rx=0 tx=0).stop 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 -- 172.21.15.111:0/2092506218 shutdown_connections 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/2092506218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140c7780 0x7fdc140c7be0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/2092506218 >> v2:172.21.15.143:3300/0 conn(0x7fdc140aea30 0x7fdc140aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 -- 172.21.15.111:0/2092506218 >> 172.21.15.111:0/2092506218 conn(0x7fdc140ccc20 msgr2=0x7fdc140cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 -- 172.21.15.111:0/2092506218 shutdown_connections 2023-12-19T09:02:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.183+0000 7fdc1a22e700 1 -- 172.21.15.111:0/2092506218 wait complete. 2023-12-19T09:02:57.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 Processor -- start 2023-12-19T09:02:57.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 -- start start 2023-12-19T09:02:57.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140c7780 0x7fdc1413b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1413f830 0x7fdc140c2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1416c830 con 0x7fdc140aea30 2023-12-19T09:02:57.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1416d0d0 con 0x7fdc1413f830 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc1a22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc14169f70 con 0x7fdc140c7780 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc137fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc137fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52088/0 (socket says 172.21.15.111:52088) 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc137fe700 1 -- 172.21.15.111:0/1150599827 learned_addr learned my addr 172.21.15.111:0/1150599827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc13fff700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1413f830 0x7fdc140c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.187+0000 7fdc12ffd700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140c7780 0x7fdc1413b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 -- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140c7780 msgr2=0x7fdc1413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140c7780 0x7fdc1413b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 -- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1413f830 msgr2=0x7fdc140c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1413f830 0x7fdc140c2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 -- 172.21.15.111:0/1150599827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c009a20 con 0x7fdc140aea30 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc137fe700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c5c0 tx=0x7fdc04007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04005b30 con 0x7fdc140aea30 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04005cb0 con 0x7fdc140aea30 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1416ca80 con 0x7fdc140aea30 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007320 con 0x7fdc140aea30 2023-12-19T09:02:57.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc14169c50 con 0x7fdc140aea30 2023-12-19T09:02:57.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.191+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc14071120 con 0x7fdc140aea30 2023-12-19T09:02:57.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.195+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 72) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fdc0401b3b0 con 0x7fdc140aea30 2023-12-19T09:02:57.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.195+0000 7fdbfb7fe700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 0x7fdc000628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.195+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fdc0407bd70 con 0x7fdc140aea30 2023-12-19T09:02:57.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.195+0000 7fdc12ffd700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 0x7fdc000628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.199+0000 7fdc12ffd700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 0x7fdc000628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdc140c2920 tx=0x7fdc080091a0 comp rx=0 tx=0).ready entity=mgr.8788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.203+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc0404c150 con 0x7fdc140aea30 2023-12-19T09:02:57.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.387+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc1416d210 con 0x7fdc000603f0 2023-12-19T09:02:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.499+0000 7fdbfb7fe700 1 -- 172.21.15.111:0/1150599827 <== mgr.8788 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdc08009ad0 con 0x7fdc000603f0 2023-12-19T09:02:57.508 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path snap_test_dir1 2023-12-19T09:02:57.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 msgr2=0x7fdc000628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 0x7fdc000628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdc140c2920 tx=0x7fdc080091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 msgr2=0x7fdc1413c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c5c0 tx=0x7fdc04007e00 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 shutdown_connections 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fdc000603f0 0x7fdc000628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1413f830 0x7fdc140c2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140c7780 0x7fdc1413b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 --2- 172.21.15.111:0/1150599827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc140aea30 0x7fdc1413c120 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 >> 172.21.15.111:0/1150599827 conn(0x7fdc140ccc20 msgr2=0x7fdc140ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 shutdown_connections 2023-12-19T09:02:57.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.507+0000 7fdc1a22e700 1 -- 172.21.15.111:0/1150599827 wait complete. 2023-12-19T09:02:57.536 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T09:02:57.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs snap_test_dir1 2023-12-19T09:02:57.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 Processor -- start 2023-12-19T09:02:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 -- start start 2023-12-19T09:02:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65100addc0 0x7f65100c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f65100b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651006da90 con 0x7f65100ad9f0 2023-12-19T09:02:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651006a620 con 0x7f65100addc0 2023-12-19T09:02:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f651648e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651006dd30 con 0x7f651015b140 2023-12-19T09:02:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f6514c8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65100addc0 0x7f65100c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f6514c8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65100addc0 0x7f65100c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:59414/0 (socket says 172.21.15.111:59414) 2023-12-19T09:02:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f6514c8b700 1 -- 172.21.15.111:0/3589930073 learned_addr learned my addr 172.21.15.111:0/3589930073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.699+0000 7f6515c8d700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f65100b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6514c8b700 1 -- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 msgr2=0x7f65100b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6514c8b700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f65100b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6514c8b700 1 -- 172.21.15.111:0/3589930073 >> v1:172.21.15.143:6789/0 conn(0x7f65100ad9f0 legacy=0x7f651006ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6514c8b700 1 -- 172.21.15.111:0/3589930073 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651006a8c0 con 0x7f65100addc0 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6514c8b700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f65100c8db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f65000099b0 tx=0x7f6500009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17aa965c2241262b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6507fff700 1 -- 172.21.15.111:0/3589930073 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500009ea0 con 0x7f65100addc0 2023-12-19T09:02:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6507fff700 1 -- 172.21.15.111:0/3589930073 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650000b860 con 0x7f65100addc0 2023-12-19T09:02:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f6507fff700 1 -- 172.21.15.111:0/3589930073 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650000bc00 con 0x7f65100addc0 2023-12-19T09:02:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 -- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 msgr2=0x7f65100c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f65100c8db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f65000099b0 tx=0x7f6500009250 comp rx=0 tx=0).stop 2023-12-19T09:02:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 -- 172.21.15.111:0/3589930073 shutdown_connections 2023-12-19T09:02:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f65100b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 --2- 172.21.15.111:0/3589930073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f65100c8db0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 -- 172.21.15.111:0/3589930073 >> 172.21.15.111:0/3589930073 conn(0x7f6510067bb0 msgr2=0x7f65100c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 -- 172.21.15.111:0/3589930073 shutdown_connections 2023-12-19T09:02:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.703+0000 7f651648e700 1 -- 172.21.15.111:0/3589930073 wait complete. 2023-12-19T09:02:57.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 Processor -- start 2023-12-19T09:02:57.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 -- start start 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65100ad9f0 0x7f65100ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f6510156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f651016c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100ac400 con 0x7f65100ad9f0 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65101695f0 con 0x7f65100addc0 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651648e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651016d710 con 0x7f651015b140 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651548c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65100ad9f0 0x7f65100ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651548c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65100ad9f0 0x7f65100ac820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52100/0 (socket says 172.21.15.111:52100) 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f651548c700 1 -- 172.21.15.111:0/2528805668 learned_addr learned my addr 172.21.15.111:0/2528805668 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:57.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f6515c8d700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f651016c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.707+0000 7f6514c8b700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f6510156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 -- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 msgr2=0x7f651016c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f651015b140 0x7f651016c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 -- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 msgr2=0x7f6510156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65100addc0 0x7f6510156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 -- 172.21.15.111:0/2528805668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65000094f0 con 0x7f65100ad9f0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651548c700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65100ad9f0 0x7f65100ac820 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c590 tx=0x7f650c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c011590 con 0x7f65100ad9f0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c011710 con 0x7f65100ad9f0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c011ad0 con 0x7f65100ad9f0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651648e700 1 -- 172.21.15.111:0/2528805668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65100acd60 con 0x7f65100ad9f0 2023-12-19T09:02:57.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651648e700 1 -- 172.21.15.111:0/2528805668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65100abd40 con 0x7f65100ad9f0 2023-12-19T09:02:57.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.711+0000 7f651648e700 1 -- 172.21.15.111:0/2528805668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651004be40 con 0x7f65100ad9f0 2023-12-19T09:02:57.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.715+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 72) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f650c01b3b0 con 0x7f65100ad9f0 2023-12-19T09:02:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.715+0000 7f65057fa700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f40603f0 0x7f64f40628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.715+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f650c07bed0 con 0x7f65100ad9f0 2023-12-19T09:02:57.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.715+0000 7f6514c8b700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f40603f0 0x7f64f40628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:57.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.719+0000 7f6514c8b700 1 --2- 172.21.15.111:0/2528805668 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f64f40603f0 0x7f64f40628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65000099b0 tx=0x7f6500017000 comp rx=0 tx=0).ready entity=mgr.8788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:57.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.719+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f650c04c1e0 con 0x7f65100ad9f0 2023-12-19T09:02:57.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.883+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 73) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f650c04c5b0 con 0x7f65100ad9f0 2023-12-19T09:02:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.919+0000 7f651648e700 1 -- 172.21.15.111:0/2528805668 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f65100c8b70 con 0x7f64f40603f0 2023-12-19T09:02:57.941 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:57.934+0000 7f6487633700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-12-19T09:02:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:57.935+0000 7f65057fa700 1 -- 172.21.15.111:0/2528805668 <== mgr.8788 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f65100c8b70 con 0x7f64f40603f0 2023-12-19T09:02:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-12-19T09:02:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T09:02:57.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:02:57.959 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T09:02:57.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T09:02:58.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.191+0000 7f0a6ed63700 1 Processor -- start 2023-12-19T09:02:58.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.195+0000 7f0a6ed63700 1 -- start start 2023-12-19T09:02:58.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.195+0000 7f0a6ed63700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0a68158b70 0x7f0a680cb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:58.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.195+0000 7f0a6ed63700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680c86b0 con 0x7f0a68067ee0 2023-12-19T09:02:58.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.195+0000 7f0a6ed63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680c8380 con 0x7f0a68158f40 2023-12-19T09:02:58.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.195+0000 7f0a6ed63700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a68068880 con 0x7f0a68158b70 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6e562700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0a68067ee0 0x7f0a680c8b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46158/0 (socket says 172.21.15.111:46158) 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6e562700 1 -- 172.21.15.111:0/3435293534 learned_addr learned my addr 172.21.15.111:0/3435293534 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6dd61700 1 --2- 172.21.15.111:0/3435293534 >> v2:172.21.15.143:3300/0 conn(0x7f0a68158b70 0x7f0a680cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152293442 0 0) 0x7f0a680c86b0 con 0x7f0a68067ee0 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a54003570 con 0x7f0a68067ee0 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210075132 0 0) 0x7f0a680c8380 con 0x7f0a68158f40 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a680c86b0 con 0x7f0a68158f40 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6dd61700 1 -- 172.21.15.111:0/3435293534 >> v1:172.21.15.143:6789/0 conn(0x7f0a68158f40 legacy=0x7f0a680677d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6dd61700 1 -- 172.21.15.111:0/3435293534 >> v1:172.21.15.111:6789/0 conn(0x7f0a68067ee0 legacy=0x7f0a680c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6dd61700 1 -- 172.21.15.111:0/3435293534 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a68068b60 con 0x7f0a68158b70 2023-12-19T09:02:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6dd61700 1 --2- 172.21.15.111:0/3435293534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a680cb840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a640098e0 tx=0x7f0a640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402e3900b4890413 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64011630 con 0x7f0a68158b70 2023-12-19T09:02:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a64011790 con 0x7f0a68158b70 2023-12-19T09:02:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6cd5f700 1 -- 172.21.15.111:0/3435293534 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64011b30 con 0x7f0a68158b70 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6ed63700 1 -- 172.21.15.111:0/3435293534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 msgr2=0x7f0a680cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.199+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/3435293534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a680cb840 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a640098e0 tx=0x7f0a640091a0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- 172.21.15.111:0/3435293534 shutdown_connections 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/3435293534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a680cb840 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- 172.21.15.111:0/3435293534 >> 172.21.15.111:0/3435293534 conn(0x7f0a680cdc00 msgr2=0x7f0a680ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- 172.21.15.111:0/3435293534 shutdown_connections 2023-12-19T09:02:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- 172.21.15.111:0/3435293534 wait complete. 2023-12-19T09:02:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 Processor -- start 2023-12-19T09:02:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- start start 2023-12-19T09:02:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a68067ee0 0x7f0a680c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a6814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680b1d00 con 0x7f0a68158f40 2023-12-19T09:02:58.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680b3880 con 0x7f0a68158b70 2023-12-19T09:02:58.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6ed63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a680b2d00 con 0x7f0a68067ee0 2023-12-19T09:02:58.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6e562700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:58.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6e562700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52116/0 (socket says 172.21.15.111:52116) 2023-12-19T09:02:58.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.203+0000 7f0a6e562700 1 -- 172.21.15.111:0/2702617834 learned_addr learned my addr 172.21.15.111:0/2702617834 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6dd61700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a68067ee0 0x7f0a680c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6d560700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a6814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 -- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a68067ee0 msgr2=0x7f0a680c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a68067ee0 0x7f0a680c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 -- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 msgr2=0x7f0a6814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a6814e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 -- 172.21.15.111:0/2702617834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64009420 con 0x7f0a68158f40 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6d560700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a6814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6e562700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000cc70 tx=0x7f0a60001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000ce80 con 0x7f0a68158f40 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6000aad0 con 0x7f0a68158f40 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a680b2fe0 con 0x7f0a68158f40 2023-12-19T09:02:58.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000ae50 con 0x7f0a68158f40 2023-12-19T09:02:58.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a680b35a0 con 0x7f0a68158f40 2023-12-19T09:02:58.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.207+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6804be40 con 0x7f0a68158f40 2023-12-19T09:02:58.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.211+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 73) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f0a60007300 con 0x7f0a68158f40 2023-12-19T09:02:58.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.211+0000 7f0a5dffb700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 0x7f0a4c062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:58.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.211+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f0a6007d480 con 0x7f0a68158f40 2023-12-19T09:02:58.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.211+0000 7f0a6dd61700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 0x7f0a4c062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:58.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.211+0000 7f0a6dd61700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 0x7f0a4c062c00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a640093f0 tx=0x7f0a64017000 comp rx=0 tx=0).ready entity=mgr.8788 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:58.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.219+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a6004d310 con 0x7f0a68158f40 2023-12-19T09:02:58.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.431+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0a68068e40 con 0x7f0a68158f40 2023-12-19T09:02:58.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.899+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f0a6007d0f0 con 0x7f0a68158f40 2023-12-19T09:02:58.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.899+0000 7f0a5dffb700 1 -- 172.21.15.111:0/2702617834 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0a60047c70 con 0x7f0a68158f40 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 msgr2=0x7f0a4c062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 0x7f0a4c062c00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a640093f0 tx=0x7f0a64017000 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 msgr2=0x7f0a6814ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000cc70 tx=0x7f0a60001a40 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 shutdown_connections 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0a4c060750 0x7f0a4c062c00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a68158b70 0x7f0a6814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a68158f40 0x7f0a6814ea90 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 --2- 172.21.15.111:0/2702617834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a68067ee0 0x7f0a680c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 >> 172.21.15.111:0/2702617834 conn(0x7f0a680cdc00 msgr2=0x7f0a680cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:58.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 shutdown_connections 2023-12-19T09:02:58.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:58.903+0000 7f0a6ed63700 1 -- 172.21.15.111:0/2702617834 wait complete. 2023-12-19T09:02:58.932 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:02:58.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:58.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:02:59.014 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:02:59.015 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:02:59.015 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:02:59.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:02:59.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[57873]: fuse finished with error 0 and tester_r 0 2023-12-19T09:02:59.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:02:59.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:59.049 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:02:59.108 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:59.099+0000 7fb24e5ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:59.110 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:59.106+0000 7f5e3e0eb000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:02:59.234 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:02:59.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:59.235 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T09:02:59.257 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T09:02:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:59.258 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:02:59.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:59.258 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T09:02:59.294 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T09:02:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:59.296 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:02:59.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:02:59.296 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T09:02:59.322 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T09:02:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:02:59.324 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:02:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.491+0000 7f52cd06e700 1 Processor -- start 2023-12-19T09:02:59.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.491+0000 7f52cd06e700 1 -- start start 2023-12-19T09:02:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52cd06e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c8158d40 0x7f52c8067960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52cd06e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c8069070 0x7f52c8159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52cd06e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c8068be0 con 0x7f52c80cf9c0 2023-12-19T09:02:59.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52cd06e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c8067fd0 con 0x7f52c8069070 2023-12-19T09:02:59.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52cd06e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80682b0 con 0x7f52c8158d40 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c8069070 0x7f52c8159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c8069070 0x7f52c8159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52148/0 (socket says 172.21.15.111:52148) 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 -- 172.21.15.111:0/2609835231 learned_addr learned my addr 172.21.15.111:0/2609835231 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c6d9d700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c8158d40 0x7f52c8067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 -- 172.21.15.111:0/2609835231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c8158d40 msgr2=0x7f52c8067960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c8158d40 0x7f52c8067960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 -- 172.21.15.111:0/2609835231 >> v1:172.21.15.143:6789/0 conn(0x7f52c80cf9c0 legacy=0x7f52c80cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 -- 172.21.15.111:0/2609835231 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8068590 con 0x7f52c8069070 2023-12-19T09:02:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.495+0000 7f52c659c700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8069070 0x7f52c8159110 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0099b0 tx=0x7f52bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02672ee37f05862 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52c5d9b700 1 -- 172.21.15.111:0/2609835231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00b860 con 0x7f52c8069070 2023-12-19T09:02:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52c5d9b700 1 -- 172.21.15.111:0/2609835231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00b9c0 con 0x7f52c8069070 2023-12-19T09:02:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52c5d9b700 1 -- 172.21.15.111:0/2609835231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00bd60 con 0x7f52c8069070 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- 172.21.15.111:0/2609835231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8069070 msgr2=0x7f52c8159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8069070 0x7f52c8159110 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0099b0 tx=0x7f52bc009250 comp rx=0 tx=0).stop 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- 172.21.15.111:0/2609835231 shutdown_connections 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8069070 0x7f52c8159110 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2609835231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c8158d40 0x7f52c8067960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- 172.21.15.111:0/2609835231 >> 172.21.15.111:0/2609835231 conn(0x7f52c80ccc40 msgr2=0x7f52c80cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- 172.21.15.111:0/2609835231 shutdown_connections 2023-12-19T09:02:59.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- 172.21.15.111:0/2609835231 wait complete. 2023-12-19T09:02:59.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 Processor -- start 2023-12-19T09:02:59.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.499+0000 7f52cd06e700 1 -- start start 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 0x7f52c80afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c80cf9c0 0x7f52c815c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80b18e0 con 0x7f52c8158d40 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c806d900 con 0x7f52c8069070 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c806dbe0 con 0x7f52c80cf9c0 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c6d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 0x7f52c80afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52164/0 (socket says 172.21.15.111:52164) 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c6d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 0x7f52c80afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:59482/0 (socket says 172.21.15.111:59482) 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 -- 172.21.15.111:0/2988115785 learned_addr learned my addr 172.21.15.111:0/2988115785 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:59.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c659c700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c80cf9c0 0x7f52c815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c80cf9c0 msgr2=0x7f52c815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c80cf9c0 0x7f52c815c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 msgr2=0x7f52c80afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 0x7f52c80afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 -- 172.21.15.111:0/2988115785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0094f0 con 0x7f52c8158d40 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52c759e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c8e0 tx=0x7f52c0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0012630 con 0x7f52c8158d40 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0012790 con 0x7f52c8158d40 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80b1b80 con 0x7f52c8158d40 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0012b50 con 0x7f52c8158d40 2023-12-19T09:02:59.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.503+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c80b00c0 con 0x7f52c8158d40 2023-12-19T09:02:59.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c80b1740 con 0x7f52c8158d40 2023-12-19T09:02:59.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f52c001a3a0 con 0x7f52c8158d40 2023-12-19T09:02:59.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52aeffd700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 0x7f52b4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f52c007d120 con 0x7f52c8158d40 2023-12-19T09:02:59.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52c6d9d700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 msgr2=0x7f52b4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:59.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.507+0000 7f52c6d9d700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 0x7f52b4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:59.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.515+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f52c80b1740 con 0x7f52c8158d40 2023-12-19T09:02:59.646 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:59.639+0000 7fb24e5ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:59.688 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:59.682+0000 7f5e3e0eb000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:02:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.711+0000 7f52c6d9d700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 msgr2=0x7f52b4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:59.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.711+0000 7f52c6d9d700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 0x7f52b4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:02:59.717 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:59.711+0000 7fb24e5ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:59.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.723+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c80b1740 con 0x7f52c8158d40 2023-12-19T09:02:59.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.723+0000 7f52aeffd700 1 -- 172.21.15.111:0/2988115785 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f52c80b1740 con 0x7f52c8158d40 2023-12-19T09:02:59.730 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:02:59.730 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:02:59.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 msgr2=0x7f52b4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:02:59.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 0x7f52b4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 msgr2=0x7f52c80b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f52c000c8e0 tx=0x7f52c0001710 comp rx=0 tx=0).stop 2023-12-19T09:02:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 shutdown_connections 2023-12-19T09:02:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f52b4060700 0x7f52b4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c8158d40 0x7f52c80b20a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52c80cf9c0 0x7f52c815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 --2- 172.21.15.111:0/2988115785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52c8069070 0x7f52c80afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 >> 172.21.15.111:0/2988115785 conn(0x7f52c80ccc40 msgr2=0x7f52c8155f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:59.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 shutdown_connections 2023-12-19T09:02:59.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.727+0000 7f52cd06e700 1 -- 172.21.15.111:0/2988115785 wait complete. 2023-12-19T09:02:59.736 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:02:59.758 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:02:59.767 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:59.762+0000 7f5e3e0eb000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:02:59.869 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:02:59.863+0000 7fb24e5ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:59.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.899+0000 7f31ca385700 1 Processor -- start 2023-12-19T09:02:59.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.899+0000 7f31ca385700 1 -- start start 2023-12-19T09:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31ca385700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31c40aba90 0x7f31c40abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31ca385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 0x7f31c40c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31ca385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c40c8730 con 0x7f31c40cb750 2023-12-19T09:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31ca385700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c40cbf80 con 0x7f31c40aba90 2023-12-19T09:02:59.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31ca385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c40c7e70 con 0x7f31c40cbb20 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31c40aba90 0x7f31c40abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c8922700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 0x7f31c40c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31c40aba90 0x7f31c40abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:59488/0 (socket says 172.21.15.111:59488) 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c8922700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 0x7f31c40c8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:42102/0 (socket says 172.21.15.111:42102) 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 -- 172.21.15.111:0/3289694988 learned_addr learned my addr 172.21.15.111:0/3289694988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:59.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c3fff700 1 --1- 172.21.15.111:0/3289694988 >> v1:172.21.15.143:6789/0 conn(0x7f31c40cb750 0x7f31c40c1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:46626/0 (socket says 172.21.15.111:46626) 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 -- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 msgr2=0x7f31c40c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 --2- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 0x7f31c40c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 -- 172.21.15.111:0/3289694988 >> v1:172.21.15.143:6789/0 conn(0x7f31c40cb750 legacy=0x7f31c40c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 -- 172.21.15.111:0/3289694988 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c40c23a0 con 0x7f31c40aba90 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c37fe700 1 --2- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40aba90 0x7f31c40abff0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f31b80099b0 tx=0x7f31b8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bbfe474b7625420 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c2ffd700 1 -- 172.21.15.111:0/3289694988 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8009ea0 con 0x7f31c40aba90 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c2ffd700 1 -- 172.21.15.111:0/3289694988 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b800b860 con 0x7f31c40aba90 2023-12-19T09:02:59.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.903+0000 7f31c2ffd700 1 -- 172.21.15.111:0/3289694988 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800bc20 con 0x7f31c40aba90 2023-12-19T09:02:59.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40aba90 msgr2=0x7f31c40abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 --2- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40aba90 0x7f31c40abff0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f31b80099b0 tx=0x7f31b8009250 comp rx=0 tx=0).stop 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- 172.21.15.111:0/3289694988 shutdown_connections 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 --2- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40cbb20 0x7f31c40c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 --2- 172.21.15.111:0/3289694988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40aba90 0x7f31c40abff0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- 172.21.15.111:0/3289694988 >> 172.21.15.111:0/3289694988 conn(0x7f31c40cd9c0 msgr2=0x7f31c40c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- 172.21.15.111:0/3289694988 shutdown_connections 2023-12-19T09:02:59.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- 172.21.15.111:0/3289694988 wait complete. 2023-12-19T09:02:59.913 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:02:59.910+0000 7f5e3e0eb000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:02:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 Processor -- start 2023-12-19T09:02:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.907+0000 7f31ca385700 1 -- start start 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40aba90 0x7f31c41559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40cb750 0x7f31c4169fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c415dcb0 con 0x7f31c40cbb20 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c416cfc0 con 0x7f31c40cb750 2023-12-19T09:02:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c415e6c0 con 0x7f31c40aba90 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40cb750 0x7f31c4169fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52170/0 (socket says 172.21.15.111:52170) 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 -- 172.21.15.111:0/3056473273 learned_addr learned my addr 172.21.15.111:0/3056473273 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c3fff700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40aba90 0x7f31c41559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:02:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 -- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40aba90 msgr2=0x7f31c41559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40aba90 0x7f31c41559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 -- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40cb750 msgr2=0x7f31c4169fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40cb750 0x7f31c4169fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 -- 172.21.15.111:0/3056473273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80094f0 con 0x7f31c40cbb20 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31c8922700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009e20 tx=0x7f31bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc00caa0 con 0x7f31c40cbb20 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31bc00f040 con 0x7f31c40cbb20 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c415e9a0 con 0x7f31c40cbb20 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31bc00ce70 con 0x7f31c40cbb20 2023-12-19T09:02:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.911+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c416cc50 con 0x7f31c40cbb20 2023-12-19T09:02:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c416d150 con 0x7f31c40cbb20 2023-12-19T09:02:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f31bc014690 con 0x7f31c40cbb20 2023-12-19T09:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31abfff700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 0x7f31b0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f31bc07d880 con 0x7f31c40cbb20 2023-12-19T09:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31c3fff700 1 -- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 msgr2=0x7f31b0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:02:59.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.915+0000 7f31c3fff700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 0x7f31b0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:02:59.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:02:59.923+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f31bc04d620 con 0x7f31c40cbb20 2023-12-19T09:03:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.091+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31c4071120 con 0x7f31c40cbb20 2023-12-19T09:03:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.091+0000 7f31abfff700 1 -- 172.21.15.111:0/3056473273 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16949 (secure 0 0 0) 0x7f31bc050c40 con 0x7f31c40cbb20 2023-12-19T09:03:00.098 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:00.098 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":162,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:55.853471+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2824146656":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1422322069":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/3492177270":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1354315856":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6833/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6832/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/272833109":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1889918723":"2023-12-20T09:02:55.853401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 msgr2=0x7f31b0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 0x7f31b0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 msgr2=0x7f31c415d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009e20 tx=0x7f31bc00b040 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 shutdown_connections 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f31b0060700 0x7f31b0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c40cb750 0x7f31c4169fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31c40cbb20 0x7f31c415d770 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 --2- 172.21.15.111:0/3056473273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31c40aba90 0x7f31c41559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 >> 172.21.15.111:0/3056473273 conn(0x7f31c40cd9c0 msgr2=0x7f31c4157a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 shutdown_connections 2023-12-19T09:03:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.095+0000 7f31ca385700 1 -- 172.21.15.111:0/3056473273 wait complete. 2023-12-19T09:03:00.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T09:02:39.584489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T09:02:40.099361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T09:03:00.120 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:03:00.134 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:00.127+0000 7fb24e5ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:03:00.191 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:00.186+0000 7f5e3e0eb000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:03:00.234 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:00.227+0000 7fb24e5ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:00.266 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:00.262+0000 7f5e3e0eb000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:00.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.275+0000 7f7ca5545700 1 Processor -- start 2023-12-19T09:03:00.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 -- start start 2023-12-19T09:03:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca00cab10 0x7f7ca00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ca006b1b0 0x7f7ca006b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca00c7450 con 0x7f7ca015d410 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca006bea0 con 0x7f7ca006b1b0 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00c6e00 con 0x7f7ca00cab10 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ca006b1b0 0x7f7ca006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ca006b1b0 0x7f7ca006b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52178/0 (socket says 172.21.15.111:52178) 2023-12-19T09:03:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/3493119379 learned_addr learned my addr 172.21.15.111:0/3493119379 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c967fc700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca00cab10 0x7f7ca00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/3493119379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca00cab10 msgr2=0x7f7ca00ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca00cab10 0x7f7ca00ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/3493119379 >> v1:172.21.15.111:6789/0 conn(0x7f7ca015d410 legacy=0x7f7ca00ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/3493119379 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca00c70e0 con 0x7f7ca006b1b0 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9e7fc700 1 -- 172.21.15.111:0/3493119379 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698668785 0 0) 0x7f7ca00c7450 con 0x7f7ca015d410 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9f7fe700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca006b650 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f7c980099b0 tx=0x7f7c98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422d04a5aa6552d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9e7fc700 1 -- 172.21.15.111:0/3493119379 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9800b860 con 0x7f7ca006b1b0 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9e7fc700 1 -- 172.21.15.111:0/3493119379 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9800b9c0 con 0x7f7ca006b1b0 2023-12-19T09:03:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7c9e7fc700 1 -- 172.21.15.111:0/3493119379 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9800bd60 con 0x7f7ca006b1b0 2023-12-19T09:03:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 -- 172.21.15.111:0/3493119379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 msgr2=0x7f7ca006b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.279+0000 7f7ca5545700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca006b650 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f7c980099b0 tx=0x7f7c98009250 comp rx=0 tx=0).stop 2023-12-19T09:03:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- 172.21.15.111:0/3493119379 shutdown_connections 2023-12-19T09:03:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca006b650 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 --2- 172.21.15.111:0/3493119379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca00cab10 0x7f7ca00ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- 172.21.15.111:0/3493119379 >> 172.21.15.111:0/3493119379 conn(0x7f7ca0067bb0 msgr2=0x7f7ca00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- 172.21.15.111:0/3493119379 shutdown_connections 2023-12-19T09:03:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- 172.21.15.111:0/3493119379 wait complete. 2023-12-19T09:03:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 Processor -- start 2023-12-19T09:03:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- start start 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca0162c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 0x7f7ca015c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00b0b30 con 0x7f7ca006b1b0 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca01627c0 con 0x7f7ca00cab10 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7ca5545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0161bb0 con 0x7f7ca015d410 2023-12-19T09:03:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7c9effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca0162c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7c9f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7c967fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 0x7f7ca015c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7c9f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:42122/0 (socket says 172.21.15.111:42122) 2023-12-19T09:03:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.283+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/1172036733 learned_addr learned my addr 172.21.15.111:0/1172036733 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 msgr2=0x7f7ca015c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 0x7f7ca015c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/1172036733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 msgr2=0x7f7ca0162c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca0162c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 -- 172.21.15.111:0/1172036733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c980094f0 con 0x7f7ca015d410 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c967fc700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 0x7f7ca015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c9f7fe700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b3c0 tx=0x7f7c98017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9800b860 con 0x7f7ca015d410 2023-12-19T09:03:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca0161e90 con 0x7f7ca015d410 2023-12-19T09:03:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9801a630 con 0x7f7ca015d410 2023-12-19T09:03:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca0162450 con 0x7f7ca015d410 2023-12-19T09:03:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9801a9b0 con 0x7f7ca015d410 2023-12-19T09:03:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca0162950 con 0x7f7ca015d410 2023-12-19T09:03:00.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.287+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f7c98024440 con 0x7f7ca015d410 2023-12-19T09:03:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.291+0000 7f7c977fe700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 0x7f7c80062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.291+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f7c9807ec00 con 0x7f7ca015d410 2023-12-19T09:03:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.291+0000 7f7c9effd700 1 -- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 msgr2=0x7f7c80062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.291+0000 7f7c9effd700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 0x7f7c80062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:00.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.299+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c98085080 con 0x7f7ca015d410 2023-12-19T09:03:00.450 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:00.443+0000 7fb24e5ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:03:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.471+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ca0160d50 con 0x7f7ca015d410 2023-12-19T09:03:00.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7c977fe700 1 -- 172.21.15.111:0/1172036733 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v162) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c9804eb70 con 0x7f7ca015d410 2023-12-19T09:03:00.480 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 msgr2=0x7f7c80062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 0x7f7c80062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 msgr2=0x7f7ca0160fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b3c0 tx=0x7f7c98017000 comp rx=0 tx=0).stop 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 shutdown_connections 2023-12-19T09:03:00.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f7c80060700 0x7f7c80062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ca00cab10 0x7f7ca015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ca015d410 0x7f7ca0160fd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 --2- 172.21.15.111:0/1172036733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ca006b1b0 0x7f7ca0162c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 >> 172.21.15.111:0/1172036733 conn(0x7f7ca0067bb0 msgr2=0x7f7ca006a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 shutdown_connections 2023-12-19T09:03:00.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.475+0000 7f7ca5545700 1 -- 172.21.15.111:0/1172036733 wait complete. 2023-12-19T09:03:00.494 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:00.490+0000 7f5e3e0eb000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:03:00.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T09:03:00.533 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:00.527+0000 7fb24e5ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:03:00.577 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:00.570+0000 7f5e3e0eb000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:03:00.628 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:00.619+0000 7fb24e5ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:03:00.662 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:00.658+0000 7f5e3e0eb000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:03:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.687+0000 7fc28f40c700 1 Processor -- start 2023-12-19T09:03:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 -- start start 2023-12-19T09:03:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2880c69a0 0x7fc2880c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc28806b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 0x7fc28815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28806d970 con 0x7fc2880c69a0 2023-12-19T09:03:00.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28806cd60 con 0x7fc28806c7c0 2023-12-19T09:03:00.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28f40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28806d040 con 0x7fc28806de00 2023-12-19T09:03:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2880c69a0 0x7fc2880c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28ec0b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 0x7fc28815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28dc09700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc28806b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28ec0b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 0x7fc28815cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:59512/0 (socket says 172.21.15.111:59512) 2023-12-19T09:03:00.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28dc09700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc28806b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:42136/0 (socket says 172.21.15.111:42136) 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28ec0b700 1 -- 172.21.15.111:0/3221548212 learned_addr learned my addr 172.21.15.111:0/3221548212 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 -- 172.21.15.111:0/3221548212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 msgr2=0x7fc28806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 --2- 172.21.15.111:0/3221548212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc28806b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 -- 172.21.15.111:0/3221548212 >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 msgr2=0x7fc28815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 --2- 172.21.15.111:0/3221548212 >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 0x7fc28815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 -- 172.21.15.111:0/3221548212 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28806d320 con 0x7fc2880c69a0 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28e40a700 1 --2- 172.21.15.111:0/3221548212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2880c69a0 0x7fc2880c6d70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc2840098e0 tx=0x7fc2840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d5797300dd4b1e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28d408700 1 -- 172.21.15.111:0/3221548212 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284011630 con 0x7fc2880c69a0 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28d408700 1 -- 172.21.15.111:0/3221548212 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284011790 con 0x7fc2880c69a0 2023-12-19T09:03:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.691+0000 7fc28d408700 1 -- 172.21.15.111:0/3221548212 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284011b30 con 0x7fc2880c69a0 2023-12-19T09:03:00.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- 172.21.15.111:0/3221548212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2880c69a0 msgr2=0x7fc2880c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- 172.21.15.111:0/3221548212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2880c69a0 0x7fc2880c6d70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc2840098e0 tx=0x7fc2840091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- 172.21.15.111:0/3221548212 shutdown_connections 2023-12-19T09:03:00.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- 172.21.15.111:0/3221548212 >> v2:172.21.15.143:3300/0 conn(0x7fc28806c7c0 0x7fc28815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- 172.21.15.111:0/3221548212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc28806b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- 172.21.15.111:0/3221548212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2880c69a0 0x7fc2880c6d70 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- 172.21.15.111:0/3221548212 >> 172.21.15.111:0/3221548212 conn(0x7fc288067b50 msgr2=0x7fc2880cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- 172.21.15.111:0/3221548212 shutdown_connections 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- 172.21.15.111:0/3221548212 wait complete. 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 Processor -- start 2023-12-19T09:03:00.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- start start 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc2880b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 0x7fc28814a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28815df10 con 0x7fc28806c7c0 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28815ea90 con 0x7fc2880c69a0 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28f40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28815e160 con 0x7fc28806de00 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28e40a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28e40a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52210/0 (socket says 172.21.15.111:52210) 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28e40a700 1 -- 172.21.15.111:0/919718387 learned_addr learned my addr 172.21.15.111:0/919718387 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28dc09700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc2880b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.695+0000 7fc28ec0b700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 0x7fc28814a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 -- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 msgr2=0x7fc2880b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc2880b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 -- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 msgr2=0x7fc28814a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 0x7fc28814a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 -- 172.21.15.111:0/919718387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284009420 con 0x7fc28806c7c0 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28ec0b700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 0x7fc28814a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:00.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28e40a700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc284011f20 tx=0x7fc284017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:00.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284011900 con 0x7fc28806c7c0 2023-12-19T09:03:00.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284011a80 con 0x7fc28806c7c0 2023-12-19T09:03:00.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28815e440 con 0x7fc28806c7c0 2023-12-19T09:03:00.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284026680 con 0x7fc28806c7c0 2023-12-19T09:03:00.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2880c12f0 con 0x7fc28806c7c0 2023-12-19T09:03:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.699+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc288161480 con 0x7fc28806c7c0 2023-12-19T09:03:00.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.703+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fc284023030 con 0x7fc28806c7c0 2023-12-19T09:03:00.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.703+0000 7fc27e7fc700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 0x7fc26c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:00.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.703+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fc28407e720 con 0x7fc28806c7c0 2023-12-19T09:03:00.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.703+0000 7fc28dc09700 1 -- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 msgr2=0x7fc26c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:00.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.703+0000 7fc28dc09700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 0x7fc26c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:00.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.707+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc288161480 con 0x7fc28806c7c0 2023-12-19T09:03:00.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.875+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc288161480 con 0x7fc28806c7c0 2023-12-19T09:03:00.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.875+0000 7fc27e7fc700 1 -- 172.21.15.111:0/919718387 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v162) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc288161480 con 0x7fc28806c7c0 2023-12-19T09:03:00.883 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T09:03:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 msgr2=0x7fc26c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 0x7fc26c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 msgr2=0x7fc2880c0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:00.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc284011f20 tx=0x7fc284017000 comp rx=0 tx=0).stop 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 shutdown_connections 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc26c060700 0x7fc26c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2880c69a0 0x7fc28814a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc28806de00 0x7fc2880b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 --2- 172.21.15.111:0/919718387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc28806c7c0 0x7fc2880c0db0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 >> 172.21.15.111:0/919718387 conn(0x7fc288067b50 msgr2=0x7fc28815aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 shutdown_connections 2023-12-19T09:03:00.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:00.879+0000 7fc28f40c700 1 -- 172.21.15.111:0/919718387 wait complete. 2023-12-19T09:03:00.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T09:03:01.040 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:01.035+0000 7fb24e5ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:03:01.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 Processor -- start 2023-12-19T09:03:01.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 -- start start 2023-12-19T09:03:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 0x7f56140ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56140c7780 0x7f56140c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140adc60 con 0x7f561413f6a0 2023-12-19T09:03:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561406d7d0 con 0x7f56140c7780 2023-12-19T09:03:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5618944700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56140c8140 con 0x7f561413f2d0 2023-12-19T09:03:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f561259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f561413f6a0 0x7f561415a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46168/0 (socket says 172.21.15.111:46168) 2023-12-19T09:03:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f561259c700 1 -- 172.21.15.111:0/3263727180 learned_addr learned my addr 172.21.15.111:0/3263727180 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f561359e700 1 --2- 172.21.15.111:0/3263727180 >> v2:172.21.15.111:3300/0 conn(0x7f56140c7780 0x7f56140c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.059+0000 7f5612d9d700 1 --2- 172.21.15.111:0/3263727180 >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 0x7f56140ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5611d9b700 1 -- 172.21.15.111:0/3263727180 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495965515 0 0) 0x7f56140adc60 con 0x7f561413f6a0 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5611d9b700 1 -- 172.21.15.111:0/3263727180 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc003570 con 0x7f561413f6a0 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f561359e700 1 -- 172.21.15.111:0/3263727180 >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 msgr2=0x7f56140ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f561359e700 1 --2- 172.21.15.111:0/3263727180 >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 0x7f56140ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f561359e700 1 -- 172.21.15.111:0/3263727180 >> v1:172.21.15.111:6789/0 conn(0x7f561413f6a0 legacy=0x7f561415a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f561359e700 1 -- 172.21.15.111:0/3263727180 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140ad050 con 0x7f56140c7780 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5612d9d700 1 --2- 172.21.15.111:0/3263727180 >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 0x7f56140ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f561359e700 1 --2- 172.21.15.111:0/3263727180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56140c7780 0x7f56140c7c00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f5614069bf0 tx=0x7f560c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc796c2d1d9ccce server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5611d9b700 1 -- 172.21.15.111:0/3263727180 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c00e070 con 0x7f56140c7780 2023-12-19T09:03:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5611d9b700 1 -- 172.21.15.111:0/3263727180 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c00bb20 con 0x7f56140c7780 2023-12-19T09:03:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5611d9b700 1 -- 172.21.15.111:0/3263727180 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c012430 con 0x7f56140c7780 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 -- 172.21.15.111:0/3263727180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56140c7780 msgr2=0x7f56140c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 --2- 172.21.15.111:0/3263727180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56140c7780 0x7f56140c7c00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f5614069bf0 tx=0x7f560c009780 comp rx=0 tx=0).stop 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 -- 172.21.15.111:0/3263727180 shutdown_connections 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 --2- 172.21.15.111:0/3263727180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56140c7780 0x7f56140c7c00 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 --2- 172.21.15.111:0/3263727180 >> v2:172.21.15.143:3300/0 conn(0x7f561413f2d0 0x7f56140ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 -- 172.21.15.111:0/3263727180 >> 172.21.15.111:0/3263727180 conn(0x7f5614067b50 msgr2=0x7f56140cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 -- 172.21.15.111:0/3263727180 shutdown_connections 2023-12-19T09:03:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.063+0000 7f5618944700 1 -- 172.21.15.111:0/3263727180 wait complete. 2023-12-19T09:03:01.073 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:01.066+0000 7f5e3e0eb000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:03:01.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 Processor -- start 2023-12-19T09:03:01.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 -- start start 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 0x7f56140c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f561413f2d0 0x7f561413dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56140afc20 con 0x7f561413f6a0 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56140af660 con 0x7f56140c7780 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.067+0000 7f5618944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56140af940 con 0x7f561413f2d0 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52226/0 (socket says 172.21.15.111:52226) 2023-12-19T09:03:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 -- 172.21.15.111:0/3912957224 learned_addr learned my addr 172.21.15.111:0/3912957224 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561259c700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f561413f2d0 0x7f561413dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f5612d9d700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 0x7f56140c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 -- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f561413f2d0 msgr2=0x7f561413dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f561413f2d0 0x7f561413dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 -- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 msgr2=0x7f56140c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 0x7f56140c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 -- 172.21.15.111:0/3912957224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c009a20 con 0x7f561413f6a0 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f5612d9d700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 0x7f56140c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f561359e700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f561406a0d0 tx=0x7f560c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0032f0 con 0x7f561413f6a0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c027470 con 0x7f561413f6a0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5614168880 con 0x7f561413f6a0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0277f0 con 0x7f561413f6a0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56140c5450 con 0x7f561413f6a0 2023-12-19T09:03:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.071+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614168d80 con 0x7f561413f6a0 2023-12-19T09:03:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.075+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f560c024070 con 0x7f561413f6a0 2023-12-19T09:03:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.075+0000 7f55faffd700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 0x7f5600062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.075+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f560c07f6c0 con 0x7f561413f6a0 2023-12-19T09:03:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.075+0000 7f5612d9d700 1 -- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 msgr2=0x7f5600062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.075+0000 7f5612d9d700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 0x7f5600062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:01.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.083+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f560c04f570 con 0x7f561413f6a0 2023-12-19T09:03:01.216 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:01.211+0000 7fb24e5ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:03:01.249 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:01.242+0000 7f5e3e0eb000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:03:01.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.263+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f561404be40 con 0x7f561413f6a0 2023-12-19T09:03:01.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.263+0000 7f55faffd700 1 -- 172.21.15.111:0/3912957224 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v162) v1 ==== 100+0+11 (secure 0 0 0) 0x7f560c052b90 con 0x7f561413f6a0 2023-12-19T09:03:01.270 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 msgr2=0x7f5600062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 0x7f5600062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 msgr2=0x7f561413e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f561406a0d0 tx=0x7f560c019000 comp rx=0 tx=0).stop 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 shutdown_connections 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5600060700 0x7f5600062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561413f6a0 0x7f561413e2a0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f561413f2d0 0x7f561413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 --2- 172.21.15.111:0/3912957224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56140c7780 0x7f56140c4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 >> 172.21.15.111:0/3912957224 conn(0x7f5614067b50 msgr2=0x7f56140c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:01.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 shutdown_connections 2023-12-19T09:03:01.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.267+0000 7f5618944700 1 -- 172.21.15.111:0/3912957224 wait complete. 2023-12-19T09:03:01.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:01.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.435+0000 7f86e1cf0700 1 Processor -- start 2023-12-19T09:03:01.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.435+0000 7f86e1cf0700 1 -- start start 2023-12-19T09:03:01.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86e1cf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f86dc068800 0x7f86dc161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86e1cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc06e150 con 0x7f86dc140ae0 2023-12-19T09:03:01.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86e1cf0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc06d540 con 0x7f86dc140710 2023-12-19T09:03:01.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86e1cf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc06d820 con 0x7f86dc068800 2023-12-19T09:03:01.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f86dc068800 0x7f86dc161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86daffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f86dc140ae0 0x7f86dc0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46184/0 (socket says 172.21.15.111:46184) 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86daffd700 1 -- 172.21.15.111:0/1341567429 learned_addr learned my addr 172.21.15.111:0/1341567429 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f86dc068800 0x7f86dc161cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52230/0 (socket says 172.21.15.111:52230) 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 -- 172.21.15.111:0/1341567429 >> v1:172.21.15.143:6789/0 conn(0x7f86dc140710 legacy=0x7f86dc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 -- 172.21.15.111:0/1341567429 >> v1:172.21.15.111:6789/0 conn(0x7f86dc140ae0 legacy=0x7f86dc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86da7fc700 1 -- 172.21.15.111:0/1341567429 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403444162 0 0) 0x7f86dc06e150 con 0x7f86dc140ae0 2023-12-19T09:03:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 -- 172.21.15.111:0/1341567429 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc06db00 con 0x7f86dc068800 2023-12-19T09:03:01.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.439+0000 7f86dbfff700 1 --2- 172.21.15.111:0/1341567429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc161cd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009e60 tx=0x7f86d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e814de4c43cce629 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86da7fc700 1 -- 172.21.15.111:0/1341567429 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d400e070 con 0x7f86dc068800 2023-12-19T09:03:01.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86da7fc700 1 -- 172.21.15.111:0/1341567429 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d400bc30 con 0x7f86dc068800 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86da7fc700 1 -- 172.21.15.111:0/1341567429 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4012430 con 0x7f86dc068800 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 -- 172.21.15.111:0/1341567429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 msgr2=0x7f86dc161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/1341567429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc161cd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009e60 tx=0x7f86d4009780 comp rx=0 tx=0).stop 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 -- 172.21.15.111:0/1341567429 shutdown_connections 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/1341567429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc161cd0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 -- 172.21.15.111:0/1341567429 >> 172.21.15.111:0/1341567429 conn(0x7f86dc0c8850 msgr2=0x7f86dc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 -- 172.21.15.111:0/1341567429 shutdown_connections 2023-12-19T09:03:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.443+0000 7f86e1cf0700 1 -- 172.21.15.111:0/1341567429 wait complete. 2023-12-19T09:03:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 Processor -- start 2023-12-19T09:03:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- start start 2023-12-19T09:03:01.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86dc140710 0x7f86dc158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86dc140ae0 0x7f86dc166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0c5d20 con 0x7f86dc068800 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc158d50 con 0x7f86dc140710 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc159030 con 0x7f86dc140ae0 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52242/0 (socket says 172.21.15.111:52242) 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 -- 172.21.15.111:0/314971672 learned_addr learned my addr 172.21.15.111:0/314971672 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86daffd700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86dc140710 0x7f86dc158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86dbfff700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86dc140ae0 0x7f86dc166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86dc140ae0 msgr2=0x7f86dc166bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86dc140ae0 0x7f86dc166bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86dc140710 msgr2=0x7f86dc158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86dc140710 0x7f86dc158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 -- 172.21.15.111:0/314971672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d4009a20 con 0x7f86dc068800 2023-12-19T09:03:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86db7fe700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c590 tx=0x7f86cc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc011590 con 0x7f86dc068800 2023-12-19T09:03:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc011710 con 0x7f86dc068800 2023-12-19T09:03:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc0c2f30 con 0x7f86dc068800 2023-12-19T09:03:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc011ad0 con 0x7f86dc068800 2023-12-19T09:03:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.447+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc0ab000 con 0x7f86dc068800 2023-12-19T09:03:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.451+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0c3430 con 0x7f86dc068800 2023-12-19T09:03:01.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.451+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f86cc01b3b0 con 0x7f86dc068800 2023-12-19T09:03:01.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.451+0000 7f86c37fe700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 0x7f86c8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.451+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f86cc07c4b0 con 0x7f86dc068800 2023-12-19T09:03:01.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.451+0000 7f86daffd700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 msgr2=0x7f86c8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:01.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.455+0000 7f86daffd700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 0x7f86c8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:01.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.459+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f86cc04c2c0 con 0x7f86dc068800 2023-12-19T09:03:01.607 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:01.599+0000 7fb24e5ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:03:01.630 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:01.626+0000 7f5e3e0eb000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:03:01.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.659+0000 7f86daffd700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 msgr2=0x7f86c8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:01.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.659+0000 7f86daffd700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 0x7f86c8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:03:01.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.663+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86dc071120 con 0x7f86dc068800 2023-12-19T09:03:01.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86c37fe700 1 -- 172.21.15.111:0/314971672 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f86cc04f8e0 con 0x7f86dc068800 2023-12-19T09:03:01.673 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:01.673 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 msgr2=0x7f86c8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 0x7f86c8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 msgr2=0x7f86dc158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c590 tx=0x7f86cc00c8a0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 shutdown_connections 2023-12-19T09:03:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f86c8060700 0x7f86c8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86dc140710 0x7f86dc158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86dc140ae0 0x7f86dc166bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 --2- 172.21.15.111:0/314971672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86dc068800 0x7f86dc158120 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.667+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 >> 172.21.15.111:0/314971672 conn(0x7f86dc0c8850 msgr2=0x7f86dc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.671+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 shutdown_connections 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.671+0000 7f86e1cf0700 1 -- 172.21.15.111:0/314971672 wait complete. 2023-12-19T09:03:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:03:01.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:01.848 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:01.839+0000 7fb24e5ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:01.857 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:01.850+0000 7f5e3e0eb000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:01.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.855+0000 7f038e585700 1 Processor -- start 2023-12-19T09:03:01.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.855+0000 7f038e585700 1 -- start start 2023-12-19T09:03:01.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038e585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0388067c90 0x7f0388068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038e585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03880c5900 0x7f03880c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038e585700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880c6630 con 0x7f038814fba0 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038e585700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880c6910 con 0x7f03880c5900 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038e585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880c6bf0 con 0x7f0388067c90 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03880c5900 0x7f03880c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03880c5900 0x7f03880c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41516/0 (socket says 172.21.15.111:41516) 2023-12-19T09:03:01.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 -- 172.21.15.111:0/3796492088 learned_addr learned my addr 172.21.15.111:0/3796492088 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f0387fff700 1 --2- 172.21.15.111:0/3796492088 >> v2:172.21.15.143:3300/0 conn(0x7f0388067c90 0x7f0388068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 -- 172.21.15.111:0/3796492088 >> v2:172.21.15.143:3300/0 conn(0x7f0388067c90 msgr2=0x7f0388068060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 --2- 172.21.15.111:0/3796492088 >> v2:172.21.15.143:3300/0 conn(0x7f0388067c90 0x7f0388068060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 -- 172.21.15.111:0/3796492088 >> v1:172.21.15.143:6789/0 conn(0x7f038814fba0 legacy=0x7f0388150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 -- 172.21.15.111:0/3796492088 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880c6ed0 con 0x7f03880c5900 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f038cb22700 1 --2- 172.21.15.111:0/3796492088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03880c5900 0x7f03880c5d60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f038806a0c0 tx=0x7f0380009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc74920cc0765b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.859+0000 7f03877fe700 1 -- 172.21.15.111:0/3796492088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000e070 con 0x7f03880c5900 2023-12-19T09:03:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f03877fe700 1 -- 172.21.15.111:0/3796492088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038000ba10 con 0x7f03880c5900 2023-12-19T09:03:01.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f03877fe700 1 -- 172.21.15.111:0/3796492088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03800092e0 con 0x7f03880c5900 2023-12-19T09:03:01.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 -- 172.21.15.111:0/3796492088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03880c5900 msgr2=0x7f03880c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 --2- 172.21.15.111:0/3796492088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03880c5900 0x7f03880c5d60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f038806a0c0 tx=0x7f0380009840 comp rx=0 tx=0).stop 2023-12-19T09:03:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 -- 172.21.15.111:0/3796492088 shutdown_connections 2023-12-19T09:03:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 --2- 172.21.15.111:0/3796492088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03880c5900 0x7f03880c5d60 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 --2- 172.21.15.111:0/3796492088 >> v2:172.21.15.143:3300/0 conn(0x7f0388067c90 0x7f0388068060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 -- 172.21.15.111:0/3796492088 >> 172.21.15.111:0/3796492088 conn(0x7f03880cd6d0 msgr2=0x7f038806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 -- 172.21.15.111:0/3796492088 shutdown_connections 2023-12-19T09:03:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.863+0000 7f038e585700 1 -- 172.21.15.111:0/3796492088 wait complete. 2023-12-19T09:03:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 Processor -- start 2023-12-19T09:03:01.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 -- start start 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0388067c90 0x7f038814aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03880c5900 0x7f038816a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880aff70 con 0x7f038814fba0 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b0250 con 0x7f03880c5900 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880b0530 con 0x7f0388067c90 2023-12-19T09:03:01.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41520/0 (socket says 172.21.15.111:41520) 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 -- 172.21.15.111:0/966043577 learned_addr learned my addr 172.21.15.111:0/966043577 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f0387fff700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0388067c90 0x7f038814aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f037ffff700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03880c5900 0x7f038816a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 -- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0388067c90 msgr2=0x7f038814aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0388067c90 0x7f038814aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 -- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03880c5900 msgr2=0x7f038816a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03880c5900 0x7f038816a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 -- 172.21.15.111:0/966043577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380009ae0 con 0x7f038814fba0 2023-12-19T09:03:01.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038cb22700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f038806a0f0 tx=0x7f0380017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03800092e0 con 0x7f038814fba0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0380012d00 con 0x7f038814fba0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038001d670 con 0x7f038814fba0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.867+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03880b0810 con 0x7f038814fba0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.871+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038816abe0 con 0x7f038814fba0 2023-12-19T09:03:01.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.871+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03880b0d10 con 0x7f038814fba0 2023-12-19T09:03:01.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.871+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f038001d9e0 con 0x7f038814fba0 2023-12-19T09:03:01.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.875+0000 7f0384ff9700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 0x7f0368062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:01.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.875+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f038007e810 con 0x7f038814fba0 2023-12-19T09:03:01.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.875+0000 7f0387fff700 1 -- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 msgr2=0x7f0368062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:01.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.875+0000 7f0387fff700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 0x7f0368062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:01.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:01.879+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f038004e510 con 0x7f038814fba0 2023-12-19T09:03:01.943 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:01.935+0000 7fb24e5ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:03:01.956 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:01.950+0000 7f5e3e0eb000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:03:02.022 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.015+0000 7fb24e5ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:03:02.045 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.038+0000 7f5e3e0eb000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:03:02.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.051+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0388071120 con 0x7f038814fba0 2023-12-19T09:03:02.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f0384ff9700 1 -- 172.21.15.111:0/966043577 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16949 (secure 0 0 0) 0x7f0380051b30 con 0x7f038814fba0 2023-12-19T09:03:02.061 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:02.061 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":162,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:55.853471+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2824146656":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1422322069":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/3492177270":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1354315856":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6833/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6832/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/272833109":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1889918723":"2023-12-20T09:02:55.853401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 msgr2=0x7f0368062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 0x7f0368062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 msgr2=0x7f03880b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f038806a0f0 tx=0x7f0380017000 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 shutdown_connections 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0368060700 0x7f0368062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03880c5900 0x7f038816a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f038814fba0 0x7f03880b1010 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 --2- 172.21.15.111:0/966043577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0388067c90 0x7f038814aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.055+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 >> 172.21.15.111:0/966043577 conn(0x7f03880cd6d0 msgr2=0x7f038814f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:02.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.059+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 shutdown_connections 2023-12-19T09:03:02.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.059+0000 7f038e585700 1 -- 172.21.15.111:0/966043577 wait complete. 2023-12-19T09:03:02.085 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:03:02.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:02.105 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.099+0000 7fb24e5ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:03:02.125 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.118+0000 7f5e3e0eb000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:03:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 Processor -- start 2023-12-19T09:03:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 -- start start 2023-12-19T09:03:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 0x7ff930161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93006e150 con 0x7ff930140710 2023-12-19T09:03:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93006d540 con 0x7ff930140ae0 2023-12-19T09:03:02.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff934e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93006d820 con 0x7ff930068800 2023-12-19T09:03:02.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff92e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff930140710 0x7ff9300c70a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54872/0 (socket says 172.21.15.111:54872) 2023-12-19T09:03:02.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff92e59c700 1 -- 172.21.15.111:0/2754514641 learned_addr learned my addr 172.21.15.111:0/2754514641 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:02.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff92ed9d700 1 --2- 172.21.15.111:0/2754514641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 0x7ff930161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355939908 0 0) 0x7ff93006e150 con 0x7ff930140710 2023-12-19T09:03:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.243+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c003570 con 0x7ff930140710 2023-12-19T09:03:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850072958 0 0) 0x7ff93006d540 con 0x7ff930140ae0 2023-12-19T09:03:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff93006e150 con 0x7ff930140ae0 2023-12-19T09:03:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92ed9d700 1 -- 172.21.15.111:0/2754514641 >> v1:172.21.15.143:6789/0 conn(0x7ff930140ae0 legacy=0x7ff9300680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92ed9d700 1 -- 172.21.15.111:0/2754514641 >> v1:172.21.15.111:6789/0 conn(0x7ff930140710 legacy=0x7ff9300c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92ed9d700 1 -- 172.21.15.111:0/2754514641 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93006db00 con 0x7ff930068800 2023-12-19T09:03:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92ed9d700 1 --2- 172.21.15.111:0/2754514641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 0x7ff930161cd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff928009e90 tx=0x7ff928009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c099b724553477b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800e070 con 0x7ff930068800 2023-12-19T09:03:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92800b870 con 0x7ff930068800 2023-12-19T09:03:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff92dd9b700 1 -- 172.21.15.111:0/2754514641 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920001470 con 0x7ff930068800 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 -- 172.21.15.111:0/2754514641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 msgr2=0x7ff930161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 --2- 172.21.15.111:0/2754514641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 0x7ff930161cd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff928009e90 tx=0x7ff928009780 comp rx=0 tx=0).stop 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 -- 172.21.15.111:0/2754514641 shutdown_connections 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 --2- 172.21.15.111:0/2754514641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930068800 0x7ff930161cd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 -- 172.21.15.111:0/2754514641 >> 172.21.15.111:0/2754514641 conn(0x7ff9300c8850 msgr2=0x7ff9300c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 -- 172.21.15.111:0/2754514641 shutdown_connections 2023-12-19T09:03:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.247+0000 7ff934e78700 1 -- 172.21.15.111:0/2754514641 wait complete. 2023-12-19T09:03:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 Processor -- start 2023-12-19T09:03:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 -- start start 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930140710 0x7ff93015d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff930140ae0 0x7ff9300c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93015d9f0 con 0x7ff930068800 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9300a9c60 con 0x7ff930140ae0 2023-12-19T09:03:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff934e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9300a9330 con 0x7ff930140710 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41526/0 (socket says 172.21.15.111:41526) 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 -- 172.21.15.111:0/939220850 learned_addr learned my addr 172.21.15.111:0/939220850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92ed9d700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff930140ae0 0x7ff9300c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff927fff700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930140710 0x7ff93015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930140710 msgr2=0x7ff93015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930140710 0x7ff93015d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff930140ae0 msgr2=0x7ff9300c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff930140ae0 0x7ff9300c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 -- 172.21.15.111:0/939220850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff928009a20 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.251+0000 7ff92e59c700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff92000cb60 tx=0x7ff920001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92000ce70 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff920007360 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920004830 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9300a9610 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93015d690 con 0x7ff930068800 2023-12-19T09:03:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93006c0c0 con 0x7ff930068800 2023-12-19T09:03:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.255+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7ff920004c20 con 0x7ff930068800 2023-12-19T09:03:02.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.259+0000 7ff9267fc700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 0x7ff910062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.259+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7ff92007c730 con 0x7ff930068800 2023-12-19T09:03:02.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.259+0000 7ff927fff700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 msgr2=0x7ff910062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:02.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.259+0000 7ff927fff700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 0x7ff910062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:02.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.263+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff92004abe0 con 0x7ff930068800 2023-12-19T09:03:02.343 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.335+0000 7fb24e5ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:02.371 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.366+0000 7f5e3e0eb000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:02.417 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.411+0000 7fb24e5ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:03:02.451 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.446+0000 7f5e3e0eb000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:03:02.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.463+0000 7ff927fff700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 msgr2=0x7ff910062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:02.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.463+0000 7ff927fff700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 0x7ff910062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:03:02.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff930071120 con 0x7ff930068800 2023-12-19T09:03:02.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff9267fc700 1 -- 172.21.15.111:0/939220850 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7ff93006c0c0 con 0x7ff930068800 2023-12-19T09:03:02.474 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:02.474 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:03:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 msgr2=0x7ff910062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 0x7ff910062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 msgr2=0x7ff93015b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7ff92000cb60 tx=0x7ff920001c30 comp rx=0 tx=0).stop 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 shutdown_connections 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff910060700 0x7ff910062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff930140ae0 0x7ff9300c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff930140710 0x7ff93015d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 --2- 172.21.15.111:0/939220850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff930068800 0x7ff93015b860 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 >> 172.21.15.111:0/939220850 conn(0x7ff9300c8850 msgr2=0x7ff93013e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:02.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.467+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 shutdown_connections 2023-12-19T09:03:02.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.471+0000 7ff934e78700 1 -- 172.21.15.111:0/939220850 wait complete. 2023-12-19T09:03:02.478 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:03:02.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:02.511 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.503+0000 7fb24e5ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:03:02.549 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.546+0000 7f5e3e0eb000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:03:02.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.663+0000 7f8d2aee3700 1 Processor -- start 2023-12-19T09:03:02.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2aee3700 1 -- start start 2023-12-19T09:03:02.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2aee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d240b1410 0x7f8d240ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2aee3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2406c7c0 con 0x7f8d2406aa00 2023-12-19T09:03:02.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2aee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2406cd80 con 0x7f8d2406a580 2023-12-19T09:03:02.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2aee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2415b680 con 0x7f8d240b1410 2023-12-19T09:03:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2a6e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8d2406aa00 0x7f8d2415af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54886/0 (socket says 172.21.15.111:54886) 2023-12-19T09:03:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/1925213911 learned_addr learned my addr 172.21.15.111:0/1925213911 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/1925213911 >> v2:172.21.15.143:3300/0 conn(0x7f8d240b1410 0x7f8d240ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067701359 0 0) 0x7f8d2406c7c0 con 0x7f8d2406aa00 2023-12-19T09:03:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d10003570 con 0x7f8d2406aa00 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173528010 0 0) 0x7f8d10003570 con 0x7f8d2406aa00 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 >> v2:172.21.15.143:3300/0 conn(0x7f8d240b1410 msgr2=0x7f8d240ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 --2- 172.21.15.111:0/1925213911 >> v2:172.21.15.143:3300/0 conn(0x7f8d240b1410 0x7f8d240ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 >> v1:172.21.15.143:6789/0 conn(0x7f8d2406a580 legacy=0x7f8d2415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2406b8f0 con 0x7f8d2406aa00 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f8d1c001470 con 0x7f8d2406aa00 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 >> v1:172.21.15.111:6789/0 conn(0x7f8d2406aa00 legacy=0x7f8d2415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 0x7f8d10003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d10004810 0x7f8d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d100051c0 0x7f8d1000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.667+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2415b680 con 0x7f8d10003a60 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1000c0d0 con 0x7f8d10004810 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1000c290 con 0x7f8d100051c0 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 0x7f8d10003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 -- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d100051c0 msgr2=0x7f8d1000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d100051c0 0x7f8d1000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 -- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d10004810 msgr2=0x7f8d10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d296e0700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d10004810 0x7f8d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d10004810 0x7f8d10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 -- 172.21.15.111:0/1925213911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10003570 con 0x7f8d10003a60 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/1925213911 reap_dead start 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d296e0700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d10004810 0x7f8d10004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 0x7f8d10003e70 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e210 tx=0x7f8d2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca904ea765f9d330 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000eac0 con 0x7f8d10003a60 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2000f040 con 0x7f8d10003a60 2023-12-19T09:03:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d28edf700 1 -- 172.21.15.111:0/1925213911 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000ee50 con 0x7f8d10003a60 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 -- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 msgr2=0x7f8d10003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 0x7f8d10003e70 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e210 tx=0x7f8d2000b040 comp rx=0 tx=0).stop 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 -- 172.21.15.111:0/1925213911 shutdown_connections 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/1925213911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d10003a60 0x7f8d10003e70 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 -- 172.21.15.111:0/1925213911 >> 172.21.15.111:0/1925213911 conn(0x7f8d24067bb0 msgr2=0x7f8d240cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 -- 172.21.15.111:0/1925213911 shutdown_connections 2023-12-19T09:03:02.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.671+0000 7f8d2aee3700 1 -- 172.21.15.111:0/1925213911 wait complete. 2023-12-19T09:03:02.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 Processor -- start 2023-12-19T09:03:02.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 -- start start 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 0x7f8d24154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d240b00d0 0x7f8d241527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24153240 con 0x7f8d240b0540 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d241534f0 con 0x7f8d240b00d0 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2aee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d241537d0 con 0x7f8d10003a60 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2a6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2a6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41548/0 (socket says 172.21.15.111:41548) 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.675+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/4207912180 learned_addr learned my addr 172.21.15.111:0/4207912180 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d296e0700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d240b00d0 0x7f8d241527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 0x7f8d24154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 msgr2=0x7f8d24154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 0x7f8d24154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d240b00d0 msgr2=0x7f8d241527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d240b00d0 0x7f8d241527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 -- 172.21.15.111:0/4207912180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d24153ab0 con 0x7f8d240b0540 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 0x7f8d24154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2a6e2700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8d240cc700 tx=0x7f8d2406e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c007360 con 0x7f8d240b0540 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c0074c0 con 0x7f8d240b0540 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c004830 con 0x7f8d240b0540 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d24153d90 con 0x7f8d240b0540 2023-12-19T09:03:02.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d24157320 con 0x7f8d240b0540 2023-12-19T09:03:02.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.679+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2404be40 con 0x7f8d240b0540 2023-12-19T09:03:02.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.683+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f8d1c004ba0 con 0x7f8d240b0540 2023-12-19T09:03:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.683+0000 7f8d19ffb700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 0x7f8d08062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.683+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f8d1c07ca60 con 0x7f8d240b0540 2023-12-19T09:03:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.683+0000 7f8d29ee1700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 msgr2=0x7f8d08062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.683+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 0x7f8d08062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:02.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.687+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8d1c04c910 con 0x7f8d240b0540 2023-12-19T09:03:02.751 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.746+0000 7f5e3e0eb000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:03:02.757 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.751+0000 7fb24e5ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:03:02.840 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:02.834+0000 7f5e3e0eb000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:03:02.888 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:02.883+0000 7fb24e5ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:03:02.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.887+0000 7f8d29ee1700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 msgr2=0x7f8d08062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:02.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.887+0000 7f8d29ee1700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 0x7f8d08062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:03:02.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.967+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d240ab220 con 0x7f8d240b0540 2023-12-19T09:03:02.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.967+0000 7f8d19ffb700 1 -- 172.21.15.111:0/4207912180 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4605 (secure 0 0 0) 0x7f8d1c01a090 con 0x7f8d240b0540 2023-12-19T09:03:02.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:02.974 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:02:41.093309+0000","modified":"2023-12-19T09:02:44.691422+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6834/4167714642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4167714642},{"type":"v1","addr":"172.21.15.111:6834","nonce":4167714642}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 msgr2=0x7f8d08062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 0x7f8d08062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 msgr2=0x7f8d24152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f8d240cc700 tx=0x7f8d2406e220 comp rx=0 tx=0).stop 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 shutdown_connections 2023-12-19T09:03:02.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8d08060730 0x7f8d08062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d240b00d0 0x7f8d241527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d240b0540 0x7f8d24152d00 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 --2- 172.21.15.111:0/4207912180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d10003a60 0x7f8d24154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 >> 172.21.15.111:0/4207912180 conn(0x7f8d24067bb0 msgr2=0x7f8d2406add0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 shutdown_connections 2023-12-19T09:03:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:02.971+0000 7f8d2aee3700 1 -- 172.21.15.111:0/4207912180 wait complete. 2023-12-19T09:03:02.979 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 72 2023-12-19T09:03:02.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.155+0000 7f78e9278700 1 Processor -- start 2023-12-19T09:03:03.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.155+0000 7f78e9278700 1 -- start start 2023-12-19T09:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e9278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f78e4068800 0x7f78e40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e9278700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e406c9f0 con 0x7f78e40c2980 2023-12-19T09:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e9278700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e40c7f20 con 0x7f78e413f890 2023-12-19T09:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e9278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e40c1c50 con 0x7f78e4068800 2023-12-19T09:03:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78dbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f78e40c2980 0x7f78e40680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54892/0 (socket says 172.21.15.111:54892) 2023-12-19T09:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78dbfff700 1 -- 172.21.15.111:0/1707289476 learned_addr learned my addr 172.21.15.111:0/1707289476 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e37fe700 1 --2- 172.21.15.111:0/1707289476 >> v2:172.21.15.111:3300/0 conn(0x7f78e4068800 0x7f78e40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427765453 0 0) 0x7f78e406c9f0 con 0x7f78e40c2980 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d0003570 con 0x7f78e40c2980 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643530231 0 0) 0x7f78e40c7f20 con 0x7f78e413f890 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e406c9f0 con 0x7f78e413f890 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e37fe700 1 -- 172.21.15.111:0/1707289476 >> v1:172.21.15.143:6789/0 conn(0x7f78e413f890 legacy=0x7f78e40c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e37fe700 1 -- 172.21.15.111:0/1707289476 >> v1:172.21.15.111:6789/0 conn(0x7f78e40c2980 legacy=0x7f78e40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793743631 0 0) 0x7f78d0003570 con 0x7f78e40c2980 2023-12-19T09:03:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e37fe700 1 -- 172.21.15.111:0/1707289476 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e40c8190 con 0x7f78e4068800 2023-12-19T09:03:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e37fe700 1 --2- 172.21.15.111:0/1707289476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e40aeba0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009e60 tx=0x7f78dc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82cfd45c29ed258 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc00e070 con 0x7f78e4068800 2023-12-19T09:03:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc00b7a0 con 0x7f78e4068800 2023-12-19T09:03:03.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.159+0000 7f78e27fc700 1 -- 172.21.15.111:0/1707289476 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc001470 con 0x7f78e4068800 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- 172.21.15.111:0/1707289476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 msgr2=0x7f78e40aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 --2- 172.21.15.111:0/1707289476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e40aeba0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009e60 tx=0x7f78dc009780 comp rx=0 tx=0).stop 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- 172.21.15.111:0/1707289476 shutdown_connections 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 --2- 172.21.15.111:0/1707289476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e40aeba0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- 172.21.15.111:0/1707289476 >> 172.21.15.111:0/1707289476 conn(0x7f78e40c8850 msgr2=0x7f78e40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- 172.21.15.111:0/1707289476 shutdown_connections 2023-12-19T09:03:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- 172.21.15.111:0/1707289476 wait complete. 2023-12-19T09:03:03.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 Processor -- start 2023-12-19T09:03:03.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- start start 2023-12-19T09:03:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e40c2980 0x7f78e40c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78e413f890 0x7f78e40c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e413ebc0 con 0x7f78e4068800 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4169220 con 0x7f78e40c2980 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e9278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40c4570 con 0x7f78e413f890 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41578/0 (socket says 172.21.15.111:41578) 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78e2ffd700 1 -- 172.21.15.111:0/2323488845 learned_addr learned my addr 172.21.15.111:0/2323488845 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.163+0000 7f78dbfff700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e40c2980 0x7f78e40c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e37fe700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78e413f890 0x7f78e40c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 -- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78e413f890 msgr2=0x7f78e40c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78e413f890 0x7f78e40c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 -- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e40c2980 msgr2=0x7f78e40c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e40c2980 0x7f78e40c3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 -- 172.21.15.111:0/2323488845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc009a20 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e2ffd700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c580 tx=0x7f78d4007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4010510 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4010670 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4010a30 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e40c4850 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40c4e10 con 0x7f78e4068800 2023-12-19T09:03:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.167+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e4071120 con 0x7f78e4068800 2023-12-19T09:03:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.171+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f78d401b400 con 0x7f78e4068800 2023-12-19T09:03:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.171+0000 7f78daffd700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 0x7f78c4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.171+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f78d407cb40 con 0x7f78e4068800 2023-12-19T09:03:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.171+0000 7f78dbfff700 1 -- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 msgr2=0x7f78c4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.171+0000 7f78dbfff700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 0x7f78c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:03.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.179+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f78d404c9f0 con 0x7f78e4068800 2023-12-19T09:03:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.351+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78e40642a0 con 0x7f78e4068800 2023-12-19T09:03:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.355+0000 7f78daffd700 1 -- 172.21.15.111:0/2323488845 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+16949 (secure 0 0 0) 0x7f78d4050010 con 0x7f78e4068800 2023-12-19T09:03:03.361 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:03.362 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":162,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:02:55.853471+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:02:39.584489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-12-19T09:02:40.099361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2824146656":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1422322069":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/3492177270":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1354315856":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6833/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6832/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/272833109":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1889918723":"2023-12-20T09:02:55.853401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 msgr2=0x7f78c4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 0x7f78c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 msgr2=0x7f78e416bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c580 tx=0x7f78d4007e10 comp rx=0 tx=0).stop 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 shutdown_connections 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f78c4060700 0x7f78c4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e40c2980 0x7f78e40c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78e413f890 0x7f78e40c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 --2- 172.21.15.111:0/2323488845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78e4068800 0x7f78e416bff0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 >> 172.21.15.111:0/2323488845 conn(0x7f78e40c8850 msgr2=0x7f78e40670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 shutdown_connections 2023-12-19T09:03:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.359+0000 7f78e9278700 1 -- 172.21.15.111:0/2323488845 wait complete. 2023-12-19T09:03:03.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T09:03:03.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.547+0000 7f63a86ae700 1 Processor -- start 2023-12-19T09:03:03.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.547+0000 7f63a86ae700 1 -- start start 2023-12-19T09:03:03.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a86ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f63a0158cc0 0x7f63a006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a86ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a006b0d0 con 0x7f63a006dc00 2023-12-19T09:03:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a86ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a00b4790 con 0x7f63a00b3870 2023-12-19T09:03:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a86ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a006d710 con 0x7f63a0158cc0 2023-12-19T09:03:03.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a644a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f63a0158cc0 0x7f63a006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a5c49700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f63a006dc00 0x7f63a006b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54900/0 (socket says 172.21.15.111:54900) 2023-12-19T09:03:03.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.551+0000 7f63a5c49700 1 -- 172.21.15.111:0/3005489634 learned_addr learned my addr 172.21.15.111:0/3005489634 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:03.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225699924 0 0) 0x7f63a00b4790 con 0x7f63a00b3870 2023-12-19T09:03:03.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c003570 con 0x7f63a00b3870 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393902546 0 0) 0x7f63a006b0d0 con 0x7f63a006dc00 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a00b4790 con 0x7f63a006dc00 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a644a700 1 -- 172.21.15.111:0/3005489634 >> v1:172.21.15.143:6789/0 conn(0x7f63a00b3870 legacy=0x7f63a00b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a644a700 1 -- 172.21.15.111:0/3005489634 >> v1:172.21.15.111:6789/0 conn(0x7f63a006dc00 legacy=0x7f63a006b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a644a700 1 -- 172.21.15.111:0/3005489634 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a006a7e0 con 0x7f63a0158cc0 2023-12-19T09:03:03.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a644a700 1 --2- 172.21.15.111:0/3005489634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a0158cc0 0x7f63a006d1d0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098e0 tx=0x7f639c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9833faa8184cd43a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011630 con 0x7f63a0158cc0 2023-12-19T09:03:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c011790 con 0x7f63a0158cc0 2023-12-19T09:03:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a5448700 1 -- 172.21.15.111:0/3005489634 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c011b30 con 0x7f63a0158cc0 2023-12-19T09:03:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 -- 172.21.15.111:0/3005489634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a0158cc0 msgr2=0x7f63a006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 --2- 172.21.15.111:0/3005489634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a0158cc0 0x7f63a006d1d0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f639c0098e0 tx=0x7f639c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 -- 172.21.15.111:0/3005489634 shutdown_connections 2023-12-19T09:03:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 --2- 172.21.15.111:0/3005489634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a0158cc0 0x7f63a006d1d0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 -- 172.21.15.111:0/3005489634 >> 172.21.15.111:0/3005489634 conn(0x7f63a0067bb0 msgr2=0x7f63a00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 -- 172.21.15.111:0/3005489634 shutdown_connections 2023-12-19T09:03:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.555+0000 7f63a86ae700 1 -- 172.21.15.111:0/3005489634 wait complete. 2023-12-19T09:03:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 Processor -- start 2023-12-19T09:03:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 -- start start 2023-12-19T09:03:03.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 0x7f63a0155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63a0158cc0 0x7f63a015ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a016a400 con 0x7f63a006dc00 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0153680 con 0x7f63a00b3870 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a86ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0153350 con 0x7f63a0158cc0 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41600/0 (socket says 172.21.15.111:41600) 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 -- 172.21.15.111:0/246861158 learned_addr learned my addr 172.21.15.111:0/246861158 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a5c49700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 0x7f63a0155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a6c4b700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63a0158cc0 0x7f63a015ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 -- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63a0158cc0 msgr2=0x7f63a015ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63a0158cc0 0x7f63a015ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 -- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 msgr2=0x7f63a0155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 0x7f63a0155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 -- 172.21.15.111:0/246861158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009420 con 0x7f63a006dc00 2023-12-19T09:03:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a5c49700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 0x7f63a0155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63a644a700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f63a006eb60 tx=0x7f639c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.559+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c01f070 con 0x7f63a006dc00 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.563+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c011900 con 0x7f63a006dc00 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.563+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a0156170 con 0x7f63a006dc00 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.563+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c023410 con 0x7f63a006dc00 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.563+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a0156730 con 0x7f63a006dc00 2023-12-19T09:03:03.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.563+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a0071120 con 0x7f63a006dc00 2023-12-19T09:03:03.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.567+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 74) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f639c02c3b0 con 0x7f63a006dc00 2023-12-19T09:03:03.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.567+0000 7f63967fc700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 0x7f6384062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.567+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6917+0+0 (secure 0 0 0) 0x7f639c07e180 con 0x7f63a006dc00 2023-12-19T09:03:03.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.567+0000 7f63a5c49700 1 -- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 msgr2=0x7f6384062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:03.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.567+0000 7f63a5c49700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 0x7f6384062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:03.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.571+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f639c04e030 con 0x7f63a006dc00 2023-12-19T09:03:03.598 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.594+0000 7f5e3e0eb000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:03:03.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.611+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 75) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f639c04e400 con 0x7f63a006dc00 2023-12-19T09:03:03.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.611+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 msgr2=0x7f6384062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:03.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.611+0000 7f63967fc700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 0x7f6384062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.669 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:03.663+0000 7fb24e5ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:03:03.707 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e0a76c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.708 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e0a76c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.708 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e0a76c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.708 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e0a76c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.708 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e0a76c700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e07766700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e07766700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e07766700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e07766700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:03.702+0000 7f5e07766700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:03.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.787+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f63a006aa80 con 0x7f63a006dc00 2023-12-19T09:03:03.794 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T09:03:03.787+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T09:03:03.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63967fc700 1 -- 172.21.15.111:0/246861158 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f639c0801a0 con 0x7f63a006dc00 2023-12-19T09:03:03.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 msgr2=0x7f63a0151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f63a006eb60 tx=0x7f639c017000 comp rx=0 tx=0).stop 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 shutdown_connections 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6384060700 0x7f6384062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63a00b3870 0x7f63a0155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63a0158cc0 0x7f63a015ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 --2- 172.21.15.111:0/246861158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a006dc00 0x7f63a0151080 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 >> 172.21.15.111:0/246861158 conn(0x7f63a0067bb0 msgr2=0x7f63a00aa040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 shutdown_connections 2023-12-19T09:03:03.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.799+0000 7f63a86ae700 1 -- 172.21.15.111:0/246861158 wait complete. 2023-12-19T09:03:03.809 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T09:03:03.813 INFO:tasks.ceph.mds.a.smithi111.stderr: -2681> 2023-12-19T09:02:43.532+0000 7fc8a43f1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:03:03.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T09:03:03.858 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:03:03.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 Processor -- start 2023-12-19T09:03:03.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 -- start start 2023-12-19T09:03:03.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23d415c520 0x7f23d40a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d40c6ab0 con 0x7f23d40ca240 2023-12-19T09:03:03.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d406b240 con 0x7f23d415c8f0 2023-12-19T09:03:03.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23dbcae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d406b520 con 0x7f23d415c520 2023-12-19T09:03:03.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.975+0000 7f23da24b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f23d40ca240 0x7f23d40ca630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54910/0 (socket says 172.21.15.111:54910) 2023-12-19T09:03:03.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23da24b700 1 -- 172.21.15.111:0/1482377211 learned_addr learned my addr 172.21.15.111:0/1482377211 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:03.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d9a4a700 1 --2- 172.21.15.111:0/1482377211 >> v2:172.21.15.143:3300/0 conn(0x7f23d415c520 0x7f23d40a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405792850 0 0) 0x7f23d40c6ab0 con 0x7f23d40ca240 2023-12-19T09:03:03.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c0003570 con 0x7f23d40ca240 2023-12-19T09:03:03.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d9a4a700 1 -- 172.21.15.111:0/1482377211 >> v1:172.21.15.143:6789/0 conn(0x7f23d415c8f0 legacy=0x7f23d406a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918160758 0 0) 0x7f23c0003570 con 0x7f23d40ca240 2023-12-19T09:03:03.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d9a4a700 1 -- 172.21.15.111:0/1482377211 >> v1:172.21.15.111:6789/0 conn(0x7f23d40ca240 legacy=0x7f23d40ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d9a4a700 1 -- 172.21.15.111:0/1482377211 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d40c6d50 con 0x7f23d415c520 2023-12-19T09:03:03.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d9a4a700 1 --2- 172.21.15.111:0/1482377211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40a8850 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f23d00098e0 tx=0x7f23d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=430ba5c6d733ca3f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0017f0 con 0x7f23d415c520 2023-12-19T09:03:03.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d0011420 con 0x7f23d415c520 2023-12-19T09:03:03.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23d8a48700 1 -- 172.21.15.111:0/1482377211 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d00117c0 con 0x7f23d415c520 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 -- 172.21.15.111:0/1482377211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 msgr2=0x7f23d40a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 --2- 172.21.15.111:0/1482377211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40a8850 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f23d00098e0 tx=0x7f23d00091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 -- 172.21.15.111:0/1482377211 shutdown_connections 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 --2- 172.21.15.111:0/1482377211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40a8850 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 -- 172.21.15.111:0/1482377211 >> 172.21.15.111:0/1482377211 conn(0x7f23d4067bb0 msgr2=0x7f23d40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 -- 172.21.15.111:0/1482377211 shutdown_connections 2023-12-19T09:03:03.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.979+0000 7f23dbcae700 1 -- 172.21.15.111:0/1482377211 wait complete. 2023-12-19T09:03:03.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 Processor -- start 2023-12-19T09:03:03.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 -- start start 2023-12-19T09:03:03.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d415c8f0 0x7f23d4157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40b0460 con 0x7f23d40ca240 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40af850 con 0x7f23d415c520 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23dbcae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d40afb30 con 0x7f23d415c8f0 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23d9a4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23d9a4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41622/0 (socket says 172.21.15.111:41622) 2023-12-19T09:03:03.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.983+0000 7f23d9a4a700 1 -- 172.21.15.111:0/3336663989 learned_addr learned my addr 172.21.15.111:0/3336663989 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:03.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23da24b700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d415c8f0 0x7f23d4157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 -- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d415c8f0 msgr2=0x7f23d4157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d415c8f0 0x7f23d4157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 -- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 msgr2=0x7f23d40c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9249700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 -- 172.21.15.111:0/3336663989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0009420 con 0x7f23d40ca240 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9249700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23d9a4a700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f23d00091e0 tx=0x7f23d0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d001cc90 con 0x7f23d40ca240 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d001cdf0 con 0x7f23d40ca240 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d40afe10 con 0x7f23d40ca240 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d001a6a0 con 0x7f23d40ca240 2023-12-19T09:03:03.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d40af560 con 0x7f23d40ca240 2023-12-19T09:03:03.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.987+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d4071120 con 0x7f23d40ca240 2023-12-19T09:03:03.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.991+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 75) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f23d001aa50 con 0x7f23d40ca240 2023-12-19T09:03:03.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.991+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 7304+0+0 (secure 0 0 0) 0x7f23d007e400 con 0x7f23d40ca240 2023-12-19T09:03:04.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:03.999+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23d004e770 con 0x7f23d40ca240 2023-12-19T09:03:04.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.199+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f23d406bdc0 con 0x7f23d40ca240 2023-12-19T09:03:04.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.619+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 76) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f23d001ad00 con 0x7f23d40ca240 2023-12-19T09:03:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.619+0000 7f23c9ffb700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 0x7f23b8068d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.619+0000 7f23d9249700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 0x7f23b8068d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:04.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.623+0000 7f23d9249700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 0x7f23b8068d30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f23d40b14f0 tx=0x7f23c4009040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:04.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.799+0000 7f23c9ffb700 1 -- 172.21.15.111:0/3336663989 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7f23d0054910 con 0x7f23d40ca240 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.803+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 msgr2=0x7f23b8068d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.803+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 0x7f23b8068d30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f23d40b14f0 tx=0x7f23c4009040 comp rx=0 tx=0).stop 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.803+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 msgr2=0x7f23d40ad2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.803+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f23d00091e0 tx=0x7f23d0016000 comp rx=0 tx=0).stop 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 shutdown_connections 2023-12-19T09:03:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f23b8066950 0x7f23b8068d30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d415c520 0x7f23d40c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d415c8f0 0x7f23d4157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 --2- 172.21.15.111:0/3336663989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23d40ca240 0x7f23d40ad2d0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 >> 172.21.15.111:0/3336663989 conn(0x7f23d4067bb0 msgr2=0x7f23d415a030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 shutdown_connections 2023-12-19T09:03:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:04.807+0000 7f23dbcae700 1 -- 172.21.15.111:0/3336663989 wait complete. 2023-12-19T09:03:04.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T09:03:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.043+0000 7f4edd66b700 1 Processor -- start 2023-12-19T09:03:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.043+0000 7f4edd66b700 1 -- start start 2023-12-19T09:03:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edd66b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ed815cf90 0x7f4ed806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edd66b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ed806dde0 0x7f4ed80af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edd66b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed806b550 con 0x7f4ed815d360 2023-12-19T09:03:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edd66b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed806af40 con 0x7f4ed815cf90 2023-12-19T09:03:05.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edd66b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80af5a0 con 0x7f4ed806dde0 2023-12-19T09:03:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ed815cf90 0x7f4ed806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4ed815d360 0x7f4ed80ae950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54918/0 (socket says 172.21.15.111:54918) 2023-12-19T09:03:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed77fe700 1 -- 172.21.15.111:0/2305676693 learned_addr learned my addr 172.21.15.111:0/2305676693 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:05.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4edce6a700 1 --2- 172.21.15.111:0/2305676693 >> v2:172.21.15.143:3300/0 conn(0x7f4ed806dde0 0x7f4ed80af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2305676693 >> v2:172.21.15.143:3300/0 conn(0x7f4ed806dde0 msgr2=0x7f4ed80af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 --2- 172.21.15.111:0/2305676693 >> v2:172.21.15.143:3300/0 conn(0x7f4ed806dde0 0x7f4ed80af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2305676693 >> v1:172.21.15.111:6789/0 conn(0x7f4ed815d360 legacy=0x7f4ed80ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2305676693 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80ae0c0 con 0x7f4ed815cf90 2023-12-19T09:03:05.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed6ffd700 1 -- 172.21.15.111:0/2305676693 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668716472 0 0) 0x7f4ed806b550 con 0x7f4ed815d360 2023-12-19T09:03:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed7fff700 1 --2- 172.21.15.111:0/2305676693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed815cf90 0x7f4ed806b9e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4ed806a000 tx=0x7f4ec80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f377f97ac5774ad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed6ffd700 1 -- 172.21.15.111:0/2305676693 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8011630 con 0x7f4ed815cf90 2023-12-19T09:03:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed6ffd700 1 -- 172.21.15.111:0/2305676693 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec8011790 con 0x7f4ed815cf90 2023-12-19T09:03:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.047+0000 7f4ed6ffd700 1 -- 172.21.15.111:0/2305676693 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8011b30 con 0x7f4ed815cf90 2023-12-19T09:03:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 -- 172.21.15.111:0/2305676693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed815cf90 msgr2=0x7f4ed806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2305676693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed815cf90 0x7f4ed806b9e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4ed806a000 tx=0x7f4ec80091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 -- 172.21.15.111:0/2305676693 shutdown_connections 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2305676693 >> v2:172.21.15.143:3300/0 conn(0x7f4ed806dde0 0x7f4ed80af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2305676693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed815cf90 0x7f4ed806b9e0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 -- 172.21.15.111:0/2305676693 >> 172.21.15.111:0/2305676693 conn(0x7f4ed8067bb0 msgr2=0x7f4ed80c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 -- 172.21.15.111:0/2305676693 shutdown_connections 2023-12-19T09:03:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 -- 172.21.15.111:0/2305676693 wait complete. 2023-12-19T09:03:05.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.051+0000 7f4edd66b700 1 Processor -- start 2023-12-19T09:03:05.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 -- start start 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 0x7f4ed816e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed815d360 0x7f4ed81467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed816f900 con 0x7f4ed806dde0 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed816ea50 con 0x7f4ed815cf90 2023-12-19T09:03:05.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edd66b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed816ecf0 con 0x7f4ed815d360 2023-12-19T09:03:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41640/0 (socket says 172.21.15.111:41640) 2023-12-19T09:03:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2582577703 learned_addr learned my addr 172.21.15.111:0/2582577703 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4edce6a700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed815d360 0x7f4ed81467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed77fe700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 0x7f4ed816e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed815d360 msgr2=0x7f4ed81467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed815d360 0x7f4ed81467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 msgr2=0x7f4ed816e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 0x7f4ed816e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 -- 172.21.15.111:0/2582577703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec8009420 con 0x7f4ed806dde0 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed77fe700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 0x7f4ed816e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.055+0000 7f4ed7fff700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80699a0 tx=0x7f4ec8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:05.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8011b10 con 0x7f4ed806dde0 2023-12-19T09:03:05.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec80253f0 con 0x7f4ed806dde0 2023-12-19T09:03:05.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8025790 con 0x7f4ed806dde0 2023-12-19T09:03:05.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed816efd0 con 0x7f4ed806dde0 2023-12-19T09:03:05.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed816f590 con 0x7f4ed806dde0 2023-12-19T09:03:05.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed804be40 con 0x7f4ed806dde0 2023-12-19T09:03:05.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.059+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 76) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f4ec8025b80 con 0x7f4ed806dde0 2023-12-19T09:03:05.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.063+0000 7f4ebffff700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 0x7f4ec4062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:05.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.063+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 7304+0+0 (secure 0 0 0) 0x7f4ec807e430 con 0x7f4ed806dde0 2023-12-19T09:03:05.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.063+0000 7f4ed77fe700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 0x7f4ec4062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:05.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.063+0000 7f4ed77fe700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 0x7f4ec4062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4ed815c950 tx=0x7f4ecc009040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:05.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.071+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4ec804e5c0 con 0x7f4ed806dde0 2023-12-19T09:03:05.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:05.243+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ed8070790 con 0x7f4ed806dde0 2023-12-19T09:03:06.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.155+0000 7f4ebffff700 1 -- 172.21.15.111:0/2582577703 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v165) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4ec8051be0 con 0x7f4ed806dde0 2023-12-19T09:03:06.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 msgr2=0x7f4ec4062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 0x7f4ec4062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4ed815c950 tx=0x7f4ecc009040 comp rx=0 tx=0).stop 2023-12-19T09:03:06.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 msgr2=0x7f4ed816df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80699a0 tx=0x7f4ec8017000 comp rx=0 tx=0).stop 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 shutdown_connections 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4ec40603a0 0x7f4ec4062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed815cf90 0x7f4ed816e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed815d360 0x7f4ed81467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 --2- 172.21.15.111:0/2582577703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed806dde0 0x7f4ed816df20 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 >> 172.21.15.111:0/2582577703 conn(0x7f4ed8067bb0 msgr2=0x7f4ed80caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:06.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 shutdown_connections 2023-12-19T09:03:06.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.159+0000 7f4edd66b700 1 -- 172.21.15.111:0/2582577703 wait complete. 2023-12-19T09:03:06.170 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T09:03:06.188 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T09:03:06.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.355+0000 7f5c8c42e700 1 Processor -- start 2023-12-19T09:03:06.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.355+0000 7f5c8c42e700 1 -- start start 2023-12-19T09:03:06.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8c42e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8406a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:06.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8c42e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c840c6560 con 0x7f5c8415d720 2023-12-19T09:03:06.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8c42e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c840c70e0 con 0x7f5c8415d350 2023-12-19T09:03:06.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8c42e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840cc900 con 0x7f5c84161da0 2023-12-19T09:03:06.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8a9cb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5c8415d720 0x7f5c840abc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54926/0 (socket says 172.21.15.111:54926) 2023-12-19T09:03:06.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8a9cb700 1 -- 172.21.15.111:0/4151241905 learned_addr learned my addr 172.21.15.111:0/4151241905 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:06.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c899c9700 1 --2- 172.21.15.111:0/4151241905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8406a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:06.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c8a1ca700 1 --1- 172.21.15.111:0/4151241905 >> v1:172.21.15.143:6789/0 conn(0x7f5c8415d350 0x7f5c840cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56202/0 (socket says 172.21.15.111:56202) 2023-12-19T09:03:06.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c891c8700 1 -- 172.21.15.111:0/4151241905 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428566608 0 0) 0x7f5c840c70e0 con 0x7f5c8415d350 2023-12-19T09:03:06.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c899c9700 1 -- 172.21.15.111:0/4151241905 >> v1:172.21.15.143:6789/0 conn(0x7f5c8415d350 legacy=0x7f5c840cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c899c9700 1 -- 172.21.15.111:0/4151241905 >> v1:172.21.15.111:6789/0 conn(0x7f5c8415d720 legacy=0x7f5c840abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c899c9700 1 -- 172.21.15.111:0/4151241905 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840c67b0 con 0x7f5c84161da0 2023-12-19T09:03:06.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.359+0000 7f5c899c9700 1 --2- 172.21.15.111:0/4151241905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8406a6a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009910 tx=0x7f5c740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97249790ea25697e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:06.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c891c8700 1 -- 172.21.15.111:0/4151241905 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400b760 con 0x7f5c84161da0 2023-12-19T09:03:06.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c891c8700 1 -- 172.21.15.111:0/4151241905 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400b8c0 con 0x7f5c84161da0 2023-12-19T09:03:06.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c891c8700 1 -- 172.21.15.111:0/4151241905 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400bc60 con 0x7f5c84161da0 2023-12-19T09:03:06.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 -- 172.21.15.111:0/4151241905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 msgr2=0x7f5c8406a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/4151241905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8406a6a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009910 tx=0x7f5c740091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 -- 172.21.15.111:0/4151241905 shutdown_connections 2023-12-19T09:03:06.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/4151241905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8406a6a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 -- 172.21.15.111:0/4151241905 >> 172.21.15.111:0/4151241905 conn(0x7f5c84067bb0 msgr2=0x7f5c840cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:06.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 -- 172.21.15.111:0/4151241905 shutdown_connections 2023-12-19T09:03:06.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.363+0000 7f5c8c42e700 1 -- 172.21.15.111:0/4151241905 wait complete. 2023-12-19T09:03:06.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 Processor -- start 2023-12-19T09:03:06.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 -- start start 2023-12-19T09:03:06.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 0x7f5c840b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:06.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:06.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8415dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:06.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c840aee40 con 0x7f5c8415d720 2023-12-19T09:03:06.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8415fce0 con 0x7f5c8415d350 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8c42e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8415ffc0 con 0x7f5c84161da0 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c899c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c899c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41650/0 (socket says 172.21.15.111:41650) 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c899c9700 1 -- 172.21.15.111:0/198010750 learned_addr learned my addr 172.21.15.111:0/198010750 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8a1ca700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 0x7f5c840b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:06.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.367+0000 7f5c8a9cb700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8415dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 -- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 msgr2=0x7f5c8415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8415dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 -- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 msgr2=0x7f5c840b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 0x7f5c840b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 -- 172.21.15.111:0/198010750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c74009420 con 0x7f5c8415d720 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c8a1ca700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 0x7f5c840b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c899c9700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f5c740030c0 tx=0x7f5c74016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74011d20 con 0x7f5c8415d720 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7401c460 con 0x7f5c8415d720 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c840aead0 con 0x7f5c8415d720 2023-12-19T09:03:06.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74011ea0 con 0x7f5c8415d720 2023-12-19T09:03:06.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c84152110 con 0x7f5c8415d720 2023-12-19T09:03:06.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.371+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c840aefb0 con 0x7f5c8415d720 2023-12-19T09:03:06.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.375+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5c7401cab0 con 0x7f5c8415d720 2023-12-19T09:03:06.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.375+0000 7f5c7a7fc700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 0x7f5c68062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:06.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.375+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6866+0+0 (secure 0 0 0) 0x7f5c7407f040 con 0x7f5c8415d720 2023-12-19T09:03:06.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.375+0000 7f5c8a1ca700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 0x7f5c68062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:06.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.379+0000 7f5c8a1ca700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 0x7f5c68062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c800011e0 tx=0x7f5c80007c90 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:06.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.383+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c7404f090 con 0x7f5c8415d720 2023-12-19T09:03:06.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:06.559+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5c84071120 con 0x7f5c8415d720 2023-12-19T09:03:07.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.171+0000 7f5c7a7fc700 1 -- 172.21.15.111:0/198010750 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v166) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5c740526b0 con 0x7f5c8415d720 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.171+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 msgr2=0x7f5c68062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.171+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 0x7f5c68062b50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c800011e0 tx=0x7f5c80007c90 comp rx=0 tx=0).stop 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.171+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 msgr2=0x7f5c84150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.171+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f5c740030c0 tx=0x7f5c74016000 comp rx=0 tx=0).stop 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 shutdown_connections 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5c680606a0 0x7f5c68062b50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c84161da0 0x7f5c8415dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c8415d720 0x7f5c84150ea0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 --2- 172.21.15.111:0/198010750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c8415d350 0x7f5c840b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 >> 172.21.15.111:0/198010750 conn(0x7f5c84067bb0 msgr2=0x7f5c841627f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:07.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 shutdown_connections 2023-12-19T09:03:07.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.175+0000 7f5c8c42e700 1 -- 172.21.15.111:0/198010750 wait complete. 2023-12-19T09:03:07.183 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T09:03:07.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T09:03:07.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 Processor -- start 2023-12-19T09:03:07.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 -- start start 2023-12-19T09:03:07.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f99d406c9f0 0x7f99d406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d406a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d406aa30 con 0x7f99d41588d0 2023-12-19T09:03:07.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d406b870 con 0x7f99d406c9f0 2023-12-19T09:03:07.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99da1cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d406ac60 con 0x7f99d406d9f0 2023-12-19T09:03:07.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99d91ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f99d406c9f0 0x7f99d406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99d91ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f99d406c9f0 0x7f99d406cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41652/0 (socket says 172.21.15.111:41652) 2023-12-19T09:03:07.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.359+0000 7f99d91ca700 1 -- 172.21.15.111:0/74029735 learned_addr learned my addr 172.21.15.111:0/74029735 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:07.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d91ca700 1 -- 172.21.15.111:0/74029735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 msgr2=0x7f99d406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:07.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d89c9700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d406a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d91ca700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d406a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d91ca700 1 -- 172.21.15.111:0/74029735 >> v1:172.21.15.143:6789/0 conn(0x7f99d41588d0 legacy=0x7f99d4158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d91ca700 1 -- 172.21.15.111:0/74029735 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d406af40 con 0x7f99d406c9f0 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d89c9700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99d91ca700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d406cdc0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009f20 tx=0x7f99d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e8fcd0d8ba6dc1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99cbfff700 1 -- 172.21.15.111:0/74029735 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000e070 con 0x7f99d406c9f0 2023-12-19T09:03:07.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99cbfff700 1 -- 172.21.15.111:0/74029735 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000ba40 con 0x7f99d406c9f0 2023-12-19T09:03:07.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99cbfff700 1 -- 172.21.15.111:0/74029735 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d00092e0 con 0x7f99d406c9f0 2023-12-19T09:03:07.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 -- 172.21.15.111:0/74029735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 msgr2=0x7f99d406cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d406cdc0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009f20 tx=0x7f99d0009840 comp rx=0 tx=0).stop 2023-12-19T09:03:07.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 -- 172.21.15.111:0/74029735 shutdown_connections 2023-12-19T09:03:07.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 --2- 172.21.15.111:0/74029735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d406cdc0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 -- 172.21.15.111:0/74029735 >> 172.21.15.111:0/74029735 conn(0x7f99d4067bb0 msgr2=0x7f99d40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:07.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 -- 172.21.15.111:0/74029735 shutdown_connections 2023-12-19T09:03:07.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.363+0000 7f99da1cc700 1 -- 172.21.15.111:0/74029735 wait complete. 2023-12-19T09:03:07.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 Processor -- start 2023-12-19T09:03:07.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 -- start start 2023-12-19T09:03:07.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d4169520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d41588d0 0x7f99d40ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d416a5e0 con 0x7f99d406c9f0 2023-12-19T09:03:07.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41530b0 con 0x7f99d41588d0 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.367+0000 7f99da1cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4144dd0 con 0x7f99d406d9f0 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41668/0 (socket says 172.21.15.111:41668) 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 -- 172.21.15.111:0/3451864879 learned_addr learned my addr 172.21.15.111:0/3451864879 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d89c9700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d4169520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d99cb700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d41588d0 0x7f99d40ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 -- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 msgr2=0x7f99d4169520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d4169520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 -- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d41588d0 msgr2=0x7f99d40ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d41588d0 0x7f99d40ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 -- 172.21.15.111:0/3451864879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0009ae0 con 0x7f99d406c9f0 2023-12-19T09:03:07.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99d91ca700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f99d4069b60 tx=0x7f99d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0012aa0 con 0x7f99d406c9f0 2023-12-19T09:03:07.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0012c20 con 0x7f99d406c9f0 2023-12-19T09:03:07.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d001d420 con 0x7f99d406c9f0 2023-12-19T09:03:07.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d4159730 con 0x7f99d406c9f0 2023-12-19T09:03:07.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d416a020 con 0x7f99d406c9f0 2023-12-19T09:03:07.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.371+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d4159500 con 0x7f99d406c9f0 2023-12-19T09:03:07.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.375+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f99d001d850 con 0x7f99d406c9f0 2023-12-19T09:03:07.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.375+0000 7f99c97fa700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 0x7f99b8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.375+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f99d007e490 con 0x7f99d406c9f0 2023-12-19T09:03:07.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.375+0000 7f99d89c9700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 0x7f99b8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.379+0000 7f99d89c9700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 0x7f99b8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f99d40cae10 tx=0x7f99c4006f90 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.383+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f99d004e570 con 0x7f99d406c9f0 2023-12-19T09:03:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.547+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f99d404be40 con 0x7f99d406c9f0 2023-12-19T09:03:07.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.555+0000 7f99c97fa700 1 -- 172.21.15.111:0/3451864879 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v55) v1 ==== 120+0+0 (secure 0 0 0) 0x7f99d0051b90 con 0x7f99d406c9f0 2023-12-19T09:03:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 msgr2=0x7f99b8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 0x7f99b8062b20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f99d40cae10 tx=0x7f99c4006f90 comp rx=0 tx=0).stop 2023-12-19T09:03:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 msgr2=0x7f99d4144890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f99d4069b60 tx=0x7f99d0017000 comp rx=0 tx=0).stop 2023-12-19T09:03:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 shutdown_connections 2023-12-19T09:03:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f99b8060670 0x7f99b8062b20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d41588d0 0x7f99d40ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d406d9f0 0x7f99d4169520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 --2- 172.21.15.111:0/3451864879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99d406c9f0 0x7f99d4144890 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 >> 172.21.15.111:0/3451864879 conn(0x7f99d4067bb0 msgr2=0x7f99d41538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 shutdown_connections 2023-12-19T09:03:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.559+0000 7f99da1cc700 1 -- 172.21.15.111:0/3451864879 wait complete. 2023-12-19T09:03:07.589 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T09:03:07.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.743+0000 7f0e0c8ad700 1 Processor -- start 2023-12-19T09:03:07.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.743+0000 7f0e0c8ad700 1 -- start start 2023-12-19T09:03:07.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0c8ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0e08154780 0x7f0e0815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0c8ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0e0806baa0 0x7f0e08157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0c8ad700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0815ae30 con 0x7f0e080adab0 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0c8ad700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e080ae5b0 con 0x7f0e0806baa0 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0c8ad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e080abf70 con 0x7f0e08154780 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0e0806baa0 0x7f0e08157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0759e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0e080adab0 0x7f0e080adea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54940/0 (socket says 172.21.15.111:54940) 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0759e700 1 -- 172.21.15.111:0/219732772 learned_addr learned my addr 172.21.15.111:0/219732772 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e06d9d700 1 --2- 172.21.15.111:0/219732772 >> v2:172.21.15.143:3300/0 conn(0x7f0e08154780 0x7f0e0815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 -- 172.21.15.111:0/219732772 >> v2:172.21.15.143:3300/0 conn(0x7f0e08154780 msgr2=0x7f0e0815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:07.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 --2- 172.21.15.111:0/219732772 >> v2:172.21.15.143:3300/0 conn(0x7f0e08154780 0x7f0e0815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 -- 172.21.15.111:0/219732772 >> v1:172.21.15.111:6789/0 conn(0x7f0e080adab0 legacy=0x7f0e080adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 -- 172.21.15.111:0/219732772 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08154b50 con 0x7f0e0806baa0 2023-12-19T09:03:07.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.747+0000 7f0e0659c700 1 --2- 172.21.15.111:0/219732772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e0806baa0 0x7f0e08157b60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0e0806a3f0 tx=0x7f0dfc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe6b180c585bb42 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e05d9b700 1 -- 172.21.15.111:0/219732772 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc011630 con 0x7f0e0806baa0 2023-12-19T09:03:07.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e05d9b700 1 -- 172.21.15.111:0/219732772 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc011790 con 0x7f0e0806baa0 2023-12-19T09:03:07.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e05d9b700 1 -- 172.21.15.111:0/219732772 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc011b30 con 0x7f0e0806baa0 2023-12-19T09:03:07.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/219732772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e0806baa0 msgr2=0x7f0e08157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/219732772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e0806baa0 0x7f0e08157b60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0e0806a3f0 tx=0x7f0dfc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/219732772 shutdown_connections 2023-12-19T09:03:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/219732772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e0806baa0 0x7f0e08157b60 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/219732772 >> v2:172.21.15.143:3300/0 conn(0x7f0e08154780 0x7f0e0815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/219732772 >> 172.21.15.111:0/219732772 conn(0x7f0e08067b50 msgr2=0x7f0e080cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/219732772 shutdown_connections 2023-12-19T09:03:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.751+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/219732772 wait complete. 2023-12-19T09:03:07.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 Processor -- start 2023-12-19T09:03:07.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 -- start start 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0806baa0 0x7f0e080c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e08154780 0x7f0e08150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08169400 con 0x7f0e080adab0 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c68c0 con 0x7f0e0806baa0 2023-12-19T09:03:07.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0c8ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080c6ba0 con 0x7f0e08154780 2023-12-19T09:03:07.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41690/0 (socket says 172.21.15.111:41690) 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 -- 172.21.15.111:0/345741842 learned_addr learned my addr 172.21.15.111:0/345741842 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e06d9d700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0806baa0 0x7f0e080c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0759e700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e08154780 0x7f0e08150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 -- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e08154780 msgr2=0x7f0e08150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e08154780 0x7f0e08150410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 -- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0806baa0 msgr2=0x7f0e080c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0806baa0 0x7f0e080c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.755+0000 7f0e0659c700 1 -- 172.21.15.111:0/345741842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc009420 con 0x7f0e080adab0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0e0659c700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc013000 tx=0x7f0dfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc000d10 con 0x7f0e080adab0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc011b10 con 0x7f0e080adab0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e080c6e80 con 0x7f0e080adab0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc025640 con 0x7f0e080adab0 2023-12-19T09:03:07.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e08150950 con 0x7f0e080adab0 2023-12-19T09:03:07.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.759+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e080c70c0 con 0x7f0e080adab0 2023-12-19T09:03:07.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.763+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0dfc025a30 con 0x7f0e080adab0 2023-12-19T09:03:07.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.763+0000 7f0deeffd700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 0x7f0df4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:07.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.763+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f0dfc07e490 con 0x7f0e080adab0 2023-12-19T09:03:07.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.763+0000 7f0e06d9d700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 0x7f0df4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:07.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.767+0000 7f0e06d9d700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 0x7f0df4062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d20 tx=0x7f0df80091a0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:07.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.771+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0e080c70c0 con 0x7f0e080adab0 2023-12-19T09:03:07.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.939+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0e08169830 con 0x7f0e080adab0 2023-12-19T09:03:07.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.951+0000 7f0deeffd700 1 -- 172.21.15.111:0/345741842 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0dfc04e5e0 con 0x7f0e080adab0 2023-12-19T09:03:07.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 msgr2=0x7f0df4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 0x7f0df4062b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d20 tx=0x7f0df80091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 msgr2=0x7f0e080c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:07.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc013000 tx=0x7f0dfc017000 comp rx=0 tx=0).stop 2023-12-19T09:03:07.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 shutdown_connections 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0df4060670 0x7f0df4062b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e08154780 0x7f0e08150410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e080adab0 0x7f0e080c7ab0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 --2- 172.21.15.111:0/345741842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e0806baa0 0x7f0e080c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 >> 172.21.15.111:0/345741842 conn(0x7f0e08067b50 msgr2=0x7f0e081515b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 shutdown_connections 2023-12-19T09:03:07.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:07.955+0000 7f0e0c8ad700 1 -- 172.21.15.111:0/345741842 wait complete. 2023-12-19T09:03:07.983 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T09:03:08.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.147+0000 7f3112c78700 1 Processor -- start 2023-12-19T09:03:08.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3112c78700 1 -- start start 2023-12-19T09:03:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3112c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 0x7f310c0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3112c78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0c9db0 con 0x7f310c06df10 2023-12-19T09:03:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3112c78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0cb2e0 con 0x7f310c0c63f0 2023-12-19T09:03:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3112c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0ca950 con 0x7f310c06db40 2023-12-19T09:03:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3111215700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f310c06df10 0x7f310c159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54954/0 (socket says 172.21.15.111:54954) 2023-12-19T09:03:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3111215700 1 -- 172.21.15.111:0/3857243360 learned_addr learned my addr 172.21.15.111:0/3857243360 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f3110a14700 1 --2- 172.21.15.111:0/3857243360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 0x7f310c0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f310b7fe700 1 -- 172.21.15.111:0/3857243360 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149038206 0 0) 0x7f310c0c9db0 con 0x7f310c06df10 2023-12-19T09:03:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.151+0000 7f310b7fe700 1 -- 172.21.15.111:0/3857243360 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc003570 con 0x7f310c06df10 2023-12-19T09:03:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3110a14700 1 -- 172.21.15.111:0/3857243360 >> v1:172.21.15.143:6789/0 conn(0x7f310c0c63f0 legacy=0x7f310c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3110a14700 1 -- 172.21.15.111:0/3857243360 >> v1:172.21.15.111:6789/0 conn(0x7f310c06df10 legacy=0x7f310c159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3110a14700 1 -- 172.21.15.111:0/3857243360 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0cb5c0 con 0x7f310c06db40 2023-12-19T09:03:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3110a14700 1 --2- 172.21.15.111:0/3857243360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 0x7f310c0cc3e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3100009910 tx=0x7f31000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbcc79c0eb6bc29d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f310b7fe700 1 -- 172.21.15.111:0/3857243360 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310000f6b0 con 0x7f310c06db40 2023-12-19T09:03:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f310b7fe700 1 -- 172.21.15.111:0/3857243360 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310000f810 con 0x7f310c06db40 2023-12-19T09:03:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f310b7fe700 1 -- 172.21.15.111:0/3857243360 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310000fbd0 con 0x7f310c06db40 2023-12-19T09:03:08.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 -- 172.21.15.111:0/3857243360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 msgr2=0x7f310c0cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 --2- 172.21.15.111:0/3857243360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 0x7f310c0cc3e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3100009910 tx=0x7f31000091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 -- 172.21.15.111:0/3857243360 shutdown_connections 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 --2- 172.21.15.111:0/3857243360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06db40 0x7f310c0cc3e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 -- 172.21.15.111:0/3857243360 >> 172.21.15.111:0/3857243360 conn(0x7f310c067bb0 msgr2=0x7f310c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 -- 172.21.15.111:0/3857243360 shutdown_connections 2023-12-19T09:03:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.155+0000 7f3112c78700 1 -- 172.21.15.111:0/3857243360 wait complete. 2023-12-19T09:03:08.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 Processor -- start 2023-12-19T09:03:08.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 -- start start 2023-12-19T09:03:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c06db40 0x7f310c0b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06df10 0x7f310c14bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c14c270 con 0x7f310c0c63f0 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0af7f0 con 0x7f310c06db40 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3112c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0c2d80 con 0x7f310c06df10 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3111215700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3110a14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c06db40 0x7f310c0b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3111215700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41700/0 (socket says 172.21.15.111:41700) 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f3111215700 1 -- 172.21.15.111:0/3185518642 learned_addr learned my addr 172.21.15.111:0/3185518642 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:08.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.159+0000 7f310bfff700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06df10 0x7f310c14bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 -- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06df10 msgr2=0x7f310c14bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06df10 0x7f310c14bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 -- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c06db40 msgr2=0x7f310c0b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c06db40 0x7f310c0b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 -- 172.21.15.111:0/3185518642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3100009420 con 0x7f310c0c63f0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3111215700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3104001460 tx=0x7f3104001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400a8e0 con 0x7f310c0c63f0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400aa40 con 0x7f310c0c63f0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c14ccc0 con 0x7f310c0c63f0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400adc0 con 0x7f310c0c63f0 2023-12-19T09:03:08.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.163+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c0c3040 con 0x7f310c0c63f0 2023-12-19T09:03:08.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.167+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c071120 con 0x7f310c0c63f0 2023-12-19T09:03:08.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.167+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3104007300 con 0x7f310c0c63f0 2023-12-19T09:03:08.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.167+0000 7f3108ff9700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 0x7f30f0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.167+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f310407cff0 con 0x7f310c0c63f0 2023-12-19T09:03:08.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.171+0000 7f3110a14700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 0x7f30f0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.171+0000 7f3110a14700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 0x7f30f0062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31000093f0 tx=0x7f3100019000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.175+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f310404d220 con 0x7f310c0c63f0 2023-12-19T09:03:08.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.351+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f310c0cb8a0 con 0x7f310c0c63f0 2023-12-19T09:03:08.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.359+0000 7f3108ff9700 1 -- 172.21.15.111:0/3185518642 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v57) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3104050840 con 0x7f310c0c63f0 2023-12-19T09:03:08.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 msgr2=0x7f30f0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 0x7f30f0062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31000093f0 tx=0x7f3100019000 comp rx=0 tx=0).stop 2023-12-19T09:03:08.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 msgr2=0x7f310c14f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3104001460 tx=0x7f3104001a80 comp rx=0 tx=0).stop 2023-12-19T09:03:08.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 shutdown_connections 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f30f0060670 0x7f30f0062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f310c0c63f0 0x7f310c14f8b0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c06df10 0x7f310c14bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 --2- 172.21.15.111:0/3185518642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c06db40 0x7f310c0b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 >> 172.21.15.111:0/3185518642 conn(0x7f310c067bb0 msgr2=0x7f310c06b040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 shutdown_connections 2023-12-19T09:03:08.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.363+0000 7f3112c78700 1 -- 172.21.15.111:0/3185518642 wait complete. 2023-12-19T09:03:08.392 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-12-19T09:03:08.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.559+0000 7f9e1f4f0700 1 Processor -- start 2023-12-19T09:03:08.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.559+0000 7f9e1f4f0700 1 -- start start 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e18159200 0x7f9e180c8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e1806b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1815a3e0 con 0x7f9e180aba30 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e180c8a90 con 0x7f9e18159200 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e180b0bf0 con 0x7f9e181595d0 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e18159200 0x7f9e180c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e18159200 0x7f9e180c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41702/0 (socket says 172.21.15.111:41702) 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2053149579 learned_addr learned my addr 172.21.15.111:0/2053149579 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:08.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1ca8b700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e1806b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2053149579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 msgr2=0x7f9e1806b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:08.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e1806b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2053149579 >> v1:172.21.15.143:6789/0 conn(0x7f9e180aba30 legacy=0x7f9e1815cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2053149579 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18159ab0 con 0x7f9e18159200 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1d28c700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e18159200 0x7f9e180c8550 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9e140099b0 tx=0x7f9e14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bc5312bcf4ce8b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e0ffff700 1 -- 172.21.15.111:0/2053149579 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14009ea0 con 0x7f9e18159200 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e0ffff700 1 -- 172.21.15.111:0/2053149579 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400b820 con 0x7f9e18159200 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e0ffff700 1 -- 172.21.15.111:0/2053149579 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400bbc0 con 0x7f9e18159200 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2053149579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e18159200 msgr2=0x7f9e180c8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.563+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e18159200 0x7f9e180c8550 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9e140099b0 tx=0x7f9e14009250 comp rx=0 tx=0).stop 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2053149579 shutdown_connections 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e1806b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2053149579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e18159200 0x7f9e180c8550 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2053149579 >> 172.21.15.111:0/2053149579 conn(0x7f9e18067410 msgr2=0x7f9e180677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2053149579 shutdown_connections 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2053149579 wait complete. 2023-12-19T09:03:08.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 Processor -- start 2023-12-19T09:03:08.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.567+0000 7f9e1f4f0700 1 -- start start 2023-12-19T09:03:08.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e18159200 0x7f9e18157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e180ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18152170 con 0x7f9e180aba30 2023-12-19T09:03:08.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e180acf40 con 0x7f9e18159200 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18151c10 con 0x7f9e181595d0 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41716/0 (socket says 172.21.15.111:41716) 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2590465988 learned_addr learned my addr 172.21.15.111:0/2590465988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1ca8b700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e18159200 0x7f9e18157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1da8d700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e180ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 msgr2=0x7f9e180ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e180ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e18159200 msgr2=0x7f9e18157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e18159200 0x7f9e18157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 -- 172.21.15.111:0/2590465988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140094f0 con 0x7f9e180aba30 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1d28c700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009290 tx=0x7f9e14016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14011aa0 con 0x7f9e180aba30 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14020070 con 0x7f9e180aba30 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e180ac330 con 0x7f9e180aba30 2023-12-19T09:03:08.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14011e70 con 0x7f9e180aba30 2023-12-19T09:03:08.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.571+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e180ac8f0 con 0x7f9e180aba30 2023-12-19T09:03:08.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.575+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e18071120 con 0x7f9e180aba30 2023-12-19T09:03:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.575+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f9e14022030 con 0x7f9e180aba30 2023-12-19T09:03:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.575+0000 7f9e0d7fa700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 0x7f9dfc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.575+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f9e1407e8f0 con 0x7f9e180aba30 2023-12-19T09:03:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.575+0000 7f9e1ca8b700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 0x7f9dfc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:08.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.579+0000 7f9e1ca8b700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 0x7f9dfc062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e180ad700 tx=0x7f9e080091a0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:08.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.583+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9e1404ea40 con 0x7f9e180aba30 2023-12-19T09:03:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:08.815+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f9e18071120 con 0x7f9e180aba30 2023-12-19T09:03:09.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.175+0000 7f9e0d7fa700 1 -- 172.21.15.111:0/2590465988 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2461) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9e14052060 con 0x7f9e180aba30 2023-12-19T09:03:09.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 msgr2=0x7f9dfc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 0x7f9dfc062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e180ad700 tx=0x7f9e080091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 msgr2=0x7f9e181516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009290 tx=0x7f9e14016000 comp rx=0 tx=0).stop 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 shutdown_connections 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9dfc060670 0x7f9dfc062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e18159200 0x7f9e18157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e181595d0 0x7f9e180ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 --2- 172.21.15.111:0/2590465988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e180aba30 0x7f9e181516d0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 >> 172.21.15.111:0/2590465988 conn(0x7f9e18067410 msgr2=0x7f9e18152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:09.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 shutdown_connections 2023-12-19T09:03:09.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.179+0000 7f9e1f4f0700 1 -- 172.21.15.111:0/2590465988 wait complete. 2023-12-19T09:03:09.209 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-12-19T09:03:09.210 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-12-19T09:03:09.210 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T09:03:09.210 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:09.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.367+0000 7faf9708e700 1 Processor -- start 2023-12-19T09:03:09.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 -- start start 2023-12-19T09:03:09.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faf9006a580 0x7faf9006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 0x7faf9006d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9006c7c0 con 0x7faf900b1410 2023-12-19T09:03:09.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9006af40 con 0x7faf9006d210 2023-12-19T09:03:09.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9708e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9006cd80 con 0x7faf9006a580 2023-12-19T09:03:09.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9688d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 0x7faf9006d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9688d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 0x7faf9006d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41732/0 (socket says 172.21.15.111:41732) 2023-12-19T09:03:09.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9688d700 1 -- 172.21.15.111:0/3544333721 learned_addr learned my addr 172.21.15.111:0/3544333721 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:09.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9588b700 1 --2- 172.21.15.111:0/3544333721 >> v2:172.21.15.143:3300/0 conn(0x7faf9006a580 0x7faf9006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9588b700 1 -- 172.21.15.111:0/3544333721 >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 msgr2=0x7faf9006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9588b700 1 --2- 172.21.15.111:0/3544333721 >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 0x7faf9006d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9588b700 1 -- 172.21.15.111:0/3544333721 >> v1:172.21.15.143:6789/0 conn(0x7faf900b1410 legacy=0x7faf900ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.371+0000 7faf9588b700 1 -- 172.21.15.111:0/3544333721 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9006b8f0 con 0x7faf9006a580 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9588b700 1 --2- 172.21.15.111:0/3544333721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf9006aa00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faf90068040 tx=0x7faf800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1c5f4318f31a488 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9508a700 1 -- 172.21.15.111:0/3544333721 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80011630 con 0x7faf9006a580 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9508a700 1 -- 172.21.15.111:0/3544333721 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf80011790 con 0x7faf9006a580 2023-12-19T09:03:09.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9508a700 1 -- 172.21.15.111:0/3544333721 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80011b30 con 0x7faf9006a580 2023-12-19T09:03:09.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 -- 172.21.15.111:0/3544333721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 msgr2=0x7faf9006aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 --2- 172.21.15.111:0/3544333721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf9006aa00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7faf90068040 tx=0x7faf800091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 -- 172.21.15.111:0/3544333721 shutdown_connections 2023-12-19T09:03:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 --2- 172.21.15.111:0/3544333721 >> v2:172.21.15.111:3300/0 conn(0x7faf9006d210 0x7faf9006d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 --2- 172.21.15.111:0/3544333721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf9006aa00 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 -- 172.21.15.111:0/3544333721 >> 172.21.15.111:0/3544333721 conn(0x7faf90067bb0 msgr2=0x7faf900cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 -- 172.21.15.111:0/3544333721 shutdown_connections 2023-12-19T09:03:09.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.375+0000 7faf9708e700 1 -- 172.21.15.111:0/3544333721 wait complete. 2023-12-19T09:03:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 Processor -- start 2023-12-19T09:03:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 -- start start 2023-12-19T09:03:09.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf900c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf9006d210 0x7faf900c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf901547e0 con 0x7faf900b1410 2023-12-19T09:03:09.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9016c900 con 0x7faf9006a580 2023-12-19T09:03:09.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.379+0000 7faf9708e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9016cbe0 con 0x7faf9006d210 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41746/0 (socket says 172.21.15.111:41746) 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 -- 172.21.15.111:0/3675880751 learned_addr learned my addr 172.21.15.111:0/3675880751 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9588b700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf9006d210 0x7faf900c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9608c700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf900c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 -- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf9006d210 msgr2=0x7faf900c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf9006d210 0x7faf900c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 -- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 msgr2=0x7faf900c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf900c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 -- 172.21.15.111:0/3675880751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80009420 con 0x7faf900b1410 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9608c700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf900c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9688d700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7faf88009e50 tx=0x7faf8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800ca10 con 0x7faf900b1410 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8800f040 con 0x7faf900b1410 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8800cdc0 con 0x7faf900b1410 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9016cec0 con 0x7faf900b1410 2023-12-19T09:03:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9016d480 con 0x7faf900b1410 2023-12-19T09:03:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.383+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9016d6a0 con 0x7faf900b1410 2023-12-19T09:03:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.387+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7faf88014650 con 0x7faf900b1410 2023-12-19T09:03:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.387+0000 7faf867fc700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 0x7faf74062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.387+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7faf8807d480 con 0x7faf900b1410 2023-12-19T09:03:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.387+0000 7faf9608c700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 0x7faf74062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.387+0000 7faf9608c700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 0x7faf74062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faf8c007fd0 tx=0x7faf8c009040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.395+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faf8804d560 con 0x7faf900b1410 2023-12-19T09:03:09.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.563+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf9004be40 con 0x7faf900b1410 2023-12-19T09:03:09.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf867fc700 1 -- 172.21.15.111:0/3675880751 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+14300 (secure 0 0 0) 0x7faf88050b80 con 0x7faf900b1410 2023-12-19T09:03:09.572 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:09.572 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":166,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:03:07.163477+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6834/4167714642":"2023-12-20T09:03:03.792255+0000","172.21.15.111:6832/4167714642":"2023-12-20T09:03:03.792255+0000","172.21.15.143:0/3597085434":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/1556330723":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/282533981":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/2870222102":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/3561775375":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/207975686":"2023-12-20T09:03:03.600467+0000","172.21.15.143:0/2824146656":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1422322069":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/3492177270":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1354315856":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/3940147408":"2023-12-20T09:03:03.600467+0000","172.21.15.143:6833/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:6832/33796":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/272833109":"2023-12-20T09:02:55.853401+0000","172.21.15.143:0/1889918723":"2023-12-20T09:02:55.853401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:09.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 msgr2=0x7faf74062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 0x7faf74062b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faf8c007fd0 tx=0x7faf8c009040 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 msgr2=0x7faf9014a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7faf88009e50 tx=0x7faf8800b040 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 shutdown_connections 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7faf74060670 0x7faf74062b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf900b1410 0x7faf9014a070 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf9006d210 0x7faf900c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 --2- 172.21.15.111:0/3675880751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf9006a580 0x7faf900c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 >> 172.21.15.111:0/3675880751 conn(0x7faf90067bb0 msgr2=0x7faf900ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 shutdown_connections 2023-12-19T09:03:09.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.567+0000 7faf9708e700 1 -- 172.21.15.111:0/3675880751 wait complete. 2023-12-19T09:03:09.594 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:03:09.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:03:09.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.755+0000 7f58f7b05700 1 Processor -- start 2023-12-19T09:03:09.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.755+0000 7f58f7b05700 1 -- start start 2023-12-19T09:03:09.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f7b05700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f58f015d410 0x7f58f00ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f7b05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 0x7f58f006b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f7b05700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00c7450 con 0x7f58f00ca400 2023-12-19T09:03:09.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f7b05700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f006bea0 con 0x7f58f015d410 2023-12-19T09:03:09.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f7b05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00c6e00 con 0x7f58f006b1b0 2023-12-19T09:03:09.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f50a0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f58f00ca400 0x7f58f00ca810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:54966/0 (socket says 172.21.15.111:54966) 2023-12-19T09:03:09.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f50a0700 1 -- 172.21.15.111:0/3416178963 learned_addr learned my addr 172.21.15.111:0/3416178963 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:09.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 --2- 172.21.15.111:0/3416178963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 0x7f58f006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f58a1700 1 --2- 172.21.15.111:0/3416178963 >> v2:172.21.15.143:3300/0 conn(0x7f58f015d410 0x7f58f00ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f489f700 1 -- 172.21.15.111:0/3416178963 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226889523 0 0) 0x7f58f00c7450 con 0x7f58f00ca400 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f489f700 1 -- 172.21.15.111:0/3416178963 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58dc003570 con 0x7f58f00ca400 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 -- 172.21.15.111:0/3416178963 >> v2:172.21.15.143:3300/0 conn(0x7f58f015d410 msgr2=0x7f58f00ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 --2- 172.21.15.111:0/3416178963 >> v2:172.21.15.143:3300/0 conn(0x7f58f015d410 0x7f58f00ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 -- 172.21.15.111:0/3416178963 >> v1:172.21.15.111:6789/0 conn(0x7f58f00ca400 legacy=0x7f58f00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 -- 172.21.15.111:0/3416178963 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00c70e0 con 0x7f58f006b1b0 2023-12-19T09:03:09.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f60a2700 1 --2- 172.21.15.111:0/3416178963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 0x7f58f006b650 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009e60 tx=0x7f58e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64c231c6ff9ce81f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f489f700 1 -- 172.21.15.111:0/3416178963 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e800e070 con 0x7f58f006b1b0 2023-12-19T09:03:09.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f489f700 1 -- 172.21.15.111:0/3416178963 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e800bab0 con 0x7f58f006b1b0 2023-12-19T09:03:09.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.759+0000 7f58f489f700 1 -- 172.21.15.111:0/3416178963 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e800be70 con 0x7f58f006b1b0 2023-12-19T09:03:09.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 -- 172.21.15.111:0/3416178963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 msgr2=0x7f58f006b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 --2- 172.21.15.111:0/3416178963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 0x7f58f006b650 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009e60 tx=0x7f58e8009780 comp rx=0 tx=0).stop 2023-12-19T09:03:09.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 -- 172.21.15.111:0/3416178963 shutdown_connections 2023-12-19T09:03:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 --2- 172.21.15.111:0/3416178963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f006b1b0 0x7f58f006b650 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 --2- 172.21.15.111:0/3416178963 >> v2:172.21.15.143:3300/0 conn(0x7f58f015d410 0x7f58f00ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 -- 172.21.15.111:0/3416178963 >> 172.21.15.111:0/3416178963 conn(0x7f58f0067bb0 msgr2=0x7f58f00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 -- 172.21.15.111:0/3416178963 shutdown_connections 2023-12-19T09:03:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.763+0000 7f58f7b05700 1 -- 172.21.15.111:0/3416178963 wait complete. 2023-12-19T09:03:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 Processor -- start 2023-12-19T09:03:09.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 -- start start 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58f006b1b0 0x7f58f014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f00ca400 0x7f58f00c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00af850 con 0x7f58f015d410 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00aec40 con 0x7f58f006b1b0 2023-12-19T09:03:09.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f7b05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00aef20 con 0x7f58f00ca400 2023-12-19T09:03:09.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41750/0 (socket says 172.21.15.111:41750) 2023-12-19T09:03:09.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 -- 172.21.15.111:0/2781864659 learned_addr learned my addr 172.21.15.111:0/2781864659 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f58a1700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58f006b1b0 0x7f58f014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f50a0700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f00ca400 0x7f58f00c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 -- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f00ca400 msgr2=0x7f58f00c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f00ca400 0x7f58f00c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 -- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58f006b1b0 msgr2=0x7f58f014ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58f006b1b0 0x7f58f014ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.767+0000 7f58f60a2700 1 -- 172.21.15.111:0/2781864659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8009a20 con 0x7f58f015d410 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58f60a2700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f58f00c6f40 tx=0x7f58e8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8012e10 con 0x7f58f015d410 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8027470 con 0x7f58f015d410 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8027830 con 0x7f58f015d410 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00af200 con 0x7f58f015d410 2023-12-19T09:03:09.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f014f440 con 0x7f58f015d410 2023-12-19T09:03:09.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f00afa00 con 0x7f58f015d410 2023-12-19T09:03:09.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.771+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f58e8024070 con 0x7f58f015d410 2023-12-19T09:03:09.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.775+0000 7f58e5ffb700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 0x7f58d4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:09.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.775+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f58e807f590 con 0x7f58f015d410 2023-12-19T09:03:09.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.775+0000 7f58f58a1700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 0x7f58d4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:09.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.775+0000 7f58f58a1700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 0x7f58d4062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58f00ab5f0 tx=0x7f58ec0091a0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:09.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.779+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f58f00afa00 con 0x7f58f015d410 2023-12-19T09:03:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.939+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58f0071120 con 0x7f58f015d410 2023-12-19T09:03:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58e5ffb700 1 -- 172.21.15.111:0/2781864659 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v166) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58e804f780 con 0x7f58f015d410 2023-12-19T09:03:09.948 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:03:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 msgr2=0x7f58d4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 0x7f58d4062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58f00ab5f0 tx=0x7f58ec0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 msgr2=0x7f58f00c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f58f00c6f40 tx=0x7f58e8019000 comp rx=0 tx=0).stop 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 shutdown_connections 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f58d4060670 0x7f58d4062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58f015d410 0x7f58f00c5770 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58f00ca400 0x7f58f00c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 --2- 172.21.15.111:0/2781864659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58f006b1b0 0x7f58f014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 >> 172.21.15.111:0/2781864659 conn(0x7f58f0067bb0 msgr2=0x7f58f00c14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 shutdown_connections 2023-12-19T09:03:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:09.943+0000 7f58f7b05700 1 -- 172.21.15.111:0/2781864659 wait complete. 2023-12-19T09:03:09.972 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-12-19T09:03:10.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.127+0000 7f97f0c6e700 1 Processor -- start 2023-12-19T09:03:10.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.127+0000 7f97f0c6e700 1 -- start start 2023-12-19T09:03:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 0x7f97ec0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec068c70 con 0x7f97ec159340 2023-12-19T09:03:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec068060 con 0x7f97ec069100 2023-12-19T09:03:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec068340 con 0x7f97ec158f70 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97e9d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 0x7f97ec0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:41734/0 (socket says 172.21.15.111:41734) 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 -- 172.21.15.111:0/2997471857 learned_addr learned my addr 172.21.15.111:0/2997471857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 -- 172.21.15.111:0/2997471857 >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 msgr2=0x7f97ec0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 --2- 172.21.15.111:0/2997471857 >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 0x7f97ec0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 -- 172.21.15.111:0/2997471857 >> v1:172.21.15.143:6789/0 conn(0x7f97ec159340 legacy=0x7f97ec0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 -- 172.21.15.111:0/2997471857 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec068620 con 0x7f97ec158f70 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97e9d9b700 1 --2- 172.21.15.111:0/2997471857 >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 0x7f97ec0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97ea59c700 1 --2- 172.21.15.111:0/2997471857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0099b0 tx=0x7f97dc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1147b63ab135eaf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97e959a700 1 -- 172.21.15.111:0/2997471857 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc009ea0 con 0x7f97ec158f70 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97e959a700 1 -- 172.21.15.111:0/2997471857 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc00b860 con 0x7f97ec158f70 2023-12-19T09:03:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97e959a700 1 -- 172.21.15.111:0/2997471857 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00bc00 con 0x7f97ec158f70 2023-12-19T09:03:10.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- 172.21.15.111:0/2997471857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 msgr2=0x7f97ec067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:10.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/2997471857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0099b0 tx=0x7f97dc009250 comp rx=0 tx=0).stop 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- 172.21.15.111:0/2997471857 shutdown_connections 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/2997471857 >> v2:172.21.15.111:3300/0 conn(0x7f97ec069100 0x7f97ec0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/2997471857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec067980 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.131+0000 7f97f0c6e700 1 -- 172.21.15.111:0/2997471857 >> 172.21.15.111:0/2997471857 conn(0x7f97ec0ccc40 msgr2=0x7f97ec0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.135+0000 7f97f0c6e700 1 -- 172.21.15.111:0/2997471857 shutdown_connections 2023-12-19T09:03:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.135+0000 7f97f0c6e700 1 -- 172.21.15.111:0/2997471857 wait complete. 2023-12-19T09:03:10.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.135+0000 7f97f0c6e700 1 Processor -- start 2023-12-19T09:03:10.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.135+0000 7f97f0c6e700 1 -- start start 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 0x7f97ec156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec155600 con 0x7f97ec069100 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0b0bc0 con 0x7f97ec159340 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0b0ea0 con 0x7f97ec158f70 2023-12-19T09:03:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41768/0 (socket says 172.21.15.111:41768) 2023-12-19T09:03:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 -- 172.21.15.111:0/1648878584 learned_addr learned my addr 172.21.15.111:0/1648878584 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ead9d700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 0x7f97ec156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97e9d9b700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 -- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 msgr2=0x7f97ec0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec0b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 -- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 msgr2=0x7f97ec156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 0x7f97ec156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 -- 172.21.15.111:0/1648878584 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e0009420 con 0x7f97ec069100 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ead9d700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 0x7f97ec156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97ea59c700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f97ec068480 tx=0x7f97dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00bbe0 con 0x7f97ec069100 2023-12-19T09:03:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc0094f0 con 0x7f97ec069100 2023-12-19T09:03:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc011420 con 0x7f97ec069100 2023-12-19T09:03:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.139+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0117c0 con 0x7f97ec069100 2023-12-19T09:03:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.143+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec0b1460 con 0x7f97ec069100 2023-12-19T09:03:10.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.143+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec071120 con 0x7f97ec069100 2023-12-19T09:03:10.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.143+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f97dc011bf0 con 0x7f97ec069100 2023-12-19T09:03:10.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.147+0000 7f97da7fc700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 0x7f97cc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:10.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.147+0000 7f97e9d9b700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 0x7f97cc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:10.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.147+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f97dc001d20 con 0x7f97ec069100 2023-12-19T09:03:10.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.147+0000 7f97e9d9b700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 0x7f97cc062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97e00093f0 tx=0x7f97e0008040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:10.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.151+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f97dc04e860 con 0x7f97ec069100 2023-12-19T09:03:10.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:10.379+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f97ec068900 con 0x7f97ec069100 2023-12-19T09:03:11.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.191+0000 7f97da7fc700 1 -- 172.21.15.111:0/1648878584 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2463) v1 ==== 168+0+0 (secure 0 0 0) 0x7f97dc0150e0 con 0x7f97ec069100 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 msgr2=0x7f97cc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 0x7f97cc062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f97e00093f0 tx=0x7f97e0008040 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 msgr2=0x7f97ec1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f97ec068480 tx=0x7f97dc017000 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 shutdown_connections 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f97cc060670 0x7f97cc062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97ec159340 0x7f97ec156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97ec158f70 0x7f97ec0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 --2- 172.21.15.111:0/1648878584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec069100 0x7f97ec1550c0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 >> 172.21.15.111:0/1648878584 conn(0x7f97ec0ccc40 msgr2=0x7f97ec157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 shutdown_connections 2023-12-19T09:03:11.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.195+0000 7f97f0c6e700 1 -- 172.21.15.111:0/1648878584 wait complete. 2023-12-19T09:03:11.224 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T09:03:11.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.391+0000 7ff3a1f23700 1 Processor -- start 2023-12-19T09:03:11.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.391+0000 7ff3a1f23700 1 -- start start 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1f23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff39c0b1440 0x7ff39c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1f23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c0ac390 con 0x7ff39c06ae20 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1f23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39c0ac670 con 0x7ff39c0b1440 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c0ac950 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff3a1722700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff39c0b1440 0x7ff39c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:41744/0 (socket says 172.21.15.111:41744) 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 -- 172.21.15.111:0/1522487015 learned_addr learned my addr 172.21.15.111:0/1522487015 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 -- 172.21.15.111:0/1522487015 >> v2:172.21.15.143:3300/0 conn(0x7ff39c0b1440 msgr2=0x7ff39c0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 --2- 172.21.15.111:0/1522487015 >> v2:172.21.15.143:3300/0 conn(0x7ff39c0b1440 0x7ff39c0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 -- 172.21.15.111:0/1522487015 >> v1:172.21.15.143:6789/0 conn(0x7ff39c06ae20 legacy=0x7ff39c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 -- 172.21.15.111:0/1522487015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c0acc30 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39bfff700 1 --2- 172.21.15.111:0/1522487015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff3880099b0 tx=0x7ff388009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1387fee43cc07b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39b7fe700 1 -- 172.21.15.111:0/1522487015 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388009ea0 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39b7fe700 1 -- 172.21.15.111:0/1522487015 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38800b860 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.395+0000 7ff39b7fe700 1 -- 172.21.15.111:0/1522487015 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38800bc00 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- 172.21.15.111:0/1522487015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 msgr2=0x7ff39c0abb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/1522487015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff3880099b0 tx=0x7ff388009250 comp rx=0 tx=0).stop 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- 172.21.15.111:0/1522487015 shutdown_connections 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/1522487015 >> v2:172.21.15.143:3300/0 conn(0x7ff39c0b1440 0x7ff39c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/1522487015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0ab6d0 0x7ff39c0abb20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- 172.21.15.111:0/1522487015 >> 172.21.15.111:0/1522487015 conn(0x7ff39c0c8cd0 msgr2=0x7ff39c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- 172.21.15.111:0/1522487015 shutdown_connections 2023-12-19T09:03:11.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- 172.21.15.111:0/1522487015 wait complete. 2023-12-19T09:03:11.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 Processor -- start 2023-12-19T09:03:11.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- start start 2023-12-19T09:03:11.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39c0ab6d0 0x7ff39c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0b1440 0x7ff39c15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c157700 con 0x7ff39c06ae20 2023-12-19T09:03:11.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c0c00b0 con 0x7ff39c0ab6d0 2023-12-19T09:03:11.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.399+0000 7ff3a1f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c15b460 con 0x7ff39c0b1440 2023-12-19T09:03:11.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39c0ab6d0 0x7ff39c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a1722700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0b1440 0x7ff39c15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41776/0 (socket says 172.21.15.111:41776) 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 -- 172.21.15.111:0/2518460498 learned_addr learned my addr 172.21.15.111:0/2518460498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 -- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0b1440 msgr2=0x7ff39c15af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0b1440 0x7ff39c15af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 -- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39c0ab6d0 msgr2=0x7ff39c15a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39c0ab6d0 0x7ff39c15a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 -- 172.21.15.111:0/2518460498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880094f0 con 0x7ff39c06ae20 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a0f21700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff39000c4e0 tx=0x7ff39000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390011590 con 0x7ff39c06ae20 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff390011710 con 0x7ff39c06ae20 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c16dab0 con 0x7ff39c06ae20 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390011ad0 con 0x7ff39c06ae20 2023-12-19T09:03:11.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c0c2c60 con 0x7ff39c06ae20 2023-12-19T09:03:11.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.403+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c04be40 con 0x7ff39c06ae20 2023-12-19T09:03:11.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.407+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7ff390000f90 con 0x7ff39c06ae20 2023-12-19T09:03:11.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.407+0000 7ff398ff9700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 0x7ff380062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.407+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7ff39007bfb0 con 0x7ff39c06ae20 2023-12-19T09:03:11.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.407+0000 7ff39bfff700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 0x7ff380062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.411+0000 7ff39bfff700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 0x7ff380062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3880099b0 tx=0x7ff388017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.411+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff39004c100 con 0x7ff39c06ae20 2023-12-19T09:03:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.583+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff39c0b1de0 con 0x7ff39c06ae20 2023-12-19T09:03:11.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.591+0000 7ff398ff9700 1 -- 172.21.15.111:0/2518460498 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff390017020 con 0x7ff39c06ae20 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 msgr2=0x7ff380062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 0x7ff380062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3880099b0 tx=0x7ff388017000 comp rx=0 tx=0).stop 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 msgr2=0x7ff39c16d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff39000c4e0 tx=0x7ff39000c7f0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 shutdown_connections 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7ff380060670 0x7ff380062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39c0ab6d0 0x7ff39c15a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff39c0b1440 0x7ff39c15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 --2- 172.21.15.111:0/2518460498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff39c06ae20 0x7ff39c16d570 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 >> 172.21.15.111:0/2518460498 conn(0x7ff39c0c8cd0 msgr2=0x7ff39c1513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:11.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 shutdown_connections 2023-12-19T09:03:11.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.595+0000 7ff3a1f23700 1 -- 172.21.15.111:0/2518460498 wait complete. 2023-12-19T09:03:11.622 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:03:11.622 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:03:11.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:03:11.714 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:03:11.714 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T09:03:11.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:11.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 Processor -- start 2023-12-19T09:03:11.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 -- start start 2023-12-19T09:03:11.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd1981556c0 0x7fd198155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980cb640 con 0x7fd1980cc530 2023-12-19T09:03:11.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980caa30 con 0x7fd1980c2220 2023-12-19T09:03:11.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.875+0000 7fd19e7c3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1980cad10 con 0x7fd1981556c0 2023-12-19T09:03:11.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19dfc2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd1980cc530 0x7fd1980c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46276/0 (socket says 172.21.15.111:46276) 2023-12-19T09:03:11.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19dfc2700 1 -- 172.21.15.111:0/447349275 learned_addr learned my addr 172.21.15.111:0/447349275 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:11.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19d7c1700 1 --2- 172.21.15.111:0/447349275 >> v2:172.21.15.143:3300/0 conn(0x7fd1981556c0 0x7fd198155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd18ffff700 1 -- 172.21.15.111:0/447349275 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691280705 0 0) 0x7fd1980cb640 con 0x7fd1980cc530 2023-12-19T09:03:11.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd18ffff700 1 -- 172.21.15.111:0/447349275 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184003570 con 0x7fd1980cc530 2023-12-19T09:03:11.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19d7c1700 1 -- 172.21.15.111:0/447349275 >> v1:172.21.15.143:6789/0 conn(0x7fd1980c2220 legacy=0x7fd1980cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19d7c1700 1 -- 172.21.15.111:0/447349275 >> v1:172.21.15.111:6789/0 conn(0x7fd1980cc530 legacy=0x7fd1980c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19d7c1700 1 -- 172.21.15.111:0/447349275 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1980caff0 con 0x7fd1981556c0 2023-12-19T09:03:11.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19d7c1700 1 --2- 172.21.15.111:0/447349275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1981556c0 0x7fd198155a90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd194009e60 tx=0x7fd194009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=653d547c25f78fef server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd18ffff700 1 -- 172.21.15.111:0/447349275 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19400e070 con 0x7fd1981556c0 2023-12-19T09:03:11.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd18ffff700 1 -- 172.21.15.111:0/447349275 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19400bb60 con 0x7fd1981556c0 2023-12-19T09:03:11.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd18ffff700 1 -- 172.21.15.111:0/447349275 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194012430 con 0x7fd1981556c0 2023-12-19T09:03:11.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19e7c3700 1 -- 172.21.15.111:0/447349275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1981556c0 msgr2=0x7fd198155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/447349275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1981556c0 0x7fd198155a90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd194009e60 tx=0x7fd194009780 comp rx=0 tx=0).stop 2023-12-19T09:03:11.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19e7c3700 1 -- 172.21.15.111:0/447349275 shutdown_connections 2023-12-19T09:03:11.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/447349275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1981556c0 0x7fd198155a90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.879+0000 7fd19e7c3700 1 -- 172.21.15.111:0/447349275 >> 172.21.15.111:0/447349275 conn(0x7fd19806b670 msgr2=0x7fd19806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:11.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.883+0000 7fd19e7c3700 1 -- 172.21.15.111:0/447349275 shutdown_connections 2023-12-19T09:03:11.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.883+0000 7fd19e7c3700 1 -- 172.21.15.111:0/447349275 wait complete. 2023-12-19T09:03:11.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.883+0000 7fd19e7c3700 1 Processor -- start 2023-12-19T09:03:11.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.883+0000 7fd19e7c3700 1 -- start start 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1980c2220 0x7fd198153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1981556c0 0x7fd19806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd198158de0 con 0x7fd1980cc530 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981581d0 con 0x7fd1980c2220 2023-12-19T09:03:11.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981584b0 con 0x7fd1981556c0 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42376/0 (socket says 172.21.15.111:42376) 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 -- 172.21.15.111:0/579050977 learned_addr learned my addr 172.21.15.111:0/579050977 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19d7c1700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1980c2220 0x7fd198153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19dfc2700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1981556c0 0x7fd19806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 -- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1981556c0 msgr2=0x7fd19806d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1981556c0 0x7fd19806d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 -- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1980c2220 msgr2=0x7fd198153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1980c2220 0x7fd198153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 -- 172.21.15.111:0/579050977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194009a20 con 0x7fd1980cc530 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19cfc0700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c590 tx=0x7fd18800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188011590 con 0x7fd1980cc530 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd188011710 con 0x7fd1980cc530 2023-12-19T09:03:11.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd198158790 con 0x7fd1980cc530 2023-12-19T09:03:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188011ad0 con 0x7fd1980cc530 2023-12-19T09:03:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.887+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd198153c70 con 0x7fd1980cc530 2023-12-19T09:03:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.891+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19804be40 con 0x7fd1980cc530 2023-12-19T09:03:11.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.891+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd18801b3b0 con 0x7fd1980cc530 2023-12-19T09:03:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.895+0000 7fd18d7fa700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 0x7fd17c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.895+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7fd18807c210 con 0x7fd1980cc530 2023-12-19T09:03:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.895+0000 7fd19d7c1700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 0x7fd17c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:11.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.895+0000 7fd19d7c1700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 0x7fd17c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097c0 tx=0x7fd194019000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:11.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:11.899+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd18804c360 con 0x7fd1980cc530 2023-12-19T09:03:12.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.107+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd198158f20 con 0x7fd1980cc530 2023-12-19T09:03:12.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.107+0000 7fd18d7fa700 1 -- 172.21.15.111:0/579050977 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fd188017090 con 0x7fd1980cc530 2023-12-19T09:03:12.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:12.113 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7661,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6836/541845061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":541845061},{"type":"v1","addr":"172.21.15.143:6836","nonce":541845061}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7670,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/326624480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6833","nonce":326624480},{"type":"v1","addr":"172.21.15.111:6835","nonce":326624480}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8629,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1435226096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6835","nonce":1435226096},{"type":"v1","addr":"172.21.15.143:6837","nonce":1435226096}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":9300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6834/2003953822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2003953822},{"type":"v1","addr":"172.21.15.111:6834","nonce":2003953822}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[]} 2023-12-19T09:03:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 msgr2=0x7fd17c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 0x7fd17c062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097c0 tx=0x7fd194019000 comp rx=0 tx=0).stop 2023-12-19T09:03:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 msgr2=0x7fd19806d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c590 tx=0x7fd18800c8a0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 shutdown_connections 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd17c060670 0x7fd17c062b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1981556c0 0x7fd19806d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1980cc530 0x7fd19806d050 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 --2- 172.21.15.111:0/579050977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1980c2220 0x7fd198153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 >> 172.21.15.111:0/579050977 conn(0x7fd19806b670 msgr2=0x7fd1980b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:12.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 shutdown_connections 2023-12-19T09:03:12.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.111+0000 7fd19e7c3700 1 -- 172.21.15.111:0/579050977 wait complete. 2023-12-19T09:03:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 74 2023-12-19T09:03:12.138 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T09:03:12.139 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T09:03:12.139 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T09:03:12.139 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:03:12.139 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T09:03:12.139 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T09:03:12.139 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T09:03:12.139 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:03:12.140 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T09:03:12.140 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T09:03:12.140 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T09:03:12.140 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:03:12.140 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T09:03:12.140 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T09:03:12.140 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T09:03:12.141 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:03:12.162 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T09:03:12.163 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T09:03:12.163 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T09:03:12.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T09:03:12.164 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T09:03:12.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T09:03:12.165 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T09:03:12.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T09:03:12.209 INFO:tasks.ceph.mds.a:Started 2023-12-19T09:03:12.210 DEBUG:teuthology.parallel:result is None 2023-12-19T09:03:12.214 INFO:tasks.ceph.mds.c:Started 2023-12-19T09:03:12.214 DEBUG:teuthology.parallel:result is None 2023-12-19T09:03:12.218 INFO:tasks.ceph.mds.b:Started 2023-12-19T09:03:12.218 DEBUG:teuthology.parallel:result is None 2023-12-19T09:03:12.222 INFO:tasks.ceph.mds.d:Started 2023-12-19T09:03:12.223 DEBUG:teuthology.parallel:result is None 2023-12-19T09:03:12.223 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T09:03:12.318 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T09:03:12.321 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:03:12.332 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:03:12.340 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T09:03:12.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 Processor -- start 2023-12-19T09:03:12.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 -- start start 2023-12-19T09:03:12.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f933c06a010 0x7f933c0c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:12.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0ae760 con 0x7f933c06d6a0 2023-12-19T09:03:12.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0adb50 con 0x7f933c160a00 2023-12-19T09:03:12.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f9341acd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0b2100 con 0x7f933c06a010 2023-12-19T09:03:12.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f933affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f933c06d6a0 0x7f933c0aebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46296/0 (socket says 172.21.15.111:46296) 2023-12-19T09:03:12.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f933affd700 1 -- 172.21.15.111:0/794610023 learned_addr learned my addr 172.21.15.111:0/794610023 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:12.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.387+0000 7f933bfff700 1 --2- 172.21.15.111:0/794610023 >> v2:172.21.15.143:3300/0 conn(0x7f933c06a010 0x7f933c0c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:12.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933bfff700 1 -- 172.21.15.111:0/794610023 >> v1:172.21.15.143:6789/0 conn(0x7f933c160a00 legacy=0x7f933c06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933bfff700 1 -- 172.21.15.111:0/794610023 >> v1:172.21.15.111:6789/0 conn(0x7f933c06d6a0 legacy=0x7f933c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933a7fc700 1 -- 172.21.15.111:0/794610023 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092853872 0 0) 0x7f933c0ae760 con 0x7f933c06d6a0 2023-12-19T09:03:12.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933bfff700 1 -- 172.21.15.111:0/794610023 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0ade30 con 0x7f933c06a010 2023-12-19T09:03:12.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933bfff700 1 --2- 172.21.15.111:0/794610023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c06a010 0x7f933c0c99e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9334009e90 tx=0x7f9334009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e268ea85a259d577 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:12.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933a7fc700 1 -- 172.21.15.111:0/794610023 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400e070 con 0x7f933c06a010 2023-12-19T09:03:12.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933a7fc700 1 -- 172.21.15.111:0/794610023 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400bbf0 con 0x7f933c06a010 2023-12-19T09:03:12.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f933a7fc700 1 -- 172.21.15.111:0/794610023 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334012430 con 0x7f933c06a010 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 -- 172.21.15.111:0/794610023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c06a010 msgr2=0x7f933c0c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 --2- 172.21.15.111:0/794610023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c06a010 0x7f933c0c99e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9334009e90 tx=0x7f9334009780 comp rx=0 tx=0).stop 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 -- 172.21.15.111:0/794610023 shutdown_connections 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 --2- 172.21.15.111:0/794610023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c06a010 0x7f933c0c99e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 -- 172.21.15.111:0/794610023 >> 172.21.15.111:0/794610023 conn(0x7f933c069310 msgr2=0x7f933c0696e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 -- 172.21.15.111:0/794610023 shutdown_connections 2023-12-19T09:03:12.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.391+0000 7f9341acd700 1 -- 172.21.15.111:0/794610023 wait complete. 2023-12-19T09:03:12.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 Processor -- start 2023-12-19T09:03:12.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 -- start start 2023-12-19T09:03:12.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c06a010 0x7f933c16d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c160a00 0x7f933c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0c9f20 con 0x7f933c06d6a0 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1463e0 con 0x7f933c160a00 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c15fb40 con 0x7f933c06a010 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42388/0 (socket says 172.21.15.111:42388) 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 -- 172.21.15.111:0/4111756512 learned_addr learned my addr 172.21.15.111:0/4111756512 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933b7fe700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c06a010 0x7f933c16d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933bfff700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c160a00 0x7f933c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 -- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c06a010 msgr2=0x7f933c16d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c06a010 0x7f933c16d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 -- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c160a00 msgr2=0x7f933c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:12.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c160a00 0x7f933c1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 -- 172.21.15.111:0/4111756512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334009a20 con 0x7f933c06d6a0 2023-12-19T09:03:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f933affd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f933000c7e0 tx=0x7f9330007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330005c00 con 0x7f933c06d6a0 2023-12-19T09:03:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330005d80 con 0x7f933c06d6a0 2023-12-19T09:03:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c15fe20 con 0x7f933c06d6a0 2023-12-19T09:03:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.395+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330007320 con 0x7f933c06d6a0 2023-12-19T09:03:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.399+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c146070 con 0x7f933c06d6a0 2023-12-19T09:03:12.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.399+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c146550 con 0x7f933c06d6a0 2023-12-19T09:03:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.399+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f933001b3b0 con 0x7f933c06d6a0 2023-12-19T09:03:12.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.403+0000 7f93237fe700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 0x7f9328062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.403+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6452+0+0 (secure 0 0 0) 0x7f933007c130 con 0x7f933c06d6a0 2023-12-19T09:03:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.403+0000 7f933b7fe700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 0x7f9328062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.403+0000 7f933b7fe700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 0x7f9328062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f932c003a80 tx=0x7f932c007ec0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:12.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.407+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f933004c280 con 0x7f933c06d6a0 2023-12-19T09:03:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:12.587+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f933c06dc70 con 0x7f933c06d6a0 2023-12-19T09:03:13.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.623+0000 7f93237fe700 1 -- 172.21.15.111:0/4111756512 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v171) v1 ==== 97+0+0 (secure 0 0 0) 0x7f933004f8a0 con 0x7f933c06d6a0 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 msgr2=0x7f9328062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 0x7f9328062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f932c003a80 tx=0x7f932c007ec0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 msgr2=0x7f933c15cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f933000c7e0 tx=0x7f9330007d70 comp rx=0 tx=0).stop 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 shutdown_connections 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f9328060670 0x7f9328062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c160a00 0x7f933c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f933c06d6a0 0x7f933c15cfb0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 --2- 172.21.15.111:0/4111756512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c06a010 0x7f933c16d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 >> 172.21.15.111:0/4111756512 conn(0x7f933c069310 msgr2=0x7f933c0afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:13.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 shutdown_connections 2023-12-19T09:03:13.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.627+0000 7f9341acd700 1 -- 172.21.15.111:0/4111756512 wait complete. 2023-12-19T09:03:13.637 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T09:03:13.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T09:03:13.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.815+0000 7fa1a9ce3700 1 Processor -- start 2023-12-19T09:03:13.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.815+0000 7fa1a9ce3700 1 -- start start 2023-12-19T09:03:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a9ce3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa1a40c6cd0 0x7fa1a406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a9ce3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1a415cc20 0x7fa1a415d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a9ce3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a406b470 con 0x7fa1a40c6900 2023-12-19T09:03:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a9ce3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a406be40 con 0x7fa1a415cc20 2023-12-19T09:03:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a9ce3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a406e110 con 0x7fa1a40c6cd0 2023-12-19T09:03:13.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1a415cc20 0x7fa1a415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1a415cc20 0x7fa1a415d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42398/0 (socket says 172.21.15.111:42398) 2023-12-19T09:03:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2360915036 learned_addr learned my addr 172.21.15.111:0/2360915036 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa19bfff700 1 --2- 172.21.15.111:0/2360915036 >> v2:172.21.15.143:3300/0 conn(0x7fa1a40c6cd0 0x7fa1a406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2360915036 >> v2:172.21.15.143:3300/0 conn(0x7fa1a40c6cd0 msgr2=0x7fa1a406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 --2- 172.21.15.111:0/2360915036 >> v2:172.21.15.143:3300/0 conn(0x7fa1a40c6cd0 0x7fa1a406b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2360915036 >> v1:172.21.15.143:6789/0 conn(0x7fa1a40c6900 legacy=0x7fa1a406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2360915036 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4161d10 con 0x7fa1a415cc20 2023-12-19T09:03:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa1a94e2700 1 --2- 172.21.15.111:0/2360915036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a415d080 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa1a406a030 tx=0x7fa19c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892a55945009d105 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:13.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa19b7fe700 1 -- 172.21.15.111:0/2360915036 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00b860 con 0x7fa1a415cc20 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa19b7fe700 1 -- 172.21.15.111:0/2360915036 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c00b9c0 con 0x7fa1a415cc20 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.819+0000 7fa19b7fe700 1 -- 172.21.15.111:0/2360915036 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00bd60 con 0x7fa1a415cc20 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2360915036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 msgr2=0x7fa1a415d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2360915036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a415d080 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa1a406a030 tx=0x7fa19c009250 comp rx=0 tx=0).stop 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2360915036 shutdown_connections 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2360915036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a415d080 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2360915036 >> v2:172.21.15.143:3300/0 conn(0x7fa1a40c6cd0 0x7fa1a406b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2360915036 >> 172.21.15.111:0/2360915036 conn(0x7fa1a4067bb0 msgr2=0x7fa1a40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:13.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2360915036 shutdown_connections 2023-12-19T09:03:13.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2360915036 wait complete. 2023-12-19T09:03:13.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 Processor -- start 2023-12-19T09:03:13.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.823+0000 7fa1a9ce3700 1 -- start start 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 0x7fa1a4150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1a40c6cd0 0x7fa1a4150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40b16e0 con 0x7fa1a415cc20 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40b0ad0 con 0x7fa1a40c6cd0 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40b0db0 con 0x7fa1a40c6900 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42412/0 (socket says 172.21.15.111:42412) 2023-12-19T09:03:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2544902363 learned_addr learned my addr 172.21.15.111:0/2544902363 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:13.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa19bfff700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1a40c6cd0 0x7fa1a4150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a8ce1700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 0x7fa1a4150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 msgr2=0x7fa1a4150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 0x7fa1a4150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1a40c6cd0 msgr2=0x7fa1a4150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1a40c6cd0 0x7fa1a4150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 -- 172.21.15.111:0/2544902363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c0094f0 con 0x7fa1a415cc20 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a8ce1700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 0x7fa1a4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a94e2700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009290 tx=0x7fa19c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c01a630 con 0x7fa1a415cc20 2023-12-19T09:03:13.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c01a790 con 0x7fa1a415cc20 2023-12-19T09:03:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a40b1090 con 0x7fa1a415cc20 2023-12-19T09:03:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c01ab50 con 0x7fa1a415cc20 2023-12-19T09:03:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.827+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4151360 con 0x7fa1a415cc20 2023-12-19T09:03:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.831+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a40b1890 con 0x7fa1a415cc20 2023-12-19T09:03:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.831+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa19c02d440 con 0x7fa1a415cc20 2023-12-19T09:03:13.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.835+0000 7fa198ff9700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 0x7fa190062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:13.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.835+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fa19c07e660 con 0x7fa1a415cc20 2023-12-19T09:03:13.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.835+0000 7fa1a8ce1700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 0x7fa190062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:13.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.835+0000 7fa1a8ce1700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 0x7fa190062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007fd0 tx=0x7fa1a0009040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:13.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:13.839+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa1a40b1890 con 0x7fa1a415cc20 2023-12-19T09:03:14.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.067+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa1a404be40 con 0x7fa1a415cc20 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa198ff9700 1 -- 172.21.15.111:0/2544902363 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa19c04ea90 con 0x7fa1a415cc20 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.077 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.079 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.082 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.083 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.084 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.086 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:03:14.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.088 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T09:03:14.089 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:03:14.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 msgr2=0x7fa190062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 0x7fa190062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0007fd0 tx=0x7fa1a0009040 comp rx=0 tx=0).stop 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 msgr2=0x7fa1a41522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009290 tx=0x7fa19c017000 comp rx=0 tx=0).stop 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 shutdown_connections 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa190060670 0x7fa190062b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1a40c6cd0 0x7fa1a4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a415cc20 0x7fa1a41522d0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 --2- 172.21.15.111:0/2544902363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1a40c6900 0x7fa1a4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.071+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 >> 172.21.15.111:0/2544902363 conn(0x7fa1a4067bb0 msgr2=0x7fa1a40abfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:14.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.075+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 shutdown_connections 2023-12-19T09:03:14.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.075+0000 7fa1a9ce3700 1 -- 172.21.15.111:0/2544902363 wait complete. 2023-12-19T09:03:14.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:14.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.259+0000 7fa4746a5700 1 Processor -- start 2023-12-19T09:03:14.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.259+0000 7fa4746a5700 1 -- start start 2023-12-19T09:03:14.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa4746a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa4746a5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46c06c9f0 con 0x7fa46c13f890 2023-12-19T09:03:14.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa4746a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46c0c7f20 con 0x7fa46c0680f0 2023-12-19T09:03:14.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa4746a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0c1c50 con 0x7fa46c0c2980 2023-12-19T09:03:14.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa472441700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa46c13f890 0x7fa46c0c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46300/0 (socket says 172.21.15.111:46300) 2023-12-19T09:03:14.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa472441700 1 -- 172.21.15.111:0/523329095 learned_addr learned my addr 172.21.15.111:0/523329095 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:14.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa471c40700 1 --2- 172.21.15.111:0/523329095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824175519 0 0) 0x7fa46c06c9f0 con 0x7fa46c13f890 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa458003570 con 0x7fa46c13f890 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732527693 0 0) 0x7fa46c0c7f20 con 0x7fa46c0680f0 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c06c9f0 con 0x7fa46c0680f0 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 257924072 0 0) 0x7fa458003570 con 0x7fa46c13f890 2023-12-19T09:03:14.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.263+0000 7fa471c40700 1 -- 172.21.15.111:0/523329095 >> v1:172.21.15.143:6789/0 conn(0x7fa46c0680f0 legacy=0x7fa46c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa471c40700 1 -- 172.21.15.111:0/523329095 >> v1:172.21.15.111:6789/0 conn(0x7fa46c13f890 legacy=0x7fa46c0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa471c40700 1 -- 172.21.15.111:0/523329095 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0c8190 con 0x7fa46c0c2980 2023-12-19T09:03:14.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa471c40700 1 --2- 172.21.15.111:0/523329095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c0aeba0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009910 tx=0x7fa45c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86df6049cad9c7a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468001470 con 0x7fa46c0c2980 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c011420 con 0x7fa46c0c2980 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa47143f700 1 -- 172.21.15.111:0/523329095 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0117c0 con 0x7fa46c0c2980 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 -- 172.21.15.111:0/523329095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 msgr2=0x7fa46c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 --2- 172.21.15.111:0/523329095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c0aeba0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009910 tx=0x7fa45c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 -- 172.21.15.111:0/523329095 shutdown_connections 2023-12-19T09:03:14.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 --2- 172.21.15.111:0/523329095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c0aeba0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 -- 172.21.15.111:0/523329095 >> 172.21.15.111:0/523329095 conn(0x7fa46c0c8850 msgr2=0x7fa46c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:14.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 -- 172.21.15.111:0/523329095 shutdown_connections 2023-12-19T09:03:14.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.267+0000 7fa4746a5700 1 -- 172.21.15.111:0/523329095 wait complete. 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 Processor -- start 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 -- start start 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 0x7fa46c0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0aa1c0 con 0x7fa46c13f890 2023-12-19T09:03:14.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c15d850 con 0x7fa46c0680f0 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa4746a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0abbc0 con 0x7fa46c0c2980 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42416/0 (socket says 172.21.15.111:42416) 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 -- 172.21.15.111:0/212250934 learned_addr learned my addr 172.21.15.111:0/212250934 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa471c40700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472441700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 0x7fa46c0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 -- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 msgr2=0x7fa46c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 -- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 msgr2=0x7fa46c0ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 0x7fa46c0ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472c42700 1 -- 172.21.15.111:0/212250934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c009420 con 0x7fa46c13f890 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.271+0000 7fa472441700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 0x7fa46c0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa472c42700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fa46400c910 tx=0x7fa4640027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464012630 con 0x7fa46c13f890 2023-12-19T09:03:14.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4640127b0 con 0x7fa46c13f890 2023-12-19T09:03:14.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c0a9ee0 con 0x7fa46c13f890 2023-12-19T09:03:14.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464012b70 con 0x7fa46c13f890 2023-12-19T09:03:14.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c15d4e0 con 0x7fa46c13f890 2023-12-19T09:03:14.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46c071120 con 0x7fa46c13f890 2023-12-19T09:03:14.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.275+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa46401b400 con 0x7fa46c13f890 2023-12-19T09:03:14.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.279+0000 7fa4627fc700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 0x7fa450062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.279+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fa46407cd30 con 0x7fa46c13f890 2023-12-19T09:03:14.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.279+0000 7fa472441700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 0x7fa450062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.279+0000 7fa472441700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 0x7fa450062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0cbc40 tx=0x7fa468000bc0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.287+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa46404d160 con 0x7fa46c13f890 2023-12-19T09:03:14.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.455+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa46c0642a0 con 0x7fa46c13f890 2023-12-19T09:03:14.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.455+0000 7fa4627fc700 1 -- 172.21.15.111:0/212250934 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+13231 (secure 0 0 0) 0x7fa464016090 con 0x7fa46c13f890 2023-12-19T09:03:14.463 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:14.463 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":171,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:03:13.619090+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 msgr2=0x7fa450062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 0x7fa450062b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0cbc40 tx=0x7fa468000bc0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 msgr2=0x7fa46c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fa46400c910 tx=0x7fa4640027a0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 shutdown_connections 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa4500606a0 0x7fa450062b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa46c13f890 0x7fa46c0cae00 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa46c0c2980 0x7fa46c15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 --2- 172.21.15.111:0/212250934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa46c0680f0 0x7fa46c0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 >> 172.21.15.111:0/212250934 conn(0x7fa46c0c8850 msgr2=0x7fa46c0adcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 shutdown_connections 2023-12-19T09:03:14.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.459+0000 7fa4746a5700 1 -- 172.21.15.111:0/212250934 wait complete. 2023-12-19T09:03:14.485 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:03:14.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:03:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.639+0000 7f84a1ae3700 1 Processor -- start 2023-12-19T09:03:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.639+0000 7f84a1ae3700 1 -- start start 2023-12-19T09:03:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a1ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 0x7f849c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a1ae3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c0c7090 con 0x7f849c0ad220 2023-12-19T09:03:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a1ae3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c06ddc0 con 0x7f849c06cd00 2023-12-19T09:03:14.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a1ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0ae9b0 con 0x7f849c155c90 2023-12-19T09:03:14.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a12e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f849c0ad220 0x7f849c0ad6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46312/0 (socket says 172.21.15.111:46312) 2023-12-19T09:03:14.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a12e2700 1 -- 172.21.15.111:0/1219605375 learned_addr learned my addr 172.21.15.111:0/1219605375 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:14.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a0ae1700 1 --2- 172.21.15.111:0/1219605375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 0x7f849c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f849b7fe700 1 -- 172.21.15.111:0/1219605375 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028788718 0 0) 0x7f849c0c7090 con 0x7f849c0ad220 2023-12-19T09:03:14.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f849b7fe700 1 -- 172.21.15.111:0/1219605375 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c003570 con 0x7f849c0ad220 2023-12-19T09:03:14.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a0ae1700 1 -- 172.21.15.111:0/1219605375 >> v1:172.21.15.143:6789/0 conn(0x7f849c06cd00 legacy=0x7f849c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a0ae1700 1 -- 172.21.15.111:0/1219605375 >> v1:172.21.15.111:6789/0 conn(0x7f849c0ad220 legacy=0x7f849c0ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a0ae1700 1 -- 172.21.15.111:0/1219605375 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0ae6d0 con 0x7f849c155c90 2023-12-19T09:03:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.643+0000 7f84a0ae1700 1 --2- 172.21.15.111:0/1219605375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 0x7f849c06c7c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8490009910 tx=0x7f84900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afae27a8e6c8103 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f849b7fe700 1 -- 172.21.15.111:0/1219605375 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000b680 con 0x7f849c155c90 2023-12-19T09:03:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f849b7fe700 1 -- 172.21.15.111:0/1219605375 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849000b7e0 con 0x7f849c155c90 2023-12-19T09:03:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f849b7fe700 1 -- 172.21.15.111:0/1219605375 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000bba0 con 0x7f849c155c90 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 -- 172.21.15.111:0/1219605375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 msgr2=0x7f849c06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/1219605375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 0x7f849c06c7c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8490009910 tx=0x7f84900091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 -- 172.21.15.111:0/1219605375 shutdown_connections 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/1219605375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c155c90 0x7f849c06c7c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 -- 172.21.15.111:0/1219605375 >> 172.21.15.111:0/1219605375 conn(0x7f849c0c8cd0 msgr2=0x7f849c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 -- 172.21.15.111:0/1219605375 shutdown_connections 2023-12-19T09:03:14.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.647+0000 7f84a1ae3700 1 -- 172.21.15.111:0/1219605375 wait complete. 2023-12-19T09:03:14.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 Processor -- start 2023-12-19T09:03:14.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 -- start start 2023-12-19T09:03:14.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c0ad220 0x7f849c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f849c155c90 0x7f849c0c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0ab940 con 0x7f849c06cd00 2023-12-19T09:03:14.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0aad30 con 0x7f849c155c90 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a1ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0ab010 con 0x7f849c0ad220 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42424/0 (socket says 172.21.15.111:42424) 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 -- 172.21.15.111:0/3279683041 learned_addr learned my addr 172.21.15.111:0/3279683041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f849bfff700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c0ad220 0x7f849c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a12e2700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f849c155c90 0x7f849c0c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 -- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c0ad220 msgr2=0x7f849c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c0ad220 0x7f849c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 -- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f849c155c90 msgr2=0x7f849c0c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f849c155c90 0x7f849c0c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.651+0000 7f84a0ae1700 1 -- 172.21.15.111:0/3279683041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009420 con 0x7f849c06cd00 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f84a0ae1700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f849c06a4d0 tx=0x7f8490017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000bb60 con 0x7f849c06cd00 2023-12-19T09:03:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849001a430 con 0x7f849c06cd00 2023-12-19T09:03:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c0ab2f0 con 0x7f849c06cd00 2023-12-19T09:03:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849001a7d0 con 0x7f849c06cd00 2023-12-19T09:03:14.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c1553f0 con 0x7f849c06cd00 2023-12-19T09:03:14.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c04be40 con 0x7f849c06cd00 2023-12-19T09:03:14.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.655+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f849001ab40 con 0x7f849c06cd00 2023-12-19T09:03:14.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.659+0000 7f8498ff9700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:14.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.659+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f849007e690 con 0x7f849c06cd00 2023-12-19T09:03:14.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.659+0000 7f849bfff700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:14.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.659+0000 7f849bfff700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 0x7f8480062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f849c063a40 tx=0x7f8488009040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:14.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.663+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f849004eb60 con 0x7f849c06cd00 2023-12-19T09:03:14.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.835+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f849c0c6a40 con 0x7f849c06cd00 2023-12-19T09:03:14.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f8498ff9700 1 -- 172.21.15.111:0/3279683041 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8490052180 con 0x7f849c06cd00 2023-12-19T09:03:14.844 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:03:14.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 msgr2=0x7f8480062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 0x7f8480062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f849c063a40 tx=0x7f8488009040 comp rx=0 tx=0).stop 2023-12-19T09:03:14.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 msgr2=0x7f849c154eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:14.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f849c06a4d0 tx=0x7f8490017000 comp rx=0 tx=0).stop 2023-12-19T09:03:14.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 shutdown_connections 2023-12-19T09:03:14.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f8480060670 0x7f8480062b20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f849c155c90 0x7f849c0c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f849c0ad220 0x7f849c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 --2- 172.21.15.111:0/3279683041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f849c06cd00 0x7f849c154eb0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 >> 172.21.15.111:0/3279683041 conn(0x7f849c0c8cd0 msgr2=0x7f849c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 shutdown_connections 2023-12-19T09:03:14.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:14.839+0000 7f84a1ae3700 1 -- 172.21.15.111:0/3279683041 wait complete. 2023-12-19T09:03:14.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T09:03:15.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.023+0000 7fb8ce078700 1 Processor -- start 2023-12-19T09:03:15.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.023+0000 7fb8ce078700 1 -- start start 2023-12-19T09:03:15.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8ce078700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb8c813f890 0x7fb8c80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8ce078700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c806c9f0 con 0x7fb8c80c2980 2023-12-19T09:03:15.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8ce078700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80c7f20 con 0x7fb8c80680f0 2023-12-19T09:03:15.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8ce078700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c80c1c50 con 0x7fb8c813f890 2023-12-19T09:03:15.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8c77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb8c813f890 0x7fb8c80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8c77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb8c813f890 0x7fb8c80aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42440/0 (socket says 172.21.15.111:42440) 2023-12-19T09:03:15.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8c77fe700 1 -- 172.21.15.111:0/111029513 learned_addr learned my addr 172.21.15.111:0/111029513 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:15.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.027+0000 7fb8c6ffd700 1 --1- 172.21.15.111:0/111029513 >> v1:172.21.15.111:6789/0 conn(0x7fb8c80c2980 0x7fb8c80c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46322/0 (socket says 172.21.15.111:46322) 2023-12-19T09:03:15.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c77fe700 1 -- 172.21.15.111:0/111029513 >> v1:172.21.15.143:6789/0 conn(0x7fb8c80680f0 legacy=0x7fb8c8068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:15.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c77fe700 1 -- 172.21.15.111:0/111029513 >> v1:172.21.15.111:6789/0 conn(0x7fb8c80c2980 legacy=0x7fb8c80c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c77fe700 1 -- 172.21.15.111:0/111029513 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80c8190 con 0x7fb8c813f890 2023-12-19T09:03:15.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c77fe700 1 --2- 172.21.15.111:0/111029513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c813f890 0x7fb8c80aeba0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80099b0 tx=0x7fb8b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2646898128f5bd3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c67fc700 1 -- 172.21.15.111:0/111029513 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800b820 con 0x7fb8c813f890 2023-12-19T09:03:15.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c67fc700 1 -- 172.21.15.111:0/111029513 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b800b980 con 0x7fb8c813f890 2023-12-19T09:03:15.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8c67fc700 1 -- 172.21.15.111:0/111029513 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800bd20 con 0x7fb8c813f890 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 -- 172.21.15.111:0/111029513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c813f890 msgr2=0x7fb8c80aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 --2- 172.21.15.111:0/111029513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c813f890 0x7fb8c80aeba0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80099b0 tx=0x7fb8b8009250 comp rx=0 tx=0).stop 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 -- 172.21.15.111:0/111029513 shutdown_connections 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 --2- 172.21.15.111:0/111029513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c813f890 0x7fb8c80aeba0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 -- 172.21.15.111:0/111029513 >> 172.21.15.111:0/111029513 conn(0x7fb8c80c8850 msgr2=0x7fb8c80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 -- 172.21.15.111:0/111029513 shutdown_connections 2023-12-19T09:03:15.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.031+0000 7fb8ce078700 1 -- 172.21.15.111:0/111029513 wait complete. 2023-12-19T09:03:15.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 Processor -- start 2023-12-19T09:03:15.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 -- start start 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c80680f0 0x7fb8c8155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c813f890 0x7fb8c80af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80cb020 con 0x7fb8c80c2980 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80ca410 con 0x7fb8c80680f0 2023-12-19T09:03:15.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8ce078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80ca6f0 con 0x7fb8c813f890 2023-12-19T09:03:15.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42450/0 (socket says 172.21.15.111:42450) 2023-12-19T09:03:15.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 -- 172.21.15.111:0/1575130007 learned_addr learned my addr 172.21.15.111:0/1575130007 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:15.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c77fe700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c80680f0 0x7fb8c8155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c7fff700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c813f890 0x7fb8c80af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 -- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c813f890 msgr2=0x7fb8c80af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c813f890 0x7fb8c80af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 -- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c80680f0 msgr2=0x7fb8c8155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c80680f0 0x7fb8c8155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 -- 172.21.15.111:0/1575130007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80094f0 con 0x7fb8c80c2980 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8c6ffd700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c790 tx=0x7fb8bc007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.035+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc005a70 con 0x7fb8c80c2980 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8bc005bf0 con 0x7fb8c80c2980 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0070d0 con 0x7fb8c80c2980 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c80ca9d0 con 0x7fb8c80c2980 2023-12-19T09:03:15.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c8159c40 con 0x7fb8c80c2980 2023-12-19T09:03:15.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c80cb1d0 con 0x7fb8c80c2980 2023-12-19T09:03:15.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.039+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb8bc007480 con 0x7fb8c80c2980 2023-12-19T09:03:15.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.043+0000 7fb8af7fe700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 0x7fb8b4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.043+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fb8bc07bd00 con 0x7fb8c80c2980 2023-12-19T09:03:15.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.043+0000 7fb8c77fe700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 0x7fb8b4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.043+0000 7fb8c77fe700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 0x7fb8b4062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80099b0 tx=0x7fb8b8016000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.047+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb8c80cb1d0 con 0x7fb8c80c2980 2023-12-19T09:03:15.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.223+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8c8159e60 con 0x7fb8c80c2980 2023-12-19T09:03:15.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.223+0000 7fb8af7fe700 1 -- 172.21.15.111:0/1575130007 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v171) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb8bc053e60 con 0x7fb8c80c2980 2023-12-19T09:03:15.233 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:15.233 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:03:15.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:15.233 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T09:03:15.234 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T09:03:15.234 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:15.234 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:03:15.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 msgr2=0x7fb8b4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 0x7fb8b4062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80099b0 tx=0x7fb8b8016000 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 msgr2=0x7fb8c80aecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c790 tx=0x7fb8bc007f40 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 shutdown_connections 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb8b40606a0 0x7fb8b4062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8c813f890 0x7fb8c80af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c80c2980 0x7fb8c80aecc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 --2- 172.21.15.111:0/1575130007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8c80680f0 0x7fb8c8155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 >> 172.21.15.111:0/1575130007 conn(0x7fb8c80c8850 msgr2=0x7fb8c813e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 shutdown_connections 2023-12-19T09:03:15.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.227+0000 7fb8ce078700 1 -- 172.21.15.111:0/1575130007 wait complete. 2023-12-19T09:03:15.253 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T09:03:15.254 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T09:03:15.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.411+0000 7f445359e700 1 Processor -- start 2023-12-19T09:03:15.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.411+0000 7f445359e700 1 -- start start 2023-12-19T09:03:15.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f445359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44540aedc0 0x7f44540af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f445359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445406b3d0 con 0x7f44540cac50 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f445359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540cb3f0 con 0x7f445406af50 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f445359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44540cc140 con 0x7f44540aedc0 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f4452d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f44540cac50 0x7f44540af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46324/0 (socket says 172.21.15.111:46324) 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f4452d9d700 1 -- 172.21.15.111:0/218960985 learned_addr learned my addr 172.21.15.111:0/218960985 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:15.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.415+0000 7f445259c700 1 --2- 172.21.15.111:0/218960985 >> v2:172.21.15.111:3300/0 conn(0x7f44540aedc0 0x7f44540af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738961003 0 0) 0x7f444c001490 con 0x7f44540cac50 2023-12-19T09:03:15.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f443c003570 con 0x7f44540cac50 2023-12-19T09:03:15.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366924036 0 0) 0x7f44540cb3f0 con 0x7f445406af50 2023-12-19T09:03:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445259c700 1 -- 172.21.15.111:0/218960985 >> v1:172.21.15.143:6789/0 conn(0x7f445406af50 legacy=0x7f445406d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445259c700 1 -- 172.21.15.111:0/218960985 >> v1:172.21.15.111:6789/0 conn(0x7f44540cac50 legacy=0x7f44540af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445259c700 1 -- 172.21.15.111:0/218960985 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445406d220 con 0x7f44540aedc0 2023-12-19T09:03:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445259c700 1 --2- 172.21.15.111:0/218960985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44540aedc0 0x7f44540af190 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f44440098e0 tx=0x7f44440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934e7224b9185057 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c001980 con 0x7f44540aedc0 2023-12-19T09:03:15.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4444011420 con 0x7f44540aedc0 2023-12-19T09:03:15.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445159a700 1 -- 172.21.15.111:0/218960985 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440117c0 con 0x7f44540aedc0 2023-12-19T09:03:15.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 -- 172.21.15.111:0/218960985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44540aedc0 msgr2=0x7f44540af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 --2- 172.21.15.111:0/218960985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44540aedc0 0x7f44540af190 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f44440098e0 tx=0x7f44440091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 -- 172.21.15.111:0/218960985 shutdown_connections 2023-12-19T09:03:15.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 --2- 172.21.15.111:0/218960985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44540aedc0 0x7f44540af190 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 -- 172.21.15.111:0/218960985 >> 172.21.15.111:0/218960985 conn(0x7f44540cc760 msgr2=0x7f44540ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:15.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 -- 172.21.15.111:0/218960985 shutdown_connections 2023-12-19T09:03:15.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.419+0000 7f445359e700 1 -- 172.21.15.111:0/218960985 wait complete. 2023-12-19T09:03:15.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 Processor -- start 2023-12-19T09:03:15.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 -- start start 2023-12-19T09:03:15.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44540aedc0 0x7f44540b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44540cac50 0x7f44540c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445415dc10 con 0x7f445406af50 2023-12-19T09:03:15.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445415d000 con 0x7f44540aedc0 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.423+0000 7f445359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445415d2e0 con 0x7f44540cac50 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42460/0 (socket says 172.21.15.111:42460) 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 -- 172.21.15.111:0/4001753143 learned_addr learned my addr 172.21.15.111:0/4001753143 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f4452d9d700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44540cac50 0x7f44540c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f4451d9b700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44540aedc0 0x7f44540b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 -- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44540cac50 msgr2=0x7f44540c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44540cac50 0x7f44540c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 -- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44540aedc0 msgr2=0x7f44540b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:15.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44540aedc0 0x7f44540b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:15.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 -- 172.21.15.111:0/4001753143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4444009420 con 0x7f445406af50 2023-12-19T09:03:15.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445259c700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f44440093f0 tx=0x7f4444017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444010b50 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4444010cd0 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444401d680 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445415d5c0 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.427+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44540b5160 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.431+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445404be40 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.431+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f444401dab0 con 0x7f445406af50 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.431+0000 7f44427fc700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 0x7f4434062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:15.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.431+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f444407e190 con 0x7f445406af50 2023-12-19T09:03:15.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.431+0000 7f4451d9b700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 0x7f4434062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:15.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.435+0000 7f4451d9b700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 0x7f4434062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f445415fc80 tx=0x7f4448007ec0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:15.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.439+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f444404e660 con 0x7f445406af50 2023-12-19T09:03:15.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:15.607+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f445406c8f0 con 0x7f445406af50 2023-12-19T09:03:16.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.227+0000 7f44427fc700 1 -- 172.21.15.111:0/4001753143 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v172) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4444051c80 con 0x7f445406af50 2023-12-19T09:03:16.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 msgr2=0x7f4434062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 0x7f4434062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f445415fc80 tx=0x7f4448007ec0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 msgr2=0x7f4454153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f44440093f0 tx=0x7f4444017000 comp rx=0 tx=0).stop 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 shutdown_connections 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f4434060670 0x7f4434062b20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44540aedc0 0x7f44540b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44540cac50 0x7f44540c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 --2- 172.21.15.111:0/4001753143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f445406af50 0x7f4454153440 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 >> 172.21.15.111:0/4001753143 conn(0x7f44540cc760 msgr2=0x7f44540afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 shutdown_connections 2023-12-19T09:03:16.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.235+0000 7f445359e700 1 -- 172.21.15.111:0/4001753143 wait complete. 2023-12-19T09:03:16.244 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T09:03:16.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T09:03:16.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.435+0000 7f1316d23700 1 Processor -- start 2023-12-19T09:03:16.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.435+0000 7f1316d23700 1 -- start start 2023-12-19T09:03:16.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316d23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1310158c90 0x7f13100caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316d23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f131006aeb0 0x7f1310159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316d23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131006c850 con 0x7f13101595a0 2023-12-19T09:03:16.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316d23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131006b470 con 0x7f1310158c90 2023-12-19T09:03:16.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316d23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131006b750 con 0x7f131006aeb0 2023-12-19T09:03:16.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315d21700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1310158c90 0x7f13100caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316522700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f13101595a0 0x7f131015c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46332/0 (socket says 172.21.15.111:46332) 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1316522700 1 -- 172.21.15.111:0/1390091641 learned_addr learned my addr 172.21.15.111:0/1390091641 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315520700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f131006aeb0 0x7f1310159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315520700 1 -- 172.21.15.111:0/1390091641 >> v2:172.21.15.111:3300/0 conn(0x7f1310158c90 msgr2=0x7f13100caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315520700 1 --2- 172.21.15.111:0/1390091641 >> v2:172.21.15.111:3300/0 conn(0x7f1310158c90 0x7f13100caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315520700 1 -- 172.21.15.111:0/1390091641 >> v1:172.21.15.111:6789/0 conn(0x7f13101595a0 legacy=0x7f131015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1315520700 1 -- 172.21.15.111:0/1390091641 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131006ba30 con 0x7f131006aeb0 2023-12-19T09:03:16.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.439+0000 7f1314d1f700 1 -- 172.21.15.111:0/1390091641 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276248496 0 0) 0x7f131006c850 con 0x7f13101595a0 2023-12-19T09:03:16.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1315520700 1 --2- 172.21.15.111:0/1390091641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f1310159060 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1310068000 tx=0x7f13000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dce8b261ee8f296a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:16.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1314d1f700 1 -- 172.21.15.111:0/1390091641 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300011630 con 0x7f131006aeb0 2023-12-19T09:03:16.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1314d1f700 1 -- 172.21.15.111:0/1390091641 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1300011790 con 0x7f131006aeb0 2023-12-19T09:03:16.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1314d1f700 1 -- 172.21.15.111:0/1390091641 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300011b30 con 0x7f131006aeb0 2023-12-19T09:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 -- 172.21.15.111:0/1390091641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 msgr2=0x7f1310159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 --2- 172.21.15.111:0/1390091641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f1310159060 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1310068000 tx=0x7f13000091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 -- 172.21.15.111:0/1390091641 shutdown_connections 2023-12-19T09:03:16.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 --2- 172.21.15.111:0/1390091641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f1310159060 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 --2- 172.21.15.111:0/1390091641 >> v2:172.21.15.111:3300/0 conn(0x7f1310158c90 0x7f13100caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 -- 172.21.15.111:0/1390091641 >> 172.21.15.111:0/1390091641 conn(0x7f1310067bb0 msgr2=0x7f13100cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 -- 172.21.15.111:0/1390091641 shutdown_connections 2023-12-19T09:03:16.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.443+0000 7f1316d23700 1 -- 172.21.15.111:0/1390091641 wait complete. 2023-12-19T09:03:16.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 Processor -- start 2023-12-19T09:03:16.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 -- start start 2023-12-19T09:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f13100c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1310158c90 0x7f1310157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13100c5c40 con 0x7f13101595a0 2023-12-19T09:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1310157580 con 0x7f131006aeb0 2023-12-19T09:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1310156c50 con 0x7f1310158c90 2023-12-19T09:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42466/0 (socket says 172.21.15.111:42466) 2023-12-19T09:03:16.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 -- 172.21.15.111:0/3795069228 learned_addr learned my addr 172.21.15.111:0/3795069228 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:16.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1315d21700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f13100c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1315520700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1310158c90 0x7f1310157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 -- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1310158c90 msgr2=0x7f1310157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1310158c90 0x7f1310157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 -- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 msgr2=0x7f13100c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f13100c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 -- 172.21.15.111:0/3795069228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300009420 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1315d21700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f13100c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316522700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f130800cc20 tx=0x7f13080019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130800a870 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130800a9f0 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130800adb0 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.447+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1310156f30 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.451+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13100c9c60 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.451+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131004be40 con 0x7f13101595a0 2023-12-19T09:03:16.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.451+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f1308007300 con 0x7f13101595a0 2023-12-19T09:03:16.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.455+0000 7f1305ffb700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 0x7f12f4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:16.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.455+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f130807cca0 con 0x7f13101595a0 2023-12-19T09:03:16.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.455+0000 7f1315d21700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 0x7f12f4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:16.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.455+0000 7f1315d21700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 0x7f12f4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13100ced80 tx=0x7f130c0091a0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.463+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f130804d080 con 0x7f13101595a0 2023-12-19T09:03:16.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:16.631+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f131006bd10 con 0x7f13101595a0 2023-12-19T09:03:17.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.243+0000 7f1305ffb700 1 -- 172.21.15.111:0/3795069228 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v173) v1 ==== 170+0+0 (secure 0 0 0) 0x7f13080506a0 con 0x7f13101595a0 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 msgr2=0x7f12f4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 0x7f12f4062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13100ced80 tx=0x7f130c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 msgr2=0x7f13100c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f130800cc20 tx=0x7f13080019b0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 shutdown_connections 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f12f4060670 0x7f12f4062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13101595a0 0x7f13100c5e70 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1310158c90 0x7f1310157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 --2- 172.21.15.111:0/3795069228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f131006aeb0 0x7f13100c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 >> 172.21.15.111:0/3795069228 conn(0x7f1310067bb0 msgr2=0x7f13100afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:17.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 shutdown_connections 2023-12-19T09:03:17.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.259+0000 7f1316d23700 1 -- 172.21.15.111:0/3795069228 wait complete. 2023-12-19T09:03:17.273 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T09:03:17.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T09:03:17.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 Processor -- start 2023-12-19T09:03:17.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 -- start start 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4104155750 0x7f4104155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410406c960 con 0x7f41040ae9c0 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410406cc40 con 0x7f41040c7e90 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4109678700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410406cf20 con 0x7f4104155750 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f41027fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f41040ae9c0 0x7f41040c7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46336/0 (socket says 172.21.15.111:46336) 2023-12-19T09:03:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f41027fc700 1 -- 172.21.15.111:0/3746683509 learned_addr learned my addr 172.21.15.111:0/3746683509 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4102ffd700 1 --2- 172.21.15.111:0/3746683509 >> v2:172.21.15.143:3300/0 conn(0x7f4104155750 0x7f4104155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231308928 0 0) 0x7f410406c960 con 0x7f41040ae9c0 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f0003570 con 0x7f41040ae9c0 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320301136 0 0) 0x7f40f0003570 con 0x7f41040ae9c0 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 >> v2:172.21.15.143:3300/0 conn(0x7f4104155750 msgr2=0x7f4104155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 --2- 172.21.15.111:0/3746683509 >> v2:172.21.15.143:3300/0 conn(0x7f4104155750 0x7f4104155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 >> v1:172.21.15.143:6789/0 conn(0x7f41040c7e90 legacy=0x7f410406d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410406d200 con 0x7f41040ae9c0 2023-12-19T09:03:17.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.443+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f40ec0017f0 con 0x7f41040ae9c0 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 >> v1:172.21.15.111:6789/0 conn(0x7f41040ae9c0 legacy=0x7f41040c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 0x7f40f0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f0004810 0x7f40f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f00051c0 0x7f40f000dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410406cf20 con 0x7f40f00051c0 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f000e370 con 0x7f40f0004810 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f000e530 con 0x7f40f0003a60 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 0x7f40f0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f41027fc700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f0004810 0x7f40f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f41037fe700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f00051c0 0x7f40f000dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 -- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f0004810 msgr2=0x7f40f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40f0004810 0x7f40f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 -- 172.21.15.111:0/3746683509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f00051c0 msgr2=0x7f40f000dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f00051c0 0x7f40f000dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 -- 172.21.15.111:0/3746683509 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0003570 con 0x7f40f0003a60 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f41037fe700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f00051c0 0x7f40f000dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f41037fe700 1 -- 172.21.15.111:0/3746683509 reap_dead start 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4102ffd700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 0x7f40f0003e70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009c00 tx=0x7f40fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=465493879bf24a68 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00e710 con 0x7f40f0003a60 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc00e870 con 0x7f40f0003a60 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4101ffb700 1 -- 172.21.15.111:0/3746683509 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00ec30 con 0x7f40f0003a60 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 -- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 msgr2=0x7f40f0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 0x7f40f0003e70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009c00 tx=0x7f40fc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 -- 172.21.15.111:0/3746683509 shutdown_connections 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 --2- 172.21.15.111:0/3746683509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40f0003a60 0x7f40f0003e70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 -- 172.21.15.111:0/3746683509 >> 172.21.15.111:0/3746683509 conn(0x7f4104067bb0 msgr2=0x7f41040cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 -- 172.21.15.111:0/3746683509 shutdown_connections 2023-12-19T09:03:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.447+0000 7f4109678700 1 -- 172.21.15.111:0/3746683509 wait complete. 2023-12-19T09:03:17.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 Processor -- start 2023-12-19T09:03:17.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 -- start start 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4104149490 0x7f410406d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4104149900 0x7f410406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4104156090 con 0x7f40f0003a60 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4104156330 con 0x7f4104149490 2023-12-19T09:03:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4109678700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104158810 con 0x7f4104149900 2023-12-19T09:03:17.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4102ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4102ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42490/0 (socket says 172.21.15.111:42490) 2023-12-19T09:03:17.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.451+0000 7f4102ffd700 1 -- 172.21.15.111:0/2009873112 learned_addr learned my addr 172.21.15.111:0/2009873112 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f41037fe700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4104149900 0x7f410406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f41027fc700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4104149490 0x7f410406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 -- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4104149900 msgr2=0x7f410406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4104149900 0x7f410406db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 -- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4104149490 msgr2=0x7f410406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4104149490 0x7f410406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 -- 172.21.15.111:0/2009873112 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104158af0 con 0x7f40f0003a60 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4102ffd700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e6c0 tx=0x7f40fc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc022070 con 0x7f40f0003a60 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc00e9e0 con 0x7f40f0003a60 2023-12-19T09:03:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104158dd0 con 0x7f40f0003a60 2023-12-19T09:03:17.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0276a0 con 0x7f40f0003a60 2023-12-19T09:03:17.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4104157c80 con 0x7f40f0003a60 2023-12-19T09:03:17.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.455+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410404be40 con 0x7f40f0003a60 2023-12-19T09:03:17.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.459+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f40fc01c450 con 0x7f40f0003a60 2023-12-19T09:03:17.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.459+0000 7f40faffd700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 0x7f40e4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:17.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.459+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f40fc07ebe0 con 0x7f40f0003a60 2023-12-19T09:03:17.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.459+0000 7f41027fc700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 0x7f40e4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.463+0000 7f41027fc700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 0x7f40e4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41040ae620 tx=0x7f40ec006040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.467+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f40fc04f390 con 0x7f40f0003a60 2023-12-19T09:03:17.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:17.675+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4104158f60 con 0x7f40f0003a60 2023-12-19T09:03:18.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.275+0000 7f40faffd700 1 -- 172.21.15.111:0/2009873112 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v76) v1 ==== 174+0+0 (secure 0 0 0) 0x7f40fc0170c0 con 0x7f40f0003a60 2023-12-19T09:03:18.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.287+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 msgr2=0x7f40e4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.287+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 0x7f40e4062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41040ae620 tx=0x7f40ec006040 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.287+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 msgr2=0x7f410414b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.287+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e6c0 tx=0x7f40fc01a000 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 shutdown_connections 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f40e4060670 0x7f40e4062b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4104149490 0x7f410406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4104149900 0x7f410406db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 --2- 172.21.15.111:0/2009873112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f0003a60 0x7f410414b490 secure :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e6c0 tx=0x7f40fc01a000 comp rx=0 tx=0).stop 2023-12-19T09:03:18.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 >> 172.21.15.111:0/2009873112 conn(0x7f4104067bb0 msgr2=0x7f41040c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:18.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 shutdown_connections 2023-12-19T09:03:18.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.291+0000 7f4109678700 1 -- 172.21.15.111:0/2009873112 wait complete. 2023-12-19T09:03:18.306 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 21 and data pool 22 2023-12-19T09:03:18.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:18.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.471+0000 7f3984ee3700 1 Processor -- start 2023-12-19T09:03:18.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.471+0000 7f3984ee3700 1 -- start start 2023-12-19T09:03:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 0x7f3980158df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 0x7f39800c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39800c1ef0 0x7f39800c2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800c7c30 con 0x7f39800c1ef0 2023-12-19T09:03:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800c9f70 con 0x7f3980158a20 2023-12-19T09:03:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3984ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800ca250 con 0x7f39800c80c0 2023-12-19T09:03:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39800c1ef0 0x7f39800c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39800c1ef0 0x7f39800c2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42500/0 (socket says 172.21.15.111:42500) 2023-12-19T09:03:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 -- 172.21.15.111:0/3529019306 learned_addr learned my addr 172.21.15.111:0/3529019306 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397f7fe700 1 --2- 172.21.15.111:0/3529019306 >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 0x7f3980158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3977fff700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 0x7f39800c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 -- 172.21.15.111:0/3529019306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 msgr2=0x7f39800c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 0x7f39800c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 -- 172.21.15.111:0/3529019306 >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 msgr2=0x7f3980158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 --2- 172.21.15.111:0/3529019306 >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 0x7f3980158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 -- 172.21.15.111:0/3529019306 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800c78c0 con 0x7f39800c1ef0 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f3977fff700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 0x7f39800c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397f7fe700 1 --2- 172.21.15.111:0/3529019306 >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 0x7f3980158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397ffff700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39800c2370 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3980063a20 tx=0x7f3978009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2942d3199f57bf7e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397effd700 1 -- 172.21.15.111:0/3529019306 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397800e070 con 0x7f39800c1ef0 2023-12-19T09:03:18.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397effd700 1 -- 172.21.15.111:0/3529019306 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397800ba50 con 0x7f39800c1ef0 2023-12-19T09:03:18.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.475+0000 7f397effd700 1 -- 172.21.15.111:0/3529019306 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39780092e0 con 0x7f39800c1ef0 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- 172.21.15.111:0/3529019306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 msgr2=0x7f39800c2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39800c2370 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3980063a20 tx=0x7f3978009840 comp rx=0 tx=0).stop 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- 172.21.15.111:0/3529019306 shutdown_connections 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39800c2370 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 --2- 172.21.15.111:0/3529019306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39800c80c0 0x7f39800c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 --2- 172.21.15.111:0/3529019306 >> v2:172.21.15.143:3300/0 conn(0x7f3980158a20 0x7f3980158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- 172.21.15.111:0/3529019306 >> 172.21.15.111:0/3529019306 conn(0x7f398006b670 msgr2=0x7f398006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- 172.21.15.111:0/3529019306 shutdown_connections 2023-12-19T09:03:18.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- 172.21.15.111:0/3529019306 wait complete. 2023-12-19T09:03:18.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 Processor -- start 2023-12-19T09:03:18.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.479+0000 7f3984ee3700 1 -- start start 2023-12-19T09:03:18.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39800c80c0 0x7f39801555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3980158a20 0x7f398016b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c57b0 con 0x7f39800c1ef0 2023-12-19T09:03:18.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3980155ed0 con 0x7f39800c80c0 2023-12-19T09:03:18.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39801561b0 con 0x7f3980158a20 2023-12-19T09:03:18.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42514/0 (socket says 172.21.15.111:42514) 2023-12-19T09:03:18.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 -- 172.21.15.111:0/877338818 learned_addr learned my addr 172.21.15.111:0/877338818 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:18.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3977fff700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39800c80c0 0x7f39801555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397ffff700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3980158a20 0x7f398016b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 -- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3980158a20 msgr2=0x7f398016b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3980158a20 0x7f398016b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 -- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39800c80c0 msgr2=0x7f39801555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39800c80c0 0x7f39801555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 -- 172.21.15.111:0/877338818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978009ae0 con 0x7f39800c1ef0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f397f7fe700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f397000d860 tx=0x7f3970008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700125f0 con 0x7f39800c1ef0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970012750 con 0x7f39800c1ef0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970012af0 con 0x7f39800c1ef0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3980156490 con 0x7f39800c1ef0 2023-12-19T09:03:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.483+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3980155b10 con 0x7f39800c1ef0 2023-12-19T09:03:18.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.487+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398004be40 con 0x7f39800c1ef0 2023-12-19T09:03:18.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.487+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3970014430 con 0x7f39800c1ef0 2023-12-19T09:03:18.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.487+0000 7f39777fe700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 0x7f3960062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.487+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f397007cf80 con 0x7f39800c1ef0 2023-12-19T09:03:18.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.491+0000 7f3977fff700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 0x7f3960062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.491+0000 7f3977fff700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 0x7f3960062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39800af470 tx=0x7f3968008040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.495+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3970050af0 con 0x7f39800c1ef0 2023-12-19T09:03:18.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.663+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39800c1aa0 con 0x7f39800c1ef0 2023-12-19T09:03:18.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.667+0000 7f39777fe700 1 -- 172.21.15.111:0/877338818 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16464 (secure 0 0 0) 0x7f3970050680 con 0x7f39800c1ef0 2023-12-19T09:03:18.673 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:18.674 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":174,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:03:17.683239+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.667+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 msgr2=0x7f3960062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.667+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 0x7f3960062b20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39800af470 tx=0x7f3968008040 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.667+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 msgr2=0x7f39801688e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.667+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f397000d860 tx=0x7f3970008040 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 shutdown_connections 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f3960060670 0x7f3960062b20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39800c80c0 0x7f39801555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3980158a20 0x7f398016b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 --2- 172.21.15.111:0/877338818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39800c1ef0 0x7f39801688e0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 >> 172.21.15.111:0/877338818 conn(0x7f398006b670 msgr2=0x7f3980067730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 shutdown_connections 2023-12-19T09:03:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.671+0000 7f3984ee3700 1 -- 172.21.15.111:0/877338818 wait complete. 2023-12-19T09:03:18.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:18.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.851+0000 7fc85f20c700 1 Processor -- start 2023-12-19T09:03:18.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.851+0000 7fc85f20c700 1 -- start start 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 0x7fc8580ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc85806a580 0x7fc85806a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc85815a860 0x7fc85815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85806c7c0 con 0x7fc85815a860 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85806cd80 con 0x7fc8580b1410 2023-12-19T09:03:18.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85806af10 con 0x7fc85806a580 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc85815a860 0x7fc85815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc85815a860 0x7fc85815aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42520/0 (socket says 172.21.15.111:42520) 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 -- 172.21.15.111:0/3147082140 learned_addr learned my addr 172.21.15.111:0/3147082140 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 -- 172.21.15.111:0/3147082140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc85806a580 msgr2=0x7fc85806a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/3147082140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc85806a580 0x7fc85806a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85e20a700 1 --2- 172.21.15.111:0/3147082140 >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 0x7fc8580ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 -- 172.21.15.111:0/3147082140 >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 msgr2=0x7fc8580ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/3147082140 >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 0x7fc8580ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 -- 172.21.15.111:0/3147082140 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85806b8f0 con 0x7fc85815a860 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85e20a700 1 --2- 172.21.15.111:0/3147082140 >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 0x7fc8580ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:18.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/3147082140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc85815aca0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fc858069f90 tx=0x7fc850009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5d53a432ca7e99 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85d208700 1 -- 172.21.15.111:0/3147082140 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85000e070 con 0x7fc85815a860 2023-12-19T09:03:18.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85d208700 1 -- 172.21.15.111:0/3147082140 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85000ba50 con 0x7fc85815a860 2023-12-19T09:03:18.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85d208700 1 -- 172.21.15.111:0/3147082140 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500092e0 con 0x7fc85815a860 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- 172.21.15.111:0/3147082140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 msgr2=0x7fc85815aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- 172.21.15.111:0/3147082140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc85815aca0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fc858069f90 tx=0x7fc850009840 comp rx=0 tx=0).stop 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- 172.21.15.111:0/3147082140 shutdown_connections 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- 172.21.15.111:0/3147082140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc85815aca0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- 172.21.15.111:0/3147082140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc85806a580 0x7fc85806a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 --2- 172.21.15.111:0/3147082140 >> v2:172.21.15.143:3300/0 conn(0x7fc8580b1410 0x7fc8580ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- 172.21.15.111:0/3147082140 >> 172.21.15.111:0/3147082140 conn(0x7fc858067bb0 msgr2=0x7fc8580cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- 172.21.15.111:0/3147082140 shutdown_connections 2023-12-19T09:03:18.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.855+0000 7fc85f20c700 1 -- 172.21.15.111:0/3147082140 wait complete. 2023-12-19T09:03:18.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 Processor -- start 2023-12-19T09:03:18.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 -- start start 2023-12-19T09:03:18.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc85806a580 0x7fc858155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 0x7fc8580c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85816a380 con 0x7fc85815a860 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc858158cb0 con 0x7fc85806a580 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85f20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8581586a0 con 0x7fc8580b1410 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42526/0 (socket says 172.21.15.111:42526) 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 -- 172.21.15.111:0/2747167421 learned_addr learned my addr 172.21.15.111:0/2747167421 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85e20a700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc85806a580 0x7fc858155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 -- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 msgr2=0x7fc8580c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85da09700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 0x7fc8580c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 0x7fc8580c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 -- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc85806a580 msgr2=0x7fc858155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:18.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc85806a580 0x7fc858155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:18.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85ea0b700 1 -- 172.21.15.111:0/2747167421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850009ae0 con 0x7fc85815a860 2023-12-19T09:03:18.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.859+0000 7fc85da09700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 0x7fc8580c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:18.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc85ea0b700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc858064040 tx=0x7fc850018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500032f0 con 0x7fc85815a860 2023-12-19T09:03:18.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc850008e80 con 0x7fc85815a860 2023-12-19T09:03:18.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85816a0a0 con 0x7fc85815a860 2023-12-19T09:03:18.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500256b0 con 0x7fc85815a860 2023-12-19T09:03:18.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85816a520 con 0x7fc85815a860 2023-12-19T09:03:18.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc858158500 con 0x7fc85815a860 2023-12-19T09:03:18.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.863+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fc850003040 con 0x7fc85815a860 2023-12-19T09:03:18.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.867+0000 7fc84e7fc700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 0x7fc83c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:18.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.867+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fc85007eb20 con 0x7fc85815a860 2023-12-19T09:03:18.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.867+0000 7fc85e20a700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 0x7fc83c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:18.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.867+0000 7fc85e20a700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 0x7fc83c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc8580c8170 tx=0x7fc854006f90 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:18.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:18.871+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc858158500 con 0x7fc85815a860 2023-12-19T09:03:19.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.047+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc858158500 con 0x7fc85815a860 2023-12-19T09:03:19.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.047+0000 7fc84e7fc700 1 -- 172.21.15.111:0/2747167421 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16464 (secure 0 0 0) 0x7fc858158500 con 0x7fc85815a860 2023-12-19T09:03:19.055 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:19.055 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":174,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:03:17.683239+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:19.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 msgr2=0x7fc83c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 0x7fc83c062b20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc8580c8170 tx=0x7fc854006f90 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 msgr2=0x7fc8580c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc858064040 tx=0x7fc850018000 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 shutdown_connections 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fc83c060670 0x7fc83c062b20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc85815a860 0x7fc8580c7a10 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8580b1410 0x7fc8580c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 --2- 172.21.15.111:0/2747167421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc85806a580 0x7fc858155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.051+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 >> 172.21.15.111:0/2747167421 conn(0x7fc858067bb0 msgr2=0x7fc8580b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.055+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 shutdown_connections 2023-12-19T09:03:19.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.055+0000 7fc85f20c700 1 -- 172.21.15.111:0/2747167421 wait complete. 2023-12-19T09:03:19.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T09:03:19.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.227+0000 7f331a7c5700 1 Processor -- start 2023-12-19T09:03:19.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.227+0000 7f331a7c5700 1 -- start start 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f331a7c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f331406c7c0 0x7f331406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f331a7c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f331406b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f331a7c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331406a7a0 con 0x7f33140c6d20 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f331a7c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331406d4b0 con 0x7f331406c7c0 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f331a7c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331406d790 con 0x7f331406ac30 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f33140c6d20 0x7f33140c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46346/0 (socket says 172.21.15.111:46346) 2023-12-19T09:03:19.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 -- 172.21.15.111:0/1546492597 learned_addr learned my addr 172.21.15.111:0/1546492597 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1546492597 >> v2:172.21.15.143:3300/0 conn(0x7f331406c7c0 0x7f331406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904205972 0 0) 0x7f331406a7a0 con 0x7f33140c6d20 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3318d62700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f331406b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003570 con 0x7f33140c6d20 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723049389 0 0) 0x7f3300003570 con 0x7f33140c6d20 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 msgr2=0x7f331406b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f331406b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 >> v2:172.21.15.143:3300/0 conn(0x7f331406c7c0 msgr2=0x7f331406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 --2- 172.21.15.111:0/1546492597 >> v2:172.21.15.143:3300/0 conn(0x7f331406c7c0 0x7f331406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331406da70 con 0x7f33140c6d20 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f3304001490 con 0x7f33140c6d20 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3318d62700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f331406b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1546492597 >> v2:172.21.15.143:3300/0 conn(0x7f331406c7c0 0x7f331406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 >> v1:172.21.15.111:6789/0 conn(0x7f33140c6d20 legacy=0x7f33140c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3300003a60 0x7f3300003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3300004810 0x7f3300004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f33000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331406d4b0 con 0x7f3300003a60 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331406d790 con 0x7f3300004810 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330000bb60 con 0x7f331406ac30 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3300003a60 0x7f3300003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3318d62700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f33000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3300004810 0x7f3300004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 -- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 msgr2=0x7f33000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f33000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 -- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3300004810 msgr2=0x7f3300004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3300004810 0x7f3300004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3313fff700 1 -- 172.21.15.111:0/1546492597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300003570 con 0x7f3300003a60 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.231+0000 7f3318d62700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f331406ac30 0x7f33000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:19.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f3313fff700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3300003a60 0x7f3300003e70 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f33140695f0 tx=0x7f3304007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1bce7d86b7bb76 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330400cc30 con 0x7f3300003a60 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330400cdb0 con 0x7f3300003a60 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f33137fe700 1 -- 172.21.15.111:0/1546492597 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304004b40 con 0x7f3300003a60 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 -- 172.21.15.111:0/1546492597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3300003a60 msgr2=0x7f3300003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1546492597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3300003a60 0x7f3300003e70 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f33140695f0 tx=0x7f3304007940 comp rx=0 tx=0).stop 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f3318d62700 1 -- 172.21.15.111:0/1546492597 reap_dead start 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 -- 172.21.15.111:0/1546492597 shutdown_connections 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 -- 172.21.15.111:0/1546492597 >> 172.21.15.111:0/1546492597 conn(0x7f3314067bb0 msgr2=0x7f33140cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 -- 172.21.15.111:0/1546492597 shutdown_connections 2023-12-19T09:03:19.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.235+0000 7f331a7c5700 1 -- 172.21.15.111:0/1546492597 wait complete. 2023-12-19T09:03:19.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 Processor -- start 2023-12-19T09:03:19.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 -- start start 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 0x7f3314146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33141523a0 0x7f3314152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314152d80 con 0x7f33140c3740 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33140c40f0 con 0x7f33141523a0 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f331a7c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33140af0f0 con 0x7f3300003a60 2023-12-19T09:03:19.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42538/0 (socket says 172.21.15.111:42538) 2023-12-19T09:03:19.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 -- 172.21.15.111:0/1635529374 learned_addr learned my addr 172.21.15.111:0/1635529374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:19.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f3318d62700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33141523a0 0x7f3314152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f3313fff700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 0x7f3314146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 -- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 msgr2=0x7f3314146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 0x7f3314146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 -- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33141523a0 msgr2=0x7f3314152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33141523a0 0x7f3314152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 -- 172.21.15.111:0/1635529374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3314146cc0 con 0x7f33140c3740 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f3313fff700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 0x7f3314146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f330b7fe700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f32fc010970 tx=0x7f32fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:19.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.239+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc010e60 con 0x7f33140c3740 2023-12-19T09:03:19.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc00c050 con 0x7f33140c3740 2023-12-19T09:03:19.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc00eaa0 con 0x7f33140c3740 2023-12-19T09:03:19.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33140ad800 con 0x7f33140c3740 2023-12-19T09:03:19.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33140c45e0 con 0x7f33140c3740 2023-12-19T09:03:19.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33140af2f0 con 0x7f33140c3740 2023-12-19T09:03:19.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.243+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f32fc015470 con 0x7f33140c3740 2023-12-19T09:03:19.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.247+0000 7f3310ff9700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 0x7f32f4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:19.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.247+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f32fc07d310 con 0x7f33140c3740 2023-12-19T09:03:19.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.247+0000 7f3313fff700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 0x7f32f4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:19.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.247+0000 7f3313fff700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 0x7f32f4062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3304007d40 tx=0x7f3304015000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:19.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.251+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f33140af2f0 con 0x7f33140c3740 2023-12-19T09:03:19.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:19.451+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f33140af2b0 con 0x7f33140c3740 2023-12-19T09:03:20.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.291+0000 7f3310ff9700 1 -- 172.21.15.111:0/1635529374 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v79) v1 ==== 118+0+0 (secure 0 0 0) 0x7f32fc04d2e0 con 0x7f33140c3740 2023-12-19T09:03:20.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 msgr2=0x7f32f4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 0x7f32f4062b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3304007d40 tx=0x7f3304015000 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 msgr2=0x7f33140c3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f32fc010970 tx=0x7f32fc008040 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 shutdown_connections 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f32f40606a0 0x7f32f4062b50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33141523a0 0x7f3314152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33140c3740 0x7f33140c3bb0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 --2- 172.21.15.111:0/1635529374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3300003a60 0x7f3314146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.295+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 >> 172.21.15.111:0/1635529374 conn(0x7f3314067bb0 msgr2=0x7f3314158530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:20.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.299+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 shutdown_connections 2023-12-19T09:03:20.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.299+0000 7f331a7c5700 1 -- 172.21.15.111:0/1635529374 wait complete. 2023-12-19T09:03:20.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T09:03:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.483+0000 7fb353003700 1 Processor -- start 2023-12-19T09:03:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.483+0000 7fb353003700 1 -- start start 2023-12-19T09:03:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb353003700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb34c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb353003700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34c06d970 con 0x7fb34c06c7c0 2023-12-19T09:03:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb353003700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34c06cd60 con 0x7fb34c06de00 2023-12-19T09:03:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb353003700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c06d040 con 0x7fb34c0c69a0 2023-12-19T09:03:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb352802700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb34c06c7c0 0x7fb34c15cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46350/0 (socket says 172.21.15.111:46350) 2023-12-19T09:03:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb352802700 1 -- 172.21.15.111:0/783362942 learned_addr learned my addr 172.21.15.111:0/783362942 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb352001700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb34c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475250241 0 0) 0x7fb34c06d970 con 0x7fb34c06c7c0 2023-12-19T09:03:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb338003570 con 0x7fb34c06c7c0 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684454347 0 0) 0x7fb34c06cd60 con 0x7fb34c06de00 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb34c06d970 con 0x7fb34c06de00 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423123946 0 0) 0x7fb338003570 con 0x7fb34c06c7c0 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 msgr2=0x7fb34c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb34c0c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 >> v1:172.21.15.143:6789/0 conn(0x7fb34c06de00 legacy=0x7fb34c06b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c06d320 con 0x7fb34c06c7c0 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fb3440017f0 con 0x7fb34c06c7c0 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 >> v1:172.21.15.111:6789/0 conn(0x7fb34c06c7c0 legacy=0x7fb34c15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb338003c20 0x7fb338004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338004810 0x7fb338004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb3380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c06d040 con 0x7fb338003c20 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33800bb40 con 0x7fb338004810 2023-12-19T09:03:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33800bd00 con 0x7fb34c0c69a0 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.487+0000 7fb352001700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb338003c20 0x7fb338004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352802700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb3380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb351800700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338004810 0x7fb338004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 -- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 msgr2=0x7fb3380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0c69a0 0x7fb3380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 -- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338004810 msgr2=0x7fb338004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338004810 0x7fb338004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 -- 172.21.15.111:0/783362942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338003570 con 0x7fb338003c20 2023-12-19T09:03:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb351800700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338004810 0x7fb338004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352001700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb338003c20 0x7fb338004010 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fb348009c00 tx=0x7fb3480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d22aa1909114bf2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34800e710 con 0x7fb338003c20 2023-12-19T09:03:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34800e870 con 0x7fb338003c20 2023-12-19T09:03:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb350fff700 1 -- 172.21.15.111:0/783362942 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34800ec30 con 0x7fb338003c20 2023-12-19T09:03:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 -- 172.21.15.111:0/783362942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb338003c20 msgr2=0x7fb338004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 --2- 172.21.15.111:0/783362942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb338003c20 0x7fb338004010 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fb348009c00 tx=0x7fb3480091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb352802700 1 -- 172.21.15.111:0/783362942 reap_dead start 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 -- 172.21.15.111:0/783362942 shutdown_connections 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 -- 172.21.15.111:0/783362942 >> 172.21.15.111:0/783362942 conn(0x7fb34c067b50 msgr2=0x7fb34c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 -- 172.21.15.111:0/783362942 shutdown_connections 2023-12-19T09:03:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.491+0000 7fb353003700 1 -- 172.21.15.111:0/783362942 wait complete. 2023-12-19T09:03:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 Processor -- start 2023-12-19T09:03:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 -- start start 2023-12-19T09:03:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338003c20 0x7fb34c170e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0cace0 0x7fb34c159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c0aefa0 con 0x7fb34c0ae990 2023-12-19T09:03:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c171360 con 0x7fb338003c20 2023-12-19T09:03:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb353003700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c171600 con 0x7fb34c0cace0 2023-12-19T09:03:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb351800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb351800700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42544/0 (socket says 172.21.15.111:42544) 2023-12-19T09:03:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.495+0000 7fb351800700 1 -- 172.21.15.111:0/3451187936 learned_addr learned my addr 172.21.15.111:0/3451187936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb352802700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0cace0 0x7fb34c159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb352001700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338003c20 0x7fb34c170e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 -- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0cace0 msgr2=0x7fb34c159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0cace0 0x7fb34c159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 -- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338003c20 msgr2=0x7fb34c170e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338003c20 0x7fb34c170e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 -- 172.21.15.111:0/3451187936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c1718e0 con 0x7fb34c0ae990 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb351800700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00e810 tx=0x7fb33c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c006820 con 0x7fb34c0ae990 2023-12-19T09:03:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c006980 con 0x7fb34c0ae990 2023-12-19T09:03:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c171bc0 con 0x7fb34c0ae990 2023-12-19T09:03:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c006d40 con 0x7fb34c0ae990 2023-12-19T09:03:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c159590 con 0x7fb34c0ae990 2023-12-19T09:03:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.499+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c159810 con 0x7fb34c0ae990 2023-12-19T09:03:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.503+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb33c007240 con 0x7fb34c0ae990 2023-12-19T09:03:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.503+0000 7fb341ffb700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 0x7fb330062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.503+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fb33c07cc40 con 0x7fb34c0ae990 2023-12-19T09:03:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.503+0000 7fb352001700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 0x7fb330062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.503+0000 7fb352001700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 0x7fb330062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb3480091e0 tx=0x7fb34801a000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.511+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb33c04cdc0 con 0x7fb34c0ae990 2023-12-19T09:03:20.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:20.719+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb34c159810 con 0x7fb34c0ae990 2023-12-19T09:03:21.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.299+0000 7fb341ffb700 1 -- 172.21.15.111:0/3451187936 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v80) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb33c0503e0 con 0x7fb34c0ae990 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 msgr2=0x7fb330062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 0x7fb330062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb3480091e0 tx=0x7fb34801a000 comp rx=0 tx=0).stop 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 msgr2=0x7fb34c0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00e810 tx=0x7fb33c009040 comp rx=0 tx=0).stop 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 shutdown_connections 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb330060670 0x7fb330062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb34c0cace0 0x7fb34c159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34c0ae990 0x7fb34c0ca7a0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 --2- 172.21.15.111:0/3451187936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb338003c20 0x7fb34c170e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 >> 172.21.15.111:0/3451187936 conn(0x7fb34c067b50 msgr2=0x7fb34c0add30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 shutdown_connections 2023-12-19T09:03:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.303+0000 7fb353003700 1 -- 172.21.15.111:0/3451187936 wait complete. 2023-12-19T09:03:21.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:21.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.491+0000 7f0bdea0c700 1 Processor -- start 2023-12-19T09:03:21.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 -- start start 2023-12-19T09:03:21.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0bd80b1410 0x7f0bd80ab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd806a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 0x7f0bd815aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd806c7c0 con 0x7f0bd80b1410 2023-12-19T09:03:21.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd806cd80 con 0x7f0bd815a860 2023-12-19T09:03:21.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdea0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd806af10 con 0x7f0bd806a580 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0bd80b1410 0x7f0bd80ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0bd80b1410 0x7f0bd80ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42548/0 (socket says 172.21.15.111:42548) 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 -- 172.21.15.111:0/3431980851 learned_addr learned my addr 172.21.15.111:0/3431980851 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdd209700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd806a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bde20b700 1 --2- 172.21.15.111:0/3431980851 >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 0x7f0bd815aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 -- 172.21.15.111:0/3431980851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 msgr2=0x7f0bd806a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd806a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 -- 172.21.15.111:0/3431980851 >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 msgr2=0x7f0bd815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/3431980851 >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 0x7f0bd815aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 -- 172.21.15.111:0/3431980851 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd806b8f0 con 0x7f0bd80b1410 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bde20b700 1 --2- 172.21.15.111:0/3431980851 >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 0x7f0bd815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:21.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd80ab440 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80ae680 tx=0x7f0bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e572a6ffff405e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdca08700 1 -- 172.21.15.111:0/3431980851 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400b860 con 0x7f0bd80b1410 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdca08700 1 -- 172.21.15.111:0/3431980851 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd400b9c0 con 0x7f0bd80b1410 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.495+0000 7f0bdca08700 1 -- 172.21.15.111:0/3431980851 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400bd60 con 0x7f0bd80b1410 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- 172.21.15.111:0/3431980851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 msgr2=0x7f0bd80ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd80ab440 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80ae680 tx=0x7f0bd4009250 comp rx=0 tx=0).stop 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- 172.21.15.111:0/3431980851 shutdown_connections 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/3431980851 >> v2:172.21.15.143:3300/0 conn(0x7f0bd815a860 0x7f0bd815aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd806a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/3431980851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd80ab440 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- 172.21.15.111:0/3431980851 >> 172.21.15.111:0/3431980851 conn(0x7f0bd8067bb0 msgr2=0x7f0bd80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- 172.21.15.111:0/3431980851 shutdown_connections 2023-12-19T09:03:21.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- 172.21.15.111:0/3431980851 wait complete. 2023-12-19T09:03:21.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 Processor -- start 2023-12-19T09:03:21.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.499+0000 7f0bdea0c700 1 -- start start 2023-12-19T09:03:21.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bd815a860 0x7f0bd81543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd81587f0 con 0x7f0bd80b1410 2023-12-19T09:03:21.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd8158a20 con 0x7f0bd815a860 2023-12-19T09:03:21.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd8158d00 con 0x7f0bd806a580 2023-12-19T09:03:21.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42556/0 (socket says 172.21.15.111:42556) 2023-12-19T09:03:21.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 -- 172.21.15.111:0/757974742 learned_addr learned my addr 172.21.15.111:0/757974742 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bde20b700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bd815a860 0x7f0bd81543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 -- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 msgr2=0x7f0bd8159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd8159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 -- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bd815a860 msgr2=0x7f0bd81543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bd815a860 0x7f0bd81543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 -- 172.21.15.111:0/757974742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40094f0 con 0x7f0bd80b1410 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdd209700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d890 tx=0x7f0bc8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80125f0 con 0x7f0bd80b1410 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc8012750 con 0x7f0bd80b1410 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd8158fe0 con 0x7f0bd80b1410 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8012ad0 con 0x7f0bd80b1410 2023-12-19T09:03:21.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.503+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd8154930 con 0x7f0bd80b1410 2023-12-19T09:03:21.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.507+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd80c0860 con 0x7f0bd80b1410 2023-12-19T09:03:21.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.507+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0bc801e460 con 0x7f0bd80b1410 2023-12-19T09:03:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.507+0000 7f0bcdffb700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 0x7f0bbc062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.507+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0bc807d2c0 con 0x7f0bd80b1410 2023-12-19T09:03:21.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.511+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 0x7f0bbc062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.511+0000 7f0bdda0a700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 0x7f0bbc062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80ae680 tx=0x7f0bd4017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.515+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0bd80c0860 con 0x7f0bd80b1410 2023-12-19T09:03:21.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.723+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd80c0860 con 0x7f0bd80b1410 2023-12-19T09:03:21.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.723+0000 7f0bcdffb700 1 -- 172.21.15.111:0/757974742 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7f0bd80c0860 con 0x7f0bd80b1410 2023-12-19T09:03:21.730 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:21.731 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 msgr2=0x7f0bbc062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 0x7f0bbc062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80ae680 tx=0x7f0bd4017000 comp rx=0 tx=0).stop 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 msgr2=0x7f0bd81582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d890 tx=0x7f0bc8008040 comp rx=0 tx=0).stop 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 shutdown_connections 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0bbc060670 0x7f0bbc062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bd815a860 0x7f0bd81543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd80b1410 0x7f0bd81582b0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 --2- 172.21.15.111:0/757974742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bd806a580 0x7f0bd8159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 >> 172.21.15.111:0/757974742 conn(0x7f0bd8067bb0 msgr2=0x7f0bd80b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:21.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 shutdown_connections 2023-12-19T09:03:21.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.727+0000 7f0bdea0c700 1 -- 172.21.15.111:0/757974742 wait complete. 2023-12-19T09:03:21.738 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:03:21.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:03:21.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.915+0000 7f0da56ae700 1 Processor -- start 2023-12-19T09:03:21.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.915+0000 7f0da56ae700 1 -- start start 2023-12-19T09:03:21.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0da56ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da013f890 0x7f0da00aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0da56ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0da00c2980 0x7f0da00c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0da56ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da006c9f0 con 0x7f0da00c1a80 2023-12-19T09:03:21.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0da56ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da00c7f20 con 0x7f0da00c2980 2023-12-19T09:03:21.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0da56ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da00c8190 con 0x7f0da013f890 2023-12-19T09:03:21.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d9f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0da00c1a80 0x7f0da00680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60496/0 (socket says 172.21.15.111:60496) 2023-12-19T09:03:21.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d9f7fe700 1 -- 172.21.15.111:0/2504918858 learned_addr learned my addr 172.21.15.111:0/2504918858 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 --2- 172.21.15.111:0/2504918858 >> v2:172.21.15.143:3300/0 conn(0x7f0da00c2980 0x7f0da00c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d9effd700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da013f890 0x7f0da00aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d9e7fc700 1 -- 172.21.15.111:0/2504918858 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828611425 0 0) 0x7f0da006c9f0 con 0x7f0da00c1a80 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d9e7fc700 1 -- 172.21.15.111:0/2504918858 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d8c003570 con 0x7f0da00c1a80 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 -- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da013f890 msgr2=0x7f0da00aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da013f890 0x7f0da00aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 -- 172.21.15.111:0/2504918858 >> v1:172.21.15.111:6789/0 conn(0x7f0da00c1a80 legacy=0x7f0da00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 -- 172.21.15.111:0/2504918858 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da00c8470 con 0x7f0da00c2980 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.919+0000 7f0d97fff700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da00c2980 0x7f0da00c1540 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0d880098e0 tx=0x7f0d880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445423d74eaaadd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0d9e7fc700 1 -- 172.21.15.111:0/2504918858 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88011630 con 0x7f0da00c2980 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0d9e7fc700 1 -- 172.21.15.111:0/2504918858 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d88011790 con 0x7f0da00c2980 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0d9e7fc700 1 -- 172.21.15.111:0/2504918858 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88011b30 con 0x7f0da00c2980 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 -- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da00c2980 msgr2=0x7f0da00c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da00c2980 0x7f0da00c1540 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0d880098e0 tx=0x7f0d880091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 -- 172.21.15.111:0/2504918858 shutdown_connections 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da00c2980 0x7f0da00c1540 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 --2- 172.21.15.111:0/2504918858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da013f890 0x7f0da00aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 -- 172.21.15.111:0/2504918858 >> 172.21.15.111:0/2504918858 conn(0x7f0da00c8850 msgr2=0x7f0da00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 -- 172.21.15.111:0/2504918858 shutdown_connections 2023-12-19T09:03:21.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 -- 172.21.15.111:0/2504918858 wait complete. 2023-12-19T09:03:21.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.923+0000 7f0da56ae700 1 Processor -- start 2023-12-19T09:03:21.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- start start 2023-12-19T09:03:21.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 0x7f0da00c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da00b07b0 0x7f0da006ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da0139050 con 0x7f0da00c2980 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da00c4e00 con 0x7f0da013f890 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da00c41f0 con 0x7f0da00b07b0 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46966/0 (socket says 172.21.15.111:46966) 2023-12-19T09:03:21.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 -- 172.21.15.111:0/3231294420 learned_addr learned my addr 172.21.15.111:0/3231294420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d97fff700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 0x7f0da00c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9f7fe700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da00b07b0 0x7f0da006ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 -- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da00b07b0 msgr2=0x7f0da006ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da00b07b0 0x7f0da006ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 -- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 msgr2=0x7f0da00c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 0x7f0da00c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 -- 172.21.15.111:0/3231294420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88009420 con 0x7f0da00c2980 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d97fff700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 0x7f0da00c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d9effd700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000e930 tx=0x7f0d900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000eed0 con 0x7f0da00c2980 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da00c44d0 con 0x7f0da00c2980 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9000c850 con 0x7f0da00c2980 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000cbf0 con 0x7f0da00c2980 2023-12-19T09:03:21.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.927+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da00c4a90 con 0x7f0da00c2980 2023-12-19T09:03:21.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.931+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da0071120 con 0x7f0da00c2980 2023-12-19T09:03:21.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.931+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0d90013470 con 0x7f0da00c2980 2023-12-19T09:03:21.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.931+0000 7f0d96ffd700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 0x7f0d80062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.935+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0d9007cea0 con 0x7f0da00c2980 2023-12-19T09:03:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.935+0000 7f0d97fff700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 0x7f0d80062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:21.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.935+0000 7f0d97fff700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 0x7f0d80062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0d880091e0 tx=0x7f0d88017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:21.939+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d9004d020 con 0x7f0da00c2980 2023-12-19T09:03:22.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.107+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0da00642a0 con 0x7f0da00c2980 2023-12-19T09:03:22.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.111+0000 7f0d96ffd700 1 -- 172.21.15.111:0/3231294420 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+16451 (secure 0 0 0) 0x7f0d90050640 con 0x7f0da00c2980 2023-12-19T09:03:22.119 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:22.119 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":175,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:03:19.271162+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:03:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 msgr2=0x7f0d80062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 0x7f0d80062b50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0d880091e0 tx=0x7f0d88017000 comp rx=0 tx=0).stop 2023-12-19T09:03:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 msgr2=0x7f0da0160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000e930 tx=0x7f0d900091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 shutdown_connections 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0d800606a0 0x7f0d80062b50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da013f890 0x7f0da00c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da00b07b0 0x7f0da006ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 --2- 172.21.15.111:0/3231294420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0da00c2980 0x7f0da0160da0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 >> 172.21.15.111:0/3231294420 conn(0x7f0da00c8850 msgr2=0x7f0da0135000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 shutdown_connections 2023-12-19T09:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.115+0000 7f0da56ae700 1 -- 172.21.15.111:0/3231294420 wait complete. 2023-12-19T09:03:22.141 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T09:03:22.141 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:03:22.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 Processor -- start 2023-12-19T09:03:22.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 -- start start 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb9e40c5380 0x7fb9e4163ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406c7c0 con 0x7fb9e4158d60 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406caa0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e8923700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e415d130 con 0x7fb9e40c5380 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb9e40c5380 0x7fb9e4163ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb9e40c5380 0x7fb9e4163ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46970/0 (socket says 172.21.15.111:46970) 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 -- 172.21.15.111:0/1891877457 learned_addr learned my addr 172.21.15.111:0/1891877457 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:22.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e37fe700 1 --1- 172.21.15.111:0/1891877457 >> v1:172.21.15.111:6789/0 conn(0x7fb9e4158d60 0x7fb9e415c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60500/0 (socket says 172.21.15.111:60500) 2023-12-19T09:03:22.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 -- 172.21.15.111:0/1891877457 >> v1:172.21.15.143:6789/0 conn(0x7fb9e406b7a0 legacy=0x7fb9e40c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 -- 172.21.15.111:0/1891877457 >> v1:172.21.15.111:6789/0 conn(0x7fb9e4158d60 legacy=0x7fb9e415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.295+0000 7fb9e3fff700 1 -- 172.21.15.111:0/1891877457 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e415d410 con 0x7fb9e40c5380 2023-12-19T09:03:22.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.299+0000 7fb9e3fff700 1 --2- 172.21.15.111:0/1891877457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e40c5380 0x7fb9e4163ca0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009f20 tx=0x7fb9dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688e477607b3bd9a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.299+0000 7fb9e27fc700 1 -- 172.21.15.111:0/1891877457 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00e070 con 0x7fb9e40c5380 2023-12-19T09:03:22.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.299+0000 7fb9e27fc700 1 -- 172.21.15.111:0/1891877457 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00ba80 con 0x7fb9e40c5380 2023-12-19T09:03:22.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.299+0000 7fb9e27fc700 1 -- 172.21.15.111:0/1891877457 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0092e0 con 0x7fb9e40c5380 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- 172.21.15.111:0/1891877457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e40c5380 msgr2=0x7fb9e4163ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 --2- 172.21.15.111:0/1891877457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e40c5380 0x7fb9e4163ca0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009f20 tx=0x7fb9dc009840 comp rx=0 tx=0).stop 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- 172.21.15.111:0/1891877457 shutdown_connections 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 --2- 172.21.15.111:0/1891877457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e40c5380 0x7fb9e4163ca0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- 172.21.15.111:0/1891877457 >> 172.21.15.111:0/1891877457 conn(0x7fb9e4067bb0 msgr2=0x7fb9e40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- 172.21.15.111:0/1891877457 shutdown_connections 2023-12-19T09:03:22.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- 172.21.15.111:0/1891877457 wait complete. 2023-12-19T09:03:22.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 Processor -- start 2023-12-19T09:03:22.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.303+0000 7fb9e8923700 1 -- start start 2023-12-19T09:03:22.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e40c5380 0x7fb9e414d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4158d60 0x7fb9e414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e414eaa0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e414de90 con 0x7fb9e4158d60 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e414e170 con 0x7fb9e40c5380 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46980/0 (socket says 172.21.15.111:46980) 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 -- 172.21.15.111:0/2344293107 learned_addr learned my addr 172.21.15.111:0/2344293107 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e3fff700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4158d60 0x7fb9e414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e2ffd700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e40c5380 0x7fb9e414d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 -- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e40c5380 msgr2=0x7fb9e414d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e40c5380 0x7fb9e414d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 -- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4158d60 msgr2=0x7fb9e414ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4158d60 0x7fb9e414ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 -- 172.21.15.111:0/2344293107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc009ae0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e37fe700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c560 tx=0x7fb9d4007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4005a70 con 0x7fb9e406b7a0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d4005bd0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d40070d0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e414e450 con 0x7fb9e406b7a0 2023-12-19T09:03:22.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.307+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e4149880 con 0x7fb9e406b7a0 2023-12-19T09:03:22.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.311+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e404be40 con 0x7fb9e406b7a0 2023-12-19T09:03:22.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:03:22.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.311+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fb9d4007480 con 0x7fb9e406b7a0 2023-12-19T09:03:22.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.311+0000 7fb9cb7fe700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 0x7fb9d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:03:22.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.315+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fb9d407c0a0 con 0x7fb9e406b7a0 2023-12-19T09:03:22.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.315+0000 7fb9e2ffd700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 0x7fb9d0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.315+0000 7fb9e2ffd700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 0x7fb9d0062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8003a80 tx=0x7fb9d8007ec0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.319+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9d404c180 con 0x7fb9e406b7a0 2023-12-19T09:03:22.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.483+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9e414ebe0 con 0x7fb9d0060670 2023-12-19T09:03:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.503+0000 7fb9cb7fe700 1 -- 172.21.15.111:0/2344293107 <== mgr.7862 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287343 (secure 0 0 0) 0x7fb9d8002480 con 0x7fb9d0060670 2023-12-19T09:03:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 msgr2=0x7fb9d0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 0x7fb9d0062b20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8003a80 tx=0x7fb9d8007ec0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 msgr2=0x7fb9e4149340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c560 tx=0x7fb9d4007f70 comp rx=0 tx=0).stop 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 shutdown_connections 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fb9d0060670 0x7fb9d0062b20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4158d60 0x7fb9e414ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e40c5380 0x7fb9e414d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 --2- 172.21.15.111:0/2344293107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e406b7a0 0x7fb9e4149340 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 >> 172.21.15.111:0/2344293107 conn(0x7fb9e4067bb0 msgr2=0x7fb9e40c4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 shutdown_connections 2023-12-19T09:03:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.511+0000 7fb9e8923700 1 -- 172.21.15.111:0/2344293107 wait complete. 2023-12-19T09:03:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:03:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:03:21.632897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466148,"kb_used_data":18212,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752508572,"statfs":{"total":773094113280,"available":770568777728,"internally_reserved":0,"allocated":18649088,"data_stored":10430265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.403559"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296171+0000","last_change":"2023-12-19T09:03:18.296171+0000","last_active":"2023-12-19T09:03:18.296171+0000","last_peered":"2023-12-19T09:03:18.296171+0000","last_clean":"2023-12-19T09:03:18.296171+0000","last_became_active":"2023-12-19T09:03:18.290561+0000","last_became_peered":"2023-12-19T09:03:18.290561+0000","last_unstale":"2023-12-19T09:03:18.296171+0000","last_undegraded":"2023-12-19T09:03:18.296171+0000","last_fullsized":"2023-12-19T09:03:18.296171+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:54.050974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279643+0000","last_change":"2023-12-19T09:03:18.279643+0000","last_active":"2023-12-19T09:03:18.279643+0000","last_peered":"2023-12-19T09:03:18.279643+0000","last_clean":"2023-12-19T09:03:18.279643+0000","last_became_active":"2023-12-19T09:03:18.279335+0000","last_became_peered":"2023-12-19T09:03:18.279335+0000","last_unstale":"2023-12-19T09:03:18.279643+0000","last_undegraded":"2023-12-19T09:03:18.279643+0000","last_fullsized":"2023-12-19T09:03:18.279643+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:51.502865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.294058+0000","last_change":"2023-12-19T09:03:18.294058+0000","last_active":"2023-12-19T09:03:18.294058+0000","last_peered":"2023-12-19T09:03:18.294058+0000","last_clean":"2023-12-19T09:03:18.294058+0000","last_became_active":"2023-12-19T09:03:18.293663+0000","last_became_peered":"2023-12-19T09:03:18.293663+0000","last_unstale":"2023-12-19T09:03:18.294058+0000","last_undegraded":"2023-12-19T09:03:18.294058+0000","last_fullsized":"2023-12-19T09:03:18.294058+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:54:26.569633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298751+0000","last_change":"2023-12-19T09:03:18.293273+0000","last_active":"2023-12-19T09:03:18.298751+0000","last_peered":"2023-12-19T09:03:18.298751+0000","last_clean":"2023-12-19T09:03:18.298751+0000","last_became_active":"2023-12-19T09:03:18.292914+0000","last_became_peered":"2023-12-19T09:03:18.292914+0000","last_unstale":"2023-12-19T09:03:18.298751+0000","last_undegraded":"2023-12-19T09:03:18.298751+0000","last_fullsized":"2023-12-19T09:03:18.298751+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:21.948191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283696+0000","last_change":"2023-12-19T09:03:18.283696+0000","last_active":"2023-12-19T09:03:18.283696+0000","last_peered":"2023-12-19T09:03:18.283696+0000","last_clean":"2023-12-19T09:03:18.283696+0000","last_became_active":"2023-12-19T09:03:18.283305+0000","last_became_peered":"2023-12-19T09:03:18.283305+0000","last_unstale":"2023-12-19T09:03:18.283696+0000","last_undegraded":"2023-12-19T09:03:18.283696+0000","last_fullsized":"2023-12-19T09:03:18.283696+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:03.411954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285897+0000","last_change":"2023-12-19T09:03:18.285897+0000","last_active":"2023-12-19T09:03:18.285897+0000","last_peered":"2023-12-19T09:03:18.285897+0000","last_clean":"2023-12-19T09:03:18.285897+0000","last_became_active":"2023-12-19T09:03:18.283544+0000","last_became_peered":"2023-12-19T09:03:18.283544+0000","last_unstale":"2023-12-19T09:03:18.285897+0000","last_undegraded":"2023-12-19T09:03:18.285897+0000","last_fullsized":"2023-12-19T09:03:18.285897+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:48.448681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297901+0000","last_change":"2023-12-19T09:03:18.297901+0000","last_active":"2023-12-19T09:03:18.297901+0000","last_peered":"2023-12-19T09:03:18.297901+0000","last_clean":"2023-12-19T09:03:18.297901+0000","last_became_active":"2023-12-19T09:03:18.296795+0000","last_became_peered":"2023-12-19T09:03:18.296795+0000","last_unstale":"2023-12-19T09:03:18.297901+0000","last_undegraded":"2023-12-19T09:03:18.297901+0000","last_fullsized":"2023-12-19T09:03:18.297901+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:08.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.284567+0000","last_change":"2023-12-19T09:03:18.284567+0000","last_active":"2023-12-19T09:03:18.284567+0000","last_peered":"2023-12-19T09:03:18.284567+0000","last_clean":"2023-12-19T09:03:18.284567+0000","last_became_active":"2023-12-19T09:03:18.283996+0000","last_became_peered":"2023-12-19T09:03:18.283996+0000","last_unstale":"2023-12-19T09:03:18.284567+0000","last_undegraded":"2023-12-19T09:03:18.284567+0000","last_fullsized":"2023-12-19T09:03:18.284567+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:45.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293353+0000","last_change":"2023-12-19T09:03:18.293353+0000","last_active":"2023-12-19T09:03:18.293353+0000","last_peered":"2023-12-19T09:03:18.293353+0000","last_clean":"2023-12-19T09:03:18.293353+0000","last_became_active":"2023-12-19T09:03:18.293038+0000","last_became_peered":"2023-12-19T09:03:18.293038+0000","last_unstale":"2023-12-19T09:03:18.293353+0000","last_undegraded":"2023-12-19T09:03:18.293353+0000","last_fullsized":"2023-12-19T09:03:18.293353+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:35:36.680028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279511+0000","last_change":"2023-12-19T09:03:18.279511+0000","last_active":"2023-12-19T09:03:18.279511+0000","last_peered":"2023-12-19T09:03:18.279511+0000","last_clean":"2023-12-19T09:03:18.279511+0000","last_became_active":"2023-12-19T09:03:18.279216+0000","last_became_peered":"2023-12-19T09:03:18.279216+0000","last_unstale":"2023-12-19T09:03:18.279511+0000","last_undegraded":"2023-12-19T09:03:18.279511+0000","last_fullsized":"2023-12-19T09:03:18.279511+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:28.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291486+0000","last_change":"2023-12-19T09:03:18.291486+0000","last_active":"2023-12-19T09:03:18.291486+0000","last_peered":"2023-12-19T09:03:18.291486+0000","last_clean":"2023-12-19T09:03:18.291486+0000","last_became_active":"2023-12-19T09:03:18.290768+0000","last_became_peered":"2023-12-19T09:03:18.290768+0000","last_unstale":"2023-12-19T09:03:18.291486+0000","last_undegraded":"2023-12-19T09:03:18.291486+0000","last_fullsized":"2023-12-19T09:03:18.291486+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:33.967037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297890+0000","last_change":"2023-12-19T09:03:18.288562+0000","last_active":"2023-12-19T09:03:18.297890+0000","last_peered":"2023-12-19T09:03:18.297890+0000","last_clean":"2023-12-19T09:03:18.297890+0000","last_became_active":"2023-12-19T09:03:18.288283+0000","last_became_peered":"2023-12-19T09:03:18.288283+0000","last_unstale":"2023-12-19T09:03:18.297890+0000","last_undegraded":"2023-12-19T09:03:18.297890+0000","last_fullsized":"2023-12-19T09:03:18.297890+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:03.164273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286906+0000","last_change":"2023-12-19T09:03:18.286906+0000","last_active":"2023-12-19T09:03:18.286906+0000","last_peered":"2023-12-19T09:03:18.286906+0000","last_clean":"2023-12-19T09:03:18.286906+0000","last_became_active":"2023-12-19T09:03:18.286542+0000","last_became_peered":"2023-12-19T09:03:18.286542+0000","last_unstale":"2023-12-19T09:03:18.286906+0000","last_undegraded":"2023-12-19T09:03:18.286906+0000","last_fullsized":"2023-12-19T09:03:18.286906+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:55:13.193453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297840+0000","last_change":"2023-12-19T09:03:18.290695+0000","last_active":"2023-12-19T09:03:18.297840+0000","last_peered":"2023-12-19T09:03:18.297840+0000","last_clean":"2023-12-19T09:03:18.297840+0000","last_became_active":"2023-12-19T09:03:18.290413+0000","last_became_peered":"2023-12-19T09:03:18.290413+0000","last_unstale":"2023-12-19T09:03:18.297840+0000","last_undegraded":"2023-12-19T09:03:18.297840+0000","last_fullsized":"2023-12-19T09:03:18.297840+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:52:59.793131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292690+0000","last_change":"2023-12-19T09:03:18.292690+0000","last_active":"2023-12-19T09:03:18.292690+0000","last_peered":"2023-12-19T09:03:18.292690+0000","last_clean":"2023-12-19T09:03:18.292690+0000","last_became_active":"2023-12-19T09:03:18.292010+0000","last_became_peered":"2023-12-19T09:03:18.292010+0000","last_unstale":"2023-12-19T09:03:18.292690+0000","last_undegraded":"2023-12-19T09:03:18.292690+0000","last_fullsized":"2023-12-19T09:03:18.292690+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:08.238021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296661+0000","last_change":"2023-12-19T09:03:18.296661+0000","last_active":"2023-12-19T09:03:18.296661+0000","last_peered":"2023-12-19T09:03:18.296661+0000","last_clean":"2023-12-19T09:03:18.296661+0000","last_became_active":"2023-12-19T09:03:18.296297+0000","last_became_peered":"2023-12-19T09:03:18.296297+0000","last_unstale":"2023-12-19T09:03:18.296661+0000","last_undegraded":"2023-12-19T09:03:18.296661+0000","last_fullsized":"2023-12-19T09:03:18.296661+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:08:54.450057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298675+0000","last_change":"2023-12-19T09:03:18.284446+0000","last_active":"2023-12-19T09:03:18.298675+0000","last_peered":"2023-12-19T09:03:18.298675+0000","last_clean":"2023-12-19T09:03:18.298675+0000","last_became_active":"2023-12-19T09:03:18.284142+0000","last_became_peered":"2023-12-19T09:03:18.284142+0000","last_unstale":"2023-12-19T09:03:18.298675+0000","last_undegraded":"2023-12-19T09:03:18.298675+0000","last_fullsized":"2023-12-19T09:03:18.298675+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:22.599375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.288655+0000","last_change":"2023-12-19T09:03:18.288655+0000","last_active":"2023-12-19T09:03:18.288655+0000","last_peered":"2023-12-19T09:03:18.288655+0000","last_clean":"2023-12-19T09:03:18.288655+0000","last_became_active":"2023-12-19T09:03:18.288183+0000","last_became_peered":"2023-12-19T09:03:18.288183+0000","last_unstale":"2023-12-19T09:03:18.288655+0000","last_undegraded":"2023-12-19T09:03:18.288655+0000","last_fullsized":"2023-12-19T09:03:18.288655+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:34.113825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.303554+0000","last_change":"2023-12-19T09:03:18.303554+0000","last_active":"2023-12-19T09:03:18.303554+0000","last_peered":"2023-12-19T09:03:18.303554+0000","last_clean":"2023-12-19T09:03:18.303554+0000","last_became_active":"2023-12-19T09:03:18.301508+0000","last_became_peered":"2023-12-19T09:03:18.301508+0000","last_unstale":"2023-12-19T09:03:18.303554+0000","last_undegraded":"2023-12-19T09:03:18.303554+0000","last_fullsized":"2023-12-19T09:03:18.303554+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:57.924691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281182+0000","last_change":"2023-12-19T09:03:18.276536+0000","last_active":"2023-12-19T09:03:18.281182+0000","last_peered":"2023-12-19T09:03:18.281182+0000","last_clean":"2023-12-19T09:03:18.281182+0000","last_became_active":"2023-12-19T09:03:18.276236+0000","last_became_peered":"2023-12-19T09:03:18.276236+0000","last_unstale":"2023-12-19T09:03:18.281182+0000","last_undegraded":"2023-12-19T09:03:18.281182+0000","last_fullsized":"2023-12-19T09:03:18.281182+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:54.050974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292743+0000","last_change":"2023-12-19T09:03:18.292743+0000","last_active":"2023-12-19T09:03:18.292743+0000","last_peered":"2023-12-19T09:03:18.292743+0000","last_clean":"2023-12-19T09:03:18.292743+0000","last_became_active":"2023-12-19T09:03:18.292470+0000","last_became_peered":"2023-12-19T09:03:18.292470+0000","last_unstale":"2023-12-19T09:03:18.292743+0000","last_undegraded":"2023-12-19T09:03:18.292743+0000","last_fullsized":"2023-12-19T09:03:18.292743+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:28.127539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290383+0000","last_change":"2023-12-19T09:03:18.290383+0000","last_active":"2023-12-19T09:03:18.290383+0000","last_peered":"2023-12-19T09:03:18.290383+0000","last_clean":"2023-12-19T09:03:18.290383+0000","last_became_active":"2023-12-19T09:03:18.290096+0000","last_became_peered":"2023-12-19T09:03:18.290096+0000","last_unstale":"2023-12-19T09:03:18.290383+0000","last_undegraded":"2023-12-19T09:03:18.290383+0000","last_fullsized":"2023-12-19T09:03:18.290383+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:58.449153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286411+0000","last_change":"2023-12-19T09:03:18.286411+0000","last_active":"2023-12-19T09:03:18.286411+0000","last_peered":"2023-12-19T09:03:18.286411+0000","last_clean":"2023-12-19T09:03:18.286411+0000","last_became_active":"2023-12-19T09:03:18.286153+0000","last_became_peered":"2023-12-19T09:03:18.286153+0000","last_unstale":"2023-12-19T09:03:18.286411+0000","last_undegraded":"2023-12-19T09:03:18.286411+0000","last_fullsized":"2023-12-19T09:03:18.286411+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:51.884295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283743+0000","last_change":"2023-12-19T09:03:18.283743+0000","last_active":"2023-12-19T09:03:18.283743+0000","last_peered":"2023-12-19T09:03:18.283743+0000","last_clean":"2023-12-19T09:03:18.283743+0000","last_became_active":"2023-12-19T09:03:18.283387+0000","last_became_peered":"2023-12-19T09:03:18.283387+0000","last_unstale":"2023-12-19T09:03:18.283743+0000","last_undegraded":"2023-12-19T09:03:18.283743+0000","last_fullsized":"2023-12-19T09:03:18.283743+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:20.644044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.289483+0000","last_change":"2023-12-19T09:03:18.289483+0000","last_active":"2023-12-19T09:03:18.289483+0000","last_peered":"2023-12-19T09:03:18.289483+0000","last_clean":"2023-12-19T09:03:18.289483+0000","last_became_active":"2023-12-19T09:03:18.288335+0000","last_became_peered":"2023-12-19T09:03:18.288335+0000","last_unstale":"2023-12-19T09:03:18.289483+0000","last_undegraded":"2023-12-19T09:03:18.289483+0000","last_fullsized":"2023-12-19T09:03:18.289483+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290177+0000","last_change":"2023-12-19T09:03:18.290177+0000","last_active":"2023-12-19T09:03:18.290177+0000","last_peered":"2023-12-19T09:03:18.290177+0000","last_clean":"2023-12-19T09:03:18.290177+0000","last_became_active":"2023-12-19T09:03:18.289526+0000","last_became_peered":"2023-12-19T09:03:18.289526+0000","last_unstale":"2023-12-19T09:03:18.290177+0000","last_undegraded":"2023-12-19T09:03:18.290177+0000","last_fullsized":"2023-12-19T09:03:18.290177+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:54.402255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301753+0000","last_change":"2023-12-19T09:03:18.301753+0000","last_active":"2023-12-19T09:03:18.301753+0000","last_peered":"2023-12-19T09:03:18.301753+0000","last_clean":"2023-12-19T09:03:18.301753+0000","last_became_active":"2023-12-19T09:03:18.301414+0000","last_became_peered":"2023-12-19T09:03:18.301414+0000","last_unstale":"2023-12-19T09:03:18.301753+0000","last_undegraded":"2023-12-19T09:03:18.301753+0000","last_fullsized":"2023-12-19T09:03:18.301753+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:37:17.109651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274655+0000","last_change":"2023-12-19T09:03:18.274655+0000","last_active":"2023-12-19T09:03:18.274655+0000","last_peered":"2023-12-19T09:03:18.274655+0000","last_clean":"2023-12-19T09:03:18.274655+0000","last_became_active":"2023-12-19T09:03:18.274282+0000","last_became_peered":"2023-12-19T09:03:18.274282+0000","last_unstale":"2023-12-19T09:03:18.274655+0000","last_undegraded":"2023-12-19T09:03:18.274655+0000","last_fullsized":"2023-12-19T09:03:18.274655+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:31.967552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285799+0000","last_change":"2023-12-19T09:03:18.285799+0000","last_active":"2023-12-19T09:03:18.285799+0000","last_peered":"2023-12-19T09:03:18.285799+0000","last_clean":"2023-12-19T09:03:18.285799+0000","last_became_active":"2023-12-19T09:03:18.284892+0000","last_became_peered":"2023-12-19T09:03:18.284892+0000","last_unstale":"2023-12-19T09:03:18.285799+0000","last_undegraded":"2023-12-19T09:03:18.285799+0000","last_fullsized":"2023-12-19T09:03:18.285799+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:23.045513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293451+0000","last_change":"2023-12-19T09:03:18.293451+0000","last_active":"2023-12-19T09:03:18.293451+0000","last_peered":"2023-12-19T09:03:18.293451+0000","last_clean":"2023-12-19T09:03:18.293451+0000","last_became_active":"2023-12-19T09:03:18.293115+0000","last_became_peered":"2023-12-19T09:03:18.293115+0000","last_unstale":"2023-12-19T09:03:18.293451+0000","last_undegraded":"2023-12-19T09:03:18.293451+0000","last_fullsized":"2023-12-19T09:03:18.293451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:53.370426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296597+0000","last_change":"2023-12-19T09:03:18.296597+0000","last_active":"2023-12-19T09:03:18.296597+0000","last_peered":"2023-12-19T09:03:18.296597+0000","last_clean":"2023-12-19T09:03:18.296597+0000","last_became_active":"2023-12-19T09:03:18.296295+0000","last_became_peered":"2023-12-19T09:03:18.296295+0000","last_unstale":"2023-12-19T09:03:18.296597+0000","last_undegraded":"2023-12-19T09:03:18.296597+0000","last_fullsized":"2023-12-19T09:03:18.296597+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:06.058067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.284307+0000","last_change":"2023-12-19T09:03:18.284307+0000","last_active":"2023-12-19T09:03:18.284307+0000","last_peered":"2023-12-19T09:03:18.284307+0000","last_clean":"2023-12-19T09:03:18.284307+0000","last_became_active":"2023-12-19T09:03:18.283621+0000","last_became_peered":"2023-12-19T09:03:18.283621+0000","last_unstale":"2023-12-19T09:03:18.284307+0000","last_undegraded":"2023-12-19T09:03:18.284307+0000","last_fullsized":"2023-12-19T09:03:18.284307+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:44.588695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292409+0000","last_change":"2023-12-19T09:03:18.292409+0000","last_active":"2023-12-19T09:03:18.292409+0000","last_peered":"2023-12-19T09:03:18.292409+0000","last_clean":"2023-12-19T09:03:18.292409+0000","last_became_active":"2023-12-19T09:03:18.292031+0000","last_became_peered":"2023-12-19T09:03:18.292031+0000","last_unstale":"2023-12-19T09:03:18.292409+0000","last_undegraded":"2023-12-19T09:03:18.292409+0000","last_fullsized":"2023-12-19T09:03:18.292409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306051+0000","last_change":"2023-12-19T09:03:17.255555+0000","last_active":"2023-12-19T09:03:18.306051+0000","last_peered":"2023-12-19T09:03:18.306051+0000","last_clean":"2023-12-19T09:03:18.306051+0000","last_became_active":"2023-12-19T09:03:17.255192+0000","last_became_peered":"2023-12-19T09:03:17.255192+0000","last_unstale":"2023-12-19T09:03:18.306051+0000","last_undegraded":"2023-12-19T09:03:18.306051+0000","last_fullsized":"2023-12-19T09:03:18.306051+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:10.364450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279417+0000","last_change":"2023-12-19T09:03:18.279417+0000","last_active":"2023-12-19T09:03:18.279417+0000","last_peered":"2023-12-19T09:03:18.279417+0000","last_clean":"2023-12-19T09:03:18.279417+0000","last_became_active":"2023-12-19T09:03:18.279101+0000","last_became_peered":"2023-12-19T09:03:18.279101+0000","last_unstale":"2023-12-19T09:03:18.279417+0000","last_undegraded":"2023-12-19T09:03:18.279417+0000","last_fullsized":"2023-12-19T09:03:18.279417+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:19:52.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.287660+0000","last_change":"2023-12-19T09:03:17.269617+0000","last_active":"2023-12-19T09:03:18.287660+0000","last_peered":"2023-12-19T09:03:18.287660+0000","last_clean":"2023-12-19T09:03:18.287660+0000","last_became_active":"2023-12-19T09:03:17.269249+0000","last_became_peered":"2023-12-19T09:03:17.269249+0000","last_unstale":"2023-12-19T09:03:18.287660+0000","last_undegraded":"2023-12-19T09:03:18.287660+0000","last_fullsized":"2023-12-19T09:03:18.287660+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:53.388587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280809+0000","last_change":"2023-12-19T09:03:18.280809+0000","last_active":"2023-12-19T09:03:18.280809+0000","last_peered":"2023-12-19T09:03:18.280809+0000","last_clean":"2023-12-19T09:03:18.280809+0000","last_became_active":"2023-12-19T09:03:18.280449+0000","last_became_peered":"2023-12-19T09:03:18.280449+0000","last_unstale":"2023-12-19T09:03:18.280809+0000","last_undegraded":"2023-12-19T09:03:18.280809+0000","last_fullsized":"2023-12-19T09:03:18.280809+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:14.067771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.308917+0000","last_change":"2023-12-19T09:03:17.271627+0000","last_active":"2023-12-19T09:03:18.308917+0000","last_peered":"2023-12-19T09:03:18.308917+0000","last_clean":"2023-12-19T09:03:18.308917+0000","last_became_active":"2023-12-19T09:03:17.271281+0000","last_became_peered":"2023-12-19T09:03:17.271281+0000","last_unstale":"2023-12-19T09:03:18.308917+0000","last_undegraded":"2023-12-19T09:03:18.308917+0000","last_fullsized":"2023-12-19T09:03:18.308917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:53.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301726+0000","last_change":"2023-12-19T09:03:18.301726+0000","last_active":"2023-12-19T09:03:18.301726+0000","last_peered":"2023-12-19T09:03:18.301726+0000","last_clean":"2023-12-19T09:03:18.301726+0000","last_became_active":"2023-12-19T09:03:18.301436+0000","last_became_peered":"2023-12-19T09:03:18.301436+0000","last_unstale":"2023-12-19T09:03:18.301726+0000","last_undegraded":"2023-12-19T09:03:18.301726+0000","last_fullsized":"2023-12-19T09:03:18.301726+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:11.378118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.303278+0000","last_change":"2023-12-19T09:03:17.262871+0000","last_active":"2023-12-19T09:03:18.303278+0000","last_peered":"2023-12-19T09:03:18.303278+0000","last_clean":"2023-12-19T09:03:18.303278+0000","last_became_active":"2023-12-19T09:03:17.262583+0000","last_became_peered":"2023-12-19T09:03:17.262583+0000","last_unstale":"2023-12-19T09:03:18.303278+0000","last_undegraded":"2023-12-19T09:03:18.303278+0000","last_fullsized":"2023-12-19T09:03:18.303278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:32.153702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292884+0000","last_change":"2023-12-19T09:03:18.282998+0000","last_active":"2023-12-19T09:03:18.292884+0000","last_peered":"2023-12-19T09:03:18.292884+0000","last_clean":"2023-12-19T09:03:18.292884+0000","last_became_active":"2023-12-19T09:03:18.282598+0000","last_became_peered":"2023-12-19T09:03:18.282598+0000","last_unstale":"2023-12-19T09:03:18.292884+0000","last_undegraded":"2023-12-19T09:03:18.292884+0000","last_fullsized":"2023-12-19T09:03:18.292884+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.266338+0000","last_change":"2023-12-19T09:03:17.258207+0000","last_active":"2023-12-19T09:03:18.266338+0000","last_peered":"2023-12-19T09:03:18.266338+0000","last_clean":"2023-12-19T09:03:18.266338+0000","last_became_active":"2023-12-19T09:03:17.257930+0000","last_became_peered":"2023-12-19T09:03:17.257930+0000","last_unstale":"2023-12-19T09:03:18.266338+0000","last_undegraded":"2023-12-19T09:03:18.266338+0000","last_fullsized":"2023-12-19T09:03:18.266338+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:38.014458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296879+0000","last_change":"2023-12-19T09:03:18.283423+0000","last_active":"2023-12-19T09:03:18.296879+0000","last_peered":"2023-12-19T09:03:18.296879+0000","last_clean":"2023-12-19T09:03:18.296879+0000","last_became_active":"2023-12-19T09:03:18.283108+0000","last_became_peered":"2023-12-19T09:03:18.283108+0000","last_unstale":"2023-12-19T09:03:18.296879+0000","last_undegraded":"2023-12-19T09:03:18.296879+0000","last_fullsized":"2023-12-19T09:03:18.296879+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:09:34.132918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279330+0000","last_change":"2023-12-19T09:03:17.262122+0000","last_active":"2023-12-19T09:03:18.279330+0000","last_peered":"2023-12-19T09:03:18.279330+0000","last_clean":"2023-12-19T09:03:18.279330+0000","last_became_active":"2023-12-19T09:03:17.261747+0000","last_became_peered":"2023-12-19T09:03:17.261747+0000","last_unstale":"2023-12-19T09:03:18.279330+0000","last_undegraded":"2023-12-19T09:03:18.279330+0000","last_fullsized":"2023-12-19T09:03:18.279330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:44.664658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281692+0000","last_change":"2023-12-19T09:03:18.281692+0000","last_active":"2023-12-19T09:03:18.281692+0000","last_peered":"2023-12-19T09:03:18.281692+0000","last_clean":"2023-12-19T09:03:18.281692+0000","last_became_active":"2023-12-19T09:03:18.281360+0000","last_became_peered":"2023-12-19T09:03:18.281360+0000","last_unstale":"2023-12-19T09:03:18.281692+0000","last_undegraded":"2023-12-19T09:03:18.281692+0000","last_fullsized":"2023-12-19T09:03:18.281692+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:09.637960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.272625+0000","last_change":"2023-12-19T09:03:17.269992+0000","last_active":"2023-12-19T09:03:17.272625+0000","last_peered":"2023-12-19T09:03:17.272625+0000","last_clean":"2023-12-19T09:03:17.272625+0000","last_became_active":"2023-12-19T09:03:17.269712+0000","last_became_peered":"2023-12-19T09:03:17.269712+0000","last_unstale":"2023-12-19T09:03:17.272625+0000","last_undegraded":"2023-12-19T09:03:17.272625+0000","last_fullsized":"2023-12-19T09:03:17.272625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:43.575027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280905+0000","last_change":"2023-12-19T09:03:18.280905+0000","last_active":"2023-12-19T09:03:18.280905+0000","last_peered":"2023-12-19T09:03:18.280905+0000","last_clean":"2023-12-19T09:03:18.280905+0000","last_became_active":"2023-12-19T09:03:18.280339+0000","last_became_peered":"2023-12-19T09:03:18.280339+0000","last_unstale":"2023-12-19T09:03:18.280905+0000","last_undegraded":"2023-12-19T09:03:18.280905+0000","last_fullsized":"2023-12-19T09:03:18.280905+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:39:52.030789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.263326+0000","last_change":"2023-12-19T09:03:17.260603+0000","last_active":"2023-12-19T09:03:17.263326+0000","last_peered":"2023-12-19T09:03:17.263326+0000","last_clean":"2023-12-19T09:03:17.263326+0000","last_became_active":"2023-12-19T09:03:17.260314+0000","last_became_peered":"2023-12-19T09:03:17.260314+0000","last_unstale":"2023-12-19T09:03:17.263326+0000","last_undegraded":"2023-12-19T09:03:17.263326+0000","last_fullsized":"2023-12-19T09:03:17.263326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:12.272351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292598+0000","last_change":"2023-12-19T09:03:18.292598+0000","last_active":"2023-12-19T09:03:18.292598+0000","last_peered":"2023-12-19T09:03:18.292598+0000","last_clean":"2023-12-19T09:03:18.292598+0000","last_became_active":"2023-12-19T09:03:18.292280+0000","last_became_peered":"2023-12-19T09:03:18.292280+0000","last_unstale":"2023-12-19T09:03:18.292598+0000","last_undegraded":"2023-12-19T09:03:18.292598+0000","last_fullsized":"2023-12-19T09:03:18.292598+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:30.094154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.274675+0000","last_change":"2023-12-19T09:03:17.272325+0000","last_active":"2023-12-19T09:03:17.274675+0000","last_peered":"2023-12-19T09:03:17.274675+0000","last_clean":"2023-12-19T09:03:17.274675+0000","last_became_active":"2023-12-19T09:03:17.271811+0000","last_became_peered":"2023-12-19T09:03:17.271811+0000","last_unstale":"2023-12-19T09:03:17.274675+0000","last_undegraded":"2023-12-19T09:03:17.274675+0000","last_fullsized":"2023-12-19T09:03:17.274675+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:08.624292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290290+0000","last_change":"2023-12-19T09:03:18.290290+0000","last_active":"2023-12-19T09:03:18.290290+0000","last_peered":"2023-12-19T09:03:18.290290+0000","last_clean":"2023-12-19T09:03:18.290290+0000","last_became_active":"2023-12-19T09:03:18.289898+0000","last_became_peered":"2023-12-19T09:03:18.289898+0000","last_unstale":"2023-12-19T09:03:18.290290+0000","last_undegraded":"2023-12-19T09:03:18.290290+0000","last_fullsized":"2023-12-19T09:03:18.290290+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:09.219048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.273062+0000","last_change":"2023-12-19T09:03:17.271126+0000","last_active":"2023-12-19T09:03:17.273062+0000","last_peered":"2023-12-19T09:03:17.273062+0000","last_clean":"2023-12-19T09:03:17.273062+0000","last_became_active":"2023-12-19T09:03:17.270876+0000","last_became_peered":"2023-12-19T09:03:17.270876+0000","last_unstale":"2023-12-19T09:03:17.273062+0000","last_undegraded":"2023-12-19T09:03:17.273062+0000","last_fullsized":"2023-12-19T09:03:17.273062+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:29.080486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.288465+0000","last_change":"2023-12-19T09:03:18.288465+0000","last_active":"2023-12-19T09:03:18.288465+0000","last_peered":"2023-12-19T09:03:18.288465+0000","last_clean":"2023-12-19T09:03:18.288465+0000","last_became_active":"2023-12-19T09:03:18.287809+0000","last_became_peered":"2023-12-19T09:03:18.287809+0000","last_unstale":"2023-12-19T09:03:18.288465+0000","last_undegraded":"2023-12-19T09:03:18.288465+0000","last_fullsized":"2023-12-19T09:03:18.288465+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:39.028126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306182+0000","last_change":"2023-12-19T09:03:17.269226+0000","last_active":"2023-12-19T09:03:18.306182+0000","last_peered":"2023-12-19T09:03:18.306182+0000","last_clean":"2023-12-19T09:03:18.306182+0000","last_became_active":"2023-12-19T09:03:17.268946+0000","last_became_peered":"2023-12-19T09:03:17.268946+0000","last_unstale":"2023-12-19T09:03:18.306182+0000","last_undegraded":"2023-12-19T09:03:18.306182+0000","last_fullsized":"2023-12-19T09:03:18.306182+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:59:19.308422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291428+0000","last_change":"2023-12-19T09:03:18.291428+0000","last_active":"2023-12-19T09:03:18.291428+0000","last_peered":"2023-12-19T09:03:18.291428+0000","last_clean":"2023-12-19T09:03:18.291428+0000","last_became_active":"2023-12-19T09:03:18.291116+0000","last_became_peered":"2023-12-19T09:03:18.291116+0000","last_unstale":"2023-12-19T09:03:18.291428+0000","last_undegraded":"2023-12-19T09:03:18.291428+0000","last_fullsized":"2023-12-19T09:03:18.291428+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:33.167370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.304096+0000","last_change":"2023-12-19T09:03:17.266754+0000","last_active":"2023-12-19T09:03:18.304096+0000","last_peered":"2023-12-19T09:03:18.304096+0000","last_clean":"2023-12-19T09:03:18.304096+0000","last_became_active":"2023-12-19T09:03:17.266220+0000","last_became_peered":"2023-12-19T09:03:17.266220+0000","last_unstale":"2023-12-19T09:03:18.304096+0000","last_undegraded":"2023-12-19T09:03:18.304096+0000","last_fullsized":"2023-12-19T09:03:18.304096+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:34.799815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281983+0000","last_change":"2023-12-19T09:03:18.281983+0000","last_active":"2023-12-19T09:03:18.281983+0000","last_peered":"2023-12-19T09:03:18.281983+0000","last_clean":"2023-12-19T09:03:18.281983+0000","last_became_active":"2023-12-19T09:03:18.281621+0000","last_became_peered":"2023-12-19T09:03:18.281621+0000","last_unstale":"2023-12-19T09:03:18.281983+0000","last_undegraded":"2023-12-19T09:03:18.281983+0000","last_fullsized":"2023-12-19T09:03:18.281983+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:35.970393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279612+0000","last_change":"2023-12-19T09:03:17.261553+0000","last_active":"2023-12-19T09:03:18.279612+0000","last_peered":"2023-12-19T09:03:18.279612+0000","last_clean":"2023-12-19T09:03:18.279612+0000","last_became_active":"2023-12-19T09:03:17.260798+0000","last_became_peered":"2023-12-19T09:03:17.260798+0000","last_unstale":"2023-12-19T09:03:18.279612+0000","last_undegraded":"2023-12-19T09:03:18.279612+0000","last_fullsized":"2023-12-19T09:03:18.279612+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:30.953884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291568+0000","last_change":"2023-12-19T09:03:18.291568+0000","last_active":"2023-12-19T09:03:18.291568+0000","last_peered":"2023-12-19T09:03:18.291568+0000","last_clean":"2023-12-19T09:03:18.291568+0000","last_became_active":"2023-12-19T09:03:18.287667+0000","last_became_peered":"2023-12-19T09:03:18.287667+0000","last_unstale":"2023-12-19T09:03:18.291568+0000","last_undegraded":"2023-12-19T09:03:18.291568+0000","last_fullsized":"2023-12-19T09:03:18.291568+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:57.437656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274484+0000","last_change":"2023-12-19T09:03:17.266076+0000","last_active":"2023-12-19T09:03:18.274484+0000","last_peered":"2023-12-19T09:03:18.274484+0000","last_clean":"2023-12-19T09:03:18.274484+0000","last_became_active":"2023-12-19T09:03:17.265731+0000","last_became_peered":"2023-12-19T09:03:17.265731+0000","last_unstale":"2023-12-19T09:03:18.274484+0000","last_undegraded":"2023-12-19T09:03:18.274484+0000","last_fullsized":"2023-12-19T09:03:18.274484+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:22.384162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285593+0000","last_change":"2023-12-19T09:03:18.285593+0000","last_active":"2023-12-19T09:03:18.285593+0000","last_peered":"2023-12-19T09:03:18.285593+0000","last_clean":"2023-12-19T09:03:18.285593+0000","last_became_active":"2023-12-19T09:03:18.285002+0000","last_became_peered":"2023-12-19T09:03:18.285002+0000","last_unstale":"2023-12-19T09:03:18.285593+0000","last_undegraded":"2023-12-19T09:03:18.285593+0000","last_fullsized":"2023-12-19T09:03:18.285593+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:30.151876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.311745+0000","last_change":"2023-12-19T09:03:17.266705+0000","last_active":"2023-12-19T09:03:18.311745+0000","last_peered":"2023-12-19T09:03:18.311745+0000","last_clean":"2023-12-19T09:03:18.311745+0000","last_became_active":"2023-12-19T09:03:17.266394+0000","last_became_peered":"2023-12-19T09:03:17.266394+0000","last_unstale":"2023-12-19T09:03:18.311745+0000","last_undegraded":"2023-12-19T09:03:18.311745+0000","last_fullsized":"2023-12-19T09:03:18.311745+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:32.153702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.295964+0000","last_change":"2023-12-19T09:03:18.295964+0000","last_active":"2023-12-19T09:03:18.295964+0000","last_peered":"2023-12-19T09:03:18.295964+0000","last_clean":"2023-12-19T09:03:18.295964+0000","last_became_active":"2023-12-19T09:03:18.295635+0000","last_became_peered":"2023-12-19T09:03:18.295635+0000","last_unstale":"2023-12-19T09:03:18.295964+0000","last_undegraded":"2023-12-19T09:03:18.295964+0000","last_fullsized":"2023-12-19T09:03:18.295964+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:35.813483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.268355+0000","last_change":"2023-12-19T09:03:17.271197+0000","last_active":"2023-12-19T09:03:18.268355+0000","last_peered":"2023-12-19T09:03:18.268355+0000","last_clean":"2023-12-19T09:03:18.268355+0000","last_became_active":"2023-12-19T09:03:17.270747+0000","last_became_peered":"2023-12-19T09:03:17.270747+0000","last_unstale":"2023-12-19T09:03:18.268355+0000","last_undegraded":"2023-12-19T09:03:18.268355+0000","last_fullsized":"2023-12-19T09:03:18.268355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:54:45.930962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301173+0000","last_change":"2023-12-19T09:03:18.301173+0000","last_active":"2023-12-19T09:03:18.301173+0000","last_peered":"2023-12-19T09:03:18.301173+0000","last_clean":"2023-12-19T09:03:18.301173+0000","last_became_active":"2023-12-19T09:03:18.300854+0000","last_became_peered":"2023-12-19T09:03:18.300854+0000","last_unstale":"2023-12-19T09:03:18.301173+0000","last_undegraded":"2023-12-19T09:03:18.301173+0000","last_fullsized":"2023-12-19T09:03:18.301173+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:13.286019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306732+0000","last_change":"2023-12-19T09:03:17.270249+0000","last_active":"2023-12-19T09:03:18.306732+0000","last_peered":"2023-12-19T09:03:18.306732+0000","last_clean":"2023-12-19T09:03:18.306732+0000","last_became_active":"2023-12-19T09:03:17.269947+0000","last_became_peered":"2023-12-19T09:03:17.269947+0000","last_unstale":"2023-12-19T09:03:18.306732+0000","last_undegraded":"2023-12-19T09:03:18.306732+0000","last_fullsized":"2023-12-19T09:03:18.306732+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:50.489197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285271+0000","last_change":"2023-12-19T09:03:18.285271+0000","last_active":"2023-12-19T09:03:18.285271+0000","last_peered":"2023-12-19T09:03:18.285271+0000","last_clean":"2023-12-19T09:03:18.285271+0000","last_became_active":"2023-12-19T09:03:18.284967+0000","last_became_peered":"2023-12-19T09:03:18.284967+0000","last_unstale":"2023-12-19T09:03:18.285271+0000","last_undegraded":"2023-12-19T09:03:18.285271+0000","last_fullsized":"2023-12-19T09:03:18.285271+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:09.637960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.272091+0000","last_change":"2023-12-19T09:03:17.261110+0000","last_active":"2023-12-19T09:03:18.272091+0000","last_peered":"2023-12-19T09:03:18.272091+0000","last_clean":"2023-12-19T09:03:18.272091+0000","last_became_active":"2023-12-19T09:03:17.260798+0000","last_became_peered":"2023-12-19T09:03:17.260798+0000","last_unstale":"2023-12-19T09:03:18.272091+0000","last_undegraded":"2023-12-19T09:03:18.272091+0000","last_fullsized":"2023-12-19T09:03:18.272091+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:00:43.821041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297093+0000","last_change":"2023-12-19T09:03:18.297093+0000","last_active":"2023-12-19T09:03:18.297093+0000","last_peered":"2023-12-19T09:03:18.297093+0000","last_clean":"2023-12-19T09:03:18.297093+0000","last_became_active":"2023-12-19T09:03:18.288739+0000","last_became_peered":"2023-12-19T09:03:18.288739+0000","last_unstale":"2023-12-19T09:03:18.297093+0000","last_undegraded":"2023-12-19T09:03:18.297093+0000","last_fullsized":"2023-12-19T09:03:18.297093+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:49.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274644+0000","last_change":"2023-12-19T09:03:17.264462+0000","last_active":"2023-12-19T09:03:18.274644+0000","last_peered":"2023-12-19T09:03:18.274644+0000","last_clean":"2023-12-19T09:03:18.274644+0000","last_became_active":"2023-12-19T09:03:17.264166+0000","last_became_peered":"2023-12-19T09:03:17.264166+0000","last_unstale":"2023-12-19T09:03:18.274644+0000","last_undegraded":"2023-12-19T09:03:18.274644+0000","last_fullsized":"2023-12-19T09:03:18.274644+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:04.792003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.299933+0000","last_change":"2023-12-19T09:03:18.299933+0000","last_active":"2023-12-19T09:03:18.299933+0000","last_peered":"2023-12-19T09:03:18.299933+0000","last_clean":"2023-12-19T09:03:18.299933+0000","last_became_active":"2023-12-19T09:03:18.299622+0000","last_became_peered":"2023-12-19T09:03:18.299622+0000","last_unstale":"2023-12-19T09:03:18.299933+0000","last_undegraded":"2023-12-19T09:03:18.299933+0000","last_fullsized":"2023-12-19T09:03:18.299933+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:45.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.305278+0000","last_change":"2023-12-19T09:03:17.257950+0000","last_active":"2023-12-19T09:03:18.305278+0000","last_peered":"2023-12-19T09:03:18.305278+0000","last_clean":"2023-12-19T09:03:18.305278+0000","last_became_active":"2023-12-19T09:03:17.257658+0000","last_became_peered":"2023-12-19T09:03:17.257658+0000","last_unstale":"2023-12-19T09:03:18.305278+0000","last_undegraded":"2023-12-19T09:03:18.305278+0000","last_fullsized":"2023-12-19T09:03:18.305278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:48.318228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283595+0000","last_change":"2023-12-19T09:03:18.283595+0000","last_active":"2023-12-19T09:03:18.283595+0000","last_peered":"2023-12-19T09:03:18.283595+0000","last_clean":"2023-12-19T09:03:18.283595+0000","last_became_active":"2023-12-19T09:03:18.283284+0000","last_became_peered":"2023-12-19T09:03:18.283284+0000","last_unstale":"2023-12-19T09:03:18.283595+0000","last_undegraded":"2023-12-19T09:03:18.283595+0000","last_fullsized":"2023-12-19T09:03:18.283595+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:37.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.266808+0000","last_change":"2023-12-19T09:03:17.260978+0000","last_active":"2023-12-19T09:03:17.266808+0000","last_peered":"2023-12-19T09:03:17.266808+0000","last_clean":"2023-12-19T09:03:17.266808+0000","last_became_active":"2023-12-19T09:03:17.260645+0000","last_became_peered":"2023-12-19T09:03:17.260645+0000","last_unstale":"2023-12-19T09:03:17.266808+0000","last_undegraded":"2023-12-19T09:03:17.266808+0000","last_fullsized":"2023-12-19T09:03:17.266808+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:33.100157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293668+0000","last_change":"2023-12-19T09:03:18.293668+0000","last_active":"2023-12-19T09:03:18.293668+0000","last_peered":"2023-12-19T09:03:18.293668+0000","last_clean":"2023-12-19T09:03:18.293668+0000","last_became_active":"2023-12-19T09:03:18.293392+0000","last_became_peered":"2023-12-19T09:03:18.293392+0000","last_unstale":"2023-12-19T09:03:18.293668+0000","last_undegraded":"2023-12-19T09:03:18.293668+0000","last_fullsized":"2023-12-19T09:03:18.293668+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:38.027468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.261438+0000","last_change":"2023-12-19T09:03:17.251954+0000","last_active":"2023-12-19T09:03:18.261438+0000","last_peered":"2023-12-19T09:03:18.261438+0000","last_clean":"2023-12-19T09:03:18.261438+0000","last_became_active":"2023-12-19T09:03:17.251514+0000","last_became_peered":"2023-12-19T09:03:17.251514+0000","last_unstale":"2023-12-19T09:03:18.261438+0000","last_undegraded":"2023-12-19T09:03:18.261438+0000","last_fullsized":"2023-12-19T09:03:18.261438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:42.943753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280905+0000","last_change":"2023-12-19T09:03:18.280905+0000","last_active":"2023-12-19T09:03:18.280905+0000","last_peered":"2023-12-19T09:03:18.280905+0000","last_clean":"2023-12-19T09:03:18.280905+0000","last_became_active":"2023-12-19T09:03:18.280612+0000","last_became_peered":"2023-12-19T09:03:18.280612+0000","last_unstale":"2023-12-19T09:03:18.280905+0000","last_undegraded":"2023-12-19T09:03:18.280905+0000","last_fullsized":"2023-12-19T09:03:18.280905+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:54.402255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.260972+0000","last_change":"2023-12-19T09:03:17.252090+0000","last_active":"2023-12-19T09:03:17.260972+0000","last_peered":"2023-12-19T09:03:17.260972+0000","last_clean":"2023-12-19T09:03:17.260972+0000","last_became_active":"2023-12-19T09:03:17.251779+0000","last_became_peered":"2023-12-19T09:03:17.251779+0000","last_unstale":"2023-12-19T09:03:17.260972+0000","last_undegraded":"2023-12-19T09:03:17.260972+0000","last_fullsized":"2023-12-19T09:03:17.260972+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:47:48.224341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286969+0000","last_change":"2023-12-19T09:03:18.286969+0000","last_active":"2023-12-19T09:03:18.286969+0000","last_peered":"2023-12-19T09:03:18.286969+0000","last_clean":"2023-12-19T09:03:18.286969+0000","last_became_active":"2023-12-19T09:03:18.283477+0000","last_became_peered":"2023-12-19T09:03:18.283477+0000","last_unstale":"2023-12-19T09:03:18.286969+0000","last_undegraded":"2023-12-19T09:03:18.286969+0000","last_fullsized":"2023-12-19T09:03:18.286969+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:35.887515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.309878+0000","last_change":"2023-12-19T09:03:17.264557+0000","last_active":"2023-12-19T09:03:18.309878+0000","last_peered":"2023-12-19T09:03:18.309878+0000","last_clean":"2023-12-19T09:03:18.309878+0000","last_became_active":"2023-12-19T09:03:17.264268+0000","last_became_peered":"2023-12-19T09:03:17.264268+0000","last_unstale":"2023-12-19T09:03:18.309878+0000","last_undegraded":"2023-12-19T09:03:18.309878+0000","last_fullsized":"2023-12-19T09:03:18.309878+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:22.387828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297918+0000","last_change":"2023-12-19T09:03:18.286955+0000","last_active":"2023-12-19T09:03:18.297918+0000","last_peered":"2023-12-19T09:03:18.297918+0000","last_clean":"2023-12-19T09:03:18.297918+0000","last_became_active":"2023-12-19T09:03:18.286641+0000","last_became_peered":"2023-12-19T09:03:18.286641+0000","last_unstale":"2023-12-19T09:03:18.297918+0000","last_undegraded":"2023-12-19T09:03:18.297918+0000","last_fullsized":"2023-12-19T09:03:18.297918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:07:20.126407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.272603+0000","last_change":"2023-12-19T09:03:17.268683+0000","last_active":"2023-12-19T09:03:17.272603+0000","last_peered":"2023-12-19T09:03:17.272603+0000","last_clean":"2023-12-19T09:03:17.272603+0000","last_became_active":"2023-12-19T09:03:17.268397+0000","last_became_peered":"2023-12-19T09:03:17.268397+0000","last_unstale":"2023-12-19T09:03:17.272603+0000","last_undegraded":"2023-12-19T09:03:17.272603+0000","last_fullsized":"2023-12-19T09:03:17.272603+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:50.870627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296826+0000","last_change":"2023-12-19T09:03:18.296826+0000","last_active":"2023-12-19T09:03:18.296826+0000","last_peered":"2023-12-19T09:03:18.296826+0000","last_clean":"2023-12-19T09:03:18.296826+0000","last_became_active":"2023-12-19T09:03:18.296509+0000","last_became_peered":"2023-12-19T09:03:18.296509+0000","last_unstale":"2023-12-19T09:03:18.296826+0000","last_undegraded":"2023-12-19T09:03:18.296826+0000","last_fullsized":"2023-12-19T09:03:18.296826+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:51.502865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.265305+0000","last_change":"2023-12-19T09:03:17.261172+0000","last_active":"2023-12-19T09:03:17.265305+0000","last_peered":"2023-12-19T09:03:17.265305+0000","last_clean":"2023-12-19T09:03:17.265305+0000","last_became_active":"2023-12-19T09:03:17.260877+0000","last_became_peered":"2023-12-19T09:03:17.260877+0000","last_unstale":"2023-12-19T09:03:17.265305+0000","last_undegraded":"2023-12-19T09:03:17.265305+0000","last_fullsized":"2023-12-19T09:03:17.265305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:50.870627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293835+0000","last_change":"2023-12-19T09:03:18.293835+0000","last_active":"2023-12-19T09:03:18.293835+0000","last_peered":"2023-12-19T09:03:18.293835+0000","last_clean":"2023-12-19T09:03:18.293835+0000","last_became_active":"2023-12-19T09:03:18.293537+0000","last_became_peered":"2023-12-19T09:03:18.293537+0000","last_unstale":"2023-12-19T09:03:18.293835+0000","last_undegraded":"2023-12-19T09:03:18.293835+0000","last_fullsized":"2023-12-19T09:03:18.293835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:18.784902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.268209+0000","last_change":"2023-12-19T09:03:17.265214+0000","last_active":"2023-12-19T09:03:17.268209+0000","last_peered":"2023-12-19T09:03:17.268209+0000","last_clean":"2023-12-19T09:03:17.268209+0000","last_became_active":"2023-12-19T09:03:17.264758+0000","last_became_peered":"2023-12-19T09:03:17.264758+0000","last_unstale":"2023-12-19T09:03:17.268209+0000","last_undegraded":"2023-12-19T09:03:17.268209+0000","last_fullsized":"2023-12-19T09:03:17.268209+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:35.707152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.268708+0000","last_change":"2023-12-19T09:03:18.268708+0000","last_active":"2023-12-19T09:03:18.268708+0000","last_peered":"2023-12-19T09:03:18.268708+0000","last_clean":"2023-12-19T09:03:18.268708+0000","last_became_active":"2023-12-19T09:03:18.268274+0000","last_became_peered":"2023-12-19T09:03:18.268274+0000","last_unstale":"2023-12-19T09:03:18.268708+0000","last_undegraded":"2023-12-19T09:03:18.268708+0000","last_fullsized":"2023-12-19T09:03:18.268708+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:28:45.573661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.268072+0000","last_change":"2023-12-19T09:03:17.262439+0000","last_active":"2023-12-19T09:03:17.268072+0000","last_peered":"2023-12-19T09:03:17.268072+0000","last_clean":"2023-12-19T09:03:17.268072+0000","last_became_active":"2023-12-19T09:03:17.262163+0000","last_became_peered":"2023-12-19T09:03:17.262163+0000","last_unstale":"2023-12-19T09:03:17.268072+0000","last_undegraded":"2023-12-19T09:03:17.268072+0000","last_fullsized":"2023-12-19T09:03:17.268072+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:13.054103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298493+0000","last_change":"2023-12-19T09:03:18.292538+0000","last_active":"2023-12-19T09:03:18.298493+0000","last_peered":"2023-12-19T09:03:18.298493+0000","last_clean":"2023-12-19T09:03:18.298493+0000","last_became_active":"2023-12-19T09:03:18.292249+0000","last_became_peered":"2023-12-19T09:03:18.292249+0000","last_unstale":"2023-12-19T09:03:18.298493+0000","last_undegraded":"2023-12-19T09:03:18.298493+0000","last_fullsized":"2023-12-19T09:03:18.298493+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:26.344233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.270210+0000","last_change":"2023-12-19T09:03:17.267694+0000","last_active":"2023-12-19T09:03:17.270210+0000","last_peered":"2023-12-19T09:03:17.270210+0000","last_clean":"2023-12-19T09:03:17.270210+0000","last_became_active":"2023-12-19T09:03:17.266891+0000","last_became_peered":"2023-12-19T09:03:17.266891+0000","last_unstale":"2023-12-19T09:03:17.270210+0000","last_undegraded":"2023-12-19T09:03:17.270210+0000","last_fullsized":"2023-12-19T09:03:17.270210+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:45.481336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.289623+0000","last_change":"2023-12-19T09:03:18.289623+0000","last_active":"2023-12-19T09:03:18.289623+0000","last_peered":"2023-12-19T09:03:18.289623+0000","last_clean":"2023-12-19T09:03:18.289623+0000","last_became_active":"2023-12-19T09:03:18.289066+0000","last_became_peered":"2023-12-19T09:03:18.289066+0000","last_unstale":"2023-12-19T09:03:18.289623+0000","last_undegraded":"2023-12-19T09:03:18.289623+0000","last_fullsized":"2023-12-19T09:03:18.289623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:08.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.273725+0000","last_change":"2023-12-19T09:03:17.271852+0000","last_active":"2023-12-19T09:03:17.273725+0000","last_peered":"2023-12-19T09:03:17.273725+0000","last_clean":"2023-12-19T09:03:17.273725+0000","last_became_active":"2023-12-19T09:03:17.270829+0000","last_became_peered":"2023-12-19T09:03:17.270829+0000","last_unstale":"2023-12-19T09:03:17.273725+0000","last_undegraded":"2023-12-19T09:03:17.273725+0000","last_fullsized":"2023-12-19T09:03:17.273725+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:44.696127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285834+0000","last_change":"2023-12-19T09:03:18.285834+0000","last_active":"2023-12-19T09:03:18.285834+0000","last_peered":"2023-12-19T09:03:18.285834+0000","last_clean":"2023-12-19T09:03:18.285834+0000","last_became_active":"2023-12-19T09:03:18.285274+0000","last_became_peered":"2023-12-19T09:03:18.285274+0000","last_unstale":"2023-12-19T09:03:18.285834+0000","last_undegraded":"2023-12-19T09:03:18.285834+0000","last_fullsized":"2023-12-19T09:03:18.285834+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:43.168774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.274637+0000","last_change":"2023-12-19T09:03:17.271777+0000","last_active":"2023-12-19T09:03:17.274637+0000","last_peered":"2023-12-19T09:03:17.274637+0000","last_clean":"2023-12-19T09:03:17.274637+0000","last_became_active":"2023-12-19T09:03:17.271426+0000","last_became_peered":"2023-12-19T09:03:17.271426+0000","last_unstale":"2023-12-19T09:03:17.274637+0000","last_undegraded":"2023-12-19T09:03:17.274637+0000","last_fullsized":"2023-12-19T09:03:17.274637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:43.372741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280844+0000","last_change":"2023-12-19T09:03:18.280844+0000","last_active":"2023-12-19T09:03:18.280844+0000","last_peered":"2023-12-19T09:03:18.280844+0000","last_clean":"2023-12-19T09:03:18.280844+0000","last_became_active":"2023-12-19T09:03:18.280506+0000","last_became_peered":"2023-12-19T09:03:18.280506+0000","last_unstale":"2023-12-19T09:03:18.280844+0000","last_undegraded":"2023-12-19T09:03:18.280844+0000","last_fullsized":"2023-12-19T09:03:18.280844+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:20:47.480393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.309302+0000","last_change":"2023-12-19T09:03:17.248634+0000","last_active":"2023-12-19T09:03:18.309302+0000","last_peered":"2023-12-19T09:03:18.309302+0000","last_clean":"2023-12-19T09:03:18.309302+0000","last_became_active":"2023-12-19T09:03:17.247915+0000","last_became_peered":"2023-12-19T09:03:17.247915+0000","last_unstale":"2023-12-19T09:03:18.309302+0000","last_undegraded":"2023-12-19T09:03:18.309302+0000","last_fullsized":"2023-12-19T09:03:18.309302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:07.923063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"163'293","reported_seq":913,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.261792+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:03:17.261792+0000","last_peered":"2023-12-19T09:03:17.261792+0000","last_clean":"2023-12-19T09:03:17.261792+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:03:17.261792+0000","last_undegraded":"2023-12-19T09:03:17.261792+0000","last_fullsized":"2023-12-19T09:03:17.261792+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575755,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299996,"kb_used_data":2708,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071844,"statfs":{"total":96636764160,"available":96329568256,"internally_reserved":0,"allocated":2772992,"data_stored":1746131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3420000000000001,"15min":1.3069999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.54400000000000004},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3260000000000001,"15min":1.321},"min":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.66800000000000004},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.218}]},{"osd":1,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1799999999999999,"15min":1.179},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.64000000000000001},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1919999999999999,"15min":1.1879999999999999},"min":{"1min":0.84999999999999998,"5min":0.81799999999999995,"15min":0.64500000000000002},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.784},"last":1.3680000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1930000000000001,"15min":1.173},"min":{"1min":0.92200000000000004,"5min":0.82299999999999995,"15min":0.57199999999999995},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.88100000000000001,"5min":0.79000000000000004,"15min":0.60899999999999999},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3500000000000001,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.625},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":4.4290000000000003},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.3420000000000001,"15min":1.3440000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.61099999999999999},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.774},"last":1.548}]},{"osd":4,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2509999999999999,"15min":1.2170000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.56799999999999995},"max":{"1min":1.6080000000000001,"5min":1.847,"15min":1.847},"last":1.27},{"interface":"front","average":{"1min":1.52,"5min":1.3220000000000001,"15min":1.238},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.42299999999999999},"max":{"1min":6.2629999999999999,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":1.4039999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.27,"15min":1.2390000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.35599999999999998},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.109},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.3660000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.498},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.6699999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.429,"15min":1.375},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.51600000000000001},"max":{"1min":1.7450000000000001,"5min":1.841,"15min":1.8540000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.3959999999999999,"15min":1.3680000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.47899999999999998},"max":{"1min":2.5609999999999999,"5min":2.5609999999999999,"15min":2.5609999999999999},"last":1.1599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575754,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311704,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060136,"statfs":{"total":96636764160,"available":96317579264,"internally_reserved":0,"allocated":2179072,"data_stored":1155811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.133,"15min":1.129},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2270000000000001,"15min":1.1819999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.61399999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":4.2809999999999997},"last":1.4570000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.234,"15min":1.196},"min":{"1min":0.755,"5min":0.627,"15min":0.627},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.333},{"interface":"front","average":{"1min":1.083,"5min":1.1559999999999999,"15min":1.175},"min":{"1min":0.78100000000000003,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":1.619,"5min":1.637,"15min":1.637},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2609999999999999,"15min":1.24},"min":{"1min":0.65600000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.159},{"interface":"front","average":{"1min":1.325,"5min":1.2629999999999999,"15min":1.248},"min":{"1min":0.748,"5min":0.748,"15min":0.63400000000000001},"max":{"1min":5.6150000000000002,"5min":5.6150000000000002,"15min":5.6150000000000002},"last":0.90800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3200000000000001,"15min":1.2789999999999999},"min":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.76300000000000001},"max":{"1min":4.5519999999999996,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":1.524},{"interface":"front","average":{"1min":1.385,"5min":1.329,"15min":1.3300000000000001},"min":{"1min":0.82199999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.413}]},{"osd":4,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.83499999999999996,"5min":0.66800000000000004,"15min":0.63400000000000001},"max":{"1min":4.125,"5min":4.125,"15min":4.125},"last":1.373},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.4590000000000001,"15min":1.4259999999999999},"min":{"1min":0.97099999999999997,"5min":0.82399999999999995,"15min":0.51100000000000001},"max":{"1min":4.3099999999999996,"5min":4.3099999999999996,"15min":4.3099999999999996},"last":1.5880000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.319,"15min":1.298},"min":{"1min":0.80000000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.228},{"interface":"front","average":{"1min":1.365,"5min":1.3049999999999999,"15min":1.2769999999999999},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.56399999999999995},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.1990000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.3959999999999999,"15min":1.351},"min":{"1min":0.95199999999999996,"5min":0.76000000000000001,"15min":0.56100000000000005},"max":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"last":1.02},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.4550000000000001,"15min":1.423},"min":{"1min":1.036,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.2829999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575754,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303508,"kb_used_data":2124,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068332,"statfs":{"total":96636764160,"available":96325971968,"internally_reserved":0,"allocated":2174976,"data_stored":1155741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.94199999999999995,"5min":0.626,"15min":0.57999999999999996},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3080000000000001,"15min":1.3149999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.2130000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.163,"15min":1.173},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.57199999999999995},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.238,"15min":1.234},"min":{"1min":0.80700000000000005,"5min":0.80400000000000005,"15min":0.72499999999999998},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.3700000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2529999999999999,"15min":1.222},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.55300000000000005},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.105},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2310000000000001,"15min":1.252},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.9399999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.89300000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.891},"last":1.169},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.302,"15min":1.3149999999999999},"min":{"1min":0.66500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.03}]},{"osd":4,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2170000000000001,"15min":1.2070000000000001},"min":{"1min":0.63400000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.006},{"interface":"front","average":{"1min":1.403,"5min":1.337,"15min":1.3120000000000001},"min":{"1min":0.85899999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.7690000000000001,"5min":2.7690000000000001,"15min":2.7690000000000001},"last":1.423}]},{"osd":6,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.66700000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.393,"15min":1.3799999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":1.081}]},{"osd":7,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.4770000000000001,"15min":1.4470000000000001},"min":{"1min":0.96899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":1.323},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.4810000000000001,"15min":1.476},"min":{"1min":0.92700000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.3440000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575754,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1157277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.622},"max":{"1min":1.5669999999999999,"5min":1.6659999999999999,"15min":1.7270000000000001},"last":5.149},{"interface":"front","average":{"1min":1.302,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.78300000000000003},"max":{"1min":1.6180000000000001,"5min":1.7270000000000001,"15min":1.768},"last":4.6429999999999998}]},{"osd":1,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.21,"15min":1.202},"min":{"1min":0.85799999999999998,"5min":0.80700000000000005,"15min":0.54500000000000004},"max":{"1min":1.5389999999999999,"5min":1.7869999999999999,"15min":1.8959999999999999},"last":4.5890000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.171,"15min":1.149},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.59999999999999998},"max":{"1min":1.389,"5min":1.712,"15min":1.712},"last":2.6440000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1890000000000001,"15min":1.173},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.66100000000000003},"max":{"1min":1.5409999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":5.0919999999999996},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6200000000000001,"5min":1.7290000000000001,"15min":1.8149999999999999},"last":4.5389999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.69199999999999995},"max":{"1min":1.575,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":5.5469999999999997},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.377,"15min":1.3779999999999999},"min":{"1min":0.95399999999999996,"5min":0.90200000000000002,"15min":0.77800000000000002},"max":{"1min":1.673,"5min":1.7589999999999999,"15min":1.863},"last":5.2069999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.258,"15min":1.228},"min":{"1min":0.92000000000000004,"5min":0.875,"15min":0.623},"max":{"1min":1.5660000000000001,"5min":1.796,"15min":1.962},"last":11.943},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.82799999999999996,"5min":0.69899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6879999999999999,"5min":1.823,"15min":1.823},"last":0.92700000000000005}]},{"osd":6,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.266,"15min":1.238},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.59399999999999997},"max":{"1min":1.899,"5min":1.909,"15min":1.909},"last":5.298},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.419,"15min":1.405},"min":{"1min":1.0700000000000001,"5min":1.034,"15min":0.64200000000000002},"max":{"1min":1.677,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":4.6980000000000004}]},{"osd":7,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.337,"15min":1.3340000000000001},"min":{"1min":0.95199999999999996,"5min":0.86499999999999999,"15min":0.71799999999999997},"max":{"1min":1.851,"5min":1.851,"15min":1.919},"last":0.625},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3999999999999999,"15min":1.393},"min":{"1min":1.0860000000000001,"5min":0.84099999999999997,"15min":0.65100000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":5.2519999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311708,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060132,"statfs":{"total":96636764160,"available":96317575168,"internally_reserved":0,"allocated":2183168,"data_stored":1156329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.84199999999999997,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1399999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.97199999999999998,"15min":0.98099999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4470000000000001,"5min":1.476,"15min":1.6180000000000001},"last":0.89900000000000002}]},{"osd":1,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94299999999999995,"15min":0.94599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.345,"5min":1.5169999999999999,"15min":2.456},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0900000000000001,"15min":1.087},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69699999999999995},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":1.1559999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0700000000000001,"15min":1.0900000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.4550000000000001,"5min":1.534,"15min":2.0150000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.194,"5min":1.196,"15min":1.1930000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2949999999999999,"15min":1.288},"min":{"1min":1.002,"5min":1.002,"15min":0.94099999999999995},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.266,"15min":1.252},"min":{"1min":0.97799999999999998,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.6950000000000001,"5min":1.742,"15min":2.6520000000000001},"last":0.82299999999999995}]},{"osd":5,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3300000000000001,"15min":1.3},"min":{"1min":1.0669999999999999,"5min":0.97099999999999997,"15min":0.85399999999999998},"max":{"1min":1.887,"5min":1.887,"15min":2.7519999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.387,"15min":1.391},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.8700000000000001,"5min":1.8759999999999999,"15min":2.5710000000000002},"last":1.226}]},{"osd":6,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.421,"15min":1.421},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.429,"15min":1.405},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.88800000000000001},"max":{"1min":1.8089999999999999,"5min":1.925,"15min":2.2330000000000001},"last":1.319}]},{"osd":7,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.498,"15min":1.488},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.8720000000000001,"5min":2.0379999999999998,"15min":2.202},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.4419999999999999,"15min":1.4410000000000001},"min":{"1min":1.115,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.8620000000000001},"last":1.131}]}]},{"osd":2,"up_from":13,"seq":55834575754,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307612,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064228,"statfs":{"total":96636764160,"available":96321769472,"internally_reserved":0,"allocated":2183168,"data_stored":1155853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.94199999999999995,"15min":0.93300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.085,"15min":1.0760000000000001},"min":{"1min":0.81799999999999995,"5min":0.81100000000000005,"15min":0.60799999999999998},"max":{"1min":1.367,"5min":1.4650000000000001,"15min":1.744},"last":0.97899999999999998}]},{"osd":1,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70599999999999996,"5min":0.66800000000000004,"15min":0.56399999999999995},"max":{"1min":1.3939999999999999,"5min":1.4119999999999999,"15min":1.948},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.42699999999999999},"max":{"1min":1.538,"5min":1.6200000000000001,"15min":1.6499999999999999},"last":1.286}]},{"osd":3,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.628},"max":{"1min":1.1579999999999999,"5min":1.6140000000000001,"15min":1.8009999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1639999999999999,"15min":1.1539999999999999},"min":{"1min":0.71499999999999997,"5min":0.63600000000000001,"15min":0.57899999999999996},"max":{"1min":1.4199999999999999,"5min":1.49,"15min":2.1760000000000002},"last":0.88400000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.327,"15min":1.3320000000000001},"min":{"1min":1.012,"5min":1.0109999999999999,"15min":0.84899999999999998},"max":{"1min":1.6040000000000001,"5min":1.6719999999999999,"15min":2.0590000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.325,"15min":1.323},"min":{"1min":0.93200000000000005,"5min":0.84099999999999997,"15min":0.82899999999999996},"max":{"1min":1.7010000000000001,"5min":1.714,"15min":3.1030000000000002},"last":1.1499999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3720000000000001,"15min":1.363},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.571,"5min":1.7270000000000001,"15min":1.9099999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.286,"15min":1.2949999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.548,"5min":2.0830000000000002,"15min":2.2959999999999998},"last":1.4810000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3440000000000001,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.72599999999999998},"max":{"1min":1.524,"5min":1.9199999999999999,"15min":2.3940000000000001},"last":1.302},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.415,"15min":1.397},"min":{"1min":1.016,"5min":1.016,"15min":0.89900000000000002},"max":{"1min":1.6930000000000001,"5min":1.8520000000000001,"15min":1.869},"last":1.2649999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.0900000000000001,"5min":1.03,"15min":0.60199999999999998},"max":{"1min":1.6839999999999999,"5min":1.8340000000000001,"15min":2.1680000000000001},"last":1.45},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4259999999999999,"15min":1.4059999999999999},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.82199999999999995},"max":{"1min":1.722,"5min":2.0859999999999999,"15min":2.7109999999999999},"last":1.3819999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575755,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307608,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064232,"statfs":{"total":96636764160,"available":96321773568,"internally_reserved":0,"allocated":2179072,"data_stored":1155811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.87,"15min":0.85199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.339,"5min":1.54,"15min":1.54},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.96899999999999997,"15min":0.97199999999999998},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.42099999999999999},"max":{"1min":1.337,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.097}]},{"osd":2,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.36499999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.113,"5min":1.1160000000000001,"15min":1.081},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.503},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.159}]},{"osd":3,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.1619999999999999,"15min":1.167},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.7030000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.04},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.698,"5min":1.706,"15min":1.706},"last":0.88600000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2230000000000001,"15min":1.2190000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5900000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.238,"5min":1.327,"15min":1.323},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.7689999999999999,"5min":1.958,"15min":2.024},"last":1.2869999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3080000000000001,"15min":1.3169999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.78400000000000003},"max":{"1min":1.7150000000000001,"5min":2.0569999999999999,"15min":2.0699999999999998},"last":1.333},{"interface":"front","average":{"1min":1.252,"5min":1.292,"15min":1.302},"min":{"1min":0.93700000000000006,"5min":0.90200000000000002,"15min":0.72199999999999998},"max":{"1min":1.534,"5min":1.96,"15min":2.052},"last":1.3620000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3740000000000001,"15min":1.3819999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8280000000000001,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.375,"15min":1.377},"min":{"1min":0.93000000000000005,"5min":0.85599999999999998,"15min":0.81499999999999995},"max":{"1min":1.8779999999999999,"5min":1.881,"15min":2.0800000000000001},"last":1.2450000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.411,"15min":1.419},"min":{"1min":0.97799999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.9750000000000001,"5min":2.117,"15min":2.117},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.377,"15min":1.3919999999999999},"min":{"1min":1.0920000000000001,"5min":0.95899999999999996,"15min":0.78800000000000003},"max":{"1min":1.6719999999999999,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.198}]}]},{"osd":0,"up_from":13,"seq":55834575754,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312300,"kb_used_data":2724,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059540,"statfs":{"total":96636764160,"available":96316968960,"internally_reserved":0,"allocated":2789376,"data_stored":1747312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83699999999999997,"15min":0.84399999999999997},"min":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.41299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.94699999999999995,"15min":0.96599999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.48599999999999999},"max":{"1min":1.3660000000000001,"5min":1.464,"15min":1.488},"last":0.97299999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.997,"15min":0.99199999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.51900000000000002},"max":{"1min":1.4239999999999999,"5min":1.431,"15min":1.625},"last":0.628},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.056,"15min":1.0569999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3899999999999999,"5min":1.4199999999999999,"15min":1.48},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.028,"15min":1.0249999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.696,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.157,"15min":1.145},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.57999999999999996},"max":{"1min":1.603,"5min":1.603,"15min":1.698},"last":0.79900000000000004}]},{"osd":4,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3660000000000001,"15min":1.3779999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.747},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.161},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.349,"15min":1.3520000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3779999999999999,"15min":1.3799999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.244},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.405,"15min":1.417},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.7310000000000001,"5min":1.7470000000000001,"15min":1.7589999999999999},"last":1.446}]},{"osd":6,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.288,"15min":1.2929999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.843,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.1080000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.361,"15min":1.361},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4079999999999999,"15min":1.4039999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.9410000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.077}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:03:22.548 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:03:22.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.735+0000 7f5dd8b85700 1 Processor -- start 2023-12-19T09:03:22.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.735+0000 7f5dd8b85700 1 -- start start 2023-12-19T09:03:22.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd8b85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5dd4140710 0x7f5dd40c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd8b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 0x7f5dd40680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd8b85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd406e150 con 0x7f5dd40c75e0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd8b85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd406d540 con 0x7f5dd4140710 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd8b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd406d820 con 0x7f5dd4140ae0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5dd4140710 0x7f5dd40c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd2d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5dd40c75e0 0x7f5dd406a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60514/0 (socket says 172.21.15.111:60514) 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd2d9d700 1 -- 172.21.15.111:0/1668109890 learned_addr learned my addr 172.21.15.111:0/1668109890 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd1d9b700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 0x7f5dd40680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd159a700 1 -- 172.21.15.111:0/1668109890 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901970809 0 0) 0x7f5dd406e150 con 0x7f5dd40c75e0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd159a700 1 -- 172.21.15.111:0/1668109890 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dbc003570 con 0x7f5dd40c75e0 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd259c700 1 -- 172.21.15.111:0/1668109890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 msgr2=0x7f5dd40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd259c700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 0x7f5dd40680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd259c700 1 -- 172.21.15.111:0/1668109890 >> v1:172.21.15.111:6789/0 conn(0x7f5dd40c75e0 legacy=0x7f5dd406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd259c700 1 -- 172.21.15.111:0/1668109890 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd406db00 con 0x7f5dd4140710 2023-12-19T09:03:22.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.739+0000 7f5dd1d9b700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 0x7f5dd40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd259c700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40c70a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40098e0 tx=0x7f5dc40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451b2850df7c2640 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd159a700 1 -- 172.21.15.111:0/1668109890 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011630 con 0x7f5dd4140710 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd159a700 1 -- 172.21.15.111:0/1668109890 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4011790 con 0x7f5dd4140710 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd159a700 1 -- 172.21.15.111:0/1668109890 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011b30 con 0x7f5dd4140710 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 -- 172.21.15.111:0/1668109890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 msgr2=0x7f5dd40c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40c70a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40098e0 tx=0x7f5dc40091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 -- 172.21.15.111:0/1668109890 shutdown_connections 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd4140ae0 0x7f5dd40680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/1668109890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40c70a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 -- 172.21.15.111:0/1668109890 >> 172.21.15.111:0/1668109890 conn(0x7f5dd40c8850 msgr2=0x7f5dd40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 -- 172.21.15.111:0/1668109890 shutdown_connections 2023-12-19T09:03:22.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.743+0000 7f5dd8b85700 1 -- 172.21.15.111:0/1668109890 wait complete. 2023-12-19T09:03:22.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 Processor -- start 2023-12-19T09:03:22.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- start start 2023-12-19T09:03:22.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd40c75e0 0x7f5dd41579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5dd4140ae0 0x7f5dd40ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40c3460 con 0x7f5dd4140710 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40ada40 con 0x7f5dd4140ae0 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40c3670 con 0x7f5dd40c75e0 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46990/0 (socket says 172.21.15.111:46990) 2023-12-19T09:03:22.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 -- 172.21.15.111:0/3341681280 learned_addr learned my addr 172.21.15.111:0/3341681280 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd259c700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd40c75e0 0x7f5dd41579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd2d9d700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5dd4140ae0 0x7f5dd40ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 -- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd40c75e0 msgr2=0x7f5dd41579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd40c75e0 0x7f5dd41579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 -- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5dd4140ae0 msgr2=0x7f5dd40ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5dd4140ae0 0x7f5dd40ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 -- 172.21.15.111:0/3341681280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009420 con 0x7f5dd4140710 2023-12-19T09:03:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd1d9b700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800ec80 tx=0x7f5dc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800c9d0 con 0x7f5dd4140710 2023-12-19T09:03:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc800cb30 con 0x7f5dd4140710 2023-12-19T09:03:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc4009420 con 0x7f5dd4140710 2023-12-19T09:03:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800ceb0 con 0x7f5dd4140710 2023-12-19T09:03:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.747+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd40ad720 con 0x7f5dd4140710 2023-12-19T09:03:22.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.751+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd4071120 con 0x7f5dd4140710 2023-12-19T09:03:22.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.751+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f5dc8013680 con 0x7f5dd4140710 2023-12-19T09:03:22.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.755+0000 7f5dc27fc700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 0x7f5db4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:22.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.755+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5dc807d070 con 0x7f5dd4140710 2023-12-19T09:03:22.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.755+0000 7f5dd259c700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 0x7f5db4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:22.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.755+0000 7f5dd259c700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 0x7f5db4062b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40091e0 tx=0x7f5dc4017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:22.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.759+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5dc80acbe0 con 0x7f5dd4140710 2023-12-19T09:03:22.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.927+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5dd40642a0 con 0x7f5db40606a0 2023-12-19T09:03:22.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.947+0000 7f5dc27fc700 1 -- 172.21.15.111:0/3341681280 <== mgr.7862 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287343 (secure 0 0 0) 0x7f5dd40642a0 con 0x7f5db40606a0 2023-12-19T09:03:22.954 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 msgr2=0x7f5db4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 0x7f5db4062b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40091e0 tx=0x7f5dc4017000 comp rx=0 tx=0).stop 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 msgr2=0x7f5dd40acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800ec80 tx=0x7f5dc80091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 shutdown_connections 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5db40606a0 0x7f5db4062b50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5dd4140ae0 0x7f5dd40ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd4140710 0x7f5dd40acca0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 --2- 172.21.15.111:0/3341681280 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5dd40c75e0 0x7f5dd41579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 >> 172.21.15.111:0/3341681280 conn(0x7f5dd40c8850 msgr2=0x7f5dd40c6800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 shutdown_connections 2023-12-19T09:03:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:22.951+0000 7f5dd8b85700 1 -- 172.21.15.111:0/3341681280 wait complete. 2023-12-19T09:03:22.961 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:03:22.980 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:03:21.632897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2466148,"kb_used_data":18212,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752508572,"statfs":{"total":773094113280,"available":770568777728,"internally_reserved":0,"allocated":18649088,"data_stored":10430265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.403559"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296171+0000","last_change":"2023-12-19T09:03:18.296171+0000","last_active":"2023-12-19T09:03:18.296171+0000","last_peered":"2023-12-19T09:03:18.296171+0000","last_clean":"2023-12-19T09:03:18.296171+0000","last_became_active":"2023-12-19T09:03:18.290561+0000","last_became_peered":"2023-12-19T09:03:18.290561+0000","last_unstale":"2023-12-19T09:03:18.296171+0000","last_undegraded":"2023-12-19T09:03:18.296171+0000","last_fullsized":"2023-12-19T09:03:18.296171+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:54.050974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279643+0000","last_change":"2023-12-19T09:03:18.279643+0000","last_active":"2023-12-19T09:03:18.279643+0000","last_peered":"2023-12-19T09:03:18.279643+0000","last_clean":"2023-12-19T09:03:18.279643+0000","last_became_active":"2023-12-19T09:03:18.279335+0000","last_became_peered":"2023-12-19T09:03:18.279335+0000","last_unstale":"2023-12-19T09:03:18.279643+0000","last_undegraded":"2023-12-19T09:03:18.279643+0000","last_fullsized":"2023-12-19T09:03:18.279643+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:51.502865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.294058+0000","last_change":"2023-12-19T09:03:18.294058+0000","last_active":"2023-12-19T09:03:18.294058+0000","last_peered":"2023-12-19T09:03:18.294058+0000","last_clean":"2023-12-19T09:03:18.294058+0000","last_became_active":"2023-12-19T09:03:18.293663+0000","last_became_peered":"2023-12-19T09:03:18.293663+0000","last_unstale":"2023-12-19T09:03:18.294058+0000","last_undegraded":"2023-12-19T09:03:18.294058+0000","last_fullsized":"2023-12-19T09:03:18.294058+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:54:26.569633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298751+0000","last_change":"2023-12-19T09:03:18.293273+0000","last_active":"2023-12-19T09:03:18.298751+0000","last_peered":"2023-12-19T09:03:18.298751+0000","last_clean":"2023-12-19T09:03:18.298751+0000","last_became_active":"2023-12-19T09:03:18.292914+0000","last_became_peered":"2023-12-19T09:03:18.292914+0000","last_unstale":"2023-12-19T09:03:18.298751+0000","last_undegraded":"2023-12-19T09:03:18.298751+0000","last_fullsized":"2023-12-19T09:03:18.298751+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:21.948191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283696+0000","last_change":"2023-12-19T09:03:18.283696+0000","last_active":"2023-12-19T09:03:18.283696+0000","last_peered":"2023-12-19T09:03:18.283696+0000","last_clean":"2023-12-19T09:03:18.283696+0000","last_became_active":"2023-12-19T09:03:18.283305+0000","last_became_peered":"2023-12-19T09:03:18.283305+0000","last_unstale":"2023-12-19T09:03:18.283696+0000","last_undegraded":"2023-12-19T09:03:18.283696+0000","last_fullsized":"2023-12-19T09:03:18.283696+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:03.411954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285897+0000","last_change":"2023-12-19T09:03:18.285897+0000","last_active":"2023-12-19T09:03:18.285897+0000","last_peered":"2023-12-19T09:03:18.285897+0000","last_clean":"2023-12-19T09:03:18.285897+0000","last_became_active":"2023-12-19T09:03:18.283544+0000","last_became_peered":"2023-12-19T09:03:18.283544+0000","last_unstale":"2023-12-19T09:03:18.285897+0000","last_undegraded":"2023-12-19T09:03:18.285897+0000","last_fullsized":"2023-12-19T09:03:18.285897+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:11:48.448681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297901+0000","last_change":"2023-12-19T09:03:18.297901+0000","last_active":"2023-12-19T09:03:18.297901+0000","last_peered":"2023-12-19T09:03:18.297901+0000","last_clean":"2023-12-19T09:03:18.297901+0000","last_became_active":"2023-12-19T09:03:18.296795+0000","last_became_peered":"2023-12-19T09:03:18.296795+0000","last_unstale":"2023-12-19T09:03:18.297901+0000","last_undegraded":"2023-12-19T09:03:18.297901+0000","last_fullsized":"2023-12-19T09:03:18.297901+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:08.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.284567+0000","last_change":"2023-12-19T09:03:18.284567+0000","last_active":"2023-12-19T09:03:18.284567+0000","last_peered":"2023-12-19T09:03:18.284567+0000","last_clean":"2023-12-19T09:03:18.284567+0000","last_became_active":"2023-12-19T09:03:18.283996+0000","last_became_peered":"2023-12-19T09:03:18.283996+0000","last_unstale":"2023-12-19T09:03:18.284567+0000","last_undegraded":"2023-12-19T09:03:18.284567+0000","last_fullsized":"2023-12-19T09:03:18.284567+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:45.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293353+0000","last_change":"2023-12-19T09:03:18.293353+0000","last_active":"2023-12-19T09:03:18.293353+0000","last_peered":"2023-12-19T09:03:18.293353+0000","last_clean":"2023-12-19T09:03:18.293353+0000","last_became_active":"2023-12-19T09:03:18.293038+0000","last_became_peered":"2023-12-19T09:03:18.293038+0000","last_unstale":"2023-12-19T09:03:18.293353+0000","last_undegraded":"2023-12-19T09:03:18.293353+0000","last_fullsized":"2023-12-19T09:03:18.293353+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:35:36.680028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279511+0000","last_change":"2023-12-19T09:03:18.279511+0000","last_active":"2023-12-19T09:03:18.279511+0000","last_peered":"2023-12-19T09:03:18.279511+0000","last_clean":"2023-12-19T09:03:18.279511+0000","last_became_active":"2023-12-19T09:03:18.279216+0000","last_became_peered":"2023-12-19T09:03:18.279216+0000","last_unstale":"2023-12-19T09:03:18.279511+0000","last_undegraded":"2023-12-19T09:03:18.279511+0000","last_fullsized":"2023-12-19T09:03:18.279511+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:07:28.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291486+0000","last_change":"2023-12-19T09:03:18.291486+0000","last_active":"2023-12-19T09:03:18.291486+0000","last_peered":"2023-12-19T09:03:18.291486+0000","last_clean":"2023-12-19T09:03:18.291486+0000","last_became_active":"2023-12-19T09:03:18.290768+0000","last_became_peered":"2023-12-19T09:03:18.290768+0000","last_unstale":"2023-12-19T09:03:18.291486+0000","last_undegraded":"2023-12-19T09:03:18.291486+0000","last_fullsized":"2023-12-19T09:03:18.291486+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:12:33.967037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297890+0000","last_change":"2023-12-19T09:03:18.288562+0000","last_active":"2023-12-19T09:03:18.297890+0000","last_peered":"2023-12-19T09:03:18.297890+0000","last_clean":"2023-12-19T09:03:18.297890+0000","last_became_active":"2023-12-19T09:03:18.288283+0000","last_became_peered":"2023-12-19T09:03:18.288283+0000","last_unstale":"2023-12-19T09:03:18.297890+0000","last_undegraded":"2023-12-19T09:03:18.297890+0000","last_fullsized":"2023-12-19T09:03:18.297890+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:03.164273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286906+0000","last_change":"2023-12-19T09:03:18.286906+0000","last_active":"2023-12-19T09:03:18.286906+0000","last_peered":"2023-12-19T09:03:18.286906+0000","last_clean":"2023-12-19T09:03:18.286906+0000","last_became_active":"2023-12-19T09:03:18.286542+0000","last_became_peered":"2023-12-19T09:03:18.286542+0000","last_unstale":"2023-12-19T09:03:18.286906+0000","last_undegraded":"2023-12-19T09:03:18.286906+0000","last_fullsized":"2023-12-19T09:03:18.286906+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:55:13.193453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297840+0000","last_change":"2023-12-19T09:03:18.290695+0000","last_active":"2023-12-19T09:03:18.297840+0000","last_peered":"2023-12-19T09:03:18.297840+0000","last_clean":"2023-12-19T09:03:18.297840+0000","last_became_active":"2023-12-19T09:03:18.290413+0000","last_became_peered":"2023-12-19T09:03:18.290413+0000","last_unstale":"2023-12-19T09:03:18.297840+0000","last_undegraded":"2023-12-19T09:03:18.297840+0000","last_fullsized":"2023-12-19T09:03:18.297840+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:52:59.793131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292690+0000","last_change":"2023-12-19T09:03:18.292690+0000","last_active":"2023-12-19T09:03:18.292690+0000","last_peered":"2023-12-19T09:03:18.292690+0000","last_clean":"2023-12-19T09:03:18.292690+0000","last_became_active":"2023-12-19T09:03:18.292010+0000","last_became_peered":"2023-12-19T09:03:18.292010+0000","last_unstale":"2023-12-19T09:03:18.292690+0000","last_undegraded":"2023-12-19T09:03:18.292690+0000","last_fullsized":"2023-12-19T09:03:18.292690+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:00:08.238021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296661+0000","last_change":"2023-12-19T09:03:18.296661+0000","last_active":"2023-12-19T09:03:18.296661+0000","last_peered":"2023-12-19T09:03:18.296661+0000","last_clean":"2023-12-19T09:03:18.296661+0000","last_became_active":"2023-12-19T09:03:18.296297+0000","last_became_peered":"2023-12-19T09:03:18.296297+0000","last_unstale":"2023-12-19T09:03:18.296661+0000","last_undegraded":"2023-12-19T09:03:18.296661+0000","last_fullsized":"2023-12-19T09:03:18.296661+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:08:54.450057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298675+0000","last_change":"2023-12-19T09:03:18.284446+0000","last_active":"2023-12-19T09:03:18.298675+0000","last_peered":"2023-12-19T09:03:18.298675+0000","last_clean":"2023-12-19T09:03:18.298675+0000","last_became_active":"2023-12-19T09:03:18.284142+0000","last_became_peered":"2023-12-19T09:03:18.284142+0000","last_unstale":"2023-12-19T09:03:18.298675+0000","last_undegraded":"2023-12-19T09:03:18.298675+0000","last_fullsized":"2023-12-19T09:03:18.298675+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:22.599375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.288655+0000","last_change":"2023-12-19T09:03:18.288655+0000","last_active":"2023-12-19T09:03:18.288655+0000","last_peered":"2023-12-19T09:03:18.288655+0000","last_clean":"2023-12-19T09:03:18.288655+0000","last_became_active":"2023-12-19T09:03:18.288183+0000","last_became_peered":"2023-12-19T09:03:18.288183+0000","last_unstale":"2023-12-19T09:03:18.288655+0000","last_undegraded":"2023-12-19T09:03:18.288655+0000","last_fullsized":"2023-12-19T09:03:18.288655+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:34.113825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.303554+0000","last_change":"2023-12-19T09:03:18.303554+0000","last_active":"2023-12-19T09:03:18.303554+0000","last_peered":"2023-12-19T09:03:18.303554+0000","last_clean":"2023-12-19T09:03:18.303554+0000","last_became_active":"2023-12-19T09:03:18.301508+0000","last_became_peered":"2023-12-19T09:03:18.301508+0000","last_unstale":"2023-12-19T09:03:18.303554+0000","last_undegraded":"2023-12-19T09:03:18.303554+0000","last_fullsized":"2023-12-19T09:03:18.303554+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:23:57.924691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281182+0000","last_change":"2023-12-19T09:03:18.276536+0000","last_active":"2023-12-19T09:03:18.281182+0000","last_peered":"2023-12-19T09:03:18.281182+0000","last_clean":"2023-12-19T09:03:18.281182+0000","last_became_active":"2023-12-19T09:03:18.276236+0000","last_became_peered":"2023-12-19T09:03:18.276236+0000","last_unstale":"2023-12-19T09:03:18.281182+0000","last_undegraded":"2023-12-19T09:03:18.281182+0000","last_fullsized":"2023-12-19T09:03:18.281182+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:54.050974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292743+0000","last_change":"2023-12-19T09:03:18.292743+0000","last_active":"2023-12-19T09:03:18.292743+0000","last_peered":"2023-12-19T09:03:18.292743+0000","last_clean":"2023-12-19T09:03:18.292743+0000","last_became_active":"2023-12-19T09:03:18.292470+0000","last_became_peered":"2023-12-19T09:03:18.292470+0000","last_unstale":"2023-12-19T09:03:18.292743+0000","last_undegraded":"2023-12-19T09:03:18.292743+0000","last_fullsized":"2023-12-19T09:03:18.292743+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:20:28.127539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290383+0000","last_change":"2023-12-19T09:03:18.290383+0000","last_active":"2023-12-19T09:03:18.290383+0000","last_peered":"2023-12-19T09:03:18.290383+0000","last_clean":"2023-12-19T09:03:18.290383+0000","last_became_active":"2023-12-19T09:03:18.290096+0000","last_became_peered":"2023-12-19T09:03:18.290096+0000","last_unstale":"2023-12-19T09:03:18.290383+0000","last_undegraded":"2023-12-19T09:03:18.290383+0000","last_fullsized":"2023-12-19T09:03:18.290383+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:01:58.449153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286411+0000","last_change":"2023-12-19T09:03:18.286411+0000","last_active":"2023-12-19T09:03:18.286411+0000","last_peered":"2023-12-19T09:03:18.286411+0000","last_clean":"2023-12-19T09:03:18.286411+0000","last_became_active":"2023-12-19T09:03:18.286153+0000","last_became_peered":"2023-12-19T09:03:18.286153+0000","last_unstale":"2023-12-19T09:03:18.286411+0000","last_undegraded":"2023-12-19T09:03:18.286411+0000","last_fullsized":"2023-12-19T09:03:18.286411+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:51.884295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283743+0000","last_change":"2023-12-19T09:03:18.283743+0000","last_active":"2023-12-19T09:03:18.283743+0000","last_peered":"2023-12-19T09:03:18.283743+0000","last_clean":"2023-12-19T09:03:18.283743+0000","last_became_active":"2023-12-19T09:03:18.283387+0000","last_became_peered":"2023-12-19T09:03:18.283387+0000","last_unstale":"2023-12-19T09:03:18.283743+0000","last_undegraded":"2023-12-19T09:03:18.283743+0000","last_fullsized":"2023-12-19T09:03:18.283743+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:20.644044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.289483+0000","last_change":"2023-12-19T09:03:18.289483+0000","last_active":"2023-12-19T09:03:18.289483+0000","last_peered":"2023-12-19T09:03:18.289483+0000","last_clean":"2023-12-19T09:03:18.289483+0000","last_became_active":"2023-12-19T09:03:18.288335+0000","last_became_peered":"2023-12-19T09:03:18.288335+0000","last_unstale":"2023-12-19T09:03:18.289483+0000","last_undegraded":"2023-12-19T09:03:18.289483+0000","last_fullsized":"2023-12-19T09:03:18.289483+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290177+0000","last_change":"2023-12-19T09:03:18.290177+0000","last_active":"2023-12-19T09:03:18.290177+0000","last_peered":"2023-12-19T09:03:18.290177+0000","last_clean":"2023-12-19T09:03:18.290177+0000","last_became_active":"2023-12-19T09:03:18.289526+0000","last_became_peered":"2023-12-19T09:03:18.289526+0000","last_unstale":"2023-12-19T09:03:18.290177+0000","last_undegraded":"2023-12-19T09:03:18.290177+0000","last_fullsized":"2023-12-19T09:03:18.290177+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:54.402255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301753+0000","last_change":"2023-12-19T09:03:18.301753+0000","last_active":"2023-12-19T09:03:18.301753+0000","last_peered":"2023-12-19T09:03:18.301753+0000","last_clean":"2023-12-19T09:03:18.301753+0000","last_became_active":"2023-12-19T09:03:18.301414+0000","last_became_peered":"2023-12-19T09:03:18.301414+0000","last_unstale":"2023-12-19T09:03:18.301753+0000","last_undegraded":"2023-12-19T09:03:18.301753+0000","last_fullsized":"2023-12-19T09:03:18.301753+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:37:17.109651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274655+0000","last_change":"2023-12-19T09:03:18.274655+0000","last_active":"2023-12-19T09:03:18.274655+0000","last_peered":"2023-12-19T09:03:18.274655+0000","last_clean":"2023-12-19T09:03:18.274655+0000","last_became_active":"2023-12-19T09:03:18.274282+0000","last_became_peered":"2023-12-19T09:03:18.274282+0000","last_unstale":"2023-12-19T09:03:18.274655+0000","last_undegraded":"2023-12-19T09:03:18.274655+0000","last_fullsized":"2023-12-19T09:03:18.274655+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:31.967552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285799+0000","last_change":"2023-12-19T09:03:18.285799+0000","last_active":"2023-12-19T09:03:18.285799+0000","last_peered":"2023-12-19T09:03:18.285799+0000","last_clean":"2023-12-19T09:03:18.285799+0000","last_became_active":"2023-12-19T09:03:18.284892+0000","last_became_peered":"2023-12-19T09:03:18.284892+0000","last_unstale":"2023-12-19T09:03:18.285799+0000","last_undegraded":"2023-12-19T09:03:18.285799+0000","last_fullsized":"2023-12-19T09:03:18.285799+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:26:23.045513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293451+0000","last_change":"2023-12-19T09:03:18.293451+0000","last_active":"2023-12-19T09:03:18.293451+0000","last_peered":"2023-12-19T09:03:18.293451+0000","last_clean":"2023-12-19T09:03:18.293451+0000","last_became_active":"2023-12-19T09:03:18.293115+0000","last_became_peered":"2023-12-19T09:03:18.293115+0000","last_unstale":"2023-12-19T09:03:18.293451+0000","last_undegraded":"2023-12-19T09:03:18.293451+0000","last_fullsized":"2023-12-19T09:03:18.293451+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:05:53.370426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296597+0000","last_change":"2023-12-19T09:03:18.296597+0000","last_active":"2023-12-19T09:03:18.296597+0000","last_peered":"2023-12-19T09:03:18.296597+0000","last_clean":"2023-12-19T09:03:18.296597+0000","last_became_active":"2023-12-19T09:03:18.296295+0000","last_became_peered":"2023-12-19T09:03:18.296295+0000","last_unstale":"2023-12-19T09:03:18.296597+0000","last_undegraded":"2023-12-19T09:03:18.296597+0000","last_fullsized":"2023-12-19T09:03:18.296597+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:37:06.058067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.284307+0000","last_change":"2023-12-19T09:03:18.284307+0000","last_active":"2023-12-19T09:03:18.284307+0000","last_peered":"2023-12-19T09:03:18.284307+0000","last_clean":"2023-12-19T09:03:18.284307+0000","last_became_active":"2023-12-19T09:03:18.283621+0000","last_became_peered":"2023-12-19T09:03:18.283621+0000","last_unstale":"2023-12-19T09:03:18.284307+0000","last_undegraded":"2023-12-19T09:03:18.284307+0000","last_fullsized":"2023-12-19T09:03:18.284307+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:44.588695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292409+0000","last_change":"2023-12-19T09:03:18.292409+0000","last_active":"2023-12-19T09:03:18.292409+0000","last_peered":"2023-12-19T09:03:18.292409+0000","last_clean":"2023-12-19T09:03:18.292409+0000","last_became_active":"2023-12-19T09:03:18.292031+0000","last_became_peered":"2023-12-19T09:03:18.292031+0000","last_unstale":"2023-12-19T09:03:18.292409+0000","last_undegraded":"2023-12-19T09:03:18.292409+0000","last_fullsized":"2023-12-19T09:03:18.292409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306051+0000","last_change":"2023-12-19T09:03:17.255555+0000","last_active":"2023-12-19T09:03:18.306051+0000","last_peered":"2023-12-19T09:03:18.306051+0000","last_clean":"2023-12-19T09:03:18.306051+0000","last_became_active":"2023-12-19T09:03:17.255192+0000","last_became_peered":"2023-12-19T09:03:17.255192+0000","last_unstale":"2023-12-19T09:03:18.306051+0000","last_undegraded":"2023-12-19T09:03:18.306051+0000","last_fullsized":"2023-12-19T09:03:18.306051+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:10.364450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279417+0000","last_change":"2023-12-19T09:03:18.279417+0000","last_active":"2023-12-19T09:03:18.279417+0000","last_peered":"2023-12-19T09:03:18.279417+0000","last_clean":"2023-12-19T09:03:18.279417+0000","last_became_active":"2023-12-19T09:03:18.279101+0000","last_became_peered":"2023-12-19T09:03:18.279101+0000","last_unstale":"2023-12-19T09:03:18.279417+0000","last_undegraded":"2023-12-19T09:03:18.279417+0000","last_fullsized":"2023-12-19T09:03:18.279417+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:19:52.511411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.287660+0000","last_change":"2023-12-19T09:03:17.269617+0000","last_active":"2023-12-19T09:03:18.287660+0000","last_peered":"2023-12-19T09:03:18.287660+0000","last_clean":"2023-12-19T09:03:18.287660+0000","last_became_active":"2023-12-19T09:03:17.269249+0000","last_became_peered":"2023-12-19T09:03:17.269249+0000","last_unstale":"2023-12-19T09:03:18.287660+0000","last_undegraded":"2023-12-19T09:03:18.287660+0000","last_fullsized":"2023-12-19T09:03:18.287660+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:53.388587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280809+0000","last_change":"2023-12-19T09:03:18.280809+0000","last_active":"2023-12-19T09:03:18.280809+0000","last_peered":"2023-12-19T09:03:18.280809+0000","last_clean":"2023-12-19T09:03:18.280809+0000","last_became_active":"2023-12-19T09:03:18.280449+0000","last_became_peered":"2023-12-19T09:03:18.280449+0000","last_unstale":"2023-12-19T09:03:18.280809+0000","last_undegraded":"2023-12-19T09:03:18.280809+0000","last_fullsized":"2023-12-19T09:03:18.280809+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:14.067771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.308917+0000","last_change":"2023-12-19T09:03:17.271627+0000","last_active":"2023-12-19T09:03:18.308917+0000","last_peered":"2023-12-19T09:03:18.308917+0000","last_clean":"2023-12-19T09:03:18.308917+0000","last_became_active":"2023-12-19T09:03:17.271281+0000","last_became_peered":"2023-12-19T09:03:17.271281+0000","last_unstale":"2023-12-19T09:03:18.308917+0000","last_undegraded":"2023-12-19T09:03:18.308917+0000","last_fullsized":"2023-12-19T09:03:18.308917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:53.037306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301726+0000","last_change":"2023-12-19T09:03:18.301726+0000","last_active":"2023-12-19T09:03:18.301726+0000","last_peered":"2023-12-19T09:03:18.301726+0000","last_clean":"2023-12-19T09:03:18.301726+0000","last_became_active":"2023-12-19T09:03:18.301436+0000","last_became_peered":"2023-12-19T09:03:18.301436+0000","last_unstale":"2023-12-19T09:03:18.301726+0000","last_undegraded":"2023-12-19T09:03:18.301726+0000","last_fullsized":"2023-12-19T09:03:18.301726+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:19:11.378118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.303278+0000","last_change":"2023-12-19T09:03:17.262871+0000","last_active":"2023-12-19T09:03:18.303278+0000","last_peered":"2023-12-19T09:03:18.303278+0000","last_clean":"2023-12-19T09:03:18.303278+0000","last_became_active":"2023-12-19T09:03:17.262583+0000","last_became_peered":"2023-12-19T09:03:17.262583+0000","last_unstale":"2023-12-19T09:03:18.303278+0000","last_undegraded":"2023-12-19T09:03:18.303278+0000","last_fullsized":"2023-12-19T09:03:18.303278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:32.153702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292884+0000","last_change":"2023-12-19T09:03:18.282998+0000","last_active":"2023-12-19T09:03:18.292884+0000","last_peered":"2023-12-19T09:03:18.292884+0000","last_clean":"2023-12-19T09:03:18.292884+0000","last_became_active":"2023-12-19T09:03:18.282598+0000","last_became_peered":"2023-12-19T09:03:18.282598+0000","last_unstale":"2023-12-19T09:03:18.292884+0000","last_undegraded":"2023-12-19T09:03:18.292884+0000","last_fullsized":"2023-12-19T09:03:18.292884+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:44.386409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.266338+0000","last_change":"2023-12-19T09:03:17.258207+0000","last_active":"2023-12-19T09:03:18.266338+0000","last_peered":"2023-12-19T09:03:18.266338+0000","last_clean":"2023-12-19T09:03:18.266338+0000","last_became_active":"2023-12-19T09:03:17.257930+0000","last_became_peered":"2023-12-19T09:03:17.257930+0000","last_unstale":"2023-12-19T09:03:18.266338+0000","last_undegraded":"2023-12-19T09:03:18.266338+0000","last_fullsized":"2023-12-19T09:03:18.266338+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:38.014458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296879+0000","last_change":"2023-12-19T09:03:18.283423+0000","last_active":"2023-12-19T09:03:18.296879+0000","last_peered":"2023-12-19T09:03:18.296879+0000","last_clean":"2023-12-19T09:03:18.296879+0000","last_became_active":"2023-12-19T09:03:18.283108+0000","last_became_peered":"2023-12-19T09:03:18.283108+0000","last_unstale":"2023-12-19T09:03:18.296879+0000","last_undegraded":"2023-12-19T09:03:18.296879+0000","last_fullsized":"2023-12-19T09:03:18.296879+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:09:34.132918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279330+0000","last_change":"2023-12-19T09:03:17.262122+0000","last_active":"2023-12-19T09:03:18.279330+0000","last_peered":"2023-12-19T09:03:18.279330+0000","last_clean":"2023-12-19T09:03:18.279330+0000","last_became_active":"2023-12-19T09:03:17.261747+0000","last_became_peered":"2023-12-19T09:03:17.261747+0000","last_unstale":"2023-12-19T09:03:18.279330+0000","last_undegraded":"2023-12-19T09:03:18.279330+0000","last_fullsized":"2023-12-19T09:03:18.279330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:44.664658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281692+0000","last_change":"2023-12-19T09:03:18.281692+0000","last_active":"2023-12-19T09:03:18.281692+0000","last_peered":"2023-12-19T09:03:18.281692+0000","last_clean":"2023-12-19T09:03:18.281692+0000","last_became_active":"2023-12-19T09:03:18.281360+0000","last_became_peered":"2023-12-19T09:03:18.281360+0000","last_unstale":"2023-12-19T09:03:18.281692+0000","last_undegraded":"2023-12-19T09:03:18.281692+0000","last_fullsized":"2023-12-19T09:03:18.281692+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:09.637960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.272625+0000","last_change":"2023-12-19T09:03:17.269992+0000","last_active":"2023-12-19T09:03:17.272625+0000","last_peered":"2023-12-19T09:03:17.272625+0000","last_clean":"2023-12-19T09:03:17.272625+0000","last_became_active":"2023-12-19T09:03:17.269712+0000","last_became_peered":"2023-12-19T09:03:17.269712+0000","last_unstale":"2023-12-19T09:03:17.272625+0000","last_undegraded":"2023-12-19T09:03:17.272625+0000","last_fullsized":"2023-12-19T09:03:17.272625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:46:43.575027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280905+0000","last_change":"2023-12-19T09:03:18.280905+0000","last_active":"2023-12-19T09:03:18.280905+0000","last_peered":"2023-12-19T09:03:18.280905+0000","last_clean":"2023-12-19T09:03:18.280905+0000","last_became_active":"2023-12-19T09:03:18.280339+0000","last_became_peered":"2023-12-19T09:03:18.280339+0000","last_unstale":"2023-12-19T09:03:18.280905+0000","last_undegraded":"2023-12-19T09:03:18.280905+0000","last_fullsized":"2023-12-19T09:03:18.280905+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:39:52.030789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.263326+0000","last_change":"2023-12-19T09:03:17.260603+0000","last_active":"2023-12-19T09:03:17.263326+0000","last_peered":"2023-12-19T09:03:17.263326+0000","last_clean":"2023-12-19T09:03:17.263326+0000","last_became_active":"2023-12-19T09:03:17.260314+0000","last_became_peered":"2023-12-19T09:03:17.260314+0000","last_unstale":"2023-12-19T09:03:17.263326+0000","last_undegraded":"2023-12-19T09:03:17.263326+0000","last_fullsized":"2023-12-19T09:03:17.263326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:12.272351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.292598+0000","last_change":"2023-12-19T09:03:18.292598+0000","last_active":"2023-12-19T09:03:18.292598+0000","last_peered":"2023-12-19T09:03:18.292598+0000","last_clean":"2023-12-19T09:03:18.292598+0000","last_became_active":"2023-12-19T09:03:18.292280+0000","last_became_peered":"2023-12-19T09:03:18.292280+0000","last_unstale":"2023-12-19T09:03:18.292598+0000","last_undegraded":"2023-12-19T09:03:18.292598+0000","last_fullsized":"2023-12-19T09:03:18.292598+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:30.094154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.274675+0000","last_change":"2023-12-19T09:03:17.272325+0000","last_active":"2023-12-19T09:03:17.274675+0000","last_peered":"2023-12-19T09:03:17.274675+0000","last_clean":"2023-12-19T09:03:17.274675+0000","last_became_active":"2023-12-19T09:03:17.271811+0000","last_became_peered":"2023-12-19T09:03:17.271811+0000","last_unstale":"2023-12-19T09:03:17.274675+0000","last_undegraded":"2023-12-19T09:03:17.274675+0000","last_fullsized":"2023-12-19T09:03:17.274675+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:08.624292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.290290+0000","last_change":"2023-12-19T09:03:18.290290+0000","last_active":"2023-12-19T09:03:18.290290+0000","last_peered":"2023-12-19T09:03:18.290290+0000","last_clean":"2023-12-19T09:03:18.290290+0000","last_became_active":"2023-12-19T09:03:18.289898+0000","last_became_peered":"2023-12-19T09:03:18.289898+0000","last_unstale":"2023-12-19T09:03:18.290290+0000","last_undegraded":"2023-12-19T09:03:18.290290+0000","last_fullsized":"2023-12-19T09:03:18.290290+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:32:09.219048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.273062+0000","last_change":"2023-12-19T09:03:17.271126+0000","last_active":"2023-12-19T09:03:17.273062+0000","last_peered":"2023-12-19T09:03:17.273062+0000","last_clean":"2023-12-19T09:03:17.273062+0000","last_became_active":"2023-12-19T09:03:17.270876+0000","last_became_peered":"2023-12-19T09:03:17.270876+0000","last_unstale":"2023-12-19T09:03:17.273062+0000","last_undegraded":"2023-12-19T09:03:17.273062+0000","last_fullsized":"2023-12-19T09:03:17.273062+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:41:29.080486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.288465+0000","last_change":"2023-12-19T09:03:18.288465+0000","last_active":"2023-12-19T09:03:18.288465+0000","last_peered":"2023-12-19T09:03:18.288465+0000","last_clean":"2023-12-19T09:03:18.288465+0000","last_became_active":"2023-12-19T09:03:18.287809+0000","last_became_peered":"2023-12-19T09:03:18.287809+0000","last_unstale":"2023-12-19T09:03:18.288465+0000","last_undegraded":"2023-12-19T09:03:18.288465+0000","last_fullsized":"2023-12-19T09:03:18.288465+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:50:39.028126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306182+0000","last_change":"2023-12-19T09:03:17.269226+0000","last_active":"2023-12-19T09:03:18.306182+0000","last_peered":"2023-12-19T09:03:18.306182+0000","last_clean":"2023-12-19T09:03:18.306182+0000","last_became_active":"2023-12-19T09:03:17.268946+0000","last_became_peered":"2023-12-19T09:03:17.268946+0000","last_unstale":"2023-12-19T09:03:18.306182+0000","last_undegraded":"2023-12-19T09:03:18.306182+0000","last_fullsized":"2023-12-19T09:03:18.306182+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:59:19.308422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291428+0000","last_change":"2023-12-19T09:03:18.291428+0000","last_active":"2023-12-19T09:03:18.291428+0000","last_peered":"2023-12-19T09:03:18.291428+0000","last_clean":"2023-12-19T09:03:18.291428+0000","last_became_active":"2023-12-19T09:03:18.291116+0000","last_became_peered":"2023-12-19T09:03:18.291116+0000","last_unstale":"2023-12-19T09:03:18.291428+0000","last_undegraded":"2023-12-19T09:03:18.291428+0000","last_fullsized":"2023-12-19T09:03:18.291428+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:33.167370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.304096+0000","last_change":"2023-12-19T09:03:17.266754+0000","last_active":"2023-12-19T09:03:18.304096+0000","last_peered":"2023-12-19T09:03:18.304096+0000","last_clean":"2023-12-19T09:03:18.304096+0000","last_became_active":"2023-12-19T09:03:17.266220+0000","last_became_peered":"2023-12-19T09:03:17.266220+0000","last_unstale":"2023-12-19T09:03:18.304096+0000","last_undegraded":"2023-12-19T09:03:18.304096+0000","last_fullsized":"2023-12-19T09:03:18.304096+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:34.799815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.281983+0000","last_change":"2023-12-19T09:03:18.281983+0000","last_active":"2023-12-19T09:03:18.281983+0000","last_peered":"2023-12-19T09:03:18.281983+0000","last_clean":"2023-12-19T09:03:18.281983+0000","last_became_active":"2023-12-19T09:03:18.281621+0000","last_became_peered":"2023-12-19T09:03:18.281621+0000","last_unstale":"2023-12-19T09:03:18.281983+0000","last_undegraded":"2023-12-19T09:03:18.281983+0000","last_fullsized":"2023-12-19T09:03:18.281983+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:49:35.970393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.279612+0000","last_change":"2023-12-19T09:03:17.261553+0000","last_active":"2023-12-19T09:03:18.279612+0000","last_peered":"2023-12-19T09:03:18.279612+0000","last_clean":"2023-12-19T09:03:18.279612+0000","last_became_active":"2023-12-19T09:03:17.260798+0000","last_became_peered":"2023-12-19T09:03:17.260798+0000","last_unstale":"2023-12-19T09:03:18.279612+0000","last_undegraded":"2023-12-19T09:03:18.279612+0000","last_fullsized":"2023-12-19T09:03:18.279612+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:22:30.953884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.291568+0000","last_change":"2023-12-19T09:03:18.291568+0000","last_active":"2023-12-19T09:03:18.291568+0000","last_peered":"2023-12-19T09:03:18.291568+0000","last_clean":"2023-12-19T09:03:18.291568+0000","last_became_active":"2023-12-19T09:03:18.287667+0000","last_became_peered":"2023-12-19T09:03:18.287667+0000","last_unstale":"2023-12-19T09:03:18.291568+0000","last_undegraded":"2023-12-19T09:03:18.291568+0000","last_fullsized":"2023-12-19T09:03:18.291568+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:14:57.437656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274484+0000","last_change":"2023-12-19T09:03:17.266076+0000","last_active":"2023-12-19T09:03:18.274484+0000","last_peered":"2023-12-19T09:03:18.274484+0000","last_clean":"2023-12-19T09:03:18.274484+0000","last_became_active":"2023-12-19T09:03:17.265731+0000","last_became_peered":"2023-12-19T09:03:17.265731+0000","last_unstale":"2023-12-19T09:03:18.274484+0000","last_undegraded":"2023-12-19T09:03:18.274484+0000","last_fullsized":"2023-12-19T09:03:18.274484+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:29:22.384162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285593+0000","last_change":"2023-12-19T09:03:18.285593+0000","last_active":"2023-12-19T09:03:18.285593+0000","last_peered":"2023-12-19T09:03:18.285593+0000","last_clean":"2023-12-19T09:03:18.285593+0000","last_became_active":"2023-12-19T09:03:18.285002+0000","last_became_peered":"2023-12-19T09:03:18.285002+0000","last_unstale":"2023-12-19T09:03:18.285593+0000","last_undegraded":"2023-12-19T09:03:18.285593+0000","last_fullsized":"2023-12-19T09:03:18.285593+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:30.151876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.311745+0000","last_change":"2023-12-19T09:03:17.266705+0000","last_active":"2023-12-19T09:03:18.311745+0000","last_peered":"2023-12-19T09:03:18.311745+0000","last_clean":"2023-12-19T09:03:18.311745+0000","last_became_active":"2023-12-19T09:03:17.266394+0000","last_became_peered":"2023-12-19T09:03:17.266394+0000","last_unstale":"2023-12-19T09:03:18.311745+0000","last_undegraded":"2023-12-19T09:03:18.311745+0000","last_fullsized":"2023-12-19T09:03:18.311745+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:32.153702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.295964+0000","last_change":"2023-12-19T09:03:18.295964+0000","last_active":"2023-12-19T09:03:18.295964+0000","last_peered":"2023-12-19T09:03:18.295964+0000","last_clean":"2023-12-19T09:03:18.295964+0000","last_became_active":"2023-12-19T09:03:18.295635+0000","last_became_peered":"2023-12-19T09:03:18.295635+0000","last_unstale":"2023-12-19T09:03:18.295964+0000","last_undegraded":"2023-12-19T09:03:18.295964+0000","last_fullsized":"2023-12-19T09:03:18.295964+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:35.813483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.268355+0000","last_change":"2023-12-19T09:03:17.271197+0000","last_active":"2023-12-19T09:03:18.268355+0000","last_peered":"2023-12-19T09:03:18.268355+0000","last_clean":"2023-12-19T09:03:18.268355+0000","last_became_active":"2023-12-19T09:03:17.270747+0000","last_became_peered":"2023-12-19T09:03:17.270747+0000","last_unstale":"2023-12-19T09:03:18.268355+0000","last_undegraded":"2023-12-19T09:03:18.268355+0000","last_fullsized":"2023-12-19T09:03:18.268355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:54:45.930962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.301173+0000","last_change":"2023-12-19T09:03:18.301173+0000","last_active":"2023-12-19T09:03:18.301173+0000","last_peered":"2023-12-19T09:03:18.301173+0000","last_clean":"2023-12-19T09:03:18.301173+0000","last_became_active":"2023-12-19T09:03:18.300854+0000","last_became_peered":"2023-12-19T09:03:18.300854+0000","last_unstale":"2023-12-19T09:03:18.301173+0000","last_undegraded":"2023-12-19T09:03:18.301173+0000","last_fullsized":"2023-12-19T09:03:18.301173+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:51:13.286019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":11,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.306732+0000","last_change":"2023-12-19T09:03:17.270249+0000","last_active":"2023-12-19T09:03:18.306732+0000","last_peered":"2023-12-19T09:03:18.306732+0000","last_clean":"2023-12-19T09:03:18.306732+0000","last_became_active":"2023-12-19T09:03:17.269947+0000","last_became_peered":"2023-12-19T09:03:17.269947+0000","last_unstale":"2023-12-19T09:03:18.306732+0000","last_undegraded":"2023-12-19T09:03:18.306732+0000","last_fullsized":"2023-12-19T09:03:18.306732+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:50.489197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285271+0000","last_change":"2023-12-19T09:03:18.285271+0000","last_active":"2023-12-19T09:03:18.285271+0000","last_peered":"2023-12-19T09:03:18.285271+0000","last_clean":"2023-12-19T09:03:18.285271+0000","last_became_active":"2023-12-19T09:03:18.284967+0000","last_became_peered":"2023-12-19T09:03:18.284967+0000","last_unstale":"2023-12-19T09:03:18.285271+0000","last_undegraded":"2023-12-19T09:03:18.285271+0000","last_fullsized":"2023-12-19T09:03:18.285271+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:09.637960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.272091+0000","last_change":"2023-12-19T09:03:17.261110+0000","last_active":"2023-12-19T09:03:18.272091+0000","last_peered":"2023-12-19T09:03:18.272091+0000","last_clean":"2023-12-19T09:03:18.272091+0000","last_became_active":"2023-12-19T09:03:17.260798+0000","last_became_peered":"2023-12-19T09:03:17.260798+0000","last_unstale":"2023-12-19T09:03:18.272091+0000","last_undegraded":"2023-12-19T09:03:18.272091+0000","last_fullsized":"2023-12-19T09:03:18.272091+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:00:43.821041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297093+0000","last_change":"2023-12-19T09:03:18.297093+0000","last_active":"2023-12-19T09:03:18.297093+0000","last_peered":"2023-12-19T09:03:18.297093+0000","last_clean":"2023-12-19T09:03:18.297093+0000","last_became_active":"2023-12-19T09:03:18.288739+0000","last_became_peered":"2023-12-19T09:03:18.288739+0000","last_unstale":"2023-12-19T09:03:18.297093+0000","last_undegraded":"2023-12-19T09:03:18.297093+0000","last_fullsized":"2023-12-19T09:03:18.297093+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:29:49.329799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.274644+0000","last_change":"2023-12-19T09:03:17.264462+0000","last_active":"2023-12-19T09:03:18.274644+0000","last_peered":"2023-12-19T09:03:18.274644+0000","last_clean":"2023-12-19T09:03:18.274644+0000","last_became_active":"2023-12-19T09:03:17.264166+0000","last_became_peered":"2023-12-19T09:03:17.264166+0000","last_unstale":"2023-12-19T09:03:18.274644+0000","last_undegraded":"2023-12-19T09:03:18.274644+0000","last_fullsized":"2023-12-19T09:03:18.274644+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:17:04.792003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.299933+0000","last_change":"2023-12-19T09:03:18.299933+0000","last_active":"2023-12-19T09:03:18.299933+0000","last_peered":"2023-12-19T09:03:18.299933+0000","last_clean":"2023-12-19T09:03:18.299933+0000","last_became_active":"2023-12-19T09:03:18.299622+0000","last_became_peered":"2023-12-19T09:03:18.299622+0000","last_unstale":"2023-12-19T09:03:18.299933+0000","last_undegraded":"2023-12-19T09:03:18.299933+0000","last_fullsized":"2023-12-19T09:03:18.299933+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:53:45.678326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.305278+0000","last_change":"2023-12-19T09:03:17.257950+0000","last_active":"2023-12-19T09:03:18.305278+0000","last_peered":"2023-12-19T09:03:18.305278+0000","last_clean":"2023-12-19T09:03:18.305278+0000","last_became_active":"2023-12-19T09:03:17.257658+0000","last_became_peered":"2023-12-19T09:03:17.257658+0000","last_unstale":"2023-12-19T09:03:18.305278+0000","last_undegraded":"2023-12-19T09:03:18.305278+0000","last_fullsized":"2023-12-19T09:03:18.305278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:48.318228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.283595+0000","last_change":"2023-12-19T09:03:18.283595+0000","last_active":"2023-12-19T09:03:18.283595+0000","last_peered":"2023-12-19T09:03:18.283595+0000","last_clean":"2023-12-19T09:03:18.283595+0000","last_became_active":"2023-12-19T09:03:18.283284+0000","last_became_peered":"2023-12-19T09:03:18.283284+0000","last_unstale":"2023-12-19T09:03:18.283595+0000","last_undegraded":"2023-12-19T09:03:18.283595+0000","last_fullsized":"2023-12-19T09:03:18.283595+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:42:37.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.266808+0000","last_change":"2023-12-19T09:03:17.260978+0000","last_active":"2023-12-19T09:03:17.266808+0000","last_peered":"2023-12-19T09:03:17.266808+0000","last_clean":"2023-12-19T09:03:17.266808+0000","last_became_active":"2023-12-19T09:03:17.260645+0000","last_became_peered":"2023-12-19T09:03:17.260645+0000","last_unstale":"2023-12-19T09:03:17.266808+0000","last_undegraded":"2023-12-19T09:03:17.266808+0000","last_fullsized":"2023-12-19T09:03:17.266808+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:22:33.100157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293668+0000","last_change":"2023-12-19T09:03:18.293668+0000","last_active":"2023-12-19T09:03:18.293668+0000","last_peered":"2023-12-19T09:03:18.293668+0000","last_clean":"2023-12-19T09:03:18.293668+0000","last_became_active":"2023-12-19T09:03:18.293392+0000","last_became_peered":"2023-12-19T09:03:18.293392+0000","last_unstale":"2023-12-19T09:03:18.293668+0000","last_undegraded":"2023-12-19T09:03:18.293668+0000","last_fullsized":"2023-12-19T09:03:18.293668+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:08:38.027468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.261438+0000","last_change":"2023-12-19T09:03:17.251954+0000","last_active":"2023-12-19T09:03:18.261438+0000","last_peered":"2023-12-19T09:03:18.261438+0000","last_clean":"2023-12-19T09:03:18.261438+0000","last_became_active":"2023-12-19T09:03:17.251514+0000","last_became_peered":"2023-12-19T09:03:17.251514+0000","last_unstale":"2023-12-19T09:03:18.261438+0000","last_undegraded":"2023-12-19T09:03:18.261438+0000","last_fullsized":"2023-12-19T09:03:18.261438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:42.943753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280905+0000","last_change":"2023-12-19T09:03:18.280905+0000","last_active":"2023-12-19T09:03:18.280905+0000","last_peered":"2023-12-19T09:03:18.280905+0000","last_clean":"2023-12-19T09:03:18.280905+0000","last_became_active":"2023-12-19T09:03:18.280612+0000","last_became_peered":"2023-12-19T09:03:18.280612+0000","last_unstale":"2023-12-19T09:03:18.280905+0000","last_undegraded":"2023-12-19T09:03:18.280905+0000","last_fullsized":"2023-12-19T09:03:18.280905+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:54.402255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.260972+0000","last_change":"2023-12-19T09:03:17.252090+0000","last_active":"2023-12-19T09:03:17.260972+0000","last_peered":"2023-12-19T09:03:17.260972+0000","last_clean":"2023-12-19T09:03:17.260972+0000","last_became_active":"2023-12-19T09:03:17.251779+0000","last_became_peered":"2023-12-19T09:03:17.251779+0000","last_unstale":"2023-12-19T09:03:17.260972+0000","last_undegraded":"2023-12-19T09:03:17.260972+0000","last_fullsized":"2023-12-19T09:03:17.260972+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:47:48.224341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.286969+0000","last_change":"2023-12-19T09:03:18.286969+0000","last_active":"2023-12-19T09:03:18.286969+0000","last_peered":"2023-12-19T09:03:18.286969+0000","last_clean":"2023-12-19T09:03:18.286969+0000","last_became_active":"2023-12-19T09:03:18.283477+0000","last_became_peered":"2023-12-19T09:03:18.283477+0000","last_unstale":"2023-12-19T09:03:18.286969+0000","last_undegraded":"2023-12-19T09:03:18.286969+0000","last_fullsized":"2023-12-19T09:03:18.286969+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:35.887515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.309878+0000","last_change":"2023-12-19T09:03:17.264557+0000","last_active":"2023-12-19T09:03:18.309878+0000","last_peered":"2023-12-19T09:03:18.309878+0000","last_clean":"2023-12-19T09:03:18.309878+0000","last_became_active":"2023-12-19T09:03:17.264268+0000","last_became_peered":"2023-12-19T09:03:17.264268+0000","last_unstale":"2023-12-19T09:03:18.309878+0000","last_undegraded":"2023-12-19T09:03:18.309878+0000","last_fullsized":"2023-12-19T09:03:18.309878+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:25:22.387828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.297918+0000","last_change":"2023-12-19T09:03:18.286955+0000","last_active":"2023-12-19T09:03:18.297918+0000","last_peered":"2023-12-19T09:03:18.297918+0000","last_clean":"2023-12-19T09:03:18.297918+0000","last_became_active":"2023-12-19T09:03:18.286641+0000","last_became_peered":"2023-12-19T09:03:18.286641+0000","last_unstale":"2023-12-19T09:03:18.297918+0000","last_undegraded":"2023-12-19T09:03:18.297918+0000","last_fullsized":"2023-12-19T09:03:18.297918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:07:20.126407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.272603+0000","last_change":"2023-12-19T09:03:17.268683+0000","last_active":"2023-12-19T09:03:17.272603+0000","last_peered":"2023-12-19T09:03:17.272603+0000","last_clean":"2023-12-19T09:03:17.272603+0000","last_became_active":"2023-12-19T09:03:17.268397+0000","last_became_peered":"2023-12-19T09:03:17.268397+0000","last_unstale":"2023-12-19T09:03:17.272603+0000","last_undegraded":"2023-12-19T09:03:17.272603+0000","last_fullsized":"2023-12-19T09:03:17.272603+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:50.870627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.296826+0000","last_change":"2023-12-19T09:03:18.296826+0000","last_active":"2023-12-19T09:03:18.296826+0000","last_peered":"2023-12-19T09:03:18.296826+0000","last_clean":"2023-12-19T09:03:18.296826+0000","last_became_active":"2023-12-19T09:03:18.296509+0000","last_became_peered":"2023-12-19T09:03:18.296509+0000","last_unstale":"2023-12-19T09:03:18.296826+0000","last_undegraded":"2023-12-19T09:03:18.296826+0000","last_fullsized":"2023-12-19T09:03:18.296826+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:35:51.502865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.265305+0000","last_change":"2023-12-19T09:03:17.261172+0000","last_active":"2023-12-19T09:03:17.265305+0000","last_peered":"2023-12-19T09:03:17.265305+0000","last_clean":"2023-12-19T09:03:17.265305+0000","last_became_active":"2023-12-19T09:03:17.260877+0000","last_became_peered":"2023-12-19T09:03:17.260877+0000","last_unstale":"2023-12-19T09:03:17.265305+0000","last_undegraded":"2023-12-19T09:03:17.265305+0000","last_fullsized":"2023-12-19T09:03:17.265305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:23:50.870627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.293835+0000","last_change":"2023-12-19T09:03:18.293835+0000","last_active":"2023-12-19T09:03:18.293835+0000","last_peered":"2023-12-19T09:03:18.293835+0000","last_clean":"2023-12-19T09:03:18.293835+0000","last_became_active":"2023-12-19T09:03:18.293537+0000","last_became_peered":"2023-12-19T09:03:18.293537+0000","last_unstale":"2023-12-19T09:03:18.293835+0000","last_undegraded":"2023-12-19T09:03:18.293835+0000","last_fullsized":"2023-12-19T09:03:18.293835+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:18.784902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.268209+0000","last_change":"2023-12-19T09:03:17.265214+0000","last_active":"2023-12-19T09:03:17.268209+0000","last_peered":"2023-12-19T09:03:17.268209+0000","last_clean":"2023-12-19T09:03:17.268209+0000","last_became_active":"2023-12-19T09:03:17.264758+0000","last_became_peered":"2023-12-19T09:03:17.264758+0000","last_unstale":"2023-12-19T09:03:17.268209+0000","last_undegraded":"2023-12-19T09:03:17.268209+0000","last_fullsized":"2023-12-19T09:03:17.268209+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:58:35.707152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.268708+0000","last_change":"2023-12-19T09:03:18.268708+0000","last_active":"2023-12-19T09:03:18.268708+0000","last_peered":"2023-12-19T09:03:18.268708+0000","last_clean":"2023-12-19T09:03:18.268708+0000","last_became_active":"2023-12-19T09:03:18.268274+0000","last_became_peered":"2023-12-19T09:03:18.268274+0000","last_unstale":"2023-12-19T09:03:18.268708+0000","last_undegraded":"2023-12-19T09:03:18.268708+0000","last_fullsized":"2023-12-19T09:03:18.268708+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:28:45.573661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.268072+0000","last_change":"2023-12-19T09:03:17.262439+0000","last_active":"2023-12-19T09:03:17.268072+0000","last_peered":"2023-12-19T09:03:17.268072+0000","last_clean":"2023-12-19T09:03:17.268072+0000","last_became_active":"2023-12-19T09:03:17.262163+0000","last_became_peered":"2023-12-19T09:03:17.262163+0000","last_unstale":"2023-12-19T09:03:17.268072+0000","last_undegraded":"2023-12-19T09:03:17.268072+0000","last_fullsized":"2023-12-19T09:03:17.268072+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:29:13.054103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.298493+0000","last_change":"2023-12-19T09:03:18.292538+0000","last_active":"2023-12-19T09:03:18.298493+0000","last_peered":"2023-12-19T09:03:18.298493+0000","last_clean":"2023-12-19T09:03:18.298493+0000","last_became_active":"2023-12-19T09:03:18.292249+0000","last_became_peered":"2023-12-19T09:03:18.292249+0000","last_unstale":"2023-12-19T09:03:18.298493+0000","last_undegraded":"2023-12-19T09:03:18.298493+0000","last_fullsized":"2023-12-19T09:03:18.298493+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:15:26.344233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.270210+0000","last_change":"2023-12-19T09:03:17.267694+0000","last_active":"2023-12-19T09:03:17.270210+0000","last_peered":"2023-12-19T09:03:17.270210+0000","last_clean":"2023-12-19T09:03:17.270210+0000","last_became_active":"2023-12-19T09:03:17.266891+0000","last_became_peered":"2023-12-19T09:03:17.266891+0000","last_unstale":"2023-12-19T09:03:17.270210+0000","last_undegraded":"2023-12-19T09:03:17.270210+0000","last_fullsized":"2023-12-19T09:03:17.270210+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:45.481336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.289623+0000","last_change":"2023-12-19T09:03:18.289623+0000","last_active":"2023-12-19T09:03:18.289623+0000","last_peered":"2023-12-19T09:03:18.289623+0000","last_clean":"2023-12-19T09:03:18.289623+0000","last_became_active":"2023-12-19T09:03:18.289066+0000","last_became_peered":"2023-12-19T09:03:18.289066+0000","last_unstale":"2023-12-19T09:03:18.289623+0000","last_undegraded":"2023-12-19T09:03:18.289623+0000","last_fullsized":"2023-12-19T09:03:18.289623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:08.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.273725+0000","last_change":"2023-12-19T09:03:17.271852+0000","last_active":"2023-12-19T09:03:17.273725+0000","last_peered":"2023-12-19T09:03:17.273725+0000","last_clean":"2023-12-19T09:03:17.273725+0000","last_became_active":"2023-12-19T09:03:17.270829+0000","last_became_peered":"2023-12-19T09:03:17.270829+0000","last_unstale":"2023-12-19T09:03:17.273725+0000","last_undegraded":"2023-12-19T09:03:17.273725+0000","last_fullsized":"2023-12-19T09:03:17.273725+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:44.696127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.285834+0000","last_change":"2023-12-19T09:03:18.285834+0000","last_active":"2023-12-19T09:03:18.285834+0000","last_peered":"2023-12-19T09:03:18.285834+0000","last_clean":"2023-12-19T09:03:18.285834+0000","last_became_active":"2023-12-19T09:03:18.285274+0000","last_became_peered":"2023-12-19T09:03:18.285274+0000","last_unstale":"2023-12-19T09:03:18.285834+0000","last_undegraded":"2023-12-19T09:03:18.285834+0000","last_fullsized":"2023-12-19T09:03:18.285834+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:28:43.168774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.274637+0000","last_change":"2023-12-19T09:03:17.271777+0000","last_active":"2023-12-19T09:03:17.274637+0000","last_peered":"2023-12-19T09:03:17.274637+0000","last_clean":"2023-12-19T09:03:17.274637+0000","last_became_active":"2023-12-19T09:03:17.271426+0000","last_became_peered":"2023-12-19T09:03:17.271426+0000","last_unstale":"2023-12-19T09:03:17.274637+0000","last_undegraded":"2023-12-19T09:03:17.274637+0000","last_fullsized":"2023-12-19T09:03:17.274637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:43.372741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":8,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.280844+0000","last_change":"2023-12-19T09:03:18.280844+0000","last_active":"2023-12-19T09:03:18.280844+0000","last_peered":"2023-12-19T09:03:18.280844+0000","last_clean":"2023-12-19T09:03:18.280844+0000","last_became_active":"2023-12-19T09:03:18.280506+0000","last_became_peered":"2023-12-19T09:03:18.280506+0000","last_unstale":"2023-12-19T09:03:18.280844+0000","last_undegraded":"2023-12-19T09:03:18.280844+0000","last_fullsized":"2023-12-19T09:03:18.280844+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:17.226996+0000","last_clean_scrub_stamp":"2023-12-19T09:03:17.226996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:20:47.480393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-12-19T09:03:18.309302+0000","last_change":"2023-12-19T09:03:17.248634+0000","last_active":"2023-12-19T09:03:18.309302+0000","last_peered":"2023-12-19T09:03:18.309302+0000","last_clean":"2023-12-19T09:03:18.309302+0000","last_became_active":"2023-12-19T09:03:17.247915+0000","last_became_peered":"2023-12-19T09:03:17.247915+0000","last_unstale":"2023-12-19T09:03:18.309302+0000","last_undegraded":"2023-12-19T09:03:18.309302+0000","last_fullsized":"2023-12-19T09:03:18.309302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:03:16.213328+0000","last_clean_scrub_stamp":"2023-12-19T09:03:16.213328+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:46:07.923063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"163'293","reported_seq":913,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-12-19T09:03:17.261792+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:03:17.261792+0000","last_peered":"2023-12-19T09:03:17.261792+0000","last_clean":"2023-12-19T09:03:17.261792+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:03:17.261792+0000","last_undegraded":"2023-12-19T09:03:17.261792+0000","last_fullsized":"2023-12-19T09:03:17.261792+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575755,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299996,"kb_used_data":2708,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071844,"statfs":{"total":96636764160,"available":96329568256,"internally_reserved":0,"allocated":2772992,"data_stored":1746131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3420000000000001,"15min":1.3069999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.54400000000000004},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3260000000000001,"15min":1.321},"min":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.66800000000000004},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.218}]},{"osd":1,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1799999999999999,"15min":1.179},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.64000000000000001},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1919999999999999,"15min":1.1879999999999999},"min":{"1min":0.84999999999999998,"5min":0.81799999999999995,"15min":0.64500000000000002},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.784},"last":1.3680000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1930000000000001,"15min":1.173},"min":{"1min":0.92200000000000004,"5min":0.82299999999999995,"15min":0.57199999999999995},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.296,"15min":1.2769999999999999},"min":{"1min":0.88100000000000001,"5min":0.79000000000000004,"15min":0.60899999999999999},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.474}]},{"osd":3,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.3500000000000001,"15min":1.331},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.625},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":4.4290000000000003},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.3420000000000001,"15min":1.3440000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.61099999999999999},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.774},"last":1.548}]},{"osd":4,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2509999999999999,"15min":1.2170000000000001},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.56799999999999995},"max":{"1min":1.6080000000000001,"5min":1.847,"15min":1.847},"last":1.27},{"interface":"front","average":{"1min":1.52,"5min":1.3220000000000001,"15min":1.238},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.42299999999999999},"max":{"1min":6.2629999999999999,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":1.4039999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.27,"15min":1.2390000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.35599999999999998},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.109},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.3660000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.498},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.6699999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:03:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.429,"15min":1.375},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.51600000000000001},"max":{"1min":1.7450000000000001,"5min":1.841,"15min":1.8540000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.3959999999999999,"15min":1.3680000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.47899999999999998},"max":{"1min":2.5609999999999999,"5min":2.5609999999999999,"15min":2.5609999999999999},"last":1.1599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575754,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311704,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060136,"statfs":{"total":96636764160,"available":96317579264,"internally_reserved":0,"allocated":2179072,"data_stored":1155811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.133,"15min":1.129},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.2270000000000001,"15min":1.1819999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.61399999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":4.2809999999999997},"last":1.4570000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.234,"15min":1.196},"min":{"1min":0.755,"5min":0.627,"15min":0.627},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.333},{"interface":"front","average":{"1min":1.083,"5min":1.1559999999999999,"15min":1.175},"min":{"1min":0.78100000000000003,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":1.619,"5min":1.637,"15min":1.637},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2609999999999999,"15min":1.24},"min":{"1min":0.65600000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.159},{"interface":"front","average":{"1min":1.325,"5min":1.2629999999999999,"15min":1.248},"min":{"1min":0.748,"5min":0.748,"15min":0.63400000000000001},"max":{"1min":5.6150000000000002,"5min":5.6150000000000002,"15min":5.6150000000000002},"last":0.90800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3200000000000001,"15min":1.2789999999999999},"min":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.76300000000000001},"max":{"1min":4.5519999999999996,"5min":4.5519999999999996,"15min":4.5519999999999996},"last":1.524},{"interface":"front","average":{"1min":1.385,"5min":1.329,"15min":1.3300000000000001},"min":{"1min":0.82199999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":1.413}]},{"osd":4,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.83499999999999996,"5min":0.66800000000000004,"15min":0.63400000000000001},"max":{"1min":4.125,"5min":4.125,"15min":4.125},"last":1.373},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.4590000000000001,"15min":1.4259999999999999},"min":{"1min":0.97099999999999997,"5min":0.82399999999999995,"15min":0.51100000000000001},"max":{"1min":4.3099999999999996,"5min":4.3099999999999996,"15min":4.3099999999999996},"last":1.5880000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.319,"15min":1.298},"min":{"1min":0.80000000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.228},{"interface":"front","average":{"1min":1.365,"5min":1.3049999999999999,"15min":1.2769999999999999},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.56399999999999995},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.1990000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:19 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.3959999999999999,"15min":1.351},"min":{"1min":0.95199999999999996,"5min":0.76000000000000001,"15min":0.56100000000000005},"max":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"last":1.02},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.4550000000000001,"15min":1.423},"min":{"1min":1.036,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":2.8620000000000001},"last":1.2829999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575754,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303508,"kb_used_data":2124,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94068332,"statfs":{"total":96636764160,"available":96325971968,"internally_reserved":0,"allocated":2174976,"data_stored":1155741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.2769999999999999,"15min":1.3069999999999999},"min":{"1min":0.94199999999999995,"5min":0.626,"15min":0.57999999999999996},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3080000000000001,"15min":1.3149999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.2130000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.163,"15min":1.173},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.57199999999999995},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.238,"15min":1.234},"min":{"1min":0.80700000000000005,"5min":0.80400000000000005,"15min":0.72499999999999998},"max":{"1min":2.8109999999999999,"5min":2.8109999999999999,"15min":2.8109999999999999},"last":1.3700000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2529999999999999,"15min":1.222},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.55300000000000005},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.105},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2310000000000001,"15min":1.252},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.9399999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.82499999999999996}]},{"osd":3,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.294,"15min":1.2869999999999999},"min":{"1min":0.89300000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.891},"last":1.169},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.302,"15min":1.3149999999999999},"min":{"1min":0.66500000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.03}]},{"osd":4,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2170000000000001,"15min":1.2070000000000001},"min":{"1min":0.63400000000000001,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.006},{"interface":"front","average":{"1min":1.403,"5min":1.337,"15min":1.3120000000000001},"min":{"1min":0.85899999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.7690000000000001,"5min":2.7690000000000001,"15min":2.7690000000000001},"last":1.423}]},{"osd":6,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.323,"15min":1.3089999999999999},"min":{"1min":0.66700000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.2770000000000001},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.393,"15min":1.3799999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":1.081}]},{"osd":7,"last update":"Tue Dec 19 09:02:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.4770000000000001,"15min":1.4470000000000001},"min":{"1min":0.96899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":1.323},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.4810000000000001,"15min":1.476},"min":{"1min":0.92700000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.3440000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575754,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311712,"kb_used_data":2136,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060128,"statfs":{"total":96636764160,"available":96317571072,"internally_reserved":0,"allocated":2187264,"data_stored":1157277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.622},"max":{"1min":1.5669999999999999,"5min":1.6659999999999999,"15min":1.7270000000000001},"last":5.149},{"interface":"front","average":{"1min":1.302,"5min":1.367,"15min":1.3480000000000001},"min":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.78300000000000003},"max":{"1min":1.6180000000000001,"5min":1.7270000000000001,"15min":1.768},"last":4.6429999999999998}]},{"osd":1,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.21,"15min":1.202},"min":{"1min":0.85799999999999998,"5min":0.80700000000000005,"15min":0.54500000000000004},"max":{"1min":1.5389999999999999,"5min":1.7869999999999999,"15min":1.8959999999999999},"last":4.5890000000000004},{"interface":"front","average":{"1min":1.159,"5min":1.171,"15min":1.149},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.59999999999999998},"max":{"1min":1.389,"5min":1.712,"15min":1.712},"last":2.6440000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.1890000000000001,"15min":1.173},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.66100000000000003},"max":{"1min":1.5409999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":5.0919999999999996},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6200000000000001,"5min":1.7290000000000001,"15min":1.8149999999999999},"last":4.5389999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.3009999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.69199999999999995},"max":{"1min":1.575,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":5.5469999999999997},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.377,"15min":1.3779999999999999},"min":{"1min":0.95399999999999996,"5min":0.90200000000000002,"15min":0.77800000000000002},"max":{"1min":1.673,"5min":1.7589999999999999,"15min":1.863},"last":5.2069999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.258,"15min":1.228},"min":{"1min":0.92000000000000004,"5min":0.875,"15min":0.623},"max":{"1min":1.5660000000000001,"5min":1.796,"15min":1.962},"last":11.943},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.82799999999999996,"5min":0.69899999999999995,"15min":0.56899999999999995},"max":{"1min":1.6879999999999999,"5min":1.823,"15min":1.823},"last":0.92700000000000005}]},{"osd":6,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.266,"15min":1.238},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.59399999999999997},"max":{"1min":1.899,"5min":1.909,"15min":1.909},"last":5.298},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.419,"15min":1.405},"min":{"1min":1.0700000000000001,"5min":1.034,"15min":0.64200000000000002},"max":{"1min":1.677,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":4.6980000000000004}]},{"osd":7,"last update":"Tue Dec 19 09:02:29 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.337,"15min":1.3340000000000001},"min":{"1min":0.95199999999999996,"5min":0.86499999999999999,"15min":0.71799999999999997},"max":{"1min":1.851,"5min":1.851,"15min":1.919},"last":0.625},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3999999999999999,"15min":1.393},"min":{"1min":1.0860000000000001,"5min":0.84099999999999997,"15min":0.65100000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":5.2519999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311708,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94060132,"statfs":{"total":96636764160,"available":96317575168,"internally_reserved":0,"allocated":2183168,"data_stored":1156329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.84199999999999997,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1399999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.97199999999999998,"15min":0.98099999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4470000000000001,"5min":1.476,"15min":1.6180000000000001},"last":0.89900000000000002}]},{"osd":1,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94299999999999995,"15min":0.94599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.345,"5min":1.5169999999999999,"15min":2.456},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0900000000000001,"15min":1.087},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69699999999999995},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":1.1559999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0700000000000001,"15min":1.0900000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.4550000000000001,"5min":1.534,"15min":2.0150000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.194,"5min":1.196,"15min":1.1930000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2949999999999999,"15min":1.288},"min":{"1min":1.002,"5min":1.002,"15min":0.94099999999999995},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.266,"15min":1.252},"min":{"1min":0.97799999999999998,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.6950000000000001,"5min":1.742,"15min":2.6520000000000001},"last":0.82299999999999995}]},{"osd":5,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3300000000000001,"15min":1.3},"min":{"1min":1.0669999999999999,"5min":0.97099999999999997,"15min":0.85399999999999998},"max":{"1min":1.887,"5min":1.887,"15min":2.7519999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.387,"15min":1.391},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.8700000000000001,"5min":1.8759999999999999,"15min":2.5710000000000002},"last":1.226}]},{"osd":6,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.421,"15min":1.421},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.429,"15min":1.405},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.88800000000000001},"max":{"1min":1.8089999999999999,"5min":1.925,"15min":2.2330000000000001},"last":1.319}]},{"osd":7,"last update":"Tue Dec 19 09:03:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.498,"15min":1.488},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.8720000000000001,"5min":2.0379999999999998,"15min":2.202},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.4419999999999999,"15min":1.4410000000000001},"min":{"1min":1.115,"5min":0.97099999999999997,"15min":0.97099999999999997},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.8620000000000001},"last":1.131}]}]},{"osd":2,"up_from":13,"seq":55834575754,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307612,"kb_used_data":2132,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064228,"statfs":{"total":96636764160,"available":96321769472,"internally_reserved":0,"allocated":2183168,"data_stored":1155853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.94199999999999995,"15min":0.93300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.085,"15min":1.0760000000000001},"min":{"1min":0.81799999999999995,"5min":0.81100000000000005,"15min":0.60799999999999998},"max":{"1min":1.367,"5min":1.4650000000000001,"15min":1.744},"last":0.97899999999999998}]},{"osd":1,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.70599999999999996,"5min":0.66800000000000004,"15min":0.56399999999999995},"max":{"1min":1.3939999999999999,"5min":1.4119999999999999,"15min":1.948},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.42699999999999999},"max":{"1min":1.538,"5min":1.6200000000000001,"15min":1.6499999999999999},"last":1.286}]},{"osd":3,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.628},"max":{"1min":1.1579999999999999,"5min":1.6140000000000001,"15min":1.8009999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1639999999999999,"15min":1.1539999999999999},"min":{"1min":0.71499999999999997,"5min":0.63600000000000001,"15min":0.57899999999999996},"max":{"1min":1.4199999999999999,"5min":1.49,"15min":2.1760000000000002},"last":0.88400000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.327,"15min":1.3320000000000001},"min":{"1min":1.012,"5min":1.0109999999999999,"15min":0.84899999999999998},"max":{"1min":1.6040000000000001,"5min":1.6719999999999999,"15min":2.0590000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.325,"15min":1.323},"min":{"1min":0.93200000000000005,"5min":0.84099999999999997,"15min":0.82899999999999996},"max":{"1min":1.7010000000000001,"5min":1.714,"15min":3.1030000000000002},"last":1.1499999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3720000000000001,"15min":1.363},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.571,"5min":1.7270000000000001,"15min":1.9099999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.286,"15min":1.2949999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.548,"5min":2.0830000000000002,"15min":2.2959999999999998},"last":1.4810000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.3440000000000001,"15min":1.327},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.72599999999999998},"max":{"1min":1.524,"5min":1.9199999999999999,"15min":2.3940000000000001},"last":1.302},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.415,"15min":1.397},"min":{"1min":1.016,"5min":1.016,"15min":0.89900000000000002},"max":{"1min":1.6930000000000001,"5min":1.8520000000000001,"15min":1.869},"last":1.2649999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.0900000000000001,"5min":1.03,"15min":0.60199999999999998},"max":{"1min":1.6839999999999999,"5min":1.8340000000000001,"15min":2.1680000000000001},"last":1.45},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4259999999999999,"15min":1.4059999999999999},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.82199999999999995},"max":{"1min":1.722,"5min":2.0859999999999999,"15min":2.7109999999999999},"last":1.3819999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575755,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307608,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064232,"statfs":{"total":96636764160,"available":96321773568,"internally_reserved":0,"allocated":2179072,"data_stored":1155811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.87,"15min":0.85199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.339,"5min":1.54,"15min":1.54},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.96899999999999997,"15min":0.97199999999999998},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.42099999999999999},"max":{"1min":1.337,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.097}]},{"osd":2,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.92800000000000005,"15min":0.93000000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.36499999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.113,"5min":1.1160000000000001,"15min":1.081},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.503},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.159}]},{"osd":3,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.1619999999999999,"15min":1.167},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.7030000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.04},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.698,"5min":1.706,"15min":1.706},"last":0.88600000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.2230000000000001,"15min":1.2190000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5900000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.238,"5min":1.327,"15min":1.323},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.7689999999999999,"5min":1.958,"15min":2.024},"last":1.2869999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3080000000000001,"15min":1.3169999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.78400000000000003},"max":{"1min":1.7150000000000001,"5min":2.0569999999999999,"15min":2.0699999999999998},"last":1.333},{"interface":"front","average":{"1min":1.252,"5min":1.292,"15min":1.302},"min":{"1min":0.93700000000000006,"5min":0.90200000000000002,"15min":0.72199999999999998},"max":{"1min":1.534,"5min":1.96,"15min":2.052},"last":1.3620000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3740000000000001,"15min":1.3819999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8280000000000001,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.375,"15min":1.377},"min":{"1min":0.93000000000000005,"5min":0.85599999999999998,"15min":0.81499999999999995},"max":{"1min":1.8779999999999999,"5min":1.881,"15min":2.0800000000000001},"last":1.2450000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:07 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.411,"15min":1.419},"min":{"1min":0.97799999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":1.9750000000000001,"5min":2.117,"15min":2.117},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.377,"15min":1.3919999999999999},"min":{"1min":1.0920000000000001,"5min":0.95899999999999996,"15min":0.78800000000000003},"max":{"1min":1.6719999999999999,"5min":2.0699999999999998,"15min":2.0699999999999998},"last":1.198}]}]},{"osd":0,"up_from":13,"seq":55834575754,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312300,"kb_used_data":2724,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059540,"statfs":{"total":96636764160,"available":96316968960,"internally_reserved":0,"allocated":2789376,"data_stored":1747312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83699999999999997,"15min":0.84399999999999997},"min":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.41299999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.94699999999999995,"15min":0.96599999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.48599999999999999},"max":{"1min":1.3660000000000001,"5min":1.464,"15min":1.488},"last":0.97299999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.997,"15min":0.99199999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.51900000000000002},"max":{"1min":1.4239999999999999,"5min":1.431,"15min":1.625},"last":0.628},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.056,"15min":1.0569999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3899999999999999,"5min":1.4199999999999999,"15min":1.48},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.028,"15min":1.0249999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.696,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.157,"15min":1.145},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.57999999999999996},"max":{"1min":1.603,"5min":1.603,"15min":1.698},"last":0.79900000000000004}]},{"osd":4,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3660000000000001,"15min":1.3779999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.747},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.161},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.349,"15min":1.3520000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3779999999999999,"15min":1.3799999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.244},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.405,"15min":1.417},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.7310000000000001,"5min":1.7470000000000001,"15min":1.7589999999999999},"last":1.446}]},{"osd":6,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.288,"15min":1.2929999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.3700000000000001,"15min":1.3700000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.843,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.1080000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.361,"15min":1.361},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.4079999999999999,"15min":1.4039999999999999},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.9410000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.077}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:03:22.984 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T09:03:22.985 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T09:03:23.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.195+0000 7f53b80ae700 1 Processor -- start 2023-12-19T09:03:23.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.195+0000 7f53b80ae700 1 -- start start 2023-12-19T09:03:23.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b80ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53b015c520 0x7f53b00a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b80ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b015c8f0 0x7f53b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b80ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b00c6ab0 con 0x7f53b006ace0 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b80ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b006b520 con 0x7f53b015c520 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b80ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00cb910 con 0x7f53b015c8f0 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53b015c520 0x7f53b00a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53b015c520 0x7f53b00a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46996/0 (socket says 172.21.15.111:46996) 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 -- 172.21.15.111:0/2708891695 learned_addr learned my addr 172.21.15.111:0/2708891695 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5649700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b015c8f0 0x7f53b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 -- 172.21.15.111:0/2708891695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b015c8f0 msgr2=0x7f53b006a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b015c8f0 0x7f53b006a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 -- 172.21.15.111:0/2708891695 >> v1:172.21.15.143:6789/0 conn(0x7f53b006ace0 legacy=0x7f53b00ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.199+0000 7f53b5e4a700 1 -- 172.21.15.111:0/2708891695 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b006b240 con 0x7f53b015c520 2023-12-19T09:03:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b5e4a700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c520 0x7f53b00a8850 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0099b0 tx=0x7f53ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bb49b2d6daef1e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b4e48700 1 -- 172.21.15.111:0/2708891695 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac009ea0 con 0x7f53b015c520 2023-12-19T09:03:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b4e48700 1 -- 172.21.15.111:0/2708891695 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac00b8f0 con 0x7f53b015c520 2023-12-19T09:03:23.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b4e48700 1 -- 172.21.15.111:0/2708891695 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac00bc90 con 0x7f53b015c520 2023-12-19T09:03:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- 172.21.15.111:0/2708891695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c520 msgr2=0x7f53b00a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c520 0x7f53b00a8850 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0099b0 tx=0x7f53ac009250 comp rx=0 tx=0).stop 2023-12-19T09:03:23.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- 172.21.15.111:0/2708891695 shutdown_connections 2023-12-19T09:03:23.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b015c8f0 0x7f53b006a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 --2- 172.21.15.111:0/2708891695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c520 0x7f53b00a8850 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- 172.21.15.111:0/2708891695 >> 172.21.15.111:0/2708891695 conn(0x7f53b0067bb0 msgr2=0x7f53b00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:23.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- 172.21.15.111:0/2708891695 shutdown_connections 2023-12-19T09:03:23.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- 172.21.15.111:0/2708891695 wait complete. 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 Processor -- start 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.203+0000 7f53b80ae700 1 -- start start 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b006ace0 0x7f53b00c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b015c520 0x7f53b014a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00ad3c0 con 0x7f53b015c8f0 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00c3420 con 0x7f53b015c520 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00acdd0 con 0x7f53b006ace0 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47006/0 (socket says 172.21.15.111:47006) 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 -- 172.21.15.111:0/4152114285 learned_addr learned my addr 172.21.15.111:0/4152114285 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b5e4a700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b006ace0 0x7f53b00c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b5649700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b015c520 0x7f53b014a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 -- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b006ace0 msgr2=0x7f53b00c2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b006ace0 0x7f53b00c2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 -- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b015c520 msgr2=0x7f53b014a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b015c520 0x7f53b014a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 -- 172.21.15.111:0/4152114285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0094f0 con 0x7f53b015c8f0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b664b700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f53a800c870 tx=0x7f53a8001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8012630 con 0x7f53b015c8f0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a80127b0 con 0x7f53b015c8f0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8012b70 con 0x7f53b015c8f0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b00aa9a0 con 0x7f53b015c8f0 2023-12-19T09:03:23.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.207+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b00ad050 con 0x7f53b015c8f0 2023-12-19T09:03:23.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.211+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b00ad570 con 0x7f53b015c8f0 2023-12-19T09:03:23.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.211+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f53a8003030 con 0x7f53b015c8f0 2023-12-19T09:03:23.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.215+0000 7f53a5ffb700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 0x7f5394062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.215+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f53a807cb30 con 0x7f53b015c8f0 2023-12-19T09:03:23.222 INFO:tasks.ceph.mds.b.smithi143.stderr:2023-12-19T09:03:23.214+0000 7f6a52321700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.215+0000 7f53b5e4a700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 0x7f5394062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.215+0000 7f53b5e4a700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 0x7f5394062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0099b0 tx=0x7f53ac017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.219+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f53b00ad570 con 0x7f53b015c8f0 2023-12-19T09:03:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.443+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f53b00ad570 con 0x7f53b015c8f0 2023-12-19T09:03:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.455+0000 7f53a5ffb700 1 -- 172.21.15.111:0/4152114285 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7f53b00ad570 con 0x7f53b015c8f0 2023-12-19T09:03:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 msgr2=0x7f5394062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 0x7f5394062b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0099b0 tx=0x7f53ac017000 comp rx=0 tx=0).stop 2023-12-19T09:03:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 msgr2=0x7f53b00c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f53a800c870 tx=0x7f53a8001670 comp rx=0 tx=0).stop 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 shutdown_connections 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5394060670 0x7f5394062b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53b015c520 0x7f53b014a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53b015c8f0 0x7f53b00c3750 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 --2- 172.21.15.111:0/4152114285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53b006ace0 0x7f53b00c2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 >> 172.21.15.111:0/4152114285 conn(0x7f53b0067bb0 msgr2=0x7f53b00ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 shutdown_connections 2023-12-19T09:03:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.459+0000 7f53b80ae700 1 -- 172.21.15.111:0/4152114285 wait complete. 2023-12-19T09:03:23.466 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T09:03:23.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:03:23.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.643+0000 7f37106cd700 1 Processor -- start 2023-12-19T09:03:23.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.643+0000 7f37106cd700 1 -- start start 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f37106cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140710 0x7f37080c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f37106cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3708140ae0 0x7f37080680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f37106cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370806e150 con 0x7f37080c75e0 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f37106cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370806d540 con 0x7f3708140ae0 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f37106cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370806d820 con 0x7f3708140710 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370ec6a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f37080c75e0 0x7f370806a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60528/0 (socket says 172.21.15.111:60528) 2023-12-19T09:03:23.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370ec6a700 1 -- 172.21.15.111:0/269350081 learned_addr learned my addr 172.21.15.111:0/269350081 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 --2- 172.21.15.111:0/269350081 >> v2:172.21.15.143:3300/0 conn(0x7f3708140ae0 0x7f37080680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370e469700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140710 0x7f37080c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370d467700 1 -- 172.21.15.111:0/269350081 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669904608 0 0) 0x7f370806e150 con 0x7f37080c75e0 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370d467700 1 -- 172.21.15.111:0/269350081 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f4003570 con 0x7f37080c75e0 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 -- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140710 msgr2=0x7f37080c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:23.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140710 0x7f37080c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 -- 172.21.15.111:0/269350081 >> v1:172.21.15.111:6789/0 conn(0x7f37080c75e0 legacy=0x7f370806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 -- 172.21.15.111:0/269350081 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370806db00 con 0x7f3708140ae0 2023-12-19T09:03:23.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370dc68700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140ae0 0x7f37080680f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f36f80098e0 tx=0x7f36f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e79876caf6dff760 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370d467700 1 -- 172.21.15.111:0/269350081 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8011630 con 0x7f3708140ae0 2023-12-19T09:03:23.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370d467700 1 -- 172.21.15.111:0/269350081 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8011790 con 0x7f3708140ae0 2023-12-19T09:03:23.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.647+0000 7f370d467700 1 -- 172.21.15.111:0/269350081 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8011b30 con 0x7f3708140ae0 2023-12-19T09:03:23.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 -- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140ae0 msgr2=0x7f37080680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140ae0 0x7f37080680f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f36f80098e0 tx=0x7f36f80091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 -- 172.21.15.111:0/269350081 shutdown_connections 2023-12-19T09:03:23.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140ae0 0x7f37080680f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 --2- 172.21.15.111:0/269350081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140710 0x7f37080c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 -- 172.21.15.111:0/269350081 >> 172.21.15.111:0/269350081 conn(0x7f37080c8850 msgr2=0x7f37080c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:23.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 -- 172.21.15.111:0/269350081 shutdown_connections 2023-12-19T09:03:23.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.651+0000 7f37106cd700 1 -- 172.21.15.111:0/269350081 wait complete. 2023-12-19T09:03:23.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 Processor -- start 2023-12-19T09:03:23.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 -- start start 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 0x7f3708159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140ae0 0x7f3708159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3708158730 con 0x7f37080c75e0 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3708165b00 con 0x7f3708140710 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f37106cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3708158bc0 con 0x7f3708140ae0 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f370e469700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f370e469700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47016/0 (socket says 172.21.15.111:47016) 2023-12-19T09:03:23.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f370e469700 1 -- 172.21.15.111:0/4111130047 learned_addr learned my addr 172.21.15.111:0/4111130047 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f370dc68700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 0x7f3708159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.655+0000 7f370ec6a700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140ae0 0x7f3708159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 -- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140ae0 msgr2=0x7f3708159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140ae0 0x7f3708159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 -- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 msgr2=0x7f3708159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 0x7f3708159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 -- 172.21.15.111:0/4111130047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8009420 con 0x7f37080c75e0 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370dc68700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 0x7f3708159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f370e469700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f370400de60 tx=0x7f370400f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370400d2e0 con 0x7f37080c75e0 2023-12-19T09:03:23.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3704013040 con 0x7f37080c75e0 2023-12-19T09:03:23.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704010c20 con 0x7f37080c75e0 2023-12-19T09:03:23.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3708157b20 con 0x7f37080c75e0 2023-12-19T09:03:23.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37081580e0 con 0x7f37080c75e0 2023-12-19T09:03:23.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.659+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3708071120 con 0x7f37080c75e0 2023-12-19T09:03:23.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.663+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f3704018480 con 0x7f37080c75e0 2023-12-19T09:03:23.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.663+0000 7f36fe7fc700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 0x7f36ec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:23.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.663+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f3704081180 con 0x7f37080c75e0 2023-12-19T09:03:23.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.663+0000 7f370dc68700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 0x7f36ec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:23.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.663+0000 7f370dc68700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 0x7f36ec062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36f80091e0 tx=0x7f36f8017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:23.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.671+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f37040511f0 con 0x7f37080c75e0 2023-12-19T09:03:23.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.871+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37080642a0 con 0x7f37080c75e0 2023-12-19T09:03:23.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.871+0000 7f36fe7fc700 1 -- 172.21.15.111:0/4111130047 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7f3704054810 con 0x7f37080c75e0 2023-12-19T09:03:23.880 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:03:23.880 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:03:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 msgr2=0x7f36ec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 0x7f36ec062b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36f80091e0 tx=0x7f36f8017000 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 msgr2=0x7f37081655c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f370400de60 tx=0x7f370400f040 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 shutdown_connections 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f36ec060670 0x7f36ec062b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3708140710 0x7f3708159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3708140ae0 0x7f3708159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 --2- 172.21.15.111:0/4111130047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37080c75e0 0x7f37081655c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 >> 172.21.15.111:0/4111130047 conn(0x7f37080c8850 msgr2=0x7f37080c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 shutdown_connections 2023-12-19T09:03:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:23.875+0000 7f37106cd700 1 -- 172.21.15.111:0/4111130047 wait complete. 2023-12-19T09:03:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:03:24.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T09:03:17.684061+0000', 'modified': '2023-12-19T09:03:21.295488+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9396}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9396': {'gid': 9396, 'name': 'b', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6835/811953301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 811953301}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 811953301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T09:03:24.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T09:03:24.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T09:03:24.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 Processor -- start 2023-12-19T09:03:24.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 -- start start 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0c78155ae0 0x7f0c780670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c78155eb0 0x7f0c78068b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780af210 con 0x7f0c780c9120 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780ccb40 con 0x7f0c78155ae0 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7faf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780cc580 con 0x7f0c78155eb0 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d88c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0c78155ae0 0x7f0c780670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7e08d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0c780c9120 0x7f0c781597a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60538/0 (socket says 172.21.15.111:60538) 2023-12-19T09:03:24.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7e08d700 1 -- 172.21.15.111:0/3317480665 learned_addr learned my addr 172.21.15.111:0/3317480665 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:24.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/3317480665 >> v2:172.21.15.143:3300/0 conn(0x7f0c78155eb0 0x7f0c78068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7c88a700 1 -- 172.21.15.111:0/3317480665 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294420117 0 0) 0x7f0c780af210 con 0x7f0c780c9120 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d88c700 1 -- 172.21.15.111:0/3317480665 >> v2:172.21.15.143:3300/0 conn(0x7f0c78155eb0 msgr2=0x7f0c78068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3317480665 >> v2:172.21.15.143:3300/0 conn(0x7f0c78155eb0 0x7f0c78068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d88c700 1 -- 172.21.15.111:0/3317480665 >> v1:172.21.15.111:6789/0 conn(0x7f0c780c9120 legacy=0x7f0c781597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.471+0000 7f0c7d88c700 1 -- 172.21.15.111:0/3317480665 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c780cc860 con 0x7f0c78155ae0 2023-12-19T09:03:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3317480665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c78155ae0 0x7f0c780670e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0c78064910 tx=0x7f0c740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f100208b3ab45285 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7c88a700 1 -- 172.21.15.111:0/3317480665 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400b6b0 con 0x7f0c78155ae0 2023-12-19T09:03:24.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7c88a700 1 -- 172.21.15.111:0/3317480665 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7400b810 con 0x7f0c78155ae0 2023-12-19T09:03:24.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7c88a700 1 -- 172.21.15.111:0/3317480665 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400bbb0 con 0x7f0c78155ae0 2023-12-19T09:03:24.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 -- 172.21.15.111:0/3317480665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c78155ae0 msgr2=0x7f0c780670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/3317480665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c78155ae0 0x7f0c780670e0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0c78064910 tx=0x7f0c740091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 -- 172.21.15.111:0/3317480665 shutdown_connections 2023-12-19T09:03:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/3317480665 >> v2:172.21.15.143:3300/0 conn(0x7f0c78155eb0 0x7f0c78068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/3317480665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c78155ae0 0x7f0c780670e0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 -- 172.21.15.111:0/3317480665 >> 172.21.15.111:0/3317480665 conn(0x7f0c780cd740 msgr2=0x7f0c780c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 -- 172.21.15.111:0/3317480665 shutdown_connections 2023-12-19T09:03:24.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.475+0000 7f0c7faf0700 1 -- 172.21.15.111:0/3317480665 wait complete. 2023-12-19T09:03:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 Processor -- start 2023-12-19T09:03:24.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 -- start start 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 0x7f0c780c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 0x7f0c7813ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780c6bd0 con 0x7f0c780c5130 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7813e300 con 0x7f0c78155ae0 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7faf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780c55f0 con 0x7f0c780c9120 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7e08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7e08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47038/0 (socket says 172.21.15.111:47038) 2023-12-19T09:03:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1424711385 learned_addr learned my addr 172.21.15.111:0/1424711385 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 0x7f0c780c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.479+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 0x7f0c7813ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 msgr2=0x7f0c780c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 0x7f0c780c6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 msgr2=0x7f0c7813ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 0x7f0c7813ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1424711385 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74009420 con 0x7f0c780c5130 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 0x7f0c780c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 0x7f0c7813ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000cbf0 tx=0x7f0c700019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c6dffb700 1 -- 172.21.15.111:0/1424711385 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7000a9c0 con 0x7f0c780c5130 2023-12-19T09:03:24.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c6dffb700 1 -- 172.21.15.111:0/1424711385 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7000ab20 con 0x7f0c780c5130 2023-12-19T09:03:24.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c6dffb700 1 -- 172.21.15.111:0/1424711385 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c700070f0 con 0x7f0c780c5130 2023-12-19T09:03:24.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c780c58d0 con 0x7f0c780c5130 2023-12-19T09:03:24.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.483+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c780c5e90 con 0x7f0c780c5130 2023-12-19T09:03:24.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.487+0000 7f0c6dffb700 1 -- 172.21.15.111:0/1424711385 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7f0c70007520 con 0x7f0c780c5130 2023-12-19T09:03:24.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.487+0000 7f0c6dffb700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 0x7f0c5c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.487+0000 7f0c6dffb700 1 -- 172.21.15.111:0/1424711385 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0c7007d180 con 0x7f0c780c5130 2023-12-19T09:03:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.487+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 0x7f0c5c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.487+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 0x7f0c5c062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c740093f0 tx=0x7f0c74017000 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 Processor -- start 2023-12-19T09:03:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 -- start start 2023-12-19T09:03:24.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4ae0 0x2a36500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a36f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x295ff30 con 0x27b4710 2023-12-19T09:03:24.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x295bed0 con 0x27ff790 2023-12-19T09:03:24.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x29220b0 con 0x27b4ae0 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47042/0 (socket says 172.21.15.111:47042) 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 -- 172.21.15.111:0/1434288450 learned_addr learned my addr 172.21.15.111:0/1434288450 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4ae0 0x2a36500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a36f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 -- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4ae0 msgr2=0x2a36500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4ae0 0x2a36500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 -- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 msgr2=0x2a36f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a36f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 -- 172.21.15.111:0/1434288450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x28009e0 con 0x27b4710 2023-12-19T09:03:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007a00 tx=0x7f0c68008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4535042122716220 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.491+0000 7f0c5affd700 1 -- 172.21.15.111:0/1434288450 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800f6d0 con 0x27b4710 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c5affd700 1 -- 172.21.15.111:0/1434288450 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800f830 con 0x27b4710 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c5affd700 1 -- 172.21.15.111:0/1434288450 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800fbf0 con 0x27b4710 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- 172.21.15.111:0/1434288450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 msgr2=0x2a36a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007a00 tx=0x7f0c68008040 comp rx=0 tx=0).stop 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- 172.21.15.111:0/1434288450 shutdown_connections 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a36f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4710 0x2a36a40 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1434288450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4ae0 0x2a36500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- 172.21.15.111:0/1434288450 >> 172.21.15.111:0/1434288450 conn(0x27b4eb0 msgr2=0x29ee930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- 172.21.15.111:0/1434288450 shutdown_connections 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- 172.21.15.111:0/1434288450 wait complete. 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 -- start start 2023-12-19T09:03:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 Processor -- start 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x29ee860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27b4ae0 0x2a17cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.495+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 0x2a181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2921d50 con 0x27ff790 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x28fa970 con 0x27b4ae0 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x2962400 con 0x27b4710 2023-12-19T09:03:24.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 10 client.0 _ll_register_callbacks cb 0x29a56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7d88c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x29ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 0x2a181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27b4ae0 0x2a17cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7d88c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x29ee860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:54828/0 (socket says 172.21.15.111:54828) 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7d88c700 1 -- 172.21.15.111:0/1727415018 learned_addr learned my addr 172.21.15.111:0/1727415018 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 msgr2=0x29ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x29ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27b4ae0 msgr2=0x2a17cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27b4ae0 0x2a17cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 -- 172.21.15.111:0/1727415018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6800e040 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 0x2a181f0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0c7007b1b0 tx=0x7f0c7007d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6fb4f035e2225d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c700542b0 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0c70057f80 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x28facc0 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c700562d0 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 10 client.9534 fetch_fsmap learned FSMap version 80 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x292a370 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x28facc0 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 80) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0c70051240 con 0x27ff790 2023-12-19T09:03:24.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 10 client.9534 fetch_fsmap finished waiting for FSMap version 80 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 10 client.9534 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 20 client.9534 populate_metadata read hostname 'smithi111' 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a3aa30 0x2a3ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 4 client.9534 mds_command: new command op to 9396 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 --> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x28fe3e0 con 0x2a3aa30 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.499+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a3aa30 0x2a3ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c53fff700 10 client.9534 ms_handle_connect on v2:172.21.15.143:6834/811953301 2023-12-19T09:03:24.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a3aa30 0x2a3ae00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69fc6cef4eb77f0b server_cookie=fbb6edcd5e94ba3c in_seq=0 out_seq=0 2023-12-19T09:03:24.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c53fff700 1 -- 172.21.15.111:0/1727415018 <== mds.0 v2:172.21.15.143:6834/811953301 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x28fe3e0 con 0x2a3aa30 2023-12-19T09:03:24.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c53fff700 10 client.9534 handle_command_reply: tid=0 2023-12-19T09:03:24.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 client.9534 shutdown 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 msgr2=0x2a181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 0x2a181f0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0c7007b1b0 tx=0x7f0c7007d040 comp rx=0 tx=0).stop 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 shutdown_connections 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a3aa30 0x2a3ae00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27b4ae0 0x2a17cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27ff790 0x2a181f0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c53fff700 0 client.9534 ms_handle_reset on v2:172.21.15.143:6834/811953301 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 --2- 172.21.15.111:0/1727415018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x29ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 >> 172.21.15.111:0/1727415018 conn(0x27b4eb0 msgr2=0x29ea6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 shutdown_connections 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.503+0000 7f0c81daf740 1 -- 172.21.15.111:0/1727415018 wait complete. 2023-12-19T09:03:24.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.507+0000 7f0c81daf740 20 client.9534 trim_cache size 0 max 16384 2023-12-19T09:03:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 Processor -- start 2023-12-19T09:03:24.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 -- start start 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a34d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x29ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2930610 con 0x27b4ae0 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2957b50 con 0x27ff790 2023-12-19T09:03:24.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x28fcb80 con 0x27b4710 2023-12-19T09:03:24.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47072/0 (socket says 172.21.15.111:47072) 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 -- 172.21.15.111:0/4093853938 learned_addr learned my addr 172.21.15.111:0/4093853938 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a34d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x29ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 -- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 msgr2=0x2a34d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a34d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 -- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 msgr2=0x29ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x29ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 -- 172.21.15.111:0/4093853938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2910d80 con 0x27b4ae0 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a34d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x29ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0c68002000 tx=0x7f0c6801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6de9d1cc6c310bf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.527+0000 7f0c5b7fe700 1 -- 172.21.15.111:0/4093853938 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68026070 con 0x27b4ae0 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c5b7fe700 1 -- 172.21.15.111:0/4093853938 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68000bc0 con 0x27b4ae0 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c5b7fe700 1 -- 172.21.15.111:0/4093853938 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800f6d0 con 0x27b4ae0 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- 172.21.15.111:0/4093853938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 msgr2=0x2a35290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0c68002000 tx=0x7f0c6801b000 comp rx=0 tx=0).stop 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- 172.21.15.111:0/4093853938 shutdown_connections 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x29ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a35290 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 --2- 172.21.15.111:0/4093853938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a34d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- 172.21.15.111:0/4093853938 >> 172.21.15.111:0/4093853938 conn(0x27b4eb0 msgr2=0x2a16ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- 172.21.15.111:0/4093853938 shutdown_connections 2023-12-19T09:03:24.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- 172.21.15.111:0/4093853938 wait complete. 2023-12-19T09:03:24.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 -- start start 2023-12-19T09:03:24.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.531+0000 7f0c81daf740 1 Processor -- start 2023-12-19T09:03:24.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a8ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a46a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2925400 con 0x27b4ae0 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2834930 con 0x27ff790 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x2907ec0 con 0x27b4710 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 10 client.0 _ll_register_callbacks cb 0x27809d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47078/0 (socket says 172.21.15.111:47078) 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 -- 172.21.15.111:0/3238009672 learned_addr learned my addr 172.21.15.111:0/3238009672 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a8ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a46a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 -- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 msgr2=0x2a8ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a8ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 -- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 msgr2=0x2a46a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a46a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 -- 172.21.15.111:0/3238009672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x28009e0 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7e08d700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a46a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a8ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c7d08b700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007710 tx=0x7f0c68023050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c318f8a02ec67e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c53fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68026040 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c6800f830 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.535+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f0c6800f670 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c68007970 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 10 client.9540 fetch_fsmap learned FSMap version 80 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) v3 -- 0x2905130 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800f830 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 80) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f0c6801b070 con 0x27b4ae0 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 10 client.9540 fetch_fsmap finished waiting for FSMap version 80 2023-12-19T09:03:24.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 10 client.9540 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T09:03:24.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 20 client.9540 populate_metadata read hostname 'smithi111' 2023-12-19T09:03:24.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a47400 0x2a477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 4 client.9540 mds_command: new command op to 9396 tid=0[{"prefix": "status"}] 2023-12-19T09:03:24.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 --> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] -- command(tid 0: {"prefix": "status"}) v1 -- 0x2930410 con 0x2a47400 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a47400 0x2a477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c7d88c700 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a47400 0x2a477d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0e282668a37ee5a server_cookie=25e0c1a9f612a34b in_seq=0 out_seq=0 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 10 client.9540 ms_handle_connect on v2:172.21.15.143:6834/811953301 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 1 -- 172.21.15.111:0/3238009672 <== mds.0 v2:172.21.15.143:6834/811953301 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x2930410 con 0x2a47400 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 10 client.9540 handle_command_reply: tid=0 2023-12-19T09:03:24.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 client.9540 shutdown 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 msgr2=0x2a904b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007710 tx=0x7f0c68023050 comp rx=0 tx=0).stop 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 shutdown_connections 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:6834/811953301,v1:172.21.15.143:6835/811953301] conn(0x2a47400 0x2a477d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x27ff790 0x2a46a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x27b4ae0 0x2a904b0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c53fff700 0 client.9540 ms_handle_reset on v2:172.21.15.143:6834/811953301 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 --2- 172.21.15.111:0/3238009672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x27b4710 0x2a8ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.539+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 >> 172.21.15.111:0/3238009672 conn(0x27b4eb0 msgr2=0x29e8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 shutdown_connections 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c81daf740 1 -- 172.21.15.111:0/3238009672 wait complete. 2023-12-19T09:03:24.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c81daf740 20 client.9540 trim_cache size 0 max 16384 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 9396, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2709366170000003, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 80, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 175, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 175, 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.208637037000001 2023-12-19T09:03:24.550 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:03:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 msgr2=0x7f0c5c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 0x7f0c5c062b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c740093f0 tx=0x7f0c74017000 comp rx=0 tx=0).stop 2023-12-19T09:03:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 msgr2=0x7f0c780b0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000cbf0 tx=0x7f0c700019b0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 shutdown_connections 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f0c5c060670 0x7f0c5c062b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c78155ae0 0x7f0c7813ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780c5130 0x7f0c780b0b80 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 --2- 172.21.15.111:0/1424711385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c780c9120 0x7f0c780c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 >> 172.21.15.111:0/1424711385 conn(0x7f0c780cd740 msgr2=0x7f0c780c28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 shutdown_connections 2023-12-19T09:03:24.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.543+0000 7f0c7faf0700 1 -- 172.21.15.111:0/1424711385 wait complete. 2023-12-19T09:03:24.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T09:03:24.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.727+0000 7fd3bbeae700 1 Processor -- start 2023-12-19T09:03:24.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.727+0000 7fd3bbeae700 1 -- start start 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3bbeae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3bbeae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3b40c15b0 0x7fd3b40aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3bbeae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40c7190 con 0x7fd3b406d690 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3bbeae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40c1b50 con 0x7fd3b40c15b0 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3bbeae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40c1e30 con 0x7fd3b415d320 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3b40c15b0 0x7fd3b40aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3b40c15b0 0x7fd3b40aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47080/0 (socket says 172.21.15.111:47080) 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 -- 172.21.15.111:0/2545109941 learned_addr learned my addr 172.21.15.111:0/2545109941 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9c4a700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 -- 172.21.15.111:0/2545109941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 msgr2=0x7fd3b415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b8c48700 1 -- 172.21.15.111:0/2545109941 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885176663 0 0) 0x7fd3b40c7190 con 0x7fd3b406d690 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 -- 172.21.15.111:0/2545109941 >> v1:172.21.15.143:6789/0 conn(0x7fd3b406d690 legacy=0x7fd3b40680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 -- 172.21.15.111:0/2545109941 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40c6e20 con 0x7fd3b40c15b0 2023-12-19T09:03:24.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b9449700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b40aeba0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40099b0 tx=0x7fd3a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1e3da0f847951c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b8c48700 1 -- 172.21.15.111:0/2545109941 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4009ea0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b8c48700 1 -- 172.21.15.111:0/2545109941 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400b860 con 0x7fd3b40c15b0 2023-12-19T09:03:24.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.731+0000 7fd3b8c48700 1 -- 172.21.15.111:0/2545109941 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400bc00 con 0x7fd3b40c15b0 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- 172.21.15.111:0/2545109941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 msgr2=0x7fd3b40aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b40aeba0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40099b0 tx=0x7fd3a4009250 comp rx=0 tx=0).stop 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- 172.21.15.111:0/2545109941 shutdown_connections 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b40aeba0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/2545109941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- 172.21.15.111:0/2545109941 >> 172.21.15.111:0/2545109941 conn(0x7fd3b40c8850 msgr2=0x7fd3b40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- 172.21.15.111:0/2545109941 shutdown_connections 2023-12-19T09:03:24.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- 172.21.15.111:0/2545109941 wait complete. 2023-12-19T09:03:24.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 Processor -- start 2023-12-19T09:03:24.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.735+0000 7fd3bbeae700 1 -- start start 2023-12-19T09:03:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 0x7fd3b4153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b416de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b415d9f0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b416e460 con 0x7fd3b406d690 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b416e740 con 0x7fd3b415d320 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47094/0 (socket says 172.21.15.111:47094) 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 -- 172.21.15.111:0/3606545168 learned_addr learned my addr 172.21.15.111:0/3606545168 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9c4a700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 0x7fd3b4153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3ba44b700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b416de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 -- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 msgr2=0x7fd3b416de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b416de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 -- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 msgr2=0x7fd3b4153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 0x7fd3b4153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 -- 172.21.15.111:0/3606545168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a40094f0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3ba44b700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b416de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9c4a700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 0x7fd3b4153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3b9449700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fd3b406a2b0 tx=0x7fd3a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400bbe0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a4011420 con 0x7fd3b40c15b0 2023-12-19T09:03:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40117a0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b40af450 con 0x7fd3b40c15b0 2023-12-19T09:03:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.739+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b40afa10 con 0x7fd3b40c15b0 2023-12-19T09:03:24.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.743+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4071120 con 0x7fd3b40c15b0 2023-12-19T09:03:24.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.743+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fd3a4011b90 con 0x7fd3b40c15b0 2023-12-19T09:03:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.743+0000 7fd3a9ffb700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 0x7fd398062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.743+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fd3a407e690 con 0x7fd3b40c15b0 2023-12-19T09:03:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.743+0000 7fd3b9c4a700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 0x7fd398062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.747+0000 7fd3b9c4a700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 0x7fd398062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40b1200 tx=0x7fd3b0008040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.751+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd3a404e770 con 0x7fd3b40c15b0 2023-12-19T09:03:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.955+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd3b40642a0 con 0x7fd3b40c15b0 2023-12-19T09:03:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.955+0000 7fd3a9ffb700 1 -- 172.21.15.111:0/3606545168 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd3a4051d90 con 0x7fd3b40c15b0 2023-12-19T09:03:24.962 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 msgr2=0x7fd398062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 0x7fd398062b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40b1200 tx=0x7fd3b0008040 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 msgr2=0x7fd3b416d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fd3b406a2b0 tx=0x7fd3a4017000 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 shutdown_connections 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fd398060670 0x7fd398062b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3b415d320 0x7fd3b416de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3b40c15b0 0x7fd3b416d8f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 --2- 172.21.15.111:0/3606545168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b406d690 0x7fd3b4153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 >> 172.21.15.111:0/3606545168 conn(0x7fd3b40c8850 msgr2=0x7fd3b415ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 shutdown_connections 2023-12-19T09:03:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:24.959+0000 7fd3bbeae700 1 -- 172.21.15.111:0/3606545168 wait complete. 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T09:03:24.987 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:03:24.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:24.988 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T09:03:25.007 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T09:03:25.008 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 37853sec preferred_lft 37853sec 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T09:03:25.009 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:03:25.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:25.010 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T09:03:25.086 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T09:03:25.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:25.089 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T09:03:25.156 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T09:03:25.156 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T09:03:25.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:25.157 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T09:03:25.157 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T09:03:25.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T09:03:25.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T09:03:25.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T09:03:25.158 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T09:03:25.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:25.301 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:03:25.320 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:03:25.320 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:03:25.321 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:03:25.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:03:25.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T09:03:25.403 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T09:03:25.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:03:25.404 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:25.469 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:03:25.469 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:03:25.469 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:25.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:03:25.610 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:03:25.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:03:25.688 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:03:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:03:25.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:25.693 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:03:25.754 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T09:03:25.754 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T09:03:25.802 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:03:25.836 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:03:25.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:03:25.916 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:03:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:03:25.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:25.920 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:03:25.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T09:03:25.931+0000 7fc06e9c1380 -1 init, newargv = 0x5585f803ee70 newargc=15 2023-12-19T09:03:25.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[59550]: starting ceph client 2023-12-19T09:03:25.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[59550]: starting fuse 2023-12-19T09:03:25.952 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T09:03:25.952 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T09:03:25.953 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:03:25.954 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T09:03:25.955 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T09:03:25.956 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:03:25.956 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T09:03:25.956 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T09:03:26.123 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.59550.asok 2023-12-19T09:03:26.302 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59550.asok 2023-12-19T09:03:26.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:26.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59550.asok status 2023-12-19T09:03:26.489 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:03:26.489 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "59550", 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 9549, 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T09:03:26.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 9549 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1447556652 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1447556652 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.9549 192.168.0.1:0/1447556652", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/1447556652", 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T09:03:26.491 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 80, 2023-12-19T09:03:26.492 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 175, 2023-12-19T09:03:26.492 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T09:03:26.492 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T09:03:26.492 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T09:03:26.492 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:03:26.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:26.505 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T09:03:26.505 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:26.524 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T09:03:26.524 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:26.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:26.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:03:26.604 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T09:03:26.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.811+0000 7fa140f85700 1 Processor -- start 2023-12-19T09:03:26.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.811+0000 7fa140f85700 1 -- start start 2023-12-19T09:03:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c06c7c0 0x7fa13c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa13c1407f0 0x7fa13c0b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c06d7d0 con 0x7fa13c140420 2023-12-19T09:03:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13c0ae280 con 0x7fa13c1407f0 2023-12-19T09:03:26.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0cb8f0 con 0x7fa13c06c7c0 2023-12-19T09:03:26.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13a59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa13c140420 0x7fa13c15b220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60542/0 (socket says 172.21.15.111:60542) 2023-12-19T09:03:26.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13a59c700 1 -- 172.21.15.111:0/798279181 learned_addr learned my addr 172.21.15.111:0/798279181 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:26.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 --2- 172.21.15.111:0/798279181 >> v2:172.21.15.143:3300/0 conn(0x7fa13c1407f0 0x7fa13c0b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa131d9b700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c06c7c0 0x7fa13c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa139d9b700 1 -- 172.21.15.111:0/798279181 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444749173 0 0) 0x7fa13c06d7d0 con 0x7fa13c140420 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa139d9b700 1 -- 172.21.15.111:0/798279181 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa128003570 con 0x7fa13c140420 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 -- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c06c7c0 msgr2=0x7fa13c0b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c06c7c0 0x7fa13c0b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 -- 172.21.15.111:0/798279181 >> v1:172.21.15.111:6789/0 conn(0x7fa13c140420 legacy=0x7fa13c15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:26.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 -- 172.21.15.111:0/798279181 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c06cc50 con 0x7fa13c1407f0 2023-12-19T09:03:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa13ad9d700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c0b1210 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa134009e60 tx=0x7fa134009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91dc9e3d0ec155cb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa139d9b700 1 -- 172.21.15.111:0/798279181 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13400e070 con 0x7fa13c1407f0 2023-12-19T09:03:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa139d9b700 1 -- 172.21.15.111:0/798279181 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13400bab0 con 0x7fa13c1407f0 2023-12-19T09:03:26.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa139d9b700 1 -- 172.21.15.111:0/798279181 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13400be70 con 0x7fa13c1407f0 2023-12-19T09:03:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 -- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 msgr2=0x7fa13c0b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.815+0000 7fa140f85700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c0b1210 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa134009e60 tx=0x7fa134009780 comp rx=0 tx=0).stop 2023-12-19T09:03:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 -- 172.21.15.111:0/798279181 shutdown_connections 2023-12-19T09:03:26.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c0b1210 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 --2- 172.21.15.111:0/798279181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c06c7c0 0x7fa13c0b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 -- 172.21.15.111:0/798279181 >> 172.21.15.111:0/798279181 conn(0x7fa13c067b60 msgr2=0x7fa13c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 -- 172.21.15.111:0/798279181 shutdown_connections 2023-12-19T09:03:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 -- 172.21.15.111:0/798279181 wait complete. 2023-12-19T09:03:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 Processor -- start 2023-12-19T09:03:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.819+0000 7fa140f85700 1 -- start start 2023-12-19T09:03:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c140420 0x7fa13c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c15e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c15bcf0 con 0x7fa13c06c7c0 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c15eba0 con 0x7fa13c1407f0 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c138d90 con 0x7fa13c140420 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47098/0 (socket says 172.21.15.111:47098) 2023-12-19T09:03:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 -- 172.21.15.111:0/536616657 learned_addr learned my addr 172.21.15.111:0/536616657 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13ad9d700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c15e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa131d9b700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c140420 0x7fa13c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 -- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c140420 msgr2=0x7fa13c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c140420 0x7fa13c15e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 -- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 msgr2=0x7fa13c15e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:26.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c15e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 -- 172.21.15.111:0/536616657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa134009a20 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa13a59c700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c7c0 tx=0x7fa12c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c005b30 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c005c90 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c139070 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c007340 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.823+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c139630 con 0x7fa13c06c7c0 2023-12-19T09:03:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.827+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c071120 con 0x7fa13c06c7c0 2023-12-19T09:03:26.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.827+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fa12c01b400 con 0x7fa13c06c7c0 2023-12-19T09:03:26.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.827+0000 7fa132ffd700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 0x7fa11c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:26.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.827+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fa12c07c710 con 0x7fa13c06c7c0 2023-12-19T09:03:26.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.827+0000 7fa131d9b700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 0x7fa11c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:26.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.831+0000 7fa131d9b700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 0x7fa11c062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa13c16c440 tx=0x7fa124008040 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:26.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:26.835+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa12c04c910 con 0x7fa13c06c7c0 2023-12-19T09:03:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.027+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa13c06cf30 con 0x7fa11c060670 2023-12-19T09:03:27.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.031+0000 7fa132ffd700 1 -- 172.21.15.111:0/536616657 <== mgr.7862 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa13c06cf30 con 0x7fa11c060670 2023-12-19T09:03:27.038 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:03:27.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:03:27.039 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T09:03:27.039 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:03:27.039 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:03:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 msgr2=0x7fa11c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 0x7fa11c062b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa13c16c440 tx=0x7fa124008040 comp rx=0 tx=0).stop 2023-12-19T09:03:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 msgr2=0x7fa13c138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c7c0 tx=0x7fa12c007e00 comp rx=0 tx=0).stop 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 shutdown_connections 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fa11c060670 0x7fa11c062b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa13c1407f0 0x7fa13c15e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa13c140420 0x7fa13c15e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 --2- 172.21.15.111:0/536616657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa13c06c7c0 0x7fa13c138350 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 >> 172.21.15.111:0/536616657 conn(0x7fa13c067b60 msgr2=0x7fa13c13f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 shutdown_connections 2023-12-19T09:03:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.035+0000 7fa140f85700 1 -- 172.21.15.111:0/536616657 wait complete. 2023-12-19T09:03:27.062 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T09:03:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.219+0000 7fbf79c03700 1 Processor -- start 2023-12-19T09:03:27.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.219+0000 7fbf79c03700 1 -- start start 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.219+0000 7fbf79c03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79c03700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf7406d1d0 0x7fbf740ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79c03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74156780 con 0x7fbf740acf30 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79c03700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf741532a0 con 0x7fbf7406d1d0 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79c03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74158f90 con 0x7fbf740ad300 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf73fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:54890/0 (socket says 172.21.15.111:54890) 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf73fff700 1 -- 172.21.15.111:0/635155755 learned_addr learned my addr 172.21.15.111:0/635155755 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 --2- 172.21.15.111:0/635155755 >> v2:172.21.15.111:3300/0 conn(0x7fbf7406d1d0 0x7fbf740ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 -- 172.21.15.111:0/635155755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 msgr2=0x7fbf741557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 -- 172.21.15.111:0/635155755 >> v1:172.21.15.143:6789/0 conn(0x7fbf740acf30 legacy=0x7fbf74159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf737fe700 1 -- 172.21.15.111:0/635155755 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590276596 0 0) 0x7fbf74156780 con 0x7fbf740acf30 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 -- 172.21.15.111:0/635155755 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7406d650 con 0x7fbf7406d1d0 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf73fff700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf79402700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406d1d0 0x7fbf740ac1e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009e60 tx=0x7fbf6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877fb16482b20b15 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf737fe700 1 -- 172.21.15.111:0/635155755 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00e070 con 0x7fbf7406d1d0 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf737fe700 1 -- 172.21.15.111:0/635155755 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00bb60 con 0x7fbf7406d1d0 2023-12-19T09:03:27.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.223+0000 7fbf737fe700 1 -- 172.21.15.111:0/635155755 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c012430 con 0x7fbf7406d1d0 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- 172.21.15.111:0/635155755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406d1d0 msgr2=0x7fbf740ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406d1d0 0x7fbf740ac1e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009e60 tx=0x7fbf6c009780 comp rx=0 tx=0).stop 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- 172.21.15.111:0/635155755 shutdown_connections 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf7406d1d0 0x7fbf740ac1e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 --2- 172.21.15.111:0/635155755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf740ad300 0x7fbf741557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- 172.21.15.111:0/635155755 >> 172.21.15.111:0/635155755 conn(0x7fbf74067b50 msgr2=0x7fbf740cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- 172.21.15.111:0/635155755 shutdown_connections 2023-12-19T09:03:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- 172.21.15.111:0/635155755 wait complete. 2023-12-19T09:03:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 Processor -- start 2023-12-19T09:03:27.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.227+0000 7fbf79c03700 1 -- start start 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406d1d0 0x7fbf74154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf740ad300 0x7fbf74150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74154b40 con 0x7fbf740acf30 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740af280 con 0x7fbf740ad300 2023-12-19T09:03:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740af4f0 con 0x7fbf7406d1d0 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47110/0 (socket says 172.21.15.111:47110) 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 -- 172.21.15.111:0/3414752871 learned_addr learned my addr 172.21.15.111:0/3414752871 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79402700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf740ad300 0x7fbf74150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf78c01700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406d1d0 0x7fbf74154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 -- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406d1d0 msgr2=0x7fbf74154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406d1d0 0x7fbf74154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 -- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf740ad300 msgr2=0x7fbf74150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf740ad300 0x7fbf74150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 -- 172.21.15.111:0/3414752871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf600094f0 con 0x7fbf740acf30 2023-12-19T09:03:27.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf73fff700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000ec80 tx=0x7fbf60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:27.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000c850 con 0x7fbf740acf30 2023-12-19T09:03:27.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000c9b0 con 0x7fbf740acf30 2023-12-19T09:03:27.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c009a20 con 0x7fbf740acf30 2023-12-19T09:03:27.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000cd30 con 0x7fbf740acf30 2023-12-19T09:03:27.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.231+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf741503f0 con 0x7fbf740acf30 2023-12-19T09:03:27.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.235+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7404be40 con 0x7fbf740acf30 2023-12-19T09:03:27.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.235+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 77) v1 ==== 80063+0+0 (secure 0 0 0) 0x7fbf60013680 con 0x7fbf740acf30 2023-12-19T09:03:27.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.239+0000 7fbf70ff9700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 0x7fbf58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.239+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fbf6007d080 con 0x7fbf740acf30 2023-12-19T09:03:27.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.239+0000 7fbf78c01700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 0x7fbf58062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.239+0000 7fbf78c01700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 0x7fbf58062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf68003a40 tx=0x7fbf68007ec0 comp rx=0 tx=0).ready entity=mgr.7862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:27.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.243+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbf600acbe0 con 0x7fbf740acf30 2023-12-19T09:03:27.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.451+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbf7406dae0 con 0x7fbf740acf30 2023-12-19T09:03:27.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.503+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 78) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fbf6004d530 con 0x7fbf740acf30 2023-12-19T09:03:27.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.507+0000 7fbf70ff9700 1 -- 172.21.15.111:0/3414752871 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v78) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbf60050bf0 con 0x7fbf740acf30 2023-12-19T09:03:27.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 msgr2=0x7fbf58062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 0x7fbf58062b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf68003a40 tx=0x7fbf68007ec0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 msgr2=0x7fbf7414c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000ec80 tx=0x7fbf60009250 comp rx=0 tx=0).stop 2023-12-19T09:03:27.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 shutdown_connections 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbf58060670 0x7fbf58062b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf740ad300 0x7fbf74150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf740acf30 0x7fbf7414c590 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 --2- 172.21.15.111:0/3414752871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf7406d1d0 0x7fbf74154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 >> 172.21.15.111:0/3414752871 conn(0x7fbf74067b50 msgr2=0x7fbf74148fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 shutdown_connections 2023-12-19T09:03:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.511+0000 7fbf79c03700 1 -- 172.21.15.111:0/3414752871 wait complete. 2023-12-19T09:03:27.571 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T09:03:27.704 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:27.698+0000 7f9e52bc3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:03:27.712 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:27.707+0000 7f5cf6182000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:03:27.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.743+0000 7fec78d23700 1 Processor -- start 2023-12-19T09:03:27.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.743+0000 7fec78d23700 1 -- start start 2023-12-19T09:03:27.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec78d23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec74158c90 0x7fec740caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec78d23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7406aeb0 0x7fec74159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec78d23700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7406c850 con 0x7fec741595a0 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec78d23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7406b470 con 0x7fec7406aeb0 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec78d23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7406b750 con 0x7fec74158c90 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7406aeb0 0x7fec74159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec73fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fec741595a0 0x7fec7415c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60548/0 (socket says 172.21.15.111:60548) 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec7406aeb0 0x7fec74159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47134/0 (socket says 172.21.15.111:47134) 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec73fff700 1 -- 172.21.15.111:0/2038691406 learned_addr learned my addr 172.21.15.111:0/2038691406 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:27.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec737fe700 1 --2- 172.21.15.111:0/2038691406 >> v2:172.21.15.143:3300/0 conn(0x7fec74158c90 0x7fec740caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 -- 172.21.15.111:0/2038691406 >> v2:172.21.15.143:3300/0 conn(0x7fec74158c90 msgr2=0x7fec740caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 --2- 172.21.15.111:0/2038691406 >> v2:172.21.15.143:3300/0 conn(0x7fec74158c90 0x7fec740caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 -- 172.21.15.111:0/2038691406 >> v1:172.21.15.111:6789/0 conn(0x7fec741595a0 legacy=0x7fec7415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 -- 172.21.15.111:0/2038691406 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7406ba30 con 0x7fec7406aeb0 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec72ffd700 1 --2- 172.21.15.111:0/2038691406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7406aeb0 0x7fec74159060 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fec740698f0 tx=0x7fec680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8aac059c1cd7939 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.747+0000 7fec727fc700 1 -- 172.21.15.111:0/2038691406 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68011630 con 0x7fec7406aeb0 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec727fc700 1 -- 172.21.15.111:0/2038691406 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68011790 con 0x7fec7406aeb0 2023-12-19T09:03:27.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec727fc700 1 -- 172.21.15.111:0/2038691406 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68011b30 con 0x7fec7406aeb0 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 -- 172.21.15.111:0/2038691406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7406aeb0 msgr2=0x7fec74159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 --2- 172.21.15.111:0/2038691406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7406aeb0 0x7fec74159060 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fec740698f0 tx=0x7fec680091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 -- 172.21.15.111:0/2038691406 shutdown_connections 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 --2- 172.21.15.111:0/2038691406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7406aeb0 0x7fec74159060 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 --2- 172.21.15.111:0/2038691406 >> v2:172.21.15.143:3300/0 conn(0x7fec74158c90 0x7fec740caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 -- 172.21.15.111:0/2038691406 >> 172.21.15.111:0/2038691406 conn(0x7fec74067bb0 msgr2=0x7fec740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:27.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 -- 172.21.15.111:0/2038691406 shutdown_connections 2023-12-19T09:03:27.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 -- 172.21.15.111:0/2038691406 wait complete. 2023-12-19T09:03:27.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.751+0000 7fec78d23700 1 Processor -- start 2023-12-19T09:03:27.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 -- start start 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec74158c90 0x7fec74153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 0x7fec740c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74155e80 con 0x7fec741595a0 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74155270 con 0x7fec7406aeb0 2023-12-19T09:03:27.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec78d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec74155550 con 0x7fec74158c90 2023-12-19T09:03:27.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.755+0000 7fec737fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:36094/0 (socket says 172.21.15.111:36094) 2023-12-19T09:03:27.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 -- 172.21.15.111:0/902070552 learned_addr learned my addr 172.21.15.111:0/902070552 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:27.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec72ffd700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec74158c90 0x7fec74153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec73fff700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 0x7fec740c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 -- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec74158c90 msgr2=0x7fec74153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec74158c90 0x7fec74153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 -- 172.21.15.111:0/902070552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 msgr2=0x7fec740c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 0x7fec740c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 -- 172.21.15.111:0/902070552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec68009420 con 0x7fec7406aeb0 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec73fff700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 0x7fec740c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:03:27.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec737fe700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fec6400ec80 tx=0x7fec640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400c9d0 con 0x7fec7406aeb0 2023-12-19T09:03:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6400cb30 con 0x7fec7406aeb0 2023-12-19T09:03:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74155830 con 0x7fec7406aeb0 2023-12-19T09:03:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64013470 con 0x7fec7406aeb0 2023-12-19T09:03:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74154f00 con 0x7fec7406aeb0 2023-12-19T09:03:27.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.759+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7404be40 con 0x7fec7406aeb0 2023-12-19T09:03:27.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.763+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 78) v1 ==== 80080+0+0 (secure 0 0 0) 0x7fec64013820 con 0x7fec7406aeb0 2023-12-19T09:03:27.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.763+0000 7fec5b7fe700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 0x7fec60062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:27.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.763+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7fec6407d0b0 con 0x7fec7406aeb0 2023-12-19T09:03:27.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.763+0000 7fec72ffd700 1 -- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 msgr2=0x7fec60062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:27.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.763+0000 7fec72ffd700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 0x7fec60062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:27.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.771+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec6404d190 con 0x7fec7406aeb0 2023-12-19T09:03:27.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.943+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fec7406bd10 con 0x7fec7406aeb0 2023-12-19T09:03:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.955+0000 7fec5b7fe700 1 -- 172.21.15.111:0/902070552 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fec640507b0 con 0x7fec7406aeb0 2023-12-19T09:03:27.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 msgr2=0x7fec60062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:27.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 0x7fec60062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 msgr2=0x7fec74156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:27.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fec6400ec80 tx=0x7fec640091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 shutdown_connections 2023-12-19T09:03:27.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fec600606c0 0x7fec60062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec741595a0 0x7fec740c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec74158c90 0x7fec74153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 --2- 172.21.15.111:0/902070552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7406aeb0 0x7fec74156310 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 >> 172.21.15.111:0/902070552 conn(0x7fec74067bb0 msgr2=0x7fec740b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 shutdown_connections 2023-12-19T09:03:27.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:27.959+0000 7fec78d23700 1 -- 172.21.15.111:0/902070552 wait complete. 2023-12-19T09:03:27.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T09:03:28.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.151+0000 7f071e42e700 1 Processor -- start 2023-12-19T09:03:28.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.151+0000 7f071e42e700 1 -- start start 2023-12-19T09:03:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f071e42e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0718158e20 0x7f07180ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f071e42e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f07180cb970 0x7f07180ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f071e42e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180cbdf0 con 0x7f07180ca2c0 2023-12-19T09:03:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f071e42e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180cc0d0 con 0x7f07180cb970 2023-12-19T09:03:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f071e42e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180cc3b0 con 0x7f0718158e20 2023-12-19T09:03:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f07180cb970 0x7f07180ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f07180cb970 0x7f07180ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47162/0 (socket says 172.21.15.111:47162) 2023-12-19T09:03:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 -- 172.21.15.111:0/3662349958 learned_addr learned my addr 172.21.15.111:0/3662349958 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f0717fff700 1 --2- 172.21.15.111:0/3662349958 >> v2:172.21.15.143:3300/0 conn(0x7f0718158e20 0x7f07180ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 -- 172.21.15.111:0/3662349958 >> v2:172.21.15.143:3300/0 conn(0x7f0718158e20 msgr2=0x7f07180ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 --2- 172.21.15.111:0/3662349958 >> v2:172.21.15.143:3300/0 conn(0x7f0718158e20 0x7f07180ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 -- 172.21.15.111:0/3662349958 >> v1:172.21.15.143:6789/0 conn(0x7f07180ca2c0 legacy=0x7f071806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 -- 172.21.15.111:0/3662349958 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180cc690 con 0x7f07180cb970 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f070ffff700 1 --2- 172.21.15.111:0/3662349958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07180cb970 0x7f07180ceb00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f07000099b0 tx=0x7f0700009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5a42d82c4f8fa0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f07177fe700 1 -- 172.21.15.111:0/3662349958 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070000b820 con 0x7f07180cb970 2023-12-19T09:03:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f07177fe700 1 -- 172.21.15.111:0/3662349958 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070000b980 con 0x7f07180cb970 2023-12-19T09:03:28.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.155+0000 7f07177fe700 1 -- 172.21.15.111:0/3662349958 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070000bd20 con 0x7f07180cb970 2023-12-19T09:03:28.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 -- 172.21.15.111:0/3662349958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07180cb970 msgr2=0x7f07180ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 --2- 172.21.15.111:0/3662349958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07180cb970 0x7f07180ceb00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f07000099b0 tx=0x7f0700009250 comp rx=0 tx=0).stop 2023-12-19T09:03:28.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 -- 172.21.15.111:0/3662349958 shutdown_connections 2023-12-19T09:03:28.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 --2- 172.21.15.111:0/3662349958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07180cb970 0x7f07180ceb00 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 --2- 172.21.15.111:0/3662349958 >> v2:172.21.15.143:3300/0 conn(0x7f0718158e20 0x7f07180ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.159+0000 7f071e42e700 1 -- 172.21.15.111:0/3662349958 >> 172.21.15.111:0/3662349958 conn(0x7f0718067bb0 msgr2=0x7f07180d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:28.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- 172.21.15.111:0/3662349958 shutdown_connections 2023-12-19T09:03:28.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- 172.21.15.111:0/3662349958 wait complete. 2023-12-19T09:03:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 Processor -- start 2023-12-19T09:03:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- start start 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 0x7f0718145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 0x7f071814b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071814f480 con 0x7f071814b030 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071814b9f0 con 0x7f07180ca2c0 2023-12-19T09:03:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071e42e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0718151820 con 0x7f0718158e20 2023-12-19T09:03:28.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f070ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f070ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:54930/0 (socket says 172.21.15.111:54930) 2023-12-19T09:03:28.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f070ffff700 1 -- 172.21.15.111:0/3637617972 learned_addr learned my addr 172.21.15.111:0/3637617972 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:28.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f071c9cb700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 0x7f071814b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.163+0000 7f0717fff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 0x7f0718145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 -- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 msgr2=0x7f0718145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 0x7f0718145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 -- 172.21.15.111:0/3637617972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 msgr2=0x7f071814b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 0x7f071814b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 -- 172.21.15.111:0/3637617972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000094f0 con 0x7f0718158e20 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f0717fff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 0x7f0718145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f071c9cb700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 0x7f071814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:03:28.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f070ffff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f07000094c0 tx=0x7f0700017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070000bed0 con 0x7f0718158e20 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070000bd00 con 0x7f0718158e20 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0718146390 con 0x7f0718158e20 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0700024660 con 0x7f0718158e20 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0718150c10 con 0x7f0718158e20 2023-12-19T09:03:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.167+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0718151990 con 0x7f0718158e20 2023-12-19T09:03:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.171+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 78) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f070001a420 con 0x7f0718158e20 2023-12-19T09:03:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.171+0000 7f0714ff9700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 0x7f06f8062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.171+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f070007e140 con 0x7f0718158e20 2023-12-19T09:03:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.171+0000 7f0717fff700 1 -- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 msgr2=0x7f06f8062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.171+0000 7f0717fff700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 0x7f06f8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:28.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.179+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f070004e220 con 0x7f0718158e20 2023-12-19T09:03:28.239 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:28.234+0000 7f9e52bc3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:03:28.271 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:28.263+0000 7f5cf6182000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:03:28.317 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:28.310+0000 7f9e52bc3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:03:28.348 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:28.343+0000 7f5cf6182000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:03:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.351+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f071806a7a0 con 0x7f0718158e20 2023-12-19T09:03:28.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.363+0000 7f0714ff9700 1 -- 172.21.15.111:0/3637617972 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0700051840 con 0x7f0718158e20 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 msgr2=0x7f06f8062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 0x7f06f8062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 msgr2=0x7f071814fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f07000094c0 tx=0x7f0700017000 comp rx=0 tx=0).stop 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 shutdown_connections 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f06f80606c0 0x7f06f8062b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071814b030 0x7f071814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0718158e20 0x7f071814fab0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 --2- 172.21.15.111:0/3637617972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07180ca2c0 0x7f0718145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 >> 172.21.15.111:0/3637617972 conn(0x7f0718067bb0 msgr2=0x7f0718157c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 shutdown_connections 2023-12-19T09:03:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.367+0000 7f071e42e700 1 -- 172.21.15.111:0/3637617972 wait complete. 2023-12-19T09:03:28.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:28.398 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T09:03:28.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:03:28.427 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-12-19T09:03:28.460 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:28.454+0000 7f9e52bc3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:28.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T09:03:28.504 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:28.499+0000 7f5cf6182000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.667+0000 7f5f7d08e700 1 Processor -- start 2023-12-19T09:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f7d08e700 1 -- start start 2023-12-19T09:03:28.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f7d08e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5f781569e0 0x7f5f78156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f7d08e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7806beb0 con 0x7f5f780aef60 2023-12-19T09:03:28.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f7d08e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7806cb80 con 0x7f5f780adf40 2023-12-19T09:03:28.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f7d08e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7806ce60 con 0x7f5f781569e0 2023-12-19T09:03:28.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f6ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5f780aef60 0x7f5f780b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60554/0 (socket says 172.21.15.111:60554) 2023-12-19T09:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f6ffff700 1 -- 172.21.15.111:0/3244431371 learned_addr learned my addr 172.21.15.111:0/3244431371 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.671+0000 7f5f77fff700 1 --2- 172.21.15.111:0/3244431371 >> v2:172.21.15.143:3300/0 conn(0x7f5f781569e0 0x7f5f78156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803678133 0 0) 0x7f5f7806beb0 con 0x7f5f780aef60 2023-12-19T09:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64003570 con 0x7f5f780aef60 2023-12-19T09:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f77fff700 1 -- 172.21.15.111:0/3244431371 >> v1:172.21.15.143:6789/0 conn(0x7f5f780adf40 legacy=0x7f5f780ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331811633 0 0) 0x7f5f64003570 con 0x7f5f780aef60 2023-12-19T09:03:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f77fff700 1 -- 172.21.15.111:0/3244431371 >> v1:172.21.15.111:6789/0 conn(0x7f5f780aef60 legacy=0x7f5f780b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f77fff700 1 -- 172.21.15.111:0/3244431371 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7806d140 con 0x7f5f781569e0 2023-12-19T09:03:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f77fff700 1 --2- 172.21.15.111:0/3244431371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f781569e0 0x7f5f78156db0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5f680098e0 tx=0x7f5f680091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f98eb4b62bc59 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60001470 con 0x7f5f781569e0 2023-12-19T09:03:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f68011420 con 0x7f5f781569e0 2023-12-19T09:03:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f777fe700 1 -- 172.21.15.111:0/3244431371 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f680117c0 con 0x7f5f781569e0 2023-12-19T09:03:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 -- 172.21.15.111:0/3244431371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f781569e0 msgr2=0x7f5f78156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/3244431371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f781569e0 0x7f5f78156db0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5f680098e0 tx=0x7f5f680091a0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 -- 172.21.15.111:0/3244431371 shutdown_connections 2023-12-19T09:03:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/3244431371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f781569e0 0x7f5f78156db0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 -- 172.21.15.111:0/3244431371 >> 172.21.15.111:0/3244431371 conn(0x7f5f780c8c00 msgr2=0x7f5f780c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 -- 172.21.15.111:0/3244431371 shutdown_connections 2023-12-19T09:03:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.675+0000 7f5f7d08e700 1 -- 172.21.15.111:0/3244431371 wait complete. 2023-12-19T09:03:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.679+0000 7f5f7d08e700 1 Processor -- start 2023-12-19T09:03:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.679+0000 7f5f7d08e700 1 -- start start 2023-12-19T09:03:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f780adf40 0x7f5f78166f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f780aef60 0x7f5f78152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7816b460 con 0x7f5f781569e0 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f781526c0 con 0x7f5f780aef60 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7816a280 con 0x7f5f780adf40 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47182/0 (socket says 172.21.15.111:47182) 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 -- 172.21.15.111:0/4011660874 learned_addr learned my addr 172.21.15.111:0/4011660874 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f6ffff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f780aef60 0x7f5f78152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f780adf40 0x7f5f78166f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f780adf40 msgr2=0x7f5f78166f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f780adf40 0x7f5f78166f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f780aef60 msgr2=0x7f5f78152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f780aef60 0x7f5f78152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f68009420 con 0x7f5f781569e0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7c88d700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c8e0 tx=0x7f5f70001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70012630 con 0x7f5f781569e0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f700127b0 con 0x7f5f781569e0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7816a850 con 0x7f5f781569e0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70012b70 con 0x7f5f781569e0 2023-12-19T09:03:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.683+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7816ae10 con 0x7f5f781569e0 2023-12-19T09:03:28.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7804be40 con 0x7f5f781569e0 2023-12-19T09:03:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 78) v1 ==== 80080+0+0 (secure 0 0 0) 0x7f5f70003030 con 0x7f5f781569e0 2023-12-19T09:03:28.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f74ff9700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:28.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6573+0+0 (secure 0 0 0) 0x7f5f7007cbb0 con 0x7f5f781569e0 2023-12-19T09:03:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f77fff700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.687+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:03:28.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.695+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5f7004cc90 con 0x7f5f781569e0 2023-12-19T09:03:28.718 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:28.714+0000 7f9e52bc3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:03:28.789 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:28.783+0000 7f5cf6182000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:03:28.792 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:28.786+0000 7f9e52bc3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:28.870 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:28.863+0000 7f5cf6182000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:28.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.883+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f7806d420 con 0x7f5f580606c0 2023-12-19T09:03:28.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.891+0000 7f5f77fff700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:28.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:28.891+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:03:29.014 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:29.010+0000 7f9e52bc3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:03:29.090 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:29.086+0000 7f9e52bc3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:03:29.098 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:29.091+0000 7f5cf6182000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:03:29.182 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:29.178+0000 7f9e52bc3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:03:29.184 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:29.179+0000 7f5cf6182000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:03:29.275 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:29.267+0000 7f5cf6182000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:03:29.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:29.295+0000 7f5f77fff700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:29.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:29.295+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T09:03:29.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:29.562+0000 7f9e52bc3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:03:29.682 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:29.675+0000 7f5cf6182000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:03:29.742 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:29.738+0000 7f9e52bc3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:03:29.857 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:29.851+0000 7f5cf6182000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:03:30.080 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.074+0000 7f9e52bc3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:03:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:30.099+0000 7f5f77fff700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:30.099+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T09:03:30.232 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.227+0000 7f5cf6182000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:03:30.297 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.290+0000 7f9e52bc3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:30.386 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.382+0000 7f9e52bc3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:03:30.450 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.443+0000 7f5cf6182000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:03:30.459 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.454+0000 7f9e52bc3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:03:30.533 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.526+0000 7f9e52bc3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:03:30.549 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.543+0000 7f5cf6182000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:03:30.635 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.627+0000 7f5cf6182000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:03:30.721 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.715+0000 7f5cf6182000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:03:30.764 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.758+0000 7f9e52bc3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:30.838 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.834+0000 7f9e52bc3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:03:30.945 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:30.938+0000 7f9e52bc3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:03:30.962 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:30.955+0000 7f5cf6182000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:03:31.047 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:31.039+0000 7f5cf6182000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:03:31.129 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:31.122+0000 7f9e52bc3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:03:31.144 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:31.139+0000 7f5cf6182000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:03:31.213 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:31.206+0000 7f9e52bc3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:03:31.344 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:31.339+0000 7f5cf6182000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:03:31.437 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:31.431+0000 7f5cf6182000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:03:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:31.703+0000 7f5f77fff700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/33796 2023-12-19T09:03:31.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:31.703+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T09:03:31.911 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:31.906+0000 7f9e52bc3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:03:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:31.923+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 79) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f5f7004d060 con 0x7f5f781569e0 2023-12-19T09:03:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:31.923+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:03:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:31.923+0000 7f5f74ff9700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:32.110 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e10622700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.110 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e10622700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e10622700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e10622700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e10622700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e14e2b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e14e2b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e14e2b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.111 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e14e2b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.112 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:03:32.106+0000 7f9e14e2b700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:03:32.242 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:03:32.235+0000 7f5cf6182000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:03:32.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:32.935+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 80) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f5f7007acc0 con 0x7f5f781569e0 2023-12-19T09:03:32.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:32.935+0000 7f5f74ff9700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:32.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:32.935+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f7806d420 con 0x7f5f580606c0 2023-12-19T09:03:32.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:32.939+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:32.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:32.939+0000 7f5f77fff700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5f680091e0 tx=0x7f5f68016000 comp rx=0 tx=0).ready entity=mgr.9262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:33.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.027+0000 7f5f74ff9700 1 -- 172.21.15.111:0/4011660874 <== mgr.9262 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5f7806d420 con 0x7f5f580606c0 2023-12-19T09:03:33.036 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-12-19T09:03:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 msgr2=0x7f5f58062af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5f680091e0 tx=0x7f5f68016000 comp rx=0 tx=0).stop 2023-12-19T09:03:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 msgr2=0x7f5f78169d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c8e0 tx=0x7f5f70001710 comp rx=0 tx=0).stop 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 shutdown_connections 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f5f580606c0 0x7f5f58062af0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f780aef60 0x7f5f78152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f781569e0 0x7f5f78169d40 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 --2- 172.21.15.111:0/4011660874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f780adf40 0x7f5f78166f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 >> 172.21.15.111:0/4011660874 conn(0x7f5f780c8c00 msgr2=0x7f5f780b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 shutdown_connections 2023-12-19T09:03:33.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.035+0000 7f5f7d08e700 1 -- 172.21.15.111:0/4011660874 wait complete. 2023-12-19T09:03:33.063 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-12-19T09:03:33.063 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T09:03:33.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.219+0000 7fbd9ac03700 1 Processor -- start 2023-12-19T09:03:33.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.223+0000 7fbd9ac03700 1 -- start start 2023-12-19T09:03:33.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd9ac03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 0x7fbd940af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd9ac03700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbd9406af50 0x7fbd940cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd9ac03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9406b3d0 con 0x7fbd9406d6b0 2023-12-19T09:03:33.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd9ac03700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940cc140 con 0x7fbd9406af50 2023-12-19T09:03:33.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd9ac03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940cb3f0 con 0x7fbd940aedc0 2023-12-19T09:03:33.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbd9406af50 0x7fbd940cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:33.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbd9406af50 0x7fbd940cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60872/0 (socket says 172.21.15.111:60872) 2023-12-19T09:03:33.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 -- 172.21.15.111:0/3036999054 learned_addr learned my addr 172.21.15.111:0/3036999054 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:33.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99c01700 1 --2- 172.21.15.111:0/3036999054 >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 0x7fbd940af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 -- 172.21.15.111:0/3036999054 >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 msgr2=0x7fbd940af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 --2- 172.21.15.111:0/3036999054 >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 0x7fbd940af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 -- 172.21.15.111:0/3036999054 >> v1:172.21.15.143:6789/0 conn(0x7fbd9406d6b0 legacy=0x7fbd940af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd98bff700 1 -- 172.21.15.111:0/3036999054 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129734405 0 0) 0x7fbd8c001490 con 0x7fbd9406d6b0 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 -- 172.21.15.111:0/3036999054 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9406d220 con 0x7fbd9406af50 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99c01700 1 --2- 172.21.15.111:0/3036999054 >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 0x7fbd940af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:03:33.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd99400700 1 --2- 172.21.15.111:0/3036999054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406af50 0x7fbd940cac50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbd940cc6c0 tx=0x7fbd84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5738ad533a9d28 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:33.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd98bff700 1 -- 172.21.15.111:0/3036999054 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8400b860 con 0x7fbd9406af50 2023-12-19T09:03:33.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd98bff700 1 -- 172.21.15.111:0/3036999054 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8400b9c0 con 0x7fbd9406af50 2023-12-19T09:03:33.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.227+0000 7fbd98bff700 1 -- 172.21.15.111:0/3036999054 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8400bd60 con 0x7fbd9406af50 2023-12-19T09:03:33.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 -- 172.21.15.111:0/3036999054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406af50 msgr2=0x7fbd940cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/3036999054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406af50 0x7fbd940cac50 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbd940cc6c0 tx=0x7fbd84009250 comp rx=0 tx=0).stop 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 -- 172.21.15.111:0/3036999054 shutdown_connections 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/3036999054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406af50 0x7fbd940cac50 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/3036999054 >> v2:172.21.15.143:3300/0 conn(0x7fbd940aedc0 0x7fbd940af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 -- 172.21.15.111:0/3036999054 >> 172.21.15.111:0/3036999054 conn(0x7fbd940cc760 msgr2=0x7fbd940ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 -- 172.21.15.111:0/3036999054 shutdown_connections 2023-12-19T09:03:33.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.231+0000 7fbd9ac03700 1 -- 172.21.15.111:0/3036999054 wait complete. 2023-12-19T09:03:33.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 Processor -- start 2023-12-19T09:03:33.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 -- start start 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd9406af50 0x7fbd940b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd940aedc0 0x7fbd940c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940c98b0 con 0x7fbd9406d6b0 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940b3d40 con 0x7fbd940aedc0 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9416ceb0 con 0x7fbd9406af50 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60888/0 (socket says 172.21.15.111:60888) 2023-12-19T09:03:33.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 -- 172.21.15.111:0/793712722 learned_addr learned my addr 172.21.15.111:0/793712722 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:03:33.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9a402700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd940aedc0 0x7fbd940c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 -- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd9406af50 msgr2=0x7fbd940b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd9406af50 0x7fbd940b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 -- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd940aedc0 msgr2=0x7fbd940c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd940aedc0 0x7fbd940c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 -- 172.21.15.111:0/793712722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840094f0 con 0x7fbd9406d6b0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd99400700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbd840090b0 tx=0x7fbd84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84011e60 con 0x7fbd9406d6b0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8401a660 con 0x7fbd9406d6b0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8401aa00 con 0x7fbd9406d6b0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.235+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9416d190 con 0x7fbd9406d6b0 2023-12-19T09:03:33.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.239+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9416d750 con 0x7fbd9406d6b0 2023-12-19T09:03:33.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.239+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9404be40 con 0x7fbd9406d6b0 2023-12-19T09:03:33.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.239+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 80) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fbd8401b430 con 0x7fbd9406d6b0 2023-12-19T09:03:33.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.243+0000 7fbd89ffb700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 0x7fbd780628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:03:33.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.243+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6917+0+0 (secure 0 0 0) 0x7fbd8407dff0 con 0x7fbd9406d6b0 2023-12-19T09:03:33.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.243+0000 7fbd99c01700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 0x7fbd780628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:03:33.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.243+0000 7fbd99c01700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 0x7fbd780628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbd90006fd0 tx=0x7fbd90008040 comp rx=0 tx=0).ready entity=mgr.9262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:03:33.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.247+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbd8404e3d0 con 0x7fbd9406d6b0 2023-12-19T09:03:33.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.447+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd940c9b10 con 0x7fbd780603f0 2023-12-19T09:03:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.531+0000 7fbd89ffb700 1 -- 172.21.15.111:0/793712722 <== mgr.9262 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fbd940c9b10 con 0x7fbd780603f0 2023-12-19T09:03:33.537 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-12-19T09:03:33.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 msgr2=0x7fbd780628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 0x7fbd780628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbd90006fd0 tx=0x7fbd90008040 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 msgr2=0x7fbd940b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbd840090b0 tx=0x7fbd84017000 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 shutdown_connections 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7fbd780603f0 0x7fbd780628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd940aedc0 0x7fbd940c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd9406d6b0 0x7fbd940b3800 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 --2- 172.21.15.111:0/793712722 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd9406af50 0x7fbd940b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 >> 172.21.15.111:0/793712722 conn(0x7fbd940cc760 msgr2=0x7fbd9406e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 shutdown_connections 2023-12-19T09:03:33.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:03:33.535+0000 7fbd9ac03700 1 -- 172.21.15.111:0/793712722 wait complete. 2023-12-19T09:03:33.566 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-12-19T09:03:52.437 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:03:52.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:04:00.556 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:04:00.549+0000 7f9e03e09700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:04:00.556 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:04:00.549+0000 7f9e03e09700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:04:00.556 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:04:00.549+0000 7f9e03e09700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:04:00.557 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:04:00.549+0000 7f9e03e09700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:04:00.557 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:04:00.549+0000 7f9e03e09700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:04:22.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:04:22.586 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:04:52.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:04:52.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:05:23.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:05:23.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:05:53.246 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:05:53.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:06:23.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:06:23.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:06:53.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:06:53.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:07:23.629 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:07:23.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:07:53.786 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:07:53.794 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:08:23.933 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:08:23.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:08:54.071 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:08:54.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:09:24.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:09:24.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:09:54.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:09:54.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:10:24.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:10:24.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:10:54.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:10:54.611 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:11:24.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:11:24.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:11:54.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:11:54.908 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:12:25.049 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:12:25.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:12:55.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:12:55.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:13:25.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:13:25.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:13:55.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:13:55.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:14:15.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:14:15.568 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T09:14:15.604 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_04_00_UTC 2023-12-19T09:14:15.604 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_05_00_UTC 2023-12-19T09:14:15.604 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_06_00_UTC 2023-12-19T09:14:15.604 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_07_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_08_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_09_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_10_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_11_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_12_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_13_00_UTC 2023-12-19T09:14:15.605 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_14_00_UTC 2023-12-19T09:14:15.606 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-09_04_00_UTC', 'scheduled-2023-12-19-09_05_00_UTC', 'scheduled-2023-12-19-09_06_00_UTC', 'scheduled-2023-12-19-09_07_00_UTC', 'scheduled-2023-12-19-09_08_00_UTC', 'scheduled-2023-12-19-09_09_00_UTC', 'scheduled-2023-12-19-09_10_00_UTC', 'scheduled-2023-12-19-09_11_00_UTC', 'scheduled-2023-12-19-09_12_00_UTC', 'scheduled-2023-12-19-09_13_00_UTC', 'scheduled-2023-12-19-09_14_00_UTC'] 2023-12-19T09:14:15.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-12-19T09:14:15.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.819+0000 7f6c8dee3700 1 Processor -- start 2023-12-19T09:14:15.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.819+0000 7f6c8dee3700 1 -- start start 2023-12-19T09:14:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8dee3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c88158da0 0x7f6c8806e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8dee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c8806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8dee3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8815d720 con 0x7f6c880caca0 2023-12-19T09:14:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8dee3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c880ca290 con 0x7f6c88158da0 2023-12-19T09:14:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8dee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c880ca4a0 con 0x7f6c8806df10 2023-12-19T09:14:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c88158da0 0x7f6c8806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c88158da0 0x7f6c8806e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51228/0 (socket says 172.21.15.111:51228) 2023-12-19T09:14:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 -- 172.21.15.111:0/877776023 learned_addr learned my addr 172.21.15.111:0/877776023 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:14:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c87fff700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c8806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 -- 172.21.15.111:0/877776023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 msgr2=0x7f6c8806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c8806ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 -- 172.21.15.111:0/877776023 >> v1:172.21.15.143:6789/0 conn(0x7f6c880caca0 legacy=0x7f6c8806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 -- 172.21.15.111:0/877776023 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880ca780 con 0x7f6c88158da0 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.823+0000 7f6c8cee1700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c8806e670 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0099b0 tx=0x7f6c7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7d12137269e0e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c877fe700 1 -- 172.21.15.111:0/877776023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c009ea0 con 0x7f6c88158da0 2023-12-19T09:14:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c877fe700 1 -- 172.21.15.111:0/877776023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c00b860 con 0x7f6c88158da0 2023-12-19T09:14:15.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c877fe700 1 -- 172.21.15.111:0/877776023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00bc00 con 0x7f6c88158da0 2023-12-19T09:14:15.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- 172.21.15.111:0/877776023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 msgr2=0x7f6c8806e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:15.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c8806e670 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0099b0 tx=0x7f6c7c009250 comp rx=0 tx=0).stop 2023-12-19T09:14:15.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- 172.21.15.111:0/877776023 shutdown_connections 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c8806ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/877776023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c8806e670 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- 172.21.15.111:0/877776023 >> 172.21.15.111:0/877776023 conn(0x7f6c880cc760 msgr2=0x7f6c880ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- 172.21.15.111:0/877776023 shutdown_connections 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- 172.21.15.111:0/877776023 wait complete. 2023-12-19T09:14:15.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 Processor -- start 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.827+0000 7f6c8dee3700 1 -- start start 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c880b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 0x7f6c880c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c880b3c90 con 0x7f6c88158da0 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c880b3080 con 0x7f6c880caca0 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c880b3360 con 0x7f6c8806df10 2023-12-19T09:14:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8cee1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c880b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8cee1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c880b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:48202/0 (socket says 172.21.15.111:48202) 2023-12-19T09:14:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8cee1700 1 -- 172.21.15.111:0/1100968345 learned_addr learned my addr 172.21.15.111:0/1100968345 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:14:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c87fff700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 0x7f6c880c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51242/0 (socket says 172.21.15.111:51242) 2023-12-19T09:14:15.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 -- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 msgr2=0x7f6c880b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c880b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 -- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 msgr2=0x7f6c880c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 0x7f6c880c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 -- 172.21.15.111:0/1100968345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c0094f0 con 0x7f6c88158da0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c87fff700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 0x7f6c880c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8d6e2700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000c830 tx=0x7f6c80001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80012630 con 0x7f6c88158da0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80012790 con 0x7f6c88158da0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c880b3640 con 0x7f6c88158da0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80012b50 con 0x7f6c88158da0 2023-12-19T09:14:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.831+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c880c47c0 con 0x7f6c88158da0 2023-12-19T09:14:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.835+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8804be40 con 0x7f6c88158da0 2023-12-19T09:14:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.835+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 83) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f6c8001a3a0 con 0x7f6c88158da0 2023-12-19T09:14:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.835+0000 7f6c84ff9700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 0x7f6c6c062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.835+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6975+0+0 (secure 0 0 0) 0x7f6c8007d0a0 con 0x7f6c88158da0 2023-12-19T09:14:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.839+0000 7f6c8cee1700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 0x7f6c6c062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:15.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.839+0000 7f6c8cee1700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 0x7f6c6c062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c006e80 tx=0x7f6c7c017000 comp rx=0 tx=0).ready entity=mgr.9262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:15.843+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c8004cff0 con 0x7f6c88158da0 2023-12-19T09:14:16.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.035+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 --> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c8808b160 con 0x7f6c6c0606c0 2023-12-19T09:14:16.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.047+0000 7f6c84ff9700 1 -- 172.21.15.111:0/1100968345 <== mgr.9262 v2:172.21.15.143:6832/33796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f6c8808b160 con 0x7f6c6c0606c0 2023-12-19T09:14:16.055 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:14:16.055 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T09:03:32", "first": "2023-12-19T09:04:00", "last": "2023-12-19T09:14:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 msgr2=0x7f6c6c062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 0x7f6c6c062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c006e80 tx=0x7f6c7c017000 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 msgr2=0x7f6c88157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000c830 tx=0x7f6c80001690 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 shutdown_connections 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f6c6c0606c0 0x7f6c6c062b70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c880caca0 0x7f6c880c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c88158da0 0x7f6c88157750 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 --2- 172.21.15.111:0/1100968345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c8806df10 0x7f6c880b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 >> 172.21.15.111:0/1100968345 conn(0x7f6c880cc760 msgr2=0x7f6c88155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:14:16.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 shutdown_connections 2023-12-19T09:14:16.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.051+0000 7f6c8dee3700 1 -- 172.21.15.111:0/1100968345 wait complete. 2023-12-19T09:14:16.081 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T09:03:32", "first": "2023-12-19T09:04:00", "last": "2023-12-19T09:14:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-12-19T09:14:16.081 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T09:03:32', 'first': '2023-12-19T09:04:00', 'last': '2023-12-19T09:14:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-12-19T09:14:16.081 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-12-19T09:14:16.082 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-12-19T09:14:16.082 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-19T09:14:16.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.239+0000 7f1069563700 1 Processor -- start 2023-12-19T09:14:16.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.239+0000 7f1069563700 1 -- start start 2023-12-19T09:14:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1069563700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1064158e20 0x7f106406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1069563700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106406b730 con 0x7f10640ce810 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1069563700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106406ab20 con 0x7f10641591f0 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1069563700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106406ae00 con 0x7f1064158e20 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1063fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1064158e20 0x7f106406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1063fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1064158e20 0x7f106406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51246/0 (socket says 172.21.15.111:51246) 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1063fff700 1 -- 172.21.15.111:0/3171570367 learned_addr learned my addr 172.21.15.111:0/3171570367 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:14:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.243+0000 7f1068d62700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f10640ce810 0x7f10640ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:41604/0 (socket says 172.21.15.111:41604) 2023-12-19T09:14:16.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1063fff700 1 -- 172.21.15.111:0/3171570367 >> v1:172.21.15.143:6789/0 conn(0x7f10641591f0 legacy=0x7f10640ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:14:16.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1063fff700 1 -- 172.21.15.111:0/3171570367 >> v1:172.21.15.111:6789/0 conn(0x7f10640ce810 legacy=0x7f10640ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1063fff700 1 -- 172.21.15.111:0/3171570367 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106406b0e0 con 0x7f1064158e20 2023-12-19T09:14:16.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1063fff700 1 --2- 172.21.15.111:0/3171570367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1064158e20 0x7f106406a540 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f10540099b0 tx=0x7f1054009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc58ea167199a14 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:16.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f10637fe700 1 -- 172.21.15.111:0/3171570367 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105400b860 con 0x7f1064158e20 2023-12-19T09:14:16.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f10637fe700 1 -- 172.21.15.111:0/3171570367 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105400b9c0 con 0x7f1064158e20 2023-12-19T09:14:16.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f10637fe700 1 -- 172.21.15.111:0/3171570367 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105400bd60 con 0x7f1064158e20 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 -- 172.21.15.111:0/3171570367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1064158e20 msgr2=0x7f106406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 --2- 172.21.15.111:0/3171570367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1064158e20 0x7f106406a540 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f10540099b0 tx=0x7f1054009250 comp rx=0 tx=0).stop 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 -- 172.21.15.111:0/3171570367 shutdown_connections 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 --2- 172.21.15.111:0/3171570367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1064158e20 0x7f106406a540 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 -- 172.21.15.111:0/3171570367 >> 172.21.15.111:0/3171570367 conn(0x7f1064067bb0 msgr2=0x7f10640cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:14:16.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 -- 172.21.15.111:0/3171570367 shutdown_connections 2023-12-19T09:14:16.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.247+0000 7f1069563700 1 -- 172.21.15.111:0/3171570367 wait complete. 2023-12-19T09:14:16.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 Processor -- start 2023-12-19T09:14:16.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 -- start start 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10640ce810 0x7f1064169100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 0x7f10640b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10640c7ae0 con 0x7f10641591f0 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106415c6e0 con 0x7f1064158e20 2023-12-19T09:14:16.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.251+0000 7f1069563700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10640c71b0 con 0x7f10640ce810 2023-12-19T09:14:16.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:16.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51260/0 (socket says 172.21.15.111:51260) 2023-12-19T09:14:16.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 -- 172.21.15.111:0/3719479318 learned_addr learned my addr 172.21.15.111:0/3719479318 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:14:16.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1063fff700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10640ce810 0x7f1064169100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:16.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f105bfff700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 0x7f10640b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 -- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10640ce810 msgr2=0x7f1064169100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10640ce810 0x7f1064169100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 -- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 msgr2=0x7f10640b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 0x7f10640b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 -- 172.21.15.111:0/3719479318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540094f0 con 0x7f10641591f0 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f105bfff700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 0x7f10640b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1068d62700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f105c00cd00 tx=0x7f105c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00ce60 con 0x7f10641591f0 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c005df0 con 0x7f10641591f0 2023-12-19T09:14:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00ab90 con 0x7f10641591f0 2023-12-19T09:14:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10640c7490 con 0x7f10641591f0 2023-12-19T09:14:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1064169640 con 0x7f10641591f0 2023-12-19T09:14:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.255+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10640c7c50 con 0x7f10641591f0 2023-12-19T09:14:16.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.259+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 83) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f105c0070f0 con 0x7f10641591f0 2023-12-19T09:14:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.259+0000 7f1060ff9700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 0x7f1044062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:14:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.259+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6975+0+0 (secure 0 0 0) 0x7f105c07cf70 con 0x7f10641591f0 2023-12-19T09:14:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.259+0000 7f1063fff700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 0x7f1044062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:14:16.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.263+0000 7f1063fff700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 0x7f1044062b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f10540099b0 tx=0x7f1054017000 comp rx=0 tx=0).ready entity=mgr.9262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:14:16.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.267+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f105c04cf40 con 0x7f10641591f0 2023-12-19T09:14:16.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.475+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f106404be40 con 0x7f10641591f0 2023-12-19T09:14:16.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.483+0000 7f1060ff9700 1 -- 172.21.15.111:0/3719479318 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v83) v1 ==== 74+0+154347 (secure 0 0 0) 0x7f105c050560 con 0x7f10641591f0 2023-12-19T09:14:16.493 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 msgr2=0x7f1044062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 0x7f1044062b70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f10540099b0 tx=0x7f1054017000 comp rx=0 tx=0).stop 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 msgr2=0x7f10640b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f105c00cd00 tx=0x7f105c001b10 comp rx=0 tx=0).stop 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 shutdown_connections 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:6832/33796,v1:172.21.15.143:6833/33796] conn(0x7f10440606c0 0x7f1044062b70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1064158e20 0x7f10640b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10641591f0 0x7f10640b4200 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 --2- 172.21.15.111:0/3719479318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10640ce810 0x7f1064169100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:14:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 >> 172.21.15.111:0/3719479318 conn(0x7f1064067bb0 msgr2=0x7f10640cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:14:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 shutdown_connections 2023-12-19T09:14:16.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:14:16.487+0000 7f1069563700 1 -- 172.21.15.111:0/3719479318 wait complete. 2023-12-19T09:14:16.514 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":83,"active_gid":9262,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":33796},{"type":"v1","addr":"172.21.15.143:6833","nonce":33796}]},"active_addr":"172.21.15.143:6833/33796","active_change":"2023-12-19T09:03:31.913770+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":9567,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":176,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2893890504}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3287144757}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2193897545}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":57208669}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3589389202}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2998303494}]}]} 2023-12-19T09:14:16.518 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-12-19T09:14:16.518 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-19T09:14:16.518 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-12-19T09:14:16.519 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-19T09:14:16.519 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:14:16.580 INFO:tasks.ceph.mgr.x:Stopped 2023-12-19T09:14:16.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-19T09:14:16.585 INFO:tasks.ceph.mgr.x:Started 2023-12-19T09:14:16.816 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:16.812+0000 7f60eafab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:14:17.331 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:17.328+0000 7f60eafab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:14:17.402 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:17.400+0000 7f60eafab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:14:17.538 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:17.536+0000 7f60eafab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:14:17.785 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:17.784+0000 7f60eafab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:14:17.853 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:17.852+0000 7f60eafab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:14:18.062 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:18.060+0000 7f60eafab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:14:18.144 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:18.140+0000 7f60eafab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:14:18.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:18.224+0000 7f60eafab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:14:18.607 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:18.604+0000 7f60eafab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:14:18.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:18.760+0000 7f60eafab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:14:19.100 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.096+0000 7f60eafab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:14:19.309 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.308+0000 7f60eafab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:14:19.400 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.396+0000 7f60eafab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:14:19.479 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.476+0000 7f60eafab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:14:19.551 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.548+0000 7f60eafab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:14:19.770 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.768+0000 7f60eafab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:14:19.849 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.848+0000 7f60eafab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:14:19.939 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:19.936+0000 7f60eafab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:14:20.145 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:20.144+0000 7f60eafab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:14:20.238 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:20.236+0000 7f60eafab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:14:20.974 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:20.972+0000 7f60eafab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:14:21.251 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60ad67e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.251 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60ad67e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.251 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60ad67e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.251 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60ad67e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60ad67e700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60aae79700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60aae79700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60aae79700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60aae79700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:21.252 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:14:21.248+0000 7f60aae79700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:14:25.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:14:25.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:14:55.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:14:55.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:15:00.244 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:15:00.239+0000 7f60b5e8f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:15:00.244 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:15:00.239+0000 7f60b5e8f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:15:00.244 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:15:00.239+0000 7f60b5e8f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:15:00.244 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:15:00.239+0000 7f60b5e8f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:15:00.245 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:15:00.239+0000 7f60b5e8f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:15:25.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:15:25.822 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:15:55.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:15:55.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:16:26.045 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:16:26.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:16:56.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:16:56.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:17:26.364 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:17:26.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:17:56.528 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:17:56.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:18:26.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:18:26.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:18:56.855 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:18:56.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:19:16.586 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T09:19:16.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.758+0000 7f9d080cd700 1 Processor -- start 2023-12-19T09:19:16.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.758+0000 7f9d080cd700 1 -- start start 2023-12-19T09:19:16.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d080cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d000ad0f0 0x7f9d000cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d080cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d000af3d0 0x7f9d0006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d080cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000adf30 con 0x7f9d000acd20 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d080cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d000cd6f0 con 0x7f9d000af3d0 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d080cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d000cd9d0 con 0x7f9d000ad0f0 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d000af3d0 0x7f9d0006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9d000af3d0 0x7f9d0006c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54808/0 (socket says 172.21.15.111:54808) 2023-12-19T09:19:16.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 -- 172.21.15.111:0/2251656859 learned_addr learned my addr 172.21.15.111:0/2251656859 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:16.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d05668700 1 --2- 172.21.15.111:0/2251656859 >> v2:172.21.15.143:3300/0 conn(0x7f9d000ad0f0 0x7f9d000cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 -- 172.21.15.111:0/2251656859 >> v2:172.21.15.143:3300/0 conn(0x7f9d000ad0f0 msgr2=0x7f9d000cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:16.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 --2- 172.21.15.111:0/2251656859 >> v2:172.21.15.143:3300/0 conn(0x7f9d000ad0f0 0x7f9d000cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:16.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 -- 172.21.15.111:0/2251656859 >> v1:172.21.15.143:6789/0 conn(0x7f9d000acd20 legacy=0x7f9d000cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:16.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.762+0000 7f9d0666a700 1 -- 172.21.15.111:0/2251656859 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d000adbc0 con 0x7f9d000af3d0 2023-12-19T09:19:16.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d0666a700 1 --2- 172.21.15.111:0/2251656859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000af3d0 0x7f9d0006c7c0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f9d000c8930 tx=0x7f9cf8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710142e64f9b064a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:16.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d04e67700 1 -- 172.21.15.111:0/2251656859 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800e070 con 0x7f9d000af3d0 2023-12-19T09:19:16.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d04e67700 1 -- 172.21.15.111:0/2251656859 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf800ba50 con 0x7f9d000af3d0 2023-12-19T09:19:16.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d04e67700 1 -- 172.21.15.111:0/2251656859 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80092e0 con 0x7f9d000af3d0 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 -- 172.21.15.111:0/2251656859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000af3d0 msgr2=0x7f9d0006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 --2- 172.21.15.111:0/2251656859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000af3d0 0x7f9d0006c7c0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f9d000c8930 tx=0x7f9cf8009840 comp rx=0 tx=0).stop 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 -- 172.21.15.111:0/2251656859 shutdown_connections 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 --2- 172.21.15.111:0/2251656859 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000af3d0 0x7f9d0006c7c0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 --2- 172.21.15.111:0/2251656859 >> v2:172.21.15.143:3300/0 conn(0x7f9d000ad0f0 0x7f9d000cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 -- 172.21.15.111:0/2251656859 >> 172.21.15.111:0/2251656859 conn(0x7f9d000cdce0 msgr2=0x7f9d000ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 -- 172.21.15.111:0/2251656859 shutdown_connections 2023-12-19T09:19:16.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.766+0000 7f9d080cd700 1 -- 172.21.15.111:0/2251656859 wait complete. 2023-12-19T09:19:16.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 Processor -- start 2023-12-19T09:19:16.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 -- start start 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d000ad0f0 0x7f9d000abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d000af3d0 0x7f9d000ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0015a9e0 con 0x7f9d000acd20 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000ac6f0 con 0x7f9d000ad0f0 2023-12-19T09:19:16.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d080cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00068540 con 0x7f9d000af3d0 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d05e69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d05e69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54812/0 (socket says 172.21.15.111:54812) 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.770+0000 7f9d05e69700 1 -- 172.21.15.111:0/1368454874 learned_addr learned my addr 172.21.15.111:0/1368454874 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05668700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d000ad0f0 0x7f9d000abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d0666a700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d000af3d0 0x7f9d000ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 -- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d000af3d0 msgr2=0x7f9d000ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d000af3d0 0x7f9d000ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 -- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d000ad0f0 msgr2=0x7f9d000abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d000ad0f0 0x7f9d000abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 -- 172.21.15.111:0/1368454874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf8009ae0 con 0x7f9d000acd20 2023-12-19T09:19:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d05e69700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00c4e0 tx=0x7f9cfc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011590 con 0x7f9d000acd20 2023-12-19T09:19:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc011710 con 0x7f9d000acd20 2023-12-19T09:19:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0015ac50 con 0x7f9d000acd20 2023-12-19T09:19:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011ad0 con 0x7f9d000acd20 2023-12-19T09:19:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00068820 con 0x7f9d000acd20 2023-12-19T09:19:16.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.774+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d00071120 con 0x7f9d000acd20 2023-12-19T09:19:16.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.778+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 88) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f9cfc01b3b0 con 0x7f9d000acd20 2023-12-19T09:19:16.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.778+0000 7f9cf5ffb700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 0x7f9ce4062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:16.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.778+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f9cfc07c460 con 0x7f9d000acd20 2023-12-19T09:19:16.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.778+0000 7f9d05668700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 0x7f9ce4062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:16.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.782+0000 7f9d05668700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 0x7f9ce4062b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9d0015c550 tx=0x7f9cf0008040 comp rx=0 tx=0).ready entity=mgr.9355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:16.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.786+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9cfc04c2a0 con 0x7f9d000acd20 2023-12-19T09:19:16.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:16.978+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d000ad7e0 con 0x7f9ce40606c0 2023-12-19T09:19:17.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.070+0000 7f9cf5ffb700 1 -- 172.21.15.111:0/1368454874 <== mgr.9355 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9d000ad7e0 con 0x7f9ce40606c0 2023-12-19T09:19:17.074 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-12-19T09:19:17.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 msgr2=0x7f9ce4062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 0x7f9ce4062b70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9d0015c550 tx=0x7f9cf0008040 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 msgr2=0x7f9d00157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00c4e0 tx=0x7f9cfc00c7f0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 shutdown_connections 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9ce40606c0 0x7f9ce4062b70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d000ad0f0 0x7f9d000abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d000af3d0 0x7f9d000ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 --2- 172.21.15.111:0/1368454874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000acd20 0x7f9d00157fe0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 >> 172.21.15.111:0/1368454874 conn(0x7f9d000cdce0 msgr2=0x7f9d000cbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 shutdown_connections 2023-12-19T09:19:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.074+0000 7f9d080cd700 1 -- 172.21.15.111:0/1368454874 wait complete. 2023-12-19T09:19:17.099 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-12-19T09:19:17.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:17.099 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_04_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_05_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_06_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_07_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_08_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_09_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_10_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_11_00_UTC 2023-12-19T09:19:17.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_12_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_13_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_14_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_15_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_16_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_17_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_18_00_UTC 2023-12-19T09:19:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_19_00_UTC 2023-12-19T09:19:17.137 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-12-19-09_04_00_UTC', 'scheduled-2023-12-19-09_05_00_UTC', 'scheduled-2023-12-19-09_06_00_UTC', 'scheduled-2023-12-19-09_07_00_UTC', 'scheduled-2023-12-19-09_08_00_UTC', 'scheduled-2023-12-19-09_09_00_UTC', 'scheduled-2023-12-19-09_10_00_UTC', 'scheduled-2023-12-19-09_11_00_UTC', 'scheduled-2023-12-19-09_12_00_UTC', 'scheduled-2023-12-19-09_13_00_UTC', 'scheduled-2023-12-19-09_14_00_UTC', 'scheduled-2023-12-19-09_15_00_UTC', 'scheduled-2023-12-19-09_16_00_UTC', 'scheduled-2023-12-19-09_17_00_UTC', 'scheduled-2023-12-19-09_18_00_UTC', 'scheduled-2023-12-19-09_19_00_UTC'] 2023-12-19T09:19:17.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --fs cephfs --format json /snap_test_dir1/test_restart 2023-12-19T09:19:17.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.354+0000 7f8f3c8e3700 1 Processor -- start 2023-12-19T09:19:17.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.354+0000 7f8f3c8e3700 1 -- start start 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f3c8e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f380b0040 0x7f8f3806af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f3c8e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380cb3f0 con 0x7f8f380aedc0 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f3c8e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3806d220 con 0x7f8f380af190 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f3c8e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f380cc140 con 0x7f8f380b0040 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f377fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8f380aedc0 0x7f8f3806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59186/0 (socket says 172.21.15.111:59186) 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f377fe700 1 -- 172.21.15.111:0/3636504499 learned_addr learned my addr 172.21.15.111:0/3636504499 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:17.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 --2- 172.21.15.111:0/3636504499 >> v2:172.21.15.143:3300/0 conn(0x7f8f380b0040 0x7f8f3806af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619753594 0 0) 0x7f8f380cb3f0 con 0x7f8f380aedc0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f20003570 con 0x7f8f380aedc0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339486418 0 0) 0x7f8f20003570 con 0x7f8f380aedc0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 >> v2:172.21.15.143:3300/0 conn(0x7f8f380b0040 msgr2=0x7f8f3806af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 --2- 172.21.15.111:0/3636504499 >> v2:172.21.15.143:3300/0 conn(0x7f8f380b0040 0x7f8f3806af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 >> v1:172.21.15.143:6789/0 conn(0x7f8f380af190 legacy=0x7f8f380af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3806c8f0 con 0x7f8f380aedc0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f8f28001470 con 0x7f8f380aedc0 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 >> v1:172.21.15.111:6789/0 conn(0x7f8f380aedc0 legacy=0x7f8f3806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f20003a60 0x7f8f20003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f20004810 0x7f8f20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 0x7f8f2000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380cc140 con 0x7f8f200051c0 2023-12-19T09:19:17.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2000c160 con 0x7f8f20004810 2023-12-19T09:19:17.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2000c320 con 0x7f8f20003a60 2023-12-19T09:19:17.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 0x7f8f2000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f377fe700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f20003a60 0x7f8f20003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f36ffd700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f20004810 0x7f8f20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 -- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f20003a60 msgr2=0x7f8f20003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f20003a60 0x7f8f20003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 -- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f20004810 msgr2=0x7f8f20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f20004810 0x7f8f20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.358+0000 7f8f37fff700 1 -- 172.21.15.111:0/3636504499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20003570 con 0x7f8f200051c0 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f37fff700 1 -- 172.21.15.111:0/3636504499 reap_dead start 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f37fff700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 0x7f8f2000ba60 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8f380b5dd0 tx=0x7f8f3000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0883450ba39482 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000ea60 con 0x7f8f200051c0 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3000f040 con 0x7f8f200051c0 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f367fc700 1 -- 172.21.15.111:0/3636504499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000edf0 con 0x7f8f200051c0 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 msgr2=0x7f8f2000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 0x7f8f2000ba60 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8f380b5dd0 tx=0x7f8f3000b040 comp rx=0 tx=0).stop 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/3636504499 shutdown_connections 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/3636504499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f200051c0 0x7f8f2000ba60 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/3636504499 >> 172.21.15.111:0/3636504499 conn(0x7f8f380cc760 msgr2=0x7f8f380ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:17.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/3636504499 shutdown_connections 2023-12-19T09:19:17.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.362+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/3636504499 wait complete. 2023-12-19T09:19:17.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 Processor -- start 2023-12-19T09:19:17.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 -- start start 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 0x7f8f380c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f38147a60 0x7f8f38169420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380af610 con 0x7f8f38169960 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380c67e0 con 0x7f8f38147a60 2023-12-19T09:19:17.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3816a340 con 0x7f8f200051c0 2023-12-19T09:19:17.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 0x7f8f380c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54820/0 (socket says 172.21.15.111:54820) 2023-12-19T09:19:17.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 0x7f8f380c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:58408/0 (socket says 172.21.15.111:58408) 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 -- 172.21.15.111:0/684884810 learned_addr learned my addr 172.21.15.111:0/684884810 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f36ffd700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f38147a60 0x7f8f38169420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 -- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 msgr2=0x7f8f380c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 0x7f8f380c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 -- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f38147a60 msgr2=0x7f8f38169420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f38147a60 0x7f8f38169420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 -- 172.21.15.111:0/684884810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3815a7d0 con 0x7f8f38169960 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f37fff700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c36f0 tx=0x7f8f3001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30022070 con 0x7f8f38169960 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30028420 con 0x7f8f38169960 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000eee0 con 0x7f8f38169960 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.366+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f380c5eb0 con 0x7f8f38169960 2023-12-19T09:19:17.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.370+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f380c6470 con 0x7f8f38169960 2023-12-19T09:19:17.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.370+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f380c6d50 con 0x7f8f38169960 2023-12-19T09:19:17.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.370+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 88) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f8f30026070 con 0x7f8f38169960 2023-12-19T09:19:17.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.374+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 0x7f8f24062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.374+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f8f3007eb50 con 0x7f8f38169960 2023-12-19T09:19:17.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.374+0000 7f8f377fe700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 0x7f8f24062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.374+0000 7f8f377fe700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 0x7f8f24062b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8f280015a0 tx=0x7f8f3806d190 comp rx=0 tx=0).ready entity=mgr.9355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.378+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8f3004e9e0 con 0x7f8f38169960 2023-12-19T09:19:17.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.566+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f3804be40 con 0x7f8f240606c0 2023-12-19T09:19:17.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.582+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/684884810 <== mgr.9355 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f8f3804be40 con 0x7f8f240606c0 2023-12-19T09:19:17.587 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:17.587 INFO:teuthology.orchestra.run.smithi111.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T09:03:32", "first": "2023-12-19T09:04:00", "last": "2023-12-19T09:19:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-12-19T09:19:17.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 msgr2=0x7f8f24062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 0x7f8f24062b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8f280015a0 tx=0x7f8f3806d190 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 msgr2=0x7f8f38169e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c36f0 tx=0x7f8f3001a000 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 shutdown_connections 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8f240606c0 0x7f8f24062b70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f38147a60 0x7f8f38169420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f38169960 0x7f8f38169e00 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 --2- 172.21.15.111:0/684884810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f200051c0 0x7f8f380c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.586+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 >> 172.21.15.111:0/684884810 conn(0x7f8f380cc760 msgr2=0x7f8f3815c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.590+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 shutdown_connections 2023-12-19T09:19:17.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.590+0000 7f8f3c8e3700 1 -- 172.21.15.111:0/684884810 wait complete. 2023-12-19T09:19:17.613 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1M", "retention": {}, "start": "2023-12-19T00:00:00", "created": "2023-12-19T09:03:32", "first": "2023-12-19T09:04:00", "last": "2023-12-19T09:19:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-12-19T09:19:17.613 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1M', 'retention': {}, 'start': '2023-12-19T00:00:00', 'created': '2023-12-19T09:03:32', 'first': '2023-12-19T09:04:00', 'last': '2023-12-19T09:19:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-12-19T09:19:17.613 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-12-19T09:19:17.613 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-12-19T09:19:17.614 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs /snap_test_dir1/test_restart 1M 2023-12-19T09:19:17.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.782+0000 7f5016878700 1 Processor -- start 2023-12-19T09:19:17.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5016878700 1 -- start start 2023-12-19T09:19:17.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5016878700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50101538c0 0x7f501006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5016878700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50101572d0 con 0x7f50100c6120 2023-12-19T09:19:17.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5016878700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5010156a60 con 0x7f50100cc4e0 2023-12-19T09:19:17.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5016878700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5010156d10 con 0x7f50101538c0 2023-12-19T09:19:17.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5014e15700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f50100c6120 0x7f50100c6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59202/0 (socket says 172.21.15.111:59202) 2023-12-19T09:19:17.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f5014e15700 1 -- 172.21.15.111:0/3736633018 learned_addr learned my addr 172.21.15.111:0/3736633018 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:17.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f500ffff700 1 --2- 172.21.15.111:0/3736633018 >> v2:172.21.15.143:3300/0 conn(0x7f50101538c0 0x7f501006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.786+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988026626 0 0) 0x7f50101572d0 con 0x7f50100c6120 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc003570 con 0x7f50100c6120 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818761756 0 0) 0x7f5010156a60 con 0x7f50100cc4e0 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50101572d0 con 0x7f50100cc4e0 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500ffff700 1 -- 172.21.15.111:0/3736633018 >> v1:172.21.15.143:6789/0 conn(0x7f50100cc4e0 legacy=0x7f5010156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500ffff700 1 -- 172.21.15.111:0/3736633018 >> v1:172.21.15.111:6789/0 conn(0x7f50100c6120 legacy=0x7f50100c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500ffff700 1 -- 172.21.15.111:0/3736633018 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010156ff0 con 0x7f50101538c0 2023-12-19T09:19:17.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500ffff700 1 --2- 172.21.15.111:0/3736633018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50101538c0 0x7f501006c7c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f50000098e0 tx=0x7f50000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0d9f0a425263e00 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000011630 con 0x7f50101538c0 2023-12-19T09:19:17.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000011790 con 0x7f50101538c0 2023-12-19T09:19:17.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f500f7fe700 1 -- 172.21.15.111:0/3736633018 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000011b30 con 0x7f50101538c0 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 -- 172.21.15.111:0/3736633018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50101538c0 msgr2=0x7f501006c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 --2- 172.21.15.111:0/3736633018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50101538c0 0x7f501006c7c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f50000098e0 tx=0x7f50000091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 -- 172.21.15.111:0/3736633018 shutdown_connections 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 --2- 172.21.15.111:0/3736633018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50101538c0 0x7f501006c7c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 -- 172.21.15.111:0/3736633018 >> 172.21.15.111:0/3736633018 conn(0x7f5010067bb0 msgr2=0x7f50100cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:17.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 -- 172.21.15.111:0/3736633018 shutdown_connections 2023-12-19T09:19:17.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.790+0000 7f5016878700 1 -- 172.21.15.111:0/3736633018 wait complete. 2023-12-19T09:19:17.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 Processor -- start 2023-12-19T09:19:17.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 -- start start 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 0x7f50100af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 0x7f50101598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50100b0aa0 con 0x7f50100c6120 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50100b0fa0 con 0x7f50100cc4e0 2023-12-19T09:19:17.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5016878700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5010154000 con 0x7f50101538c0 2023-12-19T09:19:17.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54834/0 (socket says 172.21.15.111:54834) 2023-12-19T09:19:17.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 -- 172.21.15.111:0/860720551 learned_addr learned my addr 172.21.15.111:0/860720551 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:17.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5014e15700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 0x7f50101598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5007fff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 0x7f50100af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 -- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 msgr2=0x7f50101598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 0x7f50101598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 -- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 msgr2=0x7f50100af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 0x7f50100af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 -- 172.21.15.111:0/860720551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5000009420 con 0x7f50100c6120 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5014e15700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 0x7f50101598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f5007fff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 0x7f50100af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500ffff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f5010068050 tx=0x7f5000017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000011900 con 0x7f50100c6120 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.794+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000011a60 con 0x7f50100c6120 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.798+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50100b11d0 con 0x7f50100c6120 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.798+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000025670 con 0x7f50100c6120 2023-12-19T09:19:17.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.798+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5010153c90 con 0x7f50100c6120 2023-12-19T09:19:17.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.798+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5010071120 con 0x7f50100c6120 2023-12-19T09:19:17.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.798+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 88) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f5000023070 con 0x7f50100c6120 2023-12-19T09:19:17.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.802+0000 7f500cff9700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 0x7f4ff0062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:17.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.802+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f500007e5a0 con 0x7f50100c6120 2023-12-19T09:19:17.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.802+0000 7f5007fff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 0x7f4ff0062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:17.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.802+0000 7f5007fff700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 0x7f4ff0062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5010159250 tx=0x7f4ff8009040 comp rx=0 tx=0).ready entity=mgr.9355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:17.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.806+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f500004e3e0 con 0x7f50100c6120 2023-12-19T09:19:18.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:17.997+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "/snap_test_dir1/test_restart", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f50101596c0 con 0x7f4ff00606c0 2023-12-19T09:19:18.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.109+0000 7f500cff9700 1 -- 172.21.15.111:0/860720551 <== mgr.9355 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f50101596c0 con 0x7f4ff00606c0 2023-12-19T09:19:18.114 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-12-19T09:19:18.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 msgr2=0x7f4ff0062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 0x7f4ff0062b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5010159250 tx=0x7f4ff8009040 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 msgr2=0x7f501015e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f5010068050 tx=0x7f5000017000 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 shutdown_connections 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4ff00606c0 0x7f4ff0062b70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50100cc4e0 0x7f50100af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50101538c0 0x7f50101598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 --2- 172.21.15.111:0/860720551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50100c6120 0x7f501015e390 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 >> 172.21.15.111:0/860720551 conn(0x7f5010067bb0 msgr2=0x7f501006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 shutdown_connections 2023-12-19T09:19:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:18.113+0000 7f5016878700 1 -- 172.21.15.111:0/860720551 wait complete. 2023-12-19T09:19:18.138 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-12-19T09:19:18.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.139 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-12-19T09:19:18.173 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_04_00_UTC 2023-12-19T09:19:18.173 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_05_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_06_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_07_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_08_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_09_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_10_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_11_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_12_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_13_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_14_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_15_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_16_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_17_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_18_00_UTC 2023-12-19T09:19:18.174 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_19_00_UTC 2023-12-19T09:19:18.175 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_04_00_UTC 2023-12-19T09:19:18.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.176 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_04_00_UTC) 2023-12-19T09:19:18.263 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_05_00_UTC 2023-12-19T09:19:18.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.263 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_05_00_UTC) 2023-12-19T09:19:18.342 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_06_00_UTC 2023-12-19T09:19:18.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.342 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_06_00_UTC) 2023-12-19T09:19:18.428 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_07_00_UTC 2023-12-19T09:19:18.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.429 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_07_00_UTC) 2023-12-19T09:19:18.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_08_00_UTC 2023-12-19T09:19:18.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.514 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_08_00_UTC) 2023-12-19T09:19:18.596 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_09_00_UTC 2023-12-19T09:19:18.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.596 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_09_00_UTC) 2023-12-19T09:19:18.682 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_10_00_UTC 2023-12-19T09:19:18.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.682 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_10_00_UTC) 2023-12-19T09:19:18.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_11_00_UTC 2023-12-19T09:19:18.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.763 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_11_00_UTC) 2023-12-19T09:19:18.846 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_12_00_UTC 2023-12-19T09:19:18.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.846 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_12_00_UTC) 2023-12-19T09:19:18.932 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_13_00_UTC 2023-12-19T09:19:18.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:18.932 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_13_00_UTC) 2023-12-19T09:19:19.022 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_14_00_UTC 2023-12-19T09:19:19.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.022 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_14_00_UTC) 2023-12-19T09:19:19.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_15_00_UTC 2023-12-19T09:19:19.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.102 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_15_00_UTC) 2023-12-19T09:19:19.179 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_16_00_UTC 2023-12-19T09:19:19.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.180 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_16_00_UTC) 2023-12-19T09:19:19.264 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_17_00_UTC 2023-12-19T09:19:19.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.264 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_17_00_UTC) 2023-12-19T09:19:19.354 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_18_00_UTC 2023-12-19T09:19:19.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.354 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_18_00_UTC) 2023-12-19T09:19:19.441 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_19_00_UTC 2023-12-19T09:19:19.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.442 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-12-19-09_19_00_UTC) 2023-12-19T09:19:19.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:19.528 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-12-19T09:19:19.612 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T09:19:19.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.817+0000 7f1d46b85700 1 Processor -- start 2023-12-19T09:19:19.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 -- start start 2023-12-19T09:19:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 0x7f1d4006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4006a7a0 con 0x7f1d4015aa30 2023-12-19T09:19:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d400c7630 con 0x7f1d400c6d20 2023-12-19T09:19:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d46b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4006d4b0 con 0x7f1d4006c7c0 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d44921700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d44921700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:51718/0 (socket says 172.21.15.111:51718) 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d44921700 1 -- 172.21.15.111:0/1532696575 learned_addr learned my addr 172.21.15.111:0/1532696575 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/1532696575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 0x7f1d4006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d45122700 1 --1- 172.21.15.111:0/1532696575 >> v1:172.21.15.143:6789/0 conn(0x7f1d4015aa30 0x7f1d4015ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:47174/0 (socket says 172.21.15.111:47174) 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d3ffff700 1 -- 172.21.15.111:0/1532696575 >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 msgr2=0x7f1d400c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/1532696575 >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d3ffff700 1 -- 172.21.15.111:0/1532696575 >> v1:172.21.15.143:6789/0 conn(0x7f1d4015aa30 legacy=0x7f1d4015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d3ffff700 1 -- 172.21.15.111:0/1532696575 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4006d790 con 0x7f1d4006c7c0 2023-12-19T09:19:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.821+0000 7f1d44921700 1 --2- 172.21.15.111:0/1532696575 >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/1532696575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 0x7f1d4006cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0098e0 tx=0x7f1d2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e213a41166ec60 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d3f7fe700 1 -- 172.21.15.111:0/1532696575 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00b640 con 0x7f1d4006c7c0 2023-12-19T09:19:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d3f7fe700 1 -- 172.21.15.111:0/1532696575 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00b7a0 con 0x7f1d4006c7c0 2023-12-19T09:19:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d3f7fe700 1 -- 172.21.15.111:0/1532696575 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00bb40 con 0x7f1d4006c7c0 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 -- 172.21.15.111:0/1532696575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 msgr2=0x7f1d4006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 --2- 172.21.15.111:0/1532696575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 0x7f1d4006cc60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0098e0 tx=0x7f1d2c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 -- 172.21.15.111:0/1532696575 shutdown_connections 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 --2- 172.21.15.111:0/1532696575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4006c7c0 0x7f1d4006cc60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 --2- 172.21.15.111:0/1532696575 >> v2:172.21.15.143:3300/0 conn(0x7f1d400c6d20 0x7f1d400c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 -- 172.21.15.111:0/1532696575 >> 172.21.15.111:0/1532696575 conn(0x7f1d40067bb0 msgr2=0x7f1d400cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 -- 172.21.15.111:0/1532696575 shutdown_connections 2023-12-19T09:19:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.825+0000 7f1d46b85700 1 -- 172.21.15.111:0/1532696575 wait complete. 2023-12-19T09:19:19.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 Processor -- start 2023-12-19T09:19:19.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 -- start start 2023-12-19T09:19:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 0x7f1d400abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4015aa30 0x7f1d4015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d40151c80 con 0x7f1d400c6d20 2023-12-19T09:19:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400ac300 con 0x7f1d4006c7c0 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d46b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400b10b0 con 0x7f1d4015aa30 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54844/0 (socket says 172.21.15.111:54844) 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d44921700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 0x7f1d400abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 -- 172.21.15.111:0/2844284045 learned_addr learned my addr 172.21.15.111:0/2844284045 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d45122700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4015aa30 0x7f1d4015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 -- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4015aa30 msgr2=0x7f1d4015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:19.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4015aa30 0x7f1d4015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 -- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 msgr2=0x7f1d400abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 0x7f1d400abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d3ffff700 1 -- 172.21.15.111:0/2844284045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d340094f0 con 0x7f1d400c6d20 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.829+0000 7f1d44921700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 0x7f1d400abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d3ffff700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f1d4006d5f0 tx=0x7f1d2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:19.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c01bc40 con 0x7f1d400c6d20 2023-12-19T09:19:19.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c01bdc0 con 0x7f1d400c6d20 2023-12-19T09:19:19.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c011670 con 0x7f1d400c6d20 2023-12-19T09:19:19.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c009420 con 0x7f1d400c6d20 2023-12-19T09:19:19.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d400b1670 con 0x7f1d400c6d20 2023-12-19T09:19:19.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.833+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d40071120 con 0x7f1d400c6d20 2023-12-19T09:19:19.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.837+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 88) v1 ==== 80092+0+0 (secure 0 0 0) 0x7f1d2c011a20 con 0x7f1d400c6d20 2023-12-19T09:19:19.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.837+0000 7f1d3cff9700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 0x7f1d24062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:19.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.837+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f1d2c07e4b0 con 0x7f1d400c6d20 2023-12-19T09:19:19.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.837+0000 7f1d44921700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 0x7f1d24062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:19.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.841+0000 7f1d44921700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 0x7f1d24062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d34006d20 tx=0x7f1d34009250 comp rx=0 tx=0).ready entity=mgr.9355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:19.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:19.849+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1d2c04e390 con 0x7f1d400c6d20 2023-12-19T09:19:20.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.061+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f1d400ae720 con 0x7f1d400c6d20 2023-12-19T09:19:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.857+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f1d2c04e760 con 0x7f1d400c6d20 2023-12-19T09:19:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.857+0000 7f1d3cff9700 1 -- 172.21.15.111:0/2844284045 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v89) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1d2c076a00 con 0x7f1d400c6d20 2023-12-19T09:19:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 msgr2=0x7f1d24062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 0x7f1d24062b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1d34006d20 tx=0x7f1d34009250 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 msgr2=0x7f1d400c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f1d4006d5f0 tx=0x7f1d2c017000 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 shutdown_connections 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d4015aa30 0x7f1d4015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d400c6d20 0x7f1d400c6210 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1d240606c0 0x7f1d24062b70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 --2- 172.21.15.111:0/2844284045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d4006c7c0 0x7f1d400abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 >> 172.21.15.111:0/2844284045 conn(0x7f1d40067bb0 msgr2=0x7f1d4006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 shutdown_connections 2023-12-19T09:19:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:20.861+0000 7f1d46b85700 1 -- 172.21.15.111:0/2844284045 wait complete. 2023-12-19T09:19:20.888 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:19:20.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:19:20.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:19:20.980 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:19:20.980 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:19:20.980 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:19:20.980 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:21.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[59550]: fuse finished with error 0 and tester_r 0 2023-12-19T09:19:21.014 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:19:21.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:21.014 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:19:21.053 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:21.049+0000 7f878278e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:19:21.063 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:21.060+0000 7f7733fc5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:19:21.194 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:19:21.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:21.194 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T09:19:21.215 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T09:19:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:19:21.216 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:19:21.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:21.217 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T09:19:21.255 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T09:19:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:19:21.257 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:19:21.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:21.257 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T09:19:21.278 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T09:19:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:19:21.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.465+0000 7f30b0985700 1 Processor -- start 2023-12-19T09:19:21.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.465+0000 7f30b0985700 1 -- start start 2023-12-19T09:19:21.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30b0985700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30ac140710 0x7f30ac0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30b0985700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac06e150 con 0x7f30ac140ae0 2023-12-19T09:19:21.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30b0985700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac06d540 con 0x7f30ac068800 2023-12-19T09:19:21.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30b0985700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac06d820 con 0x7f30ac140710 2023-12-19T09:19:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a9d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f30ac140ae0 0x7f30ac0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:59216/0 (socket says 172.21.15.111:59216) 2023-12-19T09:19:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a9d9b700 1 -- 172.21.15.111:0/933634595 learned_addr learned my addr 172.21.15.111:0/933634595 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30aa59c700 1 --2- 172.21.15.111:0/933634595 >> v2:172.21.15.143:3300/0 conn(0x7f30ac140710 0x7f30ac0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224888890 0 0) 0x7f30ac06e150 con 0x7f30ac140ae0 2023-12-19T09:19:21.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3094003570 con 0x7f30ac140ae0 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1496856907 0 0) 0x7f3094003570 con 0x7f30ac140ae0 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 >> v2:172.21.15.143:3300/0 conn(0x7f30ac140710 msgr2=0x7f30ac0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 --2- 172.21.15.111:0/933634595 >> v2:172.21.15.143:3300/0 conn(0x7f30ac140710 0x7f30ac0c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 >> v1:172.21.15.143:6789/0 conn(0x7f30ac068800 legacy=0x7f30ac0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac06db00 con 0x7f30ac140ae0 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f30a0001470 con 0x7f30ac140ae0 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 >> v1:172.21.15.111:6789/0 conn(0x7f30ac140ae0 legacy=0x7f30ac0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f3094003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3094004810 0x7f3094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30940051c0 0x7f309400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac06d820 con 0x7f3094003a60 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309400c0d0 con 0x7f3094004810 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309400c290 con 0x7f30940051c0 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.469+0000 7f30aa59c700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f3094003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 -- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30940051c0 msgr2=0x7f309400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30a9d9b700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3094004810 0x7f3094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30940051c0 0x7f309400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 -- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3094004810 msgr2=0x7f3094004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3094004810 0x7f3094004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 -- 172.21.15.111:0/933634595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094003570 con 0x7f3094003a60 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30a9d9b700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3094004810 0x7f3094004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aad9d700 1 -- 172.21.15.111:0/933634595 reap_dead start 2023-12-19T09:19:21.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30aa59c700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f3094003e70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f309c009c00 tx=0x7f309c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863b3327184ee5b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:21.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00e710 con 0x7f3094003a60 2023-12-19T09:19:21.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00e870 con 0x7f3094003a60 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30a959a700 1 -- 172.21.15.111:0/933634595 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00ec30 con 0x7f3094003a60 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 -- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 msgr2=0x7f3094003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f3094003e70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f309c009c00 tx=0x7f309c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 -- 172.21.15.111:0/933634595 shutdown_connections 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 --2- 172.21.15.111:0/933634595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f3094003e70 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 -- 172.21.15.111:0/933634595 >> 172.21.15.111:0/933634595 conn(0x7f30ac0c8850 msgr2=0x7f30ac0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 -- 172.21.15.111:0/933634595 shutdown_connections 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.473+0000 7f30b0985700 1 -- 172.21.15.111:0/933634595 wait complete. 2023-12-19T09:19:21.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 Processor -- start 2023-12-19T09:19:21.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- start start 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac0c2350 0x7f30ac0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac13ba80 0x7f30ac0a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c1ec0 con 0x7f3094003a60 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c12b0 con 0x7f30ac0c2350 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c1590 con 0x7f30ac13ba80 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54860/0 (socket says 172.21.15.111:54860) 2023-12-19T09:19:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 -- 172.21.15.111:0/3603836421 learned_addr learned my addr 172.21.15.111:0/3603836421 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30a9d9b700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac0c2350 0x7f30ac0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aad9d700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac13ba80 0x7f30ac0a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac13ba80 msgr2=0x7f30ac0a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac13ba80 0x7f30ac0a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac0c2350 msgr2=0x7f30ac0b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac0c2350 0x7f30ac0b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 -- 172.21.15.111:0/3603836421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0c1870 con 0x7f3094003a60 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30aa59c700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f309400c0a0 tx=0x7f309c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c022070 con 0x7f3094003a60 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00e9e0 con 0x7f3094003a60 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac0c1b50 con 0x7f3094003a60 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c0276c0 con 0x7f3094003a60 2023-12-19T09:19:21.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.477+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac13c080 con 0x7f3094003a60 2023-12-19T09:19:21.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac071120 con 0x7f3094003a60 2023-12-19T09:19:21.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f309c01c450 con 0x7f3094003a60 2023-12-19T09:19:21.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f309a7fc700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 0x7f308c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f309c07eaa0 con 0x7f3094003a60 2023-12-19T09:19:21.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f30a9d9b700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 msgr2=0x7f308c062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:21.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.481+0000 7f30a9d9b700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 0x7f308c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.489+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f309c04e9e0 con 0x7f3094003a60 2023-12-19T09:19:21.602 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:21.597+0000 7f878278e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:19:21.612 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:21.608+0000 7f7733fc5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:19:21.672 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:21.669+0000 7f878278e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:19:21.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.685+0000 7f30a9d9b700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 msgr2=0x7f308c062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:21.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.685+0000 7f30a9d9b700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 0x7f308c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:19:21.689 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:21.684+0000 7f7733fc5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:19:21.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.693+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30ac0642a0 con 0x7f3094003a60 2023-12-19T09:19:21.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.693+0000 7f309a7fc700 1 -- 172.21.15.111:0/3603836421 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7f309c006cb0 con 0x7f3094003a60 2023-12-19T09:19:21.698 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:21.699 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:19:21.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 msgr2=0x7f308c062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:21.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 0x7f308c062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 msgr2=0x7f30ac0c0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f309400c0a0 tx=0x7f309c01a000 comp rx=0 tx=0).stop 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 shutdown_connections 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f308c0606a0 0x7f308c062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac0c2350 0x7f30ac0b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac13ba80 0x7f30ac0a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 --2- 172.21.15.111:0/3603836421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3094003a60 0x7f30ac0c0bb0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 >> 172.21.15.111:0/3603836421 conn(0x7f30ac0c8850 msgr2=0x7f30ac0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 shutdown_connections 2023-12-19T09:19:21.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.697+0000 7f30b0985700 1 -- 172.21.15.111:0/3603836421 wait complete. 2023-12-19T09:19:21.704 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:19:21.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:21.816 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:21.809+0000 7f878278e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:21.835 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:21.832+0000 7f7733fc5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:21.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.869+0000 7f3b8cc0c700 1 Processor -- start 2023-12-19T09:19:21.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.869+0000 7f3b8cc0c700 1 -- start start 2023-12-19T09:19:21.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b8806de20 0x7f3b880c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3b8806c7c0 0x7f3b880c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8806d990 con 0x7f3b88155960 2023-12-19T09:19:21.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8806cd80 con 0x7f3b8806de20 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8806d060 con 0x7f3b8806c7c0 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b8806de20 0x7f3b880c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b8806de20 0x7f3b880c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55676/0 (socket says 172.21.15.111:55676) 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 -- 172.21.15.111:0/2048440021 learned_addr learned my addr 172.21.15.111:0/2048440021 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b87fff700 1 --2- 172.21.15.111:0/2048440021 >> v2:172.21.15.143:3300/0 conn(0x7f3b8806c7c0 0x7f3b880c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 -- 172.21.15.111:0/2048440021 >> v2:172.21.15.143:3300/0 conn(0x7f3b8806c7c0 msgr2=0x7f3b880c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 --2- 172.21.15.111:0/2048440021 >> v2:172.21.15.143:3300/0 conn(0x7f3b8806c7c0 0x7f3b880c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 -- 172.21.15.111:0/2048440021 >> v1:172.21.15.143:6789/0 conn(0x7f3b88155960 legacy=0x7f3b8815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 -- 172.21.15.111:0/2048440021 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8806d340 con 0x7f3b8806de20 2023-12-19T09:19:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b86ffd700 1 --2- 172.21.15.111:0/2048440021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b880c77f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f3b8806a380 tx=0x7f3b70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca92304f5d05f8e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b867fc700 1 -- 172.21.15.111:0/2048440021 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70009e70 con 0x7f3b8806de20 2023-12-19T09:19:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b867fc700 1 -- 172.21.15.111:0/2048440021 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7000b860 con 0x7f3b8806de20 2023-12-19T09:19:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b867fc700 1 -- 172.21.15.111:0/2048440021 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000bc00 con 0x7f3b8806de20 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.873+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/2048440021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 msgr2=0x7f3b880c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/2048440021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b880c77f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f3b8806a380 tx=0x7f3b70009250 comp rx=0 tx=0).stop 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/2048440021 shutdown_connections 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/2048440021 >> v2:172.21.15.143:3300/0 conn(0x7f3b8806c7c0 0x7f3b880c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/2048440021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b880c77f0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/2048440021 >> 172.21.15.111:0/2048440021 conn(0x7f3b88067bb0 msgr2=0x7f3b880cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/2048440021 shutdown_connections 2023-12-19T09:19:21.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/2048440021 wait complete. 2023-12-19T09:19:21.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 Processor -- start 2023-12-19T09:19:21.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.877+0000 7f3b8cc0c700 1 -- start start 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b8806c7c0 0x7f3b88154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b88155960 0x7f3b88146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88153c00 con 0x7f3b8806de20 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88152ff0 con 0x7f3b88155960 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b881532d0 con 0x7f3b8806c7c0 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55678/0 (socket says 172.21.15.111:55678) 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 -- 172.21.15.111:0/3861368151 learned_addr learned my addr 172.21.15.111:0/3861368151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b87fff700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b88155960 0x7f3b88146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b877fe700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b8806c7c0 0x7f3b88154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 -- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b8806c7c0 msgr2=0x7f3b88154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b8806c7c0 0x7f3b88154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 -- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b88155960 msgr2=0x7f3b88146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b88155960 0x7f3b88146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 -- 172.21.15.111:0/3861368151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b700094f0 con 0x7f3b8806de20 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b86ffd700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3b88064040 tx=0x7f3b70017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000bbe0 con 0x7f3b8806de20 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b881535b0 con 0x7f3b8806de20 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b70011420 con 0x7f3b8806de20 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700117e0 con 0x7f3b8806de20 2023-12-19T09:19:21.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.881+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b88155040 con 0x7f3b8806de20 2023-12-19T09:19:21.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.885+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b88153d70 con 0x7f3b8806de20 2023-12-19T09:19:21.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.885+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f3b70011c10 con 0x7f3b8806de20 2023-12-19T09:19:21.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.885+0000 7f3b7f7fe700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 0x7f3b68062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:21.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.889+0000 7f3b877fe700 1 -- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 msgr2=0x7f3b68062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:21.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.889+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f3b7007e7d0 con 0x7f3b8806de20 2023-12-19T09:19:21.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.889+0000 7f3b877fe700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 0x7f3b68062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:21.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:21.893+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3b7004e690 con 0x7f3b8806de20 2023-12-19T09:19:22.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.065+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b8804be40 con 0x7f3b8806de20 2023-12-19T09:19:22.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.065+0000 7f3b7f7fe700 1 -- 172.21.15.111:0/3861368151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+17424 (secure 0 0 0) 0x7f3b70051cb0 con 0x7f3b8806de20 2023-12-19T09:19:22.071 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:22.071 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":179,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:14:20.977223+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.12","mappings":[{"from":3,"to":0}]},{"pgid":"22.13","mappings":[{"from":3,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2893890504":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3589389202":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/57208669":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2193897545":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3287144757":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2770672326":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/4214132662":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3599201417":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2998303494":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/1690815342":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3967653215":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2978228531":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6833/33796":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6832/33796":"2023-12-20T09:03:31.913274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 msgr2=0x7f3b68062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 0x7f3b68062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 msgr2=0x7f3b88149360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3b88064040 tx=0x7f3b70017000 comp rx=0 tx=0).stop 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 shutdown_connections 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b68060670 0x7f3b68062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b88155960 0x7f3b88146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b8806de20 0x7f3b88149360 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 --2- 172.21.15.111:0/3861368151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b8806c7c0 0x7f3b88154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 >> 172.21.15.111:0/3861368151 conn(0x7f3b88067bb0 msgr2=0x7f3b880ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:22.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 shutdown_connections 2023-12-19T09:19:22.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.069+0000 7f3b8cc0c700 1 -- 172.21.15.111:0/3861368151 wait complete. 2023-12-19T09:19:22.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T09:03:15.680577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T09:03:16.714251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T09:19:22.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:19:22.096 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:22.093+0000 7f878278e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:19:22.123 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:22.120+0000 7f7733fc5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:19:22.169 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:22.165+0000 7f878278e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:22.203 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:22.200+0000 7f7733fc5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:22.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.261+0000 7fda7b185700 1 Processor -- start 2023-12-19T09:19:22.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.261+0000 7fda7b185700 1 -- start start 2023-12-19T09:19:22.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda7b185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda740c6d20 0x7fda740c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda7b185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda7406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda7b185700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7406a7a0 con 0x7fda7415aa30 2023-12-19T09:19:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda7b185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda740c7630 con 0x7fda740c6d20 2023-12-19T09:19:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda7b185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406d4b0 con 0x7fda7406c7c0 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda79722700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fda7415aa30 0x7fda7415ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49632/0 (socket says 172.21.15.111:49632) 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda79722700 1 -- 172.21.15.111:0/995623629 learned_addr learned my addr 172.21.15.111:0/995623629 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda73fff700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda7406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> v2:172.21.15.143:3300/0 conn(0x7fda740c6d20 0x7fda740c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386777153 0 0) 0x7fda7406a7a0 con 0x7fda7415aa30 2023-12-19T09:19:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda64003570 con 0x7fda7415aa30 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116607084 0 0) 0x7fda64003570 con 0x7fda7415aa30 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 msgr2=0x7fda7406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda7406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 >> v2:172.21.15.143:3300/0 conn(0x7fda740c6d20 msgr2=0x7fda740c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 --2- 172.21.15.111:0/995623629 >> v2:172.21.15.143:3300/0 conn(0x7fda740c6d20 0x7fda740c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7406d790 con 0x7fda7415aa30 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fda6c0017f0 con 0x7fda7415aa30 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> v2:172.21.15.143:3300/0 conn(0x7fda740c6d20 0x7fda740c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda73fff700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda7406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 >> v1:172.21.15.111:6789/0 conn(0x7fda7415aa30 legacy=0x7fda7415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda64003a60 0x7fda64003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda64004810 0x7fda64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740c7630 con 0x7fda64003a60 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406d4b0 con 0x7fda64004810 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6400ba60 con 0x7fda7406c7c0 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda64003a60 0x7fda64003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda73fff700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda64004810 0x7fda64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda79722700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 -- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 msgr2=0x7fda640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda7406c7c0 0x7fda640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 -- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda64004810 msgr2=0x7fda64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda64004810 0x7fda64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.265+0000 7fda78f21700 1 -- 172.21.15.111:0/995623629 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda60009420 con 0x7fda64003a60 2023-12-19T09:19:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda78f21700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda64003a60 0x7fda64003e70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fda68009df0 tx=0x7fda680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4e152be168faaa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6800e920 con 0x7fda64003a60 2023-12-19T09:19:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6800ea80 con 0x7fda64003a60 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda737fe700 1 -- 172.21.15.111:0/995623629 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6800ee20 con 0x7fda64003a60 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 -- 172.21.15.111:0/995623629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda64003a60 msgr2=0x7fda64003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 --2- 172.21.15.111:0/995623629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda64003a60 0x7fda64003e70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fda68009df0 tx=0x7fda680091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda79722700 1 -- 172.21.15.111:0/995623629 reap_dead start 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 -- 172.21.15.111:0/995623629 shutdown_connections 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 -- 172.21.15.111:0/995623629 >> 172.21.15.111:0/995623629 conn(0x7fda74067bb0 msgr2=0x7fda740cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 -- 172.21.15.111:0/995623629 shutdown_connections 2023-12-19T09:19:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.269+0000 7fda7b185700 1 -- 172.21.15.111:0/995623629 wait complete. 2023-12-19T09:19:22.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 Processor -- start 2023-12-19T09:19:22.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 -- start start 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda64003a60 0x7fda74167900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda740b0b20 0x7fda7413e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740abf20 con 0x7fda740aeba0 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740b1ba0 con 0x7fda740b0b20 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda7b185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda74167e40 con 0x7fda64003a60 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55704/0 (socket says 172.21.15.111:55704) 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 -- 172.21.15.111:0/3000160758 learned_addr learned my addr 172.21.15.111:0/3000160758 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:22.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda78f21700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda64003a60 0x7fda74167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda73fff700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda740b0b20 0x7fda7413e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 -- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda64003a60 msgr2=0x7fda74167900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda64003a60 0x7fda74167900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 -- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda740b0b20 msgr2=0x7fda7413e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda740b0b20 0x7fda7413e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 -- 172.21.15.111:0/3000160758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda740af510 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.273+0000 7fda79722700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fda6c007a90 tx=0x7fda6c002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c00ad00 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c00ae60 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda740b1300 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c013620 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda740b18c0 con 0x7fda740aeba0 2023-12-19T09:19:22.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda74071120 con 0x7fda740aeba0 2023-12-19T09:19:22.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.277+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fda6c013a10 con 0x7fda740aeba0 2023-12-19T09:19:22.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.281+0000 7fda70ff9700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 0x7fda58062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.281+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7fda6c07d8a0 con 0x7fda740aeba0 2023-12-19T09:19:22.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.281+0000 7fda78f21700 1 -- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 msgr2=0x7fda58062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:22.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.281+0000 7fda78f21700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 0x7fda58062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:22.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.285+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fda6c04d6e0 con 0x7fda740aeba0 2023-12-19T09:19:22.387 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:22.381+0000 7f878278e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:19:22.426 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:22.424+0000 7f7733fc5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:19:22.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.457+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda740ac060 con 0x7fda740aeba0 2023-12-19T09:19:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.457+0000 7fda70ff9700 1 -- 172.21.15.111:0/3000160758 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda6c050d00 con 0x7fda740aeba0 2023-12-19T09:19:22.463 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 msgr2=0x7fda58062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 0x7fda58062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 msgr2=0x7fda740aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fda6c007a90 tx=0x7fda6c002210 comp rx=0 tx=0).stop 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 shutdown_connections 2023-12-19T09:19:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda58060670 0x7fda58062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda740b0b20 0x7fda7413e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda740aeba0 0x7fda740aefd0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 --2- 172.21.15.111:0/3000160758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda64003a60 0x7fda74167900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 >> 172.21.15.111:0/3000160758 conn(0x7fda74067bb0 msgr2=0x7fda74157990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 shutdown_connections 2023-12-19T09:19:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.461+0000 7fda7b185700 1 -- 172.21.15.111:0/3000160758 wait complete. 2023-12-19T09:19:22.475 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:22.469+0000 7f878278e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:19:22.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T09:19:22.513 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:22.512+0000 7f7733fc5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:19:22.598 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:22.593+0000 7f878278e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:19:22.601 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:22.600+0000 7f7733fc5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:19:22.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.641+0000 7f48f266b700 1 Processor -- start 2023-12-19T09:19:22.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.645+0000 7f48f266b700 1 -- start start 2023-12-19T09:19:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.645+0000 7f48f266b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f48ec0acf30 0x7f48ec1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec156780 con 0x7f48ec06d1d0 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec158f90 con 0x7f48ec0ad300 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec1532a0 con 0x7f48ec0acf30 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1e6a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f48ec06d1d0 0x7f48ec0ac1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49636/0 (socket says 172.21.15.111:49636) 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1e6a700 1 -- 172.21.15.111:0/4053629071 learned_addr learned my addr 172.21.15.111:0/4053629071 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1669700 1 --2- 172.21.15.111:0/4053629071 >> v2:172.21.15.143:3300/0 conn(0x7f48ec0acf30 0x7f48ec1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48e3fff700 1 -- 172.21.15.111:0/4053629071 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536393294 0 0) 0x7f48ec156780 con 0x7f48ec06d1d0 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48e3fff700 1 -- 172.21.15.111:0/4053629071 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d8004680 con 0x7f48ec06d1d0 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1669700 1 -- 172.21.15.111:0/4053629071 >> v1:172.21.15.143:6789/0 conn(0x7f48ec0ad300 legacy=0x7f48ec159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1669700 1 -- 172.21.15.111:0/4053629071 >> v1:172.21.15.111:6789/0 conn(0x7f48ec06d1d0 legacy=0x7f48ec0ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1669700 1 -- 172.21.15.111:0/4053629071 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec06d630 con 0x7f48ec0acf30 2023-12-19T09:19:22.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f1669700 1 --2- 172.21.15.111:0/4053629071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec1557d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=111af4b1bd0224a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48e3fff700 1 -- 172.21.15.111:0/4053629071 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800f6b0 con 0x7f48ec0acf30 2023-12-19T09:19:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48e3fff700 1 -- 172.21.15.111:0/4053629071 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e800f810 con 0x7f48ec0acf30 2023-12-19T09:19:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48e3fff700 1 -- 172.21.15.111:0/4053629071 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800fbb0 con 0x7f48ec0acf30 2023-12-19T09:19:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- 172.21.15.111:0/4053629071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 msgr2=0x7f48ec1557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 --2- 172.21.15.111:0/4053629071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec1557d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e80091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- 172.21.15.111:0/4053629071 shutdown_connections 2023-12-19T09:19:22.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 --2- 172.21.15.111:0/4053629071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec1557d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.649+0000 7f48f266b700 1 -- 172.21.15.111:0/4053629071 >> 172.21.15.111:0/4053629071 conn(0x7f48ec067b50 msgr2=0x7f48ec0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:22.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- 172.21.15.111:0/4053629071 shutdown_connections 2023-12-19T09:19:22.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- 172.21.15.111:0/4053629071 wait complete. 2023-12-19T09:19:22.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 Processor -- start 2023-12-19T09:19:22.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- start start 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48ec06d1d0 0x7f48ec156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec0c96b0 con 0x7f48ec0ad300 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec156f10 con 0x7f48ec0acf30 2023-12-19T09:19:22.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f266b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec1571f0 con 0x7f48ec06d1d0 2023-12-19T09:19:22.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f1e6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f1e6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55708/0 (socket says 172.21.15.111:55708) 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.653+0000 7f48f1e6a700 1 -- 172.21.15.111:0/2544433887 learned_addr learned my addr 172.21.15.111:0/2544433887 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1669700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48ec06d1d0 0x7f48ec156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f0e68700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 -- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48ec06d1d0 msgr2=0x7f48ec156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48ec06d1d0 0x7f48ec156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 -- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 msgr2=0x7f48ec0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 -- 172.21.15.111:0/2544433887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009420 con 0x7f48ec0ad300 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f0e68700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f1e6a700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001460 tx=0x7f48e4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e400a950 con 0x7f48ec0ad300 2023-12-19T09:19:22.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e400aab0 con 0x7f48ec0ad300 2023-12-19T09:19:22.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e400ae30 con 0x7f48ec0ad300 2023-12-19T09:19:22.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec1574d0 con 0x7f48ec0ad300 2023-12-19T09:19:22.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec157a90 con 0x7f48ec0ad300 2023-12-19T09:19:22.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.657+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec157cb0 con 0x7f48ec0ad300 2023-12-19T09:19:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.661+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f48e4007300 con 0x7f48ec0ad300 2023-12-19T09:19:22.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.661+0000 7f48e17fa700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 0x7f48d0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:22.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.661+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 7233+0+0 (secure 0 0 0) 0x7f48e407d490 con 0x7f48ec0ad300 2023-12-19T09:19:22.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.661+0000 7f48f1669700 1 -- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 msgr2=0x7f48d0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:22.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.661+0000 7f48f1669700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 0x7f48d0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:22.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.669+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f48e404d2c0 con 0x7f48ec0ad300 2023-12-19T09:19:22.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f48ec04be40 con 0x7f48ec0ad300 2023-12-19T09:19:22.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48e17fa700 1 -- 172.21.15.111:0/2544433887 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v179) v1 ==== 104+0+11 (secure 0 0 0) 0x7f48e40508e0 con 0x7f48ec0ad300 2023-12-19T09:19:22.845 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 msgr2=0x7f48d0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 0x7f48d0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 msgr2=0x7f48ec157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.841+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f48e4001460 tx=0x7f48e4001a40 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 shutdown_connections 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f48d0060670 0x7f48d0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec0acf30 0x7f48ec0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec0ad300 0x7f48ec157fb0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 --2- 172.21.15.111:0/2544433887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48ec06d1d0 0x7f48ec156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 >> 172.21.15.111:0/2544433887 conn(0x7f48ec067b50 msgr2=0x7f48ec14b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 shutdown_connections 2023-12-19T09:19:22.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:22.845+0000 7f48f266b700 1 -- 172.21.15.111:0/2544433887 wait complete. 2023-12-19T09:19:22.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T09:19:23.020 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:23.017+0000 7f878278e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:19:23.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.025+0000 7f4875216700 1 Processor -- start 2023-12-19T09:19:23.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.025+0000 7f4875216700 1 -- start start 2023-12-19T09:19:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4875216700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4870158c90 0x7f48700caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4875216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 0x7f487006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4875216700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487006c850 con 0x7f487006aeb0 2023-12-19T09:19:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4875216700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487006b470 con 0x7f4870158c90 2023-12-19T09:19:23.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4875216700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487006b750 con 0x7f4870159770 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f487006aeb0 0x7f4870159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49642/0 (socket says 172.21.15.111:49642) 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486f7fe700 1 -- 172.21.15.111:0/3934068310 learned_addr learned my addr 172.21.15.111:0/3934068310 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4874a15700 1 --2- 172.21.15.111:0/3934068310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 0x7f487006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486ffff700 1 --2- 172.21.15.111:0/3934068310 >> v2:172.21.15.143:3300/0 conn(0x7f4870158c90 0x7f48700caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631127090 0 0) 0x7f487006c850 con 0x7f487006aeb0 2023-12-19T09:19:23.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4858003570 con 0x7f487006aeb0 2023-12-19T09:19:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730156554 0 0) 0x7f4858003570 con 0x7f487006aeb0 2023-12-19T09:19:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4874a15700 1 -- 172.21.15.111:0/3934068310 >> v2:172.21.15.143:3300/0 conn(0x7f4870158c90 msgr2=0x7f48700caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4874a15700 1 --2- 172.21.15.111:0/3934068310 >> v2:172.21.15.143:3300/0 conn(0x7f4870158c90 0x7f48700caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4874a15700 1 -- 172.21.15.111:0/3934068310 >> v1:172.21.15.111:6789/0 conn(0x7f487006aeb0 legacy=0x7f4870159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.029+0000 7f4874a15700 1 -- 172.21.15.111:0/3934068310 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487006ba30 con 0x7f4870159770 2023-12-19T09:19:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4874a15700 1 --2- 172.21.15.111:0/3934068310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 0x7f487006cce0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f48680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e5f1624700ae62 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864001470 con 0x7f4870159770 2023-12-19T09:19:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868011420 con 0x7f4870159770 2023-12-19T09:19:23.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f486effd700 1 -- 172.21.15.111:0/3934068310 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680117c0 con 0x7f4870159770 2023-12-19T09:19:23.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 -- 172.21.15.111:0/3934068310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 msgr2=0x7f487006cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 --2- 172.21.15.111:0/3934068310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 0x7f487006cce0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f48680091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 -- 172.21.15.111:0/3934068310 shutdown_connections 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 --2- 172.21.15.111:0/3934068310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870159770 0x7f487006cce0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 --2- 172.21.15.111:0/3934068310 >> v2:172.21.15.143:3300/0 conn(0x7f4870158c90 0x7f48700caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 -- 172.21.15.111:0/3934068310 >> 172.21.15.111:0/3934068310 conn(0x7f4870067bb0 msgr2=0x7f48700cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 -- 172.21.15.111:0/3934068310 shutdown_connections 2023-12-19T09:19:23.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.033+0000 7f4875216700 1 -- 172.21.15.111:0/3934068310 wait complete. 2023-12-19T09:19:23.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 Processor -- start 2023-12-19T09:19:23.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 -- start start 2023-12-19T09:19:23.041 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:23.040+0000 7f7733fc5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:19:23.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f487006aeb0 0x7f48701469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870158c90 0x7f48700b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870146f20 con 0x7f4870159770 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487015d7e0 con 0x7f487006aeb0 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4875216700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487015da80 con 0x7f4870158c90 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55714/0 (socket says 172.21.15.111:55714) 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 -- 172.21.15.111:0/3539061419 learned_addr learned my addr 172.21.15.111:0/3539061419 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f486f7fe700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870158c90 0x7f48700b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f486ffff700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f487006aeb0 0x7f48701469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 -- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870158c90 msgr2=0x7f48700b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870158c90 0x7f48700b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 -- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f487006aeb0 msgr2=0x7f48701469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f487006aeb0 0x7f48701469e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.037+0000 7f4874a15700 1 -- 172.21.15.111:0/3539061419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868009420 con 0x7f4870159770 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4874a15700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f48680093f0 tx=0x7f4868016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486801cc90 con 0x7f4870159770 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486801cdf0 con 0x7f4870159770 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486801a680 con 0x7f4870159770 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487015dd60 con 0x7f4870159770 2023-12-19T09:19:23.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487016a1e0 con 0x7f4870159770 2023-12-19T09:19:23.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487015e260 con 0x7f4870159770 2023-12-19T09:19:23.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.041+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f486801aa70 con 0x7f4870159770 2023-12-19T09:19:23.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.045+0000 7f4857fff700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 0x7f485c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.045+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7f486807e980 con 0x7f4870159770 2023-12-19T09:19:23.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.045+0000 7f486ffff700 1 -- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 msgr2=0x7f485c062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:23.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.045+0000 7f486ffff700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 0x7f485c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:23.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.049+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f486804e830 con 0x7f4870159770 2023-12-19T09:19:23.194 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:23.189+0000 7f878278e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:19:23.215 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:23.212+0000 7f7733fc5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:19:23.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.217+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f487004be40 con 0x7f4870159770 2023-12-19T09:19:23.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.217+0000 7f4857fff700 1 -- 172.21.15.111:0/3539061419 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v180) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48680290a0 con 0x7f4870159770 2023-12-19T09:19:23.222 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 msgr2=0x7f485c062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 0x7f485c062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 msgr2=0x7f48700b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f48680093f0 tx=0x7f4868016000 comp rx=0 tx=0).stop 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 shutdown_connections 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4870159770 0x7f48700b17c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4870158c90 0x7f48700b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f485c060670 0x7f485c062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 --2- 172.21.15.111:0/3539061419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f487006aeb0 0x7f48701469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.221+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 >> 172.21.15.111:0/3539061419 conn(0x7f4870067bb0 msgr2=0x7f48700afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:23.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.225+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 shutdown_connections 2023-12-19T09:19:23.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.225+0000 7f4875216700 1 -- 172.21.15.111:0/3539061419 wait complete. 2023-12-19T09:19:23.254 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:23.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.401+0000 7fc7e1723700 1 Processor -- start 2023-12-19T09:19:23.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 -- start start 2023-12-19T09:19:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 0x7fc7dc0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc06b470 con 0x7fc7dc06b900 2023-12-19T09:19:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc06e110 con 0x7fc7dc0c6900 2023-12-19T09:19:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e1723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc161d10 con 0x7fc7dc15cc20 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7dbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 0x7fc7dc0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7dbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 0x7fc7dc0c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55720/0 (socket says 172.21.15.111:55720) 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7dbfff700 1 -- 172.21.15.111:0/3970943073 learned_addr learned my addr 172.21.15.111:0/3970943073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7db7fe700 1 --1- 172.21.15.111:0/3970943073 >> v1:172.21.15.143:6789/0 conn(0x7fc7dc06b900 0x7fc7dc06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:35564/0 (socket says 172.21.15.111:35564) 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.405+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/3970943073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7daffd700 1 -- 172.21.15.111:0/3970943073 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768687484 0 0) 0x7fc7dc06b470 con 0x7fc7dc06b900 2023-12-19T09:19:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7daffd700 1 -- 172.21.15.111:0/3970943073 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8003570 con 0x7fc7dc06b900 2023-12-19T09:19:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e0f22700 1 -- 172.21.15.111:0/3970943073 >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 msgr2=0x7fc7dc0c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/3970943073 >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 0x7fc7dc0c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e0f22700 1 -- 172.21.15.111:0/3970943073 >> v1:172.21.15.143:6789/0 conn(0x7fc7dc06b900 legacy=0x7fc7dc06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e0f22700 1 -- 172.21.15.111:0/3970943073 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc06bd70 con 0x7fc7dc15cc20 2023-12-19T09:19:23.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/3970943073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc15d0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009e60 tx=0x7fc7d4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42ddea98e36e619 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7daffd700 1 -- 172.21.15.111:0/3970943073 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400e070 con 0x7fc7dc15cc20 2023-12-19T09:19:23.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7daffd700 1 -- 172.21.15.111:0/3970943073 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d400bab0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7daffd700 1 -- 172.21.15.111:0/3970943073 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400be70 con 0x7fc7dc15cc20 2023-12-19T09:19:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 -- 172.21.15.111:0/3970943073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 msgr2=0x7fc7dc15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 --2- 172.21.15.111:0/3970943073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc15d0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009e60 tx=0x7fc7d4009780 comp rx=0 tx=0).stop 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 -- 172.21.15.111:0/3970943073 shutdown_connections 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 --2- 172.21.15.111:0/3970943073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc15d0a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 --2- 172.21.15.111:0/3970943073 >> v2:172.21.15.111:3300/0 conn(0x7fc7dc0c6900 0x7fc7dc0c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 -- 172.21.15.111:0/3970943073 >> 172.21.15.111:0/3970943073 conn(0x7fc7dc067bb0 msgr2=0x7fc7dc0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 -- 172.21.15.111:0/3970943073 shutdown_connections 2023-12-19T09:19:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.409+0000 7fc7e1723700 1 -- 172.21.15.111:0/3970943073 wait complete. 2023-12-19T09:19:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 Processor -- start 2023-12-19T09:19:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 -- start start 2023-12-19T09:19:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 0x7fc7dc14dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 0x7fc7dc16d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc1717a0 con 0x7fc7dc06b900 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc15e9e0 con 0x7fc7dc0c6900 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.413+0000 7fc7e1723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc15daf0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7db7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 0x7fc7dc16d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7db7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 0x7fc7dc16d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49224/0 (socket says 172.21.15.111:49224) 2023-12-19T09:19:23.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7db7fe700 1 -- 172.21.15.111:0/1079111051 learned_addr learned my addr 172.21.15.111:0/1079111051 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7dbfff700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 0x7fc7dc14dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 msgr2=0x7fc7dc16d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 0x7fc7dc16d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 msgr2=0x7fc7dc14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 0x7fc7dc14dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 -- 172.21.15.111:0/1079111051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4009a20 con 0x7fc7dc15cc20 2023-12-19T09:19:23.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7dbfff700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 0x7fc7dc14dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e0f22700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc1602d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4016000 tx=0x7fc7d4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4027470 con 0x7fc7dc15cc20 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d40275f0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc15ddd0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40279b0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.417+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc15e390 con 0x7fc7dc15cc20 2023-12-19T09:19:23.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.421+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc15eb70 con 0x7fc7dc15cc20 2023-12-19T09:19:23.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.421+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fc7d4024070 con 0x7fc7dc15cc20 2023-12-19T09:19:23.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.421+0000 7fc7d3fff700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 0x7fc7bc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.425+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7fc7d407f530 con 0x7fc7dc15cc20 2023-12-19T09:19:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.425+0000 7fc7dbfff700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 msgr2=0x7fc7bc062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.425+0000 7fc7dbfff700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 0x7fc7bc062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.429+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc7d404f3e0 con 0x7fc7dc15cc20 2023-12-19T09:19:23.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.629+0000 7fc7dbfff700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 msgr2=0x7fc7bc062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:23.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.629+0000 7fc7dbfff700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 0x7fc7bc062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:19:23.645 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:23.641+0000 7f878278e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:19:23.645 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:23.644+0000 7f7733fc5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:19:23.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.697+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7dc04be40 con 0x7fc7dc15cc20 2023-12-19T09:19:23.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.697+0000 7fc7d3fff700 1 -- 172.21.15.111:0/1079111051 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7fc7d4052a00 con 0x7fc7dc15cc20 2023-12-19T09:19:23.704 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:23.705 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 msgr2=0x7fc7bc062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 0x7fc7bc062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 msgr2=0x7fc7dc1602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc1602d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4016000 tx=0x7fc7d4019000 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 shutdown_connections 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc7bc0606a0 0x7fc7bc062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7dc0c6900 0x7fc7dc16d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7dc15cc20 0x7fc7dc1602d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 --2- 172.21.15.111:0/1079111051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7dc06b900 0x7fc7dc14dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.701+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 >> 172.21.15.111:0/1079111051 conn(0x7fc7dc067bb0 msgr2=0x7fc7dc0ab9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:23.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.705+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 shutdown_connections 2023-12-19T09:19:23.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.705+0000 7fc7e1723700 1 -- 172.21.15.111:0/1079111051 wait complete. 2023-12-19T09:19:23.710 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:19:23.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:23.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 Processor -- start 2023-12-19T09:19:23.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 -- start start 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28d4156d10 0x7f28d40cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40d0880 con 0x7f28d4156940 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d406d1a0 con 0x7f28d4156d10 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.869+0000 7f28d94f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40cfc70 con 0x7f28d406c7c0 2023-12-19T09:19:23.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d2ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f28d4156940 0x7f28d4159f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49652/0 (socket says 172.21.15.111:49652) 2023-12-19T09:19:23.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d2ffd700 1 -- 172.21.15.111:0/4248631674 learned_addr learned my addr 172.21.15.111:0/4248631674 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:23.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 --2- 172.21.15.111:0/4248631674 >> v2:172.21.15.111:3300/0 conn(0x7f28d4156d10 0x7f28d40cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d3fff700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 -- 172.21.15.111:0/4248631674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 msgr2=0x7f28d406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 -- 172.21.15.111:0/4248631674 >> v1:172.21.15.111:6789/0 conn(0x7f28d4156940 legacy=0x7f28d4159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 -- 172.21.15.111:0/4248631674 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40cff50 con 0x7f28d4156d10 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d3fff700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:23.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.873+0000 7f28d27fc700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40cf690 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f28c80098e0 tx=0x7f28c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44a798db9bd932c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d1ffb700 1 -- 172.21.15.111:0/4248631674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800b640 con 0x7f28d4156d10 2023-12-19T09:19:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d1ffb700 1 -- 172.21.15.111:0/4248631674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c800b7a0 con 0x7f28d4156d10 2023-12-19T09:19:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d1ffb700 1 -- 172.21.15.111:0/4248631674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800bb40 con 0x7f28d4156d10 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 -- 172.21.15.111:0/4248631674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 msgr2=0x7f28d40cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40cf690 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f28c80098e0 tx=0x7f28c80091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 -- 172.21.15.111:0/4248631674 shutdown_connections 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 --2- 172.21.15.111:0/4248631674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40cf690 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 -- 172.21.15.111:0/4248631674 >> 172.21.15.111:0/4248631674 conn(0x7f28d40ce320 msgr2=0x7f28d40ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 -- 172.21.15.111:0/4248631674 shutdown_connections 2023-12-19T09:19:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.877+0000 7f28d94f0700 1 -- 172.21.15.111:0/4248631674 wait complete. 2023-12-19T09:19:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 Processor -- start 2023-12-19T09:19:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 -- start start 2023-12-19T09:19:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d415e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 0x7f28d4151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40c96b0 con 0x7f28d4156d10 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d416a400 con 0x7f28d4156940 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d94f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4151960 con 0x7f28d406c7c0 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55748/0 (socket says 172.21.15.111:55748) 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 -- 172.21.15.111:0/3980424324 learned_addr learned my addr 172.21.15.111:0/3980424324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d2ffd700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d27fc700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 0x7f28d4151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 -- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 msgr2=0x7f28d415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 -- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 msgr2=0x7f28d4151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 0x7f28d4151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d3fff700 1 -- 172.21.15.111:0/3980424324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc009a20 con 0x7f28d4156d10 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d2ffd700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.881+0000 7f28d27fc700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 0x7f28d4151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28d3fff700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009fc0 tx=0x7f28cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc00cd30 con 0x7f28d4156d10 2023-12-19T09:19:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc00f040 con 0x7f28d4156d10 2023-12-19T09:19:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8009420 con 0x7f28d4156d10 2023-12-19T09:19:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0146d0 con 0x7f28d4156d10 2023-12-19T09:19:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4169db0 con 0x7f28d4156d10 2023-12-19T09:19:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d416a570 con 0x7f28d4156d10 2023-12-19T09:19:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.885+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f28cc003020 con 0x7f28d4156d10 2023-12-19T09:19:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.889+0000 7f28baffd700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 0x7f28c0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.889+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7f28cc07db60 con 0x7f28d4156d10 2023-12-19T09:19:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.889+0000 7f28d2ffd700 1 -- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 msgr2=0x7f28c0062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.889+0000 7f28d2ffd700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 0x7f28c0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:23.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:23.893+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f28cc04d860 con 0x7f28d4156d10 2023-12-19T09:19:23.920 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:23.916+0000 7f7733fc5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:23.930 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:23.925+0000 7f878278e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:24.019 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.016+0000 7f7733fc5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:19:24.026 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.021+0000 7f878278e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:19:24.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.073+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28d4071120 con 0x7f28d4156d10 2023-12-19T09:19:24.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.073+0000 7f28baffd700 1 -- 172.21.15.111:0/3980424324 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17539 (secure 0 0 0) 0x7f28cc050e80 con 0x7f28d4156d10 2023-12-19T09:19:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:24.079 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":180,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:22.856585+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":180,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.12","mappings":[{"from":3,"to":0}]},{"pgid":"22.13","mappings":[{"from":3,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2893890504":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3589389202":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/57208669":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2193897545":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3287144757":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2770672326":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/4214132662":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3599201417":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2998303494":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/1690815342":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3967653215":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2978228531":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6833/33796":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6832/33796":"2023-12-20T09:03:31.913274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 msgr2=0x7f28c0062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 0x7f28c0062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 msgr2=0x7f28d40682c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009fc0 tx=0x7f28cc00b040 comp rx=0 tx=0).stop 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 shutdown_connections 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f28c0060670 0x7f28c0062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28d4156940 0x7f28d4151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28d4156d10 0x7f28d40682c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 --2- 172.21.15.111:0/3980424324 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28d406c7c0 0x7f28d415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 >> 172.21.15.111:0/3980424324 conn(0x7f28d40ce320 msgr2=0x7f28d40cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 shutdown_connections 2023-12-19T09:19:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.077+0000 7f28d94f0700 1 -- 172.21.15.111:0/3980424324 wait complete. 2023-12-19T09:19:24.098 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.093+0000 7f878278e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:19:24.100 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.096+0000 7f7733fc5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:19:24.102 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:19:24.102 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:24.179 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.176+0000 7f7733fc5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:19:24.184 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.181+0000 7f878278e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:19:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 Processor -- start 2023-12-19T09:19:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 -- start start 2023-12-19T09:19:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03500aba90 0x7f03500abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03500c8730 con 0x7f03500cb750 2023-12-19T09:19:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03500c7e70 con 0x7f03500cbb20 2023-12-19T09:19:24.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0358785700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03500cbf80 con 0x7f03500aba90 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0355d20700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03500aba90 0x7f03500abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0356521700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f03500cb750 0x7f03500c1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49660/0 (socket says 172.21.15.111:49660) 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0355d20700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03500aba90 0x7f03500abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55760/0 (socket says 172.21.15.111:55760) 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0356521700 1 -- 172.21.15.111:0/3194427969 learned_addr learned my addr 172.21.15.111:0/3194427969 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0355d20700 1 -- 172.21.15.111:0/3194427969 >> v1:172.21.15.143:6789/0 conn(0x7f03500cbb20 legacy=0x7f03500c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0355d20700 1 -- 172.21.15.111:0/3194427969 >> v1:172.21.15.111:6789/0 conn(0x7f03500cb750 legacy=0x7f03500c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.257+0000 7f0355d20700 1 -- 172.21.15.111:0/3194427969 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03500c23a0 con 0x7f03500aba90 2023-12-19T09:19:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0355d20700 1 --2- 172.21.15.111:0/3194427969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500abff0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f03400098e0 tx=0x7f03400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db1c002eaa46029 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f035551f700 1 -- 172.21.15.111:0/3194427969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340009e80 con 0x7f03500aba90 2023-12-19T09:19:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f035551f700 1 -- 172.21.15.111:0/3194427969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034000b860 con 0x7f03500aba90 2023-12-19T09:19:24.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f035551f700 1 -- 172.21.15.111:0/3194427969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034000bbe0 con 0x7f03500aba90 2023-12-19T09:19:24.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 -- 172.21.15.111:0/3194427969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 msgr2=0x7f03500abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 --2- 172.21.15.111:0/3194427969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500abff0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f03400098e0 tx=0x7f03400091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 -- 172.21.15.111:0/3194427969 shutdown_connections 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 --2- 172.21.15.111:0/3194427969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500abff0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 -- 172.21.15.111:0/3194427969 >> 172.21.15.111:0/3194427969 conn(0x7f03500cd9c0 msgr2=0x7f03500c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 -- 172.21.15.111:0/3194427969 shutdown_connections 2023-12-19T09:19:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.261+0000 7f0358785700 1 -- 172.21.15.111:0/3194427969 wait complete. 2023-12-19T09:19:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 Processor -- start 2023-12-19T09:19:24.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 -- start start 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03500cb750 0x7f03500c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03500cbb20 0x7f0350155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035015d690 con 0x7f03500aba90 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035015ca80 con 0x7f03500cb750 2023-12-19T09:19:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0358785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035015cd60 con 0x7f03500cbb20 2023-12-19T09:19:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55774/0 (socket says 172.21.15.111:55774) 2023-12-19T09:19:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 -- 172.21.15.111:0/1027044048 learned_addr learned my addr 172.21.15.111:0/1027044048 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0355d20700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03500cb750 0x7f03500c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356d22700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03500cbb20 0x7f0350155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03500cbb20 msgr2=0x7f0350155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03500cbb20 0x7f0350155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03500cb750 msgr2=0x7f03500c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03500cb750 0x7f03500c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.265+0000 7f0356521700 1 -- 172.21.15.111:0/1027044048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0340009420 con 0x7f03500aba90 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f0356521700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f034c00c560 tx=0x7f034c001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c005b30 con 0x7f03500aba90 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c005cb0 con 0x7f03500aba90 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c007320 con 0x7f03500aba90 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035015d040 con 0x7f03500aba90 2023-12-19T09:19:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03500c4c30 con 0x7f03500aba90 2023-12-19T09:19:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.269+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035015d4c0 con 0x7f03500aba90 2023-12-19T09:19:24.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.273+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f034c007690 con 0x7f03500aba90 2023-12-19T09:19:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.273+0000 7f03467fc700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 0x7f0334062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.273+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7f034c07c860 con 0x7f03500aba90 2023-12-19T09:19:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.273+0000 7f0355d20700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 msgr2=0x7f0334062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.273+0000 7f0355d20700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 0x7f0334062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:24.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.277+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f034c04c6f0 con 0x7f03500aba90 2023-12-19T09:19:24.417 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.412+0000 7f7733fc5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:24.422 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.417+0000 7f878278e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.477+0000 7f0355d20700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 msgr2=0x7f0334062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:24.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.477+0000 7f0355d20700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 0x7f0334062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:19:24.495 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.492+0000 7f7733fc5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:19:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.493+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0350071120 con 0x7f03500aba90 2023-12-19T09:19:24.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.493+0000 7f03467fc700 1 -- 172.21.15.111:0/1027044048 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7f034c017090 con 0x7f03500aba90 2023-12-19T09:19:24.497 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:24.497 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:19:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 msgr2=0x7f0334062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:24.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 0x7f0334062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 msgr2=0x7f03500c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f034c00c560 tx=0x7f034c001020 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 shutdown_connections 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0334060670 0x7f0334062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03500cb750 0x7f03500c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03500cbb20 0x7f0350155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 --2- 172.21.15.111:0/1027044048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03500aba90 0x7f03500c3fa0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 >> 172.21.15.111:0/1027044048 conn(0x7f03500cd9c0 msgr2=0x7f03500670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 shutdown_connections 2023-12-19T09:19:24.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.497+0000 7f0358785700 1 -- 172.21.15.111:0/1027044048 wait complete. 2023-12-19T09:19:24.502 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:19:24.504 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.501+0000 7f878278e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:19:24.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:24.589 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.588+0000 7f7733fc5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:19:24.596 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.593+0000 7f878278e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:19:24.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.685+0000 7f6c9c944700 1 Processor -- start 2023-12-19T09:19:24.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.685+0000 7f6c9c944700 1 -- start start 2023-12-19T09:19:24.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.685+0000 7f6c9c944700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c9806e1d0 0x7f6c98159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.685+0000 7f6c9c944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806b210 0x7f6c9814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.685+0000 7f6c9c944700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9815a260 con 0x7f6c9806ad70 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c980afad0 con 0x7f6c9806e1d0 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980afdb0 con 0x7f6c9806b210 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c9806e1d0 0x7f6c98159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c9806e1d0 0x7f6c98159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55780/0 (socket says 172.21.15.111:55780) 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 -- 172.21.15.111:0/481373573 learned_addr learned my addr 172.21.15.111:0/481373573 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9759e700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806b210 0x7f6c9814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9659c700 1 --1- 172.21.15.111:0/481373573 >> v1:172.21.15.143:6789/0 conn(0x7f6c9806ad70 0x7f6c9814f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:35592/0 (socket says 172.21.15.111:35592) 2023-12-19T09:19:24.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 -- 172.21.15.111:0/481373573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806b210 msgr2=0x7f6c9814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:24.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806b210 0x7f6c9814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 -- 172.21.15.111:0/481373573 >> v1:172.21.15.143:6789/0 conn(0x7f6c9806ad70 legacy=0x7f6c9814f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 -- 172.21.15.111:0/481373573 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980b0090 con 0x7f6c9806e1d0 2023-12-19T09:19:24.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c96d9d700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806e1d0 0x7f6c98159d20 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e6f596e6730df1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:24.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c95d9b700 1 -- 172.21.15.111:0/481373573 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88009ea0 con 0x7f6c9806e1d0 2023-12-19T09:19:24.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c95d9b700 1 -- 172.21.15.111:0/481373573 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800b860 con 0x7f6c9806e1d0 2023-12-19T09:19:24.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c95d9b700 1 -- 172.21.15.111:0/481373573 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800bc00 con 0x7f6c9806e1d0 2023-12-19T09:19:24.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- 172.21.15.111:0/481373573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806e1d0 msgr2=0x7f6c98159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806e1d0 0x7f6c98159d20 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).stop 2023-12-19T09:19:24.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- 172.21.15.111:0/481373573 shutdown_connections 2023-12-19T09:19:24.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806b210 0x7f6c9814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 --2- 172.21.15.111:0/481373573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806e1d0 0x7f6c98159d20 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- 172.21.15.111:0/481373573 >> 172.21.15.111:0/481373573 conn(0x7f6c98067bb0 msgr2=0x7f6c980add90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- 172.21.15.111:0/481373573 shutdown_connections 2023-12-19T09:19:24.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.689+0000 7f6c9c944700 1 -- 172.21.15.111:0/481373573 wait complete. 2023-12-19T09:19:24.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 Processor -- start 2023-12-19T09:19:24.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 -- start start 2023-12-19T09:19:24.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9806ad70 0x7f6c981552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806e1d0 0x7f6c980ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9816c8d0 con 0x7f6c9806b210 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9816d450 con 0x7f6c9806ad70 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.693+0000 7f6c9c944700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9816cb20 con 0x7f6c9806e1d0 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55794/0 (socket says 172.21.15.111:55794) 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 -- 172.21.15.111:0/358170478 learned_addr learned my addr 172.21.15.111:0/358170478 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c96d9d700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9806ad70 0x7f6c981552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9759e700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806e1d0 0x7f6c980ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806e1d0 msgr2=0x7f6c980ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806e1d0 0x7f6c980ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9806ad70 msgr2=0x7f6c981552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9806ad70 0x7f6c981552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 -- 172.21.15.111:0/358170478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880094f0 con 0x7f6c9806b210 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9659c700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c7f0 tx=0x7f6c8c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c005a70 con 0x7f6c9806b210 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c005bf0 con 0x7f6c9806b210 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9816ce00 con 0x7f6c9806b210 2023-12-19T09:19:24.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0070d0 con 0x7f6c9806b210 2023-12-19T09:19:24.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c980c10a0 con 0x7f6c9806b210 2023-12-19T09:19:24.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.697+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9816d5e0 con 0x7f6c9806b210 2023-12-19T09:19:24.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.701+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f6c8c007480 con 0x7f6c9806b210 2023-12-19T09:19:24.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.701+0000 7f6c7effd700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 0x7f6c84062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:24.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.701+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7f6c8c07c430 con 0x7f6c9806b210 2023-12-19T09:19:24.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.701+0000 7f6c96d9d700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 msgr2=0x7f6c84062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:24.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.701+0000 7f6c96d9d700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 0x7f6c84062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:24.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.709+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c8c04c240 con 0x7f6c9806b210 2023-12-19T09:19:24.796 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.792+0000 7f7733fc5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:19:24.816 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.813+0000 7f878278e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:19:24.887 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:24.884+0000 7f7733fc5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:19:24.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.905+0000 7f6c96d9d700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 msgr2=0x7f6c84062b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:24.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.905+0000 7f6c96d9d700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 0x7f6c84062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:19:24.908 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:24.905+0000 7f878278e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:19:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.913+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c9804be40 con 0x7f6c9806b210 2023-12-19T09:19:24.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c7effd700 1 -- 172.21.15.111:0/358170478 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+4610 (secure 0 0 0) 0x7f6c8c04f860 con 0x7f6c9806b210 2023-12-19T09:19:24.921 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:24.921 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":80,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:03:17.684061+0000","modified":"2023-12-19T09:03:21.295488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9396":{"gid":9396,"name":"b","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6835/811953301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":811953301},{"type":"v1","addr":"172.21.15.143:6835","nonce":811953301}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2023-12-19T09:19:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 msgr2=0x7f6c84062b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 0x7f6c84062b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 msgr2=0x7f6c980add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00c7f0 tx=0x7f6c8c007f70 comp rx=0 tx=0).stop 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 shutdown_connections 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9806e1d0 0x7f6c980ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c9806b210 0x7f6c980add90 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6c84060670 0x7f6c84062b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 --2- 172.21.15.111:0/358170478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9806ad70 0x7f6c981552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.917+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 >> 172.21.15.111:0/358170478 conn(0x7f6c98067bb0 msgr2=0x7f6c981561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.921+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 shutdown_connections 2023-12-19T09:19:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:24.921+0000 7f6c9c944700 1 -- 172.21.15.111:0/358170478 wait complete. 2023-12-19T09:19:24.926 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 80 2023-12-19T09:19:24.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:25.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.101+0000 7f180759e700 1 Processor -- start 2023-12-19T09:19:25.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.101+0000 7f180759e700 1 -- start start 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1808155960 0x7f180815a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18080c7f00 0x7f18080aca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180806d990 con 0x7f180806de20 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180815a570 con 0x7f1808155960 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180806cd80 con 0x7f18080c7f00 2023-12-19T09:19:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1805d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f180806de20 0x7f18080c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49672/0 (socket says 172.21.15.111:49672) 2023-12-19T09:19:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1805d9b700 1 -- 172.21.15.111:0/1500586796 learned_addr learned my addr 172.21.15.111:0/1500586796 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:25.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> v2:172.21.15.143:3300/0 conn(0x7f1808155960 0x7f180815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1806d9d700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18080c7f00 0x7f18080aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234689785 0 0) 0x7f180806d990 con 0x7f180806de20 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f0003570 con 0x7f180806de20 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384245375 0 0) 0x7f17f0003570 con 0x7f180806de20 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18080c7f00 msgr2=0x7f18080aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18080c7f00 0x7f18080aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 >> v2:172.21.15.143:3300/0 conn(0x7f1808155960 msgr2=0x7f180815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 --2- 172.21.15.111:0/1500586796 >> v2:172.21.15.143:3300/0 conn(0x7f1808155960 0x7f180815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180806d060 con 0x7f180806de20 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f17fc001470 con 0x7f180806de20 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 >> v1:172.21.15.111:6789/0 conn(0x7f180806de20 legacy=0x7f18080c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f0003a60 0x7f17f0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17f0004810 0x7f17f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f00051c0 0x7f17f000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180815a570 con 0x7f17f0003a60 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180806cd80 con 0x7f17f0004810 2023-12-19T09:19:25.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f000e1b0 con 0x7f17f00051c0 2023-12-19T09:19:25.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1806d9d700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18080c7f00 0x7f18080aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:25.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> v2:172.21.15.143:3300/0 conn(0x7f1808155960 0x7f180815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:25.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f0003a60 0x7f17f0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1805d9b700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17f0004810 0x7f17f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f1806d9d700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f00051c0 0x7f17f000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 -- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f00051c0 msgr2=0x7f17f000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f00051c0 0x7f17f000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 -- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17f0004810 msgr2=0x7f17f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17f0004810 0x7f17f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.105+0000 7f180659c700 1 -- 172.21.15.111:0/1500586796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0003570 con 0x7f17f0003a60 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180659c700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f0003a60 0x7f17f0003e70 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009db0 tx=0x7f17f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2561ec1a6a203728 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8009740 con 0x7f17f0003a60 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f800e920 con 0x7f17f0003a60 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180559a700 1 -- 172.21.15.111:0/1500586796 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f800eca0 con 0x7f17f0003a60 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 -- 172.21.15.111:0/1500586796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f0003a60 msgr2=0x7f17f0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 --2- 172.21.15.111:0/1500586796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f0003a60 0x7f17f0003e70 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009db0 tx=0x7f17f80091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f1806d9d700 1 -- 172.21.15.111:0/1500586796 reap_dead start 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 -- 172.21.15.111:0/1500586796 shutdown_connections 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 -- 172.21.15.111:0/1500586796 >> 172.21.15.111:0/1500586796 conn(0x7f1808067bb0 msgr2=0x7f18080cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 -- 172.21.15.111:0/1500586796 shutdown_connections 2023-12-19T09:19:25.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.109+0000 7f180759e700 1 -- 172.21.15.111:0/1500586796 wait complete. 2023-12-19T09:19:25.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 Processor -- start 2023-12-19T09:19:25.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 -- start start 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f0003a60 0x7f1808169290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1808159680 0x7f18081514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180816a230 con 0x7f18080afae0 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18080aeb10 con 0x7f1808159680 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18081519e0 con 0x7f17f0003a60 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55802/0 (socket says 172.21.15.111:55802) 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 -- 172.21.15.111:0/4165174825 learned_addr learned my addr 172.21.15.111:0/4165174825 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f180659c700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f0003a60 0x7f1808169290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1805d9b700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1808159680 0x7f18081514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 -- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f0003a60 msgr2=0x7f1808169290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f0003a60 0x7f1808169290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 -- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1808159680 msgr2=0x7f18081514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1808159680 0x7f18081514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 -- 172.21.15.111:0/4165174825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808169990 con 0x7f18080afae0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.113+0000 7f1806d9d700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f180806e720 tx=0x7f180000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000e040 con 0x7f18080afae0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000f040 con 0x7f18080afae0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1808169c70 con 0x7f18080afae0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000fd30 con 0x7f18080afae0 2023-12-19T09:19:25.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1808159af0 con 0x7f18080afae0 2023-12-19T09:19:25.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18080aecc0 con 0x7f18080afae0 2023-12-19T09:19:25.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.117+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7f1800015640 con 0x7f18080afae0 2023-12-19T09:19:25.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.121+0000 7f17f67fc700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 0x7f17e8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.121+0000 7f180659c700 1 -- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 msgr2=0x7f17e8062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:25.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.121+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7f180007cfe0 con 0x7f18080afae0 2023-12-19T09:19:25.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.121+0000 7f180659c700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 0x7f17e8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:25.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.125+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f18080aecc0 con 0x7f18080afae0 2023-12-19T09:19:25.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.297+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18080aecc0 con 0x7f18080afae0 2023-12-19T09:19:25.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.297+0000 7f17f67fc700 1 -- 172.21.15.111:0/4165174825 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17539 (secure 0 0 0) 0x7f18080aecc0 con 0x7f18080afae0 2023-12-19T09:19:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:25.304 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":180,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:22.856585+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:03:15.680577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-12-19T09:03:16.714251+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":180,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.12","mappings":[{"from":3,"to":0}]},{"pgid":"22.13","mappings":[{"from":3,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/2893890504":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3589389202":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/57208669":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2193897545":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3287144757":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2770672326":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/4214132662":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3599201417":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2998303494":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/1690815342":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3967653215":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2978228531":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6833/33796":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6832/33796":"2023-12-20T09:03:31.913274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:25.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 msgr2=0x7f17e8062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 0x7f17e8062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 msgr2=0x7f18080aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f180806e720 tx=0x7f180000b040 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 shutdown_connections 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f17e80606a0 0x7f17e8062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1808159680 0x7f18081514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18080afae0 0x7f18080aff50 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 --2- 172.21.15.111:0/4165174825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17f0003a60 0x7f1808169290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 >> 172.21.15.111:0/4165174825 conn(0x7f1808067bb0 msgr2=0x7f180814dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 shutdown_connections 2023-12-19T09:19:25.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.305+0000 7f180759e700 1 -- 172.21.15.111:0/4165174825 wait complete. 2023-12-19T09:19:25.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T09:19:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 Processor -- start 2023-12-19T09:19:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 -- start start 2023-12-19T09:19:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdfdc0afef0 0x7fdfdc06a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdfdc160e60 0x7fdfdc15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfdc06ba50 con 0x7fdfdc15d050 2023-12-19T09:19:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc06bee0 con 0x7fdfdc160e60 2023-12-19T09:19:25.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe3cf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfdc0af600 con 0x7fdfdc0afef0 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe1a8c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdfdc15d050 0x7fdfdc0af7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49686/0 (socket says 172.21.15.111:49686) 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.489+0000 7fdfe1a8c700 1 -- 172.21.15.111:0/1308482750 learned_addr learned my addr 172.21.15.111:0/1308482750 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe128b700 1 --2- 172.21.15.111:0/1308482750 >> v2:172.21.15.143:3300/0 conn(0x7fdfdc0afef0 0x7fdfdc06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 --2- 172.21.15.111:0/1308482750 >> v2:172.21.15.111:3300/0 conn(0x7fdfdc160e60 0x7fdfdc15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe0a8a700 1 -- 172.21.15.111:0/1308482750 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786112375 0 0) 0x7fdfdc06ba50 con 0x7fdfdc15d050 2023-12-19T09:19:25.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe0a8a700 1 -- 172.21.15.111:0/1308482750 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc8003570 con 0x7fdfdc15d050 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 -- 172.21.15.111:0/1308482750 >> v2:172.21.15.143:3300/0 conn(0x7fdfdc0afef0 msgr2=0x7fdfdc06a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 --2- 172.21.15.111:0/1308482750 >> v2:172.21.15.143:3300/0 conn(0x7fdfdc0afef0 0x7fdfdc06a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 -- 172.21.15.111:0/1308482750 >> v1:172.21.15.111:6789/0 conn(0x7fdfdc15d050 legacy=0x7fdfdc0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 -- 172.21.15.111:0/1308482750 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc06ae40 con 0x7fdfdc160e60 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe228d700 1 --2- 172.21.15.111:0/1308482750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc160e60 0x7fdfdc15d420 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc06a150 tx=0x7fdfd4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45f7d8440d88022 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe0a8a700 1 -- 172.21.15.111:0/1308482750 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400e070 con 0x7fdfdc160e60 2023-12-19T09:19:25.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe0a8a700 1 -- 172.21.15.111:0/1308482750 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd400bab0 con 0x7fdfdc160e60 2023-12-19T09:19:25.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe0a8a700 1 -- 172.21.15.111:0/1308482750 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400be70 con 0x7fdfdc160e60 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/1308482750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc160e60 msgr2=0x7fdfdc15d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/1308482750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc160e60 0x7fdfdc15d420 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc06a150 tx=0x7fdfd4009780 comp rx=0 tx=0).stop 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/1308482750 shutdown_connections 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/1308482750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc160e60 0x7fdfdc15d420 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/1308482750 >> v2:172.21.15.143:3300/0 conn(0x7fdfdc0afef0 0x7fdfdc06a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/1308482750 >> 172.21.15.111:0/1308482750 conn(0x7fdfdc067b60 msgr2=0x7fdfdc0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/1308482750 shutdown_connections 2023-12-19T09:19:25.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.493+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/1308482750 wait complete. 2023-12-19T09:19:25.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 Processor -- start 2023-12-19T09:19:25.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 -- start start 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfdc15d050 0x7fdfdc0a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfdc160e60 0x7fdfdc156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc150ee0 con 0x7fdfdc0afef0 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc156900 con 0x7fdfdc160e60 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe3cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1565b0 con 0x7fdfdc15d050 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe1a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe1a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55826/0 (socket says 172.21.15.111:55826) 2023-12-19T09:19:25.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.497+0000 7fdfe1a8c700 1 -- 172.21.15.111:0/2886947949 learned_addr learned my addr 172.21.15.111:0/2886947949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe128b700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfdc15d050 0x7fdfdc0a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe228d700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfdc160e60 0x7fdfdc156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 -- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfdc15d050 msgr2=0x7fdfdc0a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfdc15d050 0x7fdfdc0a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 -- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfdc160e60 msgr2=0x7fdfdc156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfdc160e60 0x7fdfdc156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 -- 172.21.15.111:0/2886947949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4009a20 con 0x7fdfdc0afef0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe1a8c700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800c7c0 tx=0x7fdfd8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8005b30 con 0x7fdfdc0afef0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd8005c90 con 0x7fdfdc0afef0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8007340 con 0x7fdfdc0afef0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc155c40 con 0x7fdfdc0afef0 2023-12-19T09:19:25.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc0a4980 con 0x7fdfdc0afef0 2023-12-19T09:19:25.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.501+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc071120 con 0x7fdfdc0afef0 2023-12-19T09:19:25.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.505+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 89) v1 ==== 80075+0+0 (secure 0 0 0) 0x7fdfd801b400 con 0x7fdfdc0afef0 2023-12-19T09:19:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.505+0000 7fdfd1ffb700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 0x7fdfc0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.505+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 7289+0+0 (secure 0 0 0) 0x7fdfd807ca80 con 0x7fdfdc0afef0 2023-12-19T09:19:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.505+0000 7fdfe128b700 1 -- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 msgr2=0x7fdfc0062b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:25.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.505+0000 7fdfe128b700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 0x7fdfc0062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:25.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.509+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfd804c930 con 0x7fdfdc0afef0 2023-12-19T09:19:25.630 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.628+0000 7f7733fc5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:19:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.649+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 90) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fdfd804cd00 con 0x7fdfdc0afef0 2023-12-19T09:19:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.649+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 msgr2=0x7fdfc0062b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.649+0000 7fdfd1ffb700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 0x7fdfc0062b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.725+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdfdc0cb8f0 con 0x7fdfdc0afef0 2023-12-19T09:19:25.732 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T09:19:25.729+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T09:19:25.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.733+0000 7fdfd1ffb700 1 -- 172.21.15.111:0/2886947949 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdfd8077850 con 0x7fdfdc0afef0 2023-12-19T09:19:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 msgr2=0x7fdfdc1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800c7c0 tx=0x7fdfd8007e00 comp rx=0 tx=0).stop 2023-12-19T09:19:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 shutdown_connections 2023-12-19T09:19:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdfc00606a0 0x7fdfc0062b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfdc160e60 0x7fdfdc156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfdc15d050 0x7fdfdc0a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 --2- 172.21.15.111:0/2886947949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfdc0afef0 0x7fdfdc1509a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 >> 172.21.15.111:0/2886947949 conn(0x7fdfdc067b60 msgr2=0x7fdfdc15fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 shutdown_connections 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.741+0000 7fdfe3cf0700 1 -- 172.21.15.111:0/2886947949 wait complete. 2023-12-19T09:19:25.745 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T09:19:25.747 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fb2d0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fb2d0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fb2d0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fb2d0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fb2d0700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fe2d6700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fe2d6700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fe2d6700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fe2d6700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.748 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:25.744+0000 7f76fe2d6700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:25.760 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:25.757+0000 7f878278e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:19:25.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T09:19:25.813 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:19:25.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.917+0000 7f80f9585700 1 Processor -- start 2023-12-19T09:19:25.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.917+0000 7f80f9585700 1 -- start start 2023-12-19T09:19:25.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f9585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80f41554b0 0x7f80f40ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f9585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f80f406a7a0 0x7f80f406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f9585700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f4159340 con 0x7f80f406c7c0 2023-12-19T09:19:25.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f9585700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f406d790 con 0x7f80f41554b0 2023-12-19T09:19:25.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f9585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f406b0f0 con 0x7f80f406a7a0 2023-12-19T09:19:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80f41554b0 0x7f80f40ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f27fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f80f406c7c0 0x7f80f41597d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49688/0 (socket says 172.21.15.111:49688) 2023-12-19T09:19:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f27fc700 1 -- 172.21.15.111:0/4146732330 learned_addr learned my addr 172.21.15.111:0/4146732330 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f37fe700 1 --2- 172.21.15.111:0/4146732330 >> v2:172.21.15.143:3300/0 conn(0x7f80f406a7a0 0x7f80f406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f1ffb700 1 -- 172.21.15.111:0/4146732330 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323381902 0 0) 0x7f80f4159340 con 0x7f80f406c7c0 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 -- 172.21.15.111:0/4146732330 >> v2:172.21.15.143:3300/0 conn(0x7f80f406a7a0 msgr2=0x7f80f406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/4146732330 >> v2:172.21.15.143:3300/0 conn(0x7f80f406a7a0 0x7f80f406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 -- 172.21.15.111:0/4146732330 >> v1:172.21.15.111:6789/0 conn(0x7f80f406c7c0 legacy=0x7f80f41597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 -- 172.21.15.111:0/4146732330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f406cef0 con 0x7f80f41554b0 2023-12-19T09:19:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.921+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/4146732330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f40ca2d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f80f40635a0 tx=0x7f80e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dd6edecfc1a09b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f1ffb700 1 -- 172.21.15.111:0/4146732330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4011630 con 0x7f80f41554b0 2023-12-19T09:19:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f1ffb700 1 -- 172.21.15.111:0/4146732330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e4011790 con 0x7f80f41554b0 2023-12-19T09:19:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f1ffb700 1 -- 172.21.15.111:0/4146732330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4011b30 con 0x7f80f41554b0 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 -- 172.21.15.111:0/4146732330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 msgr2=0x7f80f40ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 --2- 172.21.15.111:0/4146732330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f40ca2d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f80f40635a0 tx=0x7f80e40091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 -- 172.21.15.111:0/4146732330 shutdown_connections 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 --2- 172.21.15.111:0/4146732330 >> v2:172.21.15.143:3300/0 conn(0x7f80f406a7a0 0x7f80f406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 --2- 172.21.15.111:0/4146732330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f40ca2d0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 -- 172.21.15.111:0/4146732330 >> 172.21.15.111:0/4146732330 conn(0x7f80f4067bb0 msgr2=0x7f80f40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 -- 172.21.15.111:0/4146732330 shutdown_connections 2023-12-19T09:19:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.925+0000 7f80f9585700 1 -- 172.21.15.111:0/4146732330 wait complete. 2023-12-19T09:19:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 Processor -- start 2023-12-19T09:19:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 -- start start 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f406a7a0 0x7f80f40ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 0x7f80f40ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40ab840 con 0x7f80f41554b0 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40abb20 con 0x7f80f406c7c0 2023-12-19T09:19:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.929+0000 7f80f9585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f40abe00 con 0x7f80f406a7a0 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55846/0 (socket says 172.21.15.111:55846) 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 -- 172.21.15.111:0/2001736119 learned_addr learned my addr 172.21.15.111:0/2001736119 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f2ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f406a7a0 0x7f80f40ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f27fc700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 0x7f80f40ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 -- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f406a7a0 msgr2=0x7f80f40ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f406a7a0 0x7f80f40ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 -- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 msgr2=0x7f80f40ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 0x7f80f40ada50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 -- 172.21.15.111:0/2001736119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e4009420 con 0x7f80f41554b0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f27fc700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 0x7f80f40ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f37fe700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009f10 tx=0x7f80ec00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec007cb0 con 0x7f80f41554b0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec010040 con 0x7f80f41554b0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec014640 con 0x7f80f41554b0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f40ac0e0 con 0x7f80f41554b0 2023-12-19T09:19:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f40acb20 con 0x7f80f41554b0 2023-12-19T09:19:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.933+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f4071120 con 0x7f80f41554b0 2023-12-19T09:19:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.937+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 90) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f80ec014a30 con 0x7f80f41554b0 2023-12-19T09:19:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.937+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 7691+0+0 (secure 0 0 0) 0x7f80ec07d180 con 0x7f80f41554b0 2023-12-19T09:19:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:25.945+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f80ec04d370 con 0x7f80f41554b0 2023-12-19T09:19:26.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.157+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f80f406e0f0 con 0x7f80f41554b0 2023-12-19T09:19:26.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.657+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 91) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f80ec04ce20 con 0x7f80f41554b0 2023-12-19T09:19:26.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.657+0000 7f80daffd700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 0x7f80e00690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.661+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 0x7f80e00690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.665+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 0x7f80e00690e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f80e40093f0 tx=0x7f80e4017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.741+0000 7f80daffd700 1 -- 172.21.15.111:0/2001736119 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) v1 ==== 104+0+0 (secure 0 0 0) 0x7f80ec069ad0 con 0x7f80f41554b0 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 msgr2=0x7f80e00690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 0x7f80e00690e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f80e40093f0 tx=0x7f80e4017000 comp rx=0 tx=0).stop 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 msgr2=0x7f80f41579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009f10 tx=0x7f80ec00a890 comp rx=0 tx=0).stop 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 shutdown_connections 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f80e0066d00 0x7f80e00690e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f406c7c0 0x7f80f40ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80f41554b0 0x7f80f41579d0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 --2- 172.21.15.111:0/2001736119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f406a7a0 0x7f80f40ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 >> 172.21.15.111:0/2001736119 conn(0x7f80f4067bb0 msgr2=0x7f80f41523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:26.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 shutdown_connections 2023-12-19T09:19:26.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.745+0000 7f80f9585700 1 -- 172.21.15.111:0/2001736119 wait complete. 2023-12-19T09:19:26.772 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T09:19:26.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.937+0000 7f74cbfff700 1 Processor -- start 2023-12-19T09:19:26.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.937+0000 7f74cbfff700 1 -- start start 2023-12-19T09:19:26.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74cbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 0x7f74cc1607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74cbfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc0ae490 con 0x7f74cc155a40 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74cbfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc0ad880 con 0x7f74cc0ae920 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74cbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0adb60 con 0x7f74cc159d30 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74caffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f74cc155a40 0x7f74cc0ace80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49696/0 (socket says 172.21.15.111:49696) 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74caffd700 1 -- 172.21.15.111:0/3173864967 learned_addr learned my addr 172.21.15.111:0/3173864967 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:26.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74cb7fe700 1 --2- 172.21.15.111:0/3173864967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 0x7f74cc1607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368889381 0 0) 0x7f74cc0ae490 con 0x7f74cc155a40 2023-12-19T09:19:26.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.941+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b4003570 con 0x7f74cc155a40 2023-12-19T09:19:26.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373391922 0 0) 0x7f74cc0ad880 con 0x7f74cc0ae920 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc0ae490 con 0x7f74cc0ae920 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236294131 0 0) 0x7f74b4003570 con 0x7f74cc155a40 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3173864967 >> v1:172.21.15.143:6789/0 conn(0x7f74cc0ae920 legacy=0x7f74cc159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3173864967 >> v1:172.21.15.111:6789/0 conn(0x7f74cc155a40 legacy=0x7f74cc0ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3173864967 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0ade40 con 0x7f74cc159d30 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cb7fe700 1 --2- 172.21.15.111:0/3173864967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 0x7f74cc1607b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009e90 tx=0x7f74c4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6366829e321e98 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c400e070 con 0x7f74cc159d30 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c400b870 con 0x7f74cc159d30 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74c9ffb700 1 -- 172.21.15.111:0/3173864967 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc001470 con 0x7f74cc159d30 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 -- 172.21.15.111:0/3173864967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 msgr2=0x7f74cc1607b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3173864967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 0x7f74cc1607b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009e90 tx=0x7f74c4009780 comp rx=0 tx=0).stop 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 -- 172.21.15.111:0/3173864967 shutdown_connections 2023-12-19T09:19:26.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3173864967 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc159d30 0x7f74cc1607b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 -- 172.21.15.111:0/3173864967 >> 172.21.15.111:0/3173864967 conn(0x7f74cc0c8c00 msgr2=0x7f74cc0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 -- 172.21.15.111:0/3173864967 shutdown_connections 2023-12-19T09:19:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.945+0000 7f74cbfff700 1 -- 172.21.15.111:0/3173864967 wait complete. 2023-12-19T09:19:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.949+0000 7f74cbfff700 1 Processor -- start 2023-12-19T09:19:26.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.949+0000 7f74cbfff700 1 -- start start 2023-12-19T09:19:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc0ae920 0x7f74cc158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 0x7f74cc154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc154880 con 0x7f74cc159d30 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc151b70 con 0x7f74cc155a40 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0c1ff0 con 0x7f74cc0ae920 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55856/0 (socket says 172.21.15.111:55856) 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3611863399 learned_addr learned my addr 172.21.15.111:0/3611863399 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74ca7fc700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 0x7f74cc154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74caffd700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc0ae920 0x7f74cc158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc0ae920 msgr2=0x7f74cc158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc0ae920 0x7f74cc158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 msgr2=0x7f74cc154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 0x7f74cc154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 -- 172.21.15.111:0/3611863399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4009a20 con 0x7f74cc159d30 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74ca7fc700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 0x7f74cc154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cb7fe700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f74c4003000 tx=0x7f74c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4012d40 con 0x7f74cc159d30 2023-12-19T09:19:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c401e450 con 0x7f74cc159d30 2023-12-19T09:19:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc0bf920 con 0x7f74cc159d30 2023-12-19T09:19:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4012ea0 con 0x7f74cc159d30 2023-12-19T09:19:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc0c2290 con 0x7f74cc159d30 2023-12-19T09:19:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.953+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc0c24b0 con 0x7f74cc159d30 2023-12-19T09:19:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.957+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 91) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f74c401eae0 con 0x7f74cc159d30 2023-12-19T09:19:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.957+0000 7f74b2ffd700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 0x7f74b8062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.957+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 7691+0+0 (secure 0 0 0) 0x7f74c407ebf0 con 0x7f74cc159d30 2023-12-19T09:19:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.957+0000 7f74caffd700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 0x7f74b8062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.957+0000 7f74caffd700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 0x7f74b8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f74bc007c40 tx=0x7f74bc009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:26.961+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f74c404ec90 con 0x7f74cc159d30 2023-12-19T09:19:26.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:19:26.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:19:27.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.137+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f74cc04be40 con 0x7f74cc159d30 2023-12-19T09:19:27.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.893+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f74c404f060 con 0x7f74cc159d30 2023-12-19T09:19:27.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.901+0000 7f74b2ffd700 1 -- 172.21.15.111:0/3611863399 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v183) v1 ==== 181+0+0 (secure 0 0 0) 0x7f74c4052720 con 0x7f74cc159d30 2023-12-19T09:19:27.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 msgr2=0x7f74b8062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:27.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 0x7f74b8062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f74bc007c40 tx=0x7f74bc009040 comp rx=0 tx=0).stop 2023-12-19T09:19:27.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 msgr2=0x7f74cc0c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:27.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f74c4003000 tx=0x7f74c4017000 comp rx=0 tx=0).stop 2023-12-19T09:19:27.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 shutdown_connections 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f74b80603a0 0x7f74b8062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc155a40 0x7f74cc154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc159d30 0x7f74cc0c27b0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 --2- 172.21.15.111:0/3611863399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74cc0ae920 0x7f74cc158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.905+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 >> 172.21.15.111:0/3611863399 conn(0x7f74cc0c8c00 msgr2=0x7f74cc0cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.909+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 shutdown_connections 2023-12-19T09:19:27.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:27.909+0000 7f74cbfff700 1 -- 172.21.15.111:0/3611863399 wait complete. 2023-12-19T09:19:27.914 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T09:19:27.931 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T09:19:28.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.093+0000 7f15fd043700 1 Processor -- start 2023-12-19T09:19:28.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.093+0000 7f15fd043700 1 -- start start 2023-12-19T09:19:28.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15fd043700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f15f8158da0 0x7f15f806e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15fd043700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 0x7f15f806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15fd043700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f815d720 con 0x7f15f80caca0 2023-12-19T09:19:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15fd043700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f80ca290 con 0x7f15f8158da0 2023-12-19T09:19:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15fd043700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f80ca4a0 con 0x7f15f806df10 2023-12-19T09:19:28.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f15f8158da0 0x7f15f806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f15f8158da0 0x7f15f806e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55862/0 (socket says 172.21.15.111:55862) 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 -- 172.21.15.111:0/3439347259 learned_addr learned my addr 172.21.15.111:0/3439347259 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15effff700 1 --2- 172.21.15.111:0/3439347259 >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 0x7f15f806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 -- 172.21.15.111:0/3439347259 >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 msgr2=0x7f15f806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 --2- 172.21.15.111:0/3439347259 >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 0x7f15f806ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 -- 172.21.15.111:0/3439347259 >> v1:172.21.15.143:6789/0 conn(0x7f15f80caca0 legacy=0x7f15f806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 -- 172.21.15.111:0/3439347259 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f80ca780 con 0x7f15f8158da0 2023-12-19T09:19:28.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15effff700 1 --2- 172.21.15.111:0/3439347259 >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 0x7f15f806ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f7fff700 1 --2- 172.21.15.111:0/3439347259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f806e670 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f15f806a0b0 tx=0x7f15e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee4026c1b28a764 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f77fe700 1 -- 172.21.15.111:0/3439347259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e800b860 con 0x7f15f8158da0 2023-12-19T09:19:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f77fe700 1 -- 172.21.15.111:0/3439347259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e800b9c0 con 0x7f15f8158da0 2023-12-19T09:19:28.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.097+0000 7f15f77fe700 1 -- 172.21.15.111:0/3439347259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e800bd60 con 0x7f15f8158da0 2023-12-19T09:19:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 -- 172.21.15.111:0/3439347259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 msgr2=0x7f15f806e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 --2- 172.21.15.111:0/3439347259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f806e670 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f15f806a0b0 tx=0x7f15e8009250 comp rx=0 tx=0).stop 2023-12-19T09:19:28.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 -- 172.21.15.111:0/3439347259 shutdown_connections 2023-12-19T09:19:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 --2- 172.21.15.111:0/3439347259 >> v2:172.21.15.143:3300/0 conn(0x7f15f806df10 0x7f15f806ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 --2- 172.21.15.111:0/3439347259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f806e670 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 -- 172.21.15.111:0/3439347259 >> 172.21.15.111:0/3439347259 conn(0x7f15f80cc760 msgr2=0x7f15f80ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 -- 172.21.15.111:0/3439347259 shutdown_connections 2023-12-19T09:19:28.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.101+0000 7f15fd043700 1 -- 172.21.15.111:0/3439347259 wait complete. 2023-12-19T09:19:28.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 Processor -- start 2023-12-19T09:19:28.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 -- start start 2023-12-19T09:19:28.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15f806df10 0x7f15f8157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 0x7f15f80cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80c3560 con 0x7f15f8158da0 2023-12-19T09:19:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80b3710 con 0x7f15f80caca0 2023-12-19T09:19:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fd043700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80b2b40 con 0x7f15f806df10 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fc842700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fc842700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55868/0 (socket says 172.21.15.111:55868) 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15fc842700 1 -- 172.21.15.111:0/1756871093 learned_addr learned my addr 172.21.15.111:0/1756871093 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15f7fff700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15f806df10 0x7f15f8157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.105+0000 7f15effff700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 0x7f15f80cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 -- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15f806df10 msgr2=0x7f15f8157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15f806df10 0x7f15f8157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 -- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 msgr2=0x7f15f80cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 0x7f15f80cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 -- 172.21.15.111:0/1756871093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80094f0 con 0x7f15f8158da0 2023-12-19T09:19:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15effff700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 0x7f15f80cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fc842700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c8e0 tx=0x7f15f0001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000ce80 con 0x7f15f8158da0 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f000a7e0 con 0x7f15f8158da0 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000ab80 con 0x7f15f8158da0 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f8157ac0 con 0x7f15f8158da0 2023-12-19T09:19:28.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f80b30c0 con 0x7f15f8158da0 2023-12-19T09:19:28.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f80b3880 con 0x7f15f8158da0 2023-12-19T09:19:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.109+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f15f0012420 con 0x7f15f8158da0 2023-12-19T09:19:28.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.113+0000 7f15f4ff9700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 0x7f15d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:28.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.113+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 7253+0+0 (secure 0 0 0) 0x7f15f007d190 con 0x7f15f8158da0 2023-12-19T09:19:28.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.113+0000 7f15f7fff700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 0x7f15d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:28.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.113+0000 7f15f7fff700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 0x7f15d8062a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f15f806a0b0 tx=0x7f15e8017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:28.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.117+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f15f004d0d0 con 0x7f15f8158da0 2023-12-19T09:19:28.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.285+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f15f804be40 con 0x7f15f8158da0 2023-12-19T09:19:28.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.913+0000 7f15f4ff9700 1 -- 172.21.15.111:0/1756871093 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v184) v1 ==== 169+0+0 (secure 0 0 0) 0x7f15f00506f0 con 0x7f15f8158da0 2023-12-19T09:19:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 msgr2=0x7f15d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 0x7f15d8062a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f15f806a0b0 tx=0x7f15e8017000 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 msgr2=0x7f15f80cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f15f000c8e0 tx=0x7f15f0001710 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 shutdown_connections 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f15d8060550 0x7f15d8062a00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15f80caca0 0x7f15f80cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f8158da0 0x7f15f80cdd60 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 --2- 172.21.15.111:0/1756871093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15f806df10 0x7f15f8157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 >> 172.21.15.111:0/1756871093 conn(0x7f15f80cc760 msgr2=0x7f15f8158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 shutdown_connections 2023-12-19T09:19:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:28.921+0000 7f15fd043700 1 -- 172.21.15.111:0/1756871093 wait complete. 2023-12-19T09:19:28.929 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T09:19:28.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T09:19:29.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 Processor -- start 2023-12-19T09:19:29.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 -- start start 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fab200ca950 0x7fab200a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab200c6ab0 con 0x7fab2015c520 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2006b240 con 0x7fab2015c8f0 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab25105700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2006b520 con 0x7fab200ca950 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab1ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fab2015c520 0x7fab2006a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49700/0 (socket says 172.21.15.111:49700) 2023-12-19T09:19:29.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab1ed9d700 1 -- 172.21.15.111:0/2053239321 learned_addr learned my addr 172.21.15.111:0/2053239321 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:29.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab1f59e700 1 --2- 172.21.15.111:0/2053239321 >> v2:172.21.15.111:3300/0 conn(0x7fab200ca950 0x7fab200a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab1e59c700 1 -- 172.21.15.111:0/2053239321 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050319205 0 0) 0x7fab200c6ab0 con 0x7fab2015c520 2023-12-19T09:19:29.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.109+0000 7fab1e59c700 1 -- 172.21.15.111:0/2053239321 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c003570 con 0x7fab2015c520 2023-12-19T09:19:29.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1f59e700 1 -- 172.21.15.111:0/2053239321 >> v1:172.21.15.143:6789/0 conn(0x7fab2015c8f0 legacy=0x7fab200ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1f59e700 1 -- 172.21.15.111:0/2053239321 >> v1:172.21.15.111:6789/0 conn(0x7fab2015c520 legacy=0x7fab2006a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1f59e700 1 -- 172.21.15.111:0/2053239321 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab200c6d50 con 0x7fab200ca950 2023-12-19T09:19:29.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1f59e700 1 --2- 172.21.15.111:0/2053239321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab200a8850 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fab18009e60 tx=0x7fab18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5d77d6f1596193 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1e59c700 1 -- 172.21.15.111:0/2053239321 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1800e070 con 0x7fab200ca950 2023-12-19T09:19:29.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1e59c700 1 -- 172.21.15.111:0/2053239321 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1800bbf0 con 0x7fab200ca950 2023-12-19T09:19:29.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab1e59c700 1 -- 172.21.15.111:0/2053239321 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab18012430 con 0x7fab200ca950 2023-12-19T09:19:29.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 -- 172.21.15.111:0/2053239321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 msgr2=0x7fab200a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 --2- 172.21.15.111:0/2053239321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab200a8850 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fab18009e60 tx=0x7fab18009780 comp rx=0 tx=0).stop 2023-12-19T09:19:29.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 -- 172.21.15.111:0/2053239321 shutdown_connections 2023-12-19T09:19:29.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 --2- 172.21.15.111:0/2053239321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab200a8850 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 -- 172.21.15.111:0/2053239321 >> 172.21.15.111:0/2053239321 conn(0x7fab20067bb0 msgr2=0x7fab200cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:29.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 -- 172.21.15.111:0/2053239321 shutdown_connections 2023-12-19T09:19:29.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.113+0000 7fab25105700 1 -- 172.21.15.111:0/2053239321 wait complete. 2023-12-19T09:19:29.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.117+0000 7fab25105700 1 Processor -- start 2023-12-19T09:19:29.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.117+0000 7fab25105700 1 -- start start 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab2015c520 0x7fab20157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 0x7fab200b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200b0ce0 con 0x7fab200ca950 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200b00d0 con 0x7fab2015c520 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab200b03b0 con 0x7fab2015c8f0 2023-12-19T09:19:29.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55884/0 (socket says 172.21.15.111:55884) 2023-12-19T09:19:29.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 -- 172.21.15.111:0/722556960 learned_addr learned my addr 172.21.15.111:0/722556960 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1f59e700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 0x7fab200b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab17fff700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab2015c520 0x7fab20157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 -- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 msgr2=0x7fab200b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 0x7fab200b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 -- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab2015c520 msgr2=0x7fab20157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab2015c520 0x7fab20157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 -- 172.21.15.111:0/722556960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18009a20 con 0x7fab200ca950 2023-12-19T09:19:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1f59e700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 0x7fab200b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab1ed9d700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c7c0 tx=0x7fab10007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab10005b30 con 0x7fab200ca950 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab10005c90 con 0x7fab200ca950 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab10007340 con 0x7fab200ca950 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab200b0690 con 0x7fab200ca950 2023-12-19T09:19:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.121+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2015e270 con 0x7fab200ca950 2023-12-19T09:19:29.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.125+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab20071120 con 0x7fab200ca950 2023-12-19T09:19:29.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.125+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fab1001b400 con 0x7fab200ca950 2023-12-19T09:19:29.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.125+0000 7fab16ffd700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 0x7fab00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.125+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fab1007c6e0 con 0x7fab200ca950 2023-12-19T09:19:29.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.129+0000 7fab17fff700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 0x7fab00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.129+0000 7fab17fff700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 0x7fab00062a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab200c61b0 tx=0x7fab08009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.133+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fab1004c920 con 0x7fab200ca950 2023-12-19T09:19:29.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.301+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fab200688e0 con 0x7fab200ca950 2023-12-19T09:19:29.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.309+0000 7fab16ffd700 1 -- 172.21.15.111:0/722556960 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v61) v1 ==== 120+0+0 (secure 0 0 0) 0x7fab1004ff40 con 0x7fab200ca950 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 msgr2=0x7fab00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 0x7fab00062a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab200c61b0 tx=0x7fab08009040 comp rx=0 tx=0).stop 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 msgr2=0x7fab2015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c7c0 tx=0x7fab10007e00 comp rx=0 tx=0).stop 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 shutdown_connections 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fab00060550 0x7fab00062a00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab2015c520 0x7fab20157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab2015c8f0 0x7fab200b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 --2- 172.21.15.111:0/722556960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab200ca950 0x7fab2015dd30 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 >> 172.21.15.111:0/722556960 conn(0x7fab20067bb0 msgr2=0x7fab200c0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:29.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 shutdown_connections 2023-12-19T09:19:29.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.313+0000 7fab25105700 1 -- 172.21.15.111:0/722556960 wait complete. 2023-12-19T09:19:29.338 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T09:19:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.501+0000 7f93c268e700 1 Processor -- start 2023-12-19T09:19:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.501+0000 7f93c268e700 1 -- start start 2023-12-19T09:19:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c268e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f93bc06de00 0x7f93bc0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c268e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f93bc06c7c0 0x7f93bc15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c268e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc06d970 con 0x7f93bc0c69a0 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c268e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc06cd60 con 0x7f93bc06c7c0 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c268e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc06d040 con 0x7f93bc06de00 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c1e8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f93bc06c7c0 0x7f93bc15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f93bc06de00 0x7f93bc0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f93bc06de00 0x7f93bc0c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49348/0 (socket says 172.21.15.111:49348) 2023-12-19T09:19:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 -- 172.21.15.111:0/549708863 learned_addr learned my addr 172.21.15.111:0/549708863 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 -- 172.21.15.111:0/549708863 >> v2:172.21.15.111:3300/0 conn(0x7f93bc06c7c0 msgr2=0x7f93bc15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 --2- 172.21.15.111:0/549708863 >> v2:172.21.15.111:3300/0 conn(0x7f93bc06c7c0 0x7f93bc15cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 -- 172.21.15.111:0/549708863 >> v1:172.21.15.143:6789/0 conn(0x7f93bc0c69a0 legacy=0x7f93bc06b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.505+0000 7f93c0e8b700 1 -- 172.21.15.111:0/549708863 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc06d320 con 0x7f93bc06de00 2023-12-19T09:19:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c0e8b700 1 --2- 172.21.15.111:0/549708863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc06de00 0x7f93bc0c6d70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0cb990 tx=0x7f93ac009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aac5e1ec9fd1cea server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93b3fff700 1 -- 172.21.15.111:0/549708863 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac009e70 con 0x7f93bc06de00 2023-12-19T09:19:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93b3fff700 1 -- 172.21.15.111:0/549708863 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac00b860 con 0x7f93bc06de00 2023-12-19T09:19:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93b3fff700 1 -- 172.21.15.111:0/549708863 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00bc00 con 0x7f93bc06de00 2023-12-19T09:19:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 -- 172.21.15.111:0/549708863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc06de00 msgr2=0x7f93bc0c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 --2- 172.21.15.111:0/549708863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc06de00 0x7f93bc0c6d70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0cb990 tx=0x7f93ac009250 comp rx=0 tx=0).stop 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 -- 172.21.15.111:0/549708863 shutdown_connections 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 --2- 172.21.15.111:0/549708863 >> v2:172.21.15.111:3300/0 conn(0x7f93bc06c7c0 0x7f93bc15cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 --2- 172.21.15.111:0/549708863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc06de00 0x7f93bc0c6d70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 -- 172.21.15.111:0/549708863 >> 172.21.15.111:0/549708863 conn(0x7f93bc067b50 msgr2=0x7f93bc0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 -- 172.21.15.111:0/549708863 shutdown_connections 2023-12-19T09:19:29.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.509+0000 7f93c268e700 1 -- 172.21.15.111:0/549708863 wait complete. 2023-12-19T09:19:29.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 Processor -- start 2023-12-19T09:19:29.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 -- start start 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93bc06de00 0x7f93bc15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc0c69a0 0x7f93bc14cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc15dc10 con 0x7f93bc06c7c0 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc15d000 con 0x7f93bc0c69a0 2023-12-19T09:19:29.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c268e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc15d2e0 con 0x7f93bc06de00 2023-12-19T09:19:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55894/0 (socket says 172.21.15.111:55894) 2023-12-19T09:19:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 -- 172.21.15.111:0/63071028 learned_addr learned my addr 172.21.15.111:0/63071028 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:29.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c0e8b700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93bc06de00 0x7f93bc15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c1e8d700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc0c69a0 0x7f93bc14cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 -- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93bc06de00 msgr2=0x7f93bc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93bc06de00 0x7f93bc15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 -- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc0c69a0 msgr2=0x7f93bc14cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc0c69a0 0x7f93bc14cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.513+0000 7f93c168c700 1 -- 172.21.15.111:0/63071028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac0094f0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93c168c700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f93b800c8e0 tx=0x7f93b8001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ce80 con 0x7f93bc06c7c0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b800a7e0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800aba0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93bc15d5c0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc14d110 con 0x7f93bc06c7c0 2023-12-19T09:19:29.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.517+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc16d2e0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.521+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f93b8003030 con 0x7f93bc06c7c0 2023-12-19T09:19:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.521+0000 7f93b17fa700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 0x7f93a0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.521+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f93b807cce0 con 0x7f93bc06c7c0 2023-12-19T09:19:29.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.521+0000 7f93c0e8b700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 0x7f93a0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.521+0000 7f93c0e8b700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 0x7f93a0062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0cb990 tx=0x7f93ac017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.529+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f93b8050890 con 0x7f93bc06c7c0 2023-12-19T09:19:29.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.701+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f93bc04be40 con 0x7f93bc06c7c0 2023-12-19T09:19:29.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.709+0000 7f93b17fa700 1 -- 172.21.15.111:0/63071028 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f93b8050420 con 0x7f93bc06c7c0 2023-12-19T09:19:29.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 msgr2=0x7f93a0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 0x7f93a0062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0cb990 tx=0x7f93ac017000 comp rx=0 tx=0).stop 2023-12-19T09:19:29.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 msgr2=0x7f93bc15e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f93b800c8e0 tx=0x7f93b8001710 comp rx=0 tx=0).stop 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 shutdown_connections 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f93a0060550 0x7f93a0062a00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93bc0c69a0 0x7f93bc14cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93bc06de00 0x7f93bc15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 --2- 172.21.15.111:0/63071028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc06c7c0 0x7f93bc15e810 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 >> 172.21.15.111:0/63071028 conn(0x7f93bc067b50 msgr2=0x7f93bc0c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:29.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 shutdown_connections 2023-12-19T09:19:29.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.713+0000 7f93c268e700 1 -- 172.21.15.111:0/63071028 wait complete. 2023-12-19T09:19:29.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T09:19:29.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.897+0000 7fc66b0f0700 1 Processor -- start 2023-12-19T09:19:29.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.897+0000 7fc66b0f0700 1 -- start start 2023-12-19T09:19:29.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc66b0f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6640b1ed0 0x7fc6640684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc66b0f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc664067790 con 0x7fc6640b0cb0 2023-12-19T09:19:29.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc66b0f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc664068030 con 0x7fc664159790 2023-12-19T09:19:29.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc66b0f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640b1200 con 0x7fc6640b1ed0 2023-12-19T09:19:29.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6640b1ed0 0x7fc6640684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6640b1ed0 0x7fc6640684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55902/0 (socket says 172.21.15.111:55902) 2023-12-19T09:19:29.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 -- 172.21.15.111:0/2677803993 learned_addr learned my addr 172.21.15.111:0/2677803993 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:29.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 -- 172.21.15.111:0/2677803993 >> v1:172.21.15.143:6789/0 conn(0x7fc664159790 legacy=0x7fc6640b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 -- 172.21.15.111:0/2677803993 >> v1:172.21.15.111:6789/0 conn(0x7fc6640b0cb0 legacy=0x7fc664159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 -- 172.21.15.111:0/2677803993 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640679e0 con 0x7fc6640b1ed0 2023-12-19T09:19:29.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc6637fe700 1 -- 172.21.15.111:0/2677803993 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574899640 0 0) 0x7fc664067790 con 0x7fc6640b0cb0 2023-12-19T09:19:29.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc663fff700 1 --2- 172.21.15.111:0/2677803993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b1ed0 0x7fc6640684c0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc6500099b0 tx=0x7fc650009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4906aa0404ee82a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc6637fe700 1 -- 172.21.15.111:0/2677803993 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000b860 con 0x7fc6640b1ed0 2023-12-19T09:19:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc6637fe700 1 -- 172.21.15.111:0/2677803993 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65000b9c0 con 0x7fc6640b1ed0 2023-12-19T09:19:29.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.901+0000 7fc6637fe700 1 -- 172.21.15.111:0/2677803993 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000bd80 con 0x7fc6640b1ed0 2023-12-19T09:19:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2677803993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b1ed0 msgr2=0x7fc6640684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2677803993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b1ed0 0x7fc6640684c0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc6500099b0 tx=0x7fc650009250 comp rx=0 tx=0).stop 2023-12-19T09:19:29.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2677803993 shutdown_connections 2023-12-19T09:19:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2677803993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b1ed0 0x7fc6640684c0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2677803993 >> 172.21.15.111:0/2677803993 conn(0x7fc6640cdef0 msgr2=0x7fc6640ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2677803993 shutdown_connections 2023-12-19T09:19:29.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.905+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2677803993 wait complete. 2023-12-19T09:19:29.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 Processor -- start 2023-12-19T09:19:29.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 -- start start 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 0x7fc66416a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 0x7fc6640ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640cd710 con 0x7fc6640b0cb0 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66413ea80 con 0x7fc664159790 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66b0f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66415d350 con 0x7fc6640b1ed0 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55914/0 (socket says 172.21.15.111:55914) 2023-12-19T09:19:29.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 -- 172.21.15.111:0/2850649234 learned_addr learned my addr 172.21.15.111:0/2850649234 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc663fff700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 0x7fc66416a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 -- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 msgr2=0x7fc66416a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66968d700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 0x7fc6640ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 0x7fc66416a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 -- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 msgr2=0x7fc6640ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 0x7fc6640ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 -- 172.21.15.111:0/2850649234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500094f0 con 0x7fc6640b0cb0 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc663fff700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 0x7fc66416a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:29.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc66968d700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 0x7fc6640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:29.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.909+0000 7fc668e8c700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c4e0 tx=0x7fc65800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658011590 con 0x7fc6640b0cb0 2023-12-19T09:19:29.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66415ca20 con 0x7fc6640b0cb0 2023-12-19T09:19:29.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66415cfe0 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc658001150 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658011900 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6440052b0 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.913+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc658011a60 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.917+0000 7fc660ff9700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 0x7fc648062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.917+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fc65807be10 con 0x7fc6640b0cb0 2023-12-19T09:19:29.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.917+0000 7fc663fff700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 0x7fc648062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:29.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.917+0000 7fc663fff700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 0x7fc648062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6500099b0 tx=0x7fc650017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:29.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:29.925+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc658049be0 con 0x7fc6640b0cb0 2023-12-19T09:19:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.093+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc644005ae0 con 0x7fc6640b0cb0 2023-12-19T09:19:30.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.105+0000 7fc660ff9700 1 -- 172.21.15.111:0/2850649234 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v63) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc6580170b0 con 0x7fc6640b0cb0 2023-12-19T09:19:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 msgr2=0x7fc648062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 0x7fc648062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6500099b0 tx=0x7fc650017000 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 msgr2=0x7fc66413e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c4e0 tx=0x7fc65800c8a0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 shutdown_connections 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc648060550 0x7fc648062a00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc664159790 0x7fc6640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6640b1ed0 0x7fc66416a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 --2- 172.21.15.111:0/2850649234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6640b0cb0 0x7fc66413e540 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 >> 172.21.15.111:0/2850649234 conn(0x7fc6640cdef0 msgr2=0x7fc664154ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 shutdown_connections 2023-12-19T09:19:30.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.109+0000 7fc66b0f0700 1 -- 172.21.15.111:0/2850649234 wait complete. 2023-12-19T09:19:30.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-12-19T09:19:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 Processor -- start 2023-12-19T09:19:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 -- start start 2023-12-19T09:19:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 0x7f94300680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943006e150 con 0x7f9430068630 2023-12-19T09:19:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943006d540 con 0x7f9430140710 2023-12-19T09:19:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9436625700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9430068ab0 con 0x7f9430140ae0 2023-12-19T09:19:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9434bc2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9430068630 0x7f94300ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:49714/0 (socket says 172.21.15.111:49714) 2023-12-19T09:19:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9434bc2700 1 -- 172.21.15.111:0/2928036589 learned_addr learned my addr 172.21.15.111:0/2928036589 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.301+0000 7f9427fff700 1 --2- 172.21.15.111:0/2928036589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 0x7f94300680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f942f7fe700 1 -- 172.21.15.111:0/2928036589 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535408516 0 0) 0x7f943006e150 con 0x7f9430068630 2023-12-19T09:19:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f942f7fe700 1 -- 172.21.15.111:0/2928036589 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f941c003570 con 0x7f9430068630 2023-12-19T09:19:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9427fff700 1 -- 172.21.15.111:0/2928036589 >> v1:172.21.15.143:6789/0 conn(0x7f9430140710 legacy=0x7f94300c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9427fff700 1 -- 172.21.15.111:0/2928036589 >> v1:172.21.15.111:6789/0 conn(0x7f9430068630 legacy=0x7f94300ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9427fff700 1 -- 172.21.15.111:0/2928036589 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943006d820 con 0x7f9430140ae0 2023-12-19T09:19:30.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9427fff700 1 --2- 172.21.15.111:0/2928036589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 0x7f94300680f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9418009910 tx=0x7f94180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2890c1d9cd6e41 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f942f7fe700 1 -- 172.21.15.111:0/2928036589 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800b680 con 0x7f9430140ae0 2023-12-19T09:19:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f942f7fe700 1 -- 172.21.15.111:0/2928036589 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941800b7e0 con 0x7f9430140ae0 2023-12-19T09:19:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f942f7fe700 1 -- 172.21.15.111:0/2928036589 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800bba0 con 0x7f9430140ae0 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 -- 172.21.15.111:0/2928036589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 msgr2=0x7f94300680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 --2- 172.21.15.111:0/2928036589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 0x7f94300680f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9418009910 tx=0x7f94180091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 -- 172.21.15.111:0/2928036589 shutdown_connections 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 --2- 172.21.15.111:0/2928036589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140ae0 0x7f94300680f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 -- 172.21.15.111:0/2928036589 >> 172.21.15.111:0/2928036589 conn(0x7f94300c8850 msgr2=0x7f94300c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.305+0000 7f9436625700 1 -- 172.21.15.111:0/2928036589 shutdown_connections 2023-12-19T09:19:30.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 -- 172.21.15.111:0/2928036589 wait complete. 2023-12-19T09:19:30.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 Processor -- start 2023-12-19T09:19:30.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 -- start start 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9430068630 0x7f94300a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 0x7f9430132860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300a9010 con 0x7f9430140ae0 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300ab2f0 con 0x7f9430068630 2023-12-19T09:19:30.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.309+0000 7f9436625700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9430132da0 con 0x7f9430140710 2023-12-19T09:19:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9427fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 0x7f9430132860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55920/0 (socket says 172.21.15.111:55920) 2023-12-19T09:19:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 -- 172.21.15.111:0/3643333045 learned_addr learned my addr 172.21.15.111:0/3643333045 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:30.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f942ffff700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9430068630 0x7f94300a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 -- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 msgr2=0x7f9430132860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 0x7f9430132860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 -- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9430068630 msgr2=0x7f94300a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9430068630 0x7f94300a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 -- 172.21.15.111:0/3643333045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418009420 con 0x7f9430140ae0 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9427fff700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 0x7f9430132860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:30.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9434bc2700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9428001460 tx=0x7f9428001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:30.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942800a890 con 0x7f9430140ae0 2023-12-19T09:19:30.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942800a9f0 con 0x7f9430140ae0 2023-12-19T09:19:30.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942800ad70 con 0x7f9430140ae0 2023-12-19T09:19:30.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9430157d20 con 0x7f9430140ae0 2023-12-19T09:19:30.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.313+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943015a3f0 con 0x7f9430140ae0 2023-12-19T09:19:30.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.317+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94300a9150 con 0x7f9430140ae0 2023-12-19T09:19:30.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.317+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9428007300 con 0x7f9430140ae0 2023-12-19T09:19:30.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.317+0000 7f942cff9700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 0x7f9410062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:30.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.317+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f942807d070 con 0x7f9430140ae0 2023-12-19T09:19:30.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.317+0000 7f942ffff700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 0x7f9410062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:30.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.321+0000 7f942ffff700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 0x7f9410062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9420003a80 tx=0x7f9420007ec0 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:30.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.325+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f942804d180 con 0x7f9430140ae0 2023-12-19T09:19:30.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.553+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f943015a5e0 con 0x7f9430140ae0 2023-12-19T09:19:30.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.925+0000 7f942cff9700 1 -- 172.21.15.111:0/3643333045 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2972) v1 ==== 165+0+0 (secure 0 0 0) 0x7f94280507a0 con 0x7f9430140ae0 2023-12-19T09:19:30.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 msgr2=0x7f9410062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 0x7f9410062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9420003a80 tx=0x7f9420007ec0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 msgr2=0x7f94300ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f9428001460 tx=0x7f9428001ae0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 shutdown_connections 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9430140ae0 0x7f94300ad220 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9430140710 0x7f9430132860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9410060550 0x7f9410062a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 --2- 172.21.15.111:0/3643333045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9430068630 0x7f94300a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 >> 172.21.15.111:0/3643333045 conn(0x7f94300c8850 msgr2=0x7f943006a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 shutdown_connections 2023-12-19T09:19:30.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:30.929+0000 7f9436625700 1 -- 172.21.15.111:0/3643333045 wait complete. 2023-12-19T09:19:30.956 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-12-19T09:19:30.956 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-12-19T09:19:30.957 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-12-19T09:19:30.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:31.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.129+0000 7f0d98a4c700 1 Processor -- start 2023-12-19T09:19:31.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.129+0000 7f0d98a4c700 1 -- start start 2023-12-19T09:19:31.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d98a4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0d94158e70 0x7f0d940afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d98a4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d9406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d98a4c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d940ca5a0 con 0x7f0d940cc470 2023-12-19T09:19:31.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d98a4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9406dac0 con 0x7f0d94158e70 2023-12-19T09:19:31.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d98a4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940b0260 con 0x7f0d9406a540 2023-12-19T09:19:31.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0d94158e70 0x7f0d940afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0d94158e70 0x7f0d940afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55930/0 (socket says 172.21.15.111:55930) 2023-12-19T09:19:31.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 -- 172.21.15.111:0/3712061119 learned_addr learned my addr 172.21.15.111:0/3712061119 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:31.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d92ffd700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d9406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 -- 172.21.15.111:0/3712061119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 msgr2=0x7f0d9406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:31.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d9406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 -- 172.21.15.111:0/3712061119 >> v1:172.21.15.143:6789/0 conn(0x7f0d940cc470 legacy=0x7f0d940cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 -- 172.21.15.111:0/3712061119 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9406d220 con 0x7f0d94158e70 2023-12-19T09:19:31.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d937fe700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d940afd20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0d840099b0 tx=0x7f0d84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11342e5eacf7940 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d927fc700 1 -- 172.21.15.111:0/3712061119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84009ea0 con 0x7f0d94158e70 2023-12-19T09:19:31.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d927fc700 1 -- 172.21.15.111:0/3712061119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8400b860 con 0x7f0d94158e70 2023-12-19T09:19:31.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.133+0000 7f0d927fc700 1 -- 172.21.15.111:0/3712061119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8400bc20 con 0x7f0d94158e70 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- 172.21.15.111:0/3712061119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 msgr2=0x7f0d940afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d940afd20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0d840099b0 tx=0x7f0d84009250 comp rx=0 tx=0).stop 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- 172.21.15.111:0/3712061119 shutdown_connections 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d9406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/3712061119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d940afd20 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- 172.21.15.111:0/3712061119 >> 172.21.15.111:0/3712061119 conn(0x7f0d940670e0 msgr2=0x7f0d940674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- 172.21.15.111:0/3712061119 shutdown_connections 2023-12-19T09:19:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- 172.21.15.111:0/3712061119 wait complete. 2023-12-19T09:19:31.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 Processor -- start 2023-12-19T09:19:31.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.137+0000 7f0d98a4c700 1 -- start start 2023-12-19T09:19:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d94154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d940cc470 0x7f0d940c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940c9900 con 0x7f0d94158e70 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940c8cf0 con 0x7f0d940cc470 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d98a4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940c8fd0 con 0x7f0d9406a540 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55946/0 (socket says 172.21.15.111:55946) 2023-12-19T09:19:31.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 -- 172.21.15.111:0/1685362347 learned_addr learned my addr 172.21.15.111:0/1685362347 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d937fe700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d94154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d92ffd700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d940cc470 0x7f0d940c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 -- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 msgr2=0x7f0d94154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d94154c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 -- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d940cc470 msgr2=0x7f0d940c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d940cc470 0x7f0d940c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 -- 172.21.15.111:0/1685362347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840094f0 con 0x7f0d94158e70 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d937fe700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d94154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.141+0000 7f0d93fff700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00c8e0 tx=0x7f0d8c001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c012630 con 0x7f0d94158e70 2023-12-19T09:19:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8c012790 con 0x7f0d94158e70 2023-12-19T09:19:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c012b50 con 0x7f0d94158e70 2023-12-19T09:19:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d940c92b0 con 0x7f0d94158e70 2023-12-19T09:19:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d94155140 con 0x7f0d94158e70 2023-12-19T09:19:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.145+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d940c9ab0 con 0x7f0d94158e70 2023-12-19T09:19:31.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.149+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0d8c01a3a0 con 0x7f0d94158e70 2023-12-19T09:19:31.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.149+0000 7f0d7b7fe700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 0x7f0d80062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.149+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f0d8c07cdc0 con 0x7f0d94158e70 2023-12-19T09:19:31.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.149+0000 7f0d937fe700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 0x7f0d80062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.153+0000 7f0d937fe700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 0x7f0d80062a30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d840099b0 tx=0x7f0d84017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.157+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d940c9ab0 con 0x7f0d94158e70 2023-12-19T09:19:31.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.337+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d9404be40 con 0x7f0d94158e70 2023-12-19T09:19:31.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.337+0000 7f0d7b7fe700 1 -- 172.21.15.111:0/1685362347 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+14782 (secure 0 0 0) 0x7f0d8c050970 con 0x7f0d94158e70 2023-12-19T09:19:31.343 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:31.343 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":184,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:28.903249+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/1477157101":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/1130046391":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/1351482287":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/3402884462":"2023-12-20T09:19:25.637482+0000","172.21.15.143:6835/811953301":"2023-12-20T09:19:25.730504+0000","172.21.15.143:0/2893890504":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3589389202":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/57208669":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2193897545":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/3353020814":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/3287144757":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/2770672326":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/4214132662":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6833/43868":"2023-12-20T09:19:25.637482+0000","172.21.15.143:6832/43868":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/3599201417":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6834/811953301":"2023-12-20T09:19:25.730504+0000","172.21.15.143:0/2998303494":"2023-12-20T09:14:20.977129+0000","172.21.15.143:0/1690815342":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/3112216012":"2023-12-20T09:19:25.637482+0000","172.21.15.143:0/3967653215":"2023-12-20T09:03:31.913274+0000","172.21.15.143:0/2978228531":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6833/33796":"2023-12-20T09:03:31.913274+0000","172.21.15.143:6832/33796":"2023-12-20T09:03:31.913274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:31.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 msgr2=0x7f0d80062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 0x7f0d80062a30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d840099b0 tx=0x7f0d84017000 comp rx=0 tx=0).stop 2023-12-19T09:19:31.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 msgr2=0x7f0d94157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00c8e0 tx=0x7f0d8c001710 comp rx=0 tx=0).stop 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 shutdown_connections 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0d80060580 0x7f0d80062a30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d940cc470 0x7f0d940c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d94158e70 0x7f0d94157b40 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 --2- 172.21.15.111:0/1685362347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d9406a540 0x7f0d94154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 >> 172.21.15.111:0/1685362347 conn(0x7f0d940670e0 msgr2=0x7f0d940cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 shutdown_connections 2023-12-19T09:19:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.341+0000 7f0d98a4c700 1 -- 172.21.15.111:0/1685362347 wait complete. 2023-12-19T09:19:31.367 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:19:31.367 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:19:31.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.529+0000 7f9742fcc700 1 Processor -- start 2023-12-19T09:19:31.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.529+0000 7f9742fcc700 1 -- start start 2023-12-19T09:19:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f9742fcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f973c06aeb0 0x7f973c159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f9742fcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f973c1595a0 0x7f973c1599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f9742fcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c06c850 con 0x7f973c158c90 2023-12-19T09:19:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f9742fcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c0cb650 con 0x7f973c06aeb0 2023-12-19T09:19:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f9742fcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c06b470 con 0x7f973c1595a0 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f973c06aeb0 0x7f973c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f973c06aeb0 0x7f973c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55948/0 (socket says 172.21.15.111:55948) 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97427cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f973c1595a0 0x7f973c1599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 -- 172.21.15.111:0/3880311885 learned_addr learned my addr 172.21.15.111:0/3880311885 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 -- 172.21.15.111:0/3880311885 >> v2:172.21.15.143:3300/0 conn(0x7f973c1595a0 msgr2=0x7f973c1599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 --2- 172.21.15.111:0/3880311885 >> v2:172.21.15.143:3300/0 conn(0x7f973c1595a0 0x7f973c1599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 -- 172.21.15.111:0/3880311885 >> v1:172.21.15.143:6789/0 conn(0x7f973c158c90 legacy=0x7f973c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 -- 172.21.15.111:0/3880311885 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c06b750 con 0x7f973c06aeb0 2023-12-19T09:19:31.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.533+0000 7f97417c9700 1 --2- 172.21.15.111:0/3880311885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c06aeb0 0x7f973c159060 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f973c0cc7f0 tx=0x7f972c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5202749bf662e3d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9740fc8700 1 -- 172.21.15.111:0/3880311885 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c009e70 con 0x7f973c06aeb0 2023-12-19T09:19:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9740fc8700 1 -- 172.21.15.111:0/3880311885 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c00b860 con 0x7f973c06aeb0 2023-12-19T09:19:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9740fc8700 1 -- 172.21.15.111:0/3880311885 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c00bc00 con 0x7f973c06aeb0 2023-12-19T09:19:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 -- 172.21.15.111:0/3880311885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c06aeb0 msgr2=0x7f973c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3880311885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c06aeb0 0x7f973c159060 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f973c0cc7f0 tx=0x7f972c009250 comp rx=0 tx=0).stop 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 -- 172.21.15.111:0/3880311885 shutdown_connections 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3880311885 >> v2:172.21.15.143:3300/0 conn(0x7f973c1595a0 0x7f973c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3880311885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c06aeb0 0x7f973c159060 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 -- 172.21.15.111:0/3880311885 >> 172.21.15.111:0/3880311885 conn(0x7f973c067bb0 msgr2=0x7f973c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 -- 172.21.15.111:0/3880311885 shutdown_connections 2023-12-19T09:19:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 -- 172.21.15.111:0/3880311885 wait complete. 2023-12-19T09:19:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.537+0000 7f9742fcc700 1 Processor -- start 2023-12-19T09:19:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 -- start start 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973c06aeb0 0x7f973c14a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973c158c90 0x7f973c16d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c16e7a0 con 0x7f973c1595a0 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c087410 con 0x7f973c158c90 2023-12-19T09:19:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c14a540 con 0x7f973c06aeb0 2023-12-19T09:19:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55964/0 (socket says 172.21.15.111:55964) 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 -- 172.21.15.111:0/3755008064 learned_addr learned my addr 172.21.15.111:0/3755008064 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97417c9700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973c158c90 0x7f973c16d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9741fca700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973c06aeb0 0x7f973c14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 -- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973c06aeb0 msgr2=0x7f973c14a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973c06aeb0 0x7f973c14a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 -- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973c158c90 msgr2=0x7f973c16d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973c158c90 0x7f973c16d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 -- 172.21.15.111:0/3755008064 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0094f0 con 0x7f973c1595a0 2023-12-19T09:19:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f97427cb700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f9734009df0 tx=0x7f973400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400cb50 con 0x7f973c1595a0 2023-12-19T09:19:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973400f040 con 0x7f973c1595a0 2023-12-19T09:19:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c16e1e0 con 0x7f973c1595a0 2023-12-19T09:19:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.541+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734014480 con 0x7f973c1595a0 2023-12-19T09:19:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.545+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c0870a0 con 0x7f973c1595a0 2023-12-19T09:19:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.545+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c0875a0 con 0x7f973c1595a0 2023-12-19T09:19:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.549+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9734014830 con 0x7f973c1595a0 2023-12-19T09:19:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.549+0000 7f9731ffb700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 0x7f9720062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.549+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f973407d530 con 0x7f973c1595a0 2023-12-19T09:19:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.549+0000 7f9741fca700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 0x7f9720062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.553+0000 7f9741fca700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 0x7f9720062a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97380037c0 tx=0x7f9738009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.553+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f973404d650 con 0x7f973c1595a0 2023-12-19T09:19:31.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.721+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f973c04be40 con 0x7f973c1595a0 2023-12-19T09:19:31.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.721+0000 7f9731ffb700 1 -- 172.21.15.111:0/3755008064 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9734050c70 con 0x7f973c1595a0 2023-12-19T09:19:31.727 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:19:31.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 msgr2=0x7f9720062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 0x7f9720062a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97380037c0 tx=0x7f9738009040 comp rx=0 tx=0).stop 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 msgr2=0x7f973c0878a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f9734009df0 tx=0x7f973400b040 comp rx=0 tx=0).stop 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 shutdown_connections 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9720060580 0x7f9720062a30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973c158c90 0x7f973c16d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f973c1595a0 0x7f973c0878a0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 --2- 172.21.15.111:0/3755008064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973c06aeb0 0x7f973c14a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 >> 172.21.15.111:0/3755008064 conn(0x7f973c067bb0 msgr2=0x7f973c06a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 shutdown_connections 2023-12-19T09:19:31.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.725+0000 7f9742fcc700 1 -- 172.21.15.111:0/3755008064 wait complete. 2023-12-19T09:19:31.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-12-19T09:19:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 Processor -- start 2023-12-19T09:19:31.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 -- start start 2023-12-19T09:19:31.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efbfc069070 0x7efbfc067960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 0x7efbfc0cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbfc068be0 con 0x7efbfc158d40 2023-12-19T09:19:31.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc159530 con 0x7efbfc159110 2023-12-19T09:19:31.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.905+0000 7efc02465700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc067fd0 con 0x7efbfc069070 2023-12-19T09:19:31.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7efbfc158d40 0x7efbfc0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42346/0 (socket says 172.21.15.111:42346) 2023-12-19T09:19:31.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfbfff700 1 -- 172.21.15.111:0/2377810552 learned_addr learned my addr 172.21.15.111:0/2377810552 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:31.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/2377810552 >> v2:172.21.15.143:3300/0 conn(0x7efbfc069070 0x7efbfc067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc00a02700 1 --2- 172.21.15.111:0/2377810552 >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 0x7efbfc0cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfaffd700 1 -- 172.21.15.111:0/2377810552 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134963243 0 0) 0x7efbfc068be0 con 0x7efbfc158d40 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfaffd700 1 -- 172.21.15.111:0/2377810552 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003570 con 0x7efbfc158d40 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 -- 172.21.15.111:0/2377810552 >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 msgr2=0x7efbfc0cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/2377810552 >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 0x7efbfc0cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 -- 172.21.15.111:0/2377810552 >> v1:172.21.15.143:6789/0 conn(0x7efbfc158d40 legacy=0x7efbfc0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 -- 172.21.15.111:0/2377810552 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0682b0 con 0x7efbfc069070 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc00a02700 1 --2- 172.21.15.111:0/2377810552 >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 0x7efbfc0cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/2377810552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc069070 0x7efbfc067960 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ce5c0 tx=0x7efbf00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d96af07e9217961 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfaffd700 1 -- 172.21.15.111:0/2377810552 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0011630 con 0x7efbfc069070 2023-12-19T09:19:31.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfaffd700 1 -- 172.21.15.111:0/2377810552 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf00117b0 con 0x7efbfc069070 2023-12-19T09:19:31.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efbfaffd700 1 -- 172.21.15.111:0/2377810552 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0011b70 con 0x7efbfc069070 2023-12-19T09:19:31.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 -- 172.21.15.111:0/2377810552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc069070 msgr2=0x7efbfc067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 --2- 172.21.15.111:0/2377810552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc069070 0x7efbfc067960 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ce5c0 tx=0x7efbf00091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 -- 172.21.15.111:0/2377810552 shutdown_connections 2023-12-19T09:19:31.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 --2- 172.21.15.111:0/2377810552 >> v2:172.21.15.111:3300/0 conn(0x7efbfc159110 0x7efbfc0cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 --2- 172.21.15.111:0/2377810552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc069070 0x7efbfc067960 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 -- 172.21.15.111:0/2377810552 >> 172.21.15.111:0/2377810552 conn(0x7efbfc0ccc40 msgr2=0x7efbfc0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:31.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 -- 172.21.15.111:0/2377810552 shutdown_connections 2023-12-19T09:19:31.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.909+0000 7efc02465700 1 -- 172.21.15.111:0/2377810552 wait complete. 2023-12-19T09:19:31.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 Processor -- start 2023-12-19T09:19:31.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 -- start start 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbfc158d40 0x7efbfc16c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc159110 0x7efbfc0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0b00e0 con 0x7efbfc069070 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0afad0 con 0x7efbfc159110 2023-12-19T09:19:31.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efc02465700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc148190 con 0x7efbfc158d40 2023-12-19T09:19:31.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.913+0000 7efbfbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41232/0 (socket says 172.21.15.111:41232) 2023-12-19T09:19:31.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 -- 172.21.15.111:0/38881348 learned_addr learned my addr 172.21.15.111:0/38881348 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:31.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbfc158d40 0x7efbfc16c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efc00a02700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc159110 0x7efbfc0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 -- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbfc158d40 msgr2=0x7efbfc16c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbfc158d40 0x7efbfc16c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 -- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc159110 msgr2=0x7efbfc0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc159110 0x7efbfc0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 -- 172.21.15.111:0/38881348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4009a20 con 0x7efbfc069070 2023-12-19T09:19:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbfbfff700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c7c0 tx=0x7efbec007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec005b30 con 0x7efbfc069070 2023-12-19T09:19:31.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbec005cb0 con 0x7efbfc069070 2023-12-19T09:19:31.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0009420 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec007320 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc16d720 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.917+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc071120 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.921+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efbec01b3b0 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.921+0000 7efbe3fff700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 0x7efbe8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.921+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7efbec07c020 con 0x7efbfc069070 2023-12-19T09:19:31.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.925+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 0x7efbe8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.925+0000 7efbfb7fe700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 0x7efbe8062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ce5c0 tx=0x7efbf0017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:31.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:31.929+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efbec04c2a0 con 0x7efbfc069070 2023-12-19T09:19:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.157+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7efbfc068590 con 0x7efbfc069070 2023-12-19T09:19:32.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.941+0000 7efbe3fff700 1 -- 172.21.15.111:0/38881348 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v2974) v1 ==== 166+0+0 (secure 0 0 0) 0x7efbec04f8c0 con 0x7efbfc069070 2023-12-19T09:19:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 msgr2=0x7efbe8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 0x7efbe8062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0ce5c0 tx=0x7efbf0017000 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 msgr2=0x7efbfc147c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c7c0 tx=0x7efbec007e00 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 shutdown_connections 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efbe8060550 0x7efbe8062a00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbfc159110 0x7efbfc0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efbfc158d40 0x7efbfc16c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 --2- 172.21.15.111:0/38881348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbfc069070 0x7efbfc147c50 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 >> 172.21.15.111:0/38881348 conn(0x7efbfc0ccc40 msgr2=0x7efbfc0cb600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 shutdown_connections 2023-12-19T09:19:32.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:32.945+0000 7efc02465700 1 -- 172.21.15.111:0/38881348 wait complete. 2023-12-19T09:19:32.972 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T09:19:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.145+0000 7f67dd3c5700 1 Processor -- start 2023-12-19T09:19:33.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.145+0000 7f67dd3c5700 1 -- start start 2023-12-19T09:19:33.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67dd3c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f67d80cb970 0x7f67d806acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67dd3c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80cbdf0 con 0x7f67d80cecd0 2023-12-19T09:19:33.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67dd3c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80cc0d0 con 0x7f67d8158e20 2023-12-19T09:19:33.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67dd3c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80cc3b0 con 0x7f67d80cb970 2023-12-19T09:19:33.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f67d80cecd0 0x7f67d806b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33644/0 (socket says 172.21.15.111:33644) 2023-12-19T09:19:33.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d77fe700 1 -- 172.21.15.111:0/3741271913 learned_addr learned my addr 172.21.15.111:0/3741271913 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:33.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d67fc700 1 --2- 172.21.15.111:0/3741271913 >> v2:172.21.15.143:3300/0 conn(0x7f67d80cb970 0x7f67d806acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d5ffb700 1 -- 172.21.15.111:0/3741271913 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936792874 0 0) 0x7f67d80cbdf0 con 0x7f67d80cecd0 2023-12-19T09:19:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d5ffb700 1 -- 172.21.15.111:0/3741271913 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c0003570 con 0x7f67d80cecd0 2023-12-19T09:19:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d67fc700 1 -- 172.21.15.111:0/3741271913 >> v1:172.21.15.143:6789/0 conn(0x7f67d8158e20 legacy=0x7f67d80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d67fc700 1 -- 172.21.15.111:0/3741271913 >> v1:172.21.15.111:6789/0 conn(0x7f67d80cecd0 legacy=0x7f67d806b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d67fc700 1 -- 172.21.15.111:0/3741271913 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80cc690 con 0x7f67d80cb970 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d67fc700 1 --2- 172.21.15.111:0/3741271913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80cb970 0x7f67d806acc0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009910 tx=0x7f67cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9113ab400053a0db server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d5ffb700 1 -- 172.21.15.111:0/3741271913 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00b640 con 0x7f67d80cb970 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d5ffb700 1 -- 172.21.15.111:0/3741271913 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc00b7a0 con 0x7f67d80cb970 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.149+0000 7f67d5ffb700 1 -- 172.21.15.111:0/3741271913 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00bb60 con 0x7f67d80cb970 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- 172.21.15.111:0/3741271913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80cb970 msgr2=0x7f67d806acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/3741271913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80cb970 0x7f67d806acc0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009910 tx=0x7f67cc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- 172.21.15.111:0/3741271913 shutdown_connections 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/3741271913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80cb970 0x7f67d806acc0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- 172.21.15.111:0/3741271913 >> 172.21.15.111:0/3741271913 conn(0x7f67d8067bb0 msgr2=0x7f67d80d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- 172.21.15.111:0/3741271913 shutdown_connections 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- 172.21.15.111:0/3741271913 wait complete. 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 Processor -- start 2023-12-19T09:19:33.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.153+0000 7f67dd3c5700 1 -- start start 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67d80cecd0 0x7f67d80c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d8158e20 0x7f67d814bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8151eb0 con 0x7f67d814d3a0 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8152190 con 0x7f67d8158e20 2023-12-19T09:19:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8152470 con 0x7f67d80cecd0 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41246/0 (socket says 172.21.15.111:41246) 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 -- 172.21.15.111:0/2093318430 learned_addr learned my addr 172.21.15.111:0/2093318430 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d67fc700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d8158e20 0x7f67d814bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d6ffd700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67d80cecd0 0x7f67d80c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 -- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67d80cecd0 msgr2=0x7f67d80c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67d80cecd0 0x7f67d80c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 -- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d8158e20 msgr2=0x7f67d814bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d8158e20 0x7f67d814bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 -- 172.21.15.111:0/2093318430 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc009420 con 0x7f67d814d3a0 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67d77fe700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f67d000cc40 tx=0x7f67d0001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000a8b0 con 0x7f67d814d3a0 2023-12-19T09:19:33.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d000aa10 con 0x7f67d814d3a0 2023-12-19T09:19:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d8152750 con 0x7f67d814d3a0 2023-12-19T09:19:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000ad90 con 0x7f67d814d3a0 2023-12-19T09:19:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.157+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d80c9fa0 con 0x7f67d814d3a0 2023-12-19T09:19:33.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.161+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d8071120 con 0x7f67d814d3a0 2023-12-19T09:19:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.161+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f67d0007300 con 0x7f67d814d3a0 2023-12-19T09:19:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.165+0000 7f67beffd700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 0x7f67c4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.165+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f67d007cfb0 con 0x7f67d814d3a0 2023-12-19T09:19:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.165+0000 7f67d6ffd700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 0x7f67c4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.165+0000 7f67d6ffd700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 0x7f67c4062a00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67c8003a80 tx=0x7f67c8007ec0 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.169+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f67d004d150 con 0x7f67d814d3a0 2023-12-19T09:19:33.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.345+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f67d806a7a0 con 0x7f67d814d3a0 2023-12-19T09:19:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.353+0000 7f67beffd700 1 -- 172.21.15.111:0/2093318430 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f67d0050770 con 0x7f67d814d3a0 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 msgr2=0x7f67c4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 0x7f67c4062a00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f67c8003a80 tx=0x7f67c8007ec0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 msgr2=0x7f67d814d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f67d000cc40 tx=0x7f67d0001aa0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 shutdown_connections 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f67c4060550 0x7f67c4062a00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d8158e20 0x7f67d814bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67d814d3a0 0x7f67d814d810 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 --2- 172.21.15.111:0/2093318430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67d80cecd0 0x7f67d80c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 >> 172.21.15.111:0/2093318430 conn(0x7f67d8067bb0 msgr2=0x7f67d8156670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 shutdown_connections 2023-12-19T09:19:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.357+0000 7f67dd3c5700 1 -- 172.21.15.111:0/2093318430 wait complete. 2023-12-19T09:19:33.384 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:19:33.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:19:33.426 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:19:33.473 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:19:33.473 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T09:19:33.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:33.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.637+0000 7fb60a8a5700 1 Processor -- start 2023-12-19T09:19:33.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 -- start start 2023-12-19T09:19:33.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6041552f0 0x7fb6041556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6040ae520 0x7fb6040c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb60406c7c0 0x7fb60406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60406d120 con 0x7fb60406c7c0 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60406a9d0 con 0x7fb6040ae520 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb60a8a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60406acb0 con 0x7fb6041552f0 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb60406c7c0 0x7fb60406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb60406c7c0 0x7fb60406cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41248/0 (socket says 172.21.15.111:41248) 2023-12-19T09:19:33.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 -- 172.21.15.111:0/292287780 learned_addr learned my addr 172.21.15.111:0/292287780 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:33.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb603fff700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6041552f0 0x7fb6041556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 -- 172.21.15.111:0/292287780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6041552f0 msgr2=0x7fb6041556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6041552f0 0x7fb6041556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 -- 172.21.15.111:0/292287780 >> v2:172.21.15.143:3300/0 conn(0x7fb6040ae520 msgr2=0x7fb6040c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 --2- 172.21.15.111:0/292287780 >> v2:172.21.15.143:3300/0 conn(0x7fb6040ae520 0x7fb6040c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.641+0000 7fb608e42700 1 -- 172.21.15.111:0/292287780 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60406af90 con 0x7fb60406c7c0 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb608e42700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb60406c7c0 0x7fb60406cbe0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fb60406a160 tx=0x7fb5fc00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f1220fea247941 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb6037fe700 1 -- 172.21.15.111:0/292287780 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc010070 con 0x7fb60406c7c0 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb6037fe700 1 -- 172.21.15.111:0/292287780 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc00de90 con 0x7fb60406c7c0 2023-12-19T09:19:33.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb6037fe700 1 -- 172.21.15.111:0/292287780 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00b320 con 0x7fb60406c7c0 2023-12-19T09:19:33.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 -- 172.21.15.111:0/292287780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb60406c7c0 msgr2=0x7fb60406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb60406c7c0 0x7fb60406cbe0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fb60406a160 tx=0x7fb5fc00ba50 comp rx=0 tx=0).stop 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 -- 172.21.15.111:0/292287780 shutdown_connections 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb60406c7c0 0x7fb60406cbe0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/292287780 >> v2:172.21.15.143:3300/0 conn(0x7fb6040ae520 0x7fb6040c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/292287780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6041552f0 0x7fb6041556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 -- 172.21.15.111:0/292287780 >> 172.21.15.111:0/292287780 conn(0x7fb6040c8850 msgr2=0x7fb6040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 -- 172.21.15.111:0/292287780 shutdown_connections 2023-12-19T09:19:33.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 -- 172.21.15.111:0/292287780 wait complete. 2023-12-19T09:19:33.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.645+0000 7fb60a8a5700 1 Processor -- start 2023-12-19T09:19:33.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 -- start start 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60406c7c0 0x7fb6040b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6041552f0 0x7fb60415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604159930 con 0x7fb6040ae520 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604159c10 con 0x7fb6041552f0 2023-12-19T09:19:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb604159ef0 con 0x7fb60406c7c0 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41252/0 (socket says 172.21.15.111:41252) 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 -- 172.21.15.111:0/4044425679 learned_addr learned my addr 172.21.15.111:0/4044425679 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb603fff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60406c7c0 0x7fb6040b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb608e42700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6041552f0 0x7fb60415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 -- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60406c7c0 msgr2=0x7fb6040b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60406c7c0 0x7fb6040b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 -- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6041552f0 msgr2=0x7fb60415e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6041552f0 0x7fb60415e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 -- 172.21.15.111:0/4044425679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc00bcf0 con 0x7fb6040ae520 2023-12-19T09:19:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb5fbfff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00b810 tx=0x7fb5ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00f510 con 0x7fb6040ae520 2023-12-19T09:19:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec00f690 con 0x7fb6040ae520 2023-12-19T09:19:33.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60415a1d0 con 0x7fb6040ae520 2023-12-19T09:19:33.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.649+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00fa50 con 0x7fb6040ae520 2023-12-19T09:19:33.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.653+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb604152d10 con 0x7fb6040ae520 2023-12-19T09:19:33.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.653+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60415a6d0 con 0x7fb6040ae520 2023-12-19T09:19:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.653+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb5ec01a400 con 0x7fb6040ae520 2023-12-19T09:19:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.657+0000 7fb600ff9700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 0x7fb5e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.657+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fb5ec07c720 con 0x7fb6040ae520 2023-12-19T09:19:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.657+0000 7fb603fff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 0x7fb5e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.657+0000 7fb603fff700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 0x7fb5e4062a00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4006fd0 tx=0x7fb5f4008040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.661+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb5ec04c960 con 0x7fb6040ae520 2023-12-19T09:19:33.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.873+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb604071120 con 0x7fb6040ae520 2023-12-19T09:19:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb600ff9700 1 -- 172.21.15.111:0/4044425679 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fb5ec04ff80 con 0x7fb6040ae520 2023-12-19T09:19:33.880 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:33.880 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":82,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3461937706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3461937706},{"type":"v1","addr":"172.21.15.143:6837","nonce":3461937706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":8378,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2960161706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2960161706},{"type":"v1","addr":"172.21.15.143:6835","nonce":2960161706}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":82},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/2652615955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":2652615955},{"type":"v1","addr":"172.21.15.111:6833","nonce":2652615955}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75},{"gid":9384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3130398266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3130398266},{"type":"v1","addr":"172.21.15.111:6835","nonce":3130398266}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[]} 2023-12-19T09:19:33.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 msgr2=0x7fb5e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 0x7fb5e4062a00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4006fd0 tx=0x7fb5f4008040 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 msgr2=0x7fb60415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00b810 tx=0x7fb5ec009250 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 shutdown_connections 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb5e4060550 0x7fb5e4062a00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6041552f0 0x7fb60415e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6040ae520 0x7fb60415a9d0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 --2- 172.21.15.111:0/4044425679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60406c7c0 0x7fb6040b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 >> 172.21.15.111:0/4044425679 conn(0x7fb6040c8850 msgr2=0x7fb604154940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 shutdown_connections 2023-12-19T09:19:33.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:33.877+0000 7fb60a8a5700 1 -- 172.21.15.111:0/4044425679 wait complete. 2023-12-19T09:19:33.886 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 82 2023-12-19T09:19:33.905 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T09:19:33.905 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T09:19:33.905 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T09:19:33.905 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:19:33.906 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T09:19:33.906 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T09:19:33.906 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T09:19:33.906 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:19:33.906 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T09:19:33.906 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T09:19:33.906 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T09:19:33.907 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:19:33.907 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T09:19:33.907 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T09:19:33.907 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T09:19:33.907 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:19:33.925 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T09:19:33.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T09:19:33.927 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T09:19:33.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T09:19:33.928 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T09:19:33.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T09:19:33.929 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T09:19:33.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T09:19:33.979 INFO:tasks.ceph.mds.b:Started 2023-12-19T09:19:33.980 DEBUG:teuthology.parallel:result is None 2023-12-19T09:19:33.981 INFO:tasks.ceph.mds.c:Started 2023-12-19T09:19:33.982 DEBUG:teuthology.parallel:result is None 2023-12-19T09:19:33.985 INFO:tasks.ceph.mds.a:Started 2023-12-19T09:19:33.985 DEBUG:teuthology.parallel:result is None 2023-12-19T09:19:33.986 INFO:tasks.ceph.mds.d:Started 2023-12-19T09:19:33.986 DEBUG:teuthology.parallel:result is None 2023-12-19T09:19:33.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T09:19:34.088 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:19:34.089 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T09:19:34.092 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T09:19:34.094 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:19:34.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.145+0000 7fc941945700 1 Processor -- start 2023-12-19T09:19:34.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.145+0000 7fc941945700 1 -- start start 2023-12-19T09:19:34.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc941945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc93c13f880 0x7fc93c1608f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:34.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc941945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c06dc00 con 0x7fc93c0c8160 2023-12-19T09:19:34.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc941945700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c06e090 con 0x7fc93c0c7d90 2023-12-19T09:19:34.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc941945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c06cff0 con 0x7fc93c13f880 2023-12-19T09:19:34.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93a7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc93c0c8160 0x7fc93c13f170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33650/0 (socket says 172.21.15.111:33650) 2023-12-19T09:19:34.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93a7fc700 1 -- 172.21.15.111:0/1093968499 learned_addr learned my addr 172.21.15.111:0/1093968499 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:34.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93b7fe700 1 --2- 172.21.15.111:0/1093968499 >> v2:172.21.15.143:3300/0 conn(0x7fc93c13f880 0x7fc93c1608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:34.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc939ffb700 1 -- 172.21.15.111:0/1093968499 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910959749 0 0) 0x7fc93c06dc00 con 0x7fc93c0c8160 2023-12-19T09:19:34.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc939ffb700 1 -- 172.21.15.111:0/1093968499 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc924003570 con 0x7fc93c0c8160 2023-12-19T09:19:34.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1093968499 >> v1:172.21.15.143:6789/0 conn(0x7fc93c0c7d90 legacy=0x7fc93c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:34.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1093968499 >> v1:172.21.15.111:6789/0 conn(0x7fc93c0c8160 legacy=0x7fc93c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:34.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.149+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1093968499 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c06d2d0 con 0x7fc93c13f880 2023-12-19T09:19:34.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc93b7fe700 1 --2- 172.21.15.111:0/1093968499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c13f880 0x7fc93c1608f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc934009e60 tx=0x7fc934009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30378593d9680d8a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:34.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc939ffb700 1 -- 172.21.15.111:0/1093968499 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400e070 con 0x7fc93c13f880 2023-12-19T09:19:34.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc939ffb700 1 -- 172.21.15.111:0/1093968499 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400bb60 con 0x7fc93c13f880 2023-12-19T09:19:34.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc939ffb700 1 -- 172.21.15.111:0/1093968499 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934012430 con 0x7fc93c13f880 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 -- 172.21.15.111:0/1093968499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c13f880 msgr2=0x7fc93c1608f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 --2- 172.21.15.111:0/1093968499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c13f880 0x7fc93c1608f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc934009e60 tx=0x7fc934009780 comp rx=0 tx=0).stop 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 -- 172.21.15.111:0/1093968499 shutdown_connections 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 --2- 172.21.15.111:0/1093968499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c13f880 0x7fc93c1608f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 -- 172.21.15.111:0/1093968499 >> 172.21.15.111:0/1093968499 conn(0x7fc93c067b60 msgr2=0x7fc93c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 -- 172.21.15.111:0/1093968499 shutdown_connections 2023-12-19T09:19:34.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.153+0000 7fc941945700 1 -- 172.21.15.111:0/1093968499 wait complete. 2023-12-19T09:19:34.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 Processor -- start 2023-12-19T09:19:34.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 -- start start 2023-12-19T09:19:34.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c0c7d90 0x7fc93c0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:34.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc93c0c8160 0x7fc93c13b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:34.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:34.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c6360 con 0x7fc93c13f880 2023-12-19T09:19:34.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c6640 con 0x7fc93c0c7d90 2023-12-19T09:19:34.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc941945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c6920 con 0x7fc93c0c8160 2023-12-19T09:19:34.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc93b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc93b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41284/0 (socket says 172.21.15.111:41284) 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1389209845 learned_addr learned my addr 172.21.15.111:0/1389209845 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc93affd700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c0c7d90 0x7fc93c0c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.157+0000 7fc93a7fc700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc93c0c8160 0x7fc93c13b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc93c0c8160 msgr2=0x7fc93c13b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc93c0c8160 0x7fc93c13b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c0c7d90 msgr2=0x7fc93c0c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c0c7d90 0x7fc93c0c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:34.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 -- 172.21.15.111:0/1389209845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc934009a20 con 0x7fc93c13f880 2023-12-19T09:19:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc93b7fe700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c8670 tx=0x7fc934019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934027470 con 0x7fc93c13f880 2023-12-19T09:19:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9340275f0 con 0x7fc93c13f880 2023-12-19T09:19:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c0c6c00 con 0x7fc93c13f880 2023-12-19T09:19:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9340279b0 con 0x7fc93c13f880 2023-12-19T09:19:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c13bb30 con 0x7fc93c13f880 2023-12-19T09:19:34.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.161+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c13cd30 con 0x7fc93c13f880 2023-12-19T09:19:34.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.165+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc934024070 con 0x7fc93c13f880 2023-12-19T09:19:34.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.165+0000 7fc922ffd700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 0x7fc928062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:34.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.169+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fc93407f410 con 0x7fc93c13f880 2023-12-19T09:19:34.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.169+0000 7fc93affd700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 0x7fc928062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:34.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.169+0000 7fc93affd700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 0x7fc928062a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc92c003a80 tx=0x7fc92c007ec0 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:34.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.173+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc93c13cd30 con 0x7fc93c13f880 2023-12-19T09:19:34.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:34.361+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc93c13bb30 con 0x7fc93c13f880 2023-12-19T09:19:35.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.389+0000 7fc922ffd700 1 -- 172.21.15.111:0/1389209845 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v189) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc93c13bb30 con 0x7fc93c13f880 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 msgr2=0x7fc928062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 0x7fc928062a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc92c003a80 tx=0x7fc92c007ec0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 msgr2=0x7fc93c0b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c8670 tx=0x7fc934019000 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 shutdown_connections 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc93c13f880 0x7fc93c0b04e0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc93c0c8160 0x7fc93c13b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc928060550 0x7fc928062a00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 --2- 172.21.15.111:0/1389209845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc93c0c7d90 0x7fc93c0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 >> 172.21.15.111:0/1389209845 conn(0x7fc93c067b60 msgr2=0x7fc93c15af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:35.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 shutdown_connections 2023-12-19T09:19:35.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.393+0000 7fc941945700 1 -- 172.21.15.111:0/1389209845 wait complete. 2023-12-19T09:19:35.402 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T09:19:35.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T09:19:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.585+0000 7f21b14cd700 1 Processor -- start 2023-12-19T09:19:35.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.585+0000 7f21b14cd700 1 -- start start 2023-12-19T09:19:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21b14cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac068060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21b14cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac0c6630 con 0x7f21ac067c90 2023-12-19T09:19:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21b14cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac0c6910 con 0x7f21ac0c5900 2023-12-19T09:19:35.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21b14cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0c6440 con 0x7f21ac1502b0 2023-12-19T09:19:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aaffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f21ac067c90 0x7f21ac14fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33664/0 (socket says 172.21.15.111:33664) 2023-12-19T09:19:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aaffd700 1 -- 172.21.15.111:0/287460957 learned_addr learned my addr 172.21.15.111:0/287460957 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:35.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aa7fc700 1 --2- 172.21.15.111:0/287460957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac068060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835931703 0 0) 0x7f21ac0c6630 con 0x7f21ac067c90 2023-12-19T09:19:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2194003570 con 0x7f21ac067c90 2023-12-19T09:19:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302001595 0 0) 0x7f21ac0c6910 con 0x7f21ac0c5900 2023-12-19T09:19:35.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ac0c6630 con 0x7f21ac0c5900 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aa7fc700 1 -- 172.21.15.111:0/287460957 >> v1:172.21.15.143:6789/0 conn(0x7f21ac0c5900 legacy=0x7f21ac0c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aa7fc700 1 -- 172.21.15.111:0/287460957 >> v1:172.21.15.111:6789/0 conn(0x7f21ac067c90 legacy=0x7f21ac14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.589+0000 7f21aa7fc700 1 -- 172.21.15.111:0/287460957 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac0c6bf0 con 0x7f21ac1502b0 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21aa7fc700 1 --2- 172.21.15.111:0/287460957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac068060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009910 tx=0x7f21a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e449d1e3c624aa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0011630 con 0x7f21ac1502b0 2023-12-19T09:19:35.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0011790 con 0x7f21ac1502b0 2023-12-19T09:19:35.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21a9ffb700 1 -- 172.21.15.111:0/287460957 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0011b30 con 0x7f21ac1502b0 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 -- 172.21.15.111:0/287460957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 msgr2=0x7f21ac068060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 --2- 172.21.15.111:0/287460957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac068060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009910 tx=0x7f21a00091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 -- 172.21.15.111:0/287460957 shutdown_connections 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 --2- 172.21.15.111:0/287460957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac068060 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 -- 172.21.15.111:0/287460957 >> 172.21.15.111:0/287460957 conn(0x7f21ac0cd6d0 msgr2=0x7f21ac06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 -- 172.21.15.111:0/287460957 shutdown_connections 2023-12-19T09:19:35.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.593+0000 7f21b14cd700 1 -- 172.21.15.111:0/287460957 wait complete. 2023-12-19T09:19:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 Processor -- start 2023-12-19T09:19:35.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 -- start start 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21ac0c5900 0x7f21ac14f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0c8680 con 0x7f21ac067c90 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0c83a0 con 0x7f21ac0c5900 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21b14cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0b40f0 con 0x7f21ac1502b0 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41294/0 (socket says 172.21.15.111:41294) 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 -- 172.21.15.111:0/1119412253 learned_addr learned my addr 172.21.15.111:0/1119412253 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aa7fc700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21ac0c5900 0x7f21ac14f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21ab7fe700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 -- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 msgr2=0x7f21ac0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 -- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21ac0c5900 msgr2=0x7f21ac14f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21ac0c5900 0x7f21ac14f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 -- 172.21.15.111:0/1119412253 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009420 con 0x7f21ac067c90 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.597+0000 7f21aaffd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c790 tx=0x7f219c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c005b30 con 0x7f21ac067c90 2023-12-19T09:19:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c005c90 con 0x7f21ac067c90 2023-12-19T09:19:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac0b43d0 con 0x7f21ac067c90 2023-12-19T09:19:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c007300 con 0x7f21ac067c90 2023-12-19T09:19:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac0c8090 con 0x7f21ac067c90 2023-12-19T09:19:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.601+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac071120 con 0x7f21ac067c90 2023-12-19T09:19:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.605+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f219c01b3b0 con 0x7f21ac067c90 2023-12-19T09:19:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.605+0000 7f2192ffd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 0x7f2198062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.605+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f219c07bc80 con 0x7f21ac067c90 2023-12-19T09:19:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.605+0000 7f21aa7fc700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 0x7f2198062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.609+0000 7f21aa7fc700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 0x7f2198062a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f21a00091e0 tx=0x7f21a0017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:35.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.613+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f219c04fcd0 con 0x7f21ac067c90 2023-12-19T09:19:35.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.837+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f21ac0642a0 con 0x7f21ac067c90 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.837+0000 7f2192ffd700 1 -- 172.21.15.111:0/1119412253 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f219c017090 con 0x7f21ac067c90 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T09:19:35.843 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:19:35.844 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T09:19:35.846 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.848 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.850 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.852 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T09:19:35.853 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T09:19:35.854 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.855 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:19:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:19:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:19:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:19:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T09:19:35.858 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 msgr2=0x7f2198062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 0x7f2198062a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f21a00091e0 tx=0x7f21a0017000 comp rx=0 tx=0).stop 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 msgr2=0x7f21ac0c7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c790 tx=0x7f219c007ea0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 shutdown_connections 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2198060550 0x7f2198062a00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21ac0c5900 0x7f21ac14f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21ac1502b0 0x7f21ac0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 --2- 172.21.15.111:0/1119412253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21ac067c90 0x7f21ac0c7b50 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:35.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 >> 172.21.15.111:0/1119412253 conn(0x7f21ac0cd6d0 msgr2=0x7f21ac06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:35.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 shutdown_connections 2023-12-19T09:19:35.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:35.841+0000 7f21b14cd700 1 -- 172.21.15.111:0/1119412253 wait complete. 2023-12-19T09:19:35.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:36.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.029+0000 7f737f345700 1 Processor -- start 2023-12-19T09:19:36.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737f345700 1 -- start start 2023-12-19T09:19:36.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737f345700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73780b0110 0x7f73780b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737f345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73780ccb40 con 0x7f7378153290 2023-12-19T09:19:36.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737f345700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73780cd470 con 0x7f73780ccfd0 2023-12-19T09:19:36.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737f345700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73780cbf30 con 0x7f73780b0110 2023-12-19T09:19:36.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737d8e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7378153290 0x7f7378153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33668/0 (socket says 172.21.15.111:33668) 2023-12-19T09:19:36.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737d8e2700 1 -- 172.21.15.111:0/137129242 learned_addr learned my addr 172.21.15.111:0/137129242 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:36.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.033+0000 7f737d0e1700 1 --2- 172.21.15.111:0/137129242 >> v2:172.21.15.143:3300/0 conn(0x7f73780b0110 0x7f73780b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f736ffff700 1 -- 172.21.15.111:0/137129242 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157921751 0 0) 0x7f73780ccb40 con 0x7f7378153290 2023-12-19T09:19:36.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f736ffff700 1 -- 172.21.15.111:0/137129242 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003570 con 0x7f7378153290 2023-12-19T09:19:36.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737d0e1700 1 -- 172.21.15.111:0/137129242 >> v1:172.21.15.143:6789/0 conn(0x7f73780ccfd0 legacy=0x7f737815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737d0e1700 1 -- 172.21.15.111:0/137129242 >> v1:172.21.15.111:6789/0 conn(0x7f7378153290 legacy=0x7f7378153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737d0e1700 1 -- 172.21.15.111:0/137129242 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73780cc210 con 0x7f73780b0110 2023-12-19T09:19:36.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737d0e1700 1 --2- 172.21.15.111:0/137129242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780b0110 0x7f73780b04e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f73740098e0 tx=0x7f73740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6906bdd34f178bf7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f736ffff700 1 -- 172.21.15.111:0/137129242 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400b640 con 0x7f73780b0110 2023-12-19T09:19:36.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f736ffff700 1 -- 172.21.15.111:0/137129242 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737400b7a0 con 0x7f73780b0110 2023-12-19T09:19:36.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f736ffff700 1 -- 172.21.15.111:0/137129242 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400bb40 con 0x7f73780b0110 2023-12-19T09:19:36.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 -- 172.21.15.111:0/137129242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780b0110 msgr2=0x7f73780b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 --2- 172.21.15.111:0/137129242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780b0110 0x7f73780b04e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f73740098e0 tx=0x7f73740091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 -- 172.21.15.111:0/137129242 shutdown_connections 2023-12-19T09:19:36.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 --2- 172.21.15.111:0/137129242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780b0110 0x7f73780b04e0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 -- 172.21.15.111:0/137129242 >> 172.21.15.111:0/137129242 conn(0x7f737806c7c0 msgr2=0x7f737806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.037+0000 7f737f345700 1 -- 172.21.15.111:0/137129242 shutdown_connections 2023-12-19T09:19:36.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 -- 172.21.15.111:0/137129242 wait complete. 2023-12-19T09:19:36.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 Processor -- start 2023-12-19T09:19:36.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 -- start start 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 0x7f73780c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 0x7f73780c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378150f30 con 0x7f73780b0110 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378067a70 con 0x7f73780ccfd0 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737f345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73781509f0 con 0x7f7378153290 2023-12-19T09:19:36.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.041+0000 7f737d0e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41302/0 (socket says 172.21.15.111:41302) 2023-12-19T09:19:36.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 -- 172.21.15.111:0/1229563333 learned_addr learned my addr 172.21.15.111:0/1229563333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d8e2700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 0x7f73780c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737c8e0700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 0x7f73780c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 -- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 msgr2=0x7f73780c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 0x7f73780c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 -- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 msgr2=0x7f73780c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 0x7f73780c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 -- 172.21.15.111:0/1229563333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374009420 con 0x7f73780b0110 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d8e2700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 0x7f73780c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737c8e0700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 0x7f73780c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:36.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737d0e1700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f73780649b0 tx=0x7f7374017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400bb20 con 0x7f73780b0110 2023-12-19T09:19:36.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374011420 con 0x7f73780b0110 2023-12-19T09:19:36.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73780a9d40 con 0x7f73780b0110 2023-12-19T09:19:36.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73740117c0 con 0x7f73780b0110 2023-12-19T09:19:36.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73780aa300 con 0x7f73780b0110 2023-12-19T09:19:36.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.045+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7378071120 con 0x7f73780b0110 2023-12-19T09:19:36.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.049+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7374011b70 con 0x7f73780b0110 2023-12-19T09:19:36.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.049+0000 7f736d7fa700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 0x7f735c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.049+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f737407df10 con 0x7f73780b0110 2023-12-19T09:19:36.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.049+0000 7f737c8e0700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 0x7f735c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.053+0000 7f737c8e0700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 0x7f735c062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7378152820 tx=0x7f7368009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.057+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f737404e430 con 0x7f73780b0110 2023-12-19T09:19:36.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.225+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73780ada80 con 0x7f73780b0110 2023-12-19T09:19:36.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.225+0000 7f736d7fa700 1 -- 172.21.15.111:0/1229563333 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+13277 (secure 0 0 0) 0x7f7374051a50 con 0x7f73780b0110 2023-12-19T09:19:36.231 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:36.231 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":189,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:35.382626+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 msgr2=0x7f735c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 0x7f735c062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7378152820 tx=0x7f7368009040 comp rx=0 tx=0).stop 2023-12-19T09:19:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 msgr2=0x7f7378067530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f73780649b0 tx=0x7f7374017000 comp rx=0 tx=0).stop 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 shutdown_connections 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f735c060550 0x7f735c062a00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73780ccfd0 0x7f73780c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7378153290 0x7f73780c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 --2- 172.21.15.111:0/1229563333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73780b0110 0x7f7378067530 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 >> 172.21.15.111:0/1229563333 conn(0x7f737806c7c0 msgr2=0x7f7378152980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 shutdown_connections 2023-12-19T09:19:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.229+0000 7f737f345700 1 -- 172.21.15.111:0/1229563333 wait complete. 2023-12-19T09:19:36.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:19:36.254 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:19:36.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 Processor -- start 2023-12-19T09:19:36.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 -- start start 2023-12-19T09:19:36.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cb970 0x7f763406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f76340cecd0 0x7f763406b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76340cbdf0 con 0x7f7634158e20 2023-12-19T09:19:36.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76340cc0d0 con 0x7f76340cecd0 2023-12-19T09:19:36.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.417+0000 7f7639945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76340cc3b0 con 0x7f76340cb970 2023-12-19T09:19:36.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7632ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7634158e20 0x7f76340ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33684/0 (socket says 172.21.15.111:33684) 2023-12-19T09:19:36.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7632ffd700 1 -- 172.21.15.111:0/3438347922 learned_addr learned my addr 172.21.15.111:0/3438347922 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:36.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 --2- 172.21.15.111:0/3438347922 >> v2:172.21.15.143:3300/0 conn(0x7f76340cecd0 0x7f763406b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76327fc700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cb970 0x7f763406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7631ffb700 1 -- 172.21.15.111:0/3438347922 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360990459 0 0) 0x7f76340cbdf0 con 0x7f7634158e20 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7631ffb700 1 -- 172.21.15.111:0/3438347922 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c003570 con 0x7f7634158e20 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 -- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cb970 msgr2=0x7f763406acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cb970 0x7f763406acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 -- 172.21.15.111:0/3438347922 >> v1:172.21.15.111:6789/0 conn(0x7f7634158e20 legacy=0x7f76340ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 -- 172.21.15.111:0/3438347922 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76340cc690 con 0x7f76340cecd0 2023-12-19T09:19:36.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f76337fe700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76340cecd0 0x7f763406b200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f762c009e60 tx=0x7f762c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b927cdc363cf5020 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7631ffb700 1 -- 172.21.15.111:0/3438347922 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c00e070 con 0x7f76340cecd0 2023-12-19T09:19:36.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7631ffb700 1 -- 172.21.15.111:0/3438347922 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c00bab0 con 0x7f76340cecd0 2023-12-19T09:19:36.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7631ffb700 1 -- 172.21.15.111:0/3438347922 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c00be70 con 0x7f76340cecd0 2023-12-19T09:19:36.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 -- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76340cecd0 msgr2=0x7f763406b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76340cecd0 0x7f763406b200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f762c009e60 tx=0x7f762c009780 comp rx=0 tx=0).stop 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 -- 172.21.15.111:0/3438347922 shutdown_connections 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76340cecd0 0x7f763406b200 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 --2- 172.21.15.111:0/3438347922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cb970 0x7f763406acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 -- 172.21.15.111:0/3438347922 >> 172.21.15.111:0/3438347922 conn(0x7f7634067bb0 msgr2=0x7f76340d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.421+0000 7f7639945700 1 -- 172.21.15.111:0/3438347922 shutdown_connections 2023-12-19T09:19:36.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.425+0000 7f7639945700 1 -- 172.21.15.111:0/3438347922 wait complete. 2023-12-19T09:19:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.425+0000 7f7639945700 1 Processor -- start 2023-12-19T09:19:36.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.425+0000 7f7639945700 1 -- start start 2023-12-19T09:19:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 0x7f76340b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 0x7f763414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76340b4e20 con 0x7f76340c9d50 2023-12-19T09:19:36.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76340b4210 con 0x7f7634158e20 2023-12-19T09:19:36.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76340b44f0 con 0x7f76340cecd0 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41308/0 (socket says 172.21.15.111:41308) 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 -- 172.21.15.111:0/3595540762 learned_addr learned my addr 172.21.15.111:0/3595540762 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7632ffd700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 0x7f76340b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76327fc700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 0x7f763414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 -- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 msgr2=0x7f76340b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 0x7f76340b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 -- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 msgr2=0x7f763414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 0x7f763414f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 -- 172.21.15.111:0/3595540762 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c009a20 con 0x7f76340c9d50 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7632ffd700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 0x7f76340b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76327fc700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 0x7f763414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:36.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f76337fe700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f762c013000 tx=0x7f762c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c027470 con 0x7f76340c9d50 2023-12-19T09:19:36.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c0275f0 con 0x7f76340c9d50 2023-12-19T09:19:36.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c0279b0 con 0x7f76340c9d50 2023-12-19T09:19:36.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76340b47d0 con 0x7f76340c9d50 2023-12-19T09:19:36.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.429+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76340b3eb0 con 0x7f76340c9d50 2023-12-19T09:19:36.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.433+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76341594b0 con 0x7f76340c9d50 2023-12-19T09:19:36.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.433+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f762c024070 con 0x7f76340c9d50 2023-12-19T09:19:36.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.433+0000 7f761affd700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 0x7f7620062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.437+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f762c07f160 con 0x7f76340c9d50 2023-12-19T09:19:36.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.437+0000 7f7632ffd700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 0x7f7620062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.437+0000 7f7632ffd700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 0x7f7620062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76240058d0 tx=0x7f7624009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.441+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f762c04f680 con 0x7f76340c9d50 2023-12-19T09:19:36.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.613+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7634071120 con 0x7f76340c9d50 2023-12-19T09:19:36.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f761affd700 1 -- 172.21.15.111:0/3595540762 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v189) v1 ==== 93+0+10 (secure 0 0 0) 0x7f762c052ca0 con 0x7f76340c9d50 2023-12-19T09:19:36.620 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 msgr2=0x7f7620062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 0x7f7620062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76240058d0 tx=0x7f7624009040 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 msgr2=0x7f763415ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f762c013000 tx=0x7f762c019000 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 shutdown_connections 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7620060550 0x7f7620062a00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7634158e20 0x7f763414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76340c9d50 0x7f763415ecb0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 --2- 172.21.15.111:0/3595540762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76340cecd0 0x7f76340b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.617+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 >> 172.21.15.111:0/3595540762 conn(0x7f7634067bb0 msgr2=0x7f76340cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.621+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 shutdown_connections 2023-12-19T09:19:36.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.621+0000 7f7639945700 1 -- 172.21.15.111:0/3595540762 wait complete. 2023-12-19T09:19:36.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T09:19:36.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.793+0000 7fc64f2ae700 1 Processor -- start 2023-12-19T09:19:36.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.793+0000 7fc64f2ae700 1 -- start start 2023-12-19T09:19:36.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64f2ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6481558b0 0x7fc648155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64f2ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6480cb8a0 con 0x7fc648159ba0 2023-12-19T09:19:36.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64f2ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc648156250 con 0x7fc6480ca420 2023-12-19T09:19:36.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64f2ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6480cac90 con 0x7fc6481558b0 2023-12-19T09:19:36.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64d04a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc648159ba0 0x7fc6480670e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33698/0 (socket says 172.21.15.111:33698) 2023-12-19T09:19:36.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64d04a700 1 -- 172.21.15.111:0/78875960 learned_addr learned my addr 172.21.15.111:0/78875960 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:36.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc64d84b700 1 --2- 172.21.15.111:0/78875960 >> v2:172.21.15.111:3300/0 conn(0x7fc6481558b0 0x7fc648155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.797+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419720368 0 0) 0x7fc6480cb8a0 con 0x7fc648159ba0 2023-12-19T09:19:36.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc634003570 con 0x7fc648159ba0 2023-12-19T09:19:36.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038459476 0 0) 0x7fc648156250 con 0x7fc6480ca420 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6480cb8a0 con 0x7fc6480ca420 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64d84b700 1 -- 172.21.15.111:0/78875960 >> v1:172.21.15.143:6789/0 conn(0x7fc6480ca420 legacy=0x7fc648159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64d84b700 1 -- 172.21.15.111:0/78875960 >> v1:172.21.15.111:6789/0 conn(0x7fc648159ba0 legacy=0x7fc6480670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2153244777 0 0) 0x7fc634003570 con 0x7fc648159ba0 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64d84b700 1 -- 172.21.15.111:0/78875960 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6480caf70 con 0x7fc6481558b0 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64d84b700 1 --2- 172.21.15.111:0/78875960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648155d10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc640009e90 tx=0x7fc640009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5022f99710015bb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64000e070 con 0x7fc6481558b0 2023-12-19T09:19:36.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64000b870 con 0x7fc6481558b0 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc63ffff700 1 -- 172.21.15.111:0/78875960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644001470 con 0x7fc6481558b0 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 -- 172.21.15.111:0/78875960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 msgr2=0x7fc648155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/78875960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648155d10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fc640009e90 tx=0x7fc640009780 comp rx=0 tx=0).stop 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 -- 172.21.15.111:0/78875960 shutdown_connections 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/78875960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648155d10 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 -- 172.21.15.111:0/78875960 >> 172.21.15.111:0/78875960 conn(0x7fc6480cd990 msgr2=0x7fc6480ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 -- 172.21.15.111:0/78875960 shutdown_connections 2023-12-19T09:19:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.801+0000 7fc64f2ae700 1 -- 172.21.15.111:0/78875960 wait complete. 2023-12-19T09:19:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.805+0000 7fc64f2ae700 1 Processor -- start 2023-12-19T09:19:36.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.805+0000 7fc64f2ae700 1 -- start start 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6480ca420 0x7fc648159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc648159ba0 0x7fc6480c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6481597d0 con 0x7fc6481558b0 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc648154570 con 0x7fc6480ca420 2023-12-19T09:19:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc648158270 con 0x7fc648159ba0 2023-12-19T09:19:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64d04a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6480ca420 0x7fc648159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41334/0 (socket says 172.21.15.111:41334) 2023-12-19T09:19:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 -- 172.21.15.111:0/3811065202 learned_addr learned my addr 172.21.15.111:0/3811065202 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64d84b700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc648159ba0 0x7fc6480c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 -- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc648159ba0 msgr2=0x7fc6480c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc648159ba0 0x7fc6480c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 -- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6480ca420 msgr2=0x7fc648159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6480ca420 0x7fc648159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 -- 172.21.15.111:0/3811065202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc640009a20 con 0x7fc6481558b0 2023-12-19T09:19:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64c849700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fc63800c790 tx=0x7fc638007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638010510 con 0x7fc6481558b0 2023-12-19T09:19:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc638010670 con 0x7fc6481558b0 2023-12-19T09:19:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6481587b0 con 0x7fc6481558b0 2023-12-19T09:19:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638010a30 con 0x7fc6481558b0 2023-12-19T09:19:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.809+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc648158d70 con 0x7fc6481558b0 2023-12-19T09:19:36.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.813+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc648158f90 con 0x7fc6481558b0 2023-12-19T09:19:36.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.813+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc63801b400 con 0x7fc6481558b0 2023-12-19T09:19:36.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.817+0000 7fc63d7fa700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 0x7fc62c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:36.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.817+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fc63807c660 con 0x7fc6481558b0 2023-12-19T09:19:36.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.817+0000 7fc64d04a700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 0x7fc62c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:36.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.817+0000 7fc64d04a700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 0x7fc62c062a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc6440015a0 tx=0x7fc644000bc0 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:36.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.821+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc63804cb80 con 0x7fc6481558b0 2023-12-19T09:19:36.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.989+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc648071120 con 0x7fc6481558b0 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.989+0000 7fc63d7fa700 1 -- 172.21.15.111:0/3811065202 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v189) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc6380501a0 con 0x7fc6481558b0 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:36.994 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:19:36.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 msgr2=0x7fc62c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 0x7fc62c062a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc6440015a0 tx=0x7fc644000bc0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 msgr2=0x7fc648154000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:36.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fc63800c790 tx=0x7fc638007e10 comp rx=0 tx=0).stop 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 shutdown_connections 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc648159ba0 0x7fc6480c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6481558b0 0x7fc648154000 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc62c060550 0x7fc62c062a00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 --2- 172.21.15.111:0/3811065202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6480ca420 0x7fc648159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:36.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 >> 172.21.15.111:0/3811065202 conn(0x7fc6480cd990 msgr2=0x7fc648068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:36.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 shutdown_connections 2023-12-19T09:19:36.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:36.993+0000 7fc64f2ae700 1 -- 172.21.15.111:0/3811065202 wait complete. 2023-12-19T09:19:37.017 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T09:19:37.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T09:19:37.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.185+0000 7f030f59e700 1 Processor -- start 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.189+0000 7f030f59e700 1 -- start start 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03100cc430 0x7f0310161820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100cd5f0 con 0x7f031006c7c0 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100cc9e0 con 0x7f03100ae7e0 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03100cccc0 con 0x7f03100cc430 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f031006c7c0 0x7f03100cbd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33714/0 (socket says 172.21.15.111:33714) 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030e59c700 1 -- 172.21.15.111:0/1836319193 learned_addr learned my addr 172.21.15.111:0/1836319193 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030ed9d700 1 --2- 172.21.15.111:0/1836319193 >> v2:172.21.15.111:3300/0 conn(0x7f03100cc430 0x7f0310161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030d59a700 1 -- 172.21.15.111:0/1836319193 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747421883 0 0) 0x7f03100cc9e0 con 0x7f03100ae7e0 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030d59a700 1 -- 172.21.15.111:0/1836319193 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f8003570 con 0x7f03100ae7e0 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030ed9d700 1 -- 172.21.15.111:0/1836319193 >> v1:172.21.15.143:6789/0 conn(0x7f03100ae7e0 legacy=0x7f031006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030ed9d700 1 -- 172.21.15.111:0/1836319193 >> v1:172.21.15.111:6789/0 conn(0x7f031006c7c0 legacy=0x7f03100cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030ed9d700 1 -- 172.21.15.111:0/1836319193 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03100ccfa0 con 0x7f03100cc430 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030ed9d700 1 --2- 172.21.15.111:0/1836319193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03100cc430 0x7f0310161820 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f0308009e60 tx=0x7f0308009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17de1d788f498877 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030d59a700 1 -- 172.21.15.111:0/1836319193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800e070 con 0x7f03100cc430 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030d59a700 1 -- 172.21.15.111:0/1836319193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030800bb60 con 0x7f03100cc430 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030d59a700 1 -- 172.21.15.111:0/1836319193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308012430 con 0x7f03100cc430 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- 172.21.15.111:0/1836319193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03100cc430 msgr2=0x7f0310161820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 --2- 172.21.15.111:0/1836319193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03100cc430 0x7f0310161820 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f0308009e60 tx=0x7f0308009780 comp rx=0 tx=0).stop 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- 172.21.15.111:0/1836319193 shutdown_connections 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 --2- 172.21.15.111:0/1836319193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03100cc430 0x7f0310161820 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- 172.21.15.111:0/1836319193 >> 172.21.15.111:0/1836319193 conn(0x7f03100cdb50 msgr2=0x7f03100cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- 172.21.15.111:0/1836319193 shutdown_connections 2023-12-19T09:19:37.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.193+0000 7f030f59e700 1 -- 172.21.15.111:0/1836319193 wait complete. 2023-12-19T09:19:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.241+0000 7f030f59e700 1 Processor -- start 2023-12-19T09:19:37.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.241+0000 7f030f59e700 1 -- start start 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03100ae7e0 0x7f03100c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03100cc430 0x7f031013a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310139ca0 con 0x7f031006c7c0 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310139370 con 0x7f03100cc430 2023-12-19T09:19:37.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030f59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0310139650 con 0x7f03100ae7e0 2023-12-19T09:19:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41360/0 (socket says 172.21.15.111:41360) 2023-12-19T09:19:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03100ae7e0 0x7f03100c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 -- 172.21.15.111:0/2079790158 learned_addr learned my addr 172.21.15.111:0/2079790158 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:37.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030ed9d700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03100cc430 0x7f031013a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 -- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03100ae7e0 msgr2=0x7f03100c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03100ae7e0 0x7f03100c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 -- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03100cc430 msgr2=0x7f031013a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03100cc430 0x7f031013a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.245+0000 7f030e59c700 1 -- 172.21.15.111:0/2079790158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0308009a20 con 0x7f031006c7c0 2023-12-19T09:19:37.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f030e59c700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f030000c590 tx=0x7f0300007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300005b30 con 0x7f031006c7c0 2023-12-19T09:19:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300005cb0 con 0x7f031006c7c0 2023-12-19T09:19:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310139930 con 0x7f031006c7c0 2023-12-19T09:19:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300007320 con 0x7f031006c7c0 2023-12-19T09:19:37.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031013a130 con 0x7f031006c7c0 2023-12-19T09:19:37.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.249+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031004be40 con 0x7f031006c7c0 2023-12-19T09:19:37.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.253+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f030001b3b0 con 0x7f031006c7c0 2023-12-19T09:19:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.253+0000 7f02fe7fc700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 0x7f02f0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:37.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.253+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f030007ba60 con 0x7f031006c7c0 2023-12-19T09:19:37.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.253+0000 7f030dd9b700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 0x7f02f0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:37.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.253+0000 7f030dd9b700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 0x7f02f0062a00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03100b3990 tx=0x7f0304007d50 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:37.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.257+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f030004c020 con 0x7f031006c7c0 2023-12-19T09:19:37.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.433+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f03100c7cb0 con 0x7f031006c7c0 2023-12-19T09:19:37.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.973+0000 7f02fe7fc700 1 -- 172.21.15.111:0/2079790158 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v190) v1 ==== 138+0+0 (secure 0 0 0) 0x7f030004f640 con 0x7f031006c7c0 2023-12-19T09:19:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 msgr2=0x7f02f0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 0x7f02f0062a00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03100b3990 tx=0x7f0304007d50 comp rx=0 tx=0).stop 2023-12-19T09:19:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 msgr2=0x7f03100c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:37.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f030000c590 tx=0x7f0300007f90 comp rx=0 tx=0).stop 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 shutdown_connections 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f02f0060550 0x7f02f0062a00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03100cc430 0x7f031013a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03100ae7e0 0x7f03100c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 --2- 172.21.15.111:0/2079790158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031006c7c0 0x7f03100c30f0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 >> 172.21.15.111:0/2079790158 conn(0x7f03100cdb50 msgr2=0x7f03100afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 shutdown_connections 2023-12-19T09:19:37.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:37.981+0000 7f030f59e700 1 -- 172.21.15.111:0/2079790158 wait complete. 2023-12-19T09:19:37.992 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T09:19:38.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T09:19:38.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.185+0000 7fdd208ae700 1 Processor -- start 2023-12-19T09:19:38.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.189+0000 7fdd208ae700 1 -- start start 2023-12-19T09:19:38.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.189+0000 7fdd208ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 0x7fdd1c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.189+0000 7fdd208ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd1c0ca2c0 0x7fdd1c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.189+0000 7fdd208ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0cbdf0 con 0x7fdd1c0cb970 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd208ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1c0cc0d0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd208ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0cc3b0 con 0x7fdd1c158e20 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd19d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdd1c0cb970 0x7fdd1c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33728/0 (socket says 172.21.15.111:33728) 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd19d9b700 1 -- 172.21.15.111:0/905339504 learned_addr learned my addr 172.21.15.111:0/905339504 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 --2- 172.21.15.111:0/905339504 >> v2:172.21.15.143:3300/0 conn(0x7fdd1c0ca2c0 0x7fdd1c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 0x7fdd1c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1959a700 1 -- 172.21.15.111:0/905339504 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529698936 0 0) 0x7fdd1c0cbdf0 con 0x7fdd1c0cb970 2023-12-19T09:19:38.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1959a700 1 -- 172.21.15.111:0/905339504 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04003570 con 0x7fdd1c0cb970 2023-12-19T09:19:38.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 -- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 msgr2=0x7fdd1c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:38.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 0x7fdd1c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:38.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 -- 172.21.15.111:0/905339504 >> v1:172.21.15.111:6789/0 conn(0x7fdd1c0cb970 legacy=0x7fdd1c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:38.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 -- 172.21.15.111:0/905339504 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c0cc690 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 0x7fdd1c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:38.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1ad9d700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c0ca2c0 0x7fdd1c0ceb00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009e60 tx=0x7fdd14009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96c346138187832 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:38.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1959a700 1 -- 172.21.15.111:0/905339504 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400e070 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1959a700 1 -- 172.21.15.111:0/905339504 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1400bab0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd1959a700 1 -- 172.21.15.111:0/905339504 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400be70 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd208ae700 1 -- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c0ca2c0 msgr2=0x7fdd1c0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:38.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.193+0000 7fdd208ae700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c0ca2c0 0x7fdd1c0ceb00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009e60 tx=0x7fdd14009780 comp rx=0 tx=0).stop 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- 172.21.15.111:0/905339504 shutdown_connections 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c0ca2c0 0x7fdd1c0ceb00 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 --2- 172.21.15.111:0/905339504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c158e20 0x7fdd1c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- 172.21.15.111:0/905339504 >> 172.21.15.111:0/905339504 conn(0x7fdd1c067bb0 msgr2=0x7fdd1c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- 172.21.15.111:0/905339504 shutdown_connections 2023-12-19T09:19:38.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- 172.21.15.111:0/905339504 wait complete. 2023-12-19T09:19:38.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 Processor -- start 2023-12-19T09:19:38.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- start start 2023-12-19T09:19:38.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 0x7fdd1c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c158e20 0x7fdd1c1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c58f0 con 0x7fdd1c16c890 2023-12-19T09:19:38.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c16d240 con 0x7fdd1c158e20 2023-12-19T09:19:38.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd208ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c5ae0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd1ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd1ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41374/0 (socket says 172.21.15.111:41374) 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd1ad9d700 1 -- 172.21.15.111:0/1160090938 learned_addr learned my addr 172.21.15.111:0/1160090938 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 0x7fdd1c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.197+0000 7fdd19d9b700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c158e20 0x7fdd1c1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 -- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c158e20 msgr2=0x7fdd1c1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c158e20 0x7fdd1c1522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 -- 172.21.15.111:0/1160090938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 msgr2=0x7fdd1c16cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 -- 172.21.15.111:0/1160090938 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c009420 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1ad9d700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:38.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd1a59c700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 0x7fdd1c1559b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00ea00 tx=0x7fdd0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:38.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00c6a0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00c800 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00cba0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd14009a20 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c0c60a0 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.201+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c071120 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.205+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdd0c013470 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.205+0000 7fdd0a7fc700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 0x7fdcfc062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:38.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.205+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6119+0+0 (secure 0 0 0) 0x7fdd0c07cb10 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.205+0000 7fdd19d9b700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 0x7fdcfc062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:38.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.205+0000 7fdd19d9b700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 0x7fdcfc062ad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0c6b60 tx=0x7fdd10009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:38.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.213+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd0c04cf40 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.385+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdd1c153b70 con 0x7fdd1c0ca2c0 2023-12-19T09:19:38.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:38.989+0000 7fdd0a7fc700 1 -- 172.21.15.111:0/1160090938 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v191) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdd0c050560 con 0x7fdd1c0ca2c0 2023-12-19T09:19:39.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.001+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 msgr2=0x7fdcfc062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.001+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 0x7fdcfc062ad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0c6b60 tx=0x7fdd10009040 comp rx=0 tx=0).stop 2023-12-19T09:19:39.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.001+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 msgr2=0x7fdd1c1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.001+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 0x7fdd1c1559b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00ea00 tx=0x7fdd0c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 shutdown_connections 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdcfc060620 0x7fdcfc062ad0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd1c158e20 0x7fdd1c1522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd1c16c890 0x7fdd1c16cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 --2- 172.21.15.111:0/1160090938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd1c0ca2c0 0x7fdd1c1559b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.005+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 >> 172.21.15.111:0/1160090938 conn(0x7fdd1c067bb0 msgr2=0x7fdd1c14c980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:39.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.009+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 shutdown_connections 2023-12-19T09:19:39.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.009+0000 7fdd208ae700 1 -- 172.21.15.111:0/1160090938 wait complete. 2023-12-19T09:19:39.022 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T09:19:39.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T09:19:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.197+0000 7f7cf204c700 1 Processor -- start 2023-12-19T09:19:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.197+0000 7f7cf204c700 1 -- start start 2023-12-19T09:19:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf204c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 0x7f7cec159060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf204c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf204c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cec06c850 con 0x7f7cec158c90 2023-12-19T09:19:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf204c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cec0cb650 con 0x7f7cec06aeb0 2023-12-19T09:19:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf204c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec06b470 con 0x7f7cec1595a0 2023-12-19T09:19:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf0849700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 0x7f7cec159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf0849700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 0x7f7cec159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41382/0 (socket says 172.21.15.111:41382) 2023-12-19T09:19:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf0849700 1 -- 172.21.15.111:0/665228379 learned_addr learned my addr 172.21.15.111:0/665228379 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf104a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7cec158c90 0x7f7cec0caf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:42434/0 (socket says 172.21.15.111:42434) 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 -- 172.21.15.111:0/665228379 >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 msgr2=0x7f7cec159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 --2- 172.21.15.111:0/665228379 >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 0x7f7cec159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 -- 172.21.15.111:0/665228379 >> v1:172.21.15.143:6789/0 conn(0x7f7cec158c90 legacy=0x7f7cec0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 -- 172.21.15.111:0/665228379 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec06b750 con 0x7f7cec1595a0 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7ce3fff700 1 -- 172.21.15.111:0/665228379 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591286804 0 0) 0x7f7cec06c850 con 0x7f7cec158c90 2023-12-19T09:19:39.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.201+0000 7f7cf184b700 1 --2- 172.21.15.111:0/665228379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec159a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009e60 tx=0x7f7ce4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80ab4e94dabd040 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:39.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7ce3fff700 1 -- 172.21.15.111:0/665228379 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce400e070 con 0x7f7cec1595a0 2023-12-19T09:19:39.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7ce3fff700 1 -- 172.21.15.111:0/665228379 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce400bb60 con 0x7f7cec1595a0 2023-12-19T09:19:39.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7ce3fff700 1 -- 172.21.15.111:0/665228379 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4012430 con 0x7f7cec1595a0 2023-12-19T09:19:39.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- 172.21.15.111:0/665228379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 msgr2=0x7f7cec159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 --2- 172.21.15.111:0/665228379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec159a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009e60 tx=0x7f7ce4009780 comp rx=0 tx=0).stop 2023-12-19T09:19:39.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- 172.21.15.111:0/665228379 shutdown_connections 2023-12-19T09:19:39.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 --2- 172.21.15.111:0/665228379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec159a00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 --2- 172.21.15.111:0/665228379 >> v2:172.21.15.111:3300/0 conn(0x7f7cec06aeb0 0x7f7cec159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- 172.21.15.111:0/665228379 >> 172.21.15.111:0/665228379 conn(0x7f7cec067bb0 msgr2=0x7f7cec0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:39.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- 172.21.15.111:0/665228379 shutdown_connections 2023-12-19T09:19:39.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- 172.21.15.111:0/665228379 wait complete. 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 Processor -- start 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.205+0000 7f7cf204c700 1 -- start start 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 0x7f7cec0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec0c6140 con 0x7f7cec06aeb0 2023-12-19T09:19:39.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec153280 con 0x7f7cec158c90 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec0c5530 con 0x7f7cec1595a0 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41388/0 (socket says 172.21.15.111:41388) 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 -- 172.21.15.111:0/817673115 learned_addr learned my addr 172.21.15.111:0/817673115 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf184b700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf0849700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 0x7f7cec0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 -- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 msgr2=0x7f7cec154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 -- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 msgr2=0x7f7cec0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 0x7f7cec0c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 -- 172.21.15.111:0/817673115 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4009a20 con 0x7f7cec06aeb0 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf184b700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:39.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf0849700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 0x7f7cec0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf104a700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80011e0 tx=0x7f7ce8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8007300 con 0x7f7cec06aeb0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce8007460 con 0x7f7cec06aeb0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec0c5810 con 0x7f7cec06aeb0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8004830 con 0x7f7cec06aeb0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.209+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec0c5dd0 con 0x7f7cec06aeb0 2023-12-19T09:19:39.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.213+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec153440 con 0x7f7cec06aeb0 2023-12-19T09:19:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.213+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7ce8004be0 con 0x7f7cec06aeb0 2023-12-19T09:19:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.213+0000 7f7ce17fa700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 0x7f7cd0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.213+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f7ce807c070 con 0x7f7cec06aeb0 2023-12-19T09:19:39.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.217+0000 7f7cf0849700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 0x7f7cd0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:39.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.217+0000 7f7cf0849700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 0x7f7cd0062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cec153a80 tx=0x7f7cdc008040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:39.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.221+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7ce804c320 con 0x7f7cec06aeb0 2023-12-19T09:19:39.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:39.425+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7cec06ba30 con 0x7f7cec06aeb0 2023-12-19T09:19:40.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.025+0000 7f7ce17fa700 1 -- 172.21.15.111:0/817673115 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7ce80180c0 con 0x7f7cec06aeb0 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 msgr2=0x7f7cd0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 0x7f7cd0062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7cec153a80 tx=0x7f7cdc008040 comp rx=0 tx=0).stop 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 msgr2=0x7f7cec0c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80011e0 tx=0x7f7ce8007e10 comp rx=0 tx=0).stop 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 shutdown_connections 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7cd0060550 0x7f7cd0062a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cec158c90 0x7f7cec0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cec1595a0 0x7f7cec154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 --2- 172.21.15.111:0/817673115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cec06aeb0 0x7f7cec0c65d0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.029+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 >> 172.21.15.111:0/817673115 conn(0x7f7cec067bb0 msgr2=0x7f7cec0cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.033+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 shutdown_connections 2023-12-19T09:19:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.033+0000 7f7cf204c700 1 -- 172.21.15.111:0/817673115 wait complete. 2023-12-19T09:19:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 23 and data pool 24 2023-12-19T09:19:40.072 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.229+0000 7fe69359e700 1 Processor -- start 2023-12-19T09:19:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.229+0000 7fe69359e700 1 -- start start 2023-12-19T09:19:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 0x7fe69406a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69406c7c0 con 0x7fe69406d210 2023-12-19T09:19:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69406cd80 con 0x7fe6940b1410 2023-12-19T09:19:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69406af10 con 0x7fe69406a580 2023-12-19T09:19:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe692d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe69406d210 0x7fe69406d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33734/0 (socket says 172.21.15.111:33734) 2023-12-19T09:19:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe692d9d700 1 -- 172.21.15.111:0/2806324307 learned_addr learned my addr 172.21.15.111:0/2806324307 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe691d9b700 1 --2- 172.21.15.111:0/2806324307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 0x7fe69406a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561780389 0 0) 0x7fe69406c7c0 con 0x7fe69406d210 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c003570 con 0x7fe69406d210 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864546385 0 0) 0x7fe69406cd80 con 0x7fe6940b1410 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe69406c7c0 con 0x7fe6940b1410 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe691d9b700 1 -- 172.21.15.111:0/2806324307 >> v1:172.21.15.143:6789/0 conn(0x7fe6940b1410 legacy=0x7fe6940ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe691d9b700 1 -- 172.21.15.111:0/2806324307 >> v1:172.21.15.111:6789/0 conn(0x7fe69406d210 legacy=0x7fe69406d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.233+0000 7fe691d9b700 1 -- 172.21.15.111:0/2806324307 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69406b8f0 con 0x7fe69406a580 2023-12-19T09:19:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe691d9b700 1 --2- 172.21.15.111:0/2806324307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 0x7fe69406a9d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe688009910 tx=0x7fe6880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eab519ce84a45ad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688011630 con 0x7fe69406a580 2023-12-19T09:19:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688011790 con 0x7fe69406a580 2023-12-19T09:19:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69159a700 1 -- 172.21.15.111:0/2806324307 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688011b30 con 0x7fe69406a580 2023-12-19T09:19:40.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 -- 172.21.15.111:0/2806324307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 msgr2=0x7fe69406a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 --2- 172.21.15.111:0/2806324307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 0x7fe69406a9d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe688009910 tx=0x7fe6880091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 -- 172.21.15.111:0/2806324307 shutdown_connections 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 --2- 172.21.15.111:0/2806324307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69406a580 0x7fe69406a9d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 -- 172.21.15.111:0/2806324307 >> 172.21.15.111:0/2806324307 conn(0x7fe694067bb0 msgr2=0x7fe6940cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 -- 172.21.15.111:0/2806324307 shutdown_connections 2023-12-19T09:19:40.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 -- 172.21.15.111:0/2806324307 wait complete. 2023-12-19T09:19:40.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.237+0000 7fe69359e700 1 Processor -- start 2023-12-19T09:19:40.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- start start 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69406a580 0x7fe6941670f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940b1410 0x7fe694169f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69416a4a0 con 0x7fe69406d210 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c0860 con 0x7fe69406a580 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c4d10 con 0x7fe6940b1410 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41390/0 (socket says 172.21.15.111:41390) 2023-12-19T09:19:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 -- 172.21.15.111:0/412190841 learned_addr learned my addr 172.21.15.111:0/412190841 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69259c700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69406a580 0x7fe6941670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe692d9d700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940b1410 0x7fe694169f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 -- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940b1410 msgr2=0x7fe694169f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940b1410 0x7fe694169f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 -- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69406a580 msgr2=0x7fe6941670f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69406a580 0x7fe6941670f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 -- 172.21.15.111:0/412190841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688009420 con 0x7fe69406d210 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe691d9b700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe688009f00 tx=0x7fe688017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68801f070 con 0x7fe69406d210 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688002f70 con 0x7fe69406d210 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6940c4ff0 con 0x7fe69406d210 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688011900 con 0x7fe69406d210 2023-12-19T09:19:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.241+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6940c55b0 con 0x7fe69406d210 2023-12-19T09:19:40.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.245+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6940c0a20 con 0x7fe69406d210 2023-12-19T09:19:40.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.245+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe688023070 con 0x7fe69406d210 2023-12-19T09:19:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.249+0000 7fe6827fc700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.249+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe68807def0 con 0x7fe69406d210 2023-12-19T09:19:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.249+0000 7fe69259c700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.249+0000 7fe69259c700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe684007fd0 tx=0x7fe684007d50 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.253+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe68804e0c0 con 0x7fe69406d210 2023-12-19T09:19:40.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.421+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe69404be40 con 0x7fe69406d210 2023-12-19T09:19:40.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.421+0000 7fe6827fc700 1 -- 172.21.15.111:0/412190841 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16510 (secure 0 0 0) 0x7fe688015090 con 0x7fe69406d210 2023-12-19T09:19:40.427 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:40.427 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":192,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:39.430383+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:40.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 msgr2=0x7fe674062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 0x7fe674062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe684007fd0 tx=0x7fe684007d50 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 msgr2=0x7fe6940c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe688009f00 tx=0x7fe688017000 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 shutdown_connections 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940b1410 0x7fe694169f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69406d210 0x7fe6940c2b70 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe674060550 0x7fe674062a00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 --2- 172.21.15.111:0/412190841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69406a580 0x7fe6941670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 >> 172.21.15.111:0/412190841 conn(0x7fe694067bb0 msgr2=0x7fe694156410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 shutdown_connections 2023-12-19T09:19:40.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.425+0000 7fe69359e700 1 -- 172.21.15.111:0/412190841 wait complete. 2023-12-19T09:19:40.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:40.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 Processor -- start 2023-12-19T09:19:40.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 -- start start 2023-12-19T09:19:40.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe758068800 0x7fe758161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75806e150 con 0x7fe758140ae0 2023-12-19T09:19:40.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75806d540 con 0x7fe758140710 2023-12-19T09:19:40.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.597+0000 7fe75e385700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75806d820 con 0x7fe758068800 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe758068800 0x7fe758161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe7577fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe758140ae0 0x7fe7580680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33736/0 (socket says 172.21.15.111:33736) 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe758068800 0x7fe758161cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41394/0 (socket says 172.21.15.111:41394) 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe7577fe700 1 -- 172.21.15.111:0/2299835232 learned_addr learned my addr 172.21.15.111:0/2299835232 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 -- 172.21.15.111:0/2299835232 >> v1:172.21.15.143:6789/0 conn(0x7fe758140710 legacy=0x7fe7580c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 -- 172.21.15.111:0/2299835232 >> v1:172.21.15.111:6789/0 conn(0x7fe758140ae0 legacy=0x7fe7580680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 -- 172.21.15.111:0/2299835232 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75806db00 con 0x7fe758068800 2023-12-19T09:19:40.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75c922700 1 --2- 172.21.15.111:0/2299835232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758068800 0x7fe758161cd0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe750009e60 tx=0x7fe750009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1882b200f6de5215 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe756ffd700 1 -- 172.21.15.111:0/2299835232 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75000e070 con 0x7fe758068800 2023-12-19T09:19:40.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe756ffd700 1 -- 172.21.15.111:0/2299835232 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75000bc30 con 0x7fe758068800 2023-12-19T09:19:40.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe756ffd700 1 -- 172.21.15.111:0/2299835232 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750012430 con 0x7fe758068800 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 -- 172.21.15.111:0/2299835232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758068800 msgr2=0x7fe758161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 --2- 172.21.15.111:0/2299835232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758068800 0x7fe758161cd0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe750009e60 tx=0x7fe750009780 comp rx=0 tx=0).stop 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 -- 172.21.15.111:0/2299835232 shutdown_connections 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 --2- 172.21.15.111:0/2299835232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758068800 0x7fe758161cd0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 -- 172.21.15.111:0/2299835232 >> 172.21.15.111:0/2299835232 conn(0x7fe7580c8850 msgr2=0x7fe7580c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 -- 172.21.15.111:0/2299835232 shutdown_connections 2023-12-19T09:19:40.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.601+0000 7fe75e385700 1 -- 172.21.15.111:0/2299835232 wait complete. 2023-12-19T09:19:40.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 Processor -- start 2023-12-19T09:19:40.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 -- start start 2023-12-19T09:19:40.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe758068800 0x7fe7580c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe758140710 0x7fe7580c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580c4080 con 0x7fe758140ae0 2023-12-19T09:19:40.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580c5a00 con 0x7fe758140710 2023-12-19T09:19:40.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75e385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580c3470 con 0x7fe758068800 2023-12-19T09:19:40.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75c922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75c922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41404/0 (socket says 172.21.15.111:41404) 2023-12-19T09:19:40.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe75c922700 1 -- 172.21.15.111:0/2441193186 learned_addr learned my addr 172.21.15.111:0/2441193186 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe757fff700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe758068800 0x7fe7580c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.605+0000 7fe7577fe700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe758140710 0x7fe7580c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 -- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe758068800 msgr2=0x7fe7580c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe758068800 0x7fe7580c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 -- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe758140710 msgr2=0x7fe7580c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe758140710 0x7fe7580c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 -- 172.21.15.111:0/2441193186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe750009a20 con 0x7fe758140ae0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75c922700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe750013000 tx=0x7fe750019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75001e630 con 0x7fe758140ae0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75001e7b0 con 0x7fe758140ae0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75001eb70 con 0x7fe758140ae0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580c3750 con 0x7fe758140ae0 2023-12-19T09:19:40.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7580c3d10 con 0x7fe758140ae0 2023-12-19T09:19:40.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.609+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe758071120 con 0x7fe758140ae0 2023-12-19T09:19:40.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.613+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe750024070 con 0x7fe758140ae0 2023-12-19T09:19:40.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.613+0000 7fe73ffff700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 0x7fe744062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.613+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe75007f4d0 con 0x7fe758140ae0 2023-12-19T09:19:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.613+0000 7fe757fff700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 0x7fe744062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.617+0000 7fe757fff700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 0x7fe744062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe748003880 tx=0x7fe748009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.617+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe75004f6a0 con 0x7fe758140ae0 2023-12-19T09:19:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.785+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7580c41c0 con 0x7fe758140ae0 2023-12-19T09:19:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe73ffff700 1 -- 172.21.15.111:0/2441193186 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16510 (secure 0 0 0) 0x7fe7580c41c0 con 0x7fe758140ae0 2023-12-19T09:19:40.793 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:40.793 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":192,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:39.430383+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 msgr2=0x7fe744062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 0x7fe744062a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe748003880 tx=0x7fe748009040 comp rx=0 tx=0).stop 2023-12-19T09:19:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 msgr2=0x7fe75813b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fe750013000 tx=0x7fe750019000 comp rx=0 tx=0).stop 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 shutdown_connections 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe744060580 0x7fe744062a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe758140710 0x7fe7580c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe758140ae0 0x7fe75813b550 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 --2- 172.21.15.111:0/2441193186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe758068800 0x7fe7580c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.789+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 >> 172.21.15.111:0/2441193186 conn(0x7fe7580c8850 msgr2=0x7fe75813e750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.793+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 shutdown_connections 2023-12-19T09:19:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.793+0000 7fe75e385700 1 -- 172.21.15.111:0/2441193186 wait complete. 2023-12-19T09:19:40.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T09:19:40.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.973+0000 7fe78b22e700 1 Processor -- start 2023-12-19T09:19:40.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe78b22e700 1 -- start start 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe78b22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe78406bc40 0x7fe784161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe78b22e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78406a7a0 con 0x7fe78406c7c0 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe78b22e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78406d4b0 con 0x7fe7840c6d20 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe78b22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78406d790 con 0x7fe78406bc40 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe783fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe78406c7c0 0x7fe78406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33744/0 (socket says 172.21.15.111:33744) 2023-12-19T09:19:40.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe783fff700 1 -- 172.21.15.111:0/3978927779 learned_addr learned my addr 172.21.15.111:0/3978927779 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500571554 0 0) 0x7fe78406a7a0 con 0x7fe78406c7c0 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774003570 con 0x7fe78406c7c0 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7897cb700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe78406bc40 0x7fe784161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376469696 0 0) 0x7fe774003570 con 0x7fe78406c7c0 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe78406bc40 msgr2=0x7fe784161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe78406bc40 0x7fe784161dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 >> v1:172.21.15.143:6789/0 conn(0x7fe7840c6d20 legacy=0x7fe7840c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78406da70 con 0x7fe78406c7c0 2023-12-19T09:19:40.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fe770001490 con 0x7fe78406c7c0 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 >> v1:172.21.15.111:6789/0 conn(0x7fe78406c7c0 legacy=0x7fe78406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774003a90 0x7fe774003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe774004810 0x7fe774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.977+0000 7fe7837fe700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe774005210 0x7fe77400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78406d790 con 0x7fe774004810 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400c0e0 con 0x7fe774005210 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400c2a0 con 0x7fe774003a90 2023-12-19T09:19:40.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe774004810 0x7fe774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 -- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774003a90 msgr2=0x7fe774003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:40.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe788fca700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774003a90 0x7fe774003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7897cb700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe774005210 0x7fe77400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774003a90 0x7fe774003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 -- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe774005210 msgr2=0x7fe77400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe774005210 0x7fe77400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 -- 172.21.15.111:0/3978927779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774003570 con 0x7fe774004810 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe788fca700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774003a90 0x7fe774003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7897cb700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe774005210 0x7fe77400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe783fff700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe774004810 0x7fe774004c80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe77000c390 tx=0x7fe770001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be6b4b15f28d725 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000cc60 con 0x7fe774004810 2023-12-19T09:19:40.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77000cdc0 con 0x7fe774004810 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7837fe700 1 -- 172.21.15.111:0/3978927779 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770004b40 con 0x7fe774004810 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 -- 172.21.15.111:0/3978927779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe774004810 msgr2=0x7fe774004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 --2- 172.21.15.111:0/3978927779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe774004810 0x7fe774004c80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe77000c390 tx=0x7fe770001e70 comp rx=0 tx=0).stop 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe7897cb700 1 -- 172.21.15.111:0/3978927779 reap_dead start 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 -- 172.21.15.111:0/3978927779 shutdown_connections 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 -- 172.21.15.111:0/3978927779 >> 172.21.15.111:0/3978927779 conn(0x7fe784067bb0 msgr2=0x7fe7840cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 -- 172.21.15.111:0/3978927779 shutdown_connections 2023-12-19T09:19:40.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.981+0000 7fe78b22e700 1 -- 172.21.15.111:0/3978927779 wait complete. 2023-12-19T09:19:40.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 Processor -- start 2023-12-19T09:19:40.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 -- start start 2023-12-19T09:19:40.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774004810 0x7fe784166370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 0x7fe7840c3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7841668b0 con 0x7fe7840c0b20 2023-12-19T09:19:40.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7840c14d0 con 0x7fe7840c3020 2023-12-19T09:19:40.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.985+0000 7fe78b22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7840c4960 con 0x7fe774004810 2023-12-19T09:19:40.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41414/0 (socket says 172.21.15.111:41414) 2023-12-19T09:19:40.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 -- 172.21.15.111:0/4002025734 learned_addr learned my addr 172.21.15.111:0/4002025734 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:40.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe788fca700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774004810 0x7fe784166370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe7897cb700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 0x7fe7840c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:40.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 -- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774004810 msgr2=0x7fe784166370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774004810 0x7fe784166370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 -- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 msgr2=0x7fe7840c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 0x7fe7840c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 -- 172.21.15.111:0/4002025734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7840c4df0 con 0x7fe7840c0b20 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe7897cb700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 0x7fe7840c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe783fff700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c0b0 tx=0x7fe770015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770007360 con 0x7fe7840c0b20 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7700074c0 con 0x7fe7840c0b20 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7840c3d50 con 0x7fe7840c0b20 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770007860 con 0x7fe7840c0b20 2023-12-19T09:19:40.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.989+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7840c4310 con 0x7fe7840c0b20 2023-12-19T09:19:40.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.993+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe784071120 con 0x7fe7840c0b20 2023-12-19T09:19:40.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.993+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe77001f630 con 0x7fe7840c0b20 2023-12-19T09:19:40.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.993+0000 7fe780ff9700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 0x7fe768062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:40.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.993+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fe77007e930 con 0x7fe7840c0b20 2023-12-19T09:19:40.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.993+0000 7fe788fca700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 0x7fe768062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:41.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:40.997+0000 7fe788fca700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 0x7fe768062a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe778007fd0 tx=0x7fe778009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:41.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:41.001+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe77004eba0 con 0x7fe7840c0b20 2023-12-19T09:19:41.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:41.209+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe7840c4aa0 con 0x7fe7840c0b20 2023-12-19T09:19:42.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.041+0000 7fe780ff9700 1 -- 172.21.15.111:0/4002025734 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v86) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe7700521c0 con 0x7fe7840c0b20 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 msgr2=0x7fe768062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 0x7fe768062a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe778007fd0 tx=0x7fe778009040 comp rx=0 tx=0).stop 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 msgr2=0x7fe7840c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe77400c0b0 tx=0x7fe770015000 comp rx=0 tx=0).stop 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 shutdown_connections 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe768060580 0x7fe768062a30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7840c3020 0x7fe7840c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7840c0b20 0x7fe7840c0f90 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 --2- 172.21.15.111:0/4002025734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe774004810 0x7fe784166370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 >> 172.21.15.111:0/4002025734 conn(0x7fe784067bb0 msgr2=0x7fe784159ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 shutdown_connections 2023-12-19T09:19:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.045+0000 7fe78b22e700 1 -- 172.21.15.111:0/4002025734 wait complete. 2023-12-19T09:19:42.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T09:19:42.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 Processor -- start 2023-12-19T09:19:42.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 -- start start 2023-12-19T09:19:42.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 0x7fb7cc0c7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc159b80 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0c8220 con 0x7fb7cc0c53f0 2023-12-19T09:19:42.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.237+0000 7fb7cb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c7f40 con 0x7fb7cc141e70 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb7cc0c7a80 0x7fb7cc0c4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50524/0 (socket says 172.21.15.111:50524) 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3381624060 learned_addr learned my addr 172.21.15.111:0/3381624060 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7ca59c700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 0x7fb7cc0c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34028351 0 0) 0x7fb7cc159b80 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b8003570 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883200044 0 0) 0x7fb7cc0c8220 con 0x7fb7cc0c53f0 2023-12-19T09:19:42.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7cc159b80 con 0x7fb7cc0c53f0 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2388999483 0 0) 0x7fb7b8003570 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 msgr2=0x7fb7cc0c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 0x7fb7cc0c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 >> v1:172.21.15.143:6789/0 conn(0x7fb7cc0c53f0 legacy=0x7fb7cc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc159530 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fb7b4001470 con 0x7fb7cc0c7a80 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 >> v1:172.21.15.111:6789/0 conn(0x7fb7cc0c7a80 legacy=0x7fb7cc0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 0x7fb7b8003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7b8004160 0x7fb7b8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7b8004d50 0x7fb7b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c7f40 con 0x7fb7b8004160 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b800bb40 con 0x7fb7b8004d50 2023-12-19T09:19:42.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b800bd00 con 0x7fb7cc141e70 2023-12-19T09:19:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7b8004160 0x7fb7b8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 msgr2=0x7fb7b8003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7cc141e70 0x7fb7b8003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7b8004d50 msgr2=0x7fb7b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7b8004d50 0x7fb7b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3381624060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8003570 con 0x7fb7b8004160 2023-12-19T09:19:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7b8004160 0x7fb7b8004810 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c7e70 tx=0x7fb7b4008c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71210ab31c086d07 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4005920 con 0x7fb7b8004160 2023-12-19T09:19:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b4005a80 con 0x7fb7b8004160 2023-12-19T09:19:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.241+0000 7fb7c9d9b700 1 -- 172.21.15.111:0/3381624060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4005e40 con 0x7fb7b8004160 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3381624060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7b8004160 msgr2=0x7fb7b8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3381624060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7b8004160 0x7fb7b8004810 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c7e70 tx=0x7fb7b4008c30 comp rx=0 tx=0).stop 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cad9d700 1 -- 172.21.15.111:0/3381624060 reap_dead start 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3381624060 shutdown_connections 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3381624060 >> 172.21.15.111:0/3381624060 conn(0x7fb7cc067b50 msgr2=0x7fb7cc0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3381624060 shutdown_connections 2023-12-19T09:19:42.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3381624060 wait complete. 2023-12-19T09:19:42.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.245+0000 7fb7cb59e700 1 Processor -- start 2023-12-19T09:19:42.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- start start 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7b8004160 0x7fb7cc158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 0x7fb7cc133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc1331e0 con 0x7fb7cc0af360 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0b0910 con 0x7fb7cc133670 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc132eb0 con 0x7fb7b8004160 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57418/0 (socket says 172.21.15.111:57418) 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3614972861 learned_addr learned my addr 172.21.15.111:0/3614972861 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7ca59c700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7b8004160 0x7fb7cc158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cad9d700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 0x7fb7cc133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7b8004160 msgr2=0x7fb7cc158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7b8004160 0x7fb7cc158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 msgr2=0x7fb7cc133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 0x7fb7cc133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 -- 172.21.15.111:0/3614972861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0b0070 con 0x7fb7cc0af360 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cad9d700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 0x7fb7cc133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c1d9b700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4008c70 tx=0x7fb7b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4001b00 con 0x7fb7cc0af360 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b40082f0 con 0x7fb7cc0af360 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc0b0350 con 0x7fb7cc0af360 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40086b0 con 0x7fb7cc0af360 2023-12-19T09:19:42.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.249+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc0afcf0 con 0x7fb7cc0af360 2023-12-19T09:19:42.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.253+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc133390 con 0x7fb7cc0af360 2023-12-19T09:19:42.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.253+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb7b402a460 con 0x7fb7cc0af360 2023-12-19T09:19:42.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.253+0000 7fb7c2ffd700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 0x7fb7ac062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:42.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.253+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fb7b407ec00 con 0x7fb7cc0af360 2023-12-19T09:19:42.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.253+0000 7fb7ca59c700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 0x7fb7ac062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:42.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.257+0000 7fb7ca59c700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 0x7fb7ac062a80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009ce0 tx=0x7fb7bc0091a0 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:42.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.261+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb7cc133390 con 0x7fb7cc0af360 2023-12-19T09:19:42.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:42.469+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb7cc133390 con 0x7fb7cc0af360 2023-12-19T09:19:43.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.041+0000 7fb7c2ffd700 1 -- 172.21.15.111:0/3614972861 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v87) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb7cc133390 con 0x7fb7cc0af360 2023-12-19T09:19:43.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 msgr2=0x7fb7ac062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 0x7fb7ac062a80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009ce0 tx=0x7fb7bc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 msgr2=0x7fb7cc0af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4008c70 tx=0x7fb7b4016000 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 shutdown_connections 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb7ac0605d0 0x7fb7ac062a80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7cc133670 0x7fb7cc133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7cc0af360 0x7fb7cc0af7b0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 --2- 172.21.15.111:0/3614972861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7b8004160 0x7fb7cc158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.045+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 >> 172.21.15.111:0/3614972861 conn(0x7fb7cc067b50 msgr2=0x7fb7cc15c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.049+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 shutdown_connections 2023-12-19T09:19:43.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.049+0000 7fb7cb59e700 1 -- 172.21.15.111:0/3614972861 wait complete. 2023-12-19T09:19:43.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:43.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.233+0000 7ff44880c700 1 Processor -- start 2023-12-19T09:19:43.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.233+0000 7ff44880c700 1 -- start start 2023-12-19T09:19:43.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff44880c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff44415bd00 0x7ff44415c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff44880c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44406b2c0 con 0x7ff4440cad10 2023-12-19T09:19:43.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff44880c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44406e0f0 con 0x7ff4440cb0e0 2023-12-19T09:19:43.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff44880c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44406a6b0 con 0x7ff44415bd00 2023-12-19T09:19:43.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff443fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff44415bd00 0x7ff44415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff4437fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff4440cad10 0x7ff44406b750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50532/0 (socket says 172.21.15.111:50532) 2023-12-19T09:19:43.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff4437fe700 1 -- 172.21.15.111:0/3604503422 learned_addr learned my addr 172.21.15.111:0/3604503422 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:43.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff443fff700 1 -- 172.21.15.111:0/3604503422 >> v1:172.21.15.143:6789/0 conn(0x7ff4440cb0e0 legacy=0x7ff44406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff443fff700 1 -- 172.21.15.111:0/3604503422 >> v1:172.21.15.111:6789/0 conn(0x7ff4440cad10 legacy=0x7ff44406b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff443fff700 1 -- 172.21.15.111:0/3604503422 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44406a990 con 0x7ff44415bd00 2023-12-19T09:19:43.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff443fff700 1 --2- 172.21.15.111:0/3604503422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff44415c150 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0098e0 tx=0x7ff43c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f251a41cd2981214 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff442ffd700 1 -- 172.21.15.111:0/3604503422 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c009e80 con 0x7ff44415bd00 2023-12-19T09:19:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff442ffd700 1 -- 172.21.15.111:0/3604503422 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c00b860 con 0x7ff44415bd00 2023-12-19T09:19:43.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.237+0000 7ff442ffd700 1 -- 172.21.15.111:0/3604503422 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00bbe0 con 0x7ff44415bd00 2023-12-19T09:19:43.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 -- 172.21.15.111:0/3604503422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 msgr2=0x7ff44415c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 --2- 172.21.15.111:0/3604503422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff44415c150 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0098e0 tx=0x7ff43c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 -- 172.21.15.111:0/3604503422 shutdown_connections 2023-12-19T09:19:43.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 --2- 172.21.15.111:0/3604503422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff44415c150 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 -- 172.21.15.111:0/3604503422 >> 172.21.15.111:0/3604503422 conn(0x7ff444067bb0 msgr2=0x7ff4440cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:43.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 -- 172.21.15.111:0/3604503422 shutdown_connections 2023-12-19T09:19:43.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 -- 172.21.15.111:0/3604503422 wait complete. 2023-12-19T09:19:43.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.241+0000 7ff44880c700 1 Processor -- start 2023-12-19T09:19:43.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 -- start start 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4440cad10 0x7ff44415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4440cb0e0 0x7ff444169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4440c9ad0 con 0x7ff44415bd00 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff444169cb0 con 0x7ff4440cad10 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff44880c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff444159310 con 0x7ff4440cb0e0 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57440/0 (socket says 172.21.15.111:57440) 2023-12-19T09:19:43.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 -- 172.21.15.111:0/2719989587 learned_addr learned my addr 172.21.15.111:0/2719989587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:43.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff43affd700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4440cb0e0 0x7ff444169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff4437fe700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4440cad10 0x7ff44415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 -- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4440cb0e0 msgr2=0x7ff444169770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4440cb0e0 0x7ff444169770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 -- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4440cad10 msgr2=0x7ff44415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4440cad10 0x7ff44415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 -- 172.21.15.111:0/2719989587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c009420 con 0x7ff44415bd00 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.245+0000 7ff443fff700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff44406e6d0 tx=0x7ff43c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c025420 con 0x7ff44415bd00 2023-12-19T09:19:43.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c025580 con 0x7ff44415bd00 2023-12-19T09:19:43.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4441595f0 con 0x7ff44415bd00 2023-12-19T09:19:43.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c025920 con 0x7ff44415bd00 2023-12-19T09:19:43.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff444159bb0 con 0x7ff44415bd00 2023-12-19T09:19:43.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44404be40 con 0x7ff44415bd00 2023-12-19T09:19:43.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff43c023070 con 0x7ff44415bd00 2023-12-19T09:19:43.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 0x7ff424062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.249+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7ff43c07e740 con 0x7ff44415bd00 2023-12-19T09:19:43.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.253+0000 7ff4437fe700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 0x7ff424062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.253+0000 7ff4437fe700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 0x7ff424062a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4340021c0 tx=0x7ff434009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.257+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff43c04e9b0 con 0x7ff44415bd00 2023-12-19T09:19:43.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.461+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4440d08a0 con 0x7ff44415bd00 2023-12-19T09:19:43.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.461+0000 7ff43bfff700 1 -- 172.21.15.111:0/2719989587 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7ff43c051fd0 con 0x7ff44415bd00 2023-12-19T09:19:43.466 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:43.466 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 msgr2=0x7ff424062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 0x7ff424062a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4340021c0 tx=0x7ff434009040 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 msgr2=0x7ff4440c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff44406e6d0 tx=0x7ff43c017000 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 shutdown_connections 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff44415bd00 0x7ff4440c7930 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4440cb0e0 0x7ff444169770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff424060550 0x7ff424062a00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 --2- 172.21.15.111:0/2719989587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4440cad10 0x7ff44415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 >> 172.21.15.111:0/2719989587 conn(0x7ff444067bb0 msgr2=0x7ff4440cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:43.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 shutdown_connections 2023-12-19T09:19:43.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.465+0000 7ff44880c700 1 -- 172.21.15.111:0/2719989587 wait complete. 2023-12-19T09:19:43.471 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:19:43.490 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:19:43.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 Processor -- start 2023-12-19T09:19:43.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 -- start start 2023-12-19T09:19:43.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4d2406a580 0x7f4d2406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 0x7f4d2406d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2406c7c0 con 0x7f4d240b1410 2023-12-19T09:19:43.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2406af40 con 0x7f4d2406a580 2023-12-19T09:19:43.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d29ae3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2406cd80 con 0x7f4d2406d210 2023-12-19T09:19:43.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d23fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4d2406a580 0x7f4d2406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d23fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4d2406a580 0x7f4d2406aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57456/0 (socket says 172.21.15.111:57456) 2023-12-19T09:19:43.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.649+0000 7f4d23fff700 1 -- 172.21.15.111:0/270002207 learned_addr learned my addr 172.21.15.111:0/270002207 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:43.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d23fff700 1 -- 172.21.15.111:0/270002207 >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 msgr2=0x7f4d2406d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:43.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d292e2700 1 --2- 172.21.15.111:0/270002207 >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 0x7f4d2406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d23fff700 1 --2- 172.21.15.111:0/270002207 >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 0x7f4d2406d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d23fff700 1 -- 172.21.15.111:0/270002207 >> v1:172.21.15.143:6789/0 conn(0x7f4d240b1410 legacy=0x7f4d240ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d23fff700 1 -- 172.21.15.111:0/270002207 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2406b8f0 con 0x7f4d2406a580 2023-12-19T09:19:43.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d292e2700 1 --2- 172.21.15.111:0/270002207 >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 0x7f4d2406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:43.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d23fff700 1 --2- 172.21.15.111:0/270002207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d2406a580 0x7f4d2406aa00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4d2415b900 tx=0x7f4d14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a07b182d61069c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d237fe700 1 -- 172.21.15.111:0/270002207 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400b860 con 0x7f4d2406a580 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d237fe700 1 -- 172.21.15.111:0/270002207 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1400b9c0 con 0x7f4d2406a580 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d237fe700 1 -- 172.21.15.111:0/270002207 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400bd60 con 0x7f4d2406a580 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 -- 172.21.15.111:0/270002207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d2406a580 msgr2=0x7f4d2406aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/270002207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d2406a580 0x7f4d2406aa00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f4d2415b900 tx=0x7f4d14009250 comp rx=0 tx=0).stop 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 -- 172.21.15.111:0/270002207 shutdown_connections 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/270002207 >> v2:172.21.15.143:3300/0 conn(0x7f4d2406d210 0x7f4d2406d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/270002207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d2406a580 0x7f4d2406aa00 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 -- 172.21.15.111:0/270002207 >> 172.21.15.111:0/270002207 conn(0x7f4d24067bb0 msgr2=0x7f4d240cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:43.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 -- 172.21.15.111:0/270002207 shutdown_connections 2023-12-19T09:19:43.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.653+0000 7f4d29ae3700 1 -- 172.21.15.111:0/270002207 wait complete. 2023-12-19T09:19:43.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 Processor -- start 2023-12-19T09:19:43.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 -- start start 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 0x7f4d240c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d2406d210 0x7f4d240add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24171d30 con 0x7f4d240b1410 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240c6150 con 0x7f4d2406a580 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d29ae3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240c63a0 con 0x7f4d2406d210 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d292e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d292e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57466/0 (socket says 172.21.15.111:57466) 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d292e2700 1 -- 172.21.15.111:0/1842540398 learned_addr learned my addr 172.21.15.111:0/1842540398 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d23fff700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d2406d210 0x7f4d240add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 -- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d2406d210 msgr2=0x7f4d240add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d2406d210 0x7f4d240add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.657+0000 7f4d28ae1700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 0x7f4d240c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 -- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 msgr2=0x7f4d240c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 0x7f4d240c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 -- 172.21.15.111:0/1842540398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140094f0 con 0x7f4d240b1410 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d28ae1700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 0x7f4d240c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d292e2700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00a380 tx=0x7f4d1c007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c015440 con 0x7f4d240b1410 2023-12-19T09:19:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c0155a0 con 0x7f4d240b1410 2023-12-19T09:19:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d240c6680 con 0x7f4d240b1410 2023-12-19T09:19:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c015960 con 0x7f4d240b1410 2023-12-19T09:19:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d240ae280 con 0x7f4d240b1410 2023-12-19T09:19:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.661+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d240c6e60 con 0x7f4d240b1410 2023-12-19T09:19:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.665+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f4d1c00d440 con 0x7f4d240b1410 2023-12-19T09:19:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.665+0000 7f4d20ff9700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 0x7f4d10062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.665+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f4d1c07d760 con 0x7f4d240b1410 2023-12-19T09:19:43.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.665+0000 7f4d28ae1700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 0x7f4d10062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:43.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.665+0000 7f4d28ae1700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 0x7f4d10062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d180017a0 tx=0x7f4d18009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:43.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.669+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4d1c04d930 con 0x7f4d240b1410 2023-12-19T09:19:43.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.841+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d2404be40 con 0x7f4d240b1410 2023-12-19T09:19:43.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.845+0000 7f4d20ff9700 1 -- 172.21.15.111:0/1842540398 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f4d1c050f50 con 0x7f4d240b1410 2023-12-19T09:19:43.849 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:43.849 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":193,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:19:41.021560+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:19:43.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 msgr2=0x7f4d10062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 0x7f4d10062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d180017a0 tx=0x7f4d18009040 comp rx=0 tx=0).stop 2023-12-19T09:19:43.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 msgr2=0x7f4d240c11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:43.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00a380 tx=0x7f4d1c007900 comp rx=0 tx=0).stop 2023-12-19T09:19:43.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 shutdown_connections 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d240b1410 0x7f4d240c11a0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d2406d210 0x7f4d240add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4d10060550 0x7f4d10062a00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 --2- 172.21.15.111:0/1842540398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d2406a580 0x7f4d240c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 >> 172.21.15.111:0/1842540398 conn(0x7f4d24067bb0 msgr2=0x7f4d241556b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 shutdown_connections 2023-12-19T09:19:43.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:43.849+0000 7f4d29ae3700 1 -- 172.21.15.111:0/1842540398 wait complete. 2023-12-19T09:19:43.875 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T09:19:43.875 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:19:44.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 Processor -- start 2023-12-19T09:19:44.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 -- start start 2023-12-19T09:19:44.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 0x7f9f88067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f88068c70 con 0x7f9f88069100 2023-12-19T09:19:44.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f88068060 con 0x7f9f88159340 2023-12-19T09:19:44.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f9022e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f88068340 con 0x7f9f88158f70 2023-12-19T09:19:44.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8d7c9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9f88069100 0x7f9f880cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50540/0 (socket says 172.21.15.111:50540) 2023-12-19T09:19:44.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8d7c9700 1 -- 172.21.15.111:0/637267633 learned_addr learned my addr 172.21.15.111:0/637267633 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:44.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8dfca700 1 --2- 172.21.15.111:0/637267633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 0x7f9f88067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783109214 0 0) 0x7f9f88068c70 con 0x7f9f88069100 2023-12-19T09:19:44.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74003570 con 0x7f9f88069100 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8dfca700 1 -- 172.21.15.111:0/637267633 >> v1:172.21.15.143:6789/0 conn(0x7f9f88159340 legacy=0x7f9f880cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514079057 0 0) 0x7f9f88068060 con 0x7f9f88159340 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8dfca700 1 -- 172.21.15.111:0/637267633 >> v1:172.21.15.111:6789/0 conn(0x7f9f88069100 legacy=0x7f9f880cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.029+0000 7f9f8dfca700 1 -- 172.21.15.111:0/637267633 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88068620 con 0x7f9f88158f70 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f8dfca700 1 --2- 172.21.15.111:0/637267633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 0x7f9f88067980 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009910 tx=0x7f9f840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500d349765bc1301 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400b680 con 0x7f9f88158f70 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8400b7e0 con 0x7f9f88158f70 2023-12-19T09:19:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f8cfc8700 1 -- 172.21.15.111:0/637267633 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400bba0 con 0x7f9f88158f70 2023-12-19T09:19:44.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 -- 172.21.15.111:0/637267633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 msgr2=0x7f9f88067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 --2- 172.21.15.111:0/637267633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 0x7f9f88067980 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009910 tx=0x7f9f840091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 -- 172.21.15.111:0/637267633 shutdown_connections 2023-12-19T09:19:44.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 --2- 172.21.15.111:0/637267633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88158f70 0x7f9f88067980 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 -- 172.21.15.111:0/637267633 >> 172.21.15.111:0/637267633 conn(0x7f9f880ccc40 msgr2=0x7f9f880cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:44.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 -- 172.21.15.111:0/637267633 shutdown_connections 2023-12-19T09:19:44.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.033+0000 7f9f9022e700 1 -- 172.21.15.111:0/637267633 wait complete. 2023-12-19T09:19:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 Processor -- start 2023-12-19T09:19:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 -- start start 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88069100 0x7f9f881561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 0x7f9f880c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880b43e0 con 0x7f9f88158f70 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880b46c0 con 0x7f9f88159340 2023-12-19T09:19:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f9022e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880b49a0 con 0x7f9f88069100 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57478/0 (socket says 172.21.15.111:57478) 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 -- 172.21.15.111:0/3010398139 learned_addr learned my addr 172.21.15.111:0/3010398139 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8dfca700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88069100 0x7f9f881561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 -- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88069100 msgr2=0x7f9f881561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88069100 0x7f9f881561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 -- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 msgr2=0x7f9f880c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8e7cb700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 0x7f9f880c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 0x7f9f880c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 -- 172.21.15.111:0/3010398139 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f84009420 con 0x7f9f88158f70 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8e7cb700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 0x7f9f880c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.037+0000 7f9f8d7c9700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c7c0 tx=0x7f9f78007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78005b30 con 0x7f9f88158f70 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f78005c90 con 0x7f9f88158f70 2023-12-19T09:19:44.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f880b4c80 con 0x7f9f88158f70 2023-12-19T09:19:44.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78007340 con 0x7f9f88158f70 2023-12-19T09:19:44.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f88140860 con 0x7f9f88158f70 2023-12-19T09:19:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.041+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f880b4ea0 con 0x7f9f88158f70 2023-12-19T09:19:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.045+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9f7801b400 con 0x7f9f88158f70 2023-12-19T09:19:44.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.045+0000 7f9f7dffb700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 0x7f9f6c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.045+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f9f7807c630 con 0x7f9f88158f70 2023-12-19T09:19:44.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.045+0000 7f9f8dfca700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 0x7f9f6c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.045+0000 7f9f8dfca700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 0x7f9f6c062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f840091e0 tx=0x7f9f84017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.053+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9f7804c920 con 0x7f9f88158f70 2023-12-19T09:19:44.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.225+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f880b4830 con 0x7f9f6c060550 2023-12-19T09:19:44.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.245+0000 7f9f7dffb700 1 -- 172.21.15.111:0/3010398139 <== mgr.8285 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287018 (secure 0 0 0) 0x7f9f880b4830 con 0x7f9f6c060550 2023-12-19T09:19:44.252 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 msgr2=0x7f9f6c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 0x7f9f6c062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f840091e0 tx=0x7f9f84017000 comp rx=0 tx=0).stop 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 msgr2=0x7f9f88140320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c7c0 tx=0x7f9f78007e00 comp rx=0 tx=0).stop 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 shutdown_connections 2023-12-19T09:19:44.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9f6c060550 0x7f9f6c062a00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f88159340 0x7f9f880c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f88158f70 0x7f9f88140320 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 --2- 172.21.15.111:0/3010398139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9f88069100 0x7f9f881561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 >> 172.21.15.111:0/3010398139 conn(0x7f9f880ccc40 msgr2=0x7f9f880b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 shutdown_connections 2023-12-19T09:19:44.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.249+0000 7f9f9022e700 1 -- 172.21.15.111:0/3010398139 wait complete. 2023-12-19T09:19:44.258 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:19:44.276 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:19:43.669447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467964,"kb_used_data":20028,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752506756,"statfs":{"total":773094113280,"available":770566918144,"internally_reserved":0,"allocated":20508672,"data_stored":11503033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.690432"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041044+0000","last_change":"2023-12-19T09:19:40.037606+0000","last_active":"2023-12-19T09:19:40.041044+0000","last_peered":"2023-12-19T09:19:40.041044+0000","last_clean":"2023-12-19T09:19:40.041044+0000","last_became_active":"2023-12-19T09:19:40.037279+0000","last_became_peered":"2023-12-19T09:19:40.037279+0000","last_unstale":"2023-12-19T09:19:40.041044+0000","last_undegraded":"2023-12-19T09:19:40.041044+0000","last_fullsized":"2023-12-19T09:19:40.041044+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:33.077567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044043+0000","last_change":"2023-12-19T09:19:40.034805+0000","last_active":"2023-12-19T09:19:40.044043+0000","last_peered":"2023-12-19T09:19:40.044043+0000","last_clean":"2023-12-19T09:19:40.044043+0000","last_became_active":"2023-12-19T09:19:40.034407+0000","last_became_peered":"2023-12-19T09:19:40.034407+0000","last_unstale":"2023-12-19T09:19:40.044043+0000","last_undegraded":"2023-12-19T09:19:40.044043+0000","last_fullsized":"2023-12-19T09:19:40.044043+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:32.144191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.053451+0000","last_change":"2023-12-19T09:19:40.049649+0000","last_active":"2023-12-19T09:19:40.053451+0000","last_peered":"2023-12-19T09:19:40.053451+0000","last_clean":"2023-12-19T09:19:40.053451+0000","last_became_active":"2023-12-19T09:19:40.049289+0000","last_became_peered":"2023-12-19T09:19:40.049289+0000","last_unstale":"2023-12-19T09:19:40.053451+0000","last_undegraded":"2023-12-19T09:19:40.053451+0000","last_fullsized":"2023-12-19T09:19:40.053451+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.608412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039093+0000","last_change":"2023-12-19T09:19:40.032515+0000","last_active":"2023-12-19T09:19:40.039093+0000","last_peered":"2023-12-19T09:19:40.039093+0000","last_clean":"2023-12-19T09:19:40.039093+0000","last_became_active":"2023-12-19T09:19:40.032175+0000","last_became_peered":"2023-12-19T09:19:40.032175+0000","last_unstale":"2023-12-19T09:19:40.039093+0000","last_undegraded":"2023-12-19T09:19:40.039093+0000","last_fullsized":"2023-12-19T09:19:40.039093+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:56.339627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.028015+0000","last_change":"2023-12-19T09:19:40.028015+0000","last_active":"2023-12-19T09:19:40.028015+0000","last_peered":"2023-12-19T09:19:40.028015+0000","last_clean":"2023-12-19T09:19:40.028015+0000","last_became_active":"2023-12-19T09:19:40.027721+0000","last_became_peered":"2023-12-19T09:19:40.027721+0000","last_unstale":"2023-12-19T09:19:40.028015+0000","last_undegraded":"2023-12-19T09:19:40.028015+0000","last_fullsized":"2023-12-19T09:19:40.028015+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:47.920865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044106+0000","last_change":"2023-12-19T09:19:40.036192+0000","last_active":"2023-12-19T09:19:40.044106+0000","last_peered":"2023-12-19T09:19:40.044106+0000","last_clean":"2023-12-19T09:19:40.044106+0000","last_became_active":"2023-12-19T09:19:40.035867+0000","last_became_peered":"2023-12-19T09:19:40.035867+0000","last_unstale":"2023-12-19T09:19:40.044106+0000","last_undegraded":"2023-12-19T09:19:40.044106+0000","last_fullsized":"2023-12-19T09:19:40.044106+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:19.209801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043958+0000","last_change":"2023-12-19T09:19:40.043958+0000","last_active":"2023-12-19T09:19:40.043958+0000","last_peered":"2023-12-19T09:19:40.043958+0000","last_clean":"2023-12-19T09:19:40.043958+0000","last_became_active":"2023-12-19T09:19:40.043504+0000","last_became_peered":"2023-12-19T09:19:40.043504+0000","last_unstale":"2023-12-19T09:19:40.043958+0000","last_undegraded":"2023-12-19T09:19:40.043958+0000","last_fullsized":"2023-12-19T09:19:40.043958+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:57.821705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047541+0000","last_change":"2023-12-19T09:19:40.047541+0000","last_active":"2023-12-19T09:19:40.047541+0000","last_peered":"2023-12-19T09:19:40.047541+0000","last_clean":"2023-12-19T09:19:40.047541+0000","last_became_active":"2023-12-19T09:19:40.047268+0000","last_became_peered":"2023-12-19T09:19:40.047268+0000","last_unstale":"2023-12-19T09:19:40.047541+0000","last_undegraded":"2023-12-19T09:19:40.047541+0000","last_fullsized":"2023-12-19T09:19:40.047541+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:22:14.364823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040335+0000","last_change":"2023-12-19T09:19:40.040335+0000","last_active":"2023-12-19T09:19:40.040335+0000","last_peered":"2023-12-19T09:19:40.040335+0000","last_clean":"2023-12-19T09:19:40.040335+0000","last_became_active":"2023-12-19T09:19:40.039865+0000","last_became_peered":"2023-12-19T09:19:40.039865+0000","last_unstale":"2023-12-19T09:19:40.040335+0000","last_undegraded":"2023-12-19T09:19:40.040335+0000","last_fullsized":"2023-12-19T09:19:40.040335+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:37.765631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039949+0000","last_change":"2023-12-19T09:19:40.039949+0000","last_active":"2023-12-19T09:19:40.039949+0000","last_peered":"2023-12-19T09:19:40.039949+0000","last_clean":"2023-12-19T09:19:40.039949+0000","last_became_active":"2023-12-19T09:19:40.039576+0000","last_became_peered":"2023-12-19T09:19:40.039576+0000","last_unstale":"2023-12-19T09:19:40.039949+0000","last_undegraded":"2023-12-19T09:19:40.039949+0000","last_fullsized":"2023-12-19T09:19:40.039949+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:04.846619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023558+0000","last_change":"2023-12-19T09:19:40.023558+0000","last_active":"2023-12-19T09:19:40.023558+0000","last_peered":"2023-12-19T09:19:40.023558+0000","last_clean":"2023-12-19T09:19:40.023558+0000","last_became_active":"2023-12-19T09:19:40.023217+0000","last_became_peered":"2023-12-19T09:19:40.023217+0000","last_unstale":"2023-12-19T09:19:40.023558+0000","last_undegraded":"2023-12-19T09:19:40.023558+0000","last_fullsized":"2023-12-19T09:19:40.023558+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:32.290878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044414+0000","last_change":"2023-12-19T09:19:40.035154+0000","last_active":"2023-12-19T09:19:40.044414+0000","last_peered":"2023-12-19T09:19:40.044414+0000","last_clean":"2023-12-19T09:19:40.044414+0000","last_became_active":"2023-12-19T09:19:40.034906+0000","last_became_peered":"2023-12-19T09:19:40.034906+0000","last_unstale":"2023-12-19T09:19:40.044414+0000","last_undegraded":"2023-12-19T09:19:40.044414+0000","last_fullsized":"2023-12-19T09:19:40.044414+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:35.969419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047196+0000","last_change":"2023-12-19T09:19:40.035487+0000","last_active":"2023-12-19T09:19:40.047196+0000","last_peered":"2023-12-19T09:19:40.047196+0000","last_clean":"2023-12-19T09:19:40.047196+0000","last_became_active":"2023-12-19T09:19:40.034784+0000","last_became_peered":"2023-12-19T09:19:40.034784+0000","last_unstale":"2023-12-19T09:19:40.047196+0000","last_undegraded":"2023-12-19T09:19:40.047196+0000","last_fullsized":"2023-12-19T09:19:40.047196+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:22.884423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036703+0000","last_change":"2023-12-19T09:19:40.036703+0000","last_active":"2023-12-19T09:19:40.036703+0000","last_peered":"2023-12-19T09:19:40.036703+0000","last_clean":"2023-12-19T09:19:40.036703+0000","last_became_active":"2023-12-19T09:19:40.035566+0000","last_became_peered":"2023-12-19T09:19:40.035566+0000","last_unstale":"2023-12-19T09:19:40.036703+0000","last_undegraded":"2023-12-19T09:19:40.036703+0000","last_fullsized":"2023-12-19T09:19:40.036703+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:22.544550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046158+0000","last_change":"2023-12-19T09:19:40.046158+0000","last_active":"2023-12-19T09:19:40.046158+0000","last_peered":"2023-12-19T09:19:40.046158+0000","last_clean":"2023-12-19T09:19:40.046158+0000","last_became_active":"2023-12-19T09:19:40.045832+0000","last_became_peered":"2023-12-19T09:19:40.045832+0000","last_unstale":"2023-12-19T09:19:40.046158+0000","last_undegraded":"2023-12-19T09:19:40.046158+0000","last_fullsized":"2023-12-19T09:19:40.046158+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:25.397590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039207+0000","last_change":"2023-12-19T09:19:40.032633+0000","last_active":"2023-12-19T09:19:40.039207+0000","last_peered":"2023-12-19T09:19:40.039207+0000","last_clean":"2023-12-19T09:19:40.039207+0000","last_became_active":"2023-12-19T09:19:40.032282+0000","last_became_peered":"2023-12-19T09:19:40.032282+0000","last_unstale":"2023-12-19T09:19:40.039207+0000","last_undegraded":"2023-12-19T09:19:40.039207+0000","last_fullsized":"2023-12-19T09:19:40.039207+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:58.604397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040145+0000","last_change":"2023-12-19T09:19:40.040145+0000","last_active":"2023-12-19T09:19:40.040145+0000","last_peered":"2023-12-19T09:19:40.040145+0000","last_clean":"2023-12-19T09:19:40.040145+0000","last_became_active":"2023-12-19T09:19:40.039546+0000","last_became_peered":"2023-12-19T09:19:40.039546+0000","last_unstale":"2023-12-19T09:19:40.040145+0000","last_undegraded":"2023-12-19T09:19:40.040145+0000","last_fullsized":"2023-12-19T09:19:40.040145+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:43.610486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.042328+0000","last_change":"2023-12-19T09:19:40.042328+0000","last_active":"2023-12-19T09:19:40.042328+0000","last_peered":"2023-12-19T09:19:40.042328+0000","last_clean":"2023-12-19T09:19:40.042328+0000","last_became_active":"2023-12-19T09:19:40.042043+0000","last_became_peered":"2023-12-19T09:19:40.042043+0000","last_unstale":"2023-12-19T09:19:40.042328+0000","last_undegraded":"2023-12-19T09:19:40.042328+0000","last_fullsized":"2023-12-19T09:19:40.042328+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:43.448409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.045694+0000","last_change":"2023-12-19T09:19:40.045694+0000","last_active":"2023-12-19T09:19:40.045694+0000","last_peered":"2023-12-19T09:19:40.045694+0000","last_clean":"2023-12-19T09:19:40.045694+0000","last_became_active":"2023-12-19T09:19:40.045335+0000","last_became_peered":"2023-12-19T09:19:40.045335+0000","last_unstale":"2023-12-19T09:19:40.045694+0000","last_undegraded":"2023-12-19T09:19:40.045694+0000","last_fullsized":"2023-12-19T09:19:40.045694+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:19.049197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039793+0000","last_change":"2023-12-19T09:19:40.039793+0000","last_active":"2023-12-19T09:19:40.039793+0000","last_peered":"2023-12-19T09:19:40.039793+0000","last_clean":"2023-12-19T09:19:40.039793+0000","last_became_active":"2023-12-19T09:19:40.039525+0000","last_became_peered":"2023-12-19T09:19:40.039525+0000","last_unstale":"2023-12-19T09:19:40.039793+0000","last_undegraded":"2023-12-19T09:19:40.039793+0000","last_fullsized":"2023-12-19T09:19:40.039793+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:48:20.971700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034395+0000","last_change":"2023-12-19T09:19:40.028414+0000","last_active":"2023-12-19T09:19:40.034395+0000","last_peered":"2023-12-19T09:19:40.034395+0000","last_clean":"2023-12-19T09:19:40.034395+0000","last_became_active":"2023-12-19T09:19:40.028137+0000","last_became_peered":"2023-12-19T09:19:40.028137+0000","last_unstale":"2023-12-19T09:19:40.034395+0000","last_undegraded":"2023-12-19T09:19:40.034395+0000","last_fullsized":"2023-12-19T09:19:40.034395+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:08:03.116601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.030161+0000","last_change":"2023-12-19T09:19:40.027582+0000","last_active":"2023-12-19T09:19:40.030161+0000","last_peered":"2023-12-19T09:19:40.030161+0000","last_clean":"2023-12-19T09:19:40.030161+0000","last_became_active":"2023-12-19T09:19:40.027271+0000","last_became_peered":"2023-12-19T09:19:40.027271+0000","last_unstale":"2023-12-19T09:19:40.030161+0000","last_undegraded":"2023-12-19T09:19:40.030161+0000","last_fullsized":"2023-12-19T09:19:40.030161+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:16.880935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.053807+0000","last_change":"2023-12-19T09:19:40.053807+0000","last_active":"2023-12-19T09:19:40.053807+0000","last_peered":"2023-12-19T09:19:40.053807+0000","last_clean":"2023-12-19T09:19:40.053807+0000","last_became_active":"2023-12-19T09:19:40.037781+0000","last_became_peered":"2023-12-19T09:19:40.037781+0000","last_unstale":"2023-12-19T09:19:40.053807+0000","last_undegraded":"2023-12-19T09:19:40.053807+0000","last_fullsized":"2023-12-19T09:19:40.053807+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:57.821705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.048339+0000","last_change":"2023-12-19T09:19:40.046917+0000","last_active":"2023-12-19T09:19:40.048339+0000","last_peered":"2023-12-19T09:19:40.048339+0000","last_clean":"2023-12-19T09:19:40.048339+0000","last_became_active":"2023-12-19T09:19:40.046601+0000","last_became_peered":"2023-12-19T09:19:40.046601+0000","last_unstale":"2023-12-19T09:19:40.048339+0000","last_undegraded":"2023-12-19T09:19:40.048339+0000","last_fullsized":"2023-12-19T09:19:40.048339+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:57.208632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.045986+0000","last_change":"2023-12-19T09:19:40.035605+0000","last_active":"2023-12-19T09:19:40.045986+0000","last_peered":"2023-12-19T09:19:40.045986+0000","last_clean":"2023-12-19T09:19:40.045986+0000","last_became_active":"2023-12-19T09:19:40.035288+0000","last_became_peered":"2023-12-19T09:19:40.035288+0000","last_unstale":"2023-12-19T09:19:40.045986+0000","last_undegraded":"2023-12-19T09:19:40.045986+0000","last_fullsized":"2023-12-19T09:19:40.045986+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:18.566066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046150+0000","last_change":"2023-12-19T09:19:40.046150+0000","last_active":"2023-12-19T09:19:40.046150+0000","last_peered":"2023-12-19T09:19:40.046150+0000","last_clean":"2023-12-19T09:19:40.046150+0000","last_became_active":"2023-12-19T09:19:40.033802+0000","last_became_peered":"2023-12-19T09:19:40.033802+0000","last_unstale":"2023-12-19T09:19:40.046150+0000","last_undegraded":"2023-12-19T09:19:40.046150+0000","last_fullsized":"2023-12-19T09:19:40.046150+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:51.885568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.030367+0000","last_change":"2023-12-19T09:19:40.028542+0000","last_active":"2023-12-19T09:19:40.030367+0000","last_peered":"2023-12-19T09:19:40.030367+0000","last_clean":"2023-12-19T09:19:40.030367+0000","last_became_active":"2023-12-19T09:19:40.028253+0000","last_became_peered":"2023-12-19T09:19:40.028253+0000","last_unstale":"2023-12-19T09:19:40.030367+0000","last_undegraded":"2023-12-19T09:19:40.030367+0000","last_fullsized":"2023-12-19T09:19:40.030367+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:15.746493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056226+0000","last_change":"2023-12-19T09:19:40.045421+0000","last_active":"2023-12-19T09:19:40.056226+0000","last_peered":"2023-12-19T09:19:40.056226+0000","last_clean":"2023-12-19T09:19:40.056226+0000","last_became_active":"2023-12-19T09:19:40.045089+0000","last_became_peered":"2023-12-19T09:19:40.045089+0000","last_unstale":"2023-12-19T09:19:40.056226+0000","last_undegraded":"2023-12-19T09:19:40.056226+0000","last_fullsized":"2023-12-19T09:19:40.056226+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:47:44.270632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.048036+0000","last_change":"2023-12-19T09:19:40.044659+0000","last_active":"2023-12-19T09:19:40.048036+0000","last_peered":"2023-12-19T09:19:40.048036+0000","last_clean":"2023-12-19T09:19:40.048036+0000","last_became_active":"2023-12-19T09:19:40.044377+0000","last_became_peered":"2023-12-19T09:19:40.044377+0000","last_unstale":"2023-12-19T09:19:40.048036+0000","last_undegraded":"2023-12-19T09:19:40.048036+0000","last_fullsized":"2023-12-19T09:19:40.048036+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:37:16.042900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.035126+0000","last_change":"2023-12-19T09:19:40.035126+0000","last_active":"2023-12-19T09:19:40.035126+0000","last_peered":"2023-12-19T09:19:40.035126+0000","last_clean":"2023-12-19T09:19:40.035126+0000","last_became_active":"2023-12-19T09:19:40.034810+0000","last_became_peered":"2023-12-19T09:19:40.034810+0000","last_unstale":"2023-12-19T09:19:40.035126+0000","last_undegraded":"2023-12-19T09:19:40.035126+0000","last_fullsized":"2023-12-19T09:19:40.035126+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.858425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038938+0000","last_change":"2023-12-19T09:19:40.038938+0000","last_active":"2023-12-19T09:19:40.038938+0000","last_peered":"2023-12-19T09:19:40.038938+0000","last_clean":"2023-12-19T09:19:40.038938+0000","last_became_active":"2023-12-19T09:19:40.038652+0000","last_became_peered":"2023-12-19T09:19:40.038652+0000","last_unstale":"2023-12-19T09:19:40.038938+0000","last_undegraded":"2023-12-19T09:19:40.038938+0000","last_fullsized":"2023-12-19T09:19:40.038938+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:49.882654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034794+0000","last_change":"2023-12-19T09:19:40.034794+0000","last_active":"2023-12-19T09:19:40.034794+0000","last_peered":"2023-12-19T09:19:40.034794+0000","last_clean":"2023-12-19T09:19:40.034794+0000","last_became_active":"2023-12-19T09:19:40.034358+0000","last_became_peered":"2023-12-19T09:19:40.034358+0000","last_unstale":"2023-12-19T09:19:40.034794+0000","last_undegraded":"2023-12-19T09:19:40.034794+0000","last_fullsized":"2023-12-19T09:19:40.034794+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:54:24.940718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047323+0000","last_change":"2023-12-19T09:19:40.043118+0000","last_active":"2023-12-19T09:19:40.047323+0000","last_peered":"2023-12-19T09:19:40.047323+0000","last_clean":"2023-12-19T09:19:40.047323+0000","last_became_active":"2023-12-19T09:19:40.042476+0000","last_became_peered":"2023-12-19T09:19:40.042476+0000","last_unstale":"2023-12-19T09:19:40.047323+0000","last_undegraded":"2023-12-19T09:19:40.047323+0000","last_fullsized":"2023-12-19T09:19:40.047323+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:18:53.213024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"192'4","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057621+0000","last_change":"2023-12-19T09:19:39.021504+0000","last_active":"2023-12-19T09:19:40.057621+0000","last_peered":"2023-12-19T09:19:40.057621+0000","last_clean":"2023-12-19T09:19:40.057621+0000","last_became_active":"2023-12-19T09:19:39.021203+0000","last_became_peered":"2023-12-19T09:19:39.021203+0000","last_unstale":"2023-12-19T09:19:40.057621+0000","last_undegraded":"2023-12-19T09:19:40.057621+0000","last_fullsized":"2023-12-19T09:19:40.057621+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:48:14.166174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056618+0000","last_change":"2023-12-19T09:19:40.046796+0000","last_active":"2023-12-19T09:19:40.056618+0000","last_peered":"2023-12-19T09:19:40.056618+0000","last_clean":"2023-12-19T09:19:40.056618+0000","last_became_active":"2023-12-19T09:19:40.046426+0000","last_became_peered":"2023-12-19T09:19:40.046426+0000","last_unstale":"2023-12-19T09:19:40.056618+0000","last_undegraded":"2023-12-19T09:19:40.056618+0000","last_fullsized":"2023-12-19T09:19:40.056618+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:46.317043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.011232+0000","last_change":"2023-12-19T09:19:39.013895+0000","last_active":"2023-12-19T09:19:40.011232+0000","last_peered":"2023-12-19T09:19:40.011232+0000","last_clean":"2023-12-19T09:19:40.011232+0000","last_became_active":"2023-12-19T09:19:39.013638+0000","last_became_peered":"2023-12-19T09:19:39.013638+0000","last_unstale":"2023-12-19T09:19:40.011232+0000","last_undegraded":"2023-12-19T09:19:40.011232+0000","last_fullsized":"2023-12-19T09:19:40.011232+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:18.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.035727+0000","last_change":"2023-12-19T09:19:40.035727+0000","last_active":"2023-12-19T09:19:40.035727+0000","last_peered":"2023-12-19T09:19:40.035727+0000","last_clean":"2023-12-19T09:19:40.035727+0000","last_became_active":"2023-12-19T09:19:40.035180+0000","last_became_peered":"2023-12-19T09:19:40.035180+0000","last_unstale":"2023-12-19T09:19:40.035727+0000","last_undegraded":"2023-12-19T09:19:40.035727+0000","last_fullsized":"2023-12-19T09:19:40.035727+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:20.262991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.058759+0000","last_change":"2023-12-19T09:19:39.006533+0000","last_active":"2023-12-19T09:19:40.058759+0000","last_peered":"2023-12-19T09:19:40.058759+0000","last_clean":"2023-12-19T09:19:40.058759+0000","last_became_active":"2023-12-19T09:19:39.006232+0000","last_became_peered":"2023-12-19T09:19:39.006232+0000","last_unstale":"2023-12-19T09:19:40.058759+0000","last_undegraded":"2023-12-19T09:19:40.058759+0000","last_fullsized":"2023-12-19T09:19:40.058759+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:32.062982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044189+0000","last_change":"2023-12-19T09:19:40.040220+0000","last_active":"2023-12-19T09:19:40.044189+0000","last_peered":"2023-12-19T09:19:40.044189+0000","last_clean":"2023-12-19T09:19:40.044189+0000","last_became_active":"2023-12-19T09:19:40.038187+0000","last_became_peered":"2023-12-19T09:19:40.038187+0000","last_unstale":"2023-12-19T09:19:40.044189+0000","last_undegraded":"2023-12-19T09:19:40.044189+0000","last_fullsized":"2023-12-19T09:19:40.044189+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:21.237273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.055044+0000","last_change":"2023-12-19T09:19:39.010088+0000","last_active":"2023-12-19T09:19:40.055044+0000","last_peered":"2023-12-19T09:19:40.055044+0000","last_clean":"2023-12-19T09:19:40.055044+0000","last_became_active":"2023-12-19T09:19:39.009813+0000","last_became_peered":"2023-12-19T09:19:39.009813+0000","last_unstale":"2023-12-19T09:19:40.055044+0000","last_undegraded":"2023-12-19T09:19:40.055044+0000","last_fullsized":"2023-12-19T09:19:40.055044+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:44:30.593357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043044+0000","last_change":"2023-12-19T09:19:40.043044+0000","last_active":"2023-12-19T09:19:40.043044+0000","last_peered":"2023-12-19T09:19:40.043044+0000","last_clean":"2023-12-19T09:19:40.043044+0000","last_became_active":"2023-12-19T09:19:40.042762+0000","last_became_peered":"2023-12-19T09:19:40.042762+0000","last_unstale":"2023-12-19T09:19:40.043044+0000","last_undegraded":"2023-12-19T09:19:40.043044+0000","last_fullsized":"2023-12-19T09:19:40.043044+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:22.544550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.027164+0000","last_change":"2023-12-19T09:19:39.024166+0000","last_active":"2023-12-19T09:19:40.027164+0000","last_peered":"2023-12-19T09:19:40.027164+0000","last_clean":"2023-12-19T09:19:40.027164+0000","last_became_active":"2023-12-19T09:19:39.023823+0000","last_became_peered":"2023-12-19T09:19:39.023823+0000","last_unstale":"2023-12-19T09:19:40.027164+0000","last_undegraded":"2023-12-19T09:19:40.027164+0000","last_fullsized":"2023-12-19T09:19:40.027164+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:35:14.104603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.049777+0000","last_change":"2023-12-19T09:19:40.049777+0000","last_active":"2023-12-19T09:19:40.049777+0000","last_peered":"2023-12-19T09:19:40.049777+0000","last_clean":"2023-12-19T09:19:40.049777+0000","last_became_active":"2023-12-19T09:19:40.048272+0000","last_became_peered":"2023-12-19T09:19:40.048272+0000","last_unstale":"2023-12-19T09:19:40.049777+0000","last_undegraded":"2023-12-19T09:19:40.049777+0000","last_fullsized":"2023-12-19T09:19:40.049777+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:35.103493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.010420+0000","last_change":"2023-12-19T09:19:39.021851+0000","last_active":"2023-12-19T09:19:40.010420+0000","last_peered":"2023-12-19T09:19:40.010420+0000","last_clean":"2023-12-19T09:19:40.010420+0000","last_became_active":"2023-12-19T09:19:39.021322+0000","last_became_peered":"2023-12-19T09:19:39.021322+0000","last_unstale":"2023-12-19T09:19:40.010420+0000","last_undegraded":"2023-12-19T09:19:40.010420+0000","last_fullsized":"2023-12-19T09:19:40.010420+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:08.209309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046223+0000","last_change":"2023-12-19T09:19:40.046223+0000","last_active":"2023-12-19T09:19:40.046223+0000","last_peered":"2023-12-19T09:19:40.046223+0000","last_clean":"2023-12-19T09:19:40.046223+0000","last_became_active":"2023-12-19T09:19:40.035421+0000","last_became_peered":"2023-12-19T09:19:40.035421+0000","last_unstale":"2023-12-19T09:19:40.046223+0000","last_undegraded":"2023-12-19T09:19:40.046223+0000","last_fullsized":"2023-12-19T09:19:40.046223+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:35:31.210260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.022640+0000","last_change":"2023-12-19T09:19:39.021391+0000","last_active":"2023-12-19T09:19:39.022640+0000","last_peered":"2023-12-19T09:19:39.022640+0000","last_clean":"2023-12-19T09:19:39.022640+0000","last_became_active":"2023-12-19T09:19:39.021010+0000","last_became_peered":"2023-12-19T09:19:39.021010+0000","last_unstale":"2023-12-19T09:19:39.022640+0000","last_undegraded":"2023-12-19T09:19:39.022640+0000","last_fullsized":"2023-12-19T09:19:39.022640+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:50.921800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039322+0000","last_change":"2023-12-19T09:19:40.032885+0000","last_active":"2023-12-19T09:19:40.039322+0000","last_peered":"2023-12-19T09:19:40.039322+0000","last_clean":"2023-12-19T09:19:40.039322+0000","last_became_active":"2023-12-19T09:19:40.031417+0000","last_became_peered":"2023-12-19T09:19:40.031417+0000","last_unstale":"2023-12-19T09:19:40.039322+0000","last_undegraded":"2023-12-19T09:19:40.039322+0000","last_fullsized":"2023-12-19T09:19:40.039322+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:00.211869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.014579+0000","last_change":"2023-12-19T09:19:39.011773+0000","last_active":"2023-12-19T09:19:39.014579+0000","last_peered":"2023-12-19T09:19:39.014579+0000","last_clean":"2023-12-19T09:19:39.014579+0000","last_became_active":"2023-12-19T09:19:39.011475+0000","last_became_peered":"2023-12-19T09:19:39.011475+0000","last_unstale":"2023-12-19T09:19:39.014579+0000","last_undegraded":"2023-12-19T09:19:39.014579+0000","last_fullsized":"2023-12-19T09:19:39.014579+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:37.365043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047191+0000","last_change":"2023-12-19T09:19:40.042070+0000","last_active":"2023-12-19T09:19:40.047191+0000","last_peered":"2023-12-19T09:19:40.047191+0000","last_clean":"2023-12-19T09:19:40.047191+0000","last_became_active":"2023-12-19T09:19:40.036700+0000","last_became_peered":"2023-12-19T09:19:40.036700+0000","last_unstale":"2023-12-19T09:19:40.047191+0000","last_undegraded":"2023-12-19T09:19:40.047191+0000","last_fullsized":"2023-12-19T09:19:40.047191+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:47.129114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.033087+0000","last_change":"2023-12-19T09:19:39.018218+0000","last_active":"2023-12-19T09:19:40.033087+0000","last_peered":"2023-12-19T09:19:40.033087+0000","last_clean":"2023-12-19T09:19:40.033087+0000","last_became_active":"2023-12-19T09:19:39.016386+0000","last_became_peered":"2023-12-19T09:19:39.016386+0000","last_unstale":"2023-12-19T09:19:40.033087+0000","last_undegraded":"2023-12-19T09:19:40.033087+0000","last_fullsized":"2023-12-19T09:19:40.033087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:18.791303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.054846+0000","last_change":"2023-12-19T09:19:40.040412+0000","last_active":"2023-12-19T09:19:40.054846+0000","last_peered":"2023-12-19T09:19:40.054846+0000","last_clean":"2023-12-19T09:19:40.054846+0000","last_became_active":"2023-12-19T09:19:40.040135+0000","last_became_peered":"2023-12-19T09:19:40.040135+0000","last_unstale":"2023-12-19T09:19:40.054846+0000","last_undegraded":"2023-12-19T09:19:40.054846+0000","last_fullsized":"2023-12-19T09:19:40.054846+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:47:08.534346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"192'3","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047532+0000","last_change":"2023-12-19T09:19:39.011515+0000","last_active":"2023-12-19T09:19:40.047532+0000","last_peered":"2023-12-19T09:19:40.047532+0000","last_clean":"2023-12-19T09:19:40.047532+0000","last_became_active":"2023-12-19T09:19:39.010744+0000","last_became_peered":"2023-12-19T09:19:39.010744+0000","last_unstale":"2023-12-19T09:19:40.047532+0000","last_undegraded":"2023-12-19T09:19:40.047532+0000","last_fullsized":"2023-12-19T09:19:40.047532+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:50:50.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041169+0000","last_change":"2023-12-19T09:19:40.041169+0000","last_active":"2023-12-19T09:19:40.041169+0000","last_peered":"2023-12-19T09:19:40.041169+0000","last_clean":"2023-12-19T09:19:40.041169+0000","last_became_active":"2023-12-19T09:19:40.032140+0000","last_became_peered":"2023-12-19T09:19:40.032140+0000","last_unstale":"2023-12-19T09:19:40.041169+0000","last_undegraded":"2023-12-19T09:19:40.041169+0000","last_fullsized":"2023-12-19T09:19:40.041169+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:01:15.147747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"192'6","reported_seq":16,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.060156+0000","last_change":"2023-12-19T09:19:39.016988+0000","last_active":"2023-12-19T09:19:40.060156+0000","last_peered":"2023-12-19T09:19:40.060156+0000","last_clean":"2023-12-19T09:19:40.060156+0000","last_became_active":"2023-12-19T09:19:39.016713+0000","last_became_peered":"2023-12-19T09:19:39.016713+0000","last_unstale":"2023-12-19T09:19:40.060156+0000","last_undegraded":"2023-12-19T09:19:40.060156+0000","last_fullsized":"2023-12-19T09:19:40.060156+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:15.866350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.032731+0000","last_change":"2023-12-19T09:19:40.032731+0000","last_active":"2023-12-19T09:19:40.032731+0000","last_peered":"2023-12-19T09:19:40.032731+0000","last_clean":"2023-12-19T09:19:40.032731+0000","last_became_active":"2023-12-19T09:19:40.031948+0000","last_became_peered":"2023-12-19T09:19:40.031948+0000","last_unstale":"2023-12-19T09:19:40.032731+0000","last_undegraded":"2023-12-19T09:19:40.032731+0000","last_fullsized":"2023-12-19T09:19:40.032731+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.715296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.051098+0000","last_change":"2023-12-19T09:19:39.012438+0000","last_active":"2023-12-19T09:19:40.051098+0000","last_peered":"2023-12-19T09:19:40.051098+0000","last_clean":"2023-12-19T09:19:40.051098+0000","last_became_active":"2023-12-19T09:19:39.012166+0000","last_became_peered":"2023-12-19T09:19:39.012166+0000","last_unstale":"2023-12-19T09:19:40.051098+0000","last_undegraded":"2023-12-19T09:19:40.051098+0000","last_fullsized":"2023-12-19T09:19:40.051098+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:40.451458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040480+0000","last_change":"2023-12-19T09:19:40.040480+0000","last_active":"2023-12-19T09:19:40.040480+0000","last_peered":"2023-12-19T09:19:40.040480+0000","last_clean":"2023-12-19T09:19:40.040480+0000","last_became_active":"2023-12-19T09:19:40.040204+0000","last_became_peered":"2023-12-19T09:19:40.040204+0000","last_unstale":"2023-12-19T09:19:40.040480+0000","last_undegraded":"2023-12-19T09:19:40.040480+0000","last_fullsized":"2023-12-19T09:19:40.040480+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:20.262991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023463+0000","last_change":"2023-12-19T09:19:39.017955+0000","last_active":"2023-12-19T09:19:40.023463+0000","last_peered":"2023-12-19T09:19:40.023463+0000","last_clean":"2023-12-19T09:19:40.023463+0000","last_became_active":"2023-12-19T09:19:39.017503+0000","last_became_peered":"2023-12-19T09:19:39.017503+0000","last_unstale":"2023-12-19T09:19:40.023463+0000","last_undegraded":"2023-12-19T09:19:40.023463+0000","last_fullsized":"2023-12-19T09:19:40.023463+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:14.731908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.024864+0000","last_change":"2023-12-19T09:19:40.024864+0000","last_active":"2023-12-19T09:19:40.024864+0000","last_peered":"2023-12-19T09:19:40.024864+0000","last_clean":"2023-12-19T09:19:40.024864+0000","last_became_active":"2023-12-19T09:19:40.024505+0000","last_became_peered":"2023-12-19T09:19:40.024505+0000","last_unstale":"2023-12-19T09:19:40.024864+0000","last_undegraded":"2023-12-19T09:19:40.024864+0000","last_fullsized":"2023-12-19T09:19:40.024864+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:19.591882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.025174+0000","last_change":"2023-12-19T09:19:39.024477+0000","last_active":"2023-12-19T09:19:39.025174+0000","last_peered":"2023-12-19T09:19:39.025174+0000","last_clean":"2023-12-19T09:19:39.025174+0000","last_became_active":"2023-12-19T09:19:39.024192+0000","last_became_peered":"2023-12-19T09:19:39.024192+0000","last_unstale":"2023-12-19T09:19:39.025174+0000","last_undegraded":"2023-12-19T09:19:39.025174+0000","last_fullsized":"2023-12-19T09:19:39.025174+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:55.325042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050013+0000","last_change":"2023-12-19T09:19:40.050013+0000","last_active":"2023-12-19T09:19:40.050013+0000","last_peered":"2023-12-19T09:19:40.050013+0000","last_clean":"2023-12-19T09:19:40.050013+0000","last_became_active":"2023-12-19T09:19:40.048302+0000","last_became_peered":"2023-12-19T09:19:40.048302+0000","last_unstale":"2023-12-19T09:19:40.050013+0000","last_undegraded":"2023-12-19T09:19:40.050013+0000","last_fullsized":"2023-12-19T09:19:40.050013+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:12:00.735401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"192'8","reported_seq":17,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050316+0000","last_change":"2023-12-19T09:19:39.011136+0000","last_active":"2023-12-19T09:19:40.050316+0000","last_peered":"2023-12-19T09:19:40.050316+0000","last_clean":"2023-12-19T09:19:40.050316+0000","last_became_active":"2023-12-19T09:19:39.010624+0000","last_became_peered":"2023-12-19T09:19:39.010624+0000","last_unstale":"2023-12-19T09:19:40.050316+0000","last_undegraded":"2023-12-19T09:19:40.050316+0000","last_fullsized":"2023-12-19T09:19:40.050316+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:22.244327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047476+0000","last_change":"2023-12-19T09:19:40.047476+0000","last_active":"2023-12-19T09:19:40.047476+0000","last_peered":"2023-12-19T09:19:40.047476+0000","last_clean":"2023-12-19T09:19:40.047476+0000","last_became_active":"2023-12-19T09:19:40.047229+0000","last_became_peered":"2023-12-19T09:19:40.047229+0000","last_unstale":"2023-12-19T09:19:40.047476+0000","last_undegraded":"2023-12-19T09:19:40.047476+0000","last_fullsized":"2023-12-19T09:19:40.047476+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:47.920865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"192'1","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057696+0000","last_change":"2023-12-19T09:19:39.018956+0000","last_active":"2023-12-19T09:19:40.057696+0000","last_peered":"2023-12-19T09:19:40.057696+0000","last_clean":"2023-12-19T09:19:40.057696+0000","last_became_active":"2023-12-19T09:19:39.016088+0000","last_became_peered":"2023-12-19T09:19:39.016088+0000","last_unstale":"2023-12-19T09:19:40.057696+0000","last_undegraded":"2023-12-19T09:19:40.057696+0000","last_fullsized":"2023-12-19T09:19:40.057696+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:31:21.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036597+0000","last_change":"2023-12-19T09:19:40.036597+0000","last_active":"2023-12-19T09:19:40.036597+0000","last_peered":"2023-12-19T09:19:40.036597+0000","last_clean":"2023-12-19T09:19:40.036597+0000","last_became_active":"2023-12-19T09:19:40.036322+0000","last_became_peered":"2023-12-19T09:19:40.036322+0000","last_unstale":"2023-12-19T09:19:40.036597+0000","last_undegraded":"2023-12-19T09:19:40.036597+0000","last_fullsized":"2023-12-19T09:19:40.036597+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:29.533414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057909+0000","last_change":"2023-12-19T09:19:39.015824+0000","last_active":"2023-12-19T09:19:40.057909+0000","last_peered":"2023-12-19T09:19:40.057909+0000","last_clean":"2023-12-19T09:19:40.057909+0000","last_became_active":"2023-12-19T09:19:39.014088+0000","last_became_peered":"2023-12-19T09:19:39.014088+0000","last_unstale":"2023-12-19T09:19:40.057909+0000","last_undegraded":"2023-12-19T09:19:40.057909+0000","last_fullsized":"2023-12-19T09:19:40.057909+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:38:15.427681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.028486+0000","last_change":"2023-12-19T09:19:40.028486+0000","last_active":"2023-12-19T09:19:40.028486+0000","last_peered":"2023-12-19T09:19:40.028486+0000","last_clean":"2023-12-19T09:19:40.028486+0000","last_became_active":"2023-12-19T09:19:40.028108+0000","last_became_peered":"2023-12-19T09:19:40.028108+0000","last_unstale":"2023-12-19T09:19:40.028486+0000","last_undegraded":"2023-12-19T09:19:40.028486+0000","last_fullsized":"2023-12-19T09:19:40.028486+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:13:06.999965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.011148+0000","last_change":"2023-12-19T09:19:39.008233+0000","last_active":"2023-12-19T09:19:40.011148+0000","last_peered":"2023-12-19T09:19:40.011148+0000","last_clean":"2023-12-19T09:19:40.011148+0000","last_became_active":"2023-12-19T09:19:39.007929+0000","last_became_peered":"2023-12-19T09:19:39.007929+0000","last_unstale":"2023-12-19T09:19:40.011148+0000","last_undegraded":"2023-12-19T09:19:40.011148+0000","last_fullsized":"2023-12-19T09:19:40.011148+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:13:05.985380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023538+0000","last_change":"2023-12-19T09:19:40.023538+0000","last_active":"2023-12-19T09:19:40.023538+0000","last_peered":"2023-12-19T09:19:40.023538+0000","last_clean":"2023-12-19T09:19:40.023538+0000","last_became_active":"2023-12-19T09:19:40.023109+0000","last_became_peered":"2023-12-19T09:19:40.023109+0000","last_unstale":"2023-12-19T09:19:40.023538+0000","last_undegraded":"2023-12-19T09:19:40.023538+0000","last_fullsized":"2023-12-19T09:19:40.023538+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:31.467181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.027050+0000","last_change":"2023-12-19T09:19:39.026198+0000","last_active":"2023-12-19T09:19:39.027050+0000","last_peered":"2023-12-19T09:19:39.027050+0000","last_clean":"2023-12-19T09:19:39.027050+0000","last_became_active":"2023-12-19T09:19:39.025847+0000","last_became_peered":"2023-12-19T09:19:39.025847+0000","last_unstale":"2023-12-19T09:19:39.027050+0000","last_undegraded":"2023-12-19T09:19:39.027050+0000","last_fullsized":"2023-12-19T09:19:39.027050+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:08.209309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038869+0000","last_change":"2023-12-19T09:19:40.033961+0000","last_active":"2023-12-19T09:19:40.038869+0000","last_peered":"2023-12-19T09:19:40.038869+0000","last_clean":"2023-12-19T09:19:40.038869+0000","last_became_active":"2023-12-19T09:19:40.033421+0000","last_became_peered":"2023-12-19T09:19:40.033421+0000","last_unstale":"2023-12-19T09:19:40.038869+0000","last_undegraded":"2023-12-19T09:19:40.038869+0000","last_fullsized":"2023-12-19T09:19:40.038869+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:37.371893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.055376+0000","last_change":"2023-12-19T09:19:39.000781+0000","last_active":"2023-12-19T09:19:40.055376+0000","last_peered":"2023-12-19T09:19:40.055376+0000","last_clean":"2023-12-19T09:19:40.055376+0000","last_became_active":"2023-12-19T09:19:39.000456+0000","last_became_peered":"2023-12-19T09:19:39.000456+0000","last_unstale":"2023-12-19T09:19:40.055376+0000","last_undegraded":"2023-12-19T09:19:40.055376+0000","last_fullsized":"2023-12-19T09:19:40.055376+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:18.577297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.051430+0000","last_change":"2023-12-19T09:19:40.051430+0000","last_active":"2023-12-19T09:19:40.051430+0000","last_peered":"2023-12-19T09:19:40.051430+0000","last_clean":"2023-12-19T09:19:40.051430+0000","last_became_active":"2023-12-19T09:19:40.045606+0000","last_became_peered":"2023-12-19T09:19:40.045606+0000","last_unstale":"2023-12-19T09:19:40.051430+0000","last_undegraded":"2023-12-19T09:19:40.051430+0000","last_fullsized":"2023-12-19T09:19:40.051430+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:37.765631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.015732+0000","last_change":"2023-12-19T09:19:39.014180+0000","last_active":"2023-12-19T09:19:39.015732+0000","last_peered":"2023-12-19T09:19:39.015732+0000","last_clean":"2023-12-19T09:19:39.015732+0000","last_became_active":"2023-12-19T09:19:39.013496+0000","last_became_peered":"2023-12-19T09:19:39.013496+0000","last_unstale":"2023-12-19T09:19:39.015732+0000","last_undegraded":"2023-12-19T09:19:39.015732+0000","last_fullsized":"2023-12-19T09:19:39.015732+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:28:59.197284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.042893+0000","last_change":"2023-12-19T09:19:40.042893+0000","last_active":"2023-12-19T09:19:40.042893+0000","last_peered":"2023-12-19T09:19:40.042893+0000","last_clean":"2023-12-19T09:19:40.042893+0000","last_became_active":"2023-12-19T09:19:40.042561+0000","last_became_peered":"2023-12-19T09:19:40.042561+0000","last_unstale":"2023-12-19T09:19:40.042893+0000","last_undegraded":"2023-12-19T09:19:40.042893+0000","last_fullsized":"2023-12-19T09:19:40.042893+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:43.610486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.012289+0000","last_change":"2023-12-19T09:19:39.009686+0000","last_active":"2023-12-19T09:19:39.012289+0000","last_peered":"2023-12-19T09:19:39.012289+0000","last_clean":"2023-12-19T09:19:39.012289+0000","last_became_active":"2023-12-19T09:19:39.009388+0000","last_became_peered":"2023-12-19T09:19:39.009388+0000","last_unstale":"2023-12-19T09:19:39.012289+0000","last_undegraded":"2023-12-19T09:19:39.012289+0000","last_fullsized":"2023-12-19T09:19:39.012289+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:32.945772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057069+0000","last_change":"2023-12-19T09:19:40.056428+0000","last_active":"2023-12-19T09:19:40.057069+0000","last_peered":"2023-12-19T09:19:40.057069+0000","last_clean":"2023-12-19T09:19:40.057069+0000","last_became_active":"2023-12-19T09:19:40.056163+0000","last_became_peered":"2023-12-19T09:19:40.056163+0000","last_unstale":"2023-12-19T09:19:40.057069+0000","last_undegraded":"2023-12-19T09:19:40.057069+0000","last_fullsized":"2023-12-19T09:19:40.057069+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:49.619422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.022374+0000","last_change":"2023-12-19T09:19:39.006915+0000","last_active":"2023-12-19T09:19:40.022374+0000","last_peered":"2023-12-19T09:19:40.022374+0000","last_clean":"2023-12-19T09:19:40.022374+0000","last_became_active":"2023-12-19T09:19:39.006628+0000","last_became_peered":"2023-12-19T09:19:39.006628+0000","last_unstale":"2023-12-19T09:19:40.022374+0000","last_undegraded":"2023-12-19T09:19:40.022374+0000","last_fullsized":"2023-12-19T09:19:40.022374+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:02.010995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038404+0000","last_change":"2023-12-19T09:19:40.038404+0000","last_active":"2023-12-19T09:19:40.038404+0000","last_peered":"2023-12-19T09:19:40.038404+0000","last_clean":"2023-12-19T09:19:40.038404+0000","last_became_active":"2023-12-19T09:19:40.037814+0000","last_became_peered":"2023-12-19T09:19:40.037814+0000","last_unstale":"2023-12-19T09:19:40.038404+0000","last_undegraded":"2023-12-19T09:19:40.038404+0000","last_fullsized":"2023-12-19T09:19:40.038404+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:56.085814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041498+0000","last_change":"2023-12-19T09:19:39.013361+0000","last_active":"2023-12-19T09:19:40.041498+0000","last_peered":"2023-12-19T09:19:40.041498+0000","last_clean":"2023-12-19T09:19:40.041498+0000","last_became_active":"2023-12-19T09:19:39.013103+0000","last_became_peered":"2023-12-19T09:19:39.013103+0000","last_unstale":"2023-12-19T09:19:40.041498+0000","last_undegraded":"2023-12-19T09:19:40.041498+0000","last_fullsized":"2023-12-19T09:19:40.041498+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:48.868069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043859+0000","last_change":"2023-12-19T09:19:40.043859+0000","last_active":"2023-12-19T09:19:40.043859+0000","last_peered":"2023-12-19T09:19:40.043859+0000","last_clean":"2023-12-19T09:19:40.043859+0000","last_became_active":"2023-12-19T09:19:40.043146+0000","last_became_peered":"2023-12-19T09:19:40.043146+0000","last_unstale":"2023-12-19T09:19:40.043859+0000","last_undegraded":"2023-12-19T09:19:40.043859+0000","last_fullsized":"2023-12-19T09:19:40.043859+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:20:07.813243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.031964+0000","last_change":"2023-12-19T09:19:39.016260+0000","last_active":"2023-12-19T09:19:40.031964+0000","last_peered":"2023-12-19T09:19:40.031964+0000","last_clean":"2023-12-19T09:19:40.031964+0000","last_became_active":"2023-12-19T09:19:39.015859+0000","last_became_peered":"2023-12-19T09:19:39.015859+0000","last_unstale":"2023-12-19T09:19:40.031964+0000","last_undegraded":"2023-12-19T09:19:40.031964+0000","last_fullsized":"2023-12-19T09:19:40.031964+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:18.195216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036730+0000","last_change":"2023-12-19T09:19:40.036730+0000","last_active":"2023-12-19T09:19:40.036730+0000","last_peered":"2023-12-19T09:19:40.036730+0000","last_clean":"2023-12-19T09:19:40.036730+0000","last_became_active":"2023-12-19T09:19:40.036434+0000","last_became_peered":"2023-12-19T09:19:40.036434+0000","last_unstale":"2023-12-19T09:19:40.036730+0000","last_undegraded":"2023-12-19T09:19:40.036730+0000","last_fullsized":"2023-12-19T09:19:40.036730+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:48.021826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034187+0000","last_change":"2023-12-19T09:19:39.005750+0000","last_active":"2023-12-19T09:19:40.034187+0000","last_peered":"2023-12-19T09:19:40.034187+0000","last_clean":"2023-12-19T09:19:40.034187+0000","last_became_active":"2023-12-19T09:19:39.005370+0000","last_became_peered":"2023-12-19T09:19:39.005370+0000","last_unstale":"2023-12-19T09:19:40.034187+0000","last_undegraded":"2023-12-19T09:19:40.034187+0000","last_fullsized":"2023-12-19T09:19:40.034187+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:31.583174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050171+0000","last_change":"2023-12-19T09:19:40.050171+0000","last_active":"2023-12-19T09:19:40.050171+0000","last_peered":"2023-12-19T09:19:40.050171+0000","last_clean":"2023-12-19T09:19:40.050171+0000","last_became_active":"2023-12-19T09:19:40.049868+0000","last_became_peered":"2023-12-19T09:19:40.049868+0000","last_unstale":"2023-12-19T09:19:40.050171+0000","last_undegraded":"2023-12-19T09:19:40.050171+0000","last_fullsized":"2023-12-19T09:19:40.050171+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:57:42.144971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.017934+0000","last_change":"2023-12-19T09:19:39.014451+0000","last_active":"2023-12-19T09:19:39.017934+0000","last_peered":"2023-12-19T09:19:39.017934+0000","last_clean":"2023-12-19T09:19:39.017934+0000","last_became_active":"2023-12-19T09:19:39.014153+0000","last_became_peered":"2023-12-19T09:19:39.014153+0000","last_unstale":"2023-12-19T09:19:39.017934+0000","last_undegraded":"2023-12-19T09:19:39.017934+0000","last_fullsized":"2023-12-19T09:19:39.017934+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:56:59.546403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039694+0000","last_change":"2023-12-19T09:19:40.035881+0000","last_active":"2023-12-19T09:19:40.039694+0000","last_peered":"2023-12-19T09:19:40.039694+0000","last_clean":"2023-12-19T09:19:40.039694+0000","last_became_active":"2023-12-19T09:19:40.035589+0000","last_became_peered":"2023-12-19T09:19:40.035589+0000","last_unstale":"2023-12-19T09:19:40.039694+0000","last_undegraded":"2023-12-19T09:19:40.039694+0000","last_fullsized":"2023-12-19T09:19:40.039694+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:03.698560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.027757+0000","last_change":"2023-12-19T09:19:39.020965+0000","last_active":"2023-12-19T09:19:40.027757+0000","last_peered":"2023-12-19T09:19:40.027757+0000","last_clean":"2023-12-19T09:19:40.027757+0000","last_became_active":"2023-12-19T09:19:39.020693+0000","last_became_peered":"2023-12-19T09:19:39.020693+0000","last_unstale":"2023-12-19T09:19:40.027757+0000","last_undegraded":"2023-12-19T09:19:40.027757+0000","last_fullsized":"2023-12-19T09:19:40.027757+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:14.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047161+0000","last_change":"2023-12-19T09:19:40.041025+0000","last_active":"2023-12-19T09:19:40.047161+0000","last_peered":"2023-12-19T09:19:40.047161+0000","last_clean":"2023-12-19T09:19:40.047161+0000","last_became_active":"2023-12-19T09:19:40.040749+0000","last_became_peered":"2023-12-19T09:19:40.040749+0000","last_unstale":"2023-12-19T09:19:40.047161+0000","last_undegraded":"2023-12-19T09:19:40.047161+0000","last_fullsized":"2023-12-19T09:19:40.047161+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:04:55.353273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023565+0000","last_change":"2023-12-19T09:19:39.016542+0000","last_active":"2023-12-19T09:19:40.023565+0000","last_peered":"2023-12-19T09:19:40.023565+0000","last_clean":"2023-12-19T09:19:40.023565+0000","last_became_active":"2023-12-19T09:19:39.016215+0000","last_became_peered":"2023-12-19T09:19:39.016215+0000","last_unstale":"2023-12-19T09:19:40.023565+0000","last_undegraded":"2023-12-19T09:19:40.023565+0000","last_fullsized":"2023-12-19T09:19:40.023565+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:46.114529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056215+0000","last_change":"2023-12-19T09:19:40.045357+0000","last_active":"2023-12-19T09:19:40.056215+0000","last_peered":"2023-12-19T09:19:40.056215+0000","last_clean":"2023-12-19T09:19:40.056215+0000","last_became_active":"2023-12-19T09:19:40.045074+0000","last_became_peered":"2023-12-19T09:19:40.045074+0000","last_unstale":"2023-12-19T09:19:40.056215+0000","last_undegraded":"2023-12-19T09:19:40.056215+0000","last_fullsized":"2023-12-19T09:19:40.056215+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:22:24.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.010111+0000","last_change":"2023-12-19T09:19:39.006154+0000","last_active":"2023-12-19T09:19:39.010111+0000","last_peered":"2023-12-19T09:19:39.010111+0000","last_clean":"2023-12-19T09:19:39.010111+0000","last_became_active":"2023-12-19T09:19:39.004886+0000","last_became_peered":"2023-12-19T09:19:39.004886+0000","last_unstale":"2023-12-19T09:19:39.010111+0000","last_undegraded":"2023-12-19T09:19:39.010111+0000","last_fullsized":"2023-12-19T09:19:39.010111+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.593827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043748+0000","last_change":"2023-12-19T09:19:40.043748+0000","last_active":"2023-12-19T09:19:40.043748+0000","last_peered":"2023-12-19T09:19:40.043748+0000","last_clean":"2023-12-19T09:19:40.043748+0000","last_became_active":"2023-12-19T09:19:40.043425+0000","last_became_peered":"2023-12-19T09:19:40.043425+0000","last_unstale":"2023-12-19T09:19:40.043748+0000","last_undegraded":"2023-12-19T09:19:40.043748+0000","last_fullsized":"2023-12-19T09:19:40.043748+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:03.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.024029+0000","last_change":"2023-12-19T09:19:39.016725+0000","last_active":"2023-12-19T09:19:39.024029+0000","last_peered":"2023-12-19T09:19:39.024029+0000","last_clean":"2023-12-19T09:19:39.024029+0000","last_became_active":"2023-12-19T09:19:39.009371+0000","last_became_peered":"2023-12-19T09:19:39.009371+0000","last_unstale":"2023-12-19T09:19:39.024029+0000","last_undegraded":"2023-12-19T09:19:39.024029+0000","last_fullsized":"2023-12-19T09:19:39.024029+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:55.071229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047227+0000","last_change":"2023-12-19T09:19:40.042064+0000","last_active":"2023-12-19T09:19:40.047227+0000","last_peered":"2023-12-19T09:19:40.047227+0000","last_clean":"2023-12-19T09:19:40.047227+0000","last_became_active":"2023-12-19T09:19:40.040686+0000","last_became_peered":"2023-12-19T09:19:40.040686+0000","last_unstale":"2023-12-19T09:19:40.047227+0000","last_undegraded":"2023-12-19T09:19:40.047227+0000","last_fullsized":"2023-12-19T09:19:40.047227+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:30:06.729521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.026644+0000","last_change":"2023-12-19T09:19:39.017482+0000","last_active":"2023-12-19T09:19:40.026644+0000","last_peered":"2023-12-19T09:19:40.026644+0000","last_clean":"2023-12-19T09:19:40.026644+0000","last_became_active":"2023-12-19T09:19:39.016213+0000","last_became_peered":"2023-12-19T09:19:39.016213+0000","last_unstale":"2023-12-19T09:19:40.026644+0000","last_undegraded":"2023-12-19T09:19:40.026644+0000","last_fullsized":"2023-12-19T09:19:40.026644+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:18:52.198439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"182'326","reported_seq":1046,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.022492+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:19:40.022492+0000","last_peered":"2023-12-19T09:19:40.022492+0000","last_clean":"2023-12-19T09:19:40.022492+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:19:40.022492+0000","last_undegraded":"2023-12-19T09:19:40.022492+0000","last_fullsized":"2023-12-19T09:19:40.022492+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304312,"kb_used_data":2928,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067528,"statfs":{"total":96636764160,"available":96325148672,"internally_reserved":0,"allocated":2998272,"data_stored":1878409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2829999999999999,"15min":1.286},"min":{"1min":0.84999999999999998,"5min":0.72699999999999998,"15min":0.52600000000000002},"max":{"1min":1.5960000000000001,"5min":1.8360000000000001,"15min":2.0470000000000002},"last":1.444},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.276,"15min":1.296},"min":{"1min":1.0289999999999999,"5min":0.871,"15min":0.55700000000000005},"max":{"1min":1.591,"5min":1.6299999999999999,"15min":1.712},"last":1.3160000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1799999999999999,"15min":1.167},"min":{"1min":0.94299999999999995,"5min":0.747,"15min":0.747},"max":{"1min":1.385,"5min":1.615,"15min":1.698},"last":1.117},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2130000000000001,"15min":1.2090000000000001},"min":{"1min":0.90900000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.474,"5min":1.671,"15min":1.796},"last":1.3300000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1819999999999999,"15min":1.1779999999999999},"min":{"1min":0.94399999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.3360000000000001,"5min":1.681,"15min":1.681},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2909999999999999,"15min":1.2849999999999999},"min":{"1min":0.97599999999999998,"5min":0.84599999999999997,"15min":0.71999999999999997},"max":{"1min":1.681,"5min":1.754,"15min":2.1400000000000001},"last":1.1830000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3029999999999999,"15min":1.3049999999999999},"min":{"1min":0.93200000000000005,"5min":0.90600000000000003,"15min":0.70599999999999996},"max":{"1min":1.522,"5min":1.6799999999999999,"15min":1.9059999999999999},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3740000000000001,"15min":1.379},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.6120000000000001,"5min":1.8480000000000001,"15min":1.8700000000000001},"last":1.498}]},{"osd":4,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.208,"15min":1.244},"min":{"1min":0.84699999999999998,"5min":0.78800000000000003,"15min":0.77000000000000002},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.95},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.2230000000000001,"15min":1.238},"min":{"1min":0.98099999999999998,"5min":0.70399999999999996,"15min":0.59899999999999998},"max":{"1min":1.5489999999999999,"5min":1.7609999999999999,"15min":1.788},"last":1.3680000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.268,"15min":1.294},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":1.9339999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.3899999999999999,"15min":1.4159999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.81899999999999995},"max":{"1min":1.804,"5min":1.901,"15min":1.901},"last":1.5489999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.393,"15min":1.411},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.732,"5min":1.9950000000000001,"15min":2.0750000000000002},"last":1.407},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.4039999999999999,"15min":1.3959999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.752},"max":{"1min":1.734,"5min":1.8979999999999999,"15min":2.238},"last":1.268}]}]},{"osd":6,"up_from":13,"seq":55834575948,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311924,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059916,"statfs":{"total":96636764160,"available":96317353984,"internally_reserved":0,"allocated":2404352,"data_stored":1288041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1579999999999999,"15min":1.139},"min":{"1min":0.76300000000000001,"5min":0.73499999999999999,"15min":0.65300000000000002},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.784},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.153,"15min":1.167},"min":{"1min":0.90900000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.675,"5min":1.6850000000000001,"15min":1.946},"last":0.92100000000000004}]},{"osd":1,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.169,"15min":1.2050000000000001},"min":{"1min":0.72899999999999998,"5min":0.67600000000000005,"15min":0.63800000000000001},"max":{"1min":1.5980000000000001,"5min":1.6459999999999999,"15min":2.3439999999999999},"last":1.159},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.218,"15min":1.2030000000000001},"min":{"1min":0.83699999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.385}]},{"osd":2,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.274,"15min":1.2649999999999999},"min":{"1min":0.93899999999999995,"5min":0.86199999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.425},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2529999999999999,"15min":1.2549999999999999},"min":{"1min":0.92300000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.8100000000000001},"last":1.077}]},{"osd":3,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2649999999999999,"15min":1.298},"min":{"1min":0.92200000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.6100000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.331,"15min":1.331},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.68999999999999995},"max":{"1min":1.712,"5min":1.7130000000000001,"15min":1.8220000000000001},"last":1.6240000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.218,"15min":1.2250000000000001},"min":{"1min":0.84299999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.649,"5min":1.7609999999999999,"15min":1.893},"last":1.573},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":1.0509999999999999,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7929999999999999,"5min":1.831,"15min":2.1139999999999999},"last":1.5149999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":1.006,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.694,"5min":1.7749999999999999,"15min":1.921},"last":1.538},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":0.81000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6719999999999999,"5min":1.748,"15min":1.8959999999999999},"last":1.667}]},{"osd":7,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.375,"15min":1.387},"min":{"1min":0.94399999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.728,"5min":1.903,"15min":1.962},"last":1.226},{"interface":"front","average":{"1min":1.454,"5min":1.444,"15min":1.4510000000000001},"min":{"1min":1.0609999999999999,"5min":0.79100000000000004,"15min":0.72599999999999998},"max":{"1min":1.661,"5min":1.8939999999999999,"15min":2.2410000000000001},"last":1.734}]}]},{"osd":5,"up_from":13,"seq":55834575948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307836,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064004,"statfs":{"total":96636764160,"available":96321540096,"internally_reserved":0,"allocated":2412544,"data_stored":1288165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3169999999999999,"15min":1.3180000000000001},"min":{"1min":0.94499999999999995,"5min":0.80500000000000005,"15min":0.50700000000000001},"max":{"1min":1.8160000000000001,"5min":2.044,"15min":2.044},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3420000000000001,"15min":1.3440000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.58199999999999996},"max":{"1min":1.671,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.5780000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.161,"15min":1.1739999999999999},"min":{"1min":0.80900000000000005,"5min":0.78800000000000003,"15min":0.76200000000000001},"max":{"1min":1.454,"5min":1.5569999999999999,"15min":1.6479999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1859999999999999,"15min":1.208},"min":{"1min":0.84799999999999998,"5min":0.74299999999999999,"15min":0.58099999999999996},"max":{"1min":1.4139999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.004}]},{"osd":2,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.76800000000000002},"max":{"1min":1.5209999999999999,"5min":1.782,"15min":1.782},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4159999999999999,"5min":1.583,"15min":1.6180000000000001},"last":1.24}]},{"osd":3,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3029999999999999,"15min":1.304},"min":{"1min":0.91700000000000004,"5min":0.872,"15min":0.63700000000000001},"max":{"1min":1.625,"5min":1.7529999999999999,"15min":1.8180000000000001},"last":1.431},{"interface":"front","average":{"1min":1.357,"5min":1.3129999999999999,"15min":1.325},"min":{"1min":1.012,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.601,"5min":1.962,"15min":1.962},"last":1.4690000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.218,"15min":1.2330000000000001},"min":{"1min":0.79800000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.391},{"interface":"front","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3},"min":{"1min":0.88800000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7390000000000001,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.827}]},{"osd":6,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.72799999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.617,"5min":1.746,"15min":1.8720000000000001},"last":1.677},{"interface":"front","average":{"1min":1.355,"5min":1.3600000000000001,"15min":1.3740000000000001},"min":{"1min":0.90400000000000003,"5min":0.76500000000000001,"15min":0.68999999999999995},"max":{"1min":1.7430000000000001,"5min":1.8169999999999999,"15min":1.8740000000000001},"last":1.619}]},{"osd":7,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.167,"5min":0.92000000000000004,"15min":0.84199999999999997},"max":{"1min":1.8720000000000001,"5min":1.8859999999999999,"15min":2.153},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.454,"15min":1.4730000000000001},"min":{"1min":1.1419999999999999,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8089999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.978}]}]},{"osd":4,"up_from":13,"seq":55834575948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311952,"kb_used_data":2376,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059888,"statfs":{"total":96636764160,"available":96317325312,"internally_reserved":0,"allocated":2433024,"data_stored":1295961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.222,"15min":1.2390000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5860000000000001,"5min":1.645,"15min":1.796},"last":3.4420000000000002},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3500000000000001,"15min":1.3540000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.768,"5min":1.768,"15min":1.782},"last":3.4260000000000002}]},{"osd":1,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.24,"15min":1.208},"min":{"1min":0.91800000000000004,"5min":0.84099999999999997,"15min":0.70799999999999996},"max":{"1min":1.6339999999999999,"5min":1.758,"15min":1.758},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1699999999999999,"15min":1.1930000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.66600000000000004},"max":{"1min":1.456,"5min":1.5549999999999999,"15min":1.7050000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.5800000000000001,"5min":1.833,"15min":1.833},"last":1.8839999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2909999999999999,"15min":1.304},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85099999999999998},"max":{"1min":1.7110000000000001,"5min":1.925,"15min":1.925},"last":1.5589999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.325,"15min":1.3129999999999999},"min":{"1min":0.92500000000000004,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.6619999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.643},{"interface":"front","average":{"1min":1.405,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.671,"5min":2.4300000000000002,"15min":2.4300000000000002},"last":1.7150000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2270000000000001,"15min":1.2689999999999999},"min":{"1min":0.88500000000000001,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.796,"5min":1.796,"15min":2.0179999999999998},"last":1.528},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2470000000000001,"15min":1.2629999999999999},"min":{"1min":0.91500000000000004,"5min":0.83999999999999997,"15min":0.68700000000000006},"max":{"1min":1.571,"5min":1.879,"15min":1.879},"last":1.73}]},{"osd":6,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.286,"15min":1.2749999999999999},"min":{"1min":0.83799999999999997,"5min":0.753,"15min":0.753},"max":{"1min":1.6499999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.7609999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.4259999999999999,"15min":1.425},"min":{"1min":1.0720000000000001,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.9379999999999999},"last":1.478}]},{"osd":7,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.369,"15min":1.367},"min":{"1min":0.94799999999999995,"5min":0.89000000000000001,"15min":0.86599999999999999},"max":{"1min":1.762,"5min":1.7749999999999999,"15min":1.9730000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4630000000000001,"15min":1.4570000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91600000000000004},"max":{"1min":1.752,"5min":1.9219999999999999,"15min":1.9990000000000001},"last":1.847}]}]},{"osd":3,"up_from":13,"seq":55834575948,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311956,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059884,"statfs":{"total":96636764160,"available":96317321216,"internally_reserved":0,"allocated":2437120,"data_stored":1297337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88,"15min":0.875},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":1.581,"15min":1.581},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.01,"15min":0.99299999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5249999999999999,"5min":2.117,"15min":2.117},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.91700000000000004,"15min":0.93700000000000006},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1539999999999999,"5min":2.331,"15min":2.331},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.077,"15min":1.0620000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":0.85299999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.139,"15min":1.127},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.7130000000000001,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.181,"15min":1.1910000000000001},"min":{"1min":0.875,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.419,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.93799999999999994}]},{"osd":4,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.3480000000000001,"15min":1.3129999999999999},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.70199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.204},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.2809999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.0029999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3129999999999999,"15min":1.3109999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.3919999999999999,"15min":1.3859999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2470000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.421,"15min":1.423},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.157},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4399999999999999,"15min":1.425},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.825,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.355}]},{"osd":7,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.53,"15min":1.5},"min":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":2.2749999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.47,"15min":1.452},"min":{"1min":0.96499999999999997,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.1389999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.05}]}]},{"osd":2,"up_from":13,"seq":55834575948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307828,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064012,"statfs":{"total":96636764160,"available":96321548288,"internally_reserved":0,"allocated":2404352,"data_stored":1288041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93600000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.58899999999999997,"15min":0.55100000000000005},"max":{"1min":1.3300000000000001,"5min":1.4670000000000001,"15min":1.5049999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.1339999999999999,"15min":1.117},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.71899999999999997},"max":{"1min":1.377,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97599999999999998,"15min":0.995},"min":{"1min":0.747,"5min":0.73999999999999999,"15min":0.59699999999999998},"max":{"1min":1.569,"5min":1.569,"15min":1.7609999999999999},"last":0.623},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.083,"15min":1.0549999999999999},"min":{"1min":0.75,"5min":0.72099999999999997,"15min":0.63800000000000001},"max":{"1min":1.5089999999999999,"5min":1.768,"15min":1.768},"last":0.94599999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.0169999999999999,"15min":0.98299999999999998},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.54100000000000004},"max":{"1min":1.3120000000000001,"5min":1.891,"15min":1.891},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.171,"5min":1.1899999999999999,"15min":1.1659999999999999},"min":{"1min":0.89100000000000001,"5min":0.85799999999999998,"15min":0.72899999999999998},"max":{"1min":1.425,"5min":1.54,"15min":1.5589999999999999},"last":0.97699999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.365,"15min":1.3580000000000001},"min":{"1min":1.079,"5min":0.94999999999999996,"15min":0.79000000000000004},"max":{"1min":1.5640000000000001,"5min":1.754,"15min":1.754},"last":1.24},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3620000000000001,"15min":1.3400000000000001},"min":{"1min":1.111,"5min":0.92300000000000004,"15min":0.82099999999999995},"max":{"1min":1.7450000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.175}]},{"osd":5,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.401,"15min":1.395},"min":{"1min":1.139,"5min":1.0680000000000001,"15min":0.90500000000000003},"max":{"1min":1.631,"5min":2.016,"15min":2.016},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3300000000000001,"15min":1.3140000000000001},"min":{"1min":1.028,"5min":0.91400000000000003,"15min":0.68300000000000005},"max":{"1min":1.6319999999999999,"5min":1.901,"15min":1.927},"last":1.1990000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3460000000000001,"15min":1.3520000000000001},"min":{"1min":1.0580000000000001,"5min":1.014,"15min":0.89000000000000001},"max":{"1min":1.6679999999999999,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4299999999999999,"15min":1.4390000000000001},"min":{"1min":1.1870000000000001,"5min":1.1040000000000001,"15min":0.92100000000000004},"max":{"1min":1.6919999999999999,"5min":1.8580000000000001,"15min":2.1909999999999998},"last":1.3340000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.462,"15min":1.448},"min":{"1min":1.075,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8999999999999999,"5min":2.327,"15min":2.327},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.4450000000000001,"15min":1.4350000000000001},"min":{"1min":1.1080000000000001,"5min":1.04,"15min":0.81100000000000005},"max":{"1min":1.665,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.3180000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575949,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311936,"kb_used_data":2360,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059904,"statfs":{"total":96636764160,"available":96317341696,"internally_reserved":0,"allocated":2416640,"data_stored":1289577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.206,"5min":1.4239999999999999,"15min":1.9039999999999999},"last":0.873},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.99299999999999999,"15min":1.0009999999999999},"min":{"1min":0.63100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.3120000000000001,"5min":1.575,"15min":1.6850000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94099999999999995,"15min":0.95399999999999996},"min":{"1min":0.69999999999999996,"5min":0.63200000000000001,"15min":0.61699999999999999},"max":{"1min":1.1819999999999999,"5min":2.331,"15min":2.331},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0620000000000001,"15min":1.083},"min":{"1min":0.84899999999999998,"5min":0.68799999999999994,"15min":0.56999999999999995},"max":{"1min":1.45,"5min":2.302,"15min":2.302},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1839999999999999,"15min":1.181},"min":{"1min":0.85299999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.534,"5min":1.851,"15min":2.0950000000000002},"last":1},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.083,"15min":1.0900000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.694,"5min":1.694,"15min":1.76},"last":1.125}]},{"osd":4,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.24,"15min":1.246},"min":{"1min":1.0009999999999999,"5min":0.89600000000000002,"15min":0.76700000000000002},"max":{"1min":1.5880000000000001,"5min":1.7430000000000001,"15min":1.9670000000000001},"last":1.528},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3300000000000001,"15min":1.3320000000000001},"min":{"1min":1.111,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.708,"5min":1.708,"15min":2.0350000000000001},"last":1.379}]},{"osd":5,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.5649999999999999,"5min":1.921,"15min":2.1699999999999999},"last":1.409},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3029999999999999,"15min":1.3109999999999999},"min":{"1min":1.024,"5min":0.93899999999999995,"15min":0.84499999999999997},"max":{"1min":1.6419999999999999,"5min":1.6930000000000001,"15min":2.117},"last":1.4379999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.387,"15min":1.409},"min":{"1min":1.0920000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.6299999999999999,"5min":1.877,"15min":2},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.393,"15min":1.4199999999999999},"min":{"1min":0.85499999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.6739999999999999,"5min":2.0129999999999999,"15min":2.3690000000000002},"last":1.4970000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.452,"15min":1.4530000000000001},"min":{"1min":1.01,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.7470000000000001,"5min":1.8280000000000001,"15min":2.2869999999999999},"last":1.591},{"interface":"front","average":{"1min":1.417,"5min":1.4259999999999999,"15min":1.427},"min":{"1min":1.1459999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.73,"5min":1.9339999999999999,"15min":2.2360000000000002},"last":1.222}]}]},{"osd":0,"up_from":13,"seq":55834575948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300220,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071620,"statfs":{"total":96636764160,"available":96329338880,"internally_reserved":0,"allocated":3002368,"data_stored":1877502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1379999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3460000000000001,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.79200000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0149999999999999,"15min":1.0289999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.321,"5min":1.446,"15min":1.5620000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.0649999999999999,"15min":1.0680000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.538,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0289999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.018,"15min":1.016},"min":{"1min":0.71399999999999997,"5min":0.68600000000000005,"15min":0.622},"max":{"1min":1.23,"5min":1.611,"15min":2.621},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.175,"15min":1.181},"min":{"1min":0.81399999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.425,"5min":1.5920000000000001,"15min":1.718},"last":0.98299999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.98299999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6020000000000001,"5min":1.891,"15min":1.891},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.69599999999999995},"max":{"1min":1.5800000000000001,"5min":1.919,"15min":2.5339999999999998},"last":1.167}]},{"osd":5,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.4039999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.286},{"interface":"front","average":{"1min":1.369,"5min":1.405,"15min":1.421},"min":{"1min":0.83499999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.542}]},{"osd":6,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.4489999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3700000000000001,"15min":1.3740000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0379999999999998,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.3819999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.3420000000000001,"15min":1.345},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.9770000000000001,"5min":2.137,"15min":2.8570000000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4339999999999999,"15min":1.4399999999999999},"min":{"1min":0.92200000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.181},"last":1.345}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:19:44.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:19:44.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.477+0000 7f74aef85700 1 Processor -- start 2023-12-19T09:19:44.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 -- start start 2023-12-19T09:19:44.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f74a80c1d60 0x7f74a80680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a806dc20 con 0x7f74a80ae510 2023-12-19T09:19:44.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a80c2200 con 0x7f74a80c1d60 2023-12-19T09:19:44.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74aef85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a806d940 con 0x7f74a80ae140 2023-12-19T09:19:44.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74acd21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.481+0000 7f74acd21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:55692/0 (socket says 172.21.15.111:55692) 2023-12-19T09:19:44.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 -- 172.21.15.111:0/4176541140 learned_addr learned my addr 172.21.15.111:0/4176541140 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:44.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a7fff700 1 --1- 172.21.15.111:0/4176541140 >> v1:172.21.15.143:6789/0 conn(0x7f74a80ae510 0x7f74a806d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:50778/0 (socket says 172.21.15.111:50778) 2023-12-19T09:19:44.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74ad522700 1 --2- 172.21.15.111:0/4176541140 >> v2:172.21.15.143:3300/0 conn(0x7f74a80c1d60 0x7f74a80680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a77fe700 1 -- 172.21.15.111:0/4176541140 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245804360 0 0) 0x7f74a806dc20 con 0x7f74a80ae510 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a77fe700 1 -- 172.21.15.111:0/4176541140 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7498003570 con 0x7f74a80ae510 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 -- 172.21.15.111:0/4176541140 >> v2:172.21.15.143:3300/0 conn(0x7f74a80c1d60 msgr2=0x7f74a80680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 --2- 172.21.15.111:0/4176541140 >> v2:172.21.15.143:3300/0 conn(0x7f74a80c1d60 0x7f74a80680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 -- 172.21.15.111:0/4176541140 >> v1:172.21.15.143:6789/0 conn(0x7f74a80ae510 legacy=0x7f74a806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 -- 172.21.15.111:0/4176541140 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80c6e20 con 0x7f74a80ae140 2023-12-19T09:19:44.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74acd21700 1 --2- 172.21.15.111:0/4176541140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f749c0099b0 tx=0x7f749c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41e294df7daeb149 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a77fe700 1 -- 172.21.15.111:0/4176541140 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00b680 con 0x7f74a80ae140 2023-12-19T09:19:44.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a77fe700 1 -- 172.21.15.111:0/4176541140 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c00b7e0 con 0x7f74a80ae140 2023-12-19T09:19:44.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74a77fe700 1 -- 172.21.15.111:0/4176541140 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00bb80 con 0x7f74a80ae140 2023-12-19T09:19:44.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 -- 172.21.15.111:0/4176541140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 msgr2=0x7f74a80a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 --2- 172.21.15.111:0/4176541140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f749c0099b0 tx=0x7f749c009250 comp rx=0 tx=0).stop 2023-12-19T09:19:44.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 -- 172.21.15.111:0/4176541140 shutdown_connections 2023-12-19T09:19:44.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 --2- 172.21.15.111:0/4176541140 >> v2:172.21.15.143:3300/0 conn(0x7f74a80c1d60 0x7f74a80680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 --2- 172.21.15.111:0/4176541140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80a8850 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 -- 172.21.15.111:0/4176541140 >> 172.21.15.111:0/4176541140 conn(0x7f74a80c8850 msgr2=0x7f74a80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:44.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 -- 172.21.15.111:0/4176541140 shutdown_connections 2023-12-19T09:19:44.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.485+0000 7f74aef85700 1 -- 172.21.15.111:0/4176541140 wait complete. 2023-12-19T09:19:44.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.489+0000 7f74aef85700 1 Processor -- start 2023-12-19T09:19:44.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.489+0000 7f74aef85700 1 -- start start 2023-12-19T09:19:44.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74a80c1d60 0x7f74a8156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c4830 con 0x7f74a80ae510 2023-12-19T09:19:44.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c3d80 con 0x7f74a80c1d60 2023-12-19T09:19:44.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c2e90 con 0x7f74a80ae140 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57486/0 (socket says 172.21.15.111:57486) 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 -- 172.21.15.111:0/3186211646 learned_addr learned my addr 172.21.15.111:0/3186211646 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74acd21700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74ad522700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74a80c1d60 0x7f74a8156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 -- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 msgr2=0x7f74a80aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80aa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 -- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74a80c1d60 msgr2=0x7f74a8156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74a80c1d60 0x7f74a8156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 -- 172.21.15.111:0/3186211646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0094f0 con 0x7f74a80ae510 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a7fff700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f749400c7c0 tx=0x7f7494007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494005b40 con 0x7f74a80ae510 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7494005ca0 con 0x7f74a80ae510 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a80c3170 con 0x7f74a80ae510 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494007320 con 0x7f74a80ae510 2023-12-19T09:19:44.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.493+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a80c3730 con 0x7f74a80ae510 2023-12-19T09:19:44.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.497+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a816ca20 con 0x7f74a80ae510 2023-12-19T09:19:44.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.497+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f749401b3b0 con 0x7f74a80ae510 2023-12-19T09:19:44.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.501+0000 7f74a4ff9700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 0x7f748c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.501+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f749407c070 con 0x7f74a80ae510 2023-12-19T09:19:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.501+0000 7f74acd21700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 0x7f748c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.501+0000 7f74acd21700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 0x7f748c062a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f749c0099b0 tx=0x7f749c017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.505+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f749404c240 con 0x7f74a80ae510 2023-12-19T09:19:44.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.669+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74a8071120 con 0x7f748c060550 2023-12-19T09:19:44.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.681+0000 7f74a4ff9700 1 -- 172.21.15.111:0/3186211646 <== mgr.8285 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287018 (secure 0 0 0) 0x7f74a8071120 con 0x7f748c060550 2023-12-19T09:19:44.692 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 msgr2=0x7f748c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 0x7f748c062a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f749c0099b0 tx=0x7f749c017000 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 msgr2=0x7f74a80c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f749400c7c0 tx=0x7f7494007dd0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 shutdown_connections 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f748c060550 0x7f748c062a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74a80c1d60 0x7f74a8156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74a80ae510 0x7f74a80c55b0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 --2- 172.21.15.111:0/3186211646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74a80ae140 0x7f74a80aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 >> 172.21.15.111:0/3186211646 conn(0x7f74a80c8850 msgr2=0x7f74a8157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:44.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 shutdown_connections 2023-12-19T09:19:44.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.689+0000 7f74aef85700 1 -- 172.21.15.111:0/3186211646 wait complete. 2023-12-19T09:19:44.697 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:19:44.715 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:19:43.669447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2467964,"kb_used_data":20028,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752506756,"statfs":{"total":773094113280,"available":770566918144,"internally_reserved":0,"allocated":20508672,"data_stored":11503033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.690432"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041044+0000","last_change":"2023-12-19T09:19:40.037606+0000","last_active":"2023-12-19T09:19:40.041044+0000","last_peered":"2023-12-19T09:19:40.041044+0000","last_clean":"2023-12-19T09:19:40.041044+0000","last_became_active":"2023-12-19T09:19:40.037279+0000","last_became_peered":"2023-12-19T09:19:40.037279+0000","last_unstale":"2023-12-19T09:19:40.041044+0000","last_undegraded":"2023-12-19T09:19:40.041044+0000","last_fullsized":"2023-12-19T09:19:40.041044+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:33.077567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044043+0000","last_change":"2023-12-19T09:19:40.034805+0000","last_active":"2023-12-19T09:19:40.044043+0000","last_peered":"2023-12-19T09:19:40.044043+0000","last_clean":"2023-12-19T09:19:40.044043+0000","last_became_active":"2023-12-19T09:19:40.034407+0000","last_became_peered":"2023-12-19T09:19:40.034407+0000","last_unstale":"2023-12-19T09:19:40.044043+0000","last_undegraded":"2023-12-19T09:19:40.044043+0000","last_fullsized":"2023-12-19T09:19:40.044043+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:02:32.144191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.053451+0000","last_change":"2023-12-19T09:19:40.049649+0000","last_active":"2023-12-19T09:19:40.053451+0000","last_peered":"2023-12-19T09:19:40.053451+0000","last_clean":"2023-12-19T09:19:40.053451+0000","last_became_active":"2023-12-19T09:19:40.049289+0000","last_became_peered":"2023-12-19T09:19:40.049289+0000","last_unstale":"2023-12-19T09:19:40.053451+0000","last_undegraded":"2023-12-19T09:19:40.053451+0000","last_fullsized":"2023-12-19T09:19:40.053451+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:16.608412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039093+0000","last_change":"2023-12-19T09:19:40.032515+0000","last_active":"2023-12-19T09:19:40.039093+0000","last_peered":"2023-12-19T09:19:40.039093+0000","last_clean":"2023-12-19T09:19:40.039093+0000","last_became_active":"2023-12-19T09:19:40.032175+0000","last_became_peered":"2023-12-19T09:19:40.032175+0000","last_unstale":"2023-12-19T09:19:40.039093+0000","last_undegraded":"2023-12-19T09:19:40.039093+0000","last_fullsized":"2023-12-19T09:19:40.039093+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:56.339627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.028015+0000","last_change":"2023-12-19T09:19:40.028015+0000","last_active":"2023-12-19T09:19:40.028015+0000","last_peered":"2023-12-19T09:19:40.028015+0000","last_clean":"2023-12-19T09:19:40.028015+0000","last_became_active":"2023-12-19T09:19:40.027721+0000","last_became_peered":"2023-12-19T09:19:40.027721+0000","last_unstale":"2023-12-19T09:19:40.028015+0000","last_undegraded":"2023-12-19T09:19:40.028015+0000","last_fullsized":"2023-12-19T09:19:40.028015+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:47.920865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044106+0000","last_change":"2023-12-19T09:19:40.036192+0000","last_active":"2023-12-19T09:19:40.044106+0000","last_peered":"2023-12-19T09:19:40.044106+0000","last_clean":"2023-12-19T09:19:40.044106+0000","last_became_active":"2023-12-19T09:19:40.035867+0000","last_became_peered":"2023-12-19T09:19:40.035867+0000","last_unstale":"2023-12-19T09:19:40.044106+0000","last_undegraded":"2023-12-19T09:19:40.044106+0000","last_fullsized":"2023-12-19T09:19:40.044106+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:19.209801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043958+0000","last_change":"2023-12-19T09:19:40.043958+0000","last_active":"2023-12-19T09:19:40.043958+0000","last_peered":"2023-12-19T09:19:40.043958+0000","last_clean":"2023-12-19T09:19:40.043958+0000","last_became_active":"2023-12-19T09:19:40.043504+0000","last_became_peered":"2023-12-19T09:19:40.043504+0000","last_unstale":"2023-12-19T09:19:40.043958+0000","last_undegraded":"2023-12-19T09:19:40.043958+0000","last_fullsized":"2023-12-19T09:19:40.043958+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:57.821705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047541+0000","last_change":"2023-12-19T09:19:40.047541+0000","last_active":"2023-12-19T09:19:40.047541+0000","last_peered":"2023-12-19T09:19:40.047541+0000","last_clean":"2023-12-19T09:19:40.047541+0000","last_became_active":"2023-12-19T09:19:40.047268+0000","last_became_peered":"2023-12-19T09:19:40.047268+0000","last_unstale":"2023-12-19T09:19:40.047541+0000","last_undegraded":"2023-12-19T09:19:40.047541+0000","last_fullsized":"2023-12-19T09:19:40.047541+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:22:14.364823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040335+0000","last_change":"2023-12-19T09:19:40.040335+0000","last_active":"2023-12-19T09:19:40.040335+0000","last_peered":"2023-12-19T09:19:40.040335+0000","last_clean":"2023-12-19T09:19:40.040335+0000","last_became_active":"2023-12-19T09:19:40.039865+0000","last_became_peered":"2023-12-19T09:19:40.039865+0000","last_unstale":"2023-12-19T09:19:40.040335+0000","last_undegraded":"2023-12-19T09:19:40.040335+0000","last_fullsized":"2023-12-19T09:19:40.040335+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:37.765631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039949+0000","last_change":"2023-12-19T09:19:40.039949+0000","last_active":"2023-12-19T09:19:40.039949+0000","last_peered":"2023-12-19T09:19:40.039949+0000","last_clean":"2023-12-19T09:19:40.039949+0000","last_became_active":"2023-12-19T09:19:40.039576+0000","last_became_peered":"2023-12-19T09:19:40.039576+0000","last_unstale":"2023-12-19T09:19:40.039949+0000","last_undegraded":"2023-12-19T09:19:40.039949+0000","last_fullsized":"2023-12-19T09:19:40.039949+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:44:04.846619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023558+0000","last_change":"2023-12-19T09:19:40.023558+0000","last_active":"2023-12-19T09:19:40.023558+0000","last_peered":"2023-12-19T09:19:40.023558+0000","last_clean":"2023-12-19T09:19:40.023558+0000","last_became_active":"2023-12-19T09:19:40.023217+0000","last_became_peered":"2023-12-19T09:19:40.023217+0000","last_unstale":"2023-12-19T09:19:40.023558+0000","last_undegraded":"2023-12-19T09:19:40.023558+0000","last_fullsized":"2023-12-19T09:19:40.023558+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:26:32.290878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044414+0000","last_change":"2023-12-19T09:19:40.035154+0000","last_active":"2023-12-19T09:19:40.044414+0000","last_peered":"2023-12-19T09:19:40.044414+0000","last_clean":"2023-12-19T09:19:40.044414+0000","last_became_active":"2023-12-19T09:19:40.034906+0000","last_became_peered":"2023-12-19T09:19:40.034906+0000","last_unstale":"2023-12-19T09:19:40.044414+0000","last_undegraded":"2023-12-19T09:19:40.044414+0000","last_fullsized":"2023-12-19T09:19:40.044414+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:58:35.969419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047196+0000","last_change":"2023-12-19T09:19:40.035487+0000","last_active":"2023-12-19T09:19:40.047196+0000","last_peered":"2023-12-19T09:19:40.047196+0000","last_clean":"2023-12-19T09:19:40.047196+0000","last_became_active":"2023-12-19T09:19:40.034784+0000","last_became_peered":"2023-12-19T09:19:40.034784+0000","last_unstale":"2023-12-19T09:19:40.047196+0000","last_undegraded":"2023-12-19T09:19:40.047196+0000","last_fullsized":"2023-12-19T09:19:40.047196+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:25:22.884423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036703+0000","last_change":"2023-12-19T09:19:40.036703+0000","last_active":"2023-12-19T09:19:40.036703+0000","last_peered":"2023-12-19T09:19:40.036703+0000","last_clean":"2023-12-19T09:19:40.036703+0000","last_became_active":"2023-12-19T09:19:40.035566+0000","last_became_peered":"2023-12-19T09:19:40.035566+0000","last_unstale":"2023-12-19T09:19:40.036703+0000","last_undegraded":"2023-12-19T09:19:40.036703+0000","last_fullsized":"2023-12-19T09:19:40.036703+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:22.544550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046158+0000","last_change":"2023-12-19T09:19:40.046158+0000","last_active":"2023-12-19T09:19:40.046158+0000","last_peered":"2023-12-19T09:19:40.046158+0000","last_clean":"2023-12-19T09:19:40.046158+0000","last_became_active":"2023-12-19T09:19:40.045832+0000","last_became_peered":"2023-12-19T09:19:40.045832+0000","last_unstale":"2023-12-19T09:19:40.046158+0000","last_undegraded":"2023-12-19T09:19:40.046158+0000","last_fullsized":"2023-12-19T09:19:40.046158+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:31:25.397590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039207+0000","last_change":"2023-12-19T09:19:40.032633+0000","last_active":"2023-12-19T09:19:40.039207+0000","last_peered":"2023-12-19T09:19:40.039207+0000","last_clean":"2023-12-19T09:19:40.039207+0000","last_became_active":"2023-12-19T09:19:40.032282+0000","last_became_peered":"2023-12-19T09:19:40.032282+0000","last_unstale":"2023-12-19T09:19:40.039207+0000","last_undegraded":"2023-12-19T09:19:40.039207+0000","last_fullsized":"2023-12-19T09:19:40.039207+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:13:58.604397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040145+0000","last_change":"2023-12-19T09:19:40.040145+0000","last_active":"2023-12-19T09:19:40.040145+0000","last_peered":"2023-12-19T09:19:40.040145+0000","last_clean":"2023-12-19T09:19:40.040145+0000","last_became_active":"2023-12-19T09:19:40.039546+0000","last_became_peered":"2023-12-19T09:19:40.039546+0000","last_unstale":"2023-12-19T09:19:40.040145+0000","last_undegraded":"2023-12-19T09:19:40.040145+0000","last_fullsized":"2023-12-19T09:19:40.040145+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:43.610486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.042328+0000","last_change":"2023-12-19T09:19:40.042328+0000","last_active":"2023-12-19T09:19:40.042328+0000","last_peered":"2023-12-19T09:19:40.042328+0000","last_clean":"2023-12-19T09:19:40.042328+0000","last_became_active":"2023-12-19T09:19:40.042043+0000","last_became_peered":"2023-12-19T09:19:40.042043+0000","last_unstale":"2023-12-19T09:19:40.042328+0000","last_undegraded":"2023-12-19T09:19:40.042328+0000","last_fullsized":"2023-12-19T09:19:40.042328+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:43.448409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.045694+0000","last_change":"2023-12-19T09:19:40.045694+0000","last_active":"2023-12-19T09:19:40.045694+0000","last_peered":"2023-12-19T09:19:40.045694+0000","last_clean":"2023-12-19T09:19:40.045694+0000","last_became_active":"2023-12-19T09:19:40.045335+0000","last_became_peered":"2023-12-19T09:19:40.045335+0000","last_unstale":"2023-12-19T09:19:40.045694+0000","last_undegraded":"2023-12-19T09:19:40.045694+0000","last_fullsized":"2023-12-19T09:19:40.045694+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:19.049197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039793+0000","last_change":"2023-12-19T09:19:40.039793+0000","last_active":"2023-12-19T09:19:40.039793+0000","last_peered":"2023-12-19T09:19:40.039793+0000","last_clean":"2023-12-19T09:19:40.039793+0000","last_became_active":"2023-12-19T09:19:40.039525+0000","last_became_peered":"2023-12-19T09:19:40.039525+0000","last_unstale":"2023-12-19T09:19:40.039793+0000","last_undegraded":"2023-12-19T09:19:40.039793+0000","last_fullsized":"2023-12-19T09:19:40.039793+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:48:20.971700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034395+0000","last_change":"2023-12-19T09:19:40.028414+0000","last_active":"2023-12-19T09:19:40.034395+0000","last_peered":"2023-12-19T09:19:40.034395+0000","last_clean":"2023-12-19T09:19:40.034395+0000","last_became_active":"2023-12-19T09:19:40.028137+0000","last_became_peered":"2023-12-19T09:19:40.028137+0000","last_unstale":"2023-12-19T09:19:40.034395+0000","last_undegraded":"2023-12-19T09:19:40.034395+0000","last_fullsized":"2023-12-19T09:19:40.034395+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:08:03.116601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.030161+0000","last_change":"2023-12-19T09:19:40.027582+0000","last_active":"2023-12-19T09:19:40.030161+0000","last_peered":"2023-12-19T09:19:40.030161+0000","last_clean":"2023-12-19T09:19:40.030161+0000","last_became_active":"2023-12-19T09:19:40.027271+0000","last_became_peered":"2023-12-19T09:19:40.027271+0000","last_unstale":"2023-12-19T09:19:40.030161+0000","last_undegraded":"2023-12-19T09:19:40.030161+0000","last_fullsized":"2023-12-19T09:19:40.030161+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:16.880935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.053807+0000","last_change":"2023-12-19T09:19:40.053807+0000","last_active":"2023-12-19T09:19:40.053807+0000","last_peered":"2023-12-19T09:19:40.053807+0000","last_clean":"2023-12-19T09:19:40.053807+0000","last_became_active":"2023-12-19T09:19:40.037781+0000","last_became_peered":"2023-12-19T09:19:40.037781+0000","last_unstale":"2023-12-19T09:19:40.053807+0000","last_undegraded":"2023-12-19T09:19:40.053807+0000","last_fullsized":"2023-12-19T09:19:40.053807+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:39:57.821705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.048339+0000","last_change":"2023-12-19T09:19:40.046917+0000","last_active":"2023-12-19T09:19:40.048339+0000","last_peered":"2023-12-19T09:19:40.048339+0000","last_clean":"2023-12-19T09:19:40.048339+0000","last_became_active":"2023-12-19T09:19:40.046601+0000","last_became_peered":"2023-12-19T09:19:40.046601+0000","last_unstale":"2023-12-19T09:19:40.048339+0000","last_undegraded":"2023-12-19T09:19:40.048339+0000","last_fullsized":"2023-12-19T09:19:40.048339+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:53:57.208632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.045986+0000","last_change":"2023-12-19T09:19:40.035605+0000","last_active":"2023-12-19T09:19:40.045986+0000","last_peered":"2023-12-19T09:19:40.045986+0000","last_clean":"2023-12-19T09:19:40.045986+0000","last_became_active":"2023-12-19T09:19:40.035288+0000","last_became_peered":"2023-12-19T09:19:40.035288+0000","last_unstale":"2023-12-19T09:19:40.045986+0000","last_undegraded":"2023-12-19T09:19:40.045986+0000","last_fullsized":"2023-12-19T09:19:40.045986+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:14:18.566066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046150+0000","last_change":"2023-12-19T09:19:40.046150+0000","last_active":"2023-12-19T09:19:40.046150+0000","last_peered":"2023-12-19T09:19:40.046150+0000","last_clean":"2023-12-19T09:19:40.046150+0000","last_became_active":"2023-12-19T09:19:40.033802+0000","last_became_peered":"2023-12-19T09:19:40.033802+0000","last_unstale":"2023-12-19T09:19:40.046150+0000","last_undegraded":"2023-12-19T09:19:40.046150+0000","last_fullsized":"2023-12-19T09:19:40.046150+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:38:51.885568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.030367+0000","last_change":"2023-12-19T09:19:40.028542+0000","last_active":"2023-12-19T09:19:40.030367+0000","last_peered":"2023-12-19T09:19:40.030367+0000","last_clean":"2023-12-19T09:19:40.030367+0000","last_became_active":"2023-12-19T09:19:40.028253+0000","last_became_peered":"2023-12-19T09:19:40.028253+0000","last_unstale":"2023-12-19T09:19:40.030367+0000","last_undegraded":"2023-12-19T09:19:40.030367+0000","last_fullsized":"2023-12-19T09:19:40.030367+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:15.746493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056226+0000","last_change":"2023-12-19T09:19:40.045421+0000","last_active":"2023-12-19T09:19:40.056226+0000","last_peered":"2023-12-19T09:19:40.056226+0000","last_clean":"2023-12-19T09:19:40.056226+0000","last_became_active":"2023-12-19T09:19:40.045089+0000","last_became_peered":"2023-12-19T09:19:40.045089+0000","last_unstale":"2023-12-19T09:19:40.056226+0000","last_undegraded":"2023-12-19T09:19:40.056226+0000","last_fullsized":"2023-12-19T09:19:40.056226+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:47:44.270632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.048036+0000","last_change":"2023-12-19T09:19:40.044659+0000","last_active":"2023-12-19T09:19:40.048036+0000","last_peered":"2023-12-19T09:19:40.048036+0000","last_clean":"2023-12-19T09:19:40.048036+0000","last_became_active":"2023-12-19T09:19:40.044377+0000","last_became_peered":"2023-12-19T09:19:40.044377+0000","last_unstale":"2023-12-19T09:19:40.048036+0000","last_undegraded":"2023-12-19T09:19:40.048036+0000","last_fullsized":"2023-12-19T09:19:40.048036+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:37:16.042900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.035126+0000","last_change":"2023-12-19T09:19:40.035126+0000","last_active":"2023-12-19T09:19:40.035126+0000","last_peered":"2023-12-19T09:19:40.035126+0000","last_clean":"2023-12-19T09:19:40.035126+0000","last_became_active":"2023-12-19T09:19:40.034810+0000","last_became_peered":"2023-12-19T09:19:40.034810+0000","last_unstale":"2023-12-19T09:19:40.035126+0000","last_undegraded":"2023-12-19T09:19:40.035126+0000","last_fullsized":"2023-12-19T09:19:40.035126+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:16:21.858425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038938+0000","last_change":"2023-12-19T09:19:40.038938+0000","last_active":"2023-12-19T09:19:40.038938+0000","last_peered":"2023-12-19T09:19:40.038938+0000","last_clean":"2023-12-19T09:19:40.038938+0000","last_became_active":"2023-12-19T09:19:40.038652+0000","last_became_peered":"2023-12-19T09:19:40.038652+0000","last_unstale":"2023-12-19T09:19:40.038938+0000","last_undegraded":"2023-12-19T09:19:40.038938+0000","last_fullsized":"2023-12-19T09:19:40.038938+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:49.882654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034794+0000","last_change":"2023-12-19T09:19:40.034794+0000","last_active":"2023-12-19T09:19:40.034794+0000","last_peered":"2023-12-19T09:19:40.034794+0000","last_clean":"2023-12-19T09:19:40.034794+0000","last_became_active":"2023-12-19T09:19:40.034358+0000","last_became_peered":"2023-12-19T09:19:40.034358+0000","last_unstale":"2023-12-19T09:19:40.034794+0000","last_undegraded":"2023-12-19T09:19:40.034794+0000","last_fullsized":"2023-12-19T09:19:40.034794+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:54:24.940718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047323+0000","last_change":"2023-12-19T09:19:40.043118+0000","last_active":"2023-12-19T09:19:40.047323+0000","last_peered":"2023-12-19T09:19:40.047323+0000","last_clean":"2023-12-19T09:19:40.047323+0000","last_became_active":"2023-12-19T09:19:40.042476+0000","last_became_peered":"2023-12-19T09:19:40.042476+0000","last_unstale":"2023-12-19T09:19:40.047323+0000","last_undegraded":"2023-12-19T09:19:40.047323+0000","last_fullsized":"2023-12-19T09:19:40.047323+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:18:53.213024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"192'4","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057621+0000","last_change":"2023-12-19T09:19:39.021504+0000","last_active":"2023-12-19T09:19:40.057621+0000","last_peered":"2023-12-19T09:19:40.057621+0000","last_clean":"2023-12-19T09:19:40.057621+0000","last_became_active":"2023-12-19T09:19:39.021203+0000","last_became_peered":"2023-12-19T09:19:39.021203+0000","last_unstale":"2023-12-19T09:19:40.057621+0000","last_undegraded":"2023-12-19T09:19:40.057621+0000","last_fullsized":"2023-12-19T09:19:40.057621+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:48:14.166174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056618+0000","last_change":"2023-12-19T09:19:40.046796+0000","last_active":"2023-12-19T09:19:40.056618+0000","last_peered":"2023-12-19T09:19:40.056618+0000","last_clean":"2023-12-19T09:19:40.056618+0000","last_became_active":"2023-12-19T09:19:40.046426+0000","last_became_peered":"2023-12-19T09:19:40.046426+0000","last_unstale":"2023-12-19T09:19:40.056618+0000","last_undegraded":"2023-12-19T09:19:40.056618+0000","last_fullsized":"2023-12-19T09:19:40.056618+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:46.317043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.011232+0000","last_change":"2023-12-19T09:19:39.013895+0000","last_active":"2023-12-19T09:19:40.011232+0000","last_peered":"2023-12-19T09:19:40.011232+0000","last_clean":"2023-12-19T09:19:40.011232+0000","last_became_active":"2023-12-19T09:19:39.013638+0000","last_became_peered":"2023-12-19T09:19:39.013638+0000","last_unstale":"2023-12-19T09:19:40.011232+0000","last_undegraded":"2023-12-19T09:19:40.011232+0000","last_fullsized":"2023-12-19T09:19:40.011232+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:18.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.035727+0000","last_change":"2023-12-19T09:19:40.035727+0000","last_active":"2023-12-19T09:19:40.035727+0000","last_peered":"2023-12-19T09:19:40.035727+0000","last_clean":"2023-12-19T09:19:40.035727+0000","last_became_active":"2023-12-19T09:19:40.035180+0000","last_became_peered":"2023-12-19T09:19:40.035180+0000","last_unstale":"2023-12-19T09:19:40.035727+0000","last_undegraded":"2023-12-19T09:19:40.035727+0000","last_fullsized":"2023-12-19T09:19:40.035727+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:20.262991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.058759+0000","last_change":"2023-12-19T09:19:39.006533+0000","last_active":"2023-12-19T09:19:40.058759+0000","last_peered":"2023-12-19T09:19:40.058759+0000","last_clean":"2023-12-19T09:19:40.058759+0000","last_became_active":"2023-12-19T09:19:39.006232+0000","last_became_peered":"2023-12-19T09:19:39.006232+0000","last_unstale":"2023-12-19T09:19:40.058759+0000","last_undegraded":"2023-12-19T09:19:40.058759+0000","last_fullsized":"2023-12-19T09:19:40.058759+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:40:32.062982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.044189+0000","last_change":"2023-12-19T09:19:40.040220+0000","last_active":"2023-12-19T09:19:40.044189+0000","last_peered":"2023-12-19T09:19:40.044189+0000","last_clean":"2023-12-19T09:19:40.044189+0000","last_became_active":"2023-12-19T09:19:40.038187+0000","last_became_peered":"2023-12-19T09:19:40.038187+0000","last_unstale":"2023-12-19T09:19:40.044189+0000","last_undegraded":"2023-12-19T09:19:40.044189+0000","last_fullsized":"2023-12-19T09:19:40.044189+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:12:21.237273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.055044+0000","last_change":"2023-12-19T09:19:39.010088+0000","last_active":"2023-12-19T09:19:40.055044+0000","last_peered":"2023-12-19T09:19:40.055044+0000","last_clean":"2023-12-19T09:19:40.055044+0000","last_became_active":"2023-12-19T09:19:39.009813+0000","last_became_peered":"2023-12-19T09:19:39.009813+0000","last_unstale":"2023-12-19T09:19:40.055044+0000","last_undegraded":"2023-12-19T09:19:40.055044+0000","last_fullsized":"2023-12-19T09:19:40.055044+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:44:30.593357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043044+0000","last_change":"2023-12-19T09:19:40.043044+0000","last_active":"2023-12-19T09:19:40.043044+0000","last_peered":"2023-12-19T09:19:40.043044+0000","last_clean":"2023-12-19T09:19:40.043044+0000","last_became_active":"2023-12-19T09:19:40.042762+0000","last_became_peered":"2023-12-19T09:19:40.042762+0000","last_unstale":"2023-12-19T09:19:40.043044+0000","last_undegraded":"2023-12-19T09:19:40.043044+0000","last_fullsized":"2023-12-19T09:19:40.043044+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:46:22.544550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.027164+0000","last_change":"2023-12-19T09:19:39.024166+0000","last_active":"2023-12-19T09:19:40.027164+0000","last_peered":"2023-12-19T09:19:40.027164+0000","last_clean":"2023-12-19T09:19:40.027164+0000","last_became_active":"2023-12-19T09:19:39.023823+0000","last_became_peered":"2023-12-19T09:19:39.023823+0000","last_unstale":"2023-12-19T09:19:40.027164+0000","last_undegraded":"2023-12-19T09:19:40.027164+0000","last_fullsized":"2023-12-19T09:19:40.027164+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:35:14.104603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.049777+0000","last_change":"2023-12-19T09:19:40.049777+0000","last_active":"2023-12-19T09:19:40.049777+0000","last_peered":"2023-12-19T09:19:40.049777+0000","last_clean":"2023-12-19T09:19:40.049777+0000","last_became_active":"2023-12-19T09:19:40.048272+0000","last_became_peered":"2023-12-19T09:19:40.048272+0000","last_unstale":"2023-12-19T09:19:40.049777+0000","last_undegraded":"2023-12-19T09:19:40.049777+0000","last_fullsized":"2023-12-19T09:19:40.049777+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:57:35.103493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.010420+0000","last_change":"2023-12-19T09:19:39.021851+0000","last_active":"2023-12-19T09:19:40.010420+0000","last_peered":"2023-12-19T09:19:40.010420+0000","last_clean":"2023-12-19T09:19:40.010420+0000","last_became_active":"2023-12-19T09:19:39.021322+0000","last_became_peered":"2023-12-19T09:19:39.021322+0000","last_unstale":"2023-12-19T09:19:40.010420+0000","last_undegraded":"2023-12-19T09:19:40.010420+0000","last_fullsized":"2023-12-19T09:19:40.010420+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:08.209309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.046223+0000","last_change":"2023-12-19T09:19:40.046223+0000","last_active":"2023-12-19T09:19:40.046223+0000","last_peered":"2023-12-19T09:19:40.046223+0000","last_clean":"2023-12-19T09:19:40.046223+0000","last_became_active":"2023-12-19T09:19:40.035421+0000","last_became_peered":"2023-12-19T09:19:40.035421+0000","last_unstale":"2023-12-19T09:19:40.046223+0000","last_undegraded":"2023-12-19T09:19:40.046223+0000","last_fullsized":"2023-12-19T09:19:40.046223+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:35:31.210260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.022640+0000","last_change":"2023-12-19T09:19:39.021391+0000","last_active":"2023-12-19T09:19:39.022640+0000","last_peered":"2023-12-19T09:19:39.022640+0000","last_clean":"2023-12-19T09:19:39.022640+0000","last_became_active":"2023-12-19T09:19:39.021010+0000","last_became_peered":"2023-12-19T09:19:39.021010+0000","last_unstale":"2023-12-19T09:19:39.022640+0000","last_undegraded":"2023-12-19T09:19:39.022640+0000","last_fullsized":"2023-12-19T09:19:39.022640+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:05:50.921800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039322+0000","last_change":"2023-12-19T09:19:40.032885+0000","last_active":"2023-12-19T09:19:40.039322+0000","last_peered":"2023-12-19T09:19:40.039322+0000","last_clean":"2023-12-19T09:19:40.039322+0000","last_became_active":"2023-12-19T09:19:40.031417+0000","last_became_peered":"2023-12-19T09:19:40.031417+0000","last_unstale":"2023-12-19T09:19:40.039322+0000","last_undegraded":"2023-12-19T09:19:40.039322+0000","last_fullsized":"2023-12-19T09:19:40.039322+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:00.211869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.014579+0000","last_change":"2023-12-19T09:19:39.011773+0000","last_active":"2023-12-19T09:19:39.014579+0000","last_peered":"2023-12-19T09:19:39.014579+0000","last_clean":"2023-12-19T09:19:39.014579+0000","last_became_active":"2023-12-19T09:19:39.011475+0000","last_became_peered":"2023-12-19T09:19:39.011475+0000","last_unstale":"2023-12-19T09:19:39.014579+0000","last_undegraded":"2023-12-19T09:19:39.014579+0000","last_fullsized":"2023-12-19T09:19:39.014579+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:02:37.365043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047191+0000","last_change":"2023-12-19T09:19:40.042070+0000","last_active":"2023-12-19T09:19:40.047191+0000","last_peered":"2023-12-19T09:19:40.047191+0000","last_clean":"2023-12-19T09:19:40.047191+0000","last_became_active":"2023-12-19T09:19:40.036700+0000","last_became_peered":"2023-12-19T09:19:40.036700+0000","last_unstale":"2023-12-19T09:19:40.047191+0000","last_undegraded":"2023-12-19T09:19:40.047191+0000","last_fullsized":"2023-12-19T09:19:40.047191+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:47.129114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.033087+0000","last_change":"2023-12-19T09:19:39.018218+0000","last_active":"2023-12-19T09:19:40.033087+0000","last_peered":"2023-12-19T09:19:40.033087+0000","last_clean":"2023-12-19T09:19:40.033087+0000","last_became_active":"2023-12-19T09:19:39.016386+0000","last_became_peered":"2023-12-19T09:19:39.016386+0000","last_unstale":"2023-12-19T09:19:40.033087+0000","last_undegraded":"2023-12-19T09:19:40.033087+0000","last_fullsized":"2023-12-19T09:19:40.033087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:18.791303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.054846+0000","last_change":"2023-12-19T09:19:40.040412+0000","last_active":"2023-12-19T09:19:40.054846+0000","last_peered":"2023-12-19T09:19:40.054846+0000","last_clean":"2023-12-19T09:19:40.054846+0000","last_became_active":"2023-12-19T09:19:40.040135+0000","last_became_peered":"2023-12-19T09:19:40.040135+0000","last_unstale":"2023-12-19T09:19:40.054846+0000","last_undegraded":"2023-12-19T09:19:40.054846+0000","last_fullsized":"2023-12-19T09:19:40.054846+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:47:08.534346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"192'3","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047532+0000","last_change":"2023-12-19T09:19:39.011515+0000","last_active":"2023-12-19T09:19:40.047532+0000","last_peered":"2023-12-19T09:19:40.047532+0000","last_clean":"2023-12-19T09:19:40.047532+0000","last_became_active":"2023-12-19T09:19:39.010744+0000","last_became_peered":"2023-12-19T09:19:39.010744+0000","last_unstale":"2023-12-19T09:19:40.047532+0000","last_undegraded":"2023-12-19T09:19:40.047532+0000","last_fullsized":"2023-12-19T09:19:40.047532+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:50:50.379115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041169+0000","last_change":"2023-12-19T09:19:40.041169+0000","last_active":"2023-12-19T09:19:40.041169+0000","last_peered":"2023-12-19T09:19:40.041169+0000","last_clean":"2023-12-19T09:19:40.041169+0000","last_became_active":"2023-12-19T09:19:40.032140+0000","last_became_peered":"2023-12-19T09:19:40.032140+0000","last_unstale":"2023-12-19T09:19:40.041169+0000","last_undegraded":"2023-12-19T09:19:40.041169+0000","last_fullsized":"2023-12-19T09:19:40.041169+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:01:15.147747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"192'6","reported_seq":16,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.060156+0000","last_change":"2023-12-19T09:19:39.016988+0000","last_active":"2023-12-19T09:19:40.060156+0000","last_peered":"2023-12-19T09:19:40.060156+0000","last_clean":"2023-12-19T09:19:40.060156+0000","last_became_active":"2023-12-19T09:19:39.016713+0000","last_became_peered":"2023-12-19T09:19:39.016713+0000","last_unstale":"2023-12-19T09:19:40.060156+0000","last_undegraded":"2023-12-19T09:19:40.060156+0000","last_fullsized":"2023-12-19T09:19:40.060156+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:15.866350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.032731+0000","last_change":"2023-12-19T09:19:40.032731+0000","last_active":"2023-12-19T09:19:40.032731+0000","last_peered":"2023-12-19T09:19:40.032731+0000","last_clean":"2023-12-19T09:19:40.032731+0000","last_became_active":"2023-12-19T09:19:40.031948+0000","last_became_peered":"2023-12-19T09:19:40.031948+0000","last_unstale":"2023-12-19T09:19:40.032731+0000","last_undegraded":"2023-12-19T09:19:40.032731+0000","last_fullsized":"2023-12-19T09:19:40.032731+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:15.715296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.051098+0000","last_change":"2023-12-19T09:19:39.012438+0000","last_active":"2023-12-19T09:19:40.051098+0000","last_peered":"2023-12-19T09:19:40.051098+0000","last_clean":"2023-12-19T09:19:40.051098+0000","last_became_active":"2023-12-19T09:19:39.012166+0000","last_became_peered":"2023-12-19T09:19:39.012166+0000","last_unstale":"2023-12-19T09:19:40.051098+0000","last_undegraded":"2023-12-19T09:19:40.051098+0000","last_fullsized":"2023-12-19T09:19:40.051098+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:06:40.451458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.040480+0000","last_change":"2023-12-19T09:19:40.040480+0000","last_active":"2023-12-19T09:19:40.040480+0000","last_peered":"2023-12-19T09:19:40.040480+0000","last_clean":"2023-12-19T09:19:40.040480+0000","last_became_active":"2023-12-19T09:19:40.040204+0000","last_became_peered":"2023-12-19T09:19:40.040204+0000","last_unstale":"2023-12-19T09:19:40.040480+0000","last_undegraded":"2023-12-19T09:19:40.040480+0000","last_fullsized":"2023-12-19T09:19:40.040480+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:20.262991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023463+0000","last_change":"2023-12-19T09:19:39.017955+0000","last_active":"2023-12-19T09:19:40.023463+0000","last_peered":"2023-12-19T09:19:40.023463+0000","last_clean":"2023-12-19T09:19:40.023463+0000","last_became_active":"2023-12-19T09:19:39.017503+0000","last_became_peered":"2023-12-19T09:19:39.017503+0000","last_unstale":"2023-12-19T09:19:40.023463+0000","last_undegraded":"2023-12-19T09:19:40.023463+0000","last_fullsized":"2023-12-19T09:19:40.023463+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:00:14.731908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.024864+0000","last_change":"2023-12-19T09:19:40.024864+0000","last_active":"2023-12-19T09:19:40.024864+0000","last_peered":"2023-12-19T09:19:40.024864+0000","last_clean":"2023-12-19T09:19:40.024864+0000","last_became_active":"2023-12-19T09:19:40.024505+0000","last_became_peered":"2023-12-19T09:19:40.024505+0000","last_unstale":"2023-12-19T09:19:40.024864+0000","last_undegraded":"2023-12-19T09:19:40.024864+0000","last_fullsized":"2023-12-19T09:19:40.024864+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:19.591882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.025174+0000","last_change":"2023-12-19T09:19:39.024477+0000","last_active":"2023-12-19T09:19:39.025174+0000","last_peered":"2023-12-19T09:19:39.025174+0000","last_clean":"2023-12-19T09:19:39.025174+0000","last_became_active":"2023-12-19T09:19:39.024192+0000","last_became_peered":"2023-12-19T09:19:39.024192+0000","last_unstale":"2023-12-19T09:19:39.025174+0000","last_undegraded":"2023-12-19T09:19:39.025174+0000","last_fullsized":"2023-12-19T09:19:39.025174+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:55.325042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050013+0000","last_change":"2023-12-19T09:19:40.050013+0000","last_active":"2023-12-19T09:19:40.050013+0000","last_peered":"2023-12-19T09:19:40.050013+0000","last_clean":"2023-12-19T09:19:40.050013+0000","last_became_active":"2023-12-19T09:19:40.048302+0000","last_became_peered":"2023-12-19T09:19:40.048302+0000","last_unstale":"2023-12-19T09:19:40.050013+0000","last_undegraded":"2023-12-19T09:19:40.050013+0000","last_fullsized":"2023-12-19T09:19:40.050013+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:12:00.735401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"192'8","reported_seq":17,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050316+0000","last_change":"2023-12-19T09:19:39.011136+0000","last_active":"2023-12-19T09:19:40.050316+0000","last_peered":"2023-12-19T09:19:40.050316+0000","last_clean":"2023-12-19T09:19:40.050316+0000","last_became_active":"2023-12-19T09:19:39.010624+0000","last_became_peered":"2023-12-19T09:19:39.010624+0000","last_unstale":"2023-12-19T09:19:40.050316+0000","last_undegraded":"2023-12-19T09:19:40.050316+0000","last_fullsized":"2023-12-19T09:19:40.050316+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:06:22.244327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047476+0000","last_change":"2023-12-19T09:19:40.047476+0000","last_active":"2023-12-19T09:19:40.047476+0000","last_peered":"2023-12-19T09:19:40.047476+0000","last_clean":"2023-12-19T09:19:40.047476+0000","last_became_active":"2023-12-19T09:19:40.047229+0000","last_became_peered":"2023-12-19T09:19:40.047229+0000","last_unstale":"2023-12-19T09:19:40.047476+0000","last_undegraded":"2023-12-19T09:19:40.047476+0000","last_fullsized":"2023-12-19T09:19:40.047476+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:32:47.920865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"192'1","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057696+0000","last_change":"2023-12-19T09:19:39.018956+0000","last_active":"2023-12-19T09:19:40.057696+0000","last_peered":"2023-12-19T09:19:40.057696+0000","last_clean":"2023-12-19T09:19:40.057696+0000","last_became_active":"2023-12-19T09:19:39.016088+0000","last_became_peered":"2023-12-19T09:19:39.016088+0000","last_unstale":"2023-12-19T09:19:40.057696+0000","last_undegraded":"2023-12-19T09:19:40.057696+0000","last_fullsized":"2023-12-19T09:19:40.057696+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:31:21.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036597+0000","last_change":"2023-12-19T09:19:40.036597+0000","last_active":"2023-12-19T09:19:40.036597+0000","last_peered":"2023-12-19T09:19:40.036597+0000","last_clean":"2023-12-19T09:19:40.036597+0000","last_became_active":"2023-12-19T09:19:40.036322+0000","last_became_peered":"2023-12-19T09:19:40.036322+0000","last_unstale":"2023-12-19T09:19:40.036597+0000","last_undegraded":"2023-12-19T09:19:40.036597+0000","last_fullsized":"2023-12-19T09:19:40.036597+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:54:29.533414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057909+0000","last_change":"2023-12-19T09:19:39.015824+0000","last_active":"2023-12-19T09:19:40.057909+0000","last_peered":"2023-12-19T09:19:40.057909+0000","last_clean":"2023-12-19T09:19:40.057909+0000","last_became_active":"2023-12-19T09:19:39.014088+0000","last_became_peered":"2023-12-19T09:19:39.014088+0000","last_unstale":"2023-12-19T09:19:40.057909+0000","last_undegraded":"2023-12-19T09:19:40.057909+0000","last_fullsized":"2023-12-19T09:19:40.057909+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:38:15.427681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.028486+0000","last_change":"2023-12-19T09:19:40.028486+0000","last_active":"2023-12-19T09:19:40.028486+0000","last_peered":"2023-12-19T09:19:40.028486+0000","last_clean":"2023-12-19T09:19:40.028486+0000","last_became_active":"2023-12-19T09:19:40.028108+0000","last_became_peered":"2023-12-19T09:19:40.028108+0000","last_unstale":"2023-12-19T09:19:40.028486+0000","last_undegraded":"2023-12-19T09:19:40.028486+0000","last_fullsized":"2023-12-19T09:19:40.028486+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:13:06.999965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"192'1","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.011148+0000","last_change":"2023-12-19T09:19:39.008233+0000","last_active":"2023-12-19T09:19:40.011148+0000","last_peered":"2023-12-19T09:19:40.011148+0000","last_clean":"2023-12-19T09:19:40.011148+0000","last_became_active":"2023-12-19T09:19:39.007929+0000","last_became_peered":"2023-12-19T09:19:39.007929+0000","last_unstale":"2023-12-19T09:19:40.011148+0000","last_undegraded":"2023-12-19T09:19:40.011148+0000","last_fullsized":"2023-12-19T09:19:40.011148+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:13:05.985380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023538+0000","last_change":"2023-12-19T09:19:40.023538+0000","last_active":"2023-12-19T09:19:40.023538+0000","last_peered":"2023-12-19T09:19:40.023538+0000","last_clean":"2023-12-19T09:19:40.023538+0000","last_became_active":"2023-12-19T09:19:40.023109+0000","last_became_peered":"2023-12-19T09:19:40.023109+0000","last_unstale":"2023-12-19T09:19:40.023538+0000","last_undegraded":"2023-12-19T09:19:40.023538+0000","last_fullsized":"2023-12-19T09:19:40.023538+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:29:31.467181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.027050+0000","last_change":"2023-12-19T09:19:39.026198+0000","last_active":"2023-12-19T09:19:39.027050+0000","last_peered":"2023-12-19T09:19:39.027050+0000","last_clean":"2023-12-19T09:19:39.027050+0000","last_became_active":"2023-12-19T09:19:39.025847+0000","last_became_peered":"2023-12-19T09:19:39.025847+0000","last_unstale":"2023-12-19T09:19:39.027050+0000","last_undegraded":"2023-12-19T09:19:39.027050+0000","last_fullsized":"2023-12-19T09:19:39.027050+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:41:08.209309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038869+0000","last_change":"2023-12-19T09:19:40.033961+0000","last_active":"2023-12-19T09:19:40.038869+0000","last_peered":"2023-12-19T09:19:40.038869+0000","last_clean":"2023-12-19T09:19:40.038869+0000","last_became_active":"2023-12-19T09:19:40.033421+0000","last_became_peered":"2023-12-19T09:19:40.033421+0000","last_unstale":"2023-12-19T09:19:40.038869+0000","last_undegraded":"2023-12-19T09:19:40.038869+0000","last_fullsized":"2023-12-19T09:19:40.038869+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:47:37.371893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.055376+0000","last_change":"2023-12-19T09:19:39.000781+0000","last_active":"2023-12-19T09:19:40.055376+0000","last_peered":"2023-12-19T09:19:40.055376+0000","last_clean":"2023-12-19T09:19:40.055376+0000","last_became_active":"2023-12-19T09:19:39.000456+0000","last_became_peered":"2023-12-19T09:19:39.000456+0000","last_unstale":"2023-12-19T09:19:40.055376+0000","last_undegraded":"2023-12-19T09:19:40.055376+0000","last_fullsized":"2023-12-19T09:19:40.055376+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:13:18.577297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.051430+0000","last_change":"2023-12-19T09:19:40.051430+0000","last_active":"2023-12-19T09:19:40.051430+0000","last_peered":"2023-12-19T09:19:40.051430+0000","last_clean":"2023-12-19T09:19:40.051430+0000","last_became_active":"2023-12-19T09:19:40.045606+0000","last_became_peered":"2023-12-19T09:19:40.045606+0000","last_unstale":"2023-12-19T09:19:40.051430+0000","last_undegraded":"2023-12-19T09:19:40.051430+0000","last_fullsized":"2023-12-19T09:19:40.051430+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:22:37.765631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.015732+0000","last_change":"2023-12-19T09:19:39.014180+0000","last_active":"2023-12-19T09:19:39.015732+0000","last_peered":"2023-12-19T09:19:39.015732+0000","last_clean":"2023-12-19T09:19:39.015732+0000","last_became_active":"2023-12-19T09:19:39.013496+0000","last_became_peered":"2023-12-19T09:19:39.013496+0000","last_unstale":"2023-12-19T09:19:39.015732+0000","last_undegraded":"2023-12-19T09:19:39.015732+0000","last_fullsized":"2023-12-19T09:19:39.015732+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:28:59.197284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.042893+0000","last_change":"2023-12-19T09:19:40.042893+0000","last_active":"2023-12-19T09:19:40.042893+0000","last_peered":"2023-12-19T09:19:40.042893+0000","last_clean":"2023-12-19T09:19:40.042893+0000","last_became_active":"2023-12-19T09:19:40.042561+0000","last_became_peered":"2023-12-19T09:19:40.042561+0000","last_unstale":"2023-12-19T09:19:40.042893+0000","last_undegraded":"2023-12-19T09:19:40.042893+0000","last_fullsized":"2023-12-19T09:19:40.042893+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:34:43.610486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.012289+0000","last_change":"2023-12-19T09:19:39.009686+0000","last_active":"2023-12-19T09:19:39.012289+0000","last_peered":"2023-12-19T09:19:39.012289+0000","last_clean":"2023-12-19T09:19:39.012289+0000","last_became_active":"2023-12-19T09:19:39.009388+0000","last_became_peered":"2023-12-19T09:19:39.009388+0000","last_unstale":"2023-12-19T09:19:39.012289+0000","last_undegraded":"2023-12-19T09:19:39.012289+0000","last_fullsized":"2023-12-19T09:19:39.012289+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:26:32.945772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.057069+0000","last_change":"2023-12-19T09:19:40.056428+0000","last_active":"2023-12-19T09:19:40.057069+0000","last_peered":"2023-12-19T09:19:40.057069+0000","last_clean":"2023-12-19T09:19:40.057069+0000","last_became_active":"2023-12-19T09:19:40.056163+0000","last_became_peered":"2023-12-19T09:19:40.056163+0000","last_unstale":"2023-12-19T09:19:40.057069+0000","last_undegraded":"2023-12-19T09:19:40.057069+0000","last_fullsized":"2023-12-19T09:19:40.057069+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:51:49.619422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.022374+0000","last_change":"2023-12-19T09:19:39.006915+0000","last_active":"2023-12-19T09:19:40.022374+0000","last_peered":"2023-12-19T09:19:40.022374+0000","last_clean":"2023-12-19T09:19:40.022374+0000","last_became_active":"2023-12-19T09:19:39.006628+0000","last_became_peered":"2023-12-19T09:19:39.006628+0000","last_unstale":"2023-12-19T09:19:40.022374+0000","last_undegraded":"2023-12-19T09:19:40.022374+0000","last_fullsized":"2023-12-19T09:19:40.022374+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:05:02.010995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.038404+0000","last_change":"2023-12-19T09:19:40.038404+0000","last_active":"2023-12-19T09:19:40.038404+0000","last_peered":"2023-12-19T09:19:40.038404+0000","last_clean":"2023-12-19T09:19:40.038404+0000","last_became_active":"2023-12-19T09:19:40.037814+0000","last_became_peered":"2023-12-19T09:19:40.037814+0000","last_unstale":"2023-12-19T09:19:40.038404+0000","last_undegraded":"2023-12-19T09:19:40.038404+0000","last_fullsized":"2023-12-19T09:19:40.038404+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:56.085814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"192'2","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.041498+0000","last_change":"2023-12-19T09:19:39.013361+0000","last_active":"2023-12-19T09:19:40.041498+0000","last_peered":"2023-12-19T09:19:40.041498+0000","last_clean":"2023-12-19T09:19:40.041498+0000","last_became_active":"2023-12-19T09:19:39.013103+0000","last_became_peered":"2023-12-19T09:19:39.013103+0000","last_unstale":"2023-12-19T09:19:40.041498+0000","last_undegraded":"2023-12-19T09:19:40.041498+0000","last_fullsized":"2023-12-19T09:19:40.041498+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:09:48.868069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043859+0000","last_change":"2023-12-19T09:19:40.043859+0000","last_active":"2023-12-19T09:19:40.043859+0000","last_peered":"2023-12-19T09:19:40.043859+0000","last_clean":"2023-12-19T09:19:40.043859+0000","last_became_active":"2023-12-19T09:19:40.043146+0000","last_became_peered":"2023-12-19T09:19:40.043146+0000","last_unstale":"2023-12-19T09:19:40.043859+0000","last_undegraded":"2023-12-19T09:19:40.043859+0000","last_fullsized":"2023-12-19T09:19:40.043859+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:20:07.813243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.031964+0000","last_change":"2023-12-19T09:19:39.016260+0000","last_active":"2023-12-19T09:19:40.031964+0000","last_peered":"2023-12-19T09:19:40.031964+0000","last_clean":"2023-12-19T09:19:40.031964+0000","last_became_active":"2023-12-19T09:19:39.015859+0000","last_became_peered":"2023-12-19T09:19:39.015859+0000","last_unstale":"2023-12-19T09:19:40.031964+0000","last_undegraded":"2023-12-19T09:19:40.031964+0000","last_fullsized":"2023-12-19T09:19:40.031964+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:18.195216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.036730+0000","last_change":"2023-12-19T09:19:40.036730+0000","last_active":"2023-12-19T09:19:40.036730+0000","last_peered":"2023-12-19T09:19:40.036730+0000","last_clean":"2023-12-19T09:19:40.036730+0000","last_became_active":"2023-12-19T09:19:40.036434+0000","last_became_peered":"2023-12-19T09:19:40.036434+0000","last_unstale":"2023-12-19T09:19:40.036730+0000","last_undegraded":"2023-12-19T09:19:40.036730+0000","last_fullsized":"2023-12-19T09:19:40.036730+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:43:48.021826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.034187+0000","last_change":"2023-12-19T09:19:39.005750+0000","last_active":"2023-12-19T09:19:40.034187+0000","last_peered":"2023-12-19T09:19:40.034187+0000","last_clean":"2023-12-19T09:19:40.034187+0000","last_became_active":"2023-12-19T09:19:39.005370+0000","last_became_peered":"2023-12-19T09:19:39.005370+0000","last_unstale":"2023-12-19T09:19:40.034187+0000","last_undegraded":"2023-12-19T09:19:40.034187+0000","last_fullsized":"2023-12-19T09:19:40.034187+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:56:31.583174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.050171+0000","last_change":"2023-12-19T09:19:40.050171+0000","last_active":"2023-12-19T09:19:40.050171+0000","last_peered":"2023-12-19T09:19:40.050171+0000","last_clean":"2023-12-19T09:19:40.050171+0000","last_became_active":"2023-12-19T09:19:40.049868+0000","last_became_peered":"2023-12-19T09:19:40.049868+0000","last_unstale":"2023-12-19T09:19:40.050171+0000","last_undegraded":"2023-12-19T09:19:40.050171+0000","last_fullsized":"2023-12-19T09:19:40.050171+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:57:42.144971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.017934+0000","last_change":"2023-12-19T09:19:39.014451+0000","last_active":"2023-12-19T09:19:39.017934+0000","last_peered":"2023-12-19T09:19:39.017934+0000","last_clean":"2023-12-19T09:19:39.017934+0000","last_became_active":"2023-12-19T09:19:39.014153+0000","last_became_peered":"2023-12-19T09:19:39.014153+0000","last_unstale":"2023-12-19T09:19:39.017934+0000","last_undegraded":"2023-12-19T09:19:39.017934+0000","last_fullsized":"2023-12-19T09:19:39.017934+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:56:59.546403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.039694+0000","last_change":"2023-12-19T09:19:40.035881+0000","last_active":"2023-12-19T09:19:40.039694+0000","last_peered":"2023-12-19T09:19:40.039694+0000","last_clean":"2023-12-19T09:19:40.039694+0000","last_became_active":"2023-12-19T09:19:40.035589+0000","last_became_peered":"2023-12-19T09:19:40.035589+0000","last_unstale":"2023-12-19T09:19:40.039694+0000","last_undegraded":"2023-12-19T09:19:40.039694+0000","last_fullsized":"2023-12-19T09:19:40.039694+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:26:03.698560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.027757+0000","last_change":"2023-12-19T09:19:39.020965+0000","last_active":"2023-12-19T09:19:40.027757+0000","last_peered":"2023-12-19T09:19:40.027757+0000","last_clean":"2023-12-19T09:19:40.027757+0000","last_became_active":"2023-12-19T09:19:39.020693+0000","last_became_peered":"2023-12-19T09:19:39.020693+0000","last_unstale":"2023-12-19T09:19:40.027757+0000","last_undegraded":"2023-12-19T09:19:40.027757+0000","last_fullsized":"2023-12-19T09:19:40.027757+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:03:14.074567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047161+0000","last_change":"2023-12-19T09:19:40.041025+0000","last_active":"2023-12-19T09:19:40.047161+0000","last_peered":"2023-12-19T09:19:40.047161+0000","last_clean":"2023-12-19T09:19:40.047161+0000","last_became_active":"2023-12-19T09:19:40.040749+0000","last_became_peered":"2023-12-19T09:19:40.040749+0000","last_unstale":"2023-12-19T09:19:40.047161+0000","last_undegraded":"2023-12-19T09:19:40.047161+0000","last_fullsized":"2023-12-19T09:19:40.047161+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:04:55.353273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.023565+0000","last_change":"2023-12-19T09:19:39.016542+0000","last_active":"2023-12-19T09:19:40.023565+0000","last_peered":"2023-12-19T09:19:40.023565+0000","last_clean":"2023-12-19T09:19:40.023565+0000","last_became_active":"2023-12-19T09:19:39.016215+0000","last_became_peered":"2023-12-19T09:19:39.016215+0000","last_unstale":"2023-12-19T09:19:40.023565+0000","last_undegraded":"2023-12-19T09:19:40.023565+0000","last_fullsized":"2023-12-19T09:19:40.023565+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:46.114529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.056215+0000","last_change":"2023-12-19T09:19:40.045357+0000","last_active":"2023-12-19T09:19:40.056215+0000","last_peered":"2023-12-19T09:19:40.056215+0000","last_clean":"2023-12-19T09:19:40.056215+0000","last_became_active":"2023-12-19T09:19:40.045074+0000","last_became_peered":"2023-12-19T09:19:40.045074+0000","last_unstale":"2023-12-19T09:19:40.056215+0000","last_undegraded":"2023-12-19T09:19:40.056215+0000","last_fullsized":"2023-12-19T09:19:40.056215+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:22:24.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.010111+0000","last_change":"2023-12-19T09:19:39.006154+0000","last_active":"2023-12-19T09:19:39.010111+0000","last_peered":"2023-12-19T09:19:39.010111+0000","last_clean":"2023-12-19T09:19:39.010111+0000","last_became_active":"2023-12-19T09:19:39.004886+0000","last_became_peered":"2023-12-19T09:19:39.004886+0000","last_unstale":"2023-12-19T09:19:39.010111+0000","last_undegraded":"2023-12-19T09:19:39.010111+0000","last_fullsized":"2023-12-19T09:19:39.010111+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:45:15.593827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":8,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.043748+0000","last_change":"2023-12-19T09:19:40.043748+0000","last_active":"2023-12-19T09:19:40.043748+0000","last_peered":"2023-12-19T09:19:40.043748+0000","last_clean":"2023-12-19T09:19:40.043748+0000","last_became_active":"2023-12-19T09:19:40.043425+0000","last_became_peered":"2023-12-19T09:19:40.043425+0000","last_unstale":"2023-12-19T09:19:40.043748+0000","last_undegraded":"2023-12-19T09:19:40.043748+0000","last_fullsized":"2023-12-19T09:19:40.043748+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:33:03.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-12-19T09:19:39.024029+0000","last_change":"2023-12-19T09:19:39.016725+0000","last_active":"2023-12-19T09:19:39.024029+0000","last_peered":"2023-12-19T09:19:39.024029+0000","last_clean":"2023-12-19T09:19:39.024029+0000","last_became_active":"2023-12-19T09:19:39.009371+0000","last_became_peered":"2023-12-19T09:19:39.009371+0000","last_unstale":"2023-12-19T09:19:39.024029+0000","last_undegraded":"2023-12-19T09:19:39.024029+0000","last_fullsized":"2023-12-19T09:19:39.024029+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:57:55.071229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.047227+0000","last_change":"2023-12-19T09:19:40.042064+0000","last_active":"2023-12-19T09:19:40.047227+0000","last_peered":"2023-12-19T09:19:40.047227+0000","last_clean":"2023-12-19T09:19:40.047227+0000","last_became_active":"2023-12-19T09:19:40.040686+0000","last_became_peered":"2023-12-19T09:19:40.040686+0000","last_unstale":"2023-12-19T09:19:40.047227+0000","last_undegraded":"2023-12-19T09:19:40.047227+0000","last_fullsized":"2023-12-19T09:19:40.047227+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:38.975716+0000","last_clean_scrub_stamp":"2023-12-19T09:19:38.975716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:30:06.729521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.026644+0000","last_change":"2023-12-19T09:19:39.017482+0000","last_active":"2023-12-19T09:19:40.026644+0000","last_peered":"2023-12-19T09:19:40.026644+0000","last_clean":"2023-12-19T09:19:40.026644+0000","last_became_active":"2023-12-19T09:19:39.016213+0000","last_became_peered":"2023-12-19T09:19:39.016213+0000","last_unstale":"2023-12-19T09:19:40.026644+0000","last_undegraded":"2023-12-19T09:19:40.026644+0000","last_fullsized":"2023-12-19T09:19:40.026644+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:19:37.961130+0000","last_clean_scrub_stamp":"2023-12-19T09:19:37.961130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:18:52.198439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"182'326","reported_seq":1046,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-12-19T09:19:40.022492+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:19:40.022492+0000","last_peered":"2023-12-19T09:19:40.022492+0000","last_clean":"2023-12-19T09:19:40.022492+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:19:40.022492+0000","last_undegraded":"2023-12-19T09:19:40.022492+0000","last_fullsized":"2023-12-19T09:19:40.022492+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304312,"kb_used_data":2928,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067528,"statfs":{"total":96636764160,"available":96325148672,"internally_reserved":0,"allocated":2998272,"data_stored":1878409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2829999999999999,"15min":1.286},"min":{"1min":0.84999999999999998,"5min":0.72699999999999998,"15min":0.52600000000000002},"max":{"1min":1.5960000000000001,"5min":1.8360000000000001,"15min":2.0470000000000002},"last":1.444},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.276,"15min":1.296},"min":{"1min":1.0289999999999999,"5min":0.871,"15min":0.55700000000000005},"max":{"1min":1.591,"5min":1.6299999999999999,"15min":1.712},"last":1.3160000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1799999999999999,"15min":1.167},"min":{"1min":0.94299999999999995,"5min":0.747,"15min":0.747},"max":{"1min":1.385,"5min":1.615,"15min":1.698},"last":1.117},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2130000000000001,"15min":1.2090000000000001},"min":{"1min":0.90900000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.474,"5min":1.671,"15min":1.796},"last":1.3300000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1819999999999999,"15min":1.1779999999999999},"min":{"1min":0.94399999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.3360000000000001,"5min":1.681,"15min":1.681},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2909999999999999,"15min":1.2849999999999999},"min":{"1min":0.97599999999999998,"5min":0.84599999999999997,"15min":0.71999999999999997},"max":{"1min":1.681,"5min":1.754,"15min":2.1400000000000001},"last":1.1830000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3029999999999999,"15min":1.3049999999999999},"min":{"1min":0.93200000000000005,"5min":0.90600000000000003,"15min":0.70599999999999996},"max":{"1min":1.522,"5min":1.6799999999999999,"15min":1.9059999999999999},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.3740000000000001,"15min":1.379},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.6120000000000001,"5min":1.8480000000000001,"15min":1.8700000000000001},"last":1.498}]},{"osd":4,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.208,"15min":1.244},"min":{"1min":0.84699999999999998,"5min":0.78800000000000003,"15min":0.77000000000000002},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.95},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.2230000000000001,"15min":1.238},"min":{"1min":0.98099999999999998,"5min":0.70399999999999996,"15min":0.59899999999999998},"max":{"1min":1.5489999999999999,"5min":1.7609999999999999,"15min":1.788},"last":1.3680000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.268,"15min":1.294},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6679999999999999,"5min":1.835,"15min":1.9339999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.375,"5min":1.3899999999999999,"15min":1.4159999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.81899999999999995},"max":{"1min":1.804,"5min":1.901,"15min":1.901},"last":1.5489999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:18:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.393,"15min":1.411},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.732,"5min":1.9950000000000001,"15min":2.0750000000000002},"last":1.407},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.4039999999999999,"15min":1.3959999999999999},"min":{"1min":0.874,"5min":0.874,"15min":0.752},"max":{"1min":1.734,"5min":1.8979999999999999,"15min":2.238},"last":1.268}]}]},{"osd":6,"up_from":13,"seq":55834575948,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311924,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059916,"statfs":{"total":96636764160,"available":96317353984,"internally_reserved":0,"allocated":2404352,"data_stored":1288041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1579999999999999,"15min":1.139},"min":{"1min":0.76300000000000001,"5min":0.73499999999999999,"15min":0.65300000000000002},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.784},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.153,"15min":1.167},"min":{"1min":0.90900000000000003,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.675,"5min":1.6850000000000001,"15min":1.946},"last":0.92100000000000004}]},{"osd":1,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.169,"15min":1.2050000000000001},"min":{"1min":0.72899999999999998,"5min":0.67600000000000005,"15min":0.63800000000000001},"max":{"1min":1.5980000000000001,"5min":1.6459999999999999,"15min":2.3439999999999999},"last":1.159},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.218,"15min":1.2030000000000001},"min":{"1min":0.83699999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.385}]},{"osd":2,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.274,"15min":1.2649999999999999},"min":{"1min":0.93899999999999995,"5min":0.86199999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.425},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2529999999999999,"15min":1.2549999999999999},"min":{"1min":0.92300000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.8100000000000001},"last":1.077}]},{"osd":3,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2649999999999999,"15min":1.298},"min":{"1min":0.92200000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.6100000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.331,"15min":1.331},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.68999999999999995},"max":{"1min":1.712,"5min":1.7130000000000001,"15min":1.8220000000000001},"last":1.6240000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.218,"15min":1.2250000000000001},"min":{"1min":0.84299999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.649,"5min":1.7609999999999999,"15min":1.893},"last":1.573},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":1.0509999999999999,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7929999999999999,"5min":1.831,"15min":2.1139999999999999},"last":1.5149999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2749999999999999,"15min":1.2949999999999999},"min":{"1min":1.006,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.694,"5min":1.7749999999999999,"15min":1.921},"last":1.538},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":0.81000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6719999999999999,"5min":1.748,"15min":1.8959999999999999},"last":1.667}]},{"osd":7,"last update":"Tue Dec 19 09:19:02 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.375,"15min":1.387},"min":{"1min":0.94399999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.728,"5min":1.903,"15min":1.962},"last":1.226},{"interface":"front","average":{"1min":1.454,"5min":1.444,"15min":1.4510000000000001},"min":{"1min":1.0609999999999999,"5min":0.79100000000000004,"15min":0.72599999999999998},"max":{"1min":1.661,"5min":1.8939999999999999,"15min":2.2410000000000001},"last":1.734}]}]},{"osd":5,"up_from":13,"seq":55834575948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307836,"kb_used_data":2356,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064004,"statfs":{"total":96636764160,"available":96321540096,"internally_reserved":0,"allocated":2412544,"data_stored":1288165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3169999999999999,"15min":1.3180000000000001},"min":{"1min":0.94499999999999995,"5min":0.80500000000000005,"15min":0.50700000000000001},"max":{"1min":1.8160000000000001,"5min":2.044,"15min":2.044},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.3420000000000001,"15min":1.3440000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.58199999999999996},"max":{"1min":1.671,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.5780000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.161,"15min":1.1739999999999999},"min":{"1min":0.80900000000000005,"5min":0.78800000000000003,"15min":0.76200000000000001},"max":{"1min":1.454,"5min":1.5569999999999999,"15min":1.6479999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1859999999999999,"15min":1.208},"min":{"1min":0.84799999999999998,"5min":0.74299999999999999,"15min":0.58099999999999996},"max":{"1min":1.4139999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.004}]},{"osd":2,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.2090000000000001,"15min":1.202},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.76800000000000002},"max":{"1min":1.5209999999999999,"5min":1.782,"15min":1.782},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4159999999999999,"5min":1.583,"15min":1.6180000000000001},"last":1.24}]},{"osd":3,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3029999999999999,"15min":1.304},"min":{"1min":0.91700000000000004,"5min":0.872,"15min":0.63700000000000001},"max":{"1min":1.625,"5min":1.7529999999999999,"15min":1.8180000000000001},"last":1.431},{"interface":"front","average":{"1min":1.357,"5min":1.3129999999999999,"15min":1.325},"min":{"1min":1.012,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.601,"5min":1.962,"15min":1.962},"last":1.4690000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.218,"15min":1.2330000000000001},"min":{"1min":0.79800000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.391},{"interface":"front","average":{"1min":1.323,"5min":1.3120000000000001,"15min":1.3},"min":{"1min":0.88800000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7390000000000001,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.827}]},{"osd":6,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.72799999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.617,"5min":1.746,"15min":1.8720000000000001},"last":1.677},{"interface":"front","average":{"1min":1.355,"5min":1.3600000000000001,"15min":1.3740000000000001},"min":{"1min":0.90400000000000003,"5min":0.76500000000000001,"15min":0.68999999999999995},"max":{"1min":1.7430000000000001,"5min":1.8169999999999999,"15min":1.8740000000000001},"last":1.619}]},{"osd":7,"last update":"Tue Dec 19 09:19:26 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.167,"5min":0.92000000000000004,"15min":0.84199999999999997},"max":{"1min":1.8720000000000001,"5min":1.8859999999999999,"15min":2.153},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.454,"15min":1.4730000000000001},"min":{"1min":1.1419999999999999,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8089999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.978}]}]},{"osd":4,"up_from":13,"seq":55834575948,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311952,"kb_used_data":2376,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059888,"statfs":{"total":96636764160,"available":96317325312,"internally_reserved":0,"allocated":2433024,"data_stored":1295961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.222,"15min":1.2390000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5860000000000001,"5min":1.645,"15min":1.796},"last":3.4420000000000002},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3500000000000001,"15min":1.3540000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.768,"5min":1.768,"15min":1.782},"last":3.4260000000000002}]},{"osd":1,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.24,"15min":1.208},"min":{"1min":0.91800000000000004,"5min":0.84099999999999997,"15min":0.70799999999999996},"max":{"1min":1.6339999999999999,"5min":1.758,"15min":1.758},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1699999999999999,"15min":1.1930000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.66600000000000004},"max":{"1min":1.456,"5min":1.5549999999999999,"15min":1.7050000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2170000000000001,"15min":1.2030000000000001},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.5800000000000001,"5min":1.833,"15min":1.833},"last":1.8839999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2909999999999999,"15min":1.304},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85099999999999998},"max":{"1min":1.7110000000000001,"5min":1.925,"15min":1.925},"last":1.5589999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.325,"15min":1.3129999999999999},"min":{"1min":0.92500000000000004,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.6619999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.643},{"interface":"front","average":{"1min":1.405,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.671,"5min":2.4300000000000002,"15min":2.4300000000000002},"last":1.7150000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2270000000000001,"15min":1.2689999999999999},"min":{"1min":0.88500000000000001,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.796,"5min":1.796,"15min":2.0179999999999998},"last":1.528},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2470000000000001,"15min":1.2629999999999999},"min":{"1min":0.91500000000000004,"5min":0.83999999999999997,"15min":0.68700000000000006},"max":{"1min":1.571,"5min":1.879,"15min":1.879},"last":1.73}]},{"osd":6,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.286,"15min":1.2749999999999999},"min":{"1min":0.83799999999999997,"5min":0.753,"15min":0.753},"max":{"1min":1.6499999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.7609999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.4259999999999999,"15min":1.425},"min":{"1min":1.0720000000000001,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.9379999999999999},"last":1.478}]},{"osd":7,"last update":"Tue Dec 19 09:19:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.369,"15min":1.367},"min":{"1min":0.94799999999999995,"5min":0.89000000000000001,"15min":0.86599999999999999},"max":{"1min":1.762,"5min":1.7749999999999999,"15min":1.9730000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4630000000000001,"15min":1.4570000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91600000000000004},"max":{"1min":1.752,"5min":1.9219999999999999,"15min":1.9990000000000001},"last":1.847}]}]},{"osd":3,"up_from":13,"seq":55834575948,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311956,"kb_used_data":2380,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059884,"statfs":{"total":96636764160,"available":96317321216,"internally_reserved":0,"allocated":2437120,"data_stored":1297337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88,"15min":0.875},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.40600000000000003},"max":{"1min":1.4670000000000001,"5min":1.581,"15min":1.581},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.01,"15min":0.99299999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5249999999999999,"5min":2.117,"15min":2.117},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.91700000000000004,"15min":0.93700000000000006},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1539999999999999,"5min":2.331,"15min":2.331},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.077,"15min":1.0620000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":0.85299999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.139,"15min":1.127},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.7130000000000001,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.181,"15min":1.1910000000000001},"min":{"1min":0.875,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.419,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.93799999999999994}]},{"osd":4,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.3480000000000001,"15min":1.3129999999999999},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.70199999999999996},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.204},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.2809999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.0029999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.3129999999999999,"15min":1.3109999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.3919999999999999,"15min":1.3859999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2470000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.421,"15min":1.423},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.157},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4399999999999999,"15min":1.425},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.825,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.355}]},{"osd":7,"last update":"Tue Dec 19 09:19:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.53,"15min":1.5},"min":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":2.2749999999999999,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.47,"15min":1.452},"min":{"1min":0.96499999999999997,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.1389999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.05}]}]},{"osd":2,"up_from":13,"seq":55834575948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307828,"kb_used_data":2348,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94064012,"statfs":{"total":96636764160,"available":96321548288,"internally_reserved":0,"allocated":2404352,"data_stored":1288041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93600000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.58899999999999997,"15min":0.55100000000000005},"max":{"1min":1.3300000000000001,"5min":1.4670000000000001,"15min":1.5049999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.1339999999999999,"15min":1.117},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.71899999999999997},"max":{"1min":1.377,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":1.0249999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97599999999999998,"15min":0.995},"min":{"1min":0.747,"5min":0.73999999999999999,"15min":0.59699999999999998},"max":{"1min":1.569,"5min":1.569,"15min":1.7609999999999999},"last":0.623},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.083,"15min":1.0549999999999999},"min":{"1min":0.75,"5min":0.72099999999999997,"15min":0.63800000000000001},"max":{"1min":1.5089999999999999,"5min":1.768,"15min":1.768},"last":0.94599999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.0169999999999999,"15min":0.98299999999999998},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.54100000000000004},"max":{"1min":1.3120000000000001,"5min":1.891,"15min":1.891},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.171,"5min":1.1899999999999999,"15min":1.1659999999999999},"min":{"1min":0.89100000000000001,"5min":0.85799999999999998,"15min":0.72899999999999998},"max":{"1min":1.425,"5min":1.54,"15min":1.5589999999999999},"last":0.97699999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.365,"15min":1.3580000000000001},"min":{"1min":1.079,"5min":0.94999999999999996,"15min":0.79000000000000004},"max":{"1min":1.5640000000000001,"5min":1.754,"15min":1.754},"last":1.24},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3620000000000001,"15min":1.3400000000000001},"min":{"1min":1.111,"5min":0.92300000000000004,"15min":0.82099999999999995},"max":{"1min":1.7450000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.175}]},{"osd":5,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.401,"15min":1.395},"min":{"1min":1.139,"5min":1.0680000000000001,"15min":0.90500000000000003},"max":{"1min":1.631,"5min":2.016,"15min":2.016},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3300000000000001,"15min":1.3140000000000001},"min":{"1min":1.028,"5min":0.91400000000000003,"15min":0.68300000000000005},"max":{"1min":1.6319999999999999,"5min":1.901,"15min":1.927},"last":1.1990000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3460000000000001,"15min":1.3520000000000001},"min":{"1min":1.0580000000000001,"5min":1.014,"15min":0.89000000000000001},"max":{"1min":1.6679999999999999,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4299999999999999,"15min":1.4390000000000001},"min":{"1min":1.1870000000000001,"5min":1.1040000000000001,"15min":0.92100000000000004},"max":{"1min":1.6919999999999999,"5min":1.8580000000000001,"15min":2.1909999999999998},"last":1.3340000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.462,"15min":1.448},"min":{"1min":1.075,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":1.8999999999999999,"5min":2.327,"15min":2.327},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.4450000000000001,"15min":1.4350000000000001},"min":{"1min":1.1080000000000001,"5min":1.04,"15min":0.81100000000000005},"max":{"1min":1.665,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.3180000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575949,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311936,"kb_used_data":2360,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059904,"statfs":{"total":96636764160,"available":96317341696,"internally_reserved":0,"allocated":2416640,"data_stored":1289577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.206,"5min":1.4239999999999999,"15min":1.9039999999999999},"last":0.873},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.99299999999999999,"15min":1.0009999999999999},"min":{"1min":0.63100000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.3120000000000001,"5min":1.575,"15min":1.6850000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94099999999999995,"15min":0.95399999999999996},"min":{"1min":0.69999999999999996,"5min":0.63200000000000001,"15min":0.61699999999999999},"max":{"1min":1.1819999999999999,"5min":2.331,"15min":2.331},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.0620000000000001,"15min":1.083},"min":{"1min":0.84899999999999998,"5min":0.68799999999999994,"15min":0.56999999999999995},"max":{"1min":1.45,"5min":2.302,"15min":2.302},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1839999999999999,"15min":1.181},"min":{"1min":0.85299999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.534,"5min":1.851,"15min":2.0950000000000002},"last":1},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.083,"15min":1.0900000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.694,"5min":1.694,"15min":1.76},"last":1.125}]},{"osd":4,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.24,"15min":1.246},"min":{"1min":1.0009999999999999,"5min":0.89600000000000002,"15min":0.76700000000000002},"max":{"1min":1.5880000000000001,"5min":1.7430000000000001,"15min":1.9670000000000001},"last":1.528},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3300000000000001,"15min":1.3320000000000001},"min":{"1min":1.111,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.708,"5min":1.708,"15min":2.0350000000000001},"last":1.379}]},{"osd":5,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.5649999999999999,"5min":1.921,"15min":2.1699999999999999},"last":1.409},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3029999999999999,"15min":1.3109999999999999},"min":{"1min":1.024,"5min":0.93899999999999995,"15min":0.84499999999999997},"max":{"1min":1.6419999999999999,"5min":1.6930000000000001,"15min":2.117},"last":1.4379999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.387,"15min":1.409},"min":{"1min":1.0920000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.6299999999999999,"5min":1.877,"15min":2},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.393,"15min":1.4199999999999999},"min":{"1min":0.85499999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.6739999999999999,"5min":2.0129999999999999,"15min":2.3690000000000002},"last":1.4970000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:18:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.452,"15min":1.4530000000000001},"min":{"1min":1.01,"5min":1.0029999999999999,"15min":0.97899999999999998},"max":{"1min":1.7470000000000001,"5min":1.8280000000000001,"15min":2.2869999999999999},"last":1.591},{"interface":"front","average":{"1min":1.417,"5min":1.4259999999999999,"15min":1.427},"min":{"1min":1.1459999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.73,"5min":1.9339999999999999,"15min":2.2360000000000002},"last":1.222}]}]},{"osd":0,"up_from":13,"seq":55834575948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300220,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071620,"statfs":{"total":96636764160,"available":96329338880,"internally_reserved":0,"allocated":3002368,"data_stored":1877502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1379999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.96799999999999997,"15min":0.97399999999999998},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3460000000000001,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.79200000000000004}]},{"osd":2,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0149999999999999,"15min":1.0289999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.321,"5min":1.446,"15min":1.5620000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.0649999999999999,"15min":1.0680000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.538,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0289999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.018,"15min":1.016},"min":{"1min":0.71399999999999997,"5min":0.68600000000000005,"15min":0.622},"max":{"1min":1.23,"5min":1.611,"15min":2.621},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.127,"5min":1.175,"15min":1.181},"min":{"1min":0.81399999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.425,"5min":1.5920000000000001,"15min":1.718},"last":0.98299999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.98299999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6020000000000001,"5min":1.891,"15min":1.891},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.3340000000000001,"15min":1.3480000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.69599999999999995},"max":{"1min":1.5800000000000001,"5min":1.919,"15min":2.5339999999999998},"last":1.167}]},{"osd":5,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.391,"15min":1.4039999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.286},{"interface":"front","average":{"1min":1.369,"5min":1.405,"15min":1.421},"min":{"1min":0.83499999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.542}]},{"osd":6,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.4489999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3700000000000001,"15min":1.3740000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0379999999999998,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.3819999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:19:35 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.3420000000000001,"15min":1.345},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.9770000000000001,"5min":2.137,"15min":2.8570000000000002},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4339999999999999,"15min":1.4399999999999999},"min":{"1min":0.92200000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.181},"last":1.345}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:19:44.720 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T09:19:44.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T09:19:44.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.921+0000 7fed1a28e700 1 Processor -- start 2023-12-19T09:19:44.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.921+0000 7fed1a28e700 1 -- start start 2023-12-19T09:19:44.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1a28e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed1406ae20 0x7fed1406b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1a28e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 0x7fed140b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1a28e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1406b730 con 0x7fed140ab6d0 2023-12-19T09:19:44.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1a28e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed140ac390 con 0x7fed1406ae20 2023-12-19T09:19:44.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1a28e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed140ac670 con 0x7fed140b1440 2023-12-19T09:19:44.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed18a8b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fed140ab6d0 0x7fed140abb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50550/0 (socket says 172.21.15.111:50550) 2023-12-19T09:19:44.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed18a8b700 1 -- 172.21.15.111:0/3029976867 learned_addr learned my addr 172.21.15.111:0/3029976867 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:44.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed1928c700 1 --2- 172.21.15.111:0/3029976867 >> v2:172.21.15.143:3300/0 conn(0x7fed1406ae20 0x7fed1406b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed19a8d700 1 --2- 172.21.15.111:0/3029976867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 0x7fed140b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed0bfff700 1 -- 172.21.15.111:0/3029976867 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481486300 0 0) 0x7fed1406b730 con 0x7fed140ab6d0 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.925+0000 7fed0bfff700 1 -- 172.21.15.111:0/3029976867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed00003570 con 0x7fed140ab6d0 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed19a8d700 1 -- 172.21.15.111:0/3029976867 >> v2:172.21.15.143:3300/0 conn(0x7fed1406ae20 msgr2=0x7fed1406b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed19a8d700 1 --2- 172.21.15.111:0/3029976867 >> v2:172.21.15.143:3300/0 conn(0x7fed1406ae20 0x7fed1406b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed19a8d700 1 -- 172.21.15.111:0/3029976867 >> v1:172.21.15.111:6789/0 conn(0x7fed140ab6d0 legacy=0x7fed140abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed19a8d700 1 -- 172.21.15.111:0/3029976867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140ac950 con 0x7fed140b1440 2023-12-19T09:19:44.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed19a8d700 1 --2- 172.21.15.111:0/3029976867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 0x7fed140b18a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009e60 tx=0x7fed0c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=429e46b0bfee0b73 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed0bfff700 1 -- 172.21.15.111:0/3029976867 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00e070 con 0x7fed140b1440 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed0bfff700 1 -- 172.21.15.111:0/3029976867 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c00bab0 con 0x7fed140b1440 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed0bfff700 1 -- 172.21.15.111:0/3029976867 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00be70 con 0x7fed140b1440 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 -- 172.21.15.111:0/3029976867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 msgr2=0x7fed140b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3029976867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 0x7fed140b18a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009e60 tx=0x7fed0c009780 comp rx=0 tx=0).stop 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 -- 172.21.15.111:0/3029976867 shutdown_connections 2023-12-19T09:19:44.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3029976867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140b1440 0x7fed140b18a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3029976867 >> v2:172.21.15.143:3300/0 conn(0x7fed1406ae20 0x7fed1406b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 -- 172.21.15.111:0/3029976867 >> 172.21.15.111:0/3029976867 conn(0x7fed140c8cd0 msgr2=0x7fed140c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:44.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 -- 172.21.15.111:0/3029976867 shutdown_connections 2023-12-19T09:19:44.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 -- 172.21.15.111:0/3029976867 wait complete. 2023-12-19T09:19:44.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.929+0000 7fed1a28e700 1 Processor -- start 2023-12-19T09:19:44.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 -- start start 2023-12-19T09:19:44.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140ab6d0 0x7fed140a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed140b1440 0x7fed140a8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed140ca170 con 0x7fed1406ae20 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed140aafb0 con 0x7fed140b1440 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed140ab290 con 0x7fed140ab6d0 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57496/0 (socket says 172.21.15.111:57496) 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 -- 172.21.15.111:0/3315274708 learned_addr learned my addr 172.21.15.111:0/3315274708 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed19a8d700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed140b1440 0x7fed140a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed18a8b700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140ab6d0 0x7fed140a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 -- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140ab6d0 msgr2=0x7fed140a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140ab6d0 0x7fed140a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 -- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed140b1440 msgr2=0x7fed140a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:44.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed140b1440 0x7fed140a8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 -- 172.21.15.111:0/3315274708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c009a20 con 0x7fed1406ae20 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1928c700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fed1000e930 tx=0x7fed100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10013680 con 0x7fed1406ae20 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100137e0 con 0x7fed1406ae20 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed140c9e00 con 0x7fed1406ae20 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.933+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10013b80 con 0x7fed1406ae20 2023-12-19T09:19:44.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.937+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed14169eb0 con 0x7fed1406ae20 2023-12-19T09:19:44.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.937+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed140ab4b0 con 0x7fed1406ae20 2023-12-19T09:19:44.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.937+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fed1000c3e0 con 0x7fed1406ae20 2023-12-19T09:19:44.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.941+0000 7fed097fa700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 0x7fecf8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:44.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.941+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fed1007cbe0 con 0x7fed1406ae20 2023-12-19T09:19:44.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.941+0000 7fed18a8b700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 0x7fecf8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:44.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.941+0000 7fed18a8b700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 0x7fecf8062a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed1406bd30 tx=0x7fed04009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:44.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:44.945+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fed1004ce50 con 0x7fed1406ae20 2023-12-19T09:19:44.960 INFO:tasks.ceph.mds.a.smithi111.stderr:2023-12-19T09:19:44.957+0000 7f2bc7c69700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:19:45.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.177+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fed1404be40 con 0x7fed1406ae20 2023-12-19T09:19:45.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.185+0000 7fed097fa700 1 -- 172.21.15.111:0/3315274708 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) v1 ==== 190+0+0 (secure 0 0 0) 0x7fed100170e0 con 0x7fed1406ae20 2023-12-19T09:19:45.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 msgr2=0x7fecf8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 0x7fecf8062a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed1406bd30 tx=0x7fed04009040 comp rx=0 tx=0).stop 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 msgr2=0x7fed14149af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fed1000e930 tx=0x7fed100091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 shutdown_connections 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fecf8060580 0x7fecf8062a30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed140b1440 0x7fed140a8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed140ab6d0 0x7fed140a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 --2- 172.21.15.111:0/3315274708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1406ae20 0x7fed14149af0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.189+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 >> 172.21.15.111:0/3315274708 conn(0x7fed140c8cd0 msgr2=0x7fed140c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.193+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 shutdown_connections 2023-12-19T09:19:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.193+0000 7fed1a28e700 1 -- 172.21.15.111:0/3315274708 wait complete. 2023-12-19T09:19:45.198 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T09:19:45.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:19:45.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.369+0000 7fa3cac4c700 1 Processor -- start 2023-12-19T09:19:45.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 -- start start 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa3c40c6900 0x7fa3c40c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c406b900 0x7fa3c406da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 0x7fa3c415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c406b470 con 0x7fa3c40c6900 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c4161d10 con 0x7fa3c415cc20 2023-12-19T09:19:45.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.373+0000 7fa3cac4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c406bd70 con 0x7fa3c406b900 2023-12-19T09:19:45.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa3c40c6900 0x7fa3c40c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9449700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c406b900 0x7fa3c406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa3c40c6900 0x7fa3c40c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57512/0 (socket says 172.21.15.111:57512) 2023-12-19T09:19:45.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 -- 172.21.15.111:0/2531370343 learned_addr learned my addr 172.21.15.111:0/2531370343 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:45.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3ca44b700 1 --2- 172.21.15.111:0/2531370343 >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 0x7fa3c415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 -- 172.21.15.111:0/2531370343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c406b900 msgr2=0x7fa3c406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/2531370343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c406b900 0x7fa3c406da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 -- 172.21.15.111:0/2531370343 >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 msgr2=0x7fa3c415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/2531370343 >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 0x7fa3c415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 -- 172.21.15.111:0/2531370343 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c406a580 con 0x7fa3c40c6900 2023-12-19T09:19:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3ca44b700 1 --2- 172.21.15.111:0/2531370343 >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 0x7fa3c415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/2531370343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c40c6900 0x7fa3c40c6cd0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00099b0 tx=0x7fa3c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4798d51de779c9c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c8c48700 1 -- 172.21.15.111:0/2531370343 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000b640 con 0x7fa3c40c6900 2023-12-19T09:19:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c8c48700 1 -- 172.21.15.111:0/2531370343 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c000b7a0 con 0x7fa3c40c6900 2023-12-19T09:19:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3c8c48700 1 -- 172.21.15.111:0/2531370343 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000bb40 con 0x7fa3c40c6900 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 -- 172.21.15.111:0/2531370343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c40c6900 msgr2=0x7fa3c40c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/2531370343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c40c6900 0x7fa3c40c6cd0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00099b0 tx=0x7fa3c0009250 comp rx=0 tx=0).stop 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 -- 172.21.15.111:0/2531370343 shutdown_connections 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/2531370343 >> v2:172.21.15.143:3300/0 conn(0x7fa3c415cc20 0x7fa3c415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/2531370343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c406b900 0x7fa3c406da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/2531370343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c40c6900 0x7fa3c40c6cd0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.377+0000 7fa3cac4c700 1 -- 172.21.15.111:0/2531370343 >> 172.21.15.111:0/2531370343 conn(0x7fa3c4067bb0 msgr2=0x7fa3c40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:45.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.381+0000 7fa3cac4c700 1 -- 172.21.15.111:0/2531370343 shutdown_connections 2023-12-19T09:19:45.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.381+0000 7fa3cac4c700 1 -- 172.21.15.111:0/2531370343 wait complete. 2023-12-19T09:19:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.381+0000 7fa3cac4c700 1 Processor -- start 2023-12-19T09:19:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.381+0000 7fa3cac4c700 1 -- start start 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3c406b900 0x7fa3c4171080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c40c6900 0x7fa3c41715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c40b1630 con 0x7fa3c415cc20 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c416edf0 con 0x7fa3c406b900 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c416f0d0 con 0x7fa3c40c6900 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57524/0 (socket says 172.21.15.111:57524) 2023-12-19T09:19:45.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 -- 172.21.15.111:0/608568830 learned_addr learned my addr 172.21.15.111:0/608568830 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:45.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3c406b900 0x7fa3c4171080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3c9449700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c40c6900 0x7fa3c41715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 -- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c40c6900 msgr2=0x7fa3c41715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c40c6900 0x7fa3c41715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 -- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3c406b900 msgr2=0x7fa3c4171080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3c406b900 0x7fa3c4171080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 -- 172.21.15.111:0/608568830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc009a20 con 0x7fa3c415cc20 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3ca44b700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009f20 tx=0x7fa3bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00cbd0 con 0x7fa3c415cc20 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc00f040 con 0x7fa3c415cc20 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c00094f0 con 0x7fa3c415cc20 2023-12-19T09:19:45.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc014480 con 0x7fa3c415cc20 2023-12-19T09:19:45.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.385+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c416f690 con 0x7fa3c415cc20 2023-12-19T09:19:45.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.389+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c416f8b0 con 0x7fa3c415cc20 2023-12-19T09:19:45.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.389+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa3bc003020 con 0x7fa3c415cc20 2023-12-19T09:19:45.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.389+0000 7fa3b9ffb700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 0x7fa3a8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:45.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.389+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fa3bc07d400 con 0x7fa3c415cc20 2023-12-19T09:19:45.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.393+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 0x7fa3a8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:45.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.393+0000 7fa3c9c4a700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 0x7fa3a8062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00099b0 tx=0x7fa3c0017000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:45.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.397+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa3bc04d670 con 0x7fa3c415cc20 2023-12-19T09:19:45.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.597+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3c404be40 con 0x7fa3c415cc20 2023-12-19T09:19:45.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.601+0000 7fa3b9ffb700 1 -- 172.21.15.111:0/608568830 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7fa3bc050c90 con 0x7fa3c415cc20 2023-12-19T09:19:45.605 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:45.605 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:19:45.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 msgr2=0x7fa3a8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 0x7fa3a8062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00099b0 tx=0x7fa3c0017000 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 msgr2=0x7fa3c416da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009f20 tx=0x7fa3bc00b040 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 shutdown_connections 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c415cc20 0x7fa3c416da20 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3c40c6900 0x7fa3c41715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa3a8060550 0x7fa3a8062a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 --2- 172.21.15.111:0/608568830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3c406b900 0x7fa3c4171080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 >> 172.21.15.111:0/608568830 conn(0x7fa3c4067bb0 msgr2=0x7fa3c4159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 shutdown_connections 2023-12-19T09:19:45.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:45.605+0000 7fa3cac4c700 1 -- 172.21.15.111:0/608568830 wait complete. 2023-12-19T09:19:45.612 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:19:46.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T09:19:39.431568+0000', 'modified': '2023-12-19T09:19:43.040733+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9948}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9948': {'gid': 9948, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/1020011686', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1020011686}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1020011686}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T09:19:46.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T09:19:46.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-19T09:19:46.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.201+0000 7fd2c55cc700 1 Processor -- start 2023-12-19T09:19:46.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 -- start start 2023-12-19T09:19:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c01591f0 0x7fd2c006a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd2c00ce810 0x7fd2c00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c006b730 con 0x7fd2c0158e20 2023-12-19T09:19:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c006ab20 con 0x7fd2c00ce810 2023-12-19T09:19:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.205+0000 7fd2c55cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c006ae00 con 0x7fd2c01591f0 2023-12-19T09:19:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2bf7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c01591f0 0x7fd2c006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd2c00ce810 0x7fd2c00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd2c00ce810 0x7fd2c00ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38670/0 (socket says 172.21.15.111:38670) 2023-12-19T09:19:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1161573705 learned_addr learned my addr 172.21.15.111:0/1161573705 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c01591f0 msgr2=0x7fd2c006a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c01591f0 0x7fd2c006a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1161573705 >> v1:172.21.15.143:6789/0 conn(0x7fd2c0158e20 legacy=0x7fd2c00ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1161573705 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c006b0e0 con 0x7fd2c00ce810 2023-12-19T09:19:46.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c00ce810 0x7fd2c00ca950 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80099b0 tx=0x7fd2b8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72fe51e7e16f2343 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2beffd700 1 -- 172.21.15.111:0/1161573705 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8009ea0 con 0x7fd2c00ce810 2023-12-19T09:19:46.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2beffd700 1 -- 172.21.15.111:0/1161573705 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b800b860 con 0x7fd2c00ce810 2023-12-19T09:19:46.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2beffd700 1 -- 172.21.15.111:0/1161573705 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800bc00 con 0x7fd2c00ce810 2023-12-19T09:19:46.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 -- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c00ce810 msgr2=0x7fd2c00ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c00ce810 0x7fd2c00ca950 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80099b0 tx=0x7fd2b8009250 comp rx=0 tx=0).stop 2023-12-19T09:19:46.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 -- 172.21.15.111:0/1161573705 shutdown_connections 2023-12-19T09:19:46.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c00ce810 0x7fd2c00ca950 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/1161573705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c01591f0 0x7fd2c006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 -- 172.21.15.111:0/1161573705 >> 172.21.15.111:0/1161573705 conn(0x7fd2c0067bb0 msgr2=0x7fd2c00cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 -- 172.21.15.111:0/1161573705 shutdown_connections 2023-12-19T09:19:46.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.209+0000 7fd2c55cc700 1 -- 172.21.15.111:0/1161573705 wait complete. 2023-12-19T09:19:46.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 Processor -- start 2023-12-19T09:19:46.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 -- start start 2023-12-19T09:19:46.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c00ce810 0x7fd2c0155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c0158e20 0x7fd2c0158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c01585d0 con 0x7fd2c01591f0 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0170e20 con 0x7fd2c0158e20 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c55cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c016e020 con 0x7fd2c00ce810 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c4dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c4dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57530/0 (socket says 172.21.15.111:57530) 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/3858887017 learned_addr learned my addr 172.21.15.111:0/3858887017 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c0158e20 0x7fd2c0158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.213+0000 7fd2bffff700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c00ce810 0x7fd2c0155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c00ce810 msgr2=0x7fd2c0155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c00ce810 0x7fd2c0155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c0158e20 msgr2=0x7fd2c0158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c0158e20 0x7fd2c0158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/3858887017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80094f0 con 0x7fd2c01591f0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006fd0 tx=0x7fd2b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2a7fff700 1 -- 172.21.15.111:0/3858887017 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8011420 con 0x7fd2c01591f0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2a7fff700 1 -- 172.21.15.111:0/3858887017 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8011580 con 0x7fd2c01591f0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c016e300 con 0x7fd2c01591f0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2a7fff700 1 -- 172.21.15.111:0/3858887017 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8011940 con 0x7fd2c01591f0 2023-12-19T09:19:46.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.217+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c016e5e0 con 0x7fd2c01591f0 2023-12-19T09:19:46.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.221+0000 7fd2a7fff700 1 -- 172.21.15.111:0/3858887017 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd2b8011cf0 con 0x7fd2c01591f0 2023-12-19T09:19:46.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.221+0000 7fd2a7fff700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 0x7fd2ac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.221+0000 7fd2a7fff700 1 -- 172.21.15.111:0/3858887017 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fd2b807e720 con 0x7fd2c01591f0 2023-12-19T09:19:46.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.221+0000 7fd2bffff700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 0x7fd2ac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2bffff700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 0x7fd2ac062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0007fd0 tx=0x7fd2b0009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 Processor -- start 2023-12-19T09:19:46.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 -- start start 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf62f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf63980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0xe90f30 con 0xce5710 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xe8ced0 con 0xce5ae0 2023-12-19T09:19:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xe530b0 con 0xd30790 2023-12-19T09:19:46.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2bf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2bf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57534/0 (socket says 172.21.15.111:57534) 2023-12-19T09:19:46.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2bf7fe700 1 -- 172.21.15.111:0/1740349823 learned_addr learned my addr 172.21.15.111:0/1740349823 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf63980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.225+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf62f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 -- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 msgr2=0xf63980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf63980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 -- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 msgr2=0xf62f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf62f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 -- 172.21.15.111:0/1740349823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xd319e0 con 0xce5710 2023-12-19T09:19:46.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009c80 tx=0x7fd2b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9e6730fe9a43f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2a57fa700 1 -- 172.21.15.111:0/1740349823 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011a90 con 0xce5710 2023-12-19T09:19:46.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2a57fa700 1 -- 172.21.15.111:0/1740349823 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4011bf0 con 0xce5710 2023-12-19T09:19:46.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2a57fa700 1 -- 172.21.15.111:0/1740349823 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4014440 con 0xce5710 2023-12-19T09:19:46.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1740349823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 msgr2=0xf63440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009c80 tx=0x7fd2b40091a0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1740349823 shutdown_connections 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf63980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf63440 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1740349823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf62f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1740349823 >> 172.21.15.111:0/1740349823 conn(0xce5eb0 msgr2=0xf1f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1740349823 shutdown_connections 2023-12-19T09:19:46.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1740349823 wait complete. 2023-12-19T09:19:46.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.229+0000 7fd2c8ac4740 1 -- start start 2023-12-19T09:19:46.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 Processor -- start 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf67ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd30790 0xf48cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0xe52d50 con 0xce5710 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xe2b970 con 0xd30790 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xe36510 con 0xce5ae0 2023-12-19T09:19:46.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c8ac4740 10 client.0 _ll_register_callbacks cb 0xed66a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57536/0 (socket says 172.21.15.111:57536) 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 -- 172.21.15.111:0/2215210350 learned_addr learned my addr 172.21.15.111:0/2215210350 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd30790 0xf48cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf67ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 -- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 msgr2=0xf67ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf67ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 -- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd30790 msgr2=0xf48cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd30790 0xf48cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.233+0000 7fd2bffff700 1 -- 172.21.15.111:0/2215210350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b4009960 con 0xce5710 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2bffff700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001680 tx=0x7fd2b0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a8c59c0de93c04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0001e20 con 0xce5710 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xe2bcc0 con 0xce5710 2023-12-19T09:19:46.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fd2b0013630 con 0xce5710 2023-12-19T09:19:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2b0013a00 con 0xce5710 2023-12-19T09:19:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 10 client.10071 fetch_fsmap learned FSMap version 87 2023-12-19T09:19:46.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0xe5b370 con 0xce5710 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xe2bcc0 con 0xce5710 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 87) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd2b0020070 con 0xce5710 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 10 client.10071 fetch_fsmap finished waiting for FSMap version 87 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 10 client.10071 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 20 client.10071 populate_metadata read hostname 'smithi111' 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf4a420 0xf726b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 4 client.10071 mds_command: new command op to 9948 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 --> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xe2f3e0 con 0xf4a420 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf4a420 0xf726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd29e7fc700 10 client.10071 ms_handle_connect on v2:172.21.15.111:6834/1020011686 2023-12-19T09:19:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.237+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf4a420 0xf726b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fe0920f01aa4476 server_cookie=125b0f5f6730a496 in_seq=0 out_seq=0 2023-12-19T09:19:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd29e7fc700 1 -- 172.21.15.111:0/2215210350 <== mds.0 v2:172.21.15.111:6834/1020011686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0xe2f3e0 con 0xf4a420 2023-12-19T09:19:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd29e7fc700 10 client.10071 handle_command_reply: tid=0 2023-12-19T09:19:46.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 client.10071 shutdown 2023-12-19T09:19:46.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 msgr2=0xf67590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001680 tx=0x7fd2b0013000 comp rx=0 tx=0).stop 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 shutdown_connections 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf4a420 0xf726b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xd30790 0xf48cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf67ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/2215210350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf67590 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd29e7fc700 0 client.10071 ms_handle_reset on v2:172.21.15.111:6834/1020011686 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 >> 172.21.15.111:0/2215210350 conn(0xce5eb0 msgr2=0xf17f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 shutdown_connections 2023-12-19T09:19:46.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/2215210350 wait complete. 2023-12-19T09:19:46.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.241+0000 7fd2c8ac4740 20 client.10071 trim_cache size 0 max 16384 2023-12-19T09:19:46.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 Processor -- start 2023-12-19T09:19:46.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 -- start start 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf46cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 0xf47ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0xd62650 con 0xd30790 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xe909d0 con 0xce5710 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xe3bc90 con 0xce5ae0 2023-12-19T09:19:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c4dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 0xf47ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2c4dcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 0xf47ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57546/0 (socket says 172.21.15.111:57546) 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.261+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38710/0 (socket says 172.21.15.111:38710) 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1423884341 learned_addr learned my addr 172.21.15.111:0/1423884341 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf46cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 -- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 msgr2=0xf46cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf46cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 -- 172.21.15.111:0/1423884341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 msgr2=0xf47ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 0xf47ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 -- 172.21.15.111:0/1423884341 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe41d80 con 0xce5710 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001fd0 tx=0x7fd2b0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc786d4e5ac236d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2a5ffb700 1 -- 172.21.15.111:0/1423884341 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0013630 con 0xce5710 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2a5ffb700 1 -- 172.21.15.111:0/1423884341 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b00031b0 con 0xce5710 2023-12-19T09:19:46.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2a5ffb700 1 -- 172.21.15.111:0/1423884341 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0001c70 con 0xce5710 2023-12-19T09:19:46.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 msgr2=0xf46780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001fd0 tx=0x7fd2b0015000 comp rx=0 tx=0).stop 2023-12-19T09:19:46.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1423884341 shutdown_connections 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xd30790 0xf47ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xce5ae0 0xf46cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1423884341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5710 0xf46780 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1423884341 >> 172.21.15.111:0/1423884341 conn(0xce5eb0 msgr2=0xf19ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1423884341 shutdown_connections 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.265+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1423884341 wait complete. 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 -- start start 2023-12-19T09:19:46.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 Processor -- start 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf1ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf1bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0xce3430 con 0xce5710 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0xe54be0 con 0xce5ae0 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0xe64d60 con 0xd30790 2023-12-19T09:19:46.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c8ac4740 10 client.0 _ll_register_callbacks cb 0xcb19d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:19:46.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57550/0 (socket says 172.21.15.111:57550) 2023-12-19T09:19:46.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2bffff700 1 -- 172.21.15.111:0/1537889562 learned_addr learned my addr 172.21.15.111:0/1537889562 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf1bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2bf7fe700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf1ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 msgr2=0xf1ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf1ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 msgr2=0xf1b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 -- 172.21.15.111:0/1537889562 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0xe5b370 con 0xd30790 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd2c4dcb700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf1bf70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd2b804ea60 tx=0x7fd2b8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99689a335f27564a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd29e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.143:3301/0 2023-12-19T09:19:46.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.269+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8050950 con 0xd30790 2023-12-19T09:19:46.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2b4011bf0 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mon.2 v2:172.21.15.143:3301/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fd2b8051aa0 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2b8059100 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 10 client.8543 fetch_fsmap learned FSMap version 87 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0xe36130 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mon.2 v2:172.21.15.143:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8070080 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mon.2 v2:172.21.15.143:3301/0 5 ==== fsmap(e 87) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fd2b804e3a0 con 0xd30790 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 10 client.8543 fetch_fsmap finished waiting for FSMap version 87 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 10 client.8543 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 20 client.8543 populate_metadata read hostname 'smithi111' 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf1c4b0 0xf1c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 4 client.8543 mds_command: new command op to 9948 tid=0[{"prefix": "status"}] 2023-12-19T09:19:46.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 --> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] -- command(tid 0: {"prefix": "status"}) v1 -- 0xe61410 con 0xf1c4b0 2023-12-19T09:19:46.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf1c4b0 0xf1c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2bffff700 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf1c4b0 0xf1c880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad69aa9c32dbe429 server_cookie=9032ae139dbd2ef in_seq=0 out_seq=0 2023-12-19T09:19:46.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 10 client.8543 ms_handle_connect on v2:172.21.15.111:6834/1020011686 2023-12-19T09:19:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 1 -- 172.21.15.111:0/1537889562 <== mds.0 v2:172.21.15.111:6834/1020011686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0xe61410 con 0xf1c4b0 2023-12-19T09:19:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 10 client.8543 handle_command_reply: tid=0 2023-12-19T09:19:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 client.8543 shutdown 2023-12-19T09:19:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 msgr2=0xf1bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf1bf70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd2b804ea60 tx=0x7fd2b8008040 comp rx=0 tx=0).stop 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 shutdown_connections 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:6834/1020011686,v1:172.21.15.111:6835/1020011686] conn(0xf1c4b0 0xf1c880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0xce5ae0 0xf1ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd29e7fc700 0 client.8543 ms_handle_reset on v2:172.21.15.111:6834/1020011686 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0xd30790 0xf1bf70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 --2- 172.21.15.111:0/1537889562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0xce5710 0xf1b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 >> 172.21.15.111:0/1537889562 conn(0xce5eb0 msgr2=0xf667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 shutdown_connections 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.273+0000 7fd2c8ac4740 1 -- 172.21.15.111:0/1537889562 wait complete. 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c8ac4740 20 client.8543 trim_cache size 0 max 16384 2023-12-19T09:19:46.280 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 9948, 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.2537879710000004, 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 87, 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 193, 2023-12-19T09:19:46.281 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 193, 2023-12-19T09:19:46.282 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.179785851 2023-12-19T09:19:46.282 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:19:46.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 msgr2=0x7fd2ac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 0x7fd2ac062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0007fd0 tx=0x7fd2b0009040 comp rx=0 tx=0).stop 2023-12-19T09:19:46.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 msgr2=0x7fd2c016d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006fd0 tx=0x7fd2b8017000 comp rx=0 tx=0).stop 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 shutdown_connections 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2ac060550 0x7fd2ac062a00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c0158e20 0x7fd2c0158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c01591f0 0x7fd2c016d750 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 --2- 172.21.15.111:0/3858887017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c00ce810 0x7fd2c0155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 >> 172.21.15.111:0/3858887017 conn(0x7fd2c0067bb0 msgr2=0x7fd2c00b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 shutdown_connections 2023-12-19T09:19:46.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.277+0000 7fd2c55cc700 1 -- 172.21.15.111:0/3858887017 wait complete. 2023-12-19T09:19:46.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T09:19:46.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 Processor -- start 2023-12-19T09:19:46.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 -- start start 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd83c140ae0 0x7fd83c0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 0x7fd83c0ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c06e150 con 0x7fd83c140710 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c068a70 con 0x7fd83c140ae0 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd841745700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c06d540 con 0x7fd83c068630 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd833fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd83c140ae0 0x7fd83c0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd833fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd83c140ae0 0x7fd83c0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57556/0 (socket says 172.21.15.111:57556) 2023-12-19T09:19:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd833fff700 1 -- 172.21.15.111:0/2750903892 learned_addr learned my addr 172.21.15.111:0/2750903892 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.465+0000 7fd83b7fe700 1 --2- 172.21.15.111:0/2750903892 >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 0x7fd83c0ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd833fff700 1 -- 172.21.15.111:0/2750903892 >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 msgr2=0x7fd83c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd833fff700 1 --2- 172.21.15.111:0/2750903892 >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 0x7fd83c0ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd833fff700 1 -- 172.21.15.111:0/2750903892 >> v1:172.21.15.111:6789/0 conn(0x7fd83c140710 legacy=0x7fd83c0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd833fff700 1 -- 172.21.15.111:0/2750903892 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c06d820 con 0x7fd83c140ae0 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd83b7fe700 1 --2- 172.21.15.111:0/2750903892 >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 0x7fd83c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:46.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd833fff700 1 --2- 172.21.15.111:0/2750903892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140ae0 0x7fd83c0680f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd83c06bfc0 tx=0x7fd824009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4a876e84da6ad1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd83a7fc700 1 -- 172.21.15.111:0/2750903892 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400b860 con 0x7fd83c140ae0 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd83a7fc700 1 -- 172.21.15.111:0/2750903892 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82400b9c0 con 0x7fd83c140ae0 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd83a7fc700 1 -- 172.21.15.111:0/2750903892 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400bd60 con 0x7fd83c140ae0 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 -- 172.21.15.111:0/2750903892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140ae0 msgr2=0x7fd83c0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 --2- 172.21.15.111:0/2750903892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140ae0 0x7fd83c0680f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd83c06bfc0 tx=0x7fd824009250 comp rx=0 tx=0).stop 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 -- 172.21.15.111:0/2750903892 shutdown_connections 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 --2- 172.21.15.111:0/2750903892 >> v2:172.21.15.143:3300/0 conn(0x7fd83c068630 0x7fd83c0ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 --2- 172.21.15.111:0/2750903892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140ae0 0x7fd83c0680f0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 -- 172.21.15.111:0/2750903892 >> 172.21.15.111:0/2750903892 conn(0x7fd83c0c8850 msgr2=0x7fd83c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 -- 172.21.15.111:0/2750903892 shutdown_connections 2023-12-19T09:19:46.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.469+0000 7fd841745700 1 -- 172.21.15.111:0/2750903892 wait complete. 2023-12-19T09:19:46.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 Processor -- start 2023-12-19T09:19:46.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 -- start start 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd83c068630 0x7fd83c13df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd83c140ae0 0x7fd83c0a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0af550 con 0x7fd83c140710 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c13e440 con 0x7fd83c140ae0 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0c3da0 con 0x7fd83c068630 2023-12-19T09:19:46.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57564/0 (socket says 172.21.15.111:57564) 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 -- 172.21.15.111:0/143419072 learned_addr learned my addr 172.21.15.111:0/143419072 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd83affd700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd83c068630 0x7fd83c13df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd83b7fe700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd83c140ae0 0x7fd83c0a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 -- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd83c068630 msgr2=0x7fd83c13df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd83c068630 0x7fd83c13df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 -- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd83c140ae0 msgr2=0x7fd83c0a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd83c140ae0 0x7fd83c0a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 -- 172.21.15.111:0/143419072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240094f0 con 0x7fd83c140710 2023-12-19T09:19:46.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd833fff700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd83c06a280 tx=0x7fd824017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824011e60 con 0x7fd83c140710 2023-12-19T09:19:46.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82401a630 con 0x7fd83c140710 2023-12-19T09:19:46.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c0c57e0 con 0x7fd83c140710 2023-12-19T09:19:46.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.473+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82401a9f0 con 0x7fd83c140710 2023-12-19T09:19:46.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.477+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c0aaee0 con 0x7fd83c140710 2023-12-19T09:19:46.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.477+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c0af690 con 0x7fd83c140710 2023-12-19T09:19:46.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.477+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd82402d440 con 0x7fd83c140710 2023-12-19T09:19:46.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.481+0000 7fd832ffd700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 0x7fd81c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:46.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.481+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fd82407e820 con 0x7fd83c140710 2023-12-19T09:19:46.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.481+0000 7fd83affd700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 0x7fd81c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:46.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.481+0000 7fd83affd700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 0x7fd81c062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0037c0 tx=0x7fd82c009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:46.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.485+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd83c0af690 con 0x7fd83c140710 2023-12-19T09:19:46.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.689+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd83c0af690 con 0x7fd83c140710 2023-12-19T09:19:46.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.689+0000 7fd832ffd700 1 -- 172.21.15.111:0/143419072 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd83c0af690 con 0x7fd83c140710 2023-12-19T09:19:46.695 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 msgr2=0x7fd81c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 0x7fd81c062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0037c0 tx=0x7fd82c009040 comp rx=0 tx=0).stop 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 msgr2=0x7fd83c0a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd83c06a280 tx=0x7fd824017000 comp rx=0 tx=0).stop 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 shutdown_connections 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd81c060550 0x7fd81c062a00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd83c140ae0 0x7fd83c0a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd83c140710 0x7fd83c0a9820 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 --2- 172.21.15.111:0/143419072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd83c068630 0x7fd83c13df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:46.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 >> 172.21.15.111:0/143419072 conn(0x7fd83c0c8850 msgr2=0x7fd83c0c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:46.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 shutdown_connections 2023-12-19T09:19:46.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:46.693+0000 7fd841745700 1 -- 172.21.15.111:0/143419072 wait complete. 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T09:19:46.718 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:19:46.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:46.719 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T09:19:46.741 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T09:19:46.741 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T09:19:46.741 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T09:19:46.741 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.741 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 36871sec preferred_lft 36871sec 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T09:19:46.742 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T09:19:46.743 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T09:19:46.744 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:19:46.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:46.745 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T09:19:46.828 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T09:19:46.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:46.832 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T09:19:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T09:19:46.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T09:19:46.896 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T09:19:47.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:47.032 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:19:47.049 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:19:47.050 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:19:47.051 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:19:47.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:19:47.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T09:19:47.131 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T09:19:47.132 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:19:47.132 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:47.196 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:19:47.196 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:19:47.196 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:47.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:19:47.339 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:19:47.339 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:19:47.416 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:19:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:19:47.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:47.418 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:19:47.478 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T09:19:47.478 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T09:19:47.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:19:47.567 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:19:47.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:19:47.644 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:19:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:19:47.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:47.648 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:19:47.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T09:19:47.661+0000 7f865d962380 -1 init, newargv = 0x560db460a230 newargc=15 2023-12-19T09:19:47.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[61451]: starting ceph client 2023-12-19T09:19:47.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[61451]: starting fuse 2023-12-19T09:19:47.681 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T09:19:47.681 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T09:19:47.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T09:19:47.681 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T09:19:47.682 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T09:19:47.683 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T09:19:47.684 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:19:47.684 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T09:19:47.684 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T09:19:47.845 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.61451.asok 2023-12-19T09:19:48.030 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61451.asok 2023-12-19T09:19:48.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:48.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61451.asok status 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T09:19:48.209 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "61451", 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 10092, 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 10092 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:19:48.210 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2819634396 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2819634396 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.10092 192.168.0.1:0/2819634396", 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2819634396", 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 87, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 193, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T09:19:48.211 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:19:48.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:48.222 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T09:19:48.223 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:48.239 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T09:19:48.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:48.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:48.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:19:48.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T09:19:48.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 Processor -- start 2023-12-19T09:19:48.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 -- start start 2023-12-19T09:19:48.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7e48158cc0 0x7f7e4806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e4806dc00 0x7f7e4806b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4806b0d0 con 0x7f7e4806baa0 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4806bec0 con 0x7f7e48158cc0 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.533+0000 7f7e4cc78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e480b4790 con 0x7f7e4806dc00 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7e48158cc0 0x7f7e4806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7e48158cc0 0x7f7e4806d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57572/0 (socket says 172.21.15.111:57572) 2023-12-19T09:19:48.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 -- 172.21.15.111:0/3427039870 learned_addr learned my addr 172.21.15.111:0/3427039870 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 -- 172.21.15.111:0/3427039870 >> v2:172.21.15.143:3300/0 conn(0x7f7e4806dc00 msgr2=0x7f7e4806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 --2- 172.21.15.111:0/3427039870 >> v2:172.21.15.143:3300/0 conn(0x7f7e4806dc00 0x7f7e4806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 -- 172.21.15.111:0/3427039870 >> v1:172.21.15.143:6789/0 conn(0x7f7e4806baa0 legacy=0x7f7e480b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 -- 172.21.15.111:0/3427039870 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4806d710 con 0x7f7e48158cc0 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4659c700 1 --2- 172.21.15.111:0/3427039870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e4806d1d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f7e48068090 tx=0x7f7e38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838150352f6ea0c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4559a700 1 -- 172.21.15.111:0/3427039870 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3800b820 con 0x7f7e48158cc0 2023-12-19T09:19:48.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4559a700 1 -- 172.21.15.111:0/3427039870 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3800b980 con 0x7f7e48158cc0 2023-12-19T09:19:48.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4559a700 1 -- 172.21.15.111:0/3427039870 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3800bd20 con 0x7f7e48158cc0 2023-12-19T09:19:48.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 -- 172.21.15.111:0/3427039870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 msgr2=0x7f7e4806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/3427039870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e4806d1d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f7e48068090 tx=0x7f7e38009250 comp rx=0 tx=0).stop 2023-12-19T09:19:48.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 -- 172.21.15.111:0/3427039870 shutdown_connections 2023-12-19T09:19:48.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/3427039870 >> v2:172.21.15.143:3300/0 conn(0x7f7e4806dc00 0x7f7e4806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/3427039870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e4806d1d0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 -- 172.21.15.111:0/3427039870 >> 172.21.15.111:0/3427039870 conn(0x7f7e48067bb0 msgr2=0x7f7e480cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:48.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 -- 172.21.15.111:0/3427039870 shutdown_connections 2023-12-19T09:19:48.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.537+0000 7f7e4cc78700 1 -- 172.21.15.111:0/3427039870 wait complete. 2023-12-19T09:19:48.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 Processor -- start 2023-12-19T09:19:48.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 -- start start 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e4806baa0 0x7f7e48150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 0x7f7e4816a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4815eb50 con 0x7f7e48158cc0 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e48159810 con 0x7f7e4806baa0 2023-12-19T09:19:48.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.541+0000 7f7e4cc78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4815e870 con 0x7f7e4806dc00 2023-12-19T09:19:48.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57584/0 (socket says 172.21.15.111:57584) 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 -- 172.21.15.111:0/669776284 learned_addr learned my addr 172.21.15.111:0/669776284 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e4659c700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e4806baa0 0x7f7e48150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e45d9b700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 0x7f7e4816a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 -- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 msgr2=0x7f7e4816a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 0x7f7e4816a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 -- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e4806baa0 msgr2=0x7f7e48150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e4806baa0 0x7f7e48150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 -- 172.21.15.111:0/669776284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e380094f0 con 0x7f7e48158cc0 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e45d9b700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 0x7f7e4816a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:19:48.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e46d9d700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000bbb0 tx=0x7f7e40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e400092e0 con 0x7f7e48158cc0 2023-12-19T09:19:48.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4000c7a0 con 0x7f7e48158cc0 2023-12-19T09:19:48.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e481591c0 con 0x7f7e48158cc0 2023-12-19T09:19:48.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4000cb40 con 0x7f7e48158cc0 2023-12-19T09:19:48.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e48150eb0 con 0x7f7e48158cc0 2023-12-19T09:19:48.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.545+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e481599a0 con 0x7f7e48158cc0 2023-12-19T09:19:48.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.549+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7e40012070 con 0x7f7e48158cc0 2023-12-19T09:19:48.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.549+0000 7f7e367fc700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 0x7f7e28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.549+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f7e4007c8d0 con 0x7f7e48158cc0 2023-12-19T09:19:48.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.549+0000 7f7e4659c700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 0x7f7e28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.553+0000 7f7e4659c700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 0x7f7e28062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e48068090 tx=0x7f7e38016000 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.553+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7e4004caa0 con 0x7f7e48158cc0 2023-12-19T09:19:48.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.749+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e48071120 con 0x7f7e28060550 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.753+0000 7f7e367fc700 1 -- 172.21.15.111:0/669776284 <== mgr.8285 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7e48071120 con 0x7f7e28060550 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:19:48.759 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 msgr2=0x7f7e28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 0x7f7e28062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e48068090 tx=0x7f7e38016000 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 msgr2=0x7f7e48159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000bbb0 tx=0x7f7e40009840 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 shutdown_connections 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e48158cc0 0x7f7e48159ca0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e4806dc00 0x7f7e4816a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7e28060550 0x7f7e28062a00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 --2- 172.21.15.111:0/669776284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e4806baa0 0x7f7e48150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 >> 172.21.15.111:0/669776284 conn(0x7f7e48067bb0 msgr2=0x7f7e481566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:48.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 shutdown_connections 2023-12-19T09:19:48.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.757+0000 7f7e4cc78700 1 -- 172.21.15.111:0/669776284 wait complete. 2023-12-19T09:19:48.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T09:19:48.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.945+0000 7f7c058f0700 1 Processor -- start 2023-12-19T09:19:48.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.945+0000 7f7c058f0700 1 -- start start 2023-12-19T09:19:48.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7c058f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c00156a10 0x7f7c000cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7c058f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0006ac90 0x7f7c0006d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7c058f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000ac620 con 0x7f7c00156de0 2023-12-19T09:19:48.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7c058f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c000ac060 con 0x7f7c00156a10 2023-12-19T09:19:48.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7c058f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000ac340 con 0x7f7c0006ac90 2023-12-19T09:19:48.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c00156a10 0x7f7c000cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c00156a10 0x7f7c000cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57588/0 (socket says 172.21.15.111:57588) 2023-12-19T09:19:48.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 -- 172.21.15.111:0/258894882 learned_addr learned my addr 172.21.15.111:0/258894882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:48.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bff7fe700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0006ac90 0x7f7c0006d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 -- 172.21.15.111:0/258894882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0006ac90 msgr2=0x7f7c0006d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0006ac90 0x7f7c0006d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 -- 172.21.15.111:0/258894882 >> v1:172.21.15.143:6789/0 conn(0x7f7c00156de0 legacy=0x7f7c0006a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 -- 172.21.15.111:0/258894882 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0006c860 con 0x7f7c00156a10 2023-12-19T09:19:48.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfdffb700 1 -- 172.21.15.111:0/258894882 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312360484 0 0) 0x7f7c000ac620 con 0x7f7c00156de0 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfeffd700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c00156a10 0x7f7c000cc360 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00099b0 tx=0x7f7bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20450025a305dfbd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfdffb700 1 -- 172.21.15.111:0/258894882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0009ea0 con 0x7f7c00156a10 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfdffb700 1 -- 172.21.15.111:0/258894882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf000b860 con 0x7f7c00156a10 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.949+0000 7f7bfdffb700 1 -- 172.21.15.111:0/258894882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf000bc00 con 0x7f7c00156a10 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 -- 172.21.15.111:0/258894882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c00156a10 msgr2=0x7f7c000cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c00156a10 0x7f7c000cc360 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00099b0 tx=0x7f7bf0009250 comp rx=0 tx=0).stop 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 -- 172.21.15.111:0/258894882 shutdown_connections 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0006ac90 0x7f7c0006d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 --2- 172.21.15.111:0/258894882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c00156a10 0x7f7c000cc360 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 -- 172.21.15.111:0/258894882 >> 172.21.15.111:0/258894882 conn(0x7f7c00067bb0 msgr2=0x7f7c000cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 -- 172.21.15.111:0/258894882 shutdown_connections 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 -- 172.21.15.111:0/258894882 wait complete. 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.953+0000 7f7c058f0700 1 Processor -- start 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- start start 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c00156a10 0x7f7c00159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c001578f0 0x7f7c000c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00144060 con 0x7f7c0006ac90 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00158ee0 con 0x7f7c00156a10 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c001582d0 con 0x7f7c001578f0 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57594/0 (socket says 172.21.15.111:57594) 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 -- 172.21.15.111:0/4139934463 learned_addr learned my addr 172.21.15.111:0/4139934463 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bff7fe700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c001578f0 0x7f7c000c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfe7fc700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c00156a10 0x7f7c00159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 -- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c001578f0 msgr2=0x7f7c000c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c001578f0 0x7f7c000c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 -- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c00156a10 msgr2=0x7f7c00159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c00156a10 0x7f7c00159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 -- 172.21.15.111:0/4139934463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00094f0 con 0x7f7c0006ac90 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7bfeffd700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7c000c95b0 tx=0x7f7bf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf000bbe0 con 0x7f7c0006ac90 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0011420 con 0x7f7c0006ac90 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00117e0 con 0x7f7c0006ac90 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c001585b0 con 0x7f7c0006ac90 2023-12-19T09:19:48.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.957+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00158b70 con 0x7f7c0006ac90 2023-12-19T09:19:48.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.961+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0004be40 con 0x7f7c0006ac90 2023-12-19T09:19:48.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.961+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 92) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f7bf0011bd0 con 0x7f7c0006ac90 2023-12-19T09:19:48.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.961+0000 7f7be6ffd700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 0x7f7bec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:48.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.961+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f7bf007e420 con 0x7f7c0006ac90 2023-12-19T09:19:48.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.961+0000 7f7bfe7fc700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 0x7f7bec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:48.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.965+0000 7f7bfe7fc700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 0x7f7bec062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7c0006a080 tx=0x7f7bf4009040 comp rx=0 tx=0).ready entity=mgr.8285 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:48.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:48.969+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7bf004e690 con 0x7f7c0006ac90 2023-12-19T09:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.177+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7c0006cb40 con 0x7f7c0006ac90 2023-12-19T09:19:49.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.665+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 93) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f7bf007e0f0 con 0x7f7c0006ac90 2023-12-19T09:19:49.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7be6ffd700 1 -- 172.21.15.111:0/4139934463 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v93) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7bf00253f0 con 0x7f7c0006ac90 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 msgr2=0x7f7bec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 0x7f7bec062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7c0006a080 tx=0x7f7bf4009040 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 msgr2=0x7f7c0015a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7c000c95b0 tx=0x7f7bf0017000 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 shutdown_connections 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7bec060550 0x7f7bec062a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c00156a10 0x7f7c00159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c001578f0 0x7f7c000c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 --2- 172.21.15.111:0/4139934463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c0006ac90 0x7f7c0015a110 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.669+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 >> 172.21.15.111:0/4139934463 conn(0x7f7c00067bb0 msgr2=0x7f7c000ca800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.673+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 shutdown_connections 2023-12-19T09:19:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.673+0000 7f7c058f0700 1 -- 172.21.15.111:0/4139934463 wait complete. 2023-12-19T09:19:49.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T09:19:49.860 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:49.855+0000 7fae84749000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:19:49.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.857+0000 7f5abb6a5700 1 Processor -- start 2023-12-19T09:19:49.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.857+0000 7f5abb6a5700 1 -- start start 2023-12-19T09:19:49.864 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f712b1ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f5abb6a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ab40b2470 0x7f5ab40cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f5abb6a5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab406d270 con 0x7f5ab415b6b0 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f5abb6a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab406d550 con 0x7f5ab415abd0 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f5abb6a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab406d830 con 0x7f5ab40b2470 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.861+0000 7f5ab9c42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ab40b2470 0x7f5ab40cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ab40b2470 0x7f5ab40cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57620/0 (socket says 172.21.15.111:57620) 2023-12-19T09:19:49.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 -- 172.21.15.111:0/1567531478 learned_addr learned my addr 172.21.15.111:0/1567531478 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:49.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 -- 172.21.15.111:0/1567531478 >> v1:172.21.15.143:6789/0 conn(0x7f5ab415abd0 legacy=0x7f5ab415afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 -- 172.21.15.111:0/1567531478 >> v1:172.21.15.111:6789/0 conn(0x7f5ab415b6b0 legacy=0x7f5ab41567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 -- 172.21.15.111:0/1567531478 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab406db10 con 0x7f5ab40b2470 2023-12-19T09:19:49.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5ab9c42700 1 --2- 172.21.15.111:0/1567531478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab40b2470 0x7f5ab40cd3f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009f20 tx=0x7f5aac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1780e224befaa775 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:49.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5aabfff700 1 -- 172.21.15.111:0/1567531478 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00e070 con 0x7f5ab40b2470 2023-12-19T09:19:49.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5aabfff700 1 -- 172.21.15.111:0/1567531478 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac00ba80 con 0x7f5ab40b2470 2023-12-19T09:19:49.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5aabfff700 1 -- 172.21.15.111:0/1567531478 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac0092e0 con 0x7f5ab40b2470 2023-12-19T09:19:49.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 -- 172.21.15.111:0/1567531478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab40b2470 msgr2=0x7f5ab40cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/1567531478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab40b2470 0x7f5ab40cd3f0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009f20 tx=0x7f5aac009840 comp rx=0 tx=0).stop 2023-12-19T09:19:49.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 -- 172.21.15.111:0/1567531478 shutdown_connections 2023-12-19T09:19:49.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/1567531478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab40b2470 0x7f5ab40cd3f0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 -- 172.21.15.111:0/1567531478 >> 172.21.15.111:0/1567531478 conn(0x7f5ab40cdef0 msgr2=0x7f5ab40ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:49.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 -- 172.21.15.111:0/1567531478 shutdown_connections 2023-12-19T09:19:49.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.865+0000 7f5abb6a5700 1 -- 172.21.15.111:0/1567531478 wait complete. 2023-12-19T09:19:49.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 Processor -- start 2023-12-19T09:19:49.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 -- start start 2023-12-19T09:19:49.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ab40b2470 0x7f5ab415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:49.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ab415abd0 0x7f5ab4149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:49.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:49.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415cf40 con 0x7f5ab415b6b0 2023-12-19T09:19:49.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415c330 con 0x7f5ab415abd0 2023-12-19T09:19:49.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5abb6a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415c610 con 0x7f5ab40b2470 2023-12-19T09:19:49.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:49.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57626/0 (socket says 172.21.15.111:57626) 2023-12-19T09:19:49.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 -- 172.21.15.111:0/2519831755 learned_addr learned my addr 172.21.15.111:0/2519831755 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab8c40700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ab415abd0 0x7f5ab4149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9441700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ab40b2470 0x7f5ab415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 -- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ab40b2470 msgr2=0x7f5ab415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ab40b2470 0x7f5ab415bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 -- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ab415abd0 msgr2=0x7f5ab4149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ab415abd0 0x7f5ab4149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:49.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.869+0000 7f5ab9c42700 1 -- 172.21.15.111:0/2519831755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac009ae0 con 0x7f5ab415b6b0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5ab9c42700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5ab406bce0 tx=0x7f5aac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac008dd0 con 0x7f5ab415b6b0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac026460 con 0x7f5ab415b6b0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac026820 con 0x7f5ab415b6b0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab415c8f0 con 0x7f5ab415b6b0 2023-12-19T09:19:49.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab415c020 con 0x7f5ab415b6b0 2023-12-19T09:19:49.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab40690f0 con 0x7f5ab415b6b0 2023-12-19T09:19:49.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.873+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 93) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f5aac003040 con 0x7f5ab415b6b0 2023-12-19T09:19:49.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.877+0000 7f5aa97fa700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 0x7f5a98062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:49.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.877+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f5aac07ec70 con 0x7f5ab415b6b0 2023-12-19T09:19:49.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.877+0000 7f5ab9441700 1 -- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 msgr2=0x7f5a98062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:49.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.877+0000 7f5ab9441700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 0x7f5a98062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:49.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:49.881+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ab40690f0 con 0x7f5ab415b6b0 2023-12-19T09:19:50.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.049+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5ab40690f0 con 0x7f5ab415b6b0 2023-12-19T09:19:50.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5aa97fa700 1 -- 172.21.15.111:0/2519831755 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ab40690f0 con 0x7f5ab415b6b0 2023-12-19T09:19:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 msgr2=0x7f5a98062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 0x7f5a98062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 msgr2=0x7f5ab415da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5ab406bce0 tx=0x7f5aac018000 comp rx=0 tx=0).stop 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 shutdown_connections 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5a980605a0 0x7f5a98062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ab415abd0 0x7f5ab4149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ab415b6b0 0x7f5ab415da20 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.057+0000 7f5abb6a5700 1 --2- 172.21.15.111:0/2519831755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ab40b2470 0x7f5ab415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.061+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 >> 172.21.15.111:0/2519831755 conn(0x7f5ab40cdef0 msgr2=0x7f5ab415f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.061+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 shutdown_connections 2023-12-19T09:19:50.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.061+0000 7f5abb6a5700 1 -- 172.21.15.111:0/2519831755 wait complete. 2023-12-19T09:19:50.081 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T09:19:50.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 Processor -- start 2023-12-19T09:19:50.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 -- start start 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 0x7f59f4161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f406a7a0 con 0x7f59f40c6d20 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f406d4b0 con 0x7f59f406c7c0 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fc545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f406d790 con 0x7f59f406bc40 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.249+0000 7f59fa2e1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f59f40c6d20 0x7f59f40c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50574/0 (socket says 172.21.15.111:50574) 2023-12-19T09:19:50.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fa2e1700 1 -- 172.21.15.111:0/2448178705 learned_addr learned my addr 172.21.15.111:0/2448178705 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f9ae0700 1 --1- 172.21.15.111:0/2448178705 >> v1:172.21.15.143:6789/0 conn(0x7f59f406c7c0 0x7f59f406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:50846/0 (socket says 172.21.15.111:50846) 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59faae2700 1 --2- 172.21.15.111:0/2448178705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 0x7f59f4161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371301054 0 0) 0x7f59f406d4b0 con 0x7f59f406c7c0 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0003570 con 0x7f59f406c7c0 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774773178 0 0) 0x7f59f406a7a0 con 0x7f59f40c6d20 2023-12-19T09:19:50.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f406d4b0 con 0x7f59f40c6d20 2023-12-19T09:19:50.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59faae2700 1 -- 172.21.15.111:0/2448178705 >> v1:172.21.15.143:6789/0 conn(0x7f59f406c7c0 legacy=0x7f59f406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59faae2700 1 -- 172.21.15.111:0/2448178705 >> v1:172.21.15.111:6789/0 conn(0x7f59f40c6d20 legacy=0x7f59f40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3973278816 0 0) 0x7f59f406d4b0 con 0x7f59f40c6d20 2023-12-19T09:19:50.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59faae2700 1 -- 172.21.15.111:0/2448178705 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f406da70 con 0x7f59f406bc40 2023-12-19T09:19:50.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59faae2700 1 --2- 172.21.15.111:0/2448178705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 0x7f59f4161dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009e90 tx=0x7f59ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce796af550d7efbf server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec00e070 con 0x7f59f406bc40 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec00b870 con 0x7f59f406bc40 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59f92df700 1 -- 172.21.15.111:0/2448178705 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0001490 con 0x7f59f406bc40 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 -- 172.21.15.111:0/2448178705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 msgr2=0x7f59f4161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 --2- 172.21.15.111:0/2448178705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 0x7f59f4161dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009e90 tx=0x7f59ec009780 comp rx=0 tx=0).stop 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 -- 172.21.15.111:0/2448178705 shutdown_connections 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 --2- 172.21.15.111:0/2448178705 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f406bc40 0x7f59f4161dd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 -- 172.21.15.111:0/2448178705 >> 172.21.15.111:0/2448178705 conn(0x7f59f4067bb0 msgr2=0x7f59f40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 -- 172.21.15.111:0/2448178705 shutdown_connections 2023-12-19T09:19:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.253+0000 7f59fc545700 1 -- 172.21.15.111:0/2448178705 wait complete. 2023-12-19T09:19:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.257+0000 7f59fc545700 1 Processor -- start 2023-12-19T09:19:50.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.257+0000 7f59fc545700 1 -- start start 2023-12-19T09:19:50.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59f406c7c0 0x7f59f40ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f40c6d20 0x7f59f40afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40aef60 con 0x7f59f406bc40 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4153290 con 0x7f59f406c7c0 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40acfb0 con 0x7f59f40c6d20 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57640/0 (socket says 172.21.15.111:57640) 2023-12-19T09:19:50.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 -- 172.21.15.111:0/1822063374 learned_addr learned my addr 172.21.15.111:0/1822063374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:50.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59f9ae0700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59f406c7c0 0x7f59f40ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59faae2700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f40c6d20 0x7f59f40afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 -- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f40c6d20 msgr2=0x7f59f40afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f40c6d20 0x7f59f40afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 -- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59f406c7c0 msgr2=0x7f59f40ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59f406c7c0 0x7f59f40ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 -- 172.21.15.111:0/1822063374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec009a20 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fa2e1700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f59f000cb90 tx=0x7f59f0001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0007150 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00072d0 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0007690 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f40ac3a0 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.261+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f40ac960 con 0x7f59f406bc40 2023-12-19T09:19:50.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f4071120 con 0x7f59f406bc40 2023-12-19T09:19:50.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 93) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f59f0004c50 con 0x7f59f406bc40 2023-12-19T09:19:50.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59ea7fc700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 0x7f59d8062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f59f007c580 con 0x7f59f406bc40 2023-12-19T09:19:50.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59f9ae0700 1 -- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 msgr2=0x7f59d8062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:50.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.265+0000 7f59f9ae0700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 0x7f59d8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:50.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.273+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f59f004c870 con 0x7f59f406bc40 2023-12-19T09:19:50.410 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:50.407+0000 7fae84749000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:19:50.436 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:50.433+0000 7f712b1ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:19:50.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.445+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f59f40ae6f0 con 0x7f59f406bc40 2023-12-19T09:19:50.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.453+0000 7f59ea7fc700 1 -- 172.21.15.111:0/1822063374 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f59f004fe90 con 0x7f59f406bc40 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 msgr2=0x7f59d8062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 0x7f59d8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 msgr2=0x7f59f40abdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f59f000cb90 tx=0x7f59f0001a50 comp rx=0 tx=0).stop 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 shutdown_connections 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f59d80605a0 0x7f59d8062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59f406c7c0 0x7f59f40ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59f40c6d20 0x7f59f40afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 --2- 172.21.15.111:0/1822063374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f406bc40 0x7f59f40abdc0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 >> 172.21.15.111:0/1822063374 conn(0x7f59f4067bb0 msgr2=0x7f59f41579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 shutdown_connections 2023-12-19T09:19:50.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.457+0000 7f59fc545700 1 -- 172.21.15.111:0/1822063374 wait complete. 2023-12-19T09:19:50.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:50.482 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T09:19:50.484 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:50.479+0000 7fae84749000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:19:50.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T09:19:50.513 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:50.509+0000 7f712b1ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:19:50.618 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:50.615+0000 7fae84749000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:50.665 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:50.661+0000 7f712b1ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:50.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 Processor -- start 2023-12-19T09:19:50.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 -- start start 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9768142910 0x7f976806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97680c6270 con 0x7f97680c4dc0 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976806a6a0 con 0x7f9768142910 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.669+0000 7f976db85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680aea30 con 0x7f97680c5210 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:55882/0 (socket says 172.21.15.111:55882) 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 -- 172.21.15.111:0/1934717840 learned_addr learned my addr 172.21.15.111:0/1934717840 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f97677fe700 1 --2- 172.21.15.111:0/1934717840 >> v2:172.21.15.143:3300/0 conn(0x7f9768142910 0x7f976806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 -- 172.21.15.111:0/1934717840 >> v2:172.21.15.143:3300/0 conn(0x7f9768142910 msgr2=0x7f976806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 --2- 172.21.15.111:0/1934717840 >> v2:172.21.15.143:3300/0 conn(0x7f9768142910 0x7f976806d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 -- 172.21.15.111:0/1934717840 >> v1:172.21.15.143:6789/0 conn(0x7f97680c4dc0 legacy=0x7f976815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 -- 172.21.15.111:0/1934717840 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976806d7d0 con 0x7f97680c5210 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9767fff700 1 --2- 172.21.15.111:0/1934717840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9760009f20 tx=0x7f9760009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e31b51b8a0cc341 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9766ffd700 1 -- 172.21.15.111:0/1934717840 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976000e070 con 0x7f97680c5210 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9766ffd700 1 -- 172.21.15.111:0/1934717840 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976000ba80 con 0x7f97680c5210 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f9766ffd700 1 -- 172.21.15.111:0/1934717840 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97600092e0 con 0x7f97680c5210 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 -- 172.21.15.111:0/1934717840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 msgr2=0x7f976815d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 --2- 172.21.15.111:0/1934717840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9760009f20 tx=0x7f9760009840 comp rx=0 tx=0).stop 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 -- 172.21.15.111:0/1934717840 shutdown_connections 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 --2- 172.21.15.111:0/1934717840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c5210 0x7f976815d3d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 --2- 172.21.15.111:0/1934717840 >> v2:172.21.15.143:3300/0 conn(0x7f9768142910 0x7f976806d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 -- 172.21.15.111:0/1934717840 >> 172.21.15.111:0/1934717840 conn(0x7f9768067bb0 msgr2=0x7f97680cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:50.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 -- 172.21.15.111:0/1934717840 shutdown_connections 2023-12-19T09:19:50.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.673+0000 7f976db85700 1 -- 172.21.15.111:0/1934717840 wait complete. 2023-12-19T09:19:50.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.677+0000 7f976db85700 1 Processor -- start 2023-12-19T09:19:50.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.677+0000 7f976db85700 1 -- start start 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c4dc0 0x7f9768156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768142910 0x7f97681557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768157e10 con 0x7f97680c5210 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768157450 con 0x7f9768142910 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f976db85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976815cfc0 con 0x7f97680c4dc0 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57648/0 (socket says 172.21.15.111:57648) 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 -- 172.21.15.111:0/582406666 learned_addr learned my addr 172.21.15.111:0/582406666 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c4dc0 0x7f9768156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f9767fff700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768142910 0x7f97681557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c4dc0 msgr2=0x7f9768156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c4dc0 0x7f9768156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768142910 msgr2=0x7f97681557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768142910 0x7f97681557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:50.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760009ae0 con 0x7f97680c5210 2023-12-19T09:19:50.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.681+0000 7f975ffff700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f975000c590 tx=0x7f975000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:50.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750011590 con 0x7f97680c5210 2023-12-19T09:19:50.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9768157660 con 0x7f97680c5210 2023-12-19T09:19:50.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750011710 con 0x7f97680c5210 2023-12-19T09:19:50.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750011ad0 con 0x7f97680c5210 2023-12-19T09:19:50.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97681579f0 con 0x7f97680c5210 2023-12-19T09:19:50.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.685+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9768071120 con 0x7f97680c5210 2023-12-19T09:19:50.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.689+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 93) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f975001b3b0 con 0x7f97680c5210 2023-12-19T09:19:50.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.689+0000 7f975f7fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:50.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.689+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f975007c030 con 0x7f97680c5210 2023-12-19T09:19:50.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.689+0000 7f97677fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:50.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.689+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:19:50.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.693+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f975004c2a0 con 0x7f97680c5210 2023-12-19T09:19:50.868 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:50.863+0000 7fae84749000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:19:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.877+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f976806b7f0 con 0x7f97480605d0 2023-12-19T09:19:50.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.893+0000 7f97677fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:50.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:50.893+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:19:50.935 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:50.931+0000 7fae84749000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:50.945 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:50.941+0000 7f712b1ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:19:51.021 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:51.017+0000 7f712b1ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:51.144 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:51.139+0000 7fae84749000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:19:51.230 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:51.227+0000 7fae84749000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:19:51.258 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:51.253+0000 7f712b1ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:19:51.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:51.297+0000 7f97677fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:51.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:51.297+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T09:19:51.312 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:51.307+0000 7fae84749000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:19:51.338 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:51.333+0000 7f712b1ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:19:51.429 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:51.425+0000 7f712b1ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:19:51.682 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:51.679+0000 7fae84749000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:19:51.840 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:51.837+0000 7f712b1ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:19:51.843 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:51.839+0000 7fae84749000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:19:52.011 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.005+0000 7f712b1ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:19:52.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:52.101+0000 7f97677fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:52.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:52.101+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T09:19:52.191 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.187+0000 7fae84749000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:19:52.388 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.381+0000 7f712b1ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:19:52.403 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.399+0000 7fae84749000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:52.493 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.487+0000 7fae84749000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:19:52.564 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.559+0000 7fae84749000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:19:52.611 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.605+0000 7f712b1ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:19:52.643 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.639+0000 7fae84749000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:19:52.701 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.697+0000 7f712b1ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:19:52.773 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.769+0000 7f712b1ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:19:52.850 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:52.845+0000 7f712b1ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:19:52.856 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.851+0000 7fae84749000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:52.938 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:52.935+0000 7fae84749000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:19:53.029 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:53.027+0000 7fae84749000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:19:53.065 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:53.061+0000 7f712b1ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:19:53.138 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:53.133+0000 7f712b1ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:19:53.221 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:53.219+0000 7fae84749000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:19:53.224 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:53.221+0000 7f712b1ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:19:53.305 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:53.303+0000 7fae84749000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:19:53.410 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:53.405+0000 7f712b1ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:19:53.498 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:53.493+0000 7f712b1ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:19:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:53.705+0000 7f97677fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:19:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:53.705+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T09:19:54.019 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.015+0000 7fae84749000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:19:54.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:54.033+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 94) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f975004c670 con 0x7f97680c5210 2023-12-19T09:19:54.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:54.033+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:19:54.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:54.033+0000 7f975f7fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:54.227 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae44e13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.227 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae44e13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae44e13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae47e19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae44e13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae44e13700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae47e19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae47e19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae47e19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:19:54.223+0000 7fae47e19700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:19:54.228 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:19:54.221+0000 7f712b1ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:19:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.045+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 95) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f9750056ab0 con 0x7f97680c5210 2023-12-19T09:19:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.045+0000 7f975f7fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.045+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f976806b7f0 con 0x7f97480605d0 2023-12-19T09:19:55.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.053+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.057+0000 7f97677fe700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9758008010 tx=0x7f9758008040 comp rx=0 tx=0).ready entity=mgr.8564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.165+0000 7f975f7fe700 1 -- 172.21.15.111:0/582406666 <== mgr.8564 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f976806b7f0 con 0x7f97480605d0 2023-12-19T09:19:55.169 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 msgr2=0x7f9748062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9758008010 tx=0x7f9758008040 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 msgr2=0x7f9768156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f975000c590 tx=0x7f975000c8a0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 shutdown_connections 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f97480605d0 0x7f9748062a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768142910 0x7f97681557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97680c5210 0x7f9768156f10 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 --2- 172.21.15.111:0/582406666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97680c4dc0 0x7f9768156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 >> 172.21.15.111:0/582406666 conn(0x7f9768067bb0 msgr2=0x7f976815c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:55.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 shutdown_connections 2023-12-19T09:19:55.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.169+0000 7f976db85700 1 -- 172.21.15.111:0/582406666 wait complete. 2023-12-19T09:19:55.201 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T09:19:55.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-12-19T09:19:55.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.365+0000 7fe09b7c5700 1 Processor -- start 2023-12-19T09:19:55.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.365+0000 7fe09b7c5700 1 -- start start 2023-12-19T09:19:55.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe09b7c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0940b1ed0 0x7fe0940684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe09b7c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe0940b0cb0 0x7fe094159b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe09b7c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe094067790 con 0x7fe094159790 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe09b7c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe094068030 con 0x7fe0940b0cb0 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe09b7c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940b1200 con 0x7fe0940b1ed0 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe099d62700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe0940b0cb0 0x7fe094159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0940b1ed0 0x7fe0940684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0940b1ed0 0x7fe0940684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:49658/0 (socket says 172.21.15.111:49658) 2023-12-19T09:19:55.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 -- 172.21.15.111:0/1779471445 learned_addr learned my addr 172.21.15.111:0/1779471445 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 -- 172.21.15.111:0/1779471445 >> v2:172.21.15.111:3300/0 conn(0x7fe0940b0cb0 msgr2=0x7fe094159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:19:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 --2- 172.21.15.111:0/1779471445 >> v2:172.21.15.111:3300/0 conn(0x7fe0940b0cb0 0x7fe094159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 -- 172.21.15.111:0/1779471445 >> v1:172.21.15.111:6789/0 conn(0x7fe094159790 legacy=0x7fe0940b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.369+0000 7fe098d60700 1 -- 172.21.15.111:0/1779471445 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940679e0 con 0x7fe0940b1ed0 2023-12-19T09:19:55.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe098d60700 1 --2- 172.21.15.111:0/1779471445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b1ed0 0x7fe0940684c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe09406be00 tx=0x7fe084009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f77145517d3a8e02 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe08bfff700 1 -- 172.21.15.111:0/1779471445 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084009e70 con 0x7fe0940b1ed0 2023-12-19T09:19:55.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe08bfff700 1 -- 172.21.15.111:0/1779471445 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08400b860 con 0x7fe0940b1ed0 2023-12-19T09:19:55.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe08bfff700 1 -- 172.21.15.111:0/1779471445 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08400bc20 con 0x7fe0940b1ed0 2023-12-19T09:19:55.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 -- 172.21.15.111:0/1779471445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b1ed0 msgr2=0x7fe0940684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/1779471445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b1ed0 0x7fe0940684c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe09406be00 tx=0x7fe084009250 comp rx=0 tx=0).stop 2023-12-19T09:19:55.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 -- 172.21.15.111:0/1779471445 shutdown_connections 2023-12-19T09:19:55.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/1779471445 >> v2:172.21.15.111:3300/0 conn(0x7fe0940b0cb0 0x7fe094159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/1779471445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b1ed0 0x7fe0940684c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 -- 172.21.15.111:0/1779471445 >> 172.21.15.111:0/1779471445 conn(0x7fe0940cdef0 msgr2=0x7fe0940ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:55.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 -- 172.21.15.111:0/1779471445 shutdown_connections 2023-12-19T09:19:55.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.373+0000 7fe09b7c5700 1 -- 172.21.15.111:0/1779471445 wait complete. 2023-12-19T09:19:55.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 Processor -- start 2023-12-19T09:19:55.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 -- start start 2023-12-19T09:19:55.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 0x7fe094142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0940b1ed0 0x7fe0940c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094151210 con 0x7fe094159790 2023-12-19T09:19:55.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941517d0 con 0x7fe0940b0cb0 2023-12-19T09:19:55.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.377+0000 7fe09b7c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940c9e80 con 0x7fe0940b1ed0 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58454/0 (socket says 172.21.15.111:58454) 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 -- 172.21.15.111:0/447082715 learned_addr learned my addr 172.21.15.111:0/447082715 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe098d60700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0940b1ed0 0x7fe0940c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099561700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 0x7fe094142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 -- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0940b1ed0 msgr2=0x7fe0940c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0940b1ed0 0x7fe0940c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 -- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 msgr2=0x7fe094142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 0x7fe094142b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 -- 172.21.15.111:0/447082715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840094f0 con 0x7fe094159790 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099561700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 0x7fe094142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:19:55.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe099d62700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009ef0 tx=0x7fe08c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00cc20 con 0x7fe094159790 2023-12-19T09:19:55.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c00f040 con 0x7fe094159790 2023-12-19T09:19:55.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0940c8be0 con 0x7fe094159790 2023-12-19T09:19:55.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c014480 con 0x7fe094159790 2023-12-19T09:19:55.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe094151460 con 0x7fe094159790 2023-12-19T09:19:55.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.381+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094151960 con 0x7fe094159790 2023-12-19T09:19:55.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.385+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 95) v1 ==== 79816+0+0 (secure 0 0 0) 0x7fe08c014830 con 0x7fe094159790 2023-12-19T09:19:55.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.385+0000 7fe0897fa700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 0x7fe0780628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.385+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fe08c07d380 con 0x7fe094159790 2023-12-19T09:19:55.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.389+0000 7fe099561700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 0x7fe0780628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.389+0000 7fe099561700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 0x7fe0780628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe090007fd0 tx=0x7fe090009040 comp rx=0 tx=0).ready entity=mgr.8564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.393+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe08c04d6d0 con 0x7fe094159790 2023-12-19T09:19:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.581+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe094071120 con 0x7fe0780603f0 2023-12-19T09:19:55.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.689+0000 7fe0897fa700 1 -- 172.21.15.111:0/447082715 <== mgr.8564 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7fe094071120 con 0x7fe0780603f0 2023-12-19T09:19:55.692 INFO:teuthology.orchestra.run.smithi111.stdout:Retention added to path snap_test_dir1 2023-12-19T09:19:55.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.689+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 msgr2=0x7fe0780628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.689+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 0x7fe0780628a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe090007fd0 tx=0x7fe090009040 comp rx=0 tx=0).stop 2023-12-19T09:19:55.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.689+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 msgr2=0x7fe0940c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.689+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009ef0 tx=0x7fe08c00b040 comp rx=0 tx=0).stop 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 shutdown_connections 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe094159790 0x7fe0940c85b0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0940b1ed0 0x7fe0940c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe0780603f0 0x7fe0780628a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 --2- 172.21.15.111:0/447082715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0940b0cb0 0x7fe094142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 >> 172.21.15.111:0/447082715 conn(0x7fe0940cdef0 msgr2=0x7fe094158740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 shutdown_connections 2023-12-19T09:19:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.693+0000 7fe09b7c5700 1 -- 172.21.15.111:0/447082715 wait complete. 2023-12-19T09:19:55.716 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-12-19T09:19:55.717 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-09_20_00 in ~5s... 2023-12-19T09:19:55.717 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-12-19T09:19:55.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T09:19:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.873+0000 7f3b5359e700 1 Processor -- start 2023-12-19T09:19:55.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.873+0000 7f3b5359e700 1 -- start start 2023-12-19T09:19:55.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b541558d0 0x7f3b541569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b540ac2b0 con 0x7f3b540abe10 2023-12-19T09:19:55.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b540cbfb0 con 0x7f3b540c6ff0 2023-12-19T09:19:55.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b54156fa0 con 0x7f3b541558d0 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b541558d0 0x7f3b541569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b541558d0 0x7f3b541569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58470/0 (socket says 172.21.15.111:58470) 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 -- 172.21.15.111:0/861329224 learned_addr learned my addr 172.21.15.111:0/861329224 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 -- 172.21.15.111:0/861329224 >> v1:172.21.15.143:6789/0 conn(0x7f3b540c6ff0 legacy=0x7f3b540c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 -- 172.21.15.111:0/861329224 >> v1:172.21.15.111:6789/0 conn(0x7f3b540abe10 legacy=0x7f3b540ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 -- 172.21.15.111:0/861329224 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b540cc290 con 0x7f3b541558d0 2023-12-19T09:19:55.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5259c700 1 --2- 172.21.15.111:0/861329224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b541558d0 0x7f3b541569b0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f3b440099b0 tx=0x7f3b44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c64af70fae55ec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5159a700 1 -- 172.21.15.111:0/861329224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400b860 con 0x7f3b541558d0 2023-12-19T09:19:55.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5159a700 1 -- 172.21.15.111:0/861329224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4400b9c0 con 0x7f3b541558d0 2023-12-19T09:19:55.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5159a700 1 -- 172.21.15.111:0/861329224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400bd60 con 0x7f3b541558d0 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- 172.21.15.111:0/861329224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b541558d0 msgr2=0x7f3b541569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 --2- 172.21.15.111:0/861329224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b541558d0 0x7f3b541569b0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f3b440099b0 tx=0x7f3b44009250 comp rx=0 tx=0).stop 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- 172.21.15.111:0/861329224 shutdown_connections 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 --2- 172.21.15.111:0/861329224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b541558d0 0x7f3b541569b0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- 172.21.15.111:0/861329224 >> 172.21.15.111:0/861329224 conn(0x7f3b5406b5b0 msgr2=0x7f3b5406b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- 172.21.15.111:0/861329224 shutdown_connections 2023-12-19T09:19:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.877+0000 7f3b5359e700 1 -- 172.21.15.111:0/861329224 wait complete. 2023-12-19T09:19:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 Processor -- start 2023-12-19T09:19:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 -- start start 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b540c6ff0 0x7f3b541419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b541558d0 0x7f3b540b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b540b18e0 con 0x7f3b540abe10 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b540b0cd0 con 0x7f3b540c6ff0 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b540b0fb0 con 0x7f3b541558d0 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58484/0 (socket says 172.21.15.111:58484) 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 -- 172.21.15.111:0/607493096 learned_addr learned my addr 172.21.15.111:0/607493096 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b52d9d700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b541558d0 0x7f3b540b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b51d9b700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b540c6ff0 0x7f3b541419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 -- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b541558d0 msgr2=0x7f3b540b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b541558d0 0x7f3b540b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 -- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b540c6ff0 msgr2=0x7f3b541419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b540c6ff0 0x7f3b541419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.881+0000 7f3b5259c700 1 -- 172.21.15.111:0/607493096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b440094f0 con 0x7f3b540abe10 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b5259c700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406b1a0 tx=0x7f3b44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44024440 con 0x7f3b540abe10 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b440245a0 con 0x7f3b540abe10 2023-12-19T09:19:55.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44024920 con 0x7f3b540abe10 2023-12-19T09:19:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b540b1290 con 0x7f3b540abe10 2023-12-19T09:19:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b540b0910 con 0x7f3b540abe10 2023-12-19T09:19:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5404be40 con 0x7f3b540abe10 2023-12-19T09:19:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.885+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 95) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f3b4401a420 con 0x7f3b540abe10 2023-12-19T09:19:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.889+0000 7f3b427fc700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 0x7f3b340628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:19:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.889+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f3b4407e640 con 0x7f3b540abe10 2023-12-19T09:19:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.889+0000 7f3b51d9b700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 0x7f3b340628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:19:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.889+0000 7f3b51d9b700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 0x7f3b340628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b540ab3b0 tx=0x7f3b48007ec0 comp rx=0 tx=0).ready entity=mgr.8564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:19:55.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:55.893+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3b4404eb10 con 0x7f3b540abe10 2023-12-19T09:19:56.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.081+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b540b1a20 con 0x7f3b340603f0 2023-12-19T09:19:56.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b427fc700 1 -- 172.21.15.111:0/607493096 <== mgr.8564 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f3b48002480 con 0x7f3b340603f0 2023-12-19T09:19:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:19:56.101 INFO:teuthology.orchestra.run.smithi111.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 msgr2=0x7f3b340628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 0x7f3b340628a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b540ab3b0 tx=0x7f3b48007ec0 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 msgr2=0x7f3b540b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406b1a0 tx=0x7f3b44017000 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.097+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 shutdown_connections 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3b340603f0 0x7f3b340628a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b540c6ff0 0x7f3b541419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b541558d0 0x7f3b540b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 --2- 172.21.15.111:0/607493096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b540abe10 0x7f3b540b03d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 >> 172.21.15.111:0/607493096 conn(0x7f3b5406b5b0 msgr2=0x7f3b54067c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 shutdown_connections 2023-12-19T09:19:56.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:19:56.101+0000 7f3b5359e700 1 -- 172.21.15.111:0/607493096 wait complete. 2023-12-19T09:19:56.126 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T09:19:56.127 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-12-19T09:19:56.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:56.127 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:19:56.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T09:19:56.158 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:19:56.158 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:19:57.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:57.160 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:19:57.194 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T09:19:57.194 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:19:57.194 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:19:57.364 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:19:57.372 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:19:58.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:58.196 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:19:58.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T09:19:58.232 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:19:58.232 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:19:59.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:19:59.233 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:19:59.269 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T09:19:59.269 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:19:59.269 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:00.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:00.270 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:00.302 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-12-19T09:20:00.302 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:00.302 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:00.711 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:20:00.707+0000 7fae36df7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:20:00.711 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:20:00.707+0000 7fae36df7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:20:00.712 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:20:00.707+0000 7fae36df7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:20:00.712 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:20:00.707+0000 7fae36df7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:20:00.712 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:20:00.707+0000 7fae36df7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:20:01.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:01.304 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:01.333 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:01.334 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:01.334 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:01.335 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:01.335 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-12-19-09_20_00_UTC'] 2023-12-19T09:20:01.335 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702977595.7170641, now = 1702977601.33524, timo = 5 2023-12-19T09:20:02.335 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-12-19-09_20_00 in ~65s... 2023-12-19T09:20:02.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:02.335 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:02.369 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:02.371 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:02.371 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:02.371 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:03.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:03.372 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:03.405 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:03.406 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:03.406 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:03.406 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:04.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:04.407 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:04.440 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:04.441 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:04.441 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:04.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:05.443 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:05.443 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:05.477 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:05.478 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:05.478 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:05.478 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:06.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:06.479 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:06.511 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:06.512 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:06.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:06.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:07.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:07.514 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:07.545 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:07.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:07.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:07.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:08.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:08.548 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:08.579 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:08.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:08.581 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:08.581 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:09.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:09.582 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:09.616 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:09.618 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:09.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:09.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:10.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:10.618 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:10.652 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:10.654 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:10.654 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:10.654 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:11.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:11.655 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:11.687 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:11.689 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:11.689 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:11.689 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:12.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:12.690 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:12.720 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:12.722 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:12.722 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:12.722 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:13.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:13.723 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:13.754 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:13.755 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:13.755 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:13.755 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:14.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:14.756 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:14.785 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:14.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:14.786 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:14.786 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:15.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:15.787 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:15.819 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:15.821 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:15.821 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:15.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:16.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:16.822 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:16.855 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:16.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:16.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:16.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:17.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:17.858 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:17.889 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:17.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:17.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:17.890 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:18.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:18.891 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:18.921 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:18.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:18.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:18.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:19.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:19.923 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:19.954 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:19.956 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:19.956 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:19.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:20.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:20.957 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:20.992 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:20.994 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:20.994 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:20.994 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:21.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:21.995 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:22.028 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:22.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:22.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:22.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:23.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:23.032 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:23.065 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:23.067 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:23.067 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:23.067 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:24.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:24.068 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:24.103 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:24.104 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:24.104 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:24.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:25.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:25.106 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:25.135 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:25.136 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:25.137 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:25.137 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:26.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:26.138 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:26.172 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:26.173 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:26.173 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:26.173 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:27.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:27.174 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:27.207 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:27.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:27.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:27.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:27.528 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:20:27.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:20:28.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:28.210 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:28.245 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:28.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:28.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:28.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:29.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:29.247 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:29.278 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:29.280 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:29.280 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:29.280 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:30.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:30.282 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:30.316 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:30.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:30.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:30.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:31.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:31.322 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:31.356 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:31.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:31.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:31.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:32.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:32.359 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:32.391 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:32.392 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:32.392 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:32.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:33.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:33.394 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:33.428 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:33.429 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:33.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:33.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:34.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:34.430 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:34.464 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:34.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:34.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:34.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:35.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:35.468 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:35.499 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:35.501 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:35.501 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:35.501 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:36.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:36.502 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:36.535 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:36.537 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:36.537 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:36.537 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:37.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:37.538 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:37.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:37.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:37.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:38.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:38.568 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:38.600 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:38.602 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:38.602 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:38.602 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:39.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:39.603 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:39.634 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:39.636 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:39.636 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:39.636 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:40.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:40.637 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:40.671 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:40.672 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:40.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:40.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:41.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:41.673 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:41.705 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:41.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:41.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:41.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:42.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:42.708 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:42.735 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:42.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:42.737 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:42.737 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:43.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:43.737 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:43.769 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:43.770 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:43.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:43.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:44.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:44.772 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:44.801 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:44.804 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:44.804 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:44.804 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:45.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:45.805 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:45.835 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:45.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:45.837 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:45.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:46.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:46.838 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:46.872 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:46.873 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:46.873 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:46.873 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:47.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:47.875 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:47.909 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:47.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:47.910 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:47.910 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:48.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:48.911 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:48.945 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:48.946 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:48.947 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:48.947 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:49.948 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:49.948 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:49.981 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:49.983 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:49.983 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:49.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:50.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:50.984 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:51.018 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:51.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:51.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:51.020 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:52.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:52.021 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:52.049 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:52.053 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:52.053 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:52.053 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:53.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:53.054 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:53.089 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:53.089 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:53.089 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:54.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:54.093 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:54.123 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:54.124 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:54.125 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:54.125 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:55.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:55.126 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:55.158 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:55.160 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:55.160 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:55.161 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:56.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:56.162 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:56.197 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:56.198 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:56.198 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:56.198 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:57.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:57.199 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:57.229 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:57.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:57.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:57.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:57.691 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:20:57.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:20:58.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:58.232 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:58.264 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:58.266 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:58.266 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:58.266 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:20:59.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:20:59.268 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:20:59.301 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:20:59.302 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:20:59.303 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:20:59.303 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:21:00.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:00.303 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:21:00.336 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_20_00_UTC 2023-12-19T09:21:00.338 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:21:00.338 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-12-19T09:21:00.338 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-12-19T09:21:01.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:01.339 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:21:01.373 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_21_00_UTC 2023-12-19T09:21:01.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-12-19-09_21_00_UTC'} 2023-12-19T09:21:01.375 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-12-19-09_21_00_UTC'} 2023-12-19T09:21:01.375 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-12-19-09_20_00_UTC'} 2023-12-19T09:21:01.375 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-12-19-09_20_00_UTC'] 2023-12-19T09:21:01.375 DEBUG:tasks.cephfs.test_snap_schedules:exec=1702977595.7170641, now = 1702977661.3757615, timo = 65 2023-12-19T09:21:02.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T09:21:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.556+0000 7f56bc985700 1 Processor -- start 2023-12-19T09:21:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.556+0000 7f56bc985700 1 -- start start 2023-12-19T09:21:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56bc985700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f56b80cb130 0x7f56b80cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56bc985700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56b806a6a0 0x7f56b806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56bc985700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b80b0ac0 con 0x7f56b815c220 2023-12-19T09:21:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56bc985700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b806b030 con 0x7f56b806a6a0 2023-12-19T09:21:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56bc985700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b806b210 con 0x7f56b80cb130 2023-12-19T09:21:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56b806a6a0 0x7f56b806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56b806a6a0 0x7f56b806aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39174/0 (socket says 172.21.15.111:39174) 2023-12-19T09:21:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 -- 172.21.15.111:0/3176051274 learned_addr learned my addr 172.21.15.111:0/3176051274 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b659c700 1 --2- 172.21.15.111:0/3176051274 >> v2:172.21.15.143:3300/0 conn(0x7f56b80cb130 0x7f56b80cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 -- 172.21.15.111:0/3176051274 >> v2:172.21.15.143:3300/0 conn(0x7f56b80cb130 msgr2=0x7f56b80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 --2- 172.21.15.111:0/3176051274 >> v2:172.21.15.143:3300/0 conn(0x7f56b80cb130 0x7f56b80cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 -- 172.21.15.111:0/3176051274 >> v1:172.21.15.143:6789/0 conn(0x7f56b815c220 legacy=0x7f56b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 -- 172.21.15.111:0/3176051274 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80ca810 con 0x7f56b806a6a0 2023-12-19T09:21:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.560+0000 7f56b5d9b700 1 --2- 172.21.15.111:0/3176051274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b806a6a0 0x7f56b806aaf0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f56b80694e0 tx=0x7f56ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f8029a170751d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56b559a700 1 -- 172.21.15.111:0/3176051274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac00b860 con 0x7f56b806a6a0 2023-12-19T09:21:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56b559a700 1 -- 172.21.15.111:0/3176051274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac00b9c0 con 0x7f56b806a6a0 2023-12-19T09:21:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56b559a700 1 -- 172.21.15.111:0/3176051274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac00bd60 con 0x7f56b806a6a0 2023-12-19T09:21:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- 172.21.15.111:0/3176051274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b806a6a0 msgr2=0x7f56b806aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 --2- 172.21.15.111:0/3176051274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b806a6a0 0x7f56b806aaf0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f56b80694e0 tx=0x7f56ac009250 comp rx=0 tx=0).stop 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- 172.21.15.111:0/3176051274 shutdown_connections 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 --2- 172.21.15.111:0/3176051274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b806a6a0 0x7f56b806aaf0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 --2- 172.21.15.111:0/3176051274 >> v2:172.21.15.143:3300/0 conn(0x7f56b80cb130 0x7f56b80cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- 172.21.15.111:0/3176051274 >> 172.21.15.111:0/3176051274 conn(0x7f56b8067bb0 msgr2=0x7f56b80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- 172.21.15.111:0/3176051274 shutdown_connections 2023-12-19T09:21:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- 172.21.15.111:0/3176051274 wait complete. 2023-12-19T09:21:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 Processor -- start 2023-12-19T09:21:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.564+0000 7f56bc985700 1 -- start start 2023-12-19T09:21:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56b806a6a0 0x7f56b8157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56b815c220 0x7f56b80b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80c87f0 con 0x7f56b80cb130 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80c9880 con 0x7f56b815c220 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b816dcb0 con 0x7f56b806a6a0 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39176/0 (socket says 172.21.15.111:39176) 2023-12-19T09:21:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 -- 172.21.15.111:0/4148723220 learned_addr learned my addr 172.21.15.111:0/4148723220 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b659c700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56b806a6a0 0x7f56b8157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b6d9d700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56b815c220 0x7f56b80b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 -- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56b806a6a0 msgr2=0x7f56b8157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56b806a6a0 0x7f56b8157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 -- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56b815c220 msgr2=0x7f56b80b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56b815c220 0x7f56b80b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 -- 172.21.15.111:0/4148723220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac0094f0 con 0x7f56b80cb130 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56b5d9b700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f56ac006fd0 tx=0x7f56ac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac011e20 con 0x7f56b80cb130 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac01a630 con 0x7f56b80cb130 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac01a9f0 con 0x7f56b80cb130 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b816d9d0 con 0x7f56b80cb130 2023-12-19T09:21:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.568+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8169c50 con 0x7f56b80cb130 2023-12-19T09:21:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.572+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8169750 con 0x7f56b80cb130 2023-12-19T09:21:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.572+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 97) v1 ==== 80194+0+0 (secure 0 0 0) 0x7f56ac02d440 con 0x7f56b80cb130 2023-12-19T09:21:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.572+0000 7f56a67fc700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 0x7f5698062cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.576+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f56ac07ea90 con 0x7f56b80cb130 2023-12-19T09:21:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.576+0000 7f56b659c700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 0x7f5698062cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.576+0000 7f56b659c700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 0x7f5698062cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56a8006fd0 tx=0x7f56a8008040 comp rx=0 tx=0).ready entity=mgr.8564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.580+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f56ac04e970 con 0x7f56b80cb130 2023-12-19T09:21:02.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.772+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f56b8071120 con 0x7f5698060810 2023-12-19T09:21:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.888+0000 7f56a67fc700 1 -- 172.21.15.111:0/4148723220 <== mgr.8564 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f56b8071120 con 0x7f5698060810 2023-12-19T09:21:02.894 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path snap_test_dir1 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 msgr2=0x7f5698062cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 0x7f5698062cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f56a8006fd0 tx=0x7f56a8008040 comp rx=0 tx=0).stop 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 msgr2=0x7f56b80b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f56ac006fd0 tx=0x7f56ac017000 comp rx=0 tx=0).stop 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 shutdown_connections 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5698060810 0x7f5698062cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56b815c220 0x7f56b80b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56b80cb130 0x7f56b80b4f90 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 --2- 172.21.15.111:0/4148723220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56b806a6a0 0x7f56b8157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 >> 172.21.15.111:0/4148723220 conn(0x7f56b8067bb0 msgr2=0x7f56b815b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 shutdown_connections 2023-12-19T09:21:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:02.892+0000 7f56bc985700 1 -- 172.21.15.111:0/4148723220 wait complete. 2023-12-19T09:21:02.920 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T09:21:02.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:02.920 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:21:02.948 INFO:teuthology.orchestra.run.smithi111.stdout:scheduled-2023-12-19-09_21_00_UTC 2023-12-19T09:21:02.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-12-19-09_21_00_UTC 2023-12-19T09:21:02.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:02.950 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-12-19-09_21_00_UTC) 2023-12-19T09:21:03.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:03.031 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T09:21:03.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T09:21:03.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.300+0000 7fda26b63700 1 Processor -- start 2023-12-19T09:21:03.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda26b63700 1 -- start start 2023-12-19T09:21:03.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda26b63700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fda200cc040 0x7fda20159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:03.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda26b63700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200cbbb0 con 0x7fda200ce2d0 2023-12-19T09:21:03.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda26b63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2006dc90 con 0x7fda200caa10 2023-12-19T09:21:03.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda26b63700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200cafa0 con 0x7fda200cc040 2023-12-19T09:21:03.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.304+0000 7fda25b61700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fda200ce2d0 0x7fda20158900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56088/0 (socket says 172.21.15.111:56088) 2023-12-19T09:21:03.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda25b61700 1 -- 172.21.15.111:0/1452123176 learned_addr learned my addr 172.21.15.111:0/1452123176 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:03.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26362700 1 --2- 172.21.15.111:0/1452123176 >> v2:172.21.15.111:3300/0 conn(0x7fda200cc040 0x7fda20159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda24b5f700 1 -- 172.21.15.111:0/1452123176 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045170966 0 0) 0x7fda2006dc90 con 0x7fda200caa10 2023-12-19T09:21:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda24b5f700 1 -- 172.21.15.111:0/1452123176 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003570 con 0x7fda200caa10 2023-12-19T09:21:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26362700 1 -- 172.21.15.111:0/1452123176 >> v1:172.21.15.143:6789/0 conn(0x7fda200caa10 legacy=0x7fda2006a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26362700 1 -- 172.21.15.111:0/1452123176 >> v1:172.21.15.111:6789/0 conn(0x7fda200ce2d0 legacy=0x7fda20158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26362700 1 -- 172.21.15.111:0/1452123176 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200cb280 con 0x7fda200cc040 2023-12-19T09:21:03.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26362700 1 --2- 172.21.15.111:0/1452123176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda20159010 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e60 tx=0x7fda18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89f3d481a66fd80 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:03.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda24b5f700 1 -- 172.21.15.111:0/1452123176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1800e070 con 0x7fda200cc040 2023-12-19T09:21:03.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda24b5f700 1 -- 172.21.15.111:0/1452123176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1800bb60 con 0x7fda200cc040 2023-12-19T09:21:03.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda24b5f700 1 -- 172.21.15.111:0/1452123176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18012430 con 0x7fda200cc040 2023-12-19T09:21:03.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 -- 172.21.15.111:0/1452123176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 msgr2=0x7fda20159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 --2- 172.21.15.111:0/1452123176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda20159010 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e60 tx=0x7fda18009780 comp rx=0 tx=0).stop 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 -- 172.21.15.111:0/1452123176 shutdown_connections 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 --2- 172.21.15.111:0/1452123176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda20159010 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 -- 172.21.15.111:0/1452123176 >> 172.21.15.111:0/1452123176 conn(0x7fda20067bb0 msgr2=0x7fda200cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 -- 172.21.15.111:0/1452123176 shutdown_connections 2023-12-19T09:21:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.308+0000 7fda26b63700 1 -- 172.21.15.111:0/1452123176 wait complete. 2023-12-19T09:21:03.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.312+0000 7fda26b63700 1 Processor -- start 2023-12-19T09:21:03.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.312+0000 7fda26b63700 1 -- start start 2023-12-19T09:21:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda200caa10 0x7fda20144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda200ce2d0 0x7fda200c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20150b50 con 0x7fda200cc040 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20150820 con 0x7fda200ce2d0 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2014fd30 con 0x7fda200caa10 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39198/0 (socket says 172.21.15.111:39198) 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 -- 172.21.15.111:0/1625954998 learned_addr learned my addr 172.21.15.111:0/1625954998 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25b61700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda200caa10 0x7fda20144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26362700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda200ce2d0 0x7fda200c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:03.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 -- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda200caa10 msgr2=0x7fda20144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda200caa10 0x7fda20144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 -- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda200ce2d0 msgr2=0x7fda200c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda200ce2d0 0x7fda200c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 -- 172.21.15.111:0/1625954998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009a20 con 0x7fda200cc040 2023-12-19T09:21:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda25360700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fda1000c790 tx=0x7fda10007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10010510 con 0x7fda200cc040 2023-12-19T09:21:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda10010670 con 0x7fda200cc040 2023-12-19T09:21:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10010a10 con 0x7fda200cc040 2023-12-19T09:21:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200c9c30 con 0x7fda200cc040 2023-12-19T09:21:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.316+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200ca1f0 con 0x7fda200cc040 2023-12-19T09:21:03.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.320+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2004be40 con 0x7fda200cc040 2023-12-19T09:21:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.320+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 97) v1 ==== 80194+0+0 (secure 0 0 0) 0x7fda1001b3b0 con 0x7fda200cc040 2023-12-19T09:21:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.320+0000 7fda15ffb700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 0x7fda04062c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.320+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fda1007c960 con 0x7fda200cc040 2023-12-19T09:21:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.324+0000 7fda25b61700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 0x7fda04062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:03.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.324+0000 7fda25b61700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 0x7fda04062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda1c001950 tx=0x7fda1c009040 comp rx=0 tx=0).ready entity=mgr.8564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:03.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.324+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fda1004c7a0 con 0x7fda200cc040 2023-12-19T09:21:03.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.540+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fda200681f0 con 0x7fda200cc040 2023-12-19T09:21:03.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.764+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fda1004cb70 con 0x7fda200cc040 2023-12-19T09:21:03.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.764+0000 7fda15ffb700 1 -- 172.21.15.111:0/1625954998 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v98) v1 ==== 93+0+0 (secure 0 0 0) 0x7fda10050460 con 0x7fda200cc040 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 msgr2=0x7fda04062c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 0x7fda04062c90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda1c001950 tx=0x7fda1c009040 comp rx=0 tx=0).stop 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 msgr2=0x7fda200c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fda1000c790 tx=0x7fda10007e10 comp rx=0 tx=0).stop 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 shutdown_connections 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fda040607e0 0x7fda04062c90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda200ce2d0 0x7fda200c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda200cc040 0x7fda200c9040 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 --2- 172.21.15.111:0/1625954998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda200caa10 0x7fda20144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:03.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 >> 172.21.15.111:0/1625954998 conn(0x7fda20067bb0 msgr2=0x7fda200c4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:03.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 shutdown_connections 2023-12-19T09:21:03.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:03.768+0000 7fda26b63700 1 -- 172.21.15.111:0/1625954998 wait complete. 2023-12-19T09:21:03.794 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:21:03.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:03.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:03.873 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:21:03.874 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:21:03.874 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:21:03.874 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:03.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[61451]: fuse finished with error 0 and tester_r 0 2023-12-19T09:21:03.916 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:21:03.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:03.916 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:21:03.966 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:03.960+0000 7fa11af61000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:03.980 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:03.973+0000 7f1e2bb61000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:04.107 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:21:04.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:04.107 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T09:21:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:04.127 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T09:21:04.127 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:21:04.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:04.128 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T09:21:04.170 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T09:21:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:04.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:21:04.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:04.172 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T09:21:04.190 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T09:21:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:04.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:04.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 Processor -- start 2023-12-19T09:21:04.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 -- start start 2023-12-19T09:21:04.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0b7c0cc3b0 0x7f0b7c0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0cd5a0 con 0x7f0b7c156c90 2023-12-19T09:21:04.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0cc990 con 0x7f0b7c157060 2023-12-19T09:21:04.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b8166e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c157470 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b7a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0b7c0cc3b0 0x7f0b7c0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b7affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0b7c156c90 0x7f0b7c0c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56104/0 (socket says 172.21.15.111:56104) 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.372+0000 7f0b7affd700 1 -- 172.21.15.111:0/979510141 learned_addr learned my addr 172.21.15.111:0/979510141 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b7a7fc700 1 -- 172.21.15.111:0/979510141 >> v1:172.21.15.143:6789/0 conn(0x7f0b7c157060 legacy=0x7f0b7c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b7a7fc700 1 -- 172.21.15.111:0/979510141 >> v1:172.21.15.111:6789/0 conn(0x7f0b7c156c90 legacy=0x7f0b7c0c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b7a7fc700 1 -- 172.21.15.111:0/979510141 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0ccc70 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b7a7fc700 1 --2- 172.21.15.111:0/979510141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c0cda30 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0b700098e0 tx=0x7f0b700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75425732d7cf569 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b79ffb700 1 -- 172.21.15.111:0/979510141 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70009e80 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b79ffb700 1 -- 172.21.15.111:0/979510141 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7000b860 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b79ffb700 1 -- 172.21.15.111:0/979510141 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000bbe0 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 -- 172.21.15.111:0/979510141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 msgr2=0x7f0b7c0cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 --2- 172.21.15.111:0/979510141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c0cda30 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f0b700098e0 tx=0x7f0b700091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 -- 172.21.15.111:0/979510141 shutdown_connections 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 --2- 172.21.15.111:0/979510141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c0cda30 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 -- 172.21.15.111:0/979510141 >> 172.21.15.111:0/979510141 conn(0x7f0b7c06c7c0 msgr2=0x7f0b7c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 -- 172.21.15.111:0/979510141 shutdown_connections 2023-12-19T09:21:04.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.376+0000 7f0b8166e700 1 -- 172.21.15.111:0/979510141 wait complete. 2023-12-19T09:21:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 Processor -- start 2023-12-19T09:21:04.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 -- start start 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b7c156c90 0x7f0b7c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 0x7f0b7c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0c5f90 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0c50a0 con 0x7f0b7c157060 2023-12-19T09:21:04.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b8166e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0c5380 con 0x7f0b7c156c90 2023-12-19T09:21:04.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39224/0 (socket says 172.21.15.111:39224) 2023-12-19T09:21:04.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 -- 172.21.15.111:0/2802481437 learned_addr learned my addr 172.21.15.111:0/2802481437 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7a7fc700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b7c156c90 0x7f0b7c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7b7fe700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 0x7f0b7c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b7c156c90 msgr2=0x7f0b7c158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b7c156c90 0x7f0b7c158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 msgr2=0x7f0b7c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 0x7f0b7c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7affd700 1 -- 172.21.15.111:0/2802481437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70009420 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.380+0000 7f0b7b7fe700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 0x7f0b7c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b7affd700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c7c0 tx=0x7f0b6c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c005b30 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c005c90 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c0c5660 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c004a80 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c0c5c20 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c071120 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.384+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f0b6c0070d0 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.388+0000 7f0b62ffd700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 0x7f0b68062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.388+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f0b6c07c390 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.388+0000 7f0b7a7fc700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 msgr2=0x7f0b68062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:04.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.388+0000 7f0b7a7fc700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 0x7f0b68062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.392+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0b6c049be0 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.526 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:04.520+0000 7fa11af61000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:04.591 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:04.585+0000 7f1e2bb61000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:04.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.592+0000 7f0b7a7fc700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 msgr2=0x7f0b68062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:04.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.592+0000 7f0b7a7fc700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 0x7f0b68062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:04.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.600+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7c0ccf50 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.605 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:04.600+0000 7fa11af61000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:04.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.600+0000 7f0b62ffd700 1 -- 172.21.15.111:0/2802481437 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7f0b6c04c270 con 0x7f0b7c0cc3b0 2023-12-19T09:21:04.606 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:04.606 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:21:04.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 msgr2=0x7f0b68062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:04.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 0x7f0b68062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 msgr2=0x7f0b7c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c7c0 tx=0x7f0b6c007f70 comp rx=0 tx=0).stop 2023-12-19T09:21:04.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 shutdown_connections 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0b68060790 0x7f0b68062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b7c157060 0x7f0b7c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b7c156c90 0x7f0b7c158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 --2- 172.21.15.111:0/2802481437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c0cc3b0 0x7f0b7c154c00 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 >> 172.21.15.111:0/2802481437 conn(0x7f0b7c06c7c0 msgr2=0x7f0b7c06cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 shutdown_connections 2023-12-19T09:21:04.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.604+0000 7f0b8166e700 1 -- 172.21.15.111:0/2802481437 wait complete. 2023-12-19T09:21:04.610 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:21:04.632 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:04.673 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:04.669+0000 7f1e2bb61000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:04.753 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:04.748+0000 7fa11af61000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:04.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.784+0000 7f25092cd700 1 Processor -- start 2023-12-19T09:21:04.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.784+0000 7f25092cd700 1 -- start start 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2504158f70 0x7f2504067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2504068c70 con 0x7f2504069100 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2504068060 con 0x7f2504158f70 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2504068340 con 0x7f2504159340 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2504158f70 0x7f2504067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2504158f70 0x7f2504067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39232/0 (socket says 172.21.15.111:39232) 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 -- 172.21.15.111:0/2839077091 learned_addr learned my addr 172.21.15.111:0/2839077091 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25037fe700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 -- 172.21.15.111:0/2839077091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 msgr2=0x7f25040cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 -- 172.21.15.111:0/2839077091 >> v1:172.21.15.143:6789/0 conn(0x7f2504069100 legacy=0x7f25040cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 -- 172.21.15.111:0/2839077091 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504068620 con 0x7f2504158f70 2023-12-19T09:21:04.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2502ffd700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f2504067980 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f24f40099b0 tx=0x7f24f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d0034cbfd2b9e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2501ffb700 1 -- 172.21.15.111:0/2839077091 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f400b860 con 0x7f2504158f70 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2501ffb700 1 -- 172.21.15.111:0/2839077091 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f400b9c0 con 0x7f2504158f70 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f2501ffb700 1 -- 172.21.15.111:0/2839077091 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f400bd60 con 0x7f2504158f70 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- 172.21.15.111:0/2839077091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 msgr2=0x7f2504067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f2504067980 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f24f40099b0 tx=0x7f24f4009250 comp rx=0 tx=0).stop 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- 172.21.15.111:0/2839077091 shutdown_connections 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 --2- 172.21.15.111:0/2839077091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f2504067980 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- 172.21.15.111:0/2839077091 >> 172.21.15.111:0/2839077091 conn(0x7f25040ccc40 msgr2=0x7f25040cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- 172.21.15.111:0/2839077091 shutdown_connections 2023-12-19T09:21:04.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.788+0000 7f25092cd700 1 -- 172.21.15.111:0/2839077091 wait complete. 2023-12-19T09:21:04.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 Processor -- start 2023-12-19T09:21:04.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 -- start start 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504069100 0x7f250414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040afbd0 con 0x7f2504158f70 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040b0190 con 0x7f2504069100 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25092cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25040b5550 con 0x7f2504159340 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39238/0 (socket says 172.21.15.111:39238) 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f2502ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504069100 0x7f250414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 -- 172.21.15.111:0/1653420950 learned_addr learned my addr 172.21.15.111:0/1653420950 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25037fe700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 -- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 msgr2=0x7f25040b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 -- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504069100 msgr2=0x7f250414cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504069100 0x7f250414cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.792+0000 7f25027fc700 1 -- 172.21.15.111:0/1653420950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f40094f0 con 0x7f2504158f70 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f25027fc700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c590 tx=0x7f24f800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8011590 con 0x7f2504158f70 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8011710 con 0x7f2504158f70 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040b5830 con 0x7f2504158f70 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8011ad0 con 0x7f2504158f70 2023-12-19T09:21:04.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250416aa80 con 0x7f2504158f70 2023-12-19T09:21:04.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25040b0300 con 0x7f2504158f70 2023-12-19T09:21:04.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.796+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f24f801b3b0 con 0x7f2504158f70 2023-12-19T09:21:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.800+0000 7f24eaffd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 0x7f24f0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.800+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f24f807c460 con 0x7f2504158f70 2023-12-19T09:21:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.800+0000 7f2502ffd700 1 -- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 msgr2=0x7f24f0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.800+0000 7f2502ffd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 0x7f24f0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:04.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.804+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f24f804c300 con 0x7f2504158f70 2023-12-19T09:21:04.824 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:04.817+0000 7f1e2bb61000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:04.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.980+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25040b5a20 con 0x7f2504158f70 2023-12-19T09:21:04.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.984+0000 7f24eaffd700 1 -- 172.21.15.111:0/1653420950 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16981 (secure 0 0 0) 0x7f24f804f920 con 0x7f2504158f70 2023-12-19T09:21:04.990 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:04.990 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":195,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:04.762230+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3701795071":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/481543095":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3982635695":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6833/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6832/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3011711537":"2023-12-20T09:19:54.022209+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 msgr2=0x7f24f0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 0x7f24f0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 msgr2=0x7f25040b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c590 tx=0x7f24f800c8a0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 shutdown_connections 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504159340 0x7f25040b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2504158f70 0x7f25040b20f0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f24f0060790 0x7f24f0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 --2- 172.21.15.111:0/1653420950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504069100 0x7f250414cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 >> 172.21.15.111:0/1653420950 conn(0x7f25040ccc40 msgr2=0x7f2504157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 shutdown_connections 2023-12-19T09:21:04.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:04.988+0000 7f25092cd700 1 -- 172.21.15.111:0/1653420950 wait complete. 2023-12-19T09:21:05.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T09:19:37.498642+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T09:19:38.454005+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 195, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T09:21:05.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:21:05.056 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.052+0000 7fa11af61000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:05.146 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.140+0000 7fa11af61000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:05.164 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:05.157+0000 7f1e2bb61000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:05.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.184+0000 7f4691bcc700 1 Processor -- start 2023-12-19T09:21:05.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.184+0000 7f4691bcc700 1 -- start start 2023-12-19T09:21:05.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06b050 0x7f468c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f468c06c7c0 0x7f468c15d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c15da20 con 0x7f468c1594b0 2023-12-19T09:21:05.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468c06da40 con 0x7f468c06c7c0 2023-12-19T09:21:05.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c06cc10 con 0x7f468c06b050 2023-12-19T09:21:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f46913cb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f468c1594b0 0x7f468c1598f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56110/0 (socket says 172.21.15.111:56110) 2023-12-19T09:21:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f46913cb700 1 -- 172.21.15.111:0/1420610688 learned_addr learned my addr 172.21.15.111:0/1420610688 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 --2- 172.21.15.111:0/1420610688 >> v2:172.21.15.143:3300/0 conn(0x7f468c06c7c0 0x7f468c15d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4690bca700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06b050 0x7f468c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468b7fe700 1 -- 172.21.15.111:0/1420610688 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329357852 0 0) 0x7f468c15da20 con 0x7f468c1594b0 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468b7fe700 1 -- 172.21.15.111:0/1420610688 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467c003570 con 0x7f468c1594b0 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 -- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06b050 msgr2=0x7f468c06b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06b050 0x7f468c06b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 -- 172.21.15.111:0/1420610688 >> v1:172.21.15.111:6789/0 conn(0x7f468c1594b0 legacy=0x7f468c1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 -- 172.21.15.111:0/1420610688 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c06cec0 con 0x7f468c06c7c0 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468bfff700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c06c7c0 0x7f468c15d4e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f46780098e0 tx=0x7f46780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50eb1037ad43c5a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:05.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468b7fe700 1 -- 172.21.15.111:0/1420610688 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678011630 con 0x7f468c06c7c0 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468b7fe700 1 -- 172.21.15.111:0/1420610688 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4678011790 con 0x7f468c06c7c0 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f468b7fe700 1 -- 172.21.15.111:0/1420610688 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678011b30 con 0x7f468c06c7c0 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c06c7c0 msgr2=0x7f468c15d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c06c7c0 0x7f468c15d4e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f46780098e0 tx=0x7f46780091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- 172.21.15.111:0/1420610688 shutdown_connections 2023-12-19T09:21:05.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c06c7c0 0x7f468c15d4e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 --2- 172.21.15.111:0/1420610688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06b050 0x7f468c06b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- 172.21.15.111:0/1420610688 >> 172.21.15.111:0/1420610688 conn(0x7f468c0c8cd0 msgr2=0x7f468c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- 172.21.15.111:0/1420610688 shutdown_connections 2023-12-19T09:21:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.188+0000 7f4691bcc700 1 -- 172.21.15.111:0/1420610688 wait complete. 2023-12-19T09:21:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 Processor -- start 2023-12-19T09:21:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 -- start start 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 0x7f468c170ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 0x7f468c0b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 0x7f468c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c15e210 con 0x7f468c06b050 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c16ed30 con 0x7f468c1594b0 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4691bcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c171660 con 0x7f468c06c7c0 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4690bca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 0x7f468c170ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f46913cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 0x7f468c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4690bca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 0x7f468c170ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39240/0 (socket says 172.21.15.111:39240) 2023-12-19T09:21:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f4690bca700 1 -- 172.21.15.111:0/2236546913 learned_addr learned my addr 172.21.15.111:0/2236546913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.192+0000 7f468bfff700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 0x7f468c0b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 -- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 msgr2=0x7f468c0b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 0x7f468c0b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 -- 172.21.15.111:0/2236546913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 msgr2=0x7f468c170ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 0x7f468c170ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 -- 172.21.15.111:0/2236546913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009420 con 0x7f468c1594b0 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f468bfff700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 0x7f468c0b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f46913cb700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 0x7f468c0b1bb0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f468400cb90 tx=0x7f4684001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400a7e0 con 0x7f468c1594b0 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c171940 con 0x7f468c1594b0 2023-12-19T09:21:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468400a940 con 0x7f468c1594b0 2023-12-19T09:21:05.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400acc0 con 0x7f468c1594b0 2023-12-19T09:21:05.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c15e770 con 0x7f468c1594b0 2023-12-19T09:21:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.196+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c15e960 con 0x7f468c1594b0 2023-12-19T09:21:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.200+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f4684007300 con 0x7f468c1594b0 2023-12-19T09:21:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.200+0000 7f4688ff9700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 0x7f4670062cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.200+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f468407cc30 con 0x7f468c1594b0 2023-12-19T09:21:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.200+0000 7f4690bca700 1 -- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 msgr2=0x7f4670062cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.200+0000 7f4690bca700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 0x7f4670062cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.208+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4684049be0 con 0x7f468c1594b0 2023-12-19T09:21:05.262 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:05.257+0000 7f1e2bb61000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:05.376 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.372+0000 7fa11af61000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:05.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.376+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f468c04be40 con 0x7f468c1594b0 2023-12-19T09:21:05.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.380+0000 7f4688ff9700 1 -- 172.21.15.111:0/2236546913 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4684017070 con 0x7f468c1594b0 2023-12-19T09:21:05.384 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 msgr2=0x7f4670062cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 0x7f4670062cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 msgr2=0x7f468c0b1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 0x7f468c0b1bb0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f468400cb90 tx=0x7f4684001980 comp rx=0 tx=0).stop 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 shutdown_connections 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f4670060810 0x7f4670062cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f468c1594b0 0x7f468c0b1bb0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f468c06c7c0 0x7f468c0b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 --2- 172.21.15.111:0/2236546913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f468c06b050 0x7f468c170ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 >> 172.21.15.111:0/2236546913 conn(0x7f468c0c8cd0 msgr2=0x7f468c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:05.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 shutdown_connections 2023-12-19T09:21:05.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.384+0000 7f4691bcc700 1 -- 172.21.15.111:0/2236546913 wait complete. 2023-12-19T09:21:05.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T09:21:05.461 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.456+0000 7fa11af61000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:05.491 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:05.485+0000 7f1e2bb61000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:05.550 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.544+0000 7fa11af61000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:05.578 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:05.573+0000 7f1e2bb61000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:05.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.608+0000 7f7978dcc700 1 Processor -- start 2023-12-19T09:21:05.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.608+0000 7f7978dcc700 1 -- start start 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.608+0000 7f7978dcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7974158c90 0x7f79740caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.608+0000 7f7978dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7974159770 0x7f797406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797406c850 con 0x7f797406aeb0 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797406b470 con 0x7f7974158c90 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797406b750 con 0x7f7974159770 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7974158c90 0x7f79740caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f796affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f797406aeb0 0x7f7974159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56116/0 (socket says 172.21.15.111:56116) 2023-12-19T09:21:05.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f796affd700 1 -- 172.21.15.111:0/3772755249 learned_addr learned my addr 172.21.15.111:0/3772755249 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7973fff700 1 --2- 172.21.15.111:0/3772755249 >> v2:172.21.15.143:3300/0 conn(0x7f7974159770 0x7f797406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7972ffd700 1 -- 172.21.15.111:0/3772755249 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939438261 0 0) 0x7f797406c850 con 0x7f797406aeb0 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7972ffd700 1 -- 172.21.15.111:0/3772755249 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7960003570 con 0x7f797406aeb0 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 -- 172.21.15.111:0/3772755249 >> v2:172.21.15.143:3300/0 conn(0x7f7974159770 msgr2=0x7f797406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 --2- 172.21.15.111:0/3772755249 >> v2:172.21.15.143:3300/0 conn(0x7f7974159770 0x7f797406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 -- 172.21.15.111:0/3772755249 >> v1:172.21.15.111:6789/0 conn(0x7f797406aeb0 legacy=0x7f7974159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 -- 172.21.15.111:0/3772755249 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797406ba30 con 0x7f7974158c90 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f79737fe700 1 --2- 172.21.15.111:0/3772755249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974158c90 0x7f79740caf40 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f79740cc7f0 tx=0x7f79640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99384e229a44a62e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7972ffd700 1 -- 172.21.15.111:0/3772755249 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964011630 con 0x7f7974158c90 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7972ffd700 1 -- 172.21.15.111:0/3772755249 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964011790 con 0x7f7974158c90 2023-12-19T09:21:05.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7972ffd700 1 -- 172.21.15.111:0/3772755249 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964011b30 con 0x7f7974158c90 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- 172.21.15.111:0/3772755249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974158c90 msgr2=0x7f79740caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 --2- 172.21.15.111:0/3772755249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974158c90 0x7f79740caf40 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f79740cc7f0 tx=0x7f79640091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- 172.21.15.111:0/3772755249 shutdown_connections 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 --2- 172.21.15.111:0/3772755249 >> v2:172.21.15.143:3300/0 conn(0x7f7974159770 0x7f797406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 --2- 172.21.15.111:0/3772755249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974158c90 0x7f79740caf40 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- 172.21.15.111:0/3772755249 >> 172.21.15.111:0/3772755249 conn(0x7f7974067bb0 msgr2=0x7f79740cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- 172.21.15.111:0/3772755249 shutdown_connections 2023-12-19T09:21:05.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.612+0000 7f7978dcc700 1 -- 172.21.15.111:0/3772755249 wait complete. 2023-12-19T09:21:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 Processor -- start 2023-12-19T09:21:05.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 -- start start 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f797406aeb0 0x7f7974153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 0x7f79740b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974157980 con 0x7f7974159770 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797408ad90 con 0x7f7974158c90 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7978dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7974087760 con 0x7f797406aeb0 2023-12-19T09:21:05.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7973fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f79737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f797406aeb0 0x7f7974153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f796affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 0x7f79740b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7973fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39258/0 (socket says 172.21.15.111:39258) 2023-12-19T09:21:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.616+0000 7f7973fff700 1 -- 172.21.15.111:0/2613126499 learned_addr learned my addr 172.21.15.111:0/2613126499 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 -- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f797406aeb0 msgr2=0x7f7974153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f797406aeb0 0x7f7974153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 -- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 msgr2=0x7f79740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 0x7f79740b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 -- 172.21.15.111:0/2613126499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964009420 con 0x7f7974159770 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f796affd700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 0x7f79740b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7973fff700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f796c007f50 tx=0x7f796c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c00eb70 con 0x7f7974159770 2023-12-19T09:21:05.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c00ecf0 con 0x7f7974159770 2023-12-19T09:21:05.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c014670 con 0x7f7974159770 2023-12-19T09:21:05.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7974153990 con 0x7f7974159770 2023-12-19T09:21:05.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7974087110 con 0x7f7974159770 2023-12-19T09:21:05.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974086f70 con 0x7f7974159770 2023-12-19T09:21:05.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.620+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f796c0149e0 con 0x7f7974159770 2023-12-19T09:21:05.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.624+0000 7f796bfff700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 0x7f7954062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:05.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.624+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f796c07d600 con 0x7f7974159770 2023-12-19T09:21:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.624+0000 7f79737fe700 1 -- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 msgr2=0x7f7954062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:05.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.624+0000 7f79737fe700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 0x7f7954062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:05.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.628+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7974086f70 con 0x7f7974159770 2023-12-19T09:21:05.665 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:05.657+0000 7f1e2bb61000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7974086f70 con 0x7f7974159770 2023-12-19T09:21:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f796bfff700 1 -- 172.21.15.111:0/2613126499 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v195) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7974086f70 con 0x7f7974159770 2023-12-19T09:21:05.809 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T09:21:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 msgr2=0x7f7954062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 0x7f7954062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 msgr2=0x7f79740b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f796c007f50 tx=0x7f796c009860 comp rx=0 tx=0).stop 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 shutdown_connections 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7954060790 0x7f7954062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7974158c90 0x7f79740b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7974159770 0x7f79740b2120 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 --2- 172.21.15.111:0/2613126499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f797406aeb0 0x7f7974153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.804+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 >> 172.21.15.111:0/2613126499 conn(0x7f7974067bb0 msgr2=0x7f79740cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.808+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 shutdown_connections 2023-12-19T09:21:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:05.808+0000 7f7978dcc700 1 -- 172.21.15.111:0/2613126499 wait complete. 2023-12-19T09:21:05.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T09:21:05.965 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:05.960+0000 7fa11af61000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:06.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 Processor -- start 2023-12-19T09:21:06.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 -- start start 2023-12-19T09:21:06.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb9e0158e20 0x7fb9e00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 0x7fb9e00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00cbdf0 con 0x7fb9e00ca2c0 2023-12-19T09:21:06.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e00cc0d0 con 0x7fb9e0158e20 2023-12-19T09:21:06.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e7cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00cc3b0 con 0x7fb9e00cb970 2023-12-19T09:21:06.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e628d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb9e00ca2c0 0x7fb9e006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56120/0 (socket says 172.21.15.111:56120) 2023-12-19T09:21:06.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e628d700 1 -- 172.21.15.111:0/931008294 learned_addr learned my addr 172.21.15.111:0/931008294 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:06.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e528b700 1 --2- 172.21.15.111:0/931008294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 0x7fb9e00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e5a8c700 1 --2- 172.21.15.111:0/931008294 >> v2:172.21.15.143:3300/0 conn(0x7fb9e0158e20 0x7fb9e00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930795267 0 0) 0x7fb9e00cbdf0 con 0x7fb9e00ca2c0 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.040+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc003570 con 0x7fb9e00ca2c0 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922910977 0 0) 0x7fb9cc003570 con 0x7fb9e00ca2c0 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e528b700 1 -- 172.21.15.111:0/931008294 >> v2:172.21.15.143:3300/0 conn(0x7fb9e0158e20 msgr2=0x7fb9e00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:06.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e528b700 1 --2- 172.21.15.111:0/931008294 >> v2:172.21.15.143:3300/0 conn(0x7fb9e0158e20 0x7fb9e00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e528b700 1 -- 172.21.15.111:0/931008294 >> v1:172.21.15.111:6789/0 conn(0x7fb9e00ca2c0 legacy=0x7fb9e006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e528b700 1 -- 172.21.15.111:0/931008294 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00cc690 con 0x7fb9e00cb970 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e528b700 1 --2- 172.21.15.111:0/931008294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 0x7fb9e00ceb00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00098e0 tx=0x7fb9d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7395cfaf1299a9e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011630 con 0x7fb9e00cb970 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0011790 con 0x7fb9e00cb970 2023-12-19T09:21:06.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e4a8a700 1 -- 172.21.15.111:0/931008294 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011b30 con 0x7fb9e00cb970 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/931008294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 msgr2=0x7fb9e00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/931008294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 0x7fb9e00ceb00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00098e0 tx=0x7fb9d00091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/931008294 shutdown_connections 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/931008294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e00cb970 0x7fb9e00ceb00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/931008294 >> v2:172.21.15.143:3300/0 conn(0x7fb9e0158e20 0x7fb9e00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/931008294 >> 172.21.15.111:0/931008294 conn(0x7fb9e0067bb0 msgr2=0x7fb9e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:06.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/931008294 shutdown_connections 2023-12-19T09:21:06.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.044+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/931008294 wait complete. 2023-12-19T09:21:06.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 Processor -- start 2023-12-19T09:21:06.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 -- start start 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e00ca2c0 0x7fb9e0153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e014f080 0x7fb9e014d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e014ae30 con 0x7fb9e0158e20 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0152bf0 con 0x7fb9e00ca2c0 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e7cf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e014e020 con 0x7fb9e014f080 2023-12-19T09:21:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39268/0 (socket says 172.21.15.111:39268) 2023-12-19T09:21:06.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 -- 172.21.15.111:0/1475970365 learned_addr learned my addr 172.21.15.111:0/1475970365 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e5a8c700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e00ca2c0 0x7fb9e0153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e628d700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e014f080 0x7fb9e014d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 -- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e014f080 msgr2=0x7fb9e014d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e014f080 0x7fb9e014d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 -- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e00ca2c0 msgr2=0x7fb9e0153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e00ca2c0 0x7fb9e0153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 -- 172.21.15.111:0/1475970365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d0009420 con 0x7fb9e0158e20 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.048+0000 7fb9e528b700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00cc4f0 tx=0x7fb9d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d001f070 con 0x7fb9e0158e20 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0002f70 con 0x7fb9e0158e20 2023-12-19T09:21:06.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0023410 con 0x7fb9e0158e20 2023-12-19T09:21:06.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e014e300 con 0x7fb9e0158e20 2023-12-19T09:21:06.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e0152880 con 0x7fb9e0158e20 2023-12-19T09:21:06.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e006fbc0 con 0x7fb9e0158e20 2023-12-19T09:21:06.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.052+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fb9d0023800 con 0x7fb9e0158e20 2023-12-19T09:21:06.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.056+0000 7fb9d5ffb700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 0x7fb9c4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.056+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fb9d007e0b0 con 0x7fb9e0158e20 2023-12-19T09:21:06.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.056+0000 7fb9e5a8c700 1 -- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 msgr2=0x7fb9c4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:06.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.056+0000 7fb9e5a8c700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 0x7fb9c4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.064+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9e006fbc0 con 0x7fb9e0158e20 2023-12-19T09:21:06.082 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:06.077+0000 7f1e2bb61000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:06.140 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:06.136+0000 7fa11af61000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:06.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.232+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb9e006fbc0 con 0x7fb9e0158e20 2023-12-19T09:21:06.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9d5ffb700 1 -- 172.21.15.111:0/1475970365 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v195) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb9e006fbc0 con 0x7fb9e0158e20 2023-12-19T09:21:06.240 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 msgr2=0x7fb9c4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 0x7fb9c4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 msgr2=0x7fb9e014eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00cc4f0 tx=0x7fb9d0017000 comp rx=0 tx=0).stop 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 shutdown_connections 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9e014f080 0x7fb9e014d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e0158e20 0x7fb9e014eb40 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb9c4060790 0x7fb9c4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 --2- 172.21.15.111:0/1475970365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e00ca2c0 0x7fb9e0153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 >> 172.21.15.111:0/1475970365 conn(0x7fb9e0067bb0 msgr2=0x7fb9e014b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 shutdown_connections 2023-12-19T09:21:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.236+0000 7fb9e7cf0700 1 -- 172.21.15.111:0/1475970365 wait complete. 2023-12-19T09:21:06.257 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:06.253+0000 7f1e2bb61000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:06.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.416+0000 7f1fd8a03700 1 Processor -- start 2023-12-19T09:21:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.416+0000 7f1fd8a03700 1 -- start start 2023-12-19T09:21:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd8a03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fd4155960 0x7f1fd415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd8a03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406de20 0x7f1fd40c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd8a03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd406d990 con 0x7f1fd406c7c0 2023-12-19T09:21:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd8a03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd415a570 con 0x7f1fd4155960 2023-12-19T09:21:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd8a03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd406cd80 con 0x7f1fd406de20 2023-12-19T09:21:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fd4155960 0x7f1fd415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fd4155960 0x7f1fd415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45052/0 (socket says 172.21.15.111:45052) 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 -- 172.21.15.111:0/1531256394 learned_addr learned my addr 172.21.15.111:0/1531256394 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd3fff700 1 --1- 172.21.15.111:0/1531256394 >> v1:172.21.15.143:6789/0 conn(0x7f1fd406c7c0 0x7f1fd40c5fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:33672/0 (socket says 172.21.15.111:33672) 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fcaffd700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406de20 0x7f1fd40c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 -- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406de20 msgr2=0x7f1fd40c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406de20 0x7f1fd40c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 -- 172.21.15.111:0/1531256394 >> v1:172.21.15.143:6789/0 conn(0x7f1fd406c7c0 legacy=0x7f1fd40c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 -- 172.21.15.111:0/1531256394 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd406d060 con 0x7f1fd4155960 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd37fe700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd4155960 0x7f1fd415a030 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40099b0 tx=0x7f1fc4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d304e6fe81a26a02 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.420+0000 7f1fd2ffd700 1 -- 172.21.15.111:0/1531256394 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400b860 con 0x7f1fd4155960 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd2ffd700 1 -- 172.21.15.111:0/1531256394 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc400b9c0 con 0x7f1fd4155960 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd2ffd700 1 -- 172.21.15.111:0/1531256394 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400bd60 con 0x7f1fd4155960 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd4155960 msgr2=0x7f1fd415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd4155960 0x7f1fd415a030 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40099b0 tx=0x7f1fc4009250 comp rx=0 tx=0).stop 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- 172.21.15.111:0/1531256394 shutdown_connections 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406de20 0x7f1fd40c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/1531256394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd4155960 0x7f1fd415a030 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- 172.21.15.111:0/1531256394 >> 172.21.15.111:0/1531256394 conn(0x7f1fd4067bb0 msgr2=0x7f1fd40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- 172.21.15.111:0/1531256394 shutdown_connections 2023-12-19T09:21:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- 172.21.15.111:0/1531256394 wait complete. 2023-12-19T09:21:06.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 Processor -- start 2023-12-19T09:21:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.424+0000 7f1fd8a03700 1 -- start start 2023-12-19T09:21:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd406de20 0x7f1fd416d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 0x7f1fd40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd414a210 con 0x7f1fd4155960 2023-12-19T09:21:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd415b7c0 con 0x7f1fd406de20 2023-12-19T09:21:06.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd415baa0 con 0x7f1fd406c7c0 2023-12-19T09:21:06.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 0x7f1fd40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:34880/0 (socket says 172.21.15.111:34880) 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 -- 172.21.15.111:0/2996859650 learned_addr learned my addr 172.21.15.111:0/2996859650 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fcaffd700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd406de20 0x7f1fd416d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd406de20 msgr2=0x7f1fd416d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd406de20 0x7f1fd416d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 msgr2=0x7f1fd40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 0x7f1fd40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 -- 172.21.15.111:0/2996859650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc40094f0 con 0x7f1fd406c7c0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd3fff700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 0x7f1fd40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd37fe700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009290 tx=0x7f1fc4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4019db0 con 0x7f1fd406c7c0 2023-12-19T09:21:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd415bd80 con 0x7f1fd406c7c0 2023-12-19T09:21:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4010470 con 0x7f1fd406c7c0 2023-12-19T09:21:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4010830 con 0x7f1fd406c7c0 2023-12-19T09:21:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.428+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd415c340 con 0x7f1fd406c7c0 2023-12-19T09:21:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd404be40 con 0x7f1fd406c7c0 2023-12-19T09:21:06.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f1fc402c430 con 0x7f1fd406c7c0 2023-12-19T09:21:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fcbfff700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 0x7f1fb4062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f1fc4014070 con 0x7f1fd406c7c0 2023-12-19T09:21:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fcaffd700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 msgr2=0x7f1fb4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:06.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.432+0000 7f1fcaffd700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 0x7f1fb4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.440+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1fc404e880 con 0x7f1fd406c7c0 2023-12-19T09:21:06.520 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:06.516+0000 7fa11af61000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:06.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.636+0000 7f1fcaffd700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 msgr2=0x7f1fb4062c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:06.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.636+0000 7f1fcaffd700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 0x7f1fb4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:06.641 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:06.637+0000 7f1e2bb61000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:06.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.652+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd40c91f0 con 0x7f1fd406c7c0 2023-12-19T09:21:06.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.656+0000 7f1fcbfff700 1 -- 172.21.15.111:0/2996859650 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7f1fc4051ea0 con 0x7f1fd406c7c0 2023-12-19T09:21:06.661 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:06.661 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 msgr2=0x7f1fb4062c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 0x7f1fb4062c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 msgr2=0x7f1fd415da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009290 tx=0x7f1fc4019000 comp rx=0 tx=0).stop 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 shutdown_connections 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1fb40607c0 0x7f1fb4062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fd406de20 0x7f1fd416d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fd4155960 0x7f1fd40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 --2- 172.21.15.111:0/2996859650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fd406c7c0 0x7f1fd415da80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 >> 172.21.15.111:0/2996859650 conn(0x7f1fd4067bb0 msgr2=0x7f1fd4151600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:06.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 shutdown_connections 2023-12-19T09:21:06.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.660+0000 7f1fd8a03700 1 -- 172.21.15.111:0/2996859650 wait complete. 2023-12-19T09:21:06.666 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:21:06.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:06.757 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:06.752+0000 7fa11af61000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:06.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.828+0000 7f6bd6ce3700 1 Processor -- start 2023-12-19T09:21:06.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.828+0000 7f6bd6ce3700 1 -- start start 2023-12-19T09:21:06.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd6ce3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bd006d760 0x7f6bd015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd6ce3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd00c7ca0 con 0x7f6bd015dab0 2023-12-19T09:21:06.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd6ce3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd015e380 con 0x7f6bd015d1a0 2023-12-19T09:21:06.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd6ce3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd015e660 con 0x7f6bd006d760 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bd006d760 0x7f6bd015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd64e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6bd015dab0 0x7f6bd00c8130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56122/0 (socket says 172.21.15.111:56122) 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd64e2700 1 -- 172.21.15.111:0/578783313 learned_addr learned my addr 172.21.15.111:0/578783313 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6bd006d760 0x7f6bd015d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39300/0 (socket says 172.21.15.111:39300) 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 -- 172.21.15.111:0/578783313 >> v1:172.21.15.143:6789/0 conn(0x7f6bd015d1a0 legacy=0x7f6bd006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 -- 172.21.15.111:0/578783313 >> v1:172.21.15.111:6789/0 conn(0x7f6bd015dab0 legacy=0x7f6bd00c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 -- 172.21.15.111:0/578783313 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00c7930 con 0x7f6bd006d760 2023-12-19T09:21:06.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.832+0000 7f6bd54e0700 1 --2- 172.21.15.111:0/578783313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd006d760 0x7f6bd015d570 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00098e0 tx=0x7f6bc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d707429e5f33ca5f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd4cdf700 1 -- 172.21.15.111:0/578783313 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0009e80 con 0x7f6bd006d760 2023-12-19T09:21:06.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd4cdf700 1 -- 172.21.15.111:0/578783313 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc000b760 con 0x7f6bd006d760 2023-12-19T09:21:06.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd4cdf700 1 -- 172.21.15.111:0/578783313 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc000bae0 con 0x7f6bd006d760 2023-12-19T09:21:06.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/578783313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd006d760 msgr2=0x7f6bd015d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/578783313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd006d760 0x7f6bd015d570 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00098e0 tx=0x7f6bc00091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/578783313 shutdown_connections 2023-12-19T09:21:06.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/578783313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd006d760 0x7f6bd015d570 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/578783313 >> 172.21.15.111:0/578783313 conn(0x7f6bd006c080 msgr2=0x7f6bd00ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:06.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/578783313 shutdown_connections 2023-12-19T09:21:06.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/578783313 wait complete. 2023-12-19T09:21:06.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 Processor -- start 2023-12-19T09:21:06.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.836+0000 7f6bd6ce3700 1 -- start start 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd006d760 0x7f6bd015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd015dab0 0x7f6bd00cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0161cc0 con 0x7f6bd015d1a0 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00b1470 con 0x7f6bd015dab0 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0161a60 con 0x7f6bd006d760 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39316/0 (socket says 172.21.15.111:39316) 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 -- 172.21.15.111:0/1684462949 learned_addr learned my addr 172.21.15.111:0/1684462949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd64e2700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd015dab0 0x7f6bd00cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd5ce1700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd006d760 0x7f6bd015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:06.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 -- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd006d760 msgr2=0x7f6bd015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd006d760 0x7f6bd015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 -- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd015dab0 msgr2=0x7f6bd00cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:06.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd015dab0 0x7f6bd00cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:06.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 -- 172.21.15.111:0/1684462949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0009420 con 0x7f6bd015d1a0 2023-12-19T09:21:06.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd54e0700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00ca600 tx=0x7f6bc0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc000bae0 con 0x7f6bd015d1a0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc000be90 con 0x7f6bd015d1a0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd00b0b40 con 0x7f6bd015d1a0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc001a630 con 0x7f6bd015d1a0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd00b1100 con 0x7f6bd015d1a0 2023-12-19T09:21:06.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.840+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd004be40 con 0x7f6bd015d1a0 2023-12-19T09:21:06.848 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:06.844+0000 7fa11af61000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:06.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.844+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f6bc001a9a0 con 0x7f6bd015d1a0 2023-12-19T09:21:06.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.844+0000 7f6bc5ffb700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 0x7f6bb4062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:06.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.844+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f6bc007eaa0 con 0x7f6bd015d1a0 2023-12-19T09:21:06.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.844+0000 7f6bd5ce1700 1 -- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 msgr2=0x7f6bb4062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:06.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.844+0000 7f6bd5ce1700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 0x7f6bb4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:06.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:06.848+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6bc004e940 con 0x7f6bd015d1a0 2023-12-19T09:21:06.876 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:06.869+0000 7f1e2bb61000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:06.926 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:06.920+0000 7fa11af61000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:06.968 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:06.961+0000 7f1e2bb61000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:07.004 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.000+0000 7fa11af61000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:07.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.024+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bd015d600 con 0x7f6bd015d1a0 2023-12-19T09:21:07.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.024+0000 7f6bc5ffb700 1 -- 172.21.15.111:0/1684462949 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16981 (secure 0 0 0) 0x7f6bc0051f60 con 0x7f6bd015d1a0 2023-12-19T09:21:07.030 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:07.030 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":195,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:04.762230+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3701795071":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/481543095":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3982635695":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6833/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6832/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3011711537":"2023-12-20T09:19:54.022209+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 msgr2=0x7f6bb4062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 0x7f6bb4062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 msgr2=0x7f6bd0161520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00ca600 tx=0x7f6bc0016000 comp rx=0 tx=0).stop 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 shutdown_connections 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6bb4060790 0x7f6bb4062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd015dab0 0x7f6bd00cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6bd015d1a0 0x7f6bd0161520 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 --2- 172.21.15.111:0/1684462949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd006d760 0x7f6bd015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 >> 172.21.15.111:0/1684462949 conn(0x7f6bd006c080 msgr2=0x7f6bd0068460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 shutdown_connections 2023-12-19T09:21:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.028+0000 7f6bd6ce3700 1 -- 172.21.15.111:0/1684462949 wait complete. 2023-12-19T09:21:07.048 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.041+0000 7f1e2bb61000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:07.051 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:21:07.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:07.133 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.125+0000 7f1e2bb61000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 Processor -- start 2023-12-19T09:21:07.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 -- start start 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 0x7fb1bc0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 0x7fb1bc06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb1bc06b0f0 0x7fb1bc06b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1bc06bab0 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1bc06e470 con 0x7fb1bc158f00 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c34cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc06e900 con 0x7fb1bc1592d0 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c1a6a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb1bc06b0f0 0x7fb1bc06b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c1269700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 0x7fb1bc0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c0a68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 0x7fb1bc06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c0a68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 0x7fb1bc06cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:34912/0 (socket says 172.21.15.111:34912) 2023-12-19T09:21:07.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.208+0000 7fb1c0a68700 1 -- 172.21.15.111:0/2545975850 learned_addr learned my addr 172.21.15.111:0/2545975850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 -- 172.21.15.111:0/2545975850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 msgr2=0x7fb1bc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 --2- 172.21.15.111:0/2545975850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 0x7fb1bc06cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 -- 172.21.15.111:0/2545975850 >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 msgr2=0x7fb1bc0c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 --2- 172.21.15.111:0/2545975850 >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 0x7fb1bc0c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 -- 172.21.15.111:0/2545975850 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1bc06d860 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1269700 1 --2- 172.21.15.111:0/2545975850 >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 0x7fb1bc0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c1a6a700 1 --2- 172.21.15.111:0/2545975850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc06b0f0 0x7fb1bc06b570 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc064270 tx=0x7fb1b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23459dccd1dfea17 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1b3fff700 1 -- 172.21.15.111:0/2545975850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b400e070 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1b3fff700 1 -- 172.21.15.111:0/2545975850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b400bab0 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1b3fff700 1 -- 172.21.15.111:0/2545975850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b400be70 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2545975850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc06b0f0 msgr2=0x7fb1bc06b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2545975850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc06b0f0 0x7fb1bc06b570 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc064270 tx=0x7fb1b4009780 comp rx=0 tx=0).stop 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2545975850 shutdown_connections 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2545975850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc06b0f0 0x7fb1bc06b570 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2545975850 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc1592d0 0x7fb1bc06cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2545975850 >> v2:172.21.15.143:3300/0 conn(0x7fb1bc158f00 0x7fb1bc0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2545975850 >> 172.21.15.111:0/2545975850 conn(0x7fb1bc0c90f0 msgr2=0x7fb1bc0c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2545975850 shutdown_connections 2023-12-19T09:21:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.212+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2545975850 wait complete. 2023-12-19T09:21:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 Processor -- start 2023-12-19T09:21:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 -- start start 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc06b0f0 0x7fb1bc15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 0x7fb1bc155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc0b0760 con 0x7fb1bc1592d0 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc156430 con 0x7fb1bc158f00 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c34cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1bc156910 con 0x7fb1bc06b0f0 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.216+0000 7fb1c1a6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c1a6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39346/0 (socket says 172.21.15.111:39346) 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 0x7fb1bc155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c1a6a700 1 -- 172.21.15.111:0/2423145479 learned_addr learned my addr 172.21.15.111:0/2423145479 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c1269700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc06b0f0 0x7fb1bc15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc06b0f0 msgr2=0x7fb1bc15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc06b0f0 0x7fb1bc15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 msgr2=0x7fb1bc0ac010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 -- 172.21.15.111:0/2423145479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b8009420 con 0x7fb1bc158f00 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c1a6a700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c0a68700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 0x7fb1bc155ef0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac013680 con 0x7fb1bc158f00 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ac0137e0 con 0x7fb1bc158f00 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b4009a20 con 0x7fb1bc158f00 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac013ba0 con 0x7fb1bc158f00 2023-12-19T09:21:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1bc0afe30 con 0x7fb1bc158f00 2023-12-19T09:21:07.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.220+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc0b0910 con 0x7fb1bc158f00 2023-12-19T09:21:07.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.224+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fb1ac0123d0 con 0x7fb1bc158f00 2023-12-19T09:21:07.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.224+0000 7fb1b17fa700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 0x7fb1a0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.224+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fb1ac07d090 con 0x7fb1bc158f00 2023-12-19T09:21:07.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.224+0000 7fb1c1269700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 msgr2=0x7fb1a0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:07.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.224+0000 7fb1c1269700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 0x7fb1a0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:07.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.232+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb1bc0b0910 con 0x7fb1bc158f00 2023-12-19T09:21:07.241 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.236+0000 7fa11af61000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:07.320 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.312+0000 7fa11af61000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:07.371 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.365+0000 7f1e2bb61000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:07.414 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.408+0000 7fa11af61000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:07.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.428+0000 7fb1c1269700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 msgr2=0x7fb1a0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:07.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.428+0000 7fb1c1269700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 0x7fb1a0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.444+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1bc0b0910 con 0x7fb1bc158f00 2023-12-19T09:21:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.444+0000 7fb1b17fa700 1 -- 172.21.15.111:0/2423145479 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7fb1bc0b0910 con 0x7fb1bc158f00 2023-12-19T09:21:07.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:07.450 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:21:07.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 msgr2=0x7fb1a0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:07.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 0x7fb1a0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 msgr2=0x7fb1bc155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 0x7fb1bc155ef0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac009250 comp rx=0 tx=0).stop 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 shutdown_connections 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb1a0060790 0x7fb1a0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1bc158f00 0x7fb1bc155ef0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1bc1592d0 0x7fb1bc0ac010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 --2- 172.21.15.111:0/2423145479 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1bc06b0f0 0x7fb1bc15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 >> 172.21.15.111:0/2423145479 conn(0x7fb1bc0c90f0 msgr2=0x7fb1bc0ab0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 shutdown_connections 2023-12-19T09:21:07.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.448+0000 7fb1c34cd700 1 -- 172.21.15.111:0/2423145479 wait complete. 2023-12-19T09:21:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:21:07.457 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.449+0000 7f1e2bb61000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:07.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:07.551 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.545+0000 7f1e2bb61000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:07.610 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.603+0000 7fa11af61000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.619+0000 7f8888f85700 1 Processor -- start 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.619+0000 7f8888f85700 1 -- start start 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88840ae140 0x7f88840a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 0x7f88840c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888406dc20 con 0x7f888406d180 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888406d940 con 0x7f88840ae140 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88840c6e20 con 0x7f88840ae510 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88840ae140 0x7f88840a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88840ae140 0x7f88840a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39362/0 (socket says 172.21.15.111:39362) 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 -- 172.21.15.111:0/3291785094 learned_addr learned my addr 172.21.15.111:0/3291785094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8881d9b700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 0x7f88840c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 -- 172.21.15.111:0/3291785094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 msgr2=0x7f88840c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 0x7f88840c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 -- 172.21.15.111:0/3291785094 >> v1:172.21.15.143:6789/0 conn(0x7f888406d180 legacy=0x7f88840680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 -- 172.21.15.111:0/3291785094 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840c7100 con 0x7f88840ae140 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8881d9b700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 0x7f88840c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888259c700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae140 0x7f88840a8850 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f888406de10 tx=0x7f8874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd101f11e69306eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888159a700 1 -- 172.21.15.111:0/3291785094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400b820 con 0x7f88840ae140 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888159a700 1 -- 172.21.15.111:0/3291785094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400b980 con 0x7f88840ae140 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f888159a700 1 -- 172.21.15.111:0/3291785094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400bd20 con 0x7f88840ae140 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- 172.21.15.111:0/3291785094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae140 msgr2=0x7f88840a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae140 0x7f88840a8850 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f888406de10 tx=0x7f8874009250 comp rx=0 tx=0).stop 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- 172.21.15.111:0/3291785094 shutdown_connections 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88840ae510 0x7f88840c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 --2- 172.21.15.111:0/3291785094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae140 0x7f88840a8850 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.623+0000 7f8888f85700 1 -- 172.21.15.111:0/3291785094 >> 172.21.15.111:0/3291785094 conn(0x7f88840c8850 msgr2=0x7f88840c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.627+0000 7f8888f85700 1 -- 172.21.15.111:0/3291785094 shutdown_connections 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.627+0000 7f8888f85700 1 -- 172.21.15.111:0/3291785094 wait complete. 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.627+0000 7f8888f85700 1 Processor -- start 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.627+0000 7f8888f85700 1 -- start start 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 0x7f8884156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 0x7f88840b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 0x7f88840b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888415e800 con 0x7f88840ae510 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8884157530 con 0x7f88840ae140 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888415db80 con 0x7f888406d180 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8881d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 0x7f88840b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f888259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 0x7f8884156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 0x7f88840b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f888259c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 0x7f8884156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:34946/0 (socket says 172.21.15.111:34946) 2023-12-19T09:21:07.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8881d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 0x7f88840b10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:45092/0 (socket says 172.21.15.111:45092) 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f888259c700 1 -- 172.21.15.111:0/508994849 learned_addr learned my addr 172.21.15.111:0/508994849 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 msgr2=0x7f8884156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 0x7f8884156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 msgr2=0x7f88840b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 0x7f88840b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 -- 172.21.15.111:0/508994849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88740094f0 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8882d9d700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 0x7f88840b1630 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f887c00bb00 tx=0x7f887c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00c7c0 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c00c920 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888415de60 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00cce0 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.631+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888415e420 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.635+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88840aa320 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.635+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f887c014620 con 0x7f88840ae510 2023-12-19T09:21:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.639+0000 7f88727fc700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 0x7f8864062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.639+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f887c07cf70 con 0x7f88840ae510 2023-12-19T09:21:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.639+0000 7f888259c700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 msgr2=0x7f8864062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.639+0000 7f888259c700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 0x7f8864062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.643+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f887c04abe0 con 0x7f88840ae510 2023-12-19T09:21:07.700 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:07.695+0000 7fa11af61000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:07.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.757+0000 7f1e2bb61000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.839+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88840aa6f0 con 0x7f88840ae510 2023-12-19T09:21:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.839+0000 7f88727fc700 1 -- 172.21.15.111:0/508994849 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4613 (secure 0 0 0) 0x7f88840aa320 con 0x7f88840ae510 2023-12-19T09:21:07.847 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:07.847 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:19:39.431568+0000","modified":"2023-12-19T09:19:43.040733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9948":{"gid":9948,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/1020011686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1020011686},{"type":"v1","addr":"172.21.15.111:6835","nonce":1020011686}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2023-12-19T09:21:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f888259c700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 msgr2=0x7f8864062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f888259c700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 0x7f8864062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 msgr2=0x7f8864062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 0x7f8864062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 msgr2=0x7f88840b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 0x7f88840b1630 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f887c00bb00 tx=0x7f887c009780 comp rx=0 tx=0).stop 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 shutdown_connections 2023-12-19T09:21:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8864060790 0x7f8864062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88840ae140 0x7f88840b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88840ae510 0x7f88840b1630 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 --2- 172.21.15.111:0/508994849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f888406d180 0x7f8884156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.843+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 >> 172.21.15.111:0/508994849 conn(0x7f88840c8850 msgr2=0x7f8884157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.847+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 shutdown_connections 2023-12-19T09:21:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:07.847+0000 7f8888f85700 1 -- 172.21.15.111:0/508994849 wait complete. 2023-12-19T09:21:07.854 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:07.849+0000 7f1e2bb61000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 87 2023-12-19T09:21:07.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:08.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.031+0000 7fcce3ef0700 1 Processor -- start 2023-12-19T09:21:08.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.031+0000 7fcce3ef0700 1 -- start start 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce3ef0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fccdc068800 0x7fccdc0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce3ef0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc06c9f0 con 0x7fccdc0c2980 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce3ef0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc0c7f20 con 0x7fccdc13f890 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce3ef0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccdc0c1c50 con 0x7fccdc068800 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce1c8c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fccdc13f890 0x7fccdc0c1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:33682/0 (socket says 172.21.15.111:33682) 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce1c8c700 1 -- 172.21.15.111:0/1900549136 learned_addr learned my addr 172.21.15.111:0/1900549136 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce148b700 1 --1- 172.21.15.111:0/1900549136 >> v1:172.21.15.111:6789/0 conn(0x7fccdc0c2980 0x7fccdc0680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56134/0 (socket says 172.21.15.111:56134) 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce248d700 1 --2- 172.21.15.111:0/1900549136 >> v2:172.21.15.143:3300/0 conn(0x7fccdc068800 0x7fccdc0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941455369 0 0) 0x7fccdc0c7f20 con 0x7fccdc13f890 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc8003570 con 0x7fccdc13f890 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396663820 0 0) 0x7fccdc06c9f0 con 0x7fccdc0c2980 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc0c7f20 con 0x7fccdc0c2980 2023-12-19T09:21:08.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce248d700 1 -- 172.21.15.111:0/1900549136 >> v1:172.21.15.143:6789/0 conn(0x7fccdc13f890 legacy=0x7fccdc0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce248d700 1 -- 172.21.15.111:0/1900549136 >> v1:172.21.15.111:6789/0 conn(0x7fccdc0c2980 legacy=0x7fccdc0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce248d700 1 -- 172.21.15.111:0/1900549136 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0c8190 con 0x7fccdc068800 2023-12-19T09:21:08.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce248d700 1 --2- 172.21.15.111:0/1900549136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0aeba0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009e60 tx=0x7fccd4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2755d5a24b225279 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.035+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400e070 con 0x7fccdc068800 2023-12-19T09:21:08.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd400bab0 con 0x7fccdc068800 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce0c8a700 1 -- 172.21.15.111:0/1900549136 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400be70 con 0x7fccdc068800 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 -- 172.21.15.111:0/1900549136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 msgr2=0x7fccdc0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/1900549136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0aeba0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009e60 tx=0x7fccd4009780 comp rx=0 tx=0).stop 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 -- 172.21.15.111:0/1900549136 shutdown_connections 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/1900549136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0aeba0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 -- 172.21.15.111:0/1900549136 >> 172.21.15.111:0/1900549136 conn(0x7fccdc0c8850 msgr2=0x7fccdc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 -- 172.21.15.111:0/1900549136 shutdown_connections 2023-12-19T09:21:08.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 -- 172.21.15.111:0/1900549136 wait complete. 2023-12-19T09:21:08.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.039+0000 7fcce3ef0700 1 Processor -- start 2023-12-19T09:21:08.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- start start 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccdc13f890 0x7fccdc0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0a9b70 con 0x7fccdc0c2980 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0abfb0 con 0x7fccdc068800 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0a95b0 con 0x7fccdc13f890 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce1c8c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39372/0 (socket says 172.21.15.111:39372) 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 -- 172.21.15.111:0/3744269565 learned_addr learned my addr 172.21.15.111:0/3744269565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:08.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce248d700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccdc13f890 0x7fccdc0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 -- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccdc13f890 msgr2=0x7fccdc0c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccdc13f890 0x7fccdc0c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 -- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 msgr2=0x7fccdc0c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 -- 172.21.15.111:0/3744269565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009a20 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce148b700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c7c0 tx=0x7fcccc007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc005b40 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc005ca0 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc0a9890 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc007320 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.043+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc0ac530 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc0ac750 con 0x7fccdc0c2980 2023-12-19T09:21:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7fcccc01b3b0 con 0x7fccdc0c2980 2023-12-19T09:21:08.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fccd1ffb700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 0x7fccc0062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fcccc07c520 con 0x7fccdc0c2980 2023-12-19T09:21:08.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fcce1c8c700 1 -- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 msgr2=0x7fccc0062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:08.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.047+0000 7fcce1c8c700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 0x7fccc0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:08.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.055+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcccc04c320 con 0x7fccdc0c2980 2023-12-19T09:21:08.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.227+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccdc071120 con 0x7fccdc0c2980 2023-12-19T09:21:08.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fccd1ffb700 1 -- 172.21.15.111:0/3744269565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16981 (secure 0 0 0) 0x7fcccc04f940 con 0x7fccdc0c2980 2023-12-19T09:21:08.236 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:08.236 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":195,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:04.762230+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:19:37.498642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-12-19T09:19:38.454005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":195,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/3701795071":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/481543095":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3982635695":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6833/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6832/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3011711537":"2023-12-20T09:19:54.022209+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]},{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 msgr2=0x7fccc0062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 0x7fccc0062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 msgr2=0x7fccdc1306c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00c7c0 tx=0x7fcccc007dd0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 shutdown_connections 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccdc13f890 0x7fccdc0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccdc0c2980 0x7fccdc1306c0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fccc0060790 0x7fccc0062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 --2- 172.21.15.111:0/3744269565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc068800 0x7fccdc0c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 >> 172.21.15.111:0/3744269565 conn(0x7fccdc0c8850 msgr2=0x7fccdc13e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:08.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 shutdown_connections 2023-12-19T09:21:08.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.231+0000 7fcce3ef0700 1 -- 172.21.15.111:0/3744269565 wait complete. 2023-12-19T09:21:08.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T09:21:08.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 Processor -- start 2023-12-19T09:21:08.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 -- start start 2023-12-19T09:21:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f09cc153530 0x7f09cc0c8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc06ccd0 con 0x7f09cc06d870 2023-12-19T09:21:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc153900 con 0x7f09cc0c85b0 2023-12-19T09:21:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09d17c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc0cbf80 con 0x7f09cc153530 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.407+0000 7f09cbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09cc06d870 0x7f09cc0c9e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56148/0 (socket says 172.21.15.111:56148) 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09cbfff700 1 -- 172.21.15.111:0/1457039492 learned_addr learned my addr 172.21.15.111:0/1457039492 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09caffd700 1 --2- 172.21.15.111:0/1457039492 >> v2:172.21.15.143:3300/0 conn(0x7f09cc153530 0x7f09cc0c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09ca7fc700 1 -- 172.21.15.111:0/1457039492 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174789716 0 0) 0x7f09cc06ccd0 con 0x7f09cc06d870 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09ca7fc700 1 -- 172.21.15.111:0/1457039492 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8003570 con 0x7f09cc06d870 2023-12-19T09:21:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09caffd700 1 -- 172.21.15.111:0/1457039492 >> v1:172.21.15.143:6789/0 conn(0x7f09cc0c85b0 legacy=0x7f09cc06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09caffd700 1 -- 172.21.15.111:0/1457039492 >> v1:172.21.15.111:6789/0 conn(0x7f09cc06d870 legacy=0x7f09cc0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09caffd700 1 -- 172.21.15.111:0/1457039492 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0cc260 con 0x7f09cc153530 2023-12-19T09:21:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09caffd700 1 --2- 172.21.15.111:0/1457039492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c8070 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0098e0 tx=0x7f09bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=636028c66eaac969 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09ca7fc700 1 -- 172.21.15.111:0/1457039492 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00b640 con 0x7f09cc153530 2023-12-19T09:21:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09ca7fc700 1 -- 172.21.15.111:0/1457039492 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc00b7a0 con 0x7f09cc153530 2023-12-19T09:21:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09ca7fc700 1 -- 172.21.15.111:0/1457039492 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00bb20 con 0x7f09cc153530 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 -- 172.21.15.111:0/1457039492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 msgr2=0x7f09cc0c8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1457039492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c8070 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0098e0 tx=0x7f09bc0091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 -- 172.21.15.111:0/1457039492 shutdown_connections 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1457039492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c8070 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 -- 172.21.15.111:0/1457039492 >> 172.21.15.111:0/1457039492 conn(0x7f09cc068500 msgr2=0x7f09cc0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 -- 172.21.15.111:0/1457039492 shutdown_connections 2023-12-19T09:21:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.411+0000 7f09d17c5700 1 -- 172.21.15.111:0/1457039492 wait complete. 2023-12-19T09:21:08.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 Processor -- start 2023-12-19T09:21:08.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 -- start start 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09cc0c85b0 0x7f09cc0c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc0b2400 con 0x7f09cc06d870 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc15aad0 con 0x7f09cc153530 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09d17c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc15adb0 con 0x7f09cc0c85b0 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09caffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09caffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39388/0 (socket says 172.21.15.111:39388) 2023-12-19T09:21:08.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09caffd700 1 -- 172.21.15.111:0/1056323581 learned_addr learned my addr 172.21.15.111:0/1056323581 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:08.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09cbfff700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.415+0000 7f09c3fff700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09cc0c85b0 0x7f09cc0c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 -- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09cc0c85b0 msgr2=0x7f09cc0c2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09cc0c85b0 0x7f09cc0c2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 -- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 msgr2=0x7f09cc0c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 -- 172.21.15.111:0/1056323581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc009420 con 0x7f09cc06d870 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09caffd700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0093f0 tx=0x7f09bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc000d10 con 0x7f09cc06d870 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc00b910 con 0x7f09cc06d870 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc025670 con 0x7f09cc06d870 2023-12-19T09:21:08.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc15b090 con 0x7f09cc06d870 2023-12-19T09:21:08.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc15b650 con 0x7f09cc06d870 2023-12-19T09:21:08.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc071120 con 0x7f09cc06d870 2023-12-19T09:21:08.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.419+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 98) v1 ==== 80177+0+0 (secure 0 0 0) 0x7f09bc023070 con 0x7f09cc06d870 2023-12-19T09:21:08.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.423+0000 7f09c2ffd700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 0x7f09ac062c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.423+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f09bc07e5a0 con 0x7f09cc06d870 2023-12-19T09:21:08.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.423+0000 7f09c3fff700 1 -- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 msgr2=0x7f09ac062c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:08.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.423+0000 7f09c3fff700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 0x7f09ac062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:08.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.431+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f09bc04e440 con 0x7f09cc06d870 2023-12-19T09:21:08.443 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:08.439+0000 7fa11af61000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:08.601 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.597+0000 7f1e2bb61000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:08.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.615+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 99) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f09bc04e810 con 0x7f09cc06d870 2023-12-19T09:21:08.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.615+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 msgr2=0x7f09ac062c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:08.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.615+0000 7f09c2ffd700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 0x7f09ac062c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.635+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f09cc0cc540 con 0x7f09cc06d870 2023-12-19T09:21:08.644 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T09:21:08.639+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T09:21:08.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.647+0000 7f09c2ffd700 1 -- 172.21.15.111:0/1056323581 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7f09bc053320 con 0x7f09cc06d870 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 msgr2=0x7f09cc14def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0093f0 tx=0x7f09bc017000 comp rx=0 tx=0).stop 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 shutdown_connections 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f09ac060790 0x7f09ac062c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09cc153530 0x7f09cc0c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09cc0c85b0 0x7f09cc0c2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 --2- 172.21.15.111:0/1056323581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09cc06d870 0x7f09cc14def0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 >> 172.21.15.111:0/1056323581 conn(0x7f09cc068500 msgr2=0x7f09cc14e800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 shutdown_connections 2023-12-19T09:21:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.651+0000 7f09d17c5700 1 -- 172.21.15.111:0/1056323581 wait complete. 2023-12-19T09:21:08.660 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T09:21:08.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T09:21:08.713 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df67e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.713 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df67e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.713 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df67e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df67e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df67e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df1fde700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df1fde700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df1fde700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df1fde700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.714 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:08.709+0000 7f1df1fde700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:08.731 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:21:08.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 Processor -- start 2023-12-19T09:21:08.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 -- start start 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 0x7f78780c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f787806c7c0 0x7f787815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787806d970 con 0x7f787806de00 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787806cd60 con 0x7f787806c7c0 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787d40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787806d040 con 0x7f78780c69a0 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f78777fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f787806de00 0x7f787806b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:56160/0 (socket says 172.21.15.111:56160) 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f78777fe700 1 -- 172.21.15.111:0/2264106333 learned_addr learned my addr 172.21.15.111:0/2264106333 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f7877fff700 1 --2- 172.21.15.111:0/2264106333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 0x7f78780c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f787cc0b700 1 --2- 172.21.15.111:0/2264106333 >> v2:172.21.15.143:3300/0 conn(0x7f787806c7c0 0x7f787815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f7876ffd700 1 -- 172.21.15.111:0/2264106333 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174896248 0 0) 0x7f787806d970 con 0x7f787806de00 2023-12-19T09:21:08.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.831+0000 7f7876ffd700 1 -- 172.21.15.111:0/2264106333 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7860003570 con 0x7f787806de00 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7877fff700 1 -- 172.21.15.111:0/2264106333 >> v2:172.21.15.143:3300/0 conn(0x7f787806c7c0 msgr2=0x7f787815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7877fff700 1 --2- 172.21.15.111:0/2264106333 >> v2:172.21.15.143:3300/0 conn(0x7f787806c7c0 0x7f787815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7877fff700 1 -- 172.21.15.111:0/2264106333 >> v1:172.21.15.111:6789/0 conn(0x7f787806de00 legacy=0x7f787806b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7877fff700 1 -- 172.21.15.111:0/2264106333 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787806d320 con 0x7f78780c69a0 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7877fff700 1 --2- 172.21.15.111:0/2264106333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 0x7f78780c6d70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7868009910 tx=0x7f78680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cd6bd3f39206c5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7876ffd700 1 -- 172.21.15.111:0/2264106333 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868011630 con 0x7f78780c69a0 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7876ffd700 1 -- 172.21.15.111:0/2264106333 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7868011790 con 0x7f78780c69a0 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f7876ffd700 1 -- 172.21.15.111:0/2264106333 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868011b30 con 0x7f78780c69a0 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 -- 172.21.15.111:0/2264106333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 msgr2=0x7f78780c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 --2- 172.21.15.111:0/2264106333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 0x7f78780c6d70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7868009910 tx=0x7f78680091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 -- 172.21.15.111:0/2264106333 shutdown_connections 2023-12-19T09:21:08.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 --2- 172.21.15.111:0/2264106333 >> v2:172.21.15.143:3300/0 conn(0x7f787806c7c0 0x7f787815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 --2- 172.21.15.111:0/2264106333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78780c69a0 0x7f78780c6d70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 -- 172.21.15.111:0/2264106333 >> 172.21.15.111:0/2264106333 conn(0x7f7878067b50 msgr2=0x7f78780cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:08.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 -- 172.21.15.111:0/2264106333 shutdown_connections 2023-12-19T09:21:08.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.835+0000 7f787d40c700 1 -- 172.21.15.111:0/2264106333 wait complete. 2023-12-19T09:21:08.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 Processor -- start 2023-12-19T09:21:08.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 -- start start 2023-12-19T09:21:08.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f787806c7c0 0x7f787815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f787806de00 0x7f787815f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787815fac0 con 0x7f78780c69a0 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787815f0f0 con 0x7f787806c7c0 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787d40c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787815e240 con 0x7f787806de00 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39416/0 (socket says 172.21.15.111:39416) 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 -- 172.21.15.111:0/1924575620 learned_addr learned my addr 172.21.15.111:0/1924575620 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f78777fe700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f787806de00 0x7f787815f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f7877fff700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f787806c7c0 0x7f787815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 -- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f787806de00 msgr2=0x7f787815f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f787806de00 0x7f787815f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 -- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f787806c7c0 msgr2=0x7f787815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f787806c7c0 0x7f787815ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.839+0000 7f787cc0b700 1 -- 172.21.15.111:0/1924575620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868009420 con 0x7f78780c69a0 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f787cc0b700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f7870009e90 tx=0x7f787000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:08.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000cb70 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000f040 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787815e4e0 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870014480 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787815eaa0 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787804be40 con 0x7f78780c69a0 2023-12-19T09:21:08.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.843+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 99) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f78700147f0 con 0x7f78780c69a0 2023-12-19T09:21:08.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.847+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 7317+0+0 (secure 0 0 0) 0x7f787007d680 con 0x7f78780c69a0 2023-12-19T09:21:08.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:08.851+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f787004d8f0 con 0x7f78780c69a0 2023-12-19T09:21:09.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.059+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f787815f230 con 0x7f78780c69a0 2023-12-19T09:21:09.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.651+0000 7f785ffff700 1 -- 172.21.15.111:0/1924575620 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7870050f10 con 0x7f78780c69a0 2023-12-19T09:21:09.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 msgr2=0x7f787815dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:09.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f7870009e90 tx=0x7f787000b040 comp rx=0 tx=0).stop 2023-12-19T09:21:09.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 shutdown_connections 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78780c69a0 0x7f787815dd00 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f787806de00 0x7f787815f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 --2- 172.21.15.111:0/1924575620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f787806c7c0 0x7f787815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 >> 172.21.15.111:0/1924575620 conn(0x7f7878067b50 msgr2=0x7f78780add60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 shutdown_connections 2023-12-19T09:21:09.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.655+0000 7f787d40c700 1 -- 172.21.15.111:0/1924575620 wait complete. 2023-12-19T09:21:09.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.847+0000 7fde5359e700 1 Processor -- start 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.847+0000 7fde5359e700 1 -- start start 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde540865c0 0x7fde541694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde5406d9f0 0x7fde54086990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540add50 0x7fde5406a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5406e1d0 con 0x7fde540865c0 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540ae1d0 con 0x7fde5406d9f0 2023-12-19T09:21:09.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540ac720 con 0x7fde540add50 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde540865c0 0x7fde541694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fde540865c0 0x7fde541694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39428/0 (socket says 172.21.15.111:39428) 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde5259c700 1 -- 172.21.15.111:0/311785060 learned_addr learned my addr 172.21.15.111:0/311785060 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde52d9d700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540add50 0x7fde5406a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.851+0000 7fde51d9b700 1 --2- 172.21.15.111:0/311785060 >> v2:172.21.15.143:3300/0 conn(0x7fde5406d9f0 0x7fde54086990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 -- 172.21.15.111:0/311785060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540add50 msgr2=0x7fde5406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540add50 0x7fde5406a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 -- 172.21.15.111:0/311785060 >> v2:172.21.15.143:3300/0 conn(0x7fde5406d9f0 msgr2=0x7fde54086990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 --2- 172.21.15.111:0/311785060 >> v2:172.21.15.143:3300/0 conn(0x7fde5406d9f0 0x7fde54086990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 -- 172.21.15.111:0/311785060 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5406def0 con 0x7fde540865c0 2023-12-19T09:21:09.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5259c700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde540865c0 0x7fde541694a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fde54070a70 tx=0x7fde4400a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ec4ba10101fa26 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:09.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.867+0000 7fde5159a700 1 -- 172.21.15.111:0/311785060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4400ae60 con 0x7fde540865c0 2023-12-19T09:21:09.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5159a700 1 -- 172.21.15.111:0/311785060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4400b750 con 0x7fde540865c0 2023-12-19T09:21:09.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5159a700 1 -- 172.21.15.111:0/311785060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4400baf0 con 0x7fde540865c0 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 -- 172.21.15.111:0/311785060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde540865c0 msgr2=0x7fde541694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde540865c0 0x7fde541694a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fde54070a70 tx=0x7fde4400a330 comp rx=0 tx=0).stop 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 -- 172.21.15.111:0/311785060 shutdown_connections 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540add50 0x7fde5406a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 --2- 172.21.15.111:0/311785060 >> v2:172.21.15.143:3300/0 conn(0x7fde5406d9f0 0x7fde54086990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 --2- 172.21.15.111:0/311785060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde540865c0 0x7fde541694a0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 -- 172.21.15.111:0/311785060 >> 172.21.15.111:0/311785060 conn(0x7fde54067bb0 msgr2=0x7fde540c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:09.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 -- 172.21.15.111:0/311785060 shutdown_connections 2023-12-19T09:21:09.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.871+0000 7fde5359e700 1 -- 172.21.15.111:0/311785060 wait complete. 2023-12-19T09:21:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 Processor -- start 2023-12-19T09:21:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 -- start start 2023-12-19T09:21:09.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540865c0 0x7fde54162410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540add50 0x7fde540c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde540c1a30 con 0x7fde5406d9f0 2023-12-19T09:21:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde540ae950 con 0x7fde540add50 2023-12-19T09:21:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde540c17d0 con 0x7fde540865c0 2023-12-19T09:21:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39430/0 (socket says 172.21.15.111:39430) 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde5259c700 1 -- 172.21.15.111:0/393560382 learned_addr learned my addr 172.21.15.111:0/393560382 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde51d9b700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540865c0 0x7fde54162410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.875+0000 7fde52d9d700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540add50 0x7fde540c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 -- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540865c0 msgr2=0x7fde54162410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540865c0 0x7fde54162410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 -- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540add50 msgr2=0x7fde540c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540add50 0x7fde540c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 -- 172.21.15.111:0/393560382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4400a590 con 0x7fde5406d9f0 2023-12-19T09:21:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5259c700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fde5406e6d0 tx=0x7fde44019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4401f070 con 0x7fde5406d9f0 2023-12-19T09:21:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4400b8c0 con 0x7fde5406d9f0 2023-12-19T09:21:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde540c6fb0 con 0x7fde5406d9f0 2023-12-19T09:21:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44019450 con 0x7fde5406d9f0 2023-12-19T09:21:09.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde540c7570 con 0x7fde5406d9f0 2023-12-19T09:21:09.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.879+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5404be40 con 0x7fde5406d9f0 2023-12-19T09:21:09.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.883+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 100) v1 ==== 79799+0+0 (secure 0 0 0) 0x7fde440197c0 con 0x7fde5406d9f0 2023-12-19T09:21:09.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.883+0000 7fde427fc700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 0x7fde34062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:09.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.883+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 7317+0+0 (secure 0 0 0) 0x7fde4407d8b0 con 0x7fde5406d9f0 2023-12-19T09:21:09.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.883+0000 7fde51d9b700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 0x7fde34062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:09.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.887+0000 7fde51d9b700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 0x7fde34062850 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde540b1010 tx=0x7fde48007090 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:09.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:09.887+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde4404dab0 con 0x7fde5406d9f0 2023-12-19T09:21:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.063+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde540c1a30 con 0x7fde5406d9f0 2023-12-19T09:21:10.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.703+0000 7fde427fc700 1 -- 172.21.15.111:0/393560382 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v198) v1 ==== 181+0+0 (secure 0 0 0) 0x7fde4400b040 con 0x7fde5406d9f0 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 msgr2=0x7fde34062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 0x7fde34062850 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde540b1010 tx=0x7fde48007090 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 msgr2=0x7fde540c1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fde5406e6d0 tx=0x7fde44019000 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 shutdown_connections 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fde340603a0 0x7fde34062850 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde540add50 0x7fde540c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde540865c0 0x7fde54162410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 --2- 172.21.15.111:0/393560382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5406d9f0 0x7fde540c1290 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 >> 172.21.15.111:0/393560382 conn(0x7fde54067bb0 msgr2=0x7fde540ab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 shutdown_connections 2023-12-19T09:21:10.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.711+0000 7fde5359e700 1 -- 172.21.15.111:0/393560382 wait complete. 2023-12-19T09:21:10.721 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T09:21:10.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T09:21:10.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 Processor -- start 2023-12-19T09:21:10.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 -- start start 2023-12-19T09:21:10.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5d8806b6f0 0x7f5d8806bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d880b2160 0x7f5d88159d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 0x7f5d8806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880cd3d0 con 0x7f5d8806b6f0 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d881598d0 con 0x7f5d880b2160 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8d163700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b2590 con 0x7f5d8806a580 2023-12-19T09:21:10.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d87fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5d8806b6f0 0x7f5d8806bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d87fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5d8806b6f0 0x7f5d8806bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39444/0 (socket says 172.21.15.111:39444) 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d87fff700 1 -- 172.21.15.111:0/2127149807 learned_addr learned my addr 172.21.15.111:0/2127149807 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d877fe700 1 --2- 172.21.15.111:0/2127149807 >> v2:172.21.15.143:3300/0 conn(0x7f5d880b2160 0x7f5d88159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.903+0000 7f5d8c962700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 0x7f5d8806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 -- 172.21.15.111:0/2127149807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 msgr2=0x7f5d8806a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 0x7f5d8806a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 -- 172.21.15.111:0/2127149807 >> v2:172.21.15.143:3300/0 conn(0x7f5d880b2160 msgr2=0x7f5d88159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:10.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2127149807 >> v2:172.21.15.143:3300/0 conn(0x7f5d880b2160 0x7f5d88159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 -- 172.21.15.111:0/2127149807 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d881592a0 con 0x7f5d8806b6f0 2023-12-19T09:21:10.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8c962700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 0x7f5d8806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:10.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806b6f0 0x7f5d8806bac0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5d88159490 tx=0x7f5d78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae807400ac8089d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d86ffd700 1 -- 172.21.15.111:0/2127149807 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800b830 con 0x7f5d8806b6f0 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d86ffd700 1 -- 172.21.15.111:0/2127149807 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7800b990 con 0x7f5d8806b6f0 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d86ffd700 1 -- 172.21.15.111:0/2127149807 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800bd50 con 0x7f5d8806b6f0 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 -- 172.21.15.111:0/2127149807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806b6f0 msgr2=0x7f5d8806bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806b6f0 0x7f5d8806bac0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f5d88159490 tx=0x7f5d78009250 comp rx=0 tx=0).stop 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 -- 172.21.15.111:0/2127149807 shutdown_connections 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d8806a580 0x7f5d8806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2127149807 >> v2:172.21.15.143:3300/0 conn(0x7f5d880b2160 0x7f5d88159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2127149807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806b6f0 0x7f5d8806bac0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 -- 172.21.15.111:0/2127149807 >> 172.21.15.111:0/2127149807 conn(0x7f5d88067bb0 msgr2=0x7f5d880cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 -- 172.21.15.111:0/2127149807 shutdown_connections 2023-12-19T09:21:10.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.907+0000 7f5d8d163700 1 -- 172.21.15.111:0/2127149807 wait complete. 2023-12-19T09:21:10.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 Processor -- start 2023-12-19T09:21:10.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 -- start start 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d8806b6f0 0x7f5d88156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 0x7f5d881531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88156690 con 0x7f5d8806a580 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88155cc0 con 0x7f5d8806b6f0 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8d163700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88154e10 con 0x7f5d880b2160 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39454/0 (socket says 172.21.15.111:39454) 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 -- 172.21.15.111:0/2069485333 learned_addr learned my addr 172.21.15.111:0/2069485333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8c962700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 0x7f5d881531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d877fe700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d8806b6f0 0x7f5d88156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 -- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 msgr2=0x7f5d881531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 0x7f5d881531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 -- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d8806b6f0 msgr2=0x7f5d88156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d8806b6f0 0x7f5d88156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 -- 172.21.15.111:0/2069485333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780094f0 con 0x7f5d8806a580 2023-12-19T09:21:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d8c962700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 0x7f5d881531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.911+0000 7f5d87fff700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009290 tx=0x7f5d78016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78011d50 con 0x7f5d8806a580 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78020070 con 0x7f5d8806a580 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7801b6a0 con 0x7f5d8806a580 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d881536e0 con 0x7f5d8806a580 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d88155390 con 0x7f5d8806a580 2023-12-19T09:21:10.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8804be40 con 0x7f5d8806a580 2023-12-19T09:21:10.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.915+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5d78022030 con 0x7f5d8806a580 2023-12-19T09:21:10.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.919+0000 7f5d6ffff700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 0x7f5d74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:10.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.919+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5d7807eba0 con 0x7f5d8806a580 2023-12-19T09:21:10.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.919+0000 7f5d877fe700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 0x7f5d74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:10.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.919+0000 7f5d877fe700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 0x7f5d74062a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d88166ec0 tx=0x7f5d7c008040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:10.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:10.923+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5d7804ed00 con 0x7f5d8806a580 2023-12-19T09:21:11.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.095+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d880b0360 con 0x7f5d8806a580 2023-12-19T09:21:11.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.815+0000 7f5d6ffff700 1 -- 172.21.15.111:0/2069485333 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v199) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5d78006cd0 con 0x7f5d8806a580 2023-12-19T09:21:11.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 msgr2=0x7f5d74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:11.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 0x7f5d74062a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d88166ec0 tx=0x7f5d7c008040 comp rx=0 tx=0).stop 2023-12-19T09:21:11.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 msgr2=0x7f5d8814e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009290 tx=0x7f5d78016000 comp rx=0 tx=0).stop 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 shutdown_connections 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5d74060550 0x7f5d74062a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d8806b6f0 0x7f5d88156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d880b2160 0x7f5d881531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 --2- 172.21.15.111:0/2069485333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d8806a580 0x7f5d8814e3e0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:11.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.819+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 >> 172.21.15.111:0/2069485333 conn(0x7f5d88067bb0 msgr2=0x7f5d88157750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:11.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.823+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 shutdown_connections 2023-12-19T09:21:11.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:11.823+0000 7f5d8d163700 1 -- 172.21.15.111:0/2069485333 wait complete. 2023-12-19T09:21:11.831 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T09:21:11.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T09:21:12.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.003+0000 7fd941745700 1 Processor -- start 2023-12-19T09:21:12.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 -- start start 2023-12-19T09:21:12.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 0x7fd93c06acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd93c0cecd0 0x7fd93c06b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c0cbdf0 con 0x7fd93c158e20 2023-12-19T09:21:12.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0cc0d0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0cc3b0 con 0x7fd93c0cb970 2023-12-19T09:21:12.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd93c158e20 0x7fd93c0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46082/0 (socket says 172.21.15.111:46082) 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93affd700 1 -- 172.21.15.111:0/3698533987 learned_addr learned my addr 172.21.15.111:0/3698533987 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 --2- 172.21.15.111:0/3698533987 >> v2:172.21.15.143:3300/0 conn(0x7fd93c0cecd0 0x7fd93c06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93a7fc700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 0x7fd93c06acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd939ffb700 1 -- 172.21.15.111:0/3698533987 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576399698 0 0) 0x7fd93c0cbdf0 con 0x7fd93c158e20 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd939ffb700 1 -- 172.21.15.111:0/3698533987 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924003570 con 0x7fd93c158e20 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 -- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 msgr2=0x7fd93c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 0x7fd93c06acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 -- 172.21.15.111:0/3698533987 >> v1:172.21.15.111:6789/0 conn(0x7fd93c158e20 legacy=0x7fd93c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 -- 172.21.15.111:0/3698533987 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0cc690 con 0x7fd93c0cecd0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93a7fc700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 0x7fd93c06acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd93b7fe700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0cecd0 0x7fd93c06b200 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd934009e60 tx=0x7fd934009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7482ab090b4ccb5b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd939ffb700 1 -- 172.21.15.111:0/3698533987 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400e070 con 0x7fd93c0cecd0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd939ffb700 1 -- 172.21.15.111:0/3698533987 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93400bab0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd939ffb700 1 -- 172.21.15.111:0/3698533987 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400be70 con 0x7fd93c0cecd0 2023-12-19T09:21:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 -- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0cecd0 msgr2=0x7fd93c06b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.007+0000 7fd941745700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0cecd0 0x7fd93c06b200 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd934009e60 tx=0x7fd934009780 comp rx=0 tx=0).stop 2023-12-19T09:21:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- 172.21.15.111:0/3698533987 shutdown_connections 2023-12-19T09:21:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0cecd0 0x7fd93c06b200 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 --2- 172.21.15.111:0/3698533987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c0cb970 0x7fd93c06acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- 172.21.15.111:0/3698533987 >> 172.21.15.111:0/3698533987 conn(0x7fd93c067bb0 msgr2=0x7fd93c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:12.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- 172.21.15.111:0/3698533987 shutdown_connections 2023-12-19T09:21:12.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- 172.21.15.111:0/3698533987 wait complete. 2023-12-19T09:21:12.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 Processor -- start 2023-12-19T09:21:12.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- start start 2023-12-19T09:21:12.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c158e20 0x7fd93c14f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 0x7fd93c15ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b4e20 con 0x7fd93c0cecd0 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b4210 con 0x7fd93c0c9d50 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd941745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0b44f0 con 0x7fd93c158e20 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60906/0 (socket says 172.21.15.111:60906) 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 -- 172.21.15.111:0/3431543243 learned_addr learned my addr 172.21.15.111:0/3431543243 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93a7fc700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c158e20 0x7fd93c14f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93b7fe700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 0x7fd93c15ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 -- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c158e20 msgr2=0x7fd93c14f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c158e20 0x7fd93c14f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 -- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 msgr2=0x7fd93c15ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 0x7fd93c15ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93affd700 1 -- 172.21.15.111:0/3431543243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930009420 con 0x7fd93c0cecd0 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.011+0000 7fd93b7fe700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 0x7fd93c15ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:12.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd93affd700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c7c0 tx=0x7fd92c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c005cd0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c005e30 con 0x7fd93c0cecd0 2023-12-19T09:21:12.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934009a20 con 0x7fd93c0cecd0 2023-12-19T09:21:12.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c007320 con 0x7fd93c0cecd0 2023-12-19T09:21:12.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c0b4ab0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c071120 con 0x7fd93c0cecd0 2023-12-19T09:21:12.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fd92c01b3b0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.015+0000 7fd922ffd700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 0x7fd928062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.019+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7fd92c07c190 con 0x7fd93c0cecd0 2023-12-19T09:21:12.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.019+0000 7fd93a7fc700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 0x7fd928062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.019+0000 7fd93a7fc700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 0x7fd928062a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c8770 tx=0x7fd9300091a0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.023+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd92c04c400 con 0x7fd93c0cecd0 2023-12-19T09:21:12.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.195+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd93c06a7a0 con 0x7fd93c0cecd0 2023-12-19T09:21:12.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.203+0000 7fd922ffd700 1 -- 172.21.15.111:0/3431543243 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v67) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd92c04fa20 con 0x7fd93c0cecd0 2023-12-19T09:21:12.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 msgr2=0x7fd928062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 0x7fd928062a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c8770 tx=0x7fd9300091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 msgr2=0x7fd93c0b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c7c0 tx=0x7fd92c007e00 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 shutdown_connections 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd928060550 0x7fd928062a00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd93c0c9d50 0x7fd93c15ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c158e20 0x7fd93c14f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 --2- 172.21.15.111:0/3431543243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd93c0cecd0 0x7fd93c0b3970 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 >> 172.21.15.111:0/3431543243 conn(0x7fd93c067bb0 msgr2=0x7fd93c0cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 shutdown_connections 2023-12-19T09:21:12.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.207+0000 7fd941745700 1 -- 172.21.15.111:0/3431543243 wait complete. 2023-12-19T09:21:12.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T09:21:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 Processor -- start 2023-12-19T09:21:12.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 -- start start 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa27c158e20 0x7fa27c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 0x7fa27c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c0ca2c0 0x7fa27c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c0cbdf0 con 0x7fa27c158e20 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c0cc0d0 con 0x7fa27c0cb970 2023-12-19T09:21:12.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.399+0000 7fa281c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0cc3b0 con 0x7fa27c0ca2c0 2023-12-19T09:21:12.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa27c158e20 0x7fa27c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa27c158e20 0x7fa27c0ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60916/0 (socket says 172.21.15.111:60916) 2023-12-19T09:21:12.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 -- 172.21.15.111:0/708186916 learned_addr learned my addr 172.21.15.111:0/708186916 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:12.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27affd700 1 --2- 172.21.15.111:0/708186916 >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 0x7fa27c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 -- 172.21.15.111:0/708186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c0ca2c0 msgr2=0x7fa27c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/708186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c0ca2c0 0x7fa27c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 -- 172.21.15.111:0/708186916 >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 msgr2=0x7fa27c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/708186916 >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 0x7fa27c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 -- 172.21.15.111:0/708186916 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0cc690 con 0x7fa27c158e20 2023-12-19T09:21:12.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27affd700 1 --2- 172.21.15.111:0/708186916 >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 0x7fa27c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/708186916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c158e20 0x7fa27c0ce5c0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0099b0 tx=0x7fa26c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2651217e98592e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27a7fc700 1 -- 172.21.15.111:0/708186916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c00b820 con 0x7fa27c158e20 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27a7fc700 1 -- 172.21.15.111:0/708186916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c00b980 con 0x7fa27c158e20 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa27a7fc700 1 -- 172.21.15.111:0/708186916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c00bd20 con 0x7fa27c158e20 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 -- 172.21.15.111:0/708186916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c158e20 msgr2=0x7fa27c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 --2- 172.21.15.111:0/708186916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c158e20 0x7fa27c0ce5c0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0099b0 tx=0x7fa26c009250 comp rx=0 tx=0).stop 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 -- 172.21.15.111:0/708186916 shutdown_connections 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 --2- 172.21.15.111:0/708186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c0ca2c0 0x7fa27c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 --2- 172.21.15.111:0/708186916 >> v2:172.21.15.143:3300/0 conn(0x7fa27c0cb970 0x7fa27c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 --2- 172.21.15.111:0/708186916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c158e20 0x7fa27c0ce5c0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 -- 172.21.15.111:0/708186916 >> 172.21.15.111:0/708186916 conn(0x7fa27c067bb0 msgr2=0x7fa27c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 -- 172.21.15.111:0/708186916 shutdown_connections 2023-12-19T09:21:12.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.403+0000 7fa281c78700 1 -- 172.21.15.111:0/708186916 wait complete. 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 Processor -- start 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 -- start start 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa27c0ca2c0 0x7fa27c14f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c158e20 0x7fa27c0c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0c94a0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c14fcd0 con 0x7fa27c0ca2c0 2023-12-19T09:21:12.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa281c78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0c8890 con 0x7fa27c158e20 2023-12-19T09:21:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa27bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa27bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60930/0 (socket says 172.21.15.111:60930) 2023-12-19T09:21:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa27affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c158e20 0x7fa27c0c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa27b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa27c0ca2c0 0x7fa27c14f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.407+0000 7fa27bfff700 1 -- 172.21.15.111:0/81688205 learned_addr learned my addr 172.21.15.111:0/81688205 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 -- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c158e20 msgr2=0x7fa27c0c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c158e20 0x7fa27c0c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 -- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa27c0ca2c0 msgr2=0x7fa27c14f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa27c0ca2c0 0x7fa27c14f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 -- 172.21.15.111:0/81688205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c0094f0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa27bfff700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa27400bc30 tx=0x7fa274009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2740092e0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27400c630 con 0x7fa27c0c9e70 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400ca10 con 0x7fa27c0c9e70 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c0c8b70 con 0x7fa27c0c9e70 2023-12-19T09:21:12.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c0c9130 con 0x7fa27c0c9e70 2023-12-19T09:21:12.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.411+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c0c9610 con 0x7fa27c0c9e70 2023-12-19T09:21:12.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.415+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fa274012070 con 0x7fa27c0c9e70 2023-12-19T09:21:12.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.415+0000 7fa2637fe700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 0x7fa268062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.415+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7fa27407c5c0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.415+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 0x7fa268062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.415+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 0x7fa268062a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0099b0 tx=0x7fa26c017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.423+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa27404c8d0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.591+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa27c06a7a0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.603+0000 7fa2637fe700 1 -- 172.21.15.111:0/81688205 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa27404fef0 con 0x7fa27c0c9e70 2023-12-19T09:21:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 msgr2=0x7fa268062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 0x7fa268062a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0099b0 tx=0x7fa26c017000 comp rx=0 tx=0).stop 2023-12-19T09:21:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 msgr2=0x7fa27c15cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa27400bc30 tx=0x7fa274009840 comp rx=0 tx=0).stop 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 shutdown_connections 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0c9e70 0x7fa27c15cb60 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa27c158e20 0x7fa27c0c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa268060550 0x7fa268062a00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 --2- 172.21.15.111:0/81688205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa27c0ca2c0 0x7fa27c14f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 >> 172.21.15.111:0/81688205 conn(0x7fa27c067bb0 msgr2=0x7fa27c06b180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 shutdown_connections 2023-12-19T09:21:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.607+0000 7fa281c78700 1 -- 172.21.15.111:0/81688205 wait complete. 2023-12-19T09:21:12.634 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T09:21:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.791+0000 7f144df63700 1 Processor -- start 2023-12-19T09:21:12.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.791+0000 7f144df63700 1 -- start start 2023-12-19T09:21:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f14480acf30 0x7f14481557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448156780 con 0x7f14480ad300 2023-12-19T09:21:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14481532a0 con 0x7f14480acf30 2023-12-19T09:21:12.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1448158f90 con 0x7f144806d1d0 2023-12-19T09:21:12.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f14480acf30 0x7f14481557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f14480acf30 0x7f14481557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38196/0 (socket says 172.21.15.111:38196) 2023-12-19T09:21:12.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 -- 172.21.15.111:0/1667165390 learned_addr learned my addr 172.21.15.111:0/1667165390 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:12.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f1447fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f14480ad300 0x7f1448159420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:53568/0 (socket says 172.21.15.111:53568) 2023-12-19T09:21:12.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144d762700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 -- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 msgr2=0x7f14480ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480ac1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 -- 172.21.15.111:0/1667165390 >> v1:172.21.15.143:6789/0 conn(0x7f14480ad300 legacy=0x7f1448159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 -- 172.21.15.111:0/1667165390 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144806d650 con 0x7f14480acf30 2023-12-19T09:21:12.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144cf61700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14481557d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f143c00d1a0 tx=0x7f143c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf6749132e798ad server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f14477fe700 1 -- 172.21.15.111:0/1667165390 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c010070 con 0x7f14480acf30 2023-12-19T09:21:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f14477fe700 1 -- 172.21.15.111:0/1667165390 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c00de10 con 0x7f14480acf30 2023-12-19T09:21:12.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f14477fe700 1 -- 172.21.15.111:0/1667165390 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00b320 con 0x7f14480acf30 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 -- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 msgr2=0x7f14481557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.795+0000 7f144df63700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14481557d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f143c00d1a0 tx=0x7f143c00ba50 comp rx=0 tx=0).stop 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- 172.21.15.111:0/1667165390 shutdown_connections 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480ac1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 --2- 172.21.15.111:0/1667165390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14481557d0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- 172.21.15.111:0/1667165390 >> 172.21.15.111:0/1667165390 conn(0x7f1448067b50 msgr2=0x7f14480cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- 172.21.15.111:0/1667165390 shutdown_connections 2023-12-19T09:21:12.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- 172.21.15.111:0/1667165390 wait complete. 2023-12-19T09:21:12.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 Processor -- start 2023-12-19T09:21:12.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- start start 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448169450 con 0x7f14480ad300 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14480c57b0 con 0x7f14480acf30 2023-12-19T09:21:12.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.799+0000 7f144df63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1448168840 con 0x7f144806d1d0 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1447fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144cf61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60938/0 (socket says 172.21.15.111:60938) 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1447fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38210/0 (socket says 172.21.15.111:38210) 2023-12-19T09:21:12.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 -- 172.21.15.111:0/4048718824 learned_addr learned my addr 172.21.15.111:0/4048718824 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 -- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 msgr2=0x7f14480c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 -- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 msgr2=0x7f14480cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 -- 172.21.15.111:0/4048718824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c00bcf0 con 0x7f14480ad300 2023-12-19T09:21:12.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1447fff700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:12.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144d762700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f144000e8d0 tx=0x7f14400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440013680 con 0x7f14480ad300 2023-12-19T09:21:12.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14400137e0 con 0x7f14480ad300 2023-12-19T09:21:12.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440013ba0 con 0x7f14480ad300 2023-12-19T09:21:12.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1448168b20 con 0x7f14480ad300 2023-12-19T09:21:12.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.803+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14481690e0 con 0x7f14480ad300 2023-12-19T09:21:12.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.807+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1448154570 con 0x7f14480ad300 2023-12-19T09:21:12.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.807+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f14400123d0 con 0x7f14480ad300 2023-12-19T09:21:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.807+0000 7f1444ff9700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 0x7f142c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.807+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f144007cbf0 con 0x7f14480ad300 2023-12-19T09:21:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.807+0000 7f144cf61700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 0x7f142c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.811+0000 7f144cf61700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 0x7f142c062a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f143c00ba90 tx=0x7f143c01a000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:12.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.815+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1448154570 con 0x7f14480ad300 2023-12-19T09:21:12.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.987+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1448154570 con 0x7f14480ad300 2023-12-19T09:21:13.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.995+0000 7f1444ff9700 1 -- 172.21.15.111:0/4048718824 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v69) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1448154570 con 0x7f14480ad300 2023-12-19T09:21:13.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 msgr2=0x7f142c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 0x7f142c062a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f143c00ba90 tx=0x7f143c01a000 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 msgr2=0x7f144814f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f144000e8d0 tx=0x7f14400091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 shutdown_connections 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f142c060550 0x7f142c062a00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14480acf30 0x7f14480cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14480ad300 0x7f144814f240 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 --2- 172.21.15.111:0/4048718824 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f144806d1d0 0x7f14480c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 >> 172.21.15.111:0/4048718824 conn(0x7f1448067b50 msgr2=0x7f14481504c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 shutdown_connections 2023-12-19T09:21:13.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:12.999+0000 7f144df63700 1 -- 172.21.15.111:0/4048718824 wait complete. 2023-12-19T09:21:13.023 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-12-19T09:21:13.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.175+0000 7fc74e123700 1 Processor -- start 2023-12-19T09:21:13.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.175+0000 7fc74e123700 1 -- start start 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74e123700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc74815d330 0x7fc74806b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74e123700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74806b470 con 0x7fc7480c6cd0 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74e123700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74806e110 con 0x7fc7480c6900 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74e123700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74806be40 con 0x7fc74815d330 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc7480c6cd0 0x7fc74815cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46098/0 (socket says 172.21.15.111:46098) 2023-12-19T09:21:13.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 -- 172.21.15.111:0/2228165873 learned_addr learned my addr 172.21.15.111:0/2228165873 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:13.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74d922700 1 --2- 172.21.15.111:0/2228165873 >> v2:172.21.15.143:3300/0 conn(0x7fc74815d330 0x7fc74806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675922904 0 0) 0x7fc74806b470 con 0x7fc7480c6cd0 2023-12-19T09:21:13.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc734003570 con 0x7fc7480c6cd0 2023-12-19T09:21:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3889461892 0 0) 0x7fc734003570 con 0x7fc7480c6cd0 2023-12-19T09:21:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 >> v2:172.21.15.143:3300/0 conn(0x7fc74815d330 msgr2=0x7fc74806b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 --2- 172.21.15.111:0/2228165873 >> v2:172.21.15.143:3300/0 conn(0x7fc74815d330 0x7fc74806b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 >> v1:172.21.15.143:6789/0 conn(0x7fc7480c6900 legacy=0x7fc74806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748161d10 con 0x7fc7480c6cd0 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fc738001470 con 0x7fc7480c6cd0 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 >> v1:172.21.15.111:6789/0 conn(0x7fc7480c6cd0 legacy=0x7fc74815cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc734003a60 0x7fc734003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 0x7fc734004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7340051c0 0x7fc73400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73400c0a0 con 0x7fc734004810 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73400c210 con 0x7fc734003a60 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73400c3d0 con 0x7fc7340051c0 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 0x7fc734004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74d922700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7340051c0 0x7fc73400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74d121700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc734003a60 0x7fc734003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 -- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7340051c0 msgr2=0x7fc73400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7340051c0 0x7fc73400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 -- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc734003a60 msgr2=0x7fc734003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc734003a60 0x7fc734003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74c920700 1 -- 172.21.15.111:0/2228165873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734003570 con 0x7fc734004810 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.179+0000 7fc74d922700 1 -- 172.21.15.111:0/2228165873 reap_dead start 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74c920700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 0x7fc734004c80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c3c0 tx=0x7fc738001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeb9c1960ff909c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73800cc10 con 0x7fc734004810 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73800cd70 con 0x7fc734004810 2023-12-19T09:21:13.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc73ffff700 1 -- 172.21.15.111:0/2228165873 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738005b50 con 0x7fc734004810 2023-12-19T09:21:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 -- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 msgr2=0x7fc734004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 0x7fc734004c80 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c3c0 tx=0x7fc738001e50 comp rx=0 tx=0).stop 2023-12-19T09:21:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 -- 172.21.15.111:0/2228165873 shutdown_connections 2023-12-19T09:21:13.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 --2- 172.21.15.111:0/2228165873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc734004810 0x7fc734004c80 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 -- 172.21.15.111:0/2228165873 >> 172.21.15.111:0/2228165873 conn(0x7fc748067bb0 msgr2=0x7fc7480cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:13.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 -- 172.21.15.111:0/2228165873 shutdown_connections 2023-12-19T09:21:13.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.183+0000 7fc74e123700 1 -- 172.21.15.111:0/2228165873 wait complete. 2023-12-19T09:21:13.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 Processor -- start 2023-12-19T09:21:13.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 -- start start 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc734004810 0x7fc7480adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7480ad230 0x7fc7480afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b0440 con 0x7fc7480aff80 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b0720 con 0x7fc7480ad230 2023-12-19T09:21:13.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.187+0000 7fc74e123700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b0a00 con 0x7fc734004810 2023-12-19T09:21:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60956/0 (socket says 172.21.15.111:60956) 2023-12-19T09:21:13.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 -- 172.21.15.111:0/2277371932 learned_addr learned my addr 172.21.15.111:0/2277371932 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d121700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc734004810 0x7fc7480adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74c920700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7480ad230 0x7fc7480afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 -- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc734004810 msgr2=0x7fc7480adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc734004810 0x7fc7480adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 -- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7480ad230 msgr2=0x7fc7480afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7480ad230 0x7fc7480afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 -- 172.21.15.111:0/2277371932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480b0ce0 con 0x7fc7480aff80 2023-12-19T09:21:13.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74d922700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc7480c5e70 tx=0x7fc74000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74000e040 con 0x7fc7480aff80 2023-12-19T09:21:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74000f040 con 0x7fc7480aff80 2023-12-19T09:21:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7480b0fc0 con 0x7fc7480aff80 2023-12-19T09:21:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74000fe20 con 0x7fc7480aff80 2023-12-19T09:21:13.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7481612d0 con 0x7fc7480aff80 2023-12-19T09:21:13.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.191+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74804be40 con 0x7fc7480aff80 2023-12-19T09:21:13.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.195+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc740015640 con 0x7fc7480aff80 2023-12-19T09:21:13.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.195+0000 7fc73d7fa700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 0x7fc72c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:13.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.195+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7fc74007d410 con 0x7fc7480aff80 2023-12-19T09:21:13.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.195+0000 7fc74d121700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 0x7fc72c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:13.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.199+0000 7fc74d121700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 0x7fc72c062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc744000c00 tx=0x7fc744007d30 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:13.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.199+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc74004d680 con 0x7fc7480aff80 2023-12-19T09:21:13.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.423+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fc7480cc490 con 0x7fc7480aff80 2023-12-19T09:21:13.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.827+0000 7fc73d7fa700 1 -- 172.21.15.111:0/2277371932 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3039) v1 ==== 163+0+0 (secure 0 0 0) 0x7fc740050ca0 con 0x7fc7480aff80 2023-12-19T09:21:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 msgr2=0x7fc72c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 0x7fc72c062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc744000c00 tx=0x7fc744007d30 comp rx=0 tx=0).stop 2023-12-19T09:21:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 msgr2=0x7fc748160d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:13.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc7480c5e70 tx=0x7fc74000b040 comp rx=0 tx=0).stop 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 shutdown_connections 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc72c060550 0x7fc72c062a00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7480ad230 0x7fc7480afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480aff80 0x7fc748160d90 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 --2- 172.21.15.111:0/2277371932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc734004810 0x7fc7480adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 >> 172.21.15.111:0/2277371932 conn(0x7fc748067bb0 msgr2=0x7fc74806d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:13.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 shutdown_connections 2023-12-19T09:21:13.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:13.831+0000 7fc74e123700 1 -- 172.21.15.111:0/2277371932 wait complete. 2023-12-19T09:21:13.858 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-12-19T09:21:13.859 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-12-19T09:21:13.859 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-12-19T09:21:13.859 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:14.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 Processor -- start 2023-12-19T09:21:14.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 -- start start 2023-12-19T09:21:14.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e8c0c6cd0 0x7f1e8c06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c06b470 con 0x7f1e8c0c6900 2023-12-19T09:21:14.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c06be40 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9180c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c06e110 con 0x7f1e8c15cc20 2023-12-19T09:21:14.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9080a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1e8c0c6900 0x7f1e8c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46100/0 (socket says 172.21.15.111:46100) 2023-12-19T09:21:14.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9080a700 1 -- 172.21.15.111:0/3767643539 learned_addr learned my addr 172.21.15.111:0/3767643539 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:14.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e83fff700 1 --2- 172.21.15.111:0/3767643539 >> v2:172.21.15.111:3300/0 conn(0x7f1e8c0c6cd0 0x7f1e8c06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.003+0000 7f1e9100b700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312007489 0 0) 0x7f1e8c06b470 con 0x7f1e8c0c6900 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e74003570 con 0x7f1e8c0c6900 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e83fff700 1 -- 172.21.15.111:0/3767643539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 msgr2=0x7f1e8c15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e83fff700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c15d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e83fff700 1 -- 172.21.15.111:0/3767643539 >> v1:172.21.15.111:6789/0 conn(0x7f1e8c0c6900 legacy=0x7f1e8c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395099160 0 0) 0x7f1e74003570 con 0x7f1e8c0c6900 2023-12-19T09:21:14.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e83fff700 1 -- 172.21.15.111:0/3767643539 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c161d10 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e83fff700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c06b900 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0098e0 tx=0x7f1e7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f70a30fba25605 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88001470 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c011420 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e837fe700 1 -- 172.21.15.111:0/3767643539 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c0117c0 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 -- 172.21.15.111:0/3767643539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6cd0 msgr2=0x7f1e8c06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c06b900 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0098e0 tx=0x7f1e7c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 -- 172.21.15.111:0/3767643539 shutdown_connections 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c15d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 --2- 172.21.15.111:0/3767643539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c06b900 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 -- 172.21.15.111:0/3767643539 >> 172.21.15.111:0/3767643539 conn(0x7f1e8c067bb0 msgr2=0x7f1e8c0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 -- 172.21.15.111:0/3767643539 shutdown_connections 2023-12-19T09:21:14.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.007+0000 7f1e9180c700 1 -- 172.21.15.111:0/3767643539 wait complete. 2023-12-19T09:21:14.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 Processor -- start 2023-12-19T09:21:14.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 -- start start 2023-12-19T09:21:14.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c170c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15ecb0 con 0x7f1e8c0c6900 2023-12-19T09:21:14.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15ef90 con 0x7f1e8c0c6cd0 2023-12-19T09:21:14.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9180c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15f270 con 0x7f1e8c15cc20 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9080a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.011+0000 7f1e9080a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60974/0 (socket says 172.21.15.111:60974) 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 -- 172.21.15.111:0/2388957631 learned_addr learned my addr 172.21.15.111:0/2388957631 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9100b700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c170c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e83fff700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 -- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 msgr2=0x7f1e8c170c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c170c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 -- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0c6cd0 msgr2=0x7f1e8c159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 -- 172.21.15.111:0/2388957631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c009420 con 0x7f1e8c0c6900 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9100b700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c170c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:14.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9080a700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800cb90 tx=0x7f1e88001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88007360 con 0x7f1e8c0c6900 2023-12-19T09:21:14.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e880074e0 con 0x7f1e8c0c6900 2023-12-19T09:21:14.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c15f550 con 0x7f1e8c0c6900 2023-12-19T09:21:14.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88004830 con 0x7f1e8c0c6900 2023-12-19T09:21:14.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c171180 con 0x7f1e8c0c6900 2023-12-19T09:21:14.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.015+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c15fa50 con 0x7f1e8c0c6900 2023-12-19T09:21:14.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.019+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1e88004ba0 con 0x7f1e8c0c6900 2023-12-19T09:21:14.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.019+0000 7f1e80ff9700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.019+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f1e8807c400 con 0x7f1e8c0c6900 2023-12-19T09:21:14.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.019+0000 7f1e83fff700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.023+0000 7f1e83fff700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0093f0 tx=0x7f1e7c016000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.023+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1e8804c790 con 0x7f1e8c0c6900 2023-12-19T09:21:14.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.195+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e8c04be40 con 0x7f1e8c0c6900 2023-12-19T09:21:14.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.199+0000 7f1e80ff9700 1 -- 172.21.15.111:0/2388957631 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+14281 (secure 0 0 0) 0x7f1e8804fdb0 con 0x7f1e8c0c6900 2023-12-19T09:21:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:14.206 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":199,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:11.802369+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6835/1020011686":"2023-12-20T09:21:08.642469+0000","172.21.15.111:6834/1020011686":"2023-12-20T09:21:08.642469+0000","172.21.15.143:0/4220694263":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/1847708403":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/4153258983":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/2341319383":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/176361162":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/75677859":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/529911322":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/3701795071":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/481543095":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3982635695":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6833/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:6832/43868":"2023-12-20T09:19:54.022209+0000","172.21.15.143:0/3879504664":"2023-12-20T09:21:08.603827+0000","172.21.15.143:0/3011711537":"2023-12-20T09:19:54.022209+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 msgr2=0x7f1e78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0093f0 tx=0x7f1e7c016000 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 msgr2=0x7f1e8c1480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800cb90 tx=0x7f1e88001b50 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 shutdown_connections 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0c6cd0 0x7f1e8c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c15cc20 0x7f1e8c170c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 --2- 172.21.15.111:0/2388957631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0c6900 0x7f1e8c1480d0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 >> 172.21.15.111:0/2388957631 conn(0x7f1e8c067bb0 msgr2=0x7f1e8c159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:14.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 shutdown_connections 2023-12-19T09:21:14.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.203+0000 7f1e9180c700 1 -- 172.21.15.111:0/2388957631 wait complete. 2023-12-19T09:21:14.227 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:21:14.228 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:21:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 Processor -- start 2023-12-19T09:21:14.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 -- start start 2023-12-19T09:21:14.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0cbdf0 con 0x7fb96c0cb970 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0cc0d0 con 0x7fb96c158e20 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.399+0000 7fb973078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0cc3b0 con 0x7fb96c0cecd0 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb970e14700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb96c158e20 0x7fb96c0ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:53594/0 (socket says 172.21.15.111:53594) 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:51982/0 (socket says 172.21.15.111:51982) 2023-12-19T09:21:14.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb970e14700 1 -- 172.21.15.111:0/119648620 learned_addr learned my addr 172.21.15.111:0/119648620 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb96bfff700 1 --1- 172.21.15.111:0/119648620 >> v1:172.21.15.111:6789/0 conn(0x7fb96c0cb970 0x7fb96c06acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46114/0 (socket says 172.21.15.111:46114) 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb96b7fe700 1 -- 172.21.15.111:0/119648620 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795404837 0 0) 0x7fb96c0cbdf0 con 0x7fb96c0cb970 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 -- 172.21.15.111:0/119648620 >> v1:172.21.15.143:6789/0 conn(0x7fb96c158e20 legacy=0x7fb96c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 -- 172.21.15.111:0/119648620 >> v1:172.21.15.111:6789/0 conn(0x7fb96c0cb970 legacy=0x7fb96c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 -- 172.21.15.111:0/119648620 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c0cc690 con 0x7fb96c0cecd0 2023-12-19T09:21:14.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb971615700 1 --2- 172.21.15.111:0/119648620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb964009ec0 tx=0x7fb964009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31b48e79431a11e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb96b7fe700 1 -- 172.21.15.111:0/119648620 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96400e070 con 0x7fb96c0cecd0 2023-12-19T09:21:14.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb96b7fe700 1 -- 172.21.15.111:0/119648620 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96400bbf0 con 0x7fb96c0cecd0 2023-12-19T09:21:14.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb96b7fe700 1 -- 172.21.15.111:0/119648620 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964012430 con 0x7fb96c0cecd0 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 -- 172.21.15.111:0/119648620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 msgr2=0x7fb96c06b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 --2- 172.21.15.111:0/119648620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb964009ec0 tx=0x7fb964009780 comp rx=0 tx=0).stop 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 -- 172.21.15.111:0/119648620 shutdown_connections 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 --2- 172.21.15.111:0/119648620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c0cecd0 0x7fb96c06b3d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 -- 172.21.15.111:0/119648620 >> 172.21.15.111:0/119648620 conn(0x7fb96c067bb0 msgr2=0x7fb96c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 -- 172.21.15.111:0/119648620 shutdown_connections 2023-12-19T09:21:14.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.403+0000 7fb973078700 1 -- 172.21.15.111:0/119648620 wait complete. 2023-12-19T09:21:14.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 Processor -- start 2023-12-19T09:21:14.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 -- start start 2023-12-19T09:21:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 0x7fb96c1698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c158e20 0x7fb96c14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0c9650 con 0x7fb96c06abc0 2023-12-19T09:21:14.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c06b5a0 con 0x7fb96c0cecd0 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb973078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c169e10 con 0x7fb96c158e20 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb971615700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb970e14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 0x7fb96c1698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c158e20 0x7fb96c14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb971615700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60988/0 (socket says 172.21.15.111:60988) 2023-12-19T09:21:14.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.407+0000 7fb971615700 1 -- 172.21.15.111:0/2668080745 learned_addr learned my addr 172.21.15.111:0/2668080745 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 -- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c158e20 msgr2=0x7fb96c14ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c158e20 0x7fb96c14ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 -- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 msgr2=0x7fb96c1698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 0x7fb96c1698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 -- 172.21.15.111:0/2668080745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964009a20 con 0x7fb96c06abc0 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb970e14700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 0x7fb96c1698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb971615700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb9640099f0 tx=0x7fb964019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96401b420 con 0x7fb96c06abc0 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96401b580 con 0x7fb96c06abc0 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96401b920 con 0x7fb96c06abc0 2023-12-19T09:21:14.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c0c85b0 con 0x7fb96c06abc0 2023-12-19T09:21:14.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c0c8b70 con 0x7fb96c06abc0 2023-12-19T09:21:14.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.411+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c16b1a0 con 0x7fb96c06abc0 2023-12-19T09:21:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.415+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb96401bc90 con 0x7fb96c06abc0 2023-12-19T09:21:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.415+0000 7fb968ff9700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 0x7fb950062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.415+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7fb96407e680 con 0x7fb96c06abc0 2023-12-19T09:21:14.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.415+0000 7fb970e14700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 0x7fb950062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.419+0000 7fb970e14700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 0x7fb950062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb960007130 tx=0x7fb960000bc0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.423+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb96404e8f0 con 0x7fb96c06abc0 2023-12-19T09:21:14.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.599+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb96c06a7a0 con 0x7fb96c06abc0 2023-12-19T09:21:14.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.599+0000 7fb968ff9700 1 -- 172.21.15.111:0/2668080745 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v199) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb964051f10 con 0x7fb96c06abc0 2023-12-19T09:21:14.608 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:21:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 msgr2=0x7fb950062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 0x7fb950062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb960007130 tx=0x7fb960000bc0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 msgr2=0x7fb96c06b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fb9640099f0 tx=0x7fb964019000 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 shutdown_connections 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb96c06abc0 0x7fb96c06b060 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb96c158e20 0x7fb96c14ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb950060550 0x7fb950062a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 --2- 172.21.15.111:0/2668080745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c0cecd0 0x7fb96c1698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 >> 172.21.15.111:0/2668080745 conn(0x7fb96c067bb0 msgr2=0x7fb96c14fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:14.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 shutdown_connections 2023-12-19T09:21:14.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.603+0000 7fb973078700 1 -- 172.21.15.111:0/2668080745 wait complete. 2023-12-19T09:21:14.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-12-19T09:21:14.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.799+0000 7ff62e46e700 1 Processor -- start 2023-12-19T09:21:14.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.799+0000 7ff62e46e700 1 -- start start 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62e46e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff62815d5f0 0x7ff6280c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62e46e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62815da40 con 0x7ff6280c6900 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62e46e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280cc030 con 0x7ff62815d220 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62e46e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280c6d60 con 0x7ff62815d5f0 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff61f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff62815d5f0 0x7ff6280c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62ca0b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff6280c6900 0x7ff628161170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46116/0 (socket says 172.21.15.111:46116) 2023-12-19T09:21:14.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff62ca0b700 1 -- 172.21.15.111:0/4035993415 learned_addr learned my addr 172.21.15.111:0/4035993415 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615821929 0 0) 0x7ff6280cc030 con 0x7ff62815d220 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff614003570 con 0x7ff62815d220 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252852731 0 0) 0x7ff620001490 con 0x7ff6280c6900 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280cc030 con 0x7ff6280c6900 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff61f7fe700 1 -- 172.21.15.111:0/4035993415 >> v1:172.21.15.143:6789/0 conn(0x7ff62815d220 legacy=0x7ff6280cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff61f7fe700 1 -- 172.21.15.111:0/4035993415 >> v1:172.21.15.111:6789/0 conn(0x7ff6280c6900 legacy=0x7ff628161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff61f7fe700 1 -- 172.21.15.111:0/4035993415 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280cb420 con 0x7ff62815d5f0 2023-12-19T09:21:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff61f7fe700 1 --2- 172.21.15.111:0/4035993415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d5f0 0x7ff6280c63c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff6100098e0 tx=0x7ff6100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a3ab12b7db89e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610011630 con 0x7ff62815d5f0 2023-12-19T09:21:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610011790 con 0x7ff62815d5f0 2023-12-19T09:21:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.803+0000 7ff6277fe700 1 -- 172.21.15.111:0/4035993415 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610011b30 con 0x7ff62815d5f0 2023-12-19T09:21:14.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- 172.21.15.111:0/4035993415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d5f0 msgr2=0x7ff6280c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 --2- 172.21.15.111:0/4035993415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d5f0 0x7ff6280c63c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff6100098e0 tx=0x7ff6100091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- 172.21.15.111:0/4035993415 shutdown_connections 2023-12-19T09:21:14.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 --2- 172.21.15.111:0/4035993415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d5f0 0x7ff6280c63c0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- 172.21.15.111:0/4035993415 >> 172.21.15.111:0/4035993415 conn(0x7ff6280cd6d0 msgr2=0x7ff62806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- 172.21.15.111:0/4035993415 shutdown_connections 2023-12-19T09:21:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- 172.21.15.111:0/4035993415 wait complete. 2023-12-19T09:21:14.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 Processor -- start 2023-12-19T09:21:14.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.807+0000 7ff62e46e700 1 -- start start 2023-12-19T09:21:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 0x7ff628159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 0x7ff62816da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280b2fb0 con 0x7ff62815d220 2023-12-19T09:21:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280b3250 con 0x7ff6280c6900 2023-12-19T09:21:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280b3530 con 0x7ff62815d5f0 2023-12-19T09:21:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32782/0 (socket says 172.21.15.111:32782) 2023-12-19T09:21:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 -- 172.21.15.111:0/920991737 learned_addr learned my addr 172.21.15.111:0/920991737 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62ca0b700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 0x7ff62816da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 -- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 msgr2=0x7ff62816da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 0x7ff62816da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff627fff700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 0x7ff628159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 -- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 msgr2=0x7ff628159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 0x7ff628159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 -- 172.21.15.111:0/920991737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610009420 con 0x7ff62815d220 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62ca0b700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 0x7ff62816da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff627fff700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 0x7ff628159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff61f7fe700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff62806a0f0 tx=0x7ff610017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61001f070 con 0x7ff62815d220 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610000bc0 con 0x7ff62815d220 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6280b3810 con 0x7ff62815d220 2023-12-19T09:21:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610025420 con 0x7ff62815d220 2023-12-19T09:21:14.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.811+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6280b57a0 con 0x7ff62815d220 2023-12-19T09:21:14.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.815+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628071120 con 0x7ff62815d220 2023-12-19T09:21:14.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.815+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff610023070 con 0x7ff62815d220 2023-12-19T09:21:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.815+0000 7ff624ff9700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.815+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7ff61007ddb0 con 0x7ff62815d220 2023-12-19T09:21:14.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.815+0000 7ff627fff700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:14.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.819+0000 7ff627fff700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 0x7ff608062a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff628159e20 tx=0x7ff618009040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:14.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:14.823+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff61004e0c0 con 0x7ff62815d220 2023-12-19T09:21:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.051+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7ff6280642a0 con 0x7ff62815d220 2023-12-19T09:21:15.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.843+0000 7ff624ff9700 1 -- 172.21.15.111:0/920991737 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3041) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff61002a0e0 con 0x7ff62815d220 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 msgr2=0x7ff608062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 0x7ff608062a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff628159e20 tx=0x7ff618009040 comp rx=0 tx=0).stop 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 msgr2=0x7ff6280b5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff62806a0f0 tx=0x7ff610017000 comp rx=0 tx=0).stop 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 shutdown_connections 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff62815d5f0 0x7ff62816da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62815d220 0x7ff6280b5260 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff608060550 0x7ff608062a00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 --2- 172.21.15.111:0/920991737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6280c6900 0x7ff628159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 >> 172.21.15.111:0/920991737 conn(0x7ff6280cd6d0 msgr2=0x7ff628159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 shutdown_connections 2023-12-19T09:21:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:15.847+0000 7ff62e46e700 1 -- 172.21.15.111:0/920991737 wait complete. 2023-12-19T09:21:15.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-19T09:21:16.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 Processor -- start 2023-12-19T09:21:16.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 -- start start 2023-12-19T09:21:16.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 0x7f2b1415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b140adab0 0x7f2b140adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1415ae30 con 0x7f2b1406baa0 2023-12-19T09:21:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140abf70 con 0x7f2b140adab0 2023-12-19T09:21:16.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.039+0000 7f2b1904c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14154b50 con 0x7f2b14154780 2023-12-19T09:21:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 0x7f2b1415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b140adab0 0x7f2b140adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 0x7f2b1415b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52000/0 (socket says 172.21.15.111:52000) 2023-12-19T09:21:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b13fff700 1 -- 172.21.15.111:0/209076337 learned_addr learned my addr 172.21.15.111:0/209076337 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:16.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b137fe700 1 --1- 172.21.15.111:0/209076337 >> v1:172.21.15.143:6789/0 conn(0x7f2b1406baa0 0x7f2b14157b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:53608/0 (socket says 172.21.15.111:53608) 2023-12-19T09:21:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 -- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 msgr2=0x7f2b1415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 --2- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 0x7f2b1415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 -- 172.21.15.111:0/209076337 >> v1:172.21.15.143:6789/0 conn(0x7f2b1406baa0 legacy=0x7f2b14157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 -- 172.21.15.111:0/209076337 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140ac210 con 0x7f2b140adab0 2023-12-19T09:21:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1884b700 1 --2- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b140adab0 0x7f2b140adf10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009e60 tx=0x7f2b0c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a63d68cf7a7f18b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b12ffd700 1 -- 172.21.15.111:0/209076337 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c00e070 con 0x7f2b140adab0 2023-12-19T09:21:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b12ffd700 1 -- 172.21.15.111:0/209076337 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c00bb60 con 0x7f2b140adab0 2023-12-19T09:21:16.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b12ffd700 1 -- 172.21.15.111:0/209076337 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c012430 con 0x7f2b140adab0 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 -- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b140adab0 msgr2=0x7f2b140adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 --2- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b140adab0 0x7f2b140adf10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009e60 tx=0x7f2b0c009780 comp rx=0 tx=0).stop 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 -- 172.21.15.111:0/209076337 shutdown_connections 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 --2- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b140adab0 0x7f2b140adf10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 --2- 172.21.15.111:0/209076337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b14154780 0x7f2b1415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 -- 172.21.15.111:0/209076337 >> 172.21.15.111:0/209076337 conn(0x7f2b14067b50 msgr2=0x7f2b140cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 -- 172.21.15.111:0/209076337 shutdown_connections 2023-12-19T09:21:16.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.043+0000 7f2b1904c700 1 -- 172.21.15.111:0/209076337 wait complete. 2023-12-19T09:21:16.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.047+0000 7f2b1904c700 1 Processor -- start 2023-12-19T09:21:16.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.047+0000 7f2b1904c700 1 -- start start 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b140adab0 0x7f2b140c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 0x7f2b140c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141415a0 con 0x7f2b1406baa0 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141522e0 con 0x7f2b14154780 2023-12-19T09:21:16.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140c8d60 con 0x7f2b140adab0 2023-12-19T09:21:16.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32792/0 (socket says 172.21.15.111:32792) 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 -- 172.21.15.111:0/438549962 learned_addr learned my addr 172.21.15.111:0/438549962 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1884b700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 0x7f2b140c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b137fe700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b140adab0 0x7f2b140c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 -- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b140adab0 msgr2=0x7f2b140c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b140adab0 0x7f2b140c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 -- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 msgr2=0x7f2b140c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 0x7f2b140c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 -- 172.21.15.111:0/438549962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c009a20 con 0x7f2b1406baa0 2023-12-19T09:21:16.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1884b700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 0x7f2b140c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b13fff700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e980 tx=0x7f2b04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04013680 con 0x7f2b1406baa0 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b040137e0 con 0x7f2b1406baa0 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b140c9040 con 0x7f2b1406baa0 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04013b60 con 0x7f2b1406baa0 2023-12-19T09:21:16.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.051+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14151f70 con 0x7f2b1406baa0 2023-12-19T09:21:16.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.055+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1415a720 con 0x7f2b1406baa0 2023-12-19T09:21:16.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.055+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2b0400c400 con 0x7f2b1406baa0 2023-12-19T09:21:16.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.055+0000 7f2afbfff700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 0x7f2b00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.059+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f2b0407cb10 con 0x7f2b1406baa0 2023-12-19T09:21:16.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.059+0000 7f2b137fe700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 0x7f2b00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.059+0000 7f2b137fe700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 0x7f2b00062a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b1406b920 tx=0x7f2b08007eb0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.063+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2b040508b0 con 0x7f2b1406baa0 2023-12-19T09:21:16.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.227+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2b1404be40 con 0x7f2b1406baa0 2023-12-19T09:21:16.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.239+0000 7f2afbfff700 1 -- 172.21.15.111:0/438549962 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2b040170c0 con 0x7f2b1406baa0 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 msgr2=0x7f2b00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 0x7f2b00062a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b1406b920 tx=0x7f2b08007eb0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 msgr2=0x7f2b14141060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e980 tx=0x7f2b04009250 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 shutdown_connections 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b00060550 0x7f2b00062a00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b14154780 0x7f2b140c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b140adab0 0x7f2b140c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 --2- 172.21.15.111:0/438549962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1406baa0 0x7f2b14141060 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 >> 172.21.15.111:0/438549962 conn(0x7f2b14067b50 msgr2=0x7f2b14151260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 shutdown_connections 2023-12-19T09:21:16.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.243+0000 7f2b1904c700 1 -- 172.21.15.111:0/438549962 wait complete. 2023-12-19T09:21:16.271 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:21:16.271 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:16.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:16.359 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:21:16.359 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-19T09:21:16.359 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:16.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.515+0000 7f6e042cd700 1 Processor -- start 2023-12-19T09:21:16.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.515+0000 7f6e042cd700 1 -- start start 2023-12-19T09:21:16.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e042cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6dfc140710 0x7f6dfc0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e042cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6dfc140ae0 0x7f6dfc0680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e042cd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc06e150 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e042cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc06d540 con 0x7f6dfc140ae0 2023-12-19T09:21:16.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e042cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc06d820 con 0x7f6dfc140710 2023-12-19T09:21:16.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6dfc140ae0 0x7f6dfc0680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6dfc140ae0 0x7f6dfc0680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32794/0 (socket says 172.21.15.111:32794) 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 -- 172.21.15.111:0/7454397 learned_addr learned my addr 172.21.15.111:0/7454397 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e02069700 1 --2- 172.21.15.111:0/7454397 >> v2:172.21.15.143:3300/0 conn(0x7f6dfc140710 0x7f6dfc0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 -- 172.21.15.111:0/7454397 >> v2:172.21.15.143:3300/0 conn(0x7f6dfc140710 msgr2=0x7f6dfc0c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 --2- 172.21.15.111:0/7454397 >> v2:172.21.15.143:3300/0 conn(0x7f6dfc140710 0x7f6dfc0c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 -- 172.21.15.111:0/7454397 >> v1:172.21.15.143:6789/0 conn(0x7f6dfc0c75e0 legacy=0x7f6dfc06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:16.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 -- 172.21.15.111:0/7454397 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc06db00 con 0x7f6dfc140ae0 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01868700 1 --2- 172.21.15.111:0/7454397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc140ae0 0x7f6dfc0680f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc068c30 tx=0x7f6dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117fc0956e7a7f51 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01067700 1 -- 172.21.15.111:0/7454397 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00b860 con 0x7f6dfc140ae0 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01067700 1 -- 172.21.15.111:0/7454397 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec00b9c0 con 0x7f6dfc140ae0 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.519+0000 7f6e01067700 1 -- 172.21.15.111:0/7454397 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00bd60 con 0x7f6dfc140ae0 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- 172.21.15.111:0/7454397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc140ae0 msgr2=0x7f6dfc0680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 --2- 172.21.15.111:0/7454397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc140ae0 0x7f6dfc0680f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc068c30 tx=0x7f6dec009250 comp rx=0 tx=0).stop 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- 172.21.15.111:0/7454397 shutdown_connections 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 --2- 172.21.15.111:0/7454397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc140ae0 0x7f6dfc0680f0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 --2- 172.21.15.111:0/7454397 >> v2:172.21.15.143:3300/0 conn(0x7f6dfc140710 0x7f6dfc0c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- 172.21.15.111:0/7454397 >> 172.21.15.111:0/7454397 conn(0x7f6dfc0c8850 msgr2=0x7f6dfc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- 172.21.15.111:0/7454397 shutdown_connections 2023-12-19T09:21:16.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- 172.21.15.111:0/7454397 wait complete. 2023-12-19T09:21:16.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 Processor -- start 2023-12-19T09:21:16.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.523+0000 7f6e042cd700 1 -- start start 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dfc140710 0x7f6dfc0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 0x7f6dfc0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0c5be0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0ab080 con 0x7f6dfc140710 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0c2e90 con 0x7f6dfc140ae0 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32808/0 (socket says 172.21.15.111:32808) 2023-12-19T09:21:16.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 -- 172.21.15.111:0/1054202878 learned_addr learned my addr 172.21.15.111:0/1054202878 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e01868700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dfc140710 0x7f6dfc0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e0286a700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 0x7f6dfc0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 -- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 msgr2=0x7f6dfc0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 0x7f6dfc0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 -- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dfc140710 msgr2=0x7f6dfc0c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dfc140710 0x7f6dfc0c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 -- 172.21.15.111:0/1054202878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0094f0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e0286a700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 0x7f6dfc0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e02069700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e190 tx=0x7f6df8009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800ee30 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8014420 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc0c3170 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df80147e0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.527+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc0c3730 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.531+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc0c3d10 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.531+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6df8014b90 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.531+0000 7f6df27fc700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 0x7f6de0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:16.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.531+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f6df807d2b0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.535+0000 7f6e01868700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 0x7f6de0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:16.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.535+0000 7f6e01868700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 0x7f6de0062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc068c30 tx=0x7f6dec017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:16.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.539+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6df804d5c0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.747+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dfc071120 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.747+0000 7f6df27fc700 1 -- 172.21.15.111:0/1054202878 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3542 (secure 0 0 0) 0x7f6df8050be0 con 0x7f6dfc0c75e0 2023-12-19T09:21:16.753 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:16.753 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":89,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8435,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/4205671591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":4205671591},{"type":"v1","addr":"172.21.15.111:6833","nonce":4205671591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9640,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1209971783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1209971783},{"type":"v1","addr":"172.21.15.143:6835","nonce":1209971783}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":9646,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/2665966224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":2665966224},{"type":"v1","addr":"172.21.15.143:6837","nonce":2665966224}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83},{"gid":10341,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1792765774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1792765774},{"type":"v1","addr":"172.21.15.111:6835","nonce":1792765774}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[]} 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 msgr2=0x7f6de0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 0x7f6de0062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc068c30 tx=0x7f6dec017000 comp rx=0 tx=0).stop 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 msgr2=0x7f6dfc156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e190 tx=0x7f6df8009920 comp rx=0 tx=0).stop 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 shutdown_connections 2023-12-19T09:21:16.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6de0060550 0x7f6de0062a00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dfc140710 0x7f6dfc0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dfc140ae0 0x7f6dfc0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 --2- 172.21.15.111:0/1054202878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc0c75e0 0x7f6dfc156b20 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 >> 172.21.15.111:0/1054202878 conn(0x7f6dfc0c8850 msgr2=0x7f6dfc0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 shutdown_connections 2023-12-19T09:21:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:16.751+0000 7f6e042cd700 1 -- 172.21.15.111:0/1054202878 wait complete. 2023-12-19T09:21:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 89 2023-12-19T09:21:16.780 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-19T09:21:16.780 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-19T09:21:16.781 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T09:21:16.781 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:21:16.781 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-19T09:21:16.781 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-19T09:21:16.781 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T09:21:16.781 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:21:16.781 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-19T09:21:16.781 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-19T09:21:16.782 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T09:21:16.782 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:21:16.782 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-19T09:21:16.782 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-19T09:21:16.782 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T09:21:16.782 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:21:16.800 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T09:21:16.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-19T09:21:16.801 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T09:21:16.801 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-19T09:21:16.804 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T09:21:16.804 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-19T09:21:16.806 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T09:21:16.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-19T09:21:16.854 INFO:tasks.ceph.mds.b:Started 2023-12-19T09:21:16.854 DEBUG:teuthology.parallel:result is None 2023-12-19T09:21:16.855 INFO:tasks.ceph.mds.a:Started 2023-12-19T09:21:16.855 DEBUG:teuthology.parallel:result is None 2023-12-19T09:21:16.856 INFO:tasks.ceph.mds.c:Started 2023-12-19T09:21:16.857 DEBUG:teuthology.parallel:result is None 2023-12-19T09:21:16.857 INFO:tasks.ceph.mds.d:Started 2023-12-19T09:21:16.857 DEBUG:teuthology.parallel:result is None 2023-12-19T09:21:16.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-19T09:21:16.964 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-12-19T09:21:16.965 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-12-19T09:21:16.969 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-12-19T09:21:16.972 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:21:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 Processor -- start 2023-12-19T09:21:17.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 -- start start 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb98815b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb98806c7c0 0x7fb9880b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98806d7d0 con 0x7fb9881407f0 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9880ae280 con 0x7fb98806c7c0 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98ccf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880cb8f0 con 0x7fb988140420 2023-12-19T09:21:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb98659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb98815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb985d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb98806c7c0 0x7fb9880b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb986d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb9881407f0 0x7fb98806a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:53644/0 (socket says 172.21.15.111:53644) 2023-12-19T09:21:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.011+0000 7fb986d9d700 1 -- 172.21.15.111:0/1330678895 learned_addr learned my addr 172.21.15.111:0/1330678895 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb985d9b700 1 -- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 msgr2=0x7fb98815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb985d9b700 1 --2- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb98815b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb985d9b700 1 -- 172.21.15.111:0/1330678895 >> v1:172.21.15.143:6789/0 conn(0x7fb9881407f0 legacy=0x7fb98806a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb985d9b700 1 -- 172.21.15.111:0/1330678895 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98806cc50 con 0x7fb98806c7c0 2023-12-19T09:21:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98559a700 1 -- 172.21.15.111:0/1330678895 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672191146 0 0) 0x7fb98806d7d0 con 0x7fb9881407f0 2023-12-19T09:21:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb985d9b700 1 --2- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb9880b0cd0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0098e0 tx=0x7fb97c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=611f0f32f7c2e7f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98559a700 1 -- 172.21.15.111:0/1330678895 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00b640 con 0x7fb98806c7c0 2023-12-19T09:21:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98559a700 1 -- 172.21.15.111:0/1330678895 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00b7a0 con 0x7fb98806c7c0 2023-12-19T09:21:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98559a700 1 -- 172.21.15.111:0/1330678895 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00bb40 con 0x7fb98806c7c0 2023-12-19T09:21:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 msgr2=0x7fb9880b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb9880b0cd0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0098e0 tx=0x7fb97c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1330678895 shutdown_connections 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb9880b0cd0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1330678895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb98815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1330678895 >> 172.21.15.111:0/1330678895 conn(0x7fb988067b60 msgr2=0x7fb9880ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1330678895 shutdown_connections 2023-12-19T09:21:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.015+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1330678895 wait complete. 2023-12-19T09:21:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 Processor -- start 2023-12-19T09:21:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 -- start start 2023-12-19T09:21:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb988152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9881407f0 0x7fb988158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988138070 con 0x7fb9881407f0 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988138350 con 0x7fb98806c7c0 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98ccf0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb988138630 con 0x7fb988140420 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb986d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9881407f0 0x7fb988158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb988152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb988152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:38344/0 (socket says 172.21.15.111:38344) 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb98659c700 1 -- 172.21.15.111:0/1896114777 learned_addr learned my addr 172.21.15.111:0/1896114777 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:52056/0 (socket says 172.21.15.111:52056) 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 -- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 msgr2=0x7fb988152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:17.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb988152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 -- 172.21.15.111:0/1896114777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9881407f0 msgr2=0x7fb988158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:17.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9881407f0 0x7fb988158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:17.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.019+0000 7fb985d9b700 1 -- 172.21.15.111:0/1896114777 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c009420 con 0x7fb988140420 2023-12-19T09:21:17.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb985d9b700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb97c016000 tx=0x7fb97c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:17.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00bb20 con 0x7fb988140420 2023-12-19T09:21:17.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c000c90 con 0x7fb988140420 2023-12-19T09:21:17.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c011630 con 0x7fb988140420 2023-12-19T09:21:17.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb988138910 con 0x7fb988140420 2023-12-19T09:21:17.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb988152ed0 con 0x7fb988140420 2023-12-19T09:21:17.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.023+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb988138e20 con 0x7fb988140420 2023-12-19T09:21:17.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.027+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb97c0119a0 con 0x7fb988140420 2023-12-19T09:21:17.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.027+0000 7fb9767fc700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 0x7fb968062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:17.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.027+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6437+0+0 (secure 0 0 0) 0x7fb97c07dc80 con 0x7fb988140420 2023-12-19T09:21:17.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.027+0000 7fb98659c700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 0x7fb968062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:17.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.027+0000 7fb98659c700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 0x7fb968062ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e70 tx=0x7fb9780091a0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:17.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.035+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb97c04df70 con 0x7fb988140420 2023-12-19T09:21:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:17.219+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb988071120 con 0x7fb988140420 2023-12-19T09:21:18.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.271+0000 7fb9767fc700 1 -- 172.21.15.111:0/1896114777 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v204) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb97c051590 con 0x7fb988140420 2023-12-19T09:21:18.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 msgr2=0x7fb968062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 0x7fb968062ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e70 tx=0x7fb9780091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 msgr2=0x7fb988154110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb97c016000 tx=0x7fb97c017000 comp rx=0 tx=0).stop 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 shutdown_connections 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9881407f0 0x7fb988158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb988140420 0x7fb988154110 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb968060620 0x7fb968062ad0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 --2- 172.21.15.111:0/1896114777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb98806c7c0 0x7fb988152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 >> 172.21.15.111:0/1896114777 conn(0x7fb988067b60 msgr2=0x7fb98813b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:18.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 shutdown_connections 2023-12-19T09:21:18.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.275+0000 7fb98ccf0700 1 -- 172.21.15.111:0/1896114777 wait complete. 2023-12-19T09:21:18.286 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-12-19T09:21:18.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-19T09:21:18.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 Processor -- start 2023-12-19T09:21:18.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 -- start start 2023-12-19T09:21:18.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6cd0 0x7fb88406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb88415cc20 0x7fb88415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88406b470 con 0x7fb8840c6900 2023-12-19T09:21:18.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88406be40 con 0x7fb88415cc20 2023-12-19T09:21:18.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.463+0000 7fb88b0e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88406e110 con 0x7fb8840c6cd0 2023-12-19T09:21:18.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb88415cc20 0x7fb88415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb88415cc20 0x7fb88415d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32846/0 (socket says 172.21.15.111:32846) 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 -- 172.21.15.111:0/3257878975 learned_addr learned my addr 172.21.15.111:0/3257878975 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb8898e0700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6cd0 0x7fb88406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 -- 172.21.15.111:0/3257878975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6cd0 msgr2=0x7fb88406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6cd0 0x7fb88406b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 -- 172.21.15.111:0/3257878975 >> v1:172.21.15.143:6789/0 conn(0x7fb8840c6900 legacy=0x7fb88406da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 -- 172.21.15.111:0/3257878975 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb884161d10 con 0x7fb88415cc20 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88a8e2700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415d0a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009f50 tx=0x7fb87c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddeee927687b734 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb8890df700 1 -- 172.21.15.111:0/3257878975 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00e070 con 0x7fb88415cc20 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb8890df700 1 -- 172.21.15.111:0/3257878975 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c00ba70 con 0x7fb88415cc20 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb8890df700 1 -- 172.21.15.111:0/3257878975 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0092e0 con 0x7fb88415cc20 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 -- 172.21.15.111:0/3257878975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 msgr2=0x7fb88415d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415d0a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009f50 tx=0x7fb87c009840 comp rx=0 tx=0).stop 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 -- 172.21.15.111:0/3257878975 shutdown_connections 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415d0a0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/3257878975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6cd0 0x7fb88406b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 -- 172.21.15.111:0/3257878975 >> 172.21.15.111:0/3257878975 conn(0x7fb884067bb0 msgr2=0x7fb8840cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 -- 172.21.15.111:0/3257878975 shutdown_connections 2023-12-19T09:21:18.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.467+0000 7fb88b0e3700 1 -- 172.21.15.111:0/3257878975 wait complete. 2023-12-19T09:21:18.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 Processor -- start 2023-12-19T09:21:18.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 -- start start 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 0x7fb884149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8840c6cd0 0x7fb8840af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840b0200 con 0x7fb88415cc20 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840afc40 con 0x7fb8840c6cd0 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88b0e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840aff20 con 0x7fb8840c6900 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88a8e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88a8e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32860/0 (socket says 172.21.15.111:32860) 2023-12-19T09:21:18.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88a8e2700 1 -- 172.21.15.111:0/2241259262 learned_addr learned my addr 172.21.15.111:0/2241259262 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb8898e0700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8840c6cd0 0x7fb8840af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.471+0000 7fb88a0e1700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 0x7fb884149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 -- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 msgr2=0x7fb884149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 0x7fb884149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 -- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8840c6cd0 msgr2=0x7fb8840af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8840c6cd0 0x7fb8840af2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 -- 172.21.15.111:0/2241259262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009ae0 con 0x7fb88415cc20 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a0e1700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 0x7fb884149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88a8e2700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb884064040 tx=0x7fb87c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0032f0 con 0x7fb88415cc20 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c008e40 con 0x7fb88415cc20 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88415d0c0 con 0x7fb88415cc20 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c025690 con 0x7fb88415cc20 2023-12-19T09:21:18.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88414a010 con 0x7fb88415cc20 2023-12-19T09:21:18.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.475+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88404be40 con 0x7fb88415cc20 2023-12-19T09:21:18.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.479+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb87c003040 con 0x7fb88415cc20 2023-12-19T09:21:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.479+0000 7fb87a7fc700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 0x7fb868062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.479+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fb87c07e580 con 0x7fb88415cc20 2023-12-19T09:21:18.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.479+0000 7fb88a0e1700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 0x7fb868062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.479+0000 7fb88a0e1700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 0x7fb868062a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb880001840 tx=0x7fb880009040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.487+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb87c04eaa0 con 0x7fb88415cc20 2023-12-19T09:21:18.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.719+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb88406a580 con 0x7fb88415cc20 2023-12-19T09:21:18.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.719+0000 7fb87a7fc700 1 -- 172.21.15.111:0/2241259262 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb87c0520c0 con 0x7fb88415cc20 2023-12-19T09:21:18.725 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlrzPbEhAAGWuALfzc9sobws0DZHOZQQ==", 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.726 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlYjwLJRAA0SoQEJSDy1V9rdKqtYusaw==", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlUtohHRAAwQd0mP+RmEhqIDd/LNwqhw==", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.727 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlC7huLxAASkLbLlYlrdluaJPOf6HAcg==", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:21:18.728 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBCSoFl7wPsGxAAO0eMnJGktwK4+Ldt0Nhl8Q==", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBFSoFlM896KRAAXpM++S/eVqgCcjLQgNXvqQ==", 2023-12-19T09:21:18.729 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBISoFlqsvoNhAAohFUwlOWAgl74MZ7B4XZeQ==", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBMSoFlXO8tChAAZDu4xZYEqeZvWsmcTAwrcw==", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBSSoFlfqcsJRAAb1r6qncVoLGXbb7tRpMbxQ==", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.731 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBVSoFl7FIWMhAACrzuwgqdKQZ5O570sxPCnQ==", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBZSoFlmzBvAhAAM2ZBQO8GsszzkUVN9oQqMQ==", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.732 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQBcSoFlY/dTDxAAnd6ChuMJqWPkRNT4SnjZ0Q==", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-12-19T09:21:18.733 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlJsFSNRAAZw0J78Q6IFz33YAbxEbXjA==", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFlfuwBBRAAgAjR+1FS8bkqQaCbuAA8Mg==", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlkMzaOhAA/m+Ww4dtvRqji98EdpPkCQ==", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA/SoFl+VRIChAAjgvk7OsN+XdhDWE2xVHT4A==", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-12-19T09:21:18.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA9SoFl0HXUJxAAbPjetFF4IZULX6+s0UAMTQ==", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.736 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFlosI7DRAAe51G8ZSucgowrrY26x9hDw==", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQA+SoFldICHBxAA0wabpBMRwFFnfY7tU39Btw==", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-12-19T09:21:18.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:18.738 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-12-19T09:21:18.738 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 msgr2=0x7fb868062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 0x7fb868062a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb880001840 tx=0x7fb880009040 comp rx=0 tx=0).stop 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 msgr2=0x7fb88415f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb884064040 tx=0x7fb87c018000 comp rx=0 tx=0).stop 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 shutdown_connections 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb868060580 0x7fb868062a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8840c6cd0 0x7fb8840af2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb88415cc20 0x7fb88415f240 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 --2- 172.21.15.111:0/2241259262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8840c6900 0x7fb884149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 >> 172.21.15.111:0/2241259262 conn(0x7fb884067bb0 msgr2=0x7fb8840abb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:18.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 shutdown_connections 2023-12-19T09:21:18.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.723+0000 7fb88b0e3700 1 -- 172.21.15.111:0/2241259262 wait complete. 2023-12-19T09:21:18.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:18.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.899+0000 7f94914f0700 1 Processor -- start 2023-12-19T09:21:18.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 -- start start 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c1557e0 0x7f948c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f948c155bb0 0x7f948c0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c159d20 con 0x7f948c06dcb0 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c0c81f0 con 0x7f948c155bb0 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f94914f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c0c7f10 con 0x7f948c1557e0 2023-12-19T09:21:18.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f948b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f948c155bb0 0x7f948c0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f948b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f948c155bb0 0x7f948c0cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32870/0 (socket says 172.21.15.111:32870) 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.903+0000 7f948b7fe700 1 -- 172.21.15.111:0/1215428483 learned_addr learned my addr 172.21.15.111:0/1215428483 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948affd700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c1557e0 0x7f948c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948b7fe700 1 -- 172.21.15.111:0/1215428483 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c1557e0 msgr2=0x7f948c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948b7fe700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c1557e0 0x7f948c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948b7fe700 1 -- 172.21.15.111:0/1215428483 >> v1:172.21.15.111:6789/0 conn(0x7f948c06dcb0 legacy=0x7f948c0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948b7fe700 1 -- 172.21.15.111:0/1215428483 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c1596d0 con 0x7f948c155bb0 2023-12-19T09:21:18.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f948b7fe700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c155bb0 0x7f948c0cad60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f9484009f20 tx=0x7f9484009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c2f9d0be35a074 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f9489ffb700 1 -- 172.21.15.111:0/1215428483 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948400e070 con 0x7f948c155bb0 2023-12-19T09:21:18.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f9489ffb700 1 -- 172.21.15.111:0/1215428483 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948400b9d0 con 0x7f948c155bb0 2023-12-19T09:21:18.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f9489ffb700 1 -- 172.21.15.111:0/1215428483 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840092e0 con 0x7f948c155bb0 2023-12-19T09:21:18.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 -- 172.21.15.111:0/1215428483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c155bb0 msgr2=0x7f948c0cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c155bb0 0x7f948c0cad60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f9484009f20 tx=0x7f9484009840 comp rx=0 tx=0).stop 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 -- 172.21.15.111:0/1215428483 shutdown_connections 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c155bb0 0x7f948c0cad60 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 --2- 172.21.15.111:0/1215428483 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c1557e0 0x7f948c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 -- 172.21.15.111:0/1215428483 >> 172.21.15.111:0/1215428483 conn(0x7f948c067bb0 msgr2=0x7f948c0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 -- 172.21.15.111:0/1215428483 shutdown_connections 2023-12-19T09:21:18.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.907+0000 7f94914f0700 1 -- 172.21.15.111:0/1215428483 wait complete. 2023-12-19T09:21:18.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 Processor -- start 2023-12-19T09:21:18.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 -- start start 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f948c1557e0 0x7f948c0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c155bb0 0x7f948c1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c0c3030 con 0x7f948c06dcb0 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c0c3310 con 0x7f948c1557e0 2023-12-19T09:21:18.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f94914f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c0c35f0 con 0x7f948c155bb0 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f948affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f948affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32878/0 (socket says 172.21.15.111:32878) 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.911+0000 7f948affd700 1 -- 172.21.15.111:0/409260827 learned_addr learned my addr 172.21.15.111:0/409260827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948a7fc700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f948c1557e0 0x7f948c0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948b7fe700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c155bb0 0x7f948c1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 -- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c155bb0 msgr2=0x7f948c1483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c155bb0 0x7f948c1483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 -- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f948c1557e0 msgr2=0x7f948c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f948c1557e0 0x7f948c0b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 -- 172.21.15.111:0/409260827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9484009ae0 con 0x7f948c06dcb0 2023-12-19T09:21:18.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f948affd700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f947c00e930 tx=0x7f947c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c013680 con 0x7f948c06dcb0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947c0137e0 con 0x7f948c06dcb0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c0c38d0 con 0x7f948c06dcb0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c013b80 con 0x7f948c06dcb0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c0b1490 con 0x7f948c06dcb0 2023-12-19T09:21:18.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.915+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c0c3d50 con 0x7f948c06dcb0 2023-12-19T09:21:18.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.919+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f947c00c3e0 con 0x7f948c06dcb0 2023-12-19T09:21:18.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.919+0000 7f9472ffd700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 0x7f9478062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:18.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.919+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f947c07c890 con 0x7f948c06dcb0 2023-12-19T09:21:18.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.919+0000 7f948a7fc700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 0x7f9478062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:18.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.919+0000 7f948a7fc700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 0x7f9478062a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c5bc0 tx=0x7f9480007ec0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:18.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:18.927+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f947c04ce50 con 0x7f948c06dcb0 2023-12-19T09:21:19.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f948c04be40 con 0x7f948c06dcb0 2023-12-19T09:21:19.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f9472ffd700 1 -- 172.21.15.111:0/409260827 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+13277 (secure 0 0 0) 0x7f947c017090 con 0x7f948c06dcb0 2023-12-19T09:21:19.097 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:19.097 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":204,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:18.266795+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 msgr2=0x7f9478062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 0x7f9478062a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c5bc0 tx=0x7f9480007ec0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 msgr2=0x7f948c0c2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.091+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f947c00e930 tx=0x7f947c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 shutdown_connections 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9478060550 0x7f9478062a00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f948c1557e0 0x7f948c0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f948c155bb0 0x7f948c1483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 --2- 172.21.15.111:0/409260827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f948c06dcb0 0x7f948c0c2a50 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 >> 172.21.15.111:0/409260827 conn(0x7f948c067bb0 msgr2=0x7f948c14a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 shutdown_connections 2023-12-19T09:21:19.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.095+0000 7f94914f0700 1 -- 172.21.15.111:0/409260827 wait complete. 2023-12-19T09:21:19.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T09:21:19.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:21:19.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.275+0000 7ff98eee3700 1 Processor -- start 2023-12-19T09:21:19.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.275+0000 7ff98eee3700 1 -- start start 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98eee3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff988158c90 0x7ff9880caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98eee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff98806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98eee3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98806c850 con 0x7ff98806aeb0 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98eee3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98806b470 con 0x7ff988158c90 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98eee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806b750 con 0x7ff988159770 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98d6e0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff98806aeb0 0x7ff988159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46154/0 (socket says 172.21.15.111:46154) 2023-12-19T09:21:19.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98d6e0700 1 -- 172.21.15.111:0/3786707461 learned_addr learned my addr 172.21.15.111:0/3786707461 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:19.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3786707461 >> v2:172.21.15.143:3300/0 conn(0x7ff988158c90 0x7ff9880caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98e6e2700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff98806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324030258 0 0) 0x7ff98806c850 con 0x7ff98806aeb0 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff974003570 con 0x7ff98806aeb0 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.279+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444924252 0 0) 0x7ff974003570 con 0x7ff98806aeb0 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98dee1700 1 -- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 msgr2=0x7ff98806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff98806cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98dee1700 1 -- 172.21.15.111:0/3786707461 >> v1:172.21.15.111:6789/0 conn(0x7ff98806aeb0 legacy=0x7ff988159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98dee1700 1 -- 172.21.15.111:0/3786707461 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98806ba30 con 0x7ff988158c90 2023-12-19T09:21:19.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff988158c90 0x7ff9880caf40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff9840098e0 tx=0x7ff9840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=441deb64801bc505 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978001470 con 0x7ff988158c90 2023-12-19T09:21:19.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984011420 con 0x7ff988158c90 2023-12-19T09:21:19.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98cedf700 1 -- 172.21.15.111:0/3786707461 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840117c0 con 0x7ff988158c90 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 -- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff988158c90 msgr2=0x7ff9880caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff988158c90 0x7ff9880caf40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff9840098e0 tx=0x7ff9840091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 -- 172.21.15.111:0/3786707461 shutdown_connections 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff98806cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3786707461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff988158c90 0x7ff9880caf40 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 -- 172.21.15.111:0/3786707461 >> 172.21.15.111:0/3786707461 conn(0x7ff988067bb0 msgr2=0x7ff9880cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 -- 172.21.15.111:0/3786707461 shutdown_connections 2023-12-19T09:21:19.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.283+0000 7ff98eee3700 1 -- 172.21.15.111:0/3786707461 wait complete. 2023-12-19T09:21:19.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 Processor -- start 2023-12-19T09:21:19.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 -- start start 2023-12-19T09:21:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff98806aeb0 0x7ff9880c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff988148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880c9a40 con 0x7ff988158c90 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880c8be0 con 0x7ff98806aeb0 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff988148d80 con 0x7ff988159770 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32882/0 (socket says 172.21.15.111:32882) 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 -- 172.21.15.111:0/3898934114 learned_addr learned my addr 172.21.15.111:0/3898934114 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff98806aeb0 0x7ff9880c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98e6e2700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff988148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 -- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 msgr2=0x7ff988148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff988148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 -- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff98806aeb0 msgr2=0x7ff9880c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff98806aeb0 0x7ff9880c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 -- 172.21.15.111:0/3898934114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984009420 con 0x7ff988158c90 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98d6e0700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7ff97800cb90 tx=0x7ff978001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978007e90 con 0x7ff988158c90 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff978007360 con 0x7ff988158c90 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978007700 con 0x7ff988158c90 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.287+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9880c8e30 con 0x7ff988158c90 2023-12-19T09:21:19.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.291+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9880c93f0 con 0x7ff988158c90 2023-12-19T09:21:19.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.291+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98804be40 con 0x7ff988158c90 2023-12-19T09:21:19.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.291+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff978004a40 con 0x7ff988158c90 2023-12-19T09:21:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.295+0000 7ff97dffb700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 0x7ff96c062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.295+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5749+0+0 (secure 0 0 0) 0x7ff97807c020 con 0x7ff988158c90 2023-12-19T09:21:19.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.295+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 0x7ff96c062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.295+0000 7ff98dee1700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 0x7ff96c062a80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9840030c0 tx=0x7ff984016000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.299+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff97804c5c0 con 0x7ff988158c90 2023-12-19T09:21:19.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.467+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff98806bd10 con 0x7ff988158c90 2023-12-19T09:21:19.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.467+0000 7ff97dffb700 1 -- 172.21.15.111:0/3898934114 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff978018020 con 0x7ff988158c90 2023-12-19T09:21:19.474 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:21:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 msgr2=0x7ff96c062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 0x7ff96c062a80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9840030c0 tx=0x7ff984016000 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 msgr2=0x7ff9880c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7ff97800cb90 tx=0x7ff978001b50 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 shutdown_connections 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff988159770 0x7ff988148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff988158c90 0x7ff9880c86a0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff96c0605d0 0x7ff96c062a80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 --2- 172.21.15.111:0/3898934114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff98806aeb0 0x7ff9880c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 >> 172.21.15.111:0/3898934114 conn(0x7ff988067bb0 msgr2=0x7ff9880afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 shutdown_connections 2023-12-19T09:21:19.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.471+0000 7ff98eee3700 1 -- 172.21.15.111:0/3898934114 wait complete. 2023-12-19T09:21:19.498 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-19T09:21:19.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.651+0000 7f3c0c385700 1 Processor -- start 2023-12-19T09:21:19.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.655+0000 7f3c0c385700 1 -- start start 2023-12-19T09:21:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c04140710 0x7f3c040c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0406e150 con 0x7f3c04140ae0 2023-12-19T09:21:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0406d540 con 0x7f3c04140710 2023-12-19T09:21:19.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0406d820 con 0x7f3c04068800 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c04140710 0x7f3c040c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c04140710 0x7f3c040c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32894/0 (socket says 172.21.15.111:32894) 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 -- 172.21.15.111:0/1329398099 learned_addr learned my addr 172.21.15.111:0/1329398099 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a922700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 -- 172.21.15.111:0/1329398099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 msgr2=0x7f3c040ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 -- 172.21.15.111:0/1329398099 >> v1:172.21.15.143:6789/0 conn(0x7f3c04140ae0 legacy=0x7f3c040680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 -- 172.21.15.111:0/1329398099 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0406db00 con 0x7f3c04140710 2023-12-19T09:21:19.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0a121700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140710 0x7f3c040c70a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f3c000099b0 tx=0x7f3c00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6bed745db59dee server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0911f700 1 -- 172.21.15.111:0/1329398099 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000b860 con 0x7f3c04140710 2023-12-19T09:21:19.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0911f700 1 -- 172.21.15.111:0/1329398099 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0000b9c0 con 0x7f3c04140710 2023-12-19T09:21:19.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0911f700 1 -- 172.21.15.111:0/1329398099 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000bd60 con 0x7f3c04140710 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 -- 172.21.15.111:0/1329398099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140710 msgr2=0x7f3c040c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.659+0000 7f3c0c385700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140710 0x7f3c040c70a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f3c000099b0 tx=0x7f3c00009250 comp rx=0 tx=0).stop 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 -- 172.21.15.111:0/1329398099 shutdown_connections 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 --2- 172.21.15.111:0/1329398099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140710 0x7f3c040c70a0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 -- 172.21.15.111:0/1329398099 >> 172.21.15.111:0/1329398099 conn(0x7f3c040c8850 msgr2=0x7f3c040c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:19.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 -- 172.21.15.111:0/1329398099 shutdown_connections 2023-12-19T09:21:19.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 -- 172.21.15.111:0/1329398099 wait complete. 2023-12-19T09:21:19.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 Processor -- start 2023-12-19T09:21:19.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.663+0000 7f3c0c385700 1 -- start start 2023-12-19T09:21:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 0x7f3c04168f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040c43d0 con 0x7f3c04140ae0 2023-12-19T09:21:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040af4a0 con 0x7f3c04140710 2023-12-19T09:21:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c040af780 con 0x7f3c04068800 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32906/0 (socket says 172.21.15.111:32906) 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 -- 172.21.15.111:0/2346023250 learned_addr learned my addr 172.21.15.111:0/2346023250 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a121700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c09920700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 0x7f3c04168f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 -- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 msgr2=0x7f3c040aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 -- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 msgr2=0x7f3c04168f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 0x7f3c04168f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 -- 172.21.15.111:0/2346023250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c000094f0 con 0x7f3c04140ae0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c09920700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 0x7f3c04168f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0a922700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc007ed0 tx=0x7f3bfc009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00ec40 con 0x7f3c04140ae0 2023-12-19T09:21:19.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc00edc0 con 0x7f3c04140ae0 2023-12-19T09:21:19.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc014670 con 0x7f3c04140ae0 2023-12-19T09:21:19.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c040afa60 con 0x7f3c04140ae0 2023-12-19T09:21:19.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.667+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c040af0e0 con 0x7f3c04140ae0 2023-12-19T09:21:19.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.671+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c040c45d0 con 0x7f3c04140ae0 2023-12-19T09:21:19.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.671+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3bfc014aa0 con 0x7f3c04140ae0 2023-12-19T09:21:19.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.675+0000 7f3bfa7fc700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 0x7f3be8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:19.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.675+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f3bfc07ce50 con 0x7f3c04140ae0 2023-12-19T09:21:19.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.675+0000 7f3c0a121700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 0x7f3be8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:19.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.675+0000 7f3c0a121700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 0x7f3be8062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3c000099b0 tx=0x7f3c00017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:19.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.683+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c040c45d0 con 0x7f3c04140ae0 2023-12-19T09:21:19.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.851+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3c04071120 con 0x7f3c04140ae0 2023-12-19T09:21:19.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.851+0000 7f3bfa7fc700 1 -- 172.21.15.111:0/2346023250 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v204) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3bfc04d410 con 0x7f3c04140ae0 2023-12-19T09:21:19.858 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:19.859 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:21:19.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 msgr2=0x7f3be8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 0x7f3be8062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3c000099b0 tx=0x7f3c00017000 comp rx=0 tx=0).stop 2023-12-19T09:21:19.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 msgr2=0x7f3c041694c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:19.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc007ed0 tx=0x7f3bfc009880 comp rx=0 tx=0).stop 2023-12-19T09:21:19.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 shutdown_connections 2023-12-19T09:21:19.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3be8060550 0x7f3be8062a00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c04140710 0x7f3c04168f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c04140ae0 0x7f3c041694c0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 --2- 172.21.15.111:0/2346023250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c04068800 0x7f3c040aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 >> 172.21.15.111:0/2346023250 conn(0x7f3c040c8850 msgr2=0x7f3c040ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 shutdown_connections 2023-12-19T09:21:19.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:19.855+0000 7f3c0c385700 1 -- 172.21.15.111:0/2346023250 wait complete. 2023-12-19T09:21:19.884 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-19T09:21:19.885 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-19T09:21:20.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.047+0000 7f1f81523700 1 Processor -- start 2023-12-19T09:21:20.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.047+0000 7f1f81523700 1 -- start start 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f81523700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f7c0b0040 0x7f1f7c06af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f81523700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0cb3f0 con 0x7f1f7c0aedc0 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f81523700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c06d220 con 0x7f1f7c0af190 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f81523700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7c0cc140 con 0x7f1f7c0b0040 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f7c0b0040 0x7f1f7c06af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f7c0b0040 0x7f1f7c06af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32918/0 (socket says 172.21.15.111:32918) 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 -- 172.21.15.111:0/1350753998 learned_addr learned my addr 172.21.15.111:0/1350753998 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:20.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f7bfff700 1 --1- 172.21.15.111:0/1350753998 >> v1:172.21.15.111:6789/0 conn(0x7f1f7c0aedc0 0x7f1f7c06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46158/0 (socket says 172.21.15.111:46158) 2023-12-19T09:21:20.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 -- 172.21.15.111:0/1350753998 >> v1:172.21.15.143:6789/0 conn(0x7f1f7c0af190 legacy=0x7f1f7c0af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 -- 172.21.15.111:0/1350753998 >> v1:172.21.15.111:6789/0 conn(0x7f1f7c0aedc0 legacy=0x7f1f7c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 -- 172.21.15.111:0/1350753998 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c06c8f0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.051+0000 7f1f80d22700 1 --2- 172.21.15.111:0/1350753998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c06af50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009f20 tx=0x7f1f74009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777d6811b6830e04 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:20.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f7affd700 1 -- 172.21.15.111:0/1350753998 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7400e070 con 0x7f1f7c0b0040 2023-12-19T09:21:20.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f7affd700 1 -- 172.21.15.111:0/1350753998 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7400ba80 con 0x7f1f7c0b0040 2023-12-19T09:21:20.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f7affd700 1 -- 172.21.15.111:0/1350753998 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740092e0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 -- 172.21.15.111:0/1350753998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 msgr2=0x7f1f7c06af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 --2- 172.21.15.111:0/1350753998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c06af50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009f20 tx=0x7f1f74009840 comp rx=0 tx=0).stop 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 -- 172.21.15.111:0/1350753998 shutdown_connections 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 --2- 172.21.15.111:0/1350753998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c06af50 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 -- 172.21.15.111:0/1350753998 >> 172.21.15.111:0/1350753998 conn(0x7f1f7c0cc760 msgr2=0x7f1f7c0ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 -- 172.21.15.111:0/1350753998 shutdown_connections 2023-12-19T09:21:20.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.055+0000 7f1f81523700 1 -- 172.21.15.111:0/1350753998 wait complete. 2023-12-19T09:21:20.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 Processor -- start 2023-12-19T09:21:20.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 -- start start 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7c0aedc0 0x7f1f7c0b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f7c0af190 0x7f1f7c0c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0af610 con 0x7f1f7c0b0040 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b57a0 con 0x7f1f7c0aedc0 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f81523700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b48d0 con 0x7f1f7c0af190 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f80d22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f80d22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32932/0 (socket says 172.21.15.111:32932) 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f80d22700 1 -- 172.21.15.111:0/138665957 learned_addr learned my addr 172.21.15.111:0/138665957 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:20.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f7bfff700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7c0aedc0 0x7f1f7c0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:20.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.059+0000 7f1f7b7fe700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f7c0af190 0x7f1f7c0c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:20.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 -- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f7c0af190 msgr2=0x7f1f7c0c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f7c0af190 0x7f1f7c0c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 -- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7c0aedc0 msgr2=0x7f1f7c0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7c0aedc0 0x7f1f7c0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 -- 172.21.15.111:0/138665957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74009ae0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f80d22700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f1f74012000 tx=0x7f1f74018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740094e0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f74008d90 con 0x7f1f7c0b0040 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c0c8ca0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740256b0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c0b50e0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.063+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c0b58e0 con 0x7f1f7c0b0040 2023-12-19T09:21:20.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.067+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1f74003040 con 0x7f1f7c0b0040 2023-12-19T09:21:20.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.067+0000 7f1f63fff700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 0x7f1f68062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:20.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.067+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f1f7407e610 con 0x7f1f7c0b0040 2023-12-19T09:21:20.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.067+0000 7f1f7bfff700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 0x7f1f68062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:20.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.067+0000 7f1f7bfff700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 0x7f1f68062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c16c420 tx=0x7f1f6c007eb0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:20.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.075+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1f7404eb30 con 0x7f1f7c0b0040 2023-12-19T09:21:20.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.247+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1f7c04be40 con 0x7f1f7c0b0040 2023-12-19T09:21:20.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.875+0000 7f1f63fff700 1 -- 172.21.15.111:0/138665957 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v205) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1f74052150 con 0x7f1f7c0b0040 2023-12-19T09:21:20.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.879+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 msgr2=0x7f1f68062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.879+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 0x7f1f68062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c16c420 tx=0x7f1f6c007eb0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.879+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 msgr2=0x7f1f7c159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:20.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.879+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f1f74012000 tx=0x7f1f74018000 comp rx=0 tx=0).stop 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 shutdown_connections 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f7c0b0040 0x7f1f7c159bb0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f7c0af190 0x7f1f7c0c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1f68060550 0x7f1f68062a00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 --2- 172.21.15.111:0/138665957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7c0aedc0 0x7f1f7c0b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:20.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.883+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 >> 172.21.15.111:0/138665957 conn(0x7f1f7c0cc760 msgr2=0x7f1f7c15c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:20.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.887+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 shutdown_connections 2023-12-19T09:21:20.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:20.887+0000 7f1f81523700 1 -- 172.21.15.111:0/138665957 wait complete. 2023-12-19T09:21:20.895 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-12-19T09:21:20.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-19T09:21:21.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.083+0000 7fdc1a9cc700 1 Processor -- start 2023-12-19T09:21:21.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.083+0000 7fdc1a9cc700 1 -- start start 2023-12-19T09:21:21.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a9cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 0x7fdc140ac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a9cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc1415cc00 0x7fdc1415d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a9cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc140c68a0 con 0x7fdc140addf0 2023-12-19T09:21:21.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a9cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc140ae610 con 0x7fdc1415cc00 2023-12-19T09:21:21.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a9cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc140ac6b0 con 0x7fdc140ae1c0 2023-12-19T09:21:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc199ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc140addf0 0x7fdc1406b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46174/0 (socket says 172.21.15.111:46174) 2023-12-19T09:21:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc199ca700 1 -- 172.21.15.111:0/2476699931 learned_addr learned my addr 172.21.15.111:0/2476699931 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc191c9700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 0x7fdc140ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2476699931 >> v2:172.21.15.143:3300/0 conn(0x7fdc1415cc00 0x7fdc1415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075731606 0 0) 0x7fdc140c68a0 con 0x7fdc140addf0 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.087+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc00003570 con 0x7fdc140addf0 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 msgr2=0x7fdc140ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 0x7fdc140ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037574588 0 0) 0x7fdc00003570 con 0x7fdc140addf0 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2476699931 >> v1:172.21.15.111:6789/0 conn(0x7fdc140addf0 legacy=0x7fdc1406b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2476699931 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140c6b80 con 0x7fdc1415cc00 2023-12-19T09:21:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc191c9700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 0x7fdc140ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1415cc00 0x7fdc1415d080 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009e60 tx=0x7fdc0c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccdebf76b4605e63 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00e070 con 0x7fdc1415cc00 2023-12-19T09:21:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c00b7a0 con 0x7fdc1415cc00 2023-12-19T09:21:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc189c8700 1 -- 172.21.15.111:0/2476699931 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10001470 con 0x7fdc1415cc00 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1415cc00 msgr2=0x7fdc1415d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1415cc00 0x7fdc1415d080 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009e60 tx=0x7fdc0c009780 comp rx=0 tx=0).stop 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2476699931 shutdown_connections 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1415cc00 0x7fdc1415d080 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2476699931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140ae1c0 0x7fdc140ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2476699931 >> 172.21.15.111:0/2476699931 conn(0x7fdc140c8cd0 msgr2=0x7fdc140c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2476699931 shutdown_connections 2023-12-19T09:21:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.091+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2476699931 wait complete. 2023-12-19T09:21:21.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 Processor -- start 2023-12-19T09:21:21.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 -- start start 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140addf0 0x7fdc140b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc140ae1c0 0x7fdc1415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc140a8bd0 con 0x7fdc1415cc00 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1415bae0 con 0x7fdc140ae1c0 2023-12-19T09:21:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a9cc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1415bdc0 con 0x7fdc140addf0 2023-12-19T09:21:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a1cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a1cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32940/0 (socket says 172.21.15.111:32940) 2023-12-19T09:21:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2312529797 learned_addr learned my addr 172.21.15.111:0/2312529797 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc199ca700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140addf0 0x7fdc140b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.095+0000 7fdc191c9700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc140ae1c0 0x7fdc1415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140addf0 msgr2=0x7fdc140b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140addf0 0x7fdc140b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc140ae1c0 msgr2=0x7fdc1415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc140ae1c0 0x7fdc1415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 -- 172.21.15.111:0/2312529797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc04009420 con 0x7fdc1415cc00 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a1cb700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0097c0 tx=0x7fdc0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c012d20 con 0x7fdc1415cc00 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c012e80 con 0x7fdc1415cc00 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c009a20 con 0x7fdc1415cc00 2023-12-19T09:21:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c01d680 con 0x7fdc1415cc00 2023-12-19T09:21:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc140a8860 con 0x7fdc1415cc00 2023-12-19T09:21:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.099+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc140a8d40 con 0x7fdc1415cc00 2023-12-19T09:21:21.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.103+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdc0c024070 con 0x7fdc1415cc00 2023-12-19T09:21:21.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.103+0000 7fdc09ffb700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 0x7fdbf8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:21.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.103+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6119+0+0 (secure 0 0 0) 0x7fdc0c07e5c0 con 0x7fdc1415cc00 2023-12-19T09:21:21.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.103+0000 7fdc199ca700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 0x7fdbf8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:21.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.103+0000 7fdc199ca700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 0x7fdbf8062a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc10001620 tx=0x7fdc100014c0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:21.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.111+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc0c04ea10 con 0x7fdc1415cc00 2023-12-19T09:21:21.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.279+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdc1404be40 con 0x7fdc1415cc00 2023-12-19T09:21:21.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.891+0000 7fdc09ffb700 1 -- 172.21.15.111:0/2312529797 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v206) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdc0c052030 con 0x7fdc1415cc00 2023-12-19T09:21:21.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 msgr2=0x7fdbf8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 0x7fdbf8062a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc10001620 tx=0x7fdc100014c0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 msgr2=0x7fdc141575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:21.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0097c0 tx=0x7fdc0c018000 comp rx=0 tx=0).stop 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 shutdown_connections 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdbf8060550 0x7fdbf8062a00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc140ae1c0 0x7fdc1415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc1415cc00 0x7fdc141575f0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 --2- 172.21.15.111:0/2312529797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc140addf0 0x7fdc140b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:21.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 >> 172.21.15.111:0/2312529797 conn(0x7fdc140c8cd0 msgr2=0x7fdc1406cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:21.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 shutdown_connections 2023-12-19T09:21:21.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:21.911+0000 7fdc1a9cc700 1 -- 172.21.15.111:0/2312529797 wait complete. 2023-12-19T09:21:21.926 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-12-19T09:21:21.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-19T09:21:22.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.107+0000 7fedc5225700 1 Processor -- start 2023-12-19T09:21:22.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.107+0000 7fedc5225700 1 -- start start 2023-12-19T09:21:22.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedc5225700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fedc006c7c0 0x7fedc006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedc5225700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc006a7a0 con 0x7fedc006ac30 2023-12-19T09:21:22.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedc5225700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc006d4b0 con 0x7fedc00c6d20 2023-12-19T09:21:22.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedc5225700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc006d790 con 0x7fedc006c7c0 2023-12-19T09:21:22.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbf59e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fedc006ac30 0x7fedc006b000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46424/0 (socket says 172.21.15.111:46424) 2023-12-19T09:21:22.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbf59e700 1 -- 172.21.15.111:0/2383766103 learned_addr learned my addr 172.21.15.111:0/2383766103 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:22.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbe59c700 1 --2- 172.21.15.111:0/2383766103 >> v2:172.21.15.143:3300/0 conn(0x7fedc006c7c0 0x7fedc006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152828225 0 0) 0x7fedc006a7a0 con 0x7fedc006ac30 2023-12-19T09:21:22.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda8003570 con 0x7fedc006ac30 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3290116811 0 0) 0x7feda8003570 con 0x7fedc006ac30 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 >> v2:172.21.15.143:3300/0 conn(0x7fedc006c7c0 msgr2=0x7fedc006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 --2- 172.21.15.111:0/2383766103 >> v2:172.21.15.143:3300/0 conn(0x7fedc006c7c0 0x7fedc006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.111+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 >> v1:172.21.15.143:6789/0 conn(0x7fedc00c6d20 legacy=0x7fedc00c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc006da70 con 0x7fedc006ac30 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fedb80017f0 con 0x7fedc006ac30 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 >> v1:172.21.15.111:6789/0 conn(0x7fedc006ac30 legacy=0x7fedc006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feda8003a60 0x7feda8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7feda8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feda80051c0 0x7feda800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc006d790 con 0x7feda8004810 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda8010460 con 0x7feda8003a60 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda8010620 con 0x7feda80051c0 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7feda8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbed9d700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feda8003a60 0x7feda8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbf59e700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feda80051c0 0x7feda800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 -- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feda80051c0 msgr2=0x7feda800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feda80051c0 0x7feda800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 -- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feda8003a60 msgr2=0x7feda8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feda8003a60 0x7feda8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 -- 172.21.15.111:0/2383766103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda8003570 con 0x7feda8004810 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbed9d700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feda8003a60 0x7feda8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbf59e700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feda80051c0 0x7feda800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbf59e700 1 -- 172.21.15.111:0/2383766103 reap_dead start 2023-12-19T09:21:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbe59c700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7feda8004c80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009c00 tx=0x7fedb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64010ab8f0cb734d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb400e710 con 0x7feda8004810 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb400e870 con 0x7feda8004810 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedbdd9b700 1 -- 172.21.15.111:0/2383766103 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb400ec30 con 0x7feda8004810 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 -- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 msgr2=0x7feda8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7feda8004c80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009c00 tx=0x7fedb40091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 -- 172.21.15.111:0/2383766103 shutdown_connections 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 --2- 172.21.15.111:0/2383766103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7feda8004c80 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 -- 172.21.15.111:0/2383766103 >> 172.21.15.111:0/2383766103 conn(0x7fedc0067bb0 msgr2=0x7fedc00cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 -- 172.21.15.111:0/2383766103 shutdown_connections 2023-12-19T09:21:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.115+0000 7fedc5225700 1 -- 172.21.15.111:0/2383766103 wait complete. 2023-12-19T09:21:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 Processor -- start 2023-12-19T09:21:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 -- start start 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedc00c5c20 0x7fedc0158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedc0159240 0x7fedc0165440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00c38e0 con 0x7feda8004810 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc01596b0 con 0x7fedc00c5c20 2023-12-19T09:21:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedc5225700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00c5790 con 0x7fedc0159240 2023-12-19T09:21:22.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.119+0000 7fedbed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40306/0 (socket says 172.21.15.111:40306) 2023-12-19T09:21:22.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 -- 172.21.15.111:0/60339558 learned_addr learned my addr 172.21.15.111:0/60339558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:22.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbe59c700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedc00c5c20 0x7fedc0158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbf59e700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedc0159240 0x7fedc0165440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 -- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedc0159240 msgr2=0x7fedc0165440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedc0159240 0x7fedc0165440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 -- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedc00c5c20 msgr2=0x7fedc0158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedc00c5c20 0x7fedc0158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 -- 172.21.15.111:0/60339558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc0141810 con 0x7feda8004810 2023-12-19T09:21:22.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedbed9d700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fedb000d8d0 tx=0x7fedb0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:22.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0012600 con 0x7feda8004810 2023-12-19T09:21:22.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0012760 con 0x7feda8004810 2023-12-19T09:21:22.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc0166170 con 0x7feda8004810 2023-12-19T09:21:22.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0012b20 con 0x7feda8004810 2023-12-19T09:21:22.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc00c5140 con 0x7feda8004810 2023-12-19T09:21:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.123+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc0071120 con 0x7feda8004810 2023-12-19T09:21:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.127+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fedb0025480 con 0x7feda8004810 2023-12-19T09:21:22.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.131+0000 7feda6ffd700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 0x7fedac062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:22.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.131+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fedb007cf50 con 0x7feda8004810 2023-12-19T09:21:22.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.131+0000 7fedbe59c700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 0x7fedac062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:22.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.131+0000 7fedbe59c700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 0x7fedac062a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedb40091e0 tx=0x7fedb401a000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:22.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.135+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fedb004d220 con 0x7feda8004810 2023-12-19T09:21:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.343+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fedc006dd50 con 0x7feda8004810 2023-12-19T09:21:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.903+0000 7feda6ffd700 1 -- 172.21.15.111:0/60339558 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v91) v1 ==== 174+0+0 (secure 0 0 0) 0x7fedb0050840 con 0x7feda8004810 2023-12-19T09:21:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.911+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 msgr2=0x7fedac062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.911+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 0x7fedac062a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedb40091e0 tx=0x7fedb401a000 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.911+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 msgr2=0x7fedc01412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.911+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fedb000d8d0 tx=0x7fedb0009040 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 shutdown_connections 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fedac060580 0x7fedac062a30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fedc00c5c20 0x7fedc0158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fedc0159240 0x7fedc0165440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 --2- 172.21.15.111:0/60339558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feda8004810 0x7fedc01412d0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 >> 172.21.15.111:0/60339558 conn(0x7fedc0067bb0 msgr2=0x7fedc0157960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 shutdown_connections 2023-12-19T09:21:22.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:22.915+0000 7fedc5225700 1 -- 172.21.15.111:0/60339558 wait complete. 2023-12-19T09:21:22.936 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 25 and data pool 26 2023-12-19T09:21:22.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:23.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 Processor -- start 2023-12-19T09:21:23.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 -- start start 2023-12-19T09:21:23.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0cd80c6900 0x7f0cd80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd806b470 con 0x7f0cd815cc20 2023-12-19T09:21:23.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd806e110 con 0x7f0cd806b900 2023-12-19T09:21:23.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.103+0000 7f0cde80c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd815d790 con 0x7f0cd80c6900 2023-12-19T09:21:23.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cde00b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0cd815cc20 0x7f0cd815d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46426/0 (socket says 172.21.15.111:46426) 2023-12-19T09:21:23.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cde00b700 1 -- 172.21.15.111:0/4276917732 learned_addr learned my addr 172.21.15.111:0/4276917732 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:23.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/4276917732 >> v2:172.21.15.143:3300/0 conn(0x7f0cd80c6900 0x7f0cd80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455635772 0 0) 0x7f0cd806b470 con 0x7f0cd815cc20 2023-12-19T09:21:23.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4003570 con 0x7f0cd815cc20 2023-12-19T09:21:23.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276080384 0 0) 0x7f0cd806e110 con 0x7f0cd806b900 2023-12-19T09:21:23.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd806b470 con 0x7f0cd806b900 2023-12-19T09:21:23.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3060541107 0 0) 0x7f0cc4003570 con 0x7f0cd815cc20 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 >> v2:172.21.15.143:3300/0 conn(0x7f0cd80c6900 msgr2=0x7f0cd80c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 --2- 172.21.15.111:0/4276917732 >> v2:172.21.15.143:3300/0 conn(0x7f0cd80c6900 0x7f0cd80c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 >> v1:172.21.15.143:6789/0 conn(0x7f0cd806b900 legacy=0x7f0cd806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8161d10 con 0x7f0cd815cc20 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0cd00017f0 con 0x7f0cd815cc20 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 >> v1:172.21.15.111:6789/0 conn(0x7f0cd815cc20 legacy=0x7f0cd815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 0x7f0cc4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cc4004810 0x7f0cc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc40051c0 0x7f0cc400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd815d790 con 0x7f0cc4003bf0 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc400c0b0 con 0x7f0cc4004810 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc400c270 con 0x7f0cc40051c0 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 0x7f0cc4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd009700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cc4004810 0x7f0cc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cde00b700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc40051c0 0x7f0cc400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 -- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc40051c0 msgr2=0x7f0cc400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc40051c0 0x7f0cc400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 -- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cc4004810 msgr2=0x7f0cc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cc4004810 0x7f0cc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd80a700 1 -- 172.21.15.111:0/4276917732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc4003570 con 0x7f0cc4003bf0 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cdd009700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cc4004810 0x7f0cc4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.107+0000 7f0cde00b700 1 -- 172.21.15.111:0/4276917732 reap_dead start 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 0x7f0cc4003fe0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009c00 tx=0x7f0cd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb3e451d7d8b71f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400e710 con 0x7f0cc4003bf0 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd400e870 con 0x7f0cc4003bf0 2023-12-19T09:21:23.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cdc808700 1 -- 172.21.15.111:0/4276917732 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400ec30 con 0x7f0cc4003bf0 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 -- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 msgr2=0x7f0cc4003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 0x7f0cc4003fe0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009c00 tx=0x7f0cd40091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 -- 172.21.15.111:0/4276917732 shutdown_connections 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 --2- 172.21.15.111:0/4276917732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cc4003bf0 0x7f0cc4003fe0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 -- 172.21.15.111:0/4276917732 >> 172.21.15.111:0/4276917732 conn(0x7f0cd8067bb0 msgr2=0x7f0cd80cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 -- 172.21.15.111:0/4276917732 shutdown_connections 2023-12-19T09:21:23.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 -- 172.21.15.111:0/4276917732 wait complete. 2023-12-19T09:21:23.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.111+0000 7f0cde80c700 1 Processor -- start 2023-12-19T09:21:23.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 -- start start 2023-12-19T09:21:23.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc4003bf0 0x7f0cd816ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd816f140 0x7f0cd816d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8170510 con 0x7f0cd815d8d0 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd816ddb0 con 0x7f0cd816f140 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd816f900 con 0x7f0cc4003bf0 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40330/0 (socket says 172.21.15.111:40330) 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 -- 172.21.15.111:0/694825819 learned_addr learned my addr 172.21.15.111:0/694825819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc4003bf0 0x7f0cd816ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cdd009700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd816f140 0x7f0cd816d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 -- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc4003bf0 msgr2=0x7f0cd816ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc4003bf0 0x7f0cd816ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 -- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd816f140 msgr2=0x7f0cd816d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd816f140 0x7f0cd816d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 -- 172.21.15.111:0/694825819 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd816fbe0 con 0x7f0cd815d8d0 2023-12-19T09:21:23.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde00b700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0cd815cff0 tx=0x7f0cd00018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd000e830 con 0x7f0cd815d8d0 2023-12-19T09:21:23.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd000e9b0 con 0x7f0cd815d8d0 2023-12-19T09:21:23.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd816fec0 con 0x7f0cd815d8d0 2023-12-19T09:21:23.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.115+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd000ed70 con 0x7f0cd815d8d0 2023-12-19T09:21:23.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.119+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd816f5b0 con 0x7f0cd815d8d0 2023-12-19T09:21:23.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.119+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8170340 con 0x7f0cd815d8d0 2023-12-19T09:21:23.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.119+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0cd0013630 con 0x7f0cd815d8d0 2023-12-19T09:21:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.119+0000 7f0ccdffb700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 0x7f0cbc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.119+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f0cd007ce40 con 0x7f0cd815d8d0 2023-12-19T09:21:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.123+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 0x7f0cbc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.123+0000 7f0cdd80a700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 0x7f0cbc062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40091e0 tx=0x7f0cd401a000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.127+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0cd004d010 con 0x7f0cd815d8d0 2023-12-19T09:21:23.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.299+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cd804be40 con 0x7f0cd815d8d0 2023-12-19T09:21:23.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.303+0000 7f0ccdffb700 1 -- 172.21.15.111:0/694825819 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16510 (secure 0 0 0) 0x7f0cd0050630 con 0x7f0cd815d8d0 2023-12-19T09:21:23.308 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:23.309 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":207,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:22.351413+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:23.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 msgr2=0x7f0cbc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 0x7f0cbc062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40091e0 tx=0x7f0cd401a000 comp rx=0 tx=0).stop 2023-12-19T09:21:23.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 msgr2=0x7f0cd80acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0cd815cff0 tx=0x7f0cd00018a0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 shutdown_connections 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0cbc060550 0x7f0cbc062a00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd816f140 0x7f0cd816d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd815d8d0 0x7f0cd80acbb0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 --2- 172.21.15.111:0/694825819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cc4003bf0 0x7f0cd816ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 >> 172.21.15.111:0/694825819 conn(0x7f0cd8067bb0 msgr2=0x7f0cd815bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 shutdown_connections 2023-12-19T09:21:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.307+0000 7f0cde80c700 1 -- 172.21.15.111:0/694825819 wait complete. 2023-12-19T09:21:23.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:23.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 Processor -- start 2023-12-19T09:21:23.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 -- start start 2023-12-19T09:21:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 0x7f8c8406a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8406b730 con 0x7f8c841591f0 2023-12-19T09:21:23.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8406ab20 con 0x7f8c84158e20 2023-12-19T09:21:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.487+0000 7f8c8359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8406ae00 con 0x7f8c840ce810 2023-12-19T09:21:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c7bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8c841591f0 0x7f8c840ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:37652/0 (socket says 172.21.15.111:37652) 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c7bfff700 1 -- 172.21.15.111:0/2782011839 learned_addr learned my addr 172.21.15.111:0/2782011839 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:45922/0 (socket says 172.21.15.111:45922) 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c8259c700 1 --2- 172.21.15.111:0/2782011839 >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 0x7f8c8406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c81d9b700 1 -- 172.21.15.111:0/2782011839 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432406508 0 0) 0x7f8c8406b730 con 0x7f8c841591f0 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c81d9b700 1 -- 172.21.15.111:0/2782011839 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c70003570 con 0x7f8c841591f0 2023-12-19T09:21:23.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 -- 172.21.15.111:0/2782011839 >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 msgr2=0x7f8c8406a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 --2- 172.21.15.111:0/2782011839 >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 0x7f8c8406a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 -- 172.21.15.111:0/2782011839 >> v1:172.21.15.143:6789/0 conn(0x7f8c841591f0 legacy=0x7f8c840ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 -- 172.21.15.111:0/2782011839 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8406b0e0 con 0x7f8c840ce810 2023-12-19T09:21:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c8259c700 1 --2- 172.21.15.111:0/2782011839 >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 0x7f8c8406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:23.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c82d9d700 1 --2- 172.21.15.111:0/2782011839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009e60 tx=0x7f8c7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cfc615be6c78e29 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c81d9b700 1 -- 172.21.15.111:0/2782011839 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00e070 con 0x7f8c840ce810 2023-12-19T09:21:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c81d9b700 1 -- 172.21.15.111:0/2782011839 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c00bab0 con 0x7f8c840ce810 2023-12-19T09:21:23.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.491+0000 7f8c81d9b700 1 -- 172.21.15.111:0/2782011839 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00be70 con 0x7f8c840ce810 2023-12-19T09:21:23.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- 172.21.15.111:0/2782011839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 msgr2=0x7f8c840ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 --2- 172.21.15.111:0/2782011839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009e60 tx=0x7f8c7c009780 comp rx=0 tx=0).stop 2023-12-19T09:21:23.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- 172.21.15.111:0/2782011839 shutdown_connections 2023-12-19T09:21:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 --2- 172.21.15.111:0/2782011839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c840ce810 0x7f8c840ca950 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 --2- 172.21.15.111:0/2782011839 >> v2:172.21.15.143:3300/0 conn(0x7f8c84158e20 0x7f8c8406a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- 172.21.15.111:0/2782011839 >> 172.21.15.111:0/2782011839 conn(0x7f8c84067bb0 msgr2=0x7f8c840cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- 172.21.15.111:0/2782011839 shutdown_connections 2023-12-19T09:21:23.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- 172.21.15.111:0/2782011839 wait complete. 2023-12-19T09:21:23.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 Processor -- start 2023-12-19T09:21:23.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.495+0000 7f8c8359e700 1 -- start start 2023-12-19T09:21:23.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 0x7f8c8414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c84158e20 0x7f8c841530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8416d8f0 con 0x7f8c841591f0 2023-12-19T09:21:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84159bb0 con 0x7f8c840ce810 2023-12-19T09:21:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8414df00 con 0x7f8c84158e20 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40340/0 (socket says 172.21.15.111:40340) 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 -- 172.21.15.111:0/3120803070 learned_addr learned my addr 172.21.15.111:0/3120803070 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c7bfff700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c84158e20 0x7f8c841530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8259c700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 0x7f8c8414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 -- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c84158e20 msgr2=0x7f8c841530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c84158e20 0x7f8c841530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 -- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 msgr2=0x7f8c8414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 0x7f8c8414d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 -- 172.21.15.111:0/3120803070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74009420 con 0x7f8c841591f0 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8259c700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 0x7f8c8414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c82d9d700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8c8406af40 tx=0x7f8c7c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c027470 con 0x7f8c841591f0 2023-12-19T09:21:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c0275f0 con 0x7f8c841591f0 2023-12-19T09:21:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c009a20 con 0x7f8c841591f0 2023-12-19T09:21:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0279b0 con 0x7f8c841591f0 2023-12-19T09:21:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.499+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8416cfc0 con 0x7f8c841591f0 2023-12-19T09:21:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.503+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8416da60 con 0x7f8c841591f0 2023-12-19T09:21:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.503+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8c7c024070 con 0x7f8c841591f0 2023-12-19T09:21:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.503+0000 7f8c7a7fc700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 0x7f8c64062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.503+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f8c7c07f450 con 0x7f8c841591f0 2023-12-19T09:21:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.503+0000 7f8c8259c700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 0x7f8c64062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.507+0000 7f8c8259c700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 0x7f8c64062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c740093f0 tx=0x7f8c74008040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.511+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c7c04f6c0 con 0x7f8c841591f0 2023-12-19T09:21:23.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.675+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c8404be40 con 0x7f8c841591f0 2023-12-19T09:21:23.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c7a7fc700 1 -- 172.21.15.111:0/3120803070 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16510 (secure 0 0 0) 0x7f8c7c052ce0 con 0x7f8c841591f0 2023-12-19T09:21:23.684 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:23.684 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":207,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:22.351413+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 msgr2=0x7f8c64062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 0x7f8c64062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c740093f0 tx=0x7f8c74008040 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 msgr2=0x7f8c841535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8c8406af40 tx=0x7f8c7c019000 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 shutdown_connections 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c841591f0 0x7f8c841535e0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c84158e20 0x7f8c841530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c64060550 0x7f8c64062a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 --2- 172.21.15.111:0/3120803070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c840ce810 0x7f8c8414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 >> 172.21.15.111:0/3120803070 conn(0x7f8c84067bb0 msgr2=0x7f8c8415f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 shutdown_connections 2023-12-19T09:21:23.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.679+0000 7f8c8359e700 1 -- 172.21.15.111:0/3120803070 wait complete. 2023-12-19T09:21:23.710 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-19T09:21:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.875+0000 7f994826e700 1 Processor -- start 2023-12-19T09:21:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.875+0000 7f994826e700 1 -- start start 2023-12-19T09:21:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994826e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 0x7f994006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994826e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99400cbdf0 con 0x7f99400cecd0 2023-12-19T09:21:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994826e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99400cc0d0 con 0x7f9940158e20 2023-12-19T09:21:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994826e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400cc3b0 con 0x7f99400cb970 2023-12-19T09:21:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994680b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f99400cecd0 0x7f994006b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46440/0 (socket says 172.21.15.111:46440) 2023-12-19T09:21:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f994680b700 1 -- 172.21.15.111:0/3379899415 learned_addr learned my addr 172.21.15.111:0/3379899415 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945809700 1 --2- 172.21.15.111:0/3379899415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 0x7f994006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075820851 0 0) 0x7f99400cbdf0 con 0x7f99400cecd0 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992c003570 con 0x7f99400cecd0 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649258656 0 0) 0x7f99400cc0d0 con 0x7f9940158e20 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945809700 1 -- 172.21.15.111:0/3379899415 >> v1:172.21.15.143:6789/0 conn(0x7f9940158e20 legacy=0x7f99400ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945809700 1 -- 172.21.15.111:0/3379899415 >> v1:172.21.15.111:6789/0 conn(0x7f99400cecd0 legacy=0x7f994006b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945809700 1 -- 172.21.15.111:0/3379899415 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400cc690 con 0x7f99400cb970 2023-12-19T09:21:23.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945809700 1 --2- 172.21.15.111:0/3379899415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 0x7f994006acc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9930009910 tx=0x7f99300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81cf530a9d6eb13b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99380017f0 con 0x7f99400cb970 2023-12-19T09:21:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930011420 con 0x7f99400cb970 2023-12-19T09:21:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.879+0000 7f9945008700 1 -- 172.21.15.111:0/3379899415 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99300117c0 con 0x7f99400cb970 2023-12-19T09:21:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- 172.21.15.111:0/3379899415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 msgr2=0x7f994006acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 --2- 172.21.15.111:0/3379899415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 0x7f994006acc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9930009910 tx=0x7f99300091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- 172.21.15.111:0/3379899415 shutdown_connections 2023-12-19T09:21:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 --2- 172.21.15.111:0/3379899415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cb970 0x7f994006acc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- 172.21.15.111:0/3379899415 >> 172.21.15.111:0/3379899415 conn(0x7f9940067bb0 msgr2=0x7f99400d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- 172.21.15.111:0/3379899415 shutdown_connections 2023-12-19T09:21:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- 172.21.15.111:0/3379899415 wait complete. 2023-12-19T09:21:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 Processor -- start 2023-12-19T09:21:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- start start 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cecd0 0x7f99400b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015e1e0 0x7f9940151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400c7fb0 con 0x7f9940158e20 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994015d810 con 0x7f994015e1e0 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f994826e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400b5ec0 con 0x7f99400cecd0 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.883+0000 7f9945809700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40356/0 (socket says 172.21.15.111:40356) 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 -- 172.21.15.111:0/4132289813 learned_addr learned my addr 172.21.15.111:0/4132289813 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:23.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f994600a700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cecd0 0x7f99400b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f994680b700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015e1e0 0x7f9940151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 -- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cecd0 msgr2=0x7f99400b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cecd0 0x7f99400b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 -- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015e1e0 msgr2=0x7f9940151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015e1e0 0x7f9940151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:23.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 -- 172.21.15.111:0/4132289813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930009420 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f9945809700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f9930009fb0 tx=0x7f9930017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930010c10 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930010d70 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994015cc00 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993001d680 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994015d1c0 con 0x7f9940158e20 2023-12-19T09:21:23.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.887+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9940071120 con 0x7f9940158e20 2023-12-19T09:21:23.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.891+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f993001da30 con 0x7f9940158e20 2023-12-19T09:21:23.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.891+0000 7f99367fc700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 0x7f9924062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:23.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.891+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f993007e950 con 0x7f9940158e20 2023-12-19T09:21:23.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.891+0000 7f994600a700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 0x7f9924062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:23.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.891+0000 7f994600a700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 0x7f9924062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f994016c600 tx=0x7f993c009040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:23.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:23.895+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f993004ebc0 con 0x7f9940158e20 2023-12-19T09:21:24.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.107+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f994015d950 con 0x7f9940158e20 2023-12-19T09:21:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.951+0000 7f99367fc700 1 -- 172.21.15.111:0/4132289813 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v94) v1 ==== 118+0+0 (secure 0 0 0) 0x7f99300521e0 con 0x7f9940158e20 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 msgr2=0x7f9924062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 0x7f9924062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f994016c600 tx=0x7f993c009040 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 msgr2=0x7f994015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f9930009fb0 tx=0x7f9930017000 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 shutdown_connections 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9924060550 0x7f9924062a00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994015e1e0 0x7f9940151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9940158e20 0x7f994015dca0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 --2- 172.21.15.111:0/4132289813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99400cecd0 0x7f99400b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 >> 172.21.15.111:0/4132289813 conn(0x7f9940067bb0 msgr2=0x7f9940157780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:24.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 shutdown_connections 2023-12-19T09:21:24.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:24.955+0000 7f994826e700 1 -- 172.21.15.111:0/4132289813 wait complete. 2023-12-19T09:21:24.984 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-19T09:21:25.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 Processor -- start 2023-12-19T09:21:25.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 -- start start 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb0840cab10 0x7fb0840ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0840c7450 con 0x7fb08406b1b0 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08406bea0 con 0x7fb08415d410 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb08bca5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0840c6e00 con 0x7fb0840cab10 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb089240700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb0840cab10 0x7fb0840ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb089240700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb0840cab10 0x7fb0840ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40370/0 (socket says 172.21.15.111:40370) 2023-12-19T09:21:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.143+0000 7fb089240700 1 -- 172.21.15.111:0/4073971777 learned_addr learned my addr 172.21.15.111:0/4073971777 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:25.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb089240700 1 -- 172.21.15.111:0/4073971777 >> v1:172.21.15.143:6789/0 conn(0x7fb08415d410 legacy=0x7fb0840ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:25.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb089240700 1 -- 172.21.15.111:0/4073971777 >> v1:172.21.15.111:6789/0 conn(0x7fb08406b1b0 legacy=0x7fb08406b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb089240700 1 -- 172.21.15.111:0/4073971777 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0840c70e0 con 0x7fb0840cab10 2023-12-19T09:21:25.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb089240700 1 --2- 172.21.15.111:0/4073971777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840ab6c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb0740099b0 tx=0x7fb074009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a9b51f70c1364b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:25.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb088a3f700 1 -- 172.21.15.111:0/4073971777 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400b860 con 0x7fb0840cab10 2023-12-19T09:21:25.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb088a3f700 1 -- 172.21.15.111:0/4073971777 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07400b9c0 con 0x7fb0840cab10 2023-12-19T09:21:25.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb088a3f700 1 -- 172.21.15.111:0/4073971777 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400bd60 con 0x7fb0840cab10 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 -- 172.21.15.111:0/4073971777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 msgr2=0x7fb0840ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 --2- 172.21.15.111:0/4073971777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840ab6c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb0740099b0 tx=0x7fb074009250 comp rx=0 tx=0).stop 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 -- 172.21.15.111:0/4073971777 shutdown_connections 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 --2- 172.21.15.111:0/4073971777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840ab6c0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 -- 172.21.15.111:0/4073971777 >> 172.21.15.111:0/4073971777 conn(0x7fb084067bb0 msgr2=0x7fb0840cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 -- 172.21.15.111:0/4073971777 shutdown_connections 2023-12-19T09:21:25.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.147+0000 7fb08bca5700 1 -- 172.21.15.111:0/4073971777 wait complete. 2023-12-19T09:21:25.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 Processor -- start 2023-12-19T09:21:25.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 -- start start 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb08406b1b0 0x7fb084159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb08415d410 0x7fb0841717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08416dc30 con 0x7fb0840cab10 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb084171cf0 con 0x7fb08415d410 2023-12-19T09:21:25.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb08bca5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08416d900 con 0x7fb08406b1b0 2023-12-19T09:21:25.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.151+0000 7fb089240700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:25.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40378/0 (socket says 172.21.15.111:40378) 2023-12-19T09:21:25.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 -- 172.21.15.111:0/2423895898 learned_addr learned my addr 172.21.15.111:0/2423895898 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:25.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089a41700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb08406b1b0 0x7fb084159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:25.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb08a242700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb08415d410 0x7fb0841717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 -- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb08406b1b0 msgr2=0x7fb084159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb08406b1b0 0x7fb084159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 -- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb08415d410 msgr2=0x7fb0841717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb08415d410 0x7fb0841717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 -- 172.21.15.111:0/2423895898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740094f0 con 0x7fb0840cab10 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb089240700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb0840635d0 tx=0x7fb074017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:25.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074011ea0 con 0x7fb0840cab10 2023-12-19T09:21:25.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07401a630 con 0x7fb0840cab10 2023-12-19T09:21:25.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07401a9d0 con 0x7fb0840cab10 2023-12-19T09:21:25.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0840b14b0 con 0x7fb0840cab10 2023-12-19T09:21:25.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0840b1a70 con 0x7fb0840cab10 2023-12-19T09:21:25.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.155+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08416dda0 con 0x7fb0840cab10 2023-12-19T09:21:25.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.159+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb07402d440 con 0x7fb0840cab10 2023-12-19T09:21:25.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.159+0000 7fb079ffb700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 0x7fb068062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:25.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.159+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fb07407e6a0 con 0x7fb0840cab10 2023-12-19T09:21:25.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.159+0000 7fb089a41700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 0x7fb068062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:25.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.163+0000 7fb089a41700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 0x7fb068062a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb0800037c0 tx=0x7fb080009040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:25.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.163+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb07404e870 con 0x7fb0840cab10 2023-12-19T09:21:25.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.375+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb084071120 con 0x7fb0840cab10 2023-12-19T09:21:25.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.955+0000 7fb079ffb700 1 -- 172.21.15.111:0/2423895898 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v95) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb074051e90 con 0x7fb0840cab10 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 msgr2=0x7fb068062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 0x7fb068062a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb0800037c0 tx=0x7fb080009040 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 msgr2=0x7fb0840c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb0840635d0 tx=0x7fb074017000 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 shutdown_connections 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb068060550 0x7fb068062a00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb08415d410 0x7fb0841717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0840cab10 0x7fb0840c58c0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 --2- 172.21.15.111:0/2423895898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb08406b1b0 0x7fb084159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:25.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 >> 172.21.15.111:0/2423895898 conn(0x7fb084067bb0 msgr2=0x7fb084062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:25.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 shutdown_connections 2023-12-19T09:21:25.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:25.959+0000 7fb08bca5700 1 -- 172.21.15.111:0/2423895898 wait complete. 2023-12-19T09:21:25.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:26.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.163+0000 7fb2d7145700 1 Processor -- start 2023-12-19T09:21:26.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.167+0000 7fb2d7145700 1 -- start start 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2d00ae140 0x7fb2d00a8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 0x7fb2d00c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d006dc20 con 0x7fb2d006d180 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d006d940 con 0x7fb2d00ae140 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00c6e20 con 0x7fb2d00ae510 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2d00ae140 0x7fb2d00a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2d00ae140 0x7fb2d00a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40380/0 (socket says 172.21.15.111:40380) 2023-12-19T09:21:26.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 -- 172.21.15.111:0/3498487118 learned_addr learned my addr 172.21.15.111:0/3498487118 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:26.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2c7fff700 1 --2- 172.21.15.111:0/3498487118 >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 0x7fb2d00c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 -- 172.21.15.111:0/3498487118 >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 msgr2=0x7fb2d00c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 --2- 172.21.15.111:0/3498487118 >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 0x7fb2d00c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 -- 172.21.15.111:0/3498487118 >> v1:172.21.15.111:6789/0 conn(0x7fb2d006d180 legacy=0x7fb2d00680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 -- 172.21.15.111:0/3498487118 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00c7100 con 0x7fb2d00ae140 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2c7fff700 1 --2- 172.21.15.111:0/3498487118 >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 0x7fb2d00c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d4ee1700 1 --2- 172.21.15.111:0/3498487118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae140 0x7fb2d00a8850 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb2d006a0a0 tx=0x7fb2cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6556ffd1d00fd740 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2c77fe700 1 -- 172.21.15.111:0/3498487118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00b860 con 0x7fb2d00ae140 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2c77fe700 1 -- 172.21.15.111:0/3498487118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc00b9c0 con 0x7fb2d00ae140 2023-12-19T09:21:26.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2c77fe700 1 -- 172.21.15.111:0/3498487118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00bd60 con 0x7fb2d00ae140 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 -- 172.21.15.111:0/3498487118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae140 msgr2=0x7fb2d00a8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.171+0000 7fb2d7145700 1 --2- 172.21.15.111:0/3498487118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae140 0x7fb2d00a8850 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb2d006a0a0 tx=0x7fb2cc009250 comp rx=0 tx=0).stop 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 -- 172.21.15.111:0/3498487118 shutdown_connections 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 --2- 172.21.15.111:0/3498487118 >> v2:172.21.15.143:3300/0 conn(0x7fb2d00ae510 0x7fb2d00c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 --2- 172.21.15.111:0/3498487118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae140 0x7fb2d00a8850 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 -- 172.21.15.111:0/3498487118 >> 172.21.15.111:0/3498487118 conn(0x7fb2d00c8850 msgr2=0x7fb2d00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 -- 172.21.15.111:0/3498487118 shutdown_connections 2023-12-19T09:21:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 -- 172.21.15.111:0/3498487118 wait complete. 2023-12-19T09:21:26.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 Processor -- start 2023-12-19T09:21:26.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.175+0000 7fb2d7145700 1 -- start start 2023-12-19T09:21:26.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2d006d180 0x7fb2d0147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 0x7fb2d0156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0157cc0 con 0x7fb2d00ae510 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0156820 con 0x7fb2d00ae140 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d01553f0 con 0x7fb2d006d180 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 0x7fb2d0156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:56928/0 (socket says 172.21.15.111:56928) 2023-12-19T09:21:26.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c7fff700 1 -- 172.21.15.111:0/2683390376 learned_addr learned my addr 172.21.15.111:0/2683390376 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:26.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d4ee1700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2d006d180 0x7fb2d0147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 -- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2d006d180 msgr2=0x7fb2d0147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2d006d180 0x7fb2d0147ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 -- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 msgr2=0x7fb2d01562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 -- 172.21.15.111:0/2683390376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0094f0 con 0x7fb2d00ae510 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c7fff700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d56e2700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 0x7fb2d0156e70 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c860 tx=0x7fb2c8001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8012630 con 0x7fb2d00ae510 2023-12-19T09:21:26.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8012790 con 0x7fb2d00ae510 2023-12-19T09:21:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8012b50 con 0x7fb2d00ae510 2023-12-19T09:21:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d0155840 con 0x7fb2d00ae510 2023-12-19T09:21:26.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.179+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d0155ae0 con 0x7fb2d00ae510 2023-12-19T09:21:26.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.183+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d01556a0 con 0x7fb2d00ae510 2023-12-19T09:21:26.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.183+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb2c801a3a0 con 0x7fb2d00ae510 2023-12-19T09:21:26.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.183+0000 7fb2c4ff9700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 0x7fb2bc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.187+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fb2c807cc00 con 0x7fb2d00ae510 2023-12-19T09:21:26.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.187+0000 7fb2d4ee1700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 0x7fb2bc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.187+0000 7fb2d4ee1700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 0x7fb2bc062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2d006a0a0 tx=0x7fb2cc017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.191+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb2d01556a0 con 0x7fb2d00ae510 2023-12-19T09:21:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.395+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d01556a0 con 0x7fb2d00ae510 2023-12-19T09:21:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.395+0000 7fb2c4ff9700 1 -- 172.21.15.111:0/2683390376 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7fb2d01556a0 con 0x7fb2d00ae510 2023-12-19T09:21:26.403 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:26.403 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 msgr2=0x7fb2bc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 0x7fb2bc062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2d006a0a0 tx=0x7fb2cc017000 comp rx=0 tx=0).stop 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 msgr2=0x7fb2d0156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 0x7fb2d0156e70 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c860 tx=0x7fb2c8001690 comp rx=0 tx=0).stop 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 shutdown_connections 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb2bc060550 0x7fb2bc062a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d00ae140 0x7fb2d01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2d00ae510 0x7fb2d0156e70 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 --2- 172.21.15.111:0/2683390376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2d006d180 0x7fb2d0147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.399+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 >> 172.21.15.111:0/2683390376 conn(0x7fb2d00c8850 msgr2=0x7fb2d0067f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:26.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.403+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 shutdown_connections 2023-12-19T09:21:26.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.403+0000 7fb2d7145700 1 -- 172.21.15.111:0/2683390376 wait complete. 2023-12-19T09:21:26.410 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:26.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:26.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.579+0000 7f2bb300c700 1 Processor -- start 2023-12-19T09:21:26.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.579+0000 7f2bb300c700 1 -- start start 2023-12-19T09:21:26.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb300c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 0x7f2bac0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb300c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac06b470 con 0x7f2bac15cc20 2023-12-19T09:21:26.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb300c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac06e110 con 0x7f2bac06b900 2023-12-19T09:21:26.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb300c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac15d7b0 con 0x7f2bac0c6900 2023-12-19T09:21:26.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb280b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2bac15cc20 0x7f2bac15d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46460/0 (socket says 172.21.15.111:46460) 2023-12-19T09:21:26.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb280b700 1 -- 172.21.15.111:0/3939414276 learned_addr learned my addr 172.21.15.111:0/3939414276 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:26.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3939414276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 0x7f2bac0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344409384 0 0) 0x7f2bac06b470 con 0x7f2bac15cc20 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b98003570 con 0x7f2bac15cc20 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969520223 0 0) 0x7f2bac06e110 con 0x7f2bac06b900 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bac06b470 con 0x7f2bac06b900 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb200a700 1 -- 172.21.15.111:0/3939414276 >> v1:172.21.15.143:6789/0 conn(0x7f2bac06b900 legacy=0x7f2bac06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb200a700 1 -- 172.21.15.111:0/3939414276 >> v1:172.21.15.111:6789/0 conn(0x7f2bac15cc20 legacy=0x7f2bac15d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.583+0000 7f2bb200a700 1 -- 172.21.15.111:0/3939414276 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac161d10 con 0x7f2bac0c6900 2023-12-19T09:21:26.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3939414276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 0x7f2bac0c6cd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009940 tx=0x7f2ba80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67fb4241218a1cb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8011630 con 0x7f2bac0c6900 2023-12-19T09:21:26.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8011790 con 0x7f2bac0c6900 2023-12-19T09:21:26.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb1008700 1 -- 172.21.15.111:0/3939414276 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8011b30 con 0x7f2bac0c6900 2023-12-19T09:21:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 -- 172.21.15.111:0/3939414276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 msgr2=0x7f2bac0c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3939414276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 0x7f2bac0c6cd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009940 tx=0x7f2ba80091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 -- 172.21.15.111:0/3939414276 shutdown_connections 2023-12-19T09:21:26.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3939414276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac0c6900 0x7f2bac0c6cd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 -- 172.21.15.111:0/3939414276 >> 172.21.15.111:0/3939414276 conn(0x7f2bac067bb0 msgr2=0x7f2bac0cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 -- 172.21.15.111:0/3939414276 shutdown_connections 2023-12-19T09:21:26.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 -- 172.21.15.111:0/3939414276 wait complete. 2023-12-19T09:21:26.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.587+0000 7f2bb300c700 1 Processor -- start 2023-12-19T09:21:26.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 -- start start 2023-12-19T09:21:26.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 0x7f2bac158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bac15cc20 0x7f2bac15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0aff20 con 0x7f2bac0c6900 2023-12-19T09:21:26.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac158dd0 con 0x7f2bac15cc20 2023-12-19T09:21:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0af310 con 0x7f2bac06b900 2023-12-19T09:21:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40400/0 (socket says 172.21.15.111:40400) 2023-12-19T09:21:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 -- 172.21.15.111:0/3298769517 learned_addr learned my addr 172.21.15.111:0/3298769517 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 0x7f2bac158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb280b700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bac15cc20 0x7f2bac15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 -- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 msgr2=0x7f2bac158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 0x7f2bac158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 -- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bac15cc20 msgr2=0x7f2bac15a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bac15cc20 0x7f2bac15a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 -- 172.21.15.111:0/3298769517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8009420 con 0x7f2bac0c6900 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 0x7f2bac158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb1809700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00c790 tx=0x7f2b9c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c010510 con 0x7f2bac0c6900 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c010670 con 0x7f2bac0c6900 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c010a10 con 0x7f2bac0c6900 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.591+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac0af5f0 con 0x7f2bac0c6900 2023-12-19T09:21:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.595+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac0afbb0 con 0x7f2bac0c6900 2023-12-19T09:21:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.595+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac0b00d0 con 0x7f2bac0c6900 2023-12-19T09:21:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.595+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f2b9c01b3b0 con 0x7f2bac0c6900 2023-12-19T09:21:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.595+0000 7f2ba27fc700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 0x7f2b90062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.599+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f2b9c07c340 con 0x7f2bac0c6900 2023-12-19T09:21:26.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.599+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 0x7f2b90062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.599+0000 7f2bb200a700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 0x7f2b90062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80093f0 tx=0x7f2ba8017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.603+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2bac0b00d0 con 0x7f2bac0c6900 2023-12-19T09:21:26.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.775+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bac04be40 con 0x7f2bac0c6900 2023-12-19T09:21:26.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2ba27fc700 1 -- 172.21.15.111:0/3298769517 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16497 (secure 0 0 0) 0x7f2b9c017090 con 0x7f2bac0c6900 2023-12-19T09:21:26.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:26.785 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":208,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:23.918275+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 msgr2=0x7f2b90062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 0x7f2b90062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80093f0 tx=0x7f2ba8017000 comp rx=0 tx=0).stop 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 msgr2=0x7f2bac0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00c790 tx=0x7f2b9c007e10 comp rx=0 tx=0).stop 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 shutdown_connections 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f2b90060580 0x7f2b90062a30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bac15cc20 0x7f2bac15a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bac0c6900 0x7f2bac0ae8e0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 --2- 172.21.15.111:0/3298769517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bac06b900 0x7f2bac158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 >> 172.21.15.111:0/3298769517 conn(0x7f2bac067bb0 msgr2=0x7f2bac160d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.779+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 shutdown_connections 2023-12-19T09:21:26.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.783+0000 7f2bb300c700 1 -- 172.21.15.111:0/3298769517 wait complete. 2023-12-19T09:21:26.806 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-19T09:21:26.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:21:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.955+0000 7f9e332f0700 1 Processor -- start 2023-12-19T09:21:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.955+0000 7f9e332f0700 1 -- start start 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e2c158f70 0x7f9e2c067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c069100 0x7f9e2c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c068c70 con 0x7f9e2c159340 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2c068060 con 0x7f9e2c158f70 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c068340 con 0x7f9e2c069100 2023-12-19T09:21:26.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e2c158f70 0x7f9e2c067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9e2c158f70 0x7f9e2c067980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40408/0 (socket says 172.21.15.111:40408) 2023-12-19T09:21:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 -- 172.21.15.111:0/1743690262 learned_addr learned my addr 172.21.15.111:0/1743690262 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 -- 172.21.15.111:0/1743690262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c069100 msgr2=0x7f9e2c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 --2- 172.21.15.111:0/1743690262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c069100 0x7f9e2c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 -- 172.21.15.111:0/1743690262 >> v1:172.21.15.143:6789/0 conn(0x7f9e2c159340 legacy=0x7f9e2c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 -- 172.21.15.111:0/1743690262 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c068620 con 0x7f9e2c158f70 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e3108c700 1 --2- 172.21.15.111:0/1743690262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c158f70 0x7f9e2c067980 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f9e280099b0 tx=0x7f9e28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf44e42f8ac3ce0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e23fff700 1 -- 172.21.15.111:0/1743690262 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800b820 con 0x7f9e2c158f70 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e23fff700 1 -- 172.21.15.111:0/1743690262 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2800b980 con 0x7f9e2c158f70 2023-12-19T09:21:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e23fff700 1 -- 172.21.15.111:0/1743690262 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800bd20 con 0x7f9e2c158f70 2023-12-19T09:21:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 -- 172.21.15.111:0/1743690262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c158f70 msgr2=0x7f9e2c067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.959+0000 7f9e332f0700 1 --2- 172.21.15.111:0/1743690262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c158f70 0x7f9e2c067980 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f9e280099b0 tx=0x7f9e28009250 comp rx=0 tx=0).stop 2023-12-19T09:21:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 -- 172.21.15.111:0/1743690262 shutdown_connections 2023-12-19T09:21:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 --2- 172.21.15.111:0/1743690262 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c069100 0x7f9e2c0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 --2- 172.21.15.111:0/1743690262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c158f70 0x7f9e2c067980 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 -- 172.21.15.111:0/1743690262 >> 172.21.15.111:0/1743690262 conn(0x7f9e2c0ccc40 msgr2=0x7f9e2c0cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 -- 172.21.15.111:0/1743690262 shutdown_connections 2023-12-19T09:21:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 -- 172.21.15.111:0/1743690262 wait complete. 2023-12-19T09:21:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 Processor -- start 2023-12-19T09:21:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.963+0000 7f9e332f0700 1 -- start start 2023-12-19T09:21:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e2c158f70 0x7f9e2c0c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c159340 0x7f9e2c0b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c156440 con 0x7f9e2c069100 2023-12-19T09:21:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c1572a0 con 0x7f9e2c158f70 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c156690 con 0x7f9e2c159340 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40416/0 (socket says 172.21.15.111:40416) 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 -- 172.21.15.111:0/3588012848 learned_addr learned my addr 172.21.15.111:0/3588012848 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3088b700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e2c158f70 0x7f9e2c0c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3188d700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c159340 0x7f9e2c0b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 -- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c159340 msgr2=0x7f9e2c0b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c159340 0x7f9e2c0b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 -- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e2c158f70 msgr2=0x7f9e2c0c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:26.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e2c158f70 0x7f9e2c0c6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 -- 172.21.15.111:0/3588012848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280094f0 con 0x7f9e2c069100 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e3108c700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f9e280094c0 tx=0x7f9e28016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28026480 con 0x7f9e2c069100 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2801f070 con 0x7f9e2c069100 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28026850 con 0x7f9e2c069100 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c156970 con 0x7f9e2c069100 2023-12-19T09:21:26.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.967+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c156f30 con 0x7f9e2c069100 2023-12-19T09:21:26.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.971+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c070800 con 0x7f9e2c069100 2023-12-19T09:21:26.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.971+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9e2801a440 con 0x7f9e2c069100 2023-12-19T09:21:26.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.971+0000 7f9e217fa700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 0x7f9e10062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:26.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.971+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f9e28021070 con 0x7f9e2c069100 2023-12-19T09:21:26.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.971+0000 7f9e3088b700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 0x7f9e10062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:26.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.975+0000 7f9e3088b700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 0x7f9e10062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0ca400 tx=0x7f9e1c0091a0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:26.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:26.979+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9e280aebe0 con 0x7f9e2c069100 2023-12-19T09:21:27.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.143+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e2c1573e0 con 0x7f9e10060580 2023-12-19T09:21:27.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.163+0000 7f9e217fa700 1 -- 172.21.15.111:0/3588012848 <== mgr.8642 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287252 (secure 0 0 0) 0x7f9e1c00e6b0 con 0x7f9e10060580 2023-12-19T09:21:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 msgr2=0x7f9e10062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 0x7f9e10062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0ca400 tx=0x7f9e1c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 msgr2=0x7f9e2c0c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f9e280094c0 tx=0x7f9e28016000 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 shutdown_connections 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9e10060580 0x7f9e10062a30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e2c158f70 0x7f9e2c0c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e2c159340 0x7f9e2c0b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 --2- 172.21.15.111:0/3588012848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e2c069100 0x7f9e2c0c7960 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 >> 172.21.15.111:0/3588012848 conn(0x7f9e2c0ccc40 msgr2=0x7f9e2c0c5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.171+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 shutdown_connections 2023-12-19T09:21:27.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.175+0000 7f9e332f0700 1 -- 172.21.15.111:0/3588012848 wait complete. 2023-12-19T09:21:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:21:27.200 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:21:26.640528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2469468,"kb_used_data":21532,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752505252,"statfs":{"total":773094113280,"available":770565378048,"internally_reserved":0,"allocated":22048768,"data_stored":12355825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.760707"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938889+0000","last_change":"2023-12-19T09:21:22.938889+0000","last_active":"2023-12-19T09:21:22.938889+0000","last_peered":"2023-12-19T09:21:22.938889+0000","last_clean":"2023-12-19T09:21:22.938889+0000","last_became_active":"2023-12-19T09:21:22.938567+0000","last_became_peered":"2023-12-19T09:21:22.938567+0000","last_unstale":"2023-12-19T09:21:22.938889+0000","last_undegraded":"2023-12-19T09:21:22.938889+0000","last_fullsized":"2023-12-19T09:21:22.938889+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:50:09.431525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938587+0000","last_change":"2023-12-19T09:21:22.938587+0000","last_active":"2023-12-19T09:21:22.938587+0000","last_peered":"2023-12-19T09:21:22.938587+0000","last_clean":"2023-12-19T09:21:22.938587+0000","last_became_active":"2023-12-19T09:21:22.938277+0000","last_became_peered":"2023-12-19T09:21:22.938277+0000","last_unstale":"2023-12-19T09:21:22.938587+0000","last_undegraded":"2023-12-19T09:21:22.938587+0000","last_fullsized":"2023-12-19T09:21:22.938587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:04.070678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938450+0000","last_change":"2023-12-19T09:21:22.938450+0000","last_active":"2023-12-19T09:21:22.938450+0000","last_peered":"2023-12-19T09:21:22.938450+0000","last_clean":"2023-12-19T09:21:22.938450+0000","last_became_active":"2023-12-19T09:21:22.936757+0000","last_became_peered":"2023-12-19T09:21:22.936757+0000","last_unstale":"2023-12-19T09:21:22.938450+0000","last_undegraded":"2023-12-19T09:21:22.938450+0000","last_fullsized":"2023-12-19T09:21:22.938450+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:43.091347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931559+0000","last_change":"2023-12-19T09:21:22.931559+0000","last_active":"2023-12-19T09:21:22.931559+0000","last_peered":"2023-12-19T09:21:22.931559+0000","last_clean":"2023-12-19T09:21:22.931559+0000","last_became_active":"2023-12-19T09:21:22.931170+0000","last_became_peered":"2023-12-19T09:21:22.931170+0000","last_unstale":"2023-12-19T09:21:22.931559+0000","last_undegraded":"2023-12-19T09:21:22.931559+0000","last_fullsized":"2023-12-19T09:21:22.931559+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:30:13.652617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948114+0000","last_change":"2023-12-19T09:21:22.939676+0000","last_active":"2023-12-19T09:21:22.948114+0000","last_peered":"2023-12-19T09:21:22.948114+0000","last_clean":"2023-12-19T09:21:22.948114+0000","last_became_active":"2023-12-19T09:21:22.939365+0000","last_became_peered":"2023-12-19T09:21:22.939365+0000","last_unstale":"2023-12-19T09:21:22.948114+0000","last_undegraded":"2023-12-19T09:21:22.948114+0000","last_fullsized":"2023-12-19T09:21:22.948114+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:50.596361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947073+0000","last_change":"2023-12-19T09:21:22.935045+0000","last_active":"2023-12-19T09:21:22.947073+0000","last_peered":"2023-12-19T09:21:22.947073+0000","last_clean":"2023-12-19T09:21:22.947073+0000","last_became_active":"2023-12-19T09:21:22.934776+0000","last_became_peered":"2023-12-19T09:21:22.934776+0000","last_unstale":"2023-12-19T09:21:22.947073+0000","last_undegraded":"2023-12-19T09:21:22.947073+0000","last_fullsized":"2023-12-19T09:21:22.947073+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:24:36.778946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.928927+0000","last_change":"2023-12-19T09:21:22.928927+0000","last_active":"2023-12-19T09:21:22.928927+0000","last_peered":"2023-12-19T09:21:22.928927+0000","last_clean":"2023-12-19T09:21:22.928927+0000","last_became_active":"2023-12-19T09:21:22.925981+0000","last_became_peered":"2023-12-19T09:21:22.925981+0000","last_unstale":"2023-12-19T09:21:22.928927+0000","last_undegraded":"2023-12-19T09:21:22.928927+0000","last_fullsized":"2023-12-19T09:21:22.928927+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:55.880776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943995+0000","last_change":"2023-12-19T09:21:22.943995+0000","last_active":"2023-12-19T09:21:22.943995+0000","last_peered":"2023-12-19T09:21:22.943995+0000","last_clean":"2023-12-19T09:21:22.943995+0000","last_became_active":"2023-12-19T09:21:22.943155+0000","last_became_peered":"2023-12-19T09:21:22.943155+0000","last_unstale":"2023-12-19T09:21:22.943995+0000","last_undegraded":"2023-12-19T09:21:22.943995+0000","last_fullsized":"2023-12-19T09:21:22.943995+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:42.203249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.932430+0000","last_change":"2023-12-19T09:21:22.932430+0000","last_active":"2023-12-19T09:21:22.932430+0000","last_peered":"2023-12-19T09:21:22.932430+0000","last_clean":"2023-12-19T09:21:22.932430+0000","last_became_active":"2023-12-19T09:21:22.930436+0000","last_became_peered":"2023-12-19T09:21:22.930436+0000","last_unstale":"2023-12-19T09:21:22.932430+0000","last_undegraded":"2023-12-19T09:21:22.932430+0000","last_fullsized":"2023-12-19T09:21:22.932430+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:30:13.652617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948221+0000","last_change":"2023-12-19T09:21:22.942518+0000","last_active":"2023-12-19T09:21:22.948221+0000","last_peered":"2023-12-19T09:21:22.948221+0000","last_clean":"2023-12-19T09:21:22.948221+0000","last_became_active":"2023-12-19T09:21:22.942219+0000","last_became_peered":"2023-12-19T09:21:22.942219+0000","last_unstale":"2023-12-19T09:21:22.948221+0000","last_undegraded":"2023-12-19T09:21:22.948221+0000","last_fullsized":"2023-12-19T09:21:22.948221+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:04.977368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940153+0000","last_change":"2023-12-19T09:21:22.940153+0000","last_active":"2023-12-19T09:21:22.940153+0000","last_peered":"2023-12-19T09:21:22.940153+0000","last_clean":"2023-12-19T09:21:22.940153+0000","last_became_active":"2023-12-19T09:21:22.939876+0000","last_became_peered":"2023-12-19T09:21:22.939876+0000","last_unstale":"2023-12-19T09:21:22.940153+0000","last_undegraded":"2023-12-19T09:21:22.940153+0000","last_fullsized":"2023-12-19T09:21:22.940153+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:15.617941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.922251+0000","last_change":"2023-12-19T09:21:22.922251+0000","last_active":"2023-12-19T09:21:22.922251+0000","last_peered":"2023-12-19T09:21:22.922251+0000","last_clean":"2023-12-19T09:21:22.922251+0000","last_became_active":"2023-12-19T09:21:22.921922+0000","last_became_peered":"2023-12-19T09:21:22.921922+0000","last_unstale":"2023-12-19T09:21:22.922251+0000","last_undegraded":"2023-12-19T09:21:22.922251+0000","last_fullsized":"2023-12-19T09:21:22.922251+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:49:12.990854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942195+0000","last_change":"2023-12-19T09:21:22.930133+0000","last_active":"2023-12-19T09:21:22.942195+0000","last_peered":"2023-12-19T09:21:22.942195+0000","last_clean":"2023-12-19T09:21:22.942195+0000","last_became_active":"2023-12-19T09:21:22.929778+0000","last_became_peered":"2023-12-19T09:21:22.929778+0000","last_unstale":"2023-12-19T09:21:22.942195+0000","last_undegraded":"2023-12-19T09:21:22.942195+0000","last_fullsized":"2023-12-19T09:21:22.942195+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:19:11.108096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.945131+0000","last_change":"2023-12-19T09:21:22.945131+0000","last_active":"2023-12-19T09:21:22.945131+0000","last_peered":"2023-12-19T09:21:22.945131+0000","last_clean":"2023-12-19T09:21:22.945131+0000","last_became_active":"2023-12-19T09:21:22.944619+0000","last_became_peered":"2023-12-19T09:21:22.944619+0000","last_unstale":"2023-12-19T09:21:22.945131+0000","last_undegraded":"2023-12-19T09:21:22.945131+0000","last_fullsized":"2023-12-19T09:21:22.945131+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:03.099999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941048+0000","last_change":"2023-12-19T09:21:22.941048+0000","last_active":"2023-12-19T09:21:22.941048+0000","last_peered":"2023-12-19T09:21:22.941048+0000","last_clean":"2023-12-19T09:21:22.941048+0000","last_became_active":"2023-12-19T09:21:22.940754+0000","last_became_peered":"2023-12-19T09:21:22.940754+0000","last_unstale":"2023-12-19T09:21:22.941048+0000","last_undegraded":"2023-12-19T09:21:22.941048+0000","last_fullsized":"2023-12-19T09:21:22.941048+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:46.636632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947046+0000","last_change":"2023-12-19T09:21:22.934994+0000","last_active":"2023-12-19T09:21:22.947046+0000","last_peered":"2023-12-19T09:21:22.947046+0000","last_clean":"2023-12-19T09:21:22.947046+0000","last_became_active":"2023-12-19T09:21:22.933158+0000","last_became_peered":"2023-12-19T09:21:22.933158+0000","last_unstale":"2023-12-19T09:21:22.947046+0000","last_undegraded":"2023-12-19T09:21:22.947046+0000","last_fullsized":"2023-12-19T09:21:22.947046+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:50.583942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948087+0000","last_change":"2023-12-19T09:21:22.938161+0000","last_active":"2023-12-19T09:21:22.948087+0000","last_peered":"2023-12-19T09:21:22.948087+0000","last_clean":"2023-12-19T09:21:22.948087+0000","last_became_active":"2023-12-19T09:21:22.937880+0000","last_became_peered":"2023-12-19T09:21:22.937880+0000","last_unstale":"2023-12-19T09:21:22.948087+0000","last_undegraded":"2023-12-19T09:21:22.948087+0000","last_fullsized":"2023-12-19T09:21:22.948087+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:04.009857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942298+0000","last_change":"2023-12-19T09:21:22.927633+0000","last_active":"2023-12-19T09:21:22.942298+0000","last_peered":"2023-12-19T09:21:22.942298+0000","last_clean":"2023-12-19T09:21:22.942298+0000","last_became_active":"2023-12-19T09:21:22.927348+0000","last_became_peered":"2023-12-19T09:21:22.927348+0000","last_unstale":"2023-12-19T09:21:22.942298+0000","last_undegraded":"2023-12-19T09:21:22.942298+0000","last_fullsized":"2023-12-19T09:21:22.942298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:05.211274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940953+0000","last_change":"2023-12-19T09:21:22.926038+0000","last_active":"2023-12-19T09:21:22.940953+0000","last_peered":"2023-12-19T09:21:22.940953+0000","last_clean":"2023-12-19T09:21:22.940953+0000","last_became_active":"2023-12-19T09:21:22.925709+0000","last_became_peered":"2023-12-19T09:21:22.925709+0000","last_unstale":"2023-12-19T09:21:22.940953+0000","last_undegraded":"2023-12-19T09:21:22.940953+0000","last_fullsized":"2023-12-19T09:21:22.940953+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:21:05.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948061+0000","last_change":"2023-12-19T09:21:22.937870+0000","last_active":"2023-12-19T09:21:22.948061+0000","last_peered":"2023-12-19T09:21:22.948061+0000","last_clean":"2023-12-19T09:21:22.948061+0000","last_became_active":"2023-12-19T09:21:22.937585+0000","last_became_peered":"2023-12-19T09:21:22.937585+0000","last_unstale":"2023-12-19T09:21:22.948061+0000","last_undegraded":"2023-12-19T09:21:22.948061+0000","last_fullsized":"2023-12-19T09:21:22.948061+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:08:58.809426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929962+0000","last_change":"2023-12-19T09:21:22.929962+0000","last_active":"2023-12-19T09:21:22.929962+0000","last_peered":"2023-12-19T09:21:22.929962+0000","last_clean":"2023-12-19T09:21:22.929962+0000","last_became_active":"2023-12-19T09:21:22.929686+0000","last_became_peered":"2023-12-19T09:21:22.929686+0000","last_unstale":"2023-12-19T09:21:22.929962+0000","last_undegraded":"2023-12-19T09:21:22.929962+0000","last_fullsized":"2023-12-19T09:21:22.929962+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:52.610948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.928513+0000","last_change":"2023-12-19T09:21:22.928513+0000","last_active":"2023-12-19T09:21:22.928513+0000","last_peered":"2023-12-19T09:21:22.928513+0000","last_clean":"2023-12-19T09:21:22.928513+0000","last_became_active":"2023-12-19T09:21:22.928224+0000","last_became_peered":"2023-12-19T09:21:22.928224+0000","last_unstale":"2023-12-19T09:21:22.928513+0000","last_undegraded":"2023-12-19T09:21:22.928513+0000","last_fullsized":"2023-12-19T09:21:22.928513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:58.963749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.933993+0000","last_change":"2023-12-19T09:21:22.933993+0000","last_active":"2023-12-19T09:21:22.933993+0000","last_peered":"2023-12-19T09:21:22.933993+0000","last_clean":"2023-12-19T09:21:22.933993+0000","last_became_active":"2023-12-19T09:21:22.933706+0000","last_became_peered":"2023-12-19T09:21:22.933706+0000","last_unstale":"2023-12-19T09:21:22.933993+0000","last_undegraded":"2023-12-19T09:21:22.933993+0000","last_fullsized":"2023-12-19T09:21:22.933993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:20:56.456843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938092+0000","last_change":"2023-12-19T09:21:22.938092+0000","last_active":"2023-12-19T09:21:22.938092+0000","last_peered":"2023-12-19T09:21:22.938092+0000","last_clean":"2023-12-19T09:21:22.938092+0000","last_became_active":"2023-12-19T09:21:22.937806+0000","last_became_peered":"2023-12-19T09:21:22.937806+0000","last_unstale":"2023-12-19T09:21:22.938092+0000","last_undegraded":"2023-12-19T09:21:22.938092+0000","last_fullsized":"2023-12-19T09:21:22.938092+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:20.910570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.933389+0000","last_change":"2023-12-19T09:21:22.933389+0000","last_active":"2023-12-19T09:21:22.933389+0000","last_peered":"2023-12-19T09:21:22.933389+0000","last_clean":"2023-12-19T09:21:22.933389+0000","last_became_active":"2023-12-19T09:21:22.933111+0000","last_became_peered":"2023-12-19T09:21:22.933111+0000","last_unstale":"2023-12-19T09:21:22.933389+0000","last_undegraded":"2023-12-19T09:21:22.933389+0000","last_fullsized":"2023-12-19T09:21:22.933389+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:57.852046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927267+0000","last_change":"2023-12-19T09:21:22.927267+0000","last_active":"2023-12-19T09:21:22.927267+0000","last_peered":"2023-12-19T09:21:22.927267+0000","last_clean":"2023-12-19T09:21:22.927267+0000","last_became_active":"2023-12-19T09:21:22.926953+0000","last_became_peered":"2023-12-19T09:21:22.926953+0000","last_unstale":"2023-12-19T09:21:22.927267+0000","last_undegraded":"2023-12-19T09:21:22.927267+0000","last_fullsized":"2023-12-19T09:21:22.927267+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:55.220863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.926136+0000","last_change":"2023-12-19T09:21:22.926136+0000","last_active":"2023-12-19T09:21:22.926136+0000","last_peered":"2023-12-19T09:21:22.926136+0000","last_clean":"2023-12-19T09:21:22.926136+0000","last_became_active":"2023-12-19T09:21:22.925800+0000","last_became_peered":"2023-12-19T09:21:22.925800+0000","last_unstale":"2023-12-19T09:21:22.926136+0000","last_undegraded":"2023-12-19T09:21:22.926136+0000","last_fullsized":"2023-12-19T09:21:22.926136+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:04.070678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948162+0000","last_change":"2023-12-19T09:21:22.941027+0000","last_active":"2023-12-19T09:21:22.948162+0000","last_peered":"2023-12-19T09:21:22.948162+0000","last_clean":"2023-12-19T09:21:22.948162+0000","last_became_active":"2023-12-19T09:21:22.940118+0000","last_became_peered":"2023-12-19T09:21:22.940118+0000","last_unstale":"2023-12-19T09:21:22.948162+0000","last_undegraded":"2023-12-19T09:21:22.948162+0000","last_fullsized":"2023-12-19T09:21:22.948162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:42.031571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.934510+0000","last_change":"2023-12-19T09:21:22.934510+0000","last_active":"2023-12-19T09:21:22.934510+0000","last_peered":"2023-12-19T09:21:22.934510+0000","last_clean":"2023-12-19T09:21:22.934510+0000","last_became_active":"2023-12-19T09:21:22.934232+0000","last_became_peered":"2023-12-19T09:21:22.934232+0000","last_unstale":"2023-12-19T09:21:22.934510+0000","last_undegraded":"2023-12-19T09:21:22.934510+0000","last_fullsized":"2023-12-19T09:21:22.934510+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:36.056878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.935298+0000","last_change":"2023-12-19T09:21:22.935298+0000","last_active":"2023-12-19T09:21:22.935298+0000","last_peered":"2023-12-19T09:21:22.935298+0000","last_clean":"2023-12-19T09:21:22.935298+0000","last_became_active":"2023-12-19T09:21:22.935017+0000","last_became_peered":"2023-12-19T09:21:22.935017+0000","last_unstale":"2023-12-19T09:21:22.935298+0000","last_undegraded":"2023-12-19T09:21:22.935298+0000","last_fullsized":"2023-12-19T09:21:22.935298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:39:44.920814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927293+0000","last_change":"2023-12-19T09:21:22.927293+0000","last_active":"2023-12-19T09:21:22.927293+0000","last_peered":"2023-12-19T09:21:22.927293+0000","last_clean":"2023-12-19T09:21:22.927293+0000","last_became_active":"2023-12-19T09:21:22.924124+0000","last_became_peered":"2023-12-19T09:21:22.924124+0000","last_unstale":"2023-12-19T09:21:22.927293+0000","last_undegraded":"2023-12-19T09:21:22.927293+0000","last_fullsized":"2023-12-19T09:21:22.927293+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:10:30.315001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948207+0000","last_change":"2023-12-19T09:21:22.941718+0000","last_active":"2023-12-19T09:21:22.948207+0000","last_peered":"2023-12-19T09:21:22.948207+0000","last_clean":"2023-12-19T09:21:22.948207+0000","last_became_active":"2023-12-19T09:21:22.941098+0000","last_became_peered":"2023-12-19T09:21:22.941098+0000","last_unstale":"2023-12-19T09:21:22.948207+0000","last_undegraded":"2023-12-19T09:21:22.948207+0000","last_fullsized":"2023-12-19T09:21:22.948207+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:27.891719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.926684+0000","last_change":"2023-12-19T09:21:22.926684+0000","last_active":"2023-12-19T09:21:22.926684+0000","last_peered":"2023-12-19T09:21:22.926684+0000","last_clean":"2023-12-19T09:21:22.926684+0000","last_became_active":"2023-12-19T09:21:22.926341+0000","last_became_peered":"2023-12-19T09:21:22.926341+0000","last_unstale":"2023-12-19T09:21:22.926684+0000","last_undegraded":"2023-12-19T09:21:22.926684+0000","last_fullsized":"2023-12-19T09:21:22.926684+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:07:21.121732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"207'4","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947522+0000","last_change":"2023-12-19T09:21:21.920062+0000","last_active":"2023-12-19T09:21:22.947522+0000","last_peered":"2023-12-19T09:21:22.947522+0000","last_clean":"2023-12-19T09:21:22.947522+0000","last_became_active":"2023-12-19T09:21:21.919729+0000","last_became_peered":"2023-12-19T09:21:21.919729+0000","last_unstale":"2023-12-19T09:21:22.947522+0000","last_undegraded":"2023-12-19T09:21:22.947522+0000","last_fullsized":"2023-12-19T09:21:22.947522+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:39:43.908102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943984+0000","last_change":"2023-12-19T09:21:22.943984+0000","last_active":"2023-12-19T09:21:22.943984+0000","last_peered":"2023-12-19T09:21:22.943984+0000","last_clean":"2023-12-19T09:21:22.943984+0000","last_became_active":"2023-12-19T09:21:22.942882+0000","last_became_peered":"2023-12-19T09:21:22.942882+0000","last_unstale":"2023-12-19T09:21:22.943984+0000","last_undegraded":"2023-12-19T09:21:22.943984+0000","last_fullsized":"2023-12-19T09:21:22.943984+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:03.387676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"207'1","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946344+0000","last_change":"2023-12-19T09:21:21.924299+0000","last_active":"2023-12-19T09:21:22.946344+0000","last_peered":"2023-12-19T09:21:22.946344+0000","last_clean":"2023-12-19T09:21:22.946344+0000","last_became_active":"2023-12-19T09:21:21.923979+0000","last_became_peered":"2023-12-19T09:21:21.923979+0000","last_unstale":"2023-12-19T09:21:22.946344+0000","last_undegraded":"2023-12-19T09:21:22.946344+0000","last_fullsized":"2023-12-19T09:21:22.946344+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:57.857343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931157+0000","last_change":"2023-12-19T09:21:22.931157+0000","last_active":"2023-12-19T09:21:22.931157+0000","last_peered":"2023-12-19T09:21:22.931157+0000","last_clean":"2023-12-19T09:21:22.931157+0000","last_became_active":"2023-12-19T09:21:22.930673+0000","last_became_peered":"2023-12-19T09:21:22.930673+0000","last_unstale":"2023-12-19T09:21:22.931157+0000","last_undegraded":"2023-12-19T09:21:22.931157+0000","last_fullsized":"2023-12-19T09:21:22.931157+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:08.045104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942547+0000","last_change":"2023-12-19T09:21:21.922027+0000","last_active":"2023-12-19T09:21:22.942547+0000","last_peered":"2023-12-19T09:21:22.942547+0000","last_clean":"2023-12-19T09:21:22.942547+0000","last_became_active":"2023-12-19T09:21:21.921695+0000","last_became_peered":"2023-12-19T09:21:21.921695+0000","last_unstale":"2023-12-19T09:21:22.942547+0000","last_undegraded":"2023-12-19T09:21:22.942547+0000","last_fullsized":"2023-12-19T09:21:22.942547+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:29.103271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.939613+0000","last_change":"2023-12-19T09:21:22.939613+0000","last_active":"2023-12-19T09:21:22.939613+0000","last_peered":"2023-12-19T09:21:22.939613+0000","last_clean":"2023-12-19T09:21:22.939613+0000","last_became_active":"2023-12-19T09:21:22.939140+0000","last_became_peered":"2023-12-19T09:21:22.939140+0000","last_unstale":"2023-12-19T09:21:22.939613+0000","last_undegraded":"2023-12-19T09:21:22.939613+0000","last_fullsized":"2023-12-19T09:21:22.939613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:54.496034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940687+0000","last_change":"2023-12-19T09:21:21.920285+0000","last_active":"2023-12-19T09:21:22.940687+0000","last_peered":"2023-12-19T09:21:22.940687+0000","last_clean":"2023-12-19T09:21:22.940687+0000","last_became_active":"2023-12-19T09:21:21.919970+0000","last_became_peered":"2023-12-19T09:21:21.919970+0000","last_unstale":"2023-12-19T09:21:22.940687+0000","last_undegraded":"2023-12-19T09:21:22.940687+0000","last_fullsized":"2023-12-19T09:21:22.940687+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:38.377336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942006+0000","last_change":"2023-12-19T09:21:22.927043+0000","last_active":"2023-12-19T09:21:22.942006+0000","last_peered":"2023-12-19T09:21:22.942006+0000","last_clean":"2023-12-19T09:21:22.942006+0000","last_became_active":"2023-12-19T09:21:22.926733+0000","last_became_peered":"2023-12-19T09:21:22.926733+0000","last_unstale":"2023-12-19T09:21:22.942006+0000","last_undegraded":"2023-12-19T09:21:22.942006+0000","last_fullsized":"2023-12-19T09:21:22.942006+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:00.796583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"207'1","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946777+0000","last_change":"2023-12-19T09:21:21.925122+0000","last_active":"2023-12-19T09:21:22.946777+0000","last_peered":"2023-12-19T09:21:22.946777+0000","last_clean":"2023-12-19T09:21:22.946777+0000","last_became_active":"2023-12-19T09:21:21.924752+0000","last_became_peered":"2023-12-19T09:21:21.924752+0000","last_unstale":"2023-12-19T09:21:22.946777+0000","last_undegraded":"2023-12-19T09:21:22.946777+0000","last_fullsized":"2023-12-19T09:21:22.946777+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:03.057966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931799+0000","last_change":"2023-12-19T09:21:22.931799+0000","last_active":"2023-12-19T09:21:22.931799+0000","last_peered":"2023-12-19T09:21:22.931799+0000","last_clean":"2023-12-19T09:21:22.931799+0000","last_became_active":"2023-12-19T09:21:22.931405+0000","last_became_peered":"2023-12-19T09:21:22.931405+0000","last_unstale":"2023-12-19T09:21:22.931799+0000","last_undegraded":"2023-12-19T09:21:22.931799+0000","last_fullsized":"2023-12-19T09:21:22.931799+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:23:07.226827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.904262+0000","last_change":"2023-12-19T09:21:21.928658+0000","last_active":"2023-12-19T09:21:22.904262+0000","last_peered":"2023-12-19T09:21:22.904262+0000","last_clean":"2023-12-19T09:21:22.904262+0000","last_became_active":"2023-12-19T09:21:21.928342+0000","last_became_peered":"2023-12-19T09:21:21.928342+0000","last_unstale":"2023-12-19T09:21:22.904262+0000","last_undegraded":"2023-12-19T09:21:22.904262+0000","last_fullsized":"2023-12-19T09:21:22.904262+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:37.335226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941893+0000","last_change":"2023-12-19T09:21:22.941893+0000","last_active":"2023-12-19T09:21:22.941893+0000","last_peered":"2023-12-19T09:21:22.941893+0000","last_clean":"2023-12-19T09:21:22.941893+0000","last_became_active":"2023-12-19T09:21:22.941587+0000","last_became_peered":"2023-12-19T09:21:22.941587+0000","last_unstale":"2023-12-19T09:21:22.941893+0000","last_undegraded":"2023-12-19T09:21:22.941893+0000","last_fullsized":"2023-12-19T09:21:22.941893+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:51.456781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.907081+0000","last_change":"2023-12-19T09:21:21.917883+0000","last_active":"2023-12-19T09:21:22.907081+0000","last_peered":"2023-12-19T09:21:22.907081+0000","last_clean":"2023-12-19T09:21:22.907081+0000","last_became_active":"2023-12-19T09:21:21.917505+0000","last_became_peered":"2023-12-19T09:21:21.917505+0000","last_unstale":"2023-12-19T09:21:22.907081+0000","last_undegraded":"2023-12-19T09:21:22.907081+0000","last_fullsized":"2023-12-19T09:21:22.907081+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:08.727761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946786+0000","last_change":"2023-12-19T09:21:22.946786+0000","last_active":"2023-12-19T09:21:22.946786+0000","last_peered":"2023-12-19T09:21:22.946786+0000","last_clean":"2023-12-19T09:21:22.946786+0000","last_became_active":"2023-12-19T09:21:22.945991+0000","last_became_peered":"2023-12-19T09:21:22.945991+0000","last_unstale":"2023-12-19T09:21:22.946786+0000","last_undegraded":"2023-12-19T09:21:22.946786+0000","last_fullsized":"2023-12-19T09:21:22.946786+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:40.463149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.913524+0000","last_change":"2023-12-19T09:21:21.924668+0000","last_active":"2023-12-19T09:21:22.913524+0000","last_peered":"2023-12-19T09:21:22.913524+0000","last_clean":"2023-12-19T09:21:22.913524+0000","last_became_active":"2023-12-19T09:21:21.924198+0000","last_became_peered":"2023-12-19T09:21:21.924198+0000","last_unstale":"2023-12-19T09:21:22.913524+0000","last_undegraded":"2023-12-19T09:21:22.913524+0000","last_fullsized":"2023-12-19T09:21:22.913524+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:41.190537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.925540+0000","last_change":"2023-12-19T09:21:22.920989+0000","last_active":"2023-12-19T09:21:22.925540+0000","last_peered":"2023-12-19T09:21:22.925540+0000","last_clean":"2023-12-19T09:21:22.925540+0000","last_became_active":"2023-12-19T09:21:22.920672+0000","last_became_peered":"2023-12-19T09:21:22.920672+0000","last_unstale":"2023-12-19T09:21:22.925540+0000","last_undegraded":"2023-12-19T09:21:22.925540+0000","last_fullsized":"2023-12-19T09:21:22.925540+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:10:29.585036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.928880+0000","last_change":"2023-12-19T09:21:21.927320+0000","last_active":"2023-12-19T09:21:21.928880+0000","last_peered":"2023-12-19T09:21:21.928880+0000","last_clean":"2023-12-19T09:21:21.928880+0000","last_became_active":"2023-12-19T09:21:21.927000+0000","last_became_peered":"2023-12-19T09:21:21.927000+0000","last_unstale":"2023-12-19T09:21:21.928880+0000","last_undegraded":"2023-12-19T09:21:21.928880+0000","last_fullsized":"2023-12-19T09:21:21.928880+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:19:10.095384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942316+0000","last_change":"2023-12-19T09:21:22.929154+0000","last_active":"2023-12-19T09:21:22.942316+0000","last_peered":"2023-12-19T09:21:22.942316+0000","last_clean":"2023-12-19T09:21:22.942316+0000","last_became_active":"2023-12-19T09:21:22.928793+0000","last_became_peered":"2023-12-19T09:21:22.928793+0000","last_unstale":"2023-12-19T09:21:22.942316+0000","last_undegraded":"2023-12-19T09:21:22.942316+0000","last_fullsized":"2023-12-19T09:21:22.942316+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:53:28.304421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"207'3","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948470+0000","last_change":"2023-12-19T09:21:21.928897+0000","last_active":"2023-12-19T09:21:22.948470+0000","last_peered":"2023-12-19T09:21:22.948470+0000","last_clean":"2023-12-19T09:21:22.948470+0000","last_became_active":"2023-12-19T09:21:21.928413+0000","last_became_peered":"2023-12-19T09:21:21.928413+0000","last_unstale":"2023-12-19T09:21:22.948470+0000","last_undegraded":"2023-12-19T09:21:22.948470+0000","last_fullsized":"2023-12-19T09:21:22.948470+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:11:00.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942676+0000","last_change":"2023-12-19T09:21:22.942676+0000","last_active":"2023-12-19T09:21:22.942676+0000","last_peered":"2023-12-19T09:21:22.942676+0000","last_clean":"2023-12-19T09:21:22.942676+0000","last_became_active":"2023-12-19T09:21:22.942316+0000","last_became_peered":"2023-12-19T09:21:22.942316+0000","last_unstale":"2023-12-19T09:21:22.942676+0000","last_undegraded":"2023-12-19T09:21:22.942676+0000","last_fullsized":"2023-12-19T09:21:22.942676+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:58.870055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"207'6","reported_seq":16,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.953576+0000","last_change":"2023-12-19T09:21:21.919056+0000","last_active":"2023-12-19T09:21:22.953576+0000","last_peered":"2023-12-19T09:21:22.953576+0000","last_clean":"2023-12-19T09:21:22.953576+0000","last_became_active":"2023-12-19T09:21:21.918781+0000","last_became_peered":"2023-12-19T09:21:21.918781+0000","last_unstale":"2023-12-19T09:21:22.953576+0000","last_undegraded":"2023-12-19T09:21:22.953576+0000","last_fullsized":"2023-12-19T09:21:22.953576+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:12.675270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927979+0000","last_change":"2023-12-19T09:21:22.927979+0000","last_active":"2023-12-19T09:21:22.927979+0000","last_peered":"2023-12-19T09:21:22.927979+0000","last_clean":"2023-12-19T09:21:22.927979+0000","last_became_active":"2023-12-19T09:21:22.927654+0000","last_became_peered":"2023-12-19T09:21:22.927654+0000","last_unstale":"2023-12-19T09:21:22.927979+0000","last_undegraded":"2023-12-19T09:21:22.927979+0000","last_fullsized":"2023-12-19T09:21:22.927979+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:16.311057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941313+0000","last_change":"2023-12-19T09:21:21.920618+0000","last_active":"2023-12-19T09:21:22.941313+0000","last_peered":"2023-12-19T09:21:22.941313+0000","last_clean":"2023-12-19T09:21:22.941313+0000","last_became_active":"2023-12-19T09:21:21.920313+0000","last_became_peered":"2023-12-19T09:21:21.920313+0000","last_unstale":"2023-12-19T09:21:22.941313+0000","last_undegraded":"2023-12-19T09:21:22.941313+0000","last_fullsized":"2023-12-19T09:21:22.941313+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:03.968366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.936308+0000","last_change":"2023-12-19T09:21:22.925304+0000","last_active":"2023-12-19T09:21:22.936308+0000","last_peered":"2023-12-19T09:21:22.936308+0000","last_clean":"2023-12-19T09:21:22.936308+0000","last_became_active":"2023-12-19T09:21:22.923624+0000","last_became_peered":"2023-12-19T09:21:22.923624+0000","last_unstale":"2023-12-19T09:21:22.936308+0000","last_undegraded":"2023-12-19T09:21:22.936308+0000","last_fullsized":"2023-12-19T09:21:22.936308+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:11.298724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.919281+0000","last_change":"2023-12-19T09:21:21.916368+0000","last_active":"2023-12-19T09:21:21.919281+0000","last_peered":"2023-12-19T09:21:21.919281+0000","last_clean":"2023-12-19T09:21:21.919281+0000","last_became_active":"2023-12-19T09:21:21.916054+0000","last_became_peered":"2023-12-19T09:21:21.916054+0000","last_unstale":"2023-12-19T09:21:21.919281+0000","last_undegraded":"2023-12-19T09:21:21.919281+0000","last_fullsized":"2023-12-19T09:21:21.919281+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:24.820042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941090+0000","last_change":"2023-12-19T09:21:22.926505+0000","last_active":"2023-12-19T09:21:22.941090+0000","last_peered":"2023-12-19T09:21:22.941090+0000","last_clean":"2023-12-19T09:21:22.941090+0000","last_became_active":"2023-12-19T09:21:22.925568+0000","last_became_peered":"2023-12-19T09:21:22.925568+0000","last_unstale":"2023-12-19T09:21:22.941090+0000","last_undegraded":"2023-12-19T09:21:22.941090+0000","last_fullsized":"2023-12-19T09:21:22.941090+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:16:24.010684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921525+0000","last_change":"2023-12-19T09:21:21.918155+0000","last_active":"2023-12-19T09:21:21.921525+0000","last_peered":"2023-12-19T09:21:21.921525+0000","last_clean":"2023-12-19T09:21:21.921525+0000","last_became_active":"2023-12-19T09:21:21.915613+0000","last_became_peered":"2023-12-19T09:21:21.915613+0000","last_unstale":"2023-12-19T09:21:21.921525+0000","last_undegraded":"2023-12-19T09:21:21.921525+0000","last_fullsized":"2023-12-19T09:21:21.921525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:14.605229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942217+0000","last_change":"2023-12-19T09:21:22.935970+0000","last_active":"2023-12-19T09:21:22.942217+0000","last_peered":"2023-12-19T09:21:22.942217+0000","last_clean":"2023-12-19T09:21:22.942217+0000","last_became_active":"2023-12-19T09:21:22.935686+0000","last_became_peered":"2023-12-19T09:21:22.935686+0000","last_unstale":"2023-12-19T09:21:22.942217+0000","last_undegraded":"2023-12-19T09:21:22.942217+0000","last_fullsized":"2023-12-19T09:21:22.942217+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:25.832753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"207'8","reported_seq":17,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.945225+0000","last_change":"2023-12-19T09:21:21.919360+0000","last_active":"2023-12-19T09:21:22.945225+0000","last_peered":"2023-12-19T09:21:22.945225+0000","last_clean":"2023-12-19T09:21:22.945225+0000","last_became_active":"2023-12-19T09:21:21.919084+0000","last_became_peered":"2023-12-19T09:21:21.919084+0000","last_unstale":"2023-12-19T09:21:22.945225+0000","last_undegraded":"2023-12-19T09:21:22.945225+0000","last_fullsized":"2023-12-19T09:21:22.945225+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:37.685945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.930816+0000","last_change":"2023-12-19T09:21:22.930816+0000","last_active":"2023-12-19T09:21:22.930816+0000","last_peered":"2023-12-19T09:21:22.930816+0000","last_clean":"2023-12-19T09:21:22.930816+0000","last_became_active":"2023-12-19T09:21:22.930485+0000","last_became_peered":"2023-12-19T09:21:22.930485+0000","last_unstale":"2023-12-19T09:21:22.930816+0000","last_undegraded":"2023-12-19T09:21:22.930816+0000","last_fullsized":"2023-12-19T09:21:22.930816+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:51:51.527655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.913661+0000","last_change":"2023-12-19T09:21:21.923101+0000","last_active":"2023-12-19T09:21:22.913661+0000","last_peered":"2023-12-19T09:21:22.913661+0000","last_clean":"2023-12-19T09:21:22.913661+0000","last_became_active":"2023-12-19T09:21:21.922751+0000","last_became_peered":"2023-12-19T09:21:21.922751+0000","last_unstale":"2023-12-19T09:21:22.913661+0000","last_undegraded":"2023-12-19T09:21:22.913661+0000","last_fullsized":"2023-12-19T09:21:22.913661+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:36:06.201695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946763+0000","last_change":"2023-12-19T09:21:22.933229+0000","last_active":"2023-12-19T09:21:22.946763+0000","last_peered":"2023-12-19T09:21:22.946763+0000","last_clean":"2023-12-19T09:21:22.946763+0000","last_became_active":"2023-12-19T09:21:22.932910+0000","last_became_peered":"2023-12-19T09:21:22.932910+0000","last_unstale":"2023-12-19T09:21:22.946763+0000","last_undegraded":"2023-12-19T09:21:22.946763+0000","last_fullsized":"2023-12-19T09:21:22.946763+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:31:42.698439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.944100+0000","last_change":"2023-12-19T09:21:21.918258+0000","last_active":"2023-12-19T09:21:22.944100+0000","last_peered":"2023-12-19T09:21:22.944100+0000","last_clean":"2023-12-19T09:21:22.944100+0000","last_became_active":"2023-12-19T09:21:21.915644+0000","last_became_peered":"2023-12-19T09:21:21.915644+0000","last_unstale":"2023-12-19T09:21:22.944100+0000","last_undegraded":"2023-12-19T09:21:22.944100+0000","last_fullsized":"2023-12-19T09:21:22.944100+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:07.032392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.923553+0000","last_change":"2023-12-19T09:21:22.923553+0000","last_active":"2023-12-19T09:21:22.923553+0000","last_peered":"2023-12-19T09:21:22.923553+0000","last_clean":"2023-12-19T09:21:22.923553+0000","last_became_active":"2023-12-19T09:21:22.922683+0000","last_became_peered":"2023-12-19T09:21:22.922683+0000","last_unstale":"2023-12-19T09:21:22.923553+0000","last_undegraded":"2023-12-19T09:21:22.923553+0000","last_fullsized":"2023-12-19T09:21:22.923553+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:00.796583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.916821+0000","last_change":"2023-12-19T09:21:21.930923+0000","last_active":"2023-12-19T09:21:22.916821+0000","last_peered":"2023-12-19T09:21:22.916821+0000","last_clean":"2023-12-19T09:21:22.916821+0000","last_became_active":"2023-12-19T09:21:21.930551+0000","last_became_peered":"2023-12-19T09:21:21.930551+0000","last_unstale":"2023-12-19T09:21:22.916821+0000","last_undegraded":"2023-12-19T09:21:22.916821+0000","last_fullsized":"2023-12-19T09:21:22.916821+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:10:29.302289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929956+0000","last_change":"2023-12-19T09:21:22.929956+0000","last_active":"2023-12-19T09:21:22.929956+0000","last_peered":"2023-12-19T09:21:22.929956+0000","last_clean":"2023-12-19T09:21:22.929956+0000","last_became_active":"2023-12-19T09:21:22.927210+0000","last_became_peered":"2023-12-19T09:21:22.927210+0000","last_unstale":"2023-12-19T09:21:22.929956+0000","last_undegraded":"2023-12-19T09:21:22.929956+0000","last_fullsized":"2023-12-19T09:21:22.929956+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:49:12.990854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.920974+0000","last_change":"2023-12-19T09:21:21.916685+0000","last_active":"2023-12-19T09:21:21.920974+0000","last_peered":"2023-12-19T09:21:21.920974+0000","last_clean":"2023-12-19T09:21:21.920974+0000","last_became_active":"2023-12-19T09:21:21.916250+0000","last_became_peered":"2023-12-19T09:21:21.916250+0000","last_unstale":"2023-12-19T09:21:21.920974+0000","last_undegraded":"2023-12-19T09:21:21.920974+0000","last_fullsized":"2023-12-19T09:21:21.920974+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:42.812210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927595+0000","last_change":"2023-12-19T09:21:22.927595+0000","last_active":"2023-12-19T09:21:22.927595+0000","last_peered":"2023-12-19T09:21:22.927595+0000","last_clean":"2023-12-19T09:21:22.927595+0000","last_became_active":"2023-12-19T09:21:22.927190+0000","last_became_peered":"2023-12-19T09:21:22.927190+0000","last_unstale":"2023-12-19T09:21:22.927595+0000","last_undegraded":"2023-12-19T09:21:22.927595+0000","last_fullsized":"2023-12-19T09:21:22.927595+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:08:21.890500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941024+0000","last_change":"2023-12-19T09:21:21.929027+0000","last_active":"2023-12-19T09:21:22.941024+0000","last_peered":"2023-12-19T09:21:22.941024+0000","last_clean":"2023-12-19T09:21:22.941024+0000","last_became_active":"2023-12-19T09:21:21.928744+0000","last_became_peered":"2023-12-19T09:21:21.928744+0000","last_unstale":"2023-12-19T09:21:22.941024+0000","last_undegraded":"2023-12-19T09:21:22.941024+0000","last_fullsized":"2023-12-19T09:21:22.941024+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:13.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929727+0000","last_change":"2023-12-19T09:21:22.929727+0000","last_active":"2023-12-19T09:21:22.929727+0000","last_peered":"2023-12-19T09:21:22.929727+0000","last_clean":"2023-12-19T09:21:22.929727+0000","last_became_active":"2023-12-19T09:21:22.927761+0000","last_became_peered":"2023-12-19T09:21:22.927761+0000","last_unstale":"2023-12-19T09:21:22.929727+0000","last_undegraded":"2023-12-19T09:21:22.929727+0000","last_fullsized":"2023-12-19T09:21:22.929727+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:58.963749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.922380+0000","last_change":"2023-12-19T09:21:21.920440+0000","last_active":"2023-12-19T09:21:21.922380+0000","last_peered":"2023-12-19T09:21:21.922380+0000","last_clean":"2023-12-19T09:21:21.922380+0000","last_became_active":"2023-12-19T09:21:21.920169+0000","last_became_peered":"2023-12-19T09:21:21.920169+0000","last_unstale":"2023-12-19T09:21:21.922380+0000","last_undegraded":"2023-12-19T09:21:21.922380+0000","last_fullsized":"2023-12-19T09:21:21.922380+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:35.044166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.923468+0000","last_change":"2023-12-19T09:21:22.923468+0000","last_active":"2023-12-19T09:21:22.923468+0000","last_peered":"2023-12-19T09:21:22.923468+0000","last_clean":"2023-12-19T09:21:22.923468+0000","last_became_active":"2023-12-19T09:21:22.923170+0000","last_became_peered":"2023-12-19T09:21:22.923170+0000","last_unstale":"2023-12-19T09:21:22.923468+0000","last_undegraded":"2023-12-19T09:21:22.923468+0000","last_fullsized":"2023-12-19T09:21:22.923468+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:28:50.478345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.915763+0000","last_change":"2023-12-19T09:21:21.910018+0000","last_active":"2023-12-19T09:21:21.915763+0000","last_peered":"2023-12-19T09:21:21.915763+0000","last_clean":"2023-12-19T09:21:21.915763+0000","last_became_active":"2023-12-19T09:21:21.909718+0000","last_became_peered":"2023-12-19T09:21:21.909718+0000","last_unstale":"2023-12-19T09:21:21.915763+0000","last_undegraded":"2023-12-19T09:21:21.915763+0000","last_fullsized":"2023-12-19T09:21:21.915763+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:15.425555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.936341+0000","last_change":"2023-12-19T09:21:22.925665+0000","last_active":"2023-12-19T09:21:22.936341+0000","last_peered":"2023-12-19T09:21:22.936341+0000","last_clean":"2023-12-19T09:21:22.936341+0000","last_became_active":"2023-12-19T09:21:22.925396+0000","last_became_peered":"2023-12-19T09:21:22.925396+0000","last_unstale":"2023-12-19T09:21:22.936341+0000","last_undegraded":"2023-12-19T09:21:22.936341+0000","last_fullsized":"2023-12-19T09:21:22.936341+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:06:35.945108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921940+0000","last_change":"2023-12-19T09:21:21.920478+0000","last_active":"2023-12-19T09:21:21.921940+0000","last_peered":"2023-12-19T09:21:21.921940+0000","last_clean":"2023-12-19T09:21:21.921940+0000","last_became_active":"2023-12-19T09:21:21.918501+0000","last_became_peered":"2023-12-19T09:21:21.918501+0000","last_unstale":"2023-12-19T09:21:21.921940+0000","last_undegraded":"2023-12-19T09:21:21.921940+0000","last_fullsized":"2023-12-19T09:21:21.921940+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:02.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943169+0000","last_change":"2023-12-19T09:21:22.943169+0000","last_active":"2023-12-19T09:21:22.943169+0000","last_peered":"2023-12-19T09:21:22.943169+0000","last_clean":"2023-12-19T09:21:22.943169+0000","last_became_active":"2023-12-19T09:21:22.942856+0000","last_became_peered":"2023-12-19T09:21:22.942856+0000","last_unstale":"2023-12-19T09:21:22.943169+0000","last_undegraded":"2023-12-19T09:21:22.943169+0000","last_fullsized":"2023-12-19T09:21:22.943169+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:58:24.989141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947001+0000","last_change":"2023-12-19T09:21:21.917286+0000","last_active":"2023-12-19T09:21:22.947001+0000","last_peered":"2023-12-19T09:21:22.947001+0000","last_clean":"2023-12-19T09:21:22.947001+0000","last_became_active":"2023-12-19T09:21:21.916368+0000","last_became_peered":"2023-12-19T09:21:21.916368+0000","last_unstale":"2023-12-19T09:21:22.947001+0000","last_undegraded":"2023-12-19T09:21:22.947001+0000","last_fullsized":"2023-12-19T09:21:22.947001+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:45.069729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.935872+0000","last_change":"2023-12-19T09:21:22.935872+0000","last_active":"2023-12-19T09:21:22.935872+0000","last_peered":"2023-12-19T09:21:22.935872+0000","last_clean":"2023-12-19T09:21:22.935872+0000","last_became_active":"2023-12-19T09:21:22.935543+0000","last_became_peered":"2023-12-19T09:21:22.935543+0000","last_unstale":"2023-12-19T09:21:22.935872+0000","last_undegraded":"2023-12-19T09:21:22.935872+0000","last_fullsized":"2023-12-19T09:21:22.935872+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:21.450955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.919555+0000","last_change":"2023-12-19T09:21:21.916297+0000","last_active":"2023-12-19T09:21:21.919555+0000","last_peered":"2023-12-19T09:21:21.919555+0000","last_clean":"2023-12-19T09:21:21.919555+0000","last_became_active":"2023-12-19T09:21:21.916007+0000","last_became_peered":"2023-12-19T09:21:21.916007+0000","last_unstale":"2023-12-19T09:21:21.919555+0000","last_undegraded":"2023-12-19T09:21:21.919555+0000","last_fullsized":"2023-12-19T09:21:21.919555+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:19:05.707222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929663+0000","last_change":"2023-12-19T09:21:22.929663+0000","last_active":"2023-12-19T09:21:22.929663+0000","last_peered":"2023-12-19T09:21:22.929663+0000","last_clean":"2023-12-19T09:21:22.929663+0000","last_became_active":"2023-12-19T09:21:22.929393+0000","last_became_peered":"2023-12-19T09:21:22.929393+0000","last_unstale":"2023-12-19T09:21:22.929663+0000","last_undegraded":"2023-12-19T09:21:22.929663+0000","last_fullsized":"2023-12-19T09:21:22.929663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:18:14.451651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.911634+0000","last_change":"2023-12-19T09:21:21.924894+0000","last_active":"2023-12-19T09:21:22.911634+0000","last_peered":"2023-12-19T09:21:22.911634+0000","last_clean":"2023-12-19T09:21:22.911634+0000","last_became_active":"2023-12-19T09:21:21.924507+0000","last_became_peered":"2023-12-19T09:21:21.924507+0000","last_unstale":"2023-12-19T09:21:22.911634+0000","last_undegraded":"2023-12-19T09:21:22.911634+0000","last_fullsized":"2023-12-19T09:21:22.911634+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:20:55.444131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942690+0000","last_change":"2023-12-19T09:21:22.942690+0000","last_active":"2023-12-19T09:21:22.942690+0000","last_peered":"2023-12-19T09:21:22.942690+0000","last_clean":"2023-12-19T09:21:22.942690+0000","last_became_active":"2023-12-19T09:21:22.942279+0000","last_became_peered":"2023-12-19T09:21:22.942279+0000","last_unstale":"2023-12-19T09:21:22.942690+0000","last_undegraded":"2023-12-19T09:21:22.942690+0000","last_fullsized":"2023-12-19T09:21:22.942690+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:23.081488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.912879+0000","last_change":"2023-12-19T09:21:21.924446+0000","last_active":"2023-12-19T09:21:22.912879+0000","last_peered":"2023-12-19T09:21:22.912879+0000","last_clean":"2023-12-19T09:21:22.912879+0000","last_became_active":"2023-12-19T09:21:21.924119+0000","last_became_peered":"2023-12-19T09:21:21.924119+0000","last_unstale":"2023-12-19T09:21:22.912879+0000","last_undegraded":"2023-12-19T09:21:22.912879+0000","last_fullsized":"2023-12-19T09:21:22.912879+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:07.012619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.925072+0000","last_change":"2023-12-19T09:21:22.925072+0000","last_active":"2023-12-19T09:21:22.925072+0000","last_peered":"2023-12-19T09:21:22.925072+0000","last_clean":"2023-12-19T09:21:22.925072+0000","last_became_active":"2023-12-19T09:21:22.924813+0000","last_became_peered":"2023-12-19T09:21:22.924813+0000","last_unstale":"2023-12-19T09:21:22.925072+0000","last_undegraded":"2023-12-19T09:21:22.925072+0000","last_fullsized":"2023-12-19T09:21:22.925072+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:57:58.976484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.917426+0000","last_change":"2023-12-19T09:21:21.915520+0000","last_active":"2023-12-19T09:21:21.917426+0000","last_peered":"2023-12-19T09:21:21.917426+0000","last_clean":"2023-12-19T09:21:21.917426+0000","last_became_active":"2023-12-19T09:21:21.915173+0000","last_became_peered":"2023-12-19T09:21:21.915173+0000","last_unstale":"2023-12-19T09:21:21.917426+0000","last_undegraded":"2023-12-19T09:21:21.917426+0000","last_fullsized":"2023-12-19T09:21:21.917426+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:50.444069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942107+0000","last_change":"2023-12-19T09:21:22.929744+0000","last_active":"2023-12-19T09:21:22.942107+0000","last_peered":"2023-12-19T09:21:22.942107+0000","last_clean":"2023-12-19T09:21:22.942107+0000","last_became_active":"2023-12-19T09:21:22.929059+0000","last_became_peered":"2023-12-19T09:21:22.929059+0000","last_unstale":"2023-12-19T09:21:22.942107+0000","last_undegraded":"2023-12-19T09:21:22.942107+0000","last_fullsized":"2023-12-19T09:21:22.942107+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:38.698657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.926645+0000","last_change":"2023-12-19T09:21:21.925803+0000","last_active":"2023-12-19T09:21:21.926645+0000","last_peered":"2023-12-19T09:21:21.926645+0000","last_clean":"2023-12-19T09:21:21.926645+0000","last_became_active":"2023-12-19T09:21:21.925393+0000","last_became_peered":"2023-12-19T09:21:21.925393+0000","last_unstale":"2023-12-19T09:21:21.926645+0000","last_undegraded":"2023-12-19T09:21:21.926645+0000","last_fullsized":"2023-12-19T09:21:21.926645+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:45.623920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.930110+0000","last_change":"2023-12-19T09:21:22.930110+0000","last_active":"2023-12-19T09:21:22.930110+0000","last_peered":"2023-12-19T09:21:22.930110+0000","last_clean":"2023-12-19T09:21:22.930110+0000","last_became_active":"2023-12-19T09:21:22.928514+0000","last_became_peered":"2023-12-19T09:21:22.928514+0000","last_unstale":"2023-12-19T09:21:22.930110+0000","last_undegraded":"2023-12-19T09:21:22.930110+0000","last_fullsized":"2023-12-19T09:21:22.930110+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:28:50.478345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.918171+0000","last_change":"2023-12-19T09:21:21.911594+0000","last_active":"2023-12-19T09:21:21.918171+0000","last_peered":"2023-12-19T09:21:21.918171+0000","last_clean":"2023-12-19T09:21:21.918171+0000","last_became_active":"2023-12-19T09:21:21.911278+0000","last_became_peered":"2023-12-19T09:21:21.911278+0000","last_unstale":"2023-12-19T09:21:21.918171+0000","last_undegraded":"2023-12-19T09:21:21.918171+0000","last_fullsized":"2023-12-19T09:21:21.918171+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:22.556014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.934211+0000","last_change":"2023-12-19T09:21:22.934211+0000","last_active":"2023-12-19T09:21:22.934211+0000","last_peered":"2023-12-19T09:21:22.934211+0000","last_clean":"2023-12-19T09:21:22.934211+0000","last_became_active":"2023-12-19T09:21:22.933928+0000","last_became_peered":"2023-12-19T09:21:22.933928+0000","last_unstale":"2023-12-19T09:21:22.934211+0000","last_undegraded":"2023-12-19T09:21:22.934211+0000","last_fullsized":"2023-12-19T09:21:22.934211+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:08.025331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921414+0000","last_change":"2023-12-19T09:21:21.919344+0000","last_active":"2023-12-19T09:21:21.921414+0000","last_peered":"2023-12-19T09:21:21.921414+0000","last_clean":"2023-12-19T09:21:21.921414+0000","last_became_active":"2023-12-19T09:21:21.918365+0000","last_became_peered":"2023-12-19T09:21:21.918365+0000","last_unstale":"2023-12-19T09:21:21.921414+0000","last_undegraded":"2023-12-19T09:21:21.921414+0000","last_fullsized":"2023-12-19T09:21:21.921414+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:54.868064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.921251+0000","last_change":"2023-12-19T09:21:22.921251+0000","last_active":"2023-12-19T09:21:22.921251+0000","last_peered":"2023-12-19T09:21:22.921251+0000","last_clean":"2023-12-19T09:21:22.921251+0000","last_became_active":"2023-12-19T09:21:22.920844+0000","last_became_peered":"2023-12-19T09:21:22.920844+0000","last_unstale":"2023-12-19T09:21:22.921251+0000","last_undegraded":"2023-12-19T09:21:22.921251+0000","last_fullsized":"2023-12-19T09:21:22.921251+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:25.771789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921695+0000","last_change":"2023-12-19T09:21:21.917540+0000","last_active":"2023-12-19T09:21:21.921695+0000","last_peered":"2023-12-19T09:21:21.921695+0000","last_clean":"2023-12-19T09:21:21.921695+0000","last_became_active":"2023-12-19T09:21:21.916456+0000","last_became_peered":"2023-12-19T09:21:21.916456+0000","last_unstale":"2023-12-19T09:21:21.921695+0000","last_undegraded":"2023-12-19T09:21:21.921695+0000","last_fullsized":"2023-12-19T09:21:21.921695+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:34.312826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"197'348","reported_seq":1133,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.901678+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:21.901678+0000","last_peered":"2023-12-19T09:21:21.901678+0000","last_clean":"2023-12-19T09:21:21.901678+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:21.901678+0000","last_undegraded":"2023-12-19T09:21:21.901678+0000","last_fullsized":"2023-12-19T09:21:21.901678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304500,"kb_used_data":3116,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067340,"statfs":{"total":96636764160,"available":96324956160,"internally_reserved":0,"allocated":3190784,"data_stored":1985008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3009999999999999,"15min":1.2929999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.52600000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.2849999999999999,"15min":1.298},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.55700000000000005},"max":{"1min":1.5269999999999999,"5min":1.6759999999999999,"15min":1.712},"last":1.417}]},{"osd":1,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.169,"15min":1.1739999999999999},"min":{"1min":0.91000000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4990000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.206,"15min":1.212},"min":{"1min":0.85099999999999998,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6579999999999999,"5min":1.7769999999999999,"15min":1.796},"last":0.91400000000000003}]},{"osd":2,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1739999999999999,"15min":1.181},"min":{"1min":0.91500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.871},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2949999999999999,"15min":1.282},"min":{"1min":0.97499999999999998,"5min":0.79000000000000004,"15min":0.71999999999999997},"max":{"1min":1.5600000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":1.2030000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3240000000000001,"15min":1.3180000000000001},"min":{"1min":1.103,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.597,"5min":1.845,"15min":1.9059999999999999},"last":1.032},{"interface":"front","average":{"1min":1.337,"5min":1.371,"15min":1.381},"min":{"1min":0.95099999999999996,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.6919999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.1739999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.228,"15min":1.244},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77000000000000002},"max":{"1min":1.6870000000000001,"5min":1.7330000000000001,"15min":1.95},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.99099999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"max":{"1min":1.696,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.149}]},{"osd":5,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.2789999999999999,"15min":1.294},"min":{"1min":0.88200000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6459999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.4139999999999999,"15min":1.413},"min":{"1min":0.97999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.925,"5min":1.972,"15min":1.972},"last":1.27}]},{"osd":6,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4139999999999999,"15min":1.4179999999999999},"min":{"1min":1.1279999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8109999999999999,"5min":1.893,"15min":2.0750000000000002},"last":1.361},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.413,"15min":1.399},"min":{"1min":1.1479999999999999,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.833,"5min":1.9530000000000001,"15min":1.986},"last":1.234}]}]},{"osd":6,"up_from":13,"seq":55834575967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312116,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059724,"statfs":{"total":96636764160,"available":96317157376,"internally_reserved":0,"allocated":2600960,"data_stored":1394710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.165,"15min":1.1479999999999999},"min":{"1min":0.82499999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.516,"5min":1.6799999999999999,"15min":1.784},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.1839999999999999,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68000000000000005,"15min":0.55100000000000005},"max":{"1min":1.431,"5min":1.6850000000000001,"15min":1.946},"last":1.466}]},{"osd":1,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.77900000000000003,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.4730000000000001,"5min":1.603,"15min":2.3439999999999999},"last":1.506},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.94299999999999995,"5min":0.72499999999999998,"15min":0.59099999999999997},"max":{"1min":1.6259999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.387}]},{"osd":2,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2629999999999999,"15min":1.2549999999999999},"min":{"1min":1.0289999999999999,"5min":0.86899999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.417},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.2569999999999999},"min":{"1min":0.90300000000000002,"5min":0.88700000000000001,"15min":0.73999999999999999},"max":{"1min":1.8340000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.04}]},{"osd":3,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.94699999999999995,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.5720000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.641},{"interface":"front","average":{"1min":1.294,"5min":1.3360000000000001,"15min":1.3200000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68999999999999995},"max":{"1min":1.5620000000000001,"5min":1.74,"15min":1.8220000000000001},"last":1.373}]},{"osd":4,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.69999999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5760000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1},{"interface":"front","average":{"1min":1.407,"5min":1.448,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7250000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.552}]},{"osd":5,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2829999999999999,"15min":1.288},"min":{"1min":0.73299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.619,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.323,"15min":1.3049999999999999},"min":{"1min":1.034,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.7709999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3759999999999999,"15min":1.3859999999999999},"min":{"1min":0.97299999999999998,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.927,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.335},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7210000000000001,"5min":1.8899999999999999,"15min":2.2410000000000001},"last":1.22}]}]},{"osd":5,"up_from":13,"seq":55834575967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308032,"kb_used_data":2552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063808,"statfs":{"total":96636764160,"available":96321339392,"internally_reserved":0,"allocated":2613248,"data_stored":1396198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3069999999999999,"15min":1.3160000000000001},"min":{"1min":0.878,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.631,"5min":2.044,"15min":2.044},"last":1.369},{"interface":"front","average":{"1min":1.409,"5min":1.351,"15min":1.3520000000000001},"min":{"1min":1.006,"5min":0.84899999999999998,"15min":0.58199999999999996},"max":{"1min":1.994,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.4550000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1579999999999999,"15min":1.1759999999999999},"min":{"1min":0.83299999999999996,"5min":0.78800000000000003,"15min":0.76200000000000001},"max":{"1min":1.502,"5min":1.5569999999999999,"15min":1.6479999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.97899999999999998,"5min":0.74299999999999999,"15min":0.58099999999999996},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.8140000000000001},"last":1.911}]},{"osd":2,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.208,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.84699999999999998,"15min":0.76800000000000002},"max":{"1min":1.6319999999999999,"5min":1.782,"15min":1.782},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.236,"15min":1.2529999999999999},"min":{"1min":1.0489999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3169999999999999,"15min":1.3149999999999999},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.655},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.351,"15min":1.3320000000000001},"min":{"1min":1.111,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.6899999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2849999999999999,"15min":1.2450000000000001},"min":{"1min":1.2090000000000001,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3520000000000001,"15min":1.3160000000000001},"min":{"1min":1.0820000000000001,"5min":0.877,"15min":0.59099999999999997},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.542}]},{"osd":6,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3600000000000001,"15min":1.329},"min":{"1min":0.98499999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4059999999999999,"15min":1.385},"min":{"1min":0.99199999999999999,"5min":0.76500000000000001,"15min":0.68999999999999995},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.28}]},{"osd":7,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.472,"15min":1.4470000000000001},"min":{"1min":1.206,"5min":0.95799999999999996,"15min":0.84199999999999997},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.762},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.3,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.5920000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575967,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312140,"kb_used_data":2564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059700,"statfs":{"total":96636764160,"available":96317132800,"internally_reserved":0,"allocated":2625536,"data_stored":1403394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.169},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.6659999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":1.0349999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":1.5249999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":1.3},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.415}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.3939999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.49},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.8089999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.591}]}]},{"osd":3,"up_from":13,"seq":55834575967,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312132,"kb_used_data":2556,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059708,"statfs":{"total":96636764160,"available":96317140992,"internally_reserved":0,"allocated":2617344,"data_stored":1401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.89100000000000001,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.373,"5min":1.581,"15min":1.581},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55100000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3200000000000001,"5min":2.117,"15min":2.117},"last":1.0649999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.92800000000000005,"15min":0.93600000000000005},"min":{"1min":0.57099999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.512,"5min":2.331,"15min":2.331},"last":1.153},{"interface":"front","average":{"1min":0.999,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.61199999999999999,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.4550000000000001,"5min":2.23,"15min":2.23},"last":0.93999999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.101,"15min":1.1200000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4199999999999999,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1890000000000001},"min":{"1min":0.86299999999999999,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.8580000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.337,"15min":1.3169999999999999},"min":{"1min":0.86699999999999999,"5min":0.85799999999999998,"15min":0.70199999999999996},"max":{"1min":2.081,"5min":2.157,"15min":2.157},"last":1.327},{"interface":"front","average":{"1min":1.304,"5min":1.3049999999999999,"15min":1.284},"min":{"1min":0.93600000000000005,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.492}]},{"osd":5,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3100000000000001,"15min":1.306},"min":{"1min":1.0349999999999999,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.397,"15min":1.387},"min":{"1min":1.0429999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0630000000000002,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.198}]},{"osd":6,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.4139999999999999,"15min":1.421},"min":{"1min":0.89800000000000002,"5min":0.752,"15min":0.752},"max":{"1min":2.004,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.47},{"interface":"front","average":{"1min":1.401,"5min":1.4139999999999999,"15min":1.4239999999999999},"min":{"1min":0.99399999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":2.0289999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.5489999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5069999999999999,"15min":1.5},"min":{"1min":1.006,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":2.3780000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4610000000000001,"15min":1.4510000000000001},"min":{"1min":0.96999999999999997,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0129999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.581}]}]},{"osd":2,"up_from":13,"seq":55834575967,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308020,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063820,"statfs":{"total":96636764160,"available":96321351680,"internally_reserved":0,"allocated":2600960,"data_stored":1395194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2529999999999999,"5min":1.345,"15min":1.5049999999999999},"last":0.877},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.286,"5min":1.556,"15min":1.6759999999999999},"last":1.012}]},{"osd":1,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3640000000000001,"5min":1.569,"15min":1.7609999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.034,"15min":1.044},"min":{"1min":0.59499999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4339999999999999,"5min":1.5369999999999999,"15min":1.768},"last":1.3280000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.472,"5min":1.53,"15min":1.891},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.121,"5min":1.1359999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":1.302}]},{"osd":4,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.341},"min":{"1min":0.76900000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.754},"last":1.222},{"interface":"front","average":{"1min":1.242,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.556,"5min":1.76,"15min":1.8520000000000001},"last":1.5529999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3759999999999999},"min":{"1min":0.84799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6679999999999999,"5min":1.948,"15min":2.016},"last":1.468},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.91200000000000003,"5min":0.83399999999999996,"15min":0.68300000000000005},"max":{"1min":1.6000000000000001,"5min":1.901,"15min":1.927},"last":1.6599999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3120000000000001,"15min":1.345},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8200000000000001,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.738},{"interface":"front","average":{"1min":1.379,"5min":1.365,"15min":1.419},"min":{"1min":1.036,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.516}]},{"osd":7,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.4059999999999999,"15min":1.4370000000000001},"min":{"1min":0.94799999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.0489999999999999,"15min":2.327},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9059999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.802}]}]},{"osd":1,"up_from":13,"seq":55834575968,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312112,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059728,"statfs":{"total":96636764160,"available":96317161472,"internally_reserved":0,"allocated":2596864,"data_stored":1394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86299999999999999,"15min":0.87},"min":{"1min":0.45200000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.258,"5min":1.4239999999999999,"15min":1.9039999999999999},"last":1.028},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94899999999999995,"15min":0.997},"min":{"1min":0.59699999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.24,"5min":1.575,"15min":1.6850000000000001},"last":0.63}]},{"osd":2,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.95699999999999996,"15min":0.95999999999999996},"min":{"1min":0.66200000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0269999999999999,"15min":1.0669999999999999},"min":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3740000000000001,"5min":1.5760000000000001,"15min":2.302},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1459999999999999,"15min":1.173},"min":{"1min":0.68100000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.4219999999999999,"5min":1.8899999999999999,"15min":2.0950000000000002},"last":1.4470000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0580000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.101}]},{"osd":4,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2549999999999999,"15min":1.252},"min":{"1min":0.96399999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.619,"5min":1.784,"15min":1.784},"last":1.252},{"interface":"front","average":{"1min":1.284,"5min":1.3109999999999999,"15min":1.3260000000000001},"min":{"1min":0.996,"5min":0.90500000000000003,"15min":0.83899999999999997},"max":{"1min":1.5,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":1.528}]},{"osd":5,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.298,"15min":1.333},"min":{"1min":1.0609999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.661,"5min":1.944,"15min":1.944},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3220000000000001,"15min":1.3129999999999999},"min":{"1min":0.93000000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.706,"5min":1.8779999999999999,"15min":2.117},"last":1.2110000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3899999999999999,"15min":1.4059999999999999},"min":{"1min":0.98099999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.8200000000000001,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.381,"15min":1.419},"min":{"1min":1.1719999999999999,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.3999999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4490000000000001,"15min":1.458},"min":{"1min":1.2210000000000001,"5min":1.01,"15min":1.0029999999999999},"max":{"1min":1.806,"5min":1.806,"15min":2.069},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4279999999999999,"15min":1.4339999999999999},"min":{"1min":1.147,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.736,"5min":1.968,"15min":2.2360000000000002},"last":1.169}]}]},{"osd":0,"up_from":13,"seq":55834575967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300416,"kb_used_data":3128,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071424,"statfs":{"total":96636764160,"available":96329138176,"internally_reserved":0,"allocated":3203072,"data_stored":1984697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.85299999999999998,"15min":0.877},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.242,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.96399999999999997,"15min":0.97299999999999998},"min":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.50800000000000001},"max":{"1min":1.484,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.77000000000000002}]},{"osd":2,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.0029999999999999,"15min":1.0269999999999999},"min":{"1min":0.71499999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.4419999999999999,"5min":1.446,"15min":1.5269999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.0660000000000001},"min":{"1min":0.71599999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.627,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.877}]},{"osd":3,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.026,"15min":1.0129999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.622},"max":{"1min":1.3740000000000001,"5min":1.611,"15min":2.621},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.1599999999999999,"15min":1.177},"min":{"1min":0.75600000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.718},"last":1.024}]},{"osd":4,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.357,"15min":1.3799999999999999},"min":{"1min":0.80900000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6519999999999999,"5min":1.891,"15min":1.891},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.337,"15min":1.3440000000000001},"min":{"1min":0.875,"5min":0.80300000000000005,"15min":0.69599999999999995},"max":{"1min":1.8560000000000001,"5min":1.919,"15min":2.5339999999999998},"last":1.381}]},{"osd":5,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.391,"15min":1.399},"min":{"1min":0.99299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8100000000000001,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4219999999999999,"15min":1.417},"min":{"1min":1.1399999999999999,"5min":0.83499999999999996,"15min":0.79500000000000004},"max":{"1min":1.615,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.329}]},{"osd":6,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.294,"15min":1.278},"min":{"1min":0.98899999999999999,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":1.772,"5min":2.0840000000000001,"15min":2.4489999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.399,"15min":1.371},"min":{"1min":1.054,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.736,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.0609999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3460000000000001,"15min":1.343},"min":{"1min":1.0509999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.6080000000000001,"5min":2.137,"15min":2.8570000000000002},"last":1.111},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4359999999999999,"15min":1.4319999999999999},"min":{"1min":1.0289999999999999,"5min":0.92200000000000004,"15min":0.84399999999999997},"max":{"1min":1.708,"5min":2.1259999999999999,"15min":2.181},"last":1.1359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:21:27.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:21:27.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.391+0000 7f6d3ca78700 1 Processor -- start 2023-12-19T09:21:27.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 -- start start 2023-12-19T09:21:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d3815d320 0x7f6d3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d380c15b0 0x7f6d380aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d380c7190 con 0x7f6d3806d690 2023-12-19T09:21:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d380c1b50 con 0x7f6d3815d320 2023-12-19T09:21:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.395+0000 7f6d3ca78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c1e30 con 0x7f6d380c15b0 2023-12-19T09:21:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d36d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6d3806d690 0x7f6d380680f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46472/0 (socket says 172.21.15.111:46472) 2023-12-19T09:21:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d36d9d700 1 -- 172.21.15.111:0/3016896994 learned_addr learned my addr 172.21.15.111:0/3016896994 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 --2- 172.21.15.111:0/3016896994 >> v2:172.21.15.143:3300/0 conn(0x7f6d3815d320 0x7f6d3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d2ffff700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d380c15b0 0x7f6d380aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d35d9b700 1 -- 172.21.15.111:0/3016896994 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304600166 0 0) 0x7f6d380c7190 con 0x7f6d3806d690 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d35d9b700 1 -- 172.21.15.111:0/3016896994 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d24003570 con 0x7f6d3806d690 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 -- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d380c15b0 msgr2=0x7f6d380aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d380c15b0 0x7f6d380aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 -- 172.21.15.111:0/3016896994 >> v1:172.21.15.111:6789/0 conn(0x7f6d3806d690 legacy=0x7f6d380680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 -- 172.21.15.111:0/3016896994 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380c6e20 con 0x7f6d3815d320 2023-12-19T09:21:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3659c700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3815d320 0x7f6d3815d6f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d280098e0 tx=0x7f6d280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b97f47e9c09a531b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d35d9b700 1 -- 172.21.15.111:0/3016896994 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28011630 con 0x7f6d3815d320 2023-12-19T09:21:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d35d9b700 1 -- 172.21.15.111:0/3016896994 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d28011790 con 0x7f6d3815d320 2023-12-19T09:21:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d35d9b700 1 -- 172.21.15.111:0/3016896994 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28011b10 con 0x7f6d3815d320 2023-12-19T09:21:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 -- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3815d320 msgr2=0x7f6d3815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3815d320 0x7f6d3815d6f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d280098e0 tx=0x7f6d280091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 -- 172.21.15.111:0/3016896994 shutdown_connections 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d380c15b0 0x7f6d380aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/3016896994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3815d320 0x7f6d3815d6f0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 -- 172.21.15.111:0/3016896994 >> 172.21.15.111:0/3016896994 conn(0x7f6d380c8850 msgr2=0x7f6d380c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 -- 172.21.15.111:0/3016896994 shutdown_connections 2023-12-19T09:21:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.399+0000 7f6d3ca78700 1 -- 172.21.15.111:0/3016896994 wait complete. 2023-12-19T09:21:27.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 Processor -- start 2023-12-19T09:21:27.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 -- start start 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 0x7f6d3816d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d3815d320 0x7f6d380b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c4e90 con 0x7f6d380c15b0 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c4280 con 0x7f6d3806d690 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3ca78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c4560 con 0x7f6d3815d320 2023-12-19T09:21:27.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40430/0 (socket says 172.21.15.111:40430) 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 -- 172.21.15.111:0/1817882267 learned_addr learned my addr 172.21.15.111:0/1817882267 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3659c700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 0x7f6d3816d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d36d9d700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d3815d320 0x7f6d380b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 -- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d3815d320 msgr2=0x7f6d380b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d3815d320 0x7f6d380b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 -- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 msgr2=0x7f6d3816d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 0x7f6d3816d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d2ffff700 1 -- 172.21.15.111:0/1817882267 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28009420 con 0x7f6d380c15b0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.403+0000 7f6d3659c700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 0x7f6d3816d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d2ffff700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000ec80 tx=0x7f6d200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2000c9d0 con 0x7f6d380c15b0 2023-12-19T09:21:27.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2000cb30 con 0x7f6d380c15b0 2023-12-19T09:21:27.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20013470 con 0x7f6d380c15b0 2023-12-19T09:21:27.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d380c4840 con 0x7f6d380c15b0 2023-12-19T09:21:27.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3816db70 con 0x7f6d380c15b0 2023-12-19T09:21:27.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.407+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d380c5000 con 0x7f6d380c15b0 2023-12-19T09:21:27.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.411+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6d20013820 con 0x7f6d380c15b0 2023-12-19T09:21:27.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.411+0000 7f6d2e7fc700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 0x7f6d18062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.411+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f6d2007ce90 con 0x7f6d380c15b0 2023-12-19T09:21:27.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.411+0000 7f6d3659c700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 0x7f6d18062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.415+0000 7f6d3659c700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 0x7f6d18062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6d280091e0 tx=0x7f6d28017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.419+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6d2004d100 con 0x7f6d380c15b0 2023-12-19T09:21:27.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.587+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d38071120 con 0x7f6d18060550 2023-12-19T09:21:27.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.607+0000 7f6d2e7fc700 1 -- 172.21.15.111:0/1817882267 <== mgr.8642 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287252 (secure 0 0 0) 0x7f6d38071120 con 0x7f6d18060550 2023-12-19T09:21:27.620 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 msgr2=0x7f6d18062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 0x7f6d18062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6d280091e0 tx=0x7f6d28017000 comp rx=0 tx=0).stop 2023-12-19T09:21:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 msgr2=0x7f6d38170300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000ec80 tx=0x7f6d200091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 shutdown_connections 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d3815d320 0x7f6d380b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d380c15b0 0x7f6d38170300 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6d18060550 0x7f6d18062a00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 --2- 172.21.15.111:0/1817882267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d3806d690 0x7f6d3816d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 >> 172.21.15.111:0/1817882267 conn(0x7f6d380c8850 msgr2=0x7f6d380cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 shutdown_connections 2023-12-19T09:21:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.615+0000 7f6d3ca78700 1 -- 172.21.15.111:0/1817882267 wait complete. 2023-12-19T09:21:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:21:27.643 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-19T09:21:26.640528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2469468,"kb_used_data":21532,"kb_used_omap":0,"kb_used_meta":2447872,"kb_avail":752505252,"statfs":{"total":773094113280,"available":770565378048,"internally_reserved":0,"allocated":22048768,"data_stored":12355825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506620928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.760707"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938889+0000","last_change":"2023-12-19T09:21:22.938889+0000","last_active":"2023-12-19T09:21:22.938889+0000","last_peered":"2023-12-19T09:21:22.938889+0000","last_clean":"2023-12-19T09:21:22.938889+0000","last_became_active":"2023-12-19T09:21:22.938567+0000","last_became_peered":"2023-12-19T09:21:22.938567+0000","last_unstale":"2023-12-19T09:21:22.938889+0000","last_undegraded":"2023-12-19T09:21:22.938889+0000","last_fullsized":"2023-12-19T09:21:22.938889+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:50:09.431525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938587+0000","last_change":"2023-12-19T09:21:22.938587+0000","last_active":"2023-12-19T09:21:22.938587+0000","last_peered":"2023-12-19T09:21:22.938587+0000","last_clean":"2023-12-19T09:21:22.938587+0000","last_became_active":"2023-12-19T09:21:22.938277+0000","last_became_peered":"2023-12-19T09:21:22.938277+0000","last_unstale":"2023-12-19T09:21:22.938587+0000","last_undegraded":"2023-12-19T09:21:22.938587+0000","last_fullsized":"2023-12-19T09:21:22.938587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:04.070678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938450+0000","last_change":"2023-12-19T09:21:22.938450+0000","last_active":"2023-12-19T09:21:22.938450+0000","last_peered":"2023-12-19T09:21:22.938450+0000","last_clean":"2023-12-19T09:21:22.938450+0000","last_became_active":"2023-12-19T09:21:22.936757+0000","last_became_peered":"2023-12-19T09:21:22.936757+0000","last_unstale":"2023-12-19T09:21:22.938450+0000","last_undegraded":"2023-12-19T09:21:22.938450+0000","last_fullsized":"2023-12-19T09:21:22.938450+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:51:43.091347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931559+0000","last_change":"2023-12-19T09:21:22.931559+0000","last_active":"2023-12-19T09:21:22.931559+0000","last_peered":"2023-12-19T09:21:22.931559+0000","last_clean":"2023-12-19T09:21:22.931559+0000","last_became_active":"2023-12-19T09:21:22.931170+0000","last_became_peered":"2023-12-19T09:21:22.931170+0000","last_unstale":"2023-12-19T09:21:22.931559+0000","last_undegraded":"2023-12-19T09:21:22.931559+0000","last_fullsized":"2023-12-19T09:21:22.931559+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:30:13.652617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948114+0000","last_change":"2023-12-19T09:21:22.939676+0000","last_active":"2023-12-19T09:21:22.948114+0000","last_peered":"2023-12-19T09:21:22.948114+0000","last_clean":"2023-12-19T09:21:22.948114+0000","last_became_active":"2023-12-19T09:21:22.939365+0000","last_became_peered":"2023-12-19T09:21:22.939365+0000","last_unstale":"2023-12-19T09:21:22.948114+0000","last_undegraded":"2023-12-19T09:21:22.948114+0000","last_fullsized":"2023-12-19T09:21:22.948114+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:04:50.596361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947073+0000","last_change":"2023-12-19T09:21:22.935045+0000","last_active":"2023-12-19T09:21:22.947073+0000","last_peered":"2023-12-19T09:21:22.947073+0000","last_clean":"2023-12-19T09:21:22.947073+0000","last_became_active":"2023-12-19T09:21:22.934776+0000","last_became_peered":"2023-12-19T09:21:22.934776+0000","last_unstale":"2023-12-19T09:21:22.947073+0000","last_undegraded":"2023-12-19T09:21:22.947073+0000","last_fullsized":"2023-12-19T09:21:22.947073+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:24:36.778946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.928927+0000","last_change":"2023-12-19T09:21:22.928927+0000","last_active":"2023-12-19T09:21:22.928927+0000","last_peered":"2023-12-19T09:21:22.928927+0000","last_clean":"2023-12-19T09:21:22.928927+0000","last_became_active":"2023-12-19T09:21:22.925981+0000","last_became_peered":"2023-12-19T09:21:22.925981+0000","last_unstale":"2023-12-19T09:21:22.928927+0000","last_undegraded":"2023-12-19T09:21:22.928927+0000","last_fullsized":"2023-12-19T09:21:22.928927+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:55.880776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943995+0000","last_change":"2023-12-19T09:21:22.943995+0000","last_active":"2023-12-19T09:21:22.943995+0000","last_peered":"2023-12-19T09:21:22.943995+0000","last_clean":"2023-12-19T09:21:22.943995+0000","last_became_active":"2023-12-19T09:21:22.943155+0000","last_became_peered":"2023-12-19T09:21:22.943155+0000","last_unstale":"2023-12-19T09:21:22.943995+0000","last_undegraded":"2023-12-19T09:21:22.943995+0000","last_fullsized":"2023-12-19T09:21:22.943995+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:42.203249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.932430+0000","last_change":"2023-12-19T09:21:22.932430+0000","last_active":"2023-12-19T09:21:22.932430+0000","last_peered":"2023-12-19T09:21:22.932430+0000","last_clean":"2023-12-19T09:21:22.932430+0000","last_became_active":"2023-12-19T09:21:22.930436+0000","last_became_peered":"2023-12-19T09:21:22.930436+0000","last_unstale":"2023-12-19T09:21:22.932430+0000","last_undegraded":"2023-12-19T09:21:22.932430+0000","last_fullsized":"2023-12-19T09:21:22.932430+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:30:13.652617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948221+0000","last_change":"2023-12-19T09:21:22.942518+0000","last_active":"2023-12-19T09:21:22.948221+0000","last_peered":"2023-12-19T09:21:22.948221+0000","last_clean":"2023-12-19T09:21:22.948221+0000","last_became_active":"2023-12-19T09:21:22.942219+0000","last_became_peered":"2023-12-19T09:21:22.942219+0000","last_unstale":"2023-12-19T09:21:22.948221+0000","last_undegraded":"2023-12-19T09:21:22.948221+0000","last_fullsized":"2023-12-19T09:21:22.948221+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:04.977368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940153+0000","last_change":"2023-12-19T09:21:22.940153+0000","last_active":"2023-12-19T09:21:22.940153+0000","last_peered":"2023-12-19T09:21:22.940153+0000","last_clean":"2023-12-19T09:21:22.940153+0000","last_became_active":"2023-12-19T09:21:22.939876+0000","last_became_peered":"2023-12-19T09:21:22.939876+0000","last_unstale":"2023-12-19T09:21:22.940153+0000","last_undegraded":"2023-12-19T09:21:22.940153+0000","last_fullsized":"2023-12-19T09:21:22.940153+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:15.617941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.922251+0000","last_change":"2023-12-19T09:21:22.922251+0000","last_active":"2023-12-19T09:21:22.922251+0000","last_peered":"2023-12-19T09:21:22.922251+0000","last_clean":"2023-12-19T09:21:22.922251+0000","last_became_active":"2023-12-19T09:21:22.921922+0000","last_became_peered":"2023-12-19T09:21:22.921922+0000","last_unstale":"2023-12-19T09:21:22.922251+0000","last_undegraded":"2023-12-19T09:21:22.922251+0000","last_fullsized":"2023-12-19T09:21:22.922251+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:49:12.990854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942195+0000","last_change":"2023-12-19T09:21:22.930133+0000","last_active":"2023-12-19T09:21:22.942195+0000","last_peered":"2023-12-19T09:21:22.942195+0000","last_clean":"2023-12-19T09:21:22.942195+0000","last_became_active":"2023-12-19T09:21:22.929778+0000","last_became_peered":"2023-12-19T09:21:22.929778+0000","last_unstale":"2023-12-19T09:21:22.942195+0000","last_undegraded":"2023-12-19T09:21:22.942195+0000","last_fullsized":"2023-12-19T09:21:22.942195+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:19:11.108096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.945131+0000","last_change":"2023-12-19T09:21:22.945131+0000","last_active":"2023-12-19T09:21:22.945131+0000","last_peered":"2023-12-19T09:21:22.945131+0000","last_clean":"2023-12-19T09:21:22.945131+0000","last_became_active":"2023-12-19T09:21:22.944619+0000","last_became_peered":"2023-12-19T09:21:22.944619+0000","last_unstale":"2023-12-19T09:21:22.945131+0000","last_undegraded":"2023-12-19T09:21:22.945131+0000","last_fullsized":"2023-12-19T09:21:22.945131+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:21:03.099999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941048+0000","last_change":"2023-12-19T09:21:22.941048+0000","last_active":"2023-12-19T09:21:22.941048+0000","last_peered":"2023-12-19T09:21:22.941048+0000","last_clean":"2023-12-19T09:21:22.941048+0000","last_became_active":"2023-12-19T09:21:22.940754+0000","last_became_peered":"2023-12-19T09:21:22.940754+0000","last_unstale":"2023-12-19T09:21:22.941048+0000","last_undegraded":"2023-12-19T09:21:22.941048+0000","last_fullsized":"2023-12-19T09:21:22.941048+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:46.636632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947046+0000","last_change":"2023-12-19T09:21:22.934994+0000","last_active":"2023-12-19T09:21:22.947046+0000","last_peered":"2023-12-19T09:21:22.947046+0000","last_clean":"2023-12-19T09:21:22.947046+0000","last_became_active":"2023-12-19T09:21:22.933158+0000","last_became_peered":"2023-12-19T09:21:22.933158+0000","last_unstale":"2023-12-19T09:21:22.947046+0000","last_undegraded":"2023-12-19T09:21:22.947046+0000","last_fullsized":"2023-12-19T09:21:22.947046+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:17:50.583942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948087+0000","last_change":"2023-12-19T09:21:22.938161+0000","last_active":"2023-12-19T09:21:22.948087+0000","last_peered":"2023-12-19T09:21:22.948087+0000","last_clean":"2023-12-19T09:21:22.948087+0000","last_became_active":"2023-12-19T09:21:22.937880+0000","last_became_peered":"2023-12-19T09:21:22.937880+0000","last_unstale":"2023-12-19T09:21:22.948087+0000","last_undegraded":"2023-12-19T09:21:22.948087+0000","last_fullsized":"2023-12-19T09:21:22.948087+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:32:04.009857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942298+0000","last_change":"2023-12-19T09:21:22.927633+0000","last_active":"2023-12-19T09:21:22.942298+0000","last_peered":"2023-12-19T09:21:22.942298+0000","last_clean":"2023-12-19T09:21:22.942298+0000","last_became_active":"2023-12-19T09:21:22.927348+0000","last_became_peered":"2023-12-19T09:21:22.927348+0000","last_unstale":"2023-12-19T09:21:22.942298+0000","last_undegraded":"2023-12-19T09:21:22.942298+0000","last_fullsized":"2023-12-19T09:21:22.942298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:48:05.211274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940953+0000","last_change":"2023-12-19T09:21:22.926038+0000","last_active":"2023-12-19T09:21:22.940953+0000","last_peered":"2023-12-19T09:21:22.940953+0000","last_clean":"2023-12-19T09:21:22.940953+0000","last_became_active":"2023-12-19T09:21:22.925709+0000","last_became_peered":"2023-12-19T09:21:22.925709+0000","last_unstale":"2023-12-19T09:21:22.940953+0000","last_undegraded":"2023-12-19T09:21:22.940953+0000","last_fullsized":"2023-12-19T09:21:22.940953+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:21:05.884386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948061+0000","last_change":"2023-12-19T09:21:22.937870+0000","last_active":"2023-12-19T09:21:22.948061+0000","last_peered":"2023-12-19T09:21:22.948061+0000","last_clean":"2023-12-19T09:21:22.948061+0000","last_became_active":"2023-12-19T09:21:22.937585+0000","last_became_peered":"2023-12-19T09:21:22.937585+0000","last_unstale":"2023-12-19T09:21:22.948061+0000","last_undegraded":"2023-12-19T09:21:22.948061+0000","last_fullsized":"2023-12-19T09:21:22.948061+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:08:58.809426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929962+0000","last_change":"2023-12-19T09:21:22.929962+0000","last_active":"2023-12-19T09:21:22.929962+0000","last_peered":"2023-12-19T09:21:22.929962+0000","last_clean":"2023-12-19T09:21:22.929962+0000","last_became_active":"2023-12-19T09:21:22.929686+0000","last_became_peered":"2023-12-19T09:21:22.929686+0000","last_unstale":"2023-12-19T09:21:22.929962+0000","last_undegraded":"2023-12-19T09:21:22.929962+0000","last_fullsized":"2023-12-19T09:21:22.929962+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:23:52.610948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.928513+0000","last_change":"2023-12-19T09:21:22.928513+0000","last_active":"2023-12-19T09:21:22.928513+0000","last_peered":"2023-12-19T09:21:22.928513+0000","last_clean":"2023-12-19T09:21:22.928513+0000","last_became_active":"2023-12-19T09:21:22.928224+0000","last_became_peered":"2023-12-19T09:21:22.928224+0000","last_unstale":"2023-12-19T09:21:22.928513+0000","last_undegraded":"2023-12-19T09:21:22.928513+0000","last_fullsized":"2023-12-19T09:21:22.928513+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:58.963749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.933993+0000","last_change":"2023-12-19T09:21:22.933993+0000","last_active":"2023-12-19T09:21:22.933993+0000","last_peered":"2023-12-19T09:21:22.933993+0000","last_clean":"2023-12-19T09:21:22.933993+0000","last_became_active":"2023-12-19T09:21:22.933706+0000","last_became_peered":"2023-12-19T09:21:22.933706+0000","last_unstale":"2023-12-19T09:21:22.933993+0000","last_undegraded":"2023-12-19T09:21:22.933993+0000","last_fullsized":"2023-12-19T09:21:22.933993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:20:56.456843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.938092+0000","last_change":"2023-12-19T09:21:22.938092+0000","last_active":"2023-12-19T09:21:22.938092+0000","last_peered":"2023-12-19T09:21:22.938092+0000","last_clean":"2023-12-19T09:21:22.938092+0000","last_became_active":"2023-12-19T09:21:22.937806+0000","last_became_peered":"2023-12-19T09:21:22.937806+0000","last_unstale":"2023-12-19T09:21:22.938092+0000","last_undegraded":"2023-12-19T09:21:22.938092+0000","last_fullsized":"2023-12-19T09:21:22.938092+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:25:20.910570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.933389+0000","last_change":"2023-12-19T09:21:22.933389+0000","last_active":"2023-12-19T09:21:22.933389+0000","last_peered":"2023-12-19T09:21:22.933389+0000","last_clean":"2023-12-19T09:21:22.933389+0000","last_became_active":"2023-12-19T09:21:22.933111+0000","last_became_peered":"2023-12-19T09:21:22.933111+0000","last_unstale":"2023-12-19T09:21:22.933389+0000","last_undegraded":"2023-12-19T09:21:22.933389+0000","last_fullsized":"2023-12-19T09:21:22.933389+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:41:57.852046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927267+0000","last_change":"2023-12-19T09:21:22.927267+0000","last_active":"2023-12-19T09:21:22.927267+0000","last_peered":"2023-12-19T09:21:22.927267+0000","last_clean":"2023-12-19T09:21:22.927267+0000","last_became_active":"2023-12-19T09:21:22.926953+0000","last_became_peered":"2023-12-19T09:21:22.926953+0000","last_unstale":"2023-12-19T09:21:22.927267+0000","last_undegraded":"2023-12-19T09:21:22.927267+0000","last_fullsized":"2023-12-19T09:21:22.927267+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:48:55.220863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.926136+0000","last_change":"2023-12-19T09:21:22.926136+0000","last_active":"2023-12-19T09:21:22.926136+0000","last_peered":"2023-12-19T09:21:22.926136+0000","last_clean":"2023-12-19T09:21:22.926136+0000","last_became_active":"2023-12-19T09:21:22.925800+0000","last_became_peered":"2023-12-19T09:21:22.925800+0000","last_unstale":"2023-12-19T09:21:22.926136+0000","last_undegraded":"2023-12-19T09:21:22.926136+0000","last_fullsized":"2023-12-19T09:21:22.926136+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:04.070678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948162+0000","last_change":"2023-12-19T09:21:22.941027+0000","last_active":"2023-12-19T09:21:22.948162+0000","last_peered":"2023-12-19T09:21:22.948162+0000","last_clean":"2023-12-19T09:21:22.948162+0000","last_became_active":"2023-12-19T09:21:22.940118+0000","last_became_peered":"2023-12-19T09:21:22.940118+0000","last_unstale":"2023-12-19T09:21:22.948162+0000","last_undegraded":"2023-12-19T09:21:22.948162+0000","last_fullsized":"2023-12-19T09:21:22.948162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:41:42.031571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.934510+0000","last_change":"2023-12-19T09:21:22.934510+0000","last_active":"2023-12-19T09:21:22.934510+0000","last_peered":"2023-12-19T09:21:22.934510+0000","last_clean":"2023-12-19T09:21:22.934510+0000","last_became_active":"2023-12-19T09:21:22.934232+0000","last_became_peered":"2023-12-19T09:21:22.934232+0000","last_unstale":"2023-12-19T09:21:22.934510+0000","last_undegraded":"2023-12-19T09:21:22.934510+0000","last_fullsized":"2023-12-19T09:21:22.934510+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:36.056878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.935298+0000","last_change":"2023-12-19T09:21:22.935298+0000","last_active":"2023-12-19T09:21:22.935298+0000","last_peered":"2023-12-19T09:21:22.935298+0000","last_clean":"2023-12-19T09:21:22.935298+0000","last_became_active":"2023-12-19T09:21:22.935017+0000","last_became_peered":"2023-12-19T09:21:22.935017+0000","last_unstale":"2023-12-19T09:21:22.935298+0000","last_undegraded":"2023-12-19T09:21:22.935298+0000","last_fullsized":"2023-12-19T09:21:22.935298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:39:44.920814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927293+0000","last_change":"2023-12-19T09:21:22.927293+0000","last_active":"2023-12-19T09:21:22.927293+0000","last_peered":"2023-12-19T09:21:22.927293+0000","last_clean":"2023-12-19T09:21:22.927293+0000","last_became_active":"2023-12-19T09:21:22.924124+0000","last_became_peered":"2023-12-19T09:21:22.924124+0000","last_unstale":"2023-12-19T09:21:22.927293+0000","last_undegraded":"2023-12-19T09:21:22.927293+0000","last_fullsized":"2023-12-19T09:21:22.927293+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:10:30.315001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948207+0000","last_change":"2023-12-19T09:21:22.941718+0000","last_active":"2023-12-19T09:21:22.948207+0000","last_peered":"2023-12-19T09:21:22.948207+0000","last_clean":"2023-12-19T09:21:22.948207+0000","last_became_active":"2023-12-19T09:21:22.941098+0000","last_became_peered":"2023-12-19T09:21:22.941098+0000","last_unstale":"2023-12-19T09:21:22.948207+0000","last_undegraded":"2023-12-19T09:21:22.948207+0000","last_fullsized":"2023-12-19T09:21:22.948207+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:52:27.891719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.926684+0000","last_change":"2023-12-19T09:21:22.926684+0000","last_active":"2023-12-19T09:21:22.926684+0000","last_peered":"2023-12-19T09:21:22.926684+0000","last_clean":"2023-12-19T09:21:22.926684+0000","last_became_active":"2023-12-19T09:21:22.926341+0000","last_became_peered":"2023-12-19T09:21:22.926341+0000","last_unstale":"2023-12-19T09:21:22.926684+0000","last_undegraded":"2023-12-19T09:21:22.926684+0000","last_fullsized":"2023-12-19T09:21:22.926684+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:07:21.121732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"207'4","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947522+0000","last_change":"2023-12-19T09:21:21.920062+0000","last_active":"2023-12-19T09:21:22.947522+0000","last_peered":"2023-12-19T09:21:22.947522+0000","last_clean":"2023-12-19T09:21:22.947522+0000","last_became_active":"2023-12-19T09:21:21.919729+0000","last_became_peered":"2023-12-19T09:21:21.919729+0000","last_unstale":"2023-12-19T09:21:22.947522+0000","last_undegraded":"2023-12-19T09:21:22.947522+0000","last_fullsized":"2023-12-19T09:21:22.947522+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:39:43.908102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943984+0000","last_change":"2023-12-19T09:21:22.943984+0000","last_active":"2023-12-19T09:21:22.943984+0000","last_peered":"2023-12-19T09:21:22.943984+0000","last_clean":"2023-12-19T09:21:22.943984+0000","last_became_active":"2023-12-19T09:21:22.942882+0000","last_became_peered":"2023-12-19T09:21:22.942882+0000","last_unstale":"2023-12-19T09:21:22.943984+0000","last_undegraded":"2023-12-19T09:21:22.943984+0000","last_fullsized":"2023-12-19T09:21:22.943984+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:03.387676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"207'1","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946344+0000","last_change":"2023-12-19T09:21:21.924299+0000","last_active":"2023-12-19T09:21:22.946344+0000","last_peered":"2023-12-19T09:21:22.946344+0000","last_clean":"2023-12-19T09:21:22.946344+0000","last_became_active":"2023-12-19T09:21:21.923979+0000","last_became_peered":"2023-12-19T09:21:21.923979+0000","last_unstale":"2023-12-19T09:21:22.946344+0000","last_undegraded":"2023-12-19T09:21:22.946344+0000","last_fullsized":"2023-12-19T09:21:22.946344+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:57.857343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931157+0000","last_change":"2023-12-19T09:21:22.931157+0000","last_active":"2023-12-19T09:21:22.931157+0000","last_peered":"2023-12-19T09:21:22.931157+0000","last_clean":"2023-12-19T09:21:22.931157+0000","last_became_active":"2023-12-19T09:21:22.930673+0000","last_became_peered":"2023-12-19T09:21:22.930673+0000","last_unstale":"2023-12-19T09:21:22.931157+0000","last_undegraded":"2023-12-19T09:21:22.931157+0000","last_fullsized":"2023-12-19T09:21:22.931157+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:08.045104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942547+0000","last_change":"2023-12-19T09:21:21.922027+0000","last_active":"2023-12-19T09:21:22.942547+0000","last_peered":"2023-12-19T09:21:22.942547+0000","last_clean":"2023-12-19T09:21:22.942547+0000","last_became_active":"2023-12-19T09:21:21.921695+0000","last_became_peered":"2023-12-19T09:21:21.921695+0000","last_unstale":"2023-12-19T09:21:22.942547+0000","last_undegraded":"2023-12-19T09:21:22.942547+0000","last_fullsized":"2023-12-19T09:21:22.942547+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:59:29.103271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.939613+0000","last_change":"2023-12-19T09:21:22.939613+0000","last_active":"2023-12-19T09:21:22.939613+0000","last_peered":"2023-12-19T09:21:22.939613+0000","last_clean":"2023-12-19T09:21:22.939613+0000","last_became_active":"2023-12-19T09:21:22.939140+0000","last_became_peered":"2023-12-19T09:21:22.939140+0000","last_unstale":"2023-12-19T09:21:22.939613+0000","last_undegraded":"2023-12-19T09:21:22.939613+0000","last_fullsized":"2023-12-19T09:21:22.939613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:55:54.496034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.940687+0000","last_change":"2023-12-19T09:21:21.920285+0000","last_active":"2023-12-19T09:21:22.940687+0000","last_peered":"2023-12-19T09:21:22.940687+0000","last_clean":"2023-12-19T09:21:22.940687+0000","last_became_active":"2023-12-19T09:21:21.919970+0000","last_became_peered":"2023-12-19T09:21:21.919970+0000","last_unstale":"2023-12-19T09:21:22.940687+0000","last_undegraded":"2023-12-19T09:21:22.940687+0000","last_fullsized":"2023-12-19T09:21:22.940687+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:32:38.377336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942006+0000","last_change":"2023-12-19T09:21:22.927043+0000","last_active":"2023-12-19T09:21:22.942006+0000","last_peered":"2023-12-19T09:21:22.942006+0000","last_clean":"2023-12-19T09:21:22.942006+0000","last_became_active":"2023-12-19T09:21:22.926733+0000","last_became_peered":"2023-12-19T09:21:22.926733+0000","last_unstale":"2023-12-19T09:21:22.942006+0000","last_undegraded":"2023-12-19T09:21:22.942006+0000","last_fullsized":"2023-12-19T09:21:22.942006+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:00.796583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"207'1","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946777+0000","last_change":"2023-12-19T09:21:21.925122+0000","last_active":"2023-12-19T09:21:22.946777+0000","last_peered":"2023-12-19T09:21:22.946777+0000","last_clean":"2023-12-19T09:21:22.946777+0000","last_became_active":"2023-12-19T09:21:21.924752+0000","last_became_peered":"2023-12-19T09:21:21.924752+0000","last_unstale":"2023-12-19T09:21:22.946777+0000","last_undegraded":"2023-12-19T09:21:22.946777+0000","last_fullsized":"2023-12-19T09:21:22.946777+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:24:03.057966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.931799+0000","last_change":"2023-12-19T09:21:22.931799+0000","last_active":"2023-12-19T09:21:22.931799+0000","last_peered":"2023-12-19T09:21:22.931799+0000","last_clean":"2023-12-19T09:21:22.931799+0000","last_became_active":"2023-12-19T09:21:22.931405+0000","last_became_peered":"2023-12-19T09:21:22.931405+0000","last_unstale":"2023-12-19T09:21:22.931799+0000","last_undegraded":"2023-12-19T09:21:22.931799+0000","last_fullsized":"2023-12-19T09:21:22.931799+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:23:07.226827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.904262+0000","last_change":"2023-12-19T09:21:21.928658+0000","last_active":"2023-12-19T09:21:22.904262+0000","last_peered":"2023-12-19T09:21:22.904262+0000","last_clean":"2023-12-19T09:21:22.904262+0000","last_became_active":"2023-12-19T09:21:21.928342+0000","last_became_peered":"2023-12-19T09:21:21.928342+0000","last_unstale":"2023-12-19T09:21:22.904262+0000","last_undegraded":"2023-12-19T09:21:22.904262+0000","last_fullsized":"2023-12-19T09:21:22.904262+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:06:37.335226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941893+0000","last_change":"2023-12-19T09:21:22.941893+0000","last_active":"2023-12-19T09:21:22.941893+0000","last_peered":"2023-12-19T09:21:22.941893+0000","last_clean":"2023-12-19T09:21:22.941893+0000","last_became_active":"2023-12-19T09:21:22.941587+0000","last_became_peered":"2023-12-19T09:21:22.941587+0000","last_unstale":"2023-12-19T09:21:22.941893+0000","last_undegraded":"2023-12-19T09:21:22.941893+0000","last_fullsized":"2023-12-19T09:21:22.941893+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:51.456781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.907081+0000","last_change":"2023-12-19T09:21:21.917883+0000","last_active":"2023-12-19T09:21:22.907081+0000","last_peered":"2023-12-19T09:21:22.907081+0000","last_clean":"2023-12-19T09:21:22.907081+0000","last_became_active":"2023-12-19T09:21:21.917505+0000","last_became_peered":"2023-12-19T09:21:21.917505+0000","last_unstale":"2023-12-19T09:21:22.907081+0000","last_undegraded":"2023-12-19T09:21:22.907081+0000","last_fullsized":"2023-12-19T09:21:22.907081+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:02:08.727761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946786+0000","last_change":"2023-12-19T09:21:22.946786+0000","last_active":"2023-12-19T09:21:22.946786+0000","last_peered":"2023-12-19T09:21:22.946786+0000","last_clean":"2023-12-19T09:21:22.946786+0000","last_became_active":"2023-12-19T09:21:22.945991+0000","last_became_peered":"2023-12-19T09:21:22.945991+0000","last_unstale":"2023-12-19T09:21:22.946786+0000","last_undegraded":"2023-12-19T09:21:22.946786+0000","last_fullsized":"2023-12-19T09:21:22.946786+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:20:40.463149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.913524+0000","last_change":"2023-12-19T09:21:21.924668+0000","last_active":"2023-12-19T09:21:22.913524+0000","last_peered":"2023-12-19T09:21:22.913524+0000","last_clean":"2023-12-19T09:21:22.913524+0000","last_became_active":"2023-12-19T09:21:21.924198+0000","last_became_peered":"2023-12-19T09:21:21.924198+0000","last_unstale":"2023-12-19T09:21:22.913524+0000","last_undegraded":"2023-12-19T09:21:22.913524+0000","last_fullsized":"2023-12-19T09:21:22.913524+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:17:41.190537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.925540+0000","last_change":"2023-12-19T09:21:22.920989+0000","last_active":"2023-12-19T09:21:22.925540+0000","last_peered":"2023-12-19T09:21:22.925540+0000","last_clean":"2023-12-19T09:21:22.925540+0000","last_became_active":"2023-12-19T09:21:22.920672+0000","last_became_peered":"2023-12-19T09:21:22.920672+0000","last_unstale":"2023-12-19T09:21:22.925540+0000","last_undegraded":"2023-12-19T09:21:22.925540+0000","last_fullsized":"2023-12-19T09:21:22.925540+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:10:29.585036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.928880+0000","last_change":"2023-12-19T09:21:21.927320+0000","last_active":"2023-12-19T09:21:21.928880+0000","last_peered":"2023-12-19T09:21:21.928880+0000","last_clean":"2023-12-19T09:21:21.928880+0000","last_became_active":"2023-12-19T09:21:21.927000+0000","last_became_peered":"2023-12-19T09:21:21.927000+0000","last_unstale":"2023-12-19T09:21:21.928880+0000","last_undegraded":"2023-12-19T09:21:21.928880+0000","last_fullsized":"2023-12-19T09:21:21.928880+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:19:10.095384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942316+0000","last_change":"2023-12-19T09:21:22.929154+0000","last_active":"2023-12-19T09:21:22.942316+0000","last_peered":"2023-12-19T09:21:22.942316+0000","last_clean":"2023-12-19T09:21:22.942316+0000","last_became_active":"2023-12-19T09:21:22.928793+0000","last_became_peered":"2023-12-19T09:21:22.928793+0000","last_unstale":"2023-12-19T09:21:22.942316+0000","last_undegraded":"2023-12-19T09:21:22.942316+0000","last_fullsized":"2023-12-19T09:21:22.942316+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:53:28.304421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"207'3","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.948470+0000","last_change":"2023-12-19T09:21:21.928897+0000","last_active":"2023-12-19T09:21:22.948470+0000","last_peered":"2023-12-19T09:21:22.948470+0000","last_clean":"2023-12-19T09:21:22.948470+0000","last_became_active":"2023-12-19T09:21:21.928413+0000","last_became_peered":"2023-12-19T09:21:21.928413+0000","last_unstale":"2023-12-19T09:21:22.948470+0000","last_undegraded":"2023-12-19T09:21:22.948470+0000","last_fullsized":"2023-12-19T09:21:22.948470+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:11:00.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942676+0000","last_change":"2023-12-19T09:21:22.942676+0000","last_active":"2023-12-19T09:21:22.942676+0000","last_peered":"2023-12-19T09:21:22.942676+0000","last_clean":"2023-12-19T09:21:22.942676+0000","last_became_active":"2023-12-19T09:21:22.942316+0000","last_became_peered":"2023-12-19T09:21:22.942316+0000","last_unstale":"2023-12-19T09:21:22.942676+0000","last_undegraded":"2023-12-19T09:21:22.942676+0000","last_fullsized":"2023-12-19T09:21:22.942676+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:50:58.870055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"207'6","reported_seq":16,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.953576+0000","last_change":"2023-12-19T09:21:21.919056+0000","last_active":"2023-12-19T09:21:22.953576+0000","last_peered":"2023-12-19T09:21:22.953576+0000","last_clean":"2023-12-19T09:21:22.953576+0000","last_became_active":"2023-12-19T09:21:21.918781+0000","last_became_peered":"2023-12-19T09:21:21.918781+0000","last_unstale":"2023-12-19T09:21:22.953576+0000","last_undegraded":"2023-12-19T09:21:22.953576+0000","last_fullsized":"2023-12-19T09:21:22.953576+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:23:12.675270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927979+0000","last_change":"2023-12-19T09:21:22.927979+0000","last_active":"2023-12-19T09:21:22.927979+0000","last_peered":"2023-12-19T09:21:22.927979+0000","last_clean":"2023-12-19T09:21:22.927979+0000","last_became_active":"2023-12-19T09:21:22.927654+0000","last_became_peered":"2023-12-19T09:21:22.927654+0000","last_unstale":"2023-12-19T09:21:22.927979+0000","last_undegraded":"2023-12-19T09:21:22.927979+0000","last_fullsized":"2023-12-19T09:21:22.927979+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:43:16.311057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941313+0000","last_change":"2023-12-19T09:21:21.920618+0000","last_active":"2023-12-19T09:21:22.941313+0000","last_peered":"2023-12-19T09:21:22.941313+0000","last_clean":"2023-12-19T09:21:22.941313+0000","last_became_active":"2023-12-19T09:21:21.920313+0000","last_became_peered":"2023-12-19T09:21:21.920313+0000","last_unstale":"2023-12-19T09:21:22.941313+0000","last_undegraded":"2023-12-19T09:21:22.941313+0000","last_fullsized":"2023-12-19T09:21:22.941313+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:03.968366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.936308+0000","last_change":"2023-12-19T09:21:22.925304+0000","last_active":"2023-12-19T09:21:22.936308+0000","last_peered":"2023-12-19T09:21:22.936308+0000","last_clean":"2023-12-19T09:21:22.936308+0000","last_became_active":"2023-12-19T09:21:22.923624+0000","last_became_peered":"2023-12-19T09:21:22.923624+0000","last_unstale":"2023-12-19T09:21:22.936308+0000","last_undegraded":"2023-12-19T09:21:22.936308+0000","last_fullsized":"2023-12-19T09:21:22.936308+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:39:11.298724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.919281+0000","last_change":"2023-12-19T09:21:21.916368+0000","last_active":"2023-12-19T09:21:21.919281+0000","last_peered":"2023-12-19T09:21:21.919281+0000","last_clean":"2023-12-19T09:21:21.919281+0000","last_became_active":"2023-12-19T09:21:21.916054+0000","last_became_peered":"2023-12-19T09:21:21.916054+0000","last_unstale":"2023-12-19T09:21:21.919281+0000","last_undegraded":"2023-12-19T09:21:21.919281+0000","last_fullsized":"2023-12-19T09:21:21.919281+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:24.820042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941090+0000","last_change":"2023-12-19T09:21:22.926505+0000","last_active":"2023-12-19T09:21:22.941090+0000","last_peered":"2023-12-19T09:21:22.941090+0000","last_clean":"2023-12-19T09:21:22.941090+0000","last_became_active":"2023-12-19T09:21:22.925568+0000","last_became_peered":"2023-12-19T09:21:22.925568+0000","last_unstale":"2023-12-19T09:21:22.941090+0000","last_undegraded":"2023-12-19T09:21:22.941090+0000","last_fullsized":"2023-12-19T09:21:22.941090+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:16:24.010684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921525+0000","last_change":"2023-12-19T09:21:21.918155+0000","last_active":"2023-12-19T09:21:21.921525+0000","last_peered":"2023-12-19T09:21:21.921525+0000","last_clean":"2023-12-19T09:21:21.921525+0000","last_became_active":"2023-12-19T09:21:21.915613+0000","last_became_peered":"2023-12-19T09:21:21.915613+0000","last_unstale":"2023-12-19T09:21:21.921525+0000","last_undegraded":"2023-12-19T09:21:21.921525+0000","last_fullsized":"2023-12-19T09:21:21.921525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:01:14.605229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942217+0000","last_change":"2023-12-19T09:21:22.935970+0000","last_active":"2023-12-19T09:21:22.942217+0000","last_peered":"2023-12-19T09:21:22.942217+0000","last_clean":"2023-12-19T09:21:22.942217+0000","last_became_active":"2023-12-19T09:21:22.935686+0000","last_became_peered":"2023-12-19T09:21:22.935686+0000","last_unstale":"2023-12-19T09:21:22.942217+0000","last_undegraded":"2023-12-19T09:21:22.942217+0000","last_fullsized":"2023-12-19T09:21:22.942217+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:43:25.832753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"207'8","reported_seq":17,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.945225+0000","last_change":"2023-12-19T09:21:21.919360+0000","last_active":"2023-12-19T09:21:22.945225+0000","last_peered":"2023-12-19T09:21:22.945225+0000","last_clean":"2023-12-19T09:21:22.945225+0000","last_became_active":"2023-12-19T09:21:21.919084+0000","last_became_peered":"2023-12-19T09:21:21.919084+0000","last_unstale":"2023-12-19T09:21:22.945225+0000","last_undegraded":"2023-12-19T09:21:22.945225+0000","last_fullsized":"2023-12-19T09:21:22.945225+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:37.685945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.930816+0000","last_change":"2023-12-19T09:21:22.930816+0000","last_active":"2023-12-19T09:21:22.930816+0000","last_peered":"2023-12-19T09:21:22.930816+0000","last_clean":"2023-12-19T09:21:22.930816+0000","last_became_active":"2023-12-19T09:21:22.930485+0000","last_became_peered":"2023-12-19T09:21:22.930485+0000","last_unstale":"2023-12-19T09:21:22.930816+0000","last_undegraded":"2023-12-19T09:21:22.930816+0000","last_fullsized":"2023-12-19T09:21:22.930816+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:51:51.527655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.913661+0000","last_change":"2023-12-19T09:21:21.923101+0000","last_active":"2023-12-19T09:21:22.913661+0000","last_peered":"2023-12-19T09:21:22.913661+0000","last_clean":"2023-12-19T09:21:22.913661+0000","last_became_active":"2023-12-19T09:21:21.922751+0000","last_became_peered":"2023-12-19T09:21:21.922751+0000","last_unstale":"2023-12-19T09:21:22.913661+0000","last_undegraded":"2023-12-19T09:21:22.913661+0000","last_fullsized":"2023-12-19T09:21:22.913661+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:36:06.201695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.946763+0000","last_change":"2023-12-19T09:21:22.933229+0000","last_active":"2023-12-19T09:21:22.946763+0000","last_peered":"2023-12-19T09:21:22.946763+0000","last_clean":"2023-12-19T09:21:22.946763+0000","last_became_active":"2023-12-19T09:21:22.932910+0000","last_became_peered":"2023-12-19T09:21:22.932910+0000","last_unstale":"2023-12-19T09:21:22.946763+0000","last_undegraded":"2023-12-19T09:21:22.946763+0000","last_fullsized":"2023-12-19T09:21:22.946763+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:31:42.698439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.944100+0000","last_change":"2023-12-19T09:21:21.918258+0000","last_active":"2023-12-19T09:21:22.944100+0000","last_peered":"2023-12-19T09:21:22.944100+0000","last_clean":"2023-12-19T09:21:22.944100+0000","last_became_active":"2023-12-19T09:21:21.915644+0000","last_became_peered":"2023-12-19T09:21:21.915644+0000","last_unstale":"2023-12-19T09:21:22.944100+0000","last_undegraded":"2023-12-19T09:21:22.944100+0000","last_fullsized":"2023-12-19T09:21:22.944100+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:04:07.032392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.923553+0000","last_change":"2023-12-19T09:21:22.923553+0000","last_active":"2023-12-19T09:21:22.923553+0000","last_peered":"2023-12-19T09:21:22.923553+0000","last_clean":"2023-12-19T09:21:22.923553+0000","last_became_active":"2023-12-19T09:21:22.922683+0000","last_became_peered":"2023-12-19T09:21:22.922683+0000","last_unstale":"2023-12-19T09:21:22.923553+0000","last_undegraded":"2023-12-19T09:21:22.923553+0000","last_fullsized":"2023-12-19T09:21:22.923553+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:11:00.796583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"207'1","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.916821+0000","last_change":"2023-12-19T09:21:21.930923+0000","last_active":"2023-12-19T09:21:22.916821+0000","last_peered":"2023-12-19T09:21:22.916821+0000","last_clean":"2023-12-19T09:21:22.916821+0000","last_became_active":"2023-12-19T09:21:21.930551+0000","last_became_peered":"2023-12-19T09:21:21.930551+0000","last_unstale":"2023-12-19T09:21:22.916821+0000","last_undegraded":"2023-12-19T09:21:22.916821+0000","last_fullsized":"2023-12-19T09:21:22.916821+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:10:29.302289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929956+0000","last_change":"2023-12-19T09:21:22.929956+0000","last_active":"2023-12-19T09:21:22.929956+0000","last_peered":"2023-12-19T09:21:22.929956+0000","last_clean":"2023-12-19T09:21:22.929956+0000","last_became_active":"2023-12-19T09:21:22.927210+0000","last_became_peered":"2023-12-19T09:21:22.927210+0000","last_unstale":"2023-12-19T09:21:22.929956+0000","last_undegraded":"2023-12-19T09:21:22.929956+0000","last_fullsized":"2023-12-19T09:21:22.929956+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:49:12.990854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.920974+0000","last_change":"2023-12-19T09:21:21.916685+0000","last_active":"2023-12-19T09:21:21.920974+0000","last_peered":"2023-12-19T09:21:21.920974+0000","last_clean":"2023-12-19T09:21:21.920974+0000","last_became_active":"2023-12-19T09:21:21.916250+0000","last_became_peered":"2023-12-19T09:21:21.916250+0000","last_unstale":"2023-12-19T09:21:21.920974+0000","last_undegraded":"2023-12-19T09:21:21.920974+0000","last_fullsized":"2023-12-19T09:21:21.920974+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:49:42.812210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.927595+0000","last_change":"2023-12-19T09:21:22.927595+0000","last_active":"2023-12-19T09:21:22.927595+0000","last_peered":"2023-12-19T09:21:22.927595+0000","last_clean":"2023-12-19T09:21:22.927595+0000","last_became_active":"2023-12-19T09:21:22.927190+0000","last_became_peered":"2023-12-19T09:21:22.927190+0000","last_unstale":"2023-12-19T09:21:22.927595+0000","last_undegraded":"2023-12-19T09:21:22.927595+0000","last_fullsized":"2023-12-19T09:21:22.927595+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:08:21.890500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"207'2","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.941024+0000","last_change":"2023-12-19T09:21:21.929027+0000","last_active":"2023-12-19T09:21:22.941024+0000","last_peered":"2023-12-19T09:21:22.941024+0000","last_clean":"2023-12-19T09:21:22.941024+0000","last_became_active":"2023-12-19T09:21:21.928744+0000","last_became_peered":"2023-12-19T09:21:21.928744+0000","last_unstale":"2023-12-19T09:21:22.941024+0000","last_undegraded":"2023-12-19T09:21:22.941024+0000","last_fullsized":"2023-12-19T09:21:22.941024+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:00:13.878677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929727+0000","last_change":"2023-12-19T09:21:22.929727+0000","last_active":"2023-12-19T09:21:22.929727+0000","last_peered":"2023-12-19T09:21:22.929727+0000","last_clean":"2023-12-19T09:21:22.929727+0000","last_became_active":"2023-12-19T09:21:22.927761+0000","last_became_peered":"2023-12-19T09:21:22.927761+0000","last_unstale":"2023-12-19T09:21:22.929727+0000","last_undegraded":"2023-12-19T09:21:22.929727+0000","last_fullsized":"2023-12-19T09:21:22.929727+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:56:58.963749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.922380+0000","last_change":"2023-12-19T09:21:21.920440+0000","last_active":"2023-12-19T09:21:21.922380+0000","last_peered":"2023-12-19T09:21:21.922380+0000","last_clean":"2023-12-19T09:21:21.922380+0000","last_became_active":"2023-12-19T09:21:21.920169+0000","last_became_peered":"2023-12-19T09:21:21.920169+0000","last_unstale":"2023-12-19T09:21:21.922380+0000","last_undegraded":"2023-12-19T09:21:21.922380+0000","last_fullsized":"2023-12-19T09:21:21.922380+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:20:35.044166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.923468+0000","last_change":"2023-12-19T09:21:22.923468+0000","last_active":"2023-12-19T09:21:22.923468+0000","last_peered":"2023-12-19T09:21:22.923468+0000","last_clean":"2023-12-19T09:21:22.923468+0000","last_became_active":"2023-12-19T09:21:22.923170+0000","last_became_peered":"2023-12-19T09:21:22.923170+0000","last_unstale":"2023-12-19T09:21:22.923468+0000","last_undegraded":"2023-12-19T09:21:22.923468+0000","last_fullsized":"2023-12-19T09:21:22.923468+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:28:50.478345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.915763+0000","last_change":"2023-12-19T09:21:21.910018+0000","last_active":"2023-12-19T09:21:21.915763+0000","last_peered":"2023-12-19T09:21:21.915763+0000","last_clean":"2023-12-19T09:21:21.915763+0000","last_became_active":"2023-12-19T09:21:21.909718+0000","last_became_peered":"2023-12-19T09:21:21.909718+0000","last_unstale":"2023-12-19T09:21:21.915763+0000","last_undegraded":"2023-12-19T09:21:21.915763+0000","last_fullsized":"2023-12-19T09:21:21.915763+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:15.425555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.936341+0000","last_change":"2023-12-19T09:21:22.925665+0000","last_active":"2023-12-19T09:21:22.936341+0000","last_peered":"2023-12-19T09:21:22.936341+0000","last_clean":"2023-12-19T09:21:22.936341+0000","last_became_active":"2023-12-19T09:21:22.925396+0000","last_became_peered":"2023-12-19T09:21:22.925396+0000","last_unstale":"2023-12-19T09:21:22.936341+0000","last_undegraded":"2023-12-19T09:21:22.936341+0000","last_fullsized":"2023-12-19T09:21:22.936341+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:06:35.945108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921940+0000","last_change":"2023-12-19T09:21:21.920478+0000","last_active":"2023-12-19T09:21:21.921940+0000","last_peered":"2023-12-19T09:21:21.921940+0000","last_clean":"2023-12-19T09:21:21.921940+0000","last_became_active":"2023-12-19T09:21:21.918501+0000","last_became_peered":"2023-12-19T09:21:21.918501+0000","last_unstale":"2023-12-19T09:21:21.921940+0000","last_undegraded":"2023-12-19T09:21:21.921940+0000","last_fullsized":"2023-12-19T09:21:21.921940+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:28:02.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.943169+0000","last_change":"2023-12-19T09:21:22.943169+0000","last_active":"2023-12-19T09:21:22.943169+0000","last_peered":"2023-12-19T09:21:22.943169+0000","last_clean":"2023-12-19T09:21:22.943169+0000","last_became_active":"2023-12-19T09:21:22.942856+0000","last_became_peered":"2023-12-19T09:21:22.942856+0000","last_unstale":"2023-12-19T09:21:22.943169+0000","last_undegraded":"2023-12-19T09:21:22.943169+0000","last_fullsized":"2023-12-19T09:21:22.943169+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T19:58:24.989141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"207'2","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.947001+0000","last_change":"2023-12-19T09:21:21.917286+0000","last_active":"2023-12-19T09:21:22.947001+0000","last_peered":"2023-12-19T09:21:22.947001+0000","last_clean":"2023-12-19T09:21:22.947001+0000","last_became_active":"2023-12-19T09:21:21.916368+0000","last_became_peered":"2023-12-19T09:21:21.916368+0000","last_unstale":"2023-12-19T09:21:22.947001+0000","last_undegraded":"2023-12-19T09:21:22.947001+0000","last_fullsized":"2023-12-19T09:21:22.947001+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T18:18:45.069729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.935872+0000","last_change":"2023-12-19T09:21:22.935872+0000","last_active":"2023-12-19T09:21:22.935872+0000","last_peered":"2023-12-19T09:21:22.935872+0000","last_clean":"2023-12-19T09:21:22.935872+0000","last_became_active":"2023-12-19T09:21:22.935543+0000","last_became_peered":"2023-12-19T09:21:22.935543+0000","last_unstale":"2023-12-19T09:21:22.935872+0000","last_undegraded":"2023-12-19T09:21:22.935872+0000","last_fullsized":"2023-12-19T09:21:22.935872+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:03:21.450955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.919555+0000","last_change":"2023-12-19T09:21:21.916297+0000","last_active":"2023-12-19T09:21:21.919555+0000","last_peered":"2023-12-19T09:21:21.919555+0000","last_clean":"2023-12-19T09:21:21.919555+0000","last_became_active":"2023-12-19T09:21:21.916007+0000","last_became_peered":"2023-12-19T09:21:21.916007+0000","last_unstale":"2023-12-19T09:21:21.919555+0000","last_undegraded":"2023-12-19T09:21:21.919555+0000","last_fullsized":"2023-12-19T09:21:21.919555+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:19:05.707222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.929663+0000","last_change":"2023-12-19T09:21:22.929663+0000","last_active":"2023-12-19T09:21:22.929663+0000","last_peered":"2023-12-19T09:21:22.929663+0000","last_clean":"2023-12-19T09:21:22.929663+0000","last_became_active":"2023-12-19T09:21:22.929393+0000","last_became_peered":"2023-12-19T09:21:22.929393+0000","last_unstale":"2023-12-19T09:21:22.929663+0000","last_undegraded":"2023-12-19T09:21:22.929663+0000","last_fullsized":"2023-12-19T09:21:22.929663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T13:18:14.451651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.911634+0000","last_change":"2023-12-19T09:21:21.924894+0000","last_active":"2023-12-19T09:21:22.911634+0000","last_peered":"2023-12-19T09:21:22.911634+0000","last_clean":"2023-12-19T09:21:22.911634+0000","last_became_active":"2023-12-19T09:21:21.924507+0000","last_became_peered":"2023-12-19T09:21:21.924507+0000","last_unstale":"2023-12-19T09:21:22.911634+0000","last_undegraded":"2023-12-19T09:21:22.911634+0000","last_fullsized":"2023-12-19T09:21:22.911634+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:20:55.444131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942690+0000","last_change":"2023-12-19T09:21:22.942690+0000","last_active":"2023-12-19T09:21:22.942690+0000","last_peered":"2023-12-19T09:21:22.942690+0000","last_clean":"2023-12-19T09:21:22.942690+0000","last_became_active":"2023-12-19T09:21:22.942279+0000","last_became_peered":"2023-12-19T09:21:22.942279+0000","last_unstale":"2023-12-19T09:21:22.942690+0000","last_undegraded":"2023-12-19T09:21:22.942690+0000","last_fullsized":"2023-12-19T09:21:22.942690+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:58:23.081488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.912879+0000","last_change":"2023-12-19T09:21:21.924446+0000","last_active":"2023-12-19T09:21:22.912879+0000","last_peered":"2023-12-19T09:21:22.912879+0000","last_clean":"2023-12-19T09:21:22.912879+0000","last_became_active":"2023-12-19T09:21:21.924119+0000","last_became_peered":"2023-12-19T09:21:21.924119+0000","last_unstale":"2023-12-19T09:21:22.912879+0000","last_undegraded":"2023-12-19T09:21:22.912879+0000","last_fullsized":"2023-12-19T09:21:22.912879+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:07.012619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.925072+0000","last_change":"2023-12-19T09:21:22.925072+0000","last_active":"2023-12-19T09:21:22.925072+0000","last_peered":"2023-12-19T09:21:22.925072+0000","last_clean":"2023-12-19T09:21:22.925072+0000","last_became_active":"2023-12-19T09:21:22.924813+0000","last_became_peered":"2023-12-19T09:21:22.924813+0000","last_unstale":"2023-12-19T09:21:22.925072+0000","last_undegraded":"2023-12-19T09:21:22.925072+0000","last_fullsized":"2023-12-19T09:21:22.925072+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:57:58.976484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.917426+0000","last_change":"2023-12-19T09:21:21.915520+0000","last_active":"2023-12-19T09:21:21.917426+0000","last_peered":"2023-12-19T09:21:21.917426+0000","last_clean":"2023-12-19T09:21:21.917426+0000","last_became_active":"2023-12-19T09:21:21.915173+0000","last_became_peered":"2023-12-19T09:21:21.915173+0000","last_unstale":"2023-12-19T09:21:21.917426+0000","last_undegraded":"2023-12-19T09:21:21.917426+0000","last_fullsized":"2023-12-19T09:21:21.917426+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T12:10:50.444069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.942107+0000","last_change":"2023-12-19T09:21:22.929744+0000","last_active":"2023-12-19T09:21:22.942107+0000","last_peered":"2023-12-19T09:21:22.942107+0000","last_clean":"2023-12-19T09:21:22.942107+0000","last_became_active":"2023-12-19T09:21:22.929059+0000","last_became_peered":"2023-12-19T09:21:22.929059+0000","last_unstale":"2023-12-19T09:21:22.942107+0000","last_undegraded":"2023-12-19T09:21:22.942107+0000","last_fullsized":"2023-12-19T09:21:22.942107+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:47:38.698657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.926645+0000","last_change":"2023-12-19T09:21:21.925803+0000","last_active":"2023-12-19T09:21:21.926645+0000","last_peered":"2023-12-19T09:21:21.926645+0000","last_clean":"2023-12-19T09:21:21.926645+0000","last_became_active":"2023-12-19T09:21:21.925393+0000","last_became_peered":"2023-12-19T09:21:21.925393+0000","last_unstale":"2023-12-19T09:21:21.926645+0000","last_undegraded":"2023-12-19T09:21:21.926645+0000","last_fullsized":"2023-12-19T09:21:21.926645+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T20:51:45.623920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.930110+0000","last_change":"2023-12-19T09:21:22.930110+0000","last_active":"2023-12-19T09:21:22.930110+0000","last_peered":"2023-12-19T09:21:22.930110+0000","last_clean":"2023-12-19T09:21:22.930110+0000","last_became_active":"2023-12-19T09:21:22.928514+0000","last_became_peered":"2023-12-19T09:21:22.928514+0000","last_unstale":"2023-12-19T09:21:22.930110+0000","last_undegraded":"2023-12-19T09:21:22.930110+0000","last_fullsized":"2023-12-19T09:21:22.930110+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T14:28:50.478345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.918171+0000","last_change":"2023-12-19T09:21:21.911594+0000","last_active":"2023-12-19T09:21:21.918171+0000","last_peered":"2023-12-19T09:21:21.918171+0000","last_clean":"2023-12-19T09:21:21.918171+0000","last_became_active":"2023-12-19T09:21:21.911278+0000","last_became_peered":"2023-12-19T09:21:21.911278+0000","last_unstale":"2023-12-19T09:21:21.918171+0000","last_undegraded":"2023-12-19T09:21:21.918171+0000","last_fullsized":"2023-12-19T09:21:21.918171+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T16:18:22.556014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.934211+0000","last_change":"2023-12-19T09:21:22.934211+0000","last_active":"2023-12-19T09:21:22.934211+0000","last_peered":"2023-12-19T09:21:22.934211+0000","last_clean":"2023-12-19T09:21:22.934211+0000","last_became_active":"2023-12-19T09:21:22.933928+0000","last_became_peered":"2023-12-19T09:21:22.933928+0000","last_unstale":"2023-12-19T09:21:22.934211+0000","last_undegraded":"2023-12-19T09:21:22.934211+0000","last_fullsized":"2023-12-19T09:21:22.934211+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T15:46:08.025331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921414+0000","last_change":"2023-12-19T09:21:21.919344+0000","last_active":"2023-12-19T09:21:21.921414+0000","last_peered":"2023-12-19T09:21:21.921414+0000","last_clean":"2023-12-19T09:21:21.921414+0000","last_became_active":"2023-12-19T09:21:21.918365+0000","last_became_peered":"2023-12-19T09:21:21.918365+0000","last_unstale":"2023-12-19T09:21:21.921414+0000","last_undegraded":"2023-12-19T09:21:21.921414+0000","last_fullsized":"2023-12-19T09:21:21.921414+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T21:20:54.868064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-12-19T09:21:22.921251+0000","last_change":"2023-12-19T09:21:22.921251+0000","last_active":"2023-12-19T09:21:22.921251+0000","last_peered":"2023-12-19T09:21:22.921251+0000","last_clean":"2023-12-19T09:21:22.921251+0000","last_became_active":"2023-12-19T09:21:22.920844+0000","last_became_peered":"2023-12-19T09:21:22.920844+0000","last_unstale":"2023-12-19T09:21:22.921251+0000","last_undegraded":"2023-12-19T09:21:22.921251+0000","last_fullsized":"2023-12-19T09:21:22.921251+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:21.878080+0000","last_clean_scrub_stamp":"2023-12-19T09:21:21.878080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T17:04:25.771789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":9,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.921695+0000","last_change":"2023-12-19T09:21:21.917540+0000","last_active":"2023-12-19T09:21:21.921695+0000","last_peered":"2023-12-19T09:21:21.921695+0000","last_clean":"2023-12-19T09:21:21.921695+0000","last_became_active":"2023-12-19T09:21:21.916456+0000","last_became_peered":"2023-12-19T09:21:21.916456+0000","last_unstale":"2023-12-19T09:21:21.921695+0000","last_undegraded":"2023-12-19T09:21:21.921695+0000","last_fullsized":"2023-12-19T09:21:21.921695+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T09:21:20.865368+0000","last_clean_scrub_stamp":"2023-12-19T09:21:20.865368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T09:28:34.312826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"197'348","reported_seq":1133,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-12-19T09:21:21.901678+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:21.901678+0000","last_peered":"2023-12-19T09:21:21.901678+0000","last_clean":"2023-12-19T09:21:21.901678+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:21.901678+0000","last_undegraded":"2023-12-19T09:21:21.901678+0000","last_fullsized":"2023-12-19T09:21:21.901678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575968,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304500,"kb_used_data":3116,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067340,"statfs":{"total":96636764160,"available":96324956160,"internally_reserved":0,"allocated":3190784,"data_stored":1985008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.3009999999999999,"15min":1.2929999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.52600000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.2849999999999999,"15min":1.298},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.55700000000000005},"max":{"1min":1.5269999999999999,"5min":1.6759999999999999,"15min":1.712},"last":1.417}]},{"osd":1,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.169,"15min":1.1739999999999999},"min":{"1min":0.91000000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4990000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.206,"15min":1.212},"min":{"1min":0.85099999999999998,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6579999999999999,"5min":1.7769999999999999,"15min":1.796},"last":0.91400000000000003}]},{"osd":2,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1739999999999999,"15min":1.181},"min":{"1min":0.91500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":0.871},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2949999999999999,"15min":1.282},"min":{"1min":0.97499999999999998,"5min":0.79000000000000004,"15min":0.71999999999999997},"max":{"1min":1.5600000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":1.2030000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3240000000000001,"15min":1.3180000000000001},"min":{"1min":1.103,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.597,"5min":1.845,"15min":1.9059999999999999},"last":1.032},{"interface":"front","average":{"1min":1.337,"5min":1.371,"15min":1.381},"min":{"1min":0.95099999999999996,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.6919999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.1739999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.228,"15min":1.244},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77000000000000002},"max":{"1min":1.6870000000000001,"5min":1.7330000000000001,"15min":1.95},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.2450000000000001,"15min":1.2450000000000001},"min":{"1min":0.99099999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"max":{"1min":1.696,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.149}]},{"osd":5,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.2789999999999999,"15min":1.294},"min":{"1min":0.88200000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.6459999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.4139999999999999,"15min":1.413},"min":{"1min":0.97999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.925,"5min":1.972,"15min":1.972},"last":1.27}]},{"osd":6,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4139999999999999,"15min":1.4179999999999999},"min":{"1min":1.1279999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8109999999999999,"5min":1.893,"15min":2.0750000000000002},"last":1.361},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.413,"15min":1.399},"min":{"1min":1.1479999999999999,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.833,"5min":1.9530000000000001,"15min":1.986},"last":1.234}]}]},{"osd":6,"up_from":13,"seq":55834575967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312116,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059724,"statfs":{"total":96636764160,"available":96317157376,"internally_reserved":0,"allocated":2600960,"data_stored":1394710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.165,"15min":1.1479999999999999},"min":{"1min":0.82499999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.516,"5min":1.6799999999999999,"15min":1.784},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.1839999999999999,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68000000000000005,"15min":0.55100000000000005},"max":{"1min":1.431,"5min":1.6850000000000001,"15min":1.946},"last":1.466}]},{"osd":1,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.77900000000000003,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.4730000000000001,"5min":1.603,"15min":2.3439999999999999},"last":1.506},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.94299999999999995,"5min":0.72499999999999998,"15min":0.59099999999999997},"max":{"1min":1.6259999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.387}]},{"osd":2,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2629999999999999,"15min":1.2549999999999999},"min":{"1min":1.0289999999999999,"5min":0.86899999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.417},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.2569999999999999},"min":{"1min":0.90300000000000002,"5min":0.88700000000000001,"15min":0.73999999999999999},"max":{"1min":1.8340000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.04}]},{"osd":3,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.94699999999999995,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.5720000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.641},{"interface":"front","average":{"1min":1.294,"5min":1.3360000000000001,"15min":1.3200000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68999999999999995},"max":{"1min":1.5620000000000001,"5min":1.74,"15min":1.8220000000000001},"last":1.373}]},{"osd":4,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.69999999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5760000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1},{"interface":"front","average":{"1min":1.407,"5min":1.448,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7250000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.552}]},{"osd":5,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2829999999999999,"15min":1.288},"min":{"1min":0.73299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.619,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.323,"15min":1.3049999999999999},"min":{"1min":1.034,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.7709999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3759999999999999,"15min":1.3859999999999999},"min":{"1min":0.97299999999999998,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.927,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.335},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7210000000000001,"5min":1.8899999999999999,"15min":2.2410000000000001},"last":1.22}]}]},{"osd":5,"up_from":13,"seq":55834575967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308032,"kb_used_data":2552,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063808,"statfs":{"total":96636764160,"available":96321339392,"internally_reserved":0,"allocated":2613248,"data_stored":1396198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.3069999999999999,"15min":1.3160000000000001},"min":{"1min":0.878,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.631,"5min":2.044,"15min":2.044},"last":1.369},{"interface":"front","average":{"1min":1.409,"5min":1.351,"15min":1.3520000000000001},"min":{"1min":1.006,"5min":0.84899999999999998,"15min":0.58199999999999996},"max":{"1min":1.994,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.4550000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.1579999999999999,"15min":1.1759999999999999},"min":{"1min":0.83299999999999996,"5min":0.78800000000000003,"15min":0.76200000000000001},"max":{"1min":1.502,"5min":1.5569999999999999,"15min":1.6479999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1799999999999999,"15min":1.206},"min":{"1min":0.97899999999999998,"5min":0.74299999999999999,"15min":0.58099999999999996},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.8140000000000001},"last":1.911}]},{"osd":2,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.208,"15min":1.204},"min":{"1min":0.89500000000000002,"5min":0.84699999999999998,"15min":0.76800000000000002},"max":{"1min":1.6319999999999999,"5min":1.782,"15min":1.782},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.236,"15min":1.2529999999999999},"min":{"1min":1.0489999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.175}]},{"osd":3,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.3169999999999999,"15min":1.3149999999999999},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.655},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.351,"15min":1.3320000000000001},"min":{"1min":1.111,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.1749999999999998,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.6899999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2849999999999999,"15min":1.2450000000000001},"min":{"1min":1.2090000000000001,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3520000000000001,"15min":1.3160000000000001},"min":{"1min":1.0820000000000001,"5min":0.877,"15min":0.59099999999999997},"max":{"1min":2.1309999999999998,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.542}]},{"osd":6,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3600000000000001,"15min":1.329},"min":{"1min":0.98499999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.4059999999999999,"15min":1.385},"min":{"1min":0.99199999999999999,"5min":0.76500000000000001,"15min":0.68999999999999995},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.28}]},{"osd":7,"last update":"Tue Dec 19 09:20:28 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.472,"15min":1.4470000000000001},"min":{"1min":1.206,"5min":0.95799999999999996,"15min":0.84199999999999997},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.762},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.5009999999999999,"15min":1.4870000000000001},"min":{"1min":1.3,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.3639999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.5920000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575967,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312140,"kb_used_data":2564,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059700,"statfs":{"total":96636764160,"available":96317132800,"internally_reserved":0,"allocated":2625536,"data_stored":1403394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.169},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.6659999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":1.0349999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":1.5249999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":1.3},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.415}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.7470000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.3939999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.49},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.8089999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.591}]}]},{"osd":3,"up_from":13,"seq":55834575967,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312132,"kb_used_data":2556,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059708,"statfs":{"total":96636764160,"available":96317140992,"internally_reserved":0,"allocated":2617344,"data_stored":1401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.89100000000000001,"15min":0.878},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.373,"5min":1.581,"15min":1.581},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55100000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3200000000000001,"5min":2.117,"15min":2.117},"last":1.0649999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.92800000000000005,"15min":0.93600000000000005},"min":{"1min":0.57099999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.512,"5min":2.331,"15min":2.331},"last":1.153},{"interface":"front","average":{"1min":0.999,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.61199999999999999,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.4550000000000001,"5min":2.23,"15min":2.23},"last":0.93999999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.101,"15min":1.1200000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.4199999999999999,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1890000000000001},"min":{"1min":0.86299999999999999,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.8580000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.02}]},{"osd":4,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.337,"15min":1.3169999999999999},"min":{"1min":0.86699999999999999,"5min":0.85799999999999998,"15min":0.70199999999999996},"max":{"1min":2.081,"5min":2.157,"15min":2.157},"last":1.327},{"interface":"front","average":{"1min":1.304,"5min":1.3049999999999999,"15min":1.284},"min":{"1min":0.93600000000000005,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":2.149,"5min":2.149,"15min":2.149},"last":1.492}]},{"osd":5,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.3100000000000001,"15min":1.306},"min":{"1min":1.0349999999999999,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.2229999999999999,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.397,"15min":1.387},"min":{"1min":1.0429999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0630000000000002,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.198}]},{"osd":6,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.4139999999999999,"15min":1.421},"min":{"1min":0.89800000000000002,"5min":0.752,"15min":0.752},"max":{"1min":2.004,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.47},{"interface":"front","average":{"1min":1.401,"5min":1.4139999999999999,"15min":1.4239999999999999},"min":{"1min":0.99399999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":2.0289999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.5489999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5069999999999999,"15min":1.5},"min":{"1min":1.006,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":2.3780000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4610000000000001,"15min":1.4510000000000001},"min":{"1min":0.96999999999999997,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0129999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.581}]}]},{"osd":2,"up_from":13,"seq":55834575967,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308020,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063820,"statfs":{"total":96636764160,"available":96321351680,"internally_reserved":0,"allocated":2600960,"data_stored":1395194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2529999999999999,"5min":1.345,"15min":1.5049999999999999},"last":0.877},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.286,"5min":1.556,"15min":1.6759999999999999},"last":1.012}]},{"osd":1,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3640000000000001,"5min":1.569,"15min":1.7609999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.034,"15min":1.044},"min":{"1min":0.59499999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4339999999999999,"5min":1.5369999999999999,"15min":1.768},"last":1.3280000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.472,"5min":1.53,"15min":1.891},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.121,"5min":1.1359999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":1.302}]},{"osd":4,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.341},"min":{"1min":0.76900000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.754},"last":1.222},{"interface":"front","average":{"1min":1.242,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.556,"5min":1.76,"15min":1.8520000000000001},"last":1.5529999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3759999999999999},"min":{"1min":0.84799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6679999999999999,"5min":1.948,"15min":2.016},"last":1.468},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.91200000000000003,"5min":0.83399999999999996,"15min":0.68300000000000005},"max":{"1min":1.6000000000000001,"5min":1.901,"15min":1.927},"last":1.6599999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3120000000000001,"15min":1.345},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8200000000000001,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.738},{"interface":"front","average":{"1min":1.379,"5min":1.365,"15min":1.419},"min":{"1min":1.036,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.516}]},{"osd":7,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.4059999999999999,"15min":1.4370000000000001},"min":{"1min":0.94799999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.0489999999999999,"15min":2.327},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9059999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.802}]}]},{"osd":1,"up_from":13,"seq":55834575968,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312112,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059728,"statfs":{"total":96636764160,"available":96317161472,"internally_reserved":0,"allocated":2596864,"data_stored":1394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86299999999999999,"15min":0.87},"min":{"1min":0.45200000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.258,"5min":1.4239999999999999,"15min":1.9039999999999999},"last":1.028},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94899999999999995,"15min":0.997},"min":{"1min":0.59699999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.24,"5min":1.575,"15min":1.6850000000000001},"last":0.63}]},{"osd":2,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.95699999999999996,"15min":0.95999999999999996},"min":{"1min":0.66200000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0269999999999999,"15min":1.0669999999999999},"min":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3740000000000001,"5min":1.5760000000000001,"15min":2.302},"last":1.1799999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1459999999999999,"15min":1.173},"min":{"1min":0.68100000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.4219999999999999,"5min":1.8899999999999999,"15min":2.0950000000000002},"last":1.4470000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0580000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.101}]},{"osd":4,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2549999999999999,"15min":1.252},"min":{"1min":0.96399999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.619,"5min":1.784,"15min":1.784},"last":1.252},{"interface":"front","average":{"1min":1.284,"5min":1.3109999999999999,"15min":1.3260000000000001},"min":{"1min":0.996,"5min":0.90500000000000003,"15min":0.83899999999999997},"max":{"1min":1.5,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":1.528}]},{"osd":5,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.298,"15min":1.333},"min":{"1min":1.0609999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.661,"5min":1.944,"15min":1.944},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3220000000000001,"15min":1.3129999999999999},"min":{"1min":0.93000000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.706,"5min":1.8779999999999999,"15min":2.117},"last":1.2110000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3899999999999999,"15min":1.4059999999999999},"min":{"1min":0.98099999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.8200000000000001,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.381,"15min":1.419},"min":{"1min":1.1719999999999999,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.3999999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:52 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4490000000000001,"15min":1.458},"min":{"1min":1.2210000000000001,"5min":1.01,"15min":1.0029999999999999},"max":{"1min":1.806,"5min":1.806,"15min":2.069},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4279999999999999,"15min":1.4339999999999999},"min":{"1min":1.147,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.736,"5min":1.968,"15min":2.2360000000000002},"last":1.169}]}]},{"osd":0,"up_from":13,"seq":55834575967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300416,"kb_used_data":3128,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071424,"statfs":{"total":96636764160,"available":96329138176,"internally_reserved":0,"allocated":3203072,"data_stored":1984697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.85299999999999998,"15min":0.877},"min":{"1min":0.63400000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.242,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.96399999999999997,"15min":0.97299999999999998},"min":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.50800000000000001},"max":{"1min":1.484,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.77000000000000002}]},{"osd":2,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.0029999999999999,"15min":1.0269999999999999},"min":{"1min":0.71499999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.4419999999999999,"5min":1.446,"15min":1.5269999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.073,"15min":1.0660000000000001},"min":{"1min":0.71599999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.627,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.877}]},{"osd":3,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.026,"15min":1.0129999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.622},"max":{"1min":1.3740000000000001,"5min":1.611,"15min":2.621},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.1599999999999999,"15min":1.177},"min":{"1min":0.75600000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.718},"last":1.024}]},{"osd":4,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.357,"15min":1.3799999999999999},"min":{"1min":0.80900000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6519999999999999,"5min":1.891,"15min":1.891},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.337,"15min":1.3440000000000001},"min":{"1min":0.875,"5min":0.80300000000000005,"15min":0.69599999999999995},"max":{"1min":1.8560000000000001,"5min":1.919,"15min":2.5339999999999998},"last":1.381}]},{"osd":5,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.391,"15min":1.399},"min":{"1min":0.99299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.8100000000000001,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4219999999999999,"15min":1.417},"min":{"1min":1.1399999999999999,"5min":0.83499999999999996,"15min":0.79500000000000004},"max":{"1min":1.615,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.329}]},{"osd":6,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.294,"15min":1.278},"min":{"1min":0.98899999999999999,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":1.772,"5min":2.0840000000000001,"15min":2.4489999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.399,"15min":1.371},"min":{"1min":1.054,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.736,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.0609999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:20:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3460000000000001,"15min":1.343},"min":{"1min":1.0509999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.6080000000000001,"5min":2.137,"15min":2.8570000000000002},"last":1.111},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4359999999999999,"15min":1.4319999999999999},"min":{"1min":1.0289999999999999,"5min":0.92200000000000004,"15min":0.84399999999999997},"max":{"1min":1.708,"5min":2.1259999999999999,"15min":2.181},"last":1.1359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:21:27.648 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-19T09:21:27.648 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-19T09:21:27.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:21:27.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:21:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.859+0000 7fbb1fef0700 1 Processor -- start 2023-12-19T09:21:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.859+0000 7fbb1fef0700 1 -- start start 2023-12-19T09:21:27.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbb18158e20 0x7fbb180ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb180ca2c0 0x7fbb180ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb180cbdf0 con 0x7fbb180cb970 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb180cc0d0 con 0x7fbb180ca2c0 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb180cc3b0 con 0x7fbb18158e20 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1d48b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbb180cb970 0x7fbb1806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46488/0 (socket says 172.21.15.111:46488) 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1d48b700 1 -- 172.21.15.111:0/3376939785 learned_addr learned my addr 172.21.15.111:0/3376939785 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 --2- 172.21.15.111:0/3376939785 >> v2:172.21.15.111:3300/0 conn(0x7fbb180ca2c0 0x7fbb180ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1dc8c700 1 --2- 172.21.15.111:0/3376939785 >> v2:172.21.15.143:3300/0 conn(0x7fbb18158e20 0x7fbb180ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 -- 172.21.15.111:0/3376939785 >> v2:172.21.15.143:3300/0 conn(0x7fbb18158e20 msgr2=0x7fbb180ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:27.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 --2- 172.21.15.111:0/3376939785 >> v2:172.21.15.143:3300/0 conn(0x7fbb18158e20 0x7fbb180ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 -- 172.21.15.111:0/3376939785 >> v1:172.21.15.111:6789/0 conn(0x7fbb180cb970 legacy=0x7fbb1806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 -- 172.21.15.111:0/3376939785 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb180cc690 con 0x7fbb180ca2c0 2023-12-19T09:21:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1e48d700 1 --2- 172.21.15.111:0/3376939785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb180ceb00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009e60 tx=0x7fbb10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc99fd0ed104a1fc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1cc8a700 1 -- 172.21.15.111:0/3376939785 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1000e070 con 0x7fbb180ca2c0 2023-12-19T09:21:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1cc8a700 1 -- 172.21.15.111:0/3376939785 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1000bb20 con 0x7fbb180ca2c0 2023-12-19T09:21:27.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1cc8a700 1 -- 172.21.15.111:0/3376939785 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10012430 con 0x7fbb180ca2c0 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 -- 172.21.15.111:0/3376939785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 msgr2=0x7fbb180ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.863+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/3376939785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb180ceb00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009e60 tx=0x7fbb10009780 comp rx=0 tx=0).stop 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 -- 172.21.15.111:0/3376939785 shutdown_connections 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/3376939785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb180ceb00 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/3376939785 >> v2:172.21.15.143:3300/0 conn(0x7fbb18158e20 0x7fbb180ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 -- 172.21.15.111:0/3376939785 >> 172.21.15.111:0/3376939785 conn(0x7fbb18067bb0 msgr2=0x7fbb180d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 -- 172.21.15.111:0/3376939785 shutdown_connections 2023-12-19T09:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 -- 172.21.15.111:0/3376939785 wait complete. 2023-12-19T09:21:27.874 INFO:tasks.ceph.mds.b.smithi143.stderr:2023-12-19T09:21:27.868+0000 7f53d958c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:21:27.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 Processor -- start 2023-12-19T09:21:27.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.867+0000 7fbb1fef0700 1 -- start start 2023-12-19T09:21:27.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb18158e20 0x7fbb1816a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 0x7fbb1815dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1815e510 con 0x7fbb180ca2c0 2023-12-19T09:21:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1814fa60 con 0x7fbb18158e20 2023-12-19T09:21:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1816a750 con 0x7fbb1815db30 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40446/0 (socket says 172.21.15.111:40446) 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1d48b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb18158e20 0x7fbb1816a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 -- 172.21.15.111:0/452877184 learned_addr learned my addr 172.21.15.111:0/452877184 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1e48d700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 0x7fbb1815dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 -- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 msgr2=0x7fbb1815dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 0x7fbb1815dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 -- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb18158e20 msgr2=0x7fbb1816a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb18158e20 0x7fbb1816a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 -- 172.21.15.111:0/452877184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb10009a20 con 0x7fbb180ca2c0 2023-12-19T09:21:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1e48d700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 0x7fbb1815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1dc8c700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400e930 tx=0x7fbb140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14013680 con 0x7fbb180ca2c0 2023-12-19T09:21:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb140137e0 con 0x7fbb180ca2c0 2023-12-19T09:21:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1814eb70 con 0x7fbb180ca2c0 2023-12-19T09:21:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.871+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14013b80 con 0x7fbb180ca2c0 2023-12-19T09:21:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.875+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1814f130 con 0x7fbb180ca2c0 2023-12-19T09:21:27.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.875+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb18071120 con 0x7fbb180ca2c0 2023-12-19T09:21:27.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.875+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbb1400c3e0 con 0x7fbb180ca2c0 2023-12-19T09:21:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.875+0000 7fbb0dffb700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 0x7fbafc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.879+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fbb1407cfe0 con 0x7fbb180ca2c0 2023-12-19T09:21:27.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.879+0000 7fbb1d48b700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 0x7fbafc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:27.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.879+0000 7fbb1d48b700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 0x7fbafc062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb1814d2c0 tx=0x7fbb08007ed0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:27.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:27.883+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbb1404d1b0 con 0x7fbb180ca2c0 2023-12-19T09:21:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.103+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbb1806a7a0 con 0x7fbb180ca2c0 2023-12-19T09:21:28.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.115+0000 7fbb0dffb700 1 -- 172.21.15.111:0/452877184 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbb140507d0 con 0x7fbb180ca2c0 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 msgr2=0x7fbafc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 0x7fbafc062a00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb1814d2c0 tx=0x7fbb08007ed0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 msgr2=0x7fbb1814cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400e930 tx=0x7fbb140091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 shutdown_connections 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbafc060550 0x7fbafc062a00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb18158e20 0x7fbb1816a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb1815db30 0x7fbb1815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 --2- 172.21.15.111:0/452877184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb180ca2c0 0x7fbb1814cb40 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 >> 172.21.15.111:0/452877184 conn(0x7fbb18067bb0 msgr2=0x7fbb1806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:28.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 shutdown_connections 2023-12-19T09:21:28.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.119+0000 7fbb1fef0700 1 -- 172.21.15.111:0/452877184 wait complete. 2023-12-19T09:21:28.128 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-12-19T09:21:28.144 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:28.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.299+0000 7f18546ae700 1 Processor -- start 2023-12-19T09:21:28.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.299+0000 7f18546ae700 1 -- start start 2023-12-19T09:21:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f18546ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f184c140710 0x7f184c0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f18546ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c06e150 con 0x7f184c068800 2023-12-19T09:21:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f18546ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c06d540 con 0x7f184c140ae0 2023-12-19T09:21:28.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f18546ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c06d820 con 0x7f184c140710 2023-12-19T09:21:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1852c4b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f184c068800 0x7f184c0ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46500/0 (socket says 172.21.15.111:46500) 2023-12-19T09:21:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1852c4b700 1 -- 172.21.15.111:0/4072566650 learned_addr learned my addr 172.21.15.111:0/4072566650 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:28.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f185244a700 1 --2- 172.21.15.111:0/4072566650 >> v2:172.21.15.143:3300/0 conn(0x7f184c140710 0x7f184c0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488489122 0 0) 0x7f184c06e150 con 0x7f184c068800 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1838003570 con 0x7f184c068800 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264242610 0 0) 0x7f184c06d540 con 0x7f184c140ae0 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f185244a700 1 -- 172.21.15.111:0/4072566650 >> v1:172.21.15.143:6789/0 conn(0x7f184c140ae0 legacy=0x7f184c0680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f185244a700 1 -- 172.21.15.111:0/4072566650 >> v1:172.21.15.111:6789/0 conn(0x7f184c068800 legacy=0x7f184c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f185244a700 1 -- 172.21.15.111:0/4072566650 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c06db00 con 0x7f184c140710 2023-12-19T09:21:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f185244a700 1 --2- 172.21.15.111:0/4072566650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140710 0x7f184c0c70a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f18480098e0 tx=0x7f18480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55fcb3612b0ba8e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184800b640 con 0x7f184c140710 2023-12-19T09:21:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184800b7a0 con 0x7f184c140710 2023-12-19T09:21:28.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.303+0000 7f1851448700 1 -- 172.21.15.111:0/4072566650 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184800bb40 con 0x7f184c140710 2023-12-19T09:21:28.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- 172.21.15.111:0/4072566650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140710 msgr2=0x7f184c0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 --2- 172.21.15.111:0/4072566650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140710 0x7f184c0c70a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f18480098e0 tx=0x7f18480091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- 172.21.15.111:0/4072566650 shutdown_connections 2023-12-19T09:21:28.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 --2- 172.21.15.111:0/4072566650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140710 0x7f184c0c70a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- 172.21.15.111:0/4072566650 >> 172.21.15.111:0/4072566650 conn(0x7f184c0c8850 msgr2=0x7f184c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:28.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- 172.21.15.111:0/4072566650 shutdown_connections 2023-12-19T09:21:28.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- 172.21.15.111:0/4072566650 wait complete. 2023-12-19T09:21:28.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 Processor -- start 2023-12-19T09:21:28.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- start start 2023-12-19T09:21:28.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:28.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 0x7f184c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140ae0 0x7f184c13ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0aff90 con 0x7f184c068800 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0af0e0 con 0x7f184c140ae0 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.307+0000 7f18546ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0af380 con 0x7f184c140710 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40458/0 (socket says 172.21.15.111:40458) 2023-12-19T09:21:28.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 -- 172.21.15.111:0/75794957 learned_addr learned my addr 172.21.15.111:0/75794957 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:28.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f1852c4b700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140ae0 0x7f184c13ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f1851c49700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 0x7f184c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 -- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 msgr2=0x7f184c0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 0x7f184c0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 -- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140ae0 msgr2=0x7f184c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140ae0 0x7f184c13ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 -- 172.21.15.111:0/75794957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848009420 con 0x7f184c068800 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f1851c49700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 0x7f184c0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f185244a700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f184c06a280 tx=0x7f1848017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184801f070 con 0x7f184c068800 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184800bb20 con 0x7f184c068800 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c0af660 con 0x7f184c068800 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848011670 con 0x7f184c068800 2023-12-19T09:21:28.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c0afc20 con 0x7f184c068800 2023-12-19T09:21:28.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.311+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c0ad3d0 con 0x7f184c068800 2023-12-19T09:21:28.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.315+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1848011a60 con 0x7f184c068800 2023-12-19T09:21:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.315+0000 7f18427fc700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 0x7f1830062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.315+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f184807dca0 con 0x7f184c068800 2023-12-19T09:21:28.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.319+0000 7f1851c49700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 0x7f1830062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:28.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.319+0000 7f1851c49700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 0x7f1830062a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f183c0011e0 tx=0x7f183c009040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.323+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f184804df10 con 0x7f184c068800 2023-12-19T09:21:28.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.527+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f184c071120 con 0x7f184c068800 2023-12-19T09:21:28.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.527+0000 7f18427fc700 1 -- 172.21.15.111:0/75794957 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7f1848051530 con 0x7f184c068800 2023-12-19T09:21:28.534 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:28.534 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 msgr2=0x7f1830062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 0x7f1830062a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f183c0011e0 tx=0x7f183c009040 comp rx=0 tx=0).stop 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 msgr2=0x7f184c0c5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f184c06a280 tx=0x7f1848017000 comp rx=0 tx=0).stop 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 shutdown_connections 2023-12-19T09:21:28.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1830060580 0x7f1830062a30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c140ae0 0x7f184c13ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c140710 0x7f184c0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 --2- 172.21.15.111:0/75794957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f184c068800 0x7f184c0c5960 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 >> 172.21.15.111:0/75794957 conn(0x7f184c0c8850 msgr2=0x7f184c0675b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 shutdown_connections 2023-12-19T09:21:28.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:28.531+0000 7f18546ae700 1 -- 172.21.15.111:0/75794957 wait complete. 2023-12-19T09:21:28.539 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:28.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-19T09:21:22.352171+0000', 'modified': '2023-12-19T09:21:25.953156+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10120}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10120': {'gid': 10120, 'name': 'b', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.143:6837/68388372', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 68388372}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 68388372}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-12-19T09:21:28.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-19T09:21:28.962 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-19T09:21:29.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.131+0000 7f86760f0700 1 Processor -- start 2023-12-19T09:21:29.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.131+0000 7f86760f0700 1 -- start start 2023-12-19T09:21:29.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f86760f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 0x7f867006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f86760f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f86760f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867006b0d0 con 0x7f867006baa0 2023-12-19T09:21:29.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f86760f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867006beb0 con 0x7f8670158cc0 2023-12-19T09:21:29.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f86760f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86700b4790 con 0x7f867006dc00 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46050/0 (socket says 172.21.15.111:46050) 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 -- 172.21.15.111:0/1697356104 learned_addr learned my addr 172.21.15.111:0/1697356104 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f866f7fe700 1 --2- 172.21.15.111:0/1697356104 >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 0x7f867006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 -- 172.21.15.111:0/1697356104 >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 msgr2=0x7f867006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 --2- 172.21.15.111:0/1697356104 >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 0x7f867006d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 -- 172.21.15.111:0/1697356104 >> v1:172.21.15.111:6789/0 conn(0x7f867006baa0 legacy=0x7f86700b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 -- 172.21.15.111:0/1697356104 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867006d710 con 0x7f867006dc00 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f866f7fe700 1 --2- 172.21.15.111:0/1697356104 >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 0x7f867006d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.135+0000 7f8667fff700 1 --2- 172.21.15.111:0/1697356104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f86580099b0 tx=0x7f8658009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a8a0021c9f7d9e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f866effd700 1 -- 172.21.15.111:0/1697356104 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865800b860 con 0x7f867006dc00 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f866effd700 1 -- 172.21.15.111:0/1697356104 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865800b9c0 con 0x7f867006dc00 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f866effd700 1 -- 172.21.15.111:0/1697356104 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865800bd60 con 0x7f867006dc00 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- 172.21.15.111:0/1697356104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 msgr2=0x7f867006b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 --2- 172.21.15.111:0/1697356104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f86580099b0 tx=0x7f8658009250 comp rx=0 tx=0).stop 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- 172.21.15.111:0/1697356104 shutdown_connections 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 --2- 172.21.15.111:0/1697356104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f867006b560 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 --2- 172.21.15.111:0/1697356104 >> v2:172.21.15.143:3300/0 conn(0x7f8670158cc0 0x7f867006d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- 172.21.15.111:0/1697356104 >> 172.21.15.111:0/1697356104 conn(0x7f8670067bb0 msgr2=0x7f86700cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- 172.21.15.111:0/1697356104 shutdown_connections 2023-12-19T09:21:29.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- 172.21.15.111:0/1697356104 wait complete. 2023-12-19T09:21:29.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 Processor -- start 2023-12-19T09:21:29.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.139+0000 7f86760f0700 1 -- start start 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f8670153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8670158cc0 0x7f86700a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8670154010 con 0x7f867006baa0 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700aec20 con 0x7f8670158cc0 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86700af4c0 con 0x7f867006dc00 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40464/0 (socket says 172.21.15.111:40464) 2023-12-19T09:21:29.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 -- 172.21.15.111:0/1236543334 learned_addr learned my addr 172.21.15.111:0/1236543334 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f8667fff700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f8670153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866ffff700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8670158cc0 0x7f86700a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 -- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 msgr2=0x7f8670153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f8670153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 -- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8670158cc0 msgr2=0x7f86700a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8670158cc0 0x7f86700a9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 -- 172.21.15.111:0/1236543334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580094f0 con 0x7f867006baa0 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f866f7fe700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f866000e8d0 tx=0x7f8660008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86677fe700 1 -- 172.21.15.111:0/1236543334 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660013410 con 0x7f867006baa0 2023-12-19T09:21:29.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86677fe700 1 -- 172.21.15.111:0/1236543334 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8660013570 con 0x7f867006baa0 2023-12-19T09:21:29.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86700acea0 con 0x7f867006baa0 2023-12-19T09:21:29.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86677fe700 1 -- 172.21.15.111:0/1236543334 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660013910 con 0x7f867006baa0 2023-12-19T09:21:29.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.143+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700af150 con 0x7f867006baa0 2023-12-19T09:21:29.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.147+0000 7f86677fe700 1 -- 172.21.15.111:0/1236543334 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8660013d00 con 0x7f867006baa0 2023-12-19T09:21:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f86677fe700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 0x7f8650062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f86677fe700 1 -- 172.21.15.111:0/1236543334 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f866007c9b0 con 0x7f867006baa0 2023-12-19T09:21:29.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f8667fff700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 0x7f8650062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f86783b4740 1 Processor -- start 2023-12-19T09:21:29.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f86783b4740 1 -- start start 2023-12-19T09:21:29.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.151+0000 7f8667fff700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 0x7f8650062a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f86580099b0 tx=0x7f8658017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x259c790 0x27cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x26fcf30 con 0x2551ae0 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x26f8ed0 con 0x259c790 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x26bf0b0 con 0x2551710 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40480/0 (socket says 172.21.15.111:40480) 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 -- 172.21.15.111:0/2325777608 learned_addr learned my addr 172.21.15.111:0/2325777608 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866ffff700 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x259c790 0x27cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f8667fff700 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 -- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 msgr2=0x27cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 -- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x259c790 msgr2=0x27cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x259c790 0x27cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 -- 172.21.15.111:0/2325777608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x259d9e0 con 0x2551ae0 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f866f7fe700 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f866007aa80 tx=0x7f866007aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05ecf7cdf97612d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f8664ff9700 1 -- 172.21.15.111:0/2325777608 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660057820 con 0x2551ae0 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f8664ff9700 1 -- 172.21.15.111:0/2325777608 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8660050840 con 0x2551ae0 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f8664ff9700 1 -- 172.21.15.111:0/2325777608 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866004ebe0 con 0x2551ae0 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- 172.21.15.111:0/2325777608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 msgr2=0x27cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f866007aa80 tx=0x7f866007aa10 comp rx=0 tx=0).stop 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- 172.21.15.111:0/2325777608 shutdown_connections 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x259c790 0x27cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 --2- 172.21.15.111:0/2325777608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x27cef00 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- 172.21.15.111:0/2325777608 >> 172.21.15.111:0/2325777608 conn(0x2551eb0 msgr2=0x278b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- 172.21.15.111:0/2325777608 shutdown_connections 2023-12-19T09:21:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.155+0000 7f86783b4740 1 -- 172.21.15.111:0/2325777608 wait complete. 2023-12-19T09:21:29.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 -- start start 2023-12-19T09:21:29.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 Processor -- start 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x26bed50 con 0x2551710 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2697970 con 0x2551ae0 2023-12-19T09:21:29.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x26ff290 con 0x259c790 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f86783b4740 10 client.0 _ll_register_callbacks cb 0x27426a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40486/0 (socket says 172.21.15.111:40486) 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 -- 172.21.15.111:0/720745005 learned_addr learned my addr 172.21.15.111:0/720745005 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866ffff700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f8667fff700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 -- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 msgr2=0x27b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 -- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 msgr2=0x27d3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27d3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f866f7fe700 1 -- 172.21.15.111:0/720745005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700af150 con 0x2551710 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.159+0000 7f8667fff700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f866f7fe700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f8660018000 tx=0x7f8660064000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e5dd51aab51255 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660013a70 con 0x2551710 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f866004c630 con 0x2551710 2023-12-19T09:21:29.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2697cc0 con 0x2551710 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7f866006d080 con 0x2551710 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 10 client.10569 fetch_fsmap learned FSMap version 95 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x26c7370 con 0x2551710 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2697cc0 con 0x2551710 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 95) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f866004f7a0 con 0x2551710 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 10 client.10569 fetch_fsmap finished waiting for FSMap version 95 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 10 client.10569 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T09:21:29.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 20 client.10569 populate_metadata read hostname 'smithi111' 2023-12-19T09:21:29.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x27b63a0 0x27de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 4 client.10569 mds_command: new command op to 10120 tid=0[{"prefix": "get_command_descriptions"}] 2023-12-19T09:21:29.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 --> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x269b3e0 con 0x27b63a0 2023-12-19T09:21:29.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f8667fff700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x27b63a0 0x27de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f8667fff700 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x27b63a0 0x27de6b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca782e9406d294b server_cookie=f94f8fe2c7a4927c in_seq=0 out_seq=0 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.163+0000 7f864dffb700 10 client.10569 ms_handle_connect on v2:172.21.15.143:6836/68388372 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f864dffb700 1 -- 172.21.15.111:0/720745005 <== mds.0 v2:172.21.15.143:6836/68388372 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x269b3e0 con 0x27b63a0 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f864dffb700 10 client.10569 handle_command_reply: tid=0 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 client.10569 shutdown 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 msgr2=0x27d3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f8660018000 tx=0x7f8660064000 comp rx=0 tx=0).stop 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 shutdown_connections 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x27b63a0 0x27de6b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f864dffb700 0 client.10569 ms_handle_reset on v2:172.21.15.143:6836/68388372 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 --2- 172.21.15.111:0/720745005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551710 0x27d3510 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 >> 172.21.15.111:0/720745005 conn(0x2551eb0 msgr2=0x2783f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 shutdown_connections 2023-12-19T09:21:29.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 1 -- 172.21.15.111:0/720745005 wait complete. 2023-12-19T09:21:29.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.167+0000 7f86783b4740 20 client.10569 trim_cache size 0 max 16384 2023-12-19T09:21:29.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 Processor -- start 2023-12-19T09:21:29.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 -- start start 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27b2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x26fdef0 con 0x259c790 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x2699b80 con 0x2551ae0 2023-12-19T09:21:29.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x26f4b50 con 0x2551710 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.187+0000 7f866ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40502/0 (socket says 172.21.15.111:40502) 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 -- 172.21.15.111:0/407708157 learned_addr learned my addr 172.21.15.111:0/407708157 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f8667fff700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866f7fe700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 -- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 msgr2=0x27b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 -- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 msgr2=0x27b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 -- 172.21.15.111:0/407708157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x26add80 con 0x259c790 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866f7fe700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f866ffff700 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8668003ac0 tx=0x7f8668009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1cf1bf3a154c8f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86657fa700 1 -- 172.21.15.111:0/407708157 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866801c030 con 0x259c790 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86657fa700 1 -- 172.21.15.111:0/407708157 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8668001780 con 0x259c790 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86657fa700 1 -- 172.21.15.111:0/407708157 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668001b20 con 0x259c790 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- 172.21.15.111:0/407708157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 msgr2=0x278b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8668003ac0 tx=0x7f8668009040 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- 172.21.15.111:0/407708157 shutdown_connections 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551ae0 0x27b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x259c790 0x278b530 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 --2- 172.21.15.111:0/407708157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x2551710 0x27b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- 172.21.15.111:0/407708157 >> 172.21.15.111:0/407708157 conn(0x2551eb0 msgr2=0x27b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- 172.21.15.111:0/407708157 shutdown_connections 2023-12-19T09:21:29.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- 172.21.15.111:0/407708157 wait complete. 2023-12-19T09:21:29.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 -- start start 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.191+0000 7f86783b4740 1 Processor -- start 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551710 0x28286e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x2828c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x26fb590 con 0x2551ae0 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x26cd610 con 0x2551710 2023-12-19T09:21:29.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x26fc9d0 con 0x259c790 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 10 client.0 _ll_register_callbacks cb 0x251d9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f8667fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x2828c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:46110/0 (socket says 172.21.15.111:46110) 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 -- 172.21.15.111:0/3287969901 learned_addr learned my addr 172.21.15.111:0/3287969901 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866f7fe700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551710 0x28286e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 -- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551710 msgr2=0x28286e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551710 0x28286e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 -- 172.21.15.111:0/3287969901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 msgr2=0x2828c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x2828c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 -- 172.21.15.111:0/3287969901 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866800f040 con 0x259c790 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f8667fff700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x2828c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f866ffff700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x26c83a0 tx=0x7f866801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8575ad1ccb9bf42d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 10 client.0 ms_handle_connect on v2:172.21.15.143:3301/0 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668001b10 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mon.2 v2:172.21.15.143:3301/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f866801d070 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8660050840 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8668029e40 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 10 client.8897 fetch_fsmap learned FSMap version 95 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x26a2130 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mon.2 v2:172.21.15.143:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8660050840 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mon.2 v2:172.21.15.143:3301/0 5 ==== fsmap(e 95) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f866802c070 con 0x259c790 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 10 client.8897 fetch_fsmap finished waiting for FSMap version 95 2023-12-19T09:21:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 10 client.8897 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-19T09:21:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 20 client.8897 populate_metadata read hostname 'smithi111' 2023-12-19T09:21:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x2788e50 0x2789220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 4 client.8897 mds_command: new command op to 10120 tid=0[{"prefix": "status"}] 2023-12-19T09:21:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.195+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 --> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] -- command(tid 0: {"prefix": "status"}) v1 -- 0x26cd410 con 0x2788e50 2023-12-19T09:21:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f866f7fe700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x2788e50 0x2789220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f864dffb700 10 client.8897 ms_handle_connect on v2:172.21.15.143:6836/68388372 2023-12-19T09:21:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f866f7fe700 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x2788e50 0x2789220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea78e123a260e23 server_cookie=d4b61e66ae36e974 in_seq=0 out_seq=0 2023-12-19T09:21:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f864dffb700 1 -- 172.21.15.111:0/3287969901 <== mds.0 v2:172.21.15.143:6836/68388372 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x26cd410 con 0x2788e50 2023-12-19T09:21:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f864dffb700 10 client.8897 handle_command_reply: tid=0 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 client.8897 shutdown 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 msgr2=0x27884c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x26c83a0 tx=0x7f866801f000 comp rx=0 tx=0).stop 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 shutdown_connections 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x259c790 0x27884c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2551ae0 0x2828c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:6836/68388372,v1:172.21.15.143:6837/68388372] conn(0x2788e50 0x2789220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 --2- 172.21.15.111:0/3287969901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x2551710 0x28286e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 >> 172.21.15.111:0/3287969901 conn(0x2551eb0 msgr2=0x27d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f864dffb700 0 client.8897 ms_handle_reset on v2:172.21.15.143:6836/68388372 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 shutdown_connections 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 1 -- 172.21.15.111:0/3287969901 wait complete. 2023-12-19T09:21:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86783b4740 20 client.8897 trim_cache size 0 max 16384 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "4ae666be-184d-45ea-a875-cac9a231c35b", 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 10120, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.3033027400000003, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 95, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 208, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 208, 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 12.227915295000001 2023-12-19T09:21:29.210 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:21:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 msgr2=0x7f8650062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 0x7f8650062a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f86580099b0 tx=0x7f8658017000 comp rx=0 tx=0).stop 2023-12-19T09:21:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 msgr2=0x7f86700af950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.199+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f866000e8d0 tx=0x7f8660008040 comp rx=0 tx=0).stop 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 shutdown_connections 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8650060550 0x7f8650062a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8670158cc0 0x7f86700a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f867006dc00 0x7f8670153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 --2- 172.21.15.111:0/1236543334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f867006baa0 0x7f86700af950 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 >> 172.21.15.111:0/1236543334 conn(0x7f8670067bb0 msgr2=0x7f86700a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 shutdown_connections 2023-12-19T09:21:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.203+0000 7f86760f0700 1 -- 172.21.15.111:0/1236543334 wait complete. 2023-12-19T09:21:29.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T09:21:29.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.387+0000 7fdc45a2e700 1 Processor -- start 2023-12-19T09:21:29.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 -- start start 2023-12-19T09:21:29.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdc40140ae0 0x7fdc400680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc400ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4006e150 con 0x7fdc40140710 2023-12-19T09:21:29.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc40068ab0 con 0x7fdc40140ae0 2023-12-19T09:21:29.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc45a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4006d540 con 0x7fdc40068630 2023-12-19T09:21:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc37fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdc40140ae0 0x7fdc400680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc37fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdc40140ae0 0x7fdc400680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40518/0 (socket says 172.21.15.111:40518) 2023-12-19T09:21:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc37fff700 1 -- 172.21.15.111:0/3322803954 learned_addr learned my addr 172.21.15.111:0/3322803954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.391+0000 7fdc3f7fe700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc400ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc37fff700 1 -- 172.21.15.111:0/3322803954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 msgr2=0x7fdc400ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc37fff700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc400ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc37fff700 1 -- 172.21.15.111:0/3322803954 >> v1:172.21.15.143:6789/0 conn(0x7fdc40140710 legacy=0x7fdc400c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc37fff700 1 -- 172.21.15.111:0/3322803954 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4006d820 con 0x7fdc40140ae0 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc3e7fc700 1 -- 172.21.15.111:0/3322803954 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046828077 0 0) 0x7fdc4006e150 con 0x7fdc40140710 2023-12-19T09:21:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc37fff700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc400680f0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a80d3dcf5ae0a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc3e7fc700 1 -- 172.21.15.111:0/3322803954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc28009ea0 con 0x7fdc40140ae0 2023-12-19T09:21:29.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc3e7fc700 1 -- 172.21.15.111:0/3322803954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2800b860 con 0x7fdc40140ae0 2023-12-19T09:21:29.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc3e7fc700 1 -- 172.21.15.111:0/3322803954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2800bc00 con 0x7fdc40140ae0 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3322803954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 msgr2=0x7fdc400680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc400680f0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fdc280099b0 tx=0x7fdc28009250 comp rx=0 tx=0).stop 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3322803954 shutdown_connections 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc400ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3322803954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc400680f0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3322803954 >> 172.21.15.111:0/3322803954 conn(0x7fdc400c8850 msgr2=0x7fdc400c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3322803954 shutdown_connections 2023-12-19T09:21:29.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.395+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3322803954 wait complete. 2023-12-19T09:21:29.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 Processor -- start 2023-12-19T09:21:29.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 -- start start 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc4013bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc40140710 0x7fdc400c1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4013c680 con 0x7fdc40140ae0 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400c2480 con 0x7fdc40140710 2023-12-19T09:21:29.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400c1be0 con 0x7fdc40068630 2023-12-19T09:21:29.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40526/0 (socket says 172.21.15.111:40526) 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 -- 172.21.15.111:0/3806789972 learned_addr learned my addr 172.21.15.111:0/3806789972 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3effd700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc4013bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc37fff700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc40140710 0x7fdc400c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 -- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 msgr2=0x7fdc4013bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc4013bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 -- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc40140710 msgr2=0x7fdc400c1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc40140710 0x7fdc400c1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 -- 172.21.15.111:0/3806789972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280094f0 con 0x7fdc40140ae0 2023-12-19T09:21:29.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc3f7fe700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009df0 tx=0x7fdc3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800caa0 con 0x7fdc40140ae0 2023-12-19T09:21:29.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3800f040 con 0x7fdc40140ae0 2023-12-19T09:21:29.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc400c1ec0 con 0x7fdc40140ae0 2023-12-19T09:21:29.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.399+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800ce70 con 0x7fdc40140ae0 2023-12-19T09:21:29.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.403+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4015a5e0 con 0x7fdc40140ae0 2023-12-19T09:21:29.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.403+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4013c7f0 con 0x7fdc40140ae0 2023-12-19T09:21:29.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.403+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fdc38014690 con 0x7fdc40140ae0 2023-12-19T09:21:29.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.407+0000 7fdc36ffd700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 0x7fdc20062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:29.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.407+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fdc3807d3c0 con 0x7fdc40140ae0 2023-12-19T09:21:29.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.407+0000 7fdc3effd700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 0x7fdc20062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:29.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.407+0000 7fdc3effd700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 0x7fdc20062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc300039b0 tx=0x7fdc30007ec0 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:29.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.411+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdc38051020 con 0x7fdc40140ae0 2023-12-19T09:21:29.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.615+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdc40071120 con 0x7fdc40140ae0 2023-12-19T09:21:29.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.615+0000 7fdc36ffd700 1 -- 172.21.15.111:0/3806789972 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdc38050bb0 con 0x7fdc40140ae0 2023-12-19T09:21:29.621 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 msgr2=0x7fdc20062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 0x7fdc20062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdc300039b0 tx=0x7fdc30007ec0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 msgr2=0x7fdc4015a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009df0 tx=0x7fdc3800b040 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 shutdown_connections 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fdc20060550 0x7fdc20062a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc40140710 0x7fdc400c1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc40140ae0 0x7fdc4015a0a0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 --2- 172.21.15.111:0/3806789972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc40068630 0x7fdc4013bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 >> 172.21.15.111:0/3806789972 conn(0x7fdc400c8850 msgr2=0x7fdc4006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 shutdown_connections 2023-12-19T09:21:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:29.619+0000 7fdc45a2e700 1 -- 172.21.15.111:0/3806789972 wait complete. 2023-12-19T09:21:29.645 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T09:21:29.646 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:29.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.646 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.666 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 36768sec preferred_lft 36768sec 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:29.667 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T09:21:29.668 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:29.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.669 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T09:21:29.749 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T09:21:29.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.752 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T09:21:29.818 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T09:21:29.818 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T09:21:29.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.819 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T09:21:29.819 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T09:21:29.819 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T09:21:29.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T09:21:29.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T09:21:29.820 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T09:21:29.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.950 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:21:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:21:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:21:29.969 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:29.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:29.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T09:21:30.047 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T09:21:30.047 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:21:30.048 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:30.113 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:21:30.113 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:30.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:30.263 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:30.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:30.346 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:30.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:30.350 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:30.408 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T09:21:30.408 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T09:21:30.462 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:30.493 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:30.493 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:30.570 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:30.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:30.573 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T09:21:30.591+0000 7f1c07dc7380 -1 init, newargv = 0x55cfb0856340 newargc=15 2023-12-19T09:21:30.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63297]: starting ceph client 2023-12-19T09:21:30.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63297]: starting fuse 2023-12-19T09:21:31.711 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:31.750 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:31.750 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:31.825 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:31.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:31.829 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:31.889 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T09:21:31.890 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:31.890 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T09:21:31.891 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T09:21:32.053 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.63297.asok 2023-12-19T09:21:32.237 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63297.asok 2023-12-19T09:21:32.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:32.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63297.asok status 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T09:21:32.429 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "63297", 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 10587, 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 10587 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2830113293 2023-12-19T09:21:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2830113293 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.10587 192.168.0.1:0/2830113293", 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2830113293", 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 95, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 208, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T09:21:32.431 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:21:32.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:32.442 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T09:21:32.442 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:32.464 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T09:21:32.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:32.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:32.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:32.553 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-12-19T09:21:32.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.759+0000 7fc8f9723700 1 Processor -- start 2023-12-19T09:21:32.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.759+0000 7fc8f9723700 1 -- start start 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8f406a540 0x7fc8f406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 0x7fc8f40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f40ca5a0 con 0x7fc8f4158e70 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f406dac0 con 0x7fc8f406a540 2023-12-19T09:21:32.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f9723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b0260 con 0x7fc8f40cc470 2023-12-19T09:21:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8f406a540 0x7fc8f406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f8f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 0x7fc8f40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50130/0 (socket says 172.21.15.111:50130) 2023-12-19T09:21:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f3fff700 1 -- 172.21.15.111:0/1226057222 learned_addr learned my addr 172.21.15.111:0/1226057222 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 -- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 msgr2=0x7fc8f40cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 0x7fc8f40cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 -- 172.21.15.111:0/1226057222 >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 msgr2=0x7fc8f40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/1226057222 >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 -- 172.21.15.111:0/1226057222 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f406d220 con 0x7fc8f406a540 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f8f22700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 0x7fc8f40cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:32.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f3fff700 1 --2- 172.21.15.111:0/1226057222 >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.763+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f406a540 0x7fc8f406c9d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40cfd20 tx=0x7fc8e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7cca52c79b0e1cc server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f2ffd700 1 -- 172.21.15.111:0/1226057222 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8011630 con 0x7fc8f406a540 2023-12-19T09:21:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f2ffd700 1 -- 172.21.15.111:0/1226057222 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e80117b0 con 0x7fc8f406a540 2023-12-19T09:21:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f2ffd700 1 -- 172.21.15.111:0/1226057222 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8011b70 con 0x7fc8f406a540 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 -- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f406a540 msgr2=0x7fc8f406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f406a540 0x7fc8f406c9d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40cfd20 tx=0x7fc8e80091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 -- 172.21.15.111:0/1226057222 shutdown_connections 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f40cc470 0x7fc8f40cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 --2- 172.21.15.111:0/1226057222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f406a540 0x7fc8f406c9d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 --2- 172.21.15.111:0/1226057222 >> v2:172.21.15.111:3300/0 conn(0x7fc8f4158e70 0x7fc8f40afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 -- 172.21.15.111:0/1226057222 >> 172.21.15.111:0/1226057222 conn(0x7fc8f40670e0 msgr2=0x7fc8f40674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 -- 172.21.15.111:0/1226057222 shutdown_connections 2023-12-19T09:21:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.767+0000 7fc8f9723700 1 -- 172.21.15.111:0/1226057222 wait complete. 2023-12-19T09:21:32.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 Processor -- start 2023-12-19T09:21:32.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 -- start start 2023-12-19T09:21:32.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 0x7fc8f40c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f4158e70 0x7fc8f40c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b2c70 con 0x7fc8f406a540 2023-12-19T09:21:32.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b4770 con 0x7fc8f40cc470 2023-12-19T09:21:32.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f9723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b2ec0 con 0x7fc8f4158e70 2023-12-19T09:21:32.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.771+0000 7fc8f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50146/0 (socket says 172.21.15.111:50146) 2023-12-19T09:21:32.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 -- 172.21.15.111:0/4181577663 learned_addr learned my addr 172.21.15.111:0/4181577663 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f8f22700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f4158e70 0x7fc8f40c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 0x7fc8f40c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 -- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f4158e70 msgr2=0x7fc8f40c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f4158e70 0x7fc8f40c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 -- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 msgr2=0x7fc8f40c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:32.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 0x7fc8f40c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 -- 172.21.15.111:0/4181577663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009a20 con 0x7fc8f406a540 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 0x7fc8f40c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f3fff700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400ec80 tx=0x7fc8e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400c9d0 con 0x7fc8f406a540 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400cb30 con 0x7fc8f406a540 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e40094f0 con 0x7fc8f406a540 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400ceb0 con 0x7fc8f406a540 2023-12-19T09:21:32.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.775+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e8009420 con 0x7fc8f406a540 2023-12-19T09:21:32.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.779+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f404be40 con 0x7fc8f406a540 2023-12-19T09:21:32.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.779+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc8e4013680 con 0x7fc8f406a540 2023-12-19T09:21:32.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.779+0000 7fc8dbfff700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 0x7fc8e0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:32.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.779+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fc8e407d1b0 con 0x7fc8f406a540 2023-12-19T09:21:32.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.779+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 0x7fc8e0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:32.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.783+0000 7fc8f37fe700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 0x7fc8e0062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40cfd20 tx=0x7fc8e8017000 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:32.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.783+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc8e404d440 con 0x7fc8f406a540 2023-12-19T09:21:32.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.979+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f40b3030 con 0x7fc8e0060550 2023-12-19T09:21:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8dbfff700 1 -- 172.21.15.111:0/4181577663 <== mgr.8642 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc8f40b3030 con 0x7fc8e0060550 2023-12-19T09:21:32.988 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-12-19T09:21:32.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-12-19T09:21:32.989 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephfs" 2023-12-19T09:21:32.989 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:32.989 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 msgr2=0x7fc8e0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 0x7fc8e0062a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40cfd20 tx=0x7fc8e8017000 comp rx=0 tx=0).stop 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 msgr2=0x7fc8f4155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400ec80 tx=0x7fc8e4009250 comp rx=0 tx=0).stop 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 shutdown_connections 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc8e0060550 0x7fc8e0062a00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f40cc470 0x7fc8f40c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8f4158e70 0x7fc8f40c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 --2- 172.21.15.111:0/4181577663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f406a540 0x7fc8f4155320 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 >> 172.21.15.111:0/4181577663 conn(0x7fc8f40670e0 msgr2=0x7fc8f4157210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 shutdown_connections 2023-12-19T09:21:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:32.983+0000 7fc8f9723700 1 -- 172.21.15.111:0/4181577663 wait complete. 2023-12-19T09:21:33.011 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-12-19T09:21:33.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 Processor -- start 2023-12-19T09:21:33.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 -- start start 2023-12-19T09:21:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f064c06c7c0 0x7f064c15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064c06d970 con 0x7f064c0c69a0 2023-12-19T09:21:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c06cd60 con 0x7f064c06c7c0 2023-12-19T09:21:33.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.155+0000 7f0652816700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c06d040 con 0x7f064c06de00 2023-12-19T09:21:33.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0651814700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f064c0c69a0 0x7f064c06b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38790/0 (socket says 172.21.15.111:38790) 2023-12-19T09:21:33.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0651013700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0651814700 1 -- 172.21.15.111:0/87509434 learned_addr learned my addr 172.21.15.111:0/87509434 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 --2- 172.21.15.111:0/87509434 >> v2:172.21.15.111:3300/0 conn(0x7f064c06c7c0 0x7f064c15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0650812700 1 -- 172.21.15.111:0/87509434 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133321127 0 0) 0x7f064c06d970 con 0x7f064c0c69a0 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0650812700 1 -- 172.21.15.111:0/87509434 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0638003570 con 0x7f064c0c69a0 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 -- 172.21.15.111:0/87509434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 msgr2=0x7f064c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 -- 172.21.15.111:0/87509434 >> v1:172.21.15.143:6789/0 conn(0x7f064c0c69a0 legacy=0x7f064c06b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 -- 172.21.15.111:0/87509434 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c06d320 con 0x7f064c06c7c0 2023-12-19T09:21:33.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0651013700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0652015700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c15cba0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0644009e90 tx=0x7f0644009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ce51cb89802a0a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0650812700 1 -- 172.21.15.111:0/87509434 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064400e070 con 0x7f064c06c7c0 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0650812700 1 -- 172.21.15.111:0/87509434 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400bb40 con 0x7f064c06c7c0 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.159+0000 7f0650812700 1 -- 172.21.15.111:0/87509434 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644012430 con 0x7f064c06c7c0 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- 172.21.15.111:0/87509434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 msgr2=0x7f064c15cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c15cba0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0644009e90 tx=0x7f0644009780 comp rx=0 tx=0).stop 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- 172.21.15.111:0/87509434 shutdown_connections 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c15cba0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- 172.21.15.111:0/87509434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- 172.21.15.111:0/87509434 >> 172.21.15.111:0/87509434 conn(0x7f064c067b50 msgr2=0x7f064c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:33.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- 172.21.15.111:0/87509434 shutdown_connections 2023-12-19T09:21:33.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- 172.21.15.111:0/87509434 wait complete. 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 Processor -- start 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- start start 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 0x7f064c14e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c15fe10 con 0x7f064c06c7c0 2023-12-19T09:21:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c15f200 con 0x7f064c0c69a0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.163+0000 7f0652816700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c15f4e0 con 0x7f064c06de00 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50168/0 (socket says 172.21.15.111:50168) 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 -- 172.21.15.111:0/1157586176 learned_addr learned my addr 172.21.15.111:0/1157586176 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0652015700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 0x7f064c14e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 -- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 msgr2=0x7f064c0ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651013700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 -- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 msgr2=0x7f064c14e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 0x7f064c14e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 -- 172.21.15.111:0/1157586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063c009420 con 0x7f064c06c7c0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0652015700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 0x7f064c14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651013700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0651814700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f06480011e0 tx=0x7f0648007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:33.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648005b30 con 0x7f064c06c7c0 2023-12-19T09:21:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648005c90 con 0x7f064c06c7c0 2023-12-19T09:21:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0644009a20 con 0x7f064c06c7c0 2023-12-19T09:21:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648007320 con 0x7f064c06c7c0 2023-12-19T09:21:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c1602a0 con 0x7f064c06c7c0 2023-12-19T09:21:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.167+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c04be40 con 0x7f064c06c7c0 2023-12-19T09:21:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.171+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 101) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0648007690 con 0x7f064c06c7c0 2023-12-19T09:21:33.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.171+0000 7f0641ffb700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 0x7f0630062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.171+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f064807c0d0 con 0x7f064c06c7c0 2023-12-19T09:21:33.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.171+0000 7f0651013700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 0x7f0630062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.171+0000 7f0651013700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 0x7f0630062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f063c0093f0 tx=0x7f063c008040 comp rx=0 tx=0).ready entity=mgr.8642 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:33.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.179+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f064804c2a0 con 0x7f064c06c7c0 2023-12-19T09:21:33.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.395+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f064c06d600 con 0x7f064c06c7c0 2023-12-19T09:21:33.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.739+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 102) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f064804c670 con 0x7f064c06c7c0 2023-12-19T09:21:33.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.739+0000 7f0641ffb700 1 -- 172.21.15.111:0/1157586176 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v102) v1 ==== 92+0+0 (secure 0 0 0) 0x7f064804ff60 con 0x7f064c06c7c0 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 msgr2=0x7f0630062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 0x7f0630062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f063c0093f0 tx=0x7f063c008040 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 msgr2=0x7f064c0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f06480011e0 tx=0x7f0648007df0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 shutdown_connections 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0630060550 0x7f0630062a00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f064c0c69a0 0x7f064c14e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f064c06de00 0x7f064c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 --2- 172.21.15.111:0/1157586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f064c06c7c0 0x7f064c0c14c0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 >> 172.21.15.111:0/1157586176 conn(0x7f064c067b50 msgr2=0x7f064c06a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:33.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 shutdown_connections 2023-12-19T09:21:33.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.743+0000 7f0652816700 1 -- 172.21.15.111:0/1157586176 wait complete. 2023-12-19T09:21:33.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-12-19T09:21:33.929 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:33.923+0000 7f88aef6c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:33.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.927+0000 7f85fa163700 1 Processor -- start 2023-12-19T09:21:33.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.927+0000 7f85fa163700 1 -- start start 2023-12-19T09:21:33.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85f41547a0 0x7f85f40ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f40c7820 con 0x7f85f406d050 2023-12-19T09:21:33.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f4158b90 con 0x7f85f4157fc0 2023-12-19T09:21:33.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f40c7210 con 0x7f85f41547a0 2023-12-19T09:21:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85f41547a0 0x7f85f40ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85f41547a0 0x7f85f40ad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50190/0 (socket says 172.21.15.111:50190) 2023-12-19T09:21:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 -- 172.21.15.111:0/152098648 learned_addr learned my addr 172.21.15.111:0/152098648 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9962700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f85f406d050 0x7f85f40c7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46252/0 (socket says 172.21.15.111:46252) 2023-12-19T09:21:33.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f8960700 1 --1- 172.21.15.111:0/152098648 >> v1:172.21.15.143:6789/0 conn(0x7f85f4157fc0 0x7f85f4158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:38804/0 (socket says 172.21.15.111:38804) 2023-12-19T09:21:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 -- 172.21.15.111:0/152098648 >> v1:172.21.15.143:6789/0 conn(0x7f85f4157fc0 legacy=0x7f85f4158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 -- 172.21.15.111:0/152098648 >> v1:172.21.15.111:6789/0 conn(0x7f85f406d050 legacy=0x7f85f40c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85ebfff700 1 -- 172.21.15.111:0/152098648 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052416969 0 0) 0x7f85f4158b90 con 0x7f85f4157fc0 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 -- 172.21.15.111:0/152098648 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40ae030 con 0x7f85f41547a0 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85f9161700 1 --2- 172.21.15.111:0/152098648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f41547a0 0x7f85f40ad730 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f85f00099b0 tx=0x7f85f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add976a18893aeb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85ebfff700 1 -- 172.21.15.111:0/152098648 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f000b760 con 0x7f85f41547a0 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85ebfff700 1 -- 172.21.15.111:0/152098648 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f000b8c0 con 0x7f85f41547a0 2023-12-19T09:21:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85ebfff700 1 -- 172.21.15.111:0/152098648 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f000bc60 con 0x7f85f41547a0 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- 172.21.15.111:0/152098648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f41547a0 msgr2=0x7f85f40ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 --2- 172.21.15.111:0/152098648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f41547a0 0x7f85f40ad730 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f85f00099b0 tx=0x7f85f0009250 comp rx=0 tx=0).stop 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- 172.21.15.111:0/152098648 shutdown_connections 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 --2- 172.21.15.111:0/152098648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f41547a0 0x7f85f40ad730 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.931+0000 7f85fa163700 1 -- 172.21.15.111:0/152098648 >> 172.21.15.111:0/152098648 conn(0x7f85f40ccb30 msgr2=0x7f85f40ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.935+0000 7f85fa163700 1 -- 172.21.15.111:0/152098648 shutdown_connections 2023-12-19T09:21:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.935+0000 7f85fa163700 1 -- 172.21.15.111:0/152098648 wait complete. 2023-12-19T09:21:33.940 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:33.932+0000 7f4470363000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:33.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.935+0000 7f85fa163700 1 Processor -- start 2023-12-19T09:21:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.935+0000 7f85fa163700 1 -- start start 2023-12-19T09:21:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85f41547a0 0x7f85f41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85f4157fc0 0x7f85f41415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4169bc0 con 0x7f85f406d050 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41578f0 con 0x7f85f41547a0 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4169290 con 0x7f85f4157fc0 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50196/0 (socket says 172.21.15.111:50196) 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 -- 172.21.15.111:0/2516557980 learned_addr learned my addr 172.21.15.111:0/2516557980 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9962700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85f4157fc0 0x7f85f41415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f8960700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85f41547a0 0x7f85f41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 -- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85f4157fc0 msgr2=0x7f85f41415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85f4157fc0 0x7f85f41415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 -- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85f41547a0 msgr2=0x7f85f41573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85f41547a0 0x7f85f41573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 -- 172.21.15.111:0/2516557980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00094f0 con 0x7f85f406d050 2023-12-19T09:21:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85f9161700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009290 tx=0x7f85f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0011e30 con 0x7f85f406d050 2023-12-19T09:21:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f001c670 con 0x7f85f406d050 2023-12-19T09:21:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f4169570 con 0x7f85f406d050 2023-12-19T09:21:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f001ca30 con 0x7f85f406d050 2023-12-19T09:21:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.939+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f4168f10 con 0x7f85f406d050 2023-12-19T09:21:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f404be40 con 0x7f85f406d050 2023-12-19T09:21:33.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 102) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f85f001a450 con 0x7f85f406d050 2023-12-19T09:21:33.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85e97fa700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 0x7f85d8062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:33.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f85f007f090 con 0x7f85f406d050 2023-12-19T09:21:33.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85f8960700 1 -- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 msgr2=0x7f85d8062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:33.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.943+0000 7f85f8960700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 0x7f85d8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:33.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:33.947+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f85f004f300 con 0x7f85f406d050 2023-12-19T09:21:34.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.119+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f85f404be40 con 0x7f85f406d050 2023-12-19T09:21:34.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.127+0000 7f85e97fa700 1 -- 172.21.15.111:0/2516557980 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f85f001ce60 con 0x7f85f406d050 2023-12-19T09:21:34.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 msgr2=0x7f85d8062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:34.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 0x7f85d8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 msgr2=0x7f85f416d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009290 tx=0x7f85f0016000 comp rx=0 tx=0).stop 2023-12-19T09:21:34.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 shutdown_connections 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f85d80605a0 0x7f85d8062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85f41547a0 0x7f85f41573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85f4157fc0 0x7f85f41415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 --2- 172.21.15.111:0/2516557980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f406d050 0x7f85f416d070 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 >> 172.21.15.111:0/2516557980 conn(0x7f85f40ccb30 msgr2=0x7f85f40acac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 shutdown_connections 2023-12-19T09:21:34.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.131+0000 7f85fa163700 1 -- 172.21.15.111:0/2516557980 wait complete. 2023-12-19T09:21:34.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-12-19T09:21:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.315+0000 7f7ceee8e700 1 Processor -- start 2023-12-19T09:21:34.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.315+0000 7f7ceee8e700 1 -- start start 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7ceee8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ce8158c90 0x7f7ce80caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7ceee8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce806c850 con 0x7f7ce8159770 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7ceee8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce806b470 con 0x7f7ce806aeb0 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7ceee8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce806b750 con 0x7f7ce8158c90 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cede8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ce8158c90 0x7f7ce80caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cee68d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7ce8159770 0x7f7ce806cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46256/0 (socket says 172.21.15.111:46256) 2023-12-19T09:21:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cee68d700 1 -- 172.21.15.111:0/2258192121 learned_addr learned my addr 172.21.15.111:0/2258192121 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cede8c700 1 -- 172.21.15.111:0/2258192121 >> v1:172.21.15.143:6789/0 conn(0x7f7ce806aeb0 legacy=0x7f7ce8159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cede8c700 1 -- 172.21.15.111:0/2258192121 >> v1:172.21.15.111:6789/0 conn(0x7f7ce8159770 legacy=0x7f7ce806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cede8c700 1 -- 172.21.15.111:0/2258192121 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce806ba30 con 0x7f7ce8158c90 2023-12-19T09:21:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cece8a700 1 -- 172.21.15.111:0/2258192121 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273396539 0 0) 0x7f7ce806c850 con 0x7f7ce8159770 2023-12-19T09:21:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.319+0000 7f7cede8c700 1 --2- 172.21.15.111:0/2258192121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80caf40 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40098e0 tx=0x7f7ce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f0c7ed21eaba8d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7cece8a700 1 -- 172.21.15.111:0/2258192121 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4009e80 con 0x7f7ce8158c90 2023-12-19T09:21:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7cece8a700 1 -- 172.21.15.111:0/2258192121 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce400b860 con 0x7f7ce8158c90 2023-12-19T09:21:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7cece8a700 1 -- 172.21.15.111:0/2258192121 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce400bbe0 con 0x7f7ce8158c90 2023-12-19T09:21:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 -- 172.21.15.111:0/2258192121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 msgr2=0x7f7ce80caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/2258192121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80caf40 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40098e0 tx=0x7f7ce40091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 -- 172.21.15.111:0/2258192121 shutdown_connections 2023-12-19T09:21:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/2258192121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80caf40 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 -- 172.21.15.111:0/2258192121 >> 172.21.15.111:0/2258192121 conn(0x7f7ce8067bb0 msgr2=0x7f7ce80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:34.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 -- 172.21.15.111:0/2258192121 shutdown_connections 2023-12-19T09:21:34.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.323+0000 7f7ceee8e700 1 -- 172.21.15.111:0/2258192121 wait complete. 2023-12-19T09:21:34.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 Processor -- start 2023-12-19T09:21:34.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 -- start start 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce806aeb0 0x7f7ce814b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce8159770 0x7f7ce80c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce80c9e30 con 0x7f7ce8158c90 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce8154fa0 con 0x7f7ce8159770 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ceee8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce80c9870 con 0x7f7ce806aeb0 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50204/0 (socket says 172.21.15.111:50204) 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 -- 172.21.15.111:0/1845952047 learned_addr learned my addr 172.21.15.111:0/1845952047 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7cee68d700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce8159770 0x7f7ce80c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7cede8c700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce806aeb0 0x7f7ce814b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 -- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce806aeb0 msgr2=0x7f7ce814b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce806aeb0 0x7f7ce814b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 -- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce8159770 msgr2=0x7f7ce80c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce8159770 0x7f7ce80c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 -- 172.21.15.111:0/1845952047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce4009420 con 0x7f7ce8158c90 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7ced68b700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800c5c0 tx=0x7f7cd800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.327+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8011590 con 0x7f7ce8158c90 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd8011710 con 0x7f7ce8158c90 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8011ad0 con 0x7f7ce8158c90 2023-12-19T09:21:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce80c9b50 con 0x7f7ce8158c90 2023-12-19T09:21:34.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce814bd70 con 0x7f7ce8158c90 2023-12-19T09:21:34.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce8154a90 con 0x7f7ce8158c90 2023-12-19T09:21:34.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.331+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 102) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f7cd801b3b0 con 0x7f7ce8158c90 2023-12-19T09:21:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.335+0000 7f7cddffb700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 0x7f7ccc062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.335+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f7cd807bfb0 con 0x7f7ce8158c90 2023-12-19T09:21:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.335+0000 7f7cede8c700 1 -- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 msgr2=0x7f7ccc062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.335+0000 7f7cede8c700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 0x7f7ccc062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:34.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.339+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7ce8154a90 con 0x7f7ce8158c90 2023-12-19T09:21:34.468 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:34.460+0000 7f4470363000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:34.500 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:34.495+0000 7f88aef6c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:34.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.515+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7ce8154a90 con 0x7f7ce8158c90 2023-12-19T09:21:34.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.523+0000 7f7cddffb700 1 -- 172.21.15.111:0/1845952047 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7ce8154a90 con 0x7f7ce8158c90 2023-12-19T09:21:34.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 msgr2=0x7f7ccc062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:34.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 0x7f7ccc062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 msgr2=0x7f7ce80c6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800c5c0 tx=0x7f7cd800c8d0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 shutdown_connections 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7ccc0605a0 0x7f7ccc062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ce8159770 0x7f7ce80c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ce8158c90 0x7f7ce80c6590 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 --2- 172.21.15.111:0/1845952047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ce806aeb0 0x7f7ce814b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 >> 172.21.15.111:0/1845952047 conn(0x7f7ce8067bb0 msgr2=0x7f7ce8159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 shutdown_connections 2023-12-19T09:21:34.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.527+0000 7f7ceee8e700 1 -- 172.21.15.111:0/1845952047 wait complete. 2023-12-19T09:21:34.542 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:34.536+0000 7f4470363000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:34.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:34.554 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-12-19T09:21:34.587 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:34.579+0000 7f88aef6c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:34.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-12-19T09:21:34.694 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:34.688+0000 7f4470363000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:34.733 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:34.727+0000 7f88aef6c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:34.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.799+0000 7f0f6c62e700 1 Processor -- start 2023-12-19T09:21:34.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.799+0000 7f0f6c62e700 1 -- start start 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6c62e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0f640cb130 0x7f0f640cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6c62e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f6406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6c62e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f640b0ac0 con 0x7f0f6415c220 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6c62e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6406b030 con 0x7f0f640cb130 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6c62e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6406b210 con 0x7f0f6406a6a0 2023-12-19T09:21:34.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6abcb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0f6415c220 0x7f0f6406c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:46268/0 (socket says 172.21.15.111:46268) 2023-12-19T09:21:34.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6abcb700 1 -- 172.21.15.111:0/292919155 learned_addr learned my addr 172.21.15.111:0/292919155 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:34.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/292919155 >> v2:172.21.15.111:3300/0 conn(0x7f0f640cb130 0x7f0f640cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108785948 0 0) 0x7f0f640b0ac0 con 0x7f0f6415c220 2023-12-19T09:21:34.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50003570 con 0x7f0f6415c220 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f6406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142701936 0 0) 0x7f0f50003570 con 0x7f0f6415c220 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 msgr2=0x7f0f6406aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f6406aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 >> v2:172.21.15.111:3300/0 conn(0x7f0f640cb130 msgr2=0x7f0f640cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 --2- 172.21.15.111:0/292919155 >> v2:172.21.15.111:3300/0 conn(0x7f0f640cb130 0x7f0f640cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640ca810 con 0x7f0f6415c220 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0f5c0017f0 con 0x7f0f6415c220 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 >> v1:172.21.15.111:6789/0 conn(0x7f0f6415c220 legacy=0x7f0f6406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f50003a60 0x7f0f50003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f50004d50 0x7f0f500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6406b030 con 0x7f0f50004d50 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6406b210 con 0x7f0f50003a60 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.803+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5000bae0 con 0x7f0f6406a6a0 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f50004d50 0x7f0f500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f50003a60 0x7f0f50003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 -- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 msgr2=0x7f0f50004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f50004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 -- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f50003a60 msgr2=0x7f0f50003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f50003a60 0x7f0f50003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 -- 172.21.15.111:0/292919155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50003570 con 0x7f0f50004d50 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f50003a60 0x7f0f50003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:34.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6406a6a0 0x7f0f50004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:34.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f50004d50 0x7f0f500051c0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00a970 tx=0x7f0f5c002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19731920c121824c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:34.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c0070d0 con 0x7f0f50004d50 2023-12-19T09:21:34.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c007270 con 0x7f0f50004d50 2023-12-19T09:21:34.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f693c8700 1 -- 172.21.15.111:0/292919155 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c007630 con 0x7f0f50004d50 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 -- 172.21.15.111:0/292919155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f50004d50 msgr2=0x7f0f500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/292919155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f50004d50 0x7f0f500051c0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00a970 tx=0x7f0f5c002210 comp rx=0 tx=0).stop 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6abcb700 1 -- 172.21.15.111:0/292919155 reap_dead start 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 -- 172.21.15.111:0/292919155 shutdown_connections 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 -- 172.21.15.111:0/292919155 >> 172.21.15.111:0/292919155 conn(0x7f0f64067bb0 msgr2=0x7f0f640cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 -- 172.21.15.111:0/292919155 shutdown_connections 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.807+0000 7f0f6c62e700 1 -- 172.21.15.111:0/292919155 wait complete. 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 Processor -- start 2023-12-19T09:21:34.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- start start 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f50004d50 0x7f0f640af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f640b40e0 0x7f0f640b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640af860 con 0x7f0f640c80b0 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640b4e50 con 0x7f0f640b40e0 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640b5130 con 0x7f0f50004d50 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50238/0 (socket says 172.21.15.111:50238) 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 -- 172.21.15.111:0/2007507783 learned_addr learned my addr 172.21.15.111:0/2007507783 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f50004d50 0x7f0f640af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6abcb700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f640b40e0 0x7f0f640b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f50004d50 msgr2=0x7f0f640af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f50004d50 0x7f0f640af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f640b40e0 msgr2=0x7f0f640b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:34.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f640b40e0 0x7f0f640b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640b5410 con 0x7f0f640c80b0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f69bc9700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400f910 tx=0x7f0f54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400d5d0 con 0x7f0f640c80b0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5400d730 con 0x7f0f640c80b0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5400dab0 con 0x7f0f640c80b0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f640b56f0 con 0x7f0f640c80b0 2023-12-19T09:21:34.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.811+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640c8a20 con 0x7f0f640c80b0 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6416a4a0 con 0x7f0f640c80b0 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 102) v1 ==== 79978+0+0 (secure 0 0 0) 0x7f0f54014460 con 0x7f0f640c80b0 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f5a7fc700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6601+0+0 (secure 0 0 0) 0x7f0f5407d590 con 0x7f0f640c80b0 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f6a3ca700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:34.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.815+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:34.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:34.823+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0f540511f0 con 0x7f0f640c80b0 2023-12-19T09:21:34.950 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:34.944+0000 7f4470363000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:35.009 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.003+0000 7f88aef6c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:35.019 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.012+0000 7f4470363000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:35.019+0000 7f0f6a3ca700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:35.019+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:35.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:35.019+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f64071120 con 0x7f0f48060620 2023-12-19T09:21:35.083 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.075+0000 7f88aef6c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:35.233 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.228+0000 7f4470363000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:35.316 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.312+0000 7f4470363000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:35.327 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.319+0000 7f88aef6c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:35.394 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.392+0000 7f4470363000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:35.414 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.403+0000 7f88aef6c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:35.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:35.423+0000 7f0f6a3ca700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:35.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:35.423+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T09:21:35.497 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.491+0000 7f88aef6c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:35.761 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.756+0000 7f4470363000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:35.916 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:35.911+0000 7f88aef6c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:35.917 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:35.916+0000 7f4470363000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:36.094 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.087+0000 7f88aef6c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:36.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:36.227+0000 7f0f6a3ca700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:36.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:36.227+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T09:21:36.268 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.264+0000 7f4470363000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:36.470 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.463+0000 7f88aef6c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:36.476 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.472+0000 7f4470363000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:36.567 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.564+0000 7f4470363000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:36.642 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.640+0000 7f4470363000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:36.705 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.699+0000 7f88aef6c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:36.724 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.720+0000 7f4470363000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:36.798 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.791+0000 7f88aef6c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:36.886 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.879+0000 7f88aef6c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:36.942 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:36.940+0000 7f4470363000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:36.969 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:36.963+0000 7f88aef6c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:37.020 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:37.016+0000 7f4470363000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:37.107 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:37.104+0000 7f4470363000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:37.213 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:37.207+0000 7f88aef6c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:37.293 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:37.287+0000 7f88aef6c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:37.294 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:37.292+0000 7f4470363000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:37.379 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:37.376+0000 7f4470363000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:37.384 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:37.379+0000 7f88aef6c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:37.593 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:37.587+0000 7f88aef6c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:37.687 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:37.679+0000 7f88aef6c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:37.831+0000 7f0f6a3ca700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:37.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:37.831+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T09:21:38.100 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.096+0000 7f4470363000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:38.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:38.115+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 103) v1 ==== 79689+0+0 (secure 0 0 0) 0x7f0f5404dbd0 con 0x7f0f640c80b0 2023-12-19T09:21:38.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:38.115+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:38.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:38.115+0000 7f0f5a7fc700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:38.307 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f4431ee1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f4431ee1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f4431ee1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f4431ee1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f4431ee1700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f442d6d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.308 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f442d6d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.309 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f442d6d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.309 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f442d6d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.309 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:38.304+0000 7f442d6d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:38.487 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:38.479+0000 7f88aef6c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:39.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.123+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 104) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f0f54066a90 con 0x7f0f640c80b0 2023-12-19T09:21:39.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.123+0000 7f0f5a7fc700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.123+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f64071120 con 0x7f0f48060620 2023-12-19T09:21:39.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.127+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.131+0000 7f0f6a3ca700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009170 tx=0x7f0f60008040 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:39.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.239+0000 7f0f5a7fc700 1 -- 172.21.15.111:0/2007507783 <== mgr.10279 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f64071120 con 0x7f0f48060620 2023-12-19T09:21:39.245 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path snap_test_dir1 2023-12-19T09:21:39.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 msgr2=0x7f0f48062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009170 tx=0x7f0f60008040 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 msgr2=0x7f0f640c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400f910 tx=0x7f0f54008040 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 shutdown_connections 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0f48060620 0x7f0f48062ad0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f640b40e0 0x7f0f640b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f640c80b0 0x7f0f640c84e0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 --2- 172.21.15.111:0/2007507783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f50004d50 0x7f0f640af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 >> 172.21.15.111:0/2007507783 conn(0x7f0f64067bb0 msgr2=0x7f0f640cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 shutdown_connections 2023-12-19T09:21:39.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.243+0000 7f0f6c62e700 1 -- 172.21.15.111:0/2007507783 wait complete. 2023-12-19T09:21:39.271 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-12-19T09:21:39.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-12-19T09:21:39.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.439+0000 7f8e3ed45700 1 Processor -- start 2023-12-19T09:21:39.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.439+0000 7f8e3ed45700 1 -- start start 2023-12-19T09:21:39.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.439+0000 7f8e3ed45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e3815abd0 0x7f8e3815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.439+0000 7f8e3ed45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e3815b4e0 0x7f8e381567f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.439+0000 7f8e3ed45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3806d270 con 0x7f8e38156d30 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3815b960 con 0x7f8e3815b4e0 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3806d550 con 0x7f8e3815abd0 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e3815b4e0 0x7f8e381567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e3815b4e0 0x7f8e381567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50246/0 (socket says 172.21.15.111:50246) 2023-12-19T09:21:39.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 -- 172.21.15.111:0/1509557728 learned_addr learned my addr 172.21.15.111:0/1509557728 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3cae1700 1 --2- 172.21.15.111:0/1509557728 >> v2:172.21.15.143:3300/0 conn(0x7f8e3815abd0 0x7f8e3815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 -- 172.21.15.111:0/1509557728 >> v2:172.21.15.143:3300/0 conn(0x7f8e3815abd0 msgr2=0x7f8e3815afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 --2- 172.21.15.111:0/1509557728 >> v2:172.21.15.143:3300/0 conn(0x7f8e3815abd0 0x7f8e3815afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 -- 172.21.15.111:0/1509557728 >> v1:172.21.15.143:6789/0 conn(0x7f8e38156d30 legacy=0x7f8e380b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 -- 172.21.15.111:0/1509557728 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3806d830 con 0x7f8e3815b4e0 2023-12-19T09:21:39.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e37fff700 1 --2- 172.21.15.111:0/1509557728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3815b4e0 0x7f8e381567f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806be00 tx=0x7f8e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf666ab3bc37c58d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:39.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e377fe700 1 -- 172.21.15.111:0/1509557728 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2400b860 con 0x7f8e3815b4e0 2023-12-19T09:21:39.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e377fe700 1 -- 172.21.15.111:0/1509557728 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2400b9c0 con 0x7f8e3815b4e0 2023-12-19T09:21:39.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e377fe700 1 -- 172.21.15.111:0/1509557728 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2400bd60 con 0x7f8e3815b4e0 2023-12-19T09:21:39.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- 172.21.15.111:0/1509557728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3815b4e0 msgr2=0x7f8e381567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/1509557728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3815b4e0 0x7f8e381567f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806be00 tx=0x7f8e24009250 comp rx=0 tx=0).stop 2023-12-19T09:21:39.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- 172.21.15.111:0/1509557728 shutdown_connections 2023-12-19T09:21:39.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/1509557728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e3815b4e0 0x7f8e381567f0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/1509557728 >> v2:172.21.15.143:3300/0 conn(0x7f8e3815abd0 0x7f8e3815afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- 172.21.15.111:0/1509557728 >> 172.21.15.111:0/1509557728 conn(0x7f8e380cdef0 msgr2=0x7f8e380ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:39.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.443+0000 7f8e3ed45700 1 -- 172.21.15.111:0/1509557728 shutdown_connections 2023-12-19T09:21:39.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.447+0000 7f8e3ed45700 1 -- 172.21.15.111:0/1509557728 wait complete. 2023-12-19T09:21:39.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.447+0000 7f8e3ed45700 1 Processor -- start 2023-12-19T09:21:39.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.447+0000 7f8e3ed45700 1 -- start start 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3815abd0 0x7f8e380ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3815b4e0 0x7f8e38169930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3816b640 con 0x7f8e38156d30 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380ae6e0 con 0x7f8e3815abd0 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3ed45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e380ae930 con 0x7f8e3815b4e0 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50248/0 (socket says 172.21.15.111:50248) 2023-12-19T09:21:39.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 -- 172.21.15.111:0/4088401709 learned_addr learned my addr 172.21.15.111:0/4088401709 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:39.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e37fff700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3815abd0 0x7f8e380ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 -- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3815b4e0 msgr2=0x7f8e38169930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3815b4e0 0x7f8e38169930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 -- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3815abd0 msgr2=0x7f8e380ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3815abd0 0x7f8e380ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 -- 172.21.15.111:0/4088401709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e240094f0 con 0x7f8e38156d30 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e3cae1700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00d8c0 tx=0x7f8e2c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.451+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00de60 con 0x7f8e38156d30 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c00b7e0 con 0x7f8e38156d30 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e380afda0 con 0x7f8e38156d30 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c00bba0 con 0x7f8e38156d30 2023-12-19T09:21:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e380b0320 con 0x7f8e38156d30 2023-12-19T09:21:39.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e38071120 con 0x7f8e38156d30 2023-12-19T09:21:39.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.455+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 104) v1 ==== 79816+0+0 (secure 0 0 0) 0x7f8e2c01e460 con 0x7f8e38156d30 2023-12-19T09:21:39.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.459+0000 7f8e34ff9700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 0x7f8e1c0628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:39.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.459+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f8e2c07d110 con 0x7f8e38156d30 2023-12-19T09:21:39.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.459+0000 7f8e37fff700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 0x7f8e1c0628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:39.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.459+0000 7f8e37fff700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 0x7f8e1c0628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806be00 tx=0x7f8e24017000 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:39.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.467+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e2c04d530 con 0x7f8e38156d30 2023-12-19T09:21:39.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.663+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e380caea0 con 0x7f8e1c0603f0 2023-12-19T09:21:39.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.771+0000 7f8e34ff9700 1 -- 172.21.15.111:0/4088401709 <== mgr.10279 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f8e380caea0 con 0x7f8e1c0603f0 2023-12-19T09:21:39.780 INFO:teuthology.orchestra.run.smithi111.stdout:Retention added to path snap_test_dir1 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 msgr2=0x7f8e1c0628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 0x7f8e1c0628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806be00 tx=0x7f8e24017000 comp rx=0 tx=0).stop 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 msgr2=0x7f8e380b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00d8c0 tx=0x7f8e2c008040 comp rx=0 tx=0).stop 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 shutdown_connections 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8e1c0603f0 0x7f8e1c0628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3815abd0 0x7f8e380ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3815b4e0 0x7f8e38169930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 --2- 172.21.15.111:0/4088401709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e38156d30 0x7f8e380b3190 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.775+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 >> 172.21.15.111:0/4088401709 conn(0x7f8e380cdef0 msgr2=0x7f8e380ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.779+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 shutdown_connections 2023-12-19T09:21:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:39.779+0000 7f8e3ed45700 1 -- 172.21.15.111:0/4088401709 wait complete. 2023-12-19T09:21:39.804 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-12-19T09:21:39.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T09:21:39.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-12-19T09:21:39.883 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.884 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.885 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.111,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-12-19T09:21:39.886 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-12-19T09:21:39.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 20 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 20 2023-12-19T09:21:39.888 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 10 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T09:21:39.889 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug finisher = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-12-19T09:21:39.890 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T09:21:39.891 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T09:21:39.892 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = "" 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout: client snapdir = .snap 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d] 2023-12-19T09:21:39.893 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-12-19T09:21:39.894 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2023-12-19T09:21:39.895 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2023-12-19T09:21:39.896 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.111,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-12-19T09:21:39.897 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 4ae666be-184d-45ea-a875-cac9a231c35b 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.899 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2023-12-19T09:21:39.900 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-12-19T09:21:39.901 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug finisher = 20 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2023-12-19T09:21:39.902 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2023-12-19T09:21:39.903 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = "" 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout: client snapdir = .snap 2023-12-19T09:21:39.904 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.d] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2023-12-19T09:21:39.905 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-12-19T09:21:39.906 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-12-19T09:21:39.906 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-12-19T09:21:39.918 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:21:39.918 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:21:39.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:39.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63297]: fuse finished with error 0 and tester_r 0 2023-12-19T09:21:39.987 INFO:teuthology.orchestra.run:waiting for 900 2023-12-19T09:21:46.089 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:21:46.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.089 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:21:46.112 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-19T09:21:46.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.315+0000 7f68e8145700 1 Processor -- start 2023-12-19T09:21:46.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e8145700 1 -- start start 2023-12-19T09:21:46.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e8145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f68e00c7080 0x7f68e00bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e8145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e0067e10 con 0x7f68e00682a0 2023-12-19T09:21:46.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e8145700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e0068740 con 0x7f68e00bec60 2023-12-19T09:21:46.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e8145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e0067200 con 0x7f68e00c7080 2023-12-19T09:21:46.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e66e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f68e00682a0 0x7f68e006a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55136/0 (socket says 172.21.15.111:55136) 2023-12-19T09:21:46.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e66e2700 1 -- 172.21.15.111:0/360664114 learned_addr learned my addr 172.21.15.111:0/360664114 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:46.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> v2:172.21.15.111:3300/0 conn(0x7f68e00c7080 0x7f68e00bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917125913 0 0) 0x7f68e0067e10 con 0x7f68e00682a0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68cc003570 con 0x7f68e00682a0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711554974 0 0) 0x7f68cc003570 con 0x7f68e00682a0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 >> v2:172.21.15.111:3300/0 conn(0x7f68e00c7080 msgr2=0x7f68e00bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 --2- 172.21.15.111:0/360664114 >> v2:172.21.15.111:3300/0 conn(0x7f68e00c7080 0x7f68e00bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 >> v1:172.21.15.143:6789/0 conn(0x7f68e00bec60 legacy=0x7f68e00c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00674e0 con 0x7f68e00682a0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f68d80017f0 con 0x7f68e00682a0 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 >> v1:172.21.15.111:6789/0 conn(0x7f68e00682a0 legacy=0x7f68e006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.319+0000 7f68e4edf700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc003a60 0x7f68cc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0051c0 0x7f68cc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0067200 con 0x7f68cc004810 2023-12-19T09:21:46.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc00c050 con 0x7f68cc003a60 2023-12-19T09:21:46.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc00c210 con 0x7f68cc0051c0 2023-12-19T09:21:46.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> v2:172.21.15.111:3300/0 conn(0x7f68e00c7080 0x7f68e00bf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e5ee1700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc003a60 0x7f68cc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e66e2700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0051c0 0x7f68cc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 -- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0051c0 msgr2=0x7f68cc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0051c0 0x7f68cc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 -- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc003a60 msgr2=0x7f68cc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc003a60 0x7f68cc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 -- 172.21.15.111:0/360664114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc003570 con 0x7f68cc004810 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e66e2700 1 -- 172.21.15.111:0/360664114 reap_dead start 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e56e0700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68cc004c80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009db0 tx=0x7f68d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56e92c9a716e5c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:46.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0009740 con 0x7f68cc004810 2023-12-19T09:21:46.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d000e920 con 0x7f68cc004810 2023-12-19T09:21:46.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e4edf700 1 -- 172.21.15.111:0/360664114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d000eca0 con 0x7f68cc004810 2023-12-19T09:21:46.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e8145700 1 -- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 msgr2=0x7f68cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.323+0000 7f68e8145700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68cc004c80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009db0 tx=0x7f68d00091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 -- 172.21.15.111:0/360664114 shutdown_connections 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 --2- 172.21.15.111:0/360664114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68cc004c80 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 -- 172.21.15.111:0/360664114 >> 172.21.15.111:0/360664114 conn(0x7f68e00cc690 msgr2=0x7f68e00c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 -- 172.21.15.111:0/360664114 shutdown_connections 2023-12-19T09:21:46.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 -- 172.21.15.111:0/360664114 wait complete. 2023-12-19T09:21:46.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 Processor -- start 2023-12-19T09:21:46.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.327+0000 7f68e8145700 1 -- start start 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68e015cf60 0x7f68e015c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68e0160000 0x7f68e0160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0160990 con 0x7f68cc004810 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e015d750 con 0x7f68e0160000 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e015f360 con 0x7f68e015cf60 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37240/0 (socket says 172.21.15.111:37240) 2023-12-19T09:21:46.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 -- 172.21.15.111:0/660460473 learned_addr learned my addr 172.21.15.111:0/660460473 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:46.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e66e2700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68e0160000 0x7f68e0160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e56e0700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68e015cf60 0x7f68e015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 -- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68e015cf60 msgr2=0x7f68e015c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68e015cf60 0x7f68e015c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 -- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68e0160000 msgr2=0x7f68e0160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68e0160000 0x7f68e0160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 -- 172.21.15.111:0/660460473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00c1f00 con 0x7f68cc004810 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e5ee1700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f68dc002b60 tx=0x7f68dc007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00f9d0 con 0x7f68cc004810 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc00fb30 con 0x7f68cc004810 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0133f0 con 0x7f68cc004810 2023-12-19T09:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e00c21e0 con 0x7f68cc004810 2023-12-19T09:21:46.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.331+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e00bf650 con 0x7f68cc004810 2023-12-19T09:21:46.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.335+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e0071110 con 0x7f68cc004810 2023-12-19T09:21:46.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.335+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 105) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f68dc0137e0 con 0x7f68cc004810 2023-12-19T09:21:46.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.335+0000 7f68d5ffb700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 0x7f68c4062cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:46.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.335+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f68dc07d7d0 con 0x7f68cc004810 2023-12-19T09:21:46.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.335+0000 7f68e56e0700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 0x7f68c4062cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:46.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.339+0000 7f68e56e0700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 0x7f68c4062cb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f68d000f010 tx=0x7f68d0019000 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:46.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.343+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f68dc04d620 con 0x7f68cc004810 2023-12-19T09:21:46.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.551+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f68e00677c0 con 0x7f68cc004810 2023-12-19T09:21:46.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68d5ffb700 1 -- 172.21.15.111:0/660460473 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) v1 ==== 53+0+73 (secure 0 0 0) 0x7f68dc050c40 con 0x7f68cc004810 2023-12-19T09:21:46.560 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 msgr2=0x7f68c4062cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 0x7f68c4062cb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f68d000f010 tx=0x7f68d0019000 comp rx=0 tx=0).stop 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 msgr2=0x7f68e015ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f68dc002b60 tx=0x7f68dc007d60 comp rx=0 tx=0).stop 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 shutdown_connections 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f68c4060800 0x7f68c4062cb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68e0160000 0x7f68e0160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68e015cf60 0x7f68e015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 --2- 172.21.15.111:0/660460473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc004810 0x7f68e015ed50 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:46.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 >> 172.21.15.111:0/660460473 conn(0x7f68e00cc690 msgr2=0x7f68e00aea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:46.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 shutdown_connections 2023-12-19T09:21:46.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:46.555+0000 7f68e8145700 1 -- 172.21.15.111:0/660460473 wait complete. 2023-12-19T09:21:46.582 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T09:21:46.583 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:46.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.583 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-12-19T09:21:46.604 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T09:21:46.604 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T09:21:46.604 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T09:21:46.604 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.604 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 36751sec preferred_lft 36751sec 2023-12-19T09:21:46.605 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 02:42:47:e2:a3:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::709e:e2ff:fe53:9af/64 scope link 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ca:29:90:01:01:8a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::c829:90ff:fe01:18a/64 scope link 2023-12-19T09:21:46.606 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.607 INFO:teuthology.orchestra.run.smithi111.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-19T09:21:46.607 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 06:4b:6c:a6:94:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:21:46.607 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::44b:6cff:fea6:947e/64 scope link 2023-12-19T09:21:46.607 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-12-19T09:21:46.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.608 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T09:21:46.690 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-12-19T09:21:46.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.693 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-12-19T09:21:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T09:21:46.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T09:21:46.755 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-12-19T09:21:46.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.895 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:21:46.912 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:21:46.912 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:21:46.913 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:46.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:46.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-12-19T09:21:46.988 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-19T09:21:46.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:21:46.988 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:47.048 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T09:21:47.049 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T09:21:47.049 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:47.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:47.190 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:47.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:47.265 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:47.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:47.268 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:47.328 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-19T09:21:47.328 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-19T09:21:47.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:47.415 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:47.415 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:47.492 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:47.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:47.494 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:47.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2023-12-19T09:21:47.499+0000 7f96541c7380 -1 init, newargv = 0x55f1685e5460 newargc=15 2023-12-19T09:21:47.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63637]: starting ceph client 2023-12-19T09:21:47.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63637]: starting fuse 2023-12-19T09:21:48.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-12-19T09:21:48.659 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-12-19T09:21:48.659 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-12-19T09:21:48.743 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-12-19T09:21:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-12-19T09:21:48.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:48.746 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-12-19T09:21:48.809 INFO:teuthology.orchestra.run.smithi111.stdout:49 2023-12-19T09:21:48.809 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-19T09:21:48.810 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:48.811 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-12-19T09:21:48.812 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-19T09:21:48.813 DEBUG:teuthology.orchestra.run.smithi111:> 2023-12-19T09:21:48.813 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2023-12-19T09:21:48.813 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-12-19T09:21:48.976 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.63637.asok 2023-12-19T09:21:49.159 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63637.asok 2023-12-19T09:21:49.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:49.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63637.asok status 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "c7312db04d305d59a3441862c84d6d4da3817eb0", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-141-gc7312db0 (c7312db04d305d59a3441862c84d6d4da3817eb0) quincy (stable)", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "63637", 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:49.356 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 10707, 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 10707 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2276477678 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:49.357 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2276477678 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.10707 192.168.0.1:0/2276477678", 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2276477678", 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 95, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 209, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-12-19T09:21:49.358 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:21:49.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:49.369 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-12-19T09:21:49.369 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:49.386 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-12-19T09:21:49.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:49.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:49.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:49.466 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-12-19-09_22_00 in ~21s... 2023-12-19T09:21:49.466 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-12-19T09:21:49.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-12-19T09:21:49.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.663+0000 7fe415d23700 1 Processor -- start 2023-12-19T09:21:49.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.663+0000 7fe415d23700 1 -- start start 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe41015cee0 0x7fe41015d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe4100ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe410158b20 0x7fe410158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4100cea80 con 0x7fe410158b20 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4100ced60 con 0x7fe41015cee0 2023-12-19T09:21:49.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4100cf040 con 0x7fe4100cb3a0 2023-12-19T09:21:49.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe410158b20 0x7fe410158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe410158b20 0x7fe410158fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37246/0 (socket says 172.21.15.111:37246) 2023-12-19T09:21:49.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 -- 172.21.15.111:0/4167536191 learned_addr learned my addr 172.21.15.111:0/4167536191 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:49.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe40ffff700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe4100ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe414d21700 1 --2- 172.21.15.111:0/4167536191 >> v2:172.21.15.143:3300/0 conn(0x7fe41015cee0 0x7fe41015d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 -- 172.21.15.111:0/4167536191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 msgr2=0x7fe4100ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:49.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe4100ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 -- 172.21.15.111:0/4167536191 >> v2:172.21.15.143:3300/0 conn(0x7fe41015cee0 msgr2=0x7fe41015d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:49.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 --2- 172.21.15.111:0/4167536191 >> v2:172.21.15.143:3300/0 conn(0x7fe41015cee0 0x7fe41015d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 -- 172.21.15.111:0/4167536191 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4100cf320 con 0x7fe410158b20 2023-12-19T09:21:49.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe415522700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe410158b20 0x7fe410158fc0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe410070a40 tx=0x7fe408009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16dc044b320089c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:49.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe40f7fe700 1 -- 172.21.15.111:0/4167536191 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40800e070 con 0x7fe410158b20 2023-12-19T09:21:49.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe40f7fe700 1 -- 172.21.15.111:0/4167536191 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40800b970 con 0x7fe410158b20 2023-12-19T09:21:49.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.667+0000 7fe40f7fe700 1 -- 172.21.15.111:0/4167536191 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080092e0 con 0x7fe410158b20 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- 172.21.15.111:0/4167536191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe410158b20 msgr2=0x7fe410158fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe410158b20 0x7fe410158fc0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe410070a40 tx=0x7fe408009840 comp rx=0 tx=0).stop 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- 172.21.15.111:0/4167536191 shutdown_connections 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe410158b20 0x7fe410158fc0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 --2- 172.21.15.111:0/4167536191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe4100ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 --2- 172.21.15.111:0/4167536191 >> v2:172.21.15.143:3300/0 conn(0x7fe41015cee0 0x7fe41015d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- 172.21.15.111:0/4167536191 >> 172.21.15.111:0/4167536191 conn(0x7fe4100cff20 msgr2=0x7fe4100d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- 172.21.15.111:0/4167536191 shutdown_connections 2023-12-19T09:21:49.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- 172.21.15.111:0/4167536191 wait complete. 2023-12-19T09:21:49.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 Processor -- start 2023-12-19T09:21:49.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.671+0000 7fe415d23700 1 -- start start 2023-12-19T09:21:49.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe410168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe410158b20 0x7fe4101697a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 0x7fe4100c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4100c88e0 con 0x7fe41015cee0 2023-12-19T09:21:49.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe410169ce0 con 0x7fe410158b20 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4100c7cd0 con 0x7fe4100cb3a0 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe414d21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe410168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 0x7fe4100c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe414d21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe410168ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:50058/0 (socket says 172.21.15.111:50058) 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe414d21700 1 -- 172.21.15.111:0/200146702 learned_addr learned my addr 172.21.15.111:0/200146702 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:49.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe40ffff700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe410158b20 0x7fe4101697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 -- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 msgr2=0x7fe410168ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe410168ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 -- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe410158b20 msgr2=0x7fe4101697a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe410158b20 0x7fe4101697a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 -- 172.21.15.111:0/200146702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe408009ae0 con 0x7fe41015cee0 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415522700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 0x7fe4100c7430 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe408015000 tx=0x7fe408018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408008ad0 con 0x7fe41015cee0 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe408008c50 con 0x7fe41015cee0 2023-12-19T09:21:49.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408026460 con 0x7fe41015cee0 2023-12-19T09:21:49.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4100c7fb0 con 0x7fe41015cee0 2023-12-19T09:21:49.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.675+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4100c8570 con 0x7fe41015cee0 2023-12-19T09:21:49.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.679+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4100c9fc0 con 0x7fe41015cee0 2023-12-19T09:21:49.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.679+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 105) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fe408003040 con 0x7fe41015cee0 2023-12-19T09:21:49.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.679+0000 7fe40cff9700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 0x7fe3f4062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:49.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.679+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fe40807ed30 con 0x7fe41015cee0 2023-12-19T09:21:49.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.679+0000 7fe414d21700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 0x7fe3f4062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:49.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.683+0000 7fe414d21700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 0x7fe3f4062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe404009dd0 tx=0x7fe4040091a0 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:49.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.683+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe40804ec70 con 0x7fe41015cee0 2023-12-19T09:21:49.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.879+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe41004be40 con 0x7fe3f4060750 2023-12-19T09:21:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe40cff9700 1 -- 172.21.15.111:0/200146702 <== mgr.10279 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fe41004be40 con 0x7fe3f4060750 2023-12-19T09:21:49.900 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:49.900 INFO:teuthology.orchestra.run.smithi111.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T09:21:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 msgr2=0x7fe3f4062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 0x7fe3f4062c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe404009dd0 tx=0x7fe4040091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 msgr2=0x7fe4100c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 0x7fe4100c7430 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe408015000 tx=0x7fe408018000 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 shutdown_connections 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe3f4060750 0x7fe3f4062c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe410158b20 0x7fe4101697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe41015cee0 0x7fe4100c7430 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 --2- 172.21.15.111:0/200146702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4100cb3a0 0x7fe410168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.895+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 >> 172.21.15.111:0/200146702 conn(0x7fe4100cff20 msgr2=0x7fe410067060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.899+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 shutdown_connections 2023-12-19T09:21:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:49.899+0000 7fe415d23700 1 -- 172.21.15.111:0/200146702 wait complete. 2023-12-19T09:21:49.927 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-12-19T09:21:49.927 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-12-19T09:21:49.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-12-19T09:21:50.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.087+0000 7fa29542e700 1 Processor -- start 2023-12-19T09:21:50.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.087+0000 7fa29542e700 1 -- start start 2023-12-19T09:21:50.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa29542e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 0x7fa2900b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa29542e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa290069980 0x7fa2900b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa29542e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2900678d0 con 0x7fa290156f80 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa29542e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa290067bb0 con 0x7fa290069980 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa29542e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa290067e90 con 0x7fa2900abdc0 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa290069980 0x7fa2900b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa290156f80 0x7fa2900ab6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55152/0 (socket says 172.21.15.111:55152) 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28effd700 1 -- 172.21.15.111:0/2676438896 learned_addr learned my addr 172.21.15.111:0/2676438896 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa290069980 0x7fa2900b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37274/0 (socket says 172.21.15.111:37274) 2023-12-19T09:21:50.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa287fff700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 0x7fa2900b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 -- 172.21.15.111:0/2676438896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 msgr2=0x7fa2900b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 0x7fa2900b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 -- 172.21.15.111:0/2676438896 >> v1:172.21.15.111:6789/0 conn(0x7fa290156f80 legacy=0x7fa2900ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 -- 172.21.15.111:0/2676438896 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa290068170 con 0x7fa290069980 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa287fff700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 0x7fa2900b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28f7fe700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa2900b2360 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa288009e60 tx=0x7fa288009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4090b1d4dd19 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28e7fc700 1 -- 172.21.15.111:0/2676438896 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800e070 con 0x7fa290069980 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28e7fc700 1 -- 172.21.15.111:0/2676438896 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28800bc30 con 0x7fa290069980 2023-12-19T09:21:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.091+0000 7fa28e7fc700 1 -- 172.21.15.111:0/2676438896 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288012430 con 0x7fa290069980 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- 172.21.15.111:0/2676438896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 msgr2=0x7fa2900b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa2900b2360 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa288009e60 tx=0x7fa288009780 comp rx=0 tx=0).stop 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- 172.21.15.111:0/2676438896 shutdown_connections 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa2900b2360 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 --2- 172.21.15.111:0/2676438896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2900abdc0 0x7fa2900b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- 172.21.15.111:0/2676438896 >> 172.21.15.111:0/2676438896 conn(0x7fa290096190 msgr2=0x7fa2900afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- 172.21.15.111:0/2676438896 shutdown_connections 2023-12-19T09:21:50.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- 172.21.15.111:0/2676438896 wait complete. 2023-12-19T09:21:50.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 Processor -- start 2023-12-19T09:21:50.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.095+0000 7fa29542e700 1 -- start start 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2900abdc0 0x7fa2900c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa290156f80 0x7fa2900c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900cc6f0 con 0x7fa290069980 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900cc260 con 0x7fa2900abdc0 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa290156980 con 0x7fa290156f80 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37284/0 (socket says 172.21.15.111:37284) 2023-12-19T09:21:50.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 -- 172.21.15.111:0/76752109 learned_addr learned my addr 172.21.15.111:0/76752109 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:50.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28f7fe700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa290156f80 0x7fa2900c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa287fff700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2900abdc0 0x7fa2900c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 -- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa290156f80 msgr2=0x7fa2900c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa290156f80 0x7fa2900c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 -- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2900abdc0 msgr2=0x7fa2900c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2900abdc0 0x7fa2900c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 -- 172.21.15.111:0/76752109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288009a20 con 0x7fa290069980 2023-12-19T09:21:50.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa28effd700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa28000c590 tx=0x7fa280007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280005b30 con 0x7fa290069980 2023-12-19T09:21:50.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa280005cb0 con 0x7fa290069980 2023-12-19T09:21:50.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2900cb3b0 con 0x7fa290069980 2023-12-19T09:21:50.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280007320 con 0x7fa290069980 2023-12-19T09:21:50.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.099+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2900cb930 con 0x7fa290069980 2023-12-19T09:21:50.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.103+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa290064830 con 0x7fa290069980 2023-12-19T09:21:50.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.103+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 105) v1 ==== 80137+0+0 (secure 0 0 0) 0x7fa28001b3b0 con 0x7fa290069980 2023-12-19T09:21:50.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.107+0000 7fa286ffd700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 0x7fa270062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.107+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fa28007c2b0 con 0x7fa290069980 2023-12-19T09:21:50.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.107+0000 7fa287fff700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 0x7fa270062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.107+0000 7fa287fff700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 0x7fa270062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa29006a4f0 tx=0x7fa278008040 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.111+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa290064830 con 0x7fa290069980 2023-12-19T09:21:50.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.307+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2900cc3a0 con 0x7fa270060750 2023-12-19T09:21:50.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.431+0000 7fa286ffd700 1 -- 172.21.15.111:0/76752109 <== mgr.10279 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa278011700 con 0x7fa270060750 2023-12-19T09:21:50.437 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule removed for path snap_test_dir1 2023-12-19T09:21:50.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 msgr2=0x7fa270062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 0x7fa270062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa29006a4f0 tx=0x7fa278008040 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 msgr2=0x7fa29014b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa28000c590 tx=0x7fa280007f70 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 shutdown_connections 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa270060750 0x7fa270062c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2900abdc0 0x7fa2900c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa290156f80 0x7fa2900c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 --2- 172.21.15.111:0/76752109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa290069980 0x7fa29014b220 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 >> 172.21.15.111:0/76752109 conn(0x7fa290096190 msgr2=0x7fa2900c1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 shutdown_connections 2023-12-19T09:21:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.435+0000 7fa29542e700 1 -- 172.21.15.111:0/76752109 wait complete. 2023-12-19T09:21:50.462 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-12-19T09:21:50.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:50.462 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-12-19T09:21:50.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-12-19T09:21:50.500 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-12-19T09:21:50.575 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-12-19T09:21:50.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.783+0000 7f8a6564c700 1 Processor -- start 2023-12-19T09:21:50.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.783+0000 7f8a6564c700 1 -- start start 2023-12-19T09:21:50.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.783+0000 7f8a6564c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a6015aff0 0x7f8a6015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a6564c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a60069780 con 0x7f8a6015b970 2023-12-19T09:21:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a6564c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a60067960 con 0x7f8a60159d00 2023-12-19T09:21:50.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a6564c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60067770 con 0x7f8a6015aff0 2023-12-19T09:21:50.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a577fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8a6015b970 0x7f8a60067060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:55156/0 (socket says 172.21.15.111:55156) 2023-12-19T09:21:50.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a577fe700 1 -- 172.21.15.111:0/3398932112 learned_addr learned my addr 172.21.15.111:0/3398932112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:50.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5ffff700 1 --2- 172.21.15.111:0/3398932112 >> v2:172.21.15.143:3300/0 conn(0x7f8a6015aff0 0x7f8a6015b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669985315 0 0) 0x7f8a60069780 con 0x7f8a6015b970 2023-12-19T09:21:50.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a4c003570 con 0x7f8a6015b970 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841733331 0 0) 0x7f8a60067960 con 0x7f8a60159d00 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a60069780 con 0x7f8a60159d00 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5ffff700 1 -- 172.21.15.111:0/3398932112 >> v1:172.21.15.143:6789/0 conn(0x7f8a60159d00 legacy=0x7f8a6015a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5ffff700 1 -- 172.21.15.111:0/3398932112 >> v1:172.21.15.111:6789/0 conn(0x7f8a6015b970 legacy=0x7f8a60067060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5ffff700 1 -- 172.21.15.111:0/3398932112 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a60067c40 con 0x7f8a6015aff0 2023-12-19T09:21:50.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5ffff700 1 --2- 172.21.15.111:0/3398932112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6015b3c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009e60 tx=0x7f8a50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62c1f528cc8644c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000e070 con 0x7f8a6015aff0 2023-12-19T09:21:50.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5000bab0 con 0x7f8a6015aff0 2023-12-19T09:21:50.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a5f7fe700 1 -- 172.21.15.111:0/3398932112 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000be70 con 0x7f8a6015aff0 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a6564c700 1 -- 172.21.15.111:0/3398932112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 msgr2=0x7f8a6015b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.787+0000 7f8a6564c700 1 --2- 172.21.15.111:0/3398932112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6015b3c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009e60 tx=0x7f8a50009780 comp rx=0 tx=0).stop 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 -- 172.21.15.111:0/3398932112 shutdown_connections 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 --2- 172.21.15.111:0/3398932112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6015b3c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 -- 172.21.15.111:0/3398932112 >> 172.21.15.111:0/3398932112 conn(0x7f8a600cc530 msgr2=0x7f8a600c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 -- 172.21.15.111:0/3398932112 shutdown_connections 2023-12-19T09:21:50.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 -- 172.21.15.111:0/3398932112 wait complete. 2023-12-19T09:21:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 Processor -- start 2023-12-19T09:21:50.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.791+0000 7f8a6564c700 1 -- start start 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a60159d00 0x7f8a6016aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6016c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600b0990 con 0x7f8a6015b970 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600a9df0 con 0x7f8a6015aff0 2023-12-19T09:21:50.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600b06b0 con 0x7f8a60159d00 2023-12-19T09:21:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37292/0 (socket says 172.21.15.111:37292) 2023-12-19T09:21:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 -- 172.21.15.111:0/2829454691 learned_addr learned my addr 172.21.15.111:0/2829454691 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a577fe700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6016c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a5ffff700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a60159d00 0x7f8a6016aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 -- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a60159d00 msgr2=0x7f8a6016aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a60159d00 0x7f8a6016aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 -- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 msgr2=0x7f8a6016c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6016c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 -- 172.21.15.111:0/2829454691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a50009a20 con 0x7f8a6015b970 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a64e4b700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c590 tx=0x7f8a58007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58016070 con 0x7f8a6015b970 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a58011590 con 0x7f8a6015b970 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a600aa090 con 0x7f8a6015b970 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58011970 con 0x7f8a6015b970 2023-12-19T09:21:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.795+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a600aa650 con 0x7f8a6015b970 2023-12-19T09:21:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.799+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6004be40 con 0x7f8a6015b970 2023-12-19T09:21:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.799+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 105) v1 ==== 80137+0+0 (secure 0 0 0) 0x7f8a58012420 con 0x7f8a6015b970 2023-12-19T09:21:50.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.803+0000 7f8a5cff9700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 0x7f8a40062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:50.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.803+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f8a5807ca10 con 0x7f8a6015b970 2023-12-19T09:21:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.803+0000 7f8a5ffff700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 0x7f8a40062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.803+0000 7f8a5ffff700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 0x7f8a40062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009460 tx=0x7f8a50019000 comp rx=0 tx=0).ready entity=mgr.10279 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:50.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:50.811+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8a5804c900 con 0x7f8a6015b970 2023-12-19T09:21:51.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.019+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f8a60067f20 con 0x7f8a6015b970 2023-12-19T09:21:51.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.759+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f8a58011da0 con 0x7f8a6015b970 2023-12-19T09:21:51.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.763+0000 7f8a5cff9700 1 -- 172.21.15.111:0/2829454691 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v106) v1 ==== 93+0+0 (secure 0 0 0) 0x7f8a58050390 con 0x7f8a6015b970 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.763+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 msgr2=0x7f8a40062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.763+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 0x7f8a40062c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009460 tx=0x7f8a50019000 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.763+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 msgr2=0x7f8a600cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c590 tx=0x7f8a58007d80 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 shutdown_connections 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8a40060750 0x7f8a40062c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6015aff0 0x7f8a6016c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a6015b970 0x7f8a600cabc0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 --2- 172.21.15.111:0/2829454691 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a60159d00 0x7f8a6016aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 >> 172.21.15.111:0/2829454691 conn(0x7f8a600cc530 msgr2=0x7f8a60156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 shutdown_connections 2023-12-19T09:21:51.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:51.767+0000 7f8a6564c700 1 -- 172.21.15.111:0/2829454691 wait complete. 2023-12-19T09:21:51.796 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-19T09:21:51.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:51.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-19T09:21:51.877 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-19T09:21:51.878 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-12-19T09:21:51.878 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-12-19T09:21:51.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-19T09:21:51.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[63637]: fuse finished with error 0 and tester_r 0 2023-12-19T09:21:51.916 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:21:51.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:51.916 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T09:21:51.965 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:51.960+0000 7fd13bbc9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:51.970 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:51.963+0000 7ff3bb15d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T09:21:52.097 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:21:52.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:52.097 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-19T09:21:52.118 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-19T09:21:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:52.120 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:21:52.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:52.120 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-19T09:21:52.159 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-19T09:21:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:52.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:21:52.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:21:52.160 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-19T09:21:52.180 INFO:teuthology.orchestra.run.smithi143.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-19T09:21:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:21:52.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:52.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.351+0000 7fd5a359e700 1 Processor -- start 2023-12-19T09:21:52.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.351+0000 7fd5a359e700 1 -- start start 2023-12-19T09:21:52.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5a4067060 0x7fd5a40ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5a4069c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a406ab30 con 0x7fd5a40aa880 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a4067f80 con 0x7fd5a4067060 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a4068550 con 0x7fd5a4069840 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd5a40aa880 0x7fd5a4067430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44318/0 (socket says 172.21.15.111:44318) 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 -- 172.21.15.111:0/4283865721 learned_addr learned my addr 172.21.15.111:0/4283865721 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:52.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a2d9d700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5a4069c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a259c700 1 --2- 172.21.15.111:0/4283865721 >> v2:172.21.15.143:3300/0 conn(0x7fd5a4067060 0x7fd5a40ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669092341 0 0) 0x7fd5a406ab30 con 0x7fd5a40aa880 2023-12-19T09:21:52.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd584003570 con 0x7fd5a40aa880 2023-12-19T09:21:52.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327656579 0 0) 0x7fd584003570 con 0x7fd5a40aa880 2023-12-19T09:21:52.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 msgr2=0x7fd5a4069c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5a4069c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 >> v2:172.21.15.143:3300/0 conn(0x7fd5a4067060 msgr2=0x7fd5a40ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 --2- 172.21.15.111:0/4283865721 >> v2:172.21.15.143:3300/0 conn(0x7fd5a4067060 0x7fd5a40ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40acb00 con 0x7fd5a40aa880 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fd58c001470 con 0x7fd5a40aa880 2023-12-19T09:21:52.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 >> v1:172.21.15.111:6789/0 conn(0x7fd5a40aa880 legacy=0x7fd5a4067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584003a60 0x7fd584003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd584004810 0x7fd584004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a4067f80 con 0x7fd584003a60 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a4068550 con 0x7fd584004810 2023-12-19T09:21:52.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58400bb10 con 0x7fd5a4069840 2023-12-19T09:21:52.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a259c700 1 --2- 172.21.15.111:0/4283865721 >> v2:172.21.15.143:3300/0 conn(0x7fd5a4067060 0x7fd5a40ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:52.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a259c700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584003a60 0x7fd584003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd584004810 0x7fd584004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a2d9d700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 -- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 msgr2=0x7fd5840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:52.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a4069840 0x7fd5840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 -- 172.21.15.111:0/4283865721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584003a60 msgr2=0x7fd584003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:52.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584003a60 0x7fd584003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd59bfff700 1 -- 172.21.15.111:0/4283865721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584003570 con 0x7fd584004810 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.355+0000 7fd5a259c700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584003a60 0x7fd584003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd59bfff700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd584004810 0x7fd584004c80 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c3c0 tx=0x7fd5a40aa730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c847d835a86b3de4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00cc90 con 0x7fd584004810 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c00ce30 con 0x7fd584004810 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a1d9b700 1 -- 172.21.15.111:0/4283865721 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c005b70 con 0x7fd584004810 2023-12-19T09:21:52.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 -- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd584004810 msgr2=0x7fd584004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 --2- 172.21.15.111:0/4283865721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd584004810 0x7fd584004c80 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c3c0 tx=0x7fd5a40aa730 comp rx=0 tx=0).stop 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a2d9d700 1 -- 172.21.15.111:0/4283865721 reap_dead start 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 -- 172.21.15.111:0/4283865721 shutdown_connections 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 -- 172.21.15.111:0/4283865721 >> 172.21.15.111:0/4283865721 conn(0x7fd5a40cc530 msgr2=0x7fd5a40c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 -- 172.21.15.111:0/4283865721 shutdown_connections 2023-12-19T09:21:52.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 -- 172.21.15.111:0/4283865721 wait complete. 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.359+0000 7fd5a359e700 1 Processor -- start 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- start start 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a4162de0 0x7fd5a4163250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40c4230 0x7fd5a40c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40c1850 con 0x7fd584004810 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40ae830 con 0x7fd5a4162de0 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a4163790 con 0x7fd5a40c4230 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50662/0 (socket says 172.21.15.111:50662) 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 -- 172.21.15.111:0/96217526 learned_addr learned my addr 172.21.15.111:0/96217526 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:52.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd59bfff700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a4162de0 0x7fd5a4163250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a2d9d700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40c4230 0x7fd5a40c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40c4230 msgr2=0x7fd5a40c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40c4230 0x7fd5a40c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a4162de0 msgr2=0x7fd5a4163250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a4162de0 0x7fd5a4163250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 -- 172.21.15.111:0/96217526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40bf550 con 0x7fd584004810 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a259c700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fd594011ec0 tx=0x7fd5940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59400fc70 con 0x7fd584004810 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59400fdd0 con 0x7fd584004810 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a40c4f10 con 0x7fd584004810 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594014650 con 0x7fd584004810 2023-12-19T09:21:52.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.363+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a40c54d0 con 0x7fd584004810 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.367+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a404be40 con 0x7fd584004810 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.367+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fd594014a80 con 0x7fd584004810 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.367+0000 7fd59a7fc700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 0x7fd588062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.371+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fd594018070 con 0x7fd584004810 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.371+0000 7fd59bfff700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 msgr2=0x7fd588062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:52.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.371+0000 7fd59bfff700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 0x7fd588062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:52.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.375+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd59404dbf0 con 0x7fd584004810 2023-12-19T09:21:52.518 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:52.511+0000 7ff3bb15d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:52.531 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:52.528+0000 7fd13bbc9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T09:21:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.575+0000 7fd59bfff700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 msgr2=0x7fd588062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.575+0000 7fd59bfff700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 0x7fd588062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:52.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.579+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5a40687f0 con 0x7fd584004810 2023-12-19T09:21:52.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd59a7fc700 1 -- 172.21.15.111:0/96217526 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7fd59407de60 con 0x7fd584004810 2023-12-19T09:21:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:52.589 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:52.590 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:52.583+0000 7ff3bb15d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 msgr2=0x7fd588062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 0x7fd588062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 msgr2=0x7fd5a40b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fd594011ec0 tx=0x7fd5940091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 shutdown_connections 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd588060700 0x7fd588062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5a4162de0 0x7fd5a4163250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5a40c4230 0x7fd5a40c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 --2- 172.21.15.111:0/96217526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd584004810 0x7fd5a40b1100 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 >> 172.21.15.111:0/96217526 conn(0x7fd5a40cc530 msgr2=0x7fd5a40ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 shutdown_connections 2023-12-19T09:21:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.583+0000 7fd5a359e700 1 -- 172.21.15.111:0/96217526 wait complete. 2023-12-19T09:21:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:52.608 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:52.604+0000 7fd13bbc9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T09:21:52.609 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:52.727 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:52.719+0000 7ff3bb15d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:52.757 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:52.756+0000 7fd13bbc9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 Processor -- start 2023-12-19T09:21:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 -- start start 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8158e80 0x7fd2d815d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2d815cd40 0x7fd2d815d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80b37a0 con 0x7fd2d8159250 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d815d700 con 0x7fd2d815cd40 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2dd7c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80682a0 con 0x7fd2d8158e80 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2d77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2d815cd40 0x7fd2d815d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2d77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd2d815cd40 0x7fd2d815d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50668/0 (socket says 172.21.15.111:50668) 2023-12-19T09:21:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.759+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1153593545 learned_addr learned my addr 172.21.15.111:0/1153593545 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d6ffd700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8158e80 0x7fd2d815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1153593545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8158e80 msgr2=0x7fd2d815d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d77fe700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8158e80 0x7fd2d815d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1153593545 >> v1:172.21.15.111:6789/0 conn(0x7fd2d8159250 legacy=0x7fd2d8067a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1153593545 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8068580 con 0x7fd2d815cd40 2023-12-19T09:21:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d77fe700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d815d1c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00099b0 tx=0x7fd2d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f065499fd1f7f0f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d67fc700 1 -- 172.21.15.111:0/1153593545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0009ea0 con 0x7fd2d815cd40 2023-12-19T09:21:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d67fc700 1 -- 172.21.15.111:0/1153593545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d000b860 con 0x7fd2d815cd40 2023-12-19T09:21:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2d67fc700 1 -- 172.21.15.111:0/1153593545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000bc00 con 0x7fd2d815cd40 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1153593545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 msgr2=0x7fd2d815d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d815d1c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00099b0 tx=0x7fd2d0009250 comp rx=0 tx=0).stop 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1153593545 shutdown_connections 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d815d1c0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1153593545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8158e80 0x7fd2d815d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1153593545 >> 172.21.15.111:0/1153593545 conn(0x7fd2d80cfed0 msgr2=0x7fd2d80d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1153593545 shutdown_connections 2023-12-19T09:21:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.763+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1153593545 wait complete. 2023-12-19T09:21:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 Processor -- start 2023-12-19T09:21:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 -- start start 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8158e80 0x7fd2d8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8159250 0x7fd2d80a7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80a7c10 con 0x7fd2d815cd40 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80a7ef0 con 0x7fd2d8158e80 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2dd7c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80a81d0 con 0x7fd2d8159250 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50676/0 (socket says 172.21.15.111:50676) 2023-12-19T09:21:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1542330551 learned_addr learned my addr 172.21.15.111:0/1542330551 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2ce7fc700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8159250 0x7fd2d80a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.767+0000 7fd2d6ffd700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8158e80 0x7fd2d8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8159250 msgr2=0x7fd2d80a7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8159250 0x7fd2d80a7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8158e80 msgr2=0x7fd2d8152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8158e80 0x7fd2d8152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 -- 172.21.15.111:0/1542330551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00094f0 con 0x7fd2d815cd40 2023-12-19T09:21:52.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2d77fe700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8088510 tx=0x7fd2d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000bbe0 con 0x7fd2d815cd40 2023-12-19T09:21:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0011420 con 0x7fd2d815cd40 2023-12-19T09:21:52.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d80a84b0 con 0x7fd2d815cd40 2023-12-19T09:21:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.771+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00117e0 con 0x7fd2d815cd40 2023-12-19T09:21:52.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.775+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d815b8c0 con 0x7fd2d815cd40 2023-12-19T09:21:52.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.775+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d804be40 con 0x7fd2d815cd40 2023-12-19T09:21:52.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.775+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fd2d0011bd0 con 0x7fd2d815cd40 2023-12-19T09:21:52.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.775+0000 7fd2cf7fe700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 0x7fd2b8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:52.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.775+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fd2d007e960 con 0x7fd2d815cd40 2023-12-19T09:21:52.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.779+0000 7fd2d6ffd700 1 -- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 msgr2=0x7fd2b8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:52.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.779+0000 7fd2d6ffd700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 0x7fd2b8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:52.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.783+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd2d004e850 con 0x7fd2d815cd40 2023-12-19T09:21:52.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.947+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d806a2e0 con 0x7fd2d815cd40 2023-12-19T09:21:52.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.951+0000 7fd2cf7fe700 1 -- 172.21.15.111:0/1542330551 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16867 (secure 0 0 0) 0x7fd2d0051e70 con 0x7fd2d815cd40 2023-12-19T09:21:52.957 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:52.957 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":209,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:38.103636+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/278966201":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/2861944468":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6833/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6832/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/150889021":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/800113432":"2023-12-20T09:21:38.103524+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.951+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 msgr2=0x7fd2b8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.951+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 0x7fd2b8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 msgr2=0x7fd2d80b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8088510 tx=0x7fd2d0017000 comp rx=0 tx=0).stop 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 shutdown_connections 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d815cd40 0x7fd2d80b3e10 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2d8159250 0x7fd2d80a7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd2b8060700 0x7fd2b8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 --2- 172.21.15.111:0/1542330551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8158e80 0x7fd2d8152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 >> 172.21.15.111:0/1542330551 conn(0x7fd2d80cfed0 msgr2=0x7fd2d8154170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 shutdown_connections 2023-12-19T09:21:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:52.955+0000 7fd2dd7c5700 1 -- 172.21.15.111:0/1542330551 wait complete. 2023-12-19T09:21:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T07:47:06.696458+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-19T09:21:20.323729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-12-19T09:21:21.351082+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T09:21:52.977 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T09:21:52.995 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:52.991+0000 7ff3bb15d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:53.042 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.040+0000 7fd13bbc9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T09:21:53.074 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:53.067+0000 7ff3bb15d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:53.119 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.116+0000 7fd13bbc9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:53.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.119+0000 7f278eb63700 1 Processor -- start 2023-12-19T09:21:53.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.119+0000 7f278eb63700 1 -- start start 2023-12-19T09:21:53.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.119+0000 7f278eb63700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f27880b23d0 0x7f2788067a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880caf70 con 0x7f278815bca0 2023-12-19T09:21:53.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2788068350 con 0x7f278815cc40 2023-12-19T09:21:53.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2788068630 con 0x7f27880b23d0 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278d360700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f27880b23d0 0x7f2788067a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278db61700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f278815cc40 0x7f278815d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56668/0 (socket says 172.21.15.111:56668) 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278db61700 1 -- 172.21.15.111:0/2263164051 learned_addr learned my addr 172.21.15.111:0/2263164051 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278d360700 1 -- 172.21.15.111:0/2263164051 >> v1:172.21.15.143:6789/0 conn(0x7f278815cc40 legacy=0x7f278815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278d360700 1 -- 172.21.15.111:0/2263164051 >> v1:172.21.15.111:6789/0 conn(0x7f278815bca0 legacy=0x7f278815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278d360700 1 -- 172.21.15.111:0/2263164051 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788068910 con 0x7f27880b23d0 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278cb5f700 1 -- 172.21.15.111:0/2263164051 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190391510 0 0) 0x7f2788068350 con 0x7f278815cc40 2023-12-19T09:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278d360700 1 --2- 172.21.15.111:0/2263164051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27880b23d0 0x7f2788067a10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f27780098e0 tx=0x7f27780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1aa8dd101f397e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278cb5f700 1 -- 172.21.15.111:0/2263164051 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778009e80 con 0x7f27880b23d0 2023-12-19T09:21:53.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278cb5f700 1 -- 172.21.15.111:0/2263164051 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277800b860 con 0x7f27880b23d0 2023-12-19T09:21:53.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278cb5f700 1 -- 172.21.15.111:0/2263164051 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800bbe0 con 0x7f27880b23d0 2023-12-19T09:21:53.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- 172.21.15.111:0/2263164051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27880b23d0 msgr2=0x7f2788067a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 --2- 172.21.15.111:0/2263164051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27880b23d0 0x7f2788067a10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f27780098e0 tx=0x7f27780091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- 172.21.15.111:0/2263164051 shutdown_connections 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 --2- 172.21.15.111:0/2263164051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27880b23d0 0x7f2788067a10 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- 172.21.15.111:0/2263164051 >> 172.21.15.111:0/2263164051 conn(0x7f2788069b90 msgr2=0x7f27880cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- 172.21.15.111:0/2263164051 shutdown_connections 2023-12-19T09:21:53.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.123+0000 7f278eb63700 1 -- 172.21.15.111:0/2263164051 wait complete. 2023-12-19T09:21:53.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 Processor -- start 2023-12-19T09:21:53.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 -- start start 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 0x7f278815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 0x7f2788169790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2788168fd0 con 0x7f278815cc40 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880c9500 con 0x7f27880b23d0 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278eb63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27880c97e0 con 0x7f278815bca0 2023-12-19T09:21:53.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278db61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278e362700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 0x7f2788169790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278db61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:54942/0 (socket says 172.21.15.111:54942) 2023-12-19T09:21:53.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278db61700 1 -- 172.21.15.111:0/2727937933 learned_addr learned my addr 172.21.15.111:0/2727937933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:53.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.127+0000 7f278d360700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 0x7f278815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 -- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 msgr2=0x7f278815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 0x7f278815c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 -- 172.21.15.111:0/2727937933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 msgr2=0x7f2788169790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 0x7f2788169790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 -- 172.21.15.111:0/2727937933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778009420 con 0x7f27880b23d0 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278d360700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 0x7f278815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278e362700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 0x7f2788169790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:53.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278db61700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f278400c5b0 tx=0x7f2784001070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784005b30 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784005c90 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27880c9ac0 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784007300 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27880c9160 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2788068110 con 0x7f27880b23d0 2023-12-19T09:21:53.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.131+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f278401b3b0 con 0x7f27880b23d0 2023-12-19T09:21:53.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.135+0000 7f277dffb700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 0x7f276c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.135+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f278407c4f0 con 0x7f27880b23d0 2023-12-19T09:21:53.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.135+0000 7f278d360700 1 -- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 msgr2=0x7f276c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:53.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.135+0000 7f278d360700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 0x7f276c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:53.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.143+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2788068110 con 0x7f27880b23d0 2023-12-19T09:21:53.296 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:53.291+0000 7ff3bb15d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:53.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.315+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2788068110 con 0x7f27880b23d0 2023-12-19T09:21:53.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f277dffb700 1 -- 172.21.15.111:0/2727937933 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2788068110 con 0x7f27880b23d0 2023-12-19T09:21:53.325 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-12-19T09:21:53.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 msgr2=0x7f276c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:53.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 0x7f276c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 msgr2=0x7f27880c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f278400c5b0 tx=0x7f2784001070 comp rx=0 tx=0).stop 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 shutdown_connections 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f278815cc40 0x7f2788169790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f278815bca0 0x7f278815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f276c060700 0x7f276c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 --2- 172.21.15.111:0/2727937933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27880b23d0 0x7f27880c8c20 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.319+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 >> 172.21.15.111:0/2727937933 conn(0x7f2788069b90 msgr2=0x7f2788067a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.323+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 shutdown_connections 2023-12-19T09:21:53.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.323+0000 7f278eb63700 1 -- 172.21.15.111:0/2727937933 wait complete. 2023-12-19T09:21:53.346 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-19T09:21:53.346 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.344+0000 7fd13bbc9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T09:21:53.370 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:53.363+0000 7ff3bb15d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:53.430 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.428+0000 7fd13bbc9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T09:21:53.457 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:53.451+0000 7ff3bb15d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:53.519 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.516+0000 7fd13bbc9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T09:21:53.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.535+0000 7fc88b08e700 1 Processor -- start 2023-12-19T09:21:53.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.535+0000 7fc88b08e700 1 -- start start 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88b08e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc88415a040 0x7fc8840b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88b08e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8840cef90 0x7fc88415ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88b08e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884067060 con 0x7fc884158c60 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88b08e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc884067900 con 0x7fc8840cef90 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88b08e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840cf750 con 0x7fc88415a040 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8840cef90 0x7fc88415ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc8840cef90 0x7fc88415ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50692/0 (socket says 172.21.15.111:50692) 2023-12-19T09:21:53.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 -- 172.21.15.111:0/4287987675 learned_addr learned my addr 172.21.15.111:0/4287987675 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:53.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88988b700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc88415a040 0x7fc8840b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 -- 172.21.15.111:0/4287987675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc88415a040 msgr2=0x7fc8840b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc88415a040 0x7fc8840b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 -- 172.21.15.111:0/4287987675 >> v1:172.21.15.143:6789/0 conn(0x7fc884158c60 legacy=0x7fc8840b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 -- 172.21.15.111:0/4287987675 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840672b0 con 0x7fc8840cef90 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88a88d700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8840cef90 0x7fc88415ca30 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009f20 tx=0x7fc87c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae985e7e7071c61 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88908a700 1 -- 172.21.15.111:0/4287987675 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00e070 con 0x7fc8840cef90 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88908a700 1 -- 172.21.15.111:0/4287987675 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c00b9d0 con 0x7fc8840cef90 2023-12-19T09:21:53.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.539+0000 7fc88908a700 1 -- 172.21.15.111:0/4287987675 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0092e0 con 0x7fc8840cef90 2023-12-19T09:21:53.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- 172.21.15.111:0/4287987675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8840cef90 msgr2=0x7fc88415ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8840cef90 0x7fc88415ca30 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009f20 tx=0x7fc87c009840 comp rx=0 tx=0).stop 2023-12-19T09:21:53.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- 172.21.15.111:0/4287987675 shutdown_connections 2023-12-19T09:21:53.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8840cef90 0x7fc88415ca30 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 --2- 172.21.15.111:0/4287987675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc88415a040 0x7fc8840b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- 172.21.15.111:0/4287987675 >> 172.21.15.111:0/4287987675 conn(0x7fc884068f80 msgr2=0x7fc884069380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:53.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- 172.21.15.111:0/4287987675 shutdown_connections 2023-12-19T09:21:53.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- 172.21.15.111:0/4287987675 wait complete. 2023-12-19T09:21:53.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 Processor -- start 2023-12-19T09:21:53.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.543+0000 7fc88b08e700 1 -- start start 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840cef90 0x7fc884154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 0x7fc884157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840a9320 con 0x7fc884158c60 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840a74c0 con 0x7fc88415a040 2023-12-19T09:21:53.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840b2050 con 0x7fc8840cef90 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50696/0 (socket says 172.21.15.111:50696) 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 -- 172.21.15.111:0/3108416386 learned_addr learned my addr 172.21.15.111:0/3108416386 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88a88d700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 0x7fc884157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88a08c700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840cef90 0x7fc884154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 -- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840cef90 msgr2=0x7fc884154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840cef90 0x7fc884154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 -- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 msgr2=0x7fc884157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 0x7fc884157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 -- 172.21.15.111:0/3108416386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c009ae0 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88a88d700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 0x7fc884157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88988b700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc87400ec80 tx=0x7fc8740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87400c9d0 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87400cb30 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874013470 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.547+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8840a68b0 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8840a6e70 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fc874013820 con 0x7fc884158c60 2023-12-19T09:21:53.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840a7600 con 0x7fc884158c60 2023-12-19T09:21:53.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc87a7fc700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 0x7fc868062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fc87407d440 con 0x7fc884158c60 2023-12-19T09:21:53.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc88a08c700 1 -- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 msgr2=0x7fc868062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:53.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.551+0000 7fc88a08c700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 0x7fc868062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:53.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.559+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc87404d290 con 0x7fc884158c60 2023-12-19T09:21:53.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.731+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc88404be40 con 0x7fc884158c60 2023-12-19T09:21:53.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc87a7fc700 1 -- 172.21.15.111:0/3108416386 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v209) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc8740508b0 con 0x7fc884158c60 2023-12-19T09:21:53.756 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-12-19T09:21:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 msgr2=0x7fc868062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 0x7fc868062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 msgr2=0x7fc8840b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc87400ec80 tx=0x7fc8740091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 shutdown_connections 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc868060700 0x7fc868062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc88415a040 0x7fc884157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc884158c60 0x7fc8840b0c70 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 --2- 172.21.15.111:0/3108416386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8840cef90 0x7fc884154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 >> 172.21.15.111:0/3108416386 conn(0x7fc884068f80 msgr2=0x7fc88415a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 shutdown_connections 2023-12-19T09:21:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.735+0000 7fc88b08e700 1 -- 172.21.15.111:0/3108416386 wait complete. 2023-12-19T09:21:53.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-19T09:21:53.839 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:53.831+0000 7ff3bb15d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:53.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.911+0000 7f70e9585700 1 Processor -- start 2023-12-19T09:21:53.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.911+0000 7f70e9585700 1 -- start start 2023-12-19T09:21:53.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e9585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e9585700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70e415d730 0x7f70e40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e9585700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e406a530 con 0x7f70e4159350 2023-12-19T09:21:53.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e9585700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e4069c00 con 0x7f70e4158f80 2023-12-19T09:21:53.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e9585700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e4069ee0 con 0x7f70e415d730 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50702/0 (socket says 172.21.15.111:50702) 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e2ffd700 1 -- 172.21.15.111:0/290266406 learned_addr learned my addr 172.21.15.111:0/290266406 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 --2- 172.21.15.111:0/290266406 >> v2:172.21.15.143:3300/0 conn(0x7f70e415d730 0x7f70e40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 -- 172.21.15.111:0/290266406 >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 msgr2=0x7f70e40bf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 --2- 172.21.15.111:0/290266406 >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 -- 172.21.15.111:0/290266406 >> v1:172.21.15.111:6789/0 conn(0x7f70e4159350 legacy=0x7f70e406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 -- 172.21.15.111:0/290266406 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e406a1c0 con 0x7f70e415d730 2023-12-19T09:21:53.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e2ffd700 1 --2- 172.21.15.111:0/290266406 >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:53.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e27fc700 1 --2- 172.21.15.111:0/290266406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e40c6aa0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f70e406d4d0 tx=0x7f70d80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff43a3c5665f3f90 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e1ffb700 1 -- 172.21.15.111:0/290266406 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8011630 con 0x7f70e415d730 2023-12-19T09:21:53.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e1ffb700 1 -- 172.21.15.111:0/290266406 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d8011790 con 0x7f70e415d730 2023-12-19T09:21:53.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.915+0000 7f70e1ffb700 1 -- 172.21.15.111:0/290266406 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8011b30 con 0x7f70e415d730 2023-12-19T09:21:53.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- 172.21.15.111:0/290266406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 msgr2=0x7f70e40c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 --2- 172.21.15.111:0/290266406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e40c6aa0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f70e406d4d0 tx=0x7f70d80091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- 172.21.15.111:0/290266406 shutdown_connections 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 --2- 172.21.15.111:0/290266406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e40c6aa0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 --2- 172.21.15.111:0/290266406 >> v2:172.21.15.111:3300/0 conn(0x7f70e4158f80 0x7f70e40bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- 172.21.15.111:0/290266406 >> 172.21.15.111:0/290266406 conn(0x7f70e40cc6c0 msgr2=0x7f70e40c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- 172.21.15.111:0/290266406 shutdown_connections 2023-12-19T09:21:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- 172.21.15.111:0/290266406 wait complete. 2023-12-19T09:21:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 Processor -- start 2023-12-19T09:21:53.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.919+0000 7f70e9585700 1 -- start start 2023-12-19T09:21:53.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70e4159350 0x7f70e40b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e4169230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40b1900 con 0x7f70e4158f80 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40aec40 con 0x7f70e415d730 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40b0750 con 0x7f70e4159350 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50704/0 (socket says 172.21.15.111:50704) 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 -- 172.21.15.111:0/2518235168 learned_addr learned my addr 172.21.15.111:0/2518235168 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e27fc700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70e4159350 0x7f70e40b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e37fe700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e4169230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:53.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 -- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70e4159350 msgr2=0x7f70e40b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70e4159350 0x7f70e40b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 -- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 msgr2=0x7f70e4169230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:53.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e4169230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 -- 172.21.15.111:0/2518235168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40094f0 con 0x7f70e4158f80 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e2ffd700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f70d400ec80 tx=0x7f70d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400c850 con 0x7f70e4158f80 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d8009420 con 0x7f70e4158f80 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400c9b0 con 0x7f70e4158f80 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400cd50 con 0x7f70e4158f80 2023-12-19T09:21:53.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.923+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e40b1340 con 0x7f70e4158f80 2023-12-19T09:21:53.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e4071110 con 0x7f70e4158f80 2023-12-19T09:21:53.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f70d4013680 con 0x7f70e4158f80 2023-12-19T09:21:53.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70caffd700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 0x7f70d0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:53.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f70d407d390 con 0x7f70e4158f80 2023-12-19T09:21:53.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70e27fc700 1 -- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 msgr2=0x7f70d0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:53.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.927+0000 7f70e27fc700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 0x7f70d0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:53.936 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:53.932+0000 7fd13bbc9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T09:21:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:53.935+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f70d404d280 con 0x7f70e4158f80 2023-12-19T09:21:54.013 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.007+0000 7ff3bb15d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:54.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.099+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70e40b16e0 con 0x7f70e4158f80 2023-12-19T09:21:54.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.099+0000 7f70caffd700 1 -- 172.21.15.111:0/2518235168 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v209) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70d40508a0 con 0x7f70e4158f80 2023-12-19T09:21:54.106 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-12-19T09:21:54.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 msgr2=0x7f70d0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:54.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 0x7f70d0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 msgr2=0x7f70e40b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f70d400ec80 tx=0x7f70d4009250 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 shutdown_connections 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f70d0060700 0x7f70d0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e415d730 0x7f70e4169230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70e4159350 0x7f70e40b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 --2- 172.21.15.111:0/2518235168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e4158f80 0x7f70e40b0a80 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 >> 172.21.15.111:0/2518235168 conn(0x7f70e40cc6c0 msgr2=0x7f70e40cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 shutdown_connections 2023-12-19T09:21:54.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.103+0000 7f70e9585700 1 -- 172.21.15.111:0/2518235168 wait complete. 2023-12-19T09:21:54.112 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.108+0000 7fd13bbc9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T09:21:54.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:54.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 Processor -- start 2023-12-19T09:21:54.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 -- start start 2023-12-19T09:21:54.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbdf4067430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf406b1a0 con 0x7fbdf40c92b0 2023-12-19T09:21:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40ae350 con 0x7fbdf40c8e60 2023-12-19T09:21:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf9545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40ac740 con 0x7fbdf4067060 2023-12-19T09:21:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbdf40c92b0 0x7fbdf40ac030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44364/0 (socket says 172.21.15.111:44364) 2023-12-19T09:21:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf37fe700 1 -- 172.21.15.111:0/1478936259 learned_addr learned my addr 172.21.15.111:0/1478936259 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbdf4067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234939274 0 0) 0x7fbdf406b1a0 con 0x7fbdf40c92b0 2023-12-19T09:21:54.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc003570 con 0x7fbdf40c92b0 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413022832 0 0) 0x7fbdf40ae350 con 0x7fbdf40c8e60 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf406b1a0 con 0x7fbdf40c8e60 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787155449 0 0) 0x7fbddc003570 con 0x7fbdf40c92b0 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 msgr2=0x7fbdf4067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbdf4067430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 >> v1:172.21.15.143:6789/0 conn(0x7fbdf40c8e60 legacy=0x7fbdf40ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4067970 con 0x7fbdf40c92b0 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.283+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fbdec0017f0 con 0x7fbdf40c92b0 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 >> v1:172.21.15.111:6789/0 conn(0x7fbdf40c92b0 legacy=0x7fbdf40ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbddc003c20 0x7fbddc004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbddc004810 0x7fbddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbddc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40ac740 con 0x7fbddc003c20 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc00bb40 con 0x7fbddc004810 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc00bd00 con 0x7fbdf4067060 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf27fc700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbddc004810 0x7fbddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbddc003c20 0x7fbddc004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf37fe700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbddc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 -- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 msgr2=0x7fbddc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf4067060 0x7fbddc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 -- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbddc004810 msgr2=0x7fbddc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbddc004810 0x7fbddc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 -- 172.21.15.111:0/1478936259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc003570 con 0x7fbddc003c20 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbddc003c20 0x7fbddc004010 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009bf0 tx=0x7fbde4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2575cb491b13c7d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4005a00 con 0x7fbddc003c20 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4005b60 con 0x7fbddc003c20 2023-12-19T09:21:54.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf1ffb700 1 -- 172.21.15.111:0/1478936259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde400e710 con 0x7fbddc003c20 2023-12-19T09:21:54.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 -- 172.21.15.111:0/1478936259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbddc003c20 msgr2=0x7fbddc004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 --2- 172.21.15.111:0/1478936259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbddc003c20 0x7fbddc004010 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009bf0 tx=0x7fbde4009120 comp rx=0 tx=0).stop 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf37fe700 1 -- 172.21.15.111:0/1478936259 reap_dead start 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 -- 172.21.15.111:0/1478936259 shutdown_connections 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 -- 172.21.15.111:0/1478936259 >> 172.21.15.111:0/1478936259 conn(0x7fbdf40cc690 msgr2=0x7fbdf40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 -- 172.21.15.111:0/1478936259 shutdown_connections 2023-12-19T09:21:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.287+0000 7fbdf9545700 1 -- 172.21.15.111:0/1478936259 wait complete. 2023-12-19T09:21:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 Processor -- start 2023-12-19T09:21:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 -- start start 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf414f4a0 0x7fbdf414d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 0x7fbdf40bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40c5c40 con 0x7fbdf414f870 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40c4c20 con 0x7fbdf414f4a0 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf9545700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf414df10 con 0x7fbdf40bf9c0 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50730/0 (socket says 172.21.15.111:50730) 2023-12-19T09:21:54.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.291+0000 7fbdf27fc700 1 -- 172.21.15.111:0/515174230 learned_addr learned my addr 172.21.15.111:0/515174230 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf414f4a0 0x7fbdf414d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf37fe700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 0x7fbdf40bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 msgr2=0x7fbdf40bfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 0x7fbdf40bfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf414f4a0 msgr2=0x7fbdf414d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf414f4a0 0x7fbdf414d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 -- 172.21.15.111:0/515174230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40c4010 con 0x7fbdf414f870 2023-12-19T09:21:54.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf37fe700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 0x7fbdf40bfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf27fc700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fbde8002be0 tx=0x7fbde8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800f9a0 con 0x7fbdf414f870 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde800fb20 con 0x7fbdf414f870 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf40c42f0 con 0x7fbdf414f870 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800fee0 con 0x7fbdf414f870 2023-12-19T09:21:54.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf40c48b0 con 0x7fbdf414f870 2023-12-19T09:21:54.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.295+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf4071110 con 0x7fbdf414f870 2023-12-19T09:21:54.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.299+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fbde8013600 con 0x7fbdf414f870 2023-12-19T09:21:54.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.299+0000 7fbddaffd700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 0x7fbde0062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.299+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fbde807d760 con 0x7fbdf414f870 2023-12-19T09:21:54.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.299+0000 7fbdf2ffd700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 msgr2=0x7fbde0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:54.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.299+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 0x7fbde0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.303+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbde804d5b0 con 0x7fbdf414f870 2023-12-19T09:21:54.368 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.363+0000 7ff3bb15d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:54.493 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.488+0000 7fd13bbc9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T09:21:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.503+0000 7fbdf2ffd700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 msgr2=0x7fbde0062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.503+0000 7fbdf2ffd700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 0x7fbde0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.507+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf40bf780 con 0x7fbdf414f870 2023-12-19T09:21:54.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.507+0000 7fbddaffd700 1 -- 172.21.15.111:0/515174230 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7fbde8050bd0 con 0x7fbdf414f870 2023-12-19T09:21:54.513 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:54.514 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 msgr2=0x7fbde0062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 0x7fbde0062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 msgr2=0x7fbdf414fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fbde8002be0 tx=0x7fbde8007d30 comp rx=0 tx=0).stop 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 shutdown_connections 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf40bf9c0 0x7fbdf40bfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf414f870 0x7fbdf414fce0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbde0060700 0x7fbde0062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 --2- 172.21.15.111:0/515174230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf414f4a0 0x7fbdf414d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 >> 172.21.15.111:0/515174230 conn(0x7fbdf40cc690 msgr2=0x7fbdf40cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 shutdown_connections 2023-12-19T09:21:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.511+0000 7fbdf9545700 1 -- 172.21.15.111:0/515174230 wait complete. 2023-12-19T09:21:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:54.535 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:54.591 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.583+0000 7ff3bb15d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:54.684 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.679+0000 7ff3bb15d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:54.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.687+0000 7f7c308f0700 1 Processor -- start 2023-12-19T09:21:54.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c308f0700 1 -- start start 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c308f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c2c155f20 0x7f7c2c068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c308f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c067da0 con 0x7f7c2c068770 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c308f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c067190 con 0x7f7c2c0c78e0 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c308f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c067470 con 0x7f7c2c155f20 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c2c155f20 0x7f7c2c068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c2c155f20 0x7f7c2c068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50740/0 (socket says 172.21.15.111:50740) 2023-12-19T09:21:54.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 -- 172.21.15.111:0/937382450 learned_addr learned my addr 172.21.15.111:0/937382450 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:54.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 -- 172.21.15.111:0/937382450 >> v1:172.21.15.143:6789/0 conn(0x7f7c2c0c78e0 legacy=0x7f7c2c0c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 -- 172.21.15.111:0/937382450 >> v1:172.21.15.111:6789/0 conn(0x7f7c2c068770 legacy=0x7f7c2c0ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.691+0000 7f7c2a59c700 1 -- 172.21.15.111:0/937382450 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c067750 con 0x7f7c2c155f20 2023-12-19T09:21:54.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c2a59c700 1 --2- 172.21.15.111:0/937382450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c068230 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009990 tx=0x7f7c1c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998f1c209ec7393 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:54.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c2959a700 1 -- 172.21.15.111:0/937382450 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c005c10 con 0x7f7c2c155f20 2023-12-19T09:21:54.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c2959a700 1 -- 172.21.15.111:0/937382450 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c005d70 con 0x7f7c2c155f20 2023-12-19T09:21:54.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c2959a700 1 -- 172.21.15.111:0/937382450 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c00b860 con 0x7f7c2c155f20 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 -- 172.21.15.111:0/937382450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 msgr2=0x7f7c2c068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 --2- 172.21.15.111:0/937382450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c068230 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009990 tx=0x7f7c1c009220 comp rx=0 tx=0).stop 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 -- 172.21.15.111:0/937382450 shutdown_connections 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 --2- 172.21.15.111:0/937382450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c068230 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 -- 172.21.15.111:0/937382450 >> 172.21.15.111:0/937382450 conn(0x7f7c2c0cc690 msgr2=0x7f7c2c0c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:54.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 -- 172.21.15.111:0/937382450 shutdown_connections 2023-12-19T09:21:54.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.695+0000 7f7c308f0700 1 -- 172.21.15.111:0/937382450 wait complete. 2023-12-19T09:21:54.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 Processor -- start 2023-12-19T09:21:54.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 -- start start 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c068770 0x7f7c2c0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c78e0 0x7f7c2c0c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c0830 con 0x7f7c2c155f20 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c0af0 con 0x7f7c2c0c78e0 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c308f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0c0dd0 con 0x7f7c2c068770 2023-12-19T09:21:54.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50744/0 (socket says 172.21.15.111:50744) 2023-12-19T09:21:54.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 -- 172.21.15.111:0/2619106211 learned_addr learned my addr 172.21.15.111:0/2619106211 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:54.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2a59c700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c068770 0x7f7c2c0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c29d9b700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c78e0 0x7f7c2c0c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 -- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c068770 msgr2=0x7f7c2c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c068770 0x7f7c2c0c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 -- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c78e0 msgr2=0x7f7c2c0c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c78e0 0x7f7c2c0c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 -- 172.21.15.111:0/2619106211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c009480 con 0x7f7c2c155f20 2023-12-19T09:21:54.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.699+0000 7f7c2ad9d700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c840 tx=0x7f7c24001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24012630 con 0x7f7c2c155f20 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c240127b0 con 0x7f7c2c155f20 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c0c10b0 con 0x7f7c2c155f20 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24012b70 con 0x7f7c2c155f20 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c0c1df0 con 0x7f7c2c155f20 2023-12-19T09:21:54.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c0c15b0 con 0x7f7c2c155f20 2023-12-19T09:21:54.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f7c2401a3a0 con 0x7f7c2c155f20 2023-12-19T09:21:54.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 0x7f7c0c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:54.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f7c2407d130 con 0x7f7c2c155f20 2023-12-19T09:21:54.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c2a59c700 1 -- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 msgr2=0x7f7c0c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:54.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.703+0000 7f7c2a59c700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 0x7f7c0c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:54.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.711+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7c2404cf80 con 0x7f7c2c155f20 2023-12-19T09:21:54.732 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.728+0000 7fd13bbc9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T09:21:54.765 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.759+0000 7ff3bb15d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:54.828 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.824+0000 7fd13bbc9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T09:21:54.847 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:54.839+0000 7ff3bb15d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.875+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c2c071110 con 0x7f7c2c155f20 2023-12-19T09:21:54.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c1a7fc700 1 -- 172.21.15.111:0/2619106211 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16867 (secure 0 0 0) 0x7f7c240505a0 con 0x7f7c2c155f20 2023-12-19T09:21:54.885 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:54.885 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":209,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:38.103636+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/278966201":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/2861944468":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6833/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6832/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/150889021":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/800113432":"2023-12-20T09:21:38.103524+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:54.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 msgr2=0x7f7c0c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:54.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 0x7f7c0c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 msgr2=0x7f7c2c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c840 tx=0x7f7c24001670 comp rx=0 tx=0).stop 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 shutdown_connections 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7c0c060700 0x7f7c0c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c78e0 0x7f7c2c0c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c2c155f20 0x7f7c2c0a5290 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 --2- 172.21.15.111:0/2619106211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c068770 0x7f7c2c0c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 >> 172.21.15.111:0/2619106211 conn(0x7f7c2c0cc690 msgr2=0x7f7c2c0b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 shutdown_connections 2023-12-19T09:21:54.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:54.879+0000 7f7c308f0700 1 -- 172.21.15.111:0/2619106211 wait complete. 2023-12-19T09:21:54.906 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:21:54.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:54.907 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.904+0000 7fd13bbc9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T09:21:54.994 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:54.992+0000 7fd13bbc9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T09:21:55.068 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:55.063+0000 7ff3bb15d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:55.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 Processor -- start 2023-12-19T09:21:55.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 -- start start 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd8680ad980 0x7fd8680ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8680c78e0 0x7fd8680c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868067da0 con 0x7fd868155f20 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8680c8210 con 0x7fd8680c78e0 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86e425700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd868068940 con 0x7fd8680ad980 2023-12-19T09:21:55.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd867fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd868155f20 0x7fd868068230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56698/0 (socket says 172.21.15.111:56698) 2023-12-19T09:21:55.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd867fff700 1 -- 172.21.15.111:0/2832869006 learned_addr learned my addr 172.21.15.111:0/2832869006 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:55.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd8677fe700 1 --2- 172.21.15.111:0/2832869006 >> v2:172.21.15.143:3300/0 conn(0x7fd8680ad980 0x7fd8680ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.063+0000 7fd86c9c2700 1 --2- 172.21.15.111:0/2832869006 >> v2:172.21.15.111:3300/0 conn(0x7fd8680c78e0 0x7fd8680c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd866ffd700 1 -- 172.21.15.111:0/2832869006 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763268893 0 0) 0x7fd868067da0 con 0x7fd868155f20 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd866ffd700 1 -- 172.21.15.111:0/2832869006 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd850003570 con 0x7fd868155f20 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86c9c2700 1 -- 172.21.15.111:0/2832869006 >> v2:172.21.15.143:3300/0 conn(0x7fd8680ad980 msgr2=0x7fd8680ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86c9c2700 1 --2- 172.21.15.111:0/2832869006 >> v2:172.21.15.143:3300/0 conn(0x7fd8680ad980 0x7fd8680ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86c9c2700 1 -- 172.21.15.111:0/2832869006 >> v1:172.21.15.143:6789/0 conn(0x7fd868155f20 legacy=0x7fd868068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86c9c2700 1 -- 172.21.15.111:0/2832869006 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868067190 con 0x7fd8680c78e0 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86c9c2700 1 --2- 172.21.15.111:0/2832869006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8680c78e0 0x7fd8680c7cd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fd8680cbb20 tx=0x7fd860009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adef79e8d29e37f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd866ffd700 1 -- 172.21.15.111:0/2832869006 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000e070 con 0x7fd8680c78e0 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd866ffd700 1 -- 172.21.15.111:0/2832869006 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86000bab0 con 0x7fd8680c78e0 2023-12-19T09:21:55.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd866ffd700 1 -- 172.21.15.111:0/2832869006 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000be70 con 0x7fd8680c78e0 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 -- 172.21.15.111:0/2832869006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8680c78e0 msgr2=0x7fd8680c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 --2- 172.21.15.111:0/2832869006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8680c78e0 0x7fd8680c7cd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fd8680cbb20 tx=0x7fd860009780 comp rx=0 tx=0).stop 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 -- 172.21.15.111:0/2832869006 shutdown_connections 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 --2- 172.21.15.111:0/2832869006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8680c78e0 0x7fd8680c7cd0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 --2- 172.21.15.111:0/2832869006 >> v2:172.21.15.143:3300/0 conn(0x7fd8680ad980 0x7fd8680ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 -- 172.21.15.111:0/2832869006 >> 172.21.15.111:0/2832869006 conn(0x7fd8680cc690 msgr2=0x7fd8680c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:55.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 -- 172.21.15.111:0/2832869006 shutdown_connections 2023-12-19T09:21:55.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.067+0000 7fd86e425700 1 -- 172.21.15.111:0/2832869006 wait complete. 2023-12-19T09:21:55.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 Processor -- start 2023-12-19T09:21:55.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 -- start start 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8680c78e0 0x7fd868155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd868155f20 0x7fd8681580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8681556b0 con 0x7fd868155f20 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd868159250 con 0x7fd8680ad980 2023-12-19T09:21:55.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd86e425700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680c28b0 con 0x7fd8680c78e0 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd867fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.071+0000 7fd867fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:54994/0 (socket says 172.21.15.111:54994) 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 -- 172.21.15.111:0/4182631689 learned_addr learned my addr 172.21.15.111:0/4182631689 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd8677fe700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8680c78e0 0x7fd868155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd86c9c2700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd868155f20 0x7fd8681580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8680c78e0 msgr2=0x7fd868155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8680c78e0 0x7fd868155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd868155f20 msgr2=0x7fd8681580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd868155f20 0x7fd8681580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 -- 172.21.15.111:0/4182631689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860009a20 con 0x7fd8680ad980 2023-12-19T09:21:55.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd867fff700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c790 tx=0x7fd858007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858005b80 con 0x7fd8680ad980 2023-12-19T09:21:55.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858005ce0 con 0x7fd8680ad980 2023-12-19T09:21:55.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8680aeab0 con 0x7fd8680ad980 2023-12-19T09:21:55.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580072e0 con 0x7fd8680ad980 2023-12-19T09:21:55.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8680af070 con 0x7fd8680ad980 2023-12-19T09:21:55.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.075+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd868156a00 con 0x7fd8680ad980 2023-12-19T09:21:55.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.079+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fd858007690 con 0x7fd8680ad980 2023-12-19T09:21:55.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.079+0000 7fd84ffff700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 0x7fd854062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.079+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fd85807c380 con 0x7fd8680ad980 2023-12-19T09:21:55.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.079+0000 7fd8677fe700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 msgr2=0x7fd854062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:55.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.079+0000 7fd8677fe700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 0x7fd854062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.087+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd868156a00 con 0x7fd8680ad980 2023-12-19T09:21:55.151 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:55.143+0000 7ff3bb15d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:55.233 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:55.232+0000 7fd13bbc9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T09:21:55.240 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:55.235+0000 7ff3bb15d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:55.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.283+0000 7fd8677fe700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 msgr2=0x7fd854062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:55.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.283+0000 7fd8677fe700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 0x7fd854062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:55.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.291+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd868156a00 con 0x7fd8680ad980 2023-12-19T09:21:55.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.291+0000 7fd84ffff700 1 -- 172.21.15.111:0/4182631689 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7fd868156a00 con 0x7fd8680ad980 2023-12-19T09:21:55.298 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:55.298 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 msgr2=0x7fd854062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 0x7fd854062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 msgr2=0x7fd8680c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c790 tx=0x7fd858007e00 comp rx=0 tx=0).stop 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 shutdown_connections 2023-12-19T09:21:55.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd868155f20 0x7fd8681580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8680c78e0 0x7fd868155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fd854060700 0x7fd854062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 --2- 172.21.15.111:0/4182631689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8680ad980 0x7fd8680c21d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 >> 172.21.15.111:0/4182631689 conn(0x7fd8680cc690 msgr2=0x7fd8681562f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 shutdown_connections 2023-12-19T09:21:55.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.295+0000 7fd86e425700 1 -- 172.21.15.111:0/4182631689 wait complete. 2023-12-19T09:21:55.302 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:55.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:21:55.319 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:55.316+0000 7fd13bbc9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T09:21:55.413 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:55.412+0000 7fd13bbc9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T09:21:55.430 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:55.423+0000 7ff3bb15d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:55.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.467+0000 7f3f8f705700 1 Processor -- start 2023-12-19T09:21:55.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 -- start start 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f88067060 0x7f3f88067430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3f880c8e60 0x7f3f880ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8806b1a0 con 0x7f3f880c92b0 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f880ae350 con 0x7f3f880c8e60 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8f705700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f88067970 con 0x7f3f88067060 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3f880c8e60 0x7f3f880ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3f880c8e60 0x7f3f880ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50776/0 (socket says 172.21.15.111:50776) 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 -- 172.21.15.111:0/3824771988 learned_addr learned my addr 172.21.15.111:0/3824771988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:55.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8d4a1700 1 --2- 172.21.15.111:0/3824771988 >> v2:172.21.15.143:3300/0 conn(0x7f3f88067060 0x7f3f88067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 -- 172.21.15.111:0/3824771988 >> v2:172.21.15.143:3300/0 conn(0x7f3f88067060 msgr2=0x7f3f88067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:55.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 --2- 172.21.15.111:0/3824771988 >> v2:172.21.15.143:3300/0 conn(0x7f3f88067060 0x7f3f88067430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 -- 172.21.15.111:0/3824771988 >> v1:172.21.15.111:6789/0 conn(0x7f3f880c92b0 legacy=0x7f3f880abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 -- 172.21.15.111:0/3824771988 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8806a900 con 0x7f3f880c8e60 2023-12-19T09:21:55.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f8cca0700 1 --2- 172.21.15.111:0/3824771988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c8e60 0x7f3f880ab920 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3f88069500 tx=0x7f3f78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc34ea0a7b11d4c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.471+0000 7f3f7ffff700 1 -- 172.21.15.111:0/3824771988 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78009e70 con 0x7f3f880c8e60 2023-12-19T09:21:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f7ffff700 1 -- 172.21.15.111:0/3824771988 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7800b860 con 0x7f3f880c8e60 2023-12-19T09:21:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f7ffff700 1 -- 172.21.15.111:0/3824771988 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800bc00 con 0x7f3f880c8e60 2023-12-19T09:21:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 -- 172.21.15.111:0/3824771988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c8e60 msgr2=0x7f3f880ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 --2- 172.21.15.111:0/3824771988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c8e60 0x7f3f880ab920 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3f88069500 tx=0x7f3f78009250 comp rx=0 tx=0).stop 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 -- 172.21.15.111:0/3824771988 shutdown_connections 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 --2- 172.21.15.111:0/3824771988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c8e60 0x7f3f880ab920 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 --2- 172.21.15.111:0/3824771988 >> v2:172.21.15.143:3300/0 conn(0x7f3f88067060 0x7f3f88067430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 -- 172.21.15.111:0/3824771988 >> 172.21.15.111:0/3824771988 conn(0x7f3f880cc690 msgr2=0x7f3f880c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 -- 172.21.15.111:0/3824771988 shutdown_connections 2023-12-19T09:21:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 -- 172.21.15.111:0/3824771988 wait complete. 2023-12-19T09:21:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.475+0000 7f3f8f705700 1 Processor -- start 2023-12-19T09:21:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- start start 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 0x7f3f880cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f880c8e60 0x7f3f881696d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880ca360 con 0x7f3f880c92b0 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880b0300 con 0x7f3f88067060 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880af6f0 con 0x7f3f880c8e60 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50784/0 (socket says 172.21.15.111:50784) 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 -- 172.21.15.111:0/104070922 learned_addr learned my addr 172.21.15.111:0/104070922 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8d4a1700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 0x7f3f880cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8cca0700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f880c8e60 0x7f3f881696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f880c8e60 msgr2=0x7f3f881696d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f880c8e60 0x7f3f881696d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 msgr2=0x7f3f880cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 0x7f3f880cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 -- 172.21.15.111:0/104070922 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780094f0 con 0x7f3f880c92b0 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8d4a1700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 0x7f3f880cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8dca2700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c840 tx=0x7f3f80001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80012630 con 0x7f3f880c92b0 2023-12-19T09:21:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f800127b0 con 0x7f3f880c92b0 2023-12-19T09:21:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80012b70 con 0x7f3f880c92b0 2023-12-19T09:21:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f880af9d0 con 0x7f3f880c92b0 2023-12-19T09:21:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.479+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f880aff90 con 0x7f3f880c92b0 2023-12-19T09:21:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.483+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f880b0490 con 0x7f3f880c92b0 2023-12-19T09:21:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.483+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f3f80003030 con 0x7f3f880c92b0 2023-12-19T09:21:55.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.483+0000 7f3f7d7fa700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 0x7f3f6c062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.483+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f3f8007cf40 con 0x7f3f880c92b0 2023-12-19T09:21:55.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.487+0000 7f3f8d4a1700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 msgr2=0x7f3f6c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:55.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.487+0000 7f3f8d4a1700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 0x7f3f6c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:55.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.491+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3f8004ce30 con 0x7f3f880c92b0 2023-12-19T09:21:55.511 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:55.503+0000 7ff3bb15d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:55.617 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:55.616+0000 7fd13bbc9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T09:21:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.691+0000 7f3f8d4a1700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 msgr2=0x7f3f6c062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:55.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.691+0000 7f3f8d4a1700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 0x7f3f6c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:55.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.691+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f88071110 con 0x7f3f880c92b0 2023-12-19T09:21:55.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.691+0000 7f3f7d7fa700 1 -- 172.21.15.111:0/104070922 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4612 (secure 0 0 0) 0x7f3f80050450 con 0x7f3f880c92b0 2023-12-19T09:21:55.699 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:55.700 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T09:21:22.352171+0000","modified":"2023-12-19T09:21:25.953156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10120},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10120":{"gid":10120,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.143:6837/68388372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":68388372},{"type":"v1","addr":"172.21.15.143:6837","nonce":68388372}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2023-12-19T09:21:55.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 msgr2=0x7f3f6c062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:55.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 0x7f3f6c062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 msgr2=0x7f3f88169c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c840 tx=0x7f3f80001670 comp rx=0 tx=0).stop 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 shutdown_connections 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f880c92b0 0x7f3f88169c10 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f880c8e60 0x7f3f881696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3f6c060700 0x7f3f6c062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 --2- 172.21.15.111:0/104070922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f88067060 0x7f3f880cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 >> 172.21.15.111:0/104070922 conn(0x7f3f880cc690 msgr2=0x7f3f88154c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 shutdown_connections 2023-12-19T09:21:55.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.695+0000 7f3f8f705700 1 -- 172.21.15.111:0/104070922 wait complete. 2023-12-19T09:21:55.704 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 95 2023-12-19T09:21:55.707 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:55.704+0000 7fd13bbc9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T09:21:55.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:21:55.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.875+0000 7f3c390e3700 1 Processor -- start 2023-12-19T09:21:55.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.875+0000 7f3c390e3700 1 -- start start 2023-12-19T09:21:55.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.875+0000 7f3c390e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c340b0cb0 0x7f3c3406a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.875+0000 7f3c390e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340abd00 con 0x7f3c340b1080 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340abfe0 con 0x7f3c34069f50 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c340abb10 con 0x7f3c340b0cb0 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c337fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c340b1080 0x7f3c34069840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44400/0 (socket says 172.21.15.111:44400) 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c337fe700 1 -- 172.21.15.111:0/2307112471 learned_addr learned my addr 172.21.15.111:0/2307112471 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:55.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c33fff700 1 --2- 172.21.15.111:0/2307112471 >> v2:172.21.15.143:3300/0 conn(0x7f3c340b0cb0 0x7f3c3406a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842780165 0 0) 0x7f3c340abd00 con 0x7f3c340b1080 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c003570 con 0x7f3c340b1080 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c33fff700 1 -- 172.21.15.111:0/2307112471 >> v1:172.21.15.143:6789/0 conn(0x7f3c34069f50 legacy=0x7f3c340673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c33fff700 1 -- 172.21.15.111:0/2307112471 >> v1:172.21.15.111:6789/0 conn(0x7f3c340b1080 legacy=0x7f3c34069840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338892449 0 0) 0x7f3c1c003570 con 0x7f3c340b1080 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c33fff700 1 -- 172.21.15.111:0/2307112471 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340ac2c0 con 0x7f3c340b0cb0 2023-12-19T09:21:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c33fff700 1 --2- 172.21.15.111:0/2307112471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c340b0cb0 0x7f3c3406a730 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3c240098e0 tx=0x7f3c240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e174303b2047db73 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28001470 con 0x7f3c340b0cb0 2023-12-19T09:21:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c24011420 con 0x7f3c340b0cb0 2023-12-19T09:21:55.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c32ffd700 1 -- 172.21.15.111:0/2307112471 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240117c0 con 0x7f3c340b0cb0 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- 172.21.15.111:0/2307112471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c340b0cb0 msgr2=0x7f3c3406a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 --2- 172.21.15.111:0/2307112471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c340b0cb0 0x7f3c3406a730 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3c240098e0 tx=0x7f3c240091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- 172.21.15.111:0/2307112471 shutdown_connections 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 --2- 172.21.15.111:0/2307112471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c340b0cb0 0x7f3c3406a730 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- 172.21.15.111:0/2307112471 >> 172.21.15.111:0/2307112471 conn(0x7f3c340cc530 msgr2=0x7f3c340c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- 172.21.15.111:0/2307112471 shutdown_connections 2023-12-19T09:21:55.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.879+0000 7f3c390e3700 1 -- 172.21.15.111:0/2307112471 wait complete. 2023-12-19T09:21:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 Processor -- start 2023-12-19T09:21:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 -- start start 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 0x7f3c340c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c340b1080 0x7f3c340bf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340c83a0 con 0x7f3c340b0cb0 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340c74f0 con 0x7f3c34069f50 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c390e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340c7790 con 0x7f3c340b1080 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50800/0 (socket says 172.21.15.111:50800) 2023-12-19T09:21:55.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c337fe700 1 -- 172.21.15.111:0/3652648498 learned_addr learned my addr 172.21.15.111:0/3652648498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.883+0000 7f3c33fff700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 0x7f3c340c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c388e2700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c340b1080 0x7f3c340bf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 -- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c340b1080 msgr2=0x7f3c340bf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c340b1080 0x7f3c340bf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 -- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 msgr2=0x7f3c340c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 0x7f3c340c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 -- 172.21.15.111:0/3652648498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c24009420 con 0x7f3c340b0cb0 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c33fff700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 0x7f3c340c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c337fe700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c930 tx=0x7f3c28001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28005b30 con 0x7f3c340b0cb0 2023-12-19T09:21:55.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c28005c90 con 0x7f3c340b0cb0 2023-12-19T09:21:55.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28007340 con 0x7f3c340b0cb0 2023-12-19T09:21:55.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c340c7a70 con 0x7f3c340b0cb0 2023-12-19T09:21:55.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c340c8030 con 0x7f3c340b0cb0 2023-12-19T09:21:55.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.887+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c340c8550 con 0x7f3c340b0cb0 2023-12-19T09:21:55.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.891+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7f3c28012400 con 0x7f3c340b0cb0 2023-12-19T09:21:55.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.891+0000 7f3c1bfff700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 0x7f3c20062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:55.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.891+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f3c2807c4b0 con 0x7f3c340b0cb0 2023-12-19T09:21:55.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.891+0000 7f3c33fff700 1 -- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 msgr2=0x7f3c20062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:55.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.891+0000 7f3c33fff700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 0x7f3c20062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:55.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:55.895+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c28049be0 con 0x7f3c340b0cb0 2023-12-19T09:21:56.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.063+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c3404be40 con 0x7f3c340b0cb0 2023-12-19T09:21:56.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c1bfff700 1 -- 172.21.15.111:0/3652648498 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16867 (secure 0 0 0) 0x7f3c28017090 con 0x7f3c340b0cb0 2023-12-19T09:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":209,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:38.103636+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-12-19T09:21:20.323729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-12-19T09:21:21.351082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:0/278966201":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/2861944468":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6833/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6832/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/150889021":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/800113432":"2023-12-20T09:21:38.103524+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 msgr2=0x7f3c20062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 0x7f3c20062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 msgr2=0x7f3c340c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800c930 tx=0x7f3c28001970 comp rx=0 tx=0).stop 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 shutdown_connections 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c340b1080 0x7f3c340bf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340b0cb0 0x7f3c340c6fb0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3c20060700 0x7f3c20062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 --2- 172.21.15.111:0/3652648498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c34069f50 0x7f3c340c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 >> 172.21.15.111:0/3652648498 conn(0x7f3c340cc530 msgr2=0x7f3c34067c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:56.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 shutdown_connections 2023-12-19T09:21:56.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.067+0000 7f3c390e3700 1 -- 172.21.15.111:0/3652648498 wait complete. 2023-12-19T09:21:56.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-19T09:21:56.244 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-12-19T09:21:56.235+0000 7ff3bb15d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:56.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 Processor -- start 2023-12-19T09:21:56.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 -- start start 2023-12-19T09:21:56.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa770158eb0 0x7fa7700ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700a8cf0 con 0x7fa7700a88a0 2023-12-19T09:21:56.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77006c7e0 con 0x7fa770158eb0 2023-12-19T09:21:56.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa774a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa770067190 con 0x7fa7700a9960 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa770158eb0 0x7fa7700ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76f59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa770158eb0 0x7fa7700ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.111:55054/0 (socket says 172.21.15.111:55054) 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 -- 172.21.15.111:0/246270470 learned_addr learned my addr 172.21.15.111:0/246270470 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 -- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 msgr2=0x7fa770068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 -- 172.21.15.111:0/246270470 >> v1:172.21.15.143:6789/0 conn(0x7fa7700a88a0 legacy=0x7fa7700a9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.247+0000 7fa76ed9d700 1 -- 172.21.15.111:0/246270470 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa770067470 con 0x7fa770158eb0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa770158eb0 0x7fa7700ab260 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa760009990 tx=0x7fa760009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c96d0c0d06d768b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa76dd9b700 1 -- 172.21.15.111:0/246270470 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760009e80 con 0x7fa770158eb0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa76dd9b700 1 -- 172.21.15.111:0/246270470 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760005c10 con 0x7fa770158eb0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa76dd9b700 1 -- 172.21.15.111:0/246270470 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000b650 con 0x7fa770158eb0 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa770158eb0 msgr2=0x7fa7700ab260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 --2- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa770158eb0 0x7fa7700ab260 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa760009990 tx=0x7fa760009220 comp rx=0 tx=0).stop 2023-12-19T09:21:56.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- 172.21.15.111:0/246270470 shutdown_connections 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 --2- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 --2- 172.21.15.111:0/246270470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa770158eb0 0x7fa7700ab260 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- 172.21.15.111:0/246270470 >> 172.21.15.111:0/246270470 conn(0x7fa77006e160 msgr2=0x7fa7700adcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- 172.21.15.111:0/246270470 shutdown_connections 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- 172.21.15.111:0/246270470 wait complete. 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 Processor -- start 2023-12-19T09:21:56.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 -- start start 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 0x7fa770146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.251+0000 7fa774a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 0x7fa770157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700c1c90 con 0x7fa770158eb0 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700c1080 con 0x7fa7700a88a0 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7700c1360 con 0x7fa7700a9960 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 0x7fa770157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 0x7fa770157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50802/0 (socket says 172.21.15.111:50802) 2023-12-19T09:21:56.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76f59e700 1 -- 172.21.15.111:0/2835240253 learned_addr learned my addr 172.21.15.111:0/2835240253 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 0x7fa770146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76e59c700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 msgr2=0x7fa770156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 msgr2=0x7fa770157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 0x7fa770157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 -- 172.21.15.111:0/2835240253 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760009480 con 0x7fa7700a88a0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa76ed9d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 0x7fa770146be0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa760015000 tx=0x7fa760016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011950 con 0x7fa7700a88a0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760020070 con 0x7fa7700a88a0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760011ce0 con 0x7fa7700a88a0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7700c1640 con 0x7fa7700a88a0 2023-12-19T09:21:56.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa770147120 con 0x7fa7700a88a0 2023-12-19T09:21:56.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.255+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa770147340 con 0x7fa7700a88a0 2023-12-19T09:21:56.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.259+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 106) v1 ==== 80120+0+0 (secure 0 0 0) 0x7fa760022030 con 0x7fa7700a88a0 2023-12-19T09:21:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.259+0000 7fa756ffd700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 0x7fa75c062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.259+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6859+0+0 (secure 0 0 0) 0x7fa76007e900 con 0x7fa7700a88a0 2023-12-19T09:21:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.259+0000 7fa76e59c700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 msgr2=0x7fa75c062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:56.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.259+0000 7fa76e59c700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 0x7fa75c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T09:21:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.267+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa770147340 con 0x7fa7700a88a0 2023-12-19T09:21:56.459 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.456+0000 7fd13bbc9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T09:21:56.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.463+0000 7fa76e59c700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 msgr2=0x7fa75c062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6832/43868 2023-12-19T09:21:56.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.463+0000 7fa76e59c700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 0x7fa75c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T09:21:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.475+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 107) v1 ==== 79672+0+0 (secure 0 0 0) 0x7fa76004eb20 con 0x7fa7700a88a0 2023-12-19T09:21:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.475+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 msgr2=0x7fa75c062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T09:21:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.475+0000 7fa756ffd700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 0x7fa75c062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.479+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa770147340 con 0x7fa7700a88a0 2023-12-19T09:21:56.485 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-12-19T09:21:56.479+0000 7f903a5a3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T09:21:56.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.487+0000 7fa756ffd700 1 -- 172.21.15.111:0/2835240253 <== mon.1 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa770147340 con 0x7fa7700a88a0 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 msgr2=0x7fa770146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 0x7fa770146be0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa760015000 tx=0x7fa760016000 comp rx=0 tx=0).stop 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 shutdown_connections 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa770158eb0 0x7fa770157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7700a9960 0x7fa770156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fa75c060730 0x7fa75c062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 --2- 172.21.15.111:0/2835240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7700a88a0 0x7fa770146be0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 >> 172.21.15.111:0/2835240253 conn(0x7fa77006e160 msgr2=0x7fa770156010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:56.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 shutdown_connections 2023-12-19T09:21:56.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.491+0000 7fa774a2d700 1 -- 172.21.15.111:0/2835240253 wait complete. 2023-12-19T09:21:56.499 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-19T09:21:56.501 INFO:tasks.ceph.mds.b.smithi143.stderr: -3612> 2023-12-19T09:21:27.868+0000 7f53d958c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-19T09:21:56.516 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-19T09:21:56.559 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-12-19T09:21:56.568 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd102f99700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd102f99700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd102f99700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd102f99700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd102f99700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd105f9f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd105f9f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd105f9f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd105f9f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.569 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-12-19T09:21:56.564+0000 7fd105f9f700 -1 client.0 error registering admin socket command: (17) File exists 2023-12-19T09:21:56.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 Processor -- start 2023-12-19T09:21:56.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 -- start start 2023-12-19T09:21:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4067660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ed40aa890 0x7f5ed40c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40abb50 con 0x7f5ed4067ba0 2023-12-19T09:21:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed406a550 con 0x7f5ed40aa890 2023-12-19T09:21:56.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed9278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40ab440 con 0x7f5ed4067290 2023-12-19T09:21:56.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ed40aa890 0x7f5ed40c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ecbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5ed4067ba0 0x7f5ed40c6590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:56728/0 (socket says 172.21.15.111:56728) 2023-12-19T09:21:56.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ecbfff700 1 -- 172.21.15.111:0/2895540400 learned_addr learned my addr 172.21.15.111:0/2895540400 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:56.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed2ffd700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4067660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.679+0000 7f5ed27fc700 1 -- 172.21.15.111:0/2895540400 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254846401 0 0) 0x7f5ed40abb50 con 0x7f5ed4067ba0 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed27fc700 1 -- 172.21.15.111:0/2895540400 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec0003570 con 0x7f5ed4067ba0 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed37fe700 1 -- 172.21.15.111:0/2895540400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 msgr2=0x7f5ed4067660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed37fe700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4067660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed37fe700 1 -- 172.21.15.111:0/2895540400 >> v1:172.21.15.143:6789/0 conn(0x7f5ed4067ba0 legacy=0x7f5ed40c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed37fe700 1 -- 172.21.15.111:0/2895540400 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed406a750 con 0x7f5ed40aa890 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed2ffd700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4067660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed37fe700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40c6ca0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009e60 tx=0x7f5ecc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdbebe5af48e6720 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed27fc700 1 -- 172.21.15.111:0/2895540400 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00e070 con 0x7f5ed40aa890 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed27fc700 1 -- 172.21.15.111:0/2895540400 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc00bab0 con 0x7f5ed40aa890 2023-12-19T09:21:56.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed27fc700 1 -- 172.21.15.111:0/2895540400 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00be70 con 0x7f5ed40aa890 2023-12-19T09:21:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 -- 172.21.15.111:0/2895540400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 msgr2=0x7f5ed40c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40c6ca0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009e60 tx=0x7f5ecc009780 comp rx=0 tx=0).stop 2023-12-19T09:21:56.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 -- 172.21.15.111:0/2895540400 shutdown_connections 2023-12-19T09:21:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40c6ca0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 --2- 172.21.15.111:0/2895540400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4067660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 -- 172.21.15.111:0/2895540400 >> 172.21.15.111:0/2895540400 conn(0x7f5ed40cc680 msgr2=0x7f5ed40c8620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 -- 172.21.15.111:0/2895540400 shutdown_connections 2023-12-19T09:21:56.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.683+0000 7f5ed9278700 1 -- 172.21.15.111:0/2895540400 wait complete. 2023-12-19T09:21:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 Processor -- start 2023-12-19T09:21:56.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 -- start start 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ed4067ba0 0x7f5ed40c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40c5a60 con 0x7f5ed40aa890 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40c4e50 con 0x7f5ed4067ba0 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed9278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40c5130 con 0x7f5ed4067290 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50818/0 (socket says 172.21.15.111:50818) 2023-12-19T09:21:56.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 -- 172.21.15.111:0/3444390741 learned_addr learned my addr 172.21.15.111:0/3444390741 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:56.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed2ffd700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ecbfff700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ed4067ba0 0x7f5ed40c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 -- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 msgr2=0x7f5ed4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 -- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ed4067ba0 msgr2=0x7f5ed40c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:56.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ed4067ba0 0x7f5ed40c3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.687+0000 7f5ed37fe700 1 -- 172.21.15.111:0/3444390741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec4009420 con 0x7f5ed40aa890 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ed37fe700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f5ed406bad0 tx=0x7f5ecc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc027470 con 0x7f5ed40aa890 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc0275f0 con 0x7f5ed40aa890 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc009a20 con 0x7f5ed40aa890 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc0279b0 con 0x7f5ed40aa890 2023-12-19T09:21:56.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed40c56f0 con 0x7f5ed40aa890 2023-12-19T09:21:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed4071110 con 0x7f5ed40aa890 2023-12-19T09:21:56.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.691+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 107) v1 ==== 79672+0+0 (secure 0 0 0) 0x7f5ecc024070 con 0x7f5ed40aa890 2023-12-19T09:21:56.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.695+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 7246+0+0 (secure 0 0 0) 0x7f5ecc07f2c0 con 0x7f5ed40aa890 2023-12-19T09:21:56.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.699+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5ecc04f580 con 0x7f5ed40aa890 2023-12-19T09:21:56.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:56.907+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5ed406aa30 con 0x7f5ed40aa890 2023-12-19T09:21:57.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.491+0000 7f5ecaffd700 1 -- 172.21.15.111:0/3444390741 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ecc025e90 con 0x7f5ed40aa890 2023-12-19T09:21:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 msgr2=0x7f5ed40ba970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f5ed406bad0 tx=0x7f5ecc019000 comp rx=0 tx=0).stop 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 shutdown_connections 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ed4067ba0 0x7f5ed40c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ed40aa890 0x7f5ed40ba970 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 --2- 172.21.15.111:0/3444390741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ed4067290 0x7f5ed4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 >> 172.21.15.111:0/3444390741 conn(0x7f5ed40cc680 msgr2=0x7f5ed40ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 shutdown_connections 2023-12-19T09:21:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.495+0000 7f5ed9278700 1 -- 172.21.15.111:0/3444390741 wait complete. 2023-12-19T09:21:57.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-19T09:21:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.687+0000 7f53a842e700 1 Processor -- start 2023-12-19T09:21:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.687+0000 7f53a842e700 1 -- start start 2023-12-19T09:21:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a842e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53a015b260 0x7f53a00abef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a842e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a0068a20 con 0x7f53a00ae270 2023-12-19T09:21:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a842e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a0069d70 con 0x7f53a00abb20 2023-12-19T09:21:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a842e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a006a8f0 con 0x7f53a015b260 2023-12-19T09:21:57.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a69cb700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f53a00ae270 0x7f53a014d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44406/0 (socket says 172.21.15.111:44406) 2023-12-19T09:21:57.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a69cb700 1 -- 172.21.15.111:0/3295456812 learned_addr learned my addr 172.21.15.111:0/3295456812 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:57.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a59c9700 1 --2- 172.21.15.111:0/3295456812 >> v2:172.21.15.143:3300/0 conn(0x7f53a015b260 0x7f53a00abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880708240 0 0) 0x7f53a0068a20 con 0x7f53a00ae270 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c003570 con 0x7f53a00ae270 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213673643 0 0) 0x7f53a0069d70 con 0x7f53a00abb20 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a0068a20 con 0x7f53a00abb20 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a59c9700 1 -- 172.21.15.111:0/3295456812 >> v1:172.21.15.143:6789/0 conn(0x7f53a00abb20 legacy=0x7f53a015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a59c9700 1 -- 172.21.15.111:0/3295456812 >> v1:172.21.15.111:6789/0 conn(0x7f53a00ae270 legacy=0x7f53a014d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a59c9700 1 -- 172.21.15.111:0/3295456812 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0068070 con 0x7f53a015b260 2023-12-19T09:21:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.691+0000 7f53a59c9700 1 --2- 172.21.15.111:0/3295456812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a015b260 0x7f53a00abef0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f53900098e0 tx=0x7f53900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f57fdf526641934e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390011630 con 0x7f53a015b260 2023-12-19T09:21:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5390011790 con 0x7f53a015b260 2023-12-19T09:21:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a51c8700 1 -- 172.21.15.111:0/3295456812 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390011b30 con 0x7f53a015b260 2023-12-19T09:21:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 -- 172.21.15.111:0/3295456812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a015b260 msgr2=0x7f53a00abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 --2- 172.21.15.111:0/3295456812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a015b260 0x7f53a00abef0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f53900098e0 tx=0x7f53900091a0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 -- 172.21.15.111:0/3295456812 shutdown_connections 2023-12-19T09:21:57.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 --2- 172.21.15.111:0/3295456812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a015b260 0x7f53a00abef0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 -- 172.21.15.111:0/3295456812 >> 172.21.15.111:0/3295456812 conn(0x7f53a00cc7e0 msgr2=0x7f53a00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:57.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 -- 172.21.15.111:0/3295456812 shutdown_connections 2023-12-19T09:21:57.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.695+0000 7f53a842e700 1 -- 172.21.15.111:0/3295456812 wait complete. 2023-12-19T09:21:57.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 Processor -- start 2023-12-19T09:21:57.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 -- start start 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53a00abb20 0x7f53a0149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 0x7f53a00c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a01497c0 con 0x7f53a015b260 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00c0500 con 0x7f53a00ae270 2023-12-19T09:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a00c3390 con 0x7f53a00abb20 2023-12-19T09:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50828/0 (socket says 172.21.15.111:50828) 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 -- 172.21.15.111:0/4175268874 learned_addr learned my addr 172.21.15.111:0/4175268874 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a59c9700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 0x7f53a00c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a61ca700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53a00abb20 0x7f53a0149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 -- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53a00abb20 msgr2=0x7f53a0149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53a00abb20 0x7f53a0149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 -- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 msgr2=0x7f53a00c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 0x7f53a00c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 -- 172.21.15.111:0/4175268874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390009420 con 0x7f53a015b260 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a59c9700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 0x7f53a00c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:21:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a69cb700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f539800cc40 tx=0x7f5398001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539800a960 con 0x7f53a015b260 2023-12-19T09:21:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539800aae0 con 0x7f53a015b260 2023-12-19T09:21:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539800aea0 con 0x7f53a015b260 2023-12-19T09:21:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.699+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a00bf8f0 con 0x7f53a015b260 2023-12-19T09:21:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.703+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a00bfeb0 con 0x7f53a015b260 2023-12-19T09:21:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.703+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a00c0690 con 0x7f53a015b260 2023-12-19T09:21:57.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.703+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 108) v1 ==== 79799+0+0 (secure 0 0 0) 0x7f5398007300 con 0x7f53a015b260 2023-12-19T09:21:57.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.707+0000 7f53967fc700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 0x7f5384062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:57.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.707+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 7246+0+0 (secure 0 0 0) 0x7f539807cef0 con 0x7f53a015b260 2023-12-19T09:21:57.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.707+0000 7f53a61ca700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 0x7f5384062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:57.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.707+0000 7f53a61ca700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 0x7f5384062880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f539c0037e0 tx=0x7f539c009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:57.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.711+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f539804d140 con 0x7f53a015b260 2023-12-19T09:21:57.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:57.883+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f53a00708a0 con 0x7f53a015b260 2023-12-19T09:21:57.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:21:58.000 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:21:58.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.803+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f5398056080 con 0x7f53a015b260 2023-12-19T09:21:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.807+0000 7f53967fc700 1 -- 172.21.15.111:0/4175268874 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v212) v1 ==== 181+0+0 (secure 0 0 0) 0x7f539804d140 con 0x7f53a015b260 2023-12-19T09:21:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 msgr2=0x7f5384062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 0x7f5384062880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f539c0037e0 tx=0x7f539c009040 comp rx=0 tx=0).stop 2023-12-19T09:21:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 msgr2=0x7f53a00c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f539800cc40 tx=0x7f5398001a40 comp rx=0 tx=0).stop 2023-12-19T09:21:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 shutdown_connections 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f53840603d0 0x7f5384062880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53a00ae270 0x7f53a00c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53a015b260 0x7f53a00c9db0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 --2- 172.21.15.111:0/4175268874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53a00abb20 0x7f53a0149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 >> 172.21.15.111:0/4175268874 conn(0x7f53a00cc7e0 msgr2=0x7f53a0157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 shutdown_connections 2023-12-19T09:21:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:58.811+0000 7f53a842e700 1 -- 172.21.15.111:0/4175268874 wait complete. 2023-12-19T09:21:58.821 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-12-19T09:21:58.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-19T09:21:59.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.003+0000 7efc3d345700 1 Processor -- start 2023-12-19T09:21:59.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.003+0000 7efc3d345700 1 -- start start 2023-12-19T09:21:59.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc3d345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:59.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc3d345700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc380c7080 con 0x7efc38068660 2023-12-19T09:21:59.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc3d345700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc380c84c0 con 0x7efc380c64e0 2023-12-19T09:21:59.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc3d345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc380acf00 con 0x7efc380c68b0 2023-12-19T09:21:59.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc2ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc38068660 0x7efc3806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44410/0 (socket says 172.21.15.111:44410) 2023-12-19T09:21:59.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc2ffff700 1 -- 172.21.15.111:0/318272550 learned_addr learned my addr 172.21.15.111:0/318272550 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:21:59.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc377fe700 1 --2- 172.21.15.111:0/318272550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:59.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460559521 0 0) 0x7efc380c7080 con 0x7efc38068660 2023-12-19T09:21:59.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc24003570 con 0x7efc38068660 2023-12-19T09:21:59.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546766178 0 0) 0x7efc380c84c0 con 0x7efc380c64e0 2023-12-19T09:21:59.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc380c7080 con 0x7efc380c64e0 2023-12-19T09:21:59.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186810194 0 0) 0x7efc24003570 con 0x7efc38068660 2023-12-19T09:21:59.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc377fe700 1 -- 172.21.15.111:0/318272550 >> v1:172.21.15.143:6789/0 conn(0x7efc380c64e0 legacy=0x7efc38069b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc377fe700 1 -- 172.21.15.111:0/318272550 >> v1:172.21.15.111:6789/0 conn(0x7efc38068660 legacy=0x7efc3806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.007+0000 7efc377fe700 1 -- 172.21.15.111:0/318272550 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc380ad1e0 con 0x7efc380c68b0 2023-12-19T09:21:59.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc377fe700 1 --2- 172.21.15.111:0/318272550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380c7e90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efc30009e90 tx=0x7efc30009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4b7fe59845aaca5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:59.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3000e070 con 0x7efc380c68b0 2023-12-19T09:21:59.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3000b870 con 0x7efc380c68b0 2023-12-19T09:21:59.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc367fc700 1 -- 172.21.15.111:0/318272550 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc200013f0 con 0x7efc380c68b0 2023-12-19T09:21:59.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 -- 172.21.15.111:0/318272550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 msgr2=0x7efc380c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 --2- 172.21.15.111:0/318272550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380c7e90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efc30009e90 tx=0x7efc30009780 comp rx=0 tx=0).stop 2023-12-19T09:21:59.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 -- 172.21.15.111:0/318272550 shutdown_connections 2023-12-19T09:21:59.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 --2- 172.21.15.111:0/318272550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380c7e90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 -- 172.21.15.111:0/318272550 >> 172.21.15.111:0/318272550 conn(0x7efc380cc7e0 msgr2=0x7efc380c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:59.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 -- 172.21.15.111:0/318272550 shutdown_connections 2023-12-19T09:21:59.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.011+0000 7efc3d345700 1 -- 172.21.15.111:0/318272550 wait complete. 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 Processor -- start 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 -- start start 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc38068660 0x7efc3814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc380b0d70 con 0x7efc380c64e0 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc380b0160 con 0x7efc38068660 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc3d345700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc380b0440 con 0x7efc380c68b0 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50842/0 (socket says 172.21.15.111:50842) 2023-12-19T09:21:59.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 -- 172.21.15.111:0/4006593825 learned_addr learned my addr 172.21.15.111:0/4006593825 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc36ffd700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc38068660 0x7efc3814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc377fe700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 -- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 msgr2=0x7efc380b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 -- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc38068660 msgr2=0x7efc3814a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.015+0000 7efc2ffff700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc38068660 0x7efc3814a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc2ffff700 1 -- 172.21.15.111:0/4006593825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009a20 con 0x7efc380c64e0 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc2ffff700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7efc2000c7e0 tx=0x7efc20001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20007360 con 0x7efc380c64e0 2023-12-19T09:21:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc200074e0 con 0x7efc380c64e0 2023-12-19T09:21:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc20004830 con 0x7efc380c64e0 2023-12-19T09:21:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc380b0720 con 0x7efc380c64e0 2023-12-19T09:21:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3814a620 con 0x7efc380c64e0 2023-12-19T09:21:59.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.019+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc380708a0 con 0x7efc380c64e0 2023-12-19T09:21:59.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.023+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7efc20004bc0 con 0x7efc380c64e0 2023-12-19T09:21:59.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.023+0000 7efc2effd700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 0x7efc18062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:21:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.023+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6808+0+0 (secure 0 0 0) 0x7efc2007cb00 con 0x7efc380c64e0 2023-12-19T09:21:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.023+0000 7efc36ffd700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 0x7efc18062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:21:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.023+0000 7efc36ffd700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 0x7efc18062a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc280010e0 tx=0x7efc28007de0 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:21:59.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.027+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efc2004cc80 con 0x7efc380c64e0 2023-12-19T09:21:59.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.199+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc380c5dd0 con 0x7efc380c64e0 2023-12-19T09:21:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.811+0000 7efc2effd700 1 -- 172.21.15.111:0/4006593825 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v213) v1 ==== 169+0+0 (secure 0 0 0) 0x7efc200502a0 con 0x7efc380c64e0 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 msgr2=0x7efc18062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 0x7efc18062a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc280010e0 tx=0x7efc28007de0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 msgr2=0x7efc380bf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7efc2000c7e0 tx=0x7efc20001990 comp rx=0 tx=0).stop 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 shutdown_connections 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc380c68b0 0x7efc380b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc380c64e0 0x7efc380bf750 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7efc18060550 0x7efc18062a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 --2- 172.21.15.111:0/4006593825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc38068660 0x7efc3814a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:21:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 >> 172.21.15.111:0/4006593825 conn(0x7efc380cc7e0 msgr2=0x7efc380c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:21:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 shutdown_connections 2023-12-19T09:21:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:21:59.815+0000 7efc3d345700 1 -- 172.21.15.111:0/4006593825 wait complete. 2023-12-19T09:21:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-12-19T09:21:59.843 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-12-19T09:22:00.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.007+0000 7fca7e323700 1 Processor -- start 2023-12-19T09:22:00.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7e323700 1 -- start start 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7e323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca7815d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7e323700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78068120 con 0x7fca780b23d0 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7e323700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780caf70 con 0x7fca7815bca0 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7e323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7815d550 con 0x7fca7815cc40 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7cb20700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fca780b23d0 0x7fca78067a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44426/0 (socket says 172.21.15.111:44426) 2023-12-19T09:22:00.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7cb20700 1 -- 172.21.15.111:0/725821235 learned_addr learned my addr 172.21.15.111:0/725821235 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca7d321700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca7815d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269238022 0 0) 0x7fca78068120 con 0x7fca780b23d0 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003570 con 0x7fca780b23d0 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2532175628 0 0) 0x7fca64003570 con 0x7fca780b23d0 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 msgr2=0x7fca7815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca7815d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 >> v1:172.21.15.143:6789/0 conn(0x7fca7815bca0 legacy=0x7fca7815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78068350 con 0x7fca780b23d0 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.011+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7fca68001470 con 0x7fca780b23d0 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 >> v1:172.21.15.111:6789/0 conn(0x7fca780b23d0 legacy=0x7fca78067a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca64003a90 0x7fca64003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7815d550 con 0x7fca64004810 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6400bb40 con 0x7fca64003a90 2023-12-19T09:22:00.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6400bd00 con 0x7fca7815cc40 2023-12-19T09:22:00.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7d321700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca64003a90 0x7fca64003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7db22700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 -- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 msgr2=0x7fca640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca7815cc40 0x7fca640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 -- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca64003a90 msgr2=0x7fca64003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca64003a90 0x7fca64003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 -- 172.21.15.111:0/725821235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64003570 con 0x7fca64004810 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7cb20700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca64004c80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c2a0 tx=0x7fca68001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeead944bd23c3d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800cb70 con 0x7fca64004810 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6800ccd0 con 0x7fca64004810 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca6ffff700 1 -- 172.21.15.111:0/725821235 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68005b70 con 0x7fca64004810 2023-12-19T09:22:00.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 -- 172.21.15.111:0/725821235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 msgr2=0x7fca64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 --2- 172.21.15.111:0/725821235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca64004c80 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c2a0 tx=0x7fca68001da0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7db22700 1 -- 172.21.15.111:0/725821235 reap_dead start 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 -- 172.21.15.111:0/725821235 shutdown_connections 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 -- 172.21.15.111:0/725821235 >> 172.21.15.111:0/725821235 conn(0x7fca78069b90 msgr2=0x7fca780cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 -- 172.21.15.111:0/725821235 shutdown_connections 2023-12-19T09:22:00.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.015+0000 7fca7e323700 1 -- 172.21.15.111:0/725821235 wait complete. 2023-12-19T09:22:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.019+0000 7fca7e323700 1 Processor -- start 2023-12-19T09:22:00.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.019+0000 7fca7e323700 1 -- start start 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.019+0000 7fca7e323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 0x7fca780c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca780cc570 0x7fca780cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780ccf20 con 0x7fca64004810 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780ca0c0 con 0x7fca780cc570 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca780c9250 con 0x7fca78153790 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50858/0 (socket says 172.21.15.111:50858) 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 -- 172.21.15.111:0/1228615575 learned_addr learned my addr 172.21.15.111:0/1228615575 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:00.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7db22700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca780cc570 0x7fca780cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7cb20700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 0x7fca780c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 -- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 msgr2=0x7fca780c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 0x7fca780c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 -- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca780cc570 msgr2=0x7fca780cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca780cc570 0x7fca780cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 -- 172.21.15.111:0/1228615575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78153c00 con 0x7fca64004810 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7cb20700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 0x7fca780c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7d321700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fca74011d40 tx=0x7fca740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400fac0 con 0x7fca64004810 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7400fc20 con 0x7fca64004810 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74014440 con 0x7fca64004810 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca780c94b0 con 0x7fca64004810 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca780c9a70 con 0x7fca64004810 2023-12-19T09:22:00.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.023+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca780b5d80 con 0x7fca64004810 2023-12-19T09:22:00.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.027+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fca74014870 con 0x7fca64004810 2023-12-19T09:22:00.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.027+0000 7fca6d7fa700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 0x7fca5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.027+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fca74018030 con 0x7fca64004810 2023-12-19T09:22:00.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.027+0000 7fca7cb20700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 0x7fca5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.031+0000 7fca7cb20700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 0x7fca5c062a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fca78156df0 tx=0x7fca68015000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.035+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fca780b5d80 con 0x7fca64004810 2023-12-19T09:22:00.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.203+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fca780b5d80 con 0x7fca64004810 2023-12-19T09:22:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.215+0000 7fca6d7fa700 1 -- 172.21.15.111:0/1228615575 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v73) v1 ==== 120+0+0 (secure 0 0 0) 0x7fca780b5d80 con 0x7fca64004810 2023-12-19T09:22:00.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 msgr2=0x7fca5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 0x7fca5c062a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fca78156df0 tx=0x7fca68015000 comp rx=0 tx=0).stop 2023-12-19T09:22:00.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 msgr2=0x7fca780c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fca74011d40 tx=0x7fca740091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 shutdown_connections 2023-12-19T09:22:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fca5c060550 0x7fca5c062a00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca780cc570 0x7fca780cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca78153790 0x7fca780c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 --2- 172.21.15.111:0/1228615575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca64004810 0x7fca780c8440 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 >> 172.21.15.111:0/1228615575 conn(0x7fca78069b90 msgr2=0x7fca780b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:00.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 shutdown_connections 2023-12-19T09:22:00.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.219+0000 7fca7e323700 1 -- 172.21.15.111:0/1228615575 wait complete. 2023-12-19T09:22:00.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-19T09:22:00.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.403+0000 7f777759e700 1 Processor -- start 2023-12-19T09:22:00.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.403+0000 7f777759e700 1 -- start start 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f77780b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7778067060 con 0x7f77780cef90 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7778067900 con 0x7f7778158c60 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780cf750 con 0x7f777815a040 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7776d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f77780cef90 0x7f777815ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44430/0 (socket says 172.21.15.111:44430) 2023-12-19T09:22:00.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7776d9d700 1 -- 172.21.15.111:0/1452386039 learned_addr learned my addr 172.21.15.111:0/1452386039 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f77780b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025980312 0 0) 0x7f7778067060 con 0x7f77780cef90 2023-12-19T09:22:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7760003570 con 0x7f77780cef90 2023-12-19T09:22:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855660554 0 0) 0x7f7760003570 con 0x7f77780cef90 2023-12-19T09:22:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 msgr2=0x7f77780b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f77780b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 >> v1:172.21.15.143:6789/0 conn(0x7f7778158c60 legacy=0x7f77780b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780672b0 con 0x7f77780cef90 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f77700017f0 con 0x7f77780cef90 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 >> v1:172.21.15.111:6789/0 conn(0x7f77780cef90 legacy=0x7f777815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f7760003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f7760004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7760004d50 0x7f77600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780cf750 con 0x7f7760004010 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776000bbf0 con 0x7f7760004d50 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776000bdb0 con 0x7f777815a040 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f7760004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777659c700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f7760003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 -- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 msgr2=0x7f7760003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7776d9d700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7760004d50 0x7f77600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f777815a040 0x7f7760003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 -- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7760004d50 msgr2=0x7f77600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7760004d50 0x7f77600051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 -- 172.21.15.111:0/1452386039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760003570 con 0x7f7760004010 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f7760004810 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f776c006f00 tx=0x7f776c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835608277d0ce42e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c00e710 con 0x7f7760004010 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c00e870 con 0x7f7760004010 2023-12-19T09:22:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777559a700 1 -- 172.21.15.111:0/1452386039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c00ec30 con 0x7f7760004010 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 -- 172.21.15.111:0/1452386039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 msgr2=0x7f7760004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f777759e700 1 --2- 172.21.15.111:0/1452386039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f7760004810 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f776c006f00 tx=0x7f776c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.407+0000 7f7776d9d700 1 -- 172.21.15.111:0/1452386039 reap_dead start 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- 172.21.15.111:0/1452386039 shutdown_connections 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- 172.21.15.111:0/1452386039 >> 172.21.15.111:0/1452386039 conn(0x7f7778068f80 msgr2=0x7f7778069380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- 172.21.15.111:0/1452386039 shutdown_connections 2023-12-19T09:22:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- 172.21.15.111:0/1452386039 wait complete. 2023-12-19T09:22:00.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 Processor -- start 2023-12-19T09:22:00.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- start start 2023-12-19T09:22:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7778152aa0 0x7f777814b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77780af8e0 0x7f77780afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780af450 con 0x7f7760004010 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b0b00 con 0x7f7778152aa0 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b0de0 con 0x7f77780af8e0 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50888/0 (socket says 172.21.15.111:50888) 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.411+0000 7f777659c700 1 -- 172.21.15.111:0/1323028743 learned_addr learned my addr 172.21.15.111:0/1323028743 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7778152aa0 0x7f777814b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f7776d9d700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77780af8e0 0x7f77780afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 -- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77780af8e0 msgr2=0x7f77780afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77780af8e0 0x7f77780afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 -- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7778152aa0 msgr2=0x7f777814b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7778152aa0 0x7f777814b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 -- 172.21.15.111:0/1323028743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780aee00 con 0x7f7760004010 2023-12-19T09:22:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777659c700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f776800e810 tx=0x7f7768009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768012590 con 0x7f7760004010 2023-12-19T09:22:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77680126f0 con 0x7f7760004010 2023-12-19T09:22:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77780af0e0 con 0x7f7760004010 2023-12-19T09:22:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768012a90 con 0x7f7760004010 2023-12-19T09:22:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.415+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777814bd70 con 0x7f7760004010 2023-12-19T09:22:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.419+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777804be40 con 0x7f7760004010 2023-12-19T09:22:00.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.419+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f776801d460 con 0x7f7760004010 2023-12-19T09:22:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.419+0000 7f77667fc700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 0x7f7758062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.419+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f776807cc30 con 0x7f7760004010 2023-12-19T09:22:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.419+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 0x7f7758062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.423+0000 7f7775d9b700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 0x7f7758062a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f776c0091e0 tx=0x7f776c01a000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.423+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f776804ced0 con 0x7f7760004010 2023-12-19T09:22:00.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.599+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7778067590 con 0x7f7760004010 2023-12-19T09:22:00.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.607+0000 7f77667fc700 1 -- 172.21.15.111:0/1323028743 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7f77680504f0 con 0x7f7760004010 2023-12-19T09:22:00.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 msgr2=0x7f7758062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 0x7f7758062a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f776c0091e0 tx=0x7f776c01a000 comp rx=0 tx=0).stop 2023-12-19T09:22:00.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 msgr2=0x7f77780b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f776800e810 tx=0x7f7768009040 comp rx=0 tx=0).stop 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 shutdown_connections 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7758060580 0x7f7758062a30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7778152aa0 0x7f777814b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77780af8e0 0x7f77780afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 --2- 172.21.15.111:0/1323028743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7760004010 0x7f77780b0400 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 >> 172.21.15.111:0/1323028743 conn(0x7f7778068f80 msgr2=0x7f77780b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 shutdown_connections 2023-12-19T09:22:00.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.611+0000 7f777759e700 1 -- 172.21.15.111:0/1323028743 wait complete. 2023-12-19T09:22:00.639 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-12-19T09:22:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 Processor -- start 2023-12-19T09:22:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 -- start start 2023-12-19T09:22:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f050c159450 0x7f050c0b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f050c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c0b3130 con 0x7f050c159080 2023-12-19T09:22:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c0cc160 con 0x7f050c159450 2023-12-19T09:22:00.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.795+0000 7f0511c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cc440 con 0x7f050c0b22e0 2023-12-19T09:22:00.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f050c159080 0x7f050c0cb6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:44438/0 (socket says 172.21.15.111:44438) 2023-12-19T09:22:00.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050b7fe700 1 -- 172.21.15.111:0/636173646 learned_addr learned my addr 172.21.15.111:0/636173646 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 --2- 172.21.15.111:0/636173646 >> v2:172.21.15.143:3300/0 conn(0x7f050c159450 0x7f050c0b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050bfff700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f050c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050197442 0 0) 0x7f050c0b3130 con 0x7f050c159080 2023-12-19T09:22:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4003570 con 0x7f050c159080 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540218406 0 0) 0x7f04f4003570 con 0x7f050c159080 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 msgr2=0x7f050c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f050c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 >> v2:172.21.15.143:3300/0 conn(0x7f050c159450 msgr2=0x7f050c0b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 --2- 172.21.15.111:0/636173646 >> v2:172.21.15.143:3300/0 conn(0x7f050c159450 0x7f050c0b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0cc720 con 0x7f050c159080 2023-12-19T09:22:00.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f04fc001470 con 0x7f050c159080 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 >> v1:172.21.15.111:6789/0 conn(0x7f050c159080 legacy=0x7f050c0cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f4003a60 0x7f04f4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04f4004810 0x7f04f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f04f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cc160 con 0x7f04f4003a60 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cc440 con 0x7f04f4004810 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f400ba60 con 0x7f050c0b22e0 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04f4004810 0x7f04f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050b7fe700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f4003a60 0x7f04f4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050bfff700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f04f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 -- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 msgr2=0x7f04f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f050c0b22e0 0x7f04f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 -- 172.21.15.111:0/636173646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f4003a60 msgr2=0x7f04f4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f4003a60 0x7f04f4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 -- 172.21.15.111:0/636173646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f4003570 con 0x7f04f4004810 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.799+0000 7f050affd700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04f4004810 0x7f04f4004c80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0500009c00 tx=0x7f05000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a388701e16d2779b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050000e710 con 0x7f04f4004810 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050000e870 con 0x7f04f4004810 2023-12-19T09:22:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f050a7fc700 1 -- 172.21.15.111:0/636173646 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050000ec30 con 0x7f04f4004810 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 -- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04f4004810 msgr2=0x7f04f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 --2- 172.21.15.111:0/636173646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04f4004810 0x7f04f4004c80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0500009c00 tx=0x7f05000091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f050bfff700 1 -- 172.21.15.111:0/636173646 reap_dead start 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 -- 172.21.15.111:0/636173646 shutdown_connections 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 -- 172.21.15.111:0/636173646 >> 172.21.15.111:0/636173646 conn(0x7f050c069c90 msgr2=0x7f050c0cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 -- 172.21.15.111:0/636173646 shutdown_connections 2023-12-19T09:22:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.803+0000 7f0511c2e700 1 -- 172.21.15.111:0/636173646 wait complete. 2023-12-19T09:22:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 Processor -- start 2023-12-19T09:22:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 -- start start 2023-12-19T09:22:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04f4004810 0x7f050c0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f050c0c7dd0 0x7f050c14b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0c9d20 con 0x7f050c067c20 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0c37c0 con 0x7f050c0c7dd0 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f0511c2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0685d0 con 0x7f04f4004810 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50912/0 (socket says 172.21.15.111:50912) 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 -- 172.21.15.111:0/1285137566 learned_addr learned my addr 172.21.15.111:0/1285137566 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050b7fe700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04f4004810 0x7f050c0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050affd700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f050c0c7dd0 0x7f050c14b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 -- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04f4004810 msgr2=0x7f050c0c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04f4004810 0x7f050c0c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 -- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f050c0c7dd0 msgr2=0x7f050c14b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f050c0c7dd0 0x7f050c14b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.807+0000 7f050bfff700 1 -- 172.21.15.111:0/1285137566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0c9480 con 0x7f050c067c20 2023-12-19T09:22:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f050bfff700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f050c15cdf0 tx=0x7f050400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050400e070 con 0x7f050c067c20 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504011d00 con 0x7f050c067c20 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040166c0 con 0x7f050c067c20 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c0c9760 con 0x7f050c067c20 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c0c9170 con 0x7f050c067c20 2023-12-19T09:22:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.811+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c0c3950 con 0x7f050c067c20 2023-12-19T09:22:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.815+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0504016af0 con 0x7f050c067c20 2023-12-19T09:22:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.815+0000 7f04f37fe700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 0x7f04f8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.815+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f050407d5e0 con 0x7f050c067c20 2023-12-19T09:22:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.815+0000 7f050b7fe700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 0x7f04f8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.819+0000 7f050b7fe700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 0x7f04f8062a30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f04fc000e10 tx=0x7f04fc009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.823+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f050404d880 con 0x7f050c067c20 2023-12-19T09:22:00.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.987+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f050c071110 con 0x7f050c067c20 2023-12-19T09:22:01.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:00.999+0000 7f04f37fe700 1 -- 172.21.15.111:0/1285137566 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v75) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0504050ea0 con 0x7f050c067c20 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 msgr2=0x7f04f8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 0x7f04f8062a30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f04fc000e10 tx=0x7f04fc009040 comp rx=0 tx=0).stop 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 msgr2=0x7f050c068090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f050c15cdf0 tx=0x7f050400b040 comp rx=0 tx=0).stop 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 shutdown_connections 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f04f8060580 0x7f04f8062a30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f050c0c7dd0 0x7f050c14b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f050c067c20 0x7f050c068090 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 --2- 172.21.15.111:0/1285137566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04f4004810 0x7f050c0c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 >> 172.21.15.111:0/1285137566 conn(0x7f050c069c90 msgr2=0x7f050c0cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:01.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 shutdown_connections 2023-12-19T09:22:01.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.003+0000 7f0511c2e700 1 -- 172.21.15.111:0/1285137566 wait complete. 2023-12-19T09:22:01.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-12-19T09:22:01.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 Processor -- start 2023-12-19T09:22:01.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 -- start start 2023-12-19T09:22:01.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5164157500 0x7f51641578d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f51640cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640cc980 con 0x7f5164157e10 2023-12-19T09:22:01.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640cbd70 con 0x7f5164157500 2023-12-19T09:22:01.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5169745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640cc050 con 0x7f5164159710 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5164157500 0x7f51641578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5164157500 0x7f51641578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50916/0 (socket says 172.21.15.111:50916) 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 -- 172.21.15.111:0/3379390369 learned_addr learned my addr 172.21.15.111:0/3379390369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5163fff700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f51640cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 -- 172.21.15.111:0/3379390369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 msgr2=0x7f51640cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f51640cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 -- 172.21.15.111:0/3379390369 >> v1:172.21.15.143:6789/0 conn(0x7f5164157e10 legacy=0x7f51640cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 -- 172.21.15.111:0/3379390369 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640cc330 con 0x7f5164157500 2023-12-19T09:22:01.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5161ffb700 1 -- 172.21.15.111:0/3379390369 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220369730 0 0) 0x7f51640cc980 con 0x7f5164157e10 2023-12-19T09:22:01.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.191+0000 7f5162ffd700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f51641578d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f51540099b0 tx=0x7f5154009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e10322dda5e6890 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:01.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5161ffb700 1 -- 172.21.15.111:0/3379390369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515400b860 con 0x7f5164157500 2023-12-19T09:22:01.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5161ffb700 1 -- 172.21.15.111:0/3379390369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515400b9c0 con 0x7f5164157500 2023-12-19T09:22:01.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5161ffb700 1 -- 172.21.15.111:0/3379390369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515400bd60 con 0x7f5164157500 2023-12-19T09:22:01.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 -- 172.21.15.111:0/3379390369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 msgr2=0x7f51641578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f51641578d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f51540099b0 tx=0x7f5154009250 comp rx=0 tx=0).stop 2023-12-19T09:22:01.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 -- 172.21.15.111:0/3379390369 shutdown_connections 2023-12-19T09:22:01.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f51640cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 --2- 172.21.15.111:0/3379390369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f51641578d0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 -- 172.21.15.111:0/3379390369 >> 172.21.15.111:0/3379390369 conn(0x7f516406bad0 msgr2=0x7f51640c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:01.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 -- 172.21.15.111:0/3379390369 shutdown_connections 2023-12-19T09:22:01.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.195+0000 7f5169745700 1 -- 172.21.15.111:0/3379390369 wait complete. 2023-12-19T09:22:01.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 Processor -- start 2023-12-19T09:22:01.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 -- start start 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5164157e10 0x7f516415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f5164155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640c58a0 con 0x7f5164157500 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516415cc10 con 0x7f5164157e10 2023-12-19T09:22:01.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5169745700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516415cef0 con 0x7f5164159710 2023-12-19T09:22:01.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5162ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50932/0 (socket says 172.21.15.111:50932) 2023-12-19T09:22:01.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f5162ffd700 1 -- 172.21.15.111:0/696259865 learned_addr learned my addr 172.21.15.111:0/696259865 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:01.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.199+0000 7f51627fc700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5164157e10 0x7f516415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5163fff700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f5164155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 -- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 msgr2=0x7f5164155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f5164155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 -- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5164157e10 msgr2=0x7f516415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5164157e10 0x7f516415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 -- 172.21.15.111:0/696259865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51540094f0 con 0x7f5164157500 2023-12-19T09:22:01.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5162ffd700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5154009290 tx=0x7f5154017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:01.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154011e00 con 0x7f5164157500 2023-12-19T09:22:01.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5154024440 con 0x7f5164157500 2023-12-19T09:22:01.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154024800 con 0x7f5164157500 2023-12-19T09:22:01.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516415d1d0 con 0x7f5164157500 2023-12-19T09:22:01.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5164145230 con 0x7f5164157500 2023-12-19T09:22:01.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.203+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51640c56d0 con 0x7f5164157500 2023-12-19T09:22:01.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.207+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f515401a420 con 0x7f5164157500 2023-12-19T09:22:01.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.207+0000 7f514affd700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 0x7f5150062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:01.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.207+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f515407e8c0 con 0x7f5164157500 2023-12-19T09:22:01.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.207+0000 7f51627fc700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 0x7f5150062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:01.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.207+0000 7f51627fc700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 0x7f5150062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51640b3060 tx=0x7f5158007cd0 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:01.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.211+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f515404eb60 con 0x7f5164157500 2023-12-19T09:22:01.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.443+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f516404be40 con 0x7f5164157500 2023-12-19T09:22:01.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.831+0000 7f514affd700 1 -- 172.21.15.111:0/696259865 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3078) v1 ==== 155+0+0 (secure 0 0 0) 0x7f5154052180 con 0x7f5164157500 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 msgr2=0x7f5150062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 0x7f5150062a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f51640b3060 tx=0x7f5158007cd0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 msgr2=0x7f5164144cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5154009290 tx=0x7f5154017000 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 shutdown_connections 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f5150060550 0x7f5150062a00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5164157e10 0x7f516415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5164159710 0x7f5164155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 --2- 172.21.15.111:0/696259865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5164157500 0x7f5164144cf0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 >> 172.21.15.111:0/696259865 conn(0x7f516406bad0 msgr2=0x7f51640b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:01.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 shutdown_connections 2023-12-19T09:22:01.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:01.835+0000 7f5169745700 1 -- 172.21.15.111:0/696259865 wait complete. 2023-12-19T09:22:01.864 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-12-19T09:22:01.864 INFO:tasks.cephfs_test_runner: 2023-12-19T09:22:01.865 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-19T09:22:01.865 INFO:tasks.cephfs_test_runner:Ran 11 tests in 5637.356s 2023-12-19T09:22:01.865 INFO:tasks.cephfs_test_runner: 2023-12-19T09:22:01.865 INFO:tasks.cephfs_test_runner:OK 2023-12-19T09:22:01.865 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-12-19T09:22:01.877 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-19T09:22:01.887 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-19T09:22:01.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:01.888 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-12-19T09:22:01.908 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-19T09:22:01.908 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-19T09:22:01.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:01.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T09:22:01.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:01.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-19T09:22:02.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:02.070 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-12-19T09:22:02.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:02.292 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-12-19T09:22:02.311 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-19T09:22:02.311 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-12-19T09:22:02.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:02.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T09:22:02.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:02.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-19T09:22:02.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T09:22:02.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link delete ceph-brx 2023-12-19T09:22:02.708 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-19T09:22:02.717 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-19T09:22:02.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:22:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 Processor -- start 2023-12-19T09:22:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 -- start start 2023-12-19T09:22:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 0x7f9d0806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d080b1080 0x7f9d08069840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d080abd00 con 0x7f9d080adf70 2023-12-19T09:22:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080abb10 con 0x7f9d080b1080 2023-12-19T09:22:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.870+0000 7f9d0f08e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080abfe0 con 0x7f9d080b0cb0 2023-12-19T09:22:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d080b1080 0x7f9d08069840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0e08c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 0x7f9d0806a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0e88d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f9d080adf70 0x7f9d0815a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:54394/0 (socket says 172.21.15.111:54394) 2023-12-19T09:22:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0e88d700 1 -- 172.21.15.111:0/577582128 learned_addr learned my addr 172.21.15.111:0/577582128 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 -- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 msgr2=0x7f9d0806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 0x7f9d0806a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 -- 172.21.15.111:0/577582128 >> v1:172.21.15.143:6789/0 conn(0x7f9d080adf70 legacy=0x7f9d0815a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 -- 172.21.15.111:0/577582128 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080ac2c0 con 0x7f9d080b1080 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d08a700 1 -- 172.21.15.111:0/577582128 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432153563 0 0) 0x7f9d080abd00 con 0x7f9d080adf70 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0e08c700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 0x7f9d0806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d88b700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080b1080 0x7f9d08069840 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80098e0 tx=0x7f9cf80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803631eccb38c2eb server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:02.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d08a700 1 -- 172.21.15.111:0/577582128 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800b640 con 0x7f9d080b1080 2023-12-19T09:22:02.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d08a700 1 -- 172.21.15.111:0/577582128 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf800b7a0 con 0x7f9d080b1080 2023-12-19T09:22:02.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.874+0000 7f9d0d08a700 1 -- 172.21.15.111:0/577582128 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf800bb40 con 0x7f9d080b1080 2023-12-19T09:22:02.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080b1080 msgr2=0x7f9d08069840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:02.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080b1080 0x7f9d08069840 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80098e0 tx=0x7f9cf80091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- 172.21.15.111:0/577582128 shutdown_connections 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080b1080 0x7f9d08069840 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/577582128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b0cb0 0x7f9d0806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- 172.21.15.111:0/577582128 >> 172.21.15.111:0/577582128 conn(0x7f9d080cc530 msgr2=0x7f9d080c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- 172.21.15.111:0/577582128 shutdown_connections 2023-12-19T09:22:02.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- 172.21.15.111:0/577582128 wait complete. 2023-12-19T09:22:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 Processor -- start 2023-12-19T09:22:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.878+0000 7f9d0f08e700 1 -- start start 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080adf70 0x7f9d080bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b1080 0x7f9d08167dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c09b0 con 0x7f9d080b0cb0 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c4420 con 0x7f9d080adf70 2023-12-19T09:22:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c0c50 con 0x7f9d080b1080 2023-12-19T09:22:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55062/0 (socket says 172.21.15.111:55062) 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 -- 172.21.15.111:0/833011032 learned_addr learned my addr 172.21.15.111:0/833011032 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0e08c700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080adf70 0x7f9d080bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0e88d700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b1080 0x7f9d08167dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 -- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b1080 msgr2=0x7f9d08167dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b1080 0x7f9d08167dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 -- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080adf70 msgr2=0x7f9d080bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080adf70 0x7f9d080bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 -- 172.21.15.111:0/833011032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009420 con 0x7f9d080b0cb0 2023-12-19T09:22:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0d88b700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009f80 tx=0x7f9cf8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8006af0 con 0x7f9d080b0cb0 2023-12-19T09:22:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf800b910 con 0x7f9d080b0cb0 2023-12-19T09:22:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf8009420 con 0x7f9d080b0cb0 2023-12-19T09:22:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8011420 con 0x7f9d080b0cb0 2023-12-19T09:22:02.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.882+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d08159be0 con 0x7f9d080b0cb0 2023-12-19T09:22:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.886+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d080c1040 con 0x7f9d080b0cb0 2023-12-19T09:22:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.886+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9cf8011850 con 0x7f9d080b0cb0 2023-12-19T09:22:02.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.886+0000 7f9cfe7fc700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 0x7f9cec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:02.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.886+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f9cf807e210 con 0x7f9d080b0cb0 2023-12-19T09:22:02.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.890+0000 7f9d0e08c700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 0x7f9cec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:02.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.890+0000 7f9d0e08c700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 0x7f9cec062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009f40 tx=0x7f9d040091a0 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:02.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:02.894+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9cf804e550 con 0x7f9d080b0cb0 2023-12-19T09:22:03.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.062+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d080c45e0 con 0x7f9d080b0cb0 2023-12-19T09:22:03.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.066+0000 7f9cfe7fc700 1 -- 172.21.15.111:0/833011032 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+14216 (secure 0 0 0) 0x7f9cf8051b70 con 0x7f9d080b0cb0 2023-12-19T09:22:03.073 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:03.073 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":213,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:59.804259+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6837/68388372":"2023-12-20T09:21:56.484688+0000","172.21.15.143:6836/68388372":"2023-12-20T09:21:56.484688+0000","172.21.15.143:0/3433885932":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3697151201":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2906865886":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3049351864":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2337999510":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2085760645":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3085442911":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/278966201":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/2861944468":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6833/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6832/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/150889021":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/800113432":"2023-12-20T09:21:38.103524+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.066+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 msgr2=0x7f9cec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 0x7f9cec062a00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009f40 tx=0x7f9d040091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 msgr2=0x7f9d08167880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009f80 tx=0x7f9cf8017000 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 shutdown_connections 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d080b1080 0x7f9d08167dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d080b0cb0 0x7f9d08167880 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9cec060550 0x7f9cec062a00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 --2- 172.21.15.111:0/833011032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d080adf70 0x7f9d080bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 >> 172.21.15.111:0/833011032 conn(0x7f9d080cc530 msgr2=0x7f9d080aa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:03.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 shutdown_connections 2023-12-19T09:22:03.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.070+0000 7f9d0f08e700 1 -- 172.21.15.111:0/833011032 wait complete. 2023-12-19T09:22:03.093 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-19T09:22:03.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:22:03.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.230+0000 7f390d016700 1 Processor -- start 2023-12-19T09:22:03.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.230+0000 7f390d016700 1 -- start start 2023-12-19T09:22:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f390d016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f390d016700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390806ab30 con 0x7f3908069840 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f390d016700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3908067f80 con 0x7f39080aa880 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f390d016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3908068550 con 0x7f3908067060 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:35672/0 (socket says 172.21.15.111:35672) 2023-12-19T09:22:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 -- 172.21.15.111:0/1042607379 learned_addr learned my addr 172.21.15.111:0/1042607379 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f39077fe700 1 -- 172.21.15.111:0/1042607379 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795043788 0 0) 0x7f390806ab30 con 0x7f3908069840 2023-12-19T09:22:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f39077fe700 1 -- 172.21.15.111:0/1042607379 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4003570 con 0x7f3908069840 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 -- 172.21.15.111:0/1042607379 >> v1:172.21.15.143:6789/0 conn(0x7f39080aa880 legacy=0x7f3908067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 -- 172.21.15.111:0/1042607379 >> v1:172.21.15.111:6789/0 conn(0x7f3908069840 legacy=0x7f3908069ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 -- 172.21.15.111:0/1042607379 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080acb00 con 0x7f3908067060 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f3907fff700 1 --2- 172.21.15.111:0/1042607379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f38f80099e0 tx=0x7f38f8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e87efdc63098c35 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f39077fe700 1 -- 172.21.15.111:0/1042607379 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f800b860 con 0x7f3908067060 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f39077fe700 1 -- 172.21.15.111:0/1042607379 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f800b9c0 con 0x7f3908067060 2023-12-19T09:22:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.234+0000 7f39077fe700 1 -- 172.21.15.111:0/1042607379 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f800bd80 con 0x7f3908067060 2023-12-19T09:22:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 -- 172.21.15.111:0/1042607379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 msgr2=0x7f39080ab060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 --2- 172.21.15.111:0/1042607379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f38f80099e0 tx=0x7f38f8009250 comp rx=0 tx=0).stop 2023-12-19T09:22:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 -- 172.21.15.111:0/1042607379 shutdown_connections 2023-12-19T09:22:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 --2- 172.21.15.111:0/1042607379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3908067060 0x7f39080ab060 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 -- 172.21.15.111:0/1042607379 >> 172.21.15.111:0/1042607379 conn(0x7f39080cc530 msgr2=0x7f39080c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 -- 172.21.15.111:0/1042607379 shutdown_connections 2023-12-19T09:22:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 -- 172.21.15.111:0/1042607379 wait complete. 2023-12-19T09:22:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.238+0000 7f390d016700 1 Processor -- start 2023-12-19T09:22:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- start start 2023-12-19T09:22:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3908069840 0x7f390806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39080aa880 0x7f390806ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c2ca0 con 0x7f3908067060 2023-12-19T09:22:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c1760 con 0x7f3908069840 2023-12-19T09:22:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c1a40 con 0x7f39080aa880 2023-12-19T09:22:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55070/0 (socket says 172.21.15.111:55070) 2023-12-19T09:22:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 -- 172.21.15.111:0/1523306654 learned_addr learned my addr 172.21.15.111:0/1523306654 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 -- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39080aa880 msgr2=0x7f390806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39080aa880 0x7f390806ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 -- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3908069840 msgr2=0x7f390806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3908069840 0x7f390806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 -- 172.21.15.111:0/1523306654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80094f0 con 0x7f3908067060 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3907fff700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f390806d380 tx=0x7f38f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f801f070 con 0x7f3908067060 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f800bd40 con 0x7f3908067060 2023-12-19T09:22:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39080c1d20 con 0x7f3908067060 2023-12-19T09:22:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f800b860 con 0x7f3908067060 2023-12-19T09:22:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.242+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39080c4f10 con 0x7f3908067060 2023-12-19T09:22:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.246+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39080c2a70 con 0x7f3908067060 2023-12-19T09:22:03.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.246+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f38f801a8e0 con 0x7f3908067060 2023-12-19T09:22:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.250+0000 7f3904ff9700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 0x7f38e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.250+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f38f807ddf0 con 0x7f3908067060 2023-12-19T09:22:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.250+0000 7f38ff7fe700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 0x7f38e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.250+0000 7f38ff7fe700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 0x7f38e8062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39080c2460 tx=0x7f38f0007b40 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.250+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38f804e110 con 0x7f3908067060 2023-12-19T09:22:03.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.414+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39080c2de0 con 0x7f38e8060550 2023-12-19T09:22:03.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f3904ff9700 1 -- 172.21.15.111:0/1523306654 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48578 (secure 0 0 0) 0x7f38f0014410 con 0x7f38e8060550 2023-12-19T09:22:03.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 msgr2=0x7f38e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 0x7f38e8062a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39080c2460 tx=0x7f38f0007b40 comp rx=0 tx=0).stop 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 msgr2=0x7f390815c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f390806d380 tx=0x7f38f8017000 comp rx=0 tx=0).stop 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 shutdown_connections 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f38e8060550 0x7f38e8062a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3908069840 0x7f390806d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39080aa880 0x7f390806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 --2- 172.21.15.111:0/1523306654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3908067060 0x7f390815c420 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 >> 172.21.15.111:0/1523306654 conn(0x7f39080cc530 msgr2=0x7f390815cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:03.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 shutdown_connections 2023-12-19T09:22:03.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.426+0000 7f390d016700 1 -- 172.21.15.111:0/1523306654 wait complete. 2023-12-19T09:22:03.436 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:22:03.454 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-19T09:22:02.496828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2475152,"kb_used_data":23120,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752499568,"statfs":{"total":773094113280,"available":770559557632,"internally_reserved":0,"allocated":23674880,"data_stored":13855224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-666822,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-284,"num_write_kb":-2054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.973528"},"pg_stats":[{"pgid":"1.0","version":"211'370","reported_seq":1214,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T09:21:56.584541+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:56.584541+0000","last_peered":"2023-12-19T09:21:56.584541+0000","last_clean":"2023-12-19T09:21:56.584541+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:56.584541+0000","last_undegraded":"2023-12-19T09:21:56.584541+0000","last_fullsized":"2023-12-19T09:21:56.584541+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575973,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304548,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067292,"statfs":{"total":96636764160,"available":96324907008,"internally_reserved":0,"allocated":3239936,"data_stored":2014764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2709999999999999,"15min":1.298},"min":{"1min":0.82799999999999996,"5min":0.75,"15min":0.71199999999999997},"max":{"1min":1.599,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2729999999999999,"15min":1.294},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.77500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.712},"last":1.042}]},{"osd":1,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1639999999999999,"15min":1.1759999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5509999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2,"15min":1.214},"min":{"1min":0.78900000000000003,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6279999999999999,"5min":1.7769999999999999,"15min":1.796},"last":0.78900000000000003}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1619999999999999,"15min":1.177},"min":{"1min":0.67900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4530000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3049999999999999,"15min":1.3129999999999999},"min":{"1min":0.91100000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.528,"5min":1.845,"15min":1.9059999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":0.98499999999999999,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3740000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.93600000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.5069999999999999,"5min":1.7330000000000001,"15min":1.95},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.94799999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.609,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.264}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2490000000000001,"15min":1.288},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.665,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.3919999999999999,"15min":1.4139999999999999},"min":{"1min":1.121,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.714,"5min":1.972,"15min":1.972},"last":1.2310000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.97199999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.893,"15min":2.0750000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":0.83799999999999997,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.6699999999999999,"5min":1.9530000000000001,"15min":1.986},"last":1.3520000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575972,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.165,"15min":1.1479999999999999},"min":{"1min":0.82499999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.516,"5min":1.6799999999999999,"15min":1.784},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.1839999999999999,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68000000000000005,"15min":0.55100000000000005},"max":{"1min":1.431,"5min":1.6850000000000001,"15min":1.946},"last":0.83899999999999997}]},{"osd":1,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.77900000000000003,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.4730000000000001,"5min":1.603,"15min":2.3439999999999999},"last":0.747},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.94299999999999995,"5min":0.72499999999999998,"15min":0.59099999999999997},"max":{"1min":1.6259999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.0049999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2629999999999999,"15min":1.2549999999999999},"min":{"1min":1.0289999999999999,"5min":0.86899999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.2569999999999999},"min":{"1min":0.90300000000000002,"5min":0.88700000000000001,"15min":0.73999999999999999},"max":{"1min":1.8340000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1679999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.94699999999999995,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.5720000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3360000000000001,"15min":1.3200000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68999999999999995},"max":{"1min":1.5620000000000001,"5min":1.74,"15min":1.8220000000000001},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.69999999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5760000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.147},{"interface":"front","average":{"1min":1.407,"5min":1.448,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7250000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2829999999999999,"15min":1.288},"min":{"1min":0.73299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.619,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.085},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.323,"15min":1.3049999999999999},"min":{"1min":1.034,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.7709999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.056}]},{"osd":7,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3759999999999999,"15min":1.3859999999999999},"min":{"1min":0.97299999999999998,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.927,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.389},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7210000000000001,"5min":1.8899999999999999,"15min":2.2410000000000001},"last":1.2949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308076,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063764,"statfs":{"total":96636764160,"available":96321294336,"internally_reserved":0,"allocated":2658304,"data_stored":1426933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.82099999999999995,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.605,"5min":2.044,"15min":2.044},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3660000000000001,"15min":1.3580000000000001},"min":{"1min":1.0289999999999999,"5min":0.84899999999999998,"15min":0.83599999999999997},"max":{"1min":1.7490000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.4530000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.161},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.444,"5min":1.502,"15min":1.6479999999999999},"last":1.222},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.208,"15min":1.21},"min":{"1min":1.004,"5min":0.84799999999999998,"15min":0.58099999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.1870000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.222,"15min":1.2110000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6830000000000001,"5min":1.782,"15min":1.782},"last":1.131},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.236,"15min":1.244},"min":{"1min":0.873,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5429999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3149999999999999},"min":{"1min":0.89700000000000002,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.7350000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.079},{"interface":"front","average":{"1min":1.383,"5min":1.3620000000000001,"15min":1.3380000000000001},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6899999999999999,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.298}]},{"osd":4,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.278,"15min":1.24},"min":{"1min":0.98499999999999999,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":1.6120000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.333,"5min":1.3600000000000001,"15min":1.3169999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.59099999999999997},"max":{"1min":1.7250000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.345,"15min":1.325},"min":{"1min":0.82099999999999995,"5min":0.72799999999999998,"15min":0.69199999999999995},"max":{"1min":1.6910000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.389},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3919999999999999,"15min":1.3759999999999999},"min":{"1min":0.96699999999999997,"5min":0.90400000000000003,"15min":0.68999999999999995},"max":{"1min":1.6950000000000001,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.04}]},{"osd":7,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4870000000000001,"15min":1.45},"min":{"1min":1.2050000000000001,"5min":1.018,"15min":0.84199999999999997},"max":{"1min":1.796,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.492},{"interface":"front","average":{"1min":1.514,"5min":1.512,"15min":1.49},"min":{"1min":0.99399999999999999,"5min":0.90900000000000003,"15min":0.79500000000000004},"max":{"1min":1.7609999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.8300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312184,"kb_used_data":2608,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059656,"statfs":{"total":96636764160,"available":96317087744,"internally_reserved":0,"allocated":2670592,"data_stored":1434469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.353},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.369}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.014},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":0.747},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.149}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.2050000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.276},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.2949999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.4330000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575972,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88800000000000001,"15min":0.878},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.254,"5min":1.581,"15min":1.581},"last":0.996},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.96299999999999997,"15min":0.98799999999999999},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.494,"5min":2.117,"15min":2.117},"last":1.361}]},{"osd":1,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.92200000000000004,"15min":0.93500000000000005},"min":{"1min":0.54800000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1579999999999999,"5min":2.331,"15min":2.331},"last":1.206},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0529999999999999},"min":{"1min":0.69099999999999995,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.5649999999999999,"5min":2.23,"15min":2.23},"last":1.1299999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.103,"15min":1.1200000000000001},"min":{"1min":0.69499999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.677,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.175},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1799999999999999,"15min":1.1919999999999999},"min":{"1min":0.89000000000000001,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.7190000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.331,"15min":1.3200000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.70199999999999996},"max":{"1min":1.8779999999999999,"5min":2.157,"15min":2.157},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.2909999999999999},"min":{"1min":1.073,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":1.9390000000000001,"5min":2.149,"15min":2.149},"last":1.5700000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3220000000000001,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1400000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.407,"15min":1.3879999999999999},"min":{"1min":1.0529999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0409999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.4119999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.413,"15min":1.423},"min":{"1min":1.1279999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.923,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.49},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4079999999999999,"15min":1.4239999999999999},"min":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.8520000000000001,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.524}]},{"osd":7,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.5009999999999999},"min":{"1min":1.1419999999999999,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":1.992,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.4710000000000001,"15min":1.456},"min":{"1min":1.0620000000000001,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0489999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.778}]}]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1737468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2529999999999999,"5min":1.345,"15min":1.5049999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.286,"5min":1.556,"15min":1.6759999999999999},"last":1.01}]},{"osd":1,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3640000000000001,"5min":1.569,"15min":1.7609999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.034,"15min":1.044},"min":{"1min":0.59499999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4339999999999999,"5min":1.5369999999999999,"15min":1.768},"last":0.91300000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.472,"5min":1.53,"15min":1.891},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.121,"5min":1.1359999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":0.99299999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.341},"min":{"1min":0.76900000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.754},"last":1.095},{"interface":"front","average":{"1min":1.242,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.556,"5min":1.76,"15min":1.8520000000000001},"last":1.2929999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3759999999999999},"min":{"1min":0.84799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6679999999999999,"5min":1.948,"15min":2.016},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.91200000000000003,"5min":0.83399999999999996,"15min":0.68300000000000005},"max":{"1min":1.6000000000000001,"5min":1.901,"15min":1.927},"last":1.052}]},{"osd":6,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3120000000000001,"15min":1.345},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8200000000000001,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.181},{"interface":"front","average":{"1min":1.379,"5min":1.365,"15min":1.419},"min":{"1min":1.036,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.157}]},{"osd":7,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.4059999999999999,"15min":1.4370000000000001},"min":{"1min":0.94799999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.0489999999999999,"15min":2.327},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9059999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834575973,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312480,"kb_used_data":2904,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059360,"statfs":{"total":96636764160,"available":96316784640,"internally_reserved":0,"allocated":2973696,"data_stored":1752076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.87},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.9039999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.6850000000000001},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.996,"5min":1.0209999999999999,"15min":1.0629999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":2.302},"last":0.93500000000000005}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.175},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.04},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.069},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2929999999999999,"15min":1.3220000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.546,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":0.998}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.296,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.899,"5min":1.944,"15min":1.944},"last":1.425},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.536}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.877,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.95299999999999996,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.752,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.5680000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.825,"5min":1.825,"15min":2.069},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4259999999999999,"15min":1.4359999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.456}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300464,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071376,"statfs":{"total":96636764160,"available":96329089024,"internally_reserved":0,"allocated":3252224,"data_stored":2014453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.97099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3069999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.995,"15min":1.0169999999999999},"min":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3759999999999999,"5min":1.446,"15min":1.5269999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.728,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2709999999999999,"5min":1.611,"15min":2.621},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.177},"min":{"1min":0.61599999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.0029999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.341,"15min":1.377},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.605,"5min":1.891,"15min":1.891},"last":1.232},{"interface":"front","average":{"1min":1.341,"5min":1.333,"15min":1.347},"min":{"1min":0.81999999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.5339999999999998},"last":0.90600000000000003}]},{"osd":5,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3759999999999999,"15min":1.3939999999999999},"min":{"1min":0.74299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.966,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.466}]},{"osd":6,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2869999999999999,"15min":1.2849999999999999},"min":{"1min":0.88800000000000001,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.4489999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.373},"min":{"1min":0.98299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.837,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.425}]},{"osd":7,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.339,"15min":1.3400000000000001},"min":{"1min":0.86399999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.746,"5min":2.137,"15min":2.8570000000000002},"last":1.097},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4239999999999999,"15min":1.4299999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0640000000000001,"5min":2.1259999999999999,"15min":2.181},"last":1.3069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:22:03.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:22:03.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.610+0000 7ff4a620c700 1 Processor -- start 2023-12-19T09:22:03.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 -- start start 2023-12-19T09:22:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 0x7ff4a00b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff4a00b1be0 0x7ff4a00b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a00cb3e0 con 0x7ff4a0159210 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a0068960 con 0x7ff4a00b1be0 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a620c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a00caab0 con 0x7ff4a0158e40 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a4a09700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff4a0159210 0x7ff4a0067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50796/0 (socket says 172.21.15.111:50796) 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a4a09700 1 -- 172.21.15.111:0/2345432370 learned_addr learned my addr 172.21.15.111:0/2345432370 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a5a0b700 1 --2- 172.21.15.111:0/2345432370 >> v2:172.21.15.111:3300/0 conn(0x7ff4a00b1be0 0x7ff4a00b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.614+0000 7ff4a520a700 1 --2- 172.21.15.111:0/2345432370 >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 0x7ff4a00b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff497fff700 1 -- 172.21.15.111:0/2345432370 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056951033 0 0) 0x7ff4a00cb3e0 con 0x7ff4a0159210 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff497fff700 1 -- 172.21.15.111:0/2345432370 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff48c003570 con 0x7ff4a0159210 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a5a0b700 1 -- 172.21.15.111:0/2345432370 >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 msgr2=0x7ff4a00b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a5a0b700 1 --2- 172.21.15.111:0/2345432370 >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 0x7ff4a00b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a5a0b700 1 -- 172.21.15.111:0/2345432370 >> v1:172.21.15.111:6789/0 conn(0x7ff4a0159210 legacy=0x7ff4a0067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a5a0b700 1 -- 172.21.15.111:0/2345432370 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00cad90 con 0x7ff4a00b1be0 2023-12-19T09:22:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a520a700 1 --2- 172.21.15.111:0/2345432370 >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 0x7ff4a00b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a5a0b700 1 --2- 172.21.15.111:0/2345432370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00b2060 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7ff4a006a030 tx=0x7ff498009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe271e12fbb64153 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff497fff700 1 -- 172.21.15.111:0/2345432370 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49800e070 con 0x7ff4a00b1be0 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff497fff700 1 -- 172.21.15.111:0/2345432370 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49800bb60 con 0x7ff4a00b1be0 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff497fff700 1 -- 172.21.15.111:0/2345432370 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498012430 con 0x7ff4a00b1be0 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 -- 172.21.15.111:0/2345432370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 msgr2=0x7ff4a00b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 --2- 172.21.15.111:0/2345432370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00b2060 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7ff4a006a030 tx=0x7ff498009780 comp rx=0 tx=0).stop 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 -- 172.21.15.111:0/2345432370 shutdown_connections 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 --2- 172.21.15.111:0/2345432370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00b2060 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 --2- 172.21.15.111:0/2345432370 >> v2:172.21.15.143:3300/0 conn(0x7ff4a0158e40 0x7ff4a00b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 -- 172.21.15.111:0/2345432370 >> 172.21.15.111:0/2345432370 conn(0x7ff4a00d0900 msgr2=0x7ff4a00cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:03.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 -- 172.21.15.111:0/2345432370 shutdown_connections 2023-12-19T09:22:03.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.618+0000 7ff4a620c700 1 -- 172.21.15.111:0/2345432370 wait complete. 2023-12-19T09:22:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 Processor -- start 2023-12-19T09:22:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 -- start start 2023-12-19T09:22:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4a0158e40 0x7ff4a00c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4a0159210 0x7ff4a0147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0169cc0 con 0x7ff4a00b1be0 2023-12-19T09:22:03.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01690b0 con 0x7ff4a0159210 2023-12-19T09:22:03.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a620c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0169390 con 0x7ff4a0158e40 2023-12-19T09:22:03.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.622+0000 7ff4a520a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55094/0 (socket says 172.21.15.111:55094) 2023-12-19T09:22:03.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 -- 172.21.15.111:0/1284386558 learned_addr learned my addr 172.21.15.111:0/1284386558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:03.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a4a09700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4a0158e40 0x7ff4a00c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a5a0b700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4a0159210 0x7ff4a0147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 -- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4a0158e40 msgr2=0x7ff4a00c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4a0158e40 0x7ff4a00c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 -- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4a0159210 msgr2=0x7ff4a0147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4a0159210 0x7ff4a0147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 -- 172.21.15.111:0/1284386558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff498009a20 con 0x7ff4a00b1be0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a520a700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00e8d0 tx=0x7ff49c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00ee70 con 0x7ff4a00b1be0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00c5e0 con 0x7ff4a00b1be0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a0169670 con 0x7ff4a00b1be0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00c980 con 0x7ff4a00b1be0 2023-12-19T09:22:03.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a00c4e90 con 0x7ff4a00b1be0 2023-12-19T09:22:03.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.626+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a004be40 con 0x7ff4a00b1be0 2023-12-19T09:22:03.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.630+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff49c013410 con 0x7ff4a00b1be0 2023-12-19T09:22:03.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.630+0000 7ff4957fa700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 0x7ff484062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:03.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.630+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7ff49c07c950 con 0x7ff4a00b1be0 2023-12-19T09:22:03.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.630+0000 7ff4a4a09700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 0x7ff484062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:03.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.630+0000 7ff4a4a09700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 0x7ff484062a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00c6870 tx=0x7ff490009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:03.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.634+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff49c04cbf0 con 0x7ff4a00b1be0 2023-12-19T09:22:03.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.798+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4a0169e00 con 0x7ff484060550 2023-12-19T09:22:03.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.802+0000 7ff4957fa700 1 -- 172.21.15.111:0/1284386558 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48578 (secure 0 0 0) 0x7ff490005900 con 0x7ff484060550 2023-12-19T09:22:03.811 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:03.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 msgr2=0x7ff484062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 0x7ff484062a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00c6870 tx=0x7ff490009040 comp rx=0 tx=0).stop 2023-12-19T09:22:03.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 msgr2=0x7ff4a00c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:03.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00e8d0 tx=0x7ff49c008040 comp rx=0 tx=0).stop 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 shutdown_connections 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff484060550 0x7ff484062a00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4a0159210 0x7ff4a0147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4a0158e40 0x7ff4a00c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 --2- 172.21.15.111:0/1284386558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4a00b1be0 0x7ff4a00c4950 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 >> 172.21.15.111:0/1284386558 conn(0x7ff4a00d0900 msgr2=0x7ff4a01534d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 shutdown_connections 2023-12-19T09:22:03.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.806+0000 7ff4a620c700 1 -- 172.21.15.111:0/1284386558 wait complete. 2023-12-19T09:22:03.816 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:22:03.836 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-19T09:22:02.496828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2475152,"kb_used_data":23120,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752499568,"statfs":{"total":773094113280,"available":770559557632,"internally_reserved":0,"allocated":23674880,"data_stored":13855224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-666822,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-284,"num_write_kb":-2054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.973528"},"pg_stats":[{"pgid":"1.0","version":"211'370","reported_seq":1214,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T09:21:56.584541+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:56.584541+0000","last_peered":"2023-12-19T09:21:56.584541+0000","last_clean":"2023-12-19T09:21:56.584541+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:56.584541+0000","last_undegraded":"2023-12-19T09:21:56.584541+0000","last_fullsized":"2023-12-19T09:21:56.584541+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575973,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304548,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067292,"statfs":{"total":96636764160,"available":96324907008,"internally_reserved":0,"allocated":3239936,"data_stored":2014764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2709999999999999,"15min":1.298},"min":{"1min":0.82799999999999996,"5min":0.75,"15min":0.71199999999999997},"max":{"1min":1.599,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2729999999999999,"15min":1.294},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.77500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.712},"last":1.042}]},{"osd":1,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1639999999999999,"15min":1.1759999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5509999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2,"15min":1.214},"min":{"1min":0.78900000000000003,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6279999999999999,"5min":1.7769999999999999,"15min":1.796},"last":0.78900000000000003}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1619999999999999,"15min":1.177},"min":{"1min":0.67900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4530000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3049999999999999,"15min":1.3129999999999999},"min":{"1min":0.91100000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.528,"5min":1.845,"15min":1.9059999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":0.98499999999999999,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3740000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.93600000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.5069999999999999,"5min":1.7330000000000001,"15min":1.95},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.94799999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.609,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.264}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2490000000000001,"15min":1.288},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.665,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.3919999999999999,"15min":1.4139999999999999},"min":{"1min":1.121,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.714,"5min":1.972,"15min":1.972},"last":1.2310000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.97199999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.893,"15min":2.0750000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":0.83799999999999997,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.6699999999999999,"5min":1.9530000000000001,"15min":1.986},"last":1.3520000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575972,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.165,"15min":1.1479999999999999},"min":{"1min":0.82499999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.516,"5min":1.6799999999999999,"15min":1.784},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.1839999999999999,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68000000000000005,"15min":0.55100000000000005},"max":{"1min":1.431,"5min":1.6850000000000001,"15min":1.946},"last":0.83899999999999997}]},{"osd":1,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.77900000000000003,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.4730000000000001,"5min":1.603,"15min":2.3439999999999999},"last":0.747},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.94299999999999995,"5min":0.72499999999999998,"15min":0.59099999999999997},"max":{"1min":1.6259999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.0049999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2629999999999999,"15min":1.2549999999999999},"min":{"1min":1.0289999999999999,"5min":0.86899999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.2569999999999999},"min":{"1min":0.90300000000000002,"5min":0.88700000000000001,"15min":0.73999999999999999},"max":{"1min":1.8340000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1679999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.94699999999999995,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.5720000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3360000000000001,"15min":1.3200000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68999999999999995},"max":{"1min":1.5620000000000001,"5min":1.74,"15min":1.8220000000000001},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.69999999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5760000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.147},{"interface":"front","average":{"1min":1.407,"5min":1.448,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7250000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2829999999999999,"15min":1.288},"min":{"1min":0.73299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.619,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.085},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.323,"15min":1.3049999999999999},"min":{"1min":1.034,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.7709999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.056}]},{"osd":7,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3759999999999999,"15min":1.3859999999999999},"min":{"1min":0.97299999999999998,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.927,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.389},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7210000000000001,"5min":1.8899999999999999,"15min":2.2410000000000001},"last":1.2949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308076,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063764,"statfs":{"total":96636764160,"available":96321294336,"internally_reserved":0,"allocated":2658304,"data_stored":1426933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.82099999999999995,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.605,"5min":2.044,"15min":2.044},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3660000000000001,"15min":1.3580000000000001},"min":{"1min":1.0289999999999999,"5min":0.84899999999999998,"15min":0.83599999999999997},"max":{"1min":1.7490000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.4530000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.161},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.444,"5min":1.502,"15min":1.6479999999999999},"last":1.222},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.208,"15min":1.21},"min":{"1min":1.004,"5min":0.84799999999999998,"15min":0.58099999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.1870000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.222,"15min":1.2110000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6830000000000001,"5min":1.782,"15min":1.782},"last":1.131},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.236,"15min":1.244},"min":{"1min":0.873,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5429999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3149999999999999},"min":{"1min":0.89700000000000002,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.7350000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.079},{"interface":"front","average":{"1min":1.383,"5min":1.3620000000000001,"15min":1.3380000000000001},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6899999999999999,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.298}]},{"osd":4,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.278,"15min":1.24},"min":{"1min":0.98499999999999999,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":1.6120000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.333,"5min":1.3600000000000001,"15min":1.3169999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.59099999999999997},"max":{"1min":1.7250000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.345,"15min":1.325},"min":{"1min":0.82099999999999995,"5min":0.72799999999999998,"15min":0.69199999999999995},"max":{"1min":1.6910000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.389},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3919999999999999,"15min":1.3759999999999999},"min":{"1min":0.96699999999999997,"5min":0.90400000000000003,"15min":0.68999999999999995},"max":{"1min":1.6950000000000001,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.04}]},{"osd":7,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4870000000000001,"15min":1.45},"min":{"1min":1.2050000000000001,"5min":1.018,"15min":0.84199999999999997},"max":{"1min":1.796,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.492},{"interface":"front","average":{"1min":1.514,"5min":1.512,"15min":1.49},"min":{"1min":0.99399999999999999,"5min":0.90900000000000003,"15min":0.79500000000000004},"max":{"1min":1.7609999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.8300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312184,"kb_used_data":2608,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059656,"statfs":{"total":96636764160,"available":96317087744,"internally_reserved":0,"allocated":2670592,"data_stored":1434469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.353},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.369}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.014},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":0.747},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.149}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.2050000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.276},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.2949999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.4330000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575972,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88800000000000001,"15min":0.878},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.254,"5min":1.581,"15min":1.581},"last":0.996},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.96299999999999997,"15min":0.98799999999999999},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.494,"5min":2.117,"15min":2.117},"last":1.361}]},{"osd":1,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.92200000000000004,"15min":0.93500000000000005},"min":{"1min":0.54800000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1579999999999999,"5min":2.331,"15min":2.331},"last":1.206},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0529999999999999},"min":{"1min":0.69099999999999995,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.5649999999999999,"5min":2.23,"15min":2.23},"last":1.1299999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.103,"15min":1.1200000000000001},"min":{"1min":0.69499999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.677,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.175},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1799999999999999,"15min":1.1919999999999999},"min":{"1min":0.89000000000000001,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.7190000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.331,"15min":1.3200000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.70199999999999996},"max":{"1min":1.8779999999999999,"5min":2.157,"15min":2.157},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.2909999999999999},"min":{"1min":1.073,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":1.9390000000000001,"5min":2.149,"15min":2.149},"last":1.5700000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3220000000000001,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1400000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.407,"15min":1.3879999999999999},"min":{"1min":1.0529999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0409999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.4119999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.413,"15min":1.423},"min":{"1min":1.1279999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.923,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.49},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4079999999999999,"15min":1.4239999999999999},"min":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.8520000000000001,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.524}]},{"osd":7,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.5009999999999999},"min":{"1min":1.1419999999999999,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":1.992,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.4710000000000001,"15min":1.456},"min":{"1min":1.0620000000000001,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0489999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.778}]}]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1737468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2529999999999999,"5min":1.345,"15min":1.5049999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.286,"5min":1.556,"15min":1.6759999999999999},"last":1.01}]},{"osd":1,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3640000000000001,"5min":1.569,"15min":1.7609999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.034,"15min":1.044},"min":{"1min":0.59499999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4339999999999999,"5min":1.5369999999999999,"15min":1.768},"last":0.91300000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.472,"5min":1.53,"15min":1.891},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.121,"5min":1.1359999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":0.99299999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.341},"min":{"1min":0.76900000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.754},"last":1.095},{"interface":"front","average":{"1min":1.242,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.556,"5min":1.76,"15min":1.8520000000000001},"last":1.2929999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3759999999999999},"min":{"1min":0.84799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6679999999999999,"5min":1.948,"15min":2.016},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.91200000000000003,"5min":0.83399999999999996,"15min":0.68300000000000005},"max":{"1min":1.6000000000000001,"5min":1.901,"15min":1.927},"last":1.052}]},{"osd":6,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3120000000000001,"15min":1.345},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8200000000000001,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.181},{"interface":"front","average":{"1min":1.379,"5min":1.365,"15min":1.419},"min":{"1min":1.036,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.157}]},{"osd":7,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.4059999999999999,"15min":1.4370000000000001},"min":{"1min":0.94799999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.0489999999999999,"15min":2.327},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9059999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834575973,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312480,"kb_used_data":2904,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059360,"statfs":{"total":96636764160,"available":96316784640,"internally_reserved":0,"allocated":2973696,"data_stored":1752076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.87},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.9039999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.6850000000000001},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.996,"5min":1.0209999999999999,"15min":1.0629999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":2.302},"last":0.93500000000000005}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.175},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.04},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.069},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2929999999999999,"15min":1.3220000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.546,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":0.998}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.296,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.899,"5min":1.944,"15min":1.944},"last":1.425},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.536}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.877,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.95299999999999996,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.752,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.5680000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.825,"5min":1.825,"15min":2.069},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4259999999999999,"15min":1.4359999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.456}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300464,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071376,"statfs":{"total":96636764160,"available":96329089024,"internally_reserved":0,"allocated":3252224,"data_stored":2014453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.97099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3069999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.995,"15min":1.0169999999999999},"min":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3759999999999999,"5min":1.446,"15min":1.5269999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.728,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2709999999999999,"5min":1.611,"15min":2.621},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.177},"min":{"1min":0.61599999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.0029999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.341,"15min":1.377},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.605,"5min":1.891,"15min":1.891},"last":1.232},{"interface":"front","average":{"1min":1.341,"5min":1.333,"15min":1.347},"min":{"1min":0.81999999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.5339999999999998},"last":0.90600000000000003}]},{"osd":5,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3759999999999999,"15min":1.3939999999999999},"min":{"1min":0.74299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.966,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.466}]},{"osd":6,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2869999999999999,"15min":1.2849999999999999},"min":{"1min":0.88800000000000001,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.4489999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.373},"min":{"1min":0.98299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.837,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.425}]},{"osd":7,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.339,"15min":1.3400000000000001},"min":{"1min":0.86399999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.746,"5min":2.137,"15min":2.8570000000000002},"last":1.097},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4239999999999999,"15min":1.4299999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0640000000000001,"5min":2.1259999999999999,"15min":2.181},"last":1.3069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:22:03.837 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-19T09:22:03.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:22:03.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.990+0000 7f43c92ae700 1 Processor -- start 2023-12-19T09:22:03.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.990+0000 7f43c92ae700 1 -- start start 2023-12-19T09:22:04.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c92ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43c40c7640 0x7f43c40c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c92ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43c406a090 0x7f43c4159320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c92ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c415a050 con 0x7f43c4069c10 2023-12-19T09:22:04.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c92ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c406a4d0 con 0x7f43c406a090 2023-12-19T09:22:04.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c92ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c40ac920 con 0x7f43c40c7640 2023-12-19T09:22:04.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43c406a090 0x7f43c4159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43c40c7640 0x7f43c40c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43ba7fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f43c4069c10 0x7f43c40ad400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:54414/0 (socket says 172.21.15.111:54414) 2023-12-19T09:22:04.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43ba7fc700 1 -- 172.21.15.111:0/2294058682 learned_addr learned my addr 172.21.15.111:0/2294058682 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43c406a090 0x7f43c4159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55106/0 (socket says 172.21.15.111:55106) 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c27fc700 1 -- 172.21.15.111:0/2294058682 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280974802 0 0) 0x7f43c415a050 con 0x7f43c4069c10 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 -- 172.21.15.111:0/2294058682 >> v2:172.21.15.143:3300/0 conn(0x7f43c40c7640 msgr2=0x7f43c40c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 --2- 172.21.15.111:0/2294058682 >> v2:172.21.15.143:3300/0 conn(0x7f43c40c7640 0x7f43c40c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 -- 172.21.15.111:0/2294058682 >> v1:172.21.15.143:6789/0 conn(0x7f43c4069c10 legacy=0x7f43c40ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 -- 172.21.15.111:0/2294058682 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c40acc00 con 0x7f43c406a090 2023-12-19T09:22:04.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.994+0000 7f43c37fe700 1 --2- 172.21.15.111:0/2294058682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c4159320 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f43c406d770 tx=0x7f43bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a058712f9e2814 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c27fc700 1 -- 172.21.15.111:0/2294058682 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc00e070 con 0x7f43c406a090 2023-12-19T09:22:04.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c27fc700 1 -- 172.21.15.111:0/2294058682 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc00bb60 con 0x7f43c406a090 2023-12-19T09:22:04.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c27fc700 1 -- 172.21.15.111:0/2294058682 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc012430 con 0x7f43c406a090 2023-12-19T09:22:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 -- 172.21.15.111:0/2294058682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 msgr2=0x7f43c4159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2294058682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c4159320 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f43c406d770 tx=0x7f43bc009780 comp rx=0 tx=0).stop 2023-12-19T09:22:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 -- 172.21.15.111:0/2294058682 shutdown_connections 2023-12-19T09:22:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2294058682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c4159320 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2294058682 >> v2:172.21.15.143:3300/0 conn(0x7f43c40c7640 0x7f43c40c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 -- 172.21.15.111:0/2294058682 >> 172.21.15.111:0/2294058682 conn(0x7f43c40cc690 msgr2=0x7f43c40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:04.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 -- 172.21.15.111:0/2294058682 shutdown_connections 2023-12-19T09:22:04.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:03.998+0000 7f43c92ae700 1 -- 172.21.15.111:0/2294058682 wait complete. 2023-12-19T09:22:04.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 Processor -- start 2023-12-19T09:22:04.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 -- start start 2023-12-19T09:22:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43c4069c10 0x7f43c414e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43c40c7640 0x7f43c40b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40c3950 con 0x7f43c406a090 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40bff10 con 0x7f43c40c7640 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c92ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40c01f0 con 0x7f43c4069c10 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55110/0 (socket says 172.21.15.111:55110) 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 -- 172.21.15.111:0/2700860168 learned_addr learned my addr 172.21.15.111:0/2700860168 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c2ffd700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43c4069c10 0x7f43c414e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43c37fe700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43c40c7640 0x7f43c40b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 -- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43c4069c10 msgr2=0x7f43c414e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43c4069c10 0x7f43c414e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 -- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43c40c7640 msgr2=0x7f43c40b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43c40c7640 0x7f43c40b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 -- 172.21.15.111:0/2700860168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc009a20 con 0x7f43c406a090 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.002+0000 7f43ba7fc700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00c7e0 tx=0x7f43ac007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac005b30 con 0x7f43c406a090 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ac005c90 con 0x7f43c406a090 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac007300 con 0x7f43c406a090 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c40c04d0 con 0x7f43c406a090 2023-12-19T09:22:04.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c414e7d0 con 0x7f43c406a090 2023-12-19T09:22:04.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.006+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c4071110 con 0x7f43c406a090 2023-12-19T09:22:04.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.010+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f43ac01b400 con 0x7f43c406a090 2023-12-19T09:22:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.010+0000 7f43bb7fe700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 0x7f43a8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.010+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f43ac07c5b0 con 0x7f43c406a090 2023-12-19T09:22:04.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.010+0000 7f43c2ffd700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 0x7f43a8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.010+0000 7f43c2ffd700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 0x7f43a8062a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f43c416a420 tx=0x7f43b4008040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.018+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43ac04c8d0 con 0x7f43c406a090 2023-12-19T09:22:04.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.190+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43c40acee0 con 0x7f43a8060580 2023-12-19T09:22:04.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.194+0000 7f43bb7fe700 1 -- 172.21.15.111:0/2700860168 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48578 (secure 0 0 0) 0x7f43c40acee0 con 0x7f43a8060580 2023-12-19T09:22:04.204 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 msgr2=0x7f43a8062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 0x7f43a8062a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f43c416a420 tx=0x7f43b4008040 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 msgr2=0x7f43c40bf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00c7e0 tx=0x7f43ac007df0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 shutdown_connections 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f43a8060580 0x7f43a8062a30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43c40c7640 0x7f43c40b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43c406a090 0x7f43c40bf4c0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 --2- 172.21.15.111:0/2700860168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43c4069c10 0x7f43c414e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 >> 172.21.15.111:0/2700860168 conn(0x7f43c40cc690 msgr2=0x7f43c40c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:04.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 shutdown_connections 2023-12-19T09:22:04.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.198+0000 7f43c92ae700 1 -- 172.21.15.111:0/2700860168 wait complete. 2023-12-19T09:22:04.209 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:22:04.228 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-12-19T09:22:02.496828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2475152,"kb_used_data":23120,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752499568,"statfs":{"total":773094113280,"available":770559557632,"internally_reserved":0,"allocated":23674880,"data_stored":13855224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-666822,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-284,"num_write_kb":-2054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.973528"},"pg_stats":[{"pgid":"1.0","version":"211'370","reported_seq":1214,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T09:21:56.584541+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:56.584541+0000","last_peered":"2023-12-19T09:21:56.584541+0000","last_clean":"2023-12-19T09:21:56.584541+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:56.584541+0000","last_undegraded":"2023-12-19T09:21:56.584541+0000","last_fullsized":"2023-12-19T09:21:56.584541+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575973,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304548,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067292,"statfs":{"total":96636764160,"available":96324907008,"internally_reserved":0,"allocated":3239936,"data_stored":2014764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2709999999999999,"15min":1.298},"min":{"1min":0.82799999999999996,"5min":0.75,"15min":0.71199999999999997},"max":{"1min":1.599,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2729999999999999,"15min":1.294},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.77500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.712},"last":1.042}]},{"osd":1,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1639999999999999,"15min":1.1759999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5509999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2,"15min":1.214},"min":{"1min":0.78900000000000003,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6279999999999999,"5min":1.7769999999999999,"15min":1.796},"last":0.78900000000000003}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1619999999999999,"15min":1.177},"min":{"1min":0.67900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4530000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.91700000000000004}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3049999999999999,"15min":1.3129999999999999},"min":{"1min":0.91100000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.528,"5min":1.845,"15min":1.9059999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":0.98499999999999999,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3740000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.93600000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.5069999999999999,"5min":1.7330000000000001,"15min":1.95},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.94799999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.609,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.264}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2490000000000001,"15min":1.288},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.665,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.3919999999999999,"15min":1.4139999999999999},"min":{"1min":1.121,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.714,"5min":1.972,"15min":1.972},"last":1.2310000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.97199999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.893,"15min":2.0750000000000002},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":0.83799999999999997,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.6699999999999999,"5min":1.9530000000000001,"15min":1.986},"last":1.3520000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575972,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.165,"15min":1.1479999999999999},"min":{"1min":0.82499999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.516,"5min":1.6799999999999999,"15min":1.784},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.1839999999999999,"15min":1.1619999999999999},"min":{"1min":0.80900000000000005,"5min":0.68000000000000005,"15min":0.55100000000000005},"max":{"1min":1.431,"5min":1.6850000000000001,"15min":1.946},"last":0.83899999999999997}]},{"osd":1,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.179,"15min":1.1950000000000001},"min":{"1min":0.77900000000000003,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.4730000000000001,"5min":1.603,"15min":2.3439999999999999},"last":0.747},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.94299999999999995,"5min":0.72499999999999998,"15min":0.59099999999999997},"max":{"1min":1.6259999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.0049999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2629999999999999,"15min":1.2549999999999999},"min":{"1min":1.0289999999999999,"5min":0.86899999999999999,"15min":0.53400000000000003},"max":{"1min":1.5429999999999999,"5min":1.827,"15min":1.827},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.2569999999999999},"min":{"1min":0.90300000000000002,"5min":0.88700000000000001,"15min":0.73999999999999999},"max":{"1min":1.8340000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1679999999999999}]},{"osd":3,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.272,"15min":1.2889999999999999},"min":{"1min":0.94699999999999995,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.5720000000000001,"5min":1.798,"15min":1.8600000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.3360000000000001,"15min":1.3200000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.68999999999999995},"max":{"1min":1.5620000000000001,"5min":1.74,"15min":1.8220000000000001},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.69999999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.5760000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.147},{"interface":"front","average":{"1min":1.407,"5min":1.448,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.7250000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2829999999999999,"15min":1.288},"min":{"1min":0.73299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.619,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.085},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.323,"15min":1.3049999999999999},"min":{"1min":1.034,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.7709999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.056}]},{"osd":7,"last update":"Tue Dec 19 09:21:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3759999999999999,"15min":1.3859999999999999},"min":{"1min":0.97299999999999998,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.927,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.389},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.4259999999999999,"15min":1.4430000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7210000000000001,"5min":1.8899999999999999,"15min":2.2410000000000001},"last":1.2949999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308076,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063764,"statfs":{"total":96636764160,"available":96321294336,"internally_reserved":0,"allocated":2658304,"data_stored":1426933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.82099999999999995,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.605,"5min":2.044,"15min":2.044},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3660000000000001,"15min":1.3580000000000001},"min":{"1min":1.0289999999999999,"5min":0.84899999999999998,"15min":0.83599999999999997},"max":{"1min":1.7490000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.4530000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.161},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.444,"5min":1.502,"15min":1.6479999999999999},"last":1.222},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.208,"15min":1.21},"min":{"1min":1.004,"5min":0.84799999999999998,"15min":0.58099999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.1870000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.222,"15min":1.2110000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6830000000000001,"5min":1.782,"15min":1.782},"last":1.131},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.236,"15min":1.244},"min":{"1min":0.873,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5429999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3149999999999999},"min":{"1min":0.89700000000000002,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.7350000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.079},{"interface":"front","average":{"1min":1.383,"5min":1.3620000000000001,"15min":1.3380000000000001},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6899999999999999,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.298}]},{"osd":4,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.278,"15min":1.24},"min":{"1min":0.98499999999999999,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":1.6120000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.333,"5min":1.3600000000000001,"15min":1.3169999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.59099999999999997},"max":{"1min":1.7250000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.345,"15min":1.325},"min":{"1min":0.82099999999999995,"5min":0.72799999999999998,"15min":0.69199999999999995},"max":{"1min":1.6910000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.389},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3919999999999999,"15min":1.3759999999999999},"min":{"1min":0.96699999999999997,"5min":0.90400000000000003,"15min":0.68999999999999995},"max":{"1min":1.6950000000000001,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.04}]},{"osd":7,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4870000000000001,"15min":1.45},"min":{"1min":1.2050000000000001,"5min":1.018,"15min":0.84199999999999997},"max":{"1min":1.796,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.492},{"interface":"front","average":{"1min":1.514,"5min":1.512,"15min":1.49},"min":{"1min":0.99399999999999999,"5min":0.90900000000000003,"15min":0.79500000000000004},"max":{"1min":1.7609999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.8300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312184,"kb_used_data":2608,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059656,"statfs":{"total":96636764160,"available":96317087744,"internally_reserved":0,"allocated":2670592,"data_stored":1434469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.353},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.369}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":0.93899999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.014},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":0.747},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.149}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.2050000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.276},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.2949999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.4330000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575972,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312468,"kb_used_data":2892,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059372,"statfs":{"total":96636764160,"available":96316796928,"internally_reserved":0,"allocated":2961408,"data_stored":1737525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88800000000000001,"15min":0.878},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.254,"5min":1.581,"15min":1.581},"last":0.996},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.96299999999999997,"15min":0.98799999999999999},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.494,"5min":2.117,"15min":2.117},"last":1.361}]},{"osd":1,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.92200000000000004,"15min":0.93500000000000005},"min":{"1min":0.54800000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1579999999999999,"5min":2.331,"15min":2.331},"last":1.206},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0529999999999999},"min":{"1min":0.69099999999999995,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.5649999999999999,"5min":2.23,"15min":2.23},"last":1.1299999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.103,"15min":1.1200000000000001},"min":{"1min":0.69499999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.677,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.175},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1799999999999999,"15min":1.1919999999999999},"min":{"1min":0.89000000000000001,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.7190000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.329}]},{"osd":4,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.331,"15min":1.3200000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.70199999999999996},"max":{"1min":1.8779999999999999,"5min":2.157,"15min":2.157},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.2909999999999999},"min":{"1min":1.073,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":1.9390000000000001,"5min":2.149,"15min":2.149},"last":1.5700000000000001}]},{"osd":5,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3220000000000001,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1400000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.407,"15min":1.3879999999999999},"min":{"1min":1.0529999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0409999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.4119999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.413,"15min":1.423},"min":{"1min":1.1279999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.923,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.49},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4079999999999999,"15min":1.4239999999999999},"min":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.8520000000000001,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.524}]},{"osd":7,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.5009999999999999},"min":{"1min":1.1419999999999999,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":1.992,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.4710000000000001,"15min":1.456},"min":{"1min":1.0620000000000001,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0489999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.778}]}]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312464,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059376,"statfs":{"total":96636764160,"available":96316801024,"internally_reserved":0,"allocated":2957312,"data_stored":1737468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2529999999999999,"5min":1.345,"15min":1.5049999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.286,"5min":1.556,"15min":1.6759999999999999},"last":1.01}]},{"osd":1,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3640000000000001,"5min":1.569,"15min":1.7609999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.034,"15min":1.044},"min":{"1min":0.59499999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4339999999999999,"5min":1.5369999999999999,"15min":1.768},"last":0.91300000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97399999999999998,"15min":0.97799999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.472,"5min":1.53,"15min":1.891},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.121,"5min":1.1359999999999999,"15min":1.1499999999999999},"min":{"1min":0.67700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":0.99299999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3220000000000001,"15min":1.341},"min":{"1min":0.76900000000000002,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.754},"last":1.095},{"interface":"front","average":{"1min":1.242,"5min":1.3169999999999999,"15min":1.3300000000000001},"min":{"1min":0.878,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.556,"5min":1.76,"15min":1.8520000000000001},"last":1.2929999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3759999999999999},"min":{"1min":0.84799999999999998,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6679999999999999,"5min":1.948,"15min":2.016},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.294,"15min":1.3100000000000001},"min":{"1min":0.91200000000000003,"5min":0.83399999999999996,"15min":0.68300000000000005},"max":{"1min":1.6000000000000001,"5min":1.901,"15min":1.927},"last":1.052}]},{"osd":6,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3120000000000001,"15min":1.345},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.8200000000000001,"5min":2.0339999999999998,"15min":2.2189999999999999},"last":1.181},{"interface":"front","average":{"1min":1.379,"5min":1.365,"15min":1.419},"min":{"1min":1.036,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.157}]},{"osd":7,"last update":"Tue Dec 19 09:21:01 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.4059999999999999,"15min":1.4370000000000001},"min":{"1min":0.94799999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.0489999999999999,"15min":2.327},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9059999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.379}]}]},{"osd":1,"up_from":13,"seq":55834575973,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312480,"kb_used_data":2904,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059360,"statfs":{"total":96636764160,"available":96316784640,"internally_reserved":0,"allocated":2973696,"data_stored":1752076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.87},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.9039999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.6850000000000001},"last":1.1459999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.996,"5min":1.0209999999999999,"15min":1.0629999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":2.302},"last":0.93500000000000005}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.175},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.04},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.2450000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.069},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2929999999999999,"15min":1.3220000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.546,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":0.998}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.296,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.899,"5min":1.944,"15min":1.944},"last":1.425},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.536}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.877,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.95299999999999996,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.752,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.5680000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.825,"5min":1.825,"15min":2.069},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4259999999999999,"15min":1.4359999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.456}]}]},{"osd":0,"up_from":13,"seq":55834575972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300464,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071376,"statfs":{"total":96636764160,"available":96329089024,"internally_reserved":0,"allocated":3252224,"data_stored":2014453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.97099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3069999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.995,"15min":1.0169999999999999},"min":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3759999999999999,"5min":1.446,"15min":1.5269999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.728,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2709999999999999,"5min":1.611,"15min":2.621},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.177},"min":{"1min":0.61599999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.0029999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.341,"15min":1.377},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.605,"5min":1.891,"15min":1.891},"last":1.232},{"interface":"front","average":{"1min":1.341,"5min":1.333,"15min":1.347},"min":{"1min":0.81999999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.5339999999999998},"last":0.90600000000000003}]},{"osd":5,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3759999999999999,"15min":1.3939999999999999},"min":{"1min":0.74299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.966,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.466}]},{"osd":6,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2869999999999999,"15min":1.2849999999999999},"min":{"1min":0.88800000000000001,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.4489999999999998},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.373},"min":{"1min":0.98299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.837,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.425}]},{"osd":7,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.339,"15min":1.3400000000000001},"min":{"1min":0.86399999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.746,"5min":2.137,"15min":2.8570000000000002},"last":1.097},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4239999999999999,"15min":1.4299999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0640000000000001,"5min":2.1259999999999999,"15min":2.181},"last":1.3069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:22:04.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T09:22:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.382+0000 7ff10759e700 1 Processor -- start 2023-12-19T09:22:04.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.382+0000 7ff10759e700 1 -- start start 2023-12-19T09:22:04.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff10759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff108069f50 0x7ff108161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff10759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1080abd00 con 0x7ff1080b1080 2023-12-19T09:22:04.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff10759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1080abfe0 con 0x7ff1080b0cb0 2023-12-19T09:22:04.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff10759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1080abb10 con 0x7ff108069f50 2023-12-19T09:22:04.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff0fdd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff1080b1080 0x7ff108069840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50808/0 (socket says 172.21.15.111:50808) 2023-12-19T09:22:04.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff0fdd9b700 1 -- 172.21.15.111:0/2770684747 learned_addr learned my addr 172.21.15.111:0/2770684747 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:04.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff106d9d700 1 --2- 172.21.15.111:0/2770684747 >> v2:172.21.15.143:3300/0 conn(0x7ff108069f50 0x7ff108161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff105d9b700 1 -- 172.21.15.111:0/2770684747 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204783088 0 0) 0x7ff1080abd00 con 0x7ff1080b1080 2023-12-19T09:22:04.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff106d9d700 1 -- 172.21.15.111:0/2770684747 >> v1:172.21.15.143:6789/0 conn(0x7ff1080b0cb0 legacy=0x7ff10806a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff106d9d700 1 -- 172.21.15.111:0/2770684747 >> v1:172.21.15.111:6789/0 conn(0x7ff1080b1080 legacy=0x7ff108069840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff106d9d700 1 -- 172.21.15.111:0/2770684747 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080ac2c0 con 0x7ff108069f50 2023-12-19T09:22:04.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.386+0000 7ff106d9d700 1 --2- 172.21.15.111:0/2770684747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff108161cd0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff100009e60 tx=0x7ff100009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd52f57b6d9c9bb6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff105d9b700 1 -- 172.21.15.111:0/2770684747 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10000e070 con 0x7ff108069f50 2023-12-19T09:22:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff105d9b700 1 -- 172.21.15.111:0/2770684747 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10000bc30 con 0x7ff108069f50 2023-12-19T09:22:04.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff105d9b700 1 -- 172.21.15.111:0/2770684747 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100012430 con 0x7ff108069f50 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- 172.21.15.111:0/2770684747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 msgr2=0x7ff108161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 --2- 172.21.15.111:0/2770684747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff108161cd0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ff100009e60 tx=0x7ff100009780 comp rx=0 tx=0).stop 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- 172.21.15.111:0/2770684747 shutdown_connections 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 --2- 172.21.15.111:0/2770684747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff108161cd0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- 172.21.15.111:0/2770684747 >> 172.21.15.111:0/2770684747 conn(0x7ff1080cc530 msgr2=0x7ff1080c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- 172.21.15.111:0/2770684747 shutdown_connections 2023-12-19T09:22:04.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- 172.21.15.111:0/2770684747 wait complete. 2023-12-19T09:22:04.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 Processor -- start 2023-12-19T09:22:04.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.390+0000 7ff10759e700 1 -- start start 2023-12-19T09:22:04.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff1081500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 0x7ff1080c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1080aa2b0 con 0x7ff1080b1080 2023-12-19T09:22:04.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff108150600 con 0x7ff108069f50 2023-12-19T09:22:04.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1080a96a0 con 0x7ff1080b0cb0 2023-12-19T09:22:04.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55124/0 (socket says 172.21.15.111:55124) 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 -- 172.21.15.111:0/4156192259 learned_addr learned my addr 172.21.15.111:0/4156192259 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10659c700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff1081500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff0fdd9b700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 0x7ff1080c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 -- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 msgr2=0x7ff1080c3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 0x7ff1080c3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 -- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 msgr2=0x7ff1081500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff1081500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 -- 172.21.15.111:0/4156192259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff100009a20 con 0x7ff1080b1080 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff0fdd9b700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 0x7ff1080c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:04.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff106d9d700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ff100013000 tx=0x7ff100019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100012ea0 con 0x7ff1080b1080 2023-12-19T09:22:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff100007000 con 0x7ff1080b1080 2023-12-19T09:22:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1080a9980 con 0x7ff1080b1080 2023-12-19T09:22:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10001b850 con 0x7ff1080b1080 2023-12-19T09:22:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.394+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1080a9f40 con 0x7ff1080b1080 2023-12-19T09:22:04.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.398+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1080c6000 con 0x7ff1080b1080 2023-12-19T09:22:04.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.398+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7ff10001bc00 con 0x7ff1080b1080 2023-12-19T09:22:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.398+0000 7ff0feffd700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 0x7ff0e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.398+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7ff10007ebb0 con 0x7ff1080b1080 2023-12-19T09:22:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.402+0000 7ff10659c700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 0x7ff0e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.402+0000 7ff10659c700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 0x7ff0e8062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8003a80 tx=0x7ff0f8007ec0 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.406+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff1080c6000 con 0x7ff1080b1080 2023-12-19T09:22:04.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.582+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1080c6000 con 0x7ff1080b1080 2023-12-19T09:22:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.586+0000 7ff0feffd700 1 -- 172.21.15.111:0/4156192259 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+14216 (secure 0 0 0) 0x7ff1080c6000 con 0x7ff1080b1080 2023-12-19T09:22:04.592 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:04.592 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":213,"fsid":"4ae666be-184d-45ea-a875-cac9a231c35b","created":"2023-12-19T07:46:53.229589+0000","modified":"2023-12-19T09:21:59.804259+0000","last_up_change":"2023-12-19T07:47:06.074117+0000","last_in_change":"2023-12-19T07:46:56.290869+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T07:47:06.696458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"176b55d9-24fe-41e0-8a08-156ec028a881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":33928},{"type":"v1","addr":"172.21.15.111:6801","nonce":33928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":33928},{"type":"v1","addr":"172.21.15.111:6803","nonce":33928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":33928},{"type":"v1","addr":"172.21.15.111:6807","nonce":33928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":33928},{"type":"v1","addr":"172.21.15.111:6805","nonce":33928}]},"public_addr":"172.21.15.111:6801/33928","cluster_addr":"172.21.15.111:6803/33928","heartbeat_back_addr":"172.21.15.111:6807/33928","heartbeat_front_addr":"172.21.15.111:6805/33928","state":["exists","up"]},{"osd":1,"uuid":"178e68a7-dd99-4b0c-8e58-5d49ee2ff22e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":33932},{"type":"v1","addr":"172.21.15.111:6809","nonce":33932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":33932},{"type":"v1","addr":"172.21.15.111:6811","nonce":33932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":33932},{"type":"v1","addr":"172.21.15.111:6815","nonce":33932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":33932},{"type":"v1","addr":"172.21.15.111:6813","nonce":33932}]},"public_addr":"172.21.15.111:6809/33932","cluster_addr":"172.21.15.111:6811/33932","heartbeat_back_addr":"172.21.15.111:6815/33932","heartbeat_front_addr":"172.21.15.111:6813/33932","state":["exists","up"]},{"osd":2,"uuid":"fb2e19b3-ea94-4568-ba79-b7a7eb32cf64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":33933},{"type":"v1","addr":"172.21.15.111:6825","nonce":33933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":33933},{"type":"v1","addr":"172.21.15.111:6827","nonce":33933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":33933},{"type":"v1","addr":"172.21.15.111:6831","nonce":33933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":33933},{"type":"v1","addr":"172.21.15.111:6829","nonce":33933}]},"public_addr":"172.21.15.111:6825/33933","cluster_addr":"172.21.15.111:6827/33933","heartbeat_back_addr":"172.21.15.111:6831/33933","heartbeat_front_addr":"172.21.15.111:6829/33933","state":["exists","up"]},{"osd":3,"uuid":"ffa7021a-cb8e-40e0-bf4f-29e85ffc9f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":33942},{"type":"v1","addr":"172.21.15.111:6817","nonce":33942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":33942},{"type":"v1","addr":"172.21.15.111:6819","nonce":33942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":33942},{"type":"v1","addr":"172.21.15.111:6823","nonce":33942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":33942},{"type":"v1","addr":"172.21.15.111:6821","nonce":33942}]},"public_addr":"172.21.15.111:6817/33942","cluster_addr":"172.21.15.111:6819/33942","heartbeat_back_addr":"172.21.15.111:6823/33942","heartbeat_front_addr":"172.21.15.111:6821/33942","state":["exists","up"]},{"osd":4,"uuid":"33ab5082-d40a-45c3-a9e5-770ca2530812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":34138},{"type":"v1","addr":"172.21.15.143:6810","nonce":34138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":34138},{"type":"v1","addr":"172.21.15.143:6814","nonce":34138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":34138},{"type":"v1","addr":"172.21.15.143:6822","nonce":34138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6815","nonce":34138},{"type":"v1","addr":"172.21.15.143:6818","nonce":34138}]},"public_addr":"172.21.15.143:6810/34138","cluster_addr":"172.21.15.143:6814/34138","heartbeat_back_addr":"172.21.15.143:6822/34138","heartbeat_front_addr":"172.21.15.143:6818/34138","state":["exists","up"]},{"osd":5,"uuid":"303e543d-2f45-4bb2-9397-6ded49636bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":34137},{"type":"v1","addr":"172.21.15.143:6811","nonce":34137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":34137},{"type":"v1","addr":"172.21.15.143:6816","nonce":34137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":34137},{"type":"v1","addr":"172.21.15.143:6823","nonce":34137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":34137},{"type":"v1","addr":"172.21.15.143:6819","nonce":34137}]},"public_addr":"172.21.15.143:6811/34137","cluster_addr":"172.21.15.143:6816/34137","heartbeat_back_addr":"172.21.15.143:6823/34137","heartbeat_front_addr":"172.21.15.143:6819/34137","state":["exists","up"]},{"osd":6,"uuid":"8d84908d-e5c3-455b-b350-f80681f1cf24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":34136},{"type":"v1","addr":"172.21.15.143:6801","nonce":34136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":34136},{"type":"v1","addr":"172.21.15.143:6803","nonce":34136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":34136},{"type":"v1","addr":"172.21.15.143:6807","nonce":34136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":34136},{"type":"v1","addr":"172.21.15.143:6805","nonce":34136}]},"public_addr":"172.21.15.143:6801/34136","cluster_addr":"172.21.15.143:6803/34136","heartbeat_back_addr":"172.21.15.143:6807/34136","heartbeat_front_addr":"172.21.15.143:6805/34136","state":["exists","up"]},{"osd":7,"uuid":"0f7bf890-21b8-40bd-94c4-73967d7a4f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":34139},{"type":"v1","addr":"172.21.15.143:6825","nonce":34139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":34139},{"type":"v1","addr":"172.21.15.143:6827","nonce":34139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":34139},{"type":"v1","addr":"172.21.15.143:6831","nonce":34139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":34139},{"type":"v1","addr":"172.21.15.143:6829","nonce":34139}]},"public_addr":"172.21.15.143:6825/34139","cluster_addr":"172.21.15.143:6827/34139","heartbeat_back_addr":"172.21.15.143:6831/34139","heartbeat_front_addr":"172.21.15.143:6829/34139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.160768+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.044407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.200772+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.323249+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.012350+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.978414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:01.899564+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T07:47:02.057798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.143:6837/68388372":"2023-12-20T09:21:56.484688+0000","172.21.15.143:6836/68388372":"2023-12-20T09:21:56.484688+0000","172.21.15.143:0/3433885932":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3697151201":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2906865886":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3049351864":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2337999510":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/2085760645":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/3085442911":"2023-12-20T09:21:56.462250+0000","172.21.15.143:0/278966201":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/2861944468":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6833/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:6832/43868":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/150889021":"2023-12-20T09:21:38.103524+0000","172.21.15.143:0/800113432":"2023-12-20T09:21:38.103524+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":11,"length":1},{"begin":17,"length":1},{"begin":20,"length":2},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":30,"length":1},{"begin":33,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":46,"length":1},{"begin":49,"length":1},{"begin":51,"length":1},{"begin":53,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":65,"length":2},{"begin":70,"length":1},{"begin":74,"length":1},{"begin":78,"length":2},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":89,"length":1},{"begin":95,"length":1},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":102,"length":1},{"begin":106,"length":2},{"begin":110,"length":1},{"begin":188,"length":34}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":22,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T09:22:04.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 msgr2=0x7ff0e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 0x7ff0e8062a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8003a80 tx=0x7ff0f8007ec0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 msgr2=0x7ff1080c5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ff100013000 tx=0x7ff100019000 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 shutdown_connections 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1080b1080 0x7ff1080c5080 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1080b0cb0 0x7ff1080c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7ff0e8060550 0x7ff0e8062a00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 --2- 172.21.15.111:0/4156192259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff108069f50 0x7ff1081500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 >> 172.21.15.111:0/4156192259 conn(0x7ff1080cc530 msgr2=0x7ff108159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 shutdown_connections 2023-12-19T09:22:04.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:04.590+0000 7ff10759e700 1 -- 172.21.15.111:0/4156192259 wait complete. 2023-12-19T09:22:05.618 INFO:tasks.ceph:Scrubbing osd.0 2023-12-19T09:22:05.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:05.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.786+0000 7f10d3145700 1 Processor -- start 2023-12-19T09:22:05.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.786+0000 7f10d3145700 1 -- start start 2023-12-19T09:22:05.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d3145700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10cc159030 0x7f10cc0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d3145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc159400 0x7f10cc068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d3145700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc067da0 con 0x7f10cc068770 2023-12-19T09:22:05.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d3145700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc067190 con 0x7f10cc159030 2023-12-19T09:22:05.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d3145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc067470 con 0x7f10cc159400 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d16e2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f10cc068770 0x7f10cc0ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50814/0 (socket says 172.21.15.111:50814) 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d16e2700 1 -- 172.21.15.111:0/2913908785 learned_addr learned my addr 172.21.15.111:0/2913908785 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/2913908785 >> v2:172.21.15.111:3300/0 conn(0x7f10cc159030 0x7f10cc0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10cbfff700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc159400 0x7f10cc068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10cb7fe700 1 -- 172.21.15.111:0/2913908785 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807557941 0 0) 0x7f10cc067da0 con 0x7f10cc068770 2023-12-19T09:22:05.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10cb7fe700 1 -- 172.21.15.111:0/2913908785 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10bc003570 con 0x7f10cc068770 2023-12-19T09:22:05.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d0ee1700 1 -- 172.21.15.111:0/2913908785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc159400 msgr2=0x7f10cc068230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:05.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc159400 0x7f10cc068230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d0ee1700 1 -- 172.21.15.111:0/2913908785 >> v1:172.21.15.111:6789/0 conn(0x7f10cc068770 legacy=0x7f10cc0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.790+0000 7f10d0ee1700 1 -- 172.21.15.111:0/2913908785 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc067750 con 0x7f10cc159030 2023-12-19T09:22:05.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0cb170 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098e0 tx=0x7f10c00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2074377559f98b68 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:05.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10cb7fe700 1 -- 172.21.15.111:0/2913908785 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0011630 con 0x7f10cc159030 2023-12-19T09:22:05.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10cb7fe700 1 -- 172.21.15.111:0/2913908785 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0011790 con 0x7f10cc159030 2023-12-19T09:22:05.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10cb7fe700 1 -- 172.21.15.111:0/2913908785 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0011b10 con 0x7f10cc159030 2023-12-19T09:22:05.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 -- 172.21.15.111:0/2913908785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 msgr2=0x7f10cc0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0cb170 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098e0 tx=0x7f10c00091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 -- 172.21.15.111:0/2913908785 shutdown_connections 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc159400 0x7f10cc068230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 --2- 172.21.15.111:0/2913908785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0cb170 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 -- 172.21.15.111:0/2913908785 >> 172.21.15.111:0/2913908785 conn(0x7f10cc0d0170 msgr2=0x7f10cc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 -- 172.21.15.111:0/2913908785 shutdown_connections 2023-12-19T09:22:05.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 -- 172.21.15.111:0/2913908785 wait complete. 2023-12-19T09:22:05.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.794+0000 7f10d3145700 1 Processor -- start 2023-12-19T09:22:05.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 -- start start 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc068770 0x7f10cc169860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10cc159400 0x7f10cc14d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0b35f0 con 0x7f10cc159030 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0c8980 con 0x7f10cc159400 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d3145700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc14ded0 con 0x7f10cc068770 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55138/0 (socket says 172.21.15.111:55138) 2023-12-19T09:22:05.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 -- 172.21.15.111:0/5976387 learned_addr learned my addr 172.21.15.111:0/5976387 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:05.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d16e2700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10cc159400 0x7f10cc14d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc068770 0x7f10cc169860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 -- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc068770 msgr2=0x7f10cc169860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc068770 0x7f10cc169860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 -- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10cc159400 msgr2=0x7f10cc14d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10cc159400 0x7f10cc14d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0009420 con 0x7f10cc159030 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10cc068770 0x7f10cc169860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10d16e2700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10cc159400 0x7f10cc14d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.798+0000 7f10cbfff700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0c8440 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e930 tx=0x7f10b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b800eed0 con 0x7f10cc159030 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800c850 con 0x7f10cc159030 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b800cbf0 con 0x7f10cc159030 2023-12-19T09:22:05.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc16d690 con 0x7f10cc159030 2023-12-19T09:22:05.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc0b3280 con 0x7f10cc159030 2023-12-19T09:22:05.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10cc0708a0 con 0x7f10cc159030 2023-12-19T09:22:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.802+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f10b8013470 con 0x7f10cc159030 2023-12-19T09:22:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f10b0060550 0x7f10b0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f10b807ccf0 con 0x7f10cc159030 2023-12-19T09:22:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f10b0067090 0x7f10b00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10b0069b20 con 0x7f10b0067090 2023-12-19T09:22:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10b807f250 con 0x7f10cc159030 2023-12-19T09:22:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f10b0060550 0x7f10b0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10d16e2700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f10b0067090 0x7f10b00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:05.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10d0ee1700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f10b0060550 0x7f10b0062a00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098e0 tx=0x7f10c0017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:05.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10d16e2700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f10b0067090 0x7f10b00694f0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:05.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.806+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== osd.0 v2:172.21.15.111:6800/33928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f10b0069b20 con 0x7f10b0067090 2023-12-19T09:22:05.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 --> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f10cc067a30 con 0x7f10b0067090 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10c8ff9700 1 -- 172.21.15.111:0/5976387 <== osd.0 v2:172.21.15.111:6800/33928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f10cc067a30 con 0x7f10b0067090 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f10b0067090 msgr2=0x7f10b00694f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:6800/33928,v1:172.21.15.111:6801/33928] conn(0x7f10b0067090 0x7f10b00694f0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f10b0060550 msgr2=0x7f10b0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f10b0060550 0x7f10b0062a00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098e0 tx=0x7f10c0017000 comp rx=0 tx=0).stop 2023-12-19T09:22:05.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 msgr2=0x7f10cc0c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:05.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 --2- 172.21.15.111:0/5976387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10cc159030 0x7f10cc0c8440 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e930 tx=0x7f10b80091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:05.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d16e2700 1 -- 172.21.15.111:0/5976387 reap_dead start 2023-12-19T09:22:05.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 shutdown_connections 2023-12-19T09:22:05.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 >> 172.21.15.111:0/5976387 conn(0x7f10cc0d0170 msgr2=0x7f10cc0aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:05.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 shutdown_connections 2023-12-19T09:22:05.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:05.826+0000 7f10d3145700 1 -- 172.21.15.111:0/5976387 wait complete. 2023-12-19T09:22:05.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-19T09:22:06.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 Processor -- start 2023-12-19T09:22:06.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 -- start start 2023-12-19T09:22:06.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f83e013f860 0x7f83e00c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e00c7ee0 0x7f83e0067990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00ca820 con 0x7f83e013f490 2023-12-19T09:22:06.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e0067500 con 0x7f83e013f860 2023-12-19T09:22:06.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.002+0000 7f83e53c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e00ca260 con 0x7f83e00c7ee0 2023-12-19T09:22:06.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83deffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f83e013f490 0x7f83e00c97c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50822/0 (socket says 172.21.15.111:50822) 2023-12-19T09:22:06.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83deffd700 1 -- 172.21.15.111:0/3429876843 learned_addr learned my addr 172.21.15.111:0/3429876843 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:06.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3429876843 >> v2:172.21.15.143:3300/0 conn(0x7f83e013f860 0x7f83e00c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83df7fe700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e00c7ee0 0x7f83e0067990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 -- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e00c7ee0 msgr2=0x7f83e0067990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:06.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e00c7ee0 0x7f83e0067990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83ddffb700 1 -- 172.21.15.111:0/3429876843 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295833678 0 0) 0x7f83e00ca820 con 0x7f83e013f490 2023-12-19T09:22:06.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 -- 172.21.15.111:0/3429876843 >> v1:172.21.15.111:6789/0 conn(0x7f83e013f490 legacy=0x7f83e00c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 -- 172.21.15.111:0/3429876843 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e00ca540 con 0x7f83e013f860 2023-12-19T09:22:06.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e00c79a0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f83d40098e0 tx=0x7f83d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d8879425fac91 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83ddffb700 1 -- 172.21.15.111:0/3429876843 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400b640 con 0x7f83e013f860 2023-12-19T09:22:06.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83ddffb700 1 -- 172.21.15.111:0/3429876843 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d400b7a0 con 0x7f83e013f860 2023-12-19T09:22:06.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83ddffb700 1 -- 172.21.15.111:0/3429876843 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400bb40 con 0x7f83e013f860 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83e53c5700 1 -- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 msgr2=0x7f83e00c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.006+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e00c79a0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f83d40098e0 tx=0x7f83d40091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 -- 172.21.15.111:0/3429876843 shutdown_connections 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e00c7ee0 0x7f83e0067990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3429876843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e00c79a0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 -- 172.21.15.111:0/3429876843 >> 172.21.15.111:0/3429876843 conn(0x7f83e00cc6c0 msgr2=0x7f83e00c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 -- 172.21.15.111:0/3429876843 shutdown_connections 2023-12-19T09:22:06.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 -- 172.21.15.111:0/3429876843 wait complete. 2023-12-19T09:22:06.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 Processor -- start 2023-12-19T09:22:06.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.010+0000 7f83e53c5700 1 -- start start 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e013f490 0x7f83e0159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e0155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e01566b0 con 0x7f83e00c7ee0 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0157a70 con 0x7f83e013f860 2023-12-19T09:22:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e015ef60 con 0x7f83e013f490 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55140/0 (socket says 172.21.15.111:55140) 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 -- 172.21.15.111:0/3768965102 learned_addr learned my addr 172.21.15.111:0/3768965102 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83df7fe700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e0155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e013f490 0x7f83e0159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 -- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e013f490 msgr2=0x7f83e0159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e013f490 0x7f83e0159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 -- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 msgr2=0x7f83e0155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e0155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 -- 172.21.15.111:0/3768965102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d4009420 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83deffd700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c5b0 tx=0x7f83d0001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0005b30 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d0005c90 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e0158010 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0007300 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.014+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e0157ca0 con 0x7f83e00c7ee0 2023-12-19T09:22:06.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.018+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e0071110 con 0x7f83e00c7ee0 2023-12-19T09:22:06.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.022+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f83d001b3b0 con 0x7f83e00c7ee0 2023-12-19T09:22:06.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.022+0000 7f83c6ffd700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 0x7f83cc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.022+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f83d007bff0 con 0x7f83e00c7ee0 2023-12-19T09:22:06.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.026+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 0x7f83cc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.026+0000 7f83de7fc700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 0x7f83cc062a00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f83d40093f0 tx=0x7f83d4017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.030+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f83d004c2b0 con 0x7f83e00c7ee0 2023-12-19T09:22:06.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.198+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e0067190 con 0x7f83cc060550 2023-12-19T09:22:06.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.198+0000 7f83c6ffd700 1 -- 172.21.15.111:0/3768965102 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83e0067190 con 0x7f83cc060550 2023-12-19T09:22:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 msgr2=0x7f83cc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 0x7f83cc062a00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f83d40093f0 tx=0x7f83d4017000 comp rx=0 tx=0).stop 2023-12-19T09:22:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 msgr2=0x7f83e015e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c5b0 tx=0x7f83d0001070 comp rx=0 tx=0).stop 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 shutdown_connections 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f83cc060550 0x7f83cc062a00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83e013f860 0x7f83e0155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83e013f490 0x7f83e0159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 --2- 172.21.15.111:0/3768965102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e00c7ee0 0x7f83e015e150 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 >> 172.21.15.111:0/3768965102 conn(0x7f83e00cc6c0 msgr2=0x7f83e0136730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 shutdown_connections 2023-12-19T09:22:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.202+0000 7f83e53c5700 1 -- 172.21.15.111:0/3768965102 wait complete. 2023-12-19T09:22:06.211 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 0 to deep-scrub 2023-12-19T09:22:06.231 INFO:tasks.ceph:Scrubbing osd.1 2023-12-19T09:22:06.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:06.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.398+0000 7fbf5326e700 1 Processor -- start 2023-12-19T09:22:06.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 -- start start 2023-12-19T09:22:06.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 0x7fbf4c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbf4c0c9de0 0x7fbf4c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4c0c7460 con 0x7fbf4c155f70 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4c0cb150 con 0x7fbf4c0c9de0 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5326e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0ca200 con 0x7fbf4c0b2840 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5100a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbf4c155f70 0x7fbf4c0cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:54432/0 (socket says 172.21.15.111:54432) 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5100a700 1 -- 172.21.15.111:0/3833875308 learned_addr learned my addr 172.21.15.111:0/3833875308 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:06.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf5180b700 1 --2- 172.21.15.111:0/3833875308 >> v2:172.21.15.111:3300/0 conn(0x7fbf4c0c9de0 0x7fbf4c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.402+0000 7fbf50809700 1 --2- 172.21.15.111:0/3833875308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 0x7fbf4c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf43fff700 1 -- 172.21.15.111:0/3833875308 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611456565 0 0) 0x7fbf4c0c7460 con 0x7fbf4c155f70 2023-12-19T09:22:06.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf43fff700 1 -- 172.21.15.111:0/3833875308 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38003570 con 0x7fbf4c155f70 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf50809700 1 -- 172.21.15.111:0/3833875308 >> v2:172.21.15.111:3300/0 conn(0x7fbf4c0c9de0 msgr2=0x7fbf4c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf50809700 1 --2- 172.21.15.111:0/3833875308 >> v2:172.21.15.111:3300/0 conn(0x7fbf4c0c9de0 0x7fbf4c159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf50809700 1 -- 172.21.15.111:0/3833875308 >> v1:172.21.15.143:6789/0 conn(0x7fbf4c155f70 legacy=0x7fbf4c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf50809700 1 -- 172.21.15.111:0/3833875308 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0b0b60 con 0x7fbf4c0b2840 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf50809700 1 --2- 172.21.15.111:0/3833875308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 0x7fbf4c0c98a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098e0 tx=0x7fbf3c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ddf6e8698267f53 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf43fff700 1 -- 172.21.15.111:0/3833875308 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c011630 con 0x7fbf4c0b2840 2023-12-19T09:22:06.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf43fff700 1 -- 172.21.15.111:0/3833875308 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c011790 con 0x7fbf4c0b2840 2023-12-19T09:22:06.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf43fff700 1 -- 172.21.15.111:0/3833875308 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c011b30 con 0x7fbf4c0b2840 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 -- 172.21.15.111:0/3833875308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 msgr2=0x7fbf4c0c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 --2- 172.21.15.111:0/3833875308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 0x7fbf4c0c98a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098e0 tx=0x7fbf3c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 -- 172.21.15.111:0/3833875308 shutdown_connections 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 --2- 172.21.15.111:0/3833875308 >> v2:172.21.15.111:3300/0 conn(0x7fbf4c0c9de0 0x7fbf4c159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 --2- 172.21.15.111:0/3833875308 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c0b2840 0x7fbf4c0c98a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 -- 172.21.15.111:0/3833875308 >> 172.21.15.111:0/3833875308 conn(0x7fbf4c096190 msgr2=0x7fbf4c0afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 -- 172.21.15.111:0/3833875308 shutdown_connections 2023-12-19T09:22:06.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.406+0000 7fbf5326e700 1 -- 172.21.15.111:0/3833875308 wait complete. 2023-12-19T09:22:06.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 Processor -- start 2023-12-19T09:22:06.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 -- start start 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 0x7fbf4c16a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf4c0c9de0 0x7fbf4c1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c155f70 0x7fbf4c06c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c096c30 con 0x7fbf4c0b2840 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c148930 con 0x7fbf4c0c9de0 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5326e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c16b360 con 0x7fbf4c155f70 2023-12-19T09:22:06.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 0x7fbf4c16a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 0x7fbf4c16a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55152/0 (socket says 172.21.15.111:55152) 2023-12-19T09:22:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 -- 172.21.15.111:0/49053718 learned_addr learned my addr 172.21.15.111:0/49053718 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5180b700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c155f70 0x7fbf4c06c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf50809700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf4c0c9de0 0x7fbf4c1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 -- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c155f70 msgr2=0x7fbf4c06c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf4c155f70 0x7fbf4c06c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 -- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf4c0c9de0 msgr2=0x7fbf4c1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf4c0c9de0 0x7fbf4c1483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c009420 con 0x7fbf4c0b2840 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.410+0000 7fbf5100a700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 0x7fbf4c16a4e0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c790 tx=0x7fbf48007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48005c50 con 0x7fbf4c0b2840 2023-12-19T09:22:06.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf48005db0 con 0x7fbf4c0b2840 2023-12-19T09:22:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c16aac0 con 0x7fbf4c0b2840 2023-12-19T09:22:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48007320 con 0x7fbf4c0b2840 2023-12-19T09:22:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c16b080 con 0x7fbf4c0b2840 2023-12-19T09:22:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.414+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf4c0708a0 con 0x7fbf4c0b2840 2023-12-19T09:22:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fbf480076d0 con 0x7fbf4c0b2840 2023-12-19T09:22:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbf30060550 0x7fbf30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fbf4807bf60 con 0x7fbf4c0b2840 2023-12-19T09:22:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fbf30067200 0x7fbf30069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf30069c90 con 0x7fbf30067200 2023-12-19T09:22:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf4807e4c0 con 0x7fbf4c0b2840 2023-12-19T09:22:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf50809700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbf30060550 0x7fbf30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf5180b700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fbf30067200 0x7fbf30069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf5180b700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fbf30067200 0x7fbf30069660 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.418+0000 7fbf50809700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbf30060550 0x7fbf30062a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0093f0 tx=0x7fbf3c017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.422+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== osd.1 v2:172.21.15.111:6808/33932 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbf30069c90 con 0x7fbf30067200 2023-12-19T09:22:06.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 --> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbf4c0cab90 con 0x7fbf30067200 2023-12-19T09:22:06.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf417fa700 1 -- 172.21.15.111:0/49053718 <== osd.1 v2:172.21.15.111:6808/33932 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fbf4c0cab90 con 0x7fbf30067200 2023-12-19T09:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fbf30067200 msgr2=0x7fbf30069660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:6808/33932,v1:172.21.15.111:6809/33932] conn(0x7fbf30067200 0x7fbf30069660 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbf30060550 msgr2=0x7fbf30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fbf30060550 0x7fbf30062a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0093f0 tx=0x7fbf3c017000 comp rx=0 tx=0).stop 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 msgr2=0x7fbf4c16a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 --2- 172.21.15.111:0/49053718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf4c0b2840 0x7fbf4c16a4e0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c790 tx=0x7fbf48007de0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5180b700 1 -- 172.21.15.111:0/49053718 reap_dead start 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 shutdown_connections 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 >> 172.21.15.111:0/49053718 conn(0x7fbf4c096190 msgr2=0x7fbf4c1553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 shutdown_connections 2023-12-19T09:22:06.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.438+0000 7fbf5326e700 1 -- 172.21.15.111:0/49053718 wait complete. 2023-12-19T09:22:06.469 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-19T09:22:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.622+0000 7f47e386e700 1 Processor -- start 2023-12-19T09:22:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 -- start start 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47dc156fc0 0x7f47dc1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 0x7f47dc157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc069d40 con 0x7f47dc15a9d0 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc157a20 con 0x7f47dc156fc0 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e386e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc157d00 con 0x7f47dc0c7350 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47dc156fc0 0x7f47dc1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47dc156fc0 0x7f47dc1582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55164/0 (socket says 172.21.15.111:55164) 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 -- 172.21.15.111:0/1168281304 learned_addr learned my addr 172.21.15.111:0/1168281304 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:06.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e0e09700 1 --1- 172.21.15.111:0/1168281304 >> v1:172.21.15.111:6789/0 conn(0x7f47dc15a9d0 0x7f47dc0c6c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50828/0 (socket says 172.21.15.111:50828) 2023-12-19T09:22:06.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 -- 172.21.15.111:0/1168281304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 msgr2=0x7f47dc157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:06.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e1e0b700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 0x7f47dc157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 0x7f47dc157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 -- 172.21.15.111:0/1168281304 >> v1:172.21.15.111:6789/0 conn(0x7f47dc15a9d0 legacy=0x7f47dc0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 -- 172.21.15.111:0/1168281304 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc157fe0 con 0x7f47dc156fc0 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e1e0b700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 0x7f47dc157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.626+0000 7f47e160a700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc156fc0 0x7f47dc1582c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f47d80099b0 tx=0x7f47d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46bef0ac2882e61 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47d3fff700 1 -- 172.21.15.111:0/1168281304 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d800b860 con 0x7f47dc156fc0 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47d3fff700 1 -- 172.21.15.111:0/1168281304 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d800b9c0 con 0x7f47dc156fc0 2023-12-19T09:22:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47d3fff700 1 -- 172.21.15.111:0/1168281304 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d800bd60 con 0x7f47dc156fc0 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 -- 172.21.15.111:0/1168281304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc156fc0 msgr2=0x7f47dc1582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc156fc0 0x7f47dc1582c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f47d80099b0 tx=0x7f47d8009250 comp rx=0 tx=0).stop 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 -- 172.21.15.111:0/1168281304 shutdown_connections 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc0c7350 0x7f47dc157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 --2- 172.21.15.111:0/1168281304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc156fc0 0x7f47dc1582c0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 -- 172.21.15.111:0/1168281304 >> 172.21.15.111:0/1168281304 conn(0x7f47dc0cc6c0 msgr2=0x7f47dc0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 -- 172.21.15.111:0/1168281304 shutdown_connections 2023-12-19T09:22:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 -- 172.21.15.111:0/1168281304 wait complete. 2023-12-19T09:22:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.630+0000 7f47e386e700 1 Processor -- start 2023-12-19T09:22:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 -- start start 2023-12-19T09:22:06.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47dc0c7350 0x7f47dc152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc156fc0 0x7f47dc0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc152da0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc1540e0 con 0x7f47dc0c7350 2023-12-19T09:22:06.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e386e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc1534d0 con 0x7f47dc156fc0 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55174/0 (socket says 172.21.15.111:55174) 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 -- 172.21.15.111:0/3365751851 learned_addr learned my addr 172.21.15.111:0/3365751851 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e160a700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47dc0c7350 0x7f47dc152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e0e09700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc156fc0 0x7f47dc0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 -- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc156fc0 msgr2=0x7f47dc0c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc156fc0 0x7f47dc0c5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 -- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47dc0c7350 msgr2=0x7f47dc152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47dc0c7350 0x7f47dc152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 -- 172.21.15.111:0/3365751851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80094f0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.634+0000 7f47e1e0b700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f47d400cef0 tx=0x7f47d4007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d400ae30 con 0x7f47dc15a9d0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d4016420 con 0x7f47dc15a9d0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d40167e0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc1537b0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc153d70 con 0x7f47dc15a9d0 2023-12-19T09:22:06.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc154290 con 0x7f47dc15a9d0 2023-12-19T09:22:06.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f47d4016bd0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.638+0000 7f47d17fa700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 0x7f47c0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.642+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f47d4012070 con 0x7f47dc15a9d0 2023-12-19T09:22:06.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.642+0000 7f47e160a700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 0x7f47c0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:06.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.642+0000 7f47e160a700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 0x7f47c0062a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f47d80099b0 tx=0x7f47d8017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:06.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.646+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f47d404cbe0 con 0x7f47dc15a9d0 2023-12-19T09:22:06.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.818+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f47dc071110 con 0x7f47c0060550 2023-12-19T09:22:06.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.818+0000 7f47d17fa700 1 -- 172.21.15.111:0/3365751851 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f47dc071110 con 0x7f47c0060550 2023-12-19T09:22:06.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 msgr2=0x7f47c0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 0x7f47c0062a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f47d80099b0 tx=0x7f47d8017000 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 msgr2=0x7f47dc0b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f47d400cef0 tx=0x7f47d4007830 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 shutdown_connections 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47dc15a9d0 0x7f47dc0b0b60 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47dc156fc0 0x7f47dc0c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f47c0060550 0x7f47c0062a00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 --2- 172.21.15.111:0/3365751851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47dc0c7350 0x7f47dc152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 >> 172.21.15.111:0/3365751851 conn(0x7f47dc0cc6c0 msgr2=0x7f47dc06a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 shutdown_connections 2023-12-19T09:22:06.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:06.822+0000 7f47e386e700 1 -- 172.21.15.111:0/3365751851 wait complete. 2023-12-19T09:22:06.833 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 1 to deep-scrub 2023-12-19T09:22:06.853 INFO:tasks.ceph:Scrubbing osd.2 2023-12-19T09:22:06.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.010+0000 7fe15f86e700 1 Processor -- start 2023-12-19T09:22:07.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.010+0000 7fe15f86e700 1 -- start start 2023-12-19T09:22:07.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15f86e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe158155790 0x7fe1581598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15f86e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15806ac30 con 0x7fe1580c7be0 2023-12-19T09:22:07.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15f86e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe158159ef0 con 0x7fe1581553c0 2023-12-19T09:22:07.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15f86e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15815a1d0 con 0x7fe158155790 2023-12-19T09:22:07.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe158155790 0x7fe1581598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe158155790 0x7fe1581598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55186/0 (socket says 172.21.15.111:55186) 2023-12-19T09:22:07.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 -- 172.21.15.111:0/35673515 learned_addr learned my addr 172.21.15.111:0/35673515 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 -- 172.21.15.111:0/35673515 >> v1:172.21.15.143:6789/0 conn(0x7fe1581553c0 legacy=0x7fe1580c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 -- 172.21.15.111:0/35673515 >> v1:172.21.15.111:6789/0 conn(0x7fe1580c7be0 legacy=0x7fe1581591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 -- 172.21.15.111:0/35673515 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15806a5e0 con 0x7fe158155790 2023-12-19T09:22:07.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe15de0b700 1 --2- 172.21.15.111:0/35673515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe158155790 0x7fe1581598d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe150009f20 tx=0x7fe150009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a671ef666ae1e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe14ffff700 1 -- 172.21.15.111:0/35673515 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1500092e0 con 0x7fe158155790 2023-12-19T09:22:07.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe14ffff700 1 -- 172.21.15.111:0/35673515 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15000ba20 con 0x7fe158155790 2023-12-19T09:22:07.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.014+0000 7fe14ffff700 1 -- 172.21.15.111:0/35673515 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15000bde0 con 0x7fe158155790 2023-12-19T09:22:07.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 -- 172.21.15.111:0/35673515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe158155790 msgr2=0x7fe1581598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 --2- 172.21.15.111:0/35673515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe158155790 0x7fe1581598d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe150009f20 tx=0x7fe150009840 comp rx=0 tx=0).stop 2023-12-19T09:22:07.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 -- 172.21.15.111:0/35673515 shutdown_connections 2023-12-19T09:22:07.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 --2- 172.21.15.111:0/35673515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe158155790 0x7fe1581598d0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 -- 172.21.15.111:0/35673515 >> 172.21.15.111:0/35673515 conn(0x7fe1580cc7e0 msgr2=0x7fe1580c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 -- 172.21.15.111:0/35673515 shutdown_connections 2023-12-19T09:22:07.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 -- 172.21.15.111:0/35673515 wait complete. 2023-12-19T09:22:07.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.018+0000 7fe15f86e700 1 Processor -- start 2023-12-19T09:22:07.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 -- start start 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1580c7be0 0x7fe1580bf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 0x7fe15816e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe158155790 0x7fe15814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580bfc90 con 0x7fe1581553c0 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15816e920 con 0x7fe158155790 2023-12-19T09:22:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15f86e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15816df50 con 0x7fe1580c7be0 2023-12-19T09:22:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 0x7fe15816e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 0x7fe15816e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55194/0 (socket says 172.21.15.111:55194) 2023-12-19T09:22:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 -- 172.21.15.111:0/2232951420 learned_addr learned my addr 172.21.15.111:0/2232951420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15de0b700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe158155790 0x7fe15814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15d60a700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1580c7be0 0x7fe1580bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 -- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1580c7be0 msgr2=0x7fe1580bf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1580c7be0 0x7fe1580bf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.022+0000 7fe15ce09700 1 -- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe158155790 msgr2=0x7fe15814d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15ce09700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe158155790 0x7fe15814d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15ce09700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe150009ae0 con 0x7fe1581553c0 2023-12-19T09:22:07.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15de0b700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe158155790 0x7fe15814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15ce09700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 0x7fe15816e3e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fe14800c5c0 tx=0x7fe14800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148011590 con 0x7fe1581553c0 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe148011710 con 0x7fe1581553c0 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15816d980 con 0x7fe1581553c0 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148011ad0 con 0x7fe1581553c0 2023-12-19T09:22:07.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15816dbe0 con 0x7fe1581553c0 2023-12-19T09:22:07.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe1580708a0 con 0x7fe1581553c0 2023-12-19T09:22:07.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.026+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fe14801b3b0 con 0x7fe1581553c0 2023-12-19T09:22:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe13c060580 0x7fe13c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fe14807c4e0 con 0x7fe1581553c0 2023-12-19T09:22:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7fe13c067230 0x7fe13c069690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe13c069cc0 con 0x7fe13c067230 2023-12-19T09:22:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe14807c8b0 con 0x7fe1581553c0 2023-12-19T09:22:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe15d60a700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe13c060580 0x7fe13c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe15de0b700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7fe13c067230 0x7fe13c069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe15d60a700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe13c060580 0x7fe13c062a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1580b0f80 tx=0x7fe154009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe15de0b700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7fe13c067230 0x7fe13c069690 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.030+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== osd.2 v2:172.21.15.111:6824/33933 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe13c069cc0 con 0x7fe13c067230 2023-12-19T09:22:07.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.046+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 --> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe15806a8c0 con 0x7fe13c067230 2023-12-19T09:22:07.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe14d7fa700 1 -- 172.21.15.111:0/2232951420 <== osd.2 v2:172.21.15.111:6824/33933 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe15806a8c0 con 0x7fe13c067230 2023-12-19T09:22:07.056 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:07.056 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:07.056 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:07.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7fe13c067230 msgr2=0x7fe13c069690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:6824/33933,v1:172.21.15.111:6825/33933] conn(0x7fe13c067230 0x7fe13c069690 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe13c060580 msgr2=0x7fe13c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fe13c060580 0x7fe13c062a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1580b0f80 tx=0x7fe154009040 comp rx=0 tx=0).stop 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 msgr2=0x7fe15816e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 --2- 172.21.15.111:0/2232951420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1581553c0 0x7fe15816e3e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fe14800c5c0 tx=0x7fe14800c8d0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15de0b700 1 -- 172.21.15.111:0/2232951420 reap_dead start 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 shutdown_connections 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 >> 172.21.15.111:0/2232951420 conn(0x7fe1580cc7e0 msgr2=0x7fe15806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 shutdown_connections 2023-12-19T09:22:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.050+0000 7fe15f86e700 1 -- 172.21.15.111:0/2232951420 wait complete. 2023-12-19T09:22:07.081 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-19T09:22:07.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.234+0000 7face9203700 1 Processor -- start 2023-12-19T09:22:07.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.234+0000 7face9203700 1 -- start start 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face9203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face9203700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7face40b1be0 0x7face40b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face9203700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face40cb3e0 con 0x7face4158e40 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face9203700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face4068960 con 0x7face40b1be0 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face9203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face40caab0 con 0x7face4159210 2023-12-19T09:22:07.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:35790/0 (socket says 172.21.15.111:35790) 2023-12-19T09:22:07.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 -- 172.21.15.111:0/513743417 learned_addr learned my addr 172.21.15.111:0/513743417 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face8a02700 1 --2- 172.21.15.111:0/513743417 >> v2:172.21.15.143:3300/0 conn(0x7face40b1be0 0x7face40b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 -- 172.21.15.111:0/513743417 >> v2:172.21.15.143:3300/0 conn(0x7face40b1be0 msgr2=0x7face40b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 --2- 172.21.15.111:0/513743417 >> v2:172.21.15.143:3300/0 conn(0x7face40b1be0 0x7face40b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 -- 172.21.15.111:0/513743417 >> v1:172.21.15.143:6789/0 conn(0x7face4158e40 legacy=0x7face4067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 -- 172.21.15.111:0/513743417 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40cad90 con 0x7face4159210 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7facdbfff700 1 --2- 172.21.15.111:0/513743417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faccc0099b0 tx=0x7faccc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5474ab8a1db6e17 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.238+0000 7face37fe700 1 -- 172.21.15.111:0/513743417 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc00b860 con 0x7face4159210 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face37fe700 1 -- 172.21.15.111:0/513743417 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc00b9c0 con 0x7face4159210 2023-12-19T09:22:07.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face37fe700 1 -- 172.21.15.111:0/513743417 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc00bd60 con 0x7face4159210 2023-12-19T09:22:07.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 -- 172.21.15.111:0/513743417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 msgr2=0x7face40b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 --2- 172.21.15.111:0/513743417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faccc0099b0 tx=0x7faccc009250 comp rx=0 tx=0).stop 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 -- 172.21.15.111:0/513743417 shutdown_connections 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 --2- 172.21.15.111:0/513743417 >> v2:172.21.15.143:3300/0 conn(0x7face40b1be0 0x7face40b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 --2- 172.21.15.111:0/513743417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face4159210 0x7face40b2c00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 -- 172.21.15.111:0/513743417 >> 172.21.15.111:0/513743417 conn(0x7face40d0900 msgr2=0x7face40cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 -- 172.21.15.111:0/513743417 shutdown_connections 2023-12-19T09:22:07.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 -- 172.21.15.111:0/513743417 wait complete. 2023-12-19T09:22:07.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.242+0000 7face9203700 1 Processor -- start 2023-12-19T09:22:07.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 -- start start 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 0x7face414a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face4159210 0x7face415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face415be40 con 0x7face4158e40 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face414d560 con 0x7face4159210 2023-12-19T09:22:07.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face9203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face415b230 con 0x7face40b1be0 2023-12-19T09:22:07.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 0x7face414a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55202/0 (socket says 172.21.15.111:55202) 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 -- 172.21.15.111:0/799415800 learned_addr learned my addr 172.21.15.111:0/799415800 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face8a02700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face4159210 0x7face415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 -- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 msgr2=0x7face414a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 0x7face414a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 -- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face4159210 msgr2=0x7face415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face4159210 0x7face415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 -- 172.21.15.111:0/799415800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0094f0 con 0x7face4158e40 2023-12-19T09:22:07.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face3fff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 0x7face414a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7facdbfff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7face40cabf0 tx=0x7faccc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.246+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc011e40 con 0x7face4158e40 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc01a450 con 0x7face4158e40 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face9203700 1 -- 172.21.15.111:0/799415800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face415b510 con 0x7face4158e40 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc01a810 con 0x7face4158e40 2023-12-19T09:22:07.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face9203700 1 -- 172.21.15.111:0/799415800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face415bad0 con 0x7face4158e40 2023-12-19T09:22:07.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face9203700 1 -- 172.21.15.111:0/799415800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face404be40 con 0x7face4158e40 2023-12-19T09:22:07.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.250+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7faccc01ab80 con 0x7face4158e40 2023-12-19T09:22:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.254+0000 7face0ff9700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 0x7facc4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.254+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7faccc07e580 con 0x7face4158e40 2023-12-19T09:22:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.254+0000 7face3fff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 0x7facc4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.254+0000 7face3fff700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 0x7facc4062a00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facd4001dc0 tx=0x7facd4009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.258+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faccc04e820 con 0x7face4158e40 2023-12-19T09:22:07.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.430+0000 7face9203700 1 -- 172.21.15.111:0/799415800 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7face40cb070 con 0x7facc4060550 2023-12-19T09:22:07.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.430+0000 7face0ff9700 1 -- 172.21.15.111:0/799415800 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7face40cb070 con 0x7facc4060550 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 msgr2=0x7facc4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 0x7facc4062a00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7facd4001dc0 tx=0x7facd4009040 comp rx=0 tx=0).stop 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 msgr2=0x7face415a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7face40cabf0 tx=0x7faccc017000 comp rx=0 tx=0).stop 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 shutdown_connections 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7facc4060550 0x7facc4062a00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face4159210 0x7face415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7face4158e40 0x7face415a6c0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 --2- 172.21.15.111:0/799415800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40b1be0 0x7face414a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 >> 172.21.15.111:0/799415800 conn(0x7face40d0900 msgr2=0x7face4151e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 shutdown_connections 2023-12-19T09:22:07.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.434+0000 7face9203700 1 -- 172.21.15.111:0/799415800 wait complete. 2023-12-19T09:22:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 2 to deep-scrub 2023-12-19T09:22:07.463 INFO:tasks.ceph:Scrubbing osd.3 2023-12-19T09:22:07.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:07.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 Processor -- start 2023-12-19T09:22:07.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 -- start start 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8cac159400 0x7f8cac068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8cac068770 0x7f8cac0ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac067da0 con 0x7f8cac159030 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac0cb880 con 0x7f8cac159400 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb2af0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac067190 con 0x7f8cac068770 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8cac159400 0x7f8cac068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8cac159400 0x7f8cac068230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55206/0 (socket says 172.21.15.111:55206) 2023-12-19T09:22:07.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 -- 172.21.15.111:0/169315561 learned_addr learned my addr 172.21.15.111:0/169315561 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cb108d700 1 --2- 172.21.15.111:0/169315561 >> v2:172.21.15.143:3300/0 conn(0x7f8cac068770 0x7f8cac0ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 -- 172.21.15.111:0/169315561 >> v2:172.21.15.143:3300/0 conn(0x7f8cac068770 msgr2=0x7f8cac0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:07.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 --2- 172.21.15.111:0/169315561 >> v2:172.21.15.143:3300/0 conn(0x7f8cac068770 0x7f8cac0ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 -- 172.21.15.111:0/169315561 >> v1:172.21.15.143:6789/0 conn(0x7f8cac159030 legacy=0x7f8cac0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.622+0000 7f8cabfff700 1 -- 172.21.15.111:0/169315561 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac067470 con 0x7f8cac159400 2023-12-19T09:22:07.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cabfff700 1 --2- 172.21.15.111:0/169315561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac159400 0x7f8cac068230 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0d1130 tx=0x7f8c98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415f0f368a44515 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cab7fe700 1 -- 172.21.15.111:0/169315561 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800b860 con 0x7f8cac159400 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cab7fe700 1 -- 172.21.15.111:0/169315561 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9800b9c0 con 0x7f8cac159400 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cab7fe700 1 -- 172.21.15.111:0/169315561 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800bd60 con 0x7f8cac159400 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 -- 172.21.15.111:0/169315561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac159400 msgr2=0x7f8cac068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/169315561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac159400 0x7f8cac068230 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0d1130 tx=0x7f8c98009250 comp rx=0 tx=0).stop 2023-12-19T09:22:07.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 -- 172.21.15.111:0/169315561 shutdown_connections 2023-12-19T09:22:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/169315561 >> v2:172.21.15.143:3300/0 conn(0x7f8cac068770 0x7f8cac0ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/169315561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac159400 0x7f8cac068230 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 -- 172.21.15.111:0/169315561 >> 172.21.15.111:0/169315561 conn(0x7f8cac0d0170 msgr2=0x7f8cac0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 -- 172.21.15.111:0/169315561 shutdown_connections 2023-12-19T09:22:07.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 -- 172.21.15.111:0/169315561 wait complete. 2023-12-19T09:22:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.626+0000 7f8cb2af0700 1 Processor -- start 2023-12-19T09:22:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- start start 2023-12-19T09:22:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac068770 0x7f8cac154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cac159030 0x7f8cac0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 0x7f8cac0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0c8820 con 0x7f8cac0c83a0 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac155290 con 0x7f8cac068770 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac15c6f0 con 0x7f8cac159030 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 0x7f8cac0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 0x7f8cac0b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55210/0 (socket says 172.21.15.111:55210) 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 -- 172.21.15.111:0/1000694275 learned_addr learned my addr 172.21.15.111:0/1000694275 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb088c700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac068770 0x7f8cac154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cabfff700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cac159030 0x7f8cac0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 -- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cac159030 msgr2=0x7f8cac0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cac159030 0x7f8cac0c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 -- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac068770 msgr2=0x7f8cac154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac068770 0x7f8cac154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c980094f0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb108d700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 0x7f8cac0b0420 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4007f30 tx=0x7f8ca40098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca400eb90 con 0x7f8cac0c83a0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca400ecf0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac15b930 con 0x7f8cac0c83a0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4014670 con 0x7f8cac0c83a0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.630+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac15bef0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.634+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cac0708a0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.634+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f8ca40149e0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.634+0000 7f8ca8ff9700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c90060550 0x7f8c90062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.634+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f8ca407d150 con 0x7f8cac0c83a0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8ca8ff9700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7f8c90067200 0x7f8c90069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c90069c90 con 0x7f8c90067200 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ca407f6b0 con 0x7f8cac0c83a0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8cb088c700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c90060550 0x7f8c90062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8cabfff700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7f8c90067200 0x7f8c90069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8cabfff700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7f8c90067200 0x7f8c90069660 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8cb088c700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c90060550 0x7f8c90062a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b1080 tx=0x7f8ca0009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.638+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== osd.3 v2:172.21.15.111:6816/33942 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8c90069c90 con 0x7f8c90067200 2023-12-19T09:22:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.654+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 --> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8cac067750 con 0x7f8c90067200 2023-12-19T09:22:07.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.654+0000 7f8ca8ff9700 1 -- 172.21.15.111:0/1000694275 <== osd.3 v2:172.21.15.111:6816/33942 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8cac067750 con 0x7f8c90067200 2023-12-19T09:22:07.664 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:07.664 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:07.664 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:07.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7f8c90067200 msgr2=0x7f8c90069660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:6816/33942,v1:172.21.15.111:6817/33942] conn(0x7f8c90067200 0x7f8c90069660 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c90060550 msgr2=0x7f8c90062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f8c90060550 0x7f8c90062a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b1080 tx=0x7f8ca0009040 comp rx=0 tx=0).stop 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 msgr2=0x7f8cac0b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 --2- 172.21.15.111:0/1000694275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cac0c83a0 0x7f8cac0b0420 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4007f30 tx=0x7f8ca40098e0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb108d700 1 -- 172.21.15.111:0/1000694275 reap_dead start 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 shutdown_connections 2023-12-19T09:22:07.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 >> 172.21.15.111:0/1000694275 conn(0x7f8cac0d0170 msgr2=0x7f8cac0b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 shutdown_connections 2023-12-19T09:22:07.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.658+0000 7f8cb2af0700 1 -- 172.21.15.111:0/1000694275 wait complete. 2023-12-19T09:22:07.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-19T09:22:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.842+0000 7f7873078700 1 Processor -- start 2023-12-19T09:22:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.842+0000 7f7873078700 1 -- start start 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7873078700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f786c1590b0 0x7f786c15d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7873078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c0b1450 0x7f786c0b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7873078700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c069dc0 con 0x7f786c158ce0 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7873078700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c0b1e10 con 0x7f786c1590b0 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7873078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0691b0 con 0x7f786c0b1450 2023-12-19T09:22:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f786c1590b0 0x7f786c15d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7870e14700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f786c158ce0 0x7f786c06a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50852/0 (socket says 172.21.15.111:50852) 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7870e14700 1 -- 172.21.15.111:0/2911970278 learned_addr learned my addr 172.21.15.111:0/2911970278 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f7871615700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c0b1450 0x7f786c0b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786b7fe700 1 -- 172.21.15.111:0/2911970278 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225879507 0 0) 0x7f786c069dc0 con 0x7f786c158ce0 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 -- 172.21.15.111:0/2911970278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c0b1450 msgr2=0x7f786c0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c0b1450 0x7f786c0b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 -- 172.21.15.111:0/2911970278 >> v1:172.21.15.111:6789/0 conn(0x7f786c158ce0 legacy=0x7f786c06a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 -- 172.21.15.111:0/2911970278 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c069490 con 0x7f786c1590b0 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786bfff700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c1590b0 0x7f786c15d550 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f78580098e0 tx=0x7f78580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943058f41380f614 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786b7fe700 1 -- 172.21.15.111:0/2911970278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7858009e80 con 0x7f786c1590b0 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786b7fe700 1 -- 172.21.15.111:0/2911970278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785800b860 con 0x7f786c1590b0 2023-12-19T09:22:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.846+0000 7f786b7fe700 1 -- 172.21.15.111:0/2911970278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785800bbe0 con 0x7f786c1590b0 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 -- 172.21.15.111:0/2911970278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c1590b0 msgr2=0x7f786c15d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c1590b0 0x7f786c15d550 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f78580098e0 tx=0x7f78580091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 -- 172.21.15.111:0/2911970278 shutdown_connections 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c0b1450 0x7f786c0b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 --2- 172.21.15.111:0/2911970278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c1590b0 0x7f786c15d550 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 -- 172.21.15.111:0/2911970278 >> 172.21.15.111:0/2911970278 conn(0x7f786c06c0f0 msgr2=0x7f786c06c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 -- 172.21.15.111:0/2911970278 shutdown_connections 2023-12-19T09:22:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.850+0000 7f7873078700 1 -- 172.21.15.111:0/2911970278 wait complete. 2023-12-19T09:22:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 Processor -- start 2023-12-19T09:22:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 -- start start 2023-12-19T09:22:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c158ce0 0x7f786c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f786c1590b0 0x7f786c0aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c157160 con 0x7f786c0b1450 2023-12-19T09:22:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0cbb30 con 0x7f786c1590b0 2023-12-19T09:22:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7873078700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c16cd20 con 0x7f786c158ce0 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7870e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7870e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55218/0 (socket says 172.21.15.111:55218) 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7870e14700 1 -- 172.21.15.111:0/3429356472 learned_addr learned my addr 172.21.15.111:0/3429356472 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f7871615700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f786c1590b0 0x7f786c0aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.854+0000 7f786bfff700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c158ce0 0x7f786c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 -- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c158ce0 msgr2=0x7f786c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c158ce0 0x7f786c0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 -- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f786c1590b0 msgr2=0x7f786c0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f786c1590b0 0x7f786c0aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 -- 172.21.15.111:0/3429356472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858009420 con 0x7f786c0b1450 2023-12-19T09:22:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7870e14700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f786000cd80 tx=0x7f7860001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786000ab10 con 0x7f786c0b1450 2023-12-19T09:22:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000ac70 con 0x7f786c0b1450 2023-12-19T09:22:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c16d000 con 0x7f786c0b1450 2023-12-19T09:22:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78600070f0 con 0x7f786c0b1450 2023-12-19T09:22:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c16d5c0 con 0x7f786c0b1450 2023-12-19T09:22:07.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.858+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c071110 con 0x7f786c0b1450 2023-12-19T09:22:07.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.862+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f78600074e0 con 0x7f786c0b1450 2023-12-19T09:22:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.862+0000 7f7868ff9700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 0x7f7850062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.862+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f786007cfe0 con 0x7f786c0b1450 2023-12-19T09:22:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.862+0000 7f786bfff700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 0x7f7850062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:07.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.862+0000 7f786bfff700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 0x7f7850062a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78580091e0 tx=0x7f7858017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:07.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:07.866+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f786004d360 con 0x7f786c0b1450 2023-12-19T09:22:08.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.046+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f786c06ccc0 con 0x7f7850060550 2023-12-19T09:22:08.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.046+0000 7f7868ff9700 1 -- 172.21.15.111:0/3429356472 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f786c06ccc0 con 0x7f7850060550 2023-12-19T09:22:08.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 msgr2=0x7f7850062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 0x7f7850062a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78580091e0 tx=0x7f7858017000 comp rx=0 tx=0).stop 2023-12-19T09:22:08.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 msgr2=0x7f786c1479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f786000cd80 tx=0x7f7860001ae0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 shutdown_connections 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f7850060550 0x7f7850062a00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f786c1590b0 0x7f786c0aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f786c158ce0 0x7f786c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 --2- 172.21.15.111:0/3429356472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f786c0b1450 0x7f786c1479c0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 >> 172.21.15.111:0/3429356472 conn(0x7f786c06c0f0 msgr2=0x7f786c06c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 shutdown_connections 2023-12-19T09:22:08.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.050+0000 7f7873078700 1 -- 172.21.15.111:0/3429356472 wait complete. 2023-12-19T09:22:08.061 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 3 to deep-scrub 2023-12-19T09:22:08.081 INFO:tasks.ceph:Scrubbing osd.4 2023-12-19T09:22:08.081 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.250+0000 7f6ab78f0700 1 Processor -- start 2023-12-19T09:22:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.250+0000 7f6ab78f0700 1 -- start start 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 0x7f6ab0067060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab0159400 0x7f6ab00c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ab00675a0 0x7f6ab00c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00c7640 con 0x7f6ab00675a0 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab006a2c0 con 0x7f6ab00c67f0 2023-12-19T09:22:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab78f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0067b20 con 0x7f6ab0159400 2023-12-19T09:22:08.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ab00675a0 0x7f6ab00c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ab00675a0 0x7f6ab00c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55228/0 (socket says 172.21.15.111:55228) 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 -- 172.21.15.111:0/3771927357 learned_addr learned my addr 172.21.15.111:0/3771927357 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab4e8b700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab0159400 0x7f6ab00c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab568c700 1 --2- 172.21.15.111:0/3771927357 >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 0x7f6ab0067060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 -- 172.21.15.111:0/3771927357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab0159400 msgr2=0x7f6ab00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab0159400 0x7f6ab00c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 -- 172.21.15.111:0/3771927357 >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 msgr2=0x7f6ab0067060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/3771927357 >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 0x7f6ab0067060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 -- 172.21.15.111:0/3771927357 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0067e00 con 0x7f6ab00675a0 2023-12-19T09:22:08.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab568c700 1 --2- 172.21.15.111:0/3771927357 >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 0x7f6ab0067060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab00c7100 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0070770 tx=0x7f6aa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11362837b06b8a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.254+0000 7f6aa7fff700 1 -- 172.21.15.111:0/3771927357 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa800e070 con 0x7f6ab00675a0 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6aa7fff700 1 -- 172.21.15.111:0/3771927357 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa800ba50 con 0x7f6ab00675a0 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6aa7fff700 1 -- 172.21.15.111:0/3771927357 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80092e0 con 0x7f6ab00675a0 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 -- 172.21.15.111:0/3771927357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 msgr2=0x7f6ab00c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab00c7100 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0070770 tx=0x7f6aa8009840 comp rx=0 tx=0).stop 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 -- 172.21.15.111:0/3771927357 shutdown_connections 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab00c7100 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/3771927357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab0159400 0x7f6ab00c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/3771927357 >> v2:172.21.15.143:3300/0 conn(0x7f6ab00c67f0 0x7f6ab0067060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 -- 172.21.15.111:0/3771927357 >> 172.21.15.111:0/3771927357 conn(0x7f6ab00cc7e0 msgr2=0x7f6ab00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 -- 172.21.15.111:0/3771927357 shutdown_connections 2023-12-19T09:22:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 -- 172.21.15.111:0/3771927357 wait complete. 2023-12-19T09:22:08.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.258+0000 7f6ab78f0700 1 Processor -- start 2023-12-19T09:22:08.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 -- start start 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab0151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab00c67f0 0x7f6ab015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ab015aae0 0x7f6ab00c2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00c0400 con 0x7f6ab00675a0 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00bf7f0 con 0x7f6ab015aae0 2023-12-19T09:22:08.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab78f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00bfad0 con 0x7f6ab00c67f0 2023-12-19T09:22:08.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab0151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab4e8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab00c67f0 0x7f6ab015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab0151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55232/0 (socket says 172.21.15.111:55232) 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab4e8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab00c67f0 0x7f6ab015b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:35830/0 (socket says 172.21.15.111:35830) 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 -- 172.21.15.111:0/2896124765 learned_addr learned my addr 172.21.15.111:0/2896124765 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ab015aae0 0x7f6ab00c2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 -- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab00c67f0 msgr2=0x7f6ab015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ab00c67f0 0x7f6ab015b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 -- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ab015aae0 msgr2=0x7f6ab00c2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ab015aae0 0x7f6ab00c2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8009ae0 con 0x7f6ab00675a0 2023-12-19T09:22:08.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ab015aae0 0x7f6ab00c2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:08.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6ab568c700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab0151410 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00d860 tx=0x7f6aac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0125f0 con 0x7f6ab00675a0 2023-12-19T09:22:08.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.262+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac012750 con 0x7f6ab00675a0 2023-12-19T09:22:08.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab00bfdb0 con 0x7f6ab00675a0 2023-12-19T09:22:08.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac012af0 con 0x7f6ab00675a0 2023-12-19T09:22:08.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0151950 con 0x7f6ab00675a0 2023-12-19T09:22:08.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ab00708a0 con 0x7f6ab00675a0 2023-12-19T09:22:08.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6aac027430 con 0x7f6ab00675a0 2023-12-19T09:22:08.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.266+0000 7f6aa57fa700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6a94060550 0x7f6a94062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f6aac07d8c0 con 0x7f6ab00675a0 2023-12-19T09:22:08.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6aa57fa700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f6a94067200 0x7f6a94069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a94069c90 con 0x7f6a94067200 2023-12-19T09:22:08.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aac07dc90 con 0x7f6ab00675a0 2023-12-19T09:22:08.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6ab4e8b700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6a94060550 0x7f6a94062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f6a94067200 0x7f6a94069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6ab4e8b700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6a94060550 0x7f6a94062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00c28d0 tx=0x7f6aa0008040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.270+0000 7f6ab5e8d700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f6a94067200 0x7f6a94069660 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.274+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== osd.4 v2:172.21.15.143:6808/34138 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6a94069c90 con 0x7f6a94067200 2023-12-19T09:22:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.290+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 --> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6ab0068bb0 con 0x7f6a94067200 2023-12-19T09:22:08.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6aa57fa700 1 -- 172.21.15.111:0/2896124765 <== osd.4 v2:172.21.15.143:6808/34138 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6ab0068bb0 con 0x7f6a94067200 2023-12-19T09:22:08.301 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:08.301 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:08.301 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f6a94067200 msgr2=0x7f6a94069660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6808/34138,v1:172.21.15.143:6810/34138] conn(0x7f6a94067200 0x7f6a94069660 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6a94060550 msgr2=0x7f6a94062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6a94060550 0x7f6a94062a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00c28d0 tx=0x7f6aa0008040 comp rx=0 tx=0).stop 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 msgr2=0x7f6ab0151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 --2- 172.21.15.111:0/2896124765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ab00675a0 0x7f6ab0151410 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00d860 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2023-12-19T09:22:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab5e8d700 1 -- 172.21.15.111:0/2896124765 reap_dead start 2023-12-19T09:22:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 shutdown_connections 2023-12-19T09:22:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 >> 172.21.15.111:0/2896124765 conn(0x7f6ab00cc7e0 msgr2=0x7f6ab01575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 shutdown_connections 2023-12-19T09:22:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.294+0000 7f6ab78f0700 1 -- 172.21.15.111:0/2896124765 wait complete. 2023-12-19T09:22:08.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-19T09:22:08.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.486+0000 7fb51a8f0700 1 Processor -- start 2023-12-19T09:22:08.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.486+0000 7fb51a8f0700 1 -- start start 2023-12-19T09:22:08.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb51a8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 0x7fb5140b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb51a8f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5140b22f0 0x7fb5140cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb51a8f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5140cb710 con 0x7fb514159090 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb51a8f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5140af490 con 0x7fb5140b22f0 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb51a8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140b3200 con 0x7fb5140b2d90 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb513fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb514159090 0x7fb5140af620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50868/0 (socket says 172.21.15.111:50868) 2023-12-19T09:22:08.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb513fff700 1 -- 172.21.15.111:0/3543133574 learned_addr learned my addr 172.21.15.111:0/3543133574 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:08.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 --2- 172.21.15.111:0/3543133574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 0x7fb5140b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb518e8d700 1 --2- 172.21.15.111:0/3543133574 >> v2:172.21.15.143:3300/0 conn(0x7fb5140b22f0 0x7fb5140cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb5137fe700 1 -- 172.21.15.111:0/3543133574 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381459752 0 0) 0x7fb5140cb710 con 0x7fb514159090 2023-12-19T09:22:08.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb5137fe700 1 -- 172.21.15.111:0/3543133574 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb500003570 con 0x7fb514159090 2023-12-19T09:22:08.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 -- 172.21.15.111:0/3543133574 >> v2:172.21.15.143:3300/0 conn(0x7fb5140b22f0 msgr2=0x7fb5140cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:08.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 --2- 172.21.15.111:0/3543133574 >> v2:172.21.15.143:3300/0 conn(0x7fb5140b22f0 0x7fb5140cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 -- 172.21.15.111:0/3543133574 >> v1:172.21.15.111:6789/0 conn(0x7fb514159090 legacy=0x7fb5140af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 -- 172.21.15.111:0/3543133574 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140cade0 con 0x7fb5140b2d90 2023-12-19T09:22:08.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.490+0000 7fb50b7fe700 1 --2- 172.21.15.111:0/3543133574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 0x7fb5140b1db0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098d0 tx=0x7fb4fc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d41a4726c217671 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb5137fe700 1 -- 172.21.15.111:0/3543133574 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc005c10 con 0x7fb5140b2d90 2023-12-19T09:22:08.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb5137fe700 1 -- 172.21.15.111:0/3543133574 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc005d70 con 0x7fb5140b2d90 2023-12-19T09:22:08.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb5137fe700 1 -- 172.21.15.111:0/3543133574 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc006260 con 0x7fb5140b2d90 2023-12-19T09:22:08.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3543133574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 msgr2=0x7fb5140b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3543133574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 0x7fb5140b1db0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0098d0 tx=0x7fb4fc009120 comp rx=0 tx=0).stop 2023-12-19T09:22:08.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3543133574 shutdown_connections 2023-12-19T09:22:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3543133574 >> v2:172.21.15.143:3300/0 conn(0x7fb5140b22f0 0x7fb5140cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3543133574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5140b2d90 0x7fb5140b1db0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3543133574 >> 172.21.15.111:0/3543133574 conn(0x7fb5140d0ab0 msgr2=0x7fb5140cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3543133574 shutdown_connections 2023-12-19T09:22:08.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.494+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3543133574 wait complete. 2023-12-19T09:22:08.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 Processor -- start 2023-12-19T09:22:08.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 -- start start 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5140b2d90 0x7fb5140ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 0x7fb5140cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51415ad30 con 0x7fb5140b22f0 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51415a120 con 0x7fb5140b2d90 2023-12-19T09:22:08.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb51a8f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51415a400 con 0x7fb514159090 2023-12-19T09:22:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb513fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb518e8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 0x7fb5140cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb50b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5140b2d90 0x7fb5140ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb513fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55242/0 (socket says 172.21.15.111:55242) 2023-12-19T09:22:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.498+0000 7fb513fff700 1 -- 172.21.15.111:0/3067100087 learned_addr learned my addr 172.21.15.111:0/3067100087 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 -- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 msgr2=0x7fb5140cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 0x7fb5140cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 -- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5140b2d90 msgr2=0x7fb5140ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5140b2d90 0x7fb5140ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 -- 172.21.15.111:0/3067100087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0093c0 con 0x7fb5140b22f0 2023-12-19T09:22:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb518e8d700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 0x7fb5140cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb513fff700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb50400cc40 tx=0x7fb504001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50400a8b0 con 0x7fb5140b22f0 2023-12-19T09:22:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50400aa30 con 0x7fb5140b22f0 2023-12-19T09:22:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50400adf0 con 0x7fb5140b22f0 2023-12-19T09:22:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51415a6e0 con 0x7fb5140b22f0 2023-12-19T09:22:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb514156990 con 0x7fb5140b22f0 2023-12-19T09:22:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.502+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb514071110 con 0x7fb5140b22f0 2023-12-19T09:22:08.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.506+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb504007300 con 0x7fb5140b22f0 2023-12-19T09:22:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.506+0000 7fb510ff9700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 0x7fb4f4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.506+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fb50407ce20 con 0x7fb5140b22f0 2023-12-19T09:22:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.506+0000 7fb50b7fe700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 0x7fb4f4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.510+0000 7fb50b7fe700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 0x7fb4f4062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009160 tx=0x7fb4fc016000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.510+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb50404d140 con 0x7fb5140b22f0 2023-12-19T09:22:08.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.682+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51415ae70 con 0x7fb4f4060550 2023-12-19T09:22:08.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb510ff9700 1 -- 172.21.15.111:0/3067100087 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb4fc006610 con 0x7fb4f4060550 2023-12-19T09:22:08.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 msgr2=0x7fb4f4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 0x7fb4f4062a00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009160 tx=0x7fb4fc016000 comp rx=0 tx=0).stop 2023-12-19T09:22:08.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 msgr2=0x7fb514156450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb50400cc40 tx=0x7fb504001a70 comp rx=0 tx=0).stop 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 shutdown_connections 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb4f4060550 0x7fb4f4062a00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5140b2d90 0x7fb5140ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514159090 0x7fb5140cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 --2- 172.21.15.111:0/3067100087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5140b22f0 0x7fb514156450 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 >> 172.21.15.111:0/3067100087 conn(0x7fb5140d0ab0 msgr2=0x7fb5140b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 shutdown_connections 2023-12-19T09:22:08.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.686+0000 7fb51a8f0700 1 -- 172.21.15.111:0/3067100087 wait complete. 2023-12-19T09:22:08.698 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 4 to deep-scrub 2023-12-19T09:22:08.713 INFO:tasks.ceph:Scrubbing osd.5 2023-12-19T09:22:08.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.870+0000 7f6b3aa8e700 1 Processor -- start 2023-12-19T09:22:08.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 -- start start 2023-12-19T09:22:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b34156f90 0x7f6b34068630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 0x7f6b340adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3406b6b0 con 0x7f6b34157ef0 2023-12-19T09:22:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b34157360 con 0x7f6b340ad6c0 2023-12-19T09:22:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.874+0000 7f6b3aa8e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b34067470 con 0x7f6b34156f90 2023-12-19T09:22:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b3928b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6b34157ef0 0x7f6b340b0100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50870/0 (socket says 172.21.15.111:50870) 2023-12-19T09:22:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b3928b700 1 -- 172.21.15.111:0/3550463702 learned_addr learned my addr 172.21.15.111:0/3550463702 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/3550463702 >> v2:172.21.15.143:3300/0 conn(0x7f6b34156f90 0x7f6b34068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b3a28d700 1 --2- 172.21.15.111:0/3550463702 >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 0x7f6b340adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 -- 172.21.15.111:0/3550463702 >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 msgr2=0x7f6b340adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/3550463702 >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 0x7f6b340adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 -- 172.21.15.111:0/3550463702 >> v1:172.21.15.111:6789/0 conn(0x7f6b34157ef0 legacy=0x7f6b340b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 -- 172.21.15.111:0/3550463702 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340ae040 con 0x7f6b34156f90 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b3a28d700 1 --2- 172.21.15.111:0/3550463702 >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 0x7f6b340adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.878+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/3550463702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b34068630 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406cb50 tx=0x7f6b300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cac8f59473c55aec server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b38a8a700 1 -- 172.21.15.111:0/3550463702 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30011630 con 0x7f6b34156f90 2023-12-19T09:22:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b38a8a700 1 -- 172.21.15.111:0/3550463702 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b30011790 con 0x7f6b34156f90 2023-12-19T09:22:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b38a8a700 1 -- 172.21.15.111:0/3550463702 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30011b30 con 0x7f6b34156f90 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/3550463702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 msgr2=0x7f6b34068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/3550463702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b34068630 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406cb50 tx=0x7f6b300091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/3550463702 shutdown_connections 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/3550463702 >> v2:172.21.15.111:3300/0 conn(0x7f6b340ad6c0 0x7f6b340adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/3550463702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b34068630 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/3550463702 >> 172.21.15.111:0/3550463702 conn(0x7f6b340c8560 msgr2=0x7f6b340c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/3550463702 shutdown_connections 2023-12-19T09:22:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.882+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/3550463702 wait complete. 2023-12-19T09:22:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 Processor -- start 2023-12-19T09:22:08.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 -- start start 2023-12-19T09:22:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 0x7f6b3416d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b340ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b34157ef0 0x7f6b3415a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3416d3a0 con 0x7f6b340ad6c0 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3414dbb0 con 0x7f6b34156f90 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3aa8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341534a0 con 0x7f6b34157ef0 2023-12-19T09:22:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b39a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 0x7f6b3416d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b39a8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 0x7f6b3416d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55254/0 (socket says 172.21.15.111:55254) 2023-12-19T09:22:08.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b39a8c700 1 -- 172.21.15.111:0/2485498882 learned_addr learned my addr 172.21.15.111:0/2485498882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:08.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3928b700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b340ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.886+0000 7f6b3a28d700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b34157ef0 0x7f6b3415a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 -- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b34157ef0 msgr2=0x7f6b3415a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b34157ef0 0x7f6b3415a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 -- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 msgr2=0x7f6b340ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b340ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c009a20 con 0x7f6b340ad6c0 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b3928b700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b34156f90 0x7f6b340ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:08.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b39a8c700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 0x7f6b3416d820 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406ec10 tx=0x7f6b30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30011b10 con 0x7f6b340ad6c0 2023-12-19T09:22:08.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b300253f0 con 0x7f6b340ad6c0 2023-12-19T09:22:08.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30025790 con 0x7f6b340ad6c0 2023-12-19T09:22:08.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b30009420 con 0x7f6b340ad6c0 2023-12-19T09:22:08.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3414d840 con 0x7f6b340ad6c0 2023-12-19T09:22:08.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.890+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b3414d680 con 0x7f6b340ad6c0 2023-12-19T09:22:08.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f6b30025b80 con 0x7f6b340ad6c0 2023-12-19T09:22:08.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6b18060580 0x7f6b18062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f6b3007e1f0 con 0x7f6b340ad6c0 2023-12-19T09:22:08.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f6b18067230 0x7f6b18069690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:08.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b18069cc0 con 0x7f6b18067230 2023-12-19T09:22:08.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b300807d0 con 0x7f6b340ad6c0 2023-12-19T09:22:08.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b3928b700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6b18060580 0x7f6b18062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b3a28d700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f6b18067230 0x7f6b18069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:08.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b3928b700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6b18060580 0x7f6b18062a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406efd0 tx=0x7f6b24007f10 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.894+0000 7f6b3a28d700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f6b18067230 0x7f6b18069690 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:08.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.898+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== osd.5 v2:172.21.15.143:6809/34137 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6b18069cc0 con 0x7f6b18067230 2023-12-19T09:22:08.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.914+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 --> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6b3404be40 con 0x7f6b18067230 2023-12-19T09:22:08.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.914+0000 7f6b29ffb700 1 -- 172.21.15.111:0/2485498882 <== osd.5 v2:172.21.15.143:6809/34137 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6b3404be40 con 0x7f6b18067230 2023-12-19T09:22:08.923 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:08.923 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:08.923 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.914+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f6b18067230 msgr2=0x7f6b18069690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.914+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6809/34137,v1:172.21.15.143:6811/34137] conn(0x7f6b18067230 0x7f6b18069690 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6b18060580 msgr2=0x7f6b18062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f6b18060580 0x7f6b18062a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406efd0 tx=0x7f6b24007f10 comp rx=0 tx=0).stop 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 msgr2=0x7f6b3416d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 --2- 172.21.15.111:0/2485498882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b340ad6c0 0x7f6b3416d820 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f6b3406ec10 tx=0x7f6b30017000 comp rx=0 tx=0).stop 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3a28d700 1 -- 172.21.15.111:0/2485498882 reap_dead start 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 shutdown_connections 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 >> 172.21.15.111:0/2485498882 conn(0x7f6b340c8560 msgr2=0x7f6b340b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 shutdown_connections 2023-12-19T09:22:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:08.918+0000 7f6b3aa8e700 1 -- 172.21.15.111:0/2485498882 wait complete. 2023-12-19T09:22:08.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-19T09:22:09.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.114+0000 7f02454ae700 1 Processor -- start 2023-12-19T09:22:09.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.114+0000 7f02454ae700 1 -- start start 2023-12-19T09:22:09.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f02454ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f024013f860 0x7f02400c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f02454ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400ca820 con 0x7f024013f490 2023-12-19T09:22:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f02454ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0240067500 con 0x7f02400c7ee0 2023-12-19T09:22:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f02454ae700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02400ca260 con 0x7f024013f860 2023-12-19T09:22:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f024013f860 0x7f02400c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f024013f860 0x7f02400c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55270/0 (socket says 172.21.15.111:55270) 2023-12-19T09:22:09.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 -- 172.21.15.111:0/1521591912 learned_addr learned my addr 172.21.15.111:0/1521591912 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 -- 172.21.15.111:0/1521591912 >> v1:172.21.15.143:6789/0 conn(0x7f02400c7ee0 legacy=0x7f0240067990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 -- 172.21.15.111:0/1521591912 >> v1:172.21.15.111:6789/0 conn(0x7f024013f490 legacy=0x7f02400c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 -- 172.21.15.111:0/1521591912 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02400ca540 con 0x7f024013f860 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023dffb700 1 -- 172.21.15.111:0/1521591912 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287320660 0 0) 0x7f02400ca820 con 0x7f024013f490 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023e7fc700 1 --2- 172.21.15.111:0/1521591912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400c79a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f023400bc50 tx=0x7f023400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa57198f4f444c40 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023dffb700 1 -- 172.21.15.111:0/1521591912 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400be60 con 0x7f024013f860 2023-12-19T09:22:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023dffb700 1 -- 172.21.15.111:0/1521591912 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023400de00 con 0x7f024013f860 2023-12-19T09:22:09.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.118+0000 7f023dffb700 1 -- 172.21.15.111:0/1521591912 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02340196b0 con 0x7f024013f860 2023-12-19T09:22:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 -- 172.21.15.111:0/1521591912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 msgr2=0x7f02400c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 --2- 172.21.15.111:0/1521591912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400c79a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f023400bc50 tx=0x7f023400b4f0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 -- 172.21.15.111:0/1521591912 shutdown_connections 2023-12-19T09:22:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 --2- 172.21.15.111:0/1521591912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400c79a0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 -- 172.21.15.111:0/1521591912 >> 172.21.15.111:0/1521591912 conn(0x7f02400cc6c0 msgr2=0x7f02400c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 -- 172.21.15.111:0/1521591912 shutdown_connections 2023-12-19T09:22:09.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.122+0000 7f02454ae700 1 -- 172.21.15.111:0/1521591912 wait complete. 2023-12-19T09:22:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 Processor -- start 2023-12-19T09:22:09.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 -- start start 2023-12-19T09:22:09.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02400c7ee0 0x7f024016b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f024013f490 0x7f02400aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02400b0270 con 0x7f024013f860 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024016e060 con 0x7f02400c7ee0 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f02454ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02400af5f0 con 0x7f024013f490 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55272/0 (socket says 172.21.15.111:55272) 2023-12-19T09:22:09.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 -- 172.21.15.111:0/3017012673 learned_addr learned my addr 172.21.15.111:0/3017012673 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023e7fc700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f024013f490 0x7f02400aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023effd700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02400c7ee0 0x7f024016b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 -- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f024013f490 msgr2=0x7f02400aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f024013f490 0x7f02400aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 -- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02400c7ee0 msgr2=0x7f024016b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02400c7ee0 0x7f024016b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.126+0000 7f023f7fe700 1 -- 172.21.15.111:0/3017012673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023400b790 con 0x7f024013f860 2023-12-19T09:22:09.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f023f7fe700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f023800c840 tx=0x7f0238001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238012630 con 0x7f024013f860 2023-12-19T09:22:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02380127b0 con 0x7f024013f860 2023-12-19T09:22:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238012b70 con 0x7f024013f860 2023-12-19T09:22:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02400af8d0 con 0x7f024013f860 2023-12-19T09:22:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02400afe90 con 0x7f024013f860 2023-12-19T09:22:09.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.130+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02400b03b0 con 0x7f024013f860 2023-12-19T09:22:09.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.134+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f023801a3a0 con 0x7f024013f860 2023-12-19T09:22:09.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.134+0000 7f0226ffd700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 0x7f022c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.134+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f023807cc40 con 0x7f024013f860 2023-12-19T09:22:09.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.134+0000 7f023effd700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 0x7f022c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.138+0000 7f023effd700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 0x7f022c062a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f024015a5b0 tx=0x7f0230009040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.142+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f023804cee0 con 0x7f024013f860 2023-12-19T09:22:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.310+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0240071110 con 0x7f022c060580 2023-12-19T09:22:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.310+0000 7f0226ffd700 1 -- 172.21.15.111:0/3017012673 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0240071110 con 0x7f022c060580 2023-12-19T09:22:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 msgr2=0x7f022c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 0x7f022c062a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f024015a5b0 tx=0x7f0230009040 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 msgr2=0x7f02400aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f023800c840 tx=0x7f0238001670 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 shutdown_connections 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024013f860 0x7f02400aef50 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f024013f490 0x7f02400aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f022c060580 0x7f022c062a30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 --2- 172.21.15.111:0/3017012673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02400c7ee0 0x7f024016b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 >> 172.21.15.111:0/3017012673 conn(0x7f02400cc6c0 msgr2=0x7f0240062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 shutdown_connections 2023-12-19T09:22:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.314+0000 7f02454ae700 1 -- 172.21.15.111:0/3017012673 wait complete. 2023-12-19T09:22:09.326 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 5 to deep-scrub 2023-12-19T09:22:09.345 INFO:tasks.ceph:Scrubbing osd.6 2023-12-19T09:22:09.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:09.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.498+0000 7f1e90ce3700 1 Processor -- start 2023-12-19T09:22:09.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 -- start start 2023-12-19T09:22:09.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e8c0cb3a0 0x7f1e8c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0cea80 con 0x7f1e8c15cee0 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b0b40 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.502+0000 7f1e90ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0ced60 con 0x7f1e8c158b20 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e8c0cb3a0 0x7f1e8c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8b7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1e8c15cee0 0x7f1e8c0ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:54502/0 (socket says 172.21.15.111:54502) 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8b7fe700 1 -- 172.21.15.111:0/1262764812 learned_addr learned my addr 172.21.15.111:0/1262764812 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8bfff700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8a7fc700 1 -- 172.21.15.111:0/1262764812 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167716214 0 0) 0x7f1e8c0cea80 con 0x7f1e8c15cee0 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 -- 172.21.15.111:0/1262764812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 msgr2=0x7f1e8c158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 -- 172.21.15.111:0/1262764812 >> v1:172.21.15.143:6789/0 conn(0x7f1e8c15cee0 legacy=0x7f1e8c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 -- 172.21.15.111:0/1262764812 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0cf040 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8affd700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c15d2b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f1e800098e0 tx=0x7f1e800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc3bf947b9ebd95 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8a7fc700 1 -- 172.21.15.111:0/1262764812 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80009e80 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8a7fc700 1 -- 172.21.15.111:0/1262764812 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8000b860 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e8a7fc700 1 -- 172.21.15.111:0/1262764812 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8000bbe0 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 -- 172.21.15.111:0/1262764812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0cb3a0 msgr2=0x7f1e8c15d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c15d2b0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f1e800098e0 tx=0x7f1e800091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 -- 172.21.15.111:0/1262764812 shutdown_connections 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/1262764812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c15d2b0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 -- 172.21.15.111:0/1262764812 >> 172.21.15.111:0/1262764812 conn(0x7f1e8c0cff20 msgr2=0x7f1e8c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 -- 172.21.15.111:0/1262764812 shutdown_connections 2023-12-19T09:22:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.506+0000 7f1e90ce3700 1 -- 172.21.15.111:0/1262764812 wait complete. 2023-12-19T09:22:09.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.510+0000 7f1e90ce3700 1 Processor -- start 2023-12-19T09:22:09.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.510+0000 7f1e90ce3700 1 -- start start 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c067060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 0x7f1e8c0675a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b3be0 con 0x7f1e8c15cee0 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b3ec0 con 0x7f1e8c0cb3a0 2023-12-19T09:22:09.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b41a0 con 0x7f1e8c158b20 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 0x7f1e8c0675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 0x7f1e8c0675a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55302/0 (socket says 172.21.15.111:55302) 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 -- 172.21.15.111:0/2253432022 learned_addr learned my addr 172.21.15.111:0/2253432022 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8b7fe700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8affd700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c067060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 -- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 msgr2=0x7f1e8c067060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e8c158b20 0x7f1e8c067060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 -- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0cb3a0 msgr2=0x7f1e8c149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e8c0cb3a0 0x7f1e8c149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80009420 con 0x7f1e8c15cee0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e8bfff700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 0x7f1e8c0675a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009fa0 tx=0x7f1e84009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400ebc0 con 0x7f1e8c15cee0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8400ed20 con 0x7f1e8c15cee0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84014670 con 0x7f1e8c15cee0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c0b4480 con 0x7f1e8c15cee0 2023-12-19T09:22:09.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.514+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c149fe0 con 0x7f1e8c15cee0 2023-12-19T09:22:09.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e8c04be40 con 0x7f1e8c15cee0 2023-12-19T09:22:09.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f1e84014a20 con 0x7f1e8c15cee0 2023-12-19T09:22:09.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f1e8407ded0 con 0x7f1e8c15cee0 2023-12-19T09:22:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7f1e78067200 0x7f1e78069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e78069c90 con 0x7f1e78067200 2023-12-19T09:22:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.518+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e8407e2a0 con 0x7f1e8c15cee0 2023-12-19T09:22:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.522+0000 7f1e8b7fe700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.522+0000 7f1e8affd700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7f1e78067200 0x7f1e78069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.522+0000 7f1e8b7fe700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c007160 tx=0x7f1e7c007f50 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.522+0000 7f1e8affd700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7f1e78067200 0x7f1e78069660 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.522+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== osd.6 v2:172.21.15.143:6800/34136 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f1e78069c90 con 0x7f1e78067200 2023-12-19T09:22:09.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 --> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1e8c0cf320 con 0x7f1e78067200 2023-12-19T09:22:09.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e737fe700 1 -- 172.21.15.111:0/2253432022 <== osd.6 v2:172.21.15.143:6800/34136 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1e8c0cf320 con 0x7f1e78067200 2023-12-19T09:22:09.550 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:09.550 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:09.550 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:09.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7f1e78067200 msgr2=0x7f1e78069660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6800/34136,v1:172.21.15.143:6801/34136] conn(0x7f1e78067200 0x7f1e78069660 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 msgr2=0x7f1e78062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f1e78060550 0x7f1e78062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c007160 tx=0x7f1e7c007f50 comp rx=0 tx=0).stop 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 msgr2=0x7f1e8c0675a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 --2- 172.21.15.111:0/2253432022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c15cee0 0x7f1e8c0675a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009fa0 tx=0x7f1e84009900 comp rx=0 tx=0).stop 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e8bfff700 1 -- 172.21.15.111:0/2253432022 reap_dead start 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 shutdown_connections 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 >> 172.21.15.111:0/2253432022 conn(0x7f1e8c0cff20 msgr2=0x7f1e8c0682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 shutdown_connections 2023-12-19T09:22:09.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.542+0000 7f1e90ce3700 1 -- 172.21.15.111:0/2253432022 wait complete. 2023-12-19T09:22:09.571 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-19T09:22:09.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.734+0000 7fb42ece3700 1 Processor -- start 2023-12-19T09:22:09.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.734+0000 7fb42ece3700 1 -- start start 2023-12-19T09:22:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb428158e40 0x7fb4280b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb428159210 0x7fb428067f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280cb3e0 con 0x7fb428068460 2023-12-19T09:22:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb428068960 con 0x7fb428159210 2023-12-19T09:22:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4280caab0 con 0x7fb428158e40 2023-12-19T09:22:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb428159210 0x7fb428067f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb428159210 0x7fb428067f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55304/0 (socket says 172.21.15.111:55304) 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 -- 172.21.15.111:0/418313690 learned_addr learned my addr 172.21.15.111:0/418313690 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42dce1700 1 --2- 172.21.15.111:0/418313690 >> v2:172.21.15.143:3300/0 conn(0x7fb428158e40 0x7fb4280b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 -- 172.21.15.111:0/418313690 >> v2:172.21.15.143:3300/0 conn(0x7fb428158e40 msgr2=0x7fb4280b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 --2- 172.21.15.111:0/418313690 >> v2:172.21.15.143:3300/0 conn(0x7fb428158e40 0x7fb4280b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 -- 172.21.15.111:0/418313690 >> v1:172.21.15.143:6789/0 conn(0x7fb428068460 legacy=0x7fb4280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 -- 172.21.15.111:0/418313690 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4280cad90 con 0x7fb428159210 2023-12-19T09:22:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42d4e0700 1 --2- 172.21.15.111:0/418313690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428159210 0x7fb428067f20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fb42806a030 tx=0x7fb418009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487d61989d309089 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ccdf700 1 -- 172.21.15.111:0/418313690 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41800b860 con 0x7fb428159210 2023-12-19T09:22:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ccdf700 1 -- 172.21.15.111:0/418313690 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41800b9c0 con 0x7fb428159210 2023-12-19T09:22:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ccdf700 1 -- 172.21.15.111:0/418313690 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41800bd60 con 0x7fb428159210 2023-12-19T09:22:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 -- 172.21.15.111:0/418313690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428159210 msgr2=0x7fb428067f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.738+0000 7fb42ece3700 1 --2- 172.21.15.111:0/418313690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428159210 0x7fb428067f20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fb42806a030 tx=0x7fb418009250 comp rx=0 tx=0).stop 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 -- 172.21.15.111:0/418313690 shutdown_connections 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 --2- 172.21.15.111:0/418313690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428159210 0x7fb428067f20 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 --2- 172.21.15.111:0/418313690 >> v2:172.21.15.143:3300/0 conn(0x7fb428158e40 0x7fb4280b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 -- 172.21.15.111:0/418313690 >> 172.21.15.111:0/418313690 conn(0x7fb4280d0900 msgr2=0x7fb4280cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 -- 172.21.15.111:0/418313690 shutdown_connections 2023-12-19T09:22:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 -- 172.21.15.111:0/418313690 wait complete. 2023-12-19T09:22:09.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 Processor -- start 2023-12-19T09:22:09.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.742+0000 7fb42ece3700 1 -- start start 2023-12-19T09:22:09.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 0x7fb4280c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 0x7fb42815bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c4e20 con 0x7fb428158e40 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c7420 con 0x7fb428068460 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280cebc0 con 0x7fb428159210 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55310/0 (socket says 172.21.15.111:55310) 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 -- 172.21.15.111:0/1525991899 learned_addr learned my addr 172.21.15.111:0/1525991899 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 -- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 msgr2=0x7fb42815bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42dce1700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 0x7fb4280c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42e4e2700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 0x7fb42815bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 0x7fb42815bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 -- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 msgr2=0x7fb4280c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 0x7fb4280c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 -- 172.21.15.111:0/1525991899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180094f0 con 0x7fb428158e40 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42dce1700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 0x7fb4280c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42e4e2700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 0x7fb42815bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T09:22:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42d4e0700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb428088e10 tx=0x7fb418017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418011e60 con 0x7fb428158e40 2023-12-19T09:22:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41801a630 con 0x7fb428158e40 2023-12-19T09:22:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4280ceea0 con 0x7fb428158e40 2023-12-19T09:22:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41801a9f0 con 0x7fb428158e40 2023-12-19T09:22:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.746+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4280ce8c0 con 0x7fb428158e40 2023-12-19T09:22:09.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.750+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42804be40 con 0x7fb428158e40 2023-12-19T09:22:09.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.750+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fb41802d440 con 0x7fb428158e40 2023-12-19T09:22:09.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.754+0000 7fb41dffb700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 0x7fb40c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:09.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.754+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fb41807e6a0 con 0x7fb428158e40 2023-12-19T09:22:09.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.754+0000 7fb42dce1700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 0x7fb40c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:09.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.754+0000 7fb42dce1700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 0x7fb40c062a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb424006fd0 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:09.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.758+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb41804e9a0 con 0x7fb428158e40 2023-12-19T09:22:09.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4280cb070 con 0x7fb40c060580 2023-12-19T09:22:09.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb41dffb700 1 -- 172.21.15.111:0/1525991899 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb4280cb070 con 0x7fb40c060580 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 msgr2=0x7fb40c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 0x7fb40c062a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb424006fd0 tx=0x7fb424008040 comp rx=0 tx=0).stop 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 msgr2=0x7fb4280c48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.930+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb428088e10 tx=0x7fb418017000 comp rx=0 tx=0).stop 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 shutdown_connections 2023-12-19T09:22:09.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb428159210 0x7fb42815bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb428158e40 0x7fb4280c48e0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fb40c060580 0x7fb40c062a30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 --2- 172.21.15.111:0/1525991899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb428068460 0x7fb4280c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 >> 172.21.15.111:0/1525991899 conn(0x7fb4280d0900 msgr2=0x7fb4280b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 shutdown_connections 2023-12-19T09:22:09.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:09.934+0000 7fb42ece3700 1 -- 172.21.15.111:0/1525991899 wait complete. 2023-12-19T09:22:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 6 to deep-scrub 2023-12-19T09:22:09.958 INFO:tasks.ceph:Scrubbing osd.7 2023-12-19T09:22:09.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-19T09:22:10.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.114+0000 7f12ce585700 1 Processor -- start 2023-12-19T09:22:10.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.114+0000 7f12ce585700 1 -- start start 2023-12-19T09:22:10.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12ce585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c8155f70 0x7f12c80b2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12ce585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12c80cc5b0 0x7f12c80c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12ce585700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c80c7460 con 0x7f12c80c9de0 2023-12-19T09:22:10.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12ce585700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c80cb150 con 0x7f12c80cc5b0 2023-12-19T09:22:10.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12ce585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80ca200 con 0x7f12c8155f70 2023-12-19T09:22:10.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12c80cc5b0 0x7f12c80c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12c80cc5b0 0x7f12c80c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55314/0 (socket says 172.21.15.111:55314) 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 -- 172.21.15.111:0/2869393661 learned_addr learned my addr 172.21.15.111:0/2869393661 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12c7fff700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c8155f70 0x7f12c80b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 -- 172.21.15.111:0/2869393661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c8155f70 msgr2=0x7f12c80b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c8155f70 0x7f12c80b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 -- 172.21.15.111:0/2869393661 >> v1:172.21.15.143:6789/0 conn(0x7f12c80c9de0 legacy=0x7f12c8068a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.118+0000 7f12bffff700 1 -- 172.21.15.111:0/2869393661 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80b2d80 con 0x7f12c80cc5b0 2023-12-19T09:22:10.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12bffff700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c98a0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f12b00099b0 tx=0x7f12b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7c1e529567b870 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12c77fe700 1 -- 172.21.15.111:0/2869393661 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000b860 con 0x7f12c80cc5b0 2023-12-19T09:22:10.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12c77fe700 1 -- 172.21.15.111:0/2869393661 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000b9c0 con 0x7f12c80cc5b0 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12c77fe700 1 -- 172.21.15.111:0/2869393661 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000bd60 con 0x7f12c80cc5b0 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 -- 172.21.15.111:0/2869393661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c80cc5b0 msgr2=0x7f12c80c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c98a0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f12b00099b0 tx=0x7f12b0009250 comp rx=0 tx=0).stop 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 -- 172.21.15.111:0/2869393661 shutdown_connections 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c98a0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 --2- 172.21.15.111:0/2869393661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c8155f70 0x7f12c80b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 -- 172.21.15.111:0/2869393661 >> 172.21.15.111:0/2869393661 conn(0x7f12c8096190 msgr2=0x7f12c80afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 -- 172.21.15.111:0/2869393661 shutdown_connections 2023-12-19T09:22:10.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.122+0000 7f12ce585700 1 -- 172.21.15.111:0/2869393661 wait complete. 2023-12-19T09:22:10.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 Processor -- start 2023-12-19T09:22:10.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 -- start start 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c80c9de0 0x7f12c80c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 0x7f12c814c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80c64c0 con 0x7f12c8155f70 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80c4dd0 con 0x7f12c80cc5b0 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80c58b0 con 0x7f12c80c9de0 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 0x7f12c814c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 0x7f12c814c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55330/0 (socket says 172.21.15.111:55330) 2023-12-19T09:22:10.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 -- 172.21.15.111:0/1813238692 learned_addr learned my addr 172.21.15.111:0/1813238692 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12c7fff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c80c9de0 0x7f12c80c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12bffff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 -- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c80c9de0 msgr2=0x7f12c80c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12c80c9de0 0x7f12c80c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 -- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12c80cc5b0 msgr2=0x7f12c80c4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12c80cc5b0 0x7f12c80c4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00094f0 con 0x7f12c8155f70 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ccb22700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 0x7f12c814c540 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f12c000c840 tx=0x7f12c0001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0012630 con 0x7f12c8155f70 2023-12-19T09:22:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c00127b0 con 0x7f12c8155f70 2023-12-19T09:22:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0012b70 con 0x7f12c8155f70 2023-12-19T09:22:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.126+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c80c5b90 con 0x7f12c8155f70 2023-12-19T09:22:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.130+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c80c6150 con 0x7f12c8155f70 2023-12-19T09:22:10.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.130+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12c80708a0 con 0x7f12c8155f70 2023-12-19T09:22:10.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.130+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f12c001a3a0 con 0x7f12c8155f70 2023-12-19T09:22:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c4ff9700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f12a8060550 0x7f12a8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f12c007cc40 con 0x7f12c8155f70 2023-12-19T09:22:10.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c4ff9700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f12a8067200 0x7f12a8069660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12a8069c90 con 0x7f12a8067200 2023-12-19T09:22:10.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=213) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12c0081220 con 0x7f12c8155f70 2023-12-19T09:22:10.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c7fff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f12a8060550 0x7f12a8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12bffff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f12a8067200 0x7f12a8069660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12c7fff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f12a8060550 0x7f12a8062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006aa0 tx=0x7f12b8008040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.134+0000 7f12bffff700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f12a8067200 0x7f12a8069660 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.138+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== osd.7 v2:172.21.15.143:6824/34139 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f12a8069c90 con 0x7f12a8067200 2023-12-19T09:22:10.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 --> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f12c80966d0 con 0x7f12a8067200 2023-12-19T09:22:10.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12c4ff9700 1 -- 172.21.15.111:0/1813238692 <== osd.7 v2:172.21.15.143:6824/34139 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f12c80966d0 con 0x7f12a8067200 2023-12-19T09:22:10.162 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-12-19T09:22:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-19T09:22:10.162 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f12a8067200 msgr2=0x7f12a8069660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6824/34139,v1:172.21.15.143:6825/34139] conn(0x7f12a8067200 0x7f12a8069660 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f12a8060550 msgr2=0x7f12a8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f12a8060550 0x7f12a8062a00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12b8006aa0 tx=0x7f12b8008040 comp rx=0 tx=0).stop 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 msgr2=0x7f12c814c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 --2- 172.21.15.111:0/1813238692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12c8155f70 0x7f12c814c540 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f12c000c840 tx=0x7f12c0001670 comp rx=0 tx=0).stop 2023-12-19T09:22:10.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ccb22700 1 -- 172.21.15.111:0/1813238692 reap_dead start 2023-12-19T09:22:10.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 shutdown_connections 2023-12-19T09:22:10.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 >> 172.21.15.111:0/1813238692 conn(0x7f12c8096190 msgr2=0x7f12c806bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 shutdown_connections 2023-12-19T09:22:10.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.154+0000 7f12ce585700 1 -- 172.21.15.111:0/1813238692 wait complete. 2023-12-19T09:22:10.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-19T09:22:10.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.346+0000 7f9c2e616700 1 Processor -- start 2023-12-19T09:22:10.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.346+0000 7f9c2e616700 1 -- start start 2023-12-19T09:22:10.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2e616700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c28067060 0x7f9c280ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2e616700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c280aa880 0x7f9c28067430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2e616700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c28067d40 con 0x7f9c28067970 2023-12-19T09:22:10.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2e616700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2806ab30 con 0x7f9c280aa880 2023-12-19T09:22:10.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2e616700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2806a850 con 0x7f9c28067060 2023-12-19T09:22:10.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c280aa880 0x7f9c28067430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c280aa880 0x7f9c28067430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55338/0 (socket says 172.21.15.111:55338) 2023-12-19T09:22:10.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 -- 172.21.15.111:0/4249988902 learned_addr learned my addr 172.21.15.111:0/4249988902 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:10.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2d614700 1 --2- 172.21.15.111:0/4249988902 >> v2:172.21.15.143:3300/0 conn(0x7f9c28067060 0x7f9c280ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 -- 172.21.15.111:0/4249988902 >> v2:172.21.15.143:3300/0 conn(0x7f9c28067060 msgr2=0x7f9c280ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:10.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 --2- 172.21.15.111:0/4249988902 >> v2:172.21.15.143:3300/0 conn(0x7f9c28067060 0x7f9c280ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 -- 172.21.15.111:0/4249988902 >> v1:172.21.15.143:6789/0 conn(0x7f9c28067970 legacy=0x7f9c28161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 -- 172.21.15.111:0/4249988902 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28067f80 con 0x7f9c280aa880 2023-12-19T09:22:10.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c2ce13700 1 --2- 172.21.15.111:0/4249988902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c280aa880 0x7f9c28067430 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f9c2806ce10 tx=0x7f9c18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4640220b22599e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c1ffff700 1 -- 172.21.15.111:0/4249988902 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18009e70 con 0x7f9c280aa880 2023-12-19T09:22:10.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c1ffff700 1 -- 172.21.15.111:0/4249988902 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1800b860 con 0x7f9c280aa880 2023-12-19T09:22:10.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.350+0000 7f9c1ffff700 1 -- 172.21.15.111:0/4249988902 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1800bc20 con 0x7f9c280aa880 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 -- 172.21.15.111:0/4249988902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c280aa880 msgr2=0x7f9c28067430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 --2- 172.21.15.111:0/4249988902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c280aa880 0x7f9c28067430 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f9c2806ce10 tx=0x7f9c18009250 comp rx=0 tx=0).stop 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 -- 172.21.15.111:0/4249988902 shutdown_connections 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 --2- 172.21.15.111:0/4249988902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c280aa880 0x7f9c28067430 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 --2- 172.21.15.111:0/4249988902 >> v2:172.21.15.143:3300/0 conn(0x7f9c28067060 0x7f9c280ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 -- 172.21.15.111:0/4249988902 >> 172.21.15.111:0/4249988902 conn(0x7f9c280cc530 msgr2=0x7f9c280c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 -- 172.21.15.111:0/4249988902 shutdown_connections 2023-12-19T09:22:10.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 -- 172.21.15.111:0/4249988902 wait complete. 2023-12-19T09:22:10.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.354+0000 7f9c2e616700 1 Processor -- start 2023-12-19T09:22:10.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- start start 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c28067970 0x7f9c280acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c280aa880 0x7f9c280ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280c2a70 con 0x7f9c28067060 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2816dc00 con 0x7f9c28067970 2023-12-19T09:22:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c280caa30 con 0x7f9c280aa880 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55352/0 (socket says 172.21.15.111:55352) 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 -- 172.21.15.111:0/712509832 learned_addr learned my addr 172.21.15.111:0/712509832 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2ce13700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c28067970 0x7f9c280acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2de15700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c280aa880 0x7f9c280ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 -- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c280aa880 msgr2=0x7f9c280ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c280aa880 0x7f9c280ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 -- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c28067970 msgr2=0x7f9c280acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c28067970 0x7f9c280acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 -- 172.21.15.111:0/712509832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c180094f0 con 0x7f9c28067060 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2d614700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400e930 tx=0x7f9c240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24013680 con 0x7f9c28067060 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c240137e0 con 0x7f9c28067060 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c280c2cc0 con 0x7f9c28067060 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24013b80 con 0x7f9c28067060 2023-12-19T09:22:10.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.358+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c280c3280 con 0x7f9c28067060 2023-12-19T09:22:10.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.362+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c280c3470 con 0x7f9c28067060 2023-12-19T09:22:10.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.362+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f9c2400c460 con 0x7f9c28067060 2023-12-19T09:22:10.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.362+0000 7f9c1d7fa700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 0x7f9c0c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.366+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f9c2407cc00 con 0x7f9c28067060 2023-12-19T09:22:10.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.366+0000 7f9c2ce13700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 0x7f9c0c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.366+0000 7f9c2ce13700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 0x7f9c0c062a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c2806ce10 tx=0x7f9c18017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.370+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c24083080 con 0x7f9c28067060 2023-12-19T09:22:10.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.546+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c2804be40 con 0x7f9c0c060550 2023-12-19T09:22:10.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.546+0000 7f9c1d7fa700 1 -- 172.21.15.111:0/712509832 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9c2804be40 con 0x7f9c0c060550 2023-12-19T09:22:10.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 msgr2=0x7f9c0c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 0x7f9c0c062a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c2806ce10 tx=0x7f9c18017000 comp rx=0 tx=0).stop 2023-12-19T09:22:10.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 msgr2=0x7f9c280aec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400e930 tx=0x7f9c240091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 shutdown_connections 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f9c0c060550 0x7f9c0c062a00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c28067970 0x7f9c280acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c280aa880 0x7f9c280ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 --2- 172.21.15.111:0/712509832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c28067060 0x7f9c280aec90 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 >> 172.21.15.111:0/712509832 conn(0x7f9c280cc530 msgr2=0x7f9c28163040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 shutdown_connections 2023-12-19T09:22:10.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.550+0000 7f9c2e616700 1 -- 172.21.15.111:0/712509832 wait complete. 2023-12-19T09:22:10.560 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 7 to deep-scrub 2023-12-19T09:22:10.578 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:22:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.730+0000 7fc53cb85700 1 Processor -- start 2023-12-19T09:22:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.730+0000 7fc53cb85700 1 -- start start 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc538155f20 0x7fc538068230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5380c78e0 0x7fc5380c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc538067da0 con 0x7fc538068770 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5380c8250 con 0x7fc538155f20 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc538067190 con 0x7fc5380c78e0 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc535d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc538068770 0x7fc5380ad980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:50888/0 (socket says 172.21.15.111:50888) 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc535d9b700 1 -- 172.21.15.111:0/1009164997 learned_addr learned my addr 172.21.15.111:0/1009164997 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc538155f20 0x7fc538068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc536d9d700 1 --2- 172.21.15.111:0/1009164997 >> v2:172.21.15.143:3300/0 conn(0x7fc5380c78e0 0x7fc5380c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53559a700 1 -- 172.21.15.111:0/1009164997 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476914509 0 0) 0x7fc538067da0 con 0x7fc538068770 2023-12-19T09:22:10.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 -- 172.21.15.111:0/1009164997 >> v2:172.21.15.143:3300/0 conn(0x7fc5380c78e0 msgr2=0x7fc5380c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:10.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 --2- 172.21.15.111:0/1009164997 >> v2:172.21.15.143:3300/0 conn(0x7fc5380c78e0 0x7fc5380c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 -- 172.21.15.111:0/1009164997 >> v1:172.21.15.111:6789/0 conn(0x7fc538068770 legacy=0x7fc5380ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 -- 172.21.15.111:0/1009164997 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538067470 con 0x7fc538155f20 2023-12-19T09:22:10.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53659c700 1 --2- 172.21.15.111:0/1009164997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538155f20 0x7fc538068230 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc53806d6a0 tx=0x7fc5280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe3dac110d11275 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53559a700 1 -- 172.21.15.111:0/1009164997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800b6b0 con 0x7fc538155f20 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53559a700 1 -- 172.21.15.111:0/1009164997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52800b810 con 0x7fc538155f20 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53559a700 1 -- 172.21.15.111:0/1009164997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800bbb0 con 0x7fc538155f20 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- 172.21.15.111:0/1009164997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538155f20 msgr2=0x7fc538068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 --2- 172.21.15.111:0/1009164997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538155f20 0x7fc538068230 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc53806d6a0 tx=0x7fc5280091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- 172.21.15.111:0/1009164997 shutdown_connections 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 --2- 172.21.15.111:0/1009164997 >> v2:172.21.15.143:3300/0 conn(0x7fc5380c78e0 0x7fc5380c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 --2- 172.21.15.111:0/1009164997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538155f20 0x7fc538068230 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- 172.21.15.111:0/1009164997 >> 172.21.15.111:0/1009164997 conn(0x7fc5380cc690 msgr2=0x7fc5380c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- 172.21.15.111:0/1009164997 shutdown_connections 2023-12-19T09:22:10.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.734+0000 7fc53cb85700 1 -- 172.21.15.111:0/1009164997 wait complete. 2023-12-19T09:22:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 Processor -- start 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 -- start start 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 0x7fc538154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 0x7fc538153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53815eb70 con 0x7fc5380c78e0 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53815e890 con 0x7fc538155f20 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53cb85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5381696a0 con 0x7fc538068770 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55368/0 (socket says 172.21.15.111:55368) 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 -- 172.21.15.111:0/274148260 learned_addr learned my addr 172.21.15.111:0/274148260 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53659c700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 0x7fc538154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc536d9d700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 0x7fc538153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 -- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 msgr2=0x7fc538154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 0x7fc538154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 -- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 msgr2=0x7fc538153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 0x7fc538153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 -- 172.21.15.111:0/274148260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280093c0 con 0x7fc5380c78e0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc536d9d700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 0x7fc538153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc53659c700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 0x7fc538154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc535d9b700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c5e0 tx=0x7fc52c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c005b30 con 0x7fc5380c78e0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.738+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c005c90 con 0x7fc5380c78e0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.742+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c007300 con 0x7fc5380c78e0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.742+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc538169980 con 0x7fc5380c78e0 2023-12-19T09:22:10.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.742+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53815e580 con 0x7fc5380c78e0 2023-12-19T09:22:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.742+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc538169500 con 0x7fc5380c78e0 2023-12-19T09:22:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.742+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7fc52c0076b0 con 0x7fc5380c78e0 2023-12-19T09:22:10.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.746+0000 7fc5267fc700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 0x7fc518062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:10.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.746+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7fc52c07bfe0 con 0x7fc5380c78e0 2023-12-19T09:22:10.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.746+0000 7fc53659c700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 0x7fc518062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:10.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.746+0000 7fc53659c700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 0x7fc518062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc528009390 tx=0x7fc528017000 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:10.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.750+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc538169500 con 0x7fc5380c78e0 2023-12-19T09:22:10.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.930+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc538169500 con 0x7fc518060550 2023-12-19T09:22:10.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.934+0000 7fc5267fc700 1 -- 172.21.15.111:0/274148260 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48366 (secure 0 0 0) 0x7fc538169500 con 0x7fc518060550 2023-12-19T09:22:10.943 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:10.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 msgr2=0x7fc518062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 0x7fc518062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc528009390 tx=0x7fc528017000 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 msgr2=0x7fc53815e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00c5e0 tx=0x7fc52c000fe0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 shutdown_connections 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7fc518060550 0x7fc518062a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc538155f20 0x7fc538153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380c78e0 0x7fc53815e040 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 --2- 172.21.15.111:0/274148260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc538068770 0x7fc538154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 >> 172.21.15.111:0/274148260 conn(0x7fc5380cc690 msgr2=0x7fc5380ca340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 shutdown_connections 2023-12-19T09:22:10.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:10.938+0000 7fc53cb85700 1 -- 172.21.15.111:0/274148260 wait complete. 2023-12-19T09:22:10.948 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:22:10.966 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-19T09:22:10.499279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2474116,"kb_used_data":22084,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752500604,"statfs":{"total":773094113280,"available":770560618496,"internally_reserved":0,"allocated":22614016,"data_stored":12690525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.687779"},"pg_stats":[{"pgid":"1.0","version":"211'370","reported_seq":1215,"reported_epoch":211,"state":"active+clean","last_fresh":"2023-12-19T09:21:56.587486+0000","last_change":"2023-12-19T07:47:08.252099+0000","last_active":"2023-12-19T09:21:56.587486+0000","last_peered":"2023-12-19T09:21:56.587486+0000","last_clean":"2023-12-19T09:21:56.587486+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:21:56.587486+0000","last_undegraded":"2023-12-19T09:21:56.587486+0000","last_fullsized":"2023-12-19T09:21:56.587486+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T07:47:07.078657+0000","last_clean_scrub_stamp":"2023-12-19T07:47:07.078657+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T11:17:30.273702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575975,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304580,"kb_used_data":3196,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067260,"statfs":{"total":96636764160,"available":96324874240,"internally_reserved":0,"allocated":3272704,"data_stored":2029218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2709999999999999,"15min":1.298},"min":{"1min":0.82799999999999996,"5min":0.75,"15min":0.71199999999999997},"max":{"1min":1.599,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2729999999999999,"15min":1.294},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.77500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.712},"last":1.1319999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1639999999999999,"15min":1.1759999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5509999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2,"15min":1.214},"min":{"1min":0.78900000000000003,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6279999999999999,"5min":1.7769999999999999,"15min":1.796},"last":1.351}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1619999999999999,"15min":1.177},"min":{"1min":0.67900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.135,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4530000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":1.214}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3049999999999999,"15min":1.3129999999999999},"min":{"1min":0.91100000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.528,"5min":1.845,"15min":1.9059999999999999},"last":1.327},{"interface":"front","average":{"1min":1.276,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":0.98499999999999999,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.371}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.93600000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.5069999999999999,"5min":1.7330000000000001,"15min":1.95},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.94799999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.609,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.391}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2490000000000001,"15min":1.288},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.665,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.048},{"interface":"front","average":{"1min":1.329,"5min":1.3919999999999999,"15min":1.4139999999999999},"min":{"1min":1.121,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.714,"5min":1.972,"15min":1.972},"last":1.4219999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.97199999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.893,"15min":2.0750000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":0.83799999999999997,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.6699999999999999,"5min":1.9530000000000001,"15min":1.986},"last":1.1739999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.143,"15min":1.141},"min":{"1min":0.76800000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.579,"5min":1.6799999999999999,"15min":1.784},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1719999999999999,"15min":1.1619999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.55100000000000005},"max":{"1min":1.641,"5min":1.675,"15min":1.946},"last":0.59299999999999997}]},{"osd":1,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1579999999999999,"15min":1.1859999999999999},"min":{"1min":0.747,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.506,"5min":1.603,"15min":2.3439999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2150000000000001,"15min":1.2050000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.59099999999999997},"max":{"1min":1.5569999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.161}]},{"osd":2,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.53400000000000003},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.827},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.175,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.7529999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.27,"15min":1.2929999999999999},"min":{"1min":0.82799999999999996,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.6799999999999999,"5min":1.798,"15min":1.8600000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.3200000000000001,"15min":1.3089999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.579,"5min":1.74,"15min":1.8220000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2310000000000001,"15min":1.228},"min":{"1min":0.81000000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.702,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.446},"min":{"1min":1.0720000000000001,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.8160000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.121}]},{"osd":5,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3149999999999999,"15min":1.286},"min":{"1min":0.872,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.8049999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.276},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.333,"15min":1.3080000000000001},"min":{"1min":0.96699999999999997,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.8360000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.25}]},{"osd":7,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3919999999999999,"15min":1.393},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.8799999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.4339999999999999,"15min":1.4379999999999999},"min":{"1min":1.196,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.2410000000000001},"last":1.329}]}]},{"osd":5,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308096,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063744,"statfs":{"total":96636764160,"available":96321273856,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.82099999999999995,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.605,"5min":2.044,"15min":2.044},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3660000000000001,"15min":1.3580000000000001},"min":{"1min":1.0289999999999999,"5min":0.84899999999999998,"15min":0.83599999999999997},"max":{"1min":1.7490000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.383}]},{"osd":1,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.161},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.444,"5min":1.502,"15min":1.6479999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.208,"15min":1.21},"min":{"1min":1.004,"5min":0.84799999999999998,"15min":0.58099999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.2450000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.222,"15min":1.2110000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6830000000000001,"5min":1.782,"15min":1.782},"last":1.107},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.236,"15min":1.244},"min":{"1min":0.873,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5429999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.89500000000000002}]},{"osd":3,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3149999999999999},"min":{"1min":0.89700000000000002,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.7350000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.3620000000000001,"15min":1.3380000000000001},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6899999999999999,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":0.96899999999999997}]},{"osd":4,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.278,"15min":1.24},"min":{"1min":0.98499999999999999,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":1.6120000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.034},{"interface":"front","average":{"1min":1.333,"5min":1.3600000000000001,"15min":1.3169999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.59099999999999997},"max":{"1min":1.7250000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.0169999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.345,"15min":1.325},"min":{"1min":0.82099999999999995,"5min":0.72799999999999998,"15min":0.69199999999999995},"max":{"1min":1.6910000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3919999999999999,"15min":1.3759999999999999},"min":{"1min":0.96699999999999997,"5min":0.90400000000000003,"15min":0.68999999999999995},"max":{"1min":1.6950000000000001,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.1970000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4870000000000001,"15min":1.45},"min":{"1min":1.2050000000000001,"5min":1.018,"15min":0.84199999999999997},"max":{"1min":1.796,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.333},{"interface":"front","average":{"1min":1.514,"5min":1.512,"15min":1.49},"min":{"1min":0.99399999999999999,"5min":0.90900000000000003,"15min":0.79500000000000004},"max":{"1min":1.7609999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.2110000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.27,"15min":1.256},"min":{"1min":0.91000000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6579999999999999,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.008},{"interface":"front","average":{"1min":1.387,"5min":1.379,"15min":1.3700000000000001},"min":{"1min":1.0760000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7849999999999999,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.327}]},{"osd":1,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.21,"15min":1.202},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.758},"last":1.554},{"interface":"front","average":{"1min":1.163,"5min":1.1519999999999999,"15min":1.1859999999999999},"min":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.677,"5min":1.677,"15min":1.6779999999999999},"last":1.1060000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2230000000000001,"15min":1.204},"min":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.6719999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.173},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.2969999999999999,"15min":1.3009999999999999},"min":{"1min":1.0089999999999999,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.786,"5min":1.925,"15min":1.925},"last":1.45}]},{"osd":3,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.274,"15min":1.3089999999999999},"min":{"1min":0.91800000000000004,"5min":0.746,"15min":0.746},"max":{"1min":1.639,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":1.0209999999999999,"5min":0.88,"15min":0.88},"max":{"1min":1.746,"5min":1.746,"15min":2.4300000000000002},"last":1.29}]},{"osd":5,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.264},"min":{"1min":0.81999999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":2.0179999999999998},"last":1.647},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2589999999999999,"15min":1.268},"min":{"1min":0.92800000000000005,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.8220000000000001,"5min":1.879,"15min":1.879},"last":1.383}]},{"osd":6,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.28},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9239999999999999,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.423},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.4219999999999999,"15min":1.4279999999999999},"min":{"1min":0.92100000000000004,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.913,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.7150000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:09 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3680000000000001,"15min":1.371},"min":{"1min":1.115,"5min":0.624,"15min":0.624},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.468,"15min":1.4610000000000001},"min":{"1min":0.94099999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.8380000000000001,"5min":1.9710000000000001,"15min":1.972},"last":1.5940000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88800000000000001,"15min":0.878},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.254,"5min":1.581,"15min":1.581},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.96299999999999997,"15min":0.98799999999999999},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.494,"5min":2.117,"15min":2.117},"last":0.93300000000000005}]},{"osd":1,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.92200000000000004,"15min":0.93500000000000005},"min":{"1min":0.54800000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1579999999999999,"5min":2.331,"15min":2.331},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0529999999999999},"min":{"1min":0.69099999999999995,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.5649999999999999,"5min":2.23,"15min":2.23},"last":0.874}]},{"osd":2,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.103,"15min":1.1200000000000001},"min":{"1min":0.69499999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.677,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.018},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1799999999999999,"15min":1.1919999999999999},"min":{"1min":0.89000000000000001,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.7190000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.165}]},{"osd":4,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.331,"15min":1.3200000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.70199999999999996},"max":{"1min":1.8779999999999999,"5min":2.157,"15min":2.157},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.2909999999999999},"min":{"1min":1.073,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":1.9390000000000001,"5min":2.149,"15min":2.149},"last":1.1439999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3220000000000001,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1400000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.05},{"interface":"front","average":{"1min":1.371,"5min":1.407,"15min":1.3879999999999999},"min":{"1min":1.0529999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0409999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.3220000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.413,"15min":1.423},"min":{"1min":1.1279999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.923,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4079999999999999,"15min":1.4239999999999999},"min":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.8520000000000001,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.6819999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.5009999999999999},"min":{"1min":1.1419999999999999,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":1.992,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.627},{"interface":"front","average":{"1min":1.474,"5min":1.4710000000000001,"15min":1.456},"min":{"1min":1.0620000000000001,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0489999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.218}]}]},{"osd":2,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86399999999999999,"15min":0.92400000000000004},"min":{"1min":0.54400000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1000000000000001,"5min":1.345,"15min":1.5049999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0249999999999999,"15min":1.0880000000000001},"min":{"1min":0.63700000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1259999999999999,"5min":1.3839999999999999,"15min":1.6759999999999999},"last":0.76500000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.67000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1479999999999999,"5min":1.569,"15min":1.7609999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.98699999999999999,"15min":1.032},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.365,"5min":1.5369999999999999,"15min":1.768},"last":0.51800000000000002}]},{"osd":3,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.93400000000000005,"15min":0.96899999999999997},"min":{"1min":0.51500000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.627,"5min":1.627,"15min":1.891},"last":1.627},{"interface":"front","average":{"1min":0.999,"5min":1.091,"15min":1.1399999999999999},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.331,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":0.85199999999999998}]},{"osd":4,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2829999999999999,"15min":1.331},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4490000000000001,"5min":1.6539999999999999,"15min":1.754},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.2789999999999999,"15min":1.3220000000000001},"min":{"1min":0.91200000000000003,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.8520000000000001},"last":1.0409999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3180000000000001,"15min":1.3700000000000001},"min":{"1min":0.94699999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6919999999999999,"5min":1.948,"15min":2.016},"last":1.093},{"interface":"front","average":{"1min":1.169,"5min":1.2490000000000001,"15min":1.3009999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.68300000000000005},"max":{"1min":1.6599999999999999,"5min":1.756,"15min":1.927},"last":1.1419999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.268,"15min":1.3320000000000001},"min":{"1min":0.84299999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.738,"5min":1.8200000000000001,"15min":2.2189999999999999},"last":1.018},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.3260000000000001,"15min":1.405},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.571,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.409}]},{"osd":7,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.3919999999999999,"15min":1.431},"min":{"1min":0.99399999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.327},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.3480000000000001,"15min":1.413},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.802,"5min":1.913,"15min":2.1139999999999999},"last":1.288}]}]},{"osd":1,"up_from":13,"seq":55834575975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.87},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.9039999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.6850000000000001},"last":1.0209999999999999}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.996,"5min":1.0209999999999999,"15min":1.0629999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":2.302},"last":0.95099999999999996}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.175},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.097}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.073},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2929999999999999,"15min":1.3220000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.546,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":1.355}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.296,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.899,"5min":1.944,"15min":1.944},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.304}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.877,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.95299999999999996,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.752,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.204}]},{"osd":7,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.825,"5min":1.825,"15min":2.069},"last":1.335},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4259999999999999,"15min":1.4359999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.385}]}]},{"osd":0,"up_from":13,"seq":55834575974,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300480,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071360,"statfs":{"total":96636764160,"available":96329072640,"internally_reserved":0,"allocated":3268608,"data_stored":2028207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.97099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3069999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.81599999999999995}]},{"osd":2,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.995,"15min":1.0169999999999999},"min":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3759999999999999,"5min":1.446,"15min":1.5269999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.728,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2709999999999999,"5min":1.611,"15min":2.621},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.177},"min":{"1min":0.61599999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.26}]},{"osd":4,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.341,"15min":1.377},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.605,"5min":1.891,"15min":1.891},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.333,"15min":1.347},"min":{"1min":0.81999999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.5339999999999998},"last":1.52}]},{"osd":5,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3759999999999999,"15min":1.3939999999999999},"min":{"1min":0.74299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.966,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.488}]},{"osd":6,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2869999999999999,"15min":1.2849999999999999},"min":{"1min":0.88800000000000001,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.4489999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.373},"min":{"1min":0.98299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.837,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.3620000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.339,"15min":1.3400000000000001},"min":{"1min":0.86399999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.746,"5min":2.137,"15min":2.8570000000000002},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4239999999999999,"15min":1.4299999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0640000000000001,"5min":2.1259999999999999,"15min":2.181},"last":1.4570000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:22:10.970 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-19T07:47:07.078657+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=19, tm_hour=7, tm_min=47, tm_sec=7, tm_wday=1, tm_yday=353, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=19, tm_hour=9, tm_min=22, tm_sec=4, tm_wday=1, tm_yday=353, tm_isdst=0) 2023-12-19T09:22:10.971 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-19T09:22:28.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:22:28.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:22:30.972 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T09:22:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.138+0000 7f0bea2cd700 1 Processor -- start 2023-12-19T09:22:31.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.138+0000 7f0bea2cd700 1 -- start start 2023-12-19T09:22:31.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0bea2cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0be40bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0bea2cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0be40682a0 0x7f0be406a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0bea2cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be4067e10 con 0x7f0be40bec60 2023-12-19T09:22:31.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0bea2cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be4068740 con 0x7f0be40682a0 2023-12-19T09:22:31.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0bea2cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4067200 con 0x7f0be40c7080 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0be40bec60 0x7f0be40c6970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60632/0 (socket says 172.21.15.111:60632) 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 -- 172.21.15.111:0/2075959117 learned_addr learned my addr 172.21.15.111:0/2075959117 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0be40bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be886a700 1 --2- 172.21.15.111:0/2075959117 >> v2:172.21.15.143:3300/0 conn(0x7f0be40682a0 0x7f0be406a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210759500 0 0) 0x7f0be4067e10 con 0x7f0be40bec60 2023-12-19T09:22:31.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc003570 con 0x7f0be40bec60 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 135860047 0 0) 0x7f0bcc003570 con 0x7f0be40bec60 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 msgr2=0x7f0be40bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0be40bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 >> v2:172.21.15.143:3300/0 conn(0x7f0be40682a0 msgr2=0x7f0be406a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 --2- 172.21.15.111:0/2075959117 >> v2:172.21.15.143:3300/0 conn(0x7f0be40682a0 0x7f0be406a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be40674e0 con 0x7f0be40bec60 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4133348485 0 0) 0x7f0bd40015f0 con 0x7f0be40bec60 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 >> v1:172.21.15.111:6789/0 conn(0x7f0be40bec60 legacy=0x7f0be40c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0bcc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0bcc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bcc004d50 0x7f0bcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4068740 con 0x7f0bcc003a60 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4067200 con 0x7f0bcc004d50 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc00ba60 con 0x7f0be40c7080 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0bcc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0bcc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be886a700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bcc004d50 0x7f0bcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 -- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 msgr2=0x7f0bcc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0bcc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 -- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bcc004d50 msgr2=0x7f0bcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bcc004d50 0x7f0bcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be3fff700 1 -- 172.21.15.111:0/2075959117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc003570 con 0x7f0bcc003a60 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.142+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40c7080 0x7f0bcc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0bcc003e70 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400c3c0 tx=0x7f0bd4007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8e7d93bf7576f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400cc30 con 0x7f0bcc003a60 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd400cd90 con 0x7f0bcc003a60 2023-12-19T09:22:31.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0be2ffd700 1 -- 172.21.15.111:0/2075959117 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4007320 con 0x7f0bcc003a60 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2075959117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 msgr2=0x7f0bcc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2075959117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0bcc003e70 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400c3c0 tx=0x7f0bd4007920 comp rx=0 tx=0).stop 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0be886a700 1 -- 172.21.15.111:0/2075959117 reap_dead start 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2075959117 shutdown_connections 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2075959117 >> 172.21.15.111:0/2075959117 conn(0x7f0be40cc690 msgr2=0x7f0be40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2075959117 shutdown_connections 2023-12-19T09:22:31.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.146+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2075959117 wait complete. 2023-12-19T09:22:31.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 Processor -- start 2023-12-19T09:22:31.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 -- start start 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40b0850 0x7f0be40bf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 0x7f0be414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be415b6f0 con 0x7f0bcc003a60 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be415fb00 con 0x7f0be40bf570 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0bea2cd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be415fde0 con 0x7f0be40b0850 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37278/0 (socket says 172.21.15.111:37278) 2023-12-19T09:22:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0be3fff700 1 -- 172.21.15.111:0/2461863379 learned_addr learned my addr 172.21.15.111:0/2461863379 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40b0850 0x7f0be40bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.150+0000 7f0be886a700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 0x7f0be414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 -- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40b0850 msgr2=0x7f0be40bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40b0850 0x7f0be40bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 -- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 msgr2=0x7f0be414ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 0x7f0be414ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 -- 172.21.15.111:0/2461863379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be41600c0 con 0x7f0bcc003a60 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be886a700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 0x7f0be414ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0be3fff700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4011000 tx=0x7f0bd4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400cc30 con 0x7f0bcc003a60 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd4002d40 con 0x7f0bcc003a60 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4005c50 con 0x7f0bcc003a60 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be41603a0 con 0x7f0bcc003a60 2023-12-19T09:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be41610e0 con 0x7f0bcc003a60 2023-12-19T09:22:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.154+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be4071110 con 0x7f0bcc003a60 2023-12-19T09:22:31.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.158+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f0bd4019030 con 0x7f0bcc003a60 2023-12-19T09:22:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.158+0000 7f0bcbfff700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 0x7f0bd0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.158+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f0bd407e310 con 0x7f0bcc003a60 2023-12-19T09:22:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.158+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 0x7f0bd0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.158+0000 7f0be37fe700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 0x7f0bd0062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0be415f140 tx=0x7f0bd80091a0 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.166+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0bd404e5b0 con 0x7f0bcc003a60 2023-12-19T09:22:31.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.342+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 --> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0be40677c0 con 0x7f0bd0060580 2023-12-19T09:22:31.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.350+0000 7f0bcbfff700 1 -- 172.21.15.111:0/2461863379 <== mgr.9002 v2:172.21.15.143:6832/43868 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48057 (secure 0 0 0) 0x7f0be40677c0 con 0x7f0bd0060580 2023-12-19T09:22:31.354 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:31.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.350+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 msgr2=0x7f0bd0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.350+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 0x7f0bd0062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0be415f140 tx=0x7f0bd80091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.350+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 msgr2=0x7f0be4160ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.350+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4011000 tx=0x7f0bd4015000 comp rx=0 tx=0).stop 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 shutdown_connections 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f0bd0060580 0x7f0bd0062a30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0be40bf570 0x7f0be414ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0be40b0850 0x7f0be40bf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 --2- 172.21.15.111:0/2461863379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bcc003a60 0x7f0be4160ba0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 >> 172.21.15.111:0/2461863379 conn(0x7f0be40cc690 msgr2=0x7f0be415ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:31.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 shutdown_connections 2023-12-19T09:22:31.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.354+0000 7f0bea2cd700 1 -- 172.21.15.111:0/2461863379 wait complete. 2023-12-19T09:22:31.359 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-12-19T09:22:31.378 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-19T09:22:30.505433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":2474116,"kb_used_data":22084,"kb_used_omap":0,"kb_used_meta":2451968,"kb_avail":752500604,"statfs":{"total":773094113280,"available":770560618496,"internally_reserved":0,"allocated":22614016,"data_stored":12690525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2510815232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.0","version":"213'371","reported_seq":1227,"reported_epoch":213,"state":"active+clean","last_fresh":"2023-12-19T09:22:11.387130+0000","last_change":"2023-12-19T09:22:11.387076+0000","last_active":"2023-12-19T09:22:11.387130+0000","last_peered":"2023-12-19T09:22:11.387130+0000","last_clean":"2023-12-19T09:22:11.387130+0000","last_became_active":"2023-12-19T07:47:08.249383+0000","last_became_peered":"2023-12-19T07:47:08.249383+0000","last_unstale":"2023-12-19T09:22:11.387130+0000","last_undegraded":"2023-12-19T09:22:11.387130+0000","last_fullsized":"2023-12-19T09:22:11.387130+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'371","last_scrub_stamp":"2023-12-19T09:22:11.386971+0000","last_deep_scrub":"213'371","last_deep_scrub_stamp":"2023-12-19T09:22:11.386971+0000","last_clean_scrub_stamp":"2023-12-19T09:22:11.386971+0000","objects_scrubbed":2,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-20T10:07:18.447586+0000","scrub_duration":0.013919944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575979,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304580,"kb_used_data":3196,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94067260,"statfs":{"total":96636764160,"available":96324874240,"internally_reserved":0,"allocated":3272704,"data_stored":2029218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2709999999999999,"15min":1.298},"min":{"1min":0.82799999999999996,"5min":0.75,"15min":0.71199999999999997},"max":{"1min":1.599,"5min":1.7190000000000001,"15min":1.8360000000000001},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2729999999999999,"15min":1.294},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.77500000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.712},"last":1.5740000000000001}]},{"osd":1,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1639999999999999,"15min":1.1759999999999999},"min":{"1min":0.79100000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5509999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2,"15min":1.214},"min":{"1min":0.78900000000000003,"5min":0.70199999999999996,"15min":0.61299999999999999},"max":{"1min":1.6279999999999999,"5min":1.7769999999999999,"15min":1.796},"last":1.3320000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.1619999999999999,"15min":1.177},"min":{"1min":0.67900000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.038},{"interface":"front","average":{"1min":1.135,"5min":1.2609999999999999,"15min":1.2789999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4530000000000001,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":1.214}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3049999999999999,"15min":1.3129999999999999},"min":{"1min":0.91100000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.528,"5min":1.845,"15min":1.9059999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.3560000000000001,"15min":1.3759999999999999},"min":{"1min":0.98499999999999999,"5min":0.91700000000000004,"15min":0.84199999999999997},"max":{"1min":1.7290000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.462}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.93600000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.5069999999999999,"5min":1.7330000000000001,"15min":1.95},"last":1.079},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2370000000000001,"15min":1.2470000000000001},"min":{"1min":0.94799999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.609,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.2709999999999999}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2490000000000001,"15min":1.288},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.665,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.514},{"interface":"front","average":{"1min":1.329,"5min":1.3919999999999999,"15min":1.4139999999999999},"min":{"1min":1.121,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.714,"5min":1.972,"15min":1.972},"last":1.502}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.97199999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.825,"5min":1.893,"15min":2.0750000000000002},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.3839999999999999,"15min":1.395},"min":{"1min":0.83799999999999997,"5min":0.82999999999999996,"15min":0.77300000000000002},"max":{"1min":1.6699999999999999,"5min":1.9530000000000001,"15min":1.986},"last":1.409}]}]},{"osd":6,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.143,"15min":1.141},"min":{"1min":0.76800000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.579,"5min":1.6799999999999999,"15min":1.784},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1719999999999999,"15min":1.1619999999999999},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.55100000000000005},"max":{"1min":1.641,"5min":1.675,"15min":1.946},"last":1.2689999999999999}]},{"osd":1,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1579999999999999,"15min":1.1859999999999999},"min":{"1min":0.747,"5min":0.68000000000000005,"15min":0.63800000000000001},"max":{"1min":1.506,"5min":1.603,"15min":2.3439999999999999},"last":1.143},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2150000000000001,"15min":1.2050000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.59099999999999997},"max":{"1min":1.5569999999999999,"5min":1.6799999999999999,"15min":2.5550000000000002},"last":1.661}]},{"osd":2,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.53400000000000003},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.827},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.7529999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.101}]},{"osd":3,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.27,"15min":1.2929999999999999},"min":{"1min":0.82799999999999996,"5min":0.78200000000000003,"15min":0.628},"max":{"1min":1.6799999999999999,"5min":1.798,"15min":1.8600000000000001},"last":1.016},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.3200000000000001,"15min":1.3089999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.579,"5min":1.74,"15min":1.8220000000000001},"last":1.466}]},{"osd":4,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2310000000000001,"15min":1.228},"min":{"1min":0.81000000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.702,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.415},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.452,"15min":1.446},"min":{"1min":1.0720000000000001,"5min":0.76300000000000001,"15min":0.70699999999999996},"max":{"1min":1.8160000000000001,"5min":1.9690000000000001,"15min":2.1139999999999999},"last":1.179}]},{"osd":5,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3149999999999999,"15min":1.286},"min":{"1min":0.872,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.8049999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.333},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.333,"15min":1.3080000000000001},"min":{"1min":0.96699999999999997,"5min":0.81000000000000005,"15min":0.65800000000000003},"max":{"1min":1.8360000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.6080000000000001}]},{"osd":7,"last update":"Tue Dec 19 09:22:06 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3919999999999999,"15min":1.393},"min":{"1min":0.98499999999999999,"5min":0.79400000000000004,"15min":0.70799999999999996},"max":{"1min":1.8799999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.458,"5min":1.4339999999999999,"15min":1.4379999999999999},"min":{"1min":1.196,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":2.2410000000000001},"last":1.2250000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":308096,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":94063744,"statfs":{"total":96636764160,"available":96321273856,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2989999999999999,"15min":1.3169999999999999},"min":{"1min":0.82099999999999995,"5min":0.81100000000000005,"15min":0.50700000000000001},"max":{"1min":1.605,"5min":2.044,"15min":2.044},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3660000000000001,"15min":1.3580000000000001},"min":{"1min":1.0289999999999999,"5min":0.84899999999999998,"15min":0.83599999999999997},"max":{"1min":1.7490000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.367}]},{"osd":1,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.161},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.444,"5min":1.502,"15min":1.6479999999999999},"last":1.268},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.208,"15min":1.21},"min":{"1min":1.004,"5min":0.84799999999999998,"15min":0.58099999999999996},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.333}]},{"osd":2,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.222,"15min":1.2110000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.6830000000000001,"5min":1.782,"15min":1.782},"last":1.145},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.236,"15min":1.244},"min":{"1min":0.873,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5429999999999999,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.115}]},{"osd":3,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3149999999999999},"min":{"1min":0.89700000000000002,"5min":0.84999999999999998,"15min":0.63700000000000001},"max":{"1min":1.7350000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.3620000000000001,"15min":1.3380000000000001},"min":{"1min":0.89700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6899999999999999,"5min":2.1749999999999998,"15min":2.1749999999999998},"last":1.298}]},{"osd":4,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.278,"15min":1.24},"min":{"1min":0.98499999999999999,"5min":0.79800000000000004,"15min":0.67200000000000004},"max":{"1min":1.6120000000000001,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.046},{"interface":"front","average":{"1min":1.333,"5min":1.3600000000000001,"15min":1.3169999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.59099999999999997},"max":{"1min":1.7250000000000001,"5min":2.1309999999999998,"15min":2.1309999999999998},"last":1.4390000000000001}]},{"osd":6,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.345,"15min":1.325},"min":{"1min":0.82099999999999995,"5min":0.72799999999999998,"15min":0.69199999999999995},"max":{"1min":1.6910000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.409},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3919999999999999,"15min":1.3759999999999999},"min":{"1min":0.96699999999999997,"5min":0.90400000000000003,"15min":0.68999999999999995},"max":{"1min":1.6950000000000001,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.1879999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.4870000000000001,"15min":1.45},"min":{"1min":1.2050000000000001,"5min":1.018,"15min":0.84199999999999997},"max":{"1min":1.796,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.512,"15min":1.49},"min":{"1min":0.99399999999999999,"5min":0.90900000000000003,"15min":0.79500000000000004},"max":{"1min":1.7609999999999999,"5min":2.3639999999999999,"15min":2.3639999999999999},"last":1.2410000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.266,"15min":1.256},"min":{"1min":0.68000000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.71,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":1.165},{"interface":"front","average":{"1min":1.272,"5min":1.3720000000000001,"15min":1.363},"min":{"1min":0.72199999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7270000000000001,"5min":3.4260000000000002,"15min":3.4260000000000002},"last":1.458}]},{"osd":1,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.202,"15min":1.204},"min":{"1min":0.81899999999999995,"5min":0.72799999999999998,"15min":0.70799999999999996},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":1.3},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1399999999999999,"15min":1.175},"min":{"1min":0.64200000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.585,"5min":1.677,"15min":1.6779999999999999},"last":1.3320000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1899999999999999,"15min":1.1950000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.397,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.288,"15min":1.298},"min":{"1min":0.90100000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.625,"5min":1.786,"15min":1.925},"last":1.2}]},{"osd":3,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2649999999999999,"15min":1.3009999999999999},"min":{"1min":0.747,"5min":0.746,"15min":0.746},"max":{"1min":1.651,"5min":1.6619999999999999,"15min":2.7440000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.375,"15min":1.393},"min":{"1min":0.97299999999999998,"5min":0.88,"15min":0.88},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":2.4300000000000002},"last":1.379}]},{"osd":5,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2470000000000001,"15min":1.264},"min":{"1min":0.82299999999999995,"5min":0.65000000000000002,"15min":0.54200000000000004},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.536},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.2569999999999999,"15min":1.27},"min":{"1min":0.95199999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"max":{"1min":1.845,"5min":1.845,"15min":1.879},"last":1.52}]},{"osd":6,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.3160000000000001,"15min":1.292},"min":{"1min":1.012,"5min":0.82999999999999996,"15min":0.753},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9770000000000001},"last":1.246},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.427,"15min":1.4330000000000001},"min":{"1min":1.0580000000000001,"5min":0.67300000000000004,"15min":0.64700000000000002},"max":{"1min":1.8089999999999999,"5min":1.9339999999999999,"15min":1.9379999999999999},"last":1.615}]},{"osd":7,"last update":"Tue Dec 19 09:22:11 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.367,"15min":1.3740000000000001},"min":{"1min":1.1040000000000001,"5min":0.624,"15min":0.624},"max":{"1min":1.7609999999999999,"5min":1.9410000000000001,"15min":1.9730000000000001},"last":1.423},{"interface":"front","average":{"1min":1.536,"5min":1.474,"15min":1.4710000000000001},"min":{"1min":0.999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.5629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.88800000000000001,"15min":0.878},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.40600000000000003},"max":{"1min":1.254,"5min":1.581,"15min":1.581},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.96299999999999997,"15min":0.98799999999999999},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.494,"5min":2.117,"15min":2.117},"last":1.004}]},{"osd":1,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.92200000000000004,"15min":0.93500000000000005},"min":{"1min":0.54800000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1579999999999999,"5min":2.331,"15min":2.331},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0529999999999999},"min":{"1min":0.69099999999999995,"5min":0.50700000000000001,"15min":0.48399999999999999},"max":{"1min":1.5649999999999999,"5min":2.23,"15min":2.23},"last":1.2270000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.103,"15min":1.1200000000000001},"min":{"1min":0.69499999999999995,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.677,"5min":2.6040000000000001,"15min":2.6040000000000001},"last":1.143},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.1799999999999999,"15min":1.1919999999999999},"min":{"1min":0.89000000000000001,"5min":0.81599999999999995,"15min":0.52700000000000002},"max":{"1min":1.7190000000000001,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":1.0580000000000001}]},{"osd":4,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.331,"15min":1.3200000000000001},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.70199999999999996},"max":{"1min":1.8779999999999999,"5min":2.157,"15min":2.157},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.2909999999999999},"min":{"1min":1.073,"5min":0.83999999999999997,"15min":0.68200000000000005},"max":{"1min":1.9390000000000001,"5min":2.149,"15min":2.149},"last":1.181}]},{"osd":5,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3220000000000001,"15min":1.3049999999999999},"min":{"1min":0.97599999999999998,"5min":0.77000000000000002,"15min":0.55400000000000005},"max":{"1min":2.1400000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":1.371},{"interface":"front","average":{"1min":1.371,"5min":1.407,"15min":1.3879999999999999},"min":{"1min":1.0529999999999999,"5min":0.81200000000000006,"15min":0.65100000000000002},"max":{"1min":2.0409999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.4039999999999999}]},{"osd":6,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.413,"15min":1.423},"min":{"1min":1.1279999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.923,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4079999999999999,"15min":1.4239999999999999},"min":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":0.82199999999999995},"max":{"1min":1.8520000000000001,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.5529999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.5009999999999999},"min":{"1min":1.1419999999999999,"5min":0.92700000000000005,"15min":0.86399999999999999},"max":{"1min":1.992,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.4710000000000001,"15min":1.456},"min":{"1min":1.0620000000000001,"5min":0.81899999999999995,"15min":0.75800000000000001},"max":{"1min":2.0489999999999999,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.4710000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.86399999999999999,"15min":0.92400000000000004},"min":{"1min":0.54400000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1000000000000001,"5min":1.345,"15min":1.5049999999999999},"last":0.873},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.0249999999999999,"15min":1.0880000000000001},"min":{"1min":0.63700000000000001,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1259999999999999,"5min":1.3839999999999999,"15min":1.6759999999999999},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.67000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1479999999999999,"5min":1.569,"15min":1.7609999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.98699999999999999,"15min":1.032},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.365,"5min":1.5369999999999999,"15min":1.768},"last":0.82299999999999995}]},{"osd":3,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.93400000000000005,"15min":0.96899999999999997},"min":{"1min":0.51500000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.627,"5min":1.627,"15min":1.891},"last":1.2470000000000001},{"interface":"front","average":{"1min":0.999,"5min":1.091,"15min":1.1399999999999999},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.331,"5min":1.5409999999999999,"15min":1.5589999999999999},"last":1.4039999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2829999999999999,"15min":1.331},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4490000000000001,"5min":1.6539999999999999,"15min":1.754},"last":1.484},{"interface":"front","average":{"1min":1.214,"5min":1.2789999999999999,"15min":1.3220000000000001},"min":{"1min":0.91200000000000003,"5min":0.878,"15min":0.82099999999999995},"max":{"1min":1.821,"5min":1.821,"15min":1.8520000000000001},"last":1.024}]},{"osd":5,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3180000000000001,"15min":1.3700000000000001},"min":{"1min":0.94699999999999995,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.6919999999999999,"5min":1.948,"15min":2.016},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.2490000000000001,"15min":1.3009999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.68300000000000005},"max":{"1min":1.6599999999999999,"5min":1.756,"15min":1.927},"last":1.044}]},{"osd":6,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.268,"15min":1.3320000000000001},"min":{"1min":0.84299999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.738,"5min":1.8200000000000001,"15min":2.2189999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.3260000000000001,"15min":1.405},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.571,"5min":1.9059999999999999,"15min":2.1909999999999998},"last":1.5389999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:22:03 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.3919999999999999,"15min":1.431},"min":{"1min":0.99399999999999999,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.327},"last":1.383},{"interface":"front","average":{"1min":1.238,"5min":1.3480000000000001,"15min":1.413},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.802,"5min":1.913,"15min":2.1139999999999999},"last":1.4279999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":312192,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":94059648,"statfs":{"total":96636764160,"available":96317079552,"internally_reserved":0,"allocated":2678784,"data_stored":1438850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.87},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.9039999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.6850000000000001},"last":0.88600000000000001}]},{"osd":2,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3819999999999999,"5min":1.6539999999999999,"15min":2.331},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.996,"5min":1.0209999999999999,"15min":1.0629999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":2.302},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.175},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0349999999999999,"15min":1.0720000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2529999999999999,"5min":1.694,"15min":1.76},"last":1.1779999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.411,"5min":2.411,"15min":2.411},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2929999999999999,"15min":1.3220000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.546,"5min":1.9910000000000001,"15min":2.0350000000000001},"last":1.28}]},{"osd":5,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.296,"15min":1.3280000000000001},"min":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.899,"5min":1.944,"15min":1.944},"last":1.431},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.323,"15min":1.3149999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.383}]},{"osd":6,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.877,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.95299999999999996,"5min":0.85499999999999998,"15min":0.75800000000000001},"max":{"1min":1.752,"5min":1.8520000000000001,"15min":2.0699999999999998},"last":1.6419999999999999}]},{"osd":7,"last update":"Tue Dec 19 09:21:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4450000000000001,"15min":1.456},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.825,"5min":1.825,"15min":2.069},"last":1.73},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4259999999999999,"15min":1.4359999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.7010000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575978,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300480,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94071360,"statfs":{"total":96636764160,"available":96329072640,"internally_reserved":0,"allocated":3268608,"data_stored":2028207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.83099999999999996,"15min":0.86699999999999999},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.95699999999999996,"15min":0.97099999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3069999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.995,"15min":1.0169999999999999},"min":{"1min":0.52400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3759999999999999,"5min":1.446,"15min":1.5269999999999999},"last":1.284},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.728,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.149}]},{"osd":3,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2709999999999999,"5min":1.611,"15min":2.621},"last":1.093},{"interface":"front","average":{"1min":1.196,"5min":1.163,"15min":1.177},"min":{"1min":0.61599999999999999,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.341,"15min":1.377},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.605,"5min":1.891,"15min":1.891},"last":1.409},{"interface":"front","average":{"1min":1.341,"5min":1.333,"15min":1.347},"min":{"1min":0.81999999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.5339999999999998},"last":1.538}]},{"osd":5,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3759999999999999,"15min":1.3939999999999999},"min":{"1min":0.74299999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.966,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":1.601},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.407,"15min":1.4159999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.79500000000000004},"max":{"1min":1.8440000000000001,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.482}]},{"osd":6,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2869999999999999,"15min":1.2849999999999999},"min":{"1min":0.88800000000000001,"5min":0.86799999999999999,"15min":0.85399999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.4489999999999998},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.3819999999999999,"15min":1.373},"min":{"1min":0.98299999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.837,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.331}]},{"osd":7,"last update":"Tue Dec 19 09:21:39 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.339,"15min":1.3400000000000001},"min":{"1min":0.86399999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.746,"5min":2.137,"15min":2.8570000000000002},"last":1.462},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4239999999999999,"15min":1.4299999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0640000000000001,"5min":2.1259999999999999,"15min":2.181},"last":1.841}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T09:22:31.380 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-19T09:22:31.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.534+0000 7f3a72e0c700 1 Processor -- start 2023-12-19T09:22:31.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.534+0000 7f3a72e0c700 1 -- start start 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0c64a0 0x7f3a6c0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3a6c0c70c0 0x7f3a6c0aba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0c6db0 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c067b10 con 0x7f3a6c0c70c0 2023-12-19T09:22:31.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a72e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c067df0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37286/0 (socket says 172.21.15.111:37286) 2023-12-19T09:22:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 -- 172.21.15.111:0/959063536 learned_addr learned my addr 172.21.15.111:0/959063536 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a71e0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0c64a0 0x7f3a6c0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a71609700 1 --2- 172.21.15.111:0/959063536 >> v2:172.21.15.143:3300/0 conn(0x7f3a6c0c70c0 0x7f3a6c0aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 -- 172.21.15.111:0/959063536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0c64a0 msgr2=0x7f3a6c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 --2- 172.21.15.111:0/959063536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0c64a0 0x7f3a6c0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 -- 172.21.15.111:0/959063536 >> v2:172.21.15.143:3300/0 conn(0x7f3a6c0c70c0 msgr2=0x7f3a6c0aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 --2- 172.21.15.111:0/959063536 >> v2:172.21.15.143:3300/0 conn(0x7f3a6c0c70c0 0x7f3a6c0aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 -- 172.21.15.111:0/959063536 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0680d0 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a7260b700 1 --2- 172.21.15.111:0/959063536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c06eba0 tx=0x7f3a64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd289e8cf6e2cafa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a70e08700 1 -- 172.21.15.111:0/959063536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400b640 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a70e08700 1 -- 172.21.15.111:0/959063536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6400b7a0 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.538+0000 7f3a70e08700 1 -- 172.21.15.111:0/959063536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6400bb60 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- 172.21.15.111:0/959063536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0abfc0 msgr2=0x7f3a6c0ac420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/959063536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c06eba0 tx=0x7f3a64009250 comp rx=0 tx=0).stop 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- 172.21.15.111:0/959063536 shutdown_connections 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/959063536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0abfc0 0x7f3a6c0ac420 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/959063536 >> v2:172.21.15.143:3300/0 conn(0x7f3a6c0c70c0 0x7f3a6c0aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/959063536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0c64a0 0x7f3a6c0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- 172.21.15.111:0/959063536 >> 172.21.15.111:0/959063536 conn(0x7f3a6c0cc530 msgr2=0x7f3a6c0c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:31.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- 172.21.15.111:0/959063536 shutdown_connections 2023-12-19T09:22:31.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- 172.21.15.111:0/959063536 wait complete. 2023-12-19T09:22:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 Processor -- start 2023-12-19T09:22:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.542+0000 7f3a72e0c700 1 -- start start 2023-12-19T09:22:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0abfc0 0x7f3a6c170460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 0x7f3a6c16ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c16e830 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c16df70 con 0x7f3a6c0c70c0 2023-12-19T09:22:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c162420 con 0x7f3a6c0abfc0 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37294/0 (socket says 172.21.15.111:37294) 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 -- 172.21.15.111:0/2843998398 learned_addr learned my addr 172.21.15.111:0/2843998398 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71e0a700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0abfc0 0x7f3a6c170460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a7260b700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 0x7f3a6c16ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 -- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0abfc0 msgr2=0x7f3a6c170460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0abfc0 0x7f3a6c170460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 -- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 msgr2=0x7f3a6c16ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 0x7f3a6c16ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 -- 172.21.15.111:0/2843998398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640094f0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a7260b700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 0x7f3a6c16ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a71609700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00ecb0 tx=0x7f3a5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c00c920 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c00ca80 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c00ce20 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c16e1e0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.546+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c16d370 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.550+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c16e9e0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.550+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f3a5c013680 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.554+0000 7f3a61ffb700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 0x7f3a50062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.554+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f3a5c07cc80 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.554+0000 7f3a71e0a700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 0x7f3a50062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.554+0000 7f3a71e0a700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 0x7f3a50062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009fd0 tx=0x7f3a6800b040 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.558+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3a6c16e9e0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.730+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f3a6c16e9e0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.742+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3a5c04cfc0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.742+0000 7f3a61ffb700 1 -- 172.21.15.111:0/2843998398 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v76)=0 v76) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3a6c16e9e0 con 0x7f3a6c0c64a0 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 msgr2=0x7f3a50062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 0x7f3a50062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009fd0 tx=0x7f3a6800b040 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 msgr2=0x7f3a6c1709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00ecb0 tx=0x7f3a5c0091a0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 shutdown_connections 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f3a50060580 0x7f3a50062a30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a6c0c70c0 0x7f3a6c16ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0c64a0 0x7f3a6c1709a0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 --2- 172.21.15.111:0/2843998398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a6c0abfc0 0x7f3a6c170460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 >> 172.21.15.111:0/2843998398 conn(0x7f3a6c0cc530 msgr2=0x7f3a6c06a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:31.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 shutdown_connections 2023-12-19T09:22:31.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.746+0000 7f3a72e0c700 1 -- 172.21.15.111:0/2843998398 wait complete. 2023-12-19T09:22:31.765 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-19T09:22:31.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T09:22:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.922+0000 7f64fe6f0700 1 Processor -- start 2023-12-19T09:22:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.922+0000 7f64fe6f0700 1 -- start start 2023-12-19T09:22:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fe6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fe6f0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64f80674f0 0x7f64f8067950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fe6f0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f80afc30 con 0x7f64f80cc2f0 2023-12-19T09:22:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fe6f0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f80b1310 con 0x7f64f80674f0 2023-12-19T09:22:31.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fe6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8156fe0 con 0x7f64f80b24e0 2023-12-19T09:22:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64f77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64fcc8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64f80674f0 0x7f64f8067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64f77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.111:53542/0 (socket says 172.21.15.111:53542) 2023-12-19T09:22:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64f77fe700 1 -- 172.21.15.111:0/2081976316 learned_addr learned my addr 172.21.15.111:0/2081976316 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:31.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.926+0000 7f64f7fff700 1 --1- 172.21.15.111:0/2081976316 >> v1:172.21.15.143:6789/0 conn(0x7f64f80cc2f0 0x7f64f80b1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.111:52750/0 (socket says 172.21.15.111:52750) 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f77fe700 1 -- 172.21.15.111:0/2081976316 >> v2:172.21.15.143:3300/0 conn(0x7f64f80674f0 msgr2=0x7f64f8067950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f77fe700 1 --2- 172.21.15.111:0/2081976316 >> v2:172.21.15.143:3300/0 conn(0x7f64f80674f0 0x7f64f8067950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f77fe700 1 -- 172.21.15.111:0/2081976316 >> v1:172.21.15.143:6789/0 conn(0x7f64f80cc2f0 legacy=0x7f64f80b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f77fe700 1 -- 172.21.15.111:0/2081976316 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f8067060 con 0x7f64f80b24e0 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f77fe700 1 --2- 172.21.15.111:0/2081976316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0099b0 tx=0x7f64ec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c08dabb827e814a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f6ffd700 1 -- 172.21.15.111:0/2081976316 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec009ea0 con 0x7f64f80b24e0 2023-12-19T09:22:31.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f6ffd700 1 -- 172.21.15.111:0/2081976316 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f64ec00b860 con 0x7f64f80b24e0 2023-12-19T09:22:31.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64f6ffd700 1 -- 172.21.15.111:0/2081976316 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00bc20 con 0x7f64f80b24e0 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 -- 172.21.15.111:0/2081976316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 msgr2=0x7f64f80af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/2081976316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0099b0 tx=0x7f64ec009250 comp rx=0 tx=0).stop 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 -- 172.21.15.111:0/2081976316 shutdown_connections 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/2081976316 >> v2:172.21.15.143:3300/0 conn(0x7f64f80674f0 0x7f64f8067950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/2081976316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80b24e0 0x7f64f80af6f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 -- 172.21.15.111:0/2081976316 >> 172.21.15.111:0/2081976316 conn(0x7f64f80d0a70 msgr2=0x7f64f80cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 -- 172.21.15.111:0/2081976316 shutdown_connections 2023-12-19T09:22:31.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.930+0000 7f64fe6f0700 1 -- 172.21.15.111:0/2081976316 wait complete. 2023-12-19T09:22:31.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.934+0000 7f64fe6f0700 1 Processor -- start 2023-12-19T09:22:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.934+0000 7f64fe6f0700 1 -- start start 2023-12-19T09:22:31.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 0x7f64f816c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64f80b24e0 0x7f64f816aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80c6ff0 con 0x7f64f80cc2f0 2023-12-19T09:22:31.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80c7b70 con 0x7f64f80b24e0 2023-12-19T09:22:31.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f816cc80 con 0x7f64f80674f0 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fcc8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fcc8d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36972/0 (socket says 172.21.15.111:36972) 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fcc8d700 1 -- 172.21.15.111:0/1689623697 learned_addr learned my addr 172.21.15.111:0/1689623697 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 0x7f64f816c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f77fe700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64f80b24e0 0x7f64f816aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 -- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64f80b24e0 msgr2=0x7f64f816aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64f80b24e0 0x7f64f816aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 -- 172.21.15.111:0/1689623697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 msgr2=0x7f64f80c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 -- 172.21.15.111:0/1689623697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec0094f0 con 0x7f64f80674f0 2023-12-19T09:22:31.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fcc8d700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64f7fff700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 0x7f64f816c740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f64e800c7f0 tx=0x7f64e8007eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8005a70 con 0x7f64f80674f0 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f64e8005bf0 con 0x7f64f80674f0 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f816b010 con 0x7f64f80674f0 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e80070d0 con 0x7f64f80674f0 2023-12-19T09:22:31.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.938+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f80c7520 con 0x7f64f80674f0 2023-12-19T09:22:31.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.942+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f8071110 con 0x7f64f80674f0 2023-12-19T09:22:31.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.942+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 109) v1 ==== 79961+0+0 (secure 0 0 0) 0x7f64e8007480 con 0x7f64f80674f0 2023-12-19T09:22:31.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.946+0000 7f64dffff700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 0x7f64e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T09:22:31.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.946+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6394+0+0 (secure 0 0 0) 0x7f64e807bd40 con 0x7f64f80674f0 2023-12-19T09:22:31.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.946+0000 7f64f77fe700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 0x7f64e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T09:22:31.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.946+0000 7f64f77fe700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 0x7f64e4062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0099b0 tx=0x7f64ec009250 comp rx=0 tx=0).ready entity=mgr.9002 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T09:22:31.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:31.950+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f64e804c060 con 0x7f64f80674f0 2023-12-19T09:22:32.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.158+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64f80afc30 con 0x7f64f80674f0 2023-12-19T09:22:32.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.158+0000 7f64dffff700 1 -- 172.21.15.111:0/1689623697 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3544 (secure 0 0 0) 0x7f64e8017df0 con 0x7f64f80674f0 2023-12-19T09:22:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-19T09:22:32.163 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":97,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/3764811941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":3764811941},{"type":"v1","addr":"172.21.15.143:6835","nonce":3764811941}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10114,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/1711878296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1711878296},{"type":"v1","addr":"172.21.15.111:6833","nonce":1711878296}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10117,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1325073448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1325073448},{"type":"v1","addr":"172.21.15.111:6835","nonce":1325073448}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90},{"gid":10489,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1841110878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1841110878},{"type":"v1","addr":"172.21.15.143:6837","nonce":1841110878}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":97}],"filesystems":[]} 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 msgr2=0x7f64e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 0x7f64e4062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0099b0 tx=0x7f64ec009250 comp rx=0 tx=0).stop 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 msgr2=0x7f64f816c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 0x7f64f816c740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f64e800c7f0 tx=0x7f64e8007eb0 comp rx=0 tx=0).stop 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 shutdown_connections 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:6832/43868,v1:172.21.15.143:6833/43868] conn(0x7f64e4060550 0x7f64e4062a00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64f80b24e0 0x7f64f816aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f80cc2f0 0x7f64f80c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:32.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 --2- 172.21.15.111:0/1689623697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64f80674f0 0x7f64f816c740 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T09:22:32.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 >> 172.21.15.111:0/1689623697 conn(0x7f64f80d0a70 msgr2=0x7f64f80cdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T09:22:32.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 shutdown_connections 2023-12-19T09:22:32.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-12-19T09:22:32.162+0000 7f64fe6f0700 1 -- 172.21.15.111:0/1689623697 wait complete. 2023-12-19T09:22:32.169 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 97 2023-12-19T09:22:32.187 DEBUG:tasks.cephfs.filesystem:already dead... 2023-12-19T09:22:32.187 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-19T09:22:32.188 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-19T09:22:32.188 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:32.210 INFO:tasks.ceph.mds.a:Stopped 2023-12-19T09:22:32.210 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-19T09:22:32.210 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:32.280 INFO:tasks.ceph.mds.c:Stopped 2023-12-19T09:22:32.280 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-19T09:22:32.281 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:32.303 INFO:tasks.ceph.mds.b:Stopped 2023-12-19T09:22:32.303 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-19T09:22:32.303 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:32.367 INFO:tasks.ceph.mds.d:Stopped 2023-12-19T09:22:32.368 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-19T09:22:32.368 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-19T09:22:32.368 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:32.470 INFO:tasks.ceph.osd.0:Stopped 2023-12-19T09:22:32.470 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-19T09:22:32.470 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:34.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-19T09:22:38.572 INFO:tasks.ceph.osd.1:Stopped 2023-12-19T09:22:38.572 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-19T09:22:38.572 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:38.657 INFO:tasks.ceph.osd.2:Stopped 2023-12-19T09:22:38.658 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-19T09:22:38.658 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:40.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-19T09:22:40.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-19T09:22:44.759 INFO:tasks.ceph.osd.3:Stopped 2023-12-19T09:22:44.760 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-19T09:22:44.760 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:44.841 INFO:tasks.ceph.osd.4:Stopped 2023-12-19T09:22:44.842 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-19T09:22:44.842 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:45.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-19T09:22:45.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-19T09:22:50.944 INFO:tasks.ceph.osd.5:Stopped 2023-12-19T09:22:50.944 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-19T09:22:50.944 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:51.018 INFO:tasks.ceph.osd.6:Stopped 2023-12-19T09:22:51.019 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-19T09:22:51.019 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:51.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-12-19T09:22:51.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-19T09:22:57.121 INFO:tasks.ceph.osd.7:Stopped 2023-12-19T09:22:57.121 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-19T09:22:57.121 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-19T09:22:57.122 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:57.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-19T09:22:57.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-12-19T09:22:57.212 INFO:tasks.ceph.mgr.y:Stopped 2023-12-19T09:22:57.213 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-19T09:22:57.213 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:57.276 INFO:tasks.ceph.mgr.x:Stopped 2023-12-19T09:22:57.276 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-19T09:22:57.277 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-19T09:22:57.277 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:57.328 INFO:tasks.ceph.mon.a:Stopped 2023-12-19T09:22:57.329 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-19T09:22:57.329 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:57.371 INFO:tasks.ceph.mon.b:Stopped 2023-12-19T09:22:57.372 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-19T09:22:57.372 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T09:22:57.413 INFO:tasks.ceph.mon.c:Stopped 2023-12-19T09:22:57.414 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-19T09:22:57.414 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T09:22:57.460 INFO:teuthology.orchestra.run.smithi111.stdout:1702972183.1945295 mon.a (mon.0) 451 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-19T09:22:57.461 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-12-19T09:22:57.462 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T09:22:57.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T09:22:57.639 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T09:22:57.730 INFO:teuthology.orchestra.run.smithi111.stdout:1702972183.1945295 mon.a (mon.0) 451 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-19T09:22:57.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:22:57.731 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-19T09:22:58.043 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:22:58.043 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-19T09:22:58.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:22:58.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:22:58.256 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:22:58.256 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-19T09:22:58.537 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:22:58.537 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-19T09:22:58.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:22:58.741 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-19T09:22:59.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:22:59.011 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-19T09:22:59.265 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:22:59.265 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-19T09:22:59.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:22:59.446 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-19T09:22:59.664 INFO:tasks.ceph:Archiving mon data... 2023-12-19T09:22:59.665 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/data/mon.a.tgz 2023-12-19T09:22:59.666 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-12-19T09:22:59.685 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.vwQ7Soh6Fe 2023-12-19T09:22:59.686 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.vwQ7Soh6Fe 2023-12-19T09:23:00.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.vwQ7Soh6Fe 2023-12-19T09:23:00.233 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.vwQ7Soh6Fe is 2MB 2023-12-19T09:23:00.345 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.vwQ7Soh6Fe 2023-12-19T09:23:00.364 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/data/mon.b.tgz 2023-12-19T09:23:00.364 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2023-12-19T09:23:00.385 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.aKFGdMB4oZ 2023-12-19T09:23:00.385 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.aKFGdMB4oZ 2023-12-19T09:23:00.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.aKFGdMB4oZ 2023-12-19T09:23:00.931 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.aKFGdMB4oZ is 2MB 2023-12-19T09:23:01.045 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.aKFGdMB4oZ 2023-12-19T09:23:01.064 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/data/mon.c.tgz 2023-12-19T09:23:01.064 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2023-12-19T09:23:01.128 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.WGSagN8HYi 2023-12-19T09:23:01.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.WGSagN8HYi 2023-12-19T09:23:01.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.WGSagN8HYi 2023-12-19T09:23:01.679 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.WGSagN8HYi is 2MB 2023-12-19T09:23:01.789 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.WGSagN8HYi 2023-12-19T09:23:01.808 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-19T09:23:01.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-19T09:23:01.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-19T09:23:01.890 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-19T09:23:01.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-19T09:23:01.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-19T09:23:01.945 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-19T09:23:01.986 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-19T09:23:02.000 INFO:tasks.ceph:Archiving crash dumps... 2023-12-19T09:23:02.002 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi111/crash 2023-12-19T09:23:02.003 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-19T09:23:02.038 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi143/crash 2023-12-19T09:23:02.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-19T09:23:02.077 INFO:tasks.ceph:Shutting down logrotate 2023-12-19T09:23:02.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:23:02.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-19T09:23:02.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-19T09:23:02.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-19T09:23:02.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-19T09:23:02.270 INFO:tasks.ceph:Compressing logs... 2023-12-19T09:23:02.270 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T09:23:02.281 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T09:23:02.315 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47111.log 2023-12-19T09:23:02.316 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42457.log 2023-12-19T09:23:02.319 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2023-12-19T09:23:02.320 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56576.log 2023-12-19T09:23:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42457.log: /var/log/ceph/ceph-client.admin.42822.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz-5 2023-12-19T09:23:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.51609.log 2023-12-19T09:23:02.322 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42457.log.gz 2023-12-19T09:23:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56576.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56576.log.gz 2023-12-19T09:23:02.324 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55420.log 2023-12-19T09:23:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62248.log 2023-12-19T09:23:02.325 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51609.log.gz 2023-12-19T09:23:02.326 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39498.log 2023-12-19T09:23:02.326 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55420.log.gz 2023-12-19T09:23:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42403.log 2023-12-19T09:23:02.328 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62248.log.gz 2023-12-19T09:23:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39498.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39498.log.gz 2023-12-19T09:23:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2023-12-19T09:23:02.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55528.log 2023-12-19T09:23:02.332 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50738.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42403.log.gz 2023-12-19T09:23:02.332 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2023-12-19T09:23:02.333 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2023-12-19T09:23:02.334 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58878.log 2023-12-19T09:23:02.335 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55528.log.gz 2023-12-19T09:23:02.336 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59225.log 2023-12-19T09:23:02.336 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2023-12-19T09:23:02.337 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58878.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.58878.log.gz-5 2023-12-19T09:23:02.337 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.49868.log 2023-12-19T09:23:02.338 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59225.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59225.log.gz 2023-12-19T09:23:02.339 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.52157.log 2023-12-19T09:23:02.340 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49868.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2023-12-19T09:23:02.340 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2023-12-19T09:23:02.341 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52157.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.52157.log.gz 2023-12-19T09:23:02.341 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2023-12-19T09:23:02.343 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2023-12-19T09:23:02.343 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2023-12-19T09:23:02.344 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62945.log 2023-12-19T09:23:02.344 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz 2023-12-19T09:23:02.346 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50108.log 2023-12-19T09:23:02.346 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62945.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz 2023-12-19T09:23:02.347 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62945.log.gz 2023-12-19T09:23:02.347 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43628.log 2023-12-19T09:23:02.348 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47055.log 2023-12-19T09:23:02.349 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50108.log.gz 2023-12-19T09:23:02.349 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43628.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43628.log.gz 2023-12-19T09:23:02.350 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2023-12-19T09:23:02.351 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53649.log 2023-12-19T09:23:02.352 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47055.log.gz 2023-12-19T09:23:02.352 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53103.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2023-12-19T09:23:02.354 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2023-12-19T09:23:02.355 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46261.log 2023-12-19T09:23:02.355 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53649.log.gz 2023-12-19T09:23:02.356 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50446.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2023-12-19T09:23:02.356 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60382.log 2023-12-19T09:23:02.356 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-12-19T09:23:02.357 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34099.log 2023-12-19T09:23:02.358 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55190.log 2023-12-19T09:23:02.358 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46261.log.gz 2023-12-19T09:23:02.359 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-19T09:23:02.359 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60382.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60382.log.gz 2023-12-19T09:23:02.359 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55818.log 2023-12-19T09:23:02.360 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2023-12-19T09:23:02.361 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55190.log.gz 2023-12-19T09:23:02.362 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.34099.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34099.log.gz 2023-12-19T09:23:02.362 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-19T09:23:02.363 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55818.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55818.log.gz 2023-12-19T09:23:02.363 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62511.log 2023-12-19T09:23:02.363 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46462.log 2023-12-19T09:23:02.363 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2023-12-19T09:23:02.364 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34045.log 2023-12-19T09:23:02.364 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64137.log 2023-12-19T09:23:02.364 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62511.log.gz 2023-12-19T09:23:02.365 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-19T09:23:02.365 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46462.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46462.log.gz 2023-12-19T09:23:02.365 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52888.log 2023-12-19T09:23:02.366 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34045.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 87.2% 2023-12-19T09:23:02.366 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34045.log.gz 2023-12-19T09:23:02.367 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2023-12-19T09:23:02.367 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52888.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64137.log.gz 2023-12-19T09:23:02.367 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52888.log.gz 2023-12-19T09:23:02.368 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58603.log 2023-12-19T09:23:02.368 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-19T09:23:02.369 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59987.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2023-12-19T09:23:02.369 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54869.log 2023-12-19T09:23:02.370 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33989.log 2023-12-19T09:23:02.370 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58603.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58603.log.gz 2023-12-19T09:23:02.371 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58529.log 2023-12-19T09:23:02.371 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54869.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54869.log.gz 2023-12-19T09:23:02.371 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48101.log 2023-12-19T09:23:02.372 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58529.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58529.log.gz 2023-12-19T09:23:02.373 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47602.log 2023-12-19T09:23:02.373 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-19T09:23:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48101.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48101.log.gz 2023-12-19T09:23:02.374 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50336.log 2023-12-19T09:23:02.375 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.33989.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33989.log.gz 2023-12-19T09:23:02.375 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47602.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47602.log.gz 2023-12-19T09:23:02.375 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2023-12-19T09:23:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50336.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50336.log.gz 2023-12-19T09:23:02.376 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44234.log 2023-12-19T09:23:02.377 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40532.log: gzip -5 --verbose 87.7%-- -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz/var/log/ceph/ceph-mon.a.log 2023-12-19T09:23:02.377 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.379 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59792.log 2023-12-19T09:23:02.379 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44234.log.gz 2023-12-19T09:23:02.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64591.log 2023-12-19T09:23:02.380 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59792.log: 94.5% -- replaced with /var/log/ceph/ceph-client.0.47111.log.gz 2023-12-19T09:23:02.381 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34071.log 2023-12-19T09:23:02.381 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59792.log.gz 2023-12-19T09:23:02.381 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64915.log 2023-12-19T09:23:02.382 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-12-19T09:23:02.383 INFO:teuthology.orchestra.run.smithi143.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-19T09:23:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49604.log 2023-12-19T09:23:02.383 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64591.log.gz 2023-12-19T09:23:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64915.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64915.log.gz 2023-12-19T09:23:02.384 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.39323.log 2023-12-19T09:23:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64510.log 2023-12-19T09:23:02.385 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2023-12-19T09:23:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58502.log 2023-12-19T09:23:02.387 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49604.log: /var/log/ceph/ceph-client.admin.64510.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64510.log.gz 2023-12-19T09:23:02.387 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49604.log.gz 2023-12-19T09:23:02.387 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2023-12-19T09:23:02.388 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.1.39323.log: 90.6% -- replaced with /var/log/ceph/ceph-client.1.39323.log.gz 2023-12-19T09:23:02.389 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51763.log 2023-12-19T09:23:02.389 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58502.log.gz 2023-12-19T09:23:02.390 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33934.log 2023-12-19T09:23:02.390 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52829.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz 2023-12-19T09:23:02.390 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64456.log 2023-12-19T09:23:02.391 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.34018.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2023-12-19T09:23:02.392 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64483.log 2023-12-19T09:23:02.392 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51763.log.gz 2023-12-19T09:23:02.393 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33908.log 2023-12-19T09:23:02.394 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64456.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64456.log.gz 2023-12-19T09:23:02.394 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55609.log 2023-12-19T09:23:02.395 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50711.log 2023-12-19T09:23:02.395 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64483.log.gz 2023-12-19T09:23:02.396 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55609.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55609.log.gz 2023-12-19T09:23:02.396 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2023-12-19T09:23:02.396 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.34071.log: /var/log/ceph/ceph-client.admin.33934.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34071.log.gz 2023-12-19T09:23:02.397 INFO:teuthology.orchestra.run.smithi143.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33934.log.gz 2023-12-19T09:23:02.398 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60871.log 2023-12-19T09:23:02.398 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50711.log.gz 2023-12-19T09:23:02.399 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-19T09:23:02.399 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47702.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2023-12-19T09:23:02.399 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58323.log 2023-12-19T09:23:02.400 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51736.log 2023-12-19T09:23:02.400 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60871.log.gz 2023-12-19T09:23:02.401 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61881.log 2023-12-19T09:23:02.401 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58323.log.gz 2023-12-19T09:23:02.403 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.33908.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33908.log.gz 2023-12-19T09:23:02.403 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.39432.log 2023-12-19T09:23:02.404 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61099.log 2023-12-19T09:23:02.404 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2023-12-19T09:23:02.404 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61881.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61881.log.gz 2023-12-19T09:23:02.404 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2023-12-19T09:23:02.405 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61099.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61099.log.gz 2023-12-19T09:23:02.406 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59005.log 2023-12-19T09:23:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.3.39432.log: 89.9% -- replaced with /var/log/ceph/ceph-client.3.39432.log.gz 2023-12-19T09:23:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64537.log 2023-12-19T09:23:02.408 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44263.log: /var/log/ceph/ceph-client.admin.59005.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2023-12-19T09:23:02.409 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2023-12-19T09:23:02.409 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59005.log.gz 2023-12-19T09:23:02.409 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-19T09:23:02.410 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62890.log 2023-12-19T09:23:02.410 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33962.log 2023-12-19T09:23:02.411 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41087.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64537.log.gz 2023-12-19T09:23:02.411 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2023-12-19T09:23:02.412 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46759.log 2023-12-19T09:23:02.413 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62890.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62890.log.gz 2023-12-19T09:23:02.414 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53730.log 2023-12-19T09:23:02.415 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55791.log 2023-12-19T09:23:02.416 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46759.log: /var/log/ceph/ceph-client.admin.53730.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46759.log.gz 2023-12-19T09:23:02.417 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.33962.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33962.log.gz 2023-12-19T09:23:02.417 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53730.log.gz 2023-12-19T09:23:02.417 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46180.log 2023-12-19T09:23:02.418 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-19T09:23:02.421 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49364.log 2023-12-19T09:23:02.421 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55791.log.gz 2023-12-19T09:23:02.421 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44292.log 2023-12-19T09:23:02.421 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46180.log.gz 2023-12-19T09:23:02.422 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47944.log 2023-12-19T09:23:02.422 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49364.log.gz 2023-12-19T09:23:02.423 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58028.log 2023-12-19T09:23:02.424 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44292.log.gz 2023-12-19T09:23:02.425 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2023-12-19T09:23:02.425 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47944.log.gz 2023-12-19T09:23:02.426 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58028.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58028.log.gz 2023-12-19T09:23:02.426 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39440.log 2023-12-19T09:23:02.427 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56638.log 2023-12-19T09:23:02.428 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45921.log: /var/log/ceph/ceph-client.admin.39440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46370.log 2023-12-19T09:23:02.428 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz 2023-12-19T09:23:02.428 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2023-12-19T09:23:02.429 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61553.log 2023-12-19T09:23:02.429 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56638.log.gz 2023-12-19T09:23:02.430 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46370.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46370.log.gz 2023-12-19T09:23:02.430 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2023-12-19T09:23:02.431 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48704.log 2023-12-19T09:23:02.432 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61553.log.gz 2023-12-19T09:23:02.433 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41801.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2023-12-19T09:23:02.433 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60739.log 2023-12-19T09:23:02.435 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2023-12-19T09:23:02.436 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48704.log: /var/log/ceph/ceph-client.admin.60739.log: gzip -5 --verbose 89.3%-- -- replaced with /var/log/ceph/ceph-client.admin.48704.log.gz/var/log/ceph/ceph-client.admin.39552.log 2023-12-19T09:23:02.436 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.436 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60739.log.gz 2023-12-19T09:23:02.437 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51993.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 2023-12-19T09:23:02.437 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58472.log 2023-12-19T09:23:02.438 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.7.log: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-19T09:23:02.439 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60898.log 2023-12-19T09:23:02.439 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2023-12-19T09:23:02.440 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54554.log 2023-12-19T09:23:02.440 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58472.log.gz 2023-12-19T09:23:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56480.log 2023-12-19T09:23:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54554.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60898.log.gz 2023-12-19T09:23:02.442 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54554.log.gz 2023-12-19T09:23:02.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2023-12-19T09:23:02.445 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46697.log 2023-12-19T09:23:02.445 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56480.log.gz 2023-12-19T09:23:02.446 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54733.log 2023-12-19T09:23:02.446 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2023-12-19T09:23:02.447 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46697.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46697.log.gz 2023-12-19T09:23:02.447 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61499.log 2023-12-19T09:23:02.449 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63404.log 2023-12-19T09:23:02.449 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54733.log.gz 2023-12-19T09:23:02.450 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2023-12-19T09:23:02.450 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61499.log.gz 2023-12-19T09:23:02.451 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63404.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63404.log.gz 2023-12-19T09:23:02.451 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-12-19T09:23:02.453 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52738.log 2023-12-19T09:23:02.453 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gz 2023-12-19T09:23:02.455 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57739.log 2023-12-19T09:23:02.456 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52738.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52738.log.gz 2023-12-19T09:23:02.456 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2023-12-19T09:23:02.457 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42930.log 2023-12-19T09:23:02.458 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57739.log.gz 2023-12-19T09:23:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40754.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2023-12-19T09:23:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59144.log 2023-12-19T09:23:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42930.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42930.log.gz 2023-12-19T09:23:02.461 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54465.log 2023-12-19T09:23:02.462 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59494.log 2023-12-19T09:23:02.463 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8%/var/log/ceph/ceph-client.admin.54465.log: -- replaced with /var/log/ceph/ceph-client.admin.59144.log.gz 2023-12-19T09:23:02.463 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54465.log.gz 2023-12-19T09:23:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50818.log 2023-12-19T09:23:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49513.log 2023-12-19T09:23:02.465 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59494.log.gz 2023-12-19T09:23:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50818.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50818.log.gz 2023-12-19T09:23:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53327.log 2023-12-19T09:23:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50500.log 2023-12-19T09:23:02.468 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49513.log.gz 2023-12-19T09:23:02.469 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53327.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53327.log.gz 2023-12-19T09:23:02.469 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48449.log 2023-12-19T09:23:02.470 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56698.log 2023-12-19T09:23:02.471 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50500.log.gz 2023-12-19T09:23:02.472 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48449.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48449.log.gz 2023-12-19T09:23:02.472 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59362.log 2023-12-19T09:23:02.473 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2023-12-19T09:23:02.473 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56698.log.gz 2023-12-19T09:23:02.474 INFO:teuthology.orchestra.run.smithi143.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-12-19T09:23:02.474 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59362.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59362.log.gz 2023-12-19T09:23:02.475 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58738.log 2023-12-19T09:23:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47303.log 2023-12-19T09:23:02.476 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2023-12-19T09:23:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58738.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58738.log.gz 2023-12-19T09:23:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2023-12-19T09:23:02.478 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2023-12-19T09:23:02.478 INFO:teuthology.orchestra.run.smithi111.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47303.log.gz 2023-12-19T09:23:02.479 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42511.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2023-12-19T09:23:02.479 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55063.log 2023-12-19T09:23:02.481 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54896.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2023-12-19T09:23:02.481 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52103.log 2023-12-19T09:23:02.482 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62972.log 2023-12-19T09:23:02.483 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55063.log: /var/log/ceph/ceph-client.admin.52103.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55063.log.gz 2023-12-19T09:23:02.484 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52103.log.gzgzip 2023-12-19T09:23:02.484 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.60439.log 2023-12-19T09:23:02.485 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63107.log 2023-12-19T09:23:02.485 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62972.log.gz 2023-12-19T09:23:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60439.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60439.log.gz 2023-12-19T09:23:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57685.log 2023-12-19T09:23:02.487 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63107.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63107.log.gz 2023-12-19T09:23:02.487 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61182.log 2023-12-19T09:23:02.488 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57685.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57685.log.gz 2023-12-19T09:23:02.488 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2023-12-19T09:23:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61182.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61182.log.gz 2023-12-19T09:23:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52022.log 2023-12-19T09:23:02.491 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60078.log 2023-12-19T09:23:02.491 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2023-12-19T09:23:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52022.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52022.log.gz 2023-12-19T09:23:02.493 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2023-12-19T09:23:02.494 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57291.log 2023-12-19T09:23:02.494 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60078.log.gz 2023-12-19T09:23:02.495 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2023-12-19T09:23:02.495 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2023-12-19T09:23:02.496 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55329.log 2023-12-19T09:23:02.497 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57291.log.gz 2023-12-19T09:23:02.498 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54788.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2023-12-19T09:23:02.498 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62724.log 2023-12-19T09:23:02.499 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55329.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55329.log.gz 2023-12-19T09:23:02.499 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50282.log 2023-12-19T09:23:02.500 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62724.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62724.log.gz 2023-12-19T09:23:02.500 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51201.log 2023-12-19T09:23:02.501 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50282.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50282.log.gz 2023-12-19T09:23:02.501 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2023-12-19T09:23:02.502 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51201.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51201.log.gz 2023-12-19T09:23:02.502 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49668.log 2023-12-19T09:23:02.503 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43246.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2023-12-19T09:23:02.504 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55004.log 2023-12-19T09:23:02.505 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49668.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49668.log.gz 2023-12-19T09:23:02.505 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44118.log 2023-12-19T09:23:02.506 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55004.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55004.log.gz 2023-12-19T09:23:02.506 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-12-19T09:23:02.507 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44118.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44118.log.gz 2023-12-19T09:23:02.507 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43682.log 2023-12-19T09:23:02.508 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2023-12-19T09:23:02.509 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43682.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43682.log.gz 2023-12-19T09:23:02.510 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62805.log 2023-12-19T09:23:02.511 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41969.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2023-12-19T09:23:02.512 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48530.log 2023-12-19T09:23:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63637.log 2023-12-19T09:23:02.514 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62805.log.gz 2023-12-19T09:23:02.514 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48530.log.gz 2023-12-19T09:23:02.515 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44562.log 2023-12-19T09:23:02.516 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.63637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55217.log 2023-12-19T09:23:02.518 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52857.log 2023-12-19T09:23:02.518 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44562.log.gz 2023-12-19T09:23:02.519 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.63637.log.gz 2023-12-19T09:23:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57577.log 2023-12-19T09:23:02.519 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55217.log.gz 2023-12-19T09:23:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52857.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52857.log.gz 2023-12-19T09:23:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44723.log 2023-12-19T09:23:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61690.log 2023-12-19T09:23:02.523 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57577.log.gz 2023-12-19T09:23:02.524 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44723.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44723.log.gz 2023-12-19T09:23:02.525 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2023-12-19T09:23:02.526 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61690.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61690.log.gz 2023-12-19T09:23:02.526 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41384.log 2023-12-19T09:23:02.528 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44007.log 2023-12-19T09:23:02.528 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2023-12-19T09:23:02.529 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41384.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gz 2023-12-19T09:23:02.529 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43655.log 2023-12-19T09:23:02.531 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54950.log 2023-12-19T09:23:02.531 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44007.log.gz 2023-12-19T09:23:02.533 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64274.log 2023-12-19T09:23:02.533 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43655.log.gz 2023-12-19T09:23:02.533 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54950.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54950.log.gz 2023-12-19T09:23:02.534 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63489.log 2023-12-19T09:23:02.535 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64220.log 2023-12-19T09:23:02.535 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64274.log.gz 2023-12-19T09:23:02.537 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63026.log 2023-12-19T09:23:02.537 INFO:teuthology.orchestra.run.smithi111.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.63489.log.gz 2023-12-19T09:23:02.538 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64220.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64220.log.gz 2023-12-19T09:23:02.538 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46732.log 2023-12-19T09:23:02.539 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46637.log 2023-12-19T09:23:02.540 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63026.log.gz 2023-12-19T09:23:02.541 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46732.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46732.log.gz 2023-12-19T09:23:02.541 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53295.log 2023-12-19T09:23:02.542 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46637.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46637.log.gz 2023-12-19T09:23:02.543 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2023-12-19T09:23:02.543 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53295.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53295.log.gz 2023-12-19T09:23:02.544 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63134.log 2023-12-19T09:23:02.545 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2023-12-19T09:23:02.546 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2023-12-19T09:23:02.547 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63134.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63134.log.gz 2023-12-19T09:23:02.547 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61263.log 2023-12-19T09:23:02.549 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42131.log 2023-12-19T09:23:02.549 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2023-12-19T09:23:02.550 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61263.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61263.log.gz 2023-12-19T09:23:02.550 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56918.log 2023-12-19T09:23:02.551 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39413.log 2023-12-19T09:23:02.552 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42131.log.gz 2023-12-19T09:23:02.552 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56918.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56918.log.gz 2023-12-19T09:23:02.553 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58263.log 2023-12-19T09:23:02.554 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39413.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39413.log.gz 2023-12-19T09:23:02.554 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2023-12-19T09:23:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48382.log 2023-12-19T09:23:02.556 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58263.log.gz 2023-12-19T09:23:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43300.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2023-12-19T09:23:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54616.log 2023-12-19T09:23:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48382.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48382.log.gz 2023-12-19T09:23:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49632.log 2023-12-19T09:23:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54616.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54616.log.gz 2023-12-19T09:23:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2023-12-19T09:23:02.561 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49632.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49632.log.gz 2023-12-19T09:23:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45309.log 2023-12-19T09:23:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33931.log 2023-12-19T09:23:02.563 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2023-12-19T09:23:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45309.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45309.log.gz 2023-12-19T09:23:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45525.log 2023-12-19T09:23:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.33931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63843.log 2023-12-19T09:23:02.566 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33931.log.gz 2023-12-19T09:23:02.567 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45525.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45525.log.gz 2023-12-19T09:23:02.568 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2023-12-19T09:23:02.569 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2023-12-19T09:23:02.569 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63843.log.gz 2023-12-19T09:23:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47164.log 2023-12-19T09:23:02.571 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz 2023-12-19T09:23:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.41730.log 2023-12-19T09:23:02.572 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2023-12-19T09:23:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47164.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47164.log.gz 2023-12-19T09:23:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45176.log 2023-12-19T09:23:02.575 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.2.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46923.log 2023-12-19T09:23:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45176.log: 90.9% -- replaced with /var/log/ceph/ceph-client.2.41730.log.gz 2023-12-19T09:23:02.576 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45176.log.gz 2023-12-19T09:23:02.576 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48290.log 2023-12-19T09:23:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62592.log 2023-12-19T09:23:02.582 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46923.log.gz 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48290.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48290.log.gz 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54074.log 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62592.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62592.log.gz 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40756.log 2023-12-19T09:23:02.584 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54074.log.gz 2023-12-19T09:23:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41141.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2023-12-19T09:23:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50247.log 2023-12-19T09:23:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40756.log.gz 2023-12-19T09:23:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60321.log 2023-12-19T09:23:02.587 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50847.log 2023-12-19T09:23:02.587 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50247.log.gz 2023-12-19T09:23:02.588 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60321.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60321.log.gz 2023-12-19T09:23:02.588 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2023-12-19T09:23:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59306.log 2023-12-19T09:23:02.590 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50847.log.gz 2023-12-19T09:23:02.592 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54435.log 2023-12-19T09:23:02.592 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2023-12-19T09:23:02.593 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43354.log 2023-12-19T09:23:02.593 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59306.log.gz 2023-12-19T09:23:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54435.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54435.log.gz 2023-12-19T09:23:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59625.log 2023-12-19T09:23:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40113.log 2023-12-19T09:23:02.596 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43354.log.gz 2023-12-19T09:23:02.597 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59625.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59625.log.gz 2023-12-19T09:23:02.598 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2023-12-19T09:23:02.599 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40113.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40113.log.gz 2023-12-19T09:23:02.599 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41033.log 2023-12-19T09:23:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41638.log 2023-12-19T09:23:02.601 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz 2023-12-19T09:23:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41033.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41033.log.gz 2023-12-19T09:23:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55300.log 2023-12-19T09:23:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41638.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41638.log.gz 2023-12-19T09:23:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2023-12-19T09:23:02.605 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55300.log.gz 2023-12-19T09:23:02.606 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58684.log 2023-12-19T09:23:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42849.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2023-12-19T09:23:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50419.log 2023-12-19T09:23:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58684.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-19T09:23:02.609 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58684.log.gz 2023-12-19T09:23:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50419.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50419.log.gz 2023-12-19T09:23:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61317.log 2023-12-19T09:23:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2023-12-19T09:23:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61317.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61317.log.gz 2023-12-19T09:23:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2023-12-19T09:23:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2023-12-19T09:23:02.614 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2023-12-19T09:23:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53049.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2023-12-19T09:23:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53676.log 2023-12-19T09:23:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52130.log 2023-12-19T09:23:02.617 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2023-12-19T09:23:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53676.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53676.log.gz 2023-12-19T09:23:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42606.log 2023-12-19T09:23:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61914.log 2023-12-19T09:23:02.619 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2023-12-19T09:23:02.620 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2023-12-19T09:23:02.620 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42606.log.gz 2023-12-19T09:23:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45578.log 2023-12-19T09:23:02.622 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61914.log.gz 2023-12-19T09:23:02.623 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40527.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2023-12-19T09:23:02.623 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2023-12-19T09:23:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57921.log 2023-12-19T09:23:02.625 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45578.log.gz 2023-12-19T09:23:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41774.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2023-12-19T09:23:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.54047.log 2023-12-19T09:23:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2023-12-19T09:23:02.628 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57921.log.gz 2023-12-19T09:23:02.629 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 88.2% 2023-12-19T09:23:02.629 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54047.log.gz 2023-12-19T09:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51577.log: gzip -5 --verbose 88.2%-- /var/log/ceph/ceph-client.admin.64348.log -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2023-12-19T09:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40086.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2023-12-19T09:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43273.log 2023-12-19T09:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33802.log 2023-12-19T09:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64348.log.gz 2023-12-19T09:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43273.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43273.log.gz 2023-12-19T09:23:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49455.log 2023-12-19T09:23:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.33802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2023-12-19T09:23:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49455.log: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.33802.log.gz 2023-12-19T09:23:02.637 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49455.log.gz 2023-12-19T09:23:02.638 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2023-12-19T09:23:02.639 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43520.log 2023-12-19T09:23:02.639 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2023-12-19T09:23:02.641 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62376.log 2023-12-19T09:23:02.641 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2023-12-19T09:23:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43520.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43520.log.gz 2023-12-19T09:23:02.642 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2023-12-19T09:23:02.644 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60685.log 2023-12-19T09:23:02.644 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62376.log.gz 2023-12-19T09:23:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41195.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2023-12-19T09:23:02.645 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50765.log 2023-12-19T09:23:02.646 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55393.log 2023-12-19T09:23:02.647 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60685.log.gz 2023-12-19T09:23:02.648 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50765.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50765.log.gz 2023-12-19T09:23:02.648 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41168.log 2023-12-19T09:23:02.649 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62219.log 2023-12-19T09:23:02.649 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55393.log.gz 2023-12-19T09:23:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41168.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41168.log.gz 2023-12-19T09:23:02.650 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60631.log 2023-12-19T09:23:02.652 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62219.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62219.log.gz 2023-12-19T09:23:02.653 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64699.log 2023-12-19T09:23:02.654 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60631.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60631.log.gz 2023-12-19T09:23:02.654 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2023-12-19T09:23:02.656 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64699.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64699.log.gz 2023-12-19T09:23:02.656 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57134.log 2023-12-19T09:23:02.657 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49949.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz-5 2023-12-19T09:23:02.657 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.43111.log 2023-12-19T09:23:02.659 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57134.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57134.log.gz 2023-12-19T09:23:02.659 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63377.log 2023-12-19T09:23:02.660 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43111.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2023-12-19T09:23:02.660 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45861.log 2023-12-19T09:23:02.661 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50309.log 2023-12-19T09:23:02.662 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63377.log.gz 2023-12-19T09:23:02.663 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55636.log 2023-12-19T09:23:02.663 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45861.log.gz 2023-12-19T09:23:02.664 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50309.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50309.log.gz 2023-12-19T09:23:02.665 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51297.log 2023-12-19T09:23:02.666 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51334.log 2023-12-19T09:23:02.666 INFO:teuthology.orchestra.run.smithi111.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.55636.log.gz 2023-12-19T09:23:02.667 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51297.log.gz 2023-12-19T09:23:02.668 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59333.log 2023-12-19T09:23:02.669 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51334.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51334.log.gz 2023-12-19T09:23:02.670 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48733.log 2023-12-19T09:23:02.671 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59333.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59333.log.gz 2023-12-19T09:23:02.671 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61236.log 2023-12-19T09:23:02.672 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48733.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48733.log.gz 2023-12-19T09:23:02.672 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44205.log 2023-12-19T09:23:02.673 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51260.log 2023-12-19T09:23:02.674 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61236.log.gz 2023-12-19T09:23:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44205.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44205.log.gz 2023-12-19T09:23:02.675 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2023-12-19T09:23:02.676 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44400.log 2023-12-19T09:23:02.677 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51260.log.gz 2023-12-19T09:23:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40538.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2023-12-19T09:23:02.678 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62624.log 2023-12-19T09:23:02.679 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58352.log 2023-12-19T09:23:02.679 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44400.log.gz 2023-12-19T09:23:02.681 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53510.log 2023-12-19T09:23:02.681 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62624.log.gz 2023-12-19T09:23:02.682 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58352.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58352.log.gz 2023-12-19T09:23:02.682 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49981.log 2023-12-19T09:23:02.683 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60771.log 2023-12-19T09:23:02.684 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53510.log.gz 2023-12-19T09:23:02.684 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49981.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49981.log.gz 2023-12-19T09:23:02.685 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58233.log 2023-12-19T09:23:02.687 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60771.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60771.log.gz 2023-12-19T09:23:02.687 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44831.log 2023-12-19T09:23:02.688 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60143.log 2023-12-19T09:23:02.688 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58233.log.gz 2023-12-19T09:23:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44831.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44831.log.gz 2023-12-19T09:23:02.689 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2023-12-19T09:23:02.690 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59086.log 2023-12-19T09:23:02.691 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60143.log.gz 2023-12-19T09:23:02.694 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45030.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2023-12-19T09:23:02.694 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64861.log 2023-12-19T09:23:02.695 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40194.log 2023-12-19T09:23:02.695 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59086.log.gz 2023-12-19T09:23:02.695 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64861.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64861.log.gz-5 2023-12-19T09:23:02.695 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.53566.log 2023-12-19T09:23:02.697 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59740.log 2023-12-19T09:23:02.697 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40194.log.gz 2023-12-19T09:23:02.698 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56972.log 2023-12-19T09:23:02.698 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2023-12-19T09:23:02.699 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52798.log 2023-12-19T09:23:02.699 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59740.log.gz 2023-12-19T09:23:02.701 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56972.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56972.log.gz 2023-12-19T09:23:02.701 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2023-12-19T09:23:02.703 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59900.log 2023-12-19T09:23:02.703 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52798.log.gz 2023-12-19T09:23:02.704 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2023-12-19T09:23:02.705 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2023-12-19T09:23:02.706 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2023-12-19T09:23:02.706 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59900.log.gz 2023-12-19T09:23:02.707 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64375.log 2023-12-19T09:23:02.707 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2023-12-19T09:23:02.709 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45737.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2023-12-19T09:23:02.709 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44319.log 2023-12-19T09:23:02.710 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39238.log 2023-12-19T09:23:02.710 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64375.log.gz 2023-12-19T09:23:02.711 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44319.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44319.log.gz 2023-12-19T09:23:02.711 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63932.log 2023-12-19T09:23:02.712 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46041.log 2023-12-19T09:23:02.713 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39238.log.gz 2023-12-19T09:23:02.714 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63932.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63932.log.gz 2023-12-19T09:23:02.714 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45684.log 2023-12-19T09:23:02.716 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2023-12-19T09:23:02.716 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46041.log.gz 2023-12-19T09:23:02.717 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45684.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45684.log.gz 2023-12-19T09:23:02.717 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55743.log 2023-12-19T09:23:02.718 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41915.log 2023-12-19T09:23:02.718 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2023-12-19T09:23:02.719 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.55743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52049.log 2023-12-19T09:23:02.721 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2023-12-19T09:23:02.721 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41915.log.gz 2023-12-19T09:23:02.722 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46556.log 2023-12-19T09:23:02.722 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52049.log.gz 2023-12-19T09:23:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43920.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2023-12-19T09:23:02.724 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58657.log 2023-12-19T09:23:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46556.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46556.log.gz 2023-12-19T09:23:02.726 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45981.log 2023-12-19T09:23:02.727 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58657.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58657.log.gz 2023-12-19T09:23:02.728 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57107.log 2023-12-19T09:23:02.728 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-19T09:23:02.729 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2023-12-19T09:23:02.729 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45981.log.gz 2023-12-19T09:23:02.730 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55364.log 2023-12-19T09:23:02.730 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57107.log.gz 2023-12-19T09:23:02.731 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48503.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2023-12-19T09:23:02.732 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43547.log 2023-12-19T09:23:02.733 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55364.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55364.log.gz 2023-12-19T09:23:02.734 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51442.log 2023-12-19T09:23:02.736 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43547.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-19T09:23:02.736 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43547.log.gz 2023-12-19T09:23:02.737 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44508.log 2023-12-19T09:23:02.737 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51442.log.gz 2023-12-19T09:23:02.738 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63690.log 2023-12-19T09:23:02.740 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41006.log 2023-12-19T09:23:02.741 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44508.log.gz 2023-12-19T09:23:02.742 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2023-12-19T09:23:02.742 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63690.log.gz 2023-12-19T09:23:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41006.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41006.log.gz 2023-12-19T09:23:02.743 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2023-12-19T09:23:02.744 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40032.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-19T09:23:02.745 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2023-12-19T09:23:02.745 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39581.log 2023-12-19T09:23:02.747 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44777.log 2023-12-19T09:23:02.747 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2023-12-19T09:23:02.748 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2023-12-19T09:23:02.748 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39581.log.gz 2023-12-19T09:23:02.750 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61606.log 2023-12-19T09:23:02.750 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44777.log.gz 2023-12-19T09:23:02.751 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2023-12-19T09:23:02.751 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2023-12-19T09:23:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61606.log: gzip -5 87.5%--verbose -- replaced with /var/log/ceph/ceph-client.admin.61606.log.gz-- 2023-12-19T09:23:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61395.log 2023-12-19T09:23:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50365.log 2023-12-19T09:23:02.754 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2023-12-19T09:23:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55582.log 2023-12-19T09:23:02.755 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61395.log.gz 2023-12-19T09:23:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44147.log 2023-12-19T09:23:02.757 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50365.log.gz 2023-12-19T09:23:02.758 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2023-12-19T09:23:02.759 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55582.log.gz 2023-12-19T09:23:02.760 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44147.log: gzip 88.3%-5 -- replaced with /var/log/ceph/ceph-client.admin.44147.log.gz--verbose 2023-12-19T09:23:02.760 INFO:teuthology.orchestra.run.smithi111.stderr:-- /var/log/ceph/ceph-client.admin.52918.log 2023-12-19T09:23:02.761 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2023-12-19T09:23:02.761 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2023-12-19T09:23:02.763 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61209.log 2023-12-19T09:23:02.763 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52918.log.gz 2023-12-19T09:23:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58140.log 2023-12-19T09:23:02.764 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2023-12-19T09:23:02.766 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2023-12-19T09:23:02.766 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61209.log.gz 2023-12-19T09:23:02.767 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61979.log 2023-12-19T09:23:02.767 INFO:teuthology.orchestra.run.smithi111.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2023-12-19T09:23:02.768 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- 87.7%/var/log/ceph/ceph-client.admin.39716.log -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2023-12-19T09:23:02.769 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64082.log 2023-12-19T09:23:02.770 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61979.log.gz 2023-12-19T09:23:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52560.log 2023-12-19T09:23:02.772 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39716.log.gz 2023-12-19T09:23:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2023-12-19T09:23:02.773 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64082.log.gz 2023-12-19T09:23:02.775 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2023-12-19T09:23:02.775 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52560.log.gz 2023-12-19T09:23:02.776 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42579.log 2023-12-19T09:23:02.777 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2023-12-19T09:23:02.778 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2023-12-19T09:23:02.778 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2023-12-19T09:23:02.779 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49271.log 2023-12-19T09:23:02.779 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42579.log.gz 2023-12-19T09:23:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46316.log 2023-12-19T09:23:02.781 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2023-12-19T09:23:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53241.log 2023-12-19T09:23:02.782 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49271.log.gz 2023-12-19T09:23:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2023-12-19T09:23:02.784 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46316.log.gz 2023-12-19T09:23:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2023-12-19T09:23:02.785 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53241.log.gz 2023-12-19T09:23:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48841.log 2023-12-19T09:23:02.787 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2023-12-19T09:23:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51141.log 2023-12-19T09:23:02.788 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2023-12-19T09:23:02.790 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2023-12-19T09:23:02.790 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48841.log.gz 2023-12-19T09:23:02.792 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46429.log 2023-12-19T09:23:02.792 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51141.log.gz 2023-12-19T09:23:02.793 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56817.log 2023-12-19T09:23:02.793 INFO:teuthology.orchestra.run.smithi111.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2023-12-19T09:23:02.794 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2023-12-19T09:23:02.794 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46429.log.gz 2023-12-19T09:23:02.796 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63457.log 2023-12-19T09:23:02.796 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56817.log.gz 2023-12-19T09:23:02.797 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56547.log 2023-12-19T09:23:02.797 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2023-12-19T09:23:02.798 INFO:teuthology.orchestra.run.smithi111.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.0.55743.log.gz 2023-12-19T09:23:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52291.log 2023-12-19T09:23:02.798 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63457.log.gz 2023-12-19T09:23:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56547.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56547.log.gz 2023-12-19T09:23:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52708.log 2023-12-19T09:23:02.801 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2023-12-19T09:23:02.802 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.52291.log: /var/log/ceph/ceph-client.admin.52708.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52708.log.gz 2023-12-19T09:23:02.802 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63903.log 2023-12-19T09:23:02.803 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41473.log: gzip -5 --verbose -- 94.1%/var/log/ceph/ceph-client.admin.55845.log -- replaced with /var/log/ceph/ceph-client.0.52291.log.gz 2023-12-19T09:23:02.803 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.804 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2023-12-19T09:23:02.804 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63903.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63903.log.gz 2023-12-19T09:23:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2023-12-19T09:23:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44642.log 2023-12-19T09:23:02.806 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55845.log.gz 2023-12-19T09:23:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40330.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2023-12-19T09:23:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2023-12-19T09:23:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48128.log 2023-12-19T09:23:02.809 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44642.log.gz 2023-12-19T09:23:02.810 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41357.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2023-12-19T09:23:02.810 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61526.log 2023-12-19T09:23:02.811 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57766.log 2023-12-19T09:23:02.812 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48128.log.gz 2023-12-19T09:23:02.812 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61526.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61526.log.gz 2023-12-19T09:23:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2023-12-19T09:23:02.814 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44750.log 2023-12-19T09:23:02.815 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57766.log: /var/log/ceph/ceph-client.admin.40736.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.57766.log.gz 2023-12-19T09:23:02.815 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2023-12-19T09:23:02.816 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50218.log 2023-12-19T09:23:02.816 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49336.log 2023-12-19T09:23:02.817 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44750.log.gz 2023-12-19T09:23:02.818 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50218.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50218.log.gz 2023-12-19T09:23:02.818 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46011.log 2023-12-19T09:23:02.819 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59059.log 2023-12-19T09:23:02.819 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49336.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49336.log.gz 2023-12-19T09:23:02.820 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46011.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46011.log.gz 2023-12-19T09:23:02.820 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46343.log 2023-12-19T09:23:02.822 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42050.log 2023-12-19T09:23:02.822 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59059.log.gz 2023-12-19T09:23:02.823 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46343.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46343.log.gz 2023-12-19T09:23:02.823 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2023-12-19T09:23:02.824 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63717.log 2023-12-19T09:23:02.825 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42050.log: /var/log/ceph/ceph-client.admin.57459.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42050.log.gz 2023-12-19T09:23:02.825 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz 2023-12-19T09:23:02.825 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2023-12-19T09:23:02.826 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45059.log 2023-12-19T09:23:02.827 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63717.log.gz 2023-12-19T09:23:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40223.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz 2023-12-19T09:23:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56667.log 2023-12-19T09:23:02.829 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2023-12-19T09:23:02.829 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45059.log.gz 2023-12-19T09:23:02.831 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56667.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56667.log.gz 2023-12-19T09:23:02.831 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53427.log 2023-12-19T09:23:02.832 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2023-12-19T09:23:02.832 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2023-12-19T09:23:02.834 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53268.log 2023-12-19T09:23:02.834 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53427.log.gz 2023-12-19T09:23:02.835 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42741.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2023-12-19T09:23:02.835 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55687.log 2023-12-19T09:23:02.836 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53184.log 2023-12-19T09:23:02.836 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53268.log.gz 2023-12-19T09:23:02.837 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55687.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55687.log.gz 2023-12-19T09:23:02.838 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49922.log 2023-12-19T09:23:02.839 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48476.log 2023-12-19T09:23:02.839 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53184.log.gz 2023-12-19T09:23:02.840 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49922.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49922.log.gz 2023-12-19T09:23:02.840 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62538.log 2023-12-19T09:23:02.842 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48476.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48476.log.gz 2023-12-19T09:23:02.842 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43973.log 2023-12-19T09:23:02.843 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62538.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62538.log.gz 2023-12-19T09:23:02.844 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62069.log 2023-12-19T09:23:02.845 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43973.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43973.log.gz 2023-12-19T09:23:02.845 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59676.log 2023-12-19T09:23:02.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62069.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62069.log.gz 2023-12-19T09:23:02.847 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62303.log 2023-12-19T09:23:02.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59676.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59676.log.gz 2023-12-19T09:23:02.848 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43192.log 2023-12-19T09:23:02.850 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62303.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62303.log.gz-5 2023-12-19T09:23:02.850 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.46234.log 2023-12-19T09:23:02.852 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43192.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43192.log.gz 2023-12-19T09:23:02.852 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62349.log 2023-12-19T09:23:02.853 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46234.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46234.log.gz 2023-12-19T09:23:02.854 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50898.log 2023-12-19T09:23:02.855 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62349.log: gzip -5 89.3%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.62349.log.gz/var/log/ceph/ceph-client.admin.44972.log 2023-12-19T09:23:02.855 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.857 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50898.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50898.log.gz 2023-12-19T09:23:02.857 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39945.log 2023-12-19T09:23:02.858 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44972.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44972.log.gz 2023-12-19T09:23:02.858 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62430.log 2023-12-19T09:23:02.860 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56338.log 2023-12-19T09:23:02.861 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56010.log 92.4% 2023-12-19T09:23:02.862 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39945.log.gz 2023-12-19T09:23:02.862 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62430.log.gz 2023-12-19T09:23:02.863 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62565.log 2023-12-19T09:23:02.863 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56338.log.gz/var/log/ceph/ceph-client.admin.56010.log: 2023-12-19T09:23:02.864 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53454.log 2023-12-19T09:23:02.864 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56010.log.gz 2023-12-19T09:23:02.865 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62565.log: gzip 87.9%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.62565.log.gz-- 2023-12-19T09:23:02.866 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60577.log 2023-12-19T09:23:02.867 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53454.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.53454.log.gz 2023-12-19T09:23:02.867 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.47761.log 2023-12-19T09:23:02.868 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-12-19T09:23:02.868 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60351.log 2023-12-19T09:23:02.869 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60577.log.gz 2023-12-19T09:23:02.870 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47761.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47761.log.gz 2023-12-19T09:23:02.870 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43084.log 2023-12-19T09:23:02.871 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60351.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60351.log.gz 2023-12-19T09:23:02.871 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47191.log 2023-12-19T09:23:02.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43084.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43084.log.gz 2023-12-19T09:23:02.872 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48557.log 2023-12-19T09:23:02.874 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47191.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47191.log.gz 2023-12-19T09:23:02.874 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51361.log 2023-12-19T09:23:02.874 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48557.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48557.log.gz 2023-12-19T09:23:02.875 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2023-12-19T09:23:02.875 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51361.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51361.log.gz 2023-12-19T09:23:02.875 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48760.log 2023-12-19T09:23:02.877 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40534.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2023-12-19T09:23:02.877 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57430.log 2023-12-19T09:23:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48760.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48760.log.gz 2023-12-19T09:23:02.878 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55555.log 2023-12-19T09:23:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57430.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57430.log.gz 2023-12-19T09:23:02.879 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54677.log 2023-12-19T09:23:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55555.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55555.log.gz 2023-12-19T09:23:02.880 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51112.log 2023-12-19T09:23:02.881 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2023-12-19T09:23:02.882 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54677.log.gz 2023-12-19T09:23:02.882 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51112.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51112.log.gz 2023-12-19T09:23:02.883 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48317.log 2023-12-19T09:23:02.884 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2023-12-19T09:23:02.885 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2023-12-19T09:23:02.886 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42984.log 2023-12-19T09:23:02.887 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48317.log: /var/log/ceph/ceph-client.admin.51083.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48317.log.gz 2023-12-19T09:23:02.888 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2023-12-19T09:23:02.888 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64402.log 2023-12-19T09:23:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42984.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42984.log.gz 2023-12-19T09:23:02.889 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58292.log 2023-12-19T09:23:02.891 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2023-12-19T09:23:02.892 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64402.log: /var/log/ceph/ceph-client.admin.58292.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64402.log.gz 2023-12-19T09:23:02.892 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58292.log.gz 2023-12-19T09:23:02.892 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57345.log 2023-12-19T09:23:02.894 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43893.log 2023-12-19T09:23:02.894 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2023-12-19T09:23:02.895 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57345.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57345.log.gz 2023-12-19T09:23:02.895 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51415.log 2023-12-19T09:23:02.896 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45832.log 2023-12-19T09:23:02.896 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43893.log.gz 2023-12-19T09:23:02.897 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51415.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51415.log.gz 2023-12-19T09:23:02.897 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50135.log 2023-12-19T09:23:02.898 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59550.log 2023-12-19T09:23:02.898 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45832.log.gz 2023-12-19T09:23:02.899 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50135.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50135.log.gz 2023-12-19T09:23:02.900 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43017.log 2023-12-19T09:23:02.901 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.59550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60173.log 2023-12-19T09:23:02.902 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43017.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43017.log.gz 2023-12-19T09:23:02.902 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64834.log 2023-12-19T09:23:02.904 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47914.log 2023-12-19T09:23:02.904 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60173.log.gz 2023-12-19T09:23:02.905 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64834.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64834.log.gz 2023-12-19T09:23:02.905 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-19T09:23:02.906 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47914.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47914.log.gz 2023-12-19T09:23:02.907 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63581.log 2023-12-19T09:23:02.908 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59416.log 2023-12-19T09:23:02.910 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.63581.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63581.log.gz 2023-12-19T09:23:02.910 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64618.log 2023-12-19T09:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54337.log 2023-12-19T09:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59416.log.gz 2023-12-19T09:23:02.913 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2023-12-19T09:23:02.913 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64618.log.gz 2023-12-19T09:23:02.915 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54337.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54337.log.gz 2023-12-19T09:23:02.915 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42104.log 2023-12-19T09:23:02.916 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54525.log 2023-12-19T09:23:02.916 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2023-12-19T09:23:02.917 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55163.log 2023-12-19T09:23:02.918 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42104.log.gz 2023-12-19T09:23:02.919 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2023-12-19T09:23:02.919 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54525.log.gz 2023-12-19T09:23:02.920 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55163.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55163.log.gz 2023-12-19T09:23:02.920 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46207.log 2023-12-19T09:23:02.922 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64996.log 2023-12-19T09:23:02.922 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2023-12-19T09:23:02.923 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46207.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46207.log.gz 2023-12-19T09:23:02.923 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2023-12-19T09:23:02.924 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43219.log 2023-12-19T09:23:02.925 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64996.log.gz 2023-12-19T09:23:02.926 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46842.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2023-12-19T09:23:02.926 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52339.log 2023-12-19T09:23:02.927 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2023-12-19T09:23:02.927 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43219.log.gz 2023-12-19T09:23:02.928 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2023-12-19T09:23:02.929 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52339.log.gz 2023-12-19T09:23:02.930 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52235.log 2023-12-19T09:23:02.930 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2023-12-19T09:23:02.931 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49760.log 2023-12-19T09:23:02.931 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2023-12-19T09:23:02.932 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52235.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52235.log.gz 2023-12-19T09:23:02.933 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53130.log 2023-12-19T09:23:02.934 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2023-12-19T09:23:02.934 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49760.log.gz 2023-12-19T09:23:02.935 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53130.log: 95.2% -- replaced with /var/log/ceph/ceph-client.0.59550.log.gz 2023-12-19T09:23:02.936 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63963.log 2023-12-19T09:23:02.936 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53130.log.gz 2023-12-19T09:23:02.937 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47271.log 2023-12-19T09:23:02.937 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2023-12-19T09:23:02.938 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60233.log 2023-12-19T09:23:02.938 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63963.log.gz 2023-12-19T09:23:02.939 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47271.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47271.log.gz 2023-12-19T09:23:02.940 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64944.log 2023-12-19T09:23:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60233.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60233.log.gz 2023-12-19T09:23:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51875.log 2023-12-19T09:23:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60293.log 2023-12-19T09:23:02.943 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64944.log.gz 2023-12-19T09:23:02.944 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51875.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51875.log.gz 2023-12-19T09:23:02.944 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2023-12-19T09:23:02.945 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46397.log 2023-12-19T09:23:02.945 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60293.log.gz 2023-12-19T09:23:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40322.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2023-12-19T09:23:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51790.log 2023-12-19T09:23:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46397.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46397.log.gz 2023-12-19T09:23:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40546.log 2023-12-19T09:23:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47973.log 2023-12-19T09:23:02.950 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51790.log.gz 2023-12-19T09:23:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40546.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40546.log.gz 2023-12-19T09:23:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61128.log 2023-12-19T09:23:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47973.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47973.log.gz 2023-12-19T09:23:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52614.log 2023-12-19T09:23:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52446.log 2023-12-19T09:23:02.955 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61128.log.gz 2023-12-19T09:23:02.956 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52614.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52614.log.gz 2023-12-19T09:23:02.956 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47218.log 2023-12-19T09:23:02.958 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54706.log 2023-12-19T09:23:02.959 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52446.log.gz 2023-12-19T09:23:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47218.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47218.log.gz 2023-12-19T09:23:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62457.log 2023-12-19T09:23:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64780.log 2023-12-19T09:23:02.961 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54706.log.gz 2023-12-19T09:23:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62457.log.gz 2023-12-19T09:23:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64193.log 2023-12-19T09:23:02.963 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46610.log 2023-12-19T09:23:02.964 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64780.log.gz 2023-12-19T09:23:02.965 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64193.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64193.log.gz 2023-12-19T09:23:02.965 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2023-12-19T09:23:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46610.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46610.log.gz 2023-12-19T09:23:02.967 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64807.log 2023-12-19T09:23:02.969 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55271.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2023-12-19T09:23:02.970 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42768.log 2023-12-19T09:23:02.970 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2023-12-19T09:23:02.970 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64807.log.gz 2023-12-19T09:23:02.971 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42768.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42768.log.gz 2023-12-19T09:23:02.971 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2023-12-19T09:23:02.972 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42239.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2023-12-19T09:23:02.973 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49814.log 2023-12-19T09:23:02.974 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62861.log 2023-12-19T09:23:02.974 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2023-12-19T09:23:02.975 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49814.log.gz 2023-12-19T09:23:02.976 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60411.log 2023-12-19T09:23:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62861.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62861.log.gz 2023-12-19T09:23:02.978 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2023-12-19T09:23:02.979 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59706.log 2023-12-19T09:23:02.980 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60411.log.gz 2023-12-19T09:23:02.981 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40543.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2023-12-19T09:23:02.981 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61037.log 2023-12-19T09:23:02.982 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58846.log 2023-12-19T09:23:02.982 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59706.log.gz 2023-12-19T09:23:02.984 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61037.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61037.log.gz 2023-12-19T09:23:02.984 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60262.log 2023-12-19T09:23:02.985 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58846.log: gzip -5 88.7%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.58846.log.gz/var/log/ceph/ceph-client.admin.42714.log 2023-12-19T09:23:02.985 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:02.987 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 87.9% 2023-12-19T09:23:02.987 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60262.log.gz 2023-12-19T09:23:02.988 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42714.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42714.log.gz 2023-12-19T09:23:02.988 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54923.log 2023-12-19T09:23:02.990 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57712.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2023-12-19T09:23:02.990 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2023-12-19T09:23:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54923.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54923.log.gz 2023-12-19T09:23:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51846.log 2023-12-19T09:23:02.993 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54154.log 2023-12-19T09:23:02.993 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2023-12-19T09:23:02.994 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2023-12-19T09:23:02.995 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51846.log.gz 2023-12-19T09:23:02.996 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55244.log 2023-12-19T09:23:02.996 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54154.log.gz 2023-12-19T09:23:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53838.log 2023-12-19T09:23:02.998 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2023-12-19T09:23:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2023-12-19T09:23:02.999 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55244.log.gz 2023-12-19T09:23:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53838.log.gz 2023-12-19T09:23:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2023-12-19T09:23:03.001 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64109.log 2023-12-19T09:23:03.002 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2023-12-19T09:23:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39331.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2023-12-19T09:23:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57226.log 2023-12-19T09:23:03.005 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64109.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64109.log.gz 2023-12-19T09:23:03.005 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50162.log 2023-12-19T09:23:03.006 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57226.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57226.log.gz 2023-12-19T09:23:03.006 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59032.log 2023-12-19T09:23:03.007 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50162.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2023-12-19T09:23:03.007 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62918.log 2023-12-19T09:23:03.009 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49574.log 2023-12-19T09:23:03.009 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59032.log.gz 2023-12-19T09:23:03.010 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62918.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62918.log.gz 2023-12-19T09:23:03.010 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63190.log 2023-12-19T09:23:03.011 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49574.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49574.log.gz 2023-12-19T09:23:03.012 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64753.log 2023-12-19T09:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44914.log 2023-12-19T09:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64753.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63190.log.gz 2023-12-19T09:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64753.log.gz 2023-12-19T09:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28954.log 2023-12-19T09:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62189.log 2023-12-19T09:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44914.log.gz 2023-12-19T09:23:03.017 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.tmp-client.admin.28954.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28954.log.gz 2023-12-19T09:23:03.017 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53784.log 2023-12-19T09:23:03.018 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45203.log 2023-12-19T09:23:03.019 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62189.log.gz 2023-12-19T09:23:03.020 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53784.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53784.log.gz 2023-12-19T09:23:03.020 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54370.log 2023-12-19T09:23:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45203.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45203.log.gz-5 --verbose 2023-12-19T09:23:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:-- /var/log/ceph/ceph-client.admin.40750.log 2023-12-19T09:23:03.024 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60467.log 2023-12-19T09:23:03.024 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54370.log.gz 2023-12-19T09:23:03.025 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42538.log 2023-12-19T09:23:03.025 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2023-12-19T09:23:03.026 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51523.log 2023-12-19T09:23:03.026 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60467.log.gz 2023-12-19T09:23:03.028 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57521.log 2023-12-19T09:23:03.028 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42538.log.gz 2023-12-19T09:23:03.029 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46583.log 2023-12-19T09:23:03.029 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51523.log.gz 2023-12-19T09:23:03.031 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53892.log 2023-12-19T09:23:03.031 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57521.log.gz 2023-12-19T09:23:03.032 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51910.log 2023-12-19T09:23:03.032 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46583.log.gz 2023-12-19T09:23:03.033 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54761.log 2023-12-19T09:23:03.034 INFO:teuthology.orchestra.run.smithi111.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53892.log.gz 2023-12-19T09:23:03.035 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62778.log 2023-12-19T09:23:03.035 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51910.log.gz 2023-12-19T09:23:03.036 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54761.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54761.log.gz 2023-12-19T09:23:03.036 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2023-12-19T09:23:03.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62778.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62778.log.gz 2023-12-19T09:23:03.041 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62751.log 2023-12-19T09:23:03.041 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41249.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz-5 2023-12-19T09:23:03.042 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.60712.log 2023-12-19T09:23:03.042 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49217.log 2023-12-19T09:23:03.042 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62751.log.gz 2023-12-19T09:23:03.042 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2023-12-19T09:23:03.043 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60712.log.gz 2023-12-19T09:23:03.044 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47794.log 2023-12-19T09:23:03.044 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49217.log.gz 2023-12-19T09:23:03.046 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64726.log 2023-12-19T09:23:03.046 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2023-12-19T09:23:03.047 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50663.log 2023-12-19T09:23:03.047 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47794.log.gz 2023-12-19T09:23:03.049 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46289.log 2023-12-19T09:23:03.049 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64726.log.gz 2023-12-19T09:23:03.050 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.50663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60804.log 2023-12-19T09:23:03.051 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.0.50663.log.gz 2023-12-19T09:23:03.051 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46289.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46289.log.gz 2023-12-19T09:23:03.052 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55031.log 2023-12-19T09:23:03.052 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61072.log 2023-12-19T09:23:03.053 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60804.log.gz 2023-12-19T09:23:03.054 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55031.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55031.log.gz 2023-12-19T09:23:03.054 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60550.log 2023-12-19T09:23:03.055 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61072.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61072.log.gz 2023-12-19T09:23:03.056 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62129.log 2023-12-19T09:23:03.057 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60550.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60550.log.gz 2023-12-19T09:23:03.057 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34058.log 2023-12-19T09:23:03.059 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62129.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62129.log.gz 2023-12-19T09:23:03.059 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42903.log 2023-12-19T09:23:03.060 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.34058.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34058.log.gz 2023-12-19T09:23:03.061 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58792.log 2023-12-19T09:23:03.061 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42903.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42903.log.gz 2023-12-19T09:23:03.062 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48650.log 2023-12-19T09:23:03.063 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39147.log 2023-12-19T09:23:03.063 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58792.log.gz 2023-12-19T09:23:03.064 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48650.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48650.log.gz 2023-12-19T09:23:03.065 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2023-12-19T09:23:03.066 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39147.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39147.log.gz 2023-12-19T09:23:03.066 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54842.log 2023-12-19T09:23:03.068 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63992.log 2023-12-19T09:23:03.068 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2023-12-19T09:23:03.069 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2023-12-19T09:23:03.069 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54842.log.gz 2023-12-19T09:23:03.070 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63992.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63992.log.gz 2023-12-19T09:23:03.070 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56758.log 2023-12-19T09:23:03.072 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44885.log 2023-12-19T09:23:03.072 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48209.log: 88.6%/var/log/ceph/ceph-client.admin.56758.log: -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz 2023-12-19T09:23:03.073 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56758.log.gz 2023-12-19T09:23:03.073 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47354.log 2023-12-19T09:23:03.075 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44885.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44885.log.gz 2023-12-19T09:23:03.075 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41594.log 2023-12-19T09:23:03.077 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47354.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47354.log.gz 2023-12-19T09:23:03.077 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59873.log 2023-12-19T09:23:03.078 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.41594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49029.log 2023-12-19T09:23:03.078 INFO:teuthology.orchestra.run.smithi111.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.41594.log.gz 2023-12-19T09:23:03.079 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63053.log 2023-12-19T09:23:03.080 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59873.log.gz 2023-12-19T09:23:03.081 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.49029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58060.log 2023-12-19T09:23:03.082 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56311.log 2023-12-19T09:23:03.082 INFO:teuthology.orchestra.run.smithi111.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.49029.log.gz 2023-12-19T09:23:03.083 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63053.log.gz 2023-12-19T09:23:03.083 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53999.log 2023-12-19T09:23:03.084 INFO:teuthology.orchestra.run.smithi111.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58060.log.gz 2023-12-19T09:23:03.084 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56311.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56311.log.gz 2023-12-19T09:23:03.085 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56880.log 2023-12-19T09:23:03.086 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.53999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2023-12-19T09:23:03.089 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43760.log 2023-12-19T09:23:03.089 INFO:teuthology.orchestra.run.smithi111.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.56880.log.gz 2023-12-19T09:23:03.089 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48236.log 2023-12-19T09:23:03.090 INFO:teuthology.orchestra.run.smithi111.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2023-12-19T09:23:03.090 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62158.log 2023-12-19T09:23:03.090 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43760.log.gz 2023-12-19T09:23:03.093 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42957.log 2023-12-19T09:23:03.093 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48236.log.gz 2023-12-19T09:23:03.093 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63297.log 2023-12-19T09:23:03.094 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62158.log.gz 2023-12-19T09:23:03.095 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61719.log 2023-12-19T09:23:03.095 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42957.log.gz 2023-12-19T09:23:03.096 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.63297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49841.log 2023-12-19T09:23:03.098 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2023-12-19T09:23:03.098 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61719.log.gz 2023-12-19T09:23:03.098 INFO:teuthology.orchestra.run.smithi111.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.63297.log.gz 2023-12-19T09:23:03.099 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55930.log 2023-12-19T09:23:03.099 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49841.log.gz 2023-12-19T09:23:03.101 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 87.6%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz/var/log/ceph/ceph-client.admin.45256.log 2023-12-19T09:23:03.101 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:03.102 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2023-12-19T09:23:03.102 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55930.log.gz 2023-12-19T09:23:03.104 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48028.log 2023-12-19T09:23:03.104 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45256.log.gz 2023-12-19T09:23:03.105 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57948.log 2023-12-19T09:23:03.105 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2023-12-19T09:23:03.106 INFO:teuthology.orchestra.run.smithi111.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-12-19T09:23:03.106 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45419.log 2023-12-19T09:23:03.107 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48028.log.gz 2023-12-19T09:23:03.107 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57948.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57948.log.gz 2023-12-19T09:23:03.107 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47381.log 2023-12-19T09:23:03.109 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58711.log 2023-12-19T09:23:03.109 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45419.log.gz 2023-12-19T09:23:03.110 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47381.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47381.log.gz 2023-12-19T09:23:03.110 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58567.log 2023-12-19T09:23:03.112 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58711.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58711.log.gz 2023-12-19T09:23:03.112 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59389.log 2023-12-19T09:23:03.113 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58567.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58567.log.gz 2023-12-19T09:23:03.113 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42687.log 2023-12-19T09:23:03.114 INFO:teuthology.orchestra.run.smithi111.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.53999.log.gz 2023-12-19T09:23:03.114 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52527.log 2023-12-19T09:23:03.114 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59389.log.gz 2023-12-19T09:23:03.115 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42687.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42687.log.gz 2023-12-19T09:23:03.115 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58411.log 2023-12-19T09:23:03.116 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59598.log 2023-12-19T09:23:03.117 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52527.log.gz 2023-12-19T09:23:03.118 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53595.log 2023-12-19T09:23:03.118 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58411.log.gz 2023-12-19T09:23:03.119 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59598.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59598.log.gz 2023-12-19T09:23:03.119 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49425.log 2023-12-19T09:23:03.120 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2023-12-19T09:23:03.120 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53595.log.gz 2023-12-19T09:23:03.121 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49425.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49425.log.gz 2023-12-19T09:23:03.122 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57401.log 2023-12-19T09:23:03.123 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51709.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2023-12-19T09:23:03.123 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63540.log 2023-12-19T09:23:03.124 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57401.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57401.log.gz 2023-12-19T09:23:03.124 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53811.log 2023-12-19T09:23:03.125 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53483.log 2023-12-19T09:23:03.125 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63540.log.gz 2023-12-19T09:23:03.126 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53811.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53811.log.gz 2023-12-19T09:23:03.126 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46126.log 2023-12-19T09:23:03.127 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62098.log 2023-12-19T09:23:03.127 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53483.log.gz 2023-12-19T09:23:03.128 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46126.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46126.log.gz 2023-12-19T09:23:03.128 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60202.log 2023-12-19T09:23:03.129 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53943.log 2023-12-19T09:23:03.130 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62098.log.gz 2023-12-19T09:23:03.130 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60202.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60202.log.gz 2023-12-19T09:23:03.131 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53703.log 2023-12-19T09:23:03.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40763.log 2023-12-19T09:23:03.132 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53943.log.gz 2023-12-19T09:23:03.132 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53703.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53703.log.gz 2023-12-19T09:23:03.133 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2023-12-19T09:23:03.134 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2023-12-19T09:23:03.134 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40763.log.gz 2023-12-19T09:23:03.135 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40059.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2023-12-19T09:23:03.135 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45892.log 2023-12-19T09:23:03.136 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63241.log 2023-12-19T09:23:03.136 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2023-12-19T09:23:03.137 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45892.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45892.log.gz 2023-12-19T09:23:03.138 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45631.log 2023-12-19T09:23:03.139 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57548.log 2023-12-19T09:23:03.139 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63241.log.gz 2023-12-19T09:23:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45631.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45631.log.gz 2023-12-19T09:23:03.140 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2023-12-19T09:23:03.141 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2023-12-19T09:23:03.141 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57548.log.gz 2023-12-19T09:23:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44943.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2023-12-19T09:23:03.143 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57193.log 2023-12-19T09:23:03.144 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42376.log 2023-12-19T09:23:03.144 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42633.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2023-12-19T09:23:03.145 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57193.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57193.log.gz 2023-12-19T09:23:03.146 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59443.log 2023-12-19T09:23:03.146 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63080.log 2023-12-19T09:23:03.146 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42376.log.gz 2023-12-19T09:23:03.148 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62657.log 2023-12-19T09:23:03.148 INFO:teuthology.orchestra.run.smithi111.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.59443.log.gz 2023-12-19T09:23:03.149 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63080.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63080.log.gz 2023-12-19T09:23:03.149 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58383.log 2023-12-19T09:23:03.150 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62657.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62657.log.gz 2023-12-19T09:23:03.150 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48787.log 2023-12-19T09:23:03.151 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57817.log 2023-12-19T09:23:03.152 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58383.log.gz 2023-12-19T09:23:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48787.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48787.log.gz 2023-12-19T09:23:03.153 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40140.log 2023-12-19T09:23:03.154 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57817.log: gzip -5 88.5%--verbose -- replaced with /var/log/ceph/ceph-client.admin.57817.log.gz-- 2023-12-19T09:23:03.154 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43601.log 2023-12-19T09:23:03.155 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2023-12-19T09:23:03.155 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40140.log.gz 2023-12-19T09:23:03.156 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63350.log 2023-12-19T09:23:03.156 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43601.log.gz 2023-12-19T09:23:03.157 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48868.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz 2023-12-19T09:23:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42795.log 2023-12-19T09:23:03.159 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39770.log 2023-12-19T09:23:03.159 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63350.log.gz 2023-12-19T09:23:03.160 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60523.log 2023-12-19T09:23:03.160 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42795.log.gz 2023-12-19T09:23:03.161 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57631.log 2023-12-19T09:23:03.161 INFO:teuthology.orchestra.run.smithi111.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz 2023-12-19T09:23:03.162 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60523.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60523.log.gz 2023-12-19T09:23:03.162 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51171.log 2023-12-19T09:23:03.163 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61290.log 2023-12-19T09:23:03.163 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57631.log.gz 2023-12-19T09:23:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51171.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51171.log.gz 2023-12-19T09:23:03.164 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2023-12-19T09:23:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49484.log 2023-12-19T09:23:03.166 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61290.log.gz 2023-12-19T09:23:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40327.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2023-12-19T09:23:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.61637.log 2023-12-19T09:23:03.168 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58442.log 2023-12-19T09:23:03.168 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49484.log.gz 2023-12-19T09:23:03.169 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61637.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61637.log.gz 2023-12-19T09:23:03.169 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57494.log 2023-12-19T09:23:03.170 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62275.log 2023-12-19T09:23:03.170 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58442.log.gz 2023-12-19T09:23:03.171 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47732.log 2023-12-19T09:23:03.171 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57494.log.gz 2023-12-19T09:23:03.173 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48155.log 2023-12-19T09:23:03.173 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62275.log.gz 2023-12-19T09:23:03.174 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2023-12-19T09:23:03.174 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47732.log.gz 2023-12-19T09:23:03.175 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62484.log 2023-12-19T09:23:03.175 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48155.log.gz 2023-12-19T09:23:03.176 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64645.log 2023-12-19T09:23:03.176 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2023-12-19T09:23:03.177 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2023-12-19T09:23:03.177 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62484.log.gz 2023-12-19T09:23:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64645.log.gz 2023-12-19T09:23:03.178 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49706.log 2023-12-19T09:23:03.179 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-19T09:23:03.180 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2023-12-19T09:23:03.181 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49706.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49706.log.gz 2023-12-19T09:23:03.181 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52769.log 2023-12-19T09:23:03.182 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63872.log 2023-12-19T09:23:03.184 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2023-12-19T09:23:03.184 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52769.log.gz 2023-12-19T09:23:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62009.log 2023-12-19T09:23:03.186 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63872.log.gz 2023-12-19T09:23:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49895.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2023-12-19T09:23:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49104.log 2023-12-19T09:23:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47004.log 2023-12-19T09:23:03.189 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62009.log.gz 2023-12-19T09:23:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49104.log: 88.5% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-19T09:23:03.189 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49104.log.gz 2023-12-19T09:23:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51052.log 2023-12-19T09:23:03.191 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57873.log 2023-12-19T09:23:03.191 INFO:teuthology.orchestra.run.smithi111.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.47004.log.gz 2023-12-19T09:23:03.192 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2023-12-19T09:23:03.192 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51052.log.gz 2023-12-19T09:23:03.194 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.57873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40005.log 2023-12-19T09:23:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40761.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2023-12-19T09:23:03.195 INFO:teuthology.orchestra.run.smithi111.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.57873.log.gz 2023-12-19T09:23:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64429.log 2023-12-19T09:23:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52475.log 2023-12-19T09:23:03.196 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40005.log.gz 2023-12-19T09:23:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64164.log 2023-12-19T09:23:03.197 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64429.log.gz 2023-12-19T09:23:03.198 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64564.log 2023-12-19T09:23:03.198 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52475.log.gz 2023-12-19T09:23:03.199 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54494.log 2023-12-19T09:23:03.199 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64164.log.gz 2023-12-19T09:23:03.200 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64564.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64564.log.gz 2023-12-19T09:23:03.201 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46786.log 2023-12-19T09:23:03.202 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59198.log 2023-12-19T09:23:03.202 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54494.log.gz 2023-12-19T09:23:03.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46786.log: gzip -5 87.7%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46786.log.gz/var/log/ceph/ceph-client.admin.44346.log 2023-12-19T09:23:03.204 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:03.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59198.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59198.log.gz 2023-12-19T09:23:03.205 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54647.log 2023-12-19T09:23:03.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42876.log 2023-12-19T09:23:03.207 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44346.log.gz 2023-12-19T09:23:03.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54647.log.gz 2023-12-19T09:23:03.208 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-19T09:23:03.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42876.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42876.log.gz 2023-12-19T09:23:03.209 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49395.log 2023-12-19T09:23:03.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2023-12-19T09:23:03.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49395.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49395.log.gz 2023-12-19T09:23:03.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45117.log 2023-12-19T09:23:03.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60979.log 2023-12-19T09:23:03.214 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2023-12-19T09:23:03.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45117.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45117.log.gz 2023-12-19T09:23:03.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39204.log 2023-12-19T09:23:03.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60979.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60979.log.gz 2023-12-19T09:23:03.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48895.log 2023-12-19T09:23:03.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39204.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39204.log.gz 2023-12-19T09:23:03.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58819.log 2023-12-19T09:23:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48895.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48895.log.gz 2023-12-19T09:23:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39662.log 2023-12-19T09:23:03.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62832.log 2023-12-19T09:23:03.222 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58819.log.gz 2023-12-19T09:23:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39662.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39662.log.gz 2023-12-19T09:23:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58765.log 2023-12-19T09:23:03.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62832.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62832.log.gz 2023-12-19T09:23:03.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42660.log 2023-12-19T09:23:03.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43574.log 2023-12-19T09:23:03.226 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58765.log.gz 2023-12-19T09:23:03.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42660.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42660.log.gz 2023-12-19T09:23:03.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2023-12-19T09:23:03.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43574.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43574.log.gz 2023-12-19T09:23:03.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2023-12-19T09:23:03.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40316.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2023-12-19T09:23:03.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52587.log 2023-12-19T09:23:03.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53865.log 2023-12-19T09:23:03.234 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2023-12-19T09:23:03.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52587.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52587.log.gz 2023-12-19T09:23:03.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2023-12-19T09:23:03.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50527.log 2023-12-19T09:23:03.236 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53865.log.gz 2023-12-19T09:23:03.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63813.log 2023-12-19T09:23:03.238 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2023-12-19T09:23:03.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50527.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50527.log.gz 2023-12-19T09:23:03.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44427.log 2023-12-19T09:23:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2023-12-19T09:23:03.240 INFO:teuthology.orchestra.run.smithi111.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63813.log.gz 2023-12-19T09:23:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44427.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44427.log.gz 2023-12-19T09:23:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59958.log 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41834.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44804.log 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59958.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59958.log.gz 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48586.log 2023-12-19T09:23:03.247 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44804.log.gz 2023-12-19T09:23:03.248 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48677.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz 2023-12-19T09:23:03.248 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61155.log 2023-12-19T09:23:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48586.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48586.log.gz 2023-12-19T09:23:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60952.log 2023-12-19T09:23:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62038.log 2023-12-19T09:23:03.250 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61155.log.gz 2023-12-19T09:23:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45146.log 2023-12-19T09:23:03.253 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60952.log.gz 2023-12-19T09:23:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2023-12-19T09:23:03.253 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62038.log.gz 2023-12-19T09:23:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45146.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45146.log.gz 2023-12-19T09:23:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57374.log 2023-12-19T09:23:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60604.log 2023-12-19T09:23:03.256 INFO:teuthology.orchestra.run.smithi111.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2023-12-19T09:23:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57374.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.57374.log.gz 2023-12-19T09:23:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58168.log 2023-12-19T09:23:03.258 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60604.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60604.log.gz 2023-12-19T09:23:03.258 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50473.log 2023-12-19T09:23:03.260 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58168.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.58168.log.gz 2023-12-19T09:23:03.260 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.48615.log 2023-12-19T09:23:03.261 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50473.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50473.log.gz 2023-12-19T09:23:03.262 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55474.log 2023-12-19T09:23:03.263 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48615.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48615.log.gz 2023-12-19T09:23:03.263 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64888.log 2023-12-19T09:23:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55474.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55474.log.gz 2023-12-19T09:23:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2023-12-19T09:23:03.265 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53211.log 2023-12-19T09:23:03.266 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64888.log.gz 2023-12-19T09:23:03.267 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42158.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2023-12-19T09:23:03.268 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51469.log 2023-12-19T09:23:03.269 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53211.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53211.log.gz 2023-12-19T09:23:03.269 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56607.log 2023-12-19T09:23:03.270 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51469.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51469.log.gzgzip 2023-12-19T09:23:03.270 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.53076.log 2023-12-19T09:23:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56607.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56607.log.gz 2023-12-19T09:23:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2023-12-19T09:23:03.273 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49077.log 2023-12-19T09:23:03.274 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53076.log.gz 2023-12-19T09:23:03.275 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47637.log 2023-12-19T09:23:03.276 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2023-12-19T09:23:03.277 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49077.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49077.log.gz 2023-12-19T09:23:03.277 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43866.log 2023-12-19T09:23:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47637.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2023-12-19T09:23:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51550.log 2023-12-19T09:23:03.279 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51642.log 2023-12-19T09:23:03.279 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43866.log.gz 2023-12-19T09:23:03.281 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2023-12-19T09:23:03.281 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51550.log.gz 2023-12-19T09:23:03.282 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42322.log 2023-12-19T09:23:03.282 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51642.log.gz 2023-12-19T09:23:03.284 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50964.log 2023-12-19T09:23:03.284 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2023-12-19T09:23:03.285 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57053.log 2023-12-19T09:23:03.285 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42322.log.gz 2023-12-19T09:23:03.286 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50964.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50964.log.gz 2023-12-19T09:23:03.286 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57026.log 2023-12-19T09:23:03.287 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51496.log 2023-12-19T09:23:03.288 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57053.log.gz 2023-12-19T09:23:03.289 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57026.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57026.log.gz 2023-12-19T09:23:03.289 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57975.log 2023-12-19T09:23:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51496.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51496.log.gz 2023-12-19T09:23:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55096.log 2023-12-19T09:23:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45365.log 2023-12-19T09:23:03.292 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57975.log.gz 2023-12-19T09:23:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55096.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55096.log.gz 2023-12-19T09:23:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61344.log 2023-12-19T09:23:03.294 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2023-12-19T09:23:03.295 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45365.log.gz 2023-12-19T09:23:03.296 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2023-12-19T09:23:03.297 INFO:teuthology.orchestra.run.smithi111.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61344.log.gz 2023-12-19T09:23:03.298 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44373.log 2023-12-19T09:23:03.299 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2023-12-19T09:23:03.299 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54977.log 2023-12-19T09:23:03.299 INFO:teuthology.orchestra.run.smithi111.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2023-12-19T09:23:03.300 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44373.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44373.log.gz 2023-12-19T09:23:03.300 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63163.log 2023-12-19T09:23:03.302 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54977.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54977.log.gz 2023-12-19T09:23:03.302 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60658.log 2023-12-19T09:23:03.303 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63163.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63163.log.gz-5 2023-12-19T09:23:03.303 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.60925.log 2023-12-19T09:23:03.304 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2023-12-19T09:23:03.305 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60658.log.gz 2023-12-19T09:23:03.305 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64247.log 2023-12-19T09:23:03.306 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60925.log.gz 2023-12-19T09:23:03.307 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50605.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2023-12-19T09:23:03.307 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41942.log 2023-12-19T09:23:03.309 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64247.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64247.log.gz 2023-12-19T09:23:03.309 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45952.log 2023-12-19T09:23:03.311 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42185.log 2023-12-19T09:23:03.311 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41942.log.gz 2023-12-19T09:23:03.312 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45952.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45952.log.gz 2023-12-19T09:23:03.312 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48263.log 2023-12-19T09:23:03.313 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42185.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-19T09:23:03.313 INFO:teuthology.orchestra.run.smithi111.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42185.log.gz 2023-12-19T09:23:03.314 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48263.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48263.log.gz 2023-12-19T09:23:03.315 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2023-12-19T09:23:03.316 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62403.log 2023-12-19T09:23:03.318 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55898.log 2023-12-19T09:23:03.318 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2023-12-19T09:23:03.319 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51388.log 2023-12-19T09:23:03.319 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62403.log.gz 2023-12-19T09:23:03.320 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55898.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55898.log.gz 2023-12-19T09:23:03.320 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2023-12-19T09:23:03.322 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43819.log 2023-12-19T09:23:03.322 INFO:teuthology.orchestra.run.smithi111.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51388.log.gz 2023-12-19T09:23:03.323 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2023-12-19T09:23:03.323 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2023-12-19T09:23:03.325 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.43819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45088.log 2023-12-19T09:23:03.327 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2023-12-19T09:23:03.327 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2023-12-19T09:23:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45088.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.45088.log.gz-5 2023-12-19T09:23:03.328 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40757.log 2023-12-19T09:23:03.330 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43327.log 2023-12-19T09:23:03.330 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2023-12-19T09:23:03.331 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2023-12-19T09:23:03.331 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2023-12-19T09:23:03.332 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47883.log 2023-12-19T09:23:03.332 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43327.log.gz 2023-12-19T09:23:03.334 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43412.log 2023-12-19T09:23:03.334 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2023-12-19T09:23:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47883.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47883.log.gz 2023-12-19T09:23:03.335 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44696.log 2023-12-19T09:23:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43412.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.43412.log.gz 2023-12-19T09:23:03.337 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.44858.log 2023-12-19T09:23:03.338 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43383.log 2023-12-19T09:23:03.338 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44696.log.gz 2023-12-19T09:23:03.340 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57080.log 2023-12-19T09:23:03.340 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44858.log.gz 2023-12-19T09:23:03.341 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49131.log 2023-12-19T09:23:03.341 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43383.log.gz 2023-12-19T09:23:03.342 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57080.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57080.log.gz 2023-12-19T09:23:03.343 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56945.log 2023-12-19T09:23:03.344 INFO:teuthology.orchestra.run.smithi111.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49131.log.gz 2023-12-19T09:23:03.345 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55447.log 2023-12-19T09:23:03.346 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 2023-12-19T09:23:03.346 INFO:teuthology.orchestra.run.smithi111.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56945.log.gz 2023-12-19T09:23:03.348 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55447.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55447.log.gz 2023-12-19T09:23:03.348 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47854.log 2023-12-19T09:23:03.349 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54815.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2023-12-19T09:23:03.349 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2023-12-19T09:23:03.351 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39608.log 2023-12-19T09:23:03.351 INFO:teuthology.orchestra.run.smithi111.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47854.log.gz 2023-12-19T09:23:03.353 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41411.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2023-12-19T09:23:03.353 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42293.log 2023-12-19T09:23:03.354 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39608.log.gz 2023-12-19T09:23:03.354 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60014.log 2023-12-19T09:23:03.356 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2023-12-19T09:23:03.356 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42293.log.gz 2023-12-19T09:23:03.358 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52679.log 2023-12-19T09:23:03.358 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60014.log.gz 2023-12-19T09:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39635.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2023-12-19T09:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.44454.log 2023-12-19T09:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2023-12-19T09:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52679.log.gz 2023-12-19T09:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47823.log 2023-12-19T09:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44454.log.gz 2023-12-19T09:23:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40332.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2023-12-19T09:23:03.364 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49184.log 2023-12-19T09:23:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47823.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47823.log.gz 2023-12-19T09:23:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62999.log 2023-12-19T09:23:03.367 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57658.log 2023-12-19T09:23:03.367 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49184.log.gz 2023-12-19T09:23:03.369 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.62999.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62999.log.gz 2023-12-19T09:23:03.369 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.50014.log 2023-12-19T09:23:03.370 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57658.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57658.log.gz 2023-12-19T09:23:03.370 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57604.log 2023-12-19T09:23:03.372 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50014.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50014.log.gz 2023-12-19T09:23:03.372 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56844.log 2023-12-19T09:23:03.373 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57604.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57604.log.gz 2023-12-19T09:23:03.374 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51021.log 2023-12-19T09:23:03.375 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43466.log 2023-12-19T09:23:03.375 INFO:teuthology.orchestra.run.smithi111.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56844.log.gz 2023-12-19T09:23:03.376 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61008.log 2023-12-19T09:23:03.376 INFO:teuthology.orchestra.run.smithi111.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51021.log.gz 2023-12-19T09:23:03.378 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43466.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43466.log.gz 2023-12-19T09:23:03.379 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.61008.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61008.log.gz 2023-12-19T09:23:03.379 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2023-12-19T09:23:03.381 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64301.log 2023-12-19T09:23:03.383 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58978.log 2023-12-19T09:23:03.383 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2023-12-19T09:23:03.384 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64969.log 2023-12-19T09:23:03.384 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64301.log.gz 2023-12-19T09:23:03.385 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40312.log 2023-12-19T09:23:03.385 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58978.log.gz 2023-12-19T09:23:03.387 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2023-12-19T09:23:03.387 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64969.log.gz 2023-12-19T09:23:03.388 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40312.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40312.log.gz 2023-12-19T09:23:03.388 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61451.log 2023-12-19T09:23:03.390 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49733.log 2023-12-19T09:23:03.390 INFO:teuthology.orchestra.run.smithi111.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2023-12-19T09:23:03.391 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.61451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63748.log 2023-12-19T09:23:03.393 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49733.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49733.log.gz 2023-12-19T09:23:03.394 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.63748.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63748.log.gz 2023-12-19T09:23:03.394 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59279.log 2023-12-19T09:23:03.396 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56999.log 2023-12-19T09:23:03.397 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59279.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59279.log.gz 2023-12-19T09:23:03.399 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56999.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56999.log.gz 2023-12-19T09:23:03.399 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52184.log 2023-12-19T09:23:03.401 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60494.log 2023-12-19T09:23:03.403 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52184.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52184.log.gz 2023-12-19T09:23:03.403 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43709.log 2023-12-19T09:23:03.404 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60494.log.gz 2023-12-19T09:23:03.405 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51964.log 2023-12-19T09:23:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43709.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43709.log.gz 2023-12-19T09:23:03.407 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2023-12-19T09:23:03.409 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51964.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51964.log.gz 2023-12-19T09:23:03.409 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53757.log 2023-12-19T09:23:03.410 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41276.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2023-12-19T09:23:03.412 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53757.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53757.log.gz 2023-12-19T09:23:03.412 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48922.log 2023-12-19T09:23:03.414 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46869.log 2023-12-19T09:23:03.416 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48922.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48922.log.gz 2023-12-19T09:23:03.416 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46869.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46869.log.gz 2023-12-19T09:23:03.417 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46529.log 2023-12-19T09:23:03.418 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2023-12-19T09:23:03.420 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46529.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46529.log.gz 2023-12-19T09:23:03.421 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50189.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2023-12-19T09:23:03.422 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54101.log 2023-12-19T09:23:03.423 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2023-12-19T09:23:03.425 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54101.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54101.log.gz 2023-12-19T09:23:03.426 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46815.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2023-12-19T09:23:03.426 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2023-12-19T09:23:03.428 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42266.log 2023-12-19T09:23:03.430 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54186.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2023-12-19T09:23:03.431 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42266.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42266.log.gz 2023-12-19T09:23:03.431 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58113.log 2023-12-19T09:23:03.433 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44176.log 2023-12-19T09:23:03.434 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58113.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58113.log.gz 2023-12-19T09:23:03.435 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58630.log 2023-12-19T09:23:03.436 INFO:teuthology.orchestra.run.smithi111.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44176.log.gz 2023-12-19T09:23:03.437 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48349.log 2023-12-19T09:23:03.440 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58630.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58630.log.gz 2023-12-19T09:23:03.440 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48349.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48349.log.gz 2023-12-19T09:23:03.440 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52948.log 2023-12-19T09:23:03.442 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58911.log 2023-12-19T09:23:03.443 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52948.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52948.log.gz 2023-12-19T09:23:03.443 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59171.log 2023-12-19T09:23:03.446 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58911.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58911.log.gz 2023-12-19T09:23:03.446 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-19T09:23:03.446 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59171.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.59171.log.gz 2023-12-19T09:23:03.448 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.2.log: 94.5% -- replaced with /var/log/ceph/ceph-client.0.61451.log.gz 2023-12-19T09:23:03.451 INFO:teuthology.orchestra.run.smithi111.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.0.43819.log.gz 2023-12-19T09:23:03.451 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53157.log 2023-12-19T09:23:03.454 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56727.log 2023-12-19T09:23:03.455 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53157.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53157.log.gz 2023-12-19T09:23:03.457 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2023-12-19T09:23:03.458 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57318.log 2023-12-19T09:23:03.458 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56727.log.gz 2023-12-19T09:23:03.459 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64023.log 2023-12-19T09:23:03.459 INFO:teuthology.orchestra.run.smithi111.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2023-12-19T09:23:03.461 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42349.log 2023-12-19T09:23:03.461 INFO:teuthology.orchestra.run.smithi111.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57318.log.gz 2023-12-19T09:23:03.462 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64672.log 2023-12-19T09:23:03.462 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64023.log.gz 2023-12-19T09:23:03.464 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59115.log 2023-12-19T09:23:03.464 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42349.log.gz 2023-12-19T09:23:03.465 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.64672.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64672.log.gz 2023-12-19T09:23:03.465 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54585.log 2023-12-19T09:23:03.466 INFO:teuthology.orchestra.run.smithi143.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-12-19T09:23:03.467 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55983.log 2023-12-19T09:23:03.467 INFO:teuthology.orchestra.run.smithi111.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59115.log.gz 2023-12-19T09:23:03.468 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54585.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54585.log.gz 2023-12-19T09:23:03.468 INFO:teuthology.orchestra.run.smithi111.stderr:--verbose -- /var/log/ceph/ceph-client.admin.50554.log 2023-12-19T09:23:03.469 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55983.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55983.log.gz 2023-12-19T09:23:03.470 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54237.log 2023-12-19T09:23:03.471 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49787.log 2023-12-19T09:23:03.472 INFO:teuthology.orchestra.run.smithi111.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50554.log.gz 2023-12-19T09:23:03.472 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54237.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54237.log.gz 2023-12-19T09:23:03.473 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2023-12-19T09:23:03.474 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49787.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49787.log.gz 2023-12-19T09:23:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41060.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2023-12-19T09:23:04.140 INFO:teuthology.orchestra.run.smithi143.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-19T09:23:04.162 INFO:teuthology.orchestra.run.smithi143.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-19T09:23:04.464 INFO:teuthology.orchestra.run.smithi143.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-19T09:23:05.444 INFO:teuthology.orchestra.run.smithi111.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-19T09:23:06.513 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-19T09:23:06.730 INFO:teuthology.orchestra.run.smithi143.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-19T09:23:07.071 INFO:teuthology.orchestra.run.smithi143.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-19T09:23:07.444 INFO:teuthology.orchestra.run.smithi143.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-19T09:23:07.723 INFO:teuthology.orchestra.run.smithi111.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-19T09:23:07.731 INFO:teuthology.orchestra.run.smithi143.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-19T09:23:07.735 INFO:teuthology.orchestra.run.smithi143.stderr: 2023-12-19T09:23:07.735 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m5.408s 2023-12-19T09:23:07.735 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m24.336s 2023-12-19T09:23:07.735 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m1.970s 2023-12-19T09:23:07.867 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-19T09:23:09.127 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-19T09:23:09.131 INFO:teuthology.orchestra.run.smithi111.stderr: 2023-12-19T09:23:09.131 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m6.842s 2023-12-19T09:23:09.131 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m22.091s 2023-12-19T09:23:09.131 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m3.408s 2023-12-19T09:23:09.132 INFO:tasks.ceph:Archiving logs... 2023-12-19T09:23:09.132 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi111/log 2023-12-19T09:23:09.133 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-19T09:23:12.989 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi143/log 2023-12-19T09:23:12.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-19T09:23:16.126 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-19T09:23:16.136 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-19T09:23:16.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T09:23:16.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T09:23:16.226 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-19T09:23:16.236 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-19T09:23:16.236 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T09:23:16.244 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout:*hv01.front.sepi 44.4.53.2 2 u 5 128 377 0.231 -0.496 0.056 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout:+hv02.front.sepi 63.231.80.2 3 u 72 128 377 0.148 0.480 0.232 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout:+hv03.front.sepi 74.6.168.72 3 u 122 128 377 0.218 0.601 0.126 2023-12-19T09:23:16.279 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout:*hv01.front.sepi 44.4.53.2 2 u 70 128 377 0.231 -1.139 0.052 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout:+hv02.front.sepi 63.231.80.2 3 u 72 128 377 0.185 -0.328 0.073 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout:+hv03.front.sepi 74.6.168.72 3 u 50 128 377 0.217 -0.163 0.032 2023-12-19T09:23:16.288 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-19T09:23:16.290 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-19T09:23:16.300 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-19T09:23:16.300 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-19T09:23:16.309 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-19T09:23:16.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-19T09:23:16.328 INFO:teuthology.task.internal:Duration was 6529.140560 seconds 2023-12-19T09:23:16.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-19T09:23:16.345 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-19T09:23:16.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T09:23:16.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T09:23:16.510 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-19T09:23:16.510 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-12-19T09:23:16.510 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-19T09:23:16.542 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2023-12-19T09:23:16.542 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-19T09:23:16.580 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-19T09:23:16.580 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-19T09:23:16.593 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-19T09:23:16.684 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-19T09:23:16.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-19T09:23:16.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-19T09:23:16.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-19T09:23:16.820 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-19T09:23:16.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T09:23:16.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T09:23:16.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-19T09:23:16.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T09:23:16.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T09:23:16.942 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-12-19T09:23:16.977 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2023-12-19T09:23:17.012 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-19T09:23:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:23:17.037 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-19T09:23:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T09:23:17.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-19T09:23:17.086 INFO:teuthology.task.internal:Transferring archived files... 2023-12-19T09:23:17.087 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi111 2023-12-19T09:23:17.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T09:23:17.131 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886/remote/smithi143 2023-12-19T09:23:17.132 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T09:23:17.175 INFO:teuthology.task.internal:Removing archive directory... 2023-12-19T09:23:17.176 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-19T09:23:17.183 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-19T09:23:17.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-19T09:23:17.250 INFO:teuthology.task.internal:Not uploading archives. 2023-12-19T09:23:17.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-19T09:23:17.259 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-19T09:23:17.260 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T09:23:17.266 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T09:23:17.281 INFO:teuthology.orchestra.run.smithi111.stdout: 1048601 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 09:23 /home/ubuntu/cephtest 2023-12-19T09:23:17.304 INFO:teuthology.orchestra.run.smithi143.stdout: 1048601 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 09:23 /home/ubuntu/cephtest 2023-12-19T09:23:17.311 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-19T09:23:17.321 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-19T09:23:17.343 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T09:23:17.371 DEBUG:teuthology.nuke:shortname: smithi111 2023-12-19T09:23:17.371 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T09:23:17.391 DEBUG:teuthology.nuke:shortname: smithi143 2023-12-19T09:23:17.391 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T09:23:17.395 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 07:24:25.336278', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-19T09:23:17.416 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-12-19_05:13:44-fs-rishabh-mdscap-quincy-testing-default-smithi/7495886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 07:24:25.337776', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-19T09:23:17.444 INFO:teuthology.orchestra.console.smithi111:Power off 2023-12-19T09:23:17.444 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T09:23:17.468 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-12-19T09:23:17.469 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:17.487 INFO:teuthology.orchestra.console.smithi143:Power off 2023-12-19T09:23:17.487 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T09:23:17.498 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-19T09:23:17.512 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2023-12-19T09:23:17.512 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:17.540 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2023-12-19T09:23:21.500 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:21.540 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:21.611 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-19T09:23:21.651 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2023-12-19T09:23:25.613 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:25.652 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:25.724 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-12-19T09:23:25.766 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2023-12-19T09:23:25.825 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-12-19T09:23:29.769 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T09:23:29.879 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2023-12-19T09:23:29.980 INFO:teuthology.orchestra.console.smithi143:Power off completed 2023-12-19T09:23:30.088 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6529.140559673309 failure_reason: '"1702972183.1945295 mon.a (mon.0) 451 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default mon.a-kernel-sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c mon.b-kernel-sha1: cd68252d7b4d7802f3cae7f5945b38e9b20dad4c owner: scheduled_rishabh@teuthology success: false 2023-12-19T09:23:30.088 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T09:23:30.173 INFO:teuthology.run:FAIL